This blog Assume prior knowledge, this to help the reader design training program to newbies on AWS. Naturally, my big data perspective is applied here.
Learn the following in rising order of importance (in my humble opinion).
General lightweight introduction to AWS & AWS Big Data :
- Create AWS user on AWS account
Start with this. get the obvious out of the way.
- AWS S3 (GCS), AWS S3 cli
Be sure to understand the basics, upload, download, copy, move, rsnc from both the GUI and AWS CLI. only then go to other advanced features such as life cycle policy, storage tiers, encyrptions etc.
- AWS EC2(Elastic Compute), how to create a machine, how to connect via ssh
Be sure to use T instance to play a round, choose amazon linux or ubuntu. Notice the different OS users name required to ssh to each machine.
Be sure to understand what is
- SSH
- SSH tunnel
- AWS security groups how to add ip and port
Without this section you wont be to access web/ssh machines.
- AWS VPC (virtual private network) , only if you feel comfortable around network architecture, otherwise skipt this topic.
- AWS RDS (mySQL,aurora)
create Mysql , login, create table, insert data from S3, export data from s3.
understand the difference between AWS RDS aurora and AWS RDS mysql
- AWS Redshift learn how to create cluster connect to cluster, and run query , understand the basic architecture.
- AWS Athena, how to create external table, how to insert data, partitions, MSCK repair table, parquet, AVRO, querying nested data.
- AWS Glue
be sure to understand the 2 role of AWS glue: shared metastore and auto ETL features.
- AWS Kinesis, Stream, Firehose, Analytics, you need to understand messaging and streaming. I covered off topic subject here such as flume and kafka.
- AWS EMR – (need to talk about with omid).
This is HARD CORE material. highly advanced materials for enterprise grade lectures.
Be sure to understand when to use Hive and when to use SparkSQL and when to use Spark Core. Moreover, the difference between the different node: core node, task node, master node.
- AWS IAM, groups, user, role based architecture, encryption at rest, at motion, based policy, identity based policy. By now you should have a basic understanding of what is IAM about. Identity, Authentication, authorization. although, there are many fine grain security issues you need to understand. At the very minimum be sure to understand what is the difference between a role and user, how to write a custom policy, and what is resource based policy vs identity based policy.
- AWS Cloud best practices, light wight
- AWS ELB, ALB, Auto scaling. [Advanced]
- AWS Route53, TBD
- AWS security
- AWS Lambda
- AWS Sage Maker
TBD subject:
API Gateway
AWS cognito.
——————————————————————————————————————————
I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with, feel free to contact me:
https://www.linkedin.com/in/omid-vahdaty/
