AWS EMR

How to increase disk space on master node root partition in EMR

Omid

I have tested the following solution on my side using the r4.4xlarge instance type

Please find the steps-

Step-1) Increase the root volume of the EMR master node

To navigate to the EMR cluster’s master node’s root EBS volume the following steps can be taken:

– Open the EMR cluster in the EMR console
– Expand the hardware dropdown
– Click on the Instance Group ID that is labelled as the MASTER
– Click on the EC2 instance ID shown in this table, which will open the master node in the EC2 console
– In the “Description” tab in the information panel at the bottom of the console, scroll down and click on the linked device for the “Root device” entry
– The EBS volume will now open in the EC2 console, this is the root EBS Volume for the master node, and should look like the screenshot below
– Now you should be able to choose the “Modify Volume” action from the “Actions” dropdown, and change the volume size!

In this case, I adjusted the size of the EBS volume from 10GB to 50GB, simple as that! Alternatively, you could give the customer the exact CLI command to do this rather than trying to guide them through this process on the console:

aws ec2 modify-volume –region us-east-1 –volume-id vol-xxxxxxxxxxxxxxxxxxxx –size 50 –volume-type gp2

Step-2) Login to the Master node with SSH and check run the following command to check the newly attached size information under /dev/xvda.

lsblk
df -h

Step-3) However, it is important to note that at this point you will still not see the additional space on the file system (root volume /dev/xvda1). Run the following command to add the following space to root volume.

sudo /usr/bin/cloud-init -d single -n growpart
sudo /usr/bin/cloud-init -d single -n resizefs

Step-4) Now run below command to see “/” volume is increased to 50GB.

df -h
lsblk

Step-5) After the volume is increased, you can do a test create a sample test file and see the size of “/” volume increased its usage.

sudo fallocate -l 10G /test

(This will create a test file inside “/” with 10GB in size)

df -h

(Verify root volume mount point increased its usage)

Step-6) After verifying, just delete sample /test file.

sudo rm -rf /test

Note: Please back up any important file or configuration files before performing the operation.

Want to get more content about big data?



——————————————————————————————————————————

I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with, feel free to contact me:

https://www.linkedin.com/in/omid-vahdaty/



Leave a Reply