I need to provide access to persistent data or custom disk configurations across big data clusters. What options are available?
Amazon S3 is the AWS default storage option for big data and can be used for moving large amounts of data as a scalable, persistent option. EMR File System (EMRFS) and S3DistCp are available to assist with EMR workloads. However, there are other options to consider. For example, sometimes the 10 GiB root file system is too small.
EBS includes additional tiers of storage with retention prices comparable to S3, but with the added benefit of being mounted inside your Unix file system. This is ideal for adding:
- Additional libraries, binaries, jars, code sets, etc.
- Additional space for extra logging
- Small reference data sets, especially if they need to be writeable
In this example, we create a /var/log2 directory with a 20 GiB partition and a directory in /MyCustomCode on a separate 25 GiB partition, and then we mount 25 GiB as /MyPersistentData. Using a separate logging directory instead of /var/log helps protect your system from logging overflow.
EBS volumes can be created and attached using either the AWS Command Line Interface (CLI) or the AWS Management Console. In this sample code we use the EMR master instance, but it could also be done from the slave nodes.
Note: You can use these volumes only for non-HDFS persistent storage because HDFS does not automatically increase storage size. For more information, see Amazon EMR Update – Support for EBS Volumes, and M4 & C4 Instance Types.
Create the volume:
$aws ec2 create-volume --size 100 --volume-type gp2 --availability-zone us-east-1a
Example return:
{
"AvailabilityZone": "us-east-1a",
"Encrypted": false,
"VolumeType": "gp2",
"VolumeId": "vol-5XaXXaf1",
"State": "creating",
"Iops": 300,
"SnapshotId": "",
"CreateTime": "2016-07-31T19:14:18.441Z",
"Size": 100
}
Attach the volume:
$aws ec2 attach-volume --volume-id vol-5XaXXaf1 --instance-id i-0babe6963131e17b6 --device /dev/xvdg
{
"AttachTime": "2016-07-31T19:22:25.703Z",
"InstanceId": "i-0babe6963131e17b6",
"VolumeId": "vol-5XaXXaf1",
"State": "attaching",
"Device": "/dev/xvdg"
}
On the master node, run fdisk to see the device:
$sudo fdisk -l
Example return:
Disk /dev/xvdg: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Repeat to create the partitions, labels, and file systems:
$sudo fdisk /dev/xvdg
Set up labels to allow the file systems to be independent of the device names:
$sudo fdisk /dev/xvdg
Setup labels to allow the file systems to be independent of the device names:
$sudo mkfs -t ext3 /dev/xvdg1
$sudo mkfs -t ext3 /dev/xvdg2
$sudo mkfs -t ext3 /dev/xvdg3
$sudo e2label /dev/xvdg1 logs
$sudo e2label /dev/xvdg3 CustomCode
$sudo e2label /dev/xvdg2 PersistentData
Attach and mount the new file systems (these commands run on the master node):
$sudo echo "LABEL=PersistentData /MyPersistentData ext3 defaults,nofail 0 0" >> /etc/fstab
$sudo echo "LABEL=CustomCode /opt/MyCustomCode ext3 defaults,nofail 0 0" >> /etc/fstab
$sudo echo "LABEL=logs /var/log2 ext3 defaults,nofail 0 0" >> /etc/fstab
$sudo mount -a
df -H displays:
/dev/xvdg1 20G 47M 19G 1% /var/log2
/dev/xvdg3 43G 51M 40G 1% /opt/MyCustomCode
/dev/xvdg2 27G 47M 25G 1% /MyPersistentData
Note: For additional information, see Making an Amazon EBS Volume Available for Use and Create Bootstrap Actions to Install Additional Software.
VPC, endpoint, subnet, private
Did this page help you? Yes | No
Back to the AWS Support Knowledge Center
Need help? Visit the AWS Support Center
Published: 2016-12-05