I need to provide access to persistent data or custom disk configurations across big data clusters. What options are available?

Amazon S3 is the AWS default storage option for big data and can be used for moving large amounts of data as a scalable, persistent option. EMR File System (EMRFS) and S3DistCp are available to assist with EMR workloads. However, there are other options to consider. For example, sometimes the 10 GiB root file system is too small.

EBS includes additional tiers of storage with retention prices comparable to S3, but with the added benefit of being mounted inside your Unix file system. This is ideal for adding:

  • Additional libraries, binaries, jars, code sets, etc.
  • Additional space for extra logging
  • Small reference data sets, especially if they need to be writeable

In this example, we create a /var/log2 directory with a 20 GiB partition and a directory in /MyCustomCode on a separate 25 GiB partition, and then we mount 25 GiB as /MyPersistentData. Using a separate logging directory instead of /var/log helps protect your system from logging overflow.

EBS volumes can be created and attached using either the AWS Command Line Interface (CLI) or the AWS Management Console. In this sample code we use the EMR master instance, but it could also be done from the slave nodes.

Note: You can use these volumes only for non-HDFS persistent storage because HDFS does not automatically increase storage size. For more information, see Amazon EMR Update – Support for EBS Volumes, and M4 & C4 Instance Types.

Create the volume:

$aws ec2 create-volume --size 100 --volume-type gp2 --availability-zone us-east-1a

Example return:

{

    "AvailabilityZone": "us-east-1a",

    "Encrypted": false,

    "VolumeType": "gp2",

    "VolumeId": "vol-5XaXXaf1",

    "State": "creating",

    "Iops": 300,

    "SnapshotId": "",

    "CreateTime": "2016-07-31T19:14:18.441Z",

    "Size": 100

}

Attach the volume:

$aws ec2 attach-volume --volume-id vol-5XaXXaf1 --instance-id i-0babe6963131e17b6 --device /dev/xvdg

{

    "AttachTime": "2016-07-31T19:22:25.703Z",

    "InstanceId": "i-0babe6963131e17b6",

    "VolumeId": "vol-5XaXXaf1",

    "State": "attaching",

    "Device": "/dev/xvdg"

}

On the master node, run fdisk to see the device:

$sudo fdisk -l

Example return:

Disk /dev/xvdg: 107.4 GB, 107374182400 bytes, 209715200 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Repeat to create the partitions, labels, and file systems:

$sudo fdisk /dev/xvdg

Set up labels to allow the file systems to be independent of the device names:

$sudo fdisk /dev/xvdg

Setup labels to allow the file systems to be independent of the device names:

$sudo mkfs -t ext3 /dev/xvdg1

$sudo mkfs -t ext3 /dev/xvdg2

$sudo mkfs -t ext3 /dev/xvdg3

$sudo e2label /dev/xvdg1 logs

$sudo e2label /dev/xvdg3 CustomCode

$sudo e2label /dev/xvdg2 PersistentData

Attach and mount the new file systems (these commands run on the master node):

$sudo echo "LABEL=PersistentData    /MyPersistentData    ext3    defaults,nofail 0 0" >> /etc/fstab

 

$sudo echo "LABEL=CustomCode     /opt/MyCustomCode     ext3    defaults,nofail 0 0" >> /etc/fstab

 

$sudo echo "LABEL=logs         /var/log2                 ext3    defaults,nofail 0 0" >> /etc/fstab

 

$sudo mount -a

df -H displays:

/dev/xvdg1        20G    47M    19G    1% /var/log2

/dev/xvdg3        43G    51M    40G    1% /opt/MyCustomCode

/dev/xvdg2        27G    47M    25G    1% /MyPersistentData

VPC, endpoint, subnet, private


Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2016-12-05