Ctrlk
Contact UsBook a DemoGo to App
For the complete documentation index, see llms.txt. This page is also available as Markdown.

My First Autoscaling Volume

A hands-on tutorial that walks you through installing AutoScaler, activating autoscaling, and watching your volumes grow and shrink automatically.

In this tutorial, we'll go from zero to a fully autoscaling environment together. By the end, you'll have:

  • Installed the Datafy AutoScaler agent on an EC2 instance

  • Activated autoscaling on volumes both manually and automatically via an autoscaling rule

  • Watched AutoScaler automatically grow a volume when you write data

  • Watched AutoScaler automatically shrink a volume when you delete data

The whole process takes about 30 minutes of active work, plus some waiting time. Let's get started!

To complete this tutorial you will need:

  • A Datafy account set up with configured permissions and an API token

  • An AWS account with permissions to create EC2 instances and EBS volumes.

  • SSH access to EC2 instances in your environment.

Set Up Your Environment

First, we'll create an EC2 instance with two EBS volumes, tag them, set up filesystems, and write some test data. You can do this in one of three ways:

  1. Automatically launch and set up an instance and 2 volumes using a CloudFormation template.

  2. Manually launch an instance with a user data script that sets up the volumes and installs docker on the instance.

  3. Manually set up the instance and volumes.

This CloudFormation template creates an EC2 instance with two tagged 150 GiB gp3 volumes, and automatically formats, mounts, and populates them with test data on first boot.

You'll need an existing VPC, subnet, security group, and key pair.

Create a CloudFormation stack with new resources, and upload the template below.

Tutorial-CF-Template.yaml
4KB
Open

The template applies the tags Name: manual_activation and Name: auto_activation to the volumes for you.

Writing the test data takes about 15 minutes for both volumes. Once the resources are created, access the instance via SSH and track the progress of the setup with:

tail /var/log/cloud-init-output.log

When it's finished you'll see a confirmation for each volume:

/mnt/vol1: verified (45G used)
/mnt/vol2: verified (45G used)
Volume setup complete.

Launch a c5.xlarge EC2 instance using the latest Amazon Linux 2023 AMI.

During the launch setup, configure:

  • Create and attach two additional gp3 EBS volumes of 150 GiB each.

  • Insert the following script into the user data of the instance. The script formats, mounts, and writes data to the volumes, and installs Docker on the instance.

#!/bin/bash
set -e

# Install Docker (required for AutoScaler)
yum update -y
yum install -y docker
service docker start
# Wait for both volumes to be attached
echo "Waiting for volumes to attach..."
while [ $(lsblk -dn -o NAME,SIZE | grep "150G" | wc -l) -lt 2 ]; do
  sleep 5
done
# Identify the two 150 GiB volumes (excluding root)
VOLUMES=($(lsblk -dn -o NAME,SIZE | grep "150G" | awk '{print "/dev/"$1}'))
# Step 1: Format and mount all volumes
for i in 0 1; do
  VOL=${VOLUMES[$i]}
  MNT="/mnt/vol$(($i + 1))"
  mkfs.ext4 -q "$VOL"
  mkdir -p "$MNT"
  mount "$VOL" "$MNT"
  echo "$(blkid -s UUID -o value $VOL) $MNT ext4 defaults,nofail 0 2" >> /etc/fstab
done
# Step 2: Write test data to all volumes with verification
EXPECTED_KB=$((45 * 1024 * 1024))  # 45 GiB in KiB
THRESHOLD_KB=$((44 * 1024 * 1024)) # Allow minor variance
for i in 1 2; do
  MNT="/mnt/vol${i}"
  echo "Writing test data to ${MNT}..."
  for attempt in 1 2; do
    echo "${MNT}: writing file1.bin (30 GiB)..."
    dd if=/dev/urandom of=${MNT}/file1.bin bs=1M count=30720 status=progress
    # 15 GiB minus 1 MiB, so df -h rounds to 45G
    echo "${MNT}: writing file2.bin..."
    dd if=/dev/urandom of=${MNT}/file2.bin bs=1M count=15359 status=progress

    USED_KB=$(df --output=used "$MNT" | tail -1 | tr -d ' ')
    if [ "$USED_KB" -ge "$THRESHOLD_KB" ]; then
      echo "${MNT}: verified ($(df -h --output=used $MNT | tail -1 | tr -d ' ') used)"
      break
    fi

    if [ "$attempt" -eq 2 ]; then
      echo "WARNING: ${MNT} has less data than expected after retry"
    else
      echo "${MNT}: incomplete write, retrying..."
      rm -f ${MNT}/file1.bin ${MNT}/file2.bin
    fi
  done
done
echo "Volume setup complete."

After the instance launches, add a Name tag to each of the volumes in the AWS Console:

  • 1 volume → Name: manual_activation

  • 1 volume → Name: auto_activation

These tags will help us identify the volumes in the Datafy App later, and we'll use them to create an autoscaling rule.

Launch the Instance and Volumes

Launch a c5.xlarge EC2 instance using the latest Amazon Linux 2023 AMI. Create two additional gp3 EBS volumes of 150 GiB each in the same Availability Zone, and attach them to the instance.

Make sure both volumes are in the same Availability Zone as your instance. You can create and attach them during instance launch or afterward through the AWS Console.

Tag the Volumes

In the AWS Console, add a Name tag to each of the two volumes:

  • 1 volume → Name: manual_activation

  • 1 volume → Name: auto_activation

These tags will help us identify the volumes in the Datafy App later, and we'll use them to create an autoscaling rule.

Mount and Prepare the Volumes

  1. Access your instance via SSH and run the following commands to format and mount both volumes.

  2. Next, write 45 GiB of test data to each volume:

Once the setup completes, verify that each volume has roughly 45 GiB of data:

The size reported by the filesystem is always slightly smaller than the volume's 150 GiB - the difference is formatting overhead.

Autoscale Your Volumes

With the environment in place, we'll install the agent and then activate autoscaling for both volumes - one manually, and one with a rule.

1

Install AutoScaler

Now let's install the Datafy AutoScaler agent! For the full installation instructions, see Installation.

AutoScaler requires Docker to run. If your instance doesn't already have Docker installed, run the following commands to install it:

If you set up your instance with the CloudFormation template or user data script, Docker is already installed.

Define the version you want to install and your Datafy API token. Replace <VERSION> with the agent version you're installing, and <YOUR_TOKEN> with the token you generated:

Install AutoScaler with the command:

The installation takes about a minute. Once it completes, the agent starts running automatically and begins reporting the volumes' utilization.

The last line of the output confirms a successful installation:

Let's head over to the Datafy app to see the instance and volumes.

Find Your Volumes in the Fleet Manager

  1. Open the Fleet Manager in the Datafy app.

  2. In the All Volumes table, filter by instance ID to show only the volumes on our tutorial instance.

  3. You should see your two 150 GiB volumes listed, with the Name tags you assigned earlier (manual_activation and auto_activation).

All Volumes table filtered to the two tutorial volumes, each 150 GiB

Before installing the agent, these volumes would have had an inactive Autoscaling toggle showing "No AutoScaler" as the reason. Now that the agent is installed, the Autoscaling toggle is active and available - ready for you to turn on.

You can also use the views at the top left of the All Volumes table to filter all the "Volumes to Activate". This is a useful way to find volumes that are ready to be activated.

2

Activate Autoscaling

Now for the main event: we'll activate autoscaling in two ways - manually on one volume, and with a rule on the other.

Activate Manually

Let's activate autoscaling on the manual_activation volume through the UI:

  1. In the All Volumes table, find the volume tagged manual_activation.

  2. Click the AutoScale toggle to turn it on. The progress of the activation is displayed next to the toggle.

    The AutoScale toggle turned on, with activation progress beside it

  3. Select the volume ID in the All Volumes table to navigate to the volume details page. Here you can see the new smaller volumes that will replace your original volume, and the usage and utilization of the volume over time.

Volume details page showing the replacement volumes and the usage chart

The initial activation copies all of the data on the volume, and should take about 15 minutes. Volumes with more data, or with active reads/writes to the volume will take longer, see Autoscaling Activation for more details.

To find the mount point of each volume, use the following - we'll call yours /mnt/volN:

During and after the activation, your files remain the same, and the filesystem continues to report and behave like it did before:

For more information check out how autoscaling works, and how to monitor activation actions.

Activate with an Autoscaling Rule

While the first volume completes its activation, let's create a rule that automatically activates autoscaling on volumes matching the auto_activation tag. This is how you'd manage autoscaling at scale in a production environment - you tag volumes, and Datafy activates them for you.

  1. In the Fleet Manager, navigate to the Autoscaling Rules section.

  2. Click Create New Rule to open the rule creation wizard.

  3. Add a condition "Volume tag is Name: auto_activation".

    Rule wizard with a volume tag condition for Name: auto_activation

  4. Verify that the volumes you expect match the rule

    Rule wizard listing the matching auto_activation volume

  5. Select Create and Apply to activate the rule

The rule runs immediately upon creation. Head back to the All Volumes table - you'll see the AutoScale toggle is now on for the auto_activation volume too.

Autoscaling is activated for one volume at a time per instance, so this volume waits its turn: it shows as pending until the manual activation finishes.

Once both activations complete, both of your tutorial volumes are managed by Datafy. 🎉

3

Grow

Let's see autoscaling in action. We'll write additional data to one of the volumes and watch Datafy automatically grow it to accommodate the new usage.

Write another 15 GiB file to the volume you activated manually - /mnt/volN from the previous step:

Now open the Datafy UI and watch the volume. As the usage increases and crosses the growth threshold, Datafy will automatically trigger a grow operation. You'll see:

  • The volume's utilization increase in the Fleet Manager

  • As the used data approaches the current size of the volume you'll see the size increase to accommodate the new data

The grow happens seamlessly — no downtime, no interruption to your applications. Once it completes, the volume returns to its normal autoscaling state with the additional capacity in place. The whole operation takes seconds: AutoScaler expands the underlying volumes in place, without copying any data.

For more on how grow thresholds work, see Growing.

4

Shrink

Now let's see the other side of autoscaling: what happens when you no longer need all that space.

Delete the 30 GiB file you wrote during setup, on the same volume:

The shrink won't happen immediately. Two things have to clear first:

  • The cooldown - Datafy waits after any autoscaling operation to be sure the freed space is really no longer needed, rather than churning on a workload that dips and spikes. Depending on your account configuration this can be up to an hour, counted from the grow you just triggered.

  • Any activation still running on the instance - a shrink won't start while another volume on the same instance is still being activated.

After the cooldown period, AutoScaler detects the reduced usage and triggers a shrink operation. In the Datafy app, you'll see:

  • A "Shrinking" action will appear in the Optimization Actions table in the Reports page and on the details page of the relevant volume

  • Once the shrink is completed, you'll see the size of the volume drop to about 55 GiB - from the 150 GiB you provisioned.

Cleanup

When you're done exploring, there are two ways to tear the tutorial environment down:

  • Full teardown - deactivate autoscaling, uninstall the agent, then remove the AWS resources. This is the sequence to follow in production, so it's the one to walk through if you want to practice it, or if you want to keep the volumes and their data afterward.

  • Delete everything - terminate the instance and delete the volumes directly. Faster, and safe here because nothing needs to survive.

Either way, start by deleting the rule so it can't activate autoscaling on anything else in your account: in the Autoscaling Rules section of the Fleet Manager, select the rule you created and delete it.

1

Deactivate Autoscaling on Both Volumes

In the All Volumes table, turn off the AutoScale toggle for both volumes, and wait for both to finish. Datafy copies your data back to standard EBS volumes at the original 150 GiB size, and deletes the volumes it was managing.

Each restored volume gets a new AWS EBS volume ID, but keeps the original volume's tags - so you can still find them by their manual_activation and auto_activation tags in the AWS Console.

2

Uninstall the Agent

On the instance, uninstall the Datafy agent, using the same version you installed:

3

Remove the AWS Resources

Terminate the EC2 instance, then delete the two restored EBS volumes.

Because the instance and the data are both being destroyed, you can skip deactivation and the uninstall. Deactivating autoscaling exists to return your data to a volume that's readable without AutoScaler, which is not needed if it is going to be deleted.

  1. Terminate the EC2 instance.

  2. Delete the EBS volumes left behind in the AWS Console - there will be four, since each autoscaling volume sits on a pair.

Skipping deactivation is fine here only because everything is being deleted. On a real environment, always deactivate autoscaling before uninstalling AutoScaler - otherwise you lose access to the data on volumes you still need.

For more on uninstalling, see Uninstalling Datafy.

What's Next?

Now that you've seen autoscaling in action, here are some next steps:

PreviousDeployment Examples
NextHow AutoScaler Works

Last updated

Was this helpful?