> For the complete documentation index, see [llms.txt](https://docs.datafy.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datafy.io/resources/my-first-autoscaling-volume.md). # My First Autoscaling Volume In this tutorial, we'll go from zero to a fully autoscaling environment together. By the end, you'll have: * Installed the Datafy AutoScaler agent on an EC2 instance * Activated autoscaling on volumes both manually and automatically via an autoscaling rule * Watched AutoScaler automatically grow a volume when you write data * Watched AutoScaler automatically shrink a volume when you delete data The whole process takes about 30 minutes of active work, plus some waiting time. Let's get started! {% hint style="success" %} To complete this tutorial you will need: * A **Datafy account** set up with [configured permissions](/set-up-and-installation/datafy-installation/permissions-configuration.md) and an [API token](/set-up-and-installation/datafy-installation/token-generation.md) * An **AWS account** with permissions to create EC2 instances and EBS volumes. * **SSH access** to EC2 instances in your environment. {% endhint %} ## Set Up Your Environment First, we'll create an EC2 instance with two EBS volumes, tag them, set up filesystems, and write some test data. You can do this in one of three ways: 1. Automatically launch and set up an instance and 2 volumes using a CloudFormation template. 2. Manually launch an instance with a user data script that sets up the volumes and installs docker on the instance. 3. Manually set up the instance and volumes. {% tabs %} {% tab title="CloudFormation" %} This CloudFormation template creates an EC2 instance with two tagged 150 GiB gp3 volumes, and automatically formats, mounts, and populates them with test data on first boot. You'll need an existing VPC, subnet, security group, and key pair. Create a CloudFormation stack with new resources, and upload the template below. {% file src="/files/jGd5njC2NWS4hjyDbRVJ" %} The template applies the tags `Name: manual_activation` and `Name: auto_activation` to the volumes for you. Writing the test data takes about 15 minutes for both volumes. Once the resources are created, access the instance via SSH and track the progress of the setup with: {% code overflow="wrap" %} ```bash tail /var/log/cloud-init-output.log ``` {% endcode %} When it's finished you'll see a confirmation for each volume: ``` /mnt/vol1: verified (45G used) /mnt/vol2: verified (45G used) Volume setup complete. ``` {% endtab %} {% tab title="Setup with User Data" %} Launch a **c5.xlarge** EC2 instance using the latest **Amazon Linux 2023** AMI. During the launch setup, configure: * Create and attach **two additional gp3 EBS volumes** of **150 GiB** each. * Insert the following script into the *user data* of the instance. The script formats, mounts, and writes data to the volumes, and installs Docker on the instance. ```bash #!/bin/bash set -e # Install Docker (required for AutoScaler) yum update -y yum install -y docker service docker start # Wait for both volumes to be attached echo "Waiting for volumes to attach..." while [ $(lsblk -dn -o NAME,SIZE | grep "150G" | wc -l) -lt 2 ]; do sleep 5 done # Identify the two 150 GiB volumes (excluding root) VOLUMES=($(lsblk -dn -o NAME,SIZE | grep "150G" | awk '{print "/dev/"$1}')) # Step 1: Format and mount all volumes for i in 0 1; do VOL=${VOLUMES[$i]} MNT="/mnt/vol$(($i + 1))" mkfs.ext4 -q "$VOL" mkdir -p "$MNT" mount "$VOL" "$MNT" echo "$(blkid -s UUID -o value $VOL) $MNT ext4 defaults,nofail 0 2" >> /etc/fstab done # Step 2: Write test data to all volumes with verification EXPECTED_KB=$((45 * 1024 * 1024)) # 45 GiB in KiB THRESHOLD_KB=$((44 * 1024 * 1024)) # Allow minor variance for i in 1 2; do MNT="/mnt/vol${i}" echo "Writing test data to ${MNT}..." for attempt in 1 2; do echo "${MNT}: writing file1.bin (30 GiB)..." dd if=/dev/urandom of=${MNT}/file1.bin bs=1M count=30720 status=progress # 15 GiB minus 1 MiB, so df -h rounds to 45G echo "${MNT}: writing file2.bin..." dd if=/dev/urandom of=${MNT}/file2.bin bs=1M count=15359 status=progress USED_KB=$(df --output=used "$MNT" | tail -1 | tr -d ' ') if [ "$USED_KB" -ge "$THRESHOLD_KB" ]; then echo "${MNT}: verified ($(df -h --output=used $MNT | tail -1 | tr -d ' ') used)" break fi if [ "$attempt" -eq 2 ]; then echo "WARNING: ${MNT} has less data than expected after retry" else echo "${MNT}: incomplete write, retrying..." rm -f ${MNT}/file1.bin ${MNT}/file2.bin fi done done echo "Volume setup complete." ``` After the instance launches, add a Name tag to each of the volumes in the AWS Console: * **1 volume** → `Name: manual_activation` * **1 volume** → `Name: auto_activation` These tags will help us identify the volumes in the Datafy App later, and we'll use them to create an autoscaling rule. {% endtab %} {% tab title="Manual Setup" %} #### Launch the Instance and Volumes Launch a **c5.xlarge** EC2 instance using the latest **Amazon Linux 2023** AMI. Create **two additional gp3 EBS volumes** of **150 GiB** each in the same Availability Zone, and attach them to the instance. {% hint style="info" %} Make sure both volumes are in the same Availability Zone as your instance. You can create and attach them during instance launch or afterward through the AWS Console. {% endhint %} #### Tag the Volumes In the AWS Console, add a `Name` tag to each of the two volumes: * **1 volume** → `Name: manual_activation` * **1 volume** → `Name: auto_activation` These tags will help us identify the volumes in the Datafy App later, and we'll use them to create an autoscaling rule. #### Mount and Prepare the Volumes 1. Access your instance via SSH and run the following commands to format and mount both volumes. 
# The volumes will typically appear as /dev/nvme1n1 and /dev/nvme2n1
   # Adjust the device names below if yours are different

   # Create ext4 filesystems on both volumes
   sudo mkfs.ext4 /dev/nvme1n1
   sudo mkfs.ext4 /dev/nvme2n1
   # Create mount points
   sudo mkdir -p /mnt/vol1 /mnt/vol2
   # Mount the volumes
   sudo mount /dev/nvme1n1 /mnt/vol1
   sudo mount /dev/nvme2n1 /mnt/vol2
   # Add to fstab for persistence across reboots
   echo "$(blkid -s UUID -o value /dev/nvme1n1) /mnt/vol1 ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
   echo "$(blkid -s UUID -o value /dev/nvme2n1) /mnt/vol2 ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
2. Next, write 45 GiB of test data to each volume: ```bash # Write test data to both volumes - 45 GiB each, less 1 MiB, so df -h rounds to 45G # This will take a few minutes per volume for vol in /mnt/vol1 /mnt/vol2; do sudo dd if=/dev/urandom of=${vol}/file1.bin bs=1M count=30720 status=progress sudo dd if=/dev/urandom of=${vol}/file2.bin bs=1M count=15359 status=progress done ``` {% endtab %} {% endtabs %} {% hint style="success" %} Once the setup completes, verify that each volume has roughly 45 GiB of data: ```bash df -h /mnt/vol1 /mnt/vol2 ``` ``` Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1 147G 45G 95G 33% /mnt/vol1 /dev/nvme2n1 147G 45G 95G 33% /mnt/vol2 ``` {% endhint %} {% hint style="info" %} The size reported by the filesystem is always slightly smaller than the volume's 150 GiB - the difference is formatting overhead. {% endhint %} ## Autoscale Your Volumes With the environment in place, we'll install the agent and then activate autoscaling for both volumes - one manually, and one with a rule. {% stepper %} {% step %} ### Install AutoScaler Now let's install the Datafy AutoScaler agent! For the full installation instructions, see [Installation](/set-up-and-installation/datafy-installation/installation.md). {% hint style="info" %} AutoScaler requires Docker to run. If your instance doesn't already have Docker installed, run the following commands to install it: ```bash sudo yum update -y sudo yum install -y docker sudo service docker start ``` If you set up your instance with the CloudFormation template or user data script, Docker is already installed. {% endhint %} Define the version you want to install and your Datafy API token. Replace `` with the agent version you're installing, and `` with the token you generated: {% code overflow="wrap" expandable="true" %} ```bash VERSION="" TOKEN="" ``` {% endcode %} Install AutoScaler with the command: {% code overflow="wrap" expandable="true" %} ```bash curl -sSfL https://agent.datafy.io/install?version=$VERSION | AGENT_MODE="autoscaler" TOKEN=$TOKEN sh ``` {% endcode %} The installation takes about a minute. Once it completes, the agent starts running automatically and begins reporting the volumes' utilization. {% hint style="success" %} The last line of the output confirms a successful installation: ``` Datafy Agent 1.40.0 was successfully installed 🎉 ``` {% endhint %} Let's head over to the [**Datafy app**](https://app.datafy.io) to see the instance and volumes. #### Find Your Volumes in the Fleet Manager 1. Open the [Fleet Manager](https://app.datafy.io/) in the Datafy app. 2. In the [All Volumes](https://app.datafy.io/fleet-manager/all-volumes) table, filter by instance ID to show only the volumes on our tutorial instance. 3. You should see your two 150 GiB volumes listed, with the `Name` tags you assigned earlier (`manual_activation` and `auto_activation`).
All Volumes table filtered to the two tutorial volumes, each 150 GiB
{% hint style="success" %} Before installing the agent, these volumes would have had an inactive Autoscaling toggle showing **"No AutoScaler"** as the reason. Now that the agent is installed, the Autoscaling toggle is **active and available** - ready for you to turn on. {% endhint %} {% hint style="info" %} You can also use the views at the top left of the All Volumes table to filter all the "[Volumes to Activate](/volume-lifecycle/managing-autoscaling-volumes.md#identifying-supported-volumes)". This is a useful way to find volumes that are ready to be activated. {% endhint %} {% endstep %} {% step %} ### Activate Autoscaling Now for the main event: we'll activate autoscaling in two ways - manually on one volume, and with a rule on the other. #### Activate Manually Let's [activate autoscaling](/volume-lifecycle/managing-autoscaling-volumes.md#activating-autoscaling) on the `manual_activation` volume through the UI: 1. In the [**All Volumes**](https://app.datafy.io/fleet-manager/all-volumes) table, find the volume tagged `manual_activation`. 2. Click the **AutoScale** toggle to turn it on. The progress of the activation is displayed next to the toggle.
The AutoScale toggle turned on, with activation progress beside it
3. Select the volume ID in the All Volumes table to navigate to the volume details page. Here you can see the new smaller volumes that will replace your original volume, and the usage and utilization of the volume over time.
Volume details page showing the replacement volumes and the usage chart
{% hint style="info" %} The initial activation copies all of the data on the volume, and should take about 15 minutes. Volumes with more data, or with active reads/writes to the volume will take longer, see [How AutoScaler Works](/how-it-works/how-autoscaler-works.md#autoscaling-activation) for more details. {% endhint %} {% hint style="info" %} To find the mount point of each volume, use the following - we'll call yours `/mnt/volN`: ```bash lsblk -o NAME,SIZE,SERIAL,MOUNTPOINT ``` {% endhint %} {% hint style="success" %} During and after the activation, your files remain the same, and the filesystem continues to report and behave like it did before: ```bash ls -lh /mnt/volN/ df -h /mnt/volN ``` ``` -rw-r--r--. 1 root root 30G Aug 11 13:15 file1.bin -rw-r--r--. 1 root root 15G Aug 11 13:17 file2.bin Filesystem Size Used Avail Use% Mounted on /dev/nvme2n1 147G 45G 95G 33% /mnt/volN ``` {% endhint %} For more information check out how [autoscaling](/how-it-works/how-autoscaler-works.md) works, and how to [monitor activation actions](/volume-lifecycle/managing-autoscaling-volumes.md#monitoring-autoscaling-activity). #### Activate with an Autoscaling Rule While the first volume completes its activation, let's create a [rule](/volume-lifecycle/autoscaling-rules.md) that automatically activates autoscaling on volumes matching the `auto_activation` tag. This is how you'd manage autoscaling at scale in a production environment - you tag volumes, and Datafy activates them for you. 1. In the Fleet Manager, navigate to the [**Autoscaling Rules**](https://app.datafy.io/fleet-manager/autoscale-rules) section. 2. Click **Create New Rule** to open the rule creation wizard. 3. Add a condition "Volume tag is `Name: auto_activation`".
Rule wizard with a volume tag condition for Name: auto_activation
4. Verify that the volumes you expect match the rule
Rule wizard listing the matching auto_activation volume
5. Select **Create and Apply** to activate the rule The rule runs immediately upon creation. Head back to the All Volumes table - you'll see the AutoScale toggle is now on for the `auto_activation` volume too. {% hint style="info" %} Autoscaling is activated for [one volume at a time per instance](/volume-lifecycle/managing-autoscaling-volumes.md#autoscaling-multiple-volumes), so this volume waits its turn: it shows as pending until the manual activation finishes. {% endhint %} Once both activations complete, both of your tutorial volumes are managed by Datafy. 🎉 {% endstep %} {% step %} ### Grow Let's see autoscaling in action. We'll write additional data to one of the volumes and watch Datafy automatically grow it to accommodate the new usage. Write another 15 GiB file to the volume you activated manually - `/mnt/volN` from the previous step: ```bash # Write an additional 15 GiB file to trigger a grow sudo dd if=/dev/urandom of=/mnt/volN/file3.bin bs=1M count=15360 status=progress ``` Now open the Datafy UI and watch the volume. As the usage increases and crosses the growth threshold, Datafy will automatically trigger a grow operation. You'll see: * The volume's **utilization** increase in the Fleet Manager * As the used data approaches the current size of the volume you'll see the size increase to accommodate the new data The grow happens seamlessly — no downtime, no interruption to your applications. Once it completes, the volume returns to its normal autoscaling state with the additional capacity in place. The whole operation takes seconds: AutoScaler expands the underlying volumes in place, without copying any data. For more on how grow thresholds work, see [How AutoScaler Works](/how-it-works/how-autoscaler-works.md#growing). {% endstep %} {% step %} ### Shrink Now let's see the other side of autoscaling: what happens when you no longer need all that space. Delete the 30 GiB file you wrote during setup, on the same volume: ```bash sudo rm /mnt/volN/file1.bin ``` {% hint style="info" %} The shrink won't happen immediately. Two things have to clear first: * **The cooldown** - Datafy waits after any autoscaling operation to be sure the freed space is really no longer needed, rather than churning on a workload that dips and spikes. Depending on your [account configuration](/set-up-and-installation/autoscaling-configurations.md) this can be **up to an hour**, counted from the grow you just triggered. * **Any activation still running on the instance** - a shrink won't start while another volume on the same instance is still being activated. {% endhint %} After the cooldown period, AutoScaler detects the reduced usage and triggers a shrink operation. In the Datafy app, you'll see: * A "Shrinking" action will appear in the Optimization Actions table in the Reports page and on the details page of the relevant volume * Once the shrink is completed, you'll see the size of the volume drop to about 55 GiB - from the 150 GiB you provisioned. {% endstep %} {% endstepper %} ## Cleanup When you're done exploring, there are two ways to tear the tutorial environment down: * **Full teardown** - deactivate autoscaling, uninstall the agent, then remove the AWS resources. This is the sequence to follow in production, so it's the one to walk through if you want to practice it, or if you want to keep the volumes and their data afterward. * **Delete everything** - terminate the instance and delete the volumes directly. Faster, and safe here because nothing needs to survive. Either way, start by deleting the rule so it can't activate autoscaling on anything else in your account: in the [**Autoscaling Rules**](https://app.datafy.io/fleet-manager/autoscale-rules) section of the Fleet Manager, select the rule you created and delete it. {% tabs %} {% tab title="Full Teardown" %} {% stepper %} {% step %} #### Deactivate Autoscaling on Both Volumes In the [**All Volumes**](https://app.datafy.io/fleet-manager/all-volumes) table, turn off the **AutoScale** toggle for both volumes, and wait for both to finish. Datafy copies your data back to standard EBS volumes at the original 150 GiB size, and deletes the volumes it was managing. {% hint style="info" %} Each restored volume gets a **new AWS EBS volume ID**, but keeps the original volume's tags - so you can still find them by their `manual_activation` and `auto_activation` tags in the AWS Console. {% endhint %} {% endstep %} {% step %} #### Uninstall the Agent On the instance, uninstall the Datafy agent, using the same version you installed: {% code overflow="wrap" %} ```bash curl -sSfL https://agent.datafy.io/uninstall\?version=$VERSION | sh ``` {% endcode %} {% endstep %} {% step %} #### Remove the AWS Resources Terminate the EC2 instance, then delete the two restored EBS volumes. {% endstep %} {% endstepper %} {% endtab %} {% tab title="Delete Everything" %} Because the instance and the data are both being destroyed, you can skip deactivation and the uninstall. Deactivating autoscaling exists to return your data to a volume that's readable without AutoScaler, which is not needed if it is going to be deleted. 1. Terminate the EC2 instance. 2. Delete the EBS volumes left behind in the AWS Console - there will be four, since each autoscaling volume sits on a pair. {% hint style="warning" %} Skipping deactivation is fine here only because everything is being deleted. On a real environment, always [deactivate autoscaling before uninstalling AutoScaler](/set-up-and-installation/uninstalling-datafy.md) - otherwise you lose access to the data on volumes you still need. {% endhint %} {% endtab %} {% endtabs %} For more on uninstalling, see [Uninstalling Datafy](/set-up-and-installation/uninstalling-datafy.md). *** ## What's Next? Now that you've seen autoscaling in action, here are some next steps: * Learn more about how the AutoScaler works under the hood in [How AutoScaler Works](/how-it-works/how-autoscaler-works.md) * Explore [Autoscaling Rules](/volume-lifecycle/autoscaling-rules.md) to set up tag-based and instance-based rules for your production volumes * Set up [Datafy Snapshots](/volume-lifecycle/datafy-snapshots.md) for backup and recovery of your autoscaling volumes --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.datafy.io/resources/my-first-autoscaling-volume.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.