# Say Hello to Cloud Storage Savings

## Datafy in a Nutshell

Datafy is a cloud storage management platform that can help you manage your EBS volumes in a cost-effective manner. Datafy installs software on nodes and instances attached to EBS volumes, and uses that software to seamlessly manage the EBS volumes on your behalf.

## Solutions

### Datafy Sensor

The Datafy Sensor is a lightweight monitoring solution designed to track and analyze your EBS volumes' usage. With read-only permissions, it ensures minimal system impact while providing detailed insights into your EBS utilization.

### Datafy AutoScaler

The Datafy AutoScaler manages your volumes directly and automatically grows and shrinks their size based on your actual usage, reducing your EBS costs by up to 50%.

* Seamless integration - the AutoScaler agent virtualizes the storage layer, presenting a large "virtual" volume to applications. It creates the underlying EBS volumes and copies data between them in the background to ensure optimal utilization.
* Full control - even after installation, AutoScaler manages only the volumes you select, while monitoring the rest for full visibility. You can revert volumes from AutoScaling and uninstall Datafy at any time.

## Getting Started

Getting started with Datafy is as easy as 1-2-3!

{% stepper %}
{% step %}

#### [Install Datafy](/set-up-and-installation/datafy-installation)

* One line install for either EC2 instances or K8s clusters.
* Completely seamless to your applications: 0 engineering team hours.
  {% endstep %}

{% step %}

#### Monitor Your EBS Usage

* Use **Datafy Sensor** to see your EBS utilization in real-time.
  {% endstep %}

{% step %}

#### Automatic Optimization

* Use [**Datafy AutoScaler**](/how-it-works/how-autoscaler-works) to choose which volumes you want Datafy to manage.
* Sit back and watch your cloud bill go down as Datafy only provisions the disk space you use.
* Give your DevOps teams a break as you will never run out of space or need to change the EBS volumes.
  {% endstep %}
  {% endstepper %}

Ready to get started?

{% content-ref url="/pages/tO578uqOXZQju8rEP5hc" %}
[Datafy Installation](/set-up-and-installation/datafy-installation)
{% endcontent-ref %}

{% content-ref url="/pages/RJzhBGB8r8UOw4xff9X0" %}
[My First Autoscaling Volume](/resources/my-first-autoscaling-volume)
{% endcontent-ref %}


# Datafy Installation

You can install the Datafy [**AutoScaler**](/#datafy-autoscaler) directly, or start with the Datafy [**Sensor**](/#datafy-sensor) and update later on.

{% stepper %}
{% step %}

#### [Configure Permissions](/set-up-and-installation/datafy-installation/permissions-configuration)

Datafy requires basic permissions to view the EC2 instances your volumes are attached to, and to control the creation and modification of EBS volumes.
{% endstep %}

{% step %}

#### [Grant Network Access](/set-up-and-installation/datafy-installation/network-access)

Network access is required to allow the Datafy agent to communicate with our application’s backend. If access is blocked by default, access to Datafy needs to be whitelisted.
{% endstep %}

{% step %}

#### [Generate a Security Token](/set-up-and-installation/datafy-installation/token-generation)

Generate a security token used to identify your account with Datafy's application.
{% endstep %}

{% step %}

#### [Install Sensor or AutoScaler](/set-up-and-installation/datafy-installation/installation)

Install Datafy with one command line and zero downtime. It can be installed on an [EC2 instance](/set-up-and-installation/datafy-installation/installation#ec2-installation), or on a [K8s cluster](/set-up-and-installation/datafy-installation/installation#kubernetes-cluster) using a DaemonSet.
{% endstep %}
{% endstepper %}

{% hint style="success" %}
Datafy supports the following infrastructure:

* EC2 or any K8s cluster (EKS or self-managed)
* Any Linux based OS distribution - Amazon Linux, Ubuntu, and more.

The full list of supported cases can be found [here](/set-up-and-installation/supported-infrastructure).
{% endhint %}

{% hint style="info" %}
Prefer to follow along on a throwaway environment first? [My First Autoscaling Volume](/resources/my-first-autoscaling-volume) sets one up and walks through installation, activation, grow and shrink end to end.
{% endhint %}


# Permissions Configuration

Datafy requires basic AWS permissions to view the EC2 instances your volumes are attached to, and to control the creation and modification of EBS volumes.

These permissions are granted through a dedicated IAM role, and providing the ARN to Datafy.

{% hint style="info" %}
Permissions are configured per AWS account. If you wish to install Datafy on multiple AWS accounts, the following steps need to be performed for each account.
{% endhint %}

## Configure IAM Role

{% stepper %}
{% step %}

#### Create IAM Role

We provide CloudFormation and Terraform templates for creating an IAM role. The template creates an IAM OIDC[^1] identity provider and an IAM role. The identity provider is used to authenticate that only Datafy uses the created role.

When creating the role, define the permissions level of the role according to the Datafy product you intend to install:

* **Sensor** **permissions** - the role provides read-only permissions. It allows to retrieve data about EC2 instances and the volumes attached to them.
* **AutoScaler** **permissions** - the role includes all of the Sensor read-only permissions, and adds modification and creation permissions. The role allows creation, modification and deletion of EBS volumes, which are used to manage your EBS storage devices.

You can also optionally scope the role to a specific Datafy [account or organization](/set-up-and-installation/account-management). This is recommended - when scoped, only the matching Datafy account or organization can assume the role.

* Use your **Organization ID** to allow any sub-account under that organization to use the role.
* Use a specific **Account ID** to restrict the role to a single sub-account.
* Leave the value empty to trust any Datafy subject.

Both IDs are shown on the [**Permissions** page](https://app.datafy.io/settings) in the dashboard — see [#define-role-arn-in-datafy-account](#define-role-arn-in-datafy-account "mention") below.

{% tabs fullWidth="false" %}
{% tab title="CloudFormation template" %}
Create a new CloudFormation stack using the URL below as the template source (see screenshot).

{% code title="S3 URL" overflow="wrap" %}

```url
https://datafy-public-bucket.s3.amazonaws.com/cloudformation-template/aws_iam/cloudformation.yaml
```

{% endcode %}

<figure><img src="/files/c6P2Ziyz5QEMGpAQxzzV" alt=""><figcaption></figcaption></figure>

In the Parameters section, choose the desired permissions level from the **Role Permissions Level** dropdown.

<figure><img src="/files/pEB8rSmpoyHE4shhNitz" alt=""><figcaption></figcaption></figure>

To scope the role, paste the Organization ID or Account ID into the **Datafy Account ID** field under **Identity Configuration**.

<figure><img src="/files/4TLGiaZlBdzS5nxfpHyn" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="Terraform module" %}
{% hint style="info" %}
The `iam-role` module has moved. It is now a submodule of the [datafy-io/modules/aws](https://registry.terraform.io/modules/datafy-io/modules/aws/latest) collection. The standalone `datafy-io/iam-role/datafy` module is deprecated and will receive no further updates.

The inputs are unchanged — to migrate, point `source` at the new location:

```hcl
module "datafy_role" {
  source  = "datafy-io/modules/aws//modules/iam-role"
  version = "~> 1.0"
}
```

{% endhint %}

Create the role using the [iam-role Terraform module](https://registry.terraform.io/modules/datafy-io/modules/aws/latest/submodules/iam-role?tab=inputs). The main inputs are:

* `permissions_level` — `Sensor` or `AutoScaler`.
* `permissions_scope` — `Global`, or `Regional` (with a `regions` list).
* `account_id` — Datafy Organization ID or Account ID to scope the role. Leave empty to trust any Datafy subject.

If you manage your Datafy accounts with the [Datafy Terraform provider](https://registry.terraform.io/providers/datafy-io/datafy/latest/docs), you can read the ID from the [`datafy_account`](https://registry.terraform.io/providers/datafy-io/datafy/latest/docs/resources/account) resource or data source — use `.id` for a sub-account or `.parent_account_id` for the organization — and pass it to `account_id`.

See the [module documentation](https://registry.terraform.io/modules/datafy-io/modules/aws/latest/submodules/iam-role?tab=inputs) for the full list of inputs and examples.
{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

#### Define Role ARN in Datafy Account

* In the Datafy app, go to the **Permissions** section of the [account settings](https://app.datafy.io/settings). An Admin role is required to access the settings panel.
* Fill in the ARN of the IAM role you just created.
* The IAM role associated with the ARN will be validated. If the ARN is incorrect, or the permissions in the associated role are incorrect, you will receive an error.

The Permissions page also displays the **Organization ID** at the top and the **Account ID** under each sub-account. Copy either to use as the `Datafy Account ID` parameter when creating the role.

{% hint style="info" %}
If the role cannot be validated, for example because your AWS account has an attached organizational [SCP](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html), you can skip the validation and insert the role anyway. Make sure your role has all of the necessary permissions for Datafy to work as expected.
{% endhint %}

<div data-with-frame="true"><figure><img src="/files/GuxZMDVaqqk2lBcQeEAY" alt=""><figcaption></figcaption></figure></div>

{% hint style="info" %}
The role ARN can also be registered programmatically using the [`datafy_role_arn`](https://registry.terraform.io/providers/datafy-io/datafy/latest/docs/resources/role_arn) resource from the Datafy Terraform provider.
{% endhint %}
{% endstep %}
{% endstepper %}

{% hint style="info" %}
If you're using custom encryption keys, ensure that the Datafy role isn't restricted by your KMS policy. Datafy needs to be able to create volumes that "inherit" the keys of the original source volume that is being auto-scaled
{% endhint %}

## Validate Configuration

After you've created and defined the IAM role, you can see all of the volumes in your account in the [Datafy app](https://app.datafy.io/fleet-manager).

At this stage, you can view the AWS attributes of each volume, including its provisioned size, but not the actual utilization. To view the utilization of each volume, continue with the installation process.

[^1]: OpenID Connect


# Network Access

The Datafy agent makes outbound HTTPS calls to the endpoints below. No data from the instance is sent to Datafy, only the volume consumption statistics that power the management dashboard.

| Endpoint URL                | Port | Static IP Alternative                 |
| --------------------------- | ---- | ------------------------------------- |
| dso.datafy.io               | 443  | <p>76.223.36.164<br>13.248.178.22</p> |
| agent.datafy.io             | 443  | <p>76.223.36.164<br>13.248.178.22</p> |
| iac.datafy.io               | 443  | <p>76.223.36.164<br>13.248.178.22</p> |
| ec2.\<region>.amazonaws.com | 443  | —                                     |

The EC2 endpoint is regional. The agent calls the EC2 API in the region where it runs, so the simplest approach is to allow every region enabled on the AWS account.

## Installing Behind an HTTPS Proxy

{% hint style="success" %}
HTTPS proxy support is available from AutoScaler version 1.37.1 (Helm chart 3.5.1).
{% endhint %}

If your instances reach the internet through an HTTPS proxy, configure the following before or during [installation](/set-up-and-installation/datafy-installation/installation). Datafy then routes all of its outbound traffic, including the calls to the endpoints above, through the proxy.

* `HTTPS_PROXY` (Helm: `proxy.httpsProxy`) — URL of the proxy server for the agent's outbound HTTPS traffic.
* `NO_PROXY` (Helm: `proxy.noProxy`) — comma-separated host, IP, CIDR, or DNS-suffix list to bypass the proxy. Include your environment's internal ranges so local traffic doesn't route through the proxy. Optional.

{% tabs %}
{% tab title="EC2" %}
Define the proxy in environment variables before installing Datafy:

{% code overflow="wrap" %}

```sh
HTTPS_PROXY="http://10.0.0.5:3128"
NO_PROXY="10.0.0.0/8,.internal.example.com"
```

{% endcode %}

The installer writes the values to the agent's environment file and re-applies them on every upgrade.
{% endtab %}

{% tab title="Helm" %}
Add the proxy values to your `helm upgrade --install` command:

{% code overflow="wrap" %}

```bash
--set proxy.httpsProxy="http://10.0.0.5:3128" \
--set proxy.noProxy="10.0.0.0/8\,.internal.example.com"
```

{% endcode %}

The chart renders the proxy variables on the AutoScaler DaemonSet, the controller, and the CSI proxy sidecar.

{% hint style="info" %}
`proxy.noProxy` is passed through to the agent verbatim. Helm's `--set` splits values on commas, so each comma in the list must be escaped as `\,`. For longer bypass lists, pass the values through a file instead:

{% code overflow="wrap" %}

```yaml
# values.yaml
proxy:
  httpsProxy: "http://10.0.0.5:3128"
  noProxy: "10.0.0.0/8,.internal.example.com"
```

{% endcode %}
{% endhint %}
{% endtab %}
{% endtabs %}

{% hint style="info" %}
The agent always bypasses `169.254.169.254` (the EC2 instance metadata endpoint), `127.0.0.1`, and `localhost`. Anything passed in `NO_PROXY` is appended to that list.
{% endhint %}


# Token Generation

Datafy uses a security token to identify communication from the agents installed on your machines, and to associate it with your account. The token is secret, and cannot be accessed again after it is generated.

{% hint style="info" %}
The token is used for the [Datafy agent installation](/set-up-and-installation/datafy-installation/installation). If you'd rather not issue and manage a long-lived token, see [Tokenless Installation](/set-up-and-installation/advanced-setup/tokenless-installation).
{% endhint %}

### Creating a New Token

{% hint style="info" %}
Tokens are associated with a specific Datafy app account, which is associated with a single AWS account. A new token should be generated from the relevant Datafy app account for every AWS account Datafy is installed on.
{% endhint %}

1. Open the account menu at the top right corner of the Datafy app
2. Open the **Admin Portal**
3. Navigate to the **API Tokens** section in the sidebar
4. Select **Generate Token**
5. Configure the token:
   * Give it a meaningful description, like `<your-company>-datafy-token`.
   * Set the role to **Admin**
   * Set expiration date to `never`
6. Save the generated token in a secure manner, and use it when [installing the Datafy agent](/set-up-and-installation/datafy-installation/installation)

{% hint style="warning" %}
Make sure to save tokens in a secure manner, so they are accessible for future installations in the same account
{% endhint %}


# Installation

## Install Datafy

Install Datafy on any nodes or instances with attached EBS volumes that you want Datafy to monitor or manage.

All installations include two parameters:

* `$VERSION` or `$HELM_VERSION` **-** The version number of the agent (in EC2) or Helm chart (in K8s) to be installed. We recommend always specifying it, for consistent behavior.
* `$TOKEN` - Security token used for identification, associated with the relevant AWS account. The security token can be [generated by an admin](/set-up-and-installation/datafy-installation/token-generation) through the Datafy app, or omitted entirely with [tokenless installation](/set-up-and-installation/advanced-setup/tokenless-installation).

{% hint style="info" %}
If you are installing on an instance where Datafy is already installed, check out [how to upgrade](/set-up-and-installation/upgrading-datafy)
{% endhint %}

{% hint style="info" %}
Installing behind an HTTPS proxy? See [Network Access](/set-up-and-installation/datafy-installation/network-access#installing-behind-an-https-proxy).
{% endhint %}

### EC2 Instance <a href="#ec2-installation" id="ec2-installation"></a>

Datafy is installed on EC2s as a Docker image. If not already installed on your instance, [install Docker](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-docker.html) before installing Datafy.

Run the following CLI command on your instance to install Datafy:

{% tabs %}
{% tab title="Sensor" %}
{% code overflow="wrap" %}

```sh
curl -sSfL https://agent.datafy.io/install?version=$VERSION | AGENT_MODE="sensor" TOKEN=$TOKEN sh
```

{% endcode %}
{% endtab %}

{% tab title="AutoScaler" %}
{% code overflow="wrap" %}

```sh
curl -sSfL https://agent.datafy.io/install?version=$VERSION | AGENT_MODE="autoscaler" TOKEN=$TOKEN sh
```

{% endcode %}
{% endtab %}
{% endtabs %}

{% hint style="info" %}
The install command downloads and runs an install script that ensures your instance is supported by Datafy, and then installs the requested version using Docker.
{% endhint %}

{% hint style="success" %}
Check out [Deployment Examples](/resources/deployment-examples) for examples how to install Datafy with the deployment tools you already use
{% endhint %}

### Kubernetes Cluster

You can install Datafy using a single line command or with a Helm chart directly.

{% hint style="warning" %}
When installing AutoScaler, make sure the nodes you install on are [supported](/set-up-and-installation/supported-infrastructure). If your cluster has a mix of supported and unsupported nodes, see [Kubernetes Node Scheduling](/set-up-and-installation/advanced-setup/kubernetes-node-scheduling#partial-cluster-installation).
{% endhint %}

#### Helm Chart Installation

Datafy can be installed directly via a Helm chart. See the [latest Helm chart documentation](https://github.com/datafy-io/datafy-agent-helm/) for additional installation parameters and versioning.

The Helm chart installs the Datafy agent as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/), by default on every node in your cluster (see [Kubernetes Node Scheduling](/set-up-and-installation/advanced-setup/kubernetes-node-scheduling#partial-cluster-installation) to install on a subset of nodes instead). For AutoScaler installations, it also installs a Datafy controller Deployment, and adds a `datafy-proxy` sidecar container to your existing EBS CSI driver pods — see [AutoScaler on Kubernetes](/how-it-works/autoscaler-on-kubernetes) for what each component does. If the installation fails, Helm will roll back the cluster to the state it was in before the installation began.

{% hint style="info" %}
For AutoScaler installations, we recommend adding a node taint to gate scheduling around node startup and AutoScaler outages — see [Node Taints](/set-up-and-installation/advanced-setup/kubernetes-node-scheduling#node-taints).
{% endhint %}

{% stepper %}
{% step %}
**Add Datafy Repo**

```bash
helm repo add datafyio https://helm.datafy.io/datafy-agent
helm repo update
```

{% endstep %}

{% step %}
**Install Datafy**

{% tabs %}
{% tab title="Sensor" %}
{% code overflow="wrap" %}

```bash
helm upgrade --install datafy-agent \
--version "$HELM_VERSION" datafyio/datafy-agent \
--namespace datafy-agent --create-namespace \
--set agent.mode="sensor" \
--set agent.token="$TOKEN" \
--atomic   
```

{% endcode %}
{% endtab %}

{% tab title="AutoScaler" %}
{% code overflow="wrap" expandable="true" %}

```bash
helm upgrade --install datafy-agent \
--version "$HELM_VERSION" datafyio/datafy-agent \
--namespace datafy-agent --create-namespace \
--set agent.mode="autoscaler" \
--set agent.token="$TOKEN" \
--atomic
```

{% endcode %}

{% hint style="warning" %}
To successfully install Autoscaler on nodes running **Bottlerocket**, you must disable kernel lockdown prior to installation. Add the following configuration to the node's user data:

{% code overflow="wrap" %}

```bash
[settings.kernel]
lockdown = "none"
```

{% endcode %}
{% endhint %}
{% endtab %}
{% endtabs %}
{% endstep %}
{% endstepper %}

{% hint style="info" %}
To read the token from a Secret you already manage, replace `agent.token` with `agent.externalTokenSecret.name` and `agent.externalTokenSecret.key`. Both are required.
{% endhint %}

#### Single-Line Installation

{% hint style="info" %}
If your cluster is running Bottlerocket, use the Helm chart installation above
{% endhint %}

{% tabs %}
{% tab title="Sensor" %}
{% code overflow="wrap" %}

```bash
curl -sSfL https://agent.datafy.io/install-k8s?version="$HELM_VERSION" | AGENT_MODE="sensor" TOKEN="$TOKEN" sh
```

{% endcode %}
{% endtab %}

{% tab title="AutoScaler" %}
{% code overflow="wrap" %}

```sh
curl -sSfL https://agent.datafy.io/install-k8s?version="$HELM_VERSION" | AGENT_MODE="autoscaler" TOKEN="$TOKEN" sh
```

{% endcode %}
{% endtab %}
{% endtabs %}

## Validate Installation

You're all set up!

Once you've successfully completed the installation, you can see the utilization of the volumes attached to instances Datafy is installed on in the [Datafy app](https://app.datafy.io/fleet-manager).

If you've installed Autoscaler, you can now use the ["AutoScale" toggle or create an autoscaling rule to activate autoscaling](/volume-lifecycle/managing-autoscaling-volumes#activating-autoscaling) for the selected volumes.

To try that on a test volume first, follow [My First Autoscaling Volume](/resources/my-first-autoscaling-volume).


# Supported Infrastructure

## Supported Operating Systems

Datafy can be installed on the following Linux distributions:

<details>

<summary>Ubuntu</summary>

* 20.04
* 22.04
* 24.04

</details>

<details>

<summary>Amazon Linux</summary>

* AL2
* AL2023

</details>

<details>

<summary>Bottlerocket</summary>

* 10 latest versions

</details>

<details>

<summary>RHEL</summary>

* RHEL 8 - [latest version](https://access.redhat.com/articles/red-hat-enterprise-linux-release-dates#RHEL8)
* RHEL 9 - [latest version](https://access.redhat.com/articles/red-hat-enterprise-linux-release-dates#RHEL9)

{% hint style="info" %}
If you're running an older minor version contact Datafy support
{% endhint %}

</details>

{% hint style="info" %}
Don't see your OS version in the list? Let us know, we're always expanding our support!
{% endhint %}

## Supported Configurations

The following instance and volume configurations are supported for utilization reporting (Sensor and Autoscaler) and for autoscaling (AutoScaler).

<table><thead><tr><th width="198.9112548828125"></th><th align="center">Utilization Reporting</th><th align="center">Autoscaling</th></tr></thead><tbody><tr><td>Hypervisor</td><td align="center">All</td><td align="center"><a data-footnote-ref href="#user-content-fn-1">Nitro</a></td></tr><tr><td>Instance size</td><td align="center">All</td><td align="center">≥ <code>xlarge</code> recommended</td></tr><tr><td>Volume types</td><td align="center">All</td><td align="center">gp2, gp3</td></tr><tr><td>Volume size</td><td align="center">All</td><td align="center">from 20 GiB to 16 TiB</td></tr><tr><td>Boot volumes</td><td align="center"><span data-gb-custom-inline data-tag="emoji" data-code="2714">✔️</span></td><td align="center"><span data-gb-custom-inline data-tag="emoji" data-code="2716">✖️</span></td></tr><tr><td>LVM</td><td align="center">VGs with a single PV</td><td align="center"><span data-gb-custom-inline data-tag="emoji" data-code="2716">✖️</span></td></tr><tr><td>Multi-attach volumes</td><td align="center">Reported utilization may be incorrect</td><td align="center"><span data-gb-custom-inline data-tag="emoji" data-code="2716">✖️</span></td></tr><tr><td>Single- or Multi-partition volumes</td><td align="center">Reported utilization may be incorrect</td><td align="center"><span data-gb-custom-inline data-tag="emoji" data-code="2716">✖️</span></td></tr></tbody></table>

### Additional Considerations

{% hint style="info" %}
If your Kubernetes cluster contains a mix of supported and unsupported node types, see [Kubernetes Node Scheduling](/set-up-and-installation/advanced-setup/kubernetes-node-scheduling#partial-cluster-installation) to install AutoScaler only on the nodes that support it.
{% endhint %}

{% hint style="warning" %}
On EC2 instances, autoscaling will not be enabled for volumes that do not follow the [best practice](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-using-volumes.html) of using the filesystem UUID in the relevant `/etc/fstab` entry. Using the device path (e.g., `/dev/sdf` or `/dev/nvme1n1`) can result in failure to mount an autoscaling volume after reboot or mounting the volume to the wrong mountpoint, and is not supported.
{% endhint %}

[^1]: All new instance types use the Nitro hypervisor. See [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#instance-hypervisor-type) for breakdown by instance type


# Upgrading Datafy

{% hint style="info" %}
The following is relevant when installing version `1.28.0` or higher. If installing an older version, you must [uninstall](/set-up-and-installation/uninstalling-datafy) the existing agent before installing the newer version.
{% endhint %}

## Upgrading from Sensor to AutoScaler

Datafy **Sensor** and **AutoScaler** are both based on shared components, operating at different permission levels: Sensor is read-only, and AutoScaler can modify and create volumes.

If you already have Sensor installed and want to upgrade to AutoScaler, these permissions need to be updated, and additional components that manage the volume lifecycle need to be installed.

{% stepper %}
{% step %}
**Update IAM Role**

Update the IAM role you created when installing Datafy Sensor, using the AutoScaler permissions level in the [role template](/set-up-and-installation/datafy-installation/permissions-configuration).

{% hint style="info" %}
If you choose to create a new role and not update the existing one, make sure to update the role ARN in the Datafy app
{% endhint %}
{% endstep %}

{% step %}
**Install Datafy AutoScaler**

[Install Datafy AutoScaler](/set-up-and-installation/datafy-installation/installation) the same way you installed Sensor, using the AutoScaler version of the installation command.

{% hint style="info" %}
We recommend asking about the latest version before you upgrade
{% endhint %}
{% endstep %}
{% endstepper %}

## Upgrading Version of Existing Agent

### Sensor

You can upgrade the version of Datafy Sensor you have installed by running the [install command](/set-up-and-installation/datafy-installation/installation) with the new version you'd like to install.

Before upgrading, you can check which version is already installed with the following:

{% tabs %}
{% tab title="EC2" %}
You can check which version is installed on your instance with the command:

```bash
sudo grep '^VERSION=' /etc/datafy/datafy.conf
```

{% endtab %}

{% tab title="K8s" %}
You can check which version is installed on your cluster by checking the image tag of the `datafy-installer` DaemonSet, or use the command:

{% code overflow="wrap" %}

```
kubectl exec -it -n kube-system datafy-installer-<pod-id> -- /usr/local/datafy-agent/bin/datafy-agent --version
```

{% endcode %}
{% endtab %}
{% endtabs %}

{% hint style="info" %}
You can install a newer or older version this way, make sure to check which is the right one
{% endhint %}

### AutoScaler

#### Upgrading from Version `1.32.0` or Above

AutoScaler is upgraded in the same way as Sensor. Volumes with active autoscaling and the data on them will continue to be available during the upgrade, with a short period of increased latency.

If the new version includes a kernel module upgrade, it will be implemented the next time the instance reboots. If the Autoscale upgrade is performed during a scheduled maintenance window, it is recommended to reboot the instance following the upgrade to ensure the kernel is updated.

#### Upgrading from Version < `1.32.0`

AutoScaler is upgraded in the same way as Sensor. However, if you have volumes with active autoscaling, they may become temporarily unavailable during the upgrade.

In this case, after running the install command, you will be warned and prompted to approve the installation. If there is no response, the installation will be aborted.

{% code overflow="wrap" %}

```
Autoscaling volumes may be unavailable during installation. Do you want to proceed? [y/N]: y
```

{% endcode %}

You can use the flag `FORCE_INSTALL=true` to skip the prompt and approve the installation in advance. For example:

{% code overflow="wrap" %}

```bash
curl -sSfL https://agent.datafy.io/install?version=$VERSION | AGENT_MODE="autoscaler" TOKEN=$TOKEN FORCE_INSTALL=true sh
```

{% endcode %}

{% hint style="warning" %}
To ensure no impact on your application or the data on the autoscaling volumes, we recommend unmounting them before upgrading AutoScaler.
{% endhint %}


# Uninstalling Datafy

Uninstalling Datafy is just as simple as the initial installation. Except for the [preparation](#prepare-for-uninstall), the following steps are the same for uninstalling Sensor or AutoScaler.

## Prepare for Uninstall

If you have **AutoScaler** installed, before you can uninstall, you must turn off autoscaling for all volumes managed by Datafy. Turning off autoscaling returns your volumes to their original provisioned size and removes the virtualization layer managed by Datafy. This is critical for ensuring access to your data after Autoscaler is uninstalled.

{% hint style="info" %}
If you only have Sensor installed, you can skip this step
{% endhint %}

{% stepper %}
{% step %}
**Turn Off Autoscaling Rules**

If you have autoscaling rules active in your account, turn off any rules relevant to the instances you wish to uninstall from, in the [Autoscale Rules table](https://app.datafy.io/fleet-manager/autoscale-rules) in the Datafy app.

This ensures that autoscaling isn't activated for any new volumes while you are preparing for uninstalling.

<figure><img src="/files/8L8FfX9Aiu6elpKvQ1BW" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Identify Managed Volumes**

In the [All Volume table](https://app.datafy.io/fleet-manager/all-volumes) in the Datafy app, find all of the relevant volumes by filtering the status "Autoscaling active" in the dashboard

<div data-full-width="false"><figure><img src="/files/UXOnfDwTYaZfGewOrfgm" alt="" width="375"><figcaption></figcaption></figure></div>
{% endstep %}

{% step %}
**Turn off Autoscaling**

Turn off the toggle in the "AutoScale" column for all of the relevant volumes.

<figure><img src="/files/ITexFuzw17SSXsnLFivk" alt=""><figcaption></figcaption></figure>

Similarly to the initial activation when autoscaling was turned on, this stage can take time, dependent on the size of the volume. You can track the progress in the All Volumes table, or in the Optimization Actions table in the [Reports](https://app.datafy.io/reports) tab.

<figure><img src="/files/kcKQg47H0DEbTEh23VkM" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

## Uninstall Datafy

{% hint style="warning" %}
Uninstalling AutoScaler when there are managed volumes can lead to loss of access to the data stored on those volumes. Always [turn off autoscaling](#prepare-for-uninstall) before uninstalling Autoscaler.
{% endhint %}

Datafy can be uninstalled in the same ways it was installed, by substituting `install` from the [original install command](/set-up-and-installation/datafy-installation/installation) with `uninstall`. The version supplied should match the version installed.

### EC2 Instance

```sh
curl -sSfL https://agent.datafy.io/uninstall\?version=$VERSION | sh
```

{% hint style="info" %}
You can check which version is installed on your instance with the command:

```bash
sudo grep '^VERSION=' /etc/datafy/datafy.conf
```

{% endhint %}

### Kubernetes Cluster

You should use the same method to uninstall as you did to install

{% tabs %}
{% tab title="One-line installation" %}

```sh
curl -sSfL https://agent.datafy.io/uninstall-k8s\?version=$VERSION | sh
```

{% hint style="info" %}
You can check which version is installed on your cluster by checking the image tag of the `datafy-installer` DaemonSet
{% endhint %}
{% endtab %}

{% tab title="Helm chart installation" %}

```shellscript
helm uninstall datafy-agent --namespace datafy-agent
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
Some Datafy components may remain in your cluster as long as volumes that were previously autoscaling exist in the cluster. The components will remove themselves once all previously autoscaling volumes are deleted from the cluster. For details, see [AutoScaler on Kubernetes — Uninstalling](/how-it-works/autoscaler-on-kubernetes#uninstalling).
{% endhint %}

## Remove IAM Role

After uninstallation, the IAM role configured during the [initial setup](/set-up-and-installation/datafy-installation/permissions-configuration) should be removed. This can be done in one of the following ways:

1. Remove the ARN from the Permission section of the [account settings](https://app.datafy.io/settings).
2. Delete the role from your AWS account, by deleting the CloudFormation stack or Terraform module.


# Advanced Setup

Some environments have constraints the [standard install](/set-up-and-installation/datafy-installation) doesn't account for. Below are advanced setup and installation options to support these environments:

* [Tokenless Installation](/set-up-and-installation/advanced-setup/tokenless-installation) - For environments where a long-lived [security token](/set-up-and-installation/datafy-installation/token-generation) is a problem: a policy against static credentials, or a fleet where distributing and rotating tokens is more overhead than it's worth.
* [Kubernetes Node Scheduling](/set-up-and-installation/advanced-setup/kubernetes-node-scheduling) - For clusters running AutoScaler on [only some of the nodes](/set-up-and-installation/supported-infrastructure), or with enough node churn that pods move between nodes regularly.
* Additional installation configurations:
  * [Installing behind an HTTPS proxy](/set-up-and-installation/datafy-installation/network-access#installing-behind-an-https-proxy), when your instances have no direct outbound access.
  * [Reading the token from a Secret you already manage](/set-up-and-installation/datafy-installation/installation#helm-chart-installation), instead of passing it to Helm.


# Tokenless Installation

{% hint style="success" %}
Tokenless installation is supported from agent version 1.39.0 (EC2) and Helm chart version 3.6.0 (Kubernetes).
{% endhint %}

By [default](/set-up-and-installation/datafy-installation/installation), Datafy identifies your AWS account using a [security token](/set-up-and-installation/datafy-installation/token-generation) passed in as `$TOKEN`. Tokenless installation is an advanced alternative when using a static token isn't compatible with your environment: the instance or cluster proves its AWS identity with its own IAM role instead, and Datafy exchanges that proof for a short-lived runtime token behind the scenes. Nothing long-lived is stored on the instance or in the cluster.

{% hint style="warning" %}
A tokenless instance profile or IRSA role only proves identity — it grants no AWS permissions. Datafy still needs a dedicated [IAM role](/set-up-and-installation/datafy-installation/permissions-configuration) to read and manage your EC2 instances and EBS volumes, regardless of token mode.
{% endhint %}

<table><thead><tr><th width="94.85589599609375">Where</th><th width="194.6484375">Mode</th><th>What you need</th></tr></thead><tbody><tr><td>EC2</td><td>Tokenless</td><td>An <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html">IAM instance profile</a> attached to the instance, with IMDS access</td></tr><tr><td>EKS</td><td>IRSA</td><td>A dedicated IAM role for the controller's ServiceAccount, trusted by the cluster's OIDC provider</td></tr><tr><td>EKS</td><td>Node instance profile</td><td>IMDS access with a metadata hop limit of 2 or greater</td></tr></tbody></table>

A given instance or cluster can only use one mode at a time — you can't combine tokenless, IRSA, and a static token.

## EC2 Instances

Before you install Datafy:

1. Attach an [IAM instance profile](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) to the instance. The role is only used to prove the instance's identity, not to grant permissions, any role will work.
2. Make sure the instance can reach the instance metadata service (IMDS) at `169.254.169.254`. Either IMDSv1 or IMDSv2 works.

Then install Datafy without the `TOKEN` parameter:

{% code overflow="wrap" %}

```sh
curl -sSfL https://agent.datafy.io/install?version=$VERSION | AGENT_MODE="autoscaler" sh
```

{% endcode %}

The installation will fail if no instance profile is attached, or if IMDS is unreachable.

## Kubernetes Clusters

Tokenless is only available on **EKS**. On EKS, the Datafy controller can authenticate either using the underlying node's instance profile, or using its own IRSA role. On other Kubernetes distributions, use [a token](/set-up-and-installation/datafy-installation/installation#helm-chart-installation) instead.

### Node Instance Profile

When installing, set the Helm chart variable `controller.serviceAccount.tokenless` to `true`. The [Datafy controller](/how-it-works/autoscaler-on-kubernetes#installing-datafy-on-kubernetes) uses the same IMDS-based mechanism as the [EC2 flow above](#ec2-instances), reading the worker node's own instance profile, which is automatically created by EKS.

{% code overflow="wrap" %}

```bash
helm upgrade --install datafy-agent \
--version "$HELM_VERSION" datafyio/datafy-agent \
--namespace datafy-agent --create-namespace \
--set agent.mode="autoscaler" \
--set controller.serviceAccount.tokenless=true \
--atomic
```

{% endcode %}

{% hint style="warning" %}
This mode requires pods to be able to reach the node's IMDS endpoint, which on EKS means the instance metadata **hop limit must be 2 or greater**. See [AWS's guidance on configuring the metadata hop limit](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html) for your node groups.
{% endhint %}

### IRSA

[IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) lets a Kubernetes ServiceAccount assume an AWS IAM role directly, without going through the node's instance profile. The role has no permission policy — only a trust relationship, since it proves identity rather than granting AWS permissions. Create one dedicated role per EKS cluster.

{% hint style="info" %}
Your cluster needs an IAM OIDC provider associated before you can create the role. If it doesn't have one, [create it first](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html): `eksctl utils associate-iam-oidc-provider --cluster <cluster-name> --approve`.
{% endhint %}

{% stepper %}
{% step %}

#### **Create the IAM Role**

Create a role trusted by your cluster's OIDC provider, restricted to the Datafy controller's ServiceAccount. The namespace and service account name here must match what you use in the Helm install below — the [standard Helm chart examples](/set-up-and-installation/datafy-installation/installation#helm-chart-installation) use namespace `datafy-agent` and service account `datafy-controller-sa`.

{% tabs fullWidth="false" %}
{% tab title="Terraform module" %}
You can create the role using the [iam-role-for-datafy-controller-eks Terraform module](https://registry.terraform.io/modules/datafy-io/modules/aws/latest/submodules/iam-role-for-datafy-controller-eks).\
Examples and usage instructions can be found in the module documentation. Set `cluster_name`, `datafy_controller_namespace`, and `datafy_controller_service_account_name` to match your cluster and Helm install, then use the module's `iam_role_arn` output in the next step — the module looks up your cluster's OIDC provider automatically, no need to find the URL yourself.
{% endtab %}

{% tab title="CloudFormation template" %}
Create a new CloudFormation stack using the URL below as the template source:

{% code title="S3 URL" overflow="wrap" %}

```url
https://datafy-public-bucket.s3.amazonaws.com/cloudformation-template/iam-role-for-datafy-controller-eks/cloudformation.yaml
```

{% endcode %}

First, find your cluster's OIDC provider URL. In the AWS console, go to **EKS → Clusters →&#x20;*****your cluster*** and copy the **OpenID Connect provider URL** from the **Overview** tab, or run:

{% code overflow="wrap" %}

```bash
aws eks describe-cluster --name <cluster-name> --query "cluster.identity.oidc.issuer" --output text
```

{% endcode %}

Then fill in the **EKS OIDC Provider URL**, **Kubernetes Namespace**, and **Kubernetes Service Account Name** parameters to match your cluster and Helm install. Once the stack is created, find the role's ARN on its **Outputs** tab.
{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

#### **Install or Upgrade the Helm Chart with the Role ARN**

Set the ARN from the previous step on `controller.serviceAccount.roleArn`:

{% code overflow="wrap" %}

```bash
helm upgrade --install datafy-agent \
--version "$HELM_VERSION" datafyio/datafy-agent \
--namespace datafy-agent --create-namespace \
--set agent.mode="autoscaler" \
--set controller.serviceAccount.roleArn="arn:aws:iam::<account-id>:role/<role-name>" \
--atomic
```

{% endcode %}

The chart adds the `eks.amazonaws.com/role-arn` annotation to the ServiceAccount and creates an empty `datafy-token` Secret that the controller populates.
{% endstep %}
{% endstepper %}

{% hint style="info" %}
`controller.serviceAccount.tokenless` and `controller.serviceAccount.roleArn` can't be combined with each other or with `agent.token` / `agent.externalTokenSecret`. Set exactly one token source, or enable tokenless mode with none set.
{% endhint %}


# Kubernetes Node Scheduling

{% hint style="success" %}
Advanced node scheduling is supported from Helm chart version 3.6.0 with agent version 1.39.0.
{% endhint %}

An Autoscaling volume can only be mounted or accessed on a node running a healthy Datafy AutoScaler agent — the agent is what translates the underlying physical EBS volumes into a [single virtual device](/how-it-works/how-autoscaler-works). A pod with an Autoscaling volume that's scheduled to a node without a healthy AutoScaler can't mount its volume and never starts — and since Kubernetes has no reason to reschedule a pod stuck on a failed mount, it won't self-recover. A pod with an Autoscaling volume that's already running loses access to its volume's data if the agent stops running or responding.

To prevent both, Datafy uses [taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) and [node affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) as complementary mechanisms:

* [Node affinity](#partial-cluster-installation), inherited from your Helm installation, protects against a pod being rescheduled onto a node AutoScaler was never installed on, for example when AutoScaler is only installed on part of the cluster.
* [Node taints](#node-taints) protect against a node where AutoScaler is installed but not currently healthy.

{% hint style="info" %}
This is a common way to ensure pods run only on nodes with the infrastructure they need — for example, Cilium taints a node until its own networking is up, and both the NVIDIA GPU Operator and AWS EKS taint GPU nodes so only pods that request a GPU are scheduled there.
{% endhint %}

## How Datafy Manages Scheduling

The [Datafy controller](/how-it-works/autoscaler-on-kubernetes) watches pods and nodes and applies taints, tolerations, and affinity based on their state.

Whenever a pod is scheduled or rescheduled, including when it moves to a different node, the Datafy controller checks whether it has an Autoscaling volume:

* If it does, the controller applies the [node affinity](#partial-cluster-installation) from your AutoScaler Helm installation to the pod, so it's only ever scheduled where AutoScaler runs.
* If it doesn't, the controller adds a toleration for the `datafy.io` taint instead, so the pod can be scheduled anywhere regardless of AutoScaler's health there.

Separately, Datafy watches every node AutoScaler is installed on and keeps its `datafy.io` taint in sync with that node's health:

| When                                                                                                                                                 | What happens                                                                                                              |
| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| A node joins the cluster and AutoScaler hasn't been healthy there yet                                                                                | The [startup taint you configure](#node-taints) blocks scheduling of pods with Autoscaling volumes until AutoScaler is up |
| AutoScaler and its CSI sidecar are confirmed healthy on that node — on first boot, after recovering, or after the node returns to AutoScaler's scope | Datafy removes the `datafy.io` taint; any pod can now be scheduled there                                                  |
| AutoScaler or its CSI sidecar on an already-healthy node stops responding                                                                            | Datafy re-applies the `datafy.io` taint with `NoExecute`, evicting any pod with an Autoscaling volume                     |
| AutoScaler is removed from a node's scope — a narrower `agent.affinity.nodeAffinity` or a dropped node group                                         | Datafy applies the `datafy.io` taint with `NoExecute`, evicting any pod with an Autoscaling volume                        |

## Cluster Setup and Configuration

### Node Taints

Add a [startup taint](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to each node group AutoScaler runs on. A taint blocks Kubernetes from scheduling a pod on the node unless it carries a matching toleration — you only need to add the taint itself; Datafy's controller adds the matching toleration [automatically](#how-datafy-manages-scheduling) to any pod that doesn't need to avoid the node.

{% hint style="info" %}
Use the taint key format your cluster autoscaler recognizes as temporary, shown in the tabs below. A cluster autoscaler decides whether to add a new node by simulating whether a pending pod would schedule on it — if it doesn't recognize your taint as temporary, it sees the taint on the simulated node, concludes the pod still couldn't schedule there, and may decide against scaling up at all.
{% endhint %}

{% tabs %}
{% tab title="Cluster Autoscaler" %}
Add the taint to your node group's configuration. This example uses the `node_group_taints` variable from the [`terraform-aws-modules/eks`](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest/submodules/eks-managed-node-group) module — adapt the key names if you manage node groups a different way. The `startup-taint.cluster-autoscaler.kubernetes.io/` prefix is what tells Cluster Autoscaler to disregard this taint when simulating whether a new node would let a pending pod schedule:

{% code title="eks-node-groups.tf" overflow="wrap" %}

```hcl
node_group_taints = {
  dedicated = {
    key    = "startup-taint.cluster-autoscaler.kubernetes.io/datafy.io"
    value  = "autoscaler"
    effect = "NO_SCHEDULE"
  }
}
```

{% endcode %}
{% endtab %}

{% tab title="Karpenter" %}
Add `startupTaints` to the `NodePool` that provisions the nodes AutoScaler runs on. Karpenter recognizes `startupTaints` as temporary and excludes them from its own provisioning and scheduling decisions, so no special key prefix is needed:

{% code title="nodepool.yaml" overflow="wrap" %}

```yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: datafy-autoscaler
spec:
  template:
    spec:
      startupTaints:
        - key: datafy.io
          effect: NoSchedule
      # ...the rest of your existing NodePool spec
```

{% endcode %}
{% endtab %}
{% endtabs %}

Once the taint is applied, standard Kubernetes scheduling takes over — the scheduler won't place a pod on the node unless it tolerates the taint, the same as with any other taint. The cluster-autoscaler-specific handling above only affects scale-up decisions, not the scheduler itself.

{% hint style="success" %}
To confirm the taint applied, run `kubectl describe node <node-name>` and look for a line under `Taints:` matching the key you configured.
{% endhint %}

Datafy's controller automatically adds a toleration for both taint formats to any pod that doesn't need to avoid nodes without AutoScaler — for example:

{% code overflow="wrap" %}

```yaml
tolerations:
  - key: datafy.io
    operator: Exists
  - key: startup-taint.cluster-autoscaler.kubernetes.io/datafy.io
    operator: Exists
```

{% endcode %}

{% hint style="info" %}
Datafy's controller only updates a pod's tolerations and affinity when it's (re)scheduled, not immediately when Autoscaling is activated or deactivated on its volume. A mismatch between a volume's state and its pod's taints or affinity is temporary and resolves the next time the pod is scheduled.
{% endhint %}

### Partial Cluster Installation

If your cluster includes node types where [AutoScaler isn't supported](/set-up-and-installation/supported-infrastructure), use node affinity to install AutoScaler only on the nodes that support it.

{% stepper %}
{% step %}

#### **Set the Node Affinity in the Helm Chart**

Set `agent.affinity.nodeAffinity` to match the node group(s) or instance types you're installing AutoScaler on. The Helm chart uses this value to install the `datafy-agent` DaemonSet only on those nodes, and the [Datafy controller applies](#how-datafy-manages-scheduling) the same affinity to any pod using an Autoscaling volume.

{% code title="values.yaml" overflow="wrap" %}

```yaml
agent:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-group
                operator: In
                values:
                  - dedicated
```

{% endcode %}

{% hint style="info" %}
The Helm chart always excludes legacy, non-[Nitro](/set-up-and-installation/supported-infrastructure) instance types from AutoScaler mode, regardless of this setting. Any affinity you define here is combined with that default, not instead of it.
{% endhint %}
{% endstep %}

{% step %}

#### **Install AutoScaler**

Install or upgrade using the values file from the previous step, following the [standard Helm chart installation](/set-up-and-installation/datafy-installation/installation#helm-chart-installation).
{% endstep %}
{% endstepper %}


# IaC Reconciliation

When using **Datafy AutoScaler**, your original EBS volumes are replaced with Datafy-managed volumes. While the OS and applications remain unaffected thanks to Datafy's virtualization layer, Infrastructure as Code (IaC) tools like Terraform may encounter issues.

IaC tools anticipate specific EBS volumes as declared in code, leading to a mismatch between the expected original volume and the actual autoscaling volumes. Potential problems include unnecessary volume recreations, drift detection errors, failure during instance operations, and increased manual reconciliation efforts.

To resolve this, Datafy provides reconciliation plugins for common IaC tools. These plugins ensure that your IaC tool understands and accommodates the state changes introduced by Datafy.

{% hint style="info" %}
Don’t see your tool listed? Let us know—we’re actively expanding support.
{% endhint %}

## Terraform

Datafy provides a custom Terraform provider called [`datafyaws`](https://registry.terraform.io/providers/datafy-io/datafyaws/latest/docs). It integrates seamlessly into your Terraform workflows and ensures that Terraform’s state remains in sync with the actual volumes managed by AutoScaler.

{% hint style="warning" %}
The `datafyaws` provider reconciles the state of volumes that were originally created using Terraform. It cannot reconcile volumes not created or managed by Terraform.
{% endhint %}

### Getting Started

{% stepper %}
{% step %}
**Install Provider**

Add the provider to your `required_providers` block:

```hcl
terraform {
  required_providers {
    datafyaws = {
      source  = "datafy-io/datafyaws"
      version = "~> 1.0"
      // ADDITIONAL_CONFIGS = "copy values from matching aws provider"
    }
  }
}
```

A `datafyaws` provider needs to be defined for every [AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs) already defined in your Terraform files.

Make sure to use a version of `datafyaws` that supports the version of the AWS provider you use. See the documentation of the [latest version](https://registry.terraform.io/providers/datafy-io/datafyaws/latest/docs) in the Terraform registry.
{% endstep %}

{% step %}
**Configure Provider**

The `datafyaws` provider should be configured with the same parameters as the existing AWS providers. The `datafy_token` can be set via environment variable or directly in the config:

```hcl
provider "datafyaws" {
  datafy_token = "$TOKEN"
  // ADDITIONAL_CONFIGS = "copy values from matching aws provider"
}
```

{% endstep %}

{% step %}
**Add Provider to Resources**

The following Terraform resource types are supported:

* [`aws_ebs_volume`](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ebs_volume)
* [`aws_volume_attachment`](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/volume_attachment)
* [`aws_instance`](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance)

Add `provider = datafyaws` to each supported resource that is managed by AutoScaler, or associated with a volume managed by AutoScaler:

```hcl
resource "aws_ebs_volume" "example" {
  provider = datafyaws

  availability_zone = "us-east-1a"
  size              = 40
}
```

{% hint style="info" %}
`datafyaws` does not support nested volume declarations. Volumes must be defined separately from the instances to which they are attached.
{% endhint %}
{% endstep %}
{% endstepper %}

### Limitations

* The provider should only be used with the supported resource types listed above. Other resource types are currently not supported for autoscaling volumes.
* `final_snapshot` is **not supported** and must be explicitly set to `false` when destroying volumes.

### Removing Datafy Provider

As long as your AWS account is associated with an active Datafy account, there is no need to remove the `datafyaws` provider - it will work in the same manner as the AWS Terraform provider for all non-autoscaling volumes.

If you need to remove the provider, perform the following steps:

1. Turn off autoscaling for all relevant volumes. Make sure the deactivation is complete before proceeding to the next step.
2. Run `terraform apply` with `datafyaws` provider for all relevant resources. This will update the Terraform state.
3. Remove `datafyaws` from relevant resources. You can now continue to use Terraform as desired.


# AutoScaling Configurations

Datafy AutoScaler manages your volumes using a set of configurable parameters that control when and how volumes grow and shrink. Datafy's defaults are designed to work well for most workloads, but you can fine-tune behavior per account or per instance through the Configuration page.

For an overview of the grow and shrink lifecycle, see [How AutoScaler Works](/how-it-works/how-autoscaler-works).

{% hint style="success" %}
Once you activate autoscaling on your volumes they will automatically grow and shrink with the default configurations. Editing these configurations allows behavior fine-tuning and workload specific optimization, it is not needed for storage optimization to work.
{% endhint %}

{% hint style="info" %}
Only users with an **admin** role can edit configurations. Read-only users can view configuration values but cannot make changes.
{% endhint %}

## Configuration Levels

Configurations follow a three-level hierarchy. Each level inherits from the level above it, and any value set at a lower level overrides the inherited one.

* **Organization level** — applies to every account in the organization.
* **Account level** — applies to every instance in the account. Overrides the organization value.
* **Instance level** — applies to a single instance. Overrides the account value.

If an override is reverted, the value goes back to inheriting from the level above — and if that inherited value changes later, the override location automatically picks up the new value.

Organization-level configurations can only be set from the [organization view](/set-up-and-installation/account-management#accessing-accounts-and-organizations). Account and instance overrides can be set from either the organization view or the relevant account view.

## Viewing and Editing Configurations

You can access the [Configurations](https://app.datafy.io/settings/configurations) page by selecting **Settings → Configurations** in the left sidebar. The navigation tree on the left shows the configuration hierarchy: the organization at the top (when accessed from the organization view), each account below it, and any instances that have overrides. Select an item to view its values on the right.

Each field shows the current value, the default or inherited value below it, and a **Default** or **Revert** indicator showing whether it has been overridden. Click any value to edit it inline - input is validated against the allowed range in real time. Click **Save** to apply your changes. To restore a single field to the default or inherited value indicated below it, click **Revert** next to that field. To restore all fields at once, click **Revert all to default** at the bottom of the page.

To create a new instance-level override, click **Add New Configuration** and select an instance from the searchable list. Instances that already have custom values are labeled **Overridden**. Any fields you don't change will continue to inherit from the account level.

## Configuration Reference

The tables below describe each configurable parameter grouped by the autoscaling operation they affect. For more on how these operations work, see [How AutoScaler Works](/how-it-works/how-autoscaler-works).

### Autoscaling Activation Configurations

These configurations control the initial autoscaling activation of a volume, how the first managed volume is sized and how the initial data copy is performed.

<table><thead><tr><th width="150.27734375">Name</th><th>Description</th><th width="120.3671875">Default</th><th width="119.89453125">Range</th></tr></thead><tbody><tr><td><strong>Autoscaling Size Factor</strong></td><td>When autoscaling is activated, the initial managed volume size will be at least this factor multiplied by the current used data. A higher value means more free space after activation.</td><td>1.3</td><td>1.1 – 2.0</td></tr><tr><td><strong>Autoscaling Minimum Size (GiB)</strong></td><td><p>The smallest absolute size an autoscaling volume can reach following initial activation. Even if actual usage is lower, the volume will not be sized below this value.</p><p>Volumes with original sizes smaller than this cannot be autoscaled.</p></td><td>50 GiB</td><td>20 – 16,384 GiB</td></tr><tr><td><strong>Autoscaling Minimum Size (%)</strong></td><td>The smallest size an autoscaling volume can reach following initial activation, as a percentage of the original volume size. Works alongside the absolute minimum, whichever produces a larger size applies.</td><td>10%</td><td>5 – 100%</td></tr><tr><td><strong>Autoscaling Maximum Size (GiB)</strong></td><td>The maximum original volume size eligible for autoscaling. Volumes larger than this will not be autoscaled.</td><td>16,384 GiB</td><td>Up to 16,384 GiB</td></tr></tbody></table>

### Growth Configurations

These control automatic volume growth - when and by how much AutoScaler expands a managed volume as it fills up.

{% hint style="warning" %}
AutoScaler maintains a time based grow triggering mechanism that ensures the volume will never run out of space even when writing to it at the maximum possible rate. The editable configurations below are in addition to this mechanism.
{% endhint %}

<table><thead><tr><th width="149.859375">Name</th><th>Description</th><th width="119.765625">Default</th><th width="120.47265625">Range</th></tr></thead><tbody><tr><td><strong>Grow Factor</strong></td><td>When a grow is triggered, the new volume size will be this factor multiplied by the current size. Higher values add more headroom per grow but temporarily use more storage.</td><td>1.5</td><td>Min 1.1</td></tr><tr><td><strong>Grow Minimum Free Space</strong></td><td>An absolute free-space floor. Growth is triggered when free space on the volume drops below this value. Acts as a safety net to ensure volumes don't run out of space.<br></td><td>5 GiB</td><td>Min 5 GiB</td></tr></tbody></table>

### Shrink Configurations

These control automatic volume shrinking - when and by how much Datafy reduces a volume to reclaim unused capacity and reduce costs.

<table><thead><tr><th width="150.29296875">Name</th><th width="356.76953125">Description</th><th width="120.00390625">Default</th><th width="119.9921875">Range</th></tr></thead><tbody><tr><td><strong>Shrink Threshold (%)</strong></td><td>If not in shrink cooldown (see below), a shrink is triggered when the volume's utilization falls below this percentage (of the volume's current size). For example, at 50%, a volume using only 30 GiB out of a current size of 65 GiB will shrink.</td><td>50%</td><td>0 – 75%</td></tr><tr><td><strong>Shrink Abort Threshold (%)</strong></td><td>If a shrink is already in progress and the expected utilization in the new smaller size rises above this percentage, the shrink is aborted.</td><td>85%</td><td>80 – 100%</td></tr><tr><td><strong>Shrink Size Factor</strong></td><td>When shrinking, the new volume size will be at least this factor multiplied by the current used data, ensuring headroom remains after the operation.</td><td>1.3</td><td>Min 1.1</td></tr><tr><td><strong>Shrink Minimum Size (GiB)</strong></td><td>The smallest absolute size a volume can be shrunk to. Even if actual usage is lower, the volume will not be sized below this value.</td><td>50 GiB</td><td>20 – 16,384 GiB</td></tr><tr><td><strong>Shrink Minimum Size (%)</strong></td><td>The smallest size a volume can be shrunk to, as a percentage of the original volume size. Works alongside the absolute minimum, whichever produces a larger size applies.</td><td>10%</td><td>5 – 100%</td></tr><tr><td><strong>Shrink Cooldown Time</strong></td><td>Minimum wait time after any autoscaling operation (activation, grow, or shrink) before a new shrink can be triggered. Prevents rapid grow/shrink cycling on volumes with fluctuating usage.</td><td>60 min</td><td>Min 1 min</td></tr></tbody></table>


# Account and User Management

## Organizations and Accounts

An **account** in the Datafy app maps 1:1 to a single AWS account. If you have multiple AWS accounts (for example, separate AWS accounts for production, staging, and development), each one will have its own Datafy account.

An **organization** groups accounts and users under a single top-level entity, typically a company or business unit. Every account belongs to exactly one organization, and an organization can contain any number of accounts.

Users, SSO, and security settings are typically configured at the organization level, but can also be managed per account. [Datafy permissions](/set-up-and-installation/datafy-installation/permissions-configuration) (IAM roles) and autoscaling rules operate at the account level but can be viewed and edited from the organization view as well. Configurations are scoped to either level, with a hierarchy tree shown in both views.

### Account Creation

Your organization is provisioned by the Datafy team as part of onboarding. Once the organization exists, Admins can create additional sub-accounts — either from the Admin Portal in the Datafy app, or [programmatically](#programmatic-access) via API or Datafy's dedicated Terraform provider.

To create a sub-account from the app:

1. Log in to the organization (see [below](#accessing-accounts-and-organizations)).
2. Open the **Admin Portal** from the account menu.
3. Navigate to **All Accounts** and select your organization from the account list.
4. Navigate to the **Sub-Accounts** tab and select **Create Sub-Accounts**.
5. Provide the account name.

<div data-with-frame="true"><figure><img src="/files/hnEC0yBEn6xlGjeuTMzI" alt=""><figcaption></figcaption></figure></div>

To finish setup, create and configure the new account's [IAM role](/set-up-and-installation/datafy-installation/permissions-configuration).

{% hint style="info" %}
The IAM role can be configured from the [**Permissions**](https://app.datafy.io/settings/permissions) page of the sub-account, or from the organization's **Permissions** page, which lists every account in one view.
{% endhint %}

### Accessing Accounts and Organizations

The account menu in the top bar lists every organization and account you have access to, with accounts grouped under their respective organization. Select one to view it.

* **Account view** — volumes, autoscaling rules, optimization actions, and configurations for a single account.
* **Organization view** — a roll-up across every account in the organization that you have access to. Aggregated metrics (such as total managed volume size) are summed across accounts. List views show every resource from every account, with an extra **Account** column so you can see which account each one belongs to.

{% hint style="success" %}
Resources always belong to a single account, but the organization view lets you see them all together.
{% endhint %}

## Managing Users

Admins can invite users, assign roles, and remove users from the **Admin Portal**, through either the **Users** or **All Accounts** sections.

* **Users** — select **Invite User** and choose a role. The user is added to the account the admin is currently logged in to.

<div data-with-frame="true"><figure><img src="/files/IzJtvTVJgRMQNhLhyhH1" alt="" width="188"><figcaption></figcaption></figure></div>

* **All Accounts** — select an account from the list, open its **Users** tab, and click **Invite Users**. Set the desired sub-account access. The user is added to the account the admin selected.

<div data-with-frame="true"><figure><img src="/files/ZrBms00239DucB4LbSpo" alt="" width="375"><figcaption></figcaption></figure></div>

{% hint style="info" %}
Users of the organization with sub-account access enabled have access to every sub-account. Users without sub-account access can still see every account's data in the organization view. To grant access, or assign a different role, on a specific sub-account, invite the user to that account directly.
{% endhint %}

### User Roles

Datafy has three user roles:

<table><thead><tr><th width="131.5567626953125">Role</th><th>Access</th></tr></thead><tbody><tr><td><strong>Readonly</strong></td><td>View volumes, reports, rules, and configurations. Cannot make changes.</td></tr><tr><td><strong>User</strong></td><td>Everything in Readonly, plus activate and deactivate autoscaling and create and manage autoscaling rules.</td></tr><tr><td><strong>Admin</strong></td><td>Everything in User, plus edit configurations, manage users and accounts, configure the AWS IAM Role ARN, and generate API tokens.</td></tr></tbody></table>

Roles are assigned when a user is invited and can be changed at any time by an Admin.

## Authentication and Security

Authentication and security settings are configured through the Admin Portal, under **Security** and **SSO**. These settings are only available to Admins.

{% hint style="info" %}
If you have requirements not covered here — such as specific compliance standards or identity provider configurations — reach out and we'll help you find a solution.
{% endhint %}

### Single Sign-On (SSO)

Datafy supports SSO via **SAML** and **OpenID Connect (OIDC)**. You can connect your existing identity provider — such as Okta, Microsoft Entra ID (Azure AD), Google Workspace, or others — to allow users to log in with their corporate credentials.

To set up SSO, go to **Admin Portal → SSO** and select **Setup SSO Connection**. Pick the connection type and follow the on-screen wizard.

SSO is matched by **email domain**. Once configured, all users with a matching email domain are redirected to your identity provider. Which users are allowed to authenticate is controlled on the identity provider side — Datafy does not manage individual SSO access.

{% hint style="success" %}
If you have multiple accounts, configure SSO from the **organization view** rather than on each individual account. This ensures consistent authentication and avoids repeating the setup.
{% endhint %}

{% hint style="info" %}
During SSO setup, you'll be asked to validate your email domain by adding a DNS record. If your organization can't make DNS changes, contact us — we can validate the domain on your behalf.
{% endhint %}

### Other Security Controls

The Admin Portal's **Security** page exposes additional access controls:

* **Multi-factor authentication (MFA)** — Admins can make MFA optional (users can enable it themselves) or forced (all users must log in with MFA). Configure under **Security → MFA → Manage**.
* **Domain restrictions** — allow or block specific email domains from signing up or being invited. Configure under **Security → Restrictions → Manage**.
* **IP restrictions** — limit access by IP address. Configure under **Security → Restrictions → Manage**, then toggle **IP Address Restriction**.

## Programmatic Access

### API Tokens

Datafy uses bearer tokens to authenticate non-interactive access — agent installations, API calls, and Terraform-driven account management. Generate tokens from the Admin Portal under **API Tokens**.

Token scope depends on where you generate the token from:

<table><thead><tr><th width="162.68603515625">Generated from</th><th width="177.46875">Scope</th><th>Use for</th></tr></thead><tbody><tr><td>A specific sub-account</td><td>That sub-account only</td><td><a href="/pages/1pGlJGhiJ2dz7MwkzcqY">Agent installation</a>, API calls scoped to a single account</td></tr><tr><td>The organization view</td><td>The organization and all its sub-accounts</td><td>API calls across multiple accounts (with the <code>accounts=</code> parameter), <a href="#terraform-provider">Terraform-driven account management</a></td></tr></tbody></table>

See [Token Generation](/set-up-and-installation/datafy-installation/token-generation) for generation instructions, and the [API documentation](/resources/api#authentication) for full details on token scopes and the `accounts=` query parameter.

### Terraform Provider

The **`datafy` Terraform provider** lets you manage Datafy accounts, AWS IAM role associations, access tokens, and autoscaling rules programmatically. This is the recommended approach for organizations with large or dynamic numbers of AWS accounts.

The provider requires an [organization-level token](#api-tokens), since creating and managing accounts operates across the organization. Generate one from the organization view.

The provider and full documentation are available on the [Terraform Registry](https://registry.terraform.io/providers/datafy-io/datafy/latest).

{% hint style="info" %}
The `datafy` provider manages Datafy account resources (accounts, tokens, autoscaling rules). For reconciling Terraform state with Datafy-managed EBS volumes, see [IaC Reconciliation](/set-up-and-installation/iac-reconciliation), which covers the separate `datafyaws` provider.
{% endhint %}


# Managing Autoscaling Volumes

Identify eligible volumes, activate and deactivate autoscaling, and monitor grow and shrink actions in the Datafy app.

Once AutoScaler is [installed](/set-up-and-installation/datafy-installation) in your environment, you can start activating autoscaling on your volumes. This page walks through the day-to-day actions of working with autoscaling volumes - identifying which volumes are eligible, activating and deactivating autoscaling, and monitoring grow and shrink actions as Datafy optimizes your storage.

For a deeper look at how the autoscaling mechanism works under the hood, check out [How AutoScaler Works](/how-it-works/how-autoscaler-works). To walk through all of this on a test volume, follow [My First Autoscaling Volume](/resources/my-first-autoscaling-volume).

## Identifying Supported Volumes

To find which volumes in your account are eligible for autoscaling, open the [All Volumes](https://app.datafy.io/fleet-manager/all-volumes) table in Fleet Manager. The dropdown above the table switches between common views in a single click:

* **All Volumes** - all volumes in the current scope.
* **Autoscaling Volumes** - autoscaling volumes currently managed by Datafy.
* **Volumes to Activate** - supported volumes that aren't autoscaling yet and are ready to be activated. Start here to grow your autoscaling footprint.
* **Volumes Pending Setup** - volumes that are potentially eligible but need additional setup before they can be activated (most commonly, AutoScaler isn't installed on the instance).
* **Unsupported Volumes** - volumes that cannot be autoscaled. A gray label in the AutoScale column explains why. Common reasons include the volume being a boot volume, or an unsupported filesystem or volume type. For the complete list of supported configurations, see [Supported Infrastructure](/set-up-and-installation/supported-infrastructure).

{% hint style="info" %}
Updating the [configuration](/set-up-and-installation/autoscaling-configurations) or other changes to your setup can change a volume's support status.
{% endhint %}

<div data-with-frame="true"><figure><img src="/files/yMJ0G9uQ2Z7tmkJb9NY1" alt="" width="304"><figcaption></figcaption></figure></div>

For finer control, open the filter side panel — the **Autoscaling** (active vs. inactive) and **Autoscaling Eligibility** (supported, pending setup, unsupported) filters can be combined to produce any view, including ones the dropdown doesn't cover (these show as a **Custom** view).

{% hint style="info" %}
To retrieve each volume's autoscaling state and eligibility programmatically, use the [volume details endpoint](/resources/api#get-api-v1-volumes-details).
{% endhint %}

{% hint style="success" %}
If you're not sure which volumes to autoscale first, start with the largest volumes or those with the lowest utilization - these offer the highest savings potential. Sort the All Volumes table by **Original Size** or **Utilization** to find them.
{% endhint %}

<div data-with-frame="true"><figure><img src="/files/XLaAqecq32E0LNjRIUkq" alt=""><figcaption></figcaption></figure></div>

## Activating Autoscaling

There are two ways to activate autoscaling on eligible volumes:

1. **Manual Activation** - To activate autoscaling on a single volume, find it in the [All Volumes](https://app.datafy.io/fleet-manager/all-volumes) table and turn on the **AutoScale** toggle.
2. **Rule-Based Activation** - For environments with many volumes or dynamic behavior, you can define [autoscaling rules](/volume-lifecycle/autoscaling-rules) that automatically activate autoscaling on volumes matching specific criteria, such as volume tags, instance IDs, cluster names, or node groups. This is the recommended approach for managing autoscaling at scale.

Once autoscaling is activated for a volume, AutoScaler begins to copy the data from the original EBS volume to new, right-sized Datafy-managed volumes (see [How AutoScaler Works](/how-it-works/how-autoscaler-works) for more details).

The progress of the activation is indicated next to the AutoScale toggle of the volume, and in the Optimization Actions table in the Reports page. The time this takes depends on the amount of data on the volume and the volume's IOPS and throughput. Your applications continue running normally throughout - Datafy always gives precedence to your application's I/O over the activation process.

{% hint style="success" %}
Datafy offers different activation configurations optimized for different volume sizes and usage patterns. Contact us to discuss the best configuration for your environment.
{% endhint %}

{% hint style="info" %}
There is a [minimum volume size](/set-up-and-installation/autoscaling-configurations#autoscaling-activation-configurations) for activating autoscaling (default 50 GiB). Volumes close to the minimum will not shrink significantly. You can check your account's configured minimum in your [account settings](https://app.datafy.io/settings/configurations).
{% endhint %}

### Autoscaling Multiple Volumes

When autoscaling is activated for multiple volumes on the same instance - whether manually or through a rule - the volumes are copied sequentially, one at a time, to minimize the impact on your applications. Volumes on different instances are activated in parallel regardless of account size.

Each instance supports up to 8 autoscaling volumes at a time. If your instance has more than 8 volumes, we recommend selecting the largest among them. Additional volumes cannot be activated until one is deactivated.

### Tracking Environment-Wide Activation

The **Autoscaling Activation** widget at the top of the [Fleet Manager](https://app.datafy.io/fleet-manager/all-volumes) page tracks your activation progress across the account at a glance:

* **Managed Volumes** - the size and number of volumes autoscaling today.
* **Potentially Supported Volumes** - the total size and number of volumes that *could* autoscale in the current account scope, including those that still need setup before they can be activated.

The ring shows managed storage as a percentage of the total potentially supported - the closer to full, the more of your savings potential is activated.

<div data-with-frame="true"><figure><img src="/files/E3ldideHD9LDAWNh473Y" alt="Autoscaling Activation Widget" width="375"><figcaption><p>Autoscaling Activation Widget</p></figcaption></figure></div>

{% hint style="info" %}
The widget reflects your current account scope and updates as volumes are created or deleted, and as autoscaling is activated.
{% endhint %}

## Monitoring Autoscaling Activity

Once autoscaling is active, Datafy continuously monitors your volumes and performs grow and shrink actions as needed (see [How AutoScaler Works](/how-it-works/how-autoscaler-works) for details on how these operations work). You can track this activity on the [**Reports**](https://app.datafy.io/reports) page, and drill into individual volumes through the volume's details page.

### Tracking Grow and Shrink Actions

Every grow, shrink, activation, and deactivation is logged in the **Optimization Actions** table on the Reports page. Each entry shows the operation type, the volume size before and after, and the start and end times. The "Type" column indicates the progress of ongoing actions, and the final result for actions that failed or were canceled in the middle (for example, a shrink aborted because utilization increased).

{% hint style="warning" %}
Datafy grow actions will grow the underlying storage up to the filesystem's reported size, but not beyond it. If you need to enlarge your volume beyond this size, you will need to [modify](/volume-lifecycle/modify-a-volumes-parameters) its logical size.
{% endhint %}

Each action is also reflected in the **Usage** chart as a step change in the **Size** line - a drop after a shrink, a jump after a grow.

To track a specific volume, click its ID in the All Volumes table to open the volume's details page. The same Optimization Actions table and charts are available there, scoped to that individual volume.

Use the filters on the Reports page to narrow results by cluster, instance, volume, or tags, and set a date range or auto-refresh interval to follow active operations.

<div data-with-frame="true"><figure><img src="/files/vJJaGatLUgpLSNY2y7Bb" alt=""><figcaption><p>Optimization Actions table in the Reports page</p></figcaption></figure></div>

{% hint style="info" %}
For details on how grow and shrink thresholds are determined and how to tune them, see [AutoScaling Configurations](/set-up-and-installation/autoscaling-configurations).
{% endhint %}

### Understanding Your Savings

The gap between the original size of your volumes and their current size is storage Datafy has reclaimed, for which you are no longer paying.

#### Current Savings

The **Autoscaling Storage Size** widget at the top of the [Reports](https://app.datafy.io/reports) page shows the size of all of your autoscaling volumes - both their original and current sizes, and the percentage Datafy has reclaimed. This is a snapshot of the current state of the autoscaling volumes in your account.

For a dollar value estimation, check the **Monthly Savings** widget alongside it. It shows two figures:

* **Projected Monthly Savings** - your current realized savings, based on the volumes autoscaling today.
* **Potential Monthly Savings** - the full savings available if every eligible volume were autoscaled.

<div data-with-frame="true"><figure><img src="/files/vULZ3lAEHdGOqsJO9iJT" alt=""><figcaption><p>Autoscaling Storage Size and Monthly Savings Widgets</p></figcaption></figure></div>

{% hint style="info" %}
Both widgets reflect the current state of your account, and may fluctuate over time as volumes are created and deleted, or autoscaling is activated.
{% endhint %}

#### Savings over Time

To better understand your savings over time, use the **Usage** chart in the [Reports](https://app.datafy.io/reports) page (and in the details page of every volume). The chart displays three plot lines aggregated across all of your monitored[^1] volumes, according to the filters you currently have selected:

* **Used** - Actual data across your volumes.
* **Size** - Current provisioned capacity of the Datafy-managed volumes.
* **Original Size** - Total volume size before autoscaling was activated.

The area between **Size** and **Original Size** represents your total storage savings.

{% hint style="info" %}
Select the **Original Size** in the graph legend plot to hide it and zoom in on the actual size of your volumes.
{% endhint %}

The complementary **Utilization** chart shows the average disk usage percentage across monitored volumes over time. Volumes managed by Datafy will typically show higher utilization than before activation, since Datafy right-sizes the underlying storage to match actual usage.

<div data-with-frame="true"><figure><img src="/files/Ay4zPp9Gages2nhs7u4k" alt=""><figcaption><p>Utilization and Usage graphs of <strong>autoscaling volumes</strong></p></figcaption></figure></div>

<div data-with-frame="true"><figure><img src="/files/2S8rdC2AELnaSR8pLQKu" alt=""><figcaption><p>Utilization and Usage graphs of <strong>non-autoscaling volumes</strong></p></figcaption></figure></div>

{% hint style="info" %}
The same **Volume View** dropdown and **Autoscaling** / **Autoscaling Eligibility** filters from Fleet Manager are available here too, so you can easily view the size and usage of the different volume states.
{% endhint %}

### Autoscaling Volume Lifecycle

While a volume is autoscaling, Datafy manages it automatically - no manual intervention is needed for day-to-day operations.

Datafy is designed to be as seamless as possible in the day-to-day management of your storage. However, there are a few things that behave differently for autoscaling volumes that you need to be aware of:

* **Filesystem Mounting** - On EC2 instances, autoscaling will not be enabled for volumes that do not follow the [best practice](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-using-volumes.html) of using the filesystem UUID in the relevant `/etc/fstab` entry.

{% hint style="warning" %}
Mounting a volume using the device path (e.g., `/dev/sdf` or `/dev/nvme1n1`) can result in failure to mount an autoscaling volume after reboot or mounting the volume to the wrong mountpoint, and is not supported.
{% endhint %}

* **IaC Reconciliation** - If you manage your infrastructure with Terraform or other IaC tools, autoscaling volumes can cause drift and state reconciliation issues. See [IaC Reconciliation](/set-up-and-installation/iac-reconciliation) for setup instructions.
* **Modifying Volumes** - To modify the size, IOPS or throughput of autoscaling volumes use the Datafy API or CSI integration, as detailed in [Modifying Autoscaling Volumes](/volume-lifecycle/modify-a-volumes-parameters). Changing volume properties through the AWS Console or AWS API is not supported for autoscaling volumes.

{% hint style="danger" %}
Do not modify autoscaling volumes directly through AWS. This can cause unexpected behaviors in your volumes and is not supported.

If modifying volume properties is part of your regular workflow, contact us to discuss your use case.
{% endhint %}

* **Volume Snapshots** - To take snapshots of autoscaling volumes use the [Datafy API or the CSI integration](/volume-lifecycle/datafy-snapshots).

## Deactivating Autoscaling

To deactivate autoscaling, turn off the **AutoScale** toggle in the All Volumes table or on the volume's details page. Like when autoscaling is activated, the progress will be indicated in the AutoScale column and in the Reports page.

<div data-with-frame="true"><figure><img src="/files/ITexFuzw17SSXsnLFivk" alt=""><figcaption></figcaption></figure></div>

When deactivation completes, the new volume has a **new AWS EBS volume ID**, but it retains the original volume's tags and is attached to the same device path and mountpoint. The Datafy virtualization layer is removed and the Datafy-managed volumes are deleted. From this point on, the volume operates as a normal EBS volume, fully independent of Datafy.

{% hint style="warning" %}
A reboot is required after deactivation to ensure clean filesystem state and proper device mapping. Your data is accessible immediately after the copy completes, but the reboot finalizes the transition.

Re-activating autoscaling for a specific volume will be blocked until the instance has been rebooted, or the pod using it has been rescheduled to another node (in K8s).
{% endhint %}

If you'd like to programmatically deactivate autoscaling for your volumes, use the [`deactivate-autoscaling` endpoint](/resources/api#post-api-v1-volumes-volumeid-deactivate-autoscaling).

### Exceptions from Autoscaling Rules

When autoscaling is deactivated for a volume, it is automatically marked as an [exception](/volume-lifecycle/autoscaling-rules#exceptions-from-rules) to all autoscaling rules. This prevents any rule from re-activating autoscaling on that volume without your explicit action. If you want a rule to apply to the volume again in the future, you can remove the exception through the [rule's detail panel](/volume-lifecycle/autoscaling-rules#clearing-exceptions).

[^1]: All volumes that have Datafy Autoscaler or Sensor installed and are actively reporting their usage


# Datafy Snapshots

{% hint style="info" %}
Datafy Snapshots are supported from version `1.30.0`!\
To restore volumes from Datafy Snapshots make sure you have the correct version installed before attaching the new volumes
{% endhint %}

## Datafy Snapshot Lifecycle

1. [**Create a Datafy Snapshot**](#creating-a-datafy-snapshot) - use the Datafy [`create-snapshot`](/resources/api#post-api-v1-volumes-volumeid-create-snapshot) API instead of the AWS `CreateSnapshot` API.
2. [**Manage Retention**](#snapshot-retention) - use your existing logic with the relevant volume tags to manage the retention of Datafy Snapshots.
3. [**Restoration**](#restoring-a-volume-from-a-datafy-snapshot) - use the Datafy [`create-volume-from-snapshot`](/resources/api#post-api-v1-volumes-create-from-snapshots) and [`attach`](/resources/api#post-api-v1-volumes-volumeid-attach) endpoints to create and attach a new autoscaling volume.

### Creating a Datafy Snapshot

To create a new Datafy Snapshot, call the [`create-snapshot` endpoint](/resources/api#post-api-v1-volumes-volumeid-create-snapshot) with the volume ID of the original volume managed by Datafy (as it appears in the Datafy app). Call the Datafy API wherever you currently call the AWS `CreateSnapshot` API.

{% hint style="info" %}
Datafy snapshots need to be created directly, and cannot be created using AWS Data Lifecycle Manager (DLM)
{% endhint %}

Datafy Snapshots will create a separate snapshot for each underlying Datafy volume associated with the autoscaling volume. These snapshots have tags that identify them as Datafy snapshots, and help track their lineage. You can use these tags to identify the original volume the snapshot is associated with, and to identify all of the snapshots that were created together and are associated with each other.

{% hint style="success" %}
Datafy snapshots work for any volume managed by Datafy, including unattached volumes
{% endhint %}

<details>

<summary>Datafy Snapshot Tags</summary>

<table><thead><tr><th width="270.40985107421875">Tag</th><th>Description</th></tr></thead><tbody><tr><td><code>Managed-By: Datafy.io</code></td><td>Identifies the snapshot as managed by Datafy</td></tr><tr><td><code>datafy:source-volume:id</code></td><td>Volume ID of the original volume before autoscaling. The volume ID associated with the snapshot by AWS is the smaller volume after autoscaling.</td></tr><tr><td><code>datafy:snapshot:id</code><br></td><td>A snapshot ID generated by Datafy (designated with the prefix <code>dsnap</code>) , shared by all of the related snapshots created when snapshotting an autoscaling volume.</td></tr><tr><td><code>datafy:snapshot:generation-id</code></td><td>Generation number of the Datafy snapshot. The first snapshot taken during activation is generation 0, and every following snapshot the generation is increased by 1</td></tr></tbody></table>

</details>

### Snapshot Retention

Snapshots created by Datafy are stored in your AWS account like regular EBS snapshots, and their retention should be handled by your existing retention tools and policies, with the following adjustments:

* Use the `datafy:source-volume:id` tag to determine the volume ID of the original non-autoscaling volume associated with the snapshot.

If you use a backup management tool to manage snapshot retention as part of a larger backup object, check out how to integrate with backup management tools [below](#integration-with-backup-managers).

{% hint style="info" %}
Be aware that AWS Data Lifecycle Manager (DLM) will not automatically manage snapshots created by third-party tools, including Datafy. You must apply your own logic to clean up these snapshots.
{% endhint %}

### Restoring a Volume from a Datafy Snapshot

Datafy snapshots represent the underlying volumes created by Datafy when a volume is autoscaling. As such, they can be restored to autoscaling volumes with the following adjustments to your restoration flow:

1. Use the tags `datafy:source-volume:id` and `datafy:snapshot:id` and the snapshot timestamp to find all of the snapshots for the volume and time you need.
2. After selecting the desired snapshots and making sure you have all of the snapshots with the same `datafy:snapshot:id`, create a new autoscaling volume using the Datafy [`create-volume-from-snapshot`](/resources/api#post-api-v1-volumes-create-from-snapshots) endpoint.
3. Attach the new autoscaling volume to an EC2 instance with Datafy AutoScaler installed using the Datafy [`attach`](/resources/api#post-api-v1-volumes-volumeid-attach) endpoint. Mount the volume as you would any EBS volume.

#### Offline Restoration

Datafy snapshots are intended to be restored to an instance with Datafy AutoScaler installed and access to the Datafy API. For emergencies where this isn't possible, we provide local offline tools that enable access to the data saved in Datafy snapshots without any additional dependencies.

{% columns %}
{% column %}
{% content-ref url="/pages/LRYOScr01Q3bvz6MaaBn" %}
[Offline Restoration - Autoscaling](/volume-lifecycle/datafy-snapshots/offline-restoration-autoscaling)
{% endcontent-ref %}

* For restoring Datafy snapshots when the Datafy API is unavailable
  {% endcolumn %}

{% column %}
{% content-ref url="/pages/myH4NbhfZafh0pDaZimO" %}
[Offline Restoration - Non Autoscaling](/volume-lifecycle/datafy-snapshots/offline-restoration-non-autoscaling)
{% endcontent-ref %}

* For complete offline restoration of Datafy snapshots, without Datafy Autoscaler
  {% endcolumn %}
  {% endcolumns %}

## Integration with Backup Managers

Datafy snapshots can be integrated into the workflows of your existing backup management tools. They can be used seamlessly with any backup tool that uses the K8s CSI, and with minimal integration with custom backup systems.

{% hint style="info" %}
Use a different type of backup manager? Ask us about integration with other tools!
{% endhint %}

### Kubernetes CSI Integration

Datafy integrates directly with the [Kubernetes CSI Snapshotter](https://kubernetes.io/docs/concepts/storage/volume-snapshots/), allowing Kubernetes-native backup tools like Velero to seamlessly create snapshots of Datafy-managed volumes. For an overview of how Datafy extends the CSI driver in your cluster, see [AutoScaler on Kubernetes](/how-it-works/autoscaler-on-kubernetes). The resulting snapshots are associated with specific PVCs, and can be used directly in the rest of the backup and restoration flow.

To enable this integration:

{% stepper %}
{% step %}
Make sure your cluster is set up for taking snapshots through the CSI:

* The [`snapshot-controller`](https://docs.aws.amazon.com/eks/latest/userguide/csi-snapshot-controller.html) EKS add-on is installed on your cluster, and the [`csi-snapshotter`](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) sidecar is installed on your cluster's CSI-controller pod
* An appropriate [`VolumeSnapshotClass`](https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes/) resource is defined in the cluster

{% hint style="info" %}
You can test this by creating a snapshot directly through Kubernetes, by creating a `VolumeSnapshot` resource
{% endhint %}

<details>

<summary>VolumeSnapshot Examples</summary>

{% code title="Define a VolumeSnapshotClass" %}

```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: aws-ebs-snapshot-class
driver: ebs.csi.aws.com
deletionPolicy: Delete
```

{% endcode %}

{% code title="Create a Snapshot" %}

```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: ebs-volume-snapshot
spec:
  volumeSnapshotClassName: aws-ebs-snapshot-class
  source:
    persistentVolumeClaimName: aws-ebs-volume
```

{% endcode %}

</details>
{% endstep %}

{% step %}
Configure your backup tool to use the CSI Snapshotter (and not the AWS native snapshotter).

When a snapshot is triggered, it will call the Datafy API in the background for Autoscaling volumes.
{% endstep %}
{% endstepper %}

### Custom Backup Systems

If you use a custom backup manager or automation script, adjust your logic to use Datafy snapshots at each part of the [snapshot lifecycle](#datafy-snapshot-lifecycle).

1. **Snapshot creation** - Use the [Datafy API](/resources/api#post-api-v1-volumes-volumeid-create-snapshot) instead of the AWS snapshot API to create snapshots of AutoScaling volumes.
2. **Snapshot retention** - Retention policies that target all snapshots in the account will apply to Datafy snapshots as well. If you use targeted retention policies, make sure to apply them to snapshots with the tag `Managed-By: Datafy.io`.
3. **Restoration from snapshots** - As detailed [above](#restoring-a-volume-from-a-datafy-snapshot), restoration of volumes from Datafy snapshots follows a similar process as restoring volumes from regular EBS snapshots. Your restoration logic should:
   1. Ensure that Datafy AutoScaler is installed on the instances the new volumes will be attached to.
   2. Identify the relevant snapshots using the `datafy:source-volume:id` and `datafy:snapshot:id` tags, and use the Datafy API instead of AWS to [create new volumes](/resources/api#post-api-v1-volumes-create-from-snapshots) from the snapshots and to [attach](/resources/api#post-api-v1-volumes-volumeid-attach) them.


# Offline Restoration - Autoscaling

Locally create a new autoscaling volume from Datafy snapshots

The standard flow for restoring Datafy snapshots is through the Datafy `create-from-snapshots` and `attach` endpoints. In emergency cases when these endpoints are not available, you can use the manual attach tool to access data from volumes created directly from Datafy snapshots.

The tool does not require any permissions on your AWS account.

{% hint style="warning" %}
The manual attach tool is intended to enable snapshot recovery in cases of emergency, and should not be used when the `create-from-snapshots` and `attach` endpoints are available

See [#limitations-of-volumes-that-were-manually-attached](#limitations-of-volumes-that-were-manually-attached "mention")
{% endhint %}

## Manual Attachment of Restored Datafy Snapshots

Before you get started, make sure you meet the following conditions:

1. Access to snapshots in the AWS console
2. SSH access and root/sudo permissions on an instance with Datafy Autoscaler installed where the volumes will be attached

{% stepper %}
{% step %}

#### Create New Volumes from Snapshots

* For a given source volume ID and desired timeframe, find all of the snapshots that have a matching `datafy:source-volume:id` tag and timestamp. All the snapshots should have the same `datafy:snapshot:id` tag, and all of the snapshots with this tag should be used.
* Create new volumes from these snapshots through AWS, and attach them to the desired instance.
  {% endstep %}

{% step %}

#### Download and Run the Attach Tool

* Download and run the attach tool using the following command. Provide the volume IDs and device names of your volumes.

  <pre class="language-bash" data-overflow="wrap"><code class="lang-bash">curl -sSfL https://artifacts.datafy.io/attach_tool | sh -s -- vol-04d316aeec130176a /dev/nvme1n1 vol-07b9a47caa69e90e2 /dev/nvme2n1
  </code></pre>

  <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>You can find the device name by running the <code>lsblk</code> command</p></div>

  \
  You will receive a new device path that can be used to access a single autoscaling volume with access to all of the data from the restored snapshots

  <pre data-overflow="wrap"><code>Generated source volume id: vol-01afc3a269acebaf9
  Volume attached successfully. New device path: /dev/dvol01afc3a269acebaf9
  </code></pre>

  * You can also download and run the tool in 2 steps using:

    ```bash
    curl -sSfL https://artifacts.datafy.io/attach_tool | sh
    ```

    and then:

    <pre class="language-bash" data-overflow="wrap"><code class="lang-bash">sudo ./attach_tool vol-04d316aeec130176a /dev/nvme1n1 vol-07b9a47caa69e90e2 /dev/nvme2n1
    </code></pre>

{% endstep %}

{% step %}

#### Set Up the Volume

Mount the volume using the device path provided by the attach tool output

```shellscript
mount /dev/dvol01afc3a269acebaf9 /mnt/data
```

Your volume is now set up and ready to use!
{% endstep %}
{% endstepper %}

### Limitations of Volumes That Were Manually Attached

{% hint style="warning" %}
Autoscaling Volumes that were created and attached using the flow above will have limited functionality until they are registered with Datafy. The volumes will continue to grow when needed, and the data on them will remain available.
{% endhint %}

Until your new volume is registered with Datafy, it will operate under the following limitations:

* **Appearance**: The volume will not appear in the Datafy app.
* **Autoscaling**: The volume will continue to grow, but will not automatically shrink
* **Additional Actions**: API actions for the volume will not be supported

{% hint style="success" %}
To restore full functionality, please contact us to register any volumes created this way.
{% endhint %}


# Offline Restoration - Non Autoscaling

Locally convert Datafy snapshots to a regular non-autoscaling volume

New volumes that are created directly from a Datafy snapshot cannot be read without Datafy AutoScaler installed. If you wish to access the data on these volumes when AutoScaler cannot be installed, you can use the Datafy restoration tool to copy the data to a new, full-sized volume (similarly to when autoscaling is deactivated by AutoScaler). After this, the resulting volume can be used as-is for any purpose.

The tool does not require any permissions on your AWS account, and only performs local copy operations.

{% hint style="warning" %}
The offline restoration tool is intended to give access to data from a Datafy snapshot in cases of emergency, and is not intended for routinely creating a non-autoscaling volume from Datafy snapshots. For this case, restore your snapshots through the [standard flow](/volume-lifecycle/datafy-snapshots#restoring-a-volume-from-a-datafy-snapshot) and then [deactivate autoscaling](/volume-lifecycle/managing-autoscaling-volumes#deactivating-autoscaling) for the new volume.
{% endhint %}

## Offline Restoration of Datafy Snapshots

Before you get started, make sure you meet the following conditions:

1. Access to snapshots in the AWS console
2. SSH access and root/sudo permissions on a Linux instance where the volumes will be attached

{% stepper %}
{% step %}

#### Create New Volumes from Snapshots

* For a given source volume ID and desired timeframe, find all of the snapshots that have a matching `datafy:source-volume:id` tag and timestamp. All the snapshots should have the same `datafy:snapshot:id` tag, and all of the snapshots with this tag should be used.
* Create new volumes from these snapshots through AWS, and attach them to the desired instance.
  {% endstep %}

{% step %}

#### Set Up Restoration Tool

Download and run the restoration tool, on the instance the volumes are attached to. The downloader automatically retrieves the latest version that matches the instance type and architecture.

{% code overflow="wrap" %}

```bash
curl -sSfL https://artifacts.datafy.io/restore_tool | sh
```

{% endcode %}

To install a specific version, use

{% code overflow="wrap" %}

```bash
 curl -sSfL https://artifacts.datafy.io/restore_tool | sh -s -- --version <version>
```

{% endcode %}
{% endstep %}

{% step %}

#### Create a New Full-Sized Volume

1. Create a new volume, with a size equal to the original size of the snapshots' source volume (the size of the filesystem). You can find this original size in several ways:
   * If the original volume still exists:
     * If it is autoscaling, you can view its size in the Datafy app or check the size of its filesystem.
     * If it is not autoscaling, you can check its size in the AWS console as well
   * Save the size returned when creating snapshots
   * Use the restoration tool to check the required size, with:

     ```bash
     sudo ./run_restore_tool.sh \
       --restoration_source_volume_ids vol-source456 vol-source789 \
       --get_size
     ```

     \
     The required size will be printed to the console, for example:

     ```
     ✅ Required destination volume size: 100.00 GB
        Exact size in bytes: 107,374,182,400
        Chunk count: 3,276,800
        Chunk size: 32.00 KB
        RAID stripe size: 1.00 MB
     ```
2. Attach the new volume to your instance
   {% endstep %}

{% step %}

#### Complete Restoration

* Run the restoration tool, providing the IDs of the restored and new volumes. You can identify the volumes using their IDs (preferred) or their device names

{% tabs %}
{% tab title="Volume IDs" %}
{% code overflow="wrap" %}

```bash
sudo ./run_restore_tool.sh \
  --new_vol_id vol-destination123 \
  --restoration_source_volume_ids vol-source456 vol-source789
```

{% endcode %}
{% endtab %}

{% tab title="Device name" %}

```bash
sudo ./run_restore_tool.sh \
  --devices /dev/nvme0n1 /dev/nvme1n1 \
  --output /dev/nvme2n1
```

{% endtab %}
{% endtabs %}

* Monitor the progress of your restoration in the console output
* After the restoration is complete, mount the full-sized volume, and delete the volumes created from the snapshots
  {% endstep %}
  {% endstepper %}


# Autoscaling Rules

{% hint style="info" %}
Autoscaling rules are supported from AutoScaler version `1.31.0`
{% endhint %}

Once Autoscaler is [installed](/set-up-and-installation/datafy-installation) and you've selected which volumes to optimize, the next step is to activate autoscaling: you can do this manually for each volume via the [Fleet Manager](https://app.datafy.io/fleet-manager/all-volumes), or set up an [autoscaling rule](https://app.datafy.io/fleet-manager/autoscale-rules).

Autoscaling rules are templates that identify volumes matching specific conditions and automatically enable autoscaling for them. Once activated, a rule applies to all existing and future matching volumes. This is particularly beneficial for initial autoscaling setup in large environments and continuous management in dynamic environments, such as EKS clusters, where volumes are frequently created and destroyed.

## Create a New Autoscaling Rule

Autoscaling rules can be created simply and easily through the [Datafy app](https://app.datafy.io/fleet-manager/autoscale-rules).

To get started, open the rule creation wizard, by selecting the "Create new rule" button:

<figure><img src="/files/bm81UbLCihGI85g3nlNg" alt="" width="178"><figcaption></figcaption></figure>

{% hint style="success" %}
You can automate rule creation and management across multiple accounts using the `datafy` Terraform provider. For details see the [autoscaling\_rule resource](https://registry.terraform.io/providers/datafy-io/datafy/latest/docs/resources/autoscaling_rule) in the Terraform registry.

Interested but don't use Terraform? Let us know!
{% endhint %}

{% stepper %}
{% step %}

#### Define Rule Conditions

Each autoscaling rule is built from **conditions** that define which volumes it applies to. A volume will match a rule if it meets all of the conditions in the rule (logical AND).

Each condition contains a volume attribute, a logical operator and at least one value.

<details>

<summary>Volume Attribute</summary>

Select one of the following volume attributes for the condition to check:

* Cluster - cluster name, use to apply autoscaling to all volumes in a cluster
* Node group - node group name, use to apply autoscaling to certain node groups within a cluster
* Instance ID - use to apply to or exclude volumes attached to specific instance. This is not recommended for dynamic environments where volumes may move between EC2 instances.
* Volume Tags - array of volume tags, with each tag formatted as a single string `"key:value"`\ <br>

<figure><img src="/files/qYRKkekjwfGUuzZAdxPx" alt="" width="188"><figcaption></figcaption></figure>

</details>

<details>

<summary>Logical Operator</summary>

Define which operator will be used to check the selected attribute:

* **Is** – the value must match exactly
* **Is not** – the value must not match
* **Contains** – the value must include the specified text
* **Does not contain** – the value must not include the specified text

<figure><img src="/files/GwqX8TyyZntROIVIWWSW" alt="" width="188"><figcaption></figcaption></figure>

{% hint style="info" %}
For the volume tags array, the *is* and *contains* conditions check that at least one tag matches the defined values, and the *is not* and *does not contain* conditions checks that none of the tags match the defined values
{% endhint %}

</details>

<details>

<summary>Values</summary>

Use the search to find values of the existing attributes in your account. For conditions with exact match operators (*is* or *is not*) an exact value must be selected from the list, for conditions with *contains* or *does not contain* any value can be used.

If you add multiple values to a condition, a volume will match it if the attribute has any one of the defined values (logical OR).

<figure><img src="/files/Ms5mjHGxnMOBr9RCj8Jz" alt=""><figcaption><p>Representation of "<em>Cluster name contains <code>dev</code></em> <em><strong>or</strong></em> <em><code>prod</code>"</em></p></figcaption></figure>

</details>

You can add additional conditions to your rule with the "+And" and "+Add type" buttons.

<figure><img src="/files/FadxYPkt2b40kljOYUsU" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Check Matching Volumes

Once your conditions are defined, select the "Next" button to see which volumes match the rule.

The matching volumes are presented in three categories:

1. Matching volumes - volumes that matched the rule and can start autoscaling right away.
2. Potential volumes - volumes that matched the rule but don't have AutoScaler installed. These will start autoscaling once AutoScaler is installed on their instance.
3. Excluded - volumes that match the rule but are [excluded from autoscaling rules](#exceptions-from-rules).

{% hint style="info" %}
Volumes that match the rule but do not support autoscaling (for example, unattached volumes) will not appear.
{% endhint %}

<figure><img src="/files/y7Ls1fSpxHCEP0p1RH9Q" alt=""><figcaption></figcaption></figure>

If you find volumes you didn't intend to include, go back and continue editing your rule until you reach the desired result.
{% endstep %}

{% step %}

#### Create Rule

Once you're satisfied with the rule conditions and the volumes it will apply to, create the rule:

* "Create" button - your rule will be created in an inactive state. The rule will not activate autoscaling until it is turned on.
* "Create and Apply" - your rule will be created and start activating autoscaling for the matching volumes immediately.
  {% endstep %}
  {% endstepper %}

## Monitor and Manage Rules

### Activation Progress

After you create and apply a rule, autoscaling is activated for all matching volumes. To view these volumes and their status, select the rule in the [AutoScale Rules](https://app.datafy.io/fleet-manager/autoscale-rules) table and visit the "Matching Volumes" tab. You can monitor the live progress of each autoscaling activation event in the Optimization Actions on the [Reports](https://app.datafy.io/reports) page and track actual storage savings in the Usage and Utilization charts on the same page.

{% hint style="success" %}
When autoscaling is activated, volumes attached to the same instance are activated sequentially to avoid impacting your applications. The data copy runs locally on the relevant instance, so any number of volumes across your account can be activated at the same time.
{% endhint %}

### Editing Rules

All rules in your account appear in the [AutoScale Rules](https://app.datafy.io/fleet-manager/autoscale-rules) table, summarizing the number and capacity of volumes matching each rule. By selecting a rule, you can view its full conditions and list of matching volumes, divided into matching, potential, and [exceptions](#exceptions-from-rules).

From here you can:

* Edit a rule's conditions
* Disable or enable a rule
* Delete a rule

{% hint style="warning" %}
Changes to a rule (including disabling or deleting) do not reverse previously executed autoscaling activations - those volumes will continue to autoscale until [directly deactivated](/volume-lifecycle/managing-autoscaling-volumes#deactivating-autoscaling).
{% endhint %}

### Exceptions from Rules

Some volumes are defined as exceptions from the autoscaling rules - even if they match, rules will not be applied to them.

Volumes that have an exception are identified in the All Volumes table, and are listed as exceptions to the specific rules that apply to them.

<figure><img src="/files/5VHFmxYRtmLN05XULvJv" alt="" width="124"><figcaption></figcaption></figure>

{% hint style="info" %}
Exceptions are from all autoscaling rules, not from any specific rule
{% endhint %}

#### Creating an Exception

Volumes are defined as exceptions in one of two ways:

1. When [autoscaling is deactivated](/volume-lifecycle/managing-autoscaling-volumes#deactivating-autoscaling) for a volume, it is automatically made an exception to the rules. This prevents any rule from re-activating the autoscaling.
2. When viewing the details of an existing rule, select any matching volume and create an exception for it. This can be done for active and inactive rules, and matching or potential volumes.

<figure><img src="/files/cQLrbnYFigWKFX80fXoo" alt="" width="375"><figcaption></figcaption></figure>

{% hint style="warning" %}
Direct exceptions of specific volumes should be used to handle special cases. To prevent undesired autoscaling activation of a category or group of volumes, make sure to add exclusive conditions to rules when they are created.
{% endhint %}

#### Clearing Exceptions

If you want to clear an exception from a volume, find it in the exceptions list of a rule that applies to it, and select the "Remove Exception" button. If the relevant rule is active and the volume is attached to an instance with AutoScaler installed, autoscaling will be automatically activated for the volume.

<figure><img src="/files/jIKW9f6fJvD62upO1JNw" alt="" width="563"><figcaption></figcaption></figure>

{% hint style="success" %}
You don't need to clear exceptions to activate autoscaling for a specific volume, you can [activate](/volume-lifecycle/managing-autoscaling-volumes#activating-autoscaling) it directly from the [All Volumes](https://app.datafy.io/fleet-manager/all-volumes) table.
{% endhint %}


# Modifying Autoscaling Volumes

{% hint style="success" %}
Modifying the size of autoscaling volumes is supported from agent version 1.36.0. Modifying IOPS and throughput is supported from agent version 1.40.0.
{% endhint %}

## Modifying an Autoscaling Volume

You can modify an autoscaling volume's size, IOPS, and throughput. There are two options:

1. [On EC2](#volumes-attached-to-ec2-instances) - Use the Datafy [modify](/resources/api#post-api-v1-volumes-volumeid-modify) API instead of the AWS modify API
2. [In Kubernetes](#pvcs-in-kubernetes) - Edit the PVC that the volume is associated with, or apply a VolumeAttributesClass to it

{% hint style="danger" %}
Do not modify autoscaling volumes directly through the AWS console or API. This can cause unexpected behaviors in your volumes and is not supported.
{% endhint %}

{% hint style="info" %}
After autoscaling has been deactivated for a volume, you can use the AWS `ModifyVolume` API directly. As always, a reboot is recommended after [deactivation](/volume-lifecycle/managing-autoscaling-volumes#deactivating-autoscaling) in order to ensure expected behavior, especially if also uninstalling AutoScaler.
{% endhint %}

### Volumes Attached to EC2 Instances

Call the Datafy [`modify`](/resources/api#post-api-v1-volumes-volumeid-modify) endpoint with the volume ID of the original volume managed by Datafy (as it appears in the [Datafy app](https://app.datafy.io)). You should call the Datafy API wherever you currently call the AWS `ModifyVolume` API.

Pass any combination of `volumeSizeGb`, `volumeIops`, and `volumeThroughput` - at least one is required. Datafy applies the requested changes and leaves the other parameters unchanged.

When the request changes the size of the volume, provide your application with access to the new size by expanding the filesystem as well:

1. Manually - after the modification completes, resize the filesystem like you would any volume.
2. Automatically - for autoscaling volumes not currently undergoing activation or deactivation, pass `expandFilesystem=true` in your modify request and the AutoScaler agent will expand the filesystem for you.

{% hint style="info" %}
`expandFilesystem` applies only to a size change, so it requires `volumeSizeGb`. Changing IOPS or throughput never affects the filesystem.
{% endhint %}

{% hint style="info" %}
A success response means the request was accepted and the modify is in progress. Check the size of the volume or use the [volume details endpoint](/resources/api#get-api-v1-volumes-details) to confirm the modify has completed.
{% endhint %}

### PVCs in Kubernetes

Datafy integrates directly with the [Kubernetes CSI volume expansion](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims) flow and modifications via a [VolumeAttributesClass](https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/). Modifying the PVC works in the same way as for any other EBS-backed PVC - edit the relevant attribute and the CSI driver applies the change. For autoscaling volumes, Datafy's CSI integration intercepts and handles the request.

* **Size** - Edit the PVC's `spec.resources.requests.storage` to change its size.

  No additional configuration is required beyond the standard Kubernetes prerequisites for PVC expansion (`allowVolumeExpansion: true` on the StorageClass).

  <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>For consistent results, use <code>Gi</code> units when defining volume sizes</p></div>
* **IOPS and throughput** - Apply a VolumeAttributesClass to the PVC. Datafy's CSI integration accepts the `iops` and `throughput` parameters, for example:

  ```yaml
  apiVersion: storage.k8s.io/v1beta1
  kind: VolumeAttributesClass
  metadata:
    name: datafy-fast
  driverName: ebs.csi.aws.com
  parameters:
    iops: "6000"
    throughput: "250"
  ```

  Reference it from the PVC's `spec.volumeAttributesClassName` to apply it. To change the values later, create a new VolumeAttributesClass and point the PVC at it - the parameters of an existing class are immutable.

  <div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><p>Only <code>iops</code> and <code>throughput</code> can be modified this way. Any other parameter in the class is rejected. Setting IOPS or throughput through the PVC annotations used by <code>volume-modifier-for-k8s</code> is not supported for autoscaling volumes.</p></div>

{% hint style="info" %}
Editing the PVC's size and its VolumeAttributesClass together applies both changes in a single modification.
{% endhint %}

{% hint style="info" %}
Kubernetes retries failed expansion and modification requests indefinitely until they succeed or the PVC is edited back. While a change is in progress, the PVC reports it as in progress and Kubernetes keeps retrying - this is expected. Errors from individual attempts are available in the CSI controller pod logs.
{% endhint %}

{% hint style="warning" %}
Modifying autoscaling volumes works only on attached volumes. If a PVC resize or performance modification is issued while the underlying volume is detached - for example, during pod rescheduling - the request will fail. The request will be retried by Kubernetes and applied once the volume is reattached.
{% endhint %}

## What Happens When a Volume Is Modified

### When Size Is Modified

Modifying the size of an autoscaling volume will increase its **Original Size** - the size the volume would be without Datafy, as shown in the Datafy app. The **Current Size** will change depending on the state of the volume:

* **If the volume's current size is smaller than its Original Size** - Only the Original Size updates. The autoscaling logic will use the new Original Size as the maximum size when growing the volume as usage increases. In this state, no cooldown limitations apply, and you can modify the volume size at any time.
* **If the volume has already grown to its full size** - The Original Size updates, and the current size increases to keep the volume at full size. This involves modifying the underlying Datafy volumes, so the [AWS cooldown](https://docs.aws.amazon.com/ebs/latest/userguide/modify-volume-requirements.html) applies. A subsequent modify at full size will only succeed once the cooldown has elapsed, or after a [shrink](/how-it-works/how-autoscaler-works#shrinking) action (when the underlying volumes are replaced).

### When IOPS or Throughput Is Modified

The IOPS and throughput of each of the underlying Datafy Volumes will be modified to provide the new values that were requested.

Because the underlying Datafy Volumes are always modified, the [AWS cooldown](https://docs.aws.amazon.com/ebs/latest/userguide/modify-volume-requirements.html) always applies to an IOPS or throughput change - unlike a size change below the volume's Original Size. A subsequent modify will only succeed once the cooldown has elapsed, or after a [shrink](/how-it-works/how-autoscaler-works#shrinking) action (when the underlying volumes are replaced).

Supported values for autoscaling volumes:

| Parameter  | Range            |
| ---------- | ---------------- |
| IOPS       | 3,000 - 16,000   |
| Throughput | 125 - 1,000 MB/s |

{% hint style="info" %}
While the underlying Datafy Volumes are optimizing, the volume delivers somewhere between its original and its new IOPS and throughput, exactly as AWS specifies for any modified volume. Optimizing can take time, so allow for it before you expect the new values in full.
{% endhint %}

### Interaction with Ongoing Actions

A modify request interrupts any ongoing autoscaling activation, deactivation, or shrink, so your change is applied right away: AutoScaler stops the action mid-flight, then applies the modify. To resume the action after the modify completes:

* **Autoscaling Activation** - the volume will return to a non-autoscaling state. Re-enable autoscaling by turning the AutoScale toggle on for the volume, or by [clearing the exception](/volume-lifecycle/autoscaling-rules#clearing-exceptions) from autoscaling rules if they apply to the volume.
* **Autoscaling Deactivation** - the volume is still autoscaling. [Deactivate autoscaling](/volume-lifecycle/managing-autoscaling-volumes#deactivating-autoscaling) again with the AutoScale toggle or API.
* **Shrink** - the volume stays at its pre-shrink Current Size. No action is needed, shrink will begin again automatically if [conditions](/how-it-works/how-autoscaler-works) match.

{% hint style="success" %}
To check whether a volume has an ongoing action before you modify it, use the Optimization Actions table in the [Reports](https://app.datafy.io/reports) page or in the volume's details page, or use the [volume details endpoint](/resources/api#get-api-v1-volumes-details).

Ongoing grow actions will be completed before the modify takes place.
{% endhint %}


# API

## Authentication

All Datafy endpoints use a bearer token for authentication. Pass the token in the `Authorization` header:

```
Authorization: Bearer <token>
```

API tokens can be [generated](/set-up-and-installation/datafy-installation/token-generation) by an admin in the Datafy app and are scoped to the account from which they were created.

### Account Tokens

Account tokens are scoped to a single sub-account. Requests authenticated with an account token automatically target that account, no additional identifier is needed.

### Organization Tokens

Organization tokens are scoped to your organization and can target any sub-account within it. Generate an organization token the same way as an account token, by [generating a token](/set-up-and-installation/datafy-installation/token-generation) while logged in to an organization account in the Datafy app.

Specify the target sub-account using the `accounts` query parameter:

* For single-resource endpoints (e.g. create snapshot, attach, deactivate autoscaling) pass a single sub-account ID:

  ```
  ?accounts=<sub-account-id>
  ```
* For list or aggregate endpoints pass a comma-separated list of sub-account IDs, or `all` to include every sub-account in your organization. The `accounts` parameter is required if using an organization token on list and aggregate endpoints.

  ```
  ?accounts=<sub-account-id>,<sub-account-id>,...
  ?accounts=all
  ```

## Volumes

## Volumes details

> Retrieve a list of Datafy-managed volumes with their current size,\
> attachment, workload, autoscaling state, ongoing actions, and recent\
> utilization.\
> \
> Supports filtering, sorting, and pagination.<br>

```json
{"openapi":"3.0.0","info":{"title":"Datafy API","version":"1.0.0"},"tags":[{"name":"Volume-Lifecycle","description":"Endpoints for managing the lifecycle of storage volumes"}],"servers":[{"description":"Datafy API","url":"https://api.datafy.io"}],"security":[{"Authorizer":[]}],"components":{"securitySchemes":{"Authorizer":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}},"parameters":{"AccountsQueryParam":{"in":"query","name":"accounts","description":"Target account(s) for the request.\nOn list/aggregate endpoints: a comma-separated list of sub-account ids, or \"all\" to\ninclude every sub-account of the calling org. Required for org users on these endpoints.\nOn single-resource endpoints: at most one sub-account id. Multiple values or \"all\"\nare silently ignored and the request runs against the caller's effective account.\n","required":false,"schema":{"type":"array","items":{"type":"string"}}}},"schemas":{"VolumeDetails":{"type":"object","description":"Datafy volume with current size, attachment, workload, autoscaling state, ongoing action, and recent utilization.","properties":{"accountId":{"type":"string","description":"Datafy account ID"},"volumeId":{"type":"string","description":"EBS volume identifier"},"underlyingVolumeIds":{"type":"array","items":{"type":"string"},"description":"Underlying Datafy-managed volumes"},"volumeName":{"type":"string","description":"Volume name (from the `Name` tag or discovered name)"},"diskType":{"type":"string","description":"EBS volume type (gp2, gp3, io1, io2)"},"originalSize":{"type":"integer","format":"int64","description":"Original provisioned size in bytes"},"currentSize":{"type":"integer","format":"int64","description":"Actual current size in bytes of the volume. Equals the actual size of the underlying datafy-managed volumes when autoscaling is active, and the originalSize otherwise."},"iops":{"type":"integer","description":"Provisioned IOPS"},"throughput":{"type":"integer","description":"Provisioned throughput in MiB/s"},"region":{"type":"string","description":"AWS region"},"availabilityZone":{"type":"string","description":"AWS availability zone"},"instanceId":{"type":"string","nullable":true,"description":"EC2 instance the volume is attached to. null when unattached."},"device":{"type":"string","nullable":true,"description":"Block device mapping on the attached instance (e.g. /dev/sdf). null when unattached."},"deleteOnTermination":{"type":"boolean","nullable":true,"description":"Whether the volume is deleted when the instance is terminated. null when unattached."},"isBootVolume":{"type":"boolean","nullable":true,"description":"Whether this is the root/boot volume of the instance. null when unattached."},"clusterName":{"type":"string","nullable":true,"description":"Kubernetes cluster name, when applicable."},"nodeGroupName":{"type":"string","nullable":true,"description":"EKS node group name, when applicable."},"autoScalingGroupName":{"type":"string","nullable":true,"description":"EC2 Auto Scaling group name, when applicable."},"autoscalingState":{"type":"string","enum":["on","off"],"description":"Whether Datafy is currently autoscaling the volume."},"eligible":{"type":"string","enum":["supported","pending setup","unsupported"],"description":"Whether the volume is eligible for autoscaling."},"ineligibleReason":{"type":"string","nullable":true,"description":"Reason for `pending setup` or `unsupported`. null when `supported`."},"excludedFromRules":{"type":"boolean","description":"Whether the volume is excluded from autoscaling rules."},"ongoingActionType":{"type":"string","nullable":true,"enum":["activate_autoscaling","deactivate_autoscaling","grow","shrink","modify"],"description":"Long-running action currently in progress. null when none."},"ongoingActionStatus":{"type":"string","nullable":true,"enum":["queued","in_progress"],"description":"Status of the ongoing action. null when no action is in progress."},"ongoingActionProgressPercent":{"type":"integer","nullable":true,"minimum":0,"maximum":100,"description":"Progress of the ongoing action (0-100). null when no action is in progress."},"usedBytes":{"type":"integer","format":"int64","nullable":true,"description":"Used space in bytes (most recent reading)."},"usedPercent":{"type":"number","format":"float","nullable":true,"description":"Used space as a percentage."},"lastReportedAt":{"type":"string","format":"date-time","nullable":true,"description":"When the most recent utilization sample was reported."},"tags":{"type":"object","additionalProperties":{"type":"string"},"description":"Volume tags as a key/value map"}}}},"responses":{"BadRequestError":{"description":"Bad request"},"UnauthorizedError":{"description":"Authentication information is missing or invalid","headers":{"WWW_Authenticate":{"schema":{"type":"string"}}}}}},"paths":{"/api/v1/volumes/details":{"get":{"summary":"Volumes details","operationId":"listVolumesDetails","description":"Retrieve a list of Datafy-managed volumes with their current size,\nattachment, workload, autoscaling state, ongoing actions, and recent\nutilization.\n\nSupports filtering, sorting, and pagination.\n","tags":["Volume-Lifecycle"],"parameters":[{"$ref":"#/components/parameters/AccountsQueryParam"},{"in":"query","name":"filters","description":"Optional filters as a JSON array. Each filter is an object with\n`field`, `value`, and `op` properties.\n\nUse `?accounts=<id>` to scope by account — there is no `accountId` filter.\n\nSupported filter fields and the operators for each:\n- `volumeId` — EBS volume ID. Operators: `equals`, `contains`, `in`.\n- `volumeName` — volume name. Operators: `equals`, `contains`, `in`.\n- `diskType` — EBS type (`gp2`, `gp3`, `io1`, `io2`). Operators: `equals`, `in`.\n- `originalSize` — provisioned size in bytes. Operators: `equals`, `greater than`, `greater than equal`, `less than`, `less than equal`.\n- `region` — AWS region. Operators: `equals`, `in`.\n- `availabilityZone` — AWS availability zone. Operators: `equals`, `in`.\n- `instanceId` — EC2 instance the volume is attached to. Operators: `equals`, `contains`, `in`.\n- `clusterName` — Kubernetes cluster name. Operators: `equals`, `in`.\n- `nodeGroupName` — Kubernetes node group name. Operators: `equals`, `in`.\n- `autoScalingGroupName` — EC2 Auto Scaling group name. Operators: `equals`, `in`.\n- `isBootVolume` — boolean (`true`/`false`). Operator: `equals`.\n- `excludedFromRules` — boolean (`true`/`false`). Operator: `equals`.\n- `autoscalingState` — one of `on`, `off`. Operator: `equals`. Comma-separated multi-value supported.\n- `eligible` — one of `supported`, `pending setup`, `unsupported`. Operator: `equals`. Comma-separated multi-value supported.\n- `tag:<key>` — filter by tag key and value (e.g. `field: \"tag:Environment\"`, `value: \"prod\"`). Operators: `equals`, `contains`, `in`.\n","required":false,"schema":{"type":"array","items":{"type":"object","required":["field","value","op"],"properties":{"field":{"type":"string"},"value":{"type":"string"},"op":{"type":"string","enum":["equals","contains","in","greater than","greater than equal","less than","less than equal"]}}}}},{"in":"query","name":"sortBy","description":"Sort specification with `field` (any top-level VolumeDetails field)\nand `ascending` (default true).\n","required":false,"schema":{"type":"object","properties":{"field":{"type":"string"},"ascending":{"type":"boolean","default":true}}}},{"in":"query","name":"limit","description":"Maximum number of records to return (max 1000)","required":false,"schema":{"type":"integer","format":"int32","minimum":0,"maximum":1000,"default":20}},{"in":"query","name":"offset","description":"Number of records to skip for pagination","required":false,"schema":{"type":"integer","format":"int32","minimum":0,"default":0}}],"responses":{"200":{"description":"volumes matching the requested filters","content":{"application/json":{"schema":{"type":"object","properties":{"volumes":{"type":"array","items":{"$ref":"#/components/schemas/VolumeDetails"}}}}}}},"400":{"$ref":"#/components/responses/BadRequestError"},"401":{"$ref":"#/components/responses/UnauthorizedError"}}}}}}
```

## Volume Lifecycle

## Create snapshot of a volume

> Create a snapshot of the specified volume<br>

```json
{"openapi":"3.0.0","info":{"title":"Datafy API","version":"1.0.0"},"tags":[{"name":"Snapshots","description":"Endpoints for snapshot management"},{"name":"Volume-Lifecycle","description":"Endpoints for managing the lifecycle of storage volumes"}],"servers":[{"description":"Datafy API","url":"https://api.datafy.io"}],"security":[{"Authorizer":[]}],"components":{"securitySchemes":{"Authorizer":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}},"parameters":{"AccountsQueryParam":{"in":"query","name":"accounts","description":"Target account(s) for the request.\nOn list/aggregate endpoints: a comma-separated list of sub-account ids, or \"all\" to\ninclude every sub-account of the calling org. Required for org users on these endpoints.\nOn single-resource endpoints: at most one sub-account id. Multiple values or \"all\"\nare silently ignored and the request runs against the caller's effective account.\n","required":false,"schema":{"type":"array","items":{"type":"string"}}}},"schemas":{"VolumeId":{"type":"string"}},"responses":{"BadRequestError":{"description":"Bad request"},"UnauthorizedError":{"description":"Authentication information is missing or invalid","headers":{"WWW_Authenticate":{"schema":{"type":"string"}}}}}},"paths":{"/api/v1/volumes/{volumeId}/create-snapshot":{"post":{"summary":"Create snapshot of a volume","operationId":"createSnapshotCommand","description":"Create a snapshot of the specified volume\n","tags":["Volume-Lifecycle","Snapshots"],"parameters":[{"$ref":"#/components/parameters/AccountsQueryParam"},{"in":"path","name":"volumeId","required":true,"schema":{"$ref":"#/components/schemas/VolumeId"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","properties":{"autoscalingOnly":{"type":"boolean","description":"Create snapshots only for volumes managed by Datafy","default":true},"useSourceVolume":{"readOnly":true,"type":"boolean","description":"Create the snapshot from the source volume (if it exists) (for internal use only)","default":false},"description":{"type":"string","description":"Description for the new snapshot"},"tags":{"type":"array","description":"Tags specifications for the new snapshot","items":{"type":"object","properties":{"key":{"type":"string"},"value":{"type":"string"}}}}}}}}},"responses":{"200":{"description":"Sent create snapshot command","content":{"application/json":{"schema":{"type":"object","properties":{"datafySnapshotId":{"type":"string","description":"Datafy id of the snapshot"},"volumeId":{"type":"string","description":"Source volume id"},"status":{"type":"string"},"startTime":{"type":"string","format":"date-time"},"volumeSizeGb":{"type":"integer","description":"Original size of the source volume"},"description":{"type":"string"}}}}}},"400":{"$ref":"#/components/responses/BadRequestError"},"401":{"$ref":"#/components/responses/UnauthorizedError"}}}}}}
```

## Create autoscaling volume from Datafy snapshots

> Create new Datafy-managed autoscaling volume from Datafy snapshots<br>

```json
{"openapi":"3.0.0","info":{"title":"Datafy API","version":"1.0.0"},"tags":[{"name":"Snapshots","description":"Endpoints for snapshot management"},{"name":"Volume-Lifecycle","description":"Endpoints for managing the lifecycle of storage volumes"}],"servers":[{"description":"Datafy API","url":"https://api.datafy.io"}],"security":[{"Authorizer":[]}],"components":{"securitySchemes":{"Authorizer":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}},"parameters":{"AccountsQueryParam":{"in":"query","name":"accounts","description":"Target account(s) for the request.\nOn list/aggregate endpoints: a comma-separated list of sub-account ids, or \"all\" to\ninclude every sub-account of the calling org. Required for org users on these endpoints.\nOn single-resource endpoints: at most one sub-account id. Multiple values or \"all\"\nare silently ignored and the request runs against the caller's effective account.\n","required":false,"schema":{"type":"array","items":{"type":"string"}}}},"schemas":{"CreateFromSnapshotsRequest":{"type":"object","properties":{"source":{"description":"Source specification for creating the autoscaling volume","oneOf":[{"type":"object","description":"Create from a single Datafy snapshot","required":["datafySnapshotId"],"properties":{"datafySnapshotId":{"type":"string"}}},{"type":"object","description":"Create from multiple AWS snapshot IDs in a region","required":["snapshotIds","region"],"properties":{"snapshotIds":{"type":"array","items":{"type":"string"}},"region":{"type":"string"}}}]},"volumeProperties":{"type":"object","required":["availabilityZone"],"properties":{"availabilityZone":{"type":"string"},"volumeIops":{"type":"integer","default":3000},"volumeThroughput":{"type":"integer","default":125},"tags":{"type":"array","items":{"type":"object","properties":{"key":{"type":"string"},"value":{"type":"string"}}}}}}},"required":["source","volumeProperties"]},"CreateFromSnapshotsResponse":{"type":"object","properties":{"volumeId":{"type":"string"},"volumeSizeGB":{"type":"integer"}}}},"responses":{"BadRequestError":{"description":"Bad request"},"UnauthorizedError":{"description":"Authentication information is missing or invalid","headers":{"WWW_Authenticate":{"schema":{"type":"string"}}}}}},"paths":{"/api/v1/volumes/create-from-snapshots":{"post":{"parameters":[{"$ref":"#/components/parameters/AccountsQueryParam"}],"summary":"Create autoscaling volume from Datafy snapshots","operationId":"createFromSnapshotsCommand","description":"Create new Datafy-managed autoscaling volume from Datafy snapshots\n","tags":["Volume-Lifecycle","Snapshots"],"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/CreateFromSnapshotsRequest"}}}},"responses":{"200":{"description":"Created autoscaling volume from Datafy snapshots","content":{"application/json":{"schema":{"$ref":"#/components/schemas/CreateFromSnapshotsResponse"}}}},"400":{"$ref":"#/components/responses/BadRequestError"},"401":{"$ref":"#/components/responses/UnauthorizedError"}}}}}}
```

## Attach autoscaling volume

> Attach an unattached Datafy-managed autoscaling volume to an instance with Datafy AutoScaler<br>

```json
{"openapi":"3.0.0","info":{"title":"Datafy API","version":"1.0.0"},"tags":[{"name":"Volume-Lifecycle","description":"Endpoints for managing the lifecycle of storage volumes"}],"servers":[{"description":"Datafy API","url":"https://api.datafy.io"}],"security":[{"Authorizer":[]}],"components":{"securitySchemes":{"Authorizer":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}},"parameters":{"AccountsQueryParam":{"in":"query","name":"accounts","description":"Target account(s) for the request.\nOn list/aggregate endpoints: a comma-separated list of sub-account ids, or \"all\" to\ninclude every sub-account of the calling org. Required for org users on these endpoints.\nOn single-resource endpoints: at most one sub-account id. Multiple values or \"all\"\nare silently ignored and the request runs against the caller's effective account.\n","required":false,"schema":{"type":"array","items":{"type":"string"}}}},"schemas":{"VolumeId":{"type":"string"}},"responses":{"BadRequestError":{"description":"Bad request"},"UnauthorizedError":{"description":"Authentication information is missing or invalid","headers":{"WWW_Authenticate":{"schema":{"type":"string"}}}}}},"paths":{"/api/v1/volumes/{volumeId}/attach":{"post":{"summary":"Attach autoscaling volume","operationId":"attachCommand","description":"Attach an unattached Datafy-managed autoscaling volume to an instance with Datafy AutoScaler\n","tags":["Volume-Lifecycle"],"parameters":[{"$ref":"#/components/parameters/AccountsQueryParam"},{"in":"path","name":"volumeId","required":true,"schema":{"$ref":"#/components/schemas/VolumeId"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","properties":{"instanceId":{"type":"string"},"deviceName":{"type":"string"},"deleteOnTermination":{"type":"boolean","description":"Indicates whether the volume is deleted on instance termination.\nThis flag is only effective when a Datafy agent is present on the instance.\n","default":false}},"required":["instanceId","deviceName"]}}}},"responses":{"200":{"description":"Attached autoscaling volume"},"400":{"$ref":"#/components/responses/BadRequestError"},"401":{"$ref":"#/components/responses/UnauthorizedError"}}}}}}
```

## Send deactivate autoscaling command

> Send deactivate autoscaling command of a specific volume<br>

```json
{"openapi":"3.0.0","info":{"title":"Datafy API","version":"1.0.0"},"tags":[{"name":"Volume-Lifecycle","description":"Endpoints for managing the lifecycle of storage volumes"}],"servers":[{"description":"Datafy API","url":"https://api.datafy.io"}],"security":[{"Authorizer":[]}],"components":{"securitySchemes":{"Authorizer":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}},"parameters":{"AccountsQueryParam":{"in":"query","name":"accounts","description":"Target account(s) for the request.\nOn list/aggregate endpoints: a comma-separated list of sub-account ids, or \"all\" to\ninclude every sub-account of the calling org. Required for org users on these endpoints.\nOn single-resource endpoints: at most one sub-account id. Multiple values or \"all\"\nare silently ignored and the request runs against the caller's effective account.\n","required":false,"schema":{"type":"array","items":{"type":"string"}}}},"schemas":{"VolumeId":{"type":"string"}},"responses":{"BadRequestError":{"description":"Bad request"},"UnauthorizedError":{"description":"Authentication information is missing or invalid","headers":{"WWW_Authenticate":{"schema":{"type":"string"}}}},"UnprocessableContentError":{"description":"Unprocessable content"}}},"paths":{"/api/v1/volumes/{volumeId}/deactivate-autoscaling":{"post":{"summary":"Send deactivate autoscaling command","operationId":"deactivateAutoscalingCommand","description":"Send deactivate autoscaling command of a specific volume\n","tags":["Volume-Lifecycle"],"parameters":[{"$ref":"#/components/parameters/AccountsQueryParam"},{"in":"path","name":"volumeId","required":true,"schema":{"$ref":"#/components/schemas/VolumeId"}}],"responses":{"200":{"description":"send deactivate autoscaling command"},"400":{"$ref":"#/components/responses/BadRequestError"},"401":{"$ref":"#/components/responses/UnauthorizedError"},"422":{"$ref":"#/components/responses/UnprocessableContentError"}}}}}}
```

## Modify a volume

> Modify the size, IOPS or throughput of a volume. See documentation for required agent versions

```json
{"openapi":"3.0.0","info":{"title":"Datafy API","version":"1.0.0"},"tags":[{"name":"Volume-Lifecycle","description":"Endpoints for managing the lifecycle of storage volumes"}],"servers":[{"description":"Datafy API","url":"https://api.datafy.io"}],"security":[{"Authorizer":[]}],"components":{"securitySchemes":{"Authorizer":{"type":"http","scheme":"bearer","bearerFormat":"JWT"}},"parameters":{"AccountsQueryParam":{"in":"query","name":"accounts","description":"Target account(s) for the request.\nOn list/aggregate endpoints: a comma-separated list of sub-account ids, or \"all\" to\ninclude every sub-account of the calling org. Required for org users on these endpoints.\nOn single-resource endpoints: at most one sub-account id. Multiple values or \"all\"\nare silently ignored and the request runs against the caller's effective account.\n","required":false,"schema":{"type":"array","items":{"type":"string"}}}},"schemas":{"VolumeId":{"type":"string"},"ModifyVolumeProperties":{"type":"object","properties":{"volumeSizeGb":{"type":"integer","description":"Desired size of volume, in GiB. Must be larger than current original size"},"volumeIops":{"type":"integer","description":"Desired IOPS of volume"},"volumeThroughput":{"type":"integer","description":"Desired throughput of volume, in MBps"},"expandFilesystem":{"type":"boolean","description":"Expand filesystem automatically after modifying (requires volumeSizeGb set)","default":false},"autoscalingOnly":{"type":"boolean","description":"Modify only volumes managed by Datafy","default":true}}}},"responses":{"BadRequestError":{"description":"Bad request"},"UnauthorizedError":{"description":"Authentication information is missing or invalid","headers":{"WWW_Authenticate":{"schema":{"type":"string"}}}}}},"paths":{"/api/v1/volumes/{volumeId}/modify":{"post":{"summary":"Modify a volume","operationId":"modifyCommand","description":"Modify the size, IOPS or throughput of a volume. See documentation for required agent versions","tags":["Volume-Lifecycle"],"parameters":[{"$ref":"#/components/parameters/AccountsQueryParam"},{"in":"path","name":"volumeId","required":true,"schema":{"$ref":"#/components/schemas/VolumeId"}}],"requestBody":{"description":"Desired values for modifying volume. At least one of volumeSizeGb, volumeIops or volumeThroughput is required.","content":{"application/json":{"schema":{"description":"Desired values for modifying volume. At least one of volumeSizeGb, volumeIops or volumeThroughput is required.","anyOf":[{"allOf":[{"$ref":"#/components/schemas/ModifyVolumeProperties"},{"required":["volumeSizeGb"]}]},{"allOf":[{"$ref":"#/components/schemas/ModifyVolumeProperties"},{"required":["volumeIops"]}]},{"allOf":[{"$ref":"#/components/schemas/ModifyVolumeProperties"},{"required":["volumeThroughput"]}]}]}}}},"responses":{"200":{"description":"send modify command"},"400":{"$ref":"#/components/responses/BadRequestError"},"401":{"$ref":"#/components/responses/UnauthorizedError"}}}}}}
```


# Datafy Metrics Exporter

The Datafy Metrics Exporter exposes supplementary metrics about the Datafy agent's health and your autoscaling volumes, at two endpoints:

* `/metrics` - exposes metrics about the size and usage of autoscaling volumes managed by Datafy.
* `/health` - returns a health indication for the Datafy agent installed on the instance.

## Installing Datafy Metrics Exporter <a href="#installing-datafy-metrics-exporter" id="installing-datafy-metrics-exporter"></a>

Datafy Metrics Exporter can be installed with a single command or as a Docker container.

The metrics are exposed by default on port 9908, and a custom port can be configured as desired.

{% tabs %}
{% tab title="One-line install" %}
To install Datafy Metrics Exporter, run the command:

```bash
curl -sSfL https://bos.datafy.io/install_exporter | sh
```

To configure a custom port, install with the flag `PORT`. For example:

```bash
curl -sSfL https://bos.datafy.io/install_exporter | sh PORT=9100 sh
```

The exporter can be uninstalled in the same manner, using the command:

```bash
curl -sSfL https://bos.datafy.io/uninstall_exporter | sh
```

{% endtab %}

{% tab title="Docker" %}
To install Datafy Metrics Exporter using a Docker image, use the following installation command:

```bash
docker run -d \
  --name datafy_exporter \
  --restart unless-stopped \
  --pid=host --privileged \
  -v /:/host:ro,rslave \
  -v /var/lib/datafy/pipes/geoms.state:/var/lib/datafy/pipes/geoms.state \
  -p 9908:9908 \
  public.ecr.aws/datafy-io/datafy-exporter:latest
```

To configure a custom port, install with the flag `-port`. For example:

{% code overflow="wrap" %}

```bash
docker run -d \
  --name datafy_exporter \
  --restart unless-stopped \
  --pid=host --privileged \
  -v /:/host:ro,rslave \
  -v /var/lib/datafy/pipes/geoms.state:/var/lib/datafy/pipes/geoms.state \
  -p 9202:9202 \
  public.ecr.aws/datafy-io/datafy-exporter:latest \
  -port=9202
```

{% endcode %}
{% endtab %}
{% endtabs %}

## Datafy Exporter Metrics <a href="#datafy-exporter-metrics" id="datafy-exporter-metrics"></a>

### Viewing Datafy Metrics

Once the Datafy Metrics Exporter is installed and running, you can verify that metrics are being exported, and view help and type information by accessing the `/metrics` endpoint:

```bash
curl http://localhost:9908/metrics
```

{% hint style="info" %}
Make sure to use the port specified in the installation
{% endhint %}

You should see output like this:

{% code overflow="wrap" %}

```
# HELP disk_total_bytes Total disk space in bytes
# TYPE disk_total_bytes gauge
# HELP disk_used_bytes Used disk space in bytes
# TYPE disk_used_bytes gauge
# HELP disk_available_bytes Available disk space in bytes
# TYPE disk_available_bytes gauge
# HELP disk_usage_percent Disk usage percent
# TYPE disk_usage_percent gauge
# HELP service_up Service liveness status (1 = up, 0 = down)
# TYPE service_up gauge
disk_total_bytes{mount="/mnt/data",fs="/dev/nvme1n1",vol="vol-00d9752e77b9ff5be"} 5.7982058496e+10
disk_used_bytes{mount="/mnt/data",fs="/dev/nvme1n1",vol="vol-00d9752e77b9ff5be"} 45056
disk_available_bytes{mount="/mnt/data",fs="/dev/nvme1n1",vol="vol-00d9752e77b9ff5be"} 5.798201344e+10
disk_usage_percent{mount="/mnt/data",fs="/dev/nvme1n1",vol="vol-00d9752e77b9ff5be"} 7.770679615162037e-05
disk_total_bytes{fs="/dev/nvme2n1",vol="vol-05c2dd6f056df3fc8",mount="/mnt/data2"} 2.10241560576e+11
disk_used_bytes{fs="/dev/nvme2n1",vol="vol-05c2dd6f056df3fc8",mount="/mnt/data2"} 36864
disk_available_bytes{mount="/mnt/data2",fs="/dev/nvme2n1",vol="vol-05c2dd6f056df3fc8"} 1.99487328256e+11
disk_usage_percent{fs="/dev/nvme2n1",vol="vol-05c2dd6f056df3fc8",mount="/mnt/data2"} 1
service_up{service="datafy-agent"} 1
service_up{service="datafy-agent-core"} 1
```

{% endcode %}

Success! The Datafy Metrics Exporter is now exposing metrics that a service like Prometheus can scrape.

### Using Datafy Metrics

The Datafy Metrics Exporter exposes size and usage metrics for each volume on the instance. For autoscaling volumes, the actual current size of the autoscaling volume is exposed.

{% hint style="info" %}
For out of space monitoring of autoscaling volumes use the utilization reported by the filesystem, do not use the metrics from Datafy Metrics Exporter.
{% endhint %}

Other metrics exporters (such as Prometheus Node Exporter) will continue to correctly report read, write and IOPS rates for autoscaling volumes. However, autoscaling volumes will have device names in the format `dvolXXX...`. This name corresponds to the `vol` field in the Datafy metrics, which you can use to enrich other metrics with the relevant mount and filesystem paths.


# Deployment Examples

Examples for deploying Datafy in your environment using common deployment tools

Datafy can be installed on EC2 instances with a single shell command, and on Kubernetes clusters with a Helm chart. In environments with many instances or clusters, we recommend deploying Datafy Sensor and AutoScaler with the same tools you already use to deploy packages in your environment.

This page provides examples for integrating Datafy installation into common deployment and management tools. Each example assumes you're already using the tool - refer to its own documentation for initial setup and general usage.

For the standard installation steps and parameters, see the [installation guide](/set-up-and-installation/datafy-installation/installation).

The examples on this page use the following parameters:

* `$TOKEN` - Your Datafy [security token](/set-up-and-installation/datafy-installation/token-generation), generated from the Datafy app.
* `$VERSION` - The Datafy agent version to install (used in EC2 examples).
* `$HELM_VERSION` - The Datafy Helm chart version to deploy (used in Kubernetes examples).

{% hint style="warning" %}
This page contains examples for deploying Datafy using different tools - they do not cover all of the ways to deploy Datafy, with these tools or otherwise. For the best outcome, follow the standards and best practices used in your environment.
{% endhint %}

## EC2 Instances

On EC2, Datafy is installed by running a shell command on each instance. The tools below let you run that command across many instances at once.

{% hint style="info" %}
Docker must be installed on each target instance before installing Datafy. See the [EC2 installation prerequisites](/set-up-and-installation/datafy-installation/installation#ec2-installation) for details.
{% endhint %}

### Ansible

[Ansible](https://docs.ansible.com/ansible/latest/index.html) is an agentless automation tool that connects to your instances over SSH and runs tasks defined in a playbook. You can use it to install Datafy across a group of EC2 instances in a single run.

Follow the steps below to create and run a playbook that will install Datafy on the desired hosts.

{% stepper %}
{% step %}
**Create Playbook**

Create a new playbook called `install_datafy.yaml`, and add it to your ansible project. The following example playbook downloads the Datafy install script once on your Ansible control machine, copies it to each target instance, and runs it.

<details>

<summary>Install Datafy Playbook</summary>

Create an `install_datafy.yaml` file:

{% code overflow="wrap" %}

```yaml
---
- name: Install Datafy Agent
  # Set the hosts field to target the instances you want to install Datafy on
  hosts: all 
  become: yes

  vars:
    # Replace $VERSION and $TOKEN with your values.
    # To install Sensor instead of AutoScaler, change agent_mode to "sensor".
    version: "$VERSION" 
    datafy_token: "$TOKEN"
    agent_mode: "autoscaler"

  tasks:
    - name: Download Datafy install script to control machine
      ansible.builtin.get_url:
        url: "https://agent.datafy.io/install?version={{ version }}"
        dest: "/tmp/install_datafy_agent.sh"
        mode: '0755'
      delegate_to: localhost
      run_once: true
      become: no

    - name: Copy install script to target instance
      ansible.builtin.copy:
        src: /tmp/install_datafy_agent.sh
        dest: /tmp/install_datafy_agent.sh
        mode: '0755'

    - name: Run Datafy install script
      ansible.builtin.shell: /tmp/install_datafy_agent.sh
      environment:
        TOKEN: "{{ datafy_token }}"
        AGENT_MODE: "{{ agent_mode }}"
```

{% endcode %}

</details>
{% endstep %}

{% step %}
**Run the Playbook**

Run the playbook against your inventory, with:

```bash
ansible-playbook install_datafy.yaml
```

Ansible will connect to each instance, copy the install script, and run it. You'll see output for each task on each host, with a summary at the end showing how many succeeded.
{% endstep %}
{% endstepper %}

### AWS Systems Manager (SSM) Run Command

[AWS Systems Manager Run Command](https://docs.aws.amazon.com/systems-manager/latest/userguide/run-command.html) lets you run shell commands on your EC2 instances remotely, without needing SSH access. It uses the SSM Agent, which comes pre-installed on most Amazon Linux and Ubuntu AMIs.

This is a good option when your instances are already managed through Systems Manager, or when you want to run the install across instances without setting up SSH keys.

#### Prerequisites

* The SSM Agent must be installed and running on each target instance. Most Amazon-provided AMIs include it by default. See [AWS documentation on SSM Agent](https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html) for details.
* The instances must have an IAM instance profile with the `AmazonSSMManagedInstanceCore` policy (or equivalent permissions) to communicate with Systems Manager.
* Docker installed on each target instance.

#### Installation

You can set up and run the shell command from the AWS console, or through the AWS CLI.

{% tabs %}
{% tab title="AWS Console" %}

1. Open the [Run Command page](https://console.aws.amazon.com/systems-manager/run-command/send-command) in the AWS Systems Manager console.
2. In **Command document**, search for `AWS-RunShellScript` and select it, then select **Run Command**.
3. In **Command parameters:**
   1. Define the `$VERSION` and `$TOKEN` variables
   2. Enter the Datafy install command:

      <pre class="language-bash" data-overflow="wrap"><code class="lang-bash">curl -sSfL https://agent.datafy.io/install?version="$VERSION" | AGENT_MODE="autoscaler" TOKEN="$TOKEN" sh
      </code></pre>
4. In **Target selection**, choose the instances to install on. You can select targets by:
   * **Tags** - for example, all instances with a specific `Environment` or `Role` tag.
   * **Manual selection** - pick individual instances from the list.
   * **Resource group** - target all instances in an AWS resource group.
5. Optionally, in **Output options**, configure output logging to an S3 bucket or CloudWatch Logs. This is helpful for reviewing results across many instances.
6. Click **Run** to execute the command.
   {% endtab %}

{% tab title="AWS CLI" %}
{% code overflow="wrap" %}

```bash
aws ssm send-command \
  --document-name "AWS-RunShellScript" \
  --document-version "1" \
  --targets '[{"Key":"tag:YOUR_TAG_KEY","Values":["YOUR_TAG_VALUE"]}]' \
  --parameters '{"commands":["curl -sSfL https://agent.datafy.io/install?version=$VERSION | AGENT_MODE=autoscaler TOKEN=$TOKEN sh"]}' \
  --region us-east-1
```

{% endcode %}

Replace the `--targets` filter with the targeting method you prefer. You can target by instance IDs, tags, or resource groups. See the [AWS CLI reference for send-command](https://docs.aws.amazon.com/cli/latest/reference/ssm/send-command.html) for all available options.
{% endtab %}
{% endtabs %}

#### Verifying the Results

After running the command, you can check the results in the Systems Manager console:

* **Running commands** - view commands currently in progress under [Executing Commands](https://console.aws.amazon.com/systems-manager/run-command/executing-commands).
* **Completed commands** - view results under [Command History](https://console.aws.amazon.com/systems-manager/run-command/complete-commands). Select a command to see the overall status, and drill into individual instances to view their output and any errors.

## Kubernetes

On Kubernetes, Datafy is installed via its [Helm chart](/set-up-and-installation/datafy-installation/installation#kubernetes-cluster). GitOps tools can manage this Helm release declaratively, keeping your cluster configuration in sync with your desired state.

### ArgoCD

[ArgoCD](https://argo-cd.readthedocs.io/en/stable/) is a declarative GitOps continuous delivery tool for Kubernetes. It monitors a source (such as a Helm chart repository) and automatically syncs your cluster to match the desired state. You can use it to deploy and manage the Datafy Helm chart.

#### Prerequisites

* ArgoCD installed and running on your cluster. See the [ArgoCD getting started guide](https://argo-cd.readthedocs.io/en/stable/getting_started/).
* Access to create ArgoCD Applications (via the UI or `kubectl`).

#### Deploy Datafy with ArgoCD

Create an ArgoCD Application that points to the Datafy Helm chart. You can use the ArgoCD UI or CLI to create an application based on the following manifest:

<details>

<summary>Datafy Application manifest</summary>

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: datafy
  namespace: argocd
spec:
  project: default
  destination:
    namespace: datafy-agent
    server: https://kubernetes.default.svc
  source:
    repoURL: https://helm.datafy.io/datafy-agent
    chart: datafy-agent
    targetRevision: $HELM_VERSION # define variable or replace with correct version
    helm:
      parameters:
        - name: agent.token
          value: $TOKEN # define variable or replace with correct value
        - name: agent.mode
          value: "autoscaler" #To install Sensor instead of AutoScaler, change to "sensor".
  syncPolicy:
    syncOptions:
      # Create the "datafy-agent" namespace if it doesn't already exist
      - CreateNamespace=true
    automated:
      # Don't automatically delete resources removed from the chart.
      # Adjust based on your team's GitOps practices.
      prune: false
      # Don't automatically revert manual changes made to resources.
      # Adjust based on your team's GitOps practices.
      selfHeal: false
```

</details>

{% tabs %}
{% tab title="ArgoCD UI" %}

1. Open the ArgoCD dashboard and click **New App**.
2. Click **Edit as YAML** and paste the above manifest.
3. Click **Save**, then **Create**.
   {% endtab %}

{% tab title="CLI" %}
Save the above manifest to a file (for example, `datafy-app.yaml`) and apply it:

```bash
kubectl apply -f datafy-app.yaml
```

{% endtab %}
{% endtabs %}

Once created, ArgoCD will sync the application and deploy Datafy to your cluster. You can monitor the sync status in the ArgoCD dashboard.


# My First Autoscaling Volume

A hands-on tutorial that walks you through installing AutoScaler, activating autoscaling, and watching your volumes grow and shrink automatically.

In this tutorial, we'll go from zero to a fully autoscaling environment together. By the end, you'll have:

* Installed the Datafy AutoScaler agent on an EC2 instance
* Activated autoscaling on volumes both manually and automatically via an autoscaling rule
* Watched AutoScaler automatically grow a volume when you write data
* Watched AutoScaler automatically shrink a volume when you delete data

The whole process takes about 30 minutes of active work, plus some waiting time. Let's get started!

{% hint style="success" %}
To complete this tutorial you will need:

* A **Datafy account** set up with [configured permissions](/set-up-and-installation/datafy-installation/permissions-configuration) and an [API token](/set-up-and-installation/datafy-installation/token-generation)
* An **AWS account** with permissions to create EC2 instances and EBS volumes.
* **SSH access** to EC2 instances in your environment.
  {% endhint %}

## Set Up Your Environment

First, we'll create an EC2 instance with two EBS volumes, tag them, set up filesystems, and write some test data. You can do this in one of three ways:

1. Automatically launch and set up an instance and 2 volumes using a CloudFormation template.
2. Manually launch an instance with a user data script that sets up the volumes and installs docker on the instance.
3. Manually set up the instance and volumes.

{% tabs %}
{% tab title="CloudFormation" %}
This CloudFormation template creates an EC2 instance with two tagged 150 GiB gp3 volumes, and automatically formats, mounts, and populates them with test data on first boot.

You'll need an existing VPC, subnet, security group, and key pair.

Create a CloudFormation stack with new resources, and upload the template below.

{% file src="/files/jGd5njC2NWS4hjyDbRVJ" %}

The template applies the tags `Name: manual_activation` and `Name: auto_activation` to the volumes for you.

Writing the test data takes about 15 minutes for both volumes. Once the resources are created, access the instance via SSH and track the progress of the setup with:

{% code overflow="wrap" %}

```bash
tail /var/log/cloud-init-output.log
```

{% endcode %}

When it's finished you'll see a confirmation for each volume:

```
/mnt/vol1: verified (45G used)
/mnt/vol2: verified (45G used)
Volume setup complete.
```

{% endtab %}

{% tab title="Setup with User Data" %}
Launch a **c5.xlarge** EC2 instance using the latest **Amazon Linux 2023** AMI.

During the launch setup, configure:

* Create and attach **two additional gp3 EBS volumes** of **150 GiB** each.
* Insert the following script into the *user data* of the instance. The script formats, mounts, and writes data to the volumes, and installs Docker on the instance.

```bash
#!/bin/bash
set -e

# Install Docker (required for AutoScaler)
yum update -y
yum install -y docker
service docker start
# Wait for both volumes to be attached
echo "Waiting for volumes to attach..."
while [ $(lsblk -dn -o NAME,SIZE | grep "150G" | wc -l) -lt 2 ]; do
  sleep 5
done
# Identify the two 150 GiB volumes (excluding root)
VOLUMES=($(lsblk -dn -o NAME,SIZE | grep "150G" | awk '{print "/dev/"$1}'))
# Step 1: Format and mount all volumes
for i in 0 1; do
  VOL=${VOLUMES[$i]}
  MNT="/mnt/vol$(($i + 1))"
  mkfs.ext4 -q "$VOL"
  mkdir -p "$MNT"
  mount "$VOL" "$MNT"
  echo "$(blkid -s UUID -o value $VOL) $MNT ext4 defaults,nofail 0 2" >> /etc/fstab
done
# Step 2: Write test data to all volumes with verification
EXPECTED_KB=$((45 * 1024 * 1024))  # 45 GiB in KiB
THRESHOLD_KB=$((44 * 1024 * 1024)) # Allow minor variance
for i in 1 2; do
  MNT="/mnt/vol${i}"
  echo "Writing test data to ${MNT}..."
  for attempt in 1 2; do
    echo "${MNT}: writing file1.bin (30 GiB)..."
    dd if=/dev/urandom of=${MNT}/file1.bin bs=1M count=30720 status=progress
    # 15 GiB minus 1 MiB, so df -h rounds to 45G
    echo "${MNT}: writing file2.bin..."
    dd if=/dev/urandom of=${MNT}/file2.bin bs=1M count=15359 status=progress

    USED_KB=$(df --output=used "$MNT" | tail -1 | tr -d ' ')
    if [ "$USED_KB" -ge "$THRESHOLD_KB" ]; then
      echo "${MNT}: verified ($(df -h --output=used $MNT | tail -1 | tr -d ' ') used)"
      break
    fi

    if [ "$attempt" -eq 2 ]; then
      echo "WARNING: ${MNT} has less data than expected after retry"
    else
      echo "${MNT}: incomplete write, retrying..."
      rm -f ${MNT}/file1.bin ${MNT}/file2.bin
    fi
  done
done
echo "Volume setup complete."
```

After the instance launches, add a Name tag to each of the volumes in the AWS Console:

* **1 volume** → `Name: manual_activation`
* **1 volume** → `Name: auto_activation`

These tags will help us identify the volumes in the Datafy App later, and we'll use them to create an autoscaling rule.
{% endtab %}

{% tab title="Manual Setup" %}

#### Launch the Instance and Volumes

Launch a **c5.xlarge** EC2 instance using the latest **Amazon Linux 2023** AMI. Create **two additional gp3 EBS volumes** of **150 GiB** each in the same Availability Zone, and attach them to the instance.

{% hint style="info" %}
Make sure both volumes are in the same Availability Zone as your instance. You can create and attach them during instance launch or afterward through the AWS Console.
{% endhint %}

#### Tag the Volumes

In the AWS Console, add a `Name` tag to each of the two volumes:

* **1 volume** → `Name: manual_activation`
* **1 volume** → `Name: auto_activation`

These tags will help us identify the volumes in the Datafy App later, and we'll use them to create an autoscaling rule.

#### Mount and Prepare the Volumes

1. Access your instance via SSH and run the following commands to format and mount both volumes.

   <pre class="language-bash" data-overflow="wrap"><code class="lang-bash"># The volumes will typically appear as /dev/nvme1n1 and /dev/nvme2n1
   # Adjust the device names below if yours are different

   # Create ext4 filesystems on both volumes
   sudo mkfs.ext4 /dev/nvme1n1
   sudo mkfs.ext4 /dev/nvme2n1
   # Create mount points
   sudo mkdir -p /mnt/vol1 /mnt/vol2
   # Mount the volumes
   sudo mount /dev/nvme1n1 /mnt/vol1
   sudo mount /dev/nvme2n1 /mnt/vol2
   # Add to fstab for persistence across reboots
   echo "$(blkid -s UUID -o value /dev/nvme1n1) /mnt/vol1 ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
   echo "$(blkid -s UUID -o value /dev/nvme2n1) /mnt/vol2 ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
   </code></pre>
2. Next, write 45 GiB of test data to each volume:

   ```bash
   # Write test data to both volumes - 45 GiB each, less 1 MiB, so df -h rounds to 45G
   # This will take a few minutes per volume
   for vol in /mnt/vol1 /mnt/vol2; do
     sudo dd if=/dev/urandom of=${vol}/file1.bin bs=1M count=30720 status=progress
     sudo dd if=/dev/urandom of=${vol}/file2.bin bs=1M count=15359 status=progress
   done
   ```

{% endtab %}
{% endtabs %}

{% hint style="success" %}
Once the setup completes, verify that each volume has roughly 45 GiB of data:

```bash
df -h /mnt/vol1 /mnt/vol2
```

```
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    147G   45G   95G  33% /mnt/vol1
/dev/nvme2n1    147G   45G   95G  33% /mnt/vol2
```

{% endhint %}

{% hint style="info" %}
The size reported by the filesystem is always slightly smaller than the volume's 150 GiB - the difference is formatting overhead.
{% endhint %}

## Autoscale Your Volumes

With the environment in place, we'll install the agent and then activate autoscaling for both volumes - one manually, and one with a rule.

{% stepper %}
{% step %}

### Install AutoScaler

Now let's install the Datafy AutoScaler agent! For the full installation instructions, see [Installation](/set-up-and-installation/datafy-installation/installation).

{% hint style="info" %}
AutoScaler requires Docker to run. If your instance doesn't already have Docker installed, run the following commands to install it:

```bash
sudo yum update -y
sudo yum install -y docker
sudo service docker start
```

If you set up your instance with the CloudFormation template or user data script, Docker is already installed.
{% endhint %}

Define the version you want to install and your Datafy API token. Replace `<VERSION>` with the agent version you're installing, and `<YOUR_TOKEN>` with the token you generated:

{% code overflow="wrap" expandable="true" %}

```bash
VERSION="<VERSION>"
TOKEN="<YOUR_TOKEN>"
```

{% endcode %}

Install AutoScaler with the command:

{% code overflow="wrap" expandable="true" %}

```bash
curl -sSfL https://agent.datafy.io/install?version=$VERSION | AGENT_MODE="autoscaler" TOKEN=$TOKEN sh
```

{% endcode %}

The installation takes about a minute. Once it completes, the agent starts running automatically and begins reporting the volumes' utilization.

{% hint style="success" %}
The last line of the output confirms a successful installation:

```
Datafy Agent 1.40.0 was successfully installed 🎉
```

{% endhint %}

Let's head over to the [**Datafy app**](https://app.datafy.io) to see the instance and volumes.

#### Find Your Volumes in the Fleet Manager

1. Open the [Fleet Manager](https://app.datafy.io/) in the Datafy app.
2. In the [All Volumes](https://app.datafy.io/fleet-manager/all-volumes) table, filter by instance ID to show only the volumes on our tutorial instance.
3. You should see your two 150 GiB volumes listed, with the `Name` tags you assigned earlier (`manual_activation` and `auto_activation`).

<div data-with-frame="true"><figure><img src="/files/MaG9CAYTrMThGxVBFPsk" alt="All Volumes table filtered to the two tutorial volumes, each 150 GiB"><figcaption></figcaption></figure></div>

{% hint style="success" %}
Before installing the agent, these volumes would have had an inactive Autoscaling toggle showing **"No AutoScaler"** as the reason. Now that the agent is installed, the Autoscaling toggle is **active and available** - ready for you to turn on.
{% endhint %}

{% hint style="info" %}
You can also use the views at the top left of the All Volumes table to filter all the "[Volumes to Activate](/volume-lifecycle/managing-autoscaling-volumes#identifying-supported-volumes)". This is a useful way to find volumes that are ready to be activated.
{% endhint %}
{% endstep %}

{% step %}

### Activate Autoscaling

Now for the main event: we'll activate autoscaling in two ways - manually on one volume, and with a rule on the other.

#### Activate Manually

Let's [activate autoscaling](/volume-lifecycle/managing-autoscaling-volumes#activating-autoscaling) on the `manual_activation` volume through the UI:

1. In the [**All Volumes**](https://app.datafy.io/fleet-manager/all-volumes) table, find the volume tagged `manual_activation`.
2. Click the **AutoScale** toggle to turn it on. The progress of the activation is displayed next to the toggle.

   <div data-with-frame="true"><figure><img src="/files/g3NmukGecSVdZuQBGwUQ" alt="The AutoScale toggle turned on, with activation progress beside it" width="375"><figcaption></figcaption></figure></div>
3. Select the volume ID in the All Volumes table to navigate to the volume details page. Here you can see the new smaller volumes that will replace your original volume, and the usage and utilization of the volume over time.

<div data-with-frame="true"><figure><img src="/files/dsZSSMIMNz8R6PHjMAP9" alt="Volume details page showing the replacement volumes and the usage chart"><figcaption></figcaption></figure></div>

{% hint style="info" %}
The initial activation copies all of the data on the volume, and should take about 15 minutes. Volumes with more data, or with active reads/writes to the volume will take longer, see [How AutoScaler Works](/how-it-works/how-autoscaler-works#autoscaling-activation) for more details.
{% endhint %}

{% hint style="info" %}
To find the mount point of each volume, use the following - we'll call yours `/mnt/volN`:

```bash
lsblk -o NAME,SIZE,SERIAL,MOUNTPOINT
```

{% endhint %}

{% hint style="success" %}
During and after the activation, your files remain the same, and the filesystem continues to report and behave like it did before:

```bash
ls -lh /mnt/volN/
df -h /mnt/volN
```

```
-rw-r--r--. 1 root root  30G Aug 11 13:15 file1.bin
-rw-r--r--. 1 root root  15G Aug 11 13:17 file2.bin

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme2n1    147G   45G   95G  33% /mnt/volN
```

{% endhint %}

For more information check out how [autoscaling](/how-it-works/how-autoscaler-works) works, and how to [monitor activation actions](/volume-lifecycle/managing-autoscaling-volumes#monitoring-autoscaling-activity).

#### Activate with an Autoscaling Rule

While the first volume completes its activation, let's create a [rule](/volume-lifecycle/autoscaling-rules) that automatically activates autoscaling on volumes matching the `auto_activation` tag. This is how you'd manage autoscaling at scale in a production environment - you tag volumes, and Datafy activates them for you.

1. In the Fleet Manager, navigate to the [**Autoscaling Rules**](https://app.datafy.io/fleet-manager/autoscale-rules) section.
2. Click **Create New Rule** to open the rule creation wizard.
3. Add a condition "Volume tag is `Name: auto_activation`".

   <div data-with-frame="true"><figure><img src="/files/5m0KWYyOPChEg0OVzj2x" alt="Rule wizard with a volume tag condition for Name: auto_activation" width="375"><figcaption></figcaption></figure></div>
4. Verify that the volumes you expect match the rule

   <div data-with-frame="true"><figure><img src="/files/OEWzyK61glHAVyVN4ENB" alt="Rule wizard listing the matching auto_activation volume" width="375"><figcaption></figcaption></figure></div>
5. Select **Create and Apply** to activate the rule

The rule runs immediately upon creation. Head back to the All Volumes table - you'll see the AutoScale toggle is now on for the `auto_activation` volume too.

{% hint style="info" %}
Autoscaling is activated for [one volume at a time per instance](/volume-lifecycle/managing-autoscaling-volumes#autoscaling-multiple-volumes), so this volume waits its turn: it shows as pending until the manual activation finishes.
{% endhint %}

Once both activations complete, both of your tutorial volumes are managed by Datafy. 🎉
{% endstep %}

{% step %}

### Grow

Let's see autoscaling in action. We'll write additional data to one of the volumes and watch Datafy automatically grow it to accommodate the new usage.

Write another 15 GiB file to the volume you activated manually - `/mnt/volN` from the previous step:

```bash
# Write an additional 15 GiB file to trigger a grow
sudo dd if=/dev/urandom of=/mnt/volN/file3.bin bs=1M count=15360 status=progress
```

Now open the Datafy UI and watch the volume. As the usage increases and crosses the growth threshold, Datafy will automatically trigger a grow operation. You'll see:

* The volume's **utilization** increase in the Fleet Manager
* As the used data approaches the current size of the volume you'll see the size increase to accommodate the new data

The grow happens seamlessly — no downtime, no interruption to your applications. Once it completes, the volume returns to its normal autoscaling state with the additional capacity in place. The whole operation takes seconds: AutoScaler expands the underlying volumes in place, without copying any data.

For more on how grow thresholds work, see [How AutoScaler Works](/how-it-works/how-autoscaler-works#growing).
{% endstep %}

{% step %}

### Shrink

Now let's see the other side of autoscaling: what happens when you no longer need all that space.

Delete the 30 GiB file you wrote during setup, on the same volume:

```bash
sudo rm /mnt/volN/file1.bin
```

{% hint style="info" %}
The shrink won't happen immediately. Two things have to clear first:

* **The cooldown** - Datafy waits after any autoscaling operation to be sure the freed space is really no longer needed, rather than churning on a workload that dips and spikes. Depending on your [account configuration](/set-up-and-installation/autoscaling-configurations) this can be **up to an hour**, counted from the grow you just triggered.
* **Any activation still running on the instance** - a shrink won't start while another volume on the same instance is still being activated.
  {% endhint %}

After the cooldown period, AutoScaler detects the reduced usage and triggers a shrink operation. In the Datafy app, you'll see:

* A "Shrinking" action will appear in the Optimization Actions table in the Reports page and on the details page of the relevant volume
* Once the shrink is completed, you'll see the size of the volume drop to about 55 GiB - from the 150 GiB you provisioned.
  {% endstep %}
  {% endstepper %}

## Cleanup

When you're done exploring, there are two ways to tear the tutorial environment down:

* **Full teardown** - deactivate autoscaling, uninstall the agent, then remove the AWS resources. This is the sequence to follow in production, so it's the one to walk through if you want to practice it, or if you want to keep the volumes and their data afterward.
* **Delete everything** - terminate the instance and delete the volumes directly. Faster, and safe here because nothing needs to survive.

Either way, start by deleting the rule so it can't activate autoscaling on anything else in your account: in the [**Autoscaling Rules**](https://app.datafy.io/fleet-manager/autoscale-rules) section of the Fleet Manager, select the rule you created and delete it.

{% tabs %}
{% tab title="Full Teardown" %}
{% stepper %}
{% step %}

#### Deactivate Autoscaling on Both Volumes

In the [**All Volumes**](https://app.datafy.io/fleet-manager/all-volumes) table, turn off the **AutoScale** toggle for both volumes, and wait for both to finish. Datafy copies your data back to standard EBS volumes at the original 150 GiB size, and deletes the volumes it was managing.

{% hint style="info" %}
Each restored volume gets a **new AWS EBS volume ID**, but keeps the original volume's tags - so you can still find them by their `manual_activation` and `auto_activation` tags in the AWS Console.
{% endhint %}
{% endstep %}

{% step %}

#### Uninstall the Agent

On the instance, uninstall the Datafy agent, using the same version you installed:

{% code overflow="wrap" %}

```bash
curl -sSfL https://agent.datafy.io/uninstall\?version=$VERSION | sh
```

{% endcode %}
{% endstep %}

{% step %}

#### Remove the AWS Resources

Terminate the EC2 instance, then delete the two restored EBS volumes.
{% endstep %}
{% endstepper %}
{% endtab %}

{% tab title="Delete Everything" %}
Because the instance and the data are both being destroyed, you can skip deactivation and the uninstall. Deactivating autoscaling exists to return your data to a volume that's readable without AutoScaler, which is not needed if it is going to be deleted.

1. Terminate the EC2 instance.
2. Delete the EBS volumes left behind in the AWS Console - there will be four, since each autoscaling volume sits on a pair.

{% hint style="warning" %}
Skipping deactivation is fine here only because everything is being deleted. On a real environment, always [deactivate autoscaling before uninstalling AutoScaler](/set-up-and-installation/uninstalling-datafy) - otherwise you lose access to the data on volumes you still need.
{% endhint %}
{% endtab %}
{% endtabs %}

For more on uninstalling, see [Uninstalling Datafy](/set-up-and-installation/uninstalling-datafy).

***

## What's Next?

Now that you've seen autoscaling in action, here are some next steps:

* Learn more about how the AutoScaler works under the hood in [How AutoScaler Works](/how-it-works/how-autoscaler-works)
* Explore [Autoscaling Rules](/volume-lifecycle/autoscaling-rules) to set up tag-based and instance-based rules for your production volumes
* Set up [Datafy Snapshots](/volume-lifecycle/datafy-snapshots) for backup and recovery of your autoscaling volumes


# How AutoScaler Works

Learn what happens to your data when Datafy autoscales your EBS volumes - from activation through growing, shrinking, and deactivation

## Datafy AutoScaler

Datafy AutoScaler reduces your EBS costs by automatically right-sizing your volumes to match actual usage. This page explains what happens to your volumes at each stage of the autoscaling lifecycle - from activation through growing, shrinking, and deactivation.

{% hint style="success" %}
Throughout every stage, your applications continue running normally with **zero downtime** and no code changes required.
{% endhint %}

### Virtualization Layer

To understand how AutoScaler works, it helps to understand what changes when Datafy manages a volume - and, just as importantly, what stays the same.

When you activate autoscaling on a volume, Datafy AutoScaler creates a virtualization layer between your filesystem and the underlying EBS storage. This layer intercepts I/Os at the block device level and redirects them to Datafy-managed volumes, which are sized to match your actual data usage.

Your filesystem is never modified - whether you use ext4, XFS, or another supported Linux filesystem, it continues operating exactly as before. Your applications, your OS, and your filesystem all still see the same virtual volume at the same mount point, with the same reported size. The only difference is that behind the scenes, the data lives on smaller, right-sized EBS volumes managed by Datafy.

Think of it like a virtual address system: your applications write to the same "address" they always have, but Datafy translates that address to the actual location on the underlying storage.

<div align="center"><figure><img src="/files/pybYmsd0CE44G5xfd2Lz" alt="" width="563"><figcaption><p>Datafy Virtualization Layer</p></figcaption></figure></div>

### Datafy Volume Pairs

Under the hood, Datafy replaces your original EBS volume with a **pair** of smaller EBS volumes and stripes data across them using a RAID 0 scheme. This means that each read or write is distributed across both volumes in the pair, which helps balance I/O and throughput load.

The striping is fully managed by the Datafy agent - your filesystem and applications are unaware that multiple volumes are involved. From their perspective, nothing has changed.

{% hint style="info" %}
For simplicity, the diagrams on this page refer to a single new volume created during activation, shrink, and deactivation. In practice, each of these operations creates or replaces a **pair** of volumes that work together as a single logical unit.
{% endhint %}

## Optimization Actions

Throughout the lifecycle of an autoscaling volume Datafy AutoScaler will perform the following optimization actions:

* [#autoscaling-activation](#autoscaling-activation "mention") - the initial copy of the data from a standard EBS volume, to an autoscaling one.
* [#growing](#growing "mention") - when data is written to the volume, the managed volume will "grow", to ensure that there is always enough space.
* [#shrinking](#shrinking "mention") - when data is deleted from the volume and utilization drops, the data will be copied to a smaller volume, to minimize empty space and maximize savings.
* [#autoscaling-deactivation](#autoscaling-deactivation "mention") - if desired, the data will be copied back to a new full-sized volume, which will continue operating without AutoScaler.

### Autoscaling Activation

When you [activate autoscaling](/volume-lifecycle/managing-autoscaling-volumes#activating-autoscaling) on a volume - either manually through the All Volumes Table, or automatically using [autoscaling rules](/volume-lifecycle/autoscaling-rules) - the following steps happen:

{% stepper %}
{% step %}
**Datafy creates new, right-sized EBS volumes**

New, smaller EBS volumes are created in your AWS account. These volumes are sized based on how much data you are actually using, plus a buffer to accommodate normal write activity.
{% endstep %}

{% step %}
**Your data is copied live**

Datafy copies your data from the original volume to the new Datafy-managed volumes, while your applications keep running. The copy operations are automatically throttled to give priority to the I/Os from your application. This ensures minimal impact on your application's performance.
{% endstep %}

{% step %}
**Switch to Datafy volumes**

Once the copy is complete, AutoScaler activates the virtualization layer. All I/O is redirected to the new, smaller volumes, and the original volume is deleted. From this point on, the Datafy agent continuously monitors your storage usage and adjusts capacity automatically.
{% endstep %}
{% endstepper %}

For example, a volume provisioned at 250 GiB with 50 GiB of actual data will be replaced by Datafy-managed volumes totaling roughly 60 GiB - so you only pay for what you are actually using, plus a small buffer.

{% hint style="success" %}
Datafy never inspects or reads the contents of your data. The agent operates at the block device level, managing volume operations without any knowledge of your files or application data.

Learn more at [Datafy's trust center](https://trust.datafy.io/).
{% endhint %}

<figure><img src="/files/ABJPIkHOXWJbzB0BVOON" alt="" width="563"><figcaption></figcaption></figure>

### Growing

Datafy continuously monitors how much storage your volumes are actually using. When usage approaches the current capacity, Datafy automatically increases it, before you run out of space.

{% hint style="success" %}
To protect against sudden write bursts, Datafy maintains a buffer calculated by assuming writes could occur at the volume's maximum throughput. Your applications are protected even during heavy I/O spikes.
{% endhint %}

Growing is handled in one of two ways:

{% columns %}
{% column %}
**Modify**

When possible, Datafy uses the AWS volume modify API to enlarge the existing volumes in place.
{% endcolumn %}

{% column %}
**Append**

When a volume is in cooldown and AWS modify can't be performed, Datafy creates new volumes and appends them to the existing set. This ensures that growth is always possible with no delays, regardless of AWS modification limits.
{% endcolumn %}
{% endcolumns %}

In both cases, the process is seamless. Your applications continue running normally, and the filesystem is unaware that underlying volumes have changed.

<figure><img src="/files/xQRzjsSbkhZicfO35fAf" alt="" width="563"><figcaption></figcaption></figure>

### Shrinking

When your data usage decreases and capacity utilization drops below a [configurable threshold](/set-up-and-installation/autoscaling-configurations#shrink-configurations) (default 50%), AutoScaler initiates a shrink to reclaim unused storage and reduce costs.

Shrinking works similarly to the initial activation: AutoScaler creates new, smaller volumes, copies your data to them live, and then discards the old, larger volumes once the copy is complete. As with all other operations, this requires no downtime. The performance impact is minimal and can be throttled.

{% hint style="info" %}
If your volume receives new writes during the shrink process, and the utilization rises, AutoScaler automatically aborts the shrink operation.
{% endhint %}

<figure><img src="/files/9c8897XU1l4YcvxD4CJx" alt="" width="563"><figcaption></figcaption></figure>

### Autoscaling Deactivation

You can [deactivate autoscaling](/volume-lifecycle/managing-autoscaling-volumes#deactivating-autoscaling) on any volume at any time.

When you deactivate, AutoScaler copies your data back to a standard EBS volume at the original provisioned size - the same size the volume was before you activated autoscaling. The virtualization layer is removed, and the Datafy-managed volumes are deleted.

Once deactivation is complete, the volume operates as a normal EBS volume, fully independent of Datafy. As with every other operation, deactivation requires zero downtime.

<figure><img src="/files/bl0v2qudk9ooZ5k0ga4G" alt="" width="563"><figcaption></figcaption></figure>


# AutoScaler on Kubernetes

Learn how Datafy AutoScaler integrates with the AWS EBS CSI driver — what gets installed, how volume requests are translated, and what happens after uninstall

When a volume is autoscaling, the original EBS volume is replaced by smaller [Datafy-managed volumes](/how-it-works/how-autoscaler-works). In a Kubernetes cluster, the [PersistentVolume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that represents this volume still references the original volume ID. Whenever the matching [PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) requires an action at the volume level (such as attaching, mounting, or detaching), the AWS EBS CSI driver, which handles EBS volume operations in Kubernetes, cannot find the correct volume to act on.

Datafy bridges this gap with a conversion layer that intercepts requests received by the CSI driver and translates them to the underlying Datafy-managed volumes. For volumes Datafy does not manage, requests pass through unchanged.

## Installing Datafy on Kubernetes

[Installing AutoScaler](/set-up-and-installation/datafy-installation/installation#kubernetes-cluster) on a Kubernetes cluster adds three main components to your cluster:

* **Datafy agent** — `datafy-agent` DaemonSet, with one pod per node. Performs the same functions as in non-Kubernetes environments: monitoring utilization, orchestrating grow and shrink operations, and managing the virtualization layer.
* **Datafy controller** — `datafy-controller` Deployment. Tracks volume state and manages the integration with the CSI driver.
* **Datafy proxy** — `datafy-proxy` sidecar container added to each `ebs-csi-controller` and `ebs-csi-node` pod. Intercepts and translates CSI driver requests for managed volumes.

{% hint style="success" %}
To verify your installation, check that the following are present in your cluster:

* `datafy-agent` DaemonSet in the `datafy-agent` namespace, with one pod running per node
* `datafy-controller` Deployment in the `datafy-agent` namespace, running
* `ebs-csi-controller` and `ebs-csi-node` pods (in the EBS CSI driver's namespace) each contain a `datafy-proxy` container
  {% endhint %}

The Datafy proxy sidecar is added to CSI driver pods through a Kubernetes [admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) that the controller registers. The webhook adds the sidecar whenever a CSI driver pod starts — initial install, version upgrade, or routine restart — so the integration is preserved across CSI driver updates without manual reconfiguration. The controller also re-adds the sidecar if the `ebs-csi-controller` Deployment or `ebs-csi-node` DaemonSet drifts.

{% hint style="info" %}
Your existing StorageClasses, IAM roles, and CSI configuration remain unchanged — Datafy works alongside the standard driver, not in place of it.
{% endhint %}

## Volume Operations

The cluster operations you'd typically perform on autoscaling volumes work the same as on any other PersistentVolume — the Datafy proxy translates each request to the underlying volumes, and the standard CSI driver carries out the AWS calls using its existing IAM credentials.

* **Pod rescheduling** — when a pod moves between nodes during a drain, rolling update, eviction, or rescaling, the CSI sidecar translates the detach and re-attach calls, and the Datafy agent on each node tears down or sets up the virtualization layer. From the application's perspective, this is identical to a standard pod rescheduling event.
* **Snapshots and backups** — Kubernetes [`VolumeSnapshot`](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) resources are supported, including the CSI snapshotter integration used by tools like Velero. For details, see [Datafy Snapshots](/volume-lifecycle/datafy-snapshots#kubernetes-csi-integration).

## Uninstalling

{% hint style="danger" %}
Deactivate autoscaling on all managed volumes before uninstalling. If Datafy is removed while volumes are still under autoscaling, those volumes will not be accessible. Follow the [uninstall guide](/set-up-and-installation/uninstalling-datafy) to prepare correctly.
{% endhint %}

[Deactivating autoscaling](/volume-lifecycle/managing-autoscaling-volumes#deactivating-autoscaling) copies the data back to a new standard EBS volume. That new volume has a different ID than the one the PersistentVolume still references, so a 1:1 translation between the two is still required for the volume to remain accessible. For this reason, to keep your volumes accessible, uninstalling does not remove every Datafy component at once.

When you uninstall AutoScaler from a cluster that had autoscaling volumes:

* The Datafy agent is removed.
* The Datafy controller and the `datafy-proxy` sidecar containers remain on the cluster, performing the 1:1 translation for any PersistentVolume that was previously managed.

As those PersistentVolumes are removed from the cluster — for example, when their workloads are recycled or replaced — the number of translations needed shrinks. Once no PersistentVolume requires translation, the controller removes itself and the CSI sidecar containers. From that point on, the CSI driver operates exactly as it did before Datafy was installed.

PersistentVolumes, PersistentVolumeClaims, and application data remain intact throughout.


# Sensor FAQ

### **Getting Started**

<details>

<summary>How do I install Sensor? How long will it take?</summary>

Installing Datafy Sensor in your environment can be done with a 1-line command, and takes less than 10 minutes. To get started, follow the steps in our [installation guide](/set-up-and-installation/datafy-installation)

</details>

<details>

<summary>Will installing Sensor cause any downtime?</summary>

No, installing Datafy Sensor does not require any downtime at all

</details>

<details>

<summary>Where can I install Datafy Sensor? Which instances and volumes types are supported?</summary>

Datafy Sensor can be installed on any EC2 or EKS instance using a Linux operating system.

See the full list of [Supported Infrastructure](/set-up-and-installation/supported-infrastructure), which details:

* Supported OSs
* Filesystems
* Instance types

</details>

<details>

<summary>What can I learn from Datafy Sensor?</summary>

After installing Datafy Sensor, you will be able to view the utilization of all volumes attached to instances where Sensor was installed in the [Datafy app](https://app.datafy.io/). Utilization over time is available in the [Reports](https://app.datafy.io/reports) tab.

You can filter the results by various parameters, including volume tags, to compare utilization across different use-cases in your account.

</details>

<details>

<summary>Is it really free?</summary>

Yes, it is! You can use Datafy Sensor for as long as you'd like, free of charge

</details>

### Privacy and Security

<details>

<summary>Which AWS permissions does Datafy require? Why do you need them?</summary>

Datafy Sensor requires permissions to collect metadata about the instances and volumes in your account. We use this to associate the usage data collected by the agent with the relevant EBS volume and instance IDs, and to provide an indication whether the volumes are candidates for autoscaling.

All of the permissions required for the operation of Datafy Sensor are read-only. The full list of permissions is detailed in the [permissions setup instructions](/set-up-and-installation/datafy-installation/permissions-configuration).

</details>

<details>

<summary>Can Datafy access my data?</summary>

Datafy doesn't access, read, or use any of the data saved in your EBS volumes. The usage data collected by Datafy Sensor is based on the usage reported by the volumes' filesystems.

</details>

<details>

<summary>What information is collected by Datafy?</summary>

Datafy's application backend collects metadata about instances and volumes in your account, and the overall usage of each volume monitored by Datafy Sensor. You can see this data in the [Datafy app](https://app.datafy.io/). The data stored in your volumes is not accessed by Datafy and always remains in your account.

</details>


# AutoScaler FAQ

## Overview

<details>

<summary>What is Datafy AutoScaler?</summary>

Datafy AutoScaler is a cloud storage management solution that automatically grows and shrinks your EBS volumes based on actual usage. It virtualizes the storage layer, presenting a large virtual volume to your applications while managing the underlying EBS volumes in the background to ensure optimal utilization. The result is reduced EBS costs without any changes to your applications or workflows.

For more information, check out [how AutoScaler works](/how-it-works/how-autoscaler-works).

</details>

<details>

<summary>What is the difference between Datafy Sensor and Datafy AutoScaler?</summary>

Datafy Sensor is a read-only monitoring agent that tracks and reports on your EBS volume utilization. It does not modify your volumes.

Datafy AutoScaler includes all of Sensor's monitoring capabilities, and additionally can manage your volumes directly - automatically growing and shrinking them to match your actual usage.

</details>

<details>

<summary>How does AutoScaler save me money?</summary>

EBS volumes are often over-provisioned to avoid running out of space. Datafy eliminates this waste by continuously right-sizing your volumes based on real-time usage. When volumes are underutilized, Datafy shrinks them. When they need more space, Datafy grows them. You only pay for the capacity you're actually using.

</details>

## Getting Started

<details>

<summary>How do I install AutoScaler? How long will it take?</summary>

Installing Datafy AutoScaler can be done with a single command and takes less than 10 minutes. To get started, follow the steps in our [installation guide](/set-up-and-installation/datafy-installation).

</details>

<details>

<summary>Will installing AutoScaler cause any downtime?</summary>

No. Installing Datafy AutoScaler does not require any downtime. Your volumes remain fully accessible throughout the installation process.

</details>

<details>

<summary>Can I upgrade from Sensor to AutoScaler?</summary>

Yes. If you already have Datafy Sensor installed, you can upgrade to AutoScaler without reinstalling. Follow the steps in our [upgrade guide](/set-up-and-installation/upgrading-datafy).

</details>

<details>

<summary>Where can I install Datafy AutoScaler? Which instances and volume types are supported?</summary>

Datafy AutoScaler can be installed on EC2 and EKS instances running a supported Linux operating system. See the full list of [Supported Infrastructure](/set-up-and-installation/supported-infrastructure), which details supported OSs, filesystems, instance types, and volume types.

If AutoScaler is installed on an instance that has unsupported volumes, utilization will still be reported for these volumes, but you won't be able to activate autoscaling for these volumes.

</details>

<details>

<summary>Why aren't some of my volumes eligible for autoscaling?</summary>

A volume may not be eligible for autoscaling if its instance type, operating system, filesystem, or volume type is not currently supported. Check the [Supported Infrastructure](/set-up-and-installation/supported-infrastructure) page to confirm your configuration is covered. If your infrastructure matches the supported list and volumes still aren't eligible, contact Datafy support.

</details>

<details>

<summary>Do I have to install AutoScaler on my whole account? What about my whole cluster?</summary>

The AutoScaler installation is controlled completely by you - you can install it in on any relevant sub-set of your environment.

On Kubernetes clusters AutoScaler is installed as a daemonset, and should be installed on the entire cluster whenever possible. If you wish to install on part of your cluster, contact Datafy support.

</details>

<details>

<summary>How do I activate autoscaling for the first time?</summary>

After installing AutoScaler, your volumes are automatically discovered and visible in the [Datafy app](https://app.datafy.io). From there, you can [enable autoscaling](/volume-lifecycle/managing-autoscaling-volumes#activating-autoscaling) on individual volumes with the toggle in the AutoScale column, or on groups of volumes that match certain conditions with [autoscaling rules](/volume-lifecycle/autoscaling-rules).

</details>

## How It Works

<details>

<summary>What happens to my volumes when autoscaling is activated? How long does it take?</summary>

When autoscaling is activated for a volume, Datafy AutoScaler creates new, smaller volumes and copies the data from your volume to the new volumes. When the copy is complete, the original volume is removed, and you start saving! From here on, AutoScaler will automatically grow or shrink the underlying volumes to match the size of the data actually on the volume.

The volume and the data on it remain accessible for the entire time. The initial activation runs in the background and is designed to have minimal impact on your workload. The duration depends on the amount of data on the volume and can range from minutes to several hours for larger volumes. For more details, see [how autoscaling works](/how-it-works/how-autoscaler-works).

</details>

<details>

<summary>Does autoscaling my volume cause any downtime?</summary>

No. Autoscaling volumes does not cause any downtime. Your volumes remain fully accessible throughout the initial activation and any following growing and shrinking actions.

</details>

<details>

<summary>What happens when a volume grows?</summary>

When an autoscaling volume's usage increases and it begins to fill up, AutoScaler automatically adds capacity by modifying the underlying volumes when possible, or by adding new volumes when needed. The grow operation happens in the background with zero downtime - your applications continue running without interruption. See [How AutoScaler Works](/how-it-works/how-autoscaler-works#growing) for more details on how growing works.

</details>

<details>

<summary>What happens if I write data to my volume during autoscaling activation?</summary>

AutoScaler continuously monitors the storage used by your volume and adjusts the size to match it, even during the initial activation. If enough data is written during the initial activation the volume will grow automatically, and complete copying your data to a larger volume size.

</details>

<details>

<summary>Will I never run out of space?</summary>

AutoScaler continuously monitors your volumes and grows them before they run out of space, so the full originally provisioned size that your application expects to see is always available to it. If you need to grow beyond the original size of your volume (the size of the filesystem), this can be configured through the Datafy API.

</details>

<details>

<summary>What happens when a volume shrinks? How long does it take?</summary>

When a volume's utilization drops below a defined threshold, AutoScaler reclaims the unused space by copying data to new, smaller volumes. The shrink operation runs in the background and is designed to have minimal impact on your workload. The duration depends on the amount of data on the volume and can range from minutes to several hours for larger volumes. For more information, see [how shrink works](/how-it-works/how-autoscaler-works#shrinking).

</details>

<details>

<summary>Can I reverse autoscaling on a volume?</summary>

Yes. You can disable autoscaling on any volume at any time. When autoscaling is disabled, your data is copied back to a standard EBS volume matching the original configuration, and the volume is returned to its unmanaged state. This process runs in the background with no downtime. For more information see [how to deactivate a volume](/volume-lifecycle/managing-autoscaling-volumes#deactivating-autoscaling) and [how it works](/how-it-works/how-autoscaler-works#autoscaling-deactivation).

</details>

<details>

<summary>What happens if Datafy's control plane is unreachable?</summary>

If the Datafy control plane becomes temporarily unavailable, your data remains fully accessible and your applications are not affected. Mission-critical actions - including data access and volume growth - continue to operate locally on the instance, independent of the control plane. Operations that require coordination with the control plane (such as shrink) will resume once connectivity is restored.

</details>

<details>

<summary>Does Datafy use AI for optimization?</summary>

No. Datafy's autoscaling decisions are deterministic and rule-based. Datafy does not use AI or machine learning for optimization actions. Volume scaling is driven by real-time usage data and configurable thresholds.

</details>

## Privacy & Security

<details>

<summary>Can Datafy access my data?</summary>

No. Datafy does not access, read, or use any of the data stored in your EBS volumes. AutoScaler operates at the block storage level and only interacts with volume metadata and filesystem-reported usage.

</details>

<details>

<summary>Which AWS permissions does AutoScaler require? Why?</summary>

Datafy AutoScaler requires permissions to monitor and modify EBS volumes in your account - including creating, resizing, and attaching volumes - as well as read permissions for instance and volume metadata. The full list of required permissions is detailed in the [permissions setup instructions](/set-up-and-installation/datafy-installation/permissions-configuration).

</details>

<details>

<summary>Does AutoScaler upgrade automatically? Who controls agent updates?</summary>

Datafy does not push updates automatically, you control when AutoScaler is updated. Agent upgrades are performed by you, following the steps in the [upgrade guide](/set-up-and-installation/upgrading-datafy).

</details>

## Compatibility & Integration

<details>

<summary>Does AutoScaler work with EBS Snapshots?</summary>

Yes. Datafy supports EBS Snapshots. See [Datafy Snapshots](/volume-lifecycle/datafy-snapshots) for details on how snapshots work with autoscaled volumes.

</details>

<details>

<summary>Is AutoScaler compatible with Terraform, Pulumi, or other IaC tools?</summary>

Yes. Datafy is designed to work alongside your existing infrastructure-as-code workflows. See the [IaC Reconciliation](/set-up-and-installation/iac-reconciliation) page for details on how Datafy handles reconciliation with tools like Terraform and Pulumi.

</details>

<details>

<summary>Does AutoScaler work with pod rescheduling tools like Cast AI or Karpenter?</summary>

Yes. AutoScaler works alongside pod rescheduling and node management tools like Cast AI and Karpenter. When pods are rescheduled to a different node, [AutoScaler follows the volumes](/how-it-works/autoscaler-on-kubernetes#volume-operations) as they move. Note that the instance types provisioned by your node provisioner, such as Cast AI or Karpenter, must be within AutoScaler's [supported instance types](/set-up-and-installation/supported-infrastructure).

</details>

## Scale and Performance

<details>

<summary>How does AutoScaler handle large-scale accounts?</summary>

All autoscaling actions (grow, shrink, data access) are performed locally on the instance, so they are not affected by account scale. Whether you have 10 volumes or 10,000, the performance of autoscaling operations on each instance remains the same. The one constraint to be aware of is that copying data (such as during a shrink) is limited to one volume at a time per instance, to minimize resource usage. Multiple instances can copy concurrently.

</details>

<details>

<summary>I have many AWS accounts. How does Datafy handle multi-account environments?</summary>

Each AWS account is associated with its own Datafy account. You can create as many accounts as you need, and view them all in the [Datafy App](https://app.datafy.io).

</details>

<details>

<summary>How do I activate autoscaling at scale, or in a dynamic environment?</summary>

For environments where volumes are frequently created or where you want to avoid manual activation, you can use [Autoscaling Rules](/volume-lifecycle/autoscaling-rules). Rules automatically enable autoscaling on new volumes that match your defined criteria, without any manual intervention.

</details>

<details>

<summary>Is the performance of my volumes impacted?</summary>

In steady state, there is no performance impact on your volumes. During a grow operation, the process is fast and transparent to your applications. During a shrink operation, data is copied in the background. This process is throttled to use only excess I/O capacity, so the impact on your running workloads is minimal.

</details>


