> For the complete documentation index, see [llms.txt](https://docs.datafy.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datafy.io/set-up-and-installation/advanced-setup/kubernetes-node-scheduling.md).

# Kubernetes Node Scheduling

{% hint style="success" %}
Advanced node scheduling is supported from Helm chart version 3.6.0 with agent version 1.39.0.
{% endhint %}

An Autoscaling volume can only be mounted or accessed on a node running a healthy Datafy AutoScaler agent — the agent is what translates the underlying physical EBS volumes into a [single virtual device](/how-it-works/how-autoscaler-works.md). A pod with an Autoscaling volume that's scheduled to a node without a healthy AutoScaler can't mount its volume and never starts — and since Kubernetes has no reason to reschedule a pod stuck on a failed mount, it won't self-recover. A pod with an Autoscaling volume that's already running loses access to its volume's data if the agent stops running or responding.

To prevent both, Datafy uses [taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) and [node affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) as complementary mechanisms:

* [Node affinity](#partial-cluster-installation), inherited from your Helm installation, protects against a pod being rescheduled onto a node AutoScaler was never installed on, for example when AutoScaler is only installed on part of the cluster.
* [Node taints](#node-taints) protect against a node where AutoScaler is installed but not currently healthy.

{% hint style="info" %}
This is a common way to ensure pods run only on nodes with the infrastructure they need — for example, Cilium taints a node until its own networking is up, and both the NVIDIA GPU Operator and AWS EKS taint GPU nodes so only pods that request a GPU are scheduled there.
{% endhint %}

## How Datafy Manages Scheduling

The [Datafy controller](/how-it-works/autoscaler-on-kubernetes.md) watches pods and nodes and applies taints, tolerations, and affinity based on their state.

Whenever a pod is scheduled or rescheduled, including when it moves to a different node, the Datafy controller checks whether it has an Autoscaling volume:

* If it does, the controller applies the [node affinity](#partial-cluster-installation) from your AutoScaler Helm installation to the pod, so it's only ever scheduled where AutoScaler runs.
* If it doesn't, the controller adds a toleration for the `datafy.io` taint instead, so the pod can be scheduled anywhere regardless of AutoScaler's health there.

Separately, Datafy watches every node AutoScaler is installed on and keeps its `datafy.io` taint in sync with that node's health:

| When                                                                                                                                                 | What happens                                                                                                              |
| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| A node joins the cluster and AutoScaler hasn't been healthy there yet                                                                                | The [startup taint you configure](#node-taints) blocks scheduling of pods with Autoscaling volumes until AutoScaler is up |
| AutoScaler and its CSI sidecar are confirmed healthy on that node — on first boot, after recovering, or after the node returns to AutoScaler's scope | Datafy removes the `datafy.io` taint; any pod can now be scheduled there                                                  |
| AutoScaler or its CSI sidecar on an already-healthy node stops responding                                                                            | Datafy re-applies the `datafy.io` taint with `NoExecute`, evicting any pod with an Autoscaling volume                     |
| AutoScaler is removed from a node's scope — a narrower `agent.affinity.nodeAffinity` or a dropped node group                                         | Datafy applies the `datafy.io` taint with `NoExecute`, evicting any pod with an Autoscaling volume                        |

## Cluster Setup and Configuration

### Node Taints

Add a [startup taint](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to each node group AutoScaler runs on. A taint blocks Kubernetes from scheduling a pod on the node unless it carries a matching toleration — you only need to add the taint itself; Datafy's controller adds the matching toleration [automatically](#how-datafy-manages-scheduling) to any pod that doesn't need to avoid the node.

{% hint style="info" %}
Use the taint key format your cluster autoscaler recognizes as temporary, shown in the tabs below. A cluster autoscaler decides whether to add a new node by simulating whether a pending pod would schedule on it — if it doesn't recognize your taint as temporary, it sees the taint on the simulated node, concludes the pod still couldn't schedule there, and may decide against scaling up at all.
{% endhint %}

{% tabs %}
{% tab title="Cluster Autoscaler" %}
Add the taint to your node group's configuration. This example uses the `node_group_taints` variable from the [`terraform-aws-modules/eks`](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest/submodules/eks-managed-node-group) module — adapt the key names if you manage node groups a different way. The `startup-taint.cluster-autoscaler.kubernetes.io/` prefix is what tells Cluster Autoscaler to disregard this taint when simulating whether a new node would let a pending pod schedule:

{% code title="eks-node-groups.tf" overflow="wrap" %}

```hcl
node_group_taints = {
  dedicated = {
    key    = "startup-taint.cluster-autoscaler.kubernetes.io/datafy.io"
    value  = "autoscaler"
    effect = "NO_SCHEDULE"
  }
}
```

{% endcode %}
{% endtab %}

{% tab title="Karpenter" %}
Add `startupTaints` to the `NodePool` that provisions the nodes AutoScaler runs on. Karpenter recognizes `startupTaints` as temporary and excludes them from its own provisioning and scheduling decisions, so no special key prefix is needed:

{% code title="nodepool.yaml" overflow="wrap" %}

```yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: datafy-autoscaler
spec:
  template:
    spec:
      startupTaints:
        - key: datafy.io
          effect: NoSchedule
      # ...the rest of your existing NodePool spec
```

{% endcode %}
{% endtab %}
{% endtabs %}

Once the taint is applied, standard Kubernetes scheduling takes over — the scheduler won't place a pod on the node unless it tolerates the taint, the same as with any other taint. The cluster-autoscaler-specific handling above only affects scale-up decisions, not the scheduler itself.

{% hint style="success" %}
To confirm the taint applied, run `kubectl describe node <node-name>` and look for a line under `Taints:` matching the key you configured.
{% endhint %}

Datafy's controller automatically adds a toleration for both taint formats to any pod that doesn't need to avoid nodes without AutoScaler — for example:

{% code overflow="wrap" %}

```yaml
tolerations:
  - key: datafy.io
    operator: Exists
  - key: startup-taint.cluster-autoscaler.kubernetes.io/datafy.io
    operator: Exists
```

{% endcode %}

{% hint style="info" %}
Datafy's controller only updates a pod's tolerations and affinity when it's (re)scheduled, not immediately when Autoscaling is activated or deactivated on its volume. A mismatch between a volume's state and its pod's taints or affinity is temporary and resolves the next time the pod is scheduled.
{% endhint %}

### Partial Cluster Installation

If your cluster includes node types where [AutoScaler isn't supported](/set-up-and-installation/supported-infrastructure.md), use node affinity to install AutoScaler only on the nodes that support it.

{% stepper %}
{% step %}

#### **Set the Node Affinity in the Helm Chart**

Set `agent.affinity.nodeAffinity` to match the node group(s) or instance types you're installing AutoScaler on. The Helm chart uses this value to install the `datafy-agent` DaemonSet only on those nodes, and the [Datafy controller applies](#how-datafy-manages-scheduling) the same affinity to any pod using an Autoscaling volume.

{% code title="values.yaml" overflow="wrap" %}

```yaml
agent:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-group
                operator: In
                values:
                  - dedicated
```

{% endcode %}

{% hint style="info" %}
The Helm chart always excludes legacy, non-[Nitro](/set-up-and-installation/supported-infrastructure.md) instance types from AutoScaler mode, regardless of this setting. Any affinity you define here is combined with that default, not instead of it.
{% endhint %}
{% endstep %}

{% step %}

#### **Install AutoScaler**

Install or upgrade using the values file from the previous step, following the [standard Helm chart installation](/set-up-and-installation/datafy-installation/installation.md#helm-chart-installation).
{% endstep %}
{% endstepper %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.datafy.io/set-up-and-installation/advanced-setup/kubernetes-node-scheduling.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.