Kubernetes Node Scheduling

Advanced node scheduling is supported from Helm chart version 3.6.0 with agent version 1.39.0.

An Autoscaling volume can only be mounted or accessed on a node running a healthy Datafy AutoScaler agent — the agent is what translates the underlying physical EBS volumes into a single virtual device. A pod with an Autoscaling volume that's scheduled to a node without a healthy AutoScaler can't mount its volume and never starts — and since Kubernetes has no reason to reschedule a pod stuck on a failed mount, it won't self-recover. A pod with an Autoscaling volume that's already running loses access to its volume's data if the agent stops running or responding.

To prevent both, Datafy uses taints and tolerations and node affinity as complementary mechanisms:

• Node affinity, inherited from your Helm installation, protects against a pod migrating to a node AutoScaler was never installed on — for example, when AutoScaler is only installed on part of the cluster.

• Node taints protect against a node where AutoScaler is installed but not currently healthy.

This is a common way to ensure pods run only on nodes with the infrastructure they need — for example, Cilium taints a node until its own networking is up, and both the NVIDIA GPU Operator and AWS EKS taint GPU nodes so only pods that request a GPU are scheduled there.

How Datafy Manages Scheduling

The Datafy controller watches pods and nodes and applies taints, tolerations, and affinity based on their state.

Whenever a pod is scheduled or rescheduled — including when it migrates to a different node — the Datafy controller checks whether it has an Autoscaling volume:

• If it does, the controller applies the node affinity from your AutoScaler Helm installation to the pod, so it's only ever scheduled where AutoScaler runs.

• If it doesn't, the controller adds a toleration for the datafy.io taint instead, so the pod can be scheduled anywhere regardless of AutoScaler's health there.

Separately, Datafy watches every node AutoScaler is installed on and keeps its datafy.io taint in sync with that node's health:

When	What happens
A node joins the cluster and AutoScaler hasn't been healthy there yet	The startup taint you configure blocks scheduling of pods with Autoscaling volumes until AutoScaler is up
AutoScaler and its CSI sidecar are confirmed healthy on that node — on first boot, after recovering, or after the node returns to AutoScaler's scope	Datafy removes the datafy.io taint; any pod can now be scheduled there
AutoScaler or its CSI sidecar on an already-healthy node stops responding	Datafy re-applies the datafy.io taint with NoExecute, evicting any pod with an Autoscaling volume
AutoScaler is removed from a node's scope — a narrower agent.affinity.nodeAffinity or a dropped node group	Datafy applies the datafy.io taint with NoExecute, evicting any pod with an Autoscaling volume

Cluster Setup and Configuration

Node Taints

Add a startup taint to each node group AutoScaler runs on. A taint blocks Kubernetes from scheduling a pod on the node unless it carries a matching toleration — you only need to add the taint itself; Datafy's controller adds the matching toleration automatically to any pod that doesn't need to avoid the node.

Use the taint key format your cluster autoscaler recognizes as temporary, shown in the tabs below. A cluster autoscaler decides whether to add a new node by simulating whether a pending pod would schedule on it — if it doesn't recognize your taint as temporary, it sees the taint on the simulated node, concludes the pod still couldn't schedule there, and may decide against scaling up at all.

Cluster Autoscaler

Add the taint to your node group's configuration. This example uses the node_group_taints variable from the terraform-aws-modules/eks module — adapt the key names if you manage node groups a different way. The startup-taint.cluster-autoscaler.kubernetes.io/ prefix is what tells Cluster Autoscaler to disregard this taint when simulating whether a new node would let a pending pod schedule:

eks-node-groups.tf

node_group_taints = {
  dedicated = {
    key    = "startup-taint.cluster-autoscaler.kubernetes.io/datafy.io"
    value  = "autoscaler"
    effect = "NO_SCHEDULE"
  }
}

Karpenter

Add startupTaints to the NodePool that provisions the nodes AutoScaler runs on. Karpenter recognizes startupTaints as temporary and excludes them from its own provisioning and scheduling decisions, so no special key prefix is needed:

nodepool.yaml

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: datafy-autoscaler
spec:
  template:
    spec:
      startupTaints:
        - key: datafy.io
          effect: NoSchedule
      # ...the rest of your existing NodePool spec

Once the taint is applied, standard Kubernetes scheduling takes over — the scheduler won't place a pod on the node unless it tolerates the taint, the same as with any other taint. The cluster-autoscaler-specific handling above only affects scale-up decisions, not the scheduler itself.

To confirm the taint applied, run kubectl describe node <node-name> and look for a line under Taints: matching the key you configured.

Datafy's controller automatically adds a toleration for both taint formats to any pod that doesn't need to avoid nodes without AutoScaler — for example:

tolerations:
  - key: datafy.io
    operator: Exists
  - key: startup-taint.cluster-autoscaler.kubernetes.io/datafy.io
    operator: Exists

Datafy's controller only updates a pod's tolerations and affinity when it's (re)scheduled, not immediately when Autoscaling is activated or deactivated on its volume. A mismatch between a volume's state and its pod's taints or affinity is temporary and resolves the next time the pod is scheduled.

Partial Cluster Installation

If your cluster includes node types where AutoScaler isn't supported, use node affinity to install AutoScaler only on the nodes that support it.

1

Set the Node Affinity in the Helm Chart

Set agent.affinity.nodeAffinity to match the node group(s) or instance types you're installing AutoScaler on. The Helm chart uses this value to install the datafy-agent DaemonSet only on those nodes, and the Datafy controller applies the same affinity to any pod using an Autoscaling volume.

values.yaml

agent:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-group
                operator: In
                values:
                  - dedicated

The Helm chart always excludes legacy, non-Nitro instance types from AutoScaler mode, regardless of this setting. Any affinity you define here is combined with that default, not instead of it.

2

Install AutoScaler

Install or upgrade using the values file from the previous step, following the standard Helm chart installation.