Node syncer and virtual scheduler can cause conflict with node.kubernetes.io/unreachable taint

[steved]

What happened?

When starting a vcluster with fromHost.nodes.enabled=true and controlPlane.distro.k8s.scheduler.enabled=true, if a host cluster node is already down and unreachable, the virtual scheduler and node syncer can race to apply the node.kubernetes.io/unreachable taint.

node-repro-0 syncer 2026-07-31 20:39:18 ERROR   controller/controller.go:494    Reconciler error        {"component": "vcluster", "controller": "node", "namespace": "", "name": "vcluster-node-repro-worker", "reconcileID": "8039305c-c46d-4399-a5f0-c9200de29157", "error": "sync: patch virtual object: update object: Node \"vcluster-node-repro-worker\" is invalid: [metadata.taints[2]: Duplicate value: {\"Key\":\"node.kubernetes.io/unreachable\",\"Value\":\"\",\"Effect\":\"NoSchedule\",\"TimeAdded\":\"2026-07-31T20:39:08Z\"}: taints must be unique by key and effect pair, metadata.taints[3]: Duplicate value: {\"Key\":\"node.kubernetes.io/unreachable\",\"Value\":\"\",\"Effect\":\"NoExecute\",\"TimeAdded\":\"2026-07-31T20:39:18Z\"}: taints must be unique by key and effect pair]"}

What did you expect to happen?

The reconcile completes cleanly.

How can we reproduce it (as minimally and precisely as possible)?

Details

repro.sh:

#!/usr/bin/env bash
set -Eeuo pipefail

SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
readonly SCRIPT_DIR
readonly CLUSTER_NAME="vcluster-node-repro"
readonly HOST_CONTEXT="kind-${CLUSTER_NAME}"
readonly VCLUSTER_NAME="node-repro"
readonly VCLUSTER_NAMESPACE="vcluster"
readonly VCLUSTER_VERSION="0.36.0"
readonly UNREACHABLE_KEY="node.kubernetes.io/unreachable"

require() {
  if ! command -v "$1" >/dev/null 2>&1; then
    echo "error: required command not found: $1" >&2
    exit 1
  fi
}

host_kubectl() {
  kubectl --context "${HOST_CONTEXT}" "$@"
}

virtual_kubectl() {
  vcluster connect "${VCLUSTER_NAME}" \
    --namespace "${VCLUSTER_NAMESPACE}" \
    --context "${HOST_CONTEXT}" \
    --background-proxy=false \
    --silent \
    -- kubectl "$@"
}

for command_name in docker kind kubectl helm vcluster; do
  require "${command_name}"
done

if ! docker info >/dev/null 2>&1; then
  echo "error: Docker is not running" >&2
  exit 1
fi

if ! vcluster --version | grep -Fq "${VCLUSTER_VERSION}"; then
  echo "error: vcluster CLI ${VCLUSTER_VERSION} is required" >&2
  exit 1
fi

if kind get clusters | grep -Fxq "${CLUSTER_NAME}"; then
  echo "error: Kind cluster ${CLUSTER_NAME} already exists" >&2
  echo "remove it with: kind delete cluster --name ${CLUSTER_NAME}" >&2
  exit 1
fi

tmp_dir=$(mktemp -d)
trap 'rm -rf "${tmp_dir}"' EXIT

kind_config="${tmp_dir}/kind.yaml"
cat >"${kind_config}" <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
EOF

echo "==> Creating the Kind host cluster"
kind create cluster \
  --name "${CLUSTER_NAME}" \
  --config "${kind_config}" \
  --wait 2m

control_plane_node="${CLUSTER_NAME}-control-plane"
host_kubectl label node "${control_plane_node}" repro-control-plane=true

worker_node="${CLUSTER_NAME}-worker"
echo "==> Waiting for the Kind worker to become healthy"
host_ready=""
for _ in $(seq 1 60); do
  host_ready=$(host_kubectl get node "${worker_node}" \
    -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
  if [[ "${host_ready}" == "True" ]]; then
    break
  fi
  sleep 1
done
host_taints=$(host_kubectl get node "${worker_node}" -o jsonpath='{.spec.taints}')
if [[ "${host_ready}" != "True" || "${host_taints}" == *"${UNREACHABLE_KEY}"* ]]; then
  echo "FAIL: the host worker was not healthy and untainted before the test" >&2
  exit 1
fi

echo "==> Stopping the Kind worker before vCluster starts"
docker stop "${worker_node}" >/dev/null

echo "==> Waiting for the host node-lifecycle controller to mark the worker unreachable"
host_ready=""
host_taints=""
for _ in $(seq 1 90); do
  host_ready=$(host_kubectl get node "${worker_node}" \
    -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
  host_taints=$(host_kubectl get node "${worker_node}" -o jsonpath='{.spec.taints}')
  if [[ "${host_ready}" == "Unknown" && "${host_taints}" == *"${UNREACHABLE_KEY}"* ]]; then
    break
  fi
  sleep 2
done

if [[ "${host_ready}" != "Unknown" || "${host_taints}" != *"${UNREACHABLE_KEY}"* ]]; then
  echo "FAIL: the stopped host worker did not become unreachable" >&2
  exit 1
fi

echo "Observed pre-existing host taint: ${host_taints}"
host_taint_time=$(host_kubectl get node "${worker_node}" \
  -o jsonpath='{range .spec.taints[?(@.effect=="NoExecute")]}{.timeAdded}{end}')

echo "==> Installing stock vCluster ${VCLUSTER_VERSION}"
helm upgrade --install "${VCLUSTER_NAME}" vcluster \
  --repo https://charts.loft.sh \
  --version "${VCLUSTER_VERSION}" \
  --namespace "${VCLUSTER_NAMESPACE}" \
  --create-namespace \
  --kube-context "${HOST_CONTEXT}" \
  --values "${SCRIPT_DIR}/values.yaml" \
  --wait \
  --timeout 5m

echo "==> Waiting for ${worker_node} to appear in the virtual cluster"
for _ in $(seq 1 60); do
  if virtual_kubectl get node "${worker_node}" >/dev/null 2>&1; then
    break
  fi
  sleep 1
done
virtual_kubectl get node "${worker_node}" >/dev/null

echo "==> Waiting for vCluster to attempt the invalid merged update"
match=""
for _ in $(seq 1 60); do
  logs=$(host_kubectl logs \
    --namespace "${VCLUSTER_NAMESPACE}" \
    "statefulset/${VCLUSTER_NAME}" \
    --since=2m 2>&1 || true)
  match=$(printf '%s\n' "${logs}" | grep -F "taints must be unique by key and effect pair" | tail -1 || true)
  if [[ -n "${match}" ]]; then
    break
  fi
  sleep 1
done

if [[ -z "${match}" ]]; then
  echo "FAIL: vCluster did not emit the expected duplicate-taint error" >&2
  echo "Last 200 vCluster log lines:" >&2
  printf '%s\n' "${logs}" | tail -200 >&2
  exit 1
fi

virtual_taint_time=$(printf '%s\n' "${match}" \
  | grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:.]+Z' \
  | tail -1)
if [[ -z "${virtual_taint_time}" || "${virtual_taint_time}" == "${host_taint_time}" ]]; then
  echo "FAIL: the error did not contain a distinct virtual-controller taint timestamp" >&2
  printf '%s\n' "${match}" >&2
  exit 1
fi

echo
echo "PASS: reproduced the vCluster duplicate-taint bug"
printf '%s\n' "${match}"
echo "host taint time:    ${host_taint_time}"
echo "virtual taint time: ${virtual_taint_time}"
echo
echo "The Kind cluster was left running for inspection."
echo "The worker container is intentionally stopped."
echo "Cleanup: kind delete cluster --name ${CLUSTER_NAME}"

Anything else we need to know?

No response

Host cluster Kubernetes version

Details

$ kubectl version
Client Version: v1.36.3
Kustomize Version: v5.8.1
Server Version: v1.36.1

vcluster version

Details

$ vcluster --version
vcluster version 0.36.0

VCluster Config

Details

sync:
  fromHost:
    nodes:
      enabled: true
      selector:
        all: true

controlPlane:
  distro:
    k8s:
      enabled: true
      controllerManager:
        extraArgs:
          - --node-monitor-grace-period=15s
          - --node-monitor-period=100ms
          - --node-startup-grace-period=100ms
      scheduler:
        enabled: true
  statefulSet:
    scheduling:
      nodeSelector:
        repro-control-plane: "true"
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule