Upgrade

Upgrade a kagent 1.x installation and Agent Substrate to a later release.

An installation of kagent consists of separate Helm releases for the custom resource definitions (CRDs), Agent Substrate, and the kagent control plane. Upgrade the kagent CRDs first, then Agent Substrate, and then the kagent control plane.

Upgrade from one minor release to the next, such as from 1.0 to 1.1, and do not skip a minor release. Before you move to the next minor release, upgrade to the latest patch version of your current minor release. Patch releases do not change the database schema, so you can upgrade directly to the latest patch release without the patch releases in between. To move a 0.10.x installation to 1.x, see Upgrade from 0.x instead.

Prepare to upgrade

Complete every step in this section before you change any release. The steps make sure that Helm manages the installation, back up the database, save the configuration of each release, and set the target versions that the upgrade commands use.

If you installed with the kagent CLI, export the installation to Helm first. The kagent install command is for getting started, and the upgrade commands in this guide use Helm. For the steps, see Export a CLI installation to Helm. An installation that you installed with Helm needs no export.
Read the kagent release notes for every release between your current version and the target version. A release can rename or remove a Helm value, and the chart fails to render when a removed value is still set.
Confirm the installed version of each release.
```
helm list -A --filter 'kagent|substrate'
```
The CHART column shows the version of each release.
- The kagent and kagent-crds releases are in the kagent namespace.
- The substrate, substrate-atelet, and substrate-crds releases are in the ate-system namespace.
- The substrate-podcert release is in the podcertificate-controller-system namespace.
If the output does not list a substrate-atelet release, the substrate release still runs atelet, and the rest of this guide does not apply to that installation. First, back up the PostgreSQL database and record its schema version, as this section describes. Then follow the steps in Upgrading from v1.0.0-rc2 instead of the remaining steps of this guide. Those steps move atelet to its own release without deleting the atelet DaemonSet, turn on the GoldenEgressPolicy gate, and upgrade kagent. To confirm the result, see Verify the upgrade.
Back up the PostgreSQL database that kagent and Agent Substrate share. The upgrade keeps your data. However, the kagent controller and the Agent Substrate API server change the database schema when they start on the new version. To return to the previous version, you must restore the backup or roll the schema back.
```
pg_dump "postgresql://<user>:<password>@<host>:5432/kagent" \
  --format=custom \
  --file=kagent-backup.dump
```
Record the schema version of the database as well. To roll the schema back without restoring the backup, you move the schema to this version.
```
kagent db migrate version \
  --db-url "postgresql://<user>:<password>@<host>:5432/kagent" \
  --db-role <owner-role>
```
Save the values of each release to a file. The upgrade commands reapply these files, so each release keeps its database settings and model provider settings. The --reuse-values flag also reapplies the defaults of the previous chart version, and the new chart can reject a default that it removed. Do not use that flag.
```
helm get values substrate-podcert -n podcertificate-controller-system -o yaml > substrate-podcert-values.yaml
helm get values substrate-atelet -n ate-system -o yaml > substrate-atelet-values.yaml
helm get values substrate -n ate-system -o yaml > substrate-values.yaml
helm get values kagent -n kagent -o yaml > kagent-values.yaml
```
For an installation that you exported from the kagent CLI, the kagent-values.yaml file contains your model provider API key in plain text. Store the files securely.
In substrate-values.yaml and substrate-atelet-values.yaml, turn on the GoldenEgressPolicy gate. The gate lets kagent give the Actor that builds each golden snapshot its own egress policy, so that the Actor can fetch its configuration from the kagent API. If the preview list in either file does not contain GoldenEgressPolicy, add it to both lists. The two lists must match.
```
preview:
  - PerTemplateSnapshotLocation
  - GoldenEgressPolicy
```
Save the target versions in environment variables. The following values are the versions that this documentation describes. To upgrade across more than one minor release, set the versions of the next minor release, and repeat the upgrade for each minor release.
```
export KAGENT_VERSION=1.0.0-rc3
export SUBSTRATE_VERSION=1.0.0
```
Save the installed kagent chart version. The Agent Substrate upgrade uses this version to change the Worker image without upgrading kagent.
```
export CURRENT_KAGENT_VERSION=$(helm get metadata kagent -n kagent -o json | jq -r .version)
```

Upgrade the kagent CRDs

Upgrade the kagent CRDs before the controller that reads them. The chart ships the CRDs as templates, so helm upgrade updates the CRDs in place.

Upgrade the kagent CRDs.

helm upgrade kagent-crds \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
  --version ${KAGENT_VERSION} \
  --namespace kagent --wait

Confirm that the CRD release reports the target version.
```
helm list -n kagent --filter 'crds'
```
In the output, verify that the STATUS column reports deployed, and that the CHART column shows kagent-crds-<version> with the target kagent version.

Upgrade Agent Substrate

Upgrade atelet first, then the Worker image of each WorkerPool, then the Agent Substrate CRDs and the substrate release. Agent Substrate does not support a Worker on an older atelet, or a control plane that is newer than its Workers. A microVM WorkerPool that is left on the previous Worker image cannot start or restore Actors after the substrate upgrade.

Upgrade the PodCertificate controller.

helm upgrade substrate-podcert \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate-podcert \
  --version ${SUBSTRATE_VERSION} \
  --namespace podcertificate-controller-system \
  --wait --timeout 5m \
  -f substrate-podcert-values.yaml

Upgrade atelet, and wait for the rollout to finish. Do not continue until the output reports that the atelet DaemonSet rolled out.

helm upgrade substrate-atelet \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate-atelet \
  --version ${SUBSTRATE_VERSION} \
  --namespace ate-system \
  --wait --timeout 1h \
  -f substrate-atelet-values.yaml
kubectl rollout status daemonset/atelet -n ate-system --timeout=1h

Example output:

daemon set "atelet" successfully rolled out

Move the kagent-default WorkerPool to the new Worker image through the kagent release. The command keeps the installed kagent chart, so the --reuse-values flag changes only the Worker image.
- For a microVM WorkerPool, use the ateworker-microvm image.
- If Git or a pipeline applies your other WorkerPools, change workerImage there too.
```
helm upgrade kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --version ${CURRENT_KAGENT_VERSION} \
  --namespace kagent --reuse-values \
  --set substrateWorkerPool.workerImage=ghcr.io/kagent-dev/substrate/ateworker-gvisor:v${SUBSTRATE_VERSION} \
  --wait
```

Confirm that every WorkerPool lists the new image.

kubectl get workerpools -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.workerImage'

In the output, verify that the IMAGE column of every WorkerPool ends in :v<target-version>.

NAMESPACE   NAME             IMAGE
kagent      kagent-default   ghcr.io/kagent-dev/substrate/ateworker-gvisor:v1.0.0

Upgrade the Agent Substrate CRDs.

helm upgrade substrate-crds \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate-crds \
  --version ${SUBSTRATE_VERSION} \
  --namespace ate-system --wait

Upgrade the Agent Substrate control plane and data plane.

helm upgrade substrate \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate \
  --version ${SUBSTRATE_VERSION} \
  --namespace ate-system \
  --wait --timeout 10m \
  -f substrate-values.yaml

Confirm that the Agent Substrate pods are running.
```
kubectl get pods -n ate-system
```
In the output, verify that every pod reports Running and that every container is ready. A pod that belongs to a Job, such as the snapshot bucket setup Job, reports Completed.

Upgrade kagent

Upgrade the kagent control plane. The upgrade has no separate migration step.

Upgrade kagent. The saved values pin the Worker image to the previous Agent Substrate version, so the command sets the image again after the values file. A later -f file overrides an earlier one.

helm upgrade kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --version ${KAGENT_VERSION} \
  --namespace kagent --timeout 12m \
  -f kagent-values.yaml \
  -f - <<EOF
substrateWorkerPool:
  workerImage: "ghcr.io/kagent-dev/substrate/ateworker-gvisor:v${SUBSTRATE_VERSION}"
EOF

Wait for the controller to roll out.

kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s

deployment "kagent-controller" successfully rolled out

Verify the upgrade

Confirm that every release runs the new version and that the agents are ready.

Confirm that each release reports the target version.
```
helm list -A --filter 'kagent|substrate'
```
For the kagent and kagent-crds releases, the CHART column shows the target kagent version. For the substrate, substrate-atelet, substrate-crds, and substrate-podcert releases, the CHART column shows the target Agent Substrate version.

Confirm that the kagent pods are running.

kubectl get pods -n kagent

In the output, verify that every pod reports Running. An installation that uses the development database also lists a kagent-postgresql pod.

NAME                                              READY   STATUS    RESTARTS   AGE
kagent-controller-56c67b6cd7-hqj5z                1/1     Running   0          2m
kagent-default-774d4496bd-sfx4b                   1/1     Running   0          2m
kagent-postgresql-856b475f59-n258n                2/2     Running   0          2m
kagent-tools-54959b659c-lfkrz                     1/1     Running   0          2m
kagent-ui-5fdb6fd85c-vzb2z                        1/1     Running   0          2m

If the controller restarts in a loop, see The controller restarts after an upgrade.

Confirm that the WorkerPool reports a ready replica.

kubectl get workerpools -n kagent

In the output, verify that the READY column matches the DESIRED column, and that the RESOLVED column reports True.

NAME             DESIRED   REPLICAS   READY   SANDBOXCONFIG    RESOLVED   AGE
kagent-default   1         1          1       gvisor-default   True       2m

Confirm that your agents are ready.
```
kubectl get agents -A
```
In the output, verify that every Agent reports True in the READY column.
```
NAMESPACE   NAME       READY   AGE
kagent      my-agent   True    3d
```

Next steps

Operational considerationsReview the production values for replicas, resources, and storage. DebugDiagnose a controller or Agent that does not become ready.

Debug Upgrade from 0.x