--- title: "vSphere Kubernetes Service (VKS)" description: "Install Calico Enterprise as a Supervisor Service on vSphere Kubernetes Service (VKS) and deploy it to guest clusters." product: "Calico Enterprise" version: "3.23 (latest)" section: "Install and upgrade" canonical_url: "https://docs.tigera.io/calico-enterprise/latest/getting-started/install-on-clusters/vks" --- # vSphere Kubernetes Service (VKS) ## Big picture Install Calico Enterprise as a Supervisor Service on vSphere Kubernetes Service (VKS), and deploy it to guest clusters. ## Before you begin This guide requires: - **VM Images**: Ubuntu or Photon OVA images registered for a supported TKR version, with TKR in a healthy state. For Windows node pools, Windows Server OVA images are also required. - **vSphere Resources**: Sufficient capacity to provision guest clusters with at minimum: - 1 control plane node - 3 Linux worker nodes - **OCI Registry**: An OCI-compliant registry configured in vSphere for storing package manifests and container images. - **Tigera registry credentials** for accessing the Tigera container registry (contact the Tigera team). - VKS v3.8 and above - vCenter - v9.1.1 and above ### Networking | Configuration | Dataplane | Encapsulation | BGP | | -------------------- | --------- | ------------- | ------- | | Linux only (default) | IPTables | IPIP | Enabled | ## How to Setting up Calico Enterprise on VKS has two main phases: set it as a Supervisor Service (Part 1) in vSphere Client (one time), then deploy it to a guest cluster (Part 2). For Air-gapped environments follow the additional steps in the Air-gapped environment section. ### Part 1: Set up the Supervisor Service (one time cluster admin setup) 1. In the vSphere Client - Container Registries page setup the Tigera container registry's credentials for pulling Calico Enterprise images from Tigera's private container registry. - Use `quay.io` as the Registry host URL, and Username/Password provided by Tigera. For air-gapped installs, add the private registry that holds the Calico Enterprise images instead (see [Air-gapped private container registry](#air-gapped-private-container-registry-relocate-calico-enterprise-private-images-to-a-private-registry)). 2. In the vSphere Client - Supervisor Management page: - Create a new Supervisor Service (or add a new version to an existing one). - Upload Supervisor service Package and PackageMetadata resources (example below) and complete the setup workflow. - Register the Supervisor Service and set `registryName` to match the registry name configured for Tigera's private container registry. ```yaml --- apiVersion: data.packaging.carvel.dev/v1alpha1 kind: Package metadata: name: calico-enterprise-svs.tigera.io.3.23.3 spec: refName: calico-enterprise-svs.tigera.io version: 3.23.3 template: spec: fetch: - imgpkgBundle: image: quay.io/tigera/calico-enterprise-svs:v3.23.3 #Change this to the registry where the carvel package is getting picked up template: - ytt: paths: - service - kapp - kbld: paths: - "-" - .imgpkg/images.yml - kbld/ deploy: - kapp: {} --- apiVersion: data.packaging.carvel.dev/v1alpha1 kind: PackageMetadata metadata: name: calico-enterprise-svs.tigera.io spec: displayName: "Calico Enterprise SVS" shortDescription: "Manages Calico Enterprise Addon Repository and Installation" ``` ### Part 2: Deploy a guest cluster with Calico Enterprise #### Guest cluster prerequisites - **Tigera license file** (contact the Tigera team). - **Dynamic PV provisioning** via a StorageClass, required for Elasticsearch storage. #### Step 1 (optional): Apply the AddonConfig An AddonConfig is needed only if you want to change the default Calico Enterprise configuration, for example to create a cluster with Windows nodes. It is not required for a Linux-only cluster that uses the defaults. The following example AddonConfig enables the Windows dataplane and disables policy recommendation. > **INFO:** Update the following fields before applying: > > - `metadata.name` and `metadata.namespace` > - `spec.values.helmValues.installation.registry` to use Calico Enterprise images from a private registry (for air-gapped installs). This should be the same registry where Calico Enterprise images are pushed to using [Air-gapped private container registry](#air-gapped-private-container-registry-relocate-calico-enterprise-private-images-to-a-private-registry) of this doc. Optionally `spec.values.helmValues.installation.imagePath` can be set if the Calico Enterprise images were pushed to a specific folder within the private registry. > **NOTE:** All Calico Enterprise features (`logCollector`, `logStorage`, `manager`, `monitor`, `policyRecommendation`) are optional. Only `apiServer` is required. Set any unwanted features to `false`. ```bash kubectl apply -f ./addon-config.yaml ``` ```yaml apiVersion: addons.kubernetes.vmware.com/v1alpha1 kind: AddonConfig metadata: name: -calico-enterprise # The AddonConfig name has to be in this format eg. cluster1-calico-enterprise namespace: spec: values: helmValues: # Every option from tigera-helm-charts/tigera-operator/values.yaml may be # set here. Anything not specified falls back to the default declared in # the AddonConfigDefinition schema. The fields below are illustrative; installation: calicoWindowsDataplane: enabled: true policyRecommendation: enabled: false ``` > **NOTE:** The Addon framework expects the `AddonConfig` name to use the format `-`. If the name does not match this format, the framework ignores the `AddonConfig` and creates a new one with default values instead. #### Step 2 (optional): Apply the Custom AddonInstall resource A custom AddonInstall resource is needed only if you want to control which clusters get upgraded. It lets you upgrade clusters selectively. Use `spec.clusters.selector.matchLabels` to specify the labels that a cluster must have, and then add those labels to the Cluster resources you want to upgrade. ```bash kubectl apply -f ./addon-install.yaml ``` ```yaml apiVersion: addons.kubernetes.vmware.com/v1alpha1 kind: AddonInstall metadata: labels: addon.kubernetes.vmware.com/addon-name: calico-enterprise addon.kubernetes.vmware.com/addon-namespace: vmware-system-vks-public addons.kubernetes.vmware.com/category: cni name: namespace: spec: addonRef: name: calico-enterprise namespace: vmware-system-vks-public clusters: - selector: matchLabels: : # Eg. cluster-group: calico-group-1 constraints: expression: cluster.cniRefName() = "calico-enterprise" && cluster.kubeProxyConfiguration().enabled == true releaseFilter: ref: name: calico-enterprise.tigera.io.3.23.3 stopMatchingBehavior: Retain ``` #### Step 3: Apply the Cluster Resource The following example creates a cluster with Linux and Windows node pools. To create a Linux-only cluster, remove the Windows node pool (`windows-pool`). Update the following fields before applying: - `metadata.name` and `metadata.namespace` - `spec.topology.variables[].value.cniRef` (name and namespace of the Calico Enterprise addon) - If you applied a custom AddonInstall in Step 2, add its label (`: `) to `metadata.labels`. - Node pool configuration (VM class, storage class, replicas) as needed for your environment ```bash kubectl apply -f ./cluster.yaml ``` ```yaml apiVersion: cluster.x-k8s.io/v1beta2 kind: Cluster metadata: name: namespace: spec: clusterNetwork: pods: {cidrBlocks: [100.96.0.0/11]} services: {cidrBlocks: [100.64.0.0/13]} # required for Windows (becomes serviceCIDRs) serviceDomain: cluster.local topology: classRef: {name: builtin-generic-v3.8.0, namespace: vmware-system-vks-public} version: v1.36.4+vmware.2-vkr.3 controlPlane: replicas: 1 metadata: &ubuntu annotations: {run.tanzu.vmware.com/resolve-os-image: "os-name=ubuntu,os-version=24.04"} workers: machineDeployments: - {class: node-pool, name: linux-pool-1, replicas: 3, metadata: *ubuntu, variables: {overrides: [{name: vmClass, value: best-effort-large}]}} - {class: node-pool, name: linux-pool-2, replicas: 2, metadata: *ubuntu} - class: node-pool name: windows-pool replicas: 2 metadata: annotations: {run.tanzu.vmware.com/resolve-os-image: "os-name=windows,os-version=2022"} variables: {overrides: [{name: vmClass, value: best-effort-xlarge}]} variables: - {name: bootstrapAddons, value: {cniRef: {name: calico-enterprise, namespace: vmware-system-vks-public}}} - {name: vmClass, value: best-effort-xlarge} - {name: storageClass, value: vsan-esa-default-policy-raid5} - {name: vsphereOptions, value: {persistentVolumes: {defaultStorageClass: vsan-esa-default-policy-raid5}}} - {name: kubernetes, value: {certificateRotation: {enabled: true, renewalDaysBeforeExpiry: 90}}} # Must be cluster-level: the Calico ACD only reads spec.topology.variables. - {name: osConfiguration, value: {windowsToolsConfig: {excludeNICs: ["cali*"]}}} - name: node value: firewall: inboundRules: - {protocol: tcp, fromPort: 179, toPort: 179} # BGP - {protocol: udp, fromPort: 4789, toPort: 4789} # VXLAN - {protocol: tcp, fromPort: 5473, toPort: 5473} # Typha - {protocol: udp, fromPort: 51820, toPort: 51820} # WireGuard - {protocol: tcp, fromPort: 8444, toPort: 8444} # Linseed - {protocol: tcp, fromPort: 9090, toPort: 9090} # Prometheus - {protocol: tcp, fromPort: 9200, toPort: 9200} # Elasticsearch - {protocol: tcp, fromPort: 9300, toPort: 9300} # Elasticsearch transport - {protocol: tcp, fromPort: 9443, toPort: 9443} # Manager UI ``` #### Step 4: Retrieve the Guest Cluster Kubeconfig Can be copied from the Supervisor. #### Step 5: Wait for Core Components Monitor the Tigera status until `calico` and `apiserver` report as `Available`: ```bash kubectl get tigerastatus ``` At this point, all nodes should be in `Ready` state. #### Step 6 (Windows only): Configure Windows nodes Skip this step for Linux-only clusters. 1. After `apiserver` reports as `Available`, set `strictAffinity` to `true` in the IPAM configuration: ```bash kubectl patch ipamconfigurations default --type merge -p '{"spec":{"strictAffinity":true}}' ``` If the IPAM configuration does not exist yet, create it: ```bash kubectl apply -f - </g" | kubectl apply -f - ``` Replace `` with the Kubernetes version of the guest cluster, for example `v1.36.4`. > 💡 **Tip:** The K8s version can be retrieved with `kubectl version -o json | jq -r '.serverVersion.gitVersion'` > ℹ️ **Note:** If the version returned is of the form `v1.36.4+vmware.2`, strip the `+vmware.2` suffix unless a VMware-provided kube-proxy image exists in the `sigwindowstools/kube-proxy` repository. ℹ️ **Note:** For Air-gapped environments copy the kube-proxy.yaml to the air-gapped environment and make the changes. #### Step 7: Apply the Tigera License ```bash kubectl apply -f tigera-license.yaml ``` #### Step 8: Verify the Installation Wait for all remaining components to become available: ```bash kubectl get tigerastatus kubectl get pods -A | grep -E 'tigera|calico' ``` Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. ### Air-gapped environment #### Air-gapped package registry If the published bundles need to be moved into a private OCI registry (for example, an air-gapped install), follow these steps. ##### Air-gapped relocation prerequisites - `imgpkg` and `yq` available locally. - Network access to the source registry (or a tarball produced from it) and to the private registry. - Credentials to push to the private registry (`imgpkg` will prompt or use the local docker config). ##### Step 1: Copy the Supervisor Service (SVS) bundles to a tarball (source side) Run on a host that has access to the source registry: ```bash imgpkg copy --bundle /calico-enterprise-svs:v3.23.3 --to-tar calico-enterprise-svs.tar ``` > **TIP:** Direct registry-to-registry relocation is also supported. Replace `--to-tar .tar` with `--to-repo /` and skip Step 2. ##### Step 2: Push the SVS bundle into the private registry (destination side) ```bash imgpkg copy --tar calico-enterprise-svs.tar --to-repo /calico-enterprise-svs ``` ##### Step 3: Patch Package Resource in `calico-enterprise-svs.yaml` The only registry reference outside the bundles is the service-bundle pointer in the `Package` CR inside `calico-enterprise-svs.yaml`. This needs to be updated to the private registry: ```bash yq eval ' (select(.kind == "Package") | .spec.template.spec.fetch[0].imgpkgBundle.image) = "/calico-enterprise-svs:v3.23.3" ' -i calico-enterprise-svs.yaml ``` > **TIP:** **Content-pin to digest (optional):** `imgpkg copy --to-repo` prints the destination digest of the relocated bundle on its last line. Substituting `:v3.23.3` with `@sha256:` in the `yq` command above will make the reference immutable. ##### Step 4: Upload `calico-enterprise-svs.yaml` i.e. Package and PackageMetadata resource to the Supervisor Upload the new Package and PackageMetadata to the vSphere Client - Supervisor Management as in Part 1, 2 and continue with the remaining steps to create a guest cluster. ##### What does *not* work - `docker pull` / `docker push` on individual images. Re-tagging can change manifest digests, and the bundles' `images.yml` lock files are not rewritten — kapp-controller fetches will fail with `manifest unknown`. - Not Copying the svs bundle during `imgpkg copy` run. The other 2 bundles are pulled by the svs package. #### Air-gapped private container registry: Relocate Calico Enterprise private images to a private registry Follow the steps in [Push Calico Enterprise images to your private registry](https://docs.tigera.io/calico-enterprise/latest/getting-started/install-on-clusters/private-registry/private-registry-regular.md#push-calico-enterprise-images-to-your-private-registry) to pull Calico Enterprise images from Tigera's private image registry to your private registry. > **NOTE:** Only steps 1, 2, 3, 4 — i.e. `docker pull`, `docker tag` and `docker push` for Linux only need to be performed. > **CAUTION:** Do not push Calico Enterprise private images to a public registry. ## Next steps **Recommended** - [Configure access to the Calico Enterprise web console](https://docs.tigera.io/calico-enterprise/latest/operations/cnx/access-the-manager.md) - [Get started with Calico Enterprise tiered network policy](https://docs.tigera.io/calico-enterprise/latest/network-policy/policy-tiers/tiered-policy.md) **Recommended - Networking** - For all networking options, see [Determine best networking option](https://docs.tigera.io/calico-enterprise/latest/networking/determine-best-networking.md).