Deploying VCF Private AI Services on Supervisor

$paisNamespace = 'REPLACE_WITH_AUTOMATION_PAIS_NAMESPACE'
kubectl get paisconfiguration -n $paisNamespace
kubectl get modelendpoints -n $paisNamespace
kubectl get virtualmachineclasses -n $paisNamespace

This procedure builds on my DSM deployment, my PostgreSQL setup, and my Harbor model store. I keep model endpoint deployment in the next post so this procedure focuses on the service, its dependencies, and access.

All hostnames, organization names, storage classes, paths, and client IDs below are examples. Replace them before applying a manifest. I used the configuration recorded here in my lab; check the compatibility and API schema for another release.

1. Separate service installation from instance configuration

There are two deployment layers. First, I register and enable the PAIS Supervisor Service, which installs the controller. Second, I create a PAISConfiguration in my workload namespace. That resource describes the database, authentication, storage, ingress, and runtime infrastructure for my PAIS instance.

A running controller in a service namespace does not mean an application instance or a model endpoint has been created. I use wkld1-private-ai for my PAIS workload resources. The generated service namespace, such as svc-pais-xxxxx, is managed separately.

2. Check prerequisites and namespace capacity

• My Supervisor and VKS service are healthy, with supported VPC networking and working load balancer address allocation.
• My PAIS namespace has permissions, a storage policy, suitable VM classes, and enough CPU, memory, storage, and IP capacity.
• My PostgreSQL database is Ready, with working DNS and TCP 5432 access from the PAIS runtime network.
• My registry is reachable over trusted HTTPS from the Supervisor and runtime networks.
• DNS and NTP work from the relevant networks, and the required software artifacts are available locally.
• My GPU VM class and passthrough configuration are available for the subsequent model endpoint deployment.

In my lab I used best-effort-medium for the VKS control plane and a separate ai-gpu-passthrough class for model workers. My namespace eventually needed limits of 40 GHz CPU, 96 GiB memory, and 500 GiB storage. Those are recorded lab settings, not minimum requirements or production sizing guidance.

3. Connect the VCF CLI to Supervisor

Install the VCF Consumption CLI and matching Windows plugin bundle from Broadcom. My workstation initially picked up Docker Desktop’s kubectl, but the important missing piece was Supervisor authentication through the VCF CLI.

vcf plugin install all --local-source 'C:\Lab\VCF-CLI-Plugins-9.1.1'
vcf plugin list

$ssoUsername = Read-Host 'Enter your vCenter SSO username'
vcf context create lab-supervisor --endpoint 'https://supervisor.lab.example.com' --type k8s --auth-type basic --username=$ssoUsername --ca-certificate 'C:\Lab\supervisor-ca.pem'
vcf context use lab-supervisor

kubectl config current-context
kubectl get virtualmachines -n wkld1-private-ai

Use the CA bundle that validates the Supervisor endpoint. In my lab, automatic OIDC discovery initially selected a VCF Automation identity that lacked namespace access. Explicit basic authentication with my authorized vCenter SSO identity resolved that connection. An authenticated identity still needs permission to access the namespace.

If recommended plugin synchronization fails because an exact version is unavailable in the online inventory, install the matching version from the downloaded plugin bundle rather than silently changing versions.

4. Mirror the PAIS software for my offline environment

My Harbor model project contains model weights. PAIS service bundles, runtime engines, and GPU Operator images need separate staging. I use the PAIS Artifact Mirroring Tool (AMT) for that software supply.

Download the matching PAIS 3.0 service definition and CLI plugins. On a connected staging system, authenticate to the source registry named in that definition using the authorized Broadcom registry credentials, then mirror the OSS variant:

docker login pais-docker.packages.broadcom.com
vcf pais amt pull .\pais-3.0.0-external.yml --gpu-operator-variant=oss

# Transfer the complete pais-store directory if needed.
docker login harbor.lab.example.com
vcf pais amt push harbor.lab.example.com/supervisor-service --local-store .\pais-store

Create the private supervisor-service Harbor project and grant the staging identity appropriate upload rights first. Use an authentication prompt rather than placing a registry token directly in a command.

The output includes pais.yml and yaml-svc-cfg.yaml. Keep both. In my generated override, runtime engine and GPU Operator repositories pointed at the local software project:

model_runtime:
  engines_repo:
    registry: harbor.lab.example.com/supervisor-service
  gpu_operator_oss_repo:
    registry: harbor.lab.example.com/supervisor-service
  gpu_operator_nvaie_repo:
    registry: harbor.lab.example.com/supervisor-service

Use the generated output rather than this excerpt as the installation input. A repository setting for a variant does not prove its images were mirrored. NVAIE has its own access and licensing requirements. William Lam’s PAIS 3.0 AMT walkthrough covers the mirroring workflow.

5. Register and enable the Supervisor Service

1. In vSphere Client, open Supervisor Management → Services → Add New Service.
2. Register the generated pais.yml.
3. Under the target Supervisor’s Configure → Container Registries settings, configure trust for the internal registry’s issuing CA as required.
4. Open Private AI Services → Actions → Manage Service, select the target Supervisor, and supply the generated yaml-svc-cfg.yaml content when prompted for configuration overrides.
5. Verify registry authentication through the supported service configuration. Do not make the project public to bypass a pull failure.
6. Complete installation and check that the service is configured.

Find the generated PAIS service namespace, then inspect its pods. Substitute the actual namespace returned by the first command:

kubectl get namespaces
kubectl get pods -n svc-pais-xxxxx
kubectl get paisconfiguration -n wkld1-private-ai

At this point, a Running controller and no PAISConfiguration in the workload namespace is expected if no instance has been created yet.

6. Prepare database credentials and certificate trust

Follow the commands in my PostgreSQL post to create pais-db-ca and pais-db-credentials in wkld1-private-ai. My password Secret has a key named password; the CA ConfigMap has ca.crt.

kubectl get configmap pais-db-ca -n wkld1-private-ai
kubectl get secret pais-db-credentials -n wkld1-private-ai
kubectl explain paisconfiguration.spec.database --recursive
kubectl explain paisconfiguration.spec.clientTls --recursive

I also prepare a ConfigMap for the CA chain used by my VCF Automation endpoint. Use public CA certificates only, never a private key:

kubectl create configmap pais-platform-ca -n wkld1-private-ai --from-file=ca.crt='C:\Lab\PrivateAI\platform-ca-chain.pem' --dry-run=client -o yaml | kubectl apply -f -

Keep database TLS verification enabled. Workstation certificate trust alone does not configure PAIS’s outbound connections.

7. Register my PAIS client in VCF Automation

In my lab, PAIS uses a VCF Automation-backed authentication provider. Before creating the instance, register a client/relying party using an account authorized to manage that configuration in VCF Automation.

Save the generated client ID for the PAIS manifest. The issuer must match the provider’s discovery response; do not assume the portal’s base URL is the issuer. Keep the redirect URI exact, including its scheme, hostname, and trailing slash.

Inspect the installed provider schema before choosing optional restrictions:

kubectl explain paisconfiguration.spec.auth.providers.vcfa --recursive

My configuration uses allowedOrgs. An empty organization list grants no tenant access. Optional group and role restrictions should reflect the intended users. Do not combine allowedOrgs with its mutually exclusive ConfigMap-based alternative.

8. Prepare HTTPS for my PAIS hostname

1. Choose the service FQDN, such as pais.lab.example.com.
2. Generate a private key and CSR through the approved certificate process.
3. Request a server certificate with the PAIS hostname in its Subject Alternative Name.
4. Save the returned server certificate chain and matching private key in a protected location.
5. Create the ingress TLS Secret in the PAIS namespace.

kubectl create secret tls pais-ingress-tls -n wkld1-private-ai --cert='C:\Lab\PrivateAI\pais-fullchain.pem' --key='C:\Lab\PrivateAI\pais.key'
kubectl get secret pais-ingress-tls -n wkld1-private-ai

This command creates a new Secret and fails if it already exists. For a running instance, use a planned certificate rotation process instead. The server certificate and private key belong in the TLS Secret; the public CA chains belong in trust configuration.

9. Prepare the GPU Operator override used in my lab

My PAIS workers used the OSS passthrough mode, GPU Operator 25.10.1, and proprietary NVIDIA kernel modules. “OSS” here selects the operator deployment path; it does not mean the driver must use open kernel modules.

Save this as pais-gpu-values.yaml if reproducing that tested configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: pais-gpu-values
  namespace: wkld1-private-ai
data:
  values.yaml: |
    driver:
      kernelModuleType: proprietary
    node-feature-discovery:
      postDeleteCleanup: false

kubectl apply -f .\pais-gpu-values.yaml

These are the overrides recorded in my lab, not a blanket driver recommendation. My PAIS worker later passed CUDA validation with its bundled driver. I did not copy the separate DLVM’s GSP-disable workaround to the PAIS workers.

10. Create my PAISConfiguration

Save the following as paisconfiguration.yaml. Replace the example client ID, organization, DNS names, and storage class. The referenced ConfigMaps and Secrets must already exist.

apiVersion: pais.vmware.com/v1alpha1
kind: PAISConfiguration
metadata:
  name: default
  namespace: wkld1-private-ai
  annotations:
    paisconfigurations.pais.vmware.com/cname: pais.lab.example.com
spec:
  clientTls:
    caBundleRefs:
      - name: pais-db-ca
      - name: pais-platform-ca
  database:
    host: pais-db.lab.example.com
    port: 5432
    username: vcfadmin
    dbname: pais
    sslMode: VerifyFull
    passwordRef:
      name: pais-db-credentials
      fieldPath: password
  auth:
    providers:
      - name: vcfa
        displayName: My VCF Automation
        vcfa:
          issuerUrl: https://auto.lab.example.com/oidc
          clientId: REPLACE_WITH_REGISTERED_CLIENT_ID
          allowedOrgs:
            - lab-org
          scope:
            - openid
            - profile
            - email
            - groups
  ingress:
    serviceType: LoadBalancer
    tlsSecretRef:
      name: pais-ingress-tls
  vksControlPlane:
    virtualMachineClassName: best-effort-medium
    storageClassName: REPLACE_WITH_STORAGE_CLASS
  defaultStorageClassName: REPLACE_WITH_STORAGE_CLASS
  runtimeConfig:
    telemetryRuntimeConfig:
      disabled: true
  nvidiaConfig:
    gpuDriverType: OSS
    gpuOperatorVersion: "25.10.1"
    gpuOperatorOverridesRef:
      name: pais-gpu-values

My manifest keeps Agent Builder and indexing enabled through the release defaults and therefore supplies PostgreSQL. It uses a LoadBalancer and a trusted certificate for browser access. The telemetry setting reflects my lab choice.

Validate against the installed API before applying:

kubectl get storageclasses
kubectl get virtualmachineclasses -n wkld1-private-ai
kubectl apply --dry-run=server -f .\paisconfiguration.yaml
kubectl apply -f .\paisconfiguration.yaml
kubectl get paisconfiguration -n wkld1-private-ai
kubectl describe paisconfiguration default -n wkld1-private-ai

Use this for a new instance. Inspect an existing configuration before applying a replacement manifest. The PAIS Kubernetes API reference documents the resource fields; the installed CRD is the relevant schema for the deployed release.

11. Create DNS for the allocated service address

Find the ingress service reference in PAISConfiguration status and inspect the namespace services:

kubectl get paisconfiguration default -n wkld1-private-ai -o jsonpath='{.status.ingressServiceRef.name}'
kubectl get services -n wkld1-private-ai
kubectl get paisconfiguration default -n wkld1-private-ai -o yaml

Create an A record for pais.lab.example.com pointing to the allocated ingress/load balancer address. Do not use a management address simply because it appeared earlier in the build. In my lab, correcting the PAIS DNS record to the actual service address was necessary.

Resolve-DnsName pais.lab.example.com
Test-NetConnection pais.lab.example.com -Port 443
curl.exe -I https://pais.lab.example.com/

The FQDN annotation in the manifest identifies the external name for PAIS 3.0; it does not create the DNS record or issue a certificate. The annotation is illustrated in this PAIS 3.0 deployment walkthrough.

12. Validate readiness and browser sign-in

1. Review PAISConfiguration conditions and generated child resource status until the instance reports Ready.
2. Confirm my PAIS hostname resolves to the service address and presents the expected certificate.
3. Open https://pais.lab.example.com/.
4. Choose the VCF Automation provider and sign in as a user in the allowed organization.
5. Confirm the browser returns to PAIS and the intended interface is available.
6. Check authorization with the intended tenant account rather than relying solely on infrastructure administrator access.

My completed lab instance accepted VCF Automation sign-in and later served an agent successfully. This validates the configured path; it does not prove every user, organization, or future node replacement will behave identically.

Problems I encountered and what to check

• Forbidden after Supervisor login: verify the selected identity and namespace permissions. My initial automatically selected identity was different from my authorized vCenter SSO account.
• Image download timeout: check the actual runtime source address, NSX translation, registry access rules, CA trust, and proxy streaming behavior.
• Database connection failure: verify DNS, TCP 5432, the logical database name, Secret key, and CA trust with hostname verification enabled.
• Sign-in loop or rejected callback: compare the discovery issuer, registered client ID, exact redirect URI, organization name, and browser-visible FQDN.
• Runtime nodes cannot resolve internal names: inspect node DNS routing/search configuration. My lab needed a node DNS correction; a manual node change does not automatically persist across node replacement.
• GPU errors: check operator and driver state when creating the model endpoint. A healthy service controller alone does not demonstrate usable GPU inference.

My next post will deploy Llama 3.1 from the Harbor model store, configure its GPU worker and inference settings, and validate the model endpoint before connecting an agent.