Deploying Llama 3.1 on a GPU Model Endpoint with VCF PAIS

This procedure follows my Harbor model upload and my PAIS Supervisor deployment. It covers the endpoint and inference tests. Creating the agent and exposing browser chat are separate posts.

Examples use generic DNS names and placeholder artifact digests. Replace them before applying configuration. The values describe my tested lab rather than a guarantee that every model revision, GPU, or driver combination will behave the same way.

1. Check my deployment prerequisites

• PAISConfiguration reports Ready and browser sign-in works.
• My Llama model artifact exists in the private Harbor project, with its digest recorded.
• My namespace contains the harbor-ai-models pull-only registry Secret.
• The PAIS download path trusts Harbor’s certificate and can stream the complete artifact.
• My storage policy and GPU VM class are assigned to the namespace.
• A healthy passthrough GPU is available, and namespace limits allow another worker.

vcf context use lab-supervisor
kubectl config current-context
kubectl get paisconfiguration -n wkld1-private-ai
kubectl get modelendpoints -n wkld1-private-ai
kubectl get secret harbor-ai-models -n wkld1-private-ai
kubectl get virtualmachineclasses -n wkld1-private-ai
kubectl get storageclasses

I use a Supervisor context for these resources. Worker pod and GPU Operator inspection requires access to the generated VKS cluster instead.

2. Prepare the GPU VM class and capacity

In my lab, ai-gpu-passthrough provides 8 vCPU, 32 GiB RAM, and the RTX 4000 SFF Ada through DirectPath I/O. Configure the device and required memory reservation through the supported vSphere VM class workflow. Confirm that the selected host and device are available to the workload placement policy.

The 32 GiB worker memory allocation is separate from the GPU’s 20 GiB VRAM. Namespace CPU, memory, and storage capacity must also cover the PAIS infrastructure and other endpoints. I increased namespace memory to 128 GiB while attempting an additional GPU worker; that value records my deployment history rather than a minimum requirement.

A registry upload consumes storage but does not reserve GPU resources. Avoid starting multiple endpoints against a single available passthrough GPU unless the placement and resource design explicitly supports that arrangement.

3. Identify the model artifact and routing name

I deploy by OCI digest rather than a moving tag. Copy the actual digest from the artifact created by vcf pais models push and verified in Harbor.

harbor.lab.example.com/ai-models/lab/llama-3.1-8b-instruct@sha256:REPLACE_WITH_64_HEX_DIGEST

My endpoint resource name and routing name are both llama-31-8b-main. The routing name is the identifier the client uses when selecting the served model; it is separate from the Harbor repository name.

4. Create the ModelEndpoint manifest

Save this as modelendpoint-llama-31-8b.yaml. Replace the registry reference and storage class, and confirm the GPU VM class exists.

apiVersion: pais.vmware.com/v1alpha1
kind: ModelEndpoint
metadata:
  name: llama-31-8b-main
  namespace: wkld1-private-ai
spec:
  type: Completions
  engine: vLLM
  replicas: 1
  routingName: llama-31-8b-main
  virtualMachineClassName: ai-gpu-passthrough
  storageClassName: REPLACE_WITH_STORAGE_CLASS
  model:
    ociRef: harbor.lab.example.com/ai-models/lab/llama-3.1-8b-instruct@sha256:REPLACE_WITH_64_HEX_DIGEST
    pullSecrets:
      - name: harbor-ai-models
  inferenceServerCustomization:
    cliArgs:
      - --max-model-len
      - "4096"
      - --gpu-memory-utilization
      - "0.92"
      - --enforce-eager
      - --max-num-seqs
      - "4"
      - --enable-auto-tool-choice
      - --tool-call-parser
      - llama3_json

This reproduces the serving settings from my working endpoint. I use the bundled vLLM engine; the manifest does not replace its container image.

The model weights, KV cache, runtime overhead, context length, and concurrency all compete for VRAM. Increasing the context or concurrency can invalidate the fit that worked in my lab. Tool calling also depends on the model’s chat template and engine version; consult the matching vLLM tool calling documentation when changing those components.

5. Validate and apply the endpoint

kubectl explain modelendpoint.spec --recursive
kubectl apply --dry-run=server -f .\modelendpoint-llama-31-8b.yaml
kubectl apply -f .\modelendpoint-llama-31-8b.yaml
kubectl get modelendpoints -n wkld1-private-ai
kubectl describe modelendpoint llama-31-8b-main -n wkld1-private-ai
kubectl get events -n wkld1-private-ai --sort-by=.metadata.creationTimestamp

Inspect existing resources before applying a manifest. PAIS rejected in-place model and CLI setting changes during my replacement attempt because those fields were immutable. I created a separate endpoint resource instead of forcing an update.

For the resource fields and registry Secret references, use the PAIS Kubernetes API reference together with the installed CRD schema.

6. Monitor the generated worker

PAIS creates managed runtime infrastructure for the endpoint. Monitor the endpoint conditions first, then connect to the generated VKS cluster if deeper inspection is needed.

# Run against Supervisor to identify the generated cluster.
kubectl get clusters -n wkld1-private-ai

# Substitute the cluster name returned above.
$ssoUsername = Read-Host 'Enter your vCenter SSO username'
vcf context create lab-pais-runtime --endpoint 'https://supervisor.lab.example.com' --type k8s --auth-type basic --username=$ssoUsername --ca-certificate 'C:\Lab\supervisor-ca.pem' --workload-cluster-namespace wkld1-private-ai --workload-cluster-name REPLACE_WITH_VKS_CLUSTER_NAME
vcf context list

# Select the generated workload context shown by the CLI.
vcf context use REPLACE_WITH_WORKLOAD_CONTEXT
kubectl get nodes
kubectl get pods -A
kubectl get pods -n gpu-operator

Use the actual namespace and pod names returned by the cluster when reading logs. Look for driver initialization, successful model download, engine startup, and readiness. A Running driver pod alone does not prove a model can execute inference.

My successful worker used GPU Operator 25.10.1 and NVIDIA driver 580.105.08 with its default firmware setting. CUDA validation passed. I did not apply a GSP-disable override to the active Llama worker.

7. Test authenticated model access

Return to my PAIS HTTPS endpoint and authenticate with an authorized VCF Automation identity. For API tests, obtain a short-lived access token through the configured authentication flow. A Harbor robot secret and a vCenter password are not PAIS API bearer tokens.

The following PowerShell example prompts for an existing access token and submits a small chat request through the PAIS compatibility API:

$tokenInput = Read-Host 'Paste a short-lived PAIS access token' -AsSecureString
$paisToken = [pscredential]::new('token', $tokenInput).GetNetworkCredential().Password
try {
    $apiBase = 'https://pais.lab.example.com/api/v1/compatibility/openai/v1'
    $headers = @{Authorization="Bearer $paisToken"}
    Invoke-RestMethod -Uri "$apiBase/models" -Headers $headers
    $body = @{
        model = 'llama-31-8b-main'
        messages = @(@{role='user'; content='Explain a private AI model endpoint in two sentences.'})
        max_tokens = 128
        temperature = 0.2
    } | ConvertTo-Json -Depth 8
    $reply = Invoke-RestMethod -Method Post -Uri "$apiBase/chat/completions" -Headers $headers -ContentType 'application/json' -Body $body
    $reply.choices[0].message.content
} finally {
    $paisToken = $null
    $tokenInput = $null
    $headers = $null
}

Confirm the expected routing name appears and a completion is returned. Keep TLS verification enabled. Broadcom’s PAIS API guide explains authentication and the compatibility endpoint.

8. Validate tool calling separately from tool execution

My recorded tests included an ordinary chat completion and a synthetic tool-call/result exchange. The tool test supplied a function definition, checked the returned call and arguments, supplied a test result, and confirmed the model could continue the conversation.

That validates the model and parser path. It does not grant an agent access to infrastructure, register an MCP server, or execute a real action. At this stage, my PAIS tool catalog was empty and no live tools were attached.

Before enabling real tools, I will define the permitted operations, authentication, approval boundaries, and logging. The agent and MCP posts will cover those separate layers.

9. Handle replacement and rollback through PAIS

My first Llama worker hit NVIDIA initialization and Xid errors. I ultimately placed the replacement endpoint on a known-working GPU. The active endpoint became llama-31-8b-main; the previous Granite and failed Llama endpoint resources were retained with zero replicas.

Use managed scaling and placement controls rather than manually powering off or editing WCP-managed VMs. If an existing endpoint occupies the only healthy GPU, plan the interruption before scaling it down. This example is only for the Granite resource name used in my lab:

vcf context use lab-supervisor
kubectl get modelendpoints -n wkld1-private-ai

# Only after confirming the endpoint and accepting its interruption:
kubectl patch modelendpoint granite-33-2b -n wkld1-private-ai --type merge -p '{"spec":{"replicas":0}}'

Retain the previous manifest and model digest. To roll back on a single GPU, stop the replacement workload before restoring the previous endpoint, and confirm healthy placement. Zero replicas preserves the resource definition but does not keep its GPU worker running.

A failed passthrough device still needs host/device diagnosis. Moving or recreating a worker does not prove that the original GPU is healthy, and a manually applied guest workaround may not survive managed replacement.

Completion checks

• My endpoint reports Ready with the intended model digest and routing name.
• The worker is placed on a healthy GPU and CUDA validation succeeds.
• The full model artifact downloads through the private registry path.
• An authenticated request returns a chat completion.
• Tool-call parsing is tested independently of any live tool permissions.
• The previous configuration is retained for rollback, and unused endpoints do not consume GPU workers.

My Llama endpoint passed chat and synthetic tool-call tests. I later verified streaming replies and contextual follow-ups through my browser chat application. My next post will focus on creating the PAIS agent, configuring its instructions and conversation history, and testing it in the Playground.

Operating note: my deployment records also identified unresolved NTP connectivity and a faulty spare GPU. Successful inference on the active worker does not close those separate infrastructure follow-ups.