ESXi PXE Server on Windows: WDS, iPXE and IIS

The deployment

This ESXi PXE server on Windows uses WDS, iPXE and IIS to replace my customized USB installation workflow. It delivers host-specific Kickstarts for installing ESXi to disk; it is a network installation workflow, not a stateless ESXi runtime deployment.

DHCP remains on the router or another DHCP appliance. The Windows server does not own a DHCP scope. It provides the PXE handoff and content only, while a dedicated provisioning VLAN carries the temporary installation traffic and the finished ESXi management interface can remain on a different network.

The ESXi PXE WDS provisioning repository contains the reusable scripts and example layout. The public copies are sanitized: replace the example names and values with your own, and do not commit credentials, real MAC addresses, private addressing, passwords, or ESXi media.

UEFI host
    -> DHCP appliance assigns a provisioning lease
    -> WDS transfers iPXE by TFTP
    -> iPXE dispatcher selects the host launcher
    -> IIS supplies ESXi payloads over HTTP
    -> Host-specific Kickstart installs ESXi

Download and prepare

Download the provisioning repository to a setup location on the Windows Server, such as C:\Setup\esxi-pxe-wds-provisioning. Run the installation and validation scripts from an elevated PowerShell console. Review the prompts and substitute your own WDS path, IIS content root, network values, ESXi image, and host definitions.

Before starting, have a Windows Server with a stable management address, an isolated PXE/provisioning VLAN, DHCP service available from the network appliance, and an ESXi installer ISO ready to extract. Do not configure the Windows server as a DHCP server for this design.

Set-Location C:\Setup\esxi-pxe-wds-provisioning
Get-ChildItem .\*.ps1
Get-ChildItem .\config
Get-ChildItem .\ipxe

1. Install WDS and IIS

Install the WDS Deployment role and IIS, then initialize WDS with a dedicated RemoteInstall path. Configure the IIS site to serve the ESXi web root. The setup script is designed to request the environment-specific values it needs rather than assuming a local DHCP scope.

.\Install-EsxiPxeProvisioningPrereqs.ps1

.\Test-WdsPxeEndpoint.ps1 -RequireUnattendedPxe

Outcome: WDS is initialized and its WDSServer service is running; IIS is ready to serve the extracted ESXi content; the endpoint test confirms WDS is listening for PXE traffic. The Windows server is not a DHCP server and must not use DHCP option 60 in this remote-DHCP design.

2. Configure remote DHCP

On the router or DHCP appliance that owns the provisioning VLAN, set the next-server/TFTP address to the WDS server and the UEFI boot filename to Boot\x64\ipxe.efi. The DHCP appliance assigns the lease; WDS transfers the boot program.

Some UEFI firmware requests the conventional Windows filename Boot\x64\wdsmgfw.efi regardless of the configured boot option. Keeping the same iPXE binary available at both paths is a useful compatibility alias when NIC firmware differs between hosts.

3. Serve the ESXi installer

Extract the ESXi ISO completely into the IIS content directory; do not attempt to boot directly from the ISO file. Each host’s boot.cfg uses an HTTP prefix pointing to the extracted media and passes the Kickstart URL in kernelopt.

IIS must return HTTP 200 for boot.cfg, b.b00, jumpstrt.gz, and every module/archive listed by boot.cfg. Add MIME mappings for ESXi payload extensions, including .v00, .v01, .v02, .b00, .t00, .gz, and .tgz.

.\Test-EsxiInstallerContent.ps1 -HostName host01

# Every required object should return HTTP 200.

Outcome: the installer can retrieve its modules from IIS. An IIS 404.3 response for even one payload is a common cause of ESXi installer fatal error 15, so validate the entire media tree before diagnosing the ISO itself.

4. Map hosts and Kickstarts

The central iPXE dispatcher maps a PXE NIC MAC address to a small per-host launcher. The launcher determines the selected PXE NIC, sets BOOTIF from that NIC’s MAC address, and chains to the matching host boot.cfg. Keep a separate Kickstart file for every ESXi host.

Check every available iPXE network-device index, not only net0. Firmware and iPXE do not always expose the PXE NIC in the same position. A dispatcher that checks all available interfaces prevents a valid host from falling through to a “no Kickstart mapping exists” message.

Host PXE MAC
    -> ipxe/boot.ipxe dispatcher
    -> ipxe/hosts/<host>.ipxe
    -> hosts/<host>/boot.cfg
    -> kickstarts/<host>.cfg

If a Kickstart was originally built for USB installation, remove stale --ignoredrives references to a device that is no longer present. Never publish real root passwords or other secrets with the Kickstarts; use sanitized examples in source control.

5. Validate a physical boot

Before booting a physical host, run the WDS endpoint and ESXi content tests. Then boot one UEFI host on the provisioning VLAN and watch the IIS logs. If the installation stops, the last successful HTTP request is a far more useful clue than a generic installer failure.

Expected sequence

DHCP lease
    -> iPXE downloaded by TFTP
    -> dispatcher selects the host launcher
    -> boot.cfg and ESXi modules return HTTP 200
    -> Kickstart completes the ESXi installation

After a host installs, remove PXE from its normal boot order so an ordinary restart does not reinstall it. The PXE NIC can remain available for maintenance, while the installed ESXi management interface uses the NIC and VLAN intended for normal operation.

What caught me out

The Windows and UEFI defaults can make an otherwise healthy WDS server look broken. A remote-DHCP configuration needs WDS without DHCP option 60, and firmware can request the default wdsmgfw.efi filename even when the DHCP boot option specifies ipxe.efi.

The other recurring issue was file delivery rather than the ESXi ISO. Missing IIS MIME types or a bad path can produce HTTP errors for an installer module and surface at the console as fatal error 15. The content test and IIS logs make that distinction clear.

Finally, a MAC mapping can appear correct while still failing if the actual PXE NIC is not net0. Check the full iPXE network-interface list, map every relevant index, and use BOOTIF from the NIC that matched the mapping.

Final thoughts

This gives a clean route from bare metal to ESXi without a USB workflow or a dependency on an existing vCenter. Keep the provisioning network isolated, keep Kickstarts host-specific, validate every HTTP object before booting, and treat the public repository as a reusable template rather than a home for live infrastructure details.