de-clouding

Lab note: GPU passthrough

Intel Arc in a Proxmox VM: no Resizable BAR, no oneAPI

An Arc Pro B-series card passed through to a VM on a server whose BIOS has no ReBAR option. Vulkan worked; Level Zero, oneAPI and torch.xpu refused to initialize. The fix is a sysfs write on the host — and a hookscript to make it survive reboots.

Setup

Symptom

Inside the VM, the card worked for Vulkan (stable-diffusion.cpp’s Vulkan backend ran fine), but anything on the oneAPI path failed:

Cause

Resizable BAR is negotiated by firmware at boot. With no BIOS option, the host left the aperture at the 256 MB default, and passthrough faithfully handed that 256 MB window to the guest. Vulkan tolerates a small aperture (it pages); the Intel compute stack does not.

Fix: resize the BAR from the host, no BIOS needed

Linux exposes a per-BAR resourceN_resize sysfs knob on devices that advertise the PCIe Resizable BAR capability. The value is the log2 of the size in MB: 15 → 2^15 MB = 32 GB. The device must be unbound from its driver while you write it.

On the Proxmox host, with the VM stopped (device address is this card’s; yours will differ — find it with lspci -nn | grep -i arc):

qm stop 200
echo 0000:3d:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind
echo 15 > /sys/bus/pci/devices/0000:3d:00.0/resource2_resize
echo 0000:3d:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
qm start 200

Check what sizes the card supports first — cat …/resource2_resize prints a bitmask of allowed sizes; if 15 isn’t allowed the write fails with an I/O error. Inside the VM afterward, lspci -vv should show Region 2 … [size=32G], and torch.xpu.is_available() flips to True.

If the write fails with “no space” or the kernel log shows a BAR assignment failure, the root port above the card doesn’t have a large enough 64-bit MMIO window. Add pci=realloc to the host kernel command line and reboot, then retry.

Making it stick

The resize is lost on every host reboot, and a VM start with vfio already bound is too late. A Proxmox hookscript runs it at pre-start:

  1. Enable snippets on a storage: Datacenter → Storage → local → Content → add Snippets.
  2. Put the script at /var/lib/vz/snippets/rebar-vm200.sh (see lab/proxmox/rebar-hookscript.sh), chmod +x it.
  3. Attach it: qm set 200 --hookscript local:snippets/rebar-vm200.sh.

The script checks the current BAR size first and exits quietly if it’s already 32 G, unbinds whatever driver holds the card, resizes, sets driver_override to vfio-pci and rebinds — and refuses to start the VM if the resize fails, so you never silently boot into the 256 MB state again. Verified across a full host reboot: BAR2 came up at 32 G in the guest with no manual step.

Gotchas