2026.03.18 · 11 min · proxmox, homelab, gpu, rocm
A Ryzen 7700, an MI50, a 1080 Ti, and a Raphael iGPU walk into a hypervisor. The hardware was straightforward: deciding which GPU received which virtualization path was where the engineering began.
I wanted one box to handle everything (gaming Windows VM, local LLM inference, Nextcloud, and a full monitoring stack) without switching between physical machines or juggling multiple power bricks. The hardware came together over a few months and was straightforward to plan. But figuring out how to partition three disparate GPUs across ten guests without bus contention, IOMMU collisions, or driver crashes was where the real engineering began.
Type-1 hypervisor, open source, and a clean web UI that doesn't require memorizing esoteric QEMU flags every time I want to resize a virtual disk.
The bigger reason was native LXC support alongside full VMs. Most of my day-to-day workloads are headless Linux services that don't need full hardware virtualization. Containers are orders of magnitude lighter on memory, share the host kernel's scheduler, and clone in seconds. Running both lightweight LXC containers and hardware-isolated VMs on the same host without additional orchestration layers was the main architectural draw.
The node runs on an AMD Ryzen 7 7700 (8 cores / 16 threads, 3.8GHz base with a 5.4GHz boost, locked in 65W Eco mode for thermal sanity). Memory is 32GB DDR5 (Proxmox addresses ~30GB after hardware reservations).
The GPU configuration spans three distinct tiers: 1. AMD Instinct MI50 (32GB HBM2, GCN 5.1 / gfx906): Headless datacenter accelerator dedicated to local LLM inference. 2. NVIDIA GeForce GTX 1080 Ti (11GB GDDR5X, Pascal GP102): Consumer flagship dedicated to bare-metal Windows gaming. 3. AMD Raphael iGPU (2 CUs, RDNA 2): Integrated on the AM5 SoC, used for host console display and lightweight hardware video transcoding.
Storage is tiered across two NVMe drives and spinning rust: a 1.9TB NVMe for the Proxmox root pool and hot guest filesystems, a dedicated 1TB NVMe passed directly to Windows, and a 2TB HDD for cold Nextcloud storage.
:::warn LXC GPU access is not automatic Getting ROCm working inside an LXC container required setting explicit cgroup device permissions for `/dev/dri/renderD128` (major 226) and `/dev/kfd` (major 511). Proxmox doesn't configure this out of the box. I baked the device rules into a base template (`/etc/pve/lxc/The GPU partitioning dilemma
The core architectural problem was that each GPU required fundamentally different virtualization treatment:
┌────────────────────────────────────────────────────────────────────────┐ │ PROXMOX VE 9.x HOST (Ryzen 7 7700 • Linux 6.8+ pve-kernel) │ │ │ │ ┌───────────────────┐ ┌───────────────────┐ ┌──────────────────────┐ │ │ │ AMD Raphael iGPU │ │ AMD Instinct MI50 │ │ NVIDIA GTX 1080 Ti │ │ │ │ (Host Console) │ │ (32GB HBM2) │ │ (11GB GDDR5X) │ │ │ └─────────┬─────────┘ └─────────┬─────────┘ └──────────┬───────────┘ │ └───────────┼──────────────────────┼───────────────────────┼─────────────┘ │ Character Bind-Mount │ Character Bind-Mount │ Raw vfio-pci ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ LXC: Media │ │ LXC: AI / LLM│ │ Windows VM │ │ /dev/dri/ │ │ /dev/kfd │ │ Dedicated │ │ renderD128 │ │ /dev/dri/ │ │ Hardware │ └──────────────┘ └──────────────┘ └──────────────┘PCIe topology on consumer AM5
Consumer motherboards are not enterprise server chassis. On my ASUS ProArt X670E, the primary PCIe x16 slot links directly to the CPU's 28 PCIe 5.0 lanes. The secondary full-length slot doesn't go through the chipset at all. It's fed by a PCIe retimer that splits and re-clocks half the CPU's direct lanes out to the second slot, so both slots trace back to the CPU rather than one of them hanging off the X670E chipset's PCIe switch.
Early `vfio-pci` stub binding
To ensure the host never touches the 1080 Ti, I bound the card's vendor IDs (`10de:1b06` for Pascal graphics, `10de:10ef` for HDMI audio) to the `vfio-pci` stub driver before any display server or `nouveau` driver can initialize:
# /etc/modprobe.d/vfio.conf options vfio-pci ids=10de:1b06,10de:10ef disable_vga=1 softdep nouveau pre: vfio-pci softdep nvidia pre: vfio-pci softdep snd_hda_intel pre: vfio-pciCombined with `initramfs-tools` updating and setting `video=efifb:off`, the hypervisor boots cleanly without attaching an EFI framebuffer to the 1080 Ti, completely sidestepping the notorious `BAR 0: cannot reserve` allocation fault and Windows `Code 43`.
Why raw passthrough vs SR-IOV / vGPU
Why not use SR-IOV to slice the 1080 Ti into multiple virtual GPUs? NVIDIA deliberately locks vGPU and SR-IOV features in silicon firmware, reserving them for enterprise Quadro and Tesla hardware with expensive enterprise licensing. Community workarounds (like `vgpu_unlock`) require patching kernel drivers and hooking GSP firmware blobs that break on every minor Proxmox kernel update. Raw `vfio-pci` passthrough gives the Windows VM 100% bare-metal silicon access with zero virtualization overhead.
For the MI50, SR-IOV is unnecessary. Because Linux containers share the host kernel, multiple LXC guests can share the card concurrently via host character device bind-mounting (`/dev/kfd` and `/dev/dri/renderD128`). ROCm communicates directly with the host `amdgpu` driver, eliminating virtualization penalties.
lxc.cgroup2.devices.allow: c 226:* rwm lxc.cgroup2.devices.allow: c 511:* rwm lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir The storage hierarchy balances NVMe flash endurance against bulk HDD capacity: - Hot Tier (1.9TB NVMe / LVM-thin): Hosts the Proxmox root filesystem, container rootfs disks, and active database volumes for InfluxDB and Grafana. - Dedicated Gaming Tier (1TB NVMe): Formatted as NTFS and passed directly to the Windows gaming VM via raw block mapping (`qm set
Guest networking relies on a single Linux bridge (`vmbr0`) bound to the physical onboard 2.5GbE adapter.
Rather than managing static IP configurations across ten separate guest operating systems, all IP allocations are centrally managed via DHCP static MAC reservations at the upstream router.
For external access, Tailscale runs directly on the Proxmox host as both an exit node and an encrypted subnet router. The Proxmox management web UI (port 8006) and guest SSH endpoints are bound strictly to the Tailscale interface (`tailscale0`), never exposed directly to the LAN bridge or public internet. Any authorized client on my Tailnet can reach any guest transparently without opening router ports or running VPN clients inside the guests.
- Nextcloud: Enterprise file sync and storage. 4GB RAM allocation, backed by the cold storage ZFS pool. - Nginx Proxy Manager: Reverse proxy and automatic TLS termination for internal service domains. - LLM Serving (LXC): High-throughput LLM inference via custom-built llama.cpp and ROCm on the shared MI50. - InfluxDB & Grafana: Time-series telemetry pipeline collecting CPU, memory, NVMe SMART data, and GPU metrics across the rack. - AI Workspace (LXC): Rapid experimentation sandbox with access to both the MI50 and Raphael iGPU. - Windows Gaming VM (stopped on demand): Dedicated gaming environment with exclusive GTX 1080 Ti passthrough and 16GB RAM.
At the time of writing, the node has logged over 39 days of uninterrupted uptime. Sharing the MI50 between containerized inference tasks has caused zero resource contention. The GTX 1080 Ti stays in low-power D3cold state until the Windows VM is booted on demand.
Building a golden LXC container template with pre-configured KFD cgroups and ROCm user permissions saved hours of repetitive debugging. Any new test container cloned from the template inherits fully operational GPU access from the first `pveam` start.