MSN-015 · 2026 · rocm, rccl, vllm, distributed-systems, tensor-parallelism, pcie-p2p, heterogeneous-compute
First working cross-generational AMD Tensor Parallelism engine bridging enterprise datacenter silicon (Instinct MI50, gfx906) and consumer desktop GPUs (Radeon RX 6900 XT, gfx1030) in vLLM via a custom dual-target librccl.so fatbin and PCIe 4.0 P2P DMA.
With secondary market prices for the AMD Instinct MI50 surging past reasonable homelab budgets, buying a second card to scale my local LLM compute was out of the question. After pushing a single MI50 to its 422 tok/s limit under continuous batching (Project MSN-014), running 7B and 8B parameter models in 16-bit precision at high concurrency demanded more compute units and wider aggregate memory than a single card could muster.
Sitting in my workstation was an AMD Radeon RX 6900 XT (16GB GDDR6, RDNA 2).
The obvious engineering thought hit me: Why not augment my compute by pairing the two?
┌────────────────────────────────────────┐ ┌────────────────────────────────────────┐
│ GPU 0: AMD Instinct MI50 │ │ GPU 1: AMD Radeon RX 6900 XT │
│ Enterprise Datacenter Accelerator │ │ Flagship Consumer Gaming GPU │
│ • Architecture: GCN 5.1 (Vega 20) │ │ • Architecture: RDNA 2 (Navi 21) │
│ • Target ISA: gfx906 │ │ • Target ISA: gfx1030 │
│ • Wavefront: 64 threads (wave64) │ │ • Wavefront: 32 threads (wave32) │
│ • Memory: 32 GB HBM2 @ 1,024 GB/s │ │ • Memory: 16 GB GDDR6 @ 576 GB/s │
│ • Silicon: Passive Interposer Die │ │ • Silicon: Monolithic + Infinity Cache │
└───────────────────┬────────────────────┘ └───────────────────┬────────────────────┘
│ │
└───────────────► PCIe 4.0 x16 ◄─────────────────┘
(PeerDirect P2P DMA)
If you search developer forums, Reddit, or AMD’s official issue trackers, the unanimous consensus was: "Impossible." AMD deliberately segments its software stack between enterprise CDNA/GCN and consumer RDNA, and cross-generational tensor parallelism was deemed a non-starter.
This project documents how I proved the consensus wrong by building the first documented operational heterogeneous AMD Tensor Parallelism () cluster running vLLM over bare PCIe 4.0.
Bypassed AMD's artificial RCCL segmentation by compiling a custom dual-target fatbin (`gfx906;gfx1030`), injected it into vLLM's `PyNccl` ctypes loader, disarmed an Nvidia Blackwell SM100 capability assertion trap, and achieved zero-copy Ring AllReduce collectives over PCIe 4.0.
---
Both accelerators are installed in dedicated PCIe 4.0 x16 slots connected directly to the root complex of an AMD EPYC 7F52 (16 cores, 32 threads, 256MB L3 cache) on a Huananzhi H12D-8D dual-socket SP3 motherboard, passed directly into an unprivileged Proxmox LXC container.
The system runs on 128 GB (8x 16GB) DDR4-3200 Registered ECC memory across 8 memory channels, backed by a 2TB Samsung 980 Pro NVMe SSD. The passive MI50 is cooled by a dedicated 4,500 RPM Delta blower fan (38 CFM static pressure).
Heterogeneous Multi-GPU Tensor Parallelism System Topology
Metric Dimension · GPU 0: AMD Instinct MI50 · GPU 1: AMD Radeon RX 6900 XT
Silicon Architecture · GCN 5.1 (Vega 20) · RDNA 2 (Navi 21) ISA Target · `gfx906` · `gfx1030` Wavefront Execution · 64-wide vector SIMD (`wave64`) · 32-wide dual-issue SIMD (`wave32`) Compute Units (CUs) · 60 CUs (3,840 Stream Cores) · 80 CUs (5,120 Stream Cores) VRAM Capacity · 32 GB HBM2 (4 stacks) · 16 GB GDDR6 (8 discrete chips) Memory Bus Width · 4,096-bit (Silicon Interposer) · 256-bit (PCB Traces) Memory Bandwidth · 1,024 GB/s · 576 GB/s (+ 128MB Infinity Cache) KFD Character Node · `/dev/dri/renderD129` (KFD Node 1) · `/dev/dri/renderD130` (KFD Node 2) Cluster Role · Rank 0 (Primary Leader) · Rank 1 (Peer Secondary)
---
Launching `vLLM` with `--tensor-parallel-size 2` across these two cards triggered an immediate cascade of four distinct software and driver failures.
When the distributed process group initialized, Worker 0 (`gfx906`) and Worker 1 (`gfx1030`) initialized their rendezvous sockets. But the moment Rank 1 allocated its first collective tensor, the worker crashed with:
(Worker pid=3218) ERROR [gpu_worker.py:1514] init_worker_distributed_environment failed
(Worker pid=3218) ERROR [pynccl.py:189] data = torch.zeros(1, device=device)
(Worker pid=3218) ERROR torch.AcceleratorError: CUDA error: invalid kernel file
(Worker pid=3218) ERROR hipErrorInvalidKernelFile: Search for `hipErrorInvalidKernelFile` in ROCm documentation.
at /opt/rocm/rccl/src/enqueue.cc:1422
strings /opt/rocm/lib/librccl.so | grep -oP 'gfx\d+' | sort -u
gfx906 <-- Instinct MI50 (Included)
gfx908 <-- Instinct MI100 (Included)
gfx942 <-- Instinct MI300X (Included)
gfx950 <-- Next-Gen Instinct (Included)
gfx1200 <-- Datacenter APU (Included)
When RCCL initializes, it loads pre-compiled device kernels for collective primitives (AllReduce, AllGather, Broadcast). Because the vendor binary was compiled exclusively for datacenter targets, the HIP runtime on the 6900 XT encountered a binary with zero matching code objects for its architecture, threw `hipErrorInvalidKernelFile`, and terminated the worker.
I compiled a custom multi-architecture fat binary from source with both ISA targets enabled:
git clone https://github.com/ROCm/rccl.git
cd rccl && mkdir build && cd build
CXX=/opt/rocm/bin/hipcc cmake \ -DCMAKE_BUILD_TYPE=Release \ -DAMDGPU_TARGETS="gfx906;gfx1030" \ -DCMAKE_INSTALL_PREFIX=/opt/rocm/rccl_dual \ -DBUILD_TESTS=OFF \ ..
make -j32 install
To make this reproducible across any mixed cluster, I also generalized this into an automated build script supporting host auto-detection (`bash build.sh --auto`) or arbitrary pairings (e.g. `gfx1030,gfx1100`, `gfx90a,gfx1100`).
Inspecting my custom `librccl.so`:
strings /opt/rocm/rccl_dual/lib/librccl.so | grep -oP 'gfx\d+' | sort -u gfx1030 <-- RX 6900 XT Verified
gfx906 <-- Instinct MI50 Verified
---
Even after installing the custom fatbin into `/opt/rocm/rccl_dual/lib/`, vLLM continued to crash.
Tracing the python execution stack revealed why: vLLM implements its own low-overhead communicator called `PyNccl`, which bypasses PyTorch’s `torch.distributed` and directly calls `ctypes.CDLL()`.
`PyNccl` hardcoded resolution to the default library path (`/opt/rocm/lib/librccl.so`), completely ignoring `LD_LIBRARY_PATH`.
I patched `vllm/distributed/device_communicators/pynccl.py` to prioritize our dual-target fatbin:
# vllm/distributed/device_communicators/pynccl.py
def _load_nccl_lib():
candidate_paths = [
"/opt/rocm/rccl_dual/lib/librccl.so", # Custom Dual-Target Fatbin
"/usr/local/lib/librccl.so",
"librccl.so",
"/opt/rocm/lib/librccl.so"
]
for path in candidate_paths:
try:
return ctypes.CDLL(path)
except OSError:
continue
raise RuntimeError("Critical: Unable to locate multi-target librccl.so")
---
Once PyNccl loaded the fatbin, both workers initialized device contexts. But during model weight loading, Worker 1 suddenly crashed with:
AssertionError: Unsupported CUDA compute capability (10, 0).
vLLM ModelRunner does not yet support Blackwell SM100 architecture.
vLLM's `ModelRunner`, anticipating Nvidia's next-generation Blackwell architecture (Compute Capability 10.0), intercepted this number and threw an assertion!
---
In multi-GPU Tensor Parallelism, collective communication latency directly determines inference speed. If the GPUs cannot communicate peer-to-peer, every AllReduce activation exchange must bounce through host RAM:
BOUNCE BUFFER (Slow):
GPU 0 VRAM ──[PCIe]──> Host System Memory ──[PCIe]──> GPU 1 VRAM (High Latency)
PEERDIRECT P2P DMA (Optimal): GPU 0 VRAM ──────────────────[PCIe 4.0 x16]─────────────────► GPU 1 VRAM (Zero Copy)
I verified that Resizable BAR (Large BAR) was enabled in the motherboard BIOS, mapping the full 32GB and 16GB address spaces into the CPU root complex:
lspci -v -s c7:00.0 | grep "Memory at" # MI50: Full 32GB 64-bit BAR mapped
lspci -v -s 03:00.0 | grep "Memory at" # 6900 XT: Full 16GB 64-bit BAR mapped
And configured the ROCm peer communication environment:
export NCCL_P2P_DISABLE=0
export NCCL_NET_GDR_LEVEL=3
export NCCL_CROSS_NIC=0
export NCCL_BUFFSIZE=8388608
export NCCL_DEBUG=INFO
---
With all four fixes in place, the cluster booted with full bilateral peer connectivity over PCIe 4.0:
[0] NCCL INFO comm 0x59d69e674e60 rank 0 nranks 2 cudaDev 0 busId c7000 (MI50: renderD129)
[1] NCCL INFO comm 0x5ffd18a35490 rank 1 nranks 2 cudaDev 1 busId 3000 (6900XT: renderD130)
[0] NCCL INFO Channel 00/02 : 0[c7000] -> 1[3000] via P2P/IPC/Direct DMA [CONNECTED]
[1] NCCL INFO Channel 00/02 : 1[3000] -> 0[c7000] via P2P/IPC/Direct DMA [CONNECTED]
[EngineCore] Loading safetensors shards for Qwen/Qwen3-8B (16.02 GiB)
[Worker_TP0] Model loading took 8.01 GiB memory on AMD Instinct MI50 (32GB HBM2)
[Worker_TP1] Model loading took 8.01 GiB memory on AMD Radeon RX 6900 XT (16GB GDDR6)
[EngineCore] init engine (profile, create kv cache, warmup model) took 42.18 s
Both GPUs established bidirectional Direct DMA channels across PCIe 4.0, loaded their 8.01 GiB weight slices symmetrically, and began serving requests simultaneously.
---
1. Vendor Segmentation Is Purely Artificial: AMD's separation of consumer and datacenter software is a packaging choice. Compiling multi-target fatbins unlocks full hardware interoperability. 2. PyNccl Ctypes Dynamic Traps: Modern Python AI runtimes frequently bypass standard dynamic linker environment variables, requiring explicit patches. 3. Augmenting Compute Works: By combining an enterprise MI50 with a consumer RX 6900 XT, I doubled my cluster compute units and created an aggregate 48 GB VRAM testbed without buying a second overpriced enterprise card.
---
To enable the broader homelab, ML systems, and open-source ROCm community to reproduce and build upon this work, all build recipes, runtime patches, and deployment configurations are open-sourced:
- Primary Repository: `github.com/frieddeli/heterogeneous-rocm-tensor-parallelism` - Companion MI50 Runtime: `github.com/frieddeli/mi50-vllm-rocm-runtime`
---
Once the cluster was communicating cleanly and serving requests, I immediately ran into low-level hardware microarchitecture traps on large-head models like Google Gemma 4, as well as the fundamental physics of Dense vs. MoE token routing.
Those discoveries are chronicled in Project MSN-016: Heterogeneous ROCm Kernel Optimization & Architecture Limits.