2026.04.12 · 9 min · llm, rocm, gpu, homelab
Qwen3.5-35B-A3B crashed on every GPU load. Qwen-VL-30B-A3B ran fine at 75 t/s on the same card. The difference was a buggy pre-compiled kernel in rocBLAS — and the fix was building it from scratch.
I'd been running Qwen-VL-30B-A3B on the MI50 for a few weeks without issues but following the hype of Qwen 3.5, i tried to switch to Qwen3.5-35B-A3B, when I first tried to offload layers to the GPU, my llama cpp instance died everytime try to offload the model layers to the mi50s. Dropping ngl to 0 and running CPU-only was fine, so the weights weren't the problem.
AMD Instinct MI50 — 32GB HBM2, gfx906
The MI50 is a datacenter card — 32GB HBM2, GCN 5.1 (Vega20), gfx906. AMD officially dropped ROCm support for it after 6.x, so every new release is a gamble. I run it anyway; 32GB of VRAM is hard to argue with.
The crash was always the same: segfault inside libamdhip64.so during slot initialization, before the model did anything useful. The stack trace pointed into the HIP runtime, not the model code itself. Setting ngl > 0 triggered it reliably; ngl = 0 didn't. Something in the GPU offload path, specific to this model.
$ ./build/bin/llama-server -m qwen3.5-35b-a3b.gguf -ngl 99
...
ggml_hip_init: found 1 ROCm devices:
Device 0: AMD Instinct MI50, compute capability 9.0, VMM: no
...
Segmentation fault (core dumped)
Qwen3.5-35B-A3B uses a Hybrid MoE-SSM architecture, which has a different inference path from the VL model. After digging into the difference, it came down to a BLAS kernel called solve_tri — a triangular matrix solver that the SSM architecture leans on heavily. The VL model doesn't hit the same code path in the same way, which explained why one worked and the other didn't.
The apt package for rocBLAS 7.x ships a bug in how solve_tri is compiled for gfx906. When llama.cpp calls it, the kernel goes out of bounds in VRAM, the hardware raises a page fault, and the process dies. The packaged binary targets a generic AMD profile rather than GCN 5.1 specifically, and the code generation for this chip is wrong.
The apt package for rocBLAS 7.x contains a compilation bug affecting the solve_tri kernel on gfx906 (GCN 5.1). Any Hybrid MoE-SSM model that hits this kernel will segfault. CPU-only inference is unaffected.
If you are running Qwen 3.5 MoE or other Hybrid MoE-SSM models on older AMD Instinct (MI50/MI60) or Radeon cards and encounter a `Segmentation Fault (139)` in `libamdhip64.so` or `librocblas.so` during the warmup/load phase, follow these steps:
Check your GPU's architecture string. For MI50, it is `gfx906`.
You must build the version of `rocBLAS` that matches your current ROCm runtime.
# Example for ROCm 7.2.0
wget https://github.com/ROCm/rocm-libraries/releases/download/rocm-7.2.0/rocblas.tar.gz
tar -xzf rocblas.tar.gz
cd rocblas
The crucial part is the `-a` flag. This forces the Tensile code-generator to create fresh, correct assembly kernels for your specific chip, bypassing bugs in the pre-compiled generic binaries.
# Replace 'gfx906' with your architecture (e.g., gfx1030 for 6800XT)
-d: installs build dependencies
-i: installs the finished library to /opt/rocm/
./install.sh -id -a gfx906
*Note: This usually takes 5-10 minutes if restricted to one architecture.*
You must clear your previous build cache to ensure `llama.cpp` links against the new library.
cd llama.cpp
rm -rf build
cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS='gfx906' # Match your arch here too
cmake --build build --config Release -j $(nproc)
Run a benchmark with full GPU offload:
./build/bin/llama-bench -m your_qwen3.5_model.gguf -ngl 99
1. The Root Cause: A Bug in Pre-compiled rocBLAS llama.cpp uses a kernel called solve_tri (triangular matrix solver) heavily during the inference of Qwen3.5's Hybrid MoE-SSM architecture. When installing ROCm 7.2.0 via apt in Ubuntu, the pre-packaged binary for rocBLAS (librocblas.so) contains a bug specifically affecting how this kernel is executed on gfx906. When llama.cpp attempts to execute this kernel, it accesses out-of-bounds memory on the GPU, causing a VRAM page fault and immediately crashing the container with a Segmentation Fault (139). 2. The Solution: Compiling rocBLAS from Source To fix this, the math library must be compiled from scratch locally to force the AMD compiler to generate fresh, correct assembly instructions for the hardware. 3. The Role of Tensile During the build, AMD's backend code-generator, Tensile, generates tens of thousands of highly specialized, unrolled assembly kernels. It analyzes the specific registers, wavefront size (64 on the MI50), and L1/L2 cache sizes of the gfx906 architecture and writes custom assembly code for thousands of different matrix shapes. Forcing Tensile to run locally successfully bypasses the compilation bug present in AMD's official ROCm 7.2 release. 4. Linking it to llama.cpp Once the fixed librocblas.so is installed in /opt/rocm/lib, clearing the llama.cpp build cache (rm -rf build) and recompiling it allows CMake to link against the newly built library, resolving the memory fault during the solve_tri operations.