Tricking Apple Silicon into native CUDA via homelab Proxmox

I wanted to learn low-level CUDA programming from first principles: writing custom kernels, optimizing shared memory layouts, and debugging race conditions directly on hardware. To keep myself grounded in a realistic workload, I picked up the CS3210 Game of Invasions problem set: a non-trivial parallel cellular simulation with multi-kernel coordination, reduction trees, and compute capability 6.1 (Pascal) requirements under CUDA 12.4. My daily driver is an Apple Silicon MacBook. The machine is quiet and fast on battery, but Apple terminated NVIDIA driver support in macOS Catalina, and NVIDIA stopped releasing macOS CUDA toolkits after version 10.2. Modern Apple GPUs execute code through Metal Shading Language instead of PTX or SASS.

Attempting to run CUDA on Apple Silicon hits a physical wall immediately. Translation layers like CuMetal and sillyCUDA exist as experimental projects, but they fail when you need exact memory alignment checks, warp shuffle primitives (__shfl_sync), shared memory bank conflict arbitration, and hardware debuggers like cuda-gdb. I needed real NVIDIA silicon, but I refused to develop inside an SSH terminal through nano or sit behind a laggy remote desktop.

The answer sat in the homelab rack: the Huananzhi H12D-8D host running an NVIDIA GeForce GTX 1080 Ti (11GB GDDR5X, GP102 silicon, Pascal sm_61) on PCIe Bus 41:00.0. I built a developer tooling suite that makes macOS think it has a local GTX 1080 Ti plugged into its thunderbolt port.

flowchart LR
    subgraph MAC [macOS Workstation]
        A["nvcc proxy<br/>(~/scripts/nvcc)"] --> B["Local Runner Stub<br/>(./binary)"]
        D["Interactive Proxies<br/>(cuda-gdb / nvidia-smi)"]
    end

    subgraph SRV ["Proxmox VE 8 Accelerator Node"]
        C["GTX 1080 Ti 11GB<br/>(Pascal sm_61 · Driver 550)"]
    end

    A -->|"rsync delta (~50ms)"| C
    B -->|"SSH execution"| C
    C -->|"pull_outputs (*.out, *.csv)"| MAC
    D <-->|"SSH pseudo-TTY (-t)"| C

1. The Real Cost of Common Workarounds

When developers need remote GPU hardware from a Mac, they usually default to three paths. Each has unacceptable friction for systems programming:

1. Remote Desktops (VNC / RDP): Transmitting a remote graphical environment introduces display latency, frame drops, and cursor drift. It ruins native macOS window management, trackpad gestures, and local IDE configurations.
2. Manual SSH and rsync loops: Writing code locally, running a separate terminal to sync directories, SSHing into the server, and running make becomes exhausting after the tenth kernel edit. Muscle memory wants standard command line ergonomics.
3. Custom tool wrappers (gpu run make): Forcing every build and execution step through a custom wrapper breaks standard project Makefiles, test scripts, and CI runners. Handing that Makefile to anyone else or running it on a plain Linux server immediately fails because they do not have your custom CLI installed.

The target was simple: standard commands must work natively on the Mac. Running nvcc main.cu -o app must compile the code. Running ./app input.in out.out must run the program. Running nvidia-smi must display card telemetry. Running cuda-gdb must open an interactive debugger.

2. The Host Hardware and Accelerator Stack

The target machine is the primary homelab hypervisor from Build Log MSN-002-B, running Proxmox VE 8 with Debian 13 (Trixie) userspace on an AMD EPYC 7F52 16-core processor:

• GPU: NVIDIA GeForce GTX 1080 Ti 11GB (GP102, 3584 CUDA cores, 352-bit memory bus, 484 GB/s theoretical GDDR5X bandwidth).
• Driver: NVIDIA Kernel Module 550.163.01.
• Toolkit: Debian nvidia-cuda-toolkit version 12.4.131 providing nvcc, cuda-gdb, and compute-sanitizer.

Passwordless SSH key authentication configured in ~/.ssh/config routes traffic directly to the host over the local network and Tailscale mesh, allowing automated scripts to dispatch remote commands without interactive prompts.

3. The Compiler Proxy and Local Runner Stub

The core breakthrough is a transparent compiler proxy located at ~/scripts/nvcc. Because ~/scripts sits at the front of $PATH in ~/.zshrc, typing nvcc invokes the proxy instead of failing with command not found.

# In ~/.zshrc
export PATH="$HOME/scripts:$PATH"

When invoked, the proxy performs three distinct jobs:

Step 1: Argument Inspection

The script iterates over $@ to detect whether the user is running a query (--version), an object-only compile (-c), or building a full binary. It extracts the output binary target specified by -o <output_binary>, falling back to a.out if omitted.

Step 2: Remote Compilation

The proxy syncs modified source files to the remote accelerator node using delta-transfer rsync (~45ms transfer time) and executes the real Linux nvcc on the Debian host against the GTX 1080 Ti.

Step 3: Local Runner Stub Generation

If compilation succeeds and produced an executable target, the Mac cannot run the resulting Linux x86_64 ELF binary. Instead of leaving a dead file or an error, the proxy writes a lightweight executable bash script to the exact output path on macOS:

#!/usr/bin/env bash
# Auto-generated remote execution stub for 1080 Ti
BIN_NAME="$(basename "$0")"
exec gpu run "./${BIN_NAME}" "$@"

When you type ./goi_cuda sample.in out.out in your macOS terminal, the operating system executes the local runner stub. The stub calls gpu run, which connects to the remote host, runs the real Linux binary on the GTX 1080 Ti, pipes stdout and stderr back to your Mac terminal in real time, and passes through the exit code.

# This works entirely on macOS without touching the remote shell:
nvcc -O3 -arch=sm_61 test.cu -o test
./test

4. Git Context Resolution and Sync Hygiene

Projects often contain subdirectories, nested Makefiles, and shared headers. If you run a build command from Assignment 2/sb/, naive tools sync only the sb/ folder, breaking relative includes like #include "../util.h".

The central workflow script (~/scripts/gpu) resolves repository topology dynamically:

if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
    SYNC_ROOT="$(git rev-parse --show-toplevel)"
    PROJECT_NAME="$(basename "$SYNC_ROOT")"
    SUB_DIR="$(git rev-parse --show-prefix)"
else
    SYNC_ROOT="$PWD"
    PROJECT_NAME="$(basename "$PWD")"
    SUB_DIR=""
fi

TARGET_DIR="${REMOTE_BASE}/${PROJECT_NAME}"
WORK_DIR="${TARGET_DIR}/${SUB_DIR}"

This guarantees two invariants:

1. The entire project root is synchronized to the host, ensuring all relative header files, input datasets, and helper scripts exist.
2. The remote execution runs inside the matching subfolder (WORK_DIR), keeping working directories synchronized.

The Stub Clobber Bug

When syncing files from the Mac to the remote server, naive rsync would copy the auto-generated Mac runner stub back to the server and overwrite the compiled Linux ELF binary, triggering an infinite execution loop. The synchronization engine inspects local files for stub signatures and adds dynamic exclusions before every transfer.

get_stub_excludes() {
    STUB_EXCLUDES=()
    while IFS= read -r stub_file; do
        if [ -n "$stub_file" ]; then
            rel_stub="${stub_file#${SYNC_ROOT}/}"
            STUB_EXCLUDES+=(--exclude "$rel_stub")
        fi
    done < <(grep -rl "Auto-generated remote execution stub" "$SYNC_ROOT" 2>/dev/null || true)
}

The script inspects local project files, identifies auto-generated runner stubs, and excludes them from the rsync payload. The remote Linux binary remains intact.

5. Automated Output Pulling

CUDA simulations and validation tests produce output data files: simulation grids, log dumps, benchmark tables, and CSVs.

If output files remain trapped on the remote server, local verification scripts fail. Immediately after command execution completes, gpu run invokes pull_outputs:

pull_outputs() {
    rsync -az --update \
        --include='*/' \
        --include='*.out' \
        --include='*.output' \
        --include='*.txt' \
        --include='*.log' \
        --include='*.csv' \
        --include='*.json' \
        --include='*.dat' \
        --include='*.tsv' \
        --exclude='*' \
        "${REMOTE_HOST}:\"${WORK_DIR}/\"" "./" 2>/dev/null || true
}

The filter is intentionally restrictive. It pulls data artifacts while strictly ignoring binaries, object files, and source code. Local Mac runner stubs are protected, local edits are never overwritten by stale server copies, and test outputs appear locally within 50ms of kernel completion.

6. Telemetry and Debugging Proxies

Developing GPU code requires real-time telemetry and low-level debugging tools. Three dedicated proxy wrappers replicate the physical GPU workstation experience on macOS:

1. Interactive Telemetry (nvidia-smi)

The wrapper script detects whether stdout is connected to a terminal:

if [ -t 1 ]; then
    exec ssh -t "$REMOTE_HOST" "nvidia-smi" "$@"
else
    exec ssh "$REMOTE_HOST" "nvidia-smi" "$@"
fi

When run interactively in a shell, ssh -t allocates a pseudo-TTY. Running nvidia-smi -l 1 displays an in-place updating terminal dashboard with zero flicker. When piped into a unix filter (nvidia-smi --query-gpu=memory.used --format=csv | grep ...), the script avoids allocating a pseudo-TTY, producing clean, uncorrupted ASCII streams.

2. Interactive Kernel Debugging (cuda-gdb)

Invoking cuda-gdb ./binary proxies through gpu gdb. It sets up a remote pseudo-TTY and attaches the GNU debugger directly to the CUDA runtime on the Pascal card. Setting hardware breakpoints, inspecting active thread warps, and reading registers works directly inside the local macOS terminal.

3. Catching Silent Failures (compute-sanitizer)

Out-of-bounds array indexing on a GPU does not trigger a clean segmentation fault. Threads silently overwrite neighboring device memory or poison subsequent kernel launches.

Running compute-sanitizer ./goi_cuda ... routes to the remote sanitizer on the GPU host. It analyzes kernel memory accesses in real time, detecting unaligned global writes, shared memory bank conflicts, and race conditions down to exact line numbers and warp identifiers.

7. The Adaptive Makefile Pattern

To ensure code compiles cleanly across both local development and standard Linux environments without requiring custom tooling, project Makefiles use an operating system check:

OS := $(shell uname -s)
ifeq ($(OS), Darwin)
    RUN := gpu run
else
    RUN :=
endif

build:
	$(RUN) nvcc -O3 -arch=sm_61 -o goi_cuda sb/sb.cu util.cu exporter.cu goi_cuda.cu main.cu

run:
	$(RUN) "./goi_cuda sample_inputs/sample1.in out.out 1 1 1 32 1 1 && cat out.out"

check:
	$(RUN) compute-sanitizer ./goi_cuda sample_inputs/sample1.in out.out 1 1 1 32 1 1

clean:
	$(RUN) rm -f *.out *.gch goi_cuda

On macOS (Darwin), RUN delegates commands through the synchronization pipeline to the remote 1080 Ti. On any standard Linux environment or server, RUN evaluates to an empty string. The Makefile runs standard native Linux commands with zero dependencies.

8. Verification and Latency Breakdown

I benchmarked the complete round-trip pipeline while developing the Game of Invasions simulation:

Local VS Code handles syntax highlighting, code completion, and Git commits. Remote Pascal silicon handles PTX generation, shared memory execution, and memory sanitization.

Zero VM Overhead

This architecture eliminates the need to run heavy Linux virtual machines on Apple Silicon or pay for hourly cloud GPU instances. The GTX 1080 Ti in the rack delivers 11GB of dedicated GDDR5X memory directly to the local terminal, while the development workflow remains standard POSIX commands.