Ray Shao Back to the room Desktop
BlogHomelab telemetry: Proxmox, InfluxDB, Telegraf and two GPU vendors on one dashboard

← Blog

Homelab telemetry: Proxmox, InfluxDB, Telegraf and two GPU vendors on one dashboard

2026.09.29 / 8 min

On this page 6 sections
  1. The layout
  2. Host metrics with Telegraf
  3. EPYC package power through RAPL
  4. Two GPU vendors, one set of panels
  5. The Flux type collision
  6. Three weeks of silence
  7. Grafana ran out of disk
  8. Reaching it from outside

The Proxmox web UI draws a graph for one guest at a time, with no view of the host’s GPUs at all. Once the rack was running two GPU-sharing AI containers, a Nextcloud VM, a Minecraft VM, and a handful of smaller services, I wanted every guest, the host sensors, and both GPUs on one page. I set up the first version of the stack on 2025-12-28 and rebuilt the host side in September 2026 after the move to the Huananzhi H12D-8D (Build Log MSN-001-C).

The layout

Everything lives on the same Proxmox host it monitors. Two unprivileged Debian LXCs, both created from the community-scripts helper scripts:

Two things write into the bucket. Proxmox’s built-in external metric server (Datacenter → Metric Server) pushes node, guest, and storage stats with nothing installed inside any guest. Telegraf 1.37.0 runs on the host itself for everything Proxmox doesn’t report: per-core CPU, lm-sensors temperatures, EPYC package power, and the two GPUs.

flowchart LR
    subgraph HOST ["Proxmox VE 9.2 host · EPYC 7F52"]
        PVE["pvestatd<br/>external metric server"]
        TG["Telegraf 1.37<br/>10s interval"]
        PX["systemd-socket-proxyd<br/>:3000"]
    end
    subgraph L1 ["LXC · InfluxDB 2.8"]
        DB[("bucket: proxmox")]
    end
    subgraph L2 ["LXC · Grafana 12.3"]
        GF["Proxmox Cluster [Flux]<br/>33 panels"]
    end
    PVE --> DB
    TG --> DB
    GF -->|Flux| DB
    PX --> GF

After nine months the InfluxDB data directory is 42 MB. Grafana’s state directory is 55 MB, and the one dashboard has grown to 33 panels.

Host metrics with Telegraf

The agent config is short. Proxmox already covers guests, so Telegraf only collects what the hypervisor can’t see:

[agent]
  interval = "10s"
  flush_interval = "10s"
  precision = "1s"

[[outputs.influxdb_v2]]
  urls = ["http://<influxdb-lxc>:8086/"]
  token = "<redacted>"
  organization = "home"
  bucket = "proxmox"

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  report_active = true
[[inputs.mem]]
[[inputs.sensors]]
[[inputs.system]]
[[inputs.processes]]

The two GPU inputs live in a separate drop-in, /etc/telegraf/telegraf.d/gpu.conf.

EPYC package power through RAPL

The EPYC 7F52 exposes its package energy counter through the kernel’s powercap interface, under the name intel-rapl:0 even on AMD silicon. Telegraf reads the raw microjoule counter with an exec input:

[[inputs.exec]]
  commands = ["cat /sys/class/powercap/intel-rapl:0/energy_uj"]
  name_override = "cpu_power_raw"
  data_format = "value"
  data_type = "integer"
energy_uj is root-only

Since Linux 5.10 the RAPL energy counter is readable only by root, as a mitigation for the PLATYPUS power side-channel. Telegraf runs as the unprivileged telegraf user, so it can’t read the file by default. I added a root @reboot cron entry that runs chmod 444 on the file. It has to be re-applied every boot because sysfs permissions don’t persist.

The counter only ever goes up, so the panel turns it into watts in Flux:

from(bucket: "proxmox")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "cpu_power_raw")
  |> derivative(unit: 1s, nonNegative: true)
  |> map(fn: (r) => ({ r with _value: float(v: r._value) / 1000000.0 }))
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)

nonNegative: true drops the one bad sample each time the counter wraps back to zero at max_energy_range_uj.

Two GPU vendors, one set of panels

The GTX 1080 Ti on PCIe 41:00.0 is covered by Telegraf’s built-in nvidia_smi input. For the RX 6900 XT (Navi 21) on 03:00.0 I wrote a small Python collector that walks /sys/class/drm/card*/device, skips any card without gpu_busy_percent, and prints Influx line protocol. The trick is naming: it writes a measurement called amd_smi with the same field names as nvidia_smi (utilization_gpu, memory_used, temperature_gpu, power_draw, fan_speed), so a single Flux filter picks up both vendors.

The sysfs sources:

One idle sample from the host:

amd_smi,name=AMD\ Radeon\ RX\ 6900\ XT,index=0,pci=0000:03:00.0 utilization_gpu=0i,utilization_memory=0i,memory_used=73i,memory_total=16368i,memory_free=16295i,temperature_gpu=35i,temperature_hotspot=38i,power_draw=5.0,fan_speed=32i

The Flux type collision

My first combined GPU panel pulled both measurements over the last five minutes and pivoted fields into columns. It failed on the first render:

invalid: runtime error @9:6-9:75: pivot: schema collision detected:
column "_value" is both of type float and int

My collector writes integers (the i suffix), and Telegraf’s nvidia_smi input stores some of the same field names as floats. InfluxDB stores each measurement’s type independently, so both rows landed in one table with two types in the same column and pivot() refused it. I dropped the pivot and gave each metric its own panel: filter a single field from both measurements, group by the GPU name tag, then aggregate.

from(bucket: "${Bucket}")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => (r._measurement == "nvidia_smi" or r._measurement == "amd_smi")
                       and r._field == "utilization_gpu")
  |> group(columns: ["name"])
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)

aggregateWindow(fn: mean) returns floats for both inputs, so the two series end up on the same axes. A gpu dashboard variable, built from the name tag, filters to one card or shows all of them.

Three weeks of silence

The original Telegraf install didn’t use a local config file. The config was stored inside InfluxDB using its Telegraf configurations feature, which serves it back over the API, and the systemd unit started Telegraf with --config http://<influxdb-lxc>:8086/api/v2/telegrafs/<id>. That meant the agent had to reach the database before it could learn where to send data.

From 2026-09-03 onward nothing was answering on port 8086 at that address. The host journal shows Telegraf failing to fetch its config, retrying three times, and exiting, at a rate of around 11,000 error lines a day:

I! Error getting HTTP config (attempt 1 of 3): failed to connect to HTTP config server:
Get "http://<influxdb-lxc>:8086/api/v2/telegrafs/<id>": dial tcp <influxdb-lxc>:8086:
connect: connection refused

By the time I looked on 2026-09-28 it had logged 227,118 of those lines (191,715 connection refused, 26,820 no route to host). The dashboard kept the Proxmox guest panels alive because pvestatd pushes on its own, so the only visible symptom was that the host power, sensor, and GPU panels were empty. I didn’t notice for over three weeks.

Root cause

A monitoring agent that downloads its config from the thing it monitors has a circular dependency. Any time the database moves or is down at boot, the agent fails before it can buffer a single metric.

The fix, done on 2026-09-28:

Grafana ran out of disk

The next morning, after a string of Grafana restarts while I reworked the GPU panels, the log started filling with this:

logger=dashboard-service level=error msg="Failed to execute k8s dashboard cleanup" error="database or disk is full"
logger=authn.service level=error msg="Failed to create session" ... database or disk is full

Grafana keeps its state in a SQLite file, and the 2 GB root disk from the helper script was full. Logins returned HTTP 500 because Grafana couldn’t write the session row. Growing the disk is one command and needs no restart, since ext4 grows online:

pct resize 106 rootfs +4G

The container is now at 6 GB with 28% used.

Reaching it from outside

The Grafana LXC isn’t on the tailnet. Instead the Proxmox host, which is, runs a socket-activated systemd-socket-proxyd that forwards its own port 3000 to the container:

# grafana-proxy.socket
[Socket]
ListenStream=3000

# grafana-proxy.service
[Service]
ExecStart=/usr/lib/systemd/systemd-socket-proxyd <grafana-lxc>:3000

From my laptop or phone the dashboard is at the host’s Tailscale address on port 3000. That is one less Tailscale node to keep logged in, and nothing in the monitoring LXCs is exposed beyond the LAN.

Current state

InfluxDB 2.8.0, Grafana 12.3.1 and Telegraf 1.37.0 running, with 10 s host samples flowing again since 2026-09-28. One dashboard shows every guest, EPYC package watts, and both the GTX 1080 Ti and RX 6900 XT on shared axes.

All postsBack to top