gpu-monitor

Your A100 runs at 3AM. You're asleep.
gpu-monitor is watching.

Monitors your GPU fleet around the clock — alerts you on crash, OOM, overheat, and hardware failures across 20 notification channels. One Python file. Zero dependencies.

< 1MB Total size

20 Alert channels

11 Alert types

11 Prom metrics

0 Dependencies

pip install gpuwatch

command: gpu-monitor

Star on GitHub Get started in 3 steps →

Yuxuan Zhang · reacher-z MIT License · Open source

Zero Dependencies

One Python file, under 1MB. Runs anywhere nvidia-smi is available — no environment setup, no docker, no daemon.

stdlib only · < 1MB

20 Built-in Alert Channels

Slack, Discord, Telegram, PagerDuty, Teams, Feishu, DingTalk, ntfy, Pushover, iMessage and 10 more — pick what your team already uses, set one env var.

fan-out to all channels

Prometheus + Grafana

11 Prometheus metrics via built-in /metrics, a ready-made Grafana dashboard, and a drop-in docker-compose monitoring stack.

11 metrics · production-ready

This Live Dashboard

Push stats to GitHub Pages — zero server cost, zero infrastructure. Anyone with the URL sees your fleet in real time, refreshed automatically.

free forever

Smart Alerting

11 alert types: crash, idle, partial idle (hung worker), recovery, OOM, memory leak, overheat, power throttle, ECC errors, fan failure, GPU hardware drop — all with hysteresis to avoid alert spam.

fewer false positives

Kubernetes Ready

DaemonSet manifests and Kustomize overlays included. Deploy across a multi-node cluster without changing a single line of Python.

cluster-scale

gpu-monitor --watch

◈ gpu-monitor v0.3.0 — rig-01 — 2026-03-08 03:14:22 GPU Name Util Mem Used / Total Temp Power Status ─── ────────────────── ────── ───────────────── ───── ────── ────── 0 NVIDIA A100-SXM4 87% 38.2 / 80.0 GiB 62°C 312W ● active 1 NVIDIA A100-SXM4 92% 72.1 / 80.0 GiB 71°C 388W ● high mem 2 NVIDIA A100-SXM4 3% 1.2 / 80.0 GiB 34°C 52W ○ idle 3 NVIDIA A100-SXM4 0% 0.8 / 80.0 GiB 33°C 48W ○ idle Processes on GPU 0: python train.py [user: ml] — 38.2 GiB Processes on GPU 1: python train.py [user: ml] — 72.1 GiB ⚠ GPU 1 memory at 90% — alert sent to Slack Next push in 58s · Press Ctrl-C to stop

Quick Setup — up and running in 3 steps

Install
pip install gpuwatch
Set a notification channel — pick any one:
export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... # or Discord export DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... # or ntfy (zero signup — subscribe on your phone) export NTFY_URL=https://ntfy.sh/my-gpu-alerts-abc123
Run gpu-monitor --channels to see all 20 options.
Run
gpu-monitor
You'll get an alert immediately on startup, then again on crash, OOM, overheat, or idle.
Optional — live web dashboard: fork this repo, enable GitHub Pages, then set GITHUB_PAGES_REPO=your-username/gpu-monitor and GITHUB_PAGES_TOKEN=ghp_.... Stats push automatically every 60 seconds.

What you'll receive in Slack / Discord

◈ gpu-monitor · gpu-cluster-1

GPUs went idle — processes exited: 12345, 12346

Avg util 1% · Temp 38°C · Mem 2G/320G

2026-03-08 03:14:22 UTC — crash detected

◈ gpu-monitor · rig-01

GPU 1 · A100-SXM4 memory at 90% (72.1 / 80.0 GiB)

Util 92% · Temp 71°C · Power 388W

2026-03-08 03:15:01 UTC — OOM warning

No data is stored on any server. Stats are committed as JSON directly to your GitHub repo — the same one hosting this page. Your GPU data never leaves your infrastructure.

Supported notification channels

Messaging & Incident

Slack Discord Telegram Email PagerDuty Teams Google Chat Mattermost Rocket.Chat Zulip OpenClaw

Mobile & Push

Bark ntfy Gotify Pushover SMS iMessage

Enterprise Collaboration

Feishu WeCom DingTalk