prod-inference-a

Inference + training · telemetry refreshes every 1.5 s

48 × H100 · simulated

GPU utilization

—

Tokens / s

—

P95 latency

—

Requests queued

—

Cost today

—

Healthy nodes

47 / 48

Nodes

Online
Busy
Idle
Error
cordoned / draining

Rack A · 01–24Rack B · 25–48

Alerts

gpu-17 at 91 °C — throttling

SM clocks down 32 %. The three chat-v3 replicas on this node answer 40 % slower.

gpu-17 thermal alarm
14:02 · paging on-call
chat-v3 p95 above 400 ms
14:05 · SLO burn 2.1×
gpu-05 XID 79 — fell off the bus
13:41 · cordoned automatically
embed-e5 v2.4 rollout complete
12:20 · 8 / 8 replicas

Job queue

4 running · 2 queued

Job	Owner	Type	GPUs	Progress	ETA
train-llama-70b-sft	Ada L.	train	16		2 h 10 m
infer-chat-v3	Jian D.	infer	12		serving
eval-mmlu-nightly	Mira K.	eval	4		41 m
train-vision-dpo	Sam O.	train	8		5 h 02 m
eval-humaneval	Ada L.	eval	2	queued	blocked · Ada L. at GPU quota
infer-embed-e5	Jian D.	infer	4	queued	waits on 4 contiguous GPUs

Ops copilot canned replies

gpu-17 is paging — what is going on?

You · 14:03

Thought for 6 seconds

The alarm is thermal. Pull 15 minutes of node metrics first; if clocks are throttled, drain instead of rebooting so the chat-v3 replicas migrate cleanly, and cite the runbook step.

get_node_metricsDone · 0.4 s

node: "gpu-17", window: "15m"

temp 91 °C · util 97 % · sm_clock 1.19 GHz (−32 %) · fan 100 % · replicas: chat-v3 ×3

drain_nodeRunning…

node: "gpu-17", reason: "thermal", reschedule: true

gpu-17 is thermally throttling — 91 °C with SM clocks down 32 %, which is why chat-v3 p95 crossed 400 ms at 14:05. I am draining it now; its three replicas reschedule onto gpu-08 and gpu-19 (idle) and p95 should recover within about 4 minutes1.

Next: open a rack A airflow ticket — this is the second thermal event on that rack this week.

RB-14 · Thermal throttling runbook, step 2

Copilot · cited

healthy
chat-v3
12 / 12 replicas · 3 on gpu-17
canary 10%
healthy
embed-e5
8 / 8 replicas · v2.4.0
stable
scaling
rerank-bge
5 / 6 replicas · scaling up
idle
whisper-large
0 / 2 replicas · scaled to zero
idle

Model	Version	Size	P50
chat-v3	v3.2.1	70 B	212 ms
embed-e5	v2.4.0	1.3 B	18 ms
rerank-bge	v1.9.0	0.6 B	9 ms

Min replicas8

Max replicas32

deploy/chat-v3.yaml +2 −2

@@ -12,4 +12,4 @@ autoscaling:
   metric: queue_depth
-  min_replicas: 6
+  min_replicas: 8
-  max_replicas: 24
+  max_replicas: 32

Runbook RB-14 thermal throttling

1 · Confirm throttling

Check nvidia-smi -q -d PERFORMANCE for HW Slowdown: Active. Above 85 °C with fans at 100 % it is a cooling fault, not a workload spike.

2 · Drain, do not reboot

Cordon the node, then drain with reschedule: true so inference replicas migrate before training shards checkpoint. A reboot on a hot node re-triggers the alarm within minutes.

3 · Open a facilities ticket

Two thermal events on one rack in seven days means airflow, not silicon. Ticket the rack and keep the node cordoned until the fix is confirmed.

Last reviewed 6 days ago

Template with simulated telemetry. No backend — every number on this page is generated in the browser by a seeded random walk (identical on every load), and the copilot replies are canned. Swap the walk for your metrics API and the composer for your inference endpoint.