Monitoring: “Online” Is Not the Same as “Working”

I built a training dashboard early and thought I was monitoring my fleet. I was not. The dashboard happily reported 11 workers “Online” while 6 of them were dead for 47 minutes. Monitoring that doesn’t reflect reality is worse than no monitoring — it gives you false confidence.

Two bugs, one lesson. The dashboard was wrong about which workers were alive, and it was wrong about how much work they were doing. Both bugs came from measuring the thing that was easy to measure instead of the thing I actually wanted to know. Both produced numbers that looked fine until I did the arithmetic in my head and noticed they couldn’t be.

What follows is the dashboard I built first, the two ways it lied to me, and the two iterations that fixed each lie.

What I built first

Simple dashboard with the obvious metrics. The learner collects episodes from actor proxies over TCP. I added a worker tracker:

@dataclass
class WorkerStats:
    ip: str
    connected: bool = True          # true until socket close
    episodes_total: int = 0
    last_seen: float = field(default_factory=time.time)       # updated when episodes arrive
    _timestamps: deque = field(default_factory=lambda: deque(maxlen=200))

The dashboard showed:

• Status: “Online” if connected == True
• Combats/min: count of episode timestamps within the last 60 seconds
• Total episodes: running counter
• Last seen: seconds since last episode

The logic seemed reasonable when I wrote it. connected flips to False when the socket closes cleanly. The timestamps deque gets a new entry every time an episode arrives. last_seen updates with each episode. All four fields are things the learner directly observes. No derived metrics, no heuristics, no inference — just report what you see.

That was the problem. What the learner sees and what the worker is actually doing are not the same thing.

What went wrong

Problem 1: “Online” meant “still connected,” not “still working”

The connected flag only flips to False on a clean socket close. If a worker crashed without closing the socket, lost its LAN connection, hit an unhandled exception in the actor loop, or froze on a game hang — the socket remained in ESTABLISHED state on the learner side for tens of minutes. TCP keepalives eventually detect the dead connection, but the default tcp_keepalive_time is 2 hours on Linux and KeepAliveTime is the same 2 hours on Windows — useless on a 47-minute horizon.

Result: the dashboard showed “Online” for workers that hadn’t produced an episode in 47 minutes.

Problem 2: CPM was inflated by buffered flushes

The combats-per-minute calculation:

@property
def combats_per_min(self) -> float:
    now = time.time()
    recent = [t for t in self._timestamps if now - t <= 60.0]
    return round(len(recent), 1)

_timestamps was populated at receipt time — when the learner received the episode, not when the episode actually completed on the worker. Usually fine. But with flaky LAN connections, the actor proxy would buffer episodes during disconnects and flush the buffer on reconnect: 20–30 episodes arriving in 2 seconds, all stamped with the same receipt timestamp.

A worker that had been offline for a minute, then briefly reconnected, would show inflated CPM as its buffered backlog landed in a narrow window. The dashboard reported 90 CPM for one worker while others showed 0 — even though all of them should have been roughly equal.

const stale = w.last_seen_s > 120;
const activeNow = w.connected && !stale;
const status = !w.connected
  ? "Offline"
  : (stale ? "Stale" : "Online");

ep_dict = {
    "transitions": raw_transitions,
    "summary": summary,
    "completed_at": time.time(),  # added
}

def record_episode(self, completed_at: float | None = None):
    self.episodes_total += 1
    self.last_seen = time.time()  # still receipt time for "last seen"
    self._timestamps.append(
        completed_at if completed_at is not None else self.last_seen
    )

What I’d do first next time

Monitoring should reflect reality, not connection state. The question “is this worker contributing?” is not the same as “is this socket open?” Define “healthy” as “producing recent output,” not “we have a handle to it.” The socket is a necessary condition, not a sufficient one.

Staleness thresholds belong in the UI layer, not the data layer. The learner doesn’t need to know what “stale” means — it just reports last_seen in seconds. The dashboard decides what threshold makes a worker stale. Server stays simple, UI definition stays flexible, and tuning the threshold doesn’t require a learner restart.

Distinguish receipt time from event time. Any metric involving time-windowed counts should use the event’s actual timestamp, not the timestamp of when you saw it. This matters any time there’s a queue, buffer, or network between event and observer — which is, in a distributed system, everywhere. Classic distributed-systems lesson, re-learned the hard way.

Sanity-check the dashboard against basic arithmetic. The inflated-CPM bug was obvious in retrospect: 5 workers summing to roughly an 11-worker total is physically impossible. I should have caught it by eye, but I wasn’t in the habit of doing the math on what I was looking at. Now I am. If the totals don’t factor, something is lying.

Backward-compatible protocol changes save ops pain. .get(“completed_at”) with a None default means I could redeploy workers one at a time instead of coordinating an 11-machine atomic restart. Protocol additions want a default on the consumer side; protocol removals want a deprecation window on the producer side. Neither is free, but neither is expensive if you plan for it.

The lesson behind the lesson: monitoring is a product. Users — me, glancing at a dashboard at 9pm while the cluster runs overnight — consume it the way they consume any other product. If the product says “everything is fine” when everything is not fine, they act on that. The dashboard was not a neutral reporter of facts. It was an agent making claims, and I had accepted its claims without verifying the inference from the data to the claim. Monitoring that doesn’t model what you actually care about is not “minimal” — it’s misleading.

Technical details

• Data path: actor proxy → TCP → learner → workers.json (file) → HTTP → dashboard (5-second poll)
• Storage: WorkerStats dataclass in memory, serialized to JSON every 5s
• Stale threshold: 120 seconds since last episode (UI-side)
• CPM window: 60 seconds, max 200 timestamps in rolling deque

Key takeaways