Ops & observability: teaching the service to speak

Overview

43.1 Intro

Observability's three questions: is the service alive (health), what did the service do (logs), is the service still healthy (resource trends). In XServer each has a very short path: health = one route; logs = xrtLogger (76 APIs — levels/formatting/multiple sinks) or printf; resources = memory reports + RSS sampling. What matters is installing all three from day one, not bolting them on after an incident.

43.2 Structured logging

/* development: printf suffices (the demo's way) */
printf("[demo] handler done: %s\n", sPath);

/* operations: xrtLogger — levels, multiple sinks, takeover-capable */
/* create the logger in Init (file + stdout dual sinks, configurable level) */
G_pLog = xrtLoggerCreate(...);
/* request paths: with level and module tag */
xrtLogInfo(G_pLog, "api", "item %lld fetched in %lld us", iId, iUs);
xrtLogWarn(G_pLog, "api", "slow query: %lld us", iUs);
/* destroy in ServiceUnit — the logger is generation state too */

• Level discipline: INFO for normal flow (disable-able), WARN for auto-recoverable anomalies, ERROR for things needing human eyes
• Request IDs: generate a short ID at the module layer and thread it through every log line of a request — the "correlation thread" when debugging
• Output: stdout goes to journald/the service manager; file sinks rotate themselves (or let logrotate split by size)

43.3 Memory observability

• External trends: RSS sampling (drill's way: PowerShell WorkingSet64 / /proc VmRSS) into monitoring — a flat line is health, a slope needs investigation
• Internal structure: xrt's memory-debug system (canaries / UAF / allocation reports, TEXT and JSON formats) — enabled in the host build config (see xrt book ch. 06), toggled as needed in production
• Swap reconciliation: finalize counts (ch. 30) + the report's active allocation count are the daily evidence of "hot reload, zero leaks"

43.4 Inspection routes

/* /health: the minimal endpoint for load balancers and monitoring */
ReplyText(pReq, 200, "ok");            /* keep complex logic out */

/* /stats: application-level metrics (behind auth) */
xvalue* p = xrtValueObject();
xrtValueObjectSetNew(p, "uptime_s",  xrtValueInt64(UpSec()));
xrtValueObjectSetNew(p, "requests",  xrtValueInt64((int64)G_iTotal));
xrtValueObjectSetNew(p, "ws_online", xrtValueInt64(ConnCount()));
xrtValueObjectSetNew(p, "reload",    ReloadSummary()); /* terminal counts */
ReplyJSON(pReq, 200, p);  xrtValueRelease(p);

Pattern points: /health answers life or death only (dependency checks there make LBs kill good instances); /stats reports business metrics behind auth (ch. 27's discipline). Want Prometheus format? xvalue → text in a one-line loop.

43.5 Contract

• Pillars: health route, structured logging, resource trends — installed on day one.
• Discipline: /health minimal; /stats authenticated; log levels split three ways.
• Baselines: the tests and drills (func/drill/routebench) keep running across versions — they are the control group for behavioral regressions.

Common mistakes

Exercises