Debugging: logs, the fail-fast list and drills
XServer has no admin panel; debugging rests on three things: structured boot logs, fail-fast errors that name the exact field, and test / drill baselines you can run yourself. This chapter turns them into your first debugging handbook.
Overview
6.1 Intro
A server without a management UI must grow its debugging power elsewhere. XServer's choice: make diagnostics deterministic events — the same config and state always produce the same log lines in the same order; config errors always name the exact field at startup and exit, never "running while sick". Determinism means comparability: every anomalous line you receive has a twin in the source's test cases.
The three sections cover three failure domains: startup (log order + the fail-fast list), shutdown (the graceful shutdown sequence), and runtime (test and drill baselines).
6.2 The logging system
The boot-sequence dictionary
Chapter 2 read the normal sequence line by line; here we add the abnormal branches — what it means when a line "doesn't appear" or "looks different":
| Observation | Diagnosis | Action |
|---|---|---|
Stops at config load failed | An xs.json field is invalid (see 6.3) | Fix the field named in the error |
No engine started | Concurrency engine init failed (rare) | Check system resources / thread limits |
No script loaded | TCC compile failed — the error embeds compiler output | Fix the C code; errors point at devfile line numbers |
[app prefix] lines missing | ServiceInit didn't run or didn't log | Check devfile is correct and Init is exported |
Stops at bind failed | Port in use / insufficient privilege | netstat the port; with multiple listeners the bound ones roll back |
Repeatedly increasing gen N | Hot reloads happening (or repeated restarts) | Find out who is submitting reloads |
The shutdown sequence
The five phases after Ctrl+C / SIGTERM (the test baseline asserts the last two lines):
[xs] stopping (stop accepting & retire scripts → wait for network terminal states → retire custom → drain the destructor queue) [xs] engine stopped [xs] bye
Where app logs come from
Lines prefixed like [demo] are your own printf — XServer script logging is just stdout. For structure, use xrt's logger family (76 APIs: levels, formatting, multiple sinks) — see the further-reading box.
6.3 The fail-fast error list
xs.json validation has 21 test cases (the A matrix of func_test), all asserting "exit=1 and the error names the exact field". High-frequency excerpts:
| Error message (excerpt) | Cause | Fix |
|---|---|---|
field 'services' expect array | Root services is not an array | Check JSON structure and spelling |
| duplicate name ... | Repeated server / host name | server names globally unique; host names unique within the server |
| missing port | Non-custom service without port | Add port (1–65535) |
| tls on udp | tls configured on a udp service | Remove it — UDP has no TLS concept |
| recv limit out of range | e.g. udp recv_limit > 65535 | Lower it into range |
| static headers reserved header | static.headers contains a reserved header like Transfer-Encoding | Remove one of the 8 reserved headers |
| error_pages key out of range | Status code outside 400–599 | Fix the key |
| static root missing | The host path directory does not exist | Create the directory or change path |
certificate uses RSA but private key uses EC | Certificate and key algorithm families mismatch | Use a matching cert/key pair (the diagnostic names both) |
| expects exactly one supported PEM block | Key file has multiple / truncated PEM blocks | Keep exactly one complete supported block |
Design stance: refuse to start rather than silently degrade — you will never meet the most dangerous silent failure of "configured TLS but actually running plaintext".
6.4 Shutdown & drills
"Is it broken only in my environment?" — the repository's test baselines are the control group that answers exactly this:
| Baseline | Contents | Use for |
|---|---|---|
test.bat / test.sh | Build → smoke (HTTP/WS/TCP/UDP echo + 11 log assertions) → functional matrix → reload matrix, one command | Run it first when suspecting the environment |
func_test | 40 assertion scenarios: 21 config fail-fast cases, path-traversal vectors, reload storms | Pinpointing protocol boundary behavior |
reload_matrix | Handoff verification across 7 endpoints, four protocols × TLS | Hot reload behavior comparison |
drill | 6-hour soak: slowloris, pipeline bombs, bad-script reload bombs + RSS sampling | Memory / stability concerns |
routebench | Same-machine latency against Go / PHP / FastAPI | Performance regression calls |
Memory-level debugging (crashes, out-of-bounds, leaks) uses xrt's memory-debugging system: leading/trailing canaries, UAF isolation queues, a ring log of allocation events, TEXT/JSON reports — enabled in the host build config; see the xrt book's chapter 6 linked at the end.
6.5 Contract
- Determinism: identical input always produces the identical log sequence — any "black magic" points at config or environment.
- Fail-fast: startup errors all name the field and exit; persistent runtime errors surface as connection-level status codes / disconnects, never process crashes.
- Debug order: logs first (locate the phase) → the fail-fast list next (match the message) → the test baselines last (isolate environmental factors).
Common mistakes
- Symptom
- Changed port / workers / added a host; behavior unchanged.
- Cause
- Most xs.json fields are assembled at startup; only scripts and certificates hot-update at runtime (via reload). Structural config changes need a restart.
/* edit xs.json and just wait */
vim xs.json # port didn't change?/* scripts/certs: submit a reload (volume 4)
* structural changes: restart the process */- Symptom
- The script includes an old header that no longer exists at the right path; compilation behaves oddly (using wrong declarations).
- Cause
- The relative dev_inc search falls back and hits a same-named file in the current working directory — the test suite keeps a dedicated "decoy header" case guarding this boundary.
#include <config.h>
/* did it hit includes/config.h
* or some other config.h in the CWD? *//* distinctive names, avoid generic ones */
#include <myapp_config.h>- Symptom
- The process lingers long after Ctrl+C.
- Cause
- Graceful shutdown waits for live connections to end naturally (no forced teardown). Long-lived connections (WebSocket / push) make the wait long.
/* "deadlocked!" → straight to kill -9
* losing the debugging scene and the graceful semantics *//* check the log for the current phase;
* provide /shutdown at the app layer to wind connections down;
* the service manager supplies the external timeout (ch. 42) */Exercises
Deliberately create three config errors from the 6.3 list (e.g. dup name, tls on udp, error_pages out of range), record the messages, then fix them.
Shuffle a normal boot log and hand it to a partner to reorder by assembly phase; then delete one line and have them say "which step is missing and where the next step will stall".
Write a logging module at the modules layer: per request output method, path, status code, duration (hint: xrt's high-resolution clock; work out what it may access within chapter 3's contract lines). Verify with drill's canary thinking that it breaks nothing.
Summary
| Point | Conclusion |
|---|---|
| Three weapons | Deterministic logs (6.2), the fail-fast list (6.3), test baselines (6.4) |
| Boot dictionary | Every missing log line has a definite diagnostic branch |
| Fail-fast | 21 config cases name the field and exit — never run while sick |
| Shutdown | Five phases, no forced teardown; external timeout from the service manager |
| Memory debugging | xrt's memory-debug system (canary / UAF / reports), deep-linked in the xrt book |
- xrt book · memory debugging — the full use of canaries / UAF isolation / fault injection / leak reports
- xrt reference · logger — structured logging APIs (beyond bare printf)