Mean Time to Resolution
Autonomous vs. Manual
How fast can your infrastructure recover? We benchmarked five common service failures across three operational models: traditional manual triage, SaaS-assisted incident response (Rootly), and self-hosted autonomous execution (StdOut).
Scenario 01: Disk Space Exhaustion
A secondary service leaks logs, filling the root partition. Writes fail across all adjacent databases, halting transaction processing.
Recovery Flow
Triage
Local agent detects disk threshold alert. Sentinel maps the mount point to the offending compose project logs.
Resolution
Windlass executes a safe log truncation script, frees 14GB of stale Docker logs, and verifies DB write capability.
Scenario 02: Node Memory Leak
An application container slowly leaks memory over 48 hours. It is approaching the 2GB limit and will crash under the next spike.
Recovery Flow
Triage
Sentinel identifies memory consumption trend. Matches event to previous memory leak signatures on that service.
Resolution
Windlass triggers a graceful rolling restart of the container, capturing a heap snapshot to local storage first.
Scenario 03: Expired SSL Certificate
Let's Encrypt HTTP challenge path is blocked by a routing change. The cert expires, blocking external API traffic.
Recovery Flow
Triage
Scanner flags invalid SSL handshake. Matches target domains to internal certificate renew scripts.
Resolution
Windlass opens the verification path, triggers Certbot renew, validates the certificate, and reloads Nginx.
Scenario 04: Port Bind Conflict
An orphaned zombie process keeps port 8080 open. The primary container fails to restart with EADDRINUSE.
Recovery Flow
Triage
Sentinel reads container restart loop log for "address already in use" and identifies the target port.
Resolution
Windlass identifies the zombie PID using the local socket, kills the orphaned process, and boots the container.
Scenario 05: 502 Bad Gateway
The backend application node crashes silently. Nginx continues running but returns 502 Bad Gateway to all traffic.
Recovery Flow
Triage
HUD scanner registers 502 error status. Sentinel traces proxy upstream definition to the dead application container.
Resolution
Windlass restarts the container, validates connection to the backend port, and verifies the 502 is resolved.
Full Benchmark Comparison
| Incident Type | Manual MTTR | Rootly MTTR | StdOut MTTR | Total Reduction |
|---|---|---|---|---|
| Disk Space Exhaustion | 45.0 min | 8.0 min | 1.2 min | 97.3% |
| Node Memory Leak | 60.0 min | 12.0 min | 1.5 min | 97.5% |
| Expired SSL Certificate | 90.0 min | 25.0 min | 2.0 min | 97.7% |
| Port Bind Conflict | 35.0 min | 10.0 min | 0.5 min | 98.5% |
| 502 Gateway Error | 50.0 min | 15.0 min | 1.1 min | 97.8% |
Methodology
Tests were executed on a localized infrastructure simulation running 50 application containers across 12 distinct Docker Compose environments. Anomalies and alerts were generated programmatically to trigger standard monitoring pathways.
- Manual CLI Triage: Measured from the initial alert arriving in Slack to the operator logging into the server, running diagnostic CLI tools (
df,du,lsof, etc.), applying manual fixes, and verifying successful external HTTP requests. - Rootly SaaS: Measured from the initial page notification to the on-call engineer, creating the Rootly Slack incident channel, running configured Slack workflows to run remote SSH remediation buttons, and confirming resolution status in the UI.
- StdOut + Windlass: Measured from the local HUD/Scanner anomaly detection to StdOut Sentinel mapping the context against local resolution history, executing the required fix command via Windlass, and running post-fix verification.
Add Autonomy to Your Operations
Stop paging humans for predictable infrastructure failures. Install StdOut on your own local nodes to detect, diagnose, and remediate incidents in seconds.