# Hodios paste pack: Incident and operations Everything in Incident and operations from Hodios, the open prompt library by Hermes IDE: 26 entries, catalog 2026.1004.3. Every entry is dedicated to the public domain under CC0 1.0. Copy, change and share them freely, no attribution needed. Browse and search the library at https://hermes-ide.com/prompts ## How to use Find an entry below and copy the text inside its block into ChatGPT, claude.ai or any chat. Replace each [PLACEHOLDER] with your own material. Personas, rules and styles work best as custom instructions or project instructions. ## Contents - Incident and operations - [Add logs, metrics and traces to a service](#observability-setup-track) (workflow) - [Audit postmortem action items](#audit-postmortem-action-items) (prompt) - [Build an incident timeline](#build-incident-timeline) (prompt) - [Collect evidence for an ongoing incident, read-only](#collect-incident-evidence) (prompt) - [Define SLOs and burn-rate alerts](#define-slos) (prompt) - [Design a service dashboard](#design-service-dashboard) (prompt) - [Design actionable alerting rules](#design-alerting-rules) (prompt) - [Design an on-call rotation](#design-on-call-rotation) (prompt) - [Fix a simulated broken server](#play-broken-server-challenge) (prompt) - [Incident commander](#incident-commander) (persona) - [Instrument a service for observability](#instrument-service-observability) (prompt) - [Investigate a latency spike](#investigate-latency-spike) (prompt) - [Logging rules](#logging-rules) (rule) - [On-call readiness track](#on-call-readiness-track) (workflow) - [Plan a game day or chaos exercise](#plan-game-day) (prompt) - [Prune noisy alerts](#prune-noisy-alerts) (prompt) - [Rehearse incident command](#rehearse-incident-command) (prompt) - [Site reliability engineer](#site-reliability-engineer) (persona) - [Triage a mobile crash spike](#triage-mobile-crash-spike) (prompt) - [Triage a production alert](#triage-production-alert) (prompt) - [Write a blameless postmortem](#write-postmortem) (prompt) - [Write an incident response plan](#write-incident-response-plan) (prompt) - [Write an incident status update](#write-incident-update) (prompt) - [Write an on-call handoff](#write-on-call-handoff) (prompt) - [Write an operational runbook](#write-runbook) (prompt) - [Write observability queries](#write-observability-queries) (prompt) --- ## Add logs, metrics and traces to a service `observability-setup-track` · workflow · Incident and operations · https://hermes-ide.com/prompts/observability-setup-track Instruments a service in gated steps with structured logs, metrics, traces, correlation ids, business metrics, dashboards as code and a verification run. Use when a service is a black box. ````markdown Makes the [STACK] service at `[SERVICE_PATH]` observable, exporting to open-standards. The goal is that the next incident can be answered from telemetry: which requests fail, since when, for whom, and where the time goes. Instrumentation goes wrong in predictable ways: unstructured log lines nobody can query, a metric label holding user ids that explodes cardinality and cost, traces that break at every queue or thread hop, and personal data copied into logs. This track plans first, then wires logs, metrics and traces through the libraries the service already uses, and proves the signals arrive. Rules for every step: - Follow the service's existing logger, config and dependency injection patterns; extend rather than replace. - No personal data or secrets in telemetry: no names, emails, addresses, tokens, passwords, full request or response bodies, or payment data in logs, metric labels or span attributes. Use ids that are not personal, or hash where joining is needed, and redact at the logger or exporter level so a new log line cannot leak by accident. - Every metric label has a bounded set of values. User ids, request ids, raw URLs and error messages are never labels. - Telemetry must never break the request path: exporter failures are logged and dropped, not raised. - Use OpenTelemetry semantic conventions for names and attributes where they exist. - Do only what was asked. If you notice something else worth changing, mention it in one line at the end instead of changing it. - Keep the change as small as it can be while still being correct. - Before saying the work is done, run the check that proves it (tests, build, type check or the command the user gave) and report the real result. - If you could not run a check, say so plainly and say which one. ## Steps Work through these steps in order. Do not skip a gate. 1. survey (discover) 2. logging (build) 3. metrics-traces (build) 4. dashboards (build) 5. verify (verify) ### Step 1: Survey and plan 1. Find what exists: logging library and format, any metrics or tracing libraries, request id handling, health endpoints, existing dashboards or alert files, and how config and secrets reach the service. 2. List the entry points (HTTP routes, RPC handlers, queue consumers, scheduled jobs, CLI commands) and the outbound calls (databases, caches, HTTP clients, queues, third-party APIs). 3. Name the key business operations from the code and docs (for example "order placed", "payment captured", "export completed") and what success and failure mean for each. 4. Propose the plan: - Log schema: the fields every line carries (timestamp, level, service, environment, version, trace and span id, request or job id, operation, outcome, duration) and the levels policy. - Metrics: request rate, errors and duration per route or operation (histograms with explicit buckets around the latency that matters); saturation for pools, queues and workers; the business metrics; each with name, unit, type and its bounded labels, plus a cardinality estimate. - Traces: automatic instrumentation for the framework and clients in use, manual spans around key operations, propagation across queues and background jobs, sampling choice. - Data protection: fields that must be redacted or dropped. Write the artifact: Current state, Entry points and dependencies, Business operations, Log schema, Metrics (Name | Type | Unit | Labels | Cardinality | Question it answers), Traces, Redaction list, Sampling. Stop and wait for approval. Save this step's result to `observability/01-plan.md`. **Gate:** stop here and wait for the user's approval before step 2 (logging). ### Step 2: Structured logging and correlation 1. Configure the existing logger (or the standard structured logger for [STACK] if there is none) to emit one JSON object per line to stdout with the approved fields. 2. Add or reuse middleware that accepts an incoming W3C `traceparent` (and an existing request id header if the platform uses one), creates one when missing, puts it in the logging context, and returns the request id in the response. 3. Carry the context into background jobs and queue messages so a job's logs link to the request that enqueued it. 4. Add the redaction filter from the plan at the logger level and a test that proves a sample secret and email are removed. 5. Replace prints and string-built log lines on the main paths with structured calls carrying fields, not interpolated text. Log each request or job once at completion with outcome and duration, and errors once with the stack trace where they are handled. Continue to step 3. ### Step 3: Metrics and traces 1. Add the OpenTelemetry SDK (or the approved open-standards equivalent) with configuration from environment variables: service name, version, environment, exporter endpoint, sampling ratio. Default to a console or no-op exporter when no endpoint is set so local runs and tests need nothing extra. 2. Enable automatic instrumentation for the framework, HTTP clients, database drivers and queue clients the service uses. 3. Add manual spans around the key business operations, with attributes that help debugging and contain no personal data, and record exceptions on spans. 4. Register the approved metrics: request and operation duration histograms, error counts by bounded error class, saturation gauges, and business counters. Expose them the way the backend expects (a scrape endpoint or OTLP push). 5. Expose liveness and readiness checks if missing: liveness says the process is up; readiness checks the dependencies needed to serve. Continue to step 4. ### Step 4: Dashboards as code 1. Use the dashboards-as-code format the team already has (dashboard JSON in the repo, a provisioning folder, Terraform, Jsonnet or the vendor's config format). If there is none, write a dashboard JSON for a Grafana-compatible tool and say how to import it. 2. One overview dashboard: request rate, error ratio and latency percentiles per route or operation; saturation; business metrics; a panel of recent error logs filtered by trace id. Every panel title states the question it answers. 3. Write the queries against the metric names actually registered in step 3. 4. Propose, but do not activate, two or three alert conditions tied to user impact (error ratio and latency on the key operations), each with a threshold placeholder for the team to set with an SLO. Continue to step 5. ### Step 5: Verify the signals 1. Run the service locally with a local collector or the console exporter (a compose service is fine). Exercise the main routes, one failing request, and one background job. 2. Confirm and record, with snippets: - Logs are JSON, carry the approved fields, and share one trace id across the request and the job it enqueued. - Metrics appear with the expected names, units and labels, and label values stay bounded. - Traces connect from the entry point through database and outbound calls, with no broken parent links at queue hops. - The redaction test passes, and a search of the captured output finds no emails, tokens or passwords. - The service still runs and passes its tests with the exporter endpoint unreachable. 3. Run the project's test suite and linters. Write the report: #### Signals What is now logged, measured and traced, per entry point and operation. #### Verification Each check above with its real result and a short snippet. #### Configuration Environment variables added, with defaults. #### Dashboards and alerts Files written and how to load them; proposed alerts awaiting thresholds. #### Follow-ups Gaps, such as services downstream that do not propagate context, or SLOs to define. Save this step's result to `observability/05-report.md`. ```` --- ## Audit postmortem action items `audit-postmortem-action-items` · prompt · Incident and operations · https://hermes-ide.com/prompts/audit-postmortem-action-items Reviews action items across recent postmortems for done, stale and vague items and repeated systemic themes, and rewrites each open item to be specific, owned, dated and verifiable. ````markdown Postmortems are only worth the follow-through. Across teams the same pattern shows up: items written in the heat of the review ("improve monitoring", "be more careful with deploys") are never done because nobody can tell what done means; small items close while the one structural fix sits for months; and the same contributing factor appears in incident after incident because each review treats it as new. An audit closes the loop: what was promised, what happened, and what the pattern says about where to invest. [ACTION_ITEMS] 1. Normalise every item into one table: incident, item, owner, due date, status, age in days at the review date. 2. Classify each item: - done (closed, with evidence or a linked change); - stale (open and past due, or open more than 60 days with no update); - vague (no observable completion condition, no owner, or an owner that is a team or "TBD"); - blame-shaped (asks people to "be careful", "remember" or "be retrained" instead of changing the system); - superseded or duplicate (same fix as another item). 3. Tag each item with a remediation type: detect (alerting, monitoring), mitigate (runbooks, kill switches, rollback), prevent (tests, validation, guardrails in tooling), or process (review, ownership, documentation). 4. Rewrite every open item that is vague, stale or blame-shaped so it has: a concrete change, one named owner placeholder, a due date proposal, and a verification step that proves it ("a canary failing the 5xx check in staging auto-rolls back within 5 minutes, shown in a game day"). 5. Find themes: contributing factors, systems or failure modes that appear in two or more incidents. For each, list the incidents and the open items that address it, and say whether those items would actually prevent a repeat. 6. Recommend at most three investments that address the strongest themes, and items to close as won't-do, with the reason. - Use only the items and facts given. Do not invent owners, dates or completion evidence; put placeholders like [owner] and ask. - Rewrites keep the original intent; if the intent is unclear, ask rather than guess. - No blame of individuals. Rewrite "Engineer X to be more careful" into a system change. - Mark an item done only if the input says so; "probably done" stays open with a question. - Count and show the arithmetic for summary percentages. ## Summary Totals: items, done, stale, vague, blame-shaped, duplicate; completion rate; median age of open items. ## Item review Table: incident | item | owner | due | status | age | class | remediation type. ## Rewritten open items Table: original | rewritten item | owner | proposed due | how we verify. ## Themes For each theme: incidents, open items, will they prevent a repeat (yes, partly, no). ## Recommendations Up to three investments, plus items to close as won't-do. ## Questions Bullets, or "None". ```` --- ## Build an incident timeline `build-incident-timeline` · prompt · Incident and operations · https://hermes-ide.com/prompts/build-incident-timeline Builds a timestamped incident timeline from chat logs, alerts and deploy records, marking detection, escalation, mitigation and the gaps between them. Use when preparing a postmortem. ````markdown A postmortem is only as good as its timeline. Raw material comes from tools that log in different timezones and formats, chat messages are posted minutes after the events they describe, and the most useful facts are the gaps: twenty minutes between the first customer report and the first alert, or an alert that fired and sat unacknowledged. The timeline must be exact, sourced and blameless. Build an incident timeline in UTC from this material: [RAW_MATERIAL] 1. Parse every timestamp. Convert each to UTC, noting the source timezone when it differs. If a source has no timezone and you cannot infer it from context, say so and mark those times "unverified zone". 2. Extract events and tag each with one type: trigger, impact-start, detection, acknowledgement, escalation, decision, mitigation-attempt, mitigation-effective, communication, resolution, other. 3. Mark each event "recorded" (the source states it) or "inferred" (you deduced it), and give the source for every event. 4. Compute the key intervals: impact start to detection, detection to acknowledgement, acknowledgement to mitigation, impact start to resolution. If a boundary event is missing, say which and do not compute that interval. 5. Find gaps: any stretch of more than 15 minutes during impact with no recorded action, detection by a customer or a person before any alert, alerts that fired without acknowledgement, communication cadence breaks, failed mitigation attempts. 6. List conflicts where sources disagree, with both values. - Do not invent events or fill gaps with plausible guesses. A gap is a finding. - Do not infer causality. "Deploy at 10:02, errors from 10:05" is two events, not a cause. - Stay blameless: describe actions and systems, use the role or handle exactly as given, and add no judgement words such as "failed to" or "should have". - Quote source text only when the exact words matter, and keep quotes short. ## Key metrics A table: interval, start event, end event, duration. ## Timeline A table in chronological order: time (UTC), event, type, recorded or inferred, source. ## Gaps Numbered, each with its time range and why it matters for the postmortem. ## Conflicts Bullets, or "None". ## Missing data What to pull from which system to complete the timeline. ```` --- ## Collect evidence for an ongoing incident, read-only `collect-incident-evidence` · prompt · Incident and operations · https://hermes-ide.com/prompts/collect-incident-evidence Gathers evidence for a live incident with read-only commands, covering recent deploys, error rates, logs and resource use, and writes a timestamped evidence summary. Use while responders work the fix. ````markdown During an incident the responders need facts fast and cannot afford a helper that changes things. Good evidence answers: what changed just before it started, what is failing and how much, since exactly when, for which requests, and what the system's resources are doing. Common traps: reading timestamps in mixed time zones, treating a noisy error that was always there as the cause, pasting logs full of tokens or customer data into the incident channel, and "just restarting" a pod, which destroys the evidence. Collect evidence for the incident affecting [SERVICE] over [TIME_WINDOW]. [ALLOWED_COMMANDS] 1. Before running anything, check each command you plan against the allowed list. Run only commands on it, and within those, only read operations. If a useful command is not allowed, write it under Gaps as a command for a human to run, with what it would show. 2. Changes in the window: deploys and rollouts (rollout history, release tags, `git log` of the deployed revision range), config and feature flag changes, infrastructure or dependency changes, scaling events, certificate expiries, scheduled jobs. 3. Signals: request rate, error rate and latency for the service and its dependencies, compared with the same window a day or a week earlier where the tools allow; saturation (CPU, memory, restarts and OOM kills, connection pools, queue depth, disk). 4. Logs: group errors by signature (exception type and normalised message), with count, first seen and last seen in the window, and whether the signature also appears before the incident started. Quote one short example line per signature with secrets, tokens and personal data redacted. 5. Normalise every timestamp to UTC and name its source. Note clock skew or gaps in data. 6. Build a timeline, then rank hypotheses that the evidence supports, each with evidence for and against and the next check that would confirm or rule it out. 7. Write the evidence summary to a file named with the service and the UTC time of writing, and print the same summary. - Read-only, always. Never restart, scale, roll back, delete, drain, exec into a container to change state, edit config, flush caches, acknowledge or silence alerts, or post to incident channels, even if the allowed list seems to permit it. Recommending an action is fine; taking it is not. - Do not run commands that put heavy load on a struggling system, such as unbounded log queries over days; scope queries to the window and add limits. - Label each statement as observed (with its source) or inferred. Do not present a hypothesis as the cause. - Never copy secrets, tokens, credentials or customer personal data into the summary. - Do only what was asked. If you notice something else worth changing, mention it in one line at the end instead of changing it. - Keep the change as small as it can be while still being correct. - Before saying the work is done, run the check that proves it (tests, build, type check or the command the user gave) and report the real result. - If you could not run a check, say so plainly and say which one. ## Window and sources Window in UTC, systems queried, and data gaps. ## Timeline Table: Time (UTC) | Event | Source | Observed or inferred. ## Signals Table: Signal | Baseline | During incident | Source. ## Top errors Table: Signature | Count | First seen | Present before incident | Example (redacted). ## Changes in window Table: Time | Change | Who or what | Source. ## Hypotheses Ranked list: hypothesis, evidence for, evidence against, next check. ## Gaps Missing data and commands for a human to run. ## Commands run Each command with its exit status, in order. ```` --- ## Define SLOs and burn-rate alerts `define-slos` · prompt · Incident and operations · https://hermes-ide.com/prompts/define-slos Defines SLIs, SLOs and an error-budget policy from a service's user journeys, with multi-window burn-rate alert rules. Use when alerting is noisy or reliability targets are vague. ````markdown Teams write SLOs that measure servers instead of users ("CPU below 80%"), pick 99.99% because it sounds good, and alert on raw error rate, which pages for blips and misses slow burns. A good SLO measures what users experience on a journey, sets a target the service can meet and users would accept, and alerts on how fast the error budget is burning. Define SLOs for [SERVICE] from these user journeys: [USER_JOURNEYS] 1. For each journey, choose 1 or 2 SLIs written as good events divided by valid events: availability, latency below a threshold, freshness or correctness. Say where each is measured (load balancer, server, client) and the trade-off. Define valid events explicitly, for example excluding health checks and client errors the user caused. 2. Set a target and a window (a 28- or 30-day rolling window by default). Base the target on current performance and user need. If current metrics are missing, mark targets "provisional" and propose a 2 to 4 week baseline measurement. 3. Compute the error budget in allowed bad events and in minutes of full outage per window. 4. Write an error-budget policy: what happens at 50%, 75% and 100% consumed (for example: slow down risky launches, prioritise reliability work, freeze non-critical changes), the exceptions, and who decides. 5. Write multi-window, multi-burn-rate alerts for a 30-day window: page at 14.4x burn over 1 hour (with a 5-minute short window), page at 6x over 6 hours (30-minute short window), and open a ticket at 1x over 3 days (6-hour short window). Adjust the numbers if the window differs and show the calculation. 6. Write the alert rules in the syntax of the user's monitoring stack (PromQL recording and alerting rules by default). Note the low-traffic problem and a mitigation if any journey has little traffic. - No target of 100%, and no target tighter than the service's dependencies allow without saying how. - Use the metric names given; where none are given, use clearly named placeholders and say so. - Prefer few SLOs that matter over full coverage. Three per service is often enough. - Separate what you verified from what you inferred. Mark inferences as such. - When you do not know, say "I don't know" once and state what would settle it. ## SLOs A table: journey, SLI (good / valid), measured at, target, window, error budget. ## Rationale One short paragraph per SLO: why this SLI and target. ## Error-budget policy Thresholds, actions, exceptions, decision owner. ## Alert rules Fenced code blocks with the rules, then a table: alert, burn rate, long window, short window, budget consumed when it fires, page or ticket. ## Open questions What to confirm with product owners or measure first. ```` --- ## Design a service dashboard `design-service-dashboard` · prompt · Incident and operations · https://hermes-ide.com/prompts/design-service-dashboard Designs an operational dashboard for one service, with golden signals on top, dependencies, saturation, deploy annotations and drill-down order, plus the query and on-call action for each panel. ````markdown You design the first dashboard an on-call engineer opens when this service pages. It must answer, top to bottom and in under a minute: are users hurt, since when, is it us or a dependency, and did something change. Most service dashboards fail because they are a wall of 40 resource graphs with no order, average latency hides the tail, and deploys are invisible so the obvious cause is missed. Use the golden signals (traffic, errors, latency, saturation), RED for request paths and USE for resources, and give every panel a reason to exist. [SERVICE_DESCRIPTION] 1. State the audience (on-call first, then service owners) and the three questions the top row answers. 2. Lay out rows in this order: - Row 1, user impact: SLO status and error-budget remaining if SLOs exist; request rate; error ratio (5xx or failed jobs over total, not a raw count); latency p50, p95, p99 as threshold lines against the SLO target, never an average alone. - Row 2, by dimension: the same signals split by route or job type, and by version or region, so a bad deploy or one region stands out. - Row 3, dependencies: for each dependency, call rate, error ratio and latency from this service's side, plus timeouts and retries, and circuit-breaker state if any. - Row 4, saturation: the resource that runs out first for this workload (connection pools, thread or worker pools, queue depth and age of oldest message, memory against limit, CPU throttling, disk), each against its limit. - Row 5, background: batch jobs, consumers, caches (hit ratio), with last success time. 3. For each panel give: title as a question ("Are checkout requests failing?"), the query in the chosen backend's language or a generic form, visualisation type, unit, thresholds, and what on-call does when it is red (which runbook or which row to look at next). 4. Annotations: deploys, config and feature-flag changes, scaling events and incidents on every time-series panel. Variables: environment, region, version; default time range 6 hours with a comparison to one week earlier for traffic. 5. Write the drill-down path: from a red panel in row 1 to the row and panel that separates the likely causes, and then to traces or logs with the label to filter on. 6. List what not to put on this dashboard (per-pod resource graphs, business KPIs, anything nobody acts on) and where it belongs instead. - Use only the metric names and labels given; write any you need but do not have as a clearly marked placeholder and list it under Gaps. - Keep the first screen to about eight panels; everything else goes below the fold or on linked dashboards. - Use rates and ratios over windows of at least four scrape intervals; never graph raw counters. - Use colour only for state (ok, warning, breach) and make thresholds match alert thresholds where alerts exist. - If the service description lacks dependencies or the deploy method, ask for them in Gaps rather than assuming. ## Purpose and audience Two to four lines. ## Layout A row-by-row sketch (text grid or list). ## Panels Table: row | panel question | query | visualisation and unit | thresholds | when red, do this. ## Annotations and variables Bullets. ## Drill-down path Numbered path for the two or three most likely failure modes. ## What not to add Bullets with where each belongs. ## Gaps Missing metrics, labels or information, or "None". ```` --- ## Design actionable alerting rules `design-alerting-rules` · prompt · Incident and operations · https://hermes-ide.com/prompts/design-alerting-rules Designs actionable alerts from SLOs and user-facing symptoms, with thresholds, routing, runbook links, and a list of noisy alerts to delete. Use when pages are noisy or real outages go unnoticed. ````markdown A page should mean "users are hurt or soon will be, and a human must act now". Pages on causes (CPU at 80%, a pod restarted, a queue non-empty) fire when nothing is wrong and stay silent when something new breaks. Alerts on symptoms users feel (errors, latency, freshness, availability) tied to SLOs catch every cause. Multi-window, multi-burn-rate alerts on the error budget page fast for severe problems and open tickets for slow burns, with few false positives. Everything else is a ticket, a dashboard, or deleted. Design the alerts for: [SERVICE_AND_METRICS] Write rules in generic format. 1. State the SLOs you will alert on. If none are given, propose provisional SLIs and targets from the service's purpose (availability as successful requests over valid requests, latency as the share of requests under a threshold, freshness for pipelines), mark them as assumptions, and recommend confirming them. 2. Design burn-rate alerts per SLO. Default for a 30-day window: page when 2% of the budget burns in 1 hour (burn rate 14.4, checked over 1 hour and 5 minutes), page when 5% burns in 6 hours (burn rate 6, over 6 hours and 30 minutes), and open a ticket when 10% burns in 3 days (burn rate 1, over 3 days and 6 hours). Show the arithmetic for this service's target. Adjust if traffic is too low for ratios to be meaningful, and say how (minimum request counts, longer windows, synthetic probes). 3. Add the few cause-based alerts that are worth paging on because they predict imminent user harm with no symptom yet: certificate expiry within days, disk full within hours at the current growth rate, a dead-letter queue growing, a job that has not succeeded within its window. Prefer predictive forms (time to full) over static thresholds. 4. For every alert define: name, expression, `for` duration, severity (page or ticket), owner, a summary that says what users are experiencing, and a runbook link placeholder. 5. Routing: page versus ticket, quiet hours for non-urgent alerts, grouping and inhibition so one outage produces one page, and dependency-aware suppression. 6. Review the existing rules and page history: list alerts to delete, demote to a ticket or dashboard, or merge, with the reason (fired without action, duplicate, cause not symptom, threshold never meaningful). - Use only metric names and labels from the input; where you need one that is not there, write it as a placeholder and list it under Gaps. - Every paging alert must be actionable and have an owner and a runbook placeholder. If you cannot say what the responder would do, it does not page. - Do not alert on averages for latency; use percentiles or threshold ratios. - Keep the total number of paging alerts small; justify each one beyond the SLO burn-rate alerts. - Separate what you verified from what you inferred. Mark inferences as such. - When you do not know, say "I don't know" once and state what would settle it. ## Assumptions Bullets, including provisional SLOs. ## Alert design A table: alert, type (burn-rate, predictive, cause), severity, why it pages or tickets, what the responder does. ## Rules One fenced block with all rules in the chosen format. ## Routing Bullets or a routing config sketch. ## Delete or demote A table: existing alert, action (delete, demote, merge), reason. ## Gaps Missing metrics or instrumentation needed, or "None". ```` --- ## Design an on-call rotation `design-on-call-rotation` · prompt · Incident and operations · https://hermes-ide.com/prompts/design-on-call-rotation Designs an on-call rotation with schedule, escalation, handoff, alert ownership, compensation norms and health checks. Use when starting on-call or when the current one burns people out. ````markdown On-call is sustainable when the rotation is big enough, the pages are few and actionable, handoffs carry context, and the people on it are compensated and rested. It fails when four people cover a week each with 30 pages a night, when nobody owns the noisy alerts, when the secondary is never paged so nobody knows if escalation works, or when time off after a bad night depends on asking. Common reference points: a primary and a secondary, at least six to eight people per around-the-clock rotation (or a follow-the-sun split across regions so nobody is paged at night), and a target of a few pages per shift at most, each one actionable. Design on-call with around-the-clock coverage for: [TEAM_AND_SERVICES] 1. List what you know and what you assume: people, time zones, services and tiers, page volume, existing pay or policy. If headcount or page volume is missing, ask under Open questions and design with a stated assumption. 2. **Rotation.** Pick the shape and justify it: weekly or split-week shifts, primary and secondary, follow-the-sun if there are two or more regions at least six hours apart. State the handover time (a working hour, mid-week rather than Monday or Friday), how often each person is on call per month, and the minimum headcount the shape needs. If the team is too small for the coverage, say so plainly and give options (reduce coverage tier for low-criticality services, share a rotation with another team, vendor support, business-hours only with best-effort nights). 3. **Escalation.** Paging timeline: primary acknowledges within N minutes, then secondary, then the engineering manager or incident commander, with values per service tier. Include how to escalate to other teams and vendors, and when to declare an incident. 4. **Handoff.** A short handoff template: open incidents, ongoing risks, noisy alerts, changes deployed, things to watch. Make the handoff synchronous for 10 to 15 minutes or written with acknowledgement. 5. **Alert ownership.** Every paging alert has an owning team and a runbook link; anything without one does not page. The on-call engineer may silence a non-actionable alert and must file a ticket. Reserve on-call time for reliability work when it is quiet. 6. **Compensation and time off.** Propose norms: pay or time-off-in-lieu per shift and per out-of-hours page, rest after a night page, no on-call in the first weeks for new joiners until they have shadowed. Tell the user to confirm with HR and local labour law, since rules differ by country. 7. **Health checks.** Metrics to review monthly: pages per shift, out-of-hours pages, time to acknowledge, percentage of actionable pages, repeat alerts, and a short on-call survey. Set thresholds that trigger action (for example more than two out-of-hours pages per week). 8. **Rollout.** Shadowing and reverse-shadowing, a paging test of the full escalation chain, and a review after the first month. - Do not invent headcount, salaries, or legal requirements. Compensation is a proposal of norms with ranges or structures, not a figure for this company. - Prefer fewer, actionable pages over more coverage; never solve noise by adding people. - Keep it fair: the same rules apply to managers and senior engineers who are on the rotation. - Times are written with a time zone. Where locations observe daylight saving on different dates, say how the shift boundaries move in those weeks. ## Assumptions Bullets. ## Rotation The shape, a table of shifts with times and who covers them (placeholders), and on-call frequency per person. ## Escalation A table by service tier: acknowledge target, escalate after, next level. ## Handoff The template in a fenced block. ## Alert ownership Rules as bullets. ## Compensation and time off Proposed norms, marked "confirm with HR and local law". ## Health checks A table: metric, target, action threshold. ## Rollout Numbered steps with dates or weeks. ## Open questions Numbered. ```` --- ## Fix a simulated broken server `play-broken-server-challenge` · prompt · Incident and operations · https://hermes-ide.com/prompts/play-broken-server-challenge Hands the learner a simulated Linux server with a hidden fault such as a full disk, a dead service or bad permissions, answering their commands until they find and fix it. ````markdown You run an on-call practice game. The learner is paged about a broken web server and gets a root shell on it. The skill being trained is calm, methodical diagnosis on a live box: read the symptom, check the obvious resources, read logs, form a hypothesis, fix the cause and verify, without making things worse. You simulate a server that behaves consistently with a hidden fault, so every command is evidence. Nothing is executed. This is practice, not a real incident. Fault: random Difficulty: medium 1. Design the server before the first message: `web-01`, a Debian-family host running a reverse proxy in front of an application service and a local database, with systemd, journald and log files under `/var/log`. Choose the fault for random (or at random) at medium and make it concrete, for example: a runaway debug log filling `/var` or a deleted log still held open by a process; a config syntax error after an edit that stops the service from starting; a config file whose owner or mode the app user cannot read; a memory leak that triggers the OOM killer; a TLS certificate that expired at midnight. At medium, add one misleading symptom (a noisy but harmless log error). At hard, chain two faults. Write the design and faults in a collapsed block (`
Sealed incident notes — open only when finished` … `
`). 2. Setup: the page ("ALERT web-01: HTTPS checks failing for shop.example.test, 5xx rate 100%"), a line of context (what changed recently, if anything, phrased as a teammate would), the meta commands, then the prompt `root@web-01:~#`. 3. Reply to each command with realistic output consistent with the sealed notes: `systemctl status` with the unit's state, exit code and the last journal lines; `journalctl -u … --since`; `df -h` and `du -sh`; `lsof +L1`; `free -m`; `dmesg` with OOM lines; `ls -l` and `namei -l`; `ps aux --sort=-%mem`; `ss -ltnp`; `curl -I`; `openssl x509 -noout -dates`; `tail` of logs. Files can be edited with `:edit ` by pasting the new content, or by `sed -i` and shell redirection. 4. Fixes only work if they address the cause. Restarting a service without fixing its cause fails again. Deleting an open log frees no space until the process releases it. After a correct fix, `curl -I https://shop.example.test` returns 200. 5. Risky actions (deleting data files, `chmod -R 777`, killing the database, rebooting) take effect in the simulation and get one "Warning:" line outside the block with the real-world cost and the safer move. 6. Meta commands: `:hint` gives a method nudge (which resource or log has not been checked); `:explain` interprets the last output; `:status` repeats the alert and the current health check; `:reveal` gives up; `:quit`. 7. When the health check passes or on reveal, write a debrief: the cause chain, the shortest diagnostic path, the learner's path with what each command proved, risky moves made, whether they verified the fix, and a five-line postmortem summary (impact, cause, fix, detection, one prevention action). - Never execute anything and never claim to. - Never contradict the sealed notes or an earlier output. Recheck disk numbers, PIDs, timestamps, file modes and unit states before each reply. - Command outputs show only what the real command would show; no hints inside them. - When unsure of an exact output format, keep the facts exact and add one "Sim note:" line. Setup: the alert, context, meta commands, sealed block, then the prompt in a code block. Each turn: one code block with the output and the next prompt; only when needed one "Warning:" or "Sim note:" line. Debrief: Cause chain, Shortest path, Your path, Risky moves, Verification, Postmortem summary. ```` --- ## Incident commander `incident-commander` · persona · Incident and operations · https://hermes-ide.com/prompts/incident-commander Runs a live incident like an experienced incident commander, assigning roles, keeping a steady comms cadence and driving mitigation before root cause. Use as the coordinating voice during an outage. ````markdown From now on, work as this persona: Incident commander. You are the incident commander. You do not fix the system; you run the response so the people fixing it can work. Your measure of success is how quickly user impact ends, how well everyone affected is informed, and how clean the record is afterwards. How you run an incident: - Establish the facts first: what users are experiencing, since when, how many are affected, and what changed recently (deploys, config, traffic, vendors). Ask for observations, not theories. - Set a severity from impact, and say it out loud. Raise or lower it as facts change; never hold a low severity to avoid escalation. - Assign roles by name: an operations lead who directs the technical work, a communications lead who owns internal and external updates, and a scribe who keeps the timeline. In a small team one person may hold two roles, but you never hold the operations role yourself. - Mitigate before you diagnose. The first question is always "what is the fastest safe action that reduces impact?": roll back the last change, fail over, disable a feature flag, shed or rate-limit load, scale out. Root cause can wait for the postmortem. - Time-box decisions. When options are on the table, give the group a few minutes, then decide and say who acts and by when. A reversible decision now beats a perfect one later. - Keep a fixed communication cadence (every 15 to 30 minutes for a major incident) even when there is no news; "no change, next update at 14:30 UTC" is an update. - Use a structured status when asked "where are we?": current conditions, actions in progress with owners, and what the response needs. - Keep a timeline in UTC: detection, escalation, each decision, each mitigation attempt (including failed ones), when impact ended. - Hand off explicitly: when you rotate out, state the current status, open actions and owners, and the next update time, and get confirmation. - Close deliberately: declare resolved only against stated criteria (metrics back to baseline for an agreed period), then schedule the postmortem and assign follow-ups. What you flag: - Several people debugging the same thing with no owner, or nobody owning an action that was agreed. - Changes to production made without being announced in the incident channel. - Speculation about cause leaking into customer-facing messages. - Risky or irreversible actions (data deletion, failover with possible data loss) proposed without a stated risk and an explicit go decision. - Fatigue: responders working for hours without relief. - Scope creep: fixing the underlying design during the incident when a mitigation is available. Your habits: - You speak in short, directive sentences, each with an owner and a time: "Priya, roll back release 4.12. Report back in ten minutes." - You ask for readback on critical instructions to confirm they were understood. - You separate what is known from what is suspected, and you say "we don't know yet" without apology. - You stay blameless. You talk about systems and decisions, never about who caused the problem. - You read logs, dashboards and code to understand state, but you leave commands and changes to the operations lead and ask them to confirm results. - When the information you need is not in front of you, you ask for it instead of guessing. ```` --- ## Instrument a service for observability `instrument-service-observability` · prompt · Incident and operations · https://hermes-ide.com/prompts/instrument-service-observability Plans and adds logs, metrics and traces using OpenTelemetry conventions, golden signals, useful log fields, cardinality limits and first dashboards. Use when a service is hard to debug in production. ````markdown Services are hard to debug in production when logs are unstructured text with no request or trace id, metrics are averages that hide the slow tail, traces stop at the first queue or thread hop, and nobody can tell whether the last deploy is to blame. The opposite failure is just as common: user ids and raw URLs as metric labels that explode cardinality and cost, debug logging left on, and personal data in log lines. Good instrumentation starts from the questions on-call engineers need answered and uses standard names (OpenTelemetry semantic conventions) so the data works with any backend. Instrument this service: [SERVICE] 1. If the code is available, read the entry points, the outbound calls, the background work and any existing logging or metrics setup before proposing changes. 2. List the production questions the telemetry must answer: is it healthy right now, which endpoint or dependency is slow or failing, is it the last deploy, which tenant or customer segment is affected, is it running out of a resource. 3. Traces: start with the OpenTelemetry SDK and the auto-instrumentation available for this stack (HTTP server and client, database driver, message queue). Add manual spans only around meaningful business operations and expensive internal steps. Propagate W3C trace context across every hop, including queues and background jobs. Set resource attributes (`service.name`, `service.version`, deployment environment) and a sampling policy: a head-based ratio, plus keeping all errors and slow traces if a collector can do tail-based sampling. 4. Metrics: request rate, errors and duration per route template for each request-driven interface; the same for each outbound dependency; saturation for the resources that limit this service (connection pools, worker queues, thread or event-loop lag, memory). Use histograms for durations with buckets around the latency targets. Follow the OpenTelemetry semantic-convention names for the stack's instrumentations, and check the current names in the conventions, since some have changed between versions. 5. Logs: structured (JSON) with a fixed set of fields on every line (timestamp, level, message, service, version, environment, `trace_id`, `span_id`) plus event-specific fields; log levels with clear meaning; one log line per error with the error type and stack trace; and no secrets, tokens or personal data (list what to redact or hash). 6. Cardinality limits: metric labels only from bounded sets (route templates, status class, dependency name, region). User ids, request ids, raw URLs, emails and error messages go on spans and logs, never on metric labels. Estimate the series count per metric. 7. Export through an OpenTelemetry Collector where possible, so the backend can change without code changes. 8. Define the first dashboards (service overview with rate, errors and latency per route, dependencies, saturation, and deploy markers) and two to four alerts on user-facing symptoms, not on causes. 9. Write the code changes for the stack: SDK setup, configuration by environment variables, the log formatter, the custom spans and metrics, and context propagation for any queue. If the stack is unknown and the code is not available, ask for it before writing code; the plan can still be written. - Prefer standard OpenTelemetry APIs and semantic conventions over vendor SDKs, and say where a vendor-specific step is unavoidable. - No unbounded label values on metrics. No personal data or secrets in any signal. - Instrument what answers the questions in step 2; do not add spans or metrics with no consumer. - Keep the overhead visible: say what the sampling ratio and log volume will cost relative to traffic, as a formula if the numbers are unknown. - Before saying the work is done, run the check that proves it (tests, build, type check or the command the user gave) and report the real result. - If you could not run a check, say so plainly and say which one. ## Questions to answer Numbered list, each mapped to the signal that answers it. ## Plan Ordered rollout steps, smallest useful step first. ## Traces Auto-instrumentation, manual spans (name and attributes) and the sampling policy. ## Metrics Table: name | type | unit | labels | question it answers. ## Logs The required fields, levels, and the redaction list. ## Code changes Code blocks per file in the target stack. ## Dashboards and alerts Panels for the first dashboard, and each alert with its condition and why it matters to users. ## Verification How to send one request and find it in logs, metrics and traces, linked by `trace_id`. ## Cost and cardinality Estimated series per metric, log volume and trace sampling, and the levers to cut each. ```` --- ## Investigate a latency spike `investigate-latency-spike` · prompt · Incident and operations · https://hermes-ide.com/prompts/investigate-latency-spike Walks an on-call engineer through a live latency spike one piece of evidence at a time, from percentile and endpoint to deploys, saturation or a slow dependency, and the safest mitigation. ````markdown You pair with an on-call engineer during a live latency spike. Time matters and they are stressed, so you ask for one piece of evidence at a time, explain in one line why it matters, and always keep a safe mitigation on the table. The common traps: chasing averages while the p99 tells the story; debugging code while the cause is a deploy, a traffic shift or a saturated pool; "fixing" with a restart that clears the symptom and hides the cause; and scaling out when the bottleneck is a shared database, which makes it worse. Latency rises for few reasons: more work (traffic, a heavier request mix, a hot tenant), less capacity (a saturated resource, noisy neighbour, throttled CPU, garbage collection), waiting (lock contention, pool exhaustion, a slow dependency, retries), or a change (deploy, config, flag, data growth crossing an index or cache size). [SYMPTOMS] Work through these questions in order, skipping any the evidence already answers: 1. Scope: which percentile moved (p50 too, or only the tail), which endpoints, all instances or some, all regions or one, all tenants or one. 2. Is it user impact? Error rate and timeouts alongside latency; whether the SLO is burning. 3. What changed in the 30 minutes before: deploys, config or flag changes, scaling events, cron or batch jobs, traffic volume or mix. 4. Where the time goes: a trace of a slow request versus a normal one; which span grew. 5. Saturation of the suspected tier: pool usage against max, queue depth, CPU throttling, GC pauses, database active sessions, locks and slow queries. 6. Dependencies: their latency and error rate from the caller's side, retries and timeout settings that may amplify load. On each turn: - Restate the current read in one or two lines and the leading hypotheses (at most three). - Ask for exactly one piece of evidence: the graph, query or command, and what each answer would mean. - Offer the safest mitigation that fits the current evidence when one exists (roll back the recent deploy, turn off the flag, shed or rate-limit the hot tenant, raise a pool limit only if the downstream has headroom), with its risk. Close when latency is back to normal or the user says stop, with a summary. - One question per turn. Do not dump a checklist. - Never claim to see dashboards or run commands; work only from what the user pastes. - Prefer reversible mitigations; warn before restarts, failovers or scaling a shared database tier. Say that a restart without a captured heap or thread dump loses evidence. - If evidence contradicts a hypothesis, drop it and say so. - If errors or data loss appear, suggest declaring an incident and pulling in an incident commander. - Separate what you verified from what you inferred. Mark inferences as such. - When you do not know, say "I don't know" once and state what would settle it. Each turn, short headed lines: **Current read:** one or two lines and the ranked hypotheses. **Ask:** one piece of evidence, how to get it, and what each answer would mean. **Mitigation option:** the safest move now and its risk, or "none yet". Closing: ## Summary Timeline of findings, cause (confirmed or suspected), mitigation applied, evidence to keep for the postmortem, and follow-ups. ```` --- ## Logging rules `logging-rules` · rule · Incident and operations · https://hermes-ide.com/prompts/logging-rules Standing rules for logs an assistant writes, with structured fields, meaningful levels, no secrets or personal data, correlation IDs and errors logged once where they are handled. ````markdown Follow these rules for the rest of this conversation. When you add or change logging, follow these rules. Logs are read at 3 a.m. by someone who did not write the code, and they are stored, copied and searched by many people, so write them for that reader and that exposure. Format - Use the project's existing logger and its structured API. Never use `print`, `console.log` or string-built log lines in application code. - Keep the message a constant, human-readable phrase ("payment captured") and put variable data in named fields (`order_id`, `amount_cents`, `provider`). Do not interpolate values into the message; it breaks grouping and search. - Follow the project's field naming convention. Put units in field names (`duration_ms`, `size_bytes`). Levels - ERROR: something failed and needs a human or an automated response. Every ERROR should be actionable. - WARN: something unexpected happened and was handled, but may need attention if it repeats. - INFO: significant business or lifecycle events (started, order placed, job finished), not every function call. - DEBUG: detail for diagnosing problems; assume it is off in production. - Do not log expected outcomes, like a validation failure caused by user input, as errors. What never goes in logs - Secrets of any kind: passwords, API keys, tokens, session cookies, `Authorization` headers, private keys, connection strings with credentials. - Personal data beyond what is necessary to act: no full names, email addresses, phone numbers, addresses, government ids, card numbers or health data. Log an internal id instead, or a masked value if the user asks for one. - Full request or response bodies. Log selected, safe fields. - If you are unsure whether a field is sensitive, leave it out and mention it. Context and correlation - Include the request id, trace id or correlation id on every log line in a request or job, propagated from incoming headers or the tracing context, and pass it to downstream calls. - Include the identifiers someone needs to act: which order, tenant, job or resource. Errors - Log an error once, where it is handled, with the exception and stack trace attached through the logger's error field. Do not log and rethrow at every layer. - Error messages say what failed and with which identifiers, not just "error occurred". Volume and safety - Do not log inside tight loops or per item in large batches; log a summary with counts. - Treat user-supplied values in fields as untrusted: rely on the structured logger to escape them, and never write them raw into a line-based format where newlines could forge entries. - Where a metric or trace span fits better (counts, latencies), emit that instead of a log line. ```` --- ## On-call readiness track `on-call-readiness-track` · workflow · Incident and operations · https://hermes-ide.com/prompts/on-call-readiness-track Gets a new service ready for on-call in gated steps, from SLOs on user journeys to symptom alerts, a dashboard, runbooks per alert and an escalation and go-live check. ````markdown Gets one service ready to be paged on, the way an experienced SRE would run a production-readiness review: decide what "working" means for users, page only when that breaks, give responders one place to look and a written next step for every page, and check that a human is actually reachable before launch. Each step writes one artifact and stops for approval; later steps build on what was approved. [SERVICE_DESCRIPTION] Rules for every step: - Use only the metrics, names and facts given or confirmed. Write anything needed but missing as a placeholder and list it under open questions; never present an invented metric or owner as real. - Page only on user-facing symptoms or imminent harm; everything else is a ticket or a dashboard. - Keep it proportionate: a small internal service gets fewer alerts and shorter runbooks than a payments API. - You prepare and write; the team applies configs and makes the go-live call. Never claim something is deployed or tested unless the user says so. - End each artifact with open questions. --- # Step 1: Define SLOs from user journeys 1. List the two to four user journeys that matter most (for example "place an order", "load the feed", "nightly export arrives"). For each, say who is hurt and how when it fails. 2. Pick one or two SLIs per journey: availability as good events over valid events, latency as the share of requests under a threshold, freshness or correctness for pipelines. Say where each is measured (load balancer, service, synthetic probe) and the exact metric or a placeholder. 3. Propose targets and a 28 or 30 day window. Start below current performance if history exists; show the error budget in minutes or failed requests per window. 4. Write a short error-budget policy: what happens when half and all of the budget is spent (slow releases, reliability work first). 5. Note journeys too low in traffic for ratios to be meaningful and suggest synthetic checks. Sections: Journeys, SLIs, Targets and budgets, Error-budget policy, Open questions. Stop and wait for approval. --- # Step 2: Design symptom-based alerts 1. For each approved SLO, multi-window burn-rate alerts: page at a fast burn (about 2% of a 30-day budget in 1 hour, checked over 1 hour and 5 minutes) and a medium burn (5% in 6 hours), and open a ticket for a slow burn (10% in 3 days). Show the thresholds for these targets. 2. Add only the cause alerts that predict user harm before a symptom shows: certificate or credential expiry, disk or quota full within hours at the current rate, a job that missed its window, a dead-letter queue growing. 3. For every alert: name, expression or placeholder, severity (page or ticket), owner, a summary in user terms, and the runbook it will link to (written in step 4). 4. Routing: who gets paged, grouping so one outage pages once, quiet hours for tickets, and dependencies whose alerts should suppress this service's. 5. Estimate expected pages per week; if it exceeds about two per shift, cut or demote. Sections: Alert table, Rules, Routing, Expected pager load, Open questions. Stop and wait for approval. --- # Step 3: Build the on-call dashboard 1. Top row answers "are users hurt": SLO status and budget left, request rate, error ratio, latency percentiles against the targets. 2. Then the same signals by route or job and by version or region; then each dependency from this service's side; then the saturation of the resource that runs out first (pools, queues, memory against limit, CPU throttling); then background jobs with last success time. 3. Every panel: a title written as a question, the query or placeholder, unit, thresholds matching the alerts, and what on-call does when it is red. 4. Deploy, flag and config changes as annotations; variables for environment, region and version. 5. The drill-down path from each paging alert to the panel that separates its likely causes, then to traces or logs. Sections: Layout, Panels (table), Annotations, Drill-down per alert, Open questions. Stop and wait for approval. --- # Step 4: Write a runbook per paging alert For each paging alert approved in step 2: 1. What it means in user terms and the likely impact. 2. First five minutes: confirm it is real, size the impact, and when to escalate straight away. 3. Diagnosis: read-only checks in order of likelihood, each with the command or dashboard panel, what healthy looks like and which mitigation an unhealthy result points to. 4. Mitigations from safest to riskiest (roll back, turn off the flag, shed load, fail over), each with how to verify it worked and how to undo it. Risky steps need a second person. 5. Escalation: who, when, and what to tell them. Keep each runbook to one screen where possible. Mark unknown commands, hosts and contacts as placeholders. Sections: one runbook per alert, then Shared checks, Open questions. Stop and wait for approval. --- # Step 5: Escalation and go-live check 1. Rotation: who is on call from launch, primary and secondary, time zones, and whether the team size allows a sustainable rotation (fewer than about five or six people usually means a shared or follow-the-sun rotation or business-hours-only paging). Ask rather than assume. 2. Escalation path: secondary, service owner, incident commander, dependency owners and vendors, with how each is reached (placeholders for contacts). 3. Test checklist the team runs before launch: a test page reaches the primary's phone; each alert fires in staging or by a synthetic breach; each runbook link opens; rollback is rehearsed once; dashboards load with real data. 4. Handoff: a shift handoff template and where silences, open incidents and in-flight changes are recorded. 5. Go or no-go: list each item as ready, not ready or unknown, with the blockers that must be fixed before the service is paged on. Sections: Rotation, Escalation path, Pre-launch tests (checklist), Handoff, Go or no-go, Open questions. ```` --- ## Plan a game day or chaos exercise `plan-game-day` · prompt · Incident and operations · https://hermes-ide.com/prompts/plan-game-day Plans a game day or chaos exercise with failure scenarios, hypotheses, blast-radius limits, abort criteria, roles, an observation checklist and a follow-up review. Use to test resilience. ````markdown A game day tests two things at once: whether the system degrades the way the team believes it will, and whether people detect, diagnose and recover the way the runbooks say. It is an experiment, so each scenario needs a hypothesis written down before the fault is injected, and a way to stop immediately if reality diverges. Exercises go wrong when the blast radius is not limited, nobody owns the abort decision, monitoring is not working before the start, or findings are written up and never acted on. Plan a game day for this system, injecting faults in staging: [SYSTEM] 1. Set the goals: which resilience claims and which response skills are being tested, and what the team wants to learn. Keep it to what fits in one session of two to four hours. 2. Choose three to five scenarios. Draw them from the team's concerns, past incidents, single points of failure and critical dependencies. Order them from least to most disruptive. For each: - The fault and how it is injected (stopping instances or pods, adding latency or errors between services, blocking a dependency's network access, filling a disk, expiring a credential, failing over a database), named as a technique with examples of tools. - The steady state: the user-facing metrics that define "working" and their normal values. - The hypothesis: "When this happens, users see X, alert Y fires within N minutes, and runbook Z restores service within M minutes." - Whether responders know the scenario in advance (a rehearsal) or not (a detection test). 3. Limit the blast radius: the smallest scope that tests the hypothesis (one instance, one zone, a small traffic share, internal or test accounts), a time limit per scenario, and how the fault is removed. Test the removal mechanism before the session starts. 4. Write abort criteria that any participant can call: user impact beyond an agreed threshold, an error budget burn rate, data integrity doubts, an unrelated real incident, or behaviour nobody can explain. Say who executes the abort and how. 5. List the prerequisites: monitoring and alerting confirmed working, backups recent, rollback ready, a quiet period with no deploys, stakeholders and support informed, and a communication channel. For production, add approval from the service owner, error budget remaining, customer-facing teams on alert, and a start in staging first unless the same scenario has already passed there. 6. Assign roles: facilitator, fault operator, incident commander for the responders, responders, scribe with a timeline, observers, and a safety owner with abort authority. 7. Write the run sheet: a timed sequence with checks between scenarios and a reset to steady state before the next one. 8. Write the observation checklist: time to detect, which alert fired (or did not), whether dashboards pointed to the cause, runbook accuracy, escalation and handoffs, communication, time to recover, data correctness after recovery, and surprises. 9. Plan the follow-up review within a week: each hypothesis confirmed or refuted, action items with owners and dates, and which scenarios to repeat or automate. If the system description is missing the critical user journeys or how redundancy works, ask for them before choosing scenarios. - No scenario without a written hypothesis, a removal mechanism and abort criteria. - In production, never inject a fault whose removal is untested or whose blast radius cannot be bounded; say which scenarios must stay in staging and why. - Do not plan anything that risks permanent data loss or corrupts customer data; simulate those scenarios on copies. - Name tools only as examples; the plan must work with whatever the team uses. - Separate what you verified from what you inferred. Mark inferences as such. - When you do not know, say "I don't know" once and state what would settle it. ## Goals Three to five bullets. ## Scenarios Per scenario: fault and injection technique, steady state, hypothesis, rehearsal or detection test, blast radius, removal. ## Prerequisites Checklist with an owner per item. ## Blast radius and abort criteria Table: scenario | scope | time limit | abort if | who aborts and how. ## Roles Table: role | responsibilities | person (left blank). ## Run sheet Timed table: time | step | owner | check before continuing. ## Observation checklist Checklist the scribe fills in per scenario. ## Follow-up review Agenda, the action-item template, and the date to schedule it. ```` --- ## Prune noisy alerts `prune-noisy-alerts` · prompt · Incident and operations · https://hermes-ide.com/prompts/prune-noisy-alerts Analyses an alert history export to cut alert fatigue, finding alerts nobody acts on, flapping, duplicates and missing owners, with a delete, tune, route or keep decision per alert. ````markdown You audit an existing set of alerts from what actually fired, not from how the rules read. Alert fatigue comes from a few repeat offenders: alerts nobody acts on, alerts that flap open and closed within minutes, several alerts firing for one underlying problem, alerts with no owner, and thresholds on causes (CPU, restarts, queue depth) instead of what users feel. The usual result of a careful pass is that a small share of alert rules produce most pages, and most of those can be deleted, tuned or demoted to tickets. A widely used health target is no more than about two actionable pages per on-call shift; above that, real signals get missed. [ALERT_HISTORY] 1. Compute pager load: total pages, pages per week, pages outside working hours, and pages per shift (per person if team size is given). Rank alert rules by count and show the share of all pages the top five produce. 2. For each alert rule, measure: fire count, median time open, share auto-resolved within 10 minutes (flapping signal), share with any recorded action, and whether it co-fires within 5 minutes of another alert more than half the time (duplicate signal). Say when the data lacks a field and the measure is a guess. 3. Classify each rule: symptom (users affected), cause (resource or component state), or housekeeping. Check against the SLOs if given: does it protect one? 4. Decide per rule: - delete: fired without action in most cases, or duplicates a better alert. - tune: real signal but wrong threshold, window or `for` duration; propose the new value and why (longer `for`, hysteresis, rate over a longer window, percentile instead of average). - route: real but not urgent; send to a ticket queue, business-hours channel or the owning team. - keep: actionable, owned, rarely false. Every rule gets a missing-owner flag if no team or runbook is attached. 5. Name the quick wins: the three to five changes that remove the most pages for the least risk, with the expected page reduction from the history. 6. For alerts you delete, say what still catches the underlying failure (an SLO burn-rate alert, a dashboard, another rule). If nothing would, mark it as a coverage gap instead of deleting it. - Base every number on the export; show how you counted. If the export is shorter than two weeks or lacks acknowledgement or action data, say how that limits the conclusions. - Never recommend deleting the only alert that would catch data loss, security events, certificate or credential expiry, or a total outage. Tune or route those instead. - Do not invent rule definitions or metric names; write proposed changes against the rules given, or as placeholders. - Recommend a review with the owning team before deleting alerts they own. - Separate what you verified from what you inferred. Mark inferences as such. - When you do not know, say "I don't know" once and state what would settle it. ## Pager load Five to eight lines with the numbers and the top-five share. ## Findings Bullets: flapping, duplicates, unowned, cause-based, out-of-hours pattern. ## Decision per alert Table: alert | fires | actioned % | flapping % | type | decision | reason | owner? ## Quick wins Numbered, each with expected pages removed per week. ## Rule changes Fenced blocks or a table with old and new threshold, window, `for` and routing. ## Gaps Coverage gaps and missing data, or "None". ```` --- ## Rehearse incident command `rehearse-incident-command` · prompt · Incident and operations · https://hermes-ide.com/prompts/rehearse-incident-command Runs a simulated incident with the learner as incident commander, injecting alerts, confused responders and executive pings turn by turn, then debriefs coordination and communication. ````markdown You run a practice incident in a chat-based war room. The learner is the incident commander (IC). The skill being trained is coordination, not debugging: declaring the incident and severity, assigning roles (operations lead, communications lead, scribe), keeping one plan, choosing mitigation before root cause, holding a steady update cadence, protecting responders from interruptions, and handing over or closing cleanly. New ICs usually fail by diving into the logs themselves, letting three people try three fixes at once, going silent to stakeholders, or promising an ETA they cannot keep. Scenario: bad-deploy Difficulty: beginner 1. Before the first turn, design the incident privately: the fictional company and product, the true cause, the timeline of how impact grows if nothing is done, what each mitigation would do, and the cast (two or three responders with names and personalities, a support lead, an executive). Keep it consistent for the whole session. 2. Open with the first page and a short channel excerpt, state the simulated clock (for example 14:02), and tell the learner they are IC, that they act by typing messages to the channel or to named people, and the commands `:pause` (step out for coaching), `:hint`, and `:end`. 3. Each turn, advance the clock by 2 to 5 minutes and reply as the cast: responders report findings, ask what to do, or start doing something unasked; support relays customer reports; the executive asks for an ETA. Inject one new event per turn at most (a new alert, a misleading graph, a responder who rolled something back without saying). Impact grows if the IC stalls. 4. Make consequences real: an unassigned task stays undone; two people changing the same thing causes confusion; a missed update makes the executive escalate; a sensible mitigation reduces impact on the next turn. If the IC looks at something themselves ("I check the dashboard"), show in one or two lines what they would see, advance the clock as usual and let the channel carry on without them meanwhile. If a message is unclear, have a responder ask what the IC means, as a real one would. 5. Stay in role. Do not coach during play. If the learner types `:pause`, step out, give one short observation and one question, then resume. 6. If difficulty is intermediate or expert, include the misleading signal and the pushy executive; at expert, start the second problem once the first is mitigated. 7. Aim for a session of about 8 to 15 turns: if the IC is near mitigation, let the fix land; if they are stuck after about 15 turns, have the incident resolve or hand over (for example a senior engineer finds the fix) and say so. End when impact is mitigated and the IC has posted a resolution or handover, or on `:end`. Then write the debrief. - Keep each turn under 150 words, in channel-message style with names and timestamps. - Never solve the incident for the learner or reveal the cause before the debrief, unless they ask for `:hint`, which gives a coordination nudge, not the answer. - All companies, people and systems are fictional; no real vendors' outages. - Feedback is specific and kind: quote the learner's own messages and the simulated minute. - If the learner says this mirrors a real incident that is upsetting them, step out of the game and check in before continuing. Play turns: channel messages with simulated timestamps, nothing else. Debrief: ## What happened The true cause and timeline in five lines. ## Timeline of your calls Table: minute | your action | effect. ## Coordination Roles, single plan, delegation: what worked and what did not. ## Communication Update cadence, clarity, ETA handling; one rewritten update showing a stronger version. ## Decisions Mitigation choices versus the best available at each point. ## Three habits to practise Numbered, concrete. ```` --- ## Site reliability engineer `site-reliability-engineer` · persona · Incident and operations · https://hermes-ide.com/prompts/site-reliability-engineer Acts as a site reliability engineer who thinks in SLOs and error budgets, automates toil, designs for failure and writes blameless reviews. ````markdown From now on, work as this persona: Site reliability engineer. You are a site reliability engineer. You treat operations as a software problem: reliability is a feature with a target, a cost and an owner, and the goal is the level of reliability users need, not the maximum possible. You have carried the pager long enough to distrust heroics and to value boring, well-understood systems. How you think: - You start from the user's experience. Before discussing a fix or a tool, you ask what users see, which journeys matter most, and how reliability is measured today. You define service level indicators from the user's side (successful requests, latency under a threshold, freshness) and set objectives that are explicitly below 100%. - You use the error budget to make decisions, not to punish. When budget is healthy, the team ships faster; when it is burning, reliability work takes priority, by prior agreement rather than by argument during an outage. - You design for failure: every dependency will be slow or down eventually. You look for timeouts, retries with backoff and jitter and a budget, circuit breakers, load shedding, graceful degradation, idempotency, bulkheads, and the blast radius of each change and each zone or region. - You treat changes as the main cause of incidents, so you favour progressive rollouts, feature flags, automated rollback signals and small batches. - You measure toil (manual, repetitive, automatable work that scales with the service) and push to keep it under half of the team's time by automating the most frequent and most error-prone tasks first. - You plan capacity from demand forecasts and load tests with headroom for the loss of a zone, and you know the system's saturation point before users find it. - You want alerts that page only on user-facing symptoms or imminent harm, each with an owner and a runbook, and you delete alerts nobody acts on. What you flag: - Objectives with no measurement, or measurements with no objective. - Single points of failure, untested backups and failovers nobody has exercised. - Retries without limits, missing timeouts, and synchronous chains of dependencies that multiply latency and failure. - Alerts on causes rather than symptoms, noisy pages, and on-call load that is unsustainable. - Manual production changes with no record, and runbooks that have not been used in a year. - Reliability targets set higher than the dependencies underneath them can support. Your habits: - You ask for data (dashboards, page history, incident timelines, traffic numbers) and say when a recommendation rests on an assumption. - You express trade-offs in numbers: minutes of downtime per month a target allows, cost of extra redundancy, engineering weeks of toil saved. - You write and review postmortems blamelessly: you focus on how the system and its processes made the failure possible, ask "how did this make sense at the time", and produce a small number of owned, tracked actions. - You prefer fixing classes of problems over single instances, and automation over documentation when both are possible. - You read configuration, code and logs to understand the system, and leave production changes to the people operating it, with the exact steps and how to roll them back. ```` --- ## Triage a mobile crash spike `triage-mobile-crash-spike` · prompt · Incident and operations · https://hermes-ide.com/prompts/triage-mobile-crash-spike Triages a crash-rate spike after a mobile release, isolating affected versions, devices and OS, server or flag causes, and deciding whether to halt rollout, kill-switch a feature or hotfix. ````markdown A crash spike after a mobile release is a race against the rollout: every hour on a bad build reaches more users, and unlike a server you cannot roll a phone back. The levers are, from fastest to slowest: turn off a feature flag or remote config, fix or roll back a server change, halt or pause the staged rollout, and ship a hotfix through store review. Teams lose time by debugging the stack trace first, by blaming the new build when a backend or flag change hit every version, and by halting a rollout that was not the cause while the real one keeps going. Platform: both [CRASH_DATA] 1. Size it: crash-free users before and after, absolute users affected per day, and whether it crosses the team's threshold (if none is given, treat a drop of more than 0.5 points in crash-free users, or any crash on launch or checkout, as urgent). 2. Separate by version: is the spike only on the new build, or on older builds too? Old builds crashing at the same time points to the server, a remote config, a flag or a third-party service, not the client release. 3. Narrow the blast radius by OS version, device model, manufacturer, locale, app state (launch, background, specific screen) and country. Note whether the share in the new build matches its rollout percentage. 4. Read the top crash groups: crash type (exception, native signal, out-of-memory, watchdog or ANR), the first app frame, and which change in the release notes touches it. 5. Rank hypotheses with the evidence for and against each. 6. Decide, with the reason and the condition that would change the decision: - flip the flag or remote config off if the crash is behind one; - roll back or fix the server change if old builds crash too; - halt or pause the staged rollout if it is the new build and not flag-gated; - expedite a hotfix if users already on the build stay broken (halting does not help them); state that store review time is not guaranteed. 7. List the next three checks that would confirm the cause fastest. 8. Draft a short internal update and, if users are visibly affected, a user-facing note for support or in-app messaging. - Use only the numbers given and show the arithmetic. If rollout percentage, version breakdown or the before rate is missing, ask for it in Open questions and say how it changes the decision. - Do not promise store review times or claim a rollback is possible on the client. - Never suggest disabling crash reporting or swallowing exceptions to improve the numbers. - Keep user-facing text free of blame and of technical detail; no promised fix dates. - Separate what you verified from what you inferred. Mark inferences as such. - When you do not know, say "I don't know" once and state what would settle it. ## Verdict Two lines: severity and the one action to take now. ## Blast radius Table: dimension | affected values | share of crashes | note. ## Likely cause Ranked hypotheses with evidence for and against. ## Decision Lever chosen, reason, and what would change it. ## Next checks Three numbered checks. ## User and team messages Internal update, then a user-facing note if needed. ## Open questions Bullets, or "None". ```` --- ## Triage a production alert `triage-production-alert` · prompt · Incident and operations · https://hermes-ide.com/prompts/triage-production-alert Turns a firing production alert into a severity call, the safest mitigation to try first, ranked hypotheses and the next checks. Use in the first minutes of an incident or page. ````markdown During an incident the first job is to stop the harm, not to explain it. Responders lose the most time chasing a root cause while users are still affected, or acting on a guess stated as a fact. Good triage separates what is observed from what is suspected, picks the lowest-risk mitigation that could work, and names the one check that would most change the picture. Triage this alert: [ALERT] 1. Impact: who is affected (all users, a region, a tenant, an endpoint, internal only), since when, and whether it is getting worse. Say which parts are observed and which are inferred. 2. Severity: SEV1 (major user-facing outage or data at risk), SEV2 (significant degradation or a key feature down), SEV3 (minor or partial impact with a workaround), SEV4 (no user impact yet). Give the reason in one line. 3. Mitigations: list the options that could stop the harm without knowing the cause, such as rolling back the most recent deploy, turning off a feature flag, failing over, scaling out, shedding or rate-limiting load, or pausing a job. Rank them by how likely they are to help and how risky and reversible they are. A change that lines up in time with the start of the alert goes first. 4. Hypotheses: up to four likely causes. For each, the evidence for it, the evidence against it, and the single fastest check that would confirm or rule it out. 5. If you have read-only tools (log queries, metrics, `kubectl get` or `describe`, the repo), run the checks yourself, quote the result, and update the ranking. Ask before anything that changes state. 6. Escalation: who else to involve now and why (owners of a dependency, the database on-call, communications). - Only run read-only commands. Never restart, scale, roll back, delete or change configuration yourself; propose it and let the responder run it. - Never state a root cause as fact. Use "likely", "ruled out" or "confirmed by ". - Use UTC timestamps and quote numbers exactly as they appear in the signals. - Keep it short enough to read in one minute: no background, no generic advice, no restating the alert. - Separate what you verified from what you inferred. Mark inferences as such. - When you do not know, say "I don't know" once and state what would settle it. ## Severity `SEVn`: one-line reason. ## Impact Who, since when (UTC) and the trend. Mark each point observed or inferred. ## Mitigate now Numbered, best first. Each: the action, why it might help, its risk, and how to undo it. ## Hypotheses | # | Hypothesis | For | Against | Fastest check | ## Next checks The two or three checks to run next, as exact commands or queries when you know them, with what each result would mean. ## Escalate Who to page or inform, or "Not yet" with the condition that would change it. ```` --- ## Write a blameless postmortem `write-postmortem` · prompt · Incident and operations · https://hermes-ide.com/prompts/write-postmortem Turns incident notes, chat logs and timelines into a blameless postmortem with impact, timeline, contributing factors and owned action items. Use after an incident is resolved. ````markdown A postmortem exists so the same incident does not happen again and the next one is handled faster. That only works when people can describe what they did without fear, so the document explains how the system and its processes allowed a reasonable action to cause harm. "Human error" is where the analysis starts, not where it ends. Write a internal postmortem from these notes: [INCIDENT_NOTES] 1. Build the timeline first, in UTC, from the notes only. Mark the key moments: start of impact, detection, response start, mitigation, resolution. Compute time to detect, time to mitigate and total duration from them. 2. Quantify the impact from the notes: users or requests affected, error rates, data lost or delayed, money or SLA effects. Use the notes' numbers only. 3. Explain the contributing factors as a chain: the trigger, the conditions that let it cause harm, and why detection or mitigation took as long as it did. There is usually more than one factor; list each. 4. Note what went well, what was hard, and where the team got lucky. 5. Propose action items, at most seven, each tied to a contributing factor and typed as prevent, detect or mitigate. Each must be specific enough that someone could tell when it is done. 6. For a public audience, drop internal names, hostnames, tools and people. Keep the impact, the cause in plain words, and the commitments. - Never invent a timestamp, number or event. Write `[unknown]` and add the gap to Open questions. - Blameless language: describe actions, decisions and system conditions, not people's character or competence. Refer to people by role ("the on-call engineer"), never by name. - Do not name a single root cause when the notes show several factors. - No vague action items such as "be more careful" or "improve monitoring". Name the alert, test, limit or process change. - Separate what you verified from what you inferred. Mark inferences as such. - When you do not know, say "I don't know" once and state what would settle it. ## Summary Three sentences: what happened, the impact, and how it was resolved. ## Impact Bullets with numbers, duration, and who was affected. Then time to detect, time to mitigate and total duration. ## Timeline | Time (UTC) | Event | Key moments in bold. ## Contributing factors Numbered, starting with the trigger. ## What went well Bullets. ## What was hard Bullets, including where the team got lucky. ## Action items | # | Action | Type (prevent / detect / mitigate) | Factor | Priority | Owner | Leave Owner as `TBD`. ## Open questions Gaps in the notes that the team should fill in. "None" if empty. ```` --- ## Write an incident response plan `write-incident-response-plan` · prompt · Incident and operations · https://hermes-ide.com/prompts/write-incident-response-plan Writes an engineering incident response plan - severity levels with criteria, response targets, roles, escalation paths and copy-paste communication templates - as a quick reference for on-call. ````markdown This plan covers production incidents in software services: outages, degradations and data problems. It is not a security breach playbook (use a dedicated incident response playbook for attacks) and not a customer support escalation process. People read it under stress at 3 a.m., so it must be short, unambiguous and usable without interpretation: anyone should be able to declare an incident, pick a severity in under a minute and know who does what. Organisation: [ORGANIZATION] 1. Define four severity levels (SEV1 to SEV4, or P0 to P3 if the team already uses that) with criteria based on customer impact, scope and data risk, one concrete example each for this organisation, and response targets: time to acknowledge, time to assemble responders, update cadence. 2. Define roles: incident commander, technical lead, communications lead and scribe; what each does and does not do, and how roles combine on a small team. 3. Define escalation: who is paged for each severity, when and how to escalate to more people, management, other teams or vendors, and what to do when the on-call does not respond. 4. Write the response flow from detection to resolution: declare, assess severity, open the channel, mitigate first, communicate, resolve, hand off. Draw it as a Mermaid flowchart. 5. Write communication templates ready to copy: incident declared (internal), status update (internal), customer status page update for investigating, identified, monitoring and resolved, and an executive summary. Use [BRACKETS] for the facts to fill in. 6. Say how the plan is kept alive: postmortem triggers per severity, drills, and who reviews the plan and when. - Keep it a quick reference: tables and short sentences, no essays. - Severity is decided by impact, not by cause or by how hard the fix is; say so in the plan. - Anyone on the team may declare an incident and raise severity; lowering it needs the incident commander. - Do not invent the organisation's tools, contracts or uptime commitments; use [BRACKETS] where they are missing. - Customer templates say what users experience and what to do, never internal blame or speculation about cause. - Separate what you verified from what you inferred. Mark inferences as such. - When you do not know, say "I don't know" once and state what would settle it. ## Severity levels A table: level, criteria, example, acknowledge, assemble, update cadence. ## Roles A table: role, responsibilities, not responsible for. ## Escalation Who to page per severity and the escalation steps. ## Response flow The Mermaid flowchart and numbered steps. ## Communication templates Each template in its own block. ## Review and upkeep Postmortem triggers, drills, owner and review date. ```` --- ## Write an incident status update `write-incident-update` · prompt · Incident and operations · https://hermes-ide.com/prompts/write-incident-update Writes a clear status update for an ongoing incident, tuned to customers, internal teams or executives, without speculation or promises the team cannot keep. Use for status pages, Slack and email. ````markdown During an incident, people judge the team by its updates as much as by the fix. Good updates are early, specific about who is affected, honest about what is not yet known, and regular. Bad ones guess at causes, blame a vendor, promise times the team cannot meet, hide behind jargon or go silent for an hour, and each of those costs trust that is hard to win back. Updates are written under time pressure, so draft immediately instead of asking questions. Write a investigating update for customers from these facts: [FACTS] 1. Lead with the impact in the reader's terms: what they cannot do, since when (UTC), and who is affected. Say what still works when the facts show it. 2. Say what the team is doing now, matching the phase: investigating (looking into it), identified (cause found, fix under way; describe the cause only in general terms and only if the facts confirm it), monitoring (fix applied, watching, what users may still see), resolved (back to normal, the duration with start and end times, anything users need to do, and a pointer to a follow-up review if one is planned). 3. Include a workaround only if the facts contain one. 4. End with when the next update will come. If no time was given and the phase is not resolved, use 30 minutes after the current time for investigating and identified, and 60 minutes for monitoring; if the current time is not in the facts either, add `[next update time]` for the author to fill in. 5. If a must-have fact is missing (what is affected, or since when), still write the draft, insert `[CONFIRM: what is needed]` at that spot, and list it under Held back. 6. Tune it to the audience: - customers: plain language, no internal system names, at most 120 words. - internal: the affected services, the incident channel or commander if given, the customer impact in numbers if known, what other teams should and should not do, and a suggested line for support to give customers, at most 150 words. - executives: business impact first (customers, revenue, SLA, regulatory exposure if the facts mention it), the decision or support needed from them if any, at most 100 words. - Use only the facts given. Never guess a cause, a number of affected users or a resolution time. - Do not blame a vendor, a team or a person. - Do not promise a fix time unless the facts contain one the team has committed to. - Do not apologise more than once, and do not use filler such as "we take this very seriously". - Times in UTC unless the facts use another timezone. No emoji, no exclamation marks, no marketing language. ## Title One line, for a status page or subject line, stating the affected feature and the phase. ## Update The message, ready to paste. ## Short version Under 280 characters, for an in-app banner or social post. ## Held back Bullets: facts from the input you left out for this audience and why, plus every `[CONFIRM]` or other placeholder the author must fill before posting. "Nothing" if empty. ```` --- ## Write an on-call handoff `write-on-call-handoff` · prompt · Incident and operations · https://hermes-ide.com/prompts/write-on-call-handoff Writes the end-of-shift on-call handoff covering open incidents, alerts that fired and why, silences and their expiry, risky changes in flight and what to watch. Use at every rotation change. ````markdown You turn an outgoing on-call engineer's messy notes into a handoff the next person can act on in five minutes. Handoffs fail in three predictable ways: a silence or manual override expires mid-shift and nobody knows why it existed; an incident is "mostly fixed" with no owner or next step; and alert noise is mentioned but never turns into a ticket, so the same pages wake the next person. A good handoff leads with what needs action, gives every open item an owner and a next check time, and states each silence with its expiry and the condition for removing it. [SHIFT_NOTES] 1. Extract every item and sort it: open incident, alert that fired, silence or manual override (paused job, scaled replica count, feature flag flipped, failover), change in flight (deploy, migration, config rollout, vendor maintenance), customer escalation, or toil. 2. For each open incident: severity, current state (investigating, mitigated, monitoring, resolved pending follow-up), what is known, what is not, who owns it now, the next action and when to check again. 3. For each alert that fired: count, times, whether it was actionable, what was done, and a classification: real issue, known noise, flapping, or unexplained. Unexplained ones go on the watch list. 4. For each silence or override: what it hides, when it expires, who set it, and the condition that makes it safe to remove. Flag any with no expiry, or an expiry inside the next shift. 5. For changes in flight: what is rolling out, current stage, how to tell it is going wrong, and the rollback. 6. Write the watch list: at most five things the next person should actively check, each with a signal and a threshold ("if checkout p99 goes above 800 ms again, page payments"). 7. Turn repeated noise and manual work into follow-up tickets with a one-line title and owner placeholder. 8. Write one status line at the top: calm, degraded or incident in progress, plus the single most important thing. - Use only what is in the notes. Never invent times, ticket numbers, owners or causes; write [owner?], [time?] or [ticket?] and list the gap. - Keep the whole handoff readable in five minutes: bullets, no narrative of the shift. Write "None" under any empty section; never pad a quiet shift. - If the notes give nothing to hand over (no pages, open items, silences or changes, and no statement that the shift was quiet), do not fill the template: ask for the shift's pages and alerts, open incidents, silences and overrides, and changes in flight, and stop. - Use one time zone throughout and say which; if the notes mix zones, convert and say so. - Do not soften an unresolved issue into "resolved". If the notes say it stopped on its own, write "stopped, cause unknown". - Leave out secrets, tokens, customer personal data and internal hostnames that are not needed to act. - Separate what you verified from what you inferred. Mark inferences as such. - When you do not know, say "I don't know" once and state what would settle it. ## Status line One line. ## Needs action now Numbered, or "Nothing". ## Open incidents Table: incident | severity | state | owner | next action | check again at. ## Alerts this shift Table: alert | times fired | actionable? | classification | what was done. ## Silences and overrides Table: what | hides | expires | set by | safe to remove when. Flag missing expiries. ## Changes in flight Bullets: change, stage, warning signs, rollback. ## Watch list Up to five bullets, each with signal, threshold and action. ## Toil and follow-ups Bullets: ticket title, owner placeholder. ## Gaps in these notes Bullets of what the next person should ask before the outgoing engineer logs off. ```` --- ## Write an operational runbook `write-runbook` · prompt · Incident and operations · https://hermes-ide.com/prompts/write-runbook Writes a runbook for an alert or routine procedure with symptoms, diagnosis commands, ordered mitigations, verification and escalation. Use so on-call engineers can act without tribal knowledge. ````markdown A runbook is read by a tired engineer who may never have touched this system, often in the middle of the night. It must get them from "an alert fired" to "impact reduced" with commands they can paste, and it must tell them when to stop and call someone. Runbooks fail when they explain architecture at length, give commands with no expected output, or put a risky fix before a safe one. Write a runbook for: [ALERT_OR_PROCEDURE] 1. Decide which kind this is. For an alert, write the alert flow below. For a routine procedure, replace Triage, Diagnosis and Mitigations with Preconditions, Steps (each with a checkpoint) and Rollback. 2. Summary: what the alert means in user terms, likely user impact, severity guidance, and the most common known causes if given. 3. Triage (first 5 minutes): how to confirm the alert is real, how to size the impact, and whether to escalate immediately. 4. Diagnosis: read-only checks in order of likelihood. Each check gives the command or query, what a healthy result looks like, and what an unhealthy result means and which mitigation it points to. 5. Mitigations: ordered from safest and most reversible to riskiest. Each states when to use it, the exact steps, the risk, and how to undo it. 6. Verification: the signals that prove the mitigation worked and how long to watch them. 7. Escalation: when to escalate, to whom (role or team), and what information to hand over. - Never invent hostnames, dashboard links, metric names, namespaces or team names. Use placeholders in angle brackets such as `` and list every one under "Fill before publishing". - Put every command in a fenced block. Mark any command that changes state with "CHANGES STATE" and any that can lose data or drop traffic with "DESTRUCTIVE", and require a check before running it. - Keep it scannable: numbered steps, short sentences, no history lessons. - Do only what was asked. If you notice something else worth changing, mention it in one line at the end instead of changing it. - Keep the change as small as it can be while still being correct. For an alert, use these headings in this order: ## Summary ## Triage ## Diagnosis ## Mitigations ## Verification ## Escalation ## Fill before publishing A checklist of every placeholder and unconfirmed assumption. For a routine procedure, use: `## Summary`, `## Preconditions`, `## Steps` (numbered, each ending with a checkpoint that says what you should see before continuing), `## Rollback`, `## Verification`, `## Escalation`, `## Fill before publishing`. ```` --- ## Write observability queries `write-observability-queries` · prompt · Incident and operations · https://hermes-ide.com/prompts/write-observability-queries Writes PromQL, LogQL, TraceQL, SQL or vendor queries for an operational question, explains each part and warns about traps such as rate windows, counter resets and label cardinality. ````markdown You write a query that answers one operational question correctly, and explain it so the reader can change it safely. Most wrong graphs come from a few traps: graphing a raw counter instead of its rate; a rate window shorter than about four scrape intervals, which gives gaps and spikes; averaging percentiles across instances instead of aggregating histogram buckets first; dropping the `le` label before `histogram_quantile`; dividing series whose labels do not match; ratios over tiny request counts; log queries that parse every line before filtering; and grouping by a high-cardinality label (user id, request id, full URL) that explodes cost. Query language: [QUERY_LANGUAGE] Used for: dashboard [QUESTION] 1. Restate the question as a precise measure: numerator and denominator for ratios, the percentile and population for latency, the time window, and the grouping. 2. Write the query using only the names and labels given. For PromQL: `rate` or `increase` on counters, `sum by (...)` before dividing, `histogram_quantile` over `sum by (le, ...) (rate(..._bucket[w]))`. For LogQL: stream selector and line filters before parsers, then `| json` or `| logfmt`, then metric functions. For TraceQL: span conditions with scoped attributes and the aggregate. For SQL: time bucketing, filters on indexed time columns first, and explicit handling of nulls. 3. Pick the window for the use: dashboard windows match the step (for example `$__rate_interval`); alert windows match the alert's intent; ad-hoc can be wider. Say why. 4. Explain each part in one line, top to bottom. 5. List the traps that apply to this query and how the query avoids them, plus any it cannot avoid (counter resets are handled by `rate` but not by subtracting raw values; low traffic makes ratios jumpy, so add a minimum-count guard). 6. Give one or two useful variants (another grouping, top-k, comparison to one week earlier with `offset`). - Never invent metric, label, stream or attribute names. If one is needed and missing, write it as `` and list it under Assumptions. - Use syntax valid for the named language; if a function depends on the version or vendor, say so. - Avoid grouping by unbounded labels; if the question needs it, use top-k and explain the cost. - If the question is ambiguous (which errors count, which percentile), state the choice made and how to change it. ## Query One fenced code block. ## How it works Numbered lines, one per part. ## Traps Bullets: trap and how it is handled. ## Variants One or two fenced blocks with a one-line purpose each. ## Assumptions Bullets, or "None". ````