Runbook reference

Step types

Nested precheck and verification definitions have first-class typed Builder controls and remain compatible with Advanced JSON.

Step configuration schemas

MANUAL / APPROVAL

{ "instructions": "...", "description": "..." }

CONDITION

{ "field": "incident.priority", "operator": "EQUALS", "value": "P1" }

Supported condition fields

Builder fields: incident.priority, incident.urgency, incident.status, incident.title, incident.description, incident.tags, service.name, input.<key>.

Advanced JSON fields: Also supports incident/service IDs and service.teamId.

Incident/service values are read when the condition executes, not frozen at execution creation. Without an incident/service, those objects are absent. Inputs are the execution's resolved values. Legacy bare incident paths such as priority are interpreted as incident.priority.

Condition behavior

• A false CONDITION skips itself and all remaining pending workflow steps.
• It is a workflow gate, not an if/else branch.
• Operators use strict type comparison: numeric inputs require JSON numbers.
• The server rejects unsupported fields and operators when saving, publishing, or starting a workflow.
• input.<key> must name a declared Runbook input.
• Trigger conditions allow only incident/service fields, never inputs.
• Missing fields never satisfy value comparisons, including negative ones.

Condition operators

WAIT

{ "durationSeconds": 30 }

Range: 0–86400. Subject to step timeout and overall deadline. Fields named seconds are not WAIT duration controls.

HTTP

{
  "method": "GET",
  "url": "https://status.example.com/health",
  "headers": { "Accept": "application/json" },
  "body": "",
  "expectedStatus": 200
}

Risk mapping: GET/HEAD → READ_ONLY, PUT/DELETE → IDEMPOTENT_WRITE, POST/PATCH → NON_IDEMPOTENT.

Control-plane HTTP rejects private, loopback, link-local, and metadata endpoints. HTTP steps cannot consume SECRET_REF inputs.

LINUX_DIAGNOSTICS

{ "diagnostic": "<sub-type>", ...sub-type fields }

Agent-local HTTP checks require both hostname and port in networkHosts / networkPorts. Redirects and URL credentials are rejected.

SYSTEMD

{
  "action": "restart",
  "unit": "api.service",
  "lines": 100,
  "expectedState": "active"
}

Unit names support .service, .socket, .timer, .target, and other systemd unit types.

DOCKER

{
  "action": "inspect",
  "container": "checkout",
  "runtime": "docker"
}

Runtime: "docker" (default) or "podman". Podman uses independent podmanContainers allowlist.

KUBERNETES

{
  "action": "scale",
  "namespace": "payments",
  "resource": "deployment",
  "name": "checkout",
  "replicas": 3
}

Scale requires integer replicas and local kubernetesMaxReplicas authorization. Platform RBAC is an independent boundary.

BASH

{ "command": "systemctl is-active -- \"$OPSKNIGHT_INPUT_SERVICE\"" }

Max 8 KiB. Always NON_IDEMPOTENT. Uses Bash (not POSIX sh), non-login, non-interactive, without profile/rc or BASH_ENV startup files.

Input types

Maximum: 30 input definitions per workflow.

Input template syntax

${{ inputs.key }} — keys must be lowercase. Exact match preserves typed value; embedded in a string coerces to string.

Risk classes

Minimum risk is derived from the action. Non-idempotent actions require explicit approval. Authors cannot make a restart safe by relabeling its risk.

State machines

Execution states

QUEUED → RUNNING → WAITING_AGENT → WAITING_APPROVAL
                                        │
         ┌──────────────────────────────┘
         ▼
  SUCCEEDED │ FAILED │ TIMED_OUT │ CANCELLED │ CANCEL_REQUESTED

Step states

PENDING → READY → RUNNING → WAITING_AGENT → WAITING_APPROVAL
                                                  │
         ┌────────────────────────────────────────┘
         ▼
  SUCCEEDED │ FAILED │ SKIPPED │ CANCELLED │ UNKNOWN

Attempt states

PENDING → CLAIMED → RUNNING → SUCCEEDED │ FAILED │ CANCELLED │ TIMED_OUT │ UNKNOWN

Approval mechanics

• Steps with requiresApproval: true or type APPROVAL pause at WAITING_APPROVAL.
• Approval requires runbook.approve capability.
• Plan digest computed from: step key + step type + risk class + config + agent/pool ID + input values + version checksum.
• Approver signs the exact same digest — any change requires re-approval.
• Circuit breaker exhaustion also transitions to WAITING_APPROVAL.

Preview authorization

• Issued for 5 minutes, single-use.
• Bound to account, incident, service binding, and suggestion.
• Cannot be reused by another responder or incident.
• Issuance/consumption recorded in audit trail (no token stored).
• Temporary records purged 30 days after expiry.

Timeouts and leases

Deadline handling

Once the deadline passes, pending/claimed attempts lose authority. A running Agent attempt stays fenced while its result or lease expiry is reconciled. A result produced before deadline may still be recorded. A write reported after deadline is recorded as UNKNOWN, not successful or discarded.

Evidence and verified recovery

Attempts persist bounded structured preState and postState snapshots including CPU load/cores, memory, load, and relevant service/container/ Kubernetes/network/disk/process/log evidence.

Verified recovery requires:

• ✅ Successful workflow and write action.
• ✅ Successful authored verification checks.
• ✅ Positive healthy post-state.

What does not qualify:

• ❌ Failed capture or unhealthy containers/services/checks.
• ❌ Unconverged Kubernetes generations.
• ❌ Metrics or exit code zero alone.
• ❌ Scale-to-zero.

Targeting

Values can reference ${{ inputs.host }} or ${{ incident.labels.host }}. The selected identity and selector/pool provenance are snapshotted; queued writes never switch hosts when labels change.

Permissions

See Permissions guide for API scopes.

Limits

Signing and effective capabilities

Signing identities rotate through ACTIVE → RETIRING → RETIRED with NEXT staged alongside ACTIVE. Distribute OPSKNIGHT_EXECUTION_PUBLIC_KEYS before activating NEXT. Every enrolled non-revoked Agent must acknowledge its ID. Private keys remain encrypted; the UI exposes only public pins.

Agents probe configured binaries and runtime access before heartbeats. Scheduling uses effective capabilities and distinguishes Docker from Podman.

Rule and readiness contracts

ServiceAutomationRule is an independent configuration identity. Readiness states: PAUSED, BLOCKED, CHECKS_REQUIRED, CONFIGURATION_VALID, UNKNOWN.

Worker health states: HEALTHY, DEGRADED, UNKNOWN. Shared PostgreSQL evidence establishes dispatch and reconciliation freshness.

Bounded nonce maintenance

Replay nonces are purged using an isolated one-connection maintenance pool. Each batch selects at most 1,000 expired IDs through the createdAt index using FOR UPDATE SKIP LOCKED, its own advisory lock, a one-second statement deadline, 100 ms lock deadline, and 2.5-second transaction deadline.

Maintenance alerts

The bundled Prometheus rules include alerts for stalled, lagging, and failing maintenance for both nonces and artifacts:

Enterprise safety

Deadline handling is shared by worker advancement and recovery. UNKNOWN writes require independent target verification. Agent output passes through shared credential sanitization. Docker inspect/health and Kubernetes diagnostics project operational state rather than dumping payloads.

Idle Agent claim behavior

A process-local negative queue hint coalesces concurrent availability probes and expires after one second. Every actual claim checks live Agent status, policy, permissions, and lease fences. Newly available work can incur the one-second hint delay plus the 2–5 second polling cadence.

Current boundaries

Runbooks remains Beta in v2.1. Configuration checks, policy evidence, browser journeys, and candidate deployment tests each establish a limited property. Release qualification requires the published server/Agent pair, physical write and recovery evidence, HA/fault scenarios, sustained workload measurements, and human accessibility review. Follow Agent operations for deployment prerequisites.