Camera and stream health

KeepPeek computes camera and stream health on the server. Peek, Cameras, Health, camera diagnosis, fleet counts, the control API, and Prometheus metrics consume that same projection. Clients do not infer health from display text, local video playback, or issue severity.

Investigate an unhealthy camera

1. Open Cameras and use Not healthy to narrow the fleet. Select the camera for its current state, primary reason, stream profiles and recording evidence.
2. Open Health for fleet findings. Refresh to request a current snapshot, then follow a finding’s camera or timeline link to the affected source and time.
3. Compare transport, advancing frames, keyframes and recording progress. A connected socket can still deliver stale or undecodable video; a decoded live picture does not prove recording.
4. Use Recording integrity when the question is whether footage was retained. Its historical coverage and gaps answer a different question from current-process health counters.
5. Check native event and external service status separately when only detections or alerts are missing. Their failures do not make otherwise healthy camera media offline.

An active health finding clears when new evidence shows recovery. It has no dismiss action that changes the underlying state. Acknowledging an alert or marking an event reviewed is a separate workflow. See Notifications and integrations and Recording and evidence.

Health and administrative diagnostics depend on your role. If a User can view video but cannot load fleet health, check access permissions before treating the unavailable diagnostic panel as a camera failure.

The current health contract version is 1. A client that cannot understand a state or reason must treat it as unknown, never as healthy.

Evidence dimensions

Health keeps these dimensions separate:

Native events, external detectors, and other optional services have their own health. Their failure does not make otherwise healthy media or recording appear offline.

Canonical states

healthy is the only healthy state. starting, unknown, and stopped are not included in healthy counts. A health request failure makes the current view unavailable; it does not rewrite the last server camera state.

Precedence

The server applies this order. The first matching rule supplies the canonical state and primary reason code.

1. Not expected, shutting down, or stopped: stopped.
2. Fresh battery registration without a connected transport or pending wake request: stopped with battery_sleeping. An accepted wake remains pending until connected media clears it; failures therefore become reconnecting or offline, not sleeping.
3. During initial grace, incomplete or transiently unhealthy evidence remains starting; a complete healthy snapshot may promote immediately.
4. Reconnecting transport: reconnecting, or offline after recent evidence expires.
5. No report, an old report, or frames that stopped advancing: stale.
6. Missing keyframes, stalled requested recording, low frame rate, partial transport, recent errors, drops, or reconnects: degraded. The primary degraded reason is ordered as partial transport, missing keyframes, recording failure, low frame rate, errors, drops, then reconnects; all matching codes remain in reason_codes.
7. Required evidence unavailable: unknown.
8. All required evidence current: healthy.

A connected transport cannot override stale frames, missing keyframes, or failed recording. A decoded browser frame cannot promote server state. A fresh capability snapshot is not media evidence.

Thresholds

Ingress publishes every 10 seconds.

• A stream report is stale after 30 seconds.
• Reconnecting becomes offline when the latest stream evidence is older than 90 seconds.
• Frame freshness is $\max(20\text{ s}, 3 / \text{expected fps})$, capped at 120 seconds, and also requires a fresh stream report.
• Keyframe freshness is three configured GOP intervals, or three observed keyframe intervals when configuration is unavailable. It is clamped to 30 through 120 seconds.
• Recording progress is fresh for the configured short-term buffer plus flush interval plus 30 seconds.
• Initial startup grace is two ingress report intervals, currently 20 seconds.

Frame-arrival jitter measures the absolute difference between each observed frame gap and the configured frame interval. Its P99 is computed over the current 10-second ingress report window; it is not WebRTC network jitter. KeepPeek displays it as diagnostic evidence but does not create a finding from jitter alone. Stale frames, low sustained frame rate, decode failure, ingress drops/errors, recording failure, and a maximum observed frame gap above 2 seconds remain actionable health signals.

The first ingress report establishes the baseline for cumulative reconnect, drop, and error counters. An initial connection is therefore not reported as a recent reconnect. During startup grace, partial-window frame rates and other transient evidence remain starting; if they have not recovered when grace expires, the canonical degraded or offline reason becomes visible.

These thresholds are the bounded debounce window. One missed 10-second ingress report does not change a camera to stale; report evidence must age beyond 30 seconds. Recovery is intentionally asymmetric: a new advancing frame or keyframe counter can recover the state immediately. The API still returns the actual evidence ages, and clients do not add another timer or rewrite the state.

These ages use monotonic clocks inside the server. Wall-clock timestamps are included for display, but wall-clock changes do not make stale evidence fresh.

Health findings are live snapshot evidence rather than acknowledgements or retained alerts. They disappear automatically after a later ingress report no longer meets the condition. The refresh button requests a current snapshot immediately, but an active finding cannot be dismissed from the Health page. Acknowledging a delivered alert belongs to the notification inbox rather than changing server health evidence.

Operational intervals

KeepPeek projects four durable event kinds from the independent health dimensions:

These event kinds remain independent. One stream can be stale and undecodable while another stays healthy, and recording failure does not rewrite transport evidence. Native event and external analysis failures are not camera, stream, or recording outages.

A candidate interval starts as soon as evidence fails. Monotonic elapsed time controls warning, critical, and recovery thresholds; server wall time supplies the displayed start and end timestamps. The default policy is:

[operational_events]
warning_hold_down_secs = 15
outage_hold_down_secs = 60
recovery_debounce_secs = 10
record_short_flaps = false

[operational_events.cameras.front-door]
warning_hold_down_secs = 30

Camera overrides may use the stable camera ID or IP address and inherit omitted global values. A zero duration requests an immediate transition. Durations cannot exceed 24 hours, and the warning hold-down cannot exceed the outage hold-down.

Recovery before the warning hold-down removes the candidate unless record_short_flaps is true. Once visible, an interval keeps one event ID. Cause or severity changes increment its revision; stable recovery increments the same event again, records the recovery timestamp, and reports total duration from the original evidence-loss instant. Open intervals are restored from the recording catalog after restart. Stable revision replay makes notification processing at least once while the notification store collapses duplicate revisions.

The stored timeline renders open and recovered intervals beside recording availability. Event payloads include severity, bounded cause and explanation, affected streams, recording consequence, evidence source, duration, and recovery state. Health findings link to the corresponding camera and timeline instant.

Session durations

Camera health reports current-process session durations separately from historical archive coverage:

• session_duration_ms is the longest observed ingress-worker duration for the camera’s expected video streams;
• recorded_main_duration_ms and recorded_sub_duration_ms accumulate video sample durations only after those samples are successfully appended to MP4 during the current process run;
• recorded_total_duration_ms is the sum of all requested stream durations for that camera session.

The total can exceed camera session elapsed time when Main and Sub record concurrently. These counters reset when KeepPeek restarts or the corresponding recording-health registry is replaced; they do not claim lifetime or retained archive coverage. A recording indicator uses writer progress state, not a comparison between these counters and wall time.

Reasons and detail

Every camera and stream includes:

• state: the canonical state;
• reason: the primary stable reason code;
• reason_codes: all applicable ordered reason codes;
• detail: bounded human-readable text;
• dimensions: the independent raw evidence and counts.

Stable reasons include transport_disconnected, transport_reconnecting, transport_partially_connected, no_stream_report, stream_report_stale, frames_not_arriving, frames_below_expected, keyframes_missing, ingress_reconnects, ingress_drops, ingress_errors, and recording_not_progressing.

Use reason codes for automation. Display text may change.

Fleet counts

Fleet summaries name the dimension they count:

• configured cameras and expected video streams;
• cameras and streams with connected transports;
• cameras and streams with fresh frames;
• cameras and streams with recent keyframes;
• cameras and streams requested for recording;
• cameras and streams with current writer progress;
• cameras whose evidence is unknown.

Fleet aggregates count expected video streams. Audio freshness remains available on each stream snapshot and is not folded into the video decodability or recording totals.

Unknown evidence is not counted as connected, fresh, decodable, recording, or healthy. The Cameras Not healthy filter counts every state except healthy.

API and metrics

HealthCommand returns the versioned protobuf health snapshot. Version 1 includes the canonical state, stable reasons, camera and stream dimensions, media progress timestamps, and explicit fleet counts. Unknown future state or reason strings are presented as unknown.

GET /metrics exposes the same projection. Important gauges include:

• keeppeek_cameras_connected, keeppeek_cameras_fresh, keeppeek_cameras_decodable, and keeppeek_cameras_recording;
• corresponding keeppeek_video_streams_* gauges;
• keeppeek_camera_info with the canonical state label;
• keeppeek_camera_health_dimension and keeppeek_camera_health_dimension_known;
• corresponding per-stream dimension and known gauges;
• keeppeek_operational_event_active, labeled by camera, stream, event kind, and severity.

There is no separate online or degraded gauge. Consumers use the canonical state label and explicit dimensions.

Recovery

Recovery requires new positive evidence:

• a newly advancing frame counter refreshes frame age;
• a newly advancing keyframe counter refreshes decodability;
• a successful writer append clears the previous bounded recording error;
• drop, error, and reconnect reasons use per-report deltas, so old cumulative counters do not keep a recovered stream degraded;
• a fresh report can recover stale state, while cached metadata cannot.

Operational interval recovery uses this canonical projection. Delivery failure cannot change or close an interval because catalog persistence happens before notification publication and provider work runs independently from camera media.

Use Recording and evidence to inspect retained coverage and explain gaps. Use Notifications and integrations to deliver these operational transitions without coupling providers or brokers to camera ingest.