Fourteen subsystems, one contract.
percept-vision is a cognition layer for goal-driven, proactive vision agents (CONTRACT_VERSION = "1.10.0"). Its wedge is temporal cognition — entity memory, a three-state consequence gate, and reasoning over events — which a raw VLM-in-a-loop and a per-frame pipeline both lack.
These are distillations, not the full reference — each entry compresses one subsystem to its governing rule and public seams. The canonical guides live in the documentation.
A frozen vocabulary every layer speaks, versioned like an API.
Contracts is the frozen membrane of the SDK: a single, zero-dependency, stdlib-only source of truth for every vendor-neutral noun (the DTOs), every seam a backend implements (the Protocol interfaces), and the safety and identity rules that must survive being wired at 2am. It sits at the bottom of the layering — imported by the cognition core, never the reverse — and is immutable within a phase: a change to a frozen shape is a MAJOR bump of CONTRACT_VERSION, never a silent edit. The cardinal rule it protects is the firing contract: the Vision seam judges a goal's plain-language condition and returns a Verdict — it never compiles the goal to a rigid attr=value predicate.
Sample → route/cache → judge → gate → deliver, with one gate.
The cognition core. Percept is the orchestrator that ties every subsystem together: it samples a frame, resolves it cheaply where it can (routing, cache, a DirectJudge), escalates to a Judge only when it must, classifies the verdict, and drives the three-state consequence Gate to a delivered action. The governing invariant is one firing path, one gate: every fire — judged, transition, episode, absence, rhythm, boundary, coverage — reaches the user only by passing a Verdict through classify and then through Gate. Construction is safety-checked: a live run with no real vision backend, or a real backend missing its credential, raises a loud ModeError naming the fix.
Plain language, carried verbatim — never compiled to predicates.
The declarative plan layer: what to watch, for which goals, and how a plan travels from authoring to a live, armed goal. A caller states intent three ways — a bare plain-language string, a hand-built Goal, or a typed, portable WatchPlan — and this module owns the frozen watch() entrypoint, string desugaring, compilation, and a two-phase atomic apply that returns a machine-actionable WatchReceipt. The cardinal rule: a plain-language condition is carried verbatim and never compiled to a rigid predicate — the judge downstream sees the caller's exact words. Goal ids are stable content hashes, a requested judge fails closed if unavailable, and domain packs ship templates that never auto-arm.
Authorize, bind, dispatch — fail-closed at every step.
A judge is the model-as-evaluator step the SDK escalates to when a goal cannot be resolved cheaply on the local path — and a judge is never taken on trust. A WatchPlan.judge value is only a request: before any watch arms, a strictly ordered chain authorizes the ref against the deployment-owned catalog, binds its immutable static context (verified byte-for-byte against a pinned SHA-256), and only then constructs the adapter that will be dispatched. Every stage is fail-closed: an unknown ref, uninstalled factory, hash mismatch, oversize blob, or bad decode refuses the arm outright — there is never a silent fallback to the generalist vision model.
Providers are identity, models are configuration.
The concrete vendor adapters behind the vendor-neutral Contracts Protocols. Every real adapter is opt-in: it needs a key and a vendor SDK, and it lazy-imports that SDK inside its methods so the package imports cleanly with no key and nothing extra installed. The deterministic fakes need neither and power the offline lane. The organizing rule is the identity/configuration split: an adapter's identity is its provider name (claude, gemini, openrouter, scripted); the model id is configuration carried to the adapter, never its identity. Each outbound call is wrapped in retry + backoff + a circuit breaker, so a vendor outage degrades instead of hanging the hot path.
A 24/7 watch loop in one line, with safe delivery built in.
The batteries-included front door: percept.run(...), percept.start(...), and RuntimeSession own the watch-a-source-to-incidents lifecycle, so applications stop writing capture, threading, and feed glue. The runtime opens a video/audio source on background capture threads, drives the cognition core over frames, and delivers resulting incidents to notification sinks — idempotent (deduped by incident id), supersession-aware, dead-lettered, SSRF-guarded. Per frame it runs the cheap tier-0 gate; only on a rising edge does it spend a cloud judgment, turning each fire into a durable Incident. Worker code is pure stdlib; cv2/numpy/sounddevice arrive lazily via extras. RuntimeSession adds the interactive inputs: ask/barge-in, arm/disarm, pause/mute, labels.
Defer-on-miss is structural: when in doubt, the model decides.
The local fast-path: a confidence-gated, layered engine that resolves the common, confident spoken turn cheaply and instantly, before any model judge is invoked. It walks an ordered list of RouteLayer tiers, takes the first match at or above a threshold, and turns it into a Decision; if nothing clears the bar it returns None — a structural DEFER to the conductor. The current tier is a regex lexicon, but RouteLayer is the real contract: a future learned tier slots in with zero engine changes. A veto lexicon guarantees goal-authoring and ambiguous turns always defer, even when a fast-path rule matches.
Never pay a model twice for the same unchanged scene.
The frame-hash cascade cache that reuses a prior judgment instead of re-calling a vision model when a near-identical frame recurs for the same goal. It sits on the cost-avoidance fast path alongside Routing, before the Judge: when a goal's region of interest hasn't visibly changed within a bounded window, the cache serves the last verdict at a decayed confidence. It ships a contract, not a policy: an empty cascade always DEFERs (the safe default); the only short-circuit is a confident, fresh-enough near-match. TTL is a hard ceiling; a goal re-armed under new plain-language wording is a clean miss, never served a stale verdict.
Identity is established below the LLM.
The cheap perceptual layer between raw frames and the model judge. Before a goal's condition is handed to a backend, this layer extracts structured evidence (fields, OCR text), measures per-channel signals over physical time, individuates repeated observations into single events, and keeps subject bookkeeping — a continuity/appearance identity and an entity graph — so identity is established below the LLM and handed up as a stable symbol. Everything is pure stdlib and rate-invariant by construction (seconds and amplitudes, never frame indices). The output makes a judge's answer cheaper and more honest: typed enrichment facts, a stable entity id, sampling-invariant event counts, and coverage signals that say when a count is blind and must escalate.
Episodes remember what stayed true; evidence fails loudly.
The SDK's memory layer: the seam between what a single frame said and what has been true over time. It groups repeated observations into bounded episodes, retains the pixels those episodes were gated on behind opaque evidence handles with a retention/eviction policy, turns fires into durable incidents with a delivery-attention clamp, keeps a local feedback rail of labels and dispositions, gates the higher-trust act modality behind consent, and persists durable state across restarts. Everything is deliberately bounded — a 24/7 watch must never leak memory — and every drop, denial, and dereference failure is typed and observable rather than silent. Dereference never returns a silent empty: an artifact, or exactly one of four typed failures.
Every threshold answers: where did this value come from?
The configuration spine: a single env-driven Settings tree (every knob a PERCEPT_* variable), versioned name-keyed profile registries, and a layered ConfigResolver that merges six independently-authored configuration objects into one immutable resolution with per-field provenance. Every behavior-affecting knob is enumerated once in a typed tunables registry whose ownership table decides which layer may set which field — ownership gates precedence, and an out-of-ownership write is a loud ConfigError, never last-writer-wins. Profiles are content-addressed and diffable, so a tuning change is a reviewable, reversible artifact. ModeError makes silently-faked success unreachable: a live run with no backend or missing credential fails loud, naming the fix.
One funnel, counted once — no number is ever invented.
Where the SDK single-sources its diagnostic truth. There is exactly one definition of the funnel a watch loop moves evidence through — observed, admitted, judged, gate outcome, incident — and percept doctor, percept eval, and the visualizer all read it from the same object, so they cannot invent three disagreeing stories. A run emits TraceEvents; a TraceRecorder structures them against a RunTraceManifest into a persistable artifact that exports to a portable .ptrace bundle (media opt-in, off by default), replays deterministically without ever calling a model, and renders to a self-contained offline HTML visualizer that makes zero network requests. The through-line is no false numbers: an unpriced call renders a dash, not $0.00.
Silence is never a pass: bars are declared before numbers exist.
The offline, statistically-rigorous rail that decides whether a plan, profile, or tunable selection is good enough to ship. It never touches a live camera or spends API budget — it scores traces and labeled fixtures produced elsewhere. Its defining discipline is pre-registration: bars, held-out splits, plan hashes, and ladder order are content-addressed into a registration hash before any number is computed, and a configuration that moved after bars were declared raises rather than scores. Every verdict is PASS / FAIL / NOT_ASSESSED, and the subsystem refuses to invent a threshold, a bound, or a winner the operator did not declare — UNDERPOWERED and NO_EVIDENCE are refusals with reasons, never soft passes.
A cheap gate in front of an expensive eye.
The internal testing, benchmark, and edge-simulation toolkit — the server-side reference for the on-device edge, exposed publicly as the percept.edge.* namespace. It ships the tier-0 gated watch loop that decides when to spend a cloud vision call, cheap high-fps detectors (motion periodicity, acoustic onset, pose), synthetic stream/load/soak drivers, golden-episode and scripted-judge evaluation, seeded capability corpora, and keyless fixture-backed offline mode. Two invariants cut across everything: determinism (time enters only through an injected clock; randomness is always seeded) and the rule that the harness proves wiring and mechanism, never model intelligence. Gate arithmetic is deliberately bit-identical between this Python reference and the JS/WASM edge.