Outposts - distributed monitoring engines¶

Danbyte's monitoring (ICMP/TCP/SNMP checks, prefix sweeps, SNMP discovery) runs on the core server's workers by default. But a remote site with no direct path to the core (a branch office, an isolated colo, a DMZ) can't be reached from the centre - so it can't be pinged, swept, or polled.

An Outpost is a lightweight agent you install at that site. It monitors the local network and reports results back to the core over an outbound HTTPS connection - so it works through NAT and firewalls, and the core never has to connect into the remote network.

Engines¶

Every check runs on a monitoring engine:

• Local (built-in) - the core server's workers. Always present; anything not assigned to an Outpost runs here, so nothing changes if you never install one.
• Outpost - a remote engine. You create it in Danbyte, install the agent at the site, and assign it a scope.
• A driver - an external monitoring system that already watches the estate and can answer for it. Danbyte does not run the checks; it asks the system what it knows and folds the answer through the same path an Outpost reports through, so alerts, silences, flapping and every notification channel behave identically whichever engine produced the result. Zabbix is the first - see Zabbix. On the engines page a driver engine has no token, transport or agent version; it shows the system it answers through and a Configure link to that system's page.

Manage them in Governance → Monitoring engines (admin only).

How an engine is chosen for a target¶

For each monitored IP, Danbyte resolves the engine most-specific first, and each level inherits from the next when it has no engine of its own:

1. the IP's Device,
2. the IP's device Location, then its parent locations (a child location left on Inherit falls through to its parent),
3. the IP's Prefix,
4. the IP's Site,
5. the tenant default engine (set on the Monitoring engines page),
6. the built-in Local engine.

A level bound to an engine that cannot answer right now is treated as though it had no binding at all, so resolution continues down the list rather than stopping on a dead engine.

Assign an engine on the Device, Site or Location edit form (the Monitoring engine dropdown - Inherit follows the level above). So "everything at Site AMS-02" is one setting; a single rack row (a child location) can override it, one device can override that, and anything left on Inherit rolls up to the site, then the tenant default.

The form's Monitoring section shows Loading… until the stored engine and the engines on offer are both known, and a pick there is written by the form's Save with the rest of its changes. Opening the form never writes; Cancel drops the pick. The same goes for the SNMP credentials and discovered-IP VRF selects beside it.

Over the API, PUT /api/monitoring/engine-binding/<scope>/<id>/ takes {"engine_id": "<uuid>"} to bind and {"engine_id": null} to inherit. A body without engine_id, or with "", is refused with a 400 rather than read as Inherit. The SNMP profile and VRF binding endpoints follow the same rule for profile_id and vrf_id.

An SNMP poll resolves the same way for the polled object: a device by its own binding, its location (and parents), then its site - its own, else its location's; a virtual machine by its site, else its cluster's site. Both then fall to the tenant default and Local.

A driver-backed engine - a Zabbix one - answers only its own check kind. A device bound to Zabbix keeps its ICMP ping, and Danbyte's own workers still run it. Binding a target somewhere else changes who answers what it was asked, not what it was asked.

The agent repo + staying compatible¶

The agent lives in its own repo, danbyte-outpost (github.com/danbyte-net/danbyte-outpost), so sites can install and upgrade it independently. This monorepo stays the source of truth for danbyte_checks/ (the shared check engine); the agent repo vendors it and re-syncs (scripts/sync-checks.sh) before a release.

The rule that keeps remotes working: an Outpost and the core are on separate upgrade schedules - a branch office may run a months-old agent against a freshly-upgraded core. So:

• Add a check kind → the remote must be able to run it. Because the engine is shared, add the checker to danbyte_checks/ here, then sync + release the agent. Until a remote upgrades, that kind returns unknown there (handled in the agent's run_check), never a crash - the core can ship the kind before every remote supports it.
• The wire protocol is additive. New fields on hello/work/results are optional; unknown fields are ignored. A breaking change bumps PROTOCOL_VERSION (the agent sends it in hello) and the core must keep serving older-protocol Outposts.

Full contract: docs/COMPATIBILITY.md in the danbyte-outpost repo. Whenever you extend the monitoring engine, check that doc.

Sub-minute checks on an Outpost¶

A check with an interval under a minute (see the fast lane) bound to an Outpost is probed by that Outpost, from its own in-memory schedule, alongside its ordinary poll loop. The agent says fast: true in its hello; from then on the core hands sub-minute checks to GET /api/outpost/fast-work/ (the whole set, refreshed every 15 s - nothing is claimed, the Outpost owns them) and the agent reports buffered probes to POST /api/outpost/fast-results/ on its poll interval, or at once when a probe's reachability differs from the last one it reported. The core folds them through the same rise and fall as its own lane, so the status, the history and the alerts are identical whichever side did the probing.

Additive, protocol version unchanged: an agent older than 0.8 never says fast, and its sub-minute checks are handed out on the minute beat at their ordinary interval - it keeps working, just not faster.

Excluded addresses¶

An address excluded from monitoring is not handed to an Outpost at all: its checks are parked, so /work never finds them due (and skips the address even if a racing write re-armed one), and /fast-work leaves the address out of the set (the agent drops it on its next refresh, within 15 s). Whatever still arrives for it - a check claimed before the switch reported to /results, buffered probes to /fast-results - is discarded, and the check is parked again. No agent change is needed: the core simply stops asking. Only checks stop; an SNMP poll or a discovery sweep an Outpost runs for the core carries on.

Reverse DNS from an Outpost¶

PTR is the one lookup whose right answer depends on where you ask from. A branch-office Outpost can usually reach that branch's DNS; the core server often cannot, and on a split-horizon network it may get a confidently wrong answer rather than none. So an Outpost can resolve PTR itself and report the name back.

Turn on Resolve reverse DNS here on the Outpost, and optionally list Nameservers for it to ask. Leave that empty to use the Outpost machine's own resolver, which is usually already correct for its site. The core then skips its own lookup for anything the Outpost answered.

This needs no protocol bump - both halves are additive fields:

• work and hello gain "dns": {"resolve": true, "resolvers": ["10.20.0.5"]}.
• A result may carry "ptr" inside its existing detail object.

The lookup itself is shared code: danbyte_checks/reverse_dns.py, so the agent and the core cannot drift on what a PTR lookup means. Explicit nameservers need dnspython; with no resolvers configured it uses the standard library and adds no dependency to the agent.

LAG membership from an Outpost¶

Interface rows in an SNMP result may carry lag_if_index - the ifIndex of the aggregate the port belongs to, "" when it is not a member - and an aggregate row reports type_name: "lag". Like ptr, the core decides on whether the key is there: an agent that looked must send lag_if_index on every row (blank included); one that never looked sends no such key, and its devices simply show no membership drift. Additive - no protocol bump; re-sync danbyte_checks into the agent to pick it up.

MAC tables from an Outpost¶

An SNMP result may carry the MAC table with its VLANs and an fdb_meta object saying how the read went, an arp_meta object saying whether the ARP read finished, ARP rows with their type, and LLDP neighbours with a local_if_index and the neighbour's remote_caps. As with ptr and lag_if_index, the core decides on whether fdb_meta is there: a result without it comes from an agent that predates MAC tracking and is read the old way - no VLANs, the core filters out the switch's own and group addresses, and the read counts as complete only when it returned rows, so an empty table from an old agent never closes anything. The same goes for ARP without arp_meta. Additive - no protocol bump. Outpost 0.9.0 is the first agent that sends all of this, VLANs and per-VLAN tables included; upload it under Governance → Monitoring engines → Outpost versions and roll it out to the sites whose switches it polls.

The core accepts SNMP results only for the devices that Outpost polls - the ones its work list hands out (devices_for_engine). A result for any other device, even one in the same tenant, is ignored rather than written. Refresh MACs on a device an Outpost polls is queued for that Outpost, like Poll now; the agent reads the whole table on its next cycle.

Transports - which way traffic flows¶

Sites differ in what their firewall allows, so an Outpost's transport is set per engine. The agent and the check code are identical; only the channel changes.

• HTTPS (Outpost dials out, 443) - the Outpost opens an outbound connection to Danbyte, pulls its work, and pushes results. For NAT'd branch offices that can reach out but can't be reached in. Authenticated by the install token. (Shipped.)
• SSH (Danbyte dials in, 22) - for locked-down / airgapped sites where the only permitted flow is Danbyte → host on SSH. Danbyte connects out to the Outpost over SSH each cycle and runs danbyte-outpost once (work in on stdin, results out on stdout) - the site never opens a connection to Danbyte. Set the host / user / SSH key on the engine's detail dialog; Danbyte drives it on a timer (drive_outposts). Pin the host's public key (ssh_host_key, from ssh-keyscan -t ed25519 <host>) to verify the server - blank falls back to trust-on-first-use with a logged warning. (Shipped.)

Both channels are encrypted (TLS for HTTPS, SSH for SSH). A third option - HTTPS dial-in (Danbyte → Outpost:443, the Outpost running a small listener) - is possible where 443-inbound is allowed but SSH isn't; SSH is the airgapped default, so HTTPS-in is a later add if a site needs it. You pick the transport when you add the Outpost.

Getting the agent - the Danbyte package store¶

The Danbyte instance is a version store for the Outpost, so remote hosts only ever pull from Danbyte (never GitHub/PyPI) - which is what makes airgapped installs work. On the Monitoring engines page an admin adds Outpost builds four ways:

• From your repo (dropdown) - set the agent repo once (Outpost repo field, with an optional token for a private repo); the package store then lists the repo's releases in a dropdown. Pick one → Danbyte fetches that release's CI-built binary and serves it. The easiest path.
• Fetch built binary - paste a repo URL + tag (a …/tree/v1.2.0 URL auto-fills the ref); Danbyte downloads that release's binary asset.
• Upload a file - a built single-binary or wheel/tarball.
• Source install (git) - a git URL + ref; the host does pip install git+url@ref (needs Python + git access on the host, so not for airgapped or private repos).

A bare binary installs with no Python on the host (download + chmod +x); a wheel/git release installs into a venv.

Each becomes a named version (e.g. 1.4.0), one of which is the default. Enrolling an Outpost shows a version-pinned install one-liner pointed at Danbyte itself serving that exact build:

curl -fsSL https://<danbyte>/api/outpost/install.sh | sudo sh -s -- --token=<TOKEN>

The generated script writes a systemd unit wired to this instance + the token and starts it. It adapts to the release kind:

• a single binary (built by the agent repo's release CI) → downloaded and chmod +xed, no Python needed on the host;
• a wheel/sdist → installed into a venv under /opt with pip;
• a git release → a source install (pip install git+url@ref).

File/binary artifacts are served from Danbyte (/api/outpost/download/<version>/, token or admin auth), so you can pin a version per site, roll out upgrades centrally, and stay fully airgapped. Manage versions under Governance → Monitoring engines → Outpost versions. (Shipped.)

Installing an Outpost - step by step¶

1. Upload a build (once): Governance → Monitoring engines → Outpost versions → upload a wheel/sdist, or add a git repo + ref. Mark one default.
2. Add the Outpost and pick its transport (HTTPS-pull or SSH).
3. Enroll it → copy the one-liner (the token is shown once).
4. On a host at the site, run the one-liner as root. It installs + starts the agent, which immediately dials out and begins pulling work.
5. Assign it to a device, site or location (on their forms) → everything in that scope is now monitored by the Outpost. Watch it go healthy on the engine's detail dialog.

SSH transport: skip steps 3–4's one-liner - instead set the host/user/key on the engine's detail dialog; Danbyte installs & drives it over SSH.

Installing¶

The Outpost's only hard dependency is the Danbyte instance it talks to - so the Danbyte instance serves the installer and the binary, and airgapped hosts never need GitHub or PyPI. Three ways to get it on a host:

1. Danbyte-served (default). The enroll dialog gives a copy-paste one-liner pointed at this instance:

danbyte-outpost run --url=https://danbyte.example.com --token=<TOKEN>

The install script + binary are fetched from https://<danbyte>/outpost/…, so 443-only hosts install fine. 2. Danbyte-pushed over SSH (airgapped). For an SSH-transport engine, Danbyte pushes the binary and installs it over the allowed Danbyte → host:22 channel - nothing to run at the site. 3. Offline tarball. A downloadable archive for a fully-isolated host (carry it in), or for building from the public repo.

Installing an Outpost¶

1. Governance → Monitoring engines → Add Outpost, give it a name.
2. It's enrolled automatically - copy the one-time OUTPOST_URL + OUTPOST_TOKEN (shown once; Rotate issues a new token and revokes the old one).
3. Install the agent on a host at the site (from the danbyte-outpost repo, or the package-store one-liner above) with those two values. It connects outbound, appears healthy on the engines page, and starts running its assigned checks.

The token only identifies the Outpost - it doesn't encode the worklist. What to monitor is the sites/locations you assign to it, resolved live, so adding a device is picked up on the next poll with no re-enrollment.

One tenant per engine. An Outpost only ever sees its own tenant's targets and credentials. A colo serving two tenants runs two Outposts.

Status¶

• Phase 0 (shipped) - the control plane: engines, the built-in local engine, site/location/tenant assignment, the settings UI.
• Phase 1 (shipped, HTTPS pull) - the agent protocol: /api/outpost/{hello, work,results} (Bearer-token auth), the dispatch() split (local → RQ, remote left for the Outpost to pull, claimed so nothing double-runs), and results ingested through the same path as the core (worker.ingest_results).
• Shared engine (shipped) - the checkers are a standalone danbyte_checks package the core and the Outpost import, so every kind (ICMP/TCP/UDP/HTTP/ SNMP/SSH/Telnet) runs identically on both - no drift. Verified end-to-end.
• SSH transport (shipped) - Danbyte drives airgapped Outposts over SSH (danbyte-outpost once, driven by drive_outposts on a timer); the SSH host/user/key are set on the engine.
• Package store (shipped) - admins upload an Outpost build (file or git ref) as a named version; Danbyte serves it and generates the version-pinned installer, so airgapped hosts pull only from Danbyte.
• Agent auto-update (shipped) - mark a version the golden image (the default release) and toggle Auto-update on an Outpost. On its heartbeat the core tells an auto-updating agent to move to the golden version if it differs; the (binary) agent downloads it, atomically swaps its own executable, and exits so systemd restarts on the new binary. Roll out an upgrade fleet-wide by making a new version golden.
• Site/location SNMP credentials (shipped) - SnmpProfileBinding now scopes to a site or location, so a site's Outpost polls its devices with site-scoped credentials (resolver: device → role → type → location → site → tenant default).
• SNMP discovery over the Outpost (shipped) - the agent pulls GET /api/outpost/snmp-work (its devices and VMs + scoped creds), runs the same danbyte_checks.snmp_facts.fetch_snmp the core runs, and posts back to POST /api/outpost/snmp → persisted through persist_snmp_result, the same path as a local poll. Runs on a slower cadence (default 900 s) than checks.
• Subnet discovery over the Outpost (shipped) - the agent pulls GET /api/outpost/sweep-work (its site's discovery-enabled prefixes), ICMP-sweeps each locally with the shared danbyte_checks.sweep.sweep_cidr, and posts live IPs to POST /api/outpost/discovered → _create_for_alive, the same path a local sweep uses (default 600 s). The core's run_discovery leaves remote-engine prefixes for their Outpost. So an Outpost now does everything the core does - checks, SNMP discovery, and subnet sweeps - for its scope. Discover now on a prefix served by an Outpost routes to that Outpost (the core can't reach the remote subnet): it flags the prefix + pokes the engine (sweep_requested_at), the agent sees sweep_pending on its next /work poll (~15 s) and sweeps immediately, and the button spinner clears when the prefix's last_discovered_at advances. Poll now and Refresh MACs on a device follow the same route (#128): a device whose engine - its own binding, its location's (or a parent's), its site's, or the tenant default - is an Outpost queues there (snmp_requested_at → snmp_pending) and answers Queued on Outpost … - the agent runs its SNMP cycle on the next poll and the results land through the normal ingest. The Outpost's work list carries every device that resolves to it by those same levels, a device placed in a room of a bound site included, and every SNMP-target VM whose site, or cluster's site, resolves to it. A VM rides in the same devices list under device_id, with "kind": "vm" beside it ("device" on a device); an agent echoes the id back and may echo the kind, so agents that predate VM targets poll them unchanged. Central polling would report outpost-only networks as SNMP unavailable, so the scheduled polls (dispatch_snmp_polls) and the poll_snmp command skip an Outpost's devices and VMs. Only an Outpost runs SNMP for the core: a target bound to a driver engine such as Zabbix polls from the core.
• SSH host-key pinning (shipped) - pin ssh_host_key on the engine and Danbyte verifies the server on every SSH connection.
• Single-binary packaging (shipped) - the agent repo's release CI builds a single-file binary on a tag; upload it to the package store and Danbyte's installer drops it with no Python on the host.

• Engine health / dead-Outpost detection (shipped) - the dispatcher's every-minute sweep flags a remote engine with assigned checks that hasn't polled within ~3× its interval (min 3 min): a red "engine unreachable - N checks stalled" banner appears on the Monitoring pages, and the tenant's notification channels get an engine unreachable event (and a recovered event when it comes back). Engines with no assigned checks never alert. The window is configurable per tenant (#129): Outpost offline after N minutes in Monitoring settings, 0 = the automatic 3× rule.

  A driver engine - Zabbix - has no agent phoning home. It is seen each time it reaches the system it answers through, and the automatic window is 3× the shortest interval among its checks, not the poll interval, which is an Outpost's setting and meant nothing for it. A failed call is deliberately not a sighting, so a Zabbix that is actually down still raises the alarm. Checks behind a dead Outpost keep their last state rather than flipping down - no results means no transitions - until the ordinary staleness rules age them. - Attribution (shipped) - results and status changes an Outpost phones in are stamped with that Outpost, so the Checks list, an address's history and the recent-changes list can say which agent saw a host go down. A row with no engine is the core's own workers - or predates the stamp, and reads the same way.