Monitoring¶

Monitoring watches your IPs and prefixes and tells you whether they're up, degraded, or down. You define checks — ICMP ping, TCP, UDP, HTTP(S), SNMP, SSH, Telnet, a TLS certificate read, or a script — attach them to an IP or a whole prefix, and Danbyte runs them on a schedule, keeps the history, and shows live status everywhere it matters: on detail pages, in list columns, and on a global Monitoring dashboard.

This page is organised by task. Jump to:

• Set up a check — create one and attach it
• Check types — the protocols and what each measures
• Where checks apply (prefixes and inheritance)
• Schedule modes — when checks run
• Reading results — status, sparklines, history, uptime
• Run a check now
• The Monitoring dashboard
• Alerts and Notifications
• Auto-discovery and cleanup
• Settings reference

Set up a check¶

Checks are created from a target's Monitoring section, and the same builder works for both IPs and prefixes.

1. Open the IP or prefix detail page and go to its Monitoring tab.
2. Click Add check (on a prefix, Add prefix check).
3. Choose either:
4. Use existing — pick one of your saved check definitions, or
5. New check — define one from scratch (it's also saved for reuse).
6. Pick the kind (ICMP, TCP, HTTP, …). The form's fields change to match — for example a TCP check asks for a port, an ICMP check for a packet count.
7. Set the timing and credentials as needed (see Check types and Schedule modes).
8. Save. The check appears in the Monitoring section and starts running on its schedule. Use Check now if you want a result immediately.

Reusable check definitions¶

A check definition (for example HTTP health on :8080 or SSH reachability) can be attached to many IPs and prefixes. Edit the definition once and the change takes effect everywhere it's used. Manage your library from the Templates tab of the Monitoring dashboard — it lists each definition with its kind, interval, and how many places use it, and warns you before deleting one that's still in use.

Check types¶

Each check reports one of four states: up, degraded (reachable but impaired), down (genuinely unreachable), or unknown (a configuration or internal error — never treated as an outage).

TLS certificate checks¶

The TLS certificate kind reads the certificate chain an endpoint presents and files it in the certificate inventory — expiry, issuer, SANs, key strength, self-signed and trust flags. It stores public certificate data only and never a private key, and reads an untrusted certificate without weakening verification anywhere.

Each read also records a binding — which endpoint served which certificate — so "what breaks when this expires" is answerable, and endpoints inside the warning window raise ordinary alerts through this same engine. See that page for the full field list, the trust rules, and the expiry thresholds.

HTTP checks are pinned to the target¶

An HTTP check always connects to the IP it's assigned to — you choose the scheme, port, and path, and optionally a Host header to set the virtual host, but it will never be redirected to dial some arbitrary hostname.

Credentials are encrypted and write-only¶

SNMP communities, SSH passwords and keys, and SNMP v3 keys are stored encrypted at rest. You can set them, but they're never shown again or returned through the API — the UI only tells you whether a credential is saved.

Script and exec checks¶

Two options let you monitor anything that can express its health as an exit code — handy when a plain port or HTTP probe can't capture the real condition.

SSH script — the SSH check can run a command (or a multi-line script) on the target and judge it by exit code and/or an output pattern. For example, to alert when nginx isn't running on a host:

Exit 0 → up; nginx stopped → non-zero exit → degraded (the host is reachable, only the service is down); a refused or timed-out SSH connection → down.

Local exec (Nagios-plugin style) — runs a vetted plugin on the worker and maps its exit code:

The plugin's first line of output becomes the result message (e.g. OK - 12ms).

Where checks apply¶

You can attach a check directly to an IP, or to a prefix.

• A check on a prefix applies to every IP inside it (when Apply to children is on). You can exclude specific IPs from a prefix check.
• A check on an IP directly is more specific than one inherited from a prefix.
• Most specific wins. If both a prefix check and a direct IP check exist for the same definition, the IP-level one takes over. Turning a check off on a single IP cancels the inherited one for that IP.

So you can monitor a whole subnet with one prefix check, then fine-tune individual hosts without touching the rest.

Inherited checks on an IP¶

On an IP's Monitoring tab, checks inherited from a parent prefix are marked as inherited and can't be edited or removed there — edit the parent prefix instead (the IP view links to it).

Per-check overrides¶

Without forking a shared definition, you can override individual settings on a single assignment — expand the check's row on the Monitoring tab to adjust:

• Enabled — keep the assignment but stop it running.
• Schedule mode — Follow global / Always on / Off.
• Interval, rise, fall — leave blank to inherit the definition's defaults.
• Exclusions (prefix checks) — tick the IPs to exempt.

Schedule modes¶

Danbyte runs checks automatically in the background; you don't run a separate scheduler. Each check resolves how often it runs from its own interval and a schedule mode:

The global switch and default interval live in the Monitoring settings (see The Monitoring dashboard).

Check frequency¶

How often a policy-driven check runs resolves in two levels:

1. Global default — the Default check interval in Monitoring settings. Every policy-based check uses this unless something more specific overrides it.
2. Per-scope override — on Monitoring → Configuration, each row (prefix, device, device type, device role) has a Frequency picker in its policy menu. Pick Follow global default to inherit, or a concrete cadence (1 min … daily) to override. The most-specific scope that sets one wins — a prefix beats its VRF beats global — so you can make one busy subnet poll every minute while everything else follows the default.

The chosen cadence shows on the policy button (e.g. 2 items · 15m). Overrides take effect on the next materialisation pass (within a minute or two), not instantly. Hand-attached checks (the Add check flow on an IP or prefix) keep their own per-check interval and schedule mode instead — see Per-check overrides.

Monitoring devices, types, and roles¶

Checks always run against IP addresses, so a device (or every device of a type/role) is monitored through its IPs. On Monitoring → Configuration, the Devices / Device types / Device roles tabs each carry the same policy controls plus an Apply to target that picks which of the device's IPs the checks cover:

A device-type or device-role policy applies the same target to every matching device. The most-specific scope wins (a per-device policy beats the device's type/role).

Turning Monitor on with no profiles/templates selected monitors basic reachability — the policy falls back to a default ICMP Reachability (ping) check (the policy button shows Ping), so the toggle always produces something. Attach profiles or templates to check more than reachability. A policy left on Follow global contributes nothing of its own — it just rides the broader-scope (global/VRF/prefix) policies — so it never adds a stray ping. A device with no IPs (or no primary/OOB when that target is chosen) still has nothing to check. Matching checks ("services") are created on the next materialisation pass (within a minute or two) and appear on each IP's Monitoring tab tagged from policy.

Monitoring a service¶

A Service (a device/VM's name + protocol + ports — e.g. "HTTPS · TCP 443") carries a Monitored flag. Turn it on from the device's Services tab and each port is watched by a TCP/UDP check against the service's target IP (its own IP, else the parent's primary IP). The row's Monitoring badge reflects the live state — Monitored (green) once checks are scheduled, No IP (amber) if the flag is on but no target IP exists yet (it activates automatically when one appears).

To watch a service across a whole fleet, define it once on the device type (Device type → Components → Services) and tick Monitor. Every device created from that type is then born with the service and, if monitored, starts checking as soon as it has an IP. This is the smart, low-maintenance path — no per-device clicking. Full design: service monitoring.

How status changes settle (hysteresis)¶

To avoid flapping on a single blip, status changes require a streak:

• A check goes up only after a number of consecutive successes (the rise count), and down only after a number of consecutive failures (the fall count).
• Degraded shows immediately when a host is reachable-but-impaired — it doesn't wait out the rise count.
• Stale — a check that's been down for a long time (a configurable number of consecutive failures, or a number of days) is escalated to stale to mark a chronic outage versus a fresh one.
• Skipped — IPs whose status is on your skip list (for example reserved) are never dialled; their checks are marked skipped and no result is recorded.

Every status change is logged so you get a history timeline and can drive notifications.

Reading results¶

On an IP¶

The IP detail page has a Monitoring section with one row per check showing:

• A status badge — up / down / degraded / unknown.
• The check name and kind.
• An inline sparkline of recent latency/status.
• The last latency and last-run time.

Expand a row to see its recent history table. When an IP has several checks with different results (one down while others are up), the badge becomes a split badge — coloured segments sized by how many checks are in each state, with a hover breakdown — rather than collapsing to just the worst one.

On a prefix¶

The prefix Monitoring tab shows:

• A roll-up badge and breakdown (e.g. 2 down · 1 up) across the prefix's IPs, worst status winning.
• The prefix-level checks, each with Apply-to-children, schedule-mode, and excluded-count controls.
• A per-IP status grid linking to each monitored child IP.

On a device¶

Checks attach to IPs, not to devices — but the device page rolls them up for you in three places:

• A roll-up badge in the device header, next to the status badge (the same mixed-status badge as the list column).
• The IPs tab has a Monitoring column showing each IP's status badge.
• The Overview has a Monitoring summary: the roll-up badge + breakdown across every IP assigned to the device (worst status winning) and a per-IP status grid linking to each monitored IP.

Because a service's check lives on the service's IP, service monitoring rolls up here too. The summary only appears when the device has at least one monitored IP. Manage the actual checks on each IP's Monitoring section.

In list pages¶

The Prefixes list and the Devices list each carry a Monitoring column showing the row's worst-status badge with a tooltip breakdown (a device rolls up across its assigned IPs), so you can scan health across many subnets or devices at a glance.

Uptime / SLA¶

The IP Monitoring tab includes an Uptime (SLA) card with a window selector (24h / 7d / 30d / 90d). Availability is time-weighted — measured from how long the IP spent in each state, not raw sample counts — so a slow check interval doesn't skew the number. Time spent in unknown or skipped is excluded from the calculation and reported separately, so a check that simply wasn't running can't read as 100% uptime. The card also shows the number of incidents in the window and the mean time to recovery (MTTR).

Run a check now¶

Anywhere checks are listed you can force an immediate run instead of waiting for the schedule:

• Check now on an IP or prefix runs its checks right away and refreshes in place.
• The Prefixes and IPs list pages have a bulk Check now action — select rows, and Danbyte re-checks every selected IP (and every IP in selected prefixes), with a live progress bar.

A manual check rolls into the same state machine as a scheduled one — it advances the rise/fall counters, can move the status, logs the change, and fires alerts exactly like an automatic scan.

The Monitoring dashboard¶

Governance → Monitoring is the global view. It has three tabs:

• Overview — stat cards (total checks, monitored IPs, definitions, alert channels), charts (status distribution, checks by type, results over the last 24 hours), recent status changes, a flapping card (see below), and the monitoring settings.
• Checks — a global list of every check with quick-filter tabs (All / Up / Degraded / Down / Stale / Skipped / Unknown, each with a count), search, and paging. Each row links to its IP.
• Templates — your reusable check library.

Settings on the dashboard¶

The Overview tab (also reachable from Settings → Monitoring) is where you set the per-tenant monitoring options:

Flapping monitor¶

The Overview tab has a flapping card that proactively surfaces IPs bouncing between states a lot — "this host is flapping, maybe go look at it" — ranked by how noisy each one is, regardless of whether it's currently up or down. To keep expected churn out of the list you can exclude whole IP statuses (the DHCP-scope escape hatch, in settings) or flip an Ignore flapping toggle on a single known-noisy IP. It only raises visibility — it doesn't page anyone.

Reverse-DNS enrichment¶

With Sync reverse DNS turned on, each time an IP is checked Danbyte looks up its PTR record and writes the hostname to the IP's DNS name field. Two options handle the no-result case: keep the existing name when a lookup fails but the host is up (so a transient DNS blip doesn't wipe a name off a live host), or clear the name when a lookup returns nothing.

Alerts¶

Status changes are turned into stateful alerts — incidents you can see and act on, not just a stream of changes. Manage them under Governance → Alerts.

• A change into a bad state opens one firing alert per condition (down/stale → critical, degraded → warning); a recovery or skip resolves it. Unknown never opens an alert.
• The Alerts page lists firing and resolved alerts with their severity, target, the bad status, when they opened, and how long they've been firing. Filter by status and severity; the list auto-refreshes.

Alert rules¶

The Rules tab decides which failures alert and at what severity. Each rule matches on check kinds, trigger statuses (down / stale / degraded), IP tags, and an optional prefix (all ANDed together) and assigns a severity. A failing check is matched against your enabled rules in priority order, and the first match sets the severity. With no rules, a sensible default applies (down/stale → critical, degraded → warning), so alerting works out of the box.

Acknowledge an alert¶

You can acknowledge a firing alert so the team knows someone owns it (with an optional note). The alert keeps firing, but acknowledging it records who and when — and stops reminder notifications (see below).

Silences and maintenance windows¶

A silence mutes notifications for matching alerts during a time window. Matchers mirror alert rules (kinds, statuses, IP tags, a prefix, plus an optional single IP — all empty means a blanket silence). While a silence is active, alerts still open and are tracked, but no notification is sent. A silence scheduled for the future is effectively a maintenance window. Manage these under Alerts → Silences; silenced alerts are flagged in the list.

Renotify, escalation, grouping, flap dampening¶

These time-based policies are per-tenant and off by default (except grouping), and all of them respect acknowledgement and silences:

• Grouping (on by default) — when one event opens many alerts at once (a switch dies, taking 50 IPs down), they're coalesced into a single digest per channel instead of a storm of messages.
• Renotify — re-sends a reminder for an alert that's still firing, unacked, and un-silenced after a configurable interval. Acknowledging or silencing stops the reminders.
• Escalation — an alert left firing and unacknowledged past a deadline is bumped to critical and re-notified.
• Flap dampening — an alert whose condition keeps reopening is marked flapping and excluded from reminders until it settles, so a flapping host can't page on a loop.

The Alerts table surfaces escalated, flapping, silenced, and ack chips, and tracks how many times each alert has notified.

Notifications¶

When an alert opens, escalates, or resolves, Danbyte routes it to your enabled notification channels. Manage them under Alerts → Channels; each has a Send test action. Every channel applies two gates before it fires:

• Minimum severity — alerts below the channel's threshold are skipped.
• On statuses — an optional allow-list of check statuses; empty means any bad status.

Supported channels:

Notifications are best-effort: a failing channel is logged and never breaks a check run. When a public base URL is configured (see below), messages include a clickable link straight back to the alert.

Subscriptions and the Notifications page¶

Beyond a channel's free-text recipient list, you can subscribe a user or a whole group to a channel — the channel then also emails that user, or every member of that group. Subscriptions are additive: they merge with the recipient list at send time.

Two kinds:

• Mandatory (admin- or group-assigned) — the subscriber cannot remove it themselves. This is the "the NOC group is on DC-event notifications and members can't opt out" case. Group subscriptions are always mandatory for members.
• Self-assigned — a user opted themselves in and can leave again.

The top-level Notifications page has two views:

• For you (every user): what you're subscribed to — your own, your groups', and any channel that lists your address directly — each tagged with its source (Self / Assigned / via group / Direct). Self-assigned rows have an Unsubscribe button; mandatory and group ones are read-only. Channels marked self-subscribable show up under "Available to join" with a Subscribe button.
• All channels (admins): every subscription across channels — the groups and users each one reaches — with add/remove.

The quickest path — "Notify me": a prefix or IP Monitoring tab — and a device's Monitoring strip — has a Notify me button. One click emails you (your account address) whenever that prefix/IP/device changes status — no channel setup. Behind the scenes it reuses a shared, auto-created email channel scoped to that object and adds you as a self subscription (visible under Notifications → For you, where you can turn it off again). A scoped channel only ever fires for its own target — a device scope covers every IP assigned to the device — for both status changes and alerts. Manually-created channels can be scoped the same way in the channel form (Everything / a subnet / a device).

Channel Send test now surfaces delivery errors instead of always reporting success — for an email channel that means the actual SMTP error, so a silent channel can be diagnosed from the UI.

Self-service opt-in/opt-out is gated by the subscribe capability on notification channels; grant it to the users/groups who should manage their own subscriptions (like reveal/connect, it isn't in the default Administrator set — superusers always have it). Managing other people's subscriptions uses ordinary add/change/delete on notification subscriptions. Mark a channel self-subscribable in its form to let permitted users join it.

Raw status-change notifications (no alert rules)¶

A channel can also send every status change for the IPs it matches, without setting up any alert rule — for operators who just want "email me when something in this subnet goes down". Enable Send raw status changes on the channel and pick a delivery mode:

• Instant — one message per check batch, carrying all of that batch's matching changes (coalesced, so a big flap is one email, not fifty).
• Batched — a periodic mini-digest every N minutes (default 30), summarising the window's changes as the same per-prefix status-badge chains the monitoring digest uses. Nothing is sent for an empty window.

Scope it with the channel's existing On statuses filter (e.g. only down) and an optional subnet — only IPs inside that prefix notify. This rides the same delivery gates and the same effective SMTP as everything else; instant fires from the check batch, batched from the minute beat, so neither needs a new timer.

Email and outbound delivery (deployment-wide)¶

Mail server and outbound options are a single deployment-wide setting, edited under Settings → Email & Delivery by an administrator (users with the manage permission). Email channels all deliver through this one server.

A Send test email action confirms the mail settings work. A misconfigured or unreachable SMTP host fails fast (a bounded connection timeout, EMAIL_SMTP_TIMEOUT, default 10s) and returns the SMTP error, rather than hanging the request.

Preview email templates. A Preview email templates card sends a sample of any email Danbyte produces — monitoring digest, certificate digest, alert and grouped-alert notifications, the sign-in code, and the invite — filled with example data, to an address you choose (or All templates at once). Subjects are prefixed with [Preview] and delivery uses the same SMTP config, so you can see exactly how each email looks before it goes out for real. Every email shares one branded, inline-styled HTML layout (with a plain-text alternative) using Danbyte's status palette.

Auto-discovery and cleanup¶

Two opt-in background jobs manage the IP lifecycle of monitored subnets. Both default off and are controlled from Monitoring → Settings.

Discovery¶

When enabled, Danbyte periodically ICMP-sweeps the prefixes you've enrolled and records the responders it finds as new IPs.

• What's enrolled: either every prefix (a global "discover everything" switch), or each prefix you flag Auto-discover plus its descendant prefixes in the same VRF — so flagging a parent subnet enrols all its children.
• New IPs are created with a tenant-specific Auto-discovered status (amber, not "available") so a human has to review and promote them — discovery never silently marks hosts active. In keeping with zero-pre-filled-data, that status isn't seeded at install; it's created the first time a responder is found, as a normal editable status you own.
• Guards: IPv4 only, and prefixes larger than a configurable minimum length (default /22) are skipped, so nobody accidentally sweeps a huge range.

Controls: each prefix has an Auto-discover toggle and a Discover now button (on its detail header and Monitoring tab). Small prefixes scan instantly; large ones run in the background with a live progress bar, and discovered IPs appear in the table as they're found. The Prefixes list bulk bar also offers Auto-discover on/off, Discover now, and Check now over the selection.

Stale cleanup¶

When enabled, discovered IPs that have been unreachable longer than a configurable number of days are deleted automatically.

Settings¶

Most day-to-day options live in the per-tenant settings on the Monitoring dashboard. A few deployment-level options (concurrency limits, the secret key for credential encryption, default global interval and switch, exec-check enablement and plugin directory, retention windows) are set by an administrator — see Reference → Settings.

Check history is high-volume (hundreds of thousands of raw results per day on a busy install), so Danbyte automatically prunes old results (default 30 days, MONITORING_RESULT_RETENTION_DAYS) and old status-change records (default 365 days, kept longer as an audit timeline) on a schedule. The rolled-up per-check state and the status-change timeline carry the long-term story; raw results only need to cover the sparkline/history windows.

Email digest¶

A scheduled summary email of the monitoring picture — a lightweight status report (like ping-monitor "digest" mails) delivered on your cadence rather than alert-by-alert. Each digest covers, per tenant: check counts by status (up / down / degraded / stale) with a reachable %, window activity counters (how many IPs went down / came up / went stale), currently-firing alerts by severity, and a count of configuration changes.

The State changes section lists every IP that changed state in the window, grouped by prefix. Each IP is drawn as a horizontal chain of status badges — the status it entered the window with, then one coloured badge per transition (Up → Down (Jul 20 03:01) → Up (Jul 20 03:07)) — so a flapping host reads at a glance. Badges use Danbyte's status palette (green up, red down/stale, amber degraded), and a heavily-flapping network is capped so the mail stays a reasonable size.

Configure it under Settings → Deployment → General → Email digest (deployment-wide default) — enable it, choose daily or weekly (with a weekday), and set the recipients (comma/newline-separated). A tenant can override the whole group (schedule + recipients) via its own settings, so an MSP sends each customer their own digest. Use Send test digest to email one immediately for the active tenant.

Delivery uses the same effective SMTP cascade as every other email (tenant/site override → deployment relay), and the message is sent as a branded HTML email with a plain-text fallback. Sending is driven by a daily systemd timer (danbyte-digest) → manage.py send_digest, which gates each tenant on its frequency, weekday, and last-sent date; nothing is sent twice in a day. Send one by hand with:

.venv/bin/python manage.py send_digest --tenant <slug> --force

The digest also carries a compact Certificates strip — expired, expiring (critical / warning), and recently-changed counts — so the overall certificate picture rides along with the status summary.

Certificate digest¶

A separate, certificate-focused digest, because expiry is the one class of problem where "you find out when it breaks" is an outage. Immediate, per-certificate expiry alerts already fire in real time through the notification channels (see Certificates); this is the recurring "everything approaching expiry, at a glance" companion email, sent as its own message rather than buried in the monitoring digest.

Each certificate digest covers, per tenant:

• Expired and expiring (critical / warning) leaf certificates actually served on the wire.
• Declared certificates (uploaded and assigned, not yet observed) approaching expiry.
• Recent changes — endpoints now serving a different certificate than before.

Enable it under Settings → Deployment → General → Email digest → Certificate digest. It runs on the same cadence as the monitoring digest (the daily danbyte-digest timer) but is gated by its own flag and tracked separately, so a tenant can run one, both, or neither. Recipients default to the digest recipients; set Certificate-digest recipients to send it elsewhere (e.g. a security team). A scheduled certificate digest with nothing to report is skipped; --force sends it anyway:

.venv/bin/python manage.py send_digest --tenant <slug> --force