Upgrading¶

Getting a new version, and (optionally) moving an existing install to the current /opt layout.

Upgrade to a new version¶

cd ~/danbyte
git fetch --tags
git checkout vX.Y.Z

# Python deps (uv, or pip inside the venv)
uv sync --frozen  ||  .venv/bin/pip install -r requirements.txt

.venv/bin/python manage.py migrate
make collectstatic frontend-build

# Restart whatever this install runs (prod shown; dev uses danbyte-backend)
systemctl --user restart danbyte-web danbyte-ws danbyte-workers danbyte-fastlane danbyte-frontend-prod

sudo tar xzf danbyte-<version>-linux-x86_64.tar.gz
cd danbyte-<version>-linux-x86_64
sudo ./install.sh                     # upgrades the existing install

Airgapped upgrade with the installer (step by step)¶

The most thorough path for an offline box - it re-asserts the production service set (gunicorn + daphne + workers + built frontend), so it also repairs a drifted install (e.g. a leftover dev danbyte-backend/runserver unit).

1. On an internet-connected machine, download the bundle for the target version from the releases page - danbyte-<version>-linux-x86_64.tar.gz (e.g. https://github.com/danbyte-net/danbyte/releases).

2. Copy it to the server (any path; /tmp is fine):

   scp danbyte-<version>-linux-x86_64.tar.gz you@server:/tmp/
   

3. On the server, unpack and run the installer as root. It auto-detects the existing service user (danbyte) and its home, so it upgrades the install in place - you do not pass a path:

   cd /tmp
   sudo tar xzf danbyte-<version>-linux-x86_64.tar.gz
   cd danbyte-<version>-linux-x86_64
   sudo ./install.sh --host danbyte.example.com      # your real hostname/IP
   

   Add --no-nginx if you terminate TLS / manage nginx yourself and don't want the installer to touch it. Without --host it keeps the name the nginx site already answers to.

   The nginx site is re-rendered from the new release only while it is still exactly what Danbyte rendered, with the certificate paths, name, mode and owner it has now; the old file is kept as danbyte.conf.bak-<time> and comes back if nginx -t refuses the new one. A site you edited by hand is left alone and the new render is written beside it as /etc/nginx/sites-available/danbyte.conf.new for you to merge.

4. Verify once it finishes:

   curl -s http://127.0.0.1:8000/api/health/; echo
   # -> {"status": "ok", "database": true, "version": "<new version>"}
   

   Or open Settings → Updates - the version and the environment table (Python / Django / PostgreSQL / Redis) load instantly and show the new version.

sudo -u danbyte env XDG_RUNTIME_DIR=/run/user/$(id -u danbyte) \
  systemctl --user disable --now danbyte-backend.service

APP="$(getent passwd danbyte | cut -d: -f6)/danbyte"
# confirm nothing is actually upgrading first:
ps -eo pid,cmd | grep -E "danbyte-upgrade|upgrade-bundle|upgrade/stage" | grep -v grep
sudo -u danbyte rm -f "$APP/.upgrade.lock" "$APP/.upgrade.lock.guard" \
                      "$APP/.upgrade-status.json" "$APP/.upgrade-bundle.tar.gz"

What an upgrade does¶

Every path - the Updates page, an uploaded bundle, automatic updates, danbyte-admin upgrade, and a re-run of install.sh - runs the target release's scripts/upgrade/stage.sh, as the service user's danbyte-upgrade.service:

If a step before resume fails, everything goes back as it was and what ran before starts again. The database is restored from the snapshot when migrations were applied; with none pending it never changed and is left alone. Nothing but monitoring results is written between quiesce and resume, so that restore loses no one's work. From resume on users are writing again, so a later problem keeps the new release and is reported as a warning. If even the database restore fails, Danbyte stays stopped (the maintenance page shows), the digest recipients get a mail, and danbyte-admin upgrade recover retries it; the pre-upgrade backup is the last resort.

Downtime is the time from quiesce to resume: a dump of the database when migrations are pending, the migrations, and the checks - the frontend build and the dependency download happen before it.

Upgrading from 0.16 or 0.17.0-dev1¶

The first upgrade off 0.16.x or 0.17.0-dev1 is started by that release's own upgrader, which knows nothing of the stage. 0.17 steps in where it can: when that upgrader runs 0.17's manage.py migrate, the migrate stops the timers, workers and web first, migrates in one transaction, collects static files (readable for nginx), and leaves the restart of everything it stopped to a unit that waits for the old upgrader to exit - so a failed migration no longer leaves a half-migrated database. That unit (its log: journalctl --user -u 'danbyte-upgrade-resume-*'):

• after a finished upgrade, starts everything the migrate stopped, the timers included, once the database has every migration the code on disk ships;
• after a migration that was rolled back in full, once the old upgrader has put its own code back, first removes the files 0.17 added. That rollback extracts the old code over the tree and removes nothing, so 0.17's migrations stayed behind: the old release's migrate could not even load its migration files, and the next upgrade applied them - and failed with the failed release's error. The migrate lists those files before it migrates: everything not in the rollback archive the old bundle upgrader wrote for this upgrade (danbyte-backups/code-pre-*.tgz) that is older than that archive. That includes a bundle uploaded to a git install, which 0.16 allows; the git upgrader's own rollback is a checkout, which removes them itself. Then it starts everything, the timers included;
• after a partly applied migration, starts nothing and says so in the upgrade's status, which the upgrade dialog shows, as well as in its log, which has the command that starts them.

It clears the failed mark of the units first, and removes the uploaded bundle (.upgrade-bundle.tar.gz, hundreds of MB, which the old upgrader keeps after a failure) and its own files. What it cannot fix: the new code and dependencies are already in place a minute before the migrate, a failure after the migration is not rolled back, and for a second during the old upgrader's rollback its status reads backup again (the rollback archive holds the status file of that moment).

It also fills in what 0.16's upgrader never records:

• The status names the release the upgrade was for instead of uploaded, so 0.16's automatic updates answer failed before for a release that failed, until a newer one appears, instead of downloading and trying it again - another backup and another outage - at every tick.
• After a finished upgrade, who started it and when: the Last upgrade card shows it and, for an automatic one, the next auto-upgrade tick mails the digest recipients and lists it on the Jobs page as the upgrade task, as for an upgrade the stage ran. A failed one put 0.16 back, which reports nothing: it shows only in 0.16's upgrade dialog and status on the Updates page, and in the unit's log.

Two more things follow from that order:

• A timer that fires between the old upgrader's checkout and its migrate runs 0.17 code on the 0.16 schema and logs a column ... does not exist error. Those runs fail before they write anything; the migrate stops the timers, and their units' failed mark is cleared when they start again.
• A git install upgraded from the Updates page is down for the migration and the whole frontend build: 0.16's upgrader runs npm ci and the build after the migrate, when everything is already stopped - about two and a half minutes on a small VM. The launcher below builds before it stops anything.

The safest way off 0.16 is therefore re-running the 0.17 installer from the bundle (sudo ./install.sh, above): it runs the full 0.17 upgrade. On a git install, run 0.17's own launcher as the upgrade unit:

sudo -u danbyte XDG_RUNTIME_DIR=/run/user/$(id -u danbyte) systemd-run --user \
  --unit danbyte-upgrade sh -c \
  'git -C ~/danbyte fetch --tags && git -C ~/danbyte show vX.Y.Z:scripts/danbyte-upgrade.sh | sh -s -- vX.Y.Z'

0.17.0-dev1's automatic updates do not see a newer pre-release; upgrade a dev1 install by hand once.

After an upgrade¶

Some releases need a step no migration can do - an nginx location, a new volume, a system package. Each release ships its list, and after any upgrade path (in-app, bundle, automatic, Docker) deployment admins see:

• an amber After upgrade: N steps badge in the top bar,
• a card at the top of Settings → Updates with each step, its snippet and a docs link, and a Done button per step (or Mark all done),
• the same list in the in-app upgrade dialog's success message.

A step that can tell it is done (the nginx site has the location, the unit file exists) disappears by itself once it is; the others stay until marked done. A fresh install starts with nothing pending.

The steps that need root - nginx, logrotate, the certificate unit - share one card, Run this release's root steps: it lists what it covers and gives the one command that applies them all, sudo ./install.sh --host-only from this release's bundle. Do it by hand opens each step's own snippet, and Done on the card marks every one of them done.

The steps on the host - nginx, logrotate, the site-certificate unit - are what an upgrade from the app cannot do, because it never has root. Re-running install.sh from the bundle does them. On a host upgraded from the app, the bundle of the release that runs does them all at once - the command these steps lead with:

sudo tar xzf danbyte-<version>-linux-x86_64.tar.gz    # the release that runs
cd danbyte-<version>-linux-x86_64
sudo ./install.sh --host-only

Each run records what it applied in /etc/danbyte/host-sync.json; until that names this release's files, the steps list Apply this release's nginx, logrotate and certificate-unit files. A site edited by hand is not replaced: the new render lands next to it as danbyte.conf.new, and --host-only --adopt replaces it anyway, keeping a backup. An install made with --no-nginx gets no nginx site from it while it has none, and no certificate unit either; sudo make install-tls-unit APP=<app directory>, in the same bundle, installs the unit alone.

On a git install the bundle of the release it runs does the same: --host-only changes no code, so it takes a git checkout. Download it from the releases page, check its SHA-256 and unpack it as root, as above. Or make the copy root owns with git - for a checkout between releases, of the commit it runs - and run the steps from there, naming the install:

sudo git clone --depth 1 --branch vX.Y.Z https://github.com/danbyte-net/danbyte /root/danbyte-vX.Y.Z
sudo make -C /root/danbyte-vX.Y.Z host-sync APP=/opt/danbyte/danbyte

ADOPT=1 takes the new render over a site edited by hand, as --adopt does.

The steps themselves, from the shell:

manage.py upgrade_notes              # print the pending steps (the upgrade scripts do this at the end)
manage.py upgrade_notes --ack all    # or --ack <id>

Search index¶

Global search runs on an index table. An upgrade starts a rebuild in the background once the site is back (danbyte-search-reindex.service), and the container entrypoint rebuilds it after migrating; if you migrate by hand, run manage.py rebuild_search_index once afterwards (it also runs nightly).

Database extensions¶

Global search relies on the PostgreSQL pg_trgm and unaccent extensions. The migration creates them itself: both are trusted extensions in PostgreSQL 13 and later, so the danbyte role needs no superuser rights. On an older server, or one where trusted extensions are disabled, create them once as a superuser before migrating:

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS unaccent;

Which install do I have?¶

getent passwd danbyte | cut -d: -f6      # the service user's home …
# … the app is in <home>/danbyte, e.g. /srv/danbyte/danbyte or /opt/danbyte/danbyte

# or ask systemd directly:
sudo -u danbyte XDG_RUNTIME_DIR=/run/user/$(id -u danbyte) \
  systemctl --user show danbyte-web -p WorkingDirectory

Move an install to /opt¶

Optional - only to match the current default layout. It moves the service user's home (app included) from /srv/danbyte to /opt/danbyte, repoints the nginx static paths, and turns on /var/log/danbyte logging. The database is untouched. Back up first.

cd danbyte-<version>-linux-x86_64     # unpacked with sudo tar xzf
sudo ./scripts/danbyte-relocate.sh          # → /opt/danbyte
# custom target / user:
sudo ./scripts/danbyte-relocate.sh --to /opt/danbyte --user danbyte

U=danbyte; UID_=$(id -u "$U")
asuser() { sudo -u "$U" env HOME="$1" XDG_RUNTIME_DIR=/run/user/$UID_ "${@:2}"; }

# 1. stop services + the user's systemd manager
asuser /srv/danbyte systemctl --user stop \
  danbyte-web danbyte-ws danbyte-frontend-prod danbyte-workers danbyte-fastlane danbyte-docs
loginctl disable-linger "$U"; loginctl terminate-user "$U"; sleep 2

# 2. move the home (contents included) and update the passwd entry
sudo usermod -m -d /opt/danbyte "$U"
sudo chmod 755 /opt/danbyte && sudo chmod o+x /opt/danbyte /opt/danbyte/danbyte
loginctl enable-linger "$U"

# 3. repoint nginx static/media/maintenance roots
sudo sed -i 's#/srv/danbyte/#/opt/danbyte/#g' /etc/nginx/sites-available/danbyte.conf
sudo nginx -t && sudo systemctl reload nginx

# 4. logging (see below), then restart
sudo install -d -o "$U" -g "$U" -m 755 /var/log/danbyte
asuser /opt/danbyte systemctl --user daemon-reload
asuser /opt/danbyte systemctl --user start \
  danbyte-web danbyte-ws danbyte-frontend-prod danbyte-workers danbyte-fastlane danbyte-docs

Turn on /var/log/danbyte logging¶

Installs from before file logging log only to the systemd journal (journalctl --user). To also write /var/log/danbyte/danbyte.log (app - Django, workers, monitoring) and gunicorn-*.log:

# as root - dir owned by the service user
sudo install -d -o danbyte -g danbyte -m 755 /var/log/danbyte

# as the service user - point the app at it and restart
echo 'DANBYTE_LOG_DIR=/var/log/danbyte' >> ~/danbyte/.env
systemctl --user restart danbyte-web danbyte-workers danbyte-ws

make logs still follows the journal; make logs-file tails the files. See the Logs section for what lands where. Leaving DANBYTE_LOG_DIR unset keeps logging on the console/journal only (the dev default).