Back to MusePages

Historic Atlas — Deployment field guide

The original prompt, shown in full as plain text.

Prompt

# SPEC: Deployment

Status: draft. Part of [SPEC.md](../SPEC.md).

## Purpose

Define how the application is run as a whole. The first deployment is for
developers; other environments are specified when needed.

## Requirements

- The whole application runs as one Docker Compose project: ingress,
  frontend, backend, Keycloak and the database.
- It is easy to bring up and to reset to a clean state, with all data local
  to the checkout.
- Secrets are never committed and never printed.

## Decisions (developer deployment)

Introduced in [AVARC-18](../tasks/finished/TASK-deployment-setup.md).

- **Files:** everything lives in `deploy/`: `docker-compose.template`,
  the feature templates `docker-compose.<feature>.template`, `run.sh` and the systemd units `systemd/avarc-stage.service`
  and `systemd/avarc-prod.service`, each running as its own user and group from that
  user's home directory (`/home/avarc-<environment>`). On the host,
  `deploy/deploy.sh stage|prod` (root only) installs them from a checkout:
  it copies the unit to `/etc/systemd/system` and `run.sh`, the account scripts and
  the templates together into the home directory, then reloads systemd. It never starts
  the service and never touches data or secrets. It records the version of
  the checkout (`git describe --tags`) in `VERSION` next to the files and
  prints it; for prod it refuses a checkout that is not exactly a release tag
  (see [SPEC-releases.md](SPEC-releases.md)) or has uncommitted changes. `run.sh` renders the templates with `envsubst` into `docker-compose.yml`
  (gitignored, mode 0600) at every start; `$$` is a literal dollar.
- **Feature templates:** `docker-compose.template` holds what always runs.
  An optional part of the deployment is a file of its own,
  `docker-compose.<feature>.template`, that only adds services, labels and
  other keys and overrides nothing of the core. `run.sh` renders every
  template with `envsubst`, merges the core and the features whose flag is on
  with `docker compose config` (which also validates the result), and writes
  the one `docker-compose.yml` that every command uses; a feature whose flag
  is off is not in it, so `up --remove-orphans` removes its containers. Today
  the features are `db-tailnet`, the labels that serve the database on the
  tailnet, and `dbtool`, the database tool; both are merged under the same
  flag (see "Database on the tailnet"). The merged file
  is generated for Compose, not for reading; the comments stay in the
  templates.
- **Environments:** `dev`, `stage` and `prod`. Every script of `deploy/` that
  works on an environment takes `--dev`, `--stage` or `--prod`, and finds the
  environment without one: the command line wins, then `SECRETS=dev|stage|prod`
  in the environment, then the user running the script (`avarc-stage`: stage;
  `avarc-prod` and `avarc`: prod; any other user: dev). The systemd units
  therefore need no option, though they may still pass one. The scripts that
  need a database that prod never exposes (`create-db-user.sh`, `import-sql.sh`,
  `import-legacy.sh`) refuse prod. The environment
  names the Compose project (`avarc-<environment>`), the data directory and
  the default ingress port (8080 dev, 8081 stage, 8082 prod; `INGRESS_PORT`
  overrides), so environments can run side by side.
- **Start and stop:** `deploy/run.sh [--stage] up` runs the project in
  the foreground, as systemd expects (`up -d` detaches), `down` stops it
  keeping data, any other argument is passed to `docker compose`. `reset`
  (dev only) stops the project and deletes its local data and generated
  secrets. Without arguments, or with `--help`, it prints the usage.
- **Operating without access to the deployment files:** the commands that act
  on the running containers (`down`, `stop`, `start`, `restart`, `ps`, `logs`,
  `top`, `kill`, `pause`, `unpause`, `exec`, `port`, `events`) neither read
  the secrets file nor render or write any file: `run.sh` passes them to
  `docker compose -p avarc-<environment>`, which finds the containers by their
  project label. A user who may use Docker but owns none of the deployment
  files can therefore stop, restart and inspect an environment; `down` then
  leaves the in-memory directories of the secrets for the owner's next start
  to replace. `restart` restarts the containers as they are and does not apply
  a changed template, secrets file or image; that takes a start of the unit,
  which is the job of the owner. The unit restarts a project that was stopped
  this way, so a lasting stop is `systemctl stop`.
- **One-shot services:** a service that only does a job at start (today
  `keycloak-init`, which creates Keycloak's database) leaves an exited
  container behind. `run.sh up` watches such a service in the background
  and removes its container once it has exited and the services that
  depend on it have been started after that (never earlier: compose would
  fail waiting for it), so `docker compose ps -a` shows only the
  running project. A further `up` creates it again.
- **Entry point:** the ingress is published on the environment's port on
  localhost. It carries docktail labels that expose it on the tailnet as
  `avarc-<environment>` on port 443, where docktail terminates TLS; the
  ingress itself does not.
- **Database on the tailnet (optional):** on dev and stage only, never on
  prod, the PostGIS database can be served on the tailnet as
  `avarc-<environment>-db`, port 5432, by docktail labels on the `db`
  service (TCP, no published host port). It is off unless the environment's
  secrets file sets `DB_TAILNET=true`; `run.sh` ignores the variable on prod,
  with a warning. Inspection uses the read-only account `avarc_ro` (created
  by `deploy/create-db-user.sh`, which takes its password
  `DB_READONLY_PASSWORD` from the secrets file or generates and appends one);
  it reads the tables of the `avarc` database, in the schemas `public` and `avarc`, and its sessions are read-only.
  The account cannot connect to Keycloak's database (users and credential
  hashes) unless `DB_KEYCLOAK=true` in the secrets file (or the option
  `--keycloak` of `create-db-user.sh`) asks for it and the script is run,
  which then grants read access; running the script without either withdraws
  it again.
  The same holds for the schema `legacy` of an imported legacy export (see
  below), which holds personal data: `avarc_ro` may read it only when
  `DB_LEGACY=true` in the secrets file (or the option `--legacy`) asks for it
  and the script is run once the schema exists; it is a schema of the `avarc`
  database, so the database tool shows it in its `avarc` connection and no
  connection of its own is added. Running the script without either
  withdraws the access again.
  The application's own credentials are not handed out. Who may reach the
  service is decided by the tailnet's access rules.
- **Importing a SQL file:** `deploy/import-sql.sh [--stage] [--schema NAME]
  FILE.sql`, on dev and stage only, loads a plain SQL file, such as the export
  of a legacy system, into the `avarc` database. The objects go into a schema
  of their own (default `legacy`, never `public` or `avarc`) unless the file
  names a schema itself, and the load is one transaction, so a failing file
  leaves nothing behind. It is run on the host of the environment and uses the
  database's own account inside the `db` container, like `create-db-user.sh`;
  it reads no password. Mapping the imported tables onto the application's
  tables is a separate step.
- **Backups:** introduced with
  [AVARC-48](../tasks/finished/TASK-database-backup.md). `deploy/backup-db.sh
  [--stage]` takes a logical backup of the databases of an environment: a
  `pg_dump -Fc` of `avarc` (the schemas `avarc` and, when imported, `legacy`)
  and of `keycloak` when it exists, and `pg_dumpall --globals-only` for the
  roles. It runs inside the `db` container as the database's own account, like
  `create-db-user.sh`, so the dump matches the running PostgreSQL version and
  no password is read. The files `<database>-<yyyymmdd-hhmmss>.dump` and
  `globals-<yyyymmdd-hhmmss>.sql` go to `data/<environment>/backups/` next to
  the script (gitignored), mode 0600, because they hold personal data and
  credential hashes; a file appears under its final name only when complete.
  Files older than `BACKUP_KEEP_DAYS` (default 7) are removed. On stage and
  prod, `systemd/avarc-<environment>-backup.service` (oneshot, after the
  project's service, run by the same user) and `.timer` (every night at 03:00,
  `Persistent=true`) run it; `deploy.sh` installs both but does not enable the
  timer: `systemctl enable --now avarc-<environment>-backup.timer` is a step on
  the host. The service does not start the project: with the `db` service down
  the run fails with a message in the journal. The run also exports the users
  of the realm into the same directory when Keycloak is running
  (`keycloak-users-<yyyymmdd-hhmmss>.json`, see
  [SPEC-auth.md](SPEC-auth.md), "Surviving a rebuild of Keycloak's data").
  Copying the dumps off the host
  is outside this repository; the backups directory is the interface.
  - **Restore:** `deploy/restore-db.sh [--stage] [--scratch] [--globals]
    [--database NAME]... [--no-backup] [--yes] [STAMP]` restores the files of
    one backup run
    (`STAMP` like `20261018-030010`, default the newest) from the backups
    directory (`BACKUP_DIR` overrides it). The steps are the manual ones, in
    this order, all inside the `db` container as the database's own account:
    the roles (`globals-<stamp>.sql`, only with `--globals`, for a new
    cluster; errors for roles that exist are harmless), then each database
    with `pg_restore --clean --if-exists --create`, which drops and recreates
    it from the dump. Without `--database` that is every database that has a
    dump in the run; `--database avarc` or `--database keycloak` (repeatable)
    restores only those, and leaves the other database as it is.
    - **Into the environment** (the default) replaces the data, so the script
      refuses to run while `backend`, `keycloak` or `dbtool` are up (stop them
      with `run.sh [--stage] stop backend keycloak`, and `dbtool`), takes a
      fresh backup first unless `--no-backup` says the database is beyond
      that, and asks to type the environment's name unless `--yes` is given.
      Start the services again afterwards.
    - **`--scratch`** is the way to try a dump: it restores into a throwaway
      container of the same image as the `db` service, without volume or
      published port, restores the roles too (a new cluster has none),
      compares it with the running database when there is
      one, and removes the container; the environment is not touched and
      nothing needs to be stopped.
    - Either way it ends with a check of the migration history
      (`avarc.flyway_schema_history`: latest version and number of entries),
      the number of locations and users of the application and of Keycloak's
      accounts and realms; a scratch run shows the live value next to each
      one, where small differences are expected because the backup is older.
    - Verified with a dump of a database holding data in the schema `avarc`,
      a role and a second database: the restore into a new cluster, and over
      an existing database, both succeed and the data and the roles are back.
      A dump taken by the stage timer (2026-10-18) was restored into a
      scratch container: the migration history, the locations with their
      positions, the users and Keycloak's realms and accounts equal those of
      the running database, and both roles are back.
- **Database tool (optional):** introduced with
  [AVARC-38](../tasks/TASK-database-tailnet.md). With `DB_TAILNET=true`, the
  `dbtool` service is started as well: CloudBeaver (Community edition, image
  `dbeaver/cloudbeaver`, pinned to an exact version in the template), a web
  tool to browse tables, run read-only queries and show spatial columns on a
  map. It is served on the tailnet by docktail labels as
  `avarc-<environment>-dbadmin` (HTTPS on 443, docktail terminates TLS), has
  no published host port and is not routed by the ingress. Without the
  variable, and on prod always, the service is not started (it is in the
  feature template `dbtool`, which `run.sh` merges only then). It is written for "the database",
  not for one engine: a connection is listed per database of the
  environment, so an engine change or an internal SQLite database adds or
  replaces a connection, not the tool. A SQLite file would be read from a
  read-only mount and has no port of its own (but see Verified below).
  - **No setup by hand:** `run.sh` configures the tool at each start from
    files it renders into the in-memory secrets directory and mounts
    read-only, so the first-run wizard never appears. The server
    configuration is a Java properties file mounted as
    `conf/.cloudbeaver.auto.conf` (keys `CB_SERVER_NAME`, `CB_SERVER_URL`,
    `CB_ADMIN_NAME`, `CB_ADMIN_PASSWORD`); the admin name and password are
    `DBTOOL_ADMIN_NAME` (default `dbadmin`; `admin` and `user` are the names
    of the tool's built-in teams and are refused) and `DBTOOL_ADMIN_PASSWORD`
    of the secrets file, the password generated and appended when missing and
    meeting the tool's policy (at least eight characters, mixed case, a
    digit). The
    connections are `GlobalConfiguration/.dbeaver/data-sources.json`, with
    `read-only` set, the user and password of the read-only account in
    `auth-properties`, and no custom connections for users. The password is
    therefore in a file that exists only in the memory directory, never in
    the repository or on disk between starts. The configuration is
    regenerated at every start and wins over changes made in the tool.
    The tool creates the admin account only when its workspace is first
    started, so `run.sh` keeps a fingerprint (a hash) of the admin name and
    password in the workspace; when the secrets file no longer matches it,
    `run.sh` stops the tool and removes the tool's own database, and the
    account is created again from the current values at that start. A
    workspace without a fingerprint, from before this rule, is reset once.
  - **Access:** the tool's own login (the admin account above) is required;
    anonymous access is switched off. The connections use the read-only
    account of `create-db-user.sh`, never the application's role, so the
    database stays read-only whatever is typed in the tool. Keycloak's
    database is a second connection, `keycloak`, with the same account, only
    when `DB_KEYCLOAK=true` (the access itself is granted by
    `create-db-user.sh`, see above); otherwise it is not a connection. The map view loads its base map from a
    public tile server in the browser of the person using it.
  - **State:** the tool's workspace is under `deploy/data/<environment>/`,
    which `reset` removes. The container runs as the user that started the
    project, like `keycloak`, to read the secrets directory.
  - **Verified (CloudBeaver 26.2.2):** started by `run.sh` on the
    developer deployment, the tool comes up configured without the wizard,
    the admin account logs in, anonymous access is off, the one connection is
    read-only and connects to PostGIS with the read-only account (after
    `create-db-user.sh`), with the container running as the starting user and
    the configuration files mounted read-only. A second `up` over running
    containers, a start with the tool disabled (the container and its
    in-memory configuration are removed) and `down` all work. On stage, the
    Value panel of a geometry cell has a MAP tab that shows the point on a
    base map, and the `keycloak` connection works once `create-db-user.sh`
    has granted it. Not covered: SQLite. The SQLite driver is disabled in
    this web build ("Driver disabled" on connecting, also with
    `enabledDrivers` set in the server configuration), so the tool does not
    cover a SQLite database until that is solved.
- **Data:** bind-mounted under `deploy/data/<environment>/` (gitignored).
- **Secrets:** their values are in `deploy/secrets.dev.sh` (dev; generated
  with random values on the first start), `deploy/secrets.stage.sh` or
  `deploy/secrets.sh` (prod), all gitignored and exporting one variable per
  secret. `run.sh` reads each needed secret from that file in an isolated
  shell, writes one file per secret into a directory in memory
  (`secrets` below the run directory, mode 0700), and mounts it
  read-only at `/run/secrets` in the containers that need it. The run
  directory is `/run/avarc-<environment>`, which the systemd unit creates
  for its user (`RuntimeDirectory=`, mode 0700) and systemd removes when the
  unit stops, so the secrets do not outlive the service; without that
  directory (dev, no unit) it is `avarc-<environment>` in the user's runtime
  directory (`$XDG_RUNTIME_DIR`). The configuration of the database tool is
  rendered into `dbtool` below it. Secrets are
  not passed as environment variables; a start with a missing value fails
  naming it. `run.sh` keeps shell tracing off while it handles values.
- **Accounts:** the secrets file also holds the credentials of the initial
  accounts (`ADMIN_EMAIL`, `ADMIN_PASSWORD`, and optionally `EDITOR_*` and
  `VIEWER_*` pairs). These are read only by `deploy/create-admin-user.sh`, never
  mounted into containers. The dev secrets file gets an admin account
  with a random password when it is generated.
- **Other configuration:** non-secret variables (for example
  `INGRESS_PORT`, and `PUBLIC_ORIGIN`, the externally visible origin, which
  defaults to `https://avarc-<environment>.<tailnet domain>`, the domain being
  that of the tailnet the host is in, and without a tailnet to
  `http://localhost:<ingress port>` for dev) are exported by `run.sh` and referenced in `compose.yaml`.
- **Footer links:** on dev and stage `run.sh` passes the frontend
  `FOOTER_API_DOCS=true` and, if the database tool is enabled,
  `FOOTER_DBTOOL_URL` set to its address, and `FOOTER_KEYCLOAK_URL` set to the
  address of the realm's admin console; on prod all are empty
  ([SPEC-frontend.md](SPEC-frontend.md), AVARC-45).
- **Host settings:** `IMAGE_NAMESPACE` and the `*_VERSION` variables (the
  images to run) are taken from the environment if set there, otherwise
  from the environment's secrets file (an exported variable of that name,
  read in the same isolated shell), otherwise from the defaults of
  `run.sh`. This lets a host name its own registry or mirror in a file that
  is never committed, with no change to the systemd unit or this
  repository.
- **Upstream registry:** the upstream images (the database, `keycloak-init`
  and the database tool) are named `${UPSTREAM_REGISTRY}/<image>`; the host
  setting `UPSTREAM_REGISTRY` defaults to `docker.io` and names a registry
  cache instead, set like `IMAGE_NAMESPACE`. It may include a path, as a cache
  that serves several upstream registries keeps Docker Hub under a prefix of
  its own. It is independent of `IMAGE_NAMESPACE`, which names the project's
  images, a different upstream.
- **Image pull:** `run.sh up` first pulls the images when `IMAGE_NAMESPACE`
  names a registry (it contains a `/`, like `ghcr.io/avarcorg`); with the
  default namespace the images are the ones built on the host and nothing is
  pulled. A failed pull is tried again a few times with a pause in between
  (4 attempts, 15 seconds; `PULL_ATTEMPTS` and `PULL_RETRY_DELAY` change
  that). When the last attempt fails, `run.sh` stops with a message before
  anything is started, and the service manager's own restart tries again later.
- **No image monitoring:** the services that run the project's own images
  (`avarc-ingress`, `avarc-backend`, `avarc-frontend`, `avarc-keycloak`) carry
  the label `dd.watch=false`, so that Drydock neither monitors nor scans their
  containers; Drydock has no narrower switch. The services that run upstream
  images (the database, the database tool and the one-shot `keycloak-init`)
  stay watched. A new service gets the label if it runs one of our images. It
  takes effect when `deploy/run.sh up` recreates the containers.
- **Updating stage:** stage runs the moving tag `snapshot` of the four images
  (the latest build of `main`). An update is a restart of the service: the
  stack stops, `run.sh up` pulls the new images and starts them, and the
  checks are the frontend's `/health` (200), `/api/hello` (401 without a login),
  `/auth/realms/avarc/.well-known/openid-configuration` (200; the bare realm
  path answers 404 at the ingress) and the origin (307). A rollback sets the four
  `*_VERSION` variables of the host to `sha-<commit>` of the earlier build
  and restarts; going back to `snapshot` removes them again. A successful
  build of `main` prompts the update (see [SPEC-releases.md](SPEC-releases.md)).
  Prod is not updated this way: it runs release tags only.
- **Keycloak admin console:** the `keycloak` service carries docktail labels
  that expose it on the tailnet as `avarc-<environment>-kc` on port 443,
  where docktail terminates TLS. Admins reach the console of the `avarc`
  realm at
  `https://avarc-<environment>-kc.<tailnet domain>/auth/admin/avarc/console/`
  (`KEYCLOAK_ADMIN_ORIGIN` overrides the address). The `master` realm
  administrator is `keycloak-admin` with the secret
  `KEYCLOAK_ADMIN_PASSWORD`. The ingress does not route any of this. The
  service also publishes its port on `127.0.0.1` only (8180 dev, 8181
  stage, 8182 prod; `KEYCLOAK_ADMIN_PORT` overrides), which
  the account scripts use.
- **Backend service:** runs the image built by the backend
  ([SPEC-backend.md](SPEC-backend.md)), image
  `<IMAGE_NAMESPACE>/avarc-backend:<BACKEND_VERSION>`. `run.sh`
  defaults `IMAGE_NAMESPACE` to `avarcorg` and `BACKEND_VERSION` to the
  version of the Maven build (`0.1.0-SNAPSHOT`), and the environment
  overrides both (set `BACKEND_VERSION` to a release version to run a
  release, and `IMAGE_NAMESPACE=ghcr.io/avarcorg` to run the published
  images, see [SPEC-releases.md](SPEC-releases.md)). The Spring profile named after the environment is active
  (`SPRING_PROFILES_ACTIVE`); it has no published port, the ingress reaches
  it by service name. It gets `PUBLIC_ORIGIN` and, mounted as a file, only
  the secret `KEYCLOAK_CLIENT_SECRET` (not the whole secrets directory);
  `KEYCLOAK_INTERNAL_URL` (default `http://keycloak:8080/auth/realms/avarc`)
  overrides the address at which it reaches Keycloak.
- **Frontend service:** runs the image built by the frontend
  ([SPEC-frontend.md](SPEC-frontend.md)), image
  `<IMAGE_NAMESPACE>/avarc-frontend:<FRONTEND_VERSION>`, versioned
  like the backend (`FRONTEND_VERSION`); it has no published port, the
  ingress reaches it as `frontend:3000`. `BACKEND_URL` (`http://backend:8080`)
  is the address at which its server side reaches the backend.
- **Ingress service:** runs the image of the module `avarc-server-ingress`
  ([SPEC-ingress.md](SPEC-ingress.md)), image
  `<IMAGE_NAMESPACE>/avarc-ingress:<INGRESS_VERSION>`, versioned like the
  backend (`INGRESS_VERSION`). The container listens on port 8080, published
  as `INGRESS_PORT` and exposed by docktail; the routing is part of the
  image.
- **Keycloak service:** runs the image of the module `avarc-server-keycloak`
  ([SPEC-auth.md](SPEC-auth.md)), image
  `<IMAGE_NAMESPACE>/avarc-keycloak:<KEYCLOAK_VERSION>`, versioned like the
  backend (`KEYCLOAK_VERSION`, defaulted by `run.sh` to the version of the
  Maven build). A host needs the image and the secrets directory only; the
  start script and the realm import are in the image.
- **Database image:** PostGIS, as chosen in
  [SPEC-backend.md](SPEC-backend.md).
- **Keycloak's database:** decided in
  [AVARC-54](../tasks/TASK-social-login.md): Keycloak keeps its database
  (`keycloak`, created by `keycloak-init`) in the same database server as the
  application's, with no server of its own. For: no further container and
  memory on a small host, one place to back up and one major version to
  upgrade, and everything around it already works with it (dumps, restore,
  the user export of [SPEC-auth.md](SPEC-auth.md), the rule of the read-only
  account). Against: a failure or upgrade of the server hits both, and both
  connect as the same role. The users do not depend on the dump alone: the
  user export is a second protection. The decision can be reversed: moving
  the database to a server of its own is a dump, a restore and a changed
  connection address. Next steps, if needed: a role of its own for Keycloak
  with its own password (separates the credentials at little cost), and
  after that a server of its own (independent failure, upgrade and restore,
  at the price of one more container and a second backup path).

- **Observability of the project** ([AVARC-24](../tasks/TASK-observability-ingress-database.md),
  [SPEC-observability.md](SPEC-observability.md)):
  - Every service logs through Docker's `json-file` driver with rotation
    (10 MB, 5 files), which the log collector of the monitoring stack reads.
  - Every service has a container health check. The backend, the frontend
    and the ingress run the Tiny Health Checker of the Paketo health-checker
    buildpack (`/layers/paketo-buildpacks_health-checker/thc/bin/thc`, set by
    `THC_PORT` and `THC_PATH`), because the backend image has no shell: the
    backend asks `/actuator/health/readiness` on its management port (9091), the
    frontend `/health` and the ingress
    its metrics listener. Keycloak and the database use their readiness. The
    modules name the buildpack in `image.buildpack` of their POM and the
    parent POM adds the health checker to every `pack` build. The frontend
    listens on all addresses (`HOSTNAME`) so that the check reaches it on
    localhost.
  - The ingress routes `/actuator/` to the backend only where `run.sh` sets
    `FOOTER_API_DOCS=true` (dev and stage; passed on as `ACTUATOR_ENABLED`).
  - The network `<project>-metrics` carries only metrics endpoints: the
    ingress, the frontend, the backend (management port 9091), Keycloak
    (management port 9000) and the database are attached to it, and the monitoring stack
    joins it as an external network. Nothing on it is published. The
    network may exist before the project is started (the monitoring stack
    creates it if missing): `up` uses it as it is, and `down` leaves it in
    place while another container is attached.
  - The database logs statements slower than 500 ms.

## Open questions

- Deployment target and environments beyond the developer deployment.