2.0.0 • Published 2d agoCLI

@aifabrix/server-setup

Licence

—

Version

2.0.0

Deps

Size

614 kB

Vulns

Weekly

Summary Dependency Versions

AI Fabrix Builder Server Setup (af-server)

This is the one document you need to get your builder-server from zero to up and running in your own Docker. How to use the server (users, secrets, onboarding, etc.) is in the AI Fabrix documentation — start with the Builder CLI reference and developer workstation setup.

Why AI Fabrix Builder server?

Full AI Fabrix platform, one server, multi-developer: You run the full AI Fabrix platform and a separate builder-server where multiple developers each get their own dedicated, isolated environment (users, certificates, secrets, workspaces). One installer (af-server) sets up Docker (including the Compose v2 CLI plugin), nginx, SSL proxy, sync user, and backup on that server.
One tool for install and management: The same AI Fabrix Builder CLI gets you the platform and manages users, secrets, and onboarding. See AI Fabrix docs.
Security and standards: TLS, client certificates, no secrets in version control. Backups are explicit (config + DB + keys); store them encrypted.

Getting the builder-server Docker image

The builder-server Docker image is not on a public registry. You get it with the AI Fabrix platform.

Install the AI Fabrix Builder CLI: npm install -g @aifabrix/builder
Install the AI Fabrix server-setup CLI (af-server): npm install -g @aifabrix/server-setup
- If you get EACCES (permission denied), use either:
  sudo npm install -g @aifabrix/server-setup
Use the Builder CLI to run or deploy the platform (including builder-server). See Build AI-ready systems and developer workstation setup.

Once you have the platform (and the builder-server image), use af-server to install that server on your own host.

Prerequisites

Node.js ≥ 18 (where you run af-server)
SSH access to the server (key-based auth). af-server uses ~/.ssh/id_ed25519.pub or ~/.ssh/id_rsa.pub by default; use -i path/to/key if your key is elsewhere.
Server: Ubuntu. For remote install the script can install Docker and Docker Compose v2. For local default you need hostname (e.g. builder01.local) and no pre-existing certs; for custom domain you provide domain, SSL directory, and certificates (see below).

Local default (recommended)

The default setup is local: service name builder01.local, automatic Root CA and wildcard server certificate.

Set the server hostname to builder01.local: sudo hostnamectl set-hostname builder01.local
Run install (bootstrap runs in the server phase and creates Root CA + wildcard cert in /opt/aifabrix/certs; certs are copied to the SSL directory)
DNS: Add *.builder01.local to your DNS pointing to this server's IP. The server IP is shown at the end of installation.
Then use https://builder01.local/install-ca to download the Root CA for your PC; aifabrix dev init with server URL https://builder01.local uses that URL to fetch and install the CA. See Builder CLI reference.

Manual prerequisites (before install)

Do the steps in What you must do before running af-server (below). For local default: Ubuntu, root/sudo, SSH, hostname (e.g. builder01.local); after install, add *.builder01.local in DNS to the printed server IP. For custom domain: DNS (CNAME or A), SSL directory, and certificate + key; the install script does not create certificates and configures nginx to use the files you provide.

Install: from zero to running

Complete the manual prerequisites first. For local default you do not copy SSL (step 3); bootstrap creates certs during the server phase.

Flow summary: Only step 1 runs over SSH from your PC. Steps 5 and 7 run on the server after you log in. Install output prints the DNS instruction with the server IP when using local default.

Step	Where	Action
1	From PC	`af-server install-init $SSH` — only command over SSH; installs on server: SSH (if needed), Node 20+, npm, and `af-server` CLI.
2	One-time	Log in to the server once (e.g. with password) to approve passwordless SSH; then from PC: `af-server ssh-cert install $SSH`.
3	From PC	Custom domain only: Copy SSL certificate and key to the server (see SSL directory and certificates). For local default (builder01.local), skip—bootstrap creates certs in server phase.
4	From PC	Log in to the server via SSH (passwordless).
5	On server	`sudo af-server install` — install all services (Docker, Docker Compose v2, nginx package, Mutagen, cron, data dir); no builder-server container yet.
6	On server	Get the builder-server image (e.g. `az login` / `docker pull`).
7	On server	`sudo af-server install-server --dev-domain $DOMAIN` — nginx vhost, builder-server container (with `REMOTE_SERVER=https://$DOMAIN` and `DOCKER_ENDPOINT=tcp://$DOMAIN:2376`), Docker TLS. For local default (`$DOMAIN=builder01.local`), bootstrap runs and creates Root CA + wildcard cert; nginx gets `/install-ca` for the Root CA.
8	—	Done. At end of install (local default), the script prints: Add .builder01.local to your DNS pointing to <IP>* and Then open https://builder01.local/install-ca to get the Root CA for your PC.

Step 1: Bootstrap the server (from PC)

Set your target and run the only command that uses SSH from your PC. For local default use the server IP or hostname (e.g. builder01.local once DNS is set):

export SSH=serveradmin@builder01.local
# Or use the server IP until DNS is set: export SSH=serveradmin@192.168.1.30
af-server install-init $SSH

This installs on the server: openssh-server (if needed), Node 20+, npm, and @aifabrix/builder + @aifabrix/server-setup so af-server is available there. You’ll see progress messages and live output from the bootstrap (package lists, SSH, Node, npm install). No Docker, nginx, or builder-server yet.

Step 2: Passwordless SSH (from PC)

af-server ssh-cert install $SSH

Step 3: Copy SSL (from PC) — custom domain only

For custom domain (e.g. builder01.aifabrix.dev), put wildcard.crt and wildcard.key in /opt/aifabrix/ssl on the server. For local default (builder01.local), skip this step—bootstrap creates certs during install-server.

Example (replace $HDD with the folder on your PC that has the cert and key):

export HDD=/path/on/your/pc/to/certs
ssh $SSH "sudo mkdir -p /opt/aifabrix/ssl"
scp $HDD/wildcard.crt $SSH:/tmp/wildcard.crt
scp $HDD/wildcard.key $SSH:/tmp/wildcard.key
ssh $SSH "sudo mv /tmp/wildcard.crt /tmp/wildcard.key /opt/aifabrix/ssl/ && sudo chmod 600 /opt/aifabrix/ssl/wildcard.key"

Step 4: Log in to the server

ssh $SSH

Step 5: Install services (on server)

sudo af-server install

This installs Docker, the Docker Compose v2 plugin (docker compose), nginx (package only), Mutagen, data dir, apply-dev-users script and cron. It does not write the builder nginx vhost or start the builder-server container.

Step 6: Get the builder-server image (on server)

The image is not on a public registry. Use your platform’s method (e.g. Azure CLI or Docker login), then pull. Example:

az login
az acr login --name youracr
docker pull youracr.azurecr.io/aifabrix/builder-server:latest

Or with Docker login (username/password from your registry):

docker login <registry> -u <user> -p <password>
docker pull <registry>/aifabrix/builder-server:latest

Step 7: Install server (nginx vhost + container) (on server)

Local default:

sudo hostnamectl set-hostname builder01.local
sudo af-server install-server --dev-domain builder01.local

Bootstrap runs and creates Root CA + wildcard cert; nginx is configured with /install-ca serving the Root CA. At the end you'll see: Add *.builder01.local to your DNS pointing to <IP> — add that DNS record, then use https://builder01.local/install-ca to get the Root CA for your PC.

Custom domain:

export DOMAIN=builder01.aifabrix.dev
sudo af-server install-server --dev-domain $DOMAIN

Use the same domain as your DNS and SSL. Optional: --ssl-dir /opt/aifabrix/ssl, --data-dir /opt/aifabrix/builder-server/data, --builder-port 3000.

Production layout (flat app hostnames + legacy workspaces): When the Builder API stays at one host (e.g. builder01.aifabrix.dev) but per-developer apps use devNN.<root> and workspaces live under /home/devNN (not $DATA_DIR/workspace), pass both flags on install and install-server (same values):

sudo af-server install \
  --dev-domain builder01.aifabrix.dev \
  --developer-root-domain aifabrix.dev \
  --workspace-root /home

sudo af-server install-server \
  --dev-domain builder01.aifabrix.dev \
  --developer-root-domain aifabrix.dev \
  --workspace-root /home

--developer-root-domain — app URL root (e.g. dev06.aifabrix.dev via Traefik/nginx sync).
--workspace-root — canonical developer data at /home/devNN; SSH sessions bind /workspace → that path (PAM mount script). Default without this flag: $DATA_DIR/workspace/devNN.

If the builder-server image is not on the host yet, the first run writes an nginx vhost without client-cert verification (so nginx -t passes). After you pull the image and re-run install-server, the container will start, create ca.crt, and a subsequent run will enable client-cert verification and reload nginx.

Step 8: Done

Your builder-server is up. Use the AI Fabrix Builder CLI for users, secrets, certs, etc.—see Platform developer access.

Onboarding the first user

Use the AI Fabrix Builder CLI to onboard the first developer. The first user comes with a single-use PIN 123456 valid for 30 days. Run (from your PC, after installing the Root CA from https://builder01.local/install-ca):

aifabrix dev init --developer-id 01 --server https://builder01.local --pin 123456

For a custom domain, use your server URL instead (e.g. https://builder01.aifabrix.dev). See Builder CLI reference.

Migrating from a legacy dev-box (per developer)

When moving existing devNN OS users without replacing their SSH keys immediately:

aifabrix dev add <id> (Builder CLI, from an enrolled admin PC) — register the user in builder-server (developer, docker groups as needed).
sudo af-server import-legacy-ssh-keys <id> — copy existing ~devNN/.ssh/authorized_keys into builder.db and $DATA_DIR/ssh-keys before dev init on the PC (otherwise onboarding replaces keys). Omit <id> to scan all dev* accounts.
aifabrix dev pin <id> — optional PIN for PC dev init (mTLS + remote Docker).
On the PC: point SSH/Mutagen at the new builder host; run aifabrix dev init when ready for certificates.

After apply-dev-users runs, migrated users get /bin/bash login (not legacy dev-box shell), /workspace bind-mount on SSH when --workspace-root is set, and docker group membership from builder-server state.

Per-user Builder CLI on the host: Legacy dev-box setups often left a stale global /usr/local/bin/aifabrix or broken pnpm shims under ~/.local/bin. After migration, install the current @aifabrix/builder for each devNN account in one command — see Builder CLI on the server (install / clean).

If the onboarding PIN was already used or expired, prefer POST /api/dev/users/:id/pin with a valid TLS client certificate (admin or self) to create a new PIN. Full request and response shapes are in the builder-server OpenAPI UI at https://<your-builder-host>/api-docs (use the same host you use for the Builder CLI; client-certificate auth applies). If you are locked out but have shell access on the builder host, you can mint a new single-use PIN with sudo af-server reset pin <developer-id> (see Commands and Troubleshooting). Then use the printed aifabrix dev init … --pin … line (server URL from the running builder-server container when available).

Builder-server logs

Use these when you need to confirm what the API did or why a request failed. PINs, private keys, decrypted secrets, and certificate request bodies are not written to logs.

Container stdout / stderr
The builder-server process logs through Docker. Inspect with docker logs on your builder-server container (name depends on how it was created). Normal HTTP error responses (4xx) are logged at warn with method, URL path, status code, and a short message—the same kind of message the client sees (for example “invalid or expired PIN”), not the PIN value. Unexpected failures are logged at error with a stack trace.

Audit file: audit.log in the data directory
On the host, this file lives next to other builder data (for example /opt/aifabrix/builder-server/data/audit.log when you use the default data dir from af-server install-server). Each line is one JSON record for security-relevant events: certificate issuance outcomes, onboarding PIN creation, user create/update/delete, SSH key add/remove, secret key create/delete (key names only), and similar actions. Server errors (5xx) and uncaught exceptions also add lines when file auditing is on.

Turn off the audit file (stdout logging unchanged): set AUDIT_LOG_ENABLED=false or 0 in the builder-server container environment (alongside DATA_DIR and other variables), then restart the container.
Rotation: Treat audit.log like any growing log on the host—logrotate, truncation, or shipping to your log stack. The data directory must stay writable by the container.

Nginx on the host has separate access and error logs for TLS and reverse proxy behaviour; they complement but do not replace application logs above.

Commands

Command	Description
`af-server install-init <user@host> [ -i SSH_KEY ]`	From PC only. One-time bootstrap over SSH: install on server SSH (if needed), Node 20+, npm, and af-server CLI. Shows progress and streams server output.
`af-server install [ user@host ] [ -d DATA_DIR ] [ --dev-domain DOMAIN ] [ --developer-root-domain DOMAIN ] [ --workspace-root PATH ] [ --ssl-dir PATH ] [ -i SSH_KEY ]`	Run on server (omit target): `sudo af-server install`. Infra: Docker, Docker Compose v2, nginx pkg, Mutagen, data dir, cron, helper scripts. No builder vhost or container. For `DEV_DOMAIN` ending in `.local`, also runs bootstrap (Root CA + wildcard under `/opt/aifabrix/certs`, copies to `SSL_DIR`) and installs the dev root CA into the Debian/Ubuntu OS trust store (same PEM as `/install-ca`). With target: same over SSH.
`af-server install-server --dev-domain DOMAIN [ -d DATA_DIR ] [ --developer-root-domain DOMAIN ] [ --workspace-root PATH ] [ --ssl-dir PATH ] [ --builder-port PORT ]`	On server only. Nginx vhost, builder-server container, Docker TLS. Run after `sudo af-server install`. For `.local`, runs bootstrap again if needed and re-applies OS trust for the dev root CA (idempotent; first install already did this during `install`).
`af-server install-trust-dev-ca [ -s, --source PATH ]`	On server only (root). Re-install bootstrap dev root CA into OS trust (default source `/opt/aifabrix/certs/root-ca.crt`). Use after CA rotation or if `install-server` ran before the PEM existed.
`af-server backup [ user@host ] [ -d DATA_DIR ] [ -o output.zip ] [ -i SSH_KEY ]`	On-demand backup (config + DB + keys).
`af-server backup [ user@host ] --schedule [ --backup-dir PATH ] [ --keep-days N ] [ -i SSH_KEY ]`	Cron backup (daily 02:00, keep last N, default 7).
`af-server restore backup.zip [ user@host ] [ -d DATA_DIR ] [ --force ] [ -i SSH_KEY ]`	Restore backup to DATA_DIR.
`af-server ssh-cert install [ user@host ] [ -i SSH_KEY ]`	Add your SSH public key to server (passwordless auth).
`af-server install-ssh [ user@host ] [ -i SSH_KEY ]`	Activate SSH server (install openssh-server, enable and start ssh) without login. Omit target for local.
`af-server reset pin <developer-id> [ -d DATA_DIR ] [ --no-restart ] [ --no-apply-dev-users ]`	On server only (break-glass). Writes `builder.db`: removes existing PIN tokens for that user and inserts a new random 6-digit PIN (24h TTL, same rules as API `createPin`). Prints the PIN, expiry (UTC), and a copy-paste `aifabrix dev init … --pin …` line (server URL from builder container `REMOTE_SERVER` when the container is running). When run as root and `/usr/local/bin/aifabrix-apply-dev-users.sh` exists, runs that script with `DATA_DIR` set (same as cron) so dev SSH users get immediate host sync (workspace, `git safe.directory` for clones, etc.); skip with `--no-apply-dev-users`. Then restarts containers (unless `--no-restart`). Docker output is suppressed so the terminal stays readable. Run with `sudo` if `DATA_DIR` is not writable. Anyone who can run this effectively has host-level access to mint PINs—use only when mTLS admin PIN creation is not available.
`af-server import-legacy-ssh-keys [ developer-id ] [ -d DATA_DIR ] [ --no-apply ]`	On server only (root). Migration: copy existing `dev<id>` `~/.ssh/authorized_keys` into `builder.db` and `$DATA_DIR/ssh-keys/<id>/authorized_keys`. Omit `developer-id` to import for all `dev` OS users. Runs apply-dev-users afterward unless `--no-apply`. Run after* `aifabrix dev add` and before PC `dev init` so legacy keys are preserved.
`af-server fix-workspace-access [ linux-user ] [ -d DATA_DIR ]`	On server only (root). Runs `chown -R linux-user:linux-user` on `$DATA_DIR/workspace/<linux-user>` (creates the directory if missing). Omit `linux-user` when using `sudo` so `SUDO_USER` is used. Does not chown `WORKSPACE_ROOT/dev<id>` when `--workspace-root` is set—use `sudo chown -R devNN:devNN /home/devNN` for that layout.
`af-server install-builder-cli [ developer-id ] [ --builder-version VERSION ] [ --include IDS ] [ --exclude IDS ]`	On server only (root). Installs `@aifabrix/builder` into each selected *`dev` user's `~/.local` (`~/.local/bin/aifabrix` and `af`**). See Builder CLI on the server.
`af-server clean-builder-cli [ developer-id ] [ --include IDS ] [ --exclude IDS ] [ --dry-run ]`	On server only (root). Removes `~/.local` Builder CLI binaries and package tree (broken pnpm wrappers, partial installs). `--dry-run` previews only. See Builder CLI on the server.
`af-server install-dev-platform [ developer-id ] …`	On server only (root). Run `aifabrix setup --mode fresh` as each `devNN` (first-time platform). See Per-developer platform install and update.
`af-server update-dev-platform [ developer-id ] …`	On server only (root). Run `aifabrix setup --mode update-images` as each `devNN` (refresh images, keep data). See Per-developer platform install and update.
`af-server provision-dev-user [ developer-id ] …`	On server only (root). One-call server readiness after `aifabrix dev add`. See Per-developer platform install and update.
`af-server remove-dev-platform [ developer-id ] …`	On server only (root). Tear down `devNN` platform only (`aifabrix teardown --yes`). Keeps OS user and `builder.db`. See Per-developer platform install and update.
`af-server remove-dev-user [ developer-id ] …`	On server only (root). Full developer removal: platform + `builder.db` + workspace + OS user. See Per-developer platform install and update.

Backups contain secrets; store encrypted. Cron backup needs SQLite (builder.db) and zip on the server; default backup dir: /opt/aifabrix/backups.

Command examples (quick reference)

Copy-paste patterns; replace user@host, paths, and domains. Defaults match af-server <command> --help. On-demand backup without -o writes ./aifabrix-backup-YYYYMMDD-HHMM.zip in the current working directory.

install-init (from your PC)

af-server install-init serveradmin@192.168.1.10
af-server install-init serveradmin@builder01.local -i ~/.ssh/id_ed25519

install

On the server (infra only; no builder-server container yet):

sudo af-server install

From your PC over SSH:

af-server install serveradmin@builder01.local -i ~/.ssh/id_ed25519
af-server install serveradmin@host \
  --dev-domain builder01.aifabrix.dev \
  --developer-root-domain aifabrix.dev \
  --workspace-root /home \
  --ssl-dir /opt/aifabrix/ssl \
  --data-dir /opt/aifabrix/builder-server/data \
  --builder-port 3000 \
  -i ~/.ssh/id_ed25519

install-server (on server only, after `sudo af-server install`)

sudo af-server install-server --dev-domain builder01.local
export DOMAIN=builder01.aifabrix.dev
sudo af-server install-server --dev-domain "$DOMAIN" \
  --developer-root-domain aifabrix.dev \
  --workspace-root /home \
  --ssl-dir /opt/aifabrix/ssl \
  --data-dir /opt/aifabrix/builder-server/data

Re-run after pulling a new builder-server image or when nginx must pick up new devNN routes. See Host install and update.

install-dev-platform / update-dev-platform / provision-dev-user

See Per-developer platform install and update.

sudo af-server install-dev-platform --include 03,04,05 --check
sudo af-server update-dev-platform --include 03,04,05 --check
sudo af-server provision-dev-user 09 --check

install-trust-dev-ca (on server only; after bootstrap PEM exists)

sudo af-server install-trust-dev-ca
sudo af-server install-trust-dev-ca --source /opt/aifabrix/certs/root-ca.crt

backup

af-server backup
af-server backup -o ~/backups/builder-snapshot.zip
af-server backup serveradmin@host -i ~/.ssh/id_ed25519 -o ./snapshot.zip
sudo af-server backup --schedule
sudo af-server backup --schedule --backup-dir /opt/aifabrix/backups --keep-days 14

restore

af-server restore /path/to/aifabrix-backup-20250402-1200.zip
af-server restore ./backup.zip serveradmin@host -i ~/.ssh/id_ed25519
af-server restore ./backup.zip -d /opt/aifabrix/builder-server/data --force

install-ssh

sudo af-server install-ssh
af-server install-ssh serveradmin@host -i ~/.ssh/id_ed25519

reset pin (on builder host; usually sudo)

sudo af-server reset pin 01
sudo af-server reset pin 01 -d /var/lib/docker/volumes/aifabrix_builder-server_data/_data
sudo af-server reset pin 01 --no-restart
sudo af-server reset pin 01 --no-apply-dev-users

import-legacy-ssh-keys (on builder host; sudo)

Use when migrating legacy devNN users so existing SSH keys are kept in builder-server before PC dev init:

sudo af-server import-legacy-ssh-keys 06
sudo af-server import-legacy-ssh-keys
sudo af-server import-legacy-ssh-keys 06 --no-apply
sudo af-server import-legacy-ssh-keys -d /opt/aifabrix/builder-server/data

install-builder-cli (on builder host; sudo)

Install or upgrade the AI Fabrix Builder CLI for dev* OS users. See the full section Builder CLI on the server (install / clean) for behaviour, user selection, and troubleshooting.

sudo af-server install-builder-cli
sudo af-server install-builder-cli --builder-version 2.49.1
sudo af-server install-builder-cli 03
sudo af-server install-builder-cli --include 01,03,05 --exclude 02
sudo af-server install-builder-cli 03 --builder-version 2.49.1

clean-builder-cli (on builder host; sudo)

Remove broken or unwanted ~/.local Builder CLI installs before reinstalling. Use --dry-run first to preview.

sudo af-server clean-builder-cli --dry-run
sudo af-server clean-builder-cli
sudo af-server clean-builder-cli 03
sudo af-server clean-builder-cli --include 01,03,05 --exclude 02

fix-workspace-access (on builder host; sudo)

Use when cp, sync, or similar fails with permission denied under your DATA_DIR/workspace/<user> tree (files owned by root or the container user).

sudo af-server fix-workspace-access
sudo af-server fix-workspace-access dev02
sudo af-server fix-workspace-access dev02 -d /opt/aifabrix/builder-server/data

ssh-cert install

export SSH=serveradmin@builder01.local
af-server ssh-cert install "$SSH"
af-server ssh-cert install serveradmin@host -i ~/.ssh/id_ed25519

ssh-cert request

Not implemented yet; reserved for a future SSH CA flow. Use ssh-cert install for passwordless key-based SSH.

Builder CLI on the server (install / clean)

On a multi-developer builder host, each devNN Linux account needs its own copy of @aifabrix/builder on PATH so commands like aifabrix dev init, make dev-install, and af work inside SSH sessions. The global npm install -g on the host (used by serveradmin) does not apply to dev01…dev16.

af-server install-builder-cli and af-server clean-builder-cli manage per-user installs under ~/.local. Run both on the builder host as root (sudo).

Cron (aifabrix-apply-dev-users.sh) does not upgrade or downgrade Builder CLI. It only installs @aifabrix/builder@${BUILDER_CLI_VERSION} when ~/.local/bin/aifabrix is missing or broken. Any version already working there is left alone so developers can pin a local version. Use install-builder-cli for deliberate batch upgrades.

When to use

Situation	Command
Batch-upgrade all developers to a known Builder CLI version	`sudo af-server install-builder-cli --builder-version 2.49.1`
Install latest from npm for everyone	`sudo af-server install-builder-cli`
One developer after onboarding	`sudo af-server install-builder-cli 03`
Legacy host still has `/usr/local/bin/aifabrix` (wrong version) or broken pnpm `af` shims	`sudo af-server clean-builder-cli` then `install-builder-cli`
Preview what would be removed	`sudo af-server clean-builder-cli --dry-run`
Migration from dev-box (with `import-legacy-ssh-keys`)	Run `install-builder-cli` after SSH keys and workspace are set

What install-builder-cli does (per user)

Skips users who already have the target version at ~/.local/bin/aifabrix and ~/.local/bin/af (login-shell PATH).
Ensures export PATH="$HOME/.local/bin:$PATH" is present in ~/.bashrc / ~/.profile when missing.
Removes stale ~/.local/bin/aifabrix, ~/.local/bin/af, and ~/.local/lib/node_modules/@aifabrix/builder (same as clean-builder-cli).
Runs npm install -g @aifabrix/builder@<version> --prefix "$HOME/.local" as that user.
Verifies aifabrix --version, af --version, and command -v point at ~/.local/bin.

Version flag: use --builder-version (not --version — that prints the af-server package version). Omit --builder-version to resolve the latest @aifabrix/builder from npm (npm view on the host).

npm registry: @aifabrix/builder must be reachable for each devNN user (GitHub Packages or your configured registry). If install fails with 404 or permission, fix ~devNN/.npmrc / registry auth on the host before retrying.

What clean-builder-cli does (per user)

Removes these paths when present (working symlinks included — this is a full local uninstall, not only “broken” files):

~/.local/bin/aifabrix
~/.local/bin/af
~/.local/lib/node_modules/@aifabrix/builder

Then chown -R devNN:devNN ~/.local. Does not remove /usr/local/bin/aifabrix (host-wide); after clean + install, login shells should prefer ~/.local/bin via PATH.

Use --dry-run to list what would be removed without changing disk. Always run install-builder-cli afterward if users still need the CLI.

Selecting users

All three forms can be combined; developer-id positional arg targets one user (03 → dev03).

Pattern	Example	Users affected
All *`dev`** OS accounts on the host	`sudo af-server install-builder-cli`	Every `dev01`, `dev02`, … from `getent passwd`
Single id	`sudo af-server install-builder-cli 06`	`dev06` only
Explicit list	`sudo af-server install-builder-cli --include 01,03,16`	`dev01`, `dev03`, `dev16`
All except some	`sudo af-server install-builder-cli --exclude 02,06`	All *`dev` minus `dev02` and `dev06`**
List minus exclusions	`sudo af-server install-builder-cli --include 01,02,03,04,05 --exclude 02`	`dev01`, `dev03`, `dev04`, `dev05`

Ids accept 3 or 03; --include / --exclude use comma-separated lists without spaces required.

Examples

# Upgrade everyone to a pinned release
sudo af-server install-builder-cli --builder-version 2.49.1

# Latest from npm, all dev* users
sudo af-server install-builder-cli

# Subset during migration
sudo af-server install-builder-cli --include 01,03,04,05,07,08,09,10,11,12,13,14,15,16

# Clean broken shims, then reinstall
sudo af-server clean-builder-cli --dry-run
sudo af-server clean-builder-cli --include 01,03
sudo af-server install-builder-cli --include 01,03 --builder-version 2.49.1

# Verify as the developer (SSH login shell)
sudo -u dev03 bash -lc 'aifabrix --version && command -v aifabrix'

Output

Both commands print a per-user summary (ok / failed / skipped for install; cleaned / unchanged / skipped for clean). Install exits non-zero if any user failed; clean does not fail on unchanged users.

Help: af-server install-builder-cli --help, af-server clean-builder-cli --help.

Documentation

This README covers @aifabrix/server-setup (af-server) only — install, backup, restore, and host maintenance for builder-server.

AI Fabrix Builder CLI (users, secrets, onboarding, remote Docker, developer isolation):

Documentation index — full page list for AI Fabrix docs
Build AI-ready systems — integrator and platform overview
Builder CLI reference — command index
Developer workstation setup — install Builder CLI and connect to your environment
Platform developer access — developer personas and access model

Server operations on the host: use af-server <command> --help for flags and examples. Per-developer Builder CLI install/clean: Builder CLI on the server (install / clean). Re-run / upgrade host infra: Host install and update. Per-developer platform (install / update / provision): Per-developer platform install and update. Advanced topics covered here include Docker TLS on :2376, nginx mTLS, and env vars (DEV_DOMAIN, WORKSPACE_ROOT, DEVELOPER_ROOT_DOMAIN) — see High level: what install vs install-server does and Troubleshooting.

This README is: zero to up and running your builder server.

What you must do before running af-server

Complete these manually before af-server install. The install script does not create DNS, certificates, or the admin account; it expects them to exist.

Server and access

Ubuntu — 22.04 LTS or later (recommended). Other Debian-based systems may work but are not guaranteed.
Root or sudo — The install script must run with root (or sudo). From your PC you SSH as a user that can sudo to root; on the server you run sudo af-server install (or as root).
SSH access — You need key-based or password SSH to the server so you can run af-server install user@host, or you run install locally on the server.

DNS (CNAME or A record)

Local default (builder01.local): Set hostname to builder01.local; after install, add *.builder01.local to your DNS pointing to this server's IP (the IP is printed at the end of installation).
Custom domain: Create a DNS A record or CNAME so your chosen domain points to the server's IP. Set --dev-domain or env DEV_DOMAIN (e.g. builder01.aifabrix.dev). Ensure DNS has propagated before install.

SSL directory and certificates

Local default: No manual certs. The install runs a bootstrap that creates a Root CA and wildcard server cert; they are placed in /opt/aifabrix/certs and copied to the SSL directory. The Root CA is served at https://<domain>/install-ca.
Custom domain: Create the SSL directory (default /opt/aifabrix/ssl):
sudo mkdir -p /opt/aifabrix/ssl
To use another path, set --ssl-dir or env SSL_DIR.
Obtain a TLS certificate and private key (e.g. Let’s Encrypt, internal CA, or purchased). Full chain if applicable.
Place in the SSL directory:
- wildcard.crt — certificate (full chain if applicable)
- wildcard.key — private key
  Set permissions: chmod 600 /opt/aifabrix/ssl/wildcard.key
  The install script does not create certificates for custom domain; it only configures nginx to use these files.

Admin user (optional but typical)
If you SSH as a non-root user, that user must be able to sudo. The script will add an admin user (default serveradmin) to the docker group and grant it passwordless sudo. Ensure that user exists on the server if you rely on it, or set SETUP_ADMIN_USER to your SSH user.

High level: what install vs install-server does

af-server install (step 5 — run on the server) does infra only:

System — apt update and apt upgrade; optional hostname if SETUP_HOSTNAME is set.
Docker — Installs Docker if missing; enables and starts it.
Admin user — Adds the admin user (default serveradmin) to the docker group and grants passwordless sudo.
Nginx — Installs the nginx package only; enables and starts it. Does not write the builder vhost yet.
Data dir — Creates the data directory (default /opt/aifabrix/builder-server/data), workspace and ssh-keys subdirs, ownership for the container.
Apply-dev-users — Installs the script and cron job (every 2 minutes) that sync per-developer OS users from builder-server state. Creates dev<id> Linux accounts, merges managed SSH keys, sets login shell to /bin/bash, adds docker group membership from $DATA_DIR/aifabrix-os-docker-user-ids.json, and writes per-user ~/.aifabrix/config.yaml stubs (remote-server, docker-endpoint). When WORKSPACE_ROOT is set (via --workspace-root at install), workspaces live at $WORKSPACE_ROOT/dev<id> and PAM mounts /workspace on SSH login. The script configures a user-writable npm/pnpm prefix (~/.local) for dev users and the aifabrix user so they can run npm install -g and pnpm add -g without sudo. Builder CLI: bootstraps @aifabrix/builder only when ~/.local/bin/aifabrix is missing or broken — it does not change an existing version (use af-server install-builder-cli or aifabrix dev platform-install-cli for upgrades). When run as root, it can also grant passwordless sudo to the aifabrix user (override with SUDO_NOPASSWD_USER in apply-dev-users-defaults). Without sudo: run the script as the current user (e.g. SUDO_NOPASSWD_USER=aifabrix); it will only set up that user's ~/.local and ~/.npmrc/~/.pnpmrc so npm and pnpm global installs work without sudo.
Host-ops processor — Installs /usr/local/bin/aifabrix-host-ops.sh and cron /etc/cron.d/aifabrix-host-ops (every minute). Processes one pending job from $DATA_DIR/host-ops/pending/ at a time (flock single-flight), runs af-server host-ops process, moves jobs to running/ / completed/ and writes logs under host-ops/logs/. Used when admins queue work via aifabrix dev platform-*, dev add (provision), or dev delete from their PC. Passwords in job files are redacted in API responses.
Mutagen — Downloads and installs Mutagen; systemd service and daemon.
Optional — If INSTALL_PORTAINER=1, installs the Portainer container.

af-server install-server (step 7 — run on the server) does the server phase:

Nginx vhost — Writes the builder site config from template (domain, SSL dir, proxy to builder-server), reloads nginx.
Builder-server container — Creates data dir (if needed), starts the builder-server container (if the image is present).
Docker TLS — Copies certs and configures /etc/docker/daemon.json for TLS on tcp://0.0.0.0:2376. Server certificate: same wildcard as nginx (wildcard.crt / public CA or bootstrap). Client auth: builder-server internal CA ($DATA_DIR/ca.crt → /etc/docker/ca.pem). Developers use client certs from aifabrix dev init; the Builder CLI merges the daemon’s public trust chain into ~/.aifabrix/certs/<id>/ca.pem when the server cert is not signed by the issue-cert CA.

Result: after both steps, the server has Docker, nginx (HTTPS for your domain), the builder-server container (if image present), and Mutagen—ready for the Builder CLI to use.

Host install and update (`af-server install` / `install-server`)

Use this section when the builder host is already running and you need to refresh infra scripts, nginx, or the builder-server container — not for first-time setup (see Install: from zero to running).

When to re-run `sudo af-server install`

Situation	Action
Upgraded `@aifabrix/server-setup` on the host and need new helper scripts	`sudo af-server install` with the same flags as the original install (`--dev-domain`, `--developer-root-domain`, `--workspace-root`, `-d`, `--ssl-dir`)
`aifabrix-apply-dev-users.sh` on disk is stale (cron behaviour changed)	Re-run `install` — each run copies the bundled script to `/usr/local/bin/aifabrix-apply-dev-users.sh`
`aifabrix-host-ops.sh` missing or stale (host-ops jobs never run)	Re-run `install` — copies script to `/usr/local/bin/aifabrix-host-ops.sh` and refreshes `/etc/cron.d/aifabrix-host-ops`
Added `--developer-root-domain` or `--workspace-root` after first install	Re-run `install` with both flags so `apply-dev-users-defaults` and PAM workspace mounts match production layout
`.local` dev domain: OS should trust the bootstrap Root CA	`install` re-applies `install-trust-dev-ca` logic when bootstrap PEM exists (idempotent)

Safe to re-run: apt upgrade, Docker enablement, Mutagen, cron, data-dir layout, apply-dev-users script install, nginx package only. Does not recreate the builder-server container or rewrite the builder nginx vhost (that is install-server).

From your PC over SSH (same as first install):

af-server install serveradmin@builder01.aifabrix.dev \
  --dev-domain builder01.aifabrix.dev \
  --developer-root-domain aifabrix.dev \
  --workspace-root /home \
  -i ~/.ssh/id_ed25519

When to re-run `sudo af-server install-server`

Situation	Action
Pulled a new builder-server image	`sudo af-server install-server --dev-domain …` — recreates container when config/image changed
First run had no image (nginx without client-cert verify)	After `docker pull`, re-run `install-server` so `ca.crt` exists and nginx enables mTLS
Changed `--dev-domain`, `--ssl-dir`, `--builder-port`, or Traefik nginx templates	Re-run with updated flags; nginx vhost and `aifabrix-sync-dev-traefik-nginx.sh` refresh
New `devNN` developers added in builder-server	Re-run `install-server` (or wait for `systemd.path`) so `/etc/hosts` and `devNN.<developer-root-domain>` nginx map update

Pass the same --dev-domain, --developer-root-domain, and --workspace-root values you used on install.

sudo af-server install-server \
  --dev-domain builder01.aifabrix.dev \
  --developer-root-domain aifabrix.dev \
  --workspace-root /home

Upgrade the `af-server` CLI itself

On the builder host (or your PC):

sudo npm install -g @aifabrix/server-setup@latest
af-server --version

Then re-run sudo af-server install on the server so bundled assets under /usr/local/bin match the new package.

Per-developer platform install and update

After aifabrix dev add on an admin PC, use provision-dev-user for one-call server readiness, or run install-dev-platform / update-dev-platform separately.

Builder CLI prerequisite: automation calls aifabrix setup --mode fresh or --mode update-images. The installed @aifabrix/builder must expose --mode on aifabrix setup (upgrade with install-builder-cli when a release ships it). Until then, --check still runs host preflight; apply will exit with an upgrade hint.

Command	Purpose	Builder equivalent (as `devNN`)
`install-dev-platform`	First-time AI Fabrix platform for selected developers	`aifabrix setup --mode fresh`
`update-dev-platform`	Refresh platform container images; keep volumes/data	`aifabrix setup --mode update-images`
`provision-dev-user`	Full server provision (includes platform + cert steps)	Combines host steps + `install-dev-platform`

All three will:

Run on the server only as root (sudo af-server …).
Execute aifabrix as each devNN user (never as root).
Support [developer-id], --include, --exclude (same as install-builder-cli).
Print a per-user report (PASS / SKIP / FAIL / BLOCKED) and exit non-zero if any user FAILs.

`install-dev-platform`

When: Developer has Builder CLI and secrets file on the host, but no running platform (Keycloak + miso-controller + dataplane) yet.

Default platform mode on shared hosts: single-traefik (Traefik + TLS, public URLs like https://devNN.<developer-root-domain>/…), not single (localhost ports only).

Admin credentials: Batch runs read $DATA_DIR/apply-dev-platform-defaults (ADMIN_EMAIL, ADMIN_PASSWORD, optional DEFAULT_PLATFORM_MODE, BUILDER_SERVER_URL). Create that file on the host before the first batch install (mode 600; not in git). There is no silent admin123 default in automation.

Skip when: Platform already healthy for that user (infra + three apps up).

# One developer
sudo af-server install-dev-platform 09

# Batch
sudo af-server install-dev-platform --include 03,04,05

# Preflight only
sudo af-server install-dev-platform 09 --check

Planned flags: --platform, --installation, --admin-email, --admin-password, --builder-version, --timeout-minutes, --check, --fail-on-blocked — see af-server install-dev-platform --help.

`update-dev-platform`

When: Platform is already installed; you need newer images after ACR/build bump without wiping Postgres/Redis volumes.

Skip when: Platform not installed or not healthy for that user.

sudo af-server update-dev-platform
sudo af-server update-dev-platform --include 01,05
sudo af-server update-dev-platform --check

Update ignores install-only flags (--admin-password, --platform fresh wipe, etc.).

`remove-dev-platform`

When: Platform install failed or you need a clean aifabrix setup --mode fresh retry. Removes Docker stack and local platform state only.

Keeps: OS user, builder.db, SSH keys, ~/.local Builder CLI, config.yaml, client certs.

sudo af-server remove-dev-platform 03
sudo af-server remove-dev-platform 03 --check
sudo af-server remove-dev-platform --include 03,04

Then re-install: sudo af-server install-dev-platform 03.

`remove-dev-user`

When: Remove the developer completely from the host (inverse of provision-dev-user).

Removes: Platform (unless --skip-teardown), builder.db entry, workspace data, SSH keys, OS user (userdel -r via apply-dev-users).

sudo af-server remove-dev-user 03
sudo af-server remove-dev-user 03 --check
sudo af-server remove-dev-user 03 --skip-teardown   # platform already stopped

Re-add with aifabrix dev add then provision-dev-user.

Relationship to other commands

Step	Command
Register user in builder-server	`aifabrix dev add` (admin PC)
Host SSH, `config.yaml`, Builder CLI	`install-builder-cli`, `import-legacy-ssh-keys`, migration prep script
Platform Docker stack per `devNN`	`install-dev-platform`
Image refresh	`update-dev-platform`
Platform teardown only (retry install)	`remove-dev-platform`
Full developer removal	`remove-dev-user`
One-call after `dev add`	`provision-dev-user`
PC mTLS / remote Docker	`aifabrix dev pin` + `aifabrix dev init` on laptop

Integrator groups at dev add: developer,secret-reader (read-only central secrets) or developer,secret-manager (read/write). SSH-first developers: developer,docker.

Troubleshooting

`error: unknown command 'reset'`

The af-server reset pin command needs @aifabrix/server-setup 1.6.0 or newer (current release 1.7.x). If af-server --help does not list reset, install a current release from npm:

sudo npm install -g @aifabrix/server-setup@latest

Check af-server --version, then af-server reset pin --help.

Onboarding PIN used up or expired (`reset pin`)

With a client certificate, create a new PIN via POST /api/dev/users/<developer-id>/pin (documented under /api-docs on your builder host).
On the server, with write access to the data directory (usually sudo):
```
sudo af-server reset pin 01
```
That default DATA_DIR is /opt/aifabrix/builder-server/data (af-server install layout). If you run builder-server with the AI Fabrix Builder CLI (aifabrix run builder-server), data often lives in a Docker named volume instead, e.g. mount destination /mnt/data with host path like /var/lib/docker/volumes/aifabrix_builder-server_data/_data. In that case you must point reset at the same place the container uses, or the API will still see the old PINs:
```
sudo af-server reset pin 01 -d /var/lib/docker/volumes/aifabrix_builder-server_data/_data
```
Your volume name may differ; confirm with docker inspect <container> --format '{{json .Mounts}}' and use the Source directory that maps to /mnt/data (or whatever DATA_DIR is inside the container).

Copy the printed PIN (do not log or commit it) and run the printed aifabrix dev init … --pin … line (or complete onboarding per Builder CLI reference).
If reset pin fails with Could not resolve builder server URL, start the builder-server container first (or pass the correct --server URL manually in the printed command pattern).
If SQLite reports database is locked, stop the builder-server container, run reset pin with --no-restart, then start the container again—or retry after the default restart finishes.

Workspace `Permission denied` when copying or syncing (e.g. under `DATA_DIR/workspace/<user>`)

If cp, make sync, or similar fails because files under your builder workspace were created as root or the container user, reclaim ownership on the builder host:

sudo af-server fix-workspace-access

That uses your login name (SUDO_USER) and the default DATA_DIR. To fix another account (e.g. as an admin):

sudo af-server fix-workspace-access dev02

Use -d if DATA_DIR is not the default /opt/aifabrix/builder-server/data.

When --workspace-root /home is in use, workspaces are under /home/devNN—this command does not fix those paths; run sudo chown -R devNN:devNN /home/devNN instead.

Builder CLI wrong version or `command not found` for `devNN` (SSH)

Symptoms: aifabrix --version shows an old number (e.g. from /usr/local/bin/aifabrix), af is missing, or pnpm left a non-working shim under ~/.local/bin.

Install per-user CLI on the host (see Builder CLI on the server):
```
sudo af-server install-builder-cli --builder-version 2.49.1
```
Or for one user: sudo af-server install-builder-cli 03 --builder-version 2.49.1.

If install keeps picking up stale files, clean first:

sudo af-server clean-builder-cli --dry-run
sudo af-server clean-builder-cli 03
sudo af-server install-builder-cli 03 --builder-version 2.49.1

Verify in a login shell (matches SSH and make dev-install):
```
sudo -u dev03 bash -lc 'aifabrix --version; command -v aifabrix; command -v af'
```
Expected: version matches --builder-version, paths are /home/dev03/.local/bin/aifabrix and .../af.
npm 404 / permission errors during install: each devNN user needs registry access to @aifabrix/builder (check ~devNN/.npmrc and GitHub Packages token scope). Fix auth on the host, then re-run install-builder-cli.
af-server install-builder-cli unknown command: install @aifabrix/server-setup 1.7.2 or newer on the host (sudo npm install -g @aifabrix/server-setup@latest).
Version changed back after install-builder-cli (older hosts): aifabrix-apply-dev-users.sh used to reinstall BUILDER_CLI_VERSION every 2 minutes. Current script only bootstraps when CLI is missing — refresh with sudo af-server install (copies the script to /usr/local/bin) or redeploy @aifabrix/server-setup. Deliberate upgrades are install-builder-cli only; cron will not override a working ~/.local install.

AI Fabrix banner missing in some terminals

The welcome line comes from the # aifabrix dev shell block in ~/.bashrc (written by apply-dev-users). It runs only for interactive shells ($- contains i). Non-interactive sessions and some IDE reconnects show a plain prompt without the banner — that is normal. af / aifabrix still resolve via ~/.local/bin when .bashrc or .profile sets PATH.

`af-server --version` shows an old version after installing a newer one

If you installed a newer @aifabrix/server-setup from npm but af-server --version still shows an older number:

Check which binary is used
Run which af-server and sudo which af-server. If they differ, your shell may be using a user-level install while the new one was installed for root.

Uninstall, clear cache, then reinstall

sudo npm uninstall -g @aifabrix/server-setup
npm cache clean --force
sudo npm install -g @aifabrix/server-setup@latest
hash -r   # clear shell’s command cache
af-server --version