Operations
This page describes how to run Stacklane in production: the service, logs, backups, upgrades, background work, health checks and common problems.
The systemd service#
The package installs stacklane.service. It is enabled and started during installation.
sudo systemctl status stacklane
sudo systemctl restart stacklane
sudo systemctl stop stacklane
What the unit does#
| Setting | Value | Effect |
|---|---|---|
User, Group | stacklane | Runs as the unprivileged stacklane user. |
Environment | STACKLANE_CONFIG=/etc/stacklane/stacklane.toml | Configuration file. |
ExecStartPre | stacklane migrate | Applies database migrations before every start. |
ExecStart | stacklane serve | Runs the server. |
Restart | on-failure, after 3 seconds | Restarts Stacklane if it exits with an error. |
TimeoutStopSec | 30 | Time Stacklane gets to stop cleanly. |
StateDirectory | stacklane, mode 0750 | Creates /var/lib/stacklane. |
UMask | 0027 | New files are not readable by other users. |
LimitNOFILE | 65536 | Maximum number of open files and connections. |
Sandbox#
The unit uses the systemd sandbox to limit what the process can do:
| Setting | Effect |
|---|---|
ProtectSystem=strict | The file system is read-only for the service, except for the paths named in ReadWritePaths. |
ReadWritePaths=/var/lib/stacklane | The only writable directory. The database must be here. |
ProtectHome=yes | Home directories are not visible. |
PrivateTmp=yes, PrivateDevices=yes | Private /tmp and no access to physical devices. |
NoNewPrivileges=yes | The process and its children cannot gain privileges. |
ProtectKernelTunables, ProtectKernelModules, ProtectKernelLogs, ProtectControlGroups, ProtectClock, ProtectHostname | No changes to kernel settings, modules, logs, cgroups, the clock or the host name. |
ProtectProc=invisible | Processes of other users are not visible. |
RestrictNamespaces, RestrictRealtime, RestrictSUIDSGID, LockPersonality, MemoryDenyWriteExecute | No namespaces, real-time scheduling, set-user-ID files, personality changes or writable executable memory. |
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX | Only IPv4, IPv6 and Unix sockets. |
SystemCallArchitectures=native, SystemCallFilter=@system-service ~@privileged | Only the system calls a normal service needs. |
CapabilityBoundingSet=CAP_NET_BIND_SERVICE, AmbientCapabilities=CAP_NET_BIND_SERVICE | The only capability is binding to ports below 1024. |
Because of ProtectSystem=strict, the service cannot write to /etc/stacklane. Create the secret key with stacklane keygen before the first start. The package does this for you.
Changing the unit#
Do not edit the installed unit file. An upgrade replaces it. Use a drop-in file instead:
sudo systemctl edit stacklane
Add your settings in the [Service] section, save, and restart:
[Service]
Environment=STACKLANE_LOG=info,tower_http=debug
sudo systemctl restart stacklane
Logs#
Stacklane writes its logs to standard error. systemd stores them in the journal.
# Follow the log
sudo journalctl -u stacklane -f
# Logs since the last boot
sudo journalctl -u stacklane -b
# Logs of the last hour
sudo journalctl -u stacklane --since "1 hour ago"
# Only warnings and errors
sudo journalctl -u stacklane -p warning
Log level#
The STACKLANE_LOG environment variable sets the log filter. The default is info,tower_http=warn,sqlx=warn.
| Value | Effect |
|---|---|
info,tower_http=warn,sqlx=warn | Default. Start, stop, tasks, warnings and errors. |
info,tower_http=debug | Also log every HTTP request. |
debug | Detailed logs from all components. |
Set it for the service with a drop-in file, as shown in Changing the unit. For a single command, set it on the command line:
sudo -u stacklane env STACKLANE_LOG=debug stacklane check
Messages to know#
| Message | Meaning |
|---|---|
listening | The server has started. The log line shows the address and the public URL. |
mail.smtp_url is not set; outgoing email is written to the log | Email is not sent. Each message is written to the log instead. |
no platform owner exists yet; run stacklane setup | Create the owner account. |
cloud.simulate is enabled ..., billing.simulate is enabled ... | Simulate mode is on. Never use it in production. |
marked tasks interrupted by a restart as failed | Tasks were running when Stacklane stopped. See Background workers. |
task failed | A provisioning task failed. The line shows the task, its kind and the error. |
mail delivery failed | The SMTP server did not accept a message. |
insufficient credit for automatic renewal | A domain could not be renewed from the customer's credit. |
no renewal price configured | A domain could not be renewed because its extension has no price. |
attack reported | The DDoS detector reported an attack. |
removed expired sessions and tokens | Hourly cleanup. |
Backups#
What to back up#
| File | Why |
|---|---|
The database, /var/lib/stacklane/stacklane.db | All data. Back it up with stacklane backup, not by copying the file. |
The secret key, /etc/stacklane/secret.key | Needed to decrypt stored secrets and two-factor secrets. |
The configuration, /etc/stacklane/stacklane.toml | Your settings. |
Keep the secret key with the database backups, but store both in a safe place. Anyone with the database and the key can read all stored credentials.
A backup of the database without the key is not enough. Without the key, Stacklane cannot decrypt the Proxmox tokens, panel credentials, registrar and payment provider keys, or the two-factor secrets of your users.
Make a backup#
stacklane backup writes a consistent copy of the database to a new file. You can run it while Stacklane is running.
sudo -u stacklane mkdir -p /var/lib/stacklane/backups
sudo -u stacklane stacklane backup /var/lib/stacklane/backups/stacklane-$(date +%Y%m%d%H%M%S).db
The target file must not exist. The stacklane user must be able to write to the directory.
Do not copy stacklane.db with cp or rsync while the service runs. The database is in WAL mode, and recent changes can still be in stacklane.db-wal.
Automatic backups#
This example makes a backup every night at 03:00 and keeps 14 days. Create /etc/cron.d/stacklane-backup:
0 3 * * * stacklane /usr/bin/stacklane backup /var/lib/stacklane/backups/stacklane-$(date +\%Y\%m\%d).db && find /var/lib/stacklane/backups -name 'stacklane-*.db' -mtime +14 -delete
Copy the backups to another machine as well. A backup on the same disk does not help when the disk fails.
Backups before upgrades#
When you upgrade the package, it makes a backup first:
/var/lib/stacklane/backups/stacklane-<old version>-<date and time>.db
If this backup fails, the package prints a warning and continues the upgrade. These files are not deleted automatically.
Restore a backup#
-
Stop Stacklane:
sudo systemctl stop stacklane -
Move the current database away:
sudo mv /var/lib/stacklane/stacklane.db /var/lib/stacklane/stacklane.db.old sudo rm -f /var/lib/stacklane/stacklane.db-wal /var/lib/stacklane/stacklane.db-shm -
Copy the backup in place with the right owner and mode:
sudo install -o stacklane -g stacklane -m 0640 \ /var/lib/stacklane/backups/stacklane-20261121.db /var/lib/stacklane/stacklane.db -
Make sure
/etc/stacklane/secret.keyis the key that belongs to this backup. -
Start Stacklane. It applies any missing migrations before it starts:
sudo systemctl start stacklane -
Check the result:
sudo -u stacklane stacklane check
You can restore a backup from an older version into a newer version. The migrations update it. Do not restore a backup made by a newer version into an older version.
Upgrades#
Install the new package with apt:
sudo apt install ./stacklane_0.1.1-1_amd64.deb
During the upgrade:
- The package backs up the database to
/var/lib/stacklane/backups/. - The new files are installed. Your changes in
/etc/stacklane/stacklane.tomlare kept. - The post-install script runs
stacklane migrate. - The service restarts. The unit runs
stacklane migrateagain before it starts, which does nothing when the database is up to date.
After the upgrade, check the service:
sudo systemctl status stacklane
sudo -u stacklane stacklane check
stacklane --version
Restarting stops tasks that are running at that moment. See Background workers. Upgrade at a quiet time.
Migrations#
Migrations are built into the binary. They run in order and each runs once. You do not need to run them by hand, but you can:
sudo -u stacklane stacklane migrate
Removing Stacklane#
# Remove the program, keep configuration and data
sudo apt remove stacklane
# Also remove configuration, the secret key and the stacklane user
sudo apt purge stacklane
apt purge deletes /etc/stacklane/secret.key. Back up the key first if you may need the data again. The data in /var/lib/stacklane is never deleted automatically. Remove it by hand when you no longer need it.
Background workers#
Stacklane runs these workers inside the same process as the web server:
| Worker | What it does | When it runs |
|---|---|---|
| Instances | Creates and changes cloud servers on Proxmox VE: create, power actions, firewall rules, snapshots, volumes, delete. | When a task is queued, and every 3 seconds when idle |
| Services | Creates, suspends, unsuspends and terminates accounts on cPanel/WHM, DirectAdmin and Pterodactyl. | When a task is queued, and every 3 seconds when idle |
| Domains | Registers and renews domains and updates nameservers. Renews domains from credit when they get close to expiry. | Tasks as above. The renewal check runs at start and then every hour. |
| Billing | Records usage, issues monthly invoices, sends payment reminders, suspends overdue accounts and restores them after payment. | At start and then every 5 minutes |
| Maintenance | Removes expired sessions, email tokens and two-factor challenges, and optimizes the database. | At start and then every hour |
Tasks and restarts#
Long-running work goes through a task queue in the database. The API records the task and returns at once. A worker picks up the task and runs it.
When Stacklane stops, tasks that are still running are not finished. At the next start, Stacklane marks every task that was still running as failed with the error interrupted by a restart. It does not retry them automatically, because the action may be half done on the remote system. Tasks that were waiting and had not started yet run normally after the restart.
After a restart with interrupted tasks:
- Look for
marked tasks interrupted by a restart as failedin the log. - Check the affected servers, services or domains on the remote system.
- Start the action again if needed.
Usage metering and downtime#
The billing worker records usage for the current hour each time it runs. Hours in which Stacklane is not running at all are not recorded. Keep downtime short.
One process per database#
Run only one Stacklane process per database. Rate limits and cached platform settings are kept in the memory of the process.
Health checks#
stacklane check#
stacklane check checks the secret key, the database, the platform owner and the email configuration:
sudo -u stacklane stacklane check
It exits with status 0 when everything is fine and 1 when there is a problem, so you can use it in scripts. See Command line for all messages.
HTTP health endpoint#
GET /api/health returns 204 No Content when Stacklane can reach its database, and 503 Service Unavailable when it cannot. Use it for your monitoring or load balancer:
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/api/health
204
What to monitor#
- the
/api/healthendpoint systemctl is-active stacklane- free disk space in
/var/lib/stacklane - the age of the newest backup
task failed,mail delivery failedandbilling run failedin the log
Troubleshooting#
The service does not start#
Look at the log first:
sudo journalctl -u stacklane -n 50 --no-pager
| Error | Cause and fix |
|---|---|
configuration file ... not found | The file does not exist. Check the path in STACKLANE_CONFIG or --config. |
invalid configuration in ...: unknown field ... | A key name is wrong. See Configuration. |
server.public_url must use http or https | Fix public_url. |
security.access_token_minutes must be between 1 and 60 | Fix the value. |
security.session_days must be between 1 and 365 | Fix the value. |
secret key missing; run stacklane keygen first | Run sudo -u stacklane stacklane keygen. Check that stacklane can read the key file. |
secret key file must contain 32 bytes encoded as 64 hex characters | The key file is damaged. Restore it from your backup. |
cannot open database at ... | Check that the database is in /var/lib/stacklane and owned by stacklane. |
mail.from is not a valid address | Fix mail.from. |
mail.smtp_url is invalid | Fix mail.smtp_url. |
| An error about the address being in use | Another program uses the port in server.listen. |
Database owned by root#
If you ran a stacklane command as root on a new system, the database files can belong to root. Fix the owner:
sudo systemctl stop stacklane
sudo chown stacklane:stacklane /var/lib/stacklane/stacklane.db*
sudo systemctl start stacklane
Users are signed out after a few minutes#
The session refresh fails. Stacklane only accepts a refresh when the Origin header of the browser is the same as server.public_url. Check that public_url matches the address in the browser exactly, including https and the port. Check that your proxy does not change the Origin header.
Everyone gets "too many requests"#
Stacklane is behind a proxy, but server.behind_proxy is false. Every request then seems to come from the proxy, and all users share one rate limit. Set behind_proxy = true and restart.
Rate limits reset after a restart#
This is expected. Rate limits are kept in memory.
Emails do not arrive#
- Run
sudo -u stacklane stacklane check. If it showswarn smtp not configured, setmail.smtp_url. - Look for
mail delivery failedin the log. The line shows the SMTP error. - Check that the server can reach the SMTP server on the port in the URL.
- Check that
mail.fromuses an address your SMTP server allows you to send from.
The page shows "The web interface is not included in this build."#
The binary was built without the frontend. Build the frontend before the backend. See Development.
Requests fail with 503#
Stacklane answers with 503 Service Unavailable when more than 1024 requests are open at the same time, or when /api/health cannot reach the database. Check the log and the load on the server.
Requests fail with 413 or 408#
The request body was larger than 256 KiB (413), or the request took longer than 30 seconds (408).
Payments are not confirmed#
Payment providers send webhooks to <public_url>/api/billing/v1/webhooks/<provider>. Check that this address is reachable from the internet and that your proxy does not block it.
Tasks fail with "interrupted by a restart"#
Stacklane was stopped while the task was running. Check the resource on the remote system and start the action again. See Tasks and restarts.