Operations

This page describes how to run Stacklane in production: the service, logs, backups, upgrades, background work, health checks and common problems.

The systemd service#

The package installs stacklane.service. It is enabled and started during installation.

sudo systemctl status stacklane
sudo systemctl restart stacklane
sudo systemctl stop stacklane

What the unit does#

SettingValueEffect
User, GroupstacklaneRuns as the unprivileged stacklane user.
EnvironmentSTACKLANE_CONFIG=/etc/stacklane/stacklane.tomlConfiguration file.
ExecStartPrestacklane migrateApplies database migrations before every start.
ExecStartstacklane serveRuns the server.
Restarton-failure, after 3 secondsRestarts Stacklane if it exits with an error.
TimeoutStopSec30Time Stacklane gets to stop cleanly.
StateDirectorystacklane, mode 0750Creates /var/lib/stacklane.
UMask0027New files are not readable by other users.
LimitNOFILE65536Maximum number of open files and connections.

Sandbox#

The unit uses the systemd sandbox to limit what the process can do:

SettingEffect
ProtectSystem=strictThe file system is read-only for the service, except for the paths named in ReadWritePaths.
ReadWritePaths=/var/lib/stacklaneThe only writable directory. The database must be here.
ProtectHome=yesHome directories are not visible.
PrivateTmp=yes, PrivateDevices=yesPrivate /tmp and no access to physical devices.
NoNewPrivileges=yesThe process and its children cannot gain privileges.
ProtectKernelTunables, ProtectKernelModules, ProtectKernelLogs, ProtectControlGroups, ProtectClock, ProtectHostnameNo changes to kernel settings, modules, logs, cgroups, the clock or the host name.
ProtectProc=invisibleProcesses of other users are not visible.
RestrictNamespaces, RestrictRealtime, RestrictSUIDSGID, LockPersonality, MemoryDenyWriteExecuteNo namespaces, real-time scheduling, set-user-ID files, personality changes or writable executable memory.
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIXOnly IPv4, IPv6 and Unix sockets.
SystemCallArchitectures=native, SystemCallFilter=@system-service ~@privilegedOnly the system calls a normal service needs.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE, AmbientCapabilities=CAP_NET_BIND_SERVICEThe only capability is binding to ports below 1024.

Because of ProtectSystem=strict, the service cannot write to /etc/stacklane. Create the secret key with stacklane keygen before the first start. The package does this for you.

Changing the unit#

Do not edit the installed unit file. An upgrade replaces it. Use a drop-in file instead:

sudo systemctl edit stacklane

Add your settings in the [Service] section, save, and restart:

[Service]
Environment=STACKLANE_LOG=info,tower_http=debug
sudo systemctl restart stacklane

Logs#

Stacklane writes its logs to standard error. systemd stores them in the journal.

# Follow the log
sudo journalctl -u stacklane -f

# Logs since the last boot
sudo journalctl -u stacklane -b

# Logs of the last hour
sudo journalctl -u stacklane --since "1 hour ago"

# Only warnings and errors
sudo journalctl -u stacklane -p warning

Log level#

The STACKLANE_LOG environment variable sets the log filter. The default is info,tower_http=warn,sqlx=warn.

ValueEffect
info,tower_http=warn,sqlx=warnDefault. Start, stop, tasks, warnings and errors.
info,tower_http=debugAlso log every HTTP request.
debugDetailed logs from all components.

Set it for the service with a drop-in file, as shown in Changing the unit. For a single command, set it on the command line:

sudo -u stacklane env STACKLANE_LOG=debug stacklane check

Messages to know#

MessageMeaning
listeningThe server has started. The log line shows the address and the public URL.
mail.smtp_url is not set; outgoing email is written to the logEmail is not sent. Each message is written to the log instead.
no platform owner exists yet; run stacklane setupCreate the owner account.
cloud.simulate is enabled ..., billing.simulate is enabled ...Simulate mode is on. Never use it in production.
marked tasks interrupted by a restart as failedTasks were running when Stacklane stopped. See Background workers.
task failedA provisioning task failed. The line shows the task, its kind and the error.
mail delivery failedThe SMTP server did not accept a message.
insufficient credit for automatic renewalA domain could not be renewed from the customer's credit.
no renewal price configuredA domain could not be renewed because its extension has no price.
attack reportedThe DDoS detector reported an attack.
removed expired sessions and tokensHourly cleanup.

Backups#

What to back up#

FileWhy
The database, /var/lib/stacklane/stacklane.dbAll data. Back it up with stacklane backup, not by copying the file.
The secret key, /etc/stacklane/secret.keyNeeded to decrypt stored secrets and two-factor secrets.
The configuration, /etc/stacklane/stacklane.tomlYour settings.

Keep the secret key with the database backups, but store both in a safe place. Anyone with the database and the key can read all stored credentials.

A backup of the database without the key is not enough. Without the key, Stacklane cannot decrypt the Proxmox tokens, panel credentials, registrar and payment provider keys, or the two-factor secrets of your users.

Make a backup#

stacklane backup writes a consistent copy of the database to a new file. You can run it while Stacklane is running.

sudo -u stacklane mkdir -p /var/lib/stacklane/backups
sudo -u stacklane stacklane backup /var/lib/stacklane/backups/stacklane-$(date +%Y%m%d%H%M%S).db

The target file must not exist. The stacklane user must be able to write to the directory.

Do not copy stacklane.db with cp or rsync while the service runs. The database is in WAL mode, and recent changes can still be in stacklane.db-wal.

Automatic backups#

This example makes a backup every night at 03:00 and keeps 14 days. Create /etc/cron.d/stacklane-backup:

0 3 * * * stacklane /usr/bin/stacklane backup /var/lib/stacklane/backups/stacklane-$(date +\%Y\%m\%d).db && find /var/lib/stacklane/backups -name 'stacklane-*.db' -mtime +14 -delete

Copy the backups to another machine as well. A backup on the same disk does not help when the disk fails.

Backups before upgrades#

When you upgrade the package, it makes a backup first:

/var/lib/stacklane/backups/stacklane-<old version>-<date and time>.db

If this backup fails, the package prints a warning and continues the upgrade. These files are not deleted automatically.

Restore a backup#

  1. Stop Stacklane:

    sudo systemctl stop stacklane
    
  2. Move the current database away:

    sudo mv /var/lib/stacklane/stacklane.db /var/lib/stacklane/stacklane.db.old
    sudo rm -f /var/lib/stacklane/stacklane.db-wal /var/lib/stacklane/stacklane.db-shm
    
  3. Copy the backup in place with the right owner and mode:

    sudo install -o stacklane -g stacklane -m 0640 \
      /var/lib/stacklane/backups/stacklane-20261121.db /var/lib/stacklane/stacklane.db
    
  4. Make sure /etc/stacklane/secret.key is the key that belongs to this backup.

  5. Start Stacklane. It applies any missing migrations before it starts:

    sudo systemctl start stacklane
    
  6. Check the result:

    sudo -u stacklane stacklane check
    

You can restore a backup from an older version into a newer version. The migrations update it. Do not restore a backup made by a newer version into an older version.

Upgrades#

Install the new package with apt:

sudo apt install ./stacklane_0.1.1-1_amd64.deb

During the upgrade:

  1. The package backs up the database to /var/lib/stacklane/backups/.
  2. The new files are installed. Your changes in /etc/stacklane/stacklane.toml are kept.
  3. The post-install script runs stacklane migrate.
  4. The service restarts. The unit runs stacklane migrate again before it starts, which does nothing when the database is up to date.

After the upgrade, check the service:

sudo systemctl status stacklane
sudo -u stacklane stacklane check
stacklane --version

Restarting stops tasks that are running at that moment. See Background workers. Upgrade at a quiet time.

Migrations#

Migrations are built into the binary. They run in order and each runs once. You do not need to run them by hand, but you can:

sudo -u stacklane stacklane migrate

Removing Stacklane#

# Remove the program, keep configuration and data
sudo apt remove stacklane

# Also remove configuration, the secret key and the stacklane user
sudo apt purge stacklane

apt purge deletes /etc/stacklane/secret.key. Back up the key first if you may need the data again. The data in /var/lib/stacklane is never deleted automatically. Remove it by hand when you no longer need it.

Background workers#

Stacklane runs these workers inside the same process as the web server:

WorkerWhat it doesWhen it runs
InstancesCreates and changes cloud servers on Proxmox VE: create, power actions, firewall rules, snapshots, volumes, delete.When a task is queued, and every 3 seconds when idle
ServicesCreates, suspends, unsuspends and terminates accounts on cPanel/WHM, DirectAdmin and Pterodactyl.When a task is queued, and every 3 seconds when idle
DomainsRegisters and renews domains and updates nameservers. Renews domains from credit when they get close to expiry.Tasks as above. The renewal check runs at start and then every hour.
BillingRecords usage, issues monthly invoices, sends payment reminders, suspends overdue accounts and restores them after payment.At start and then every 5 minutes
MaintenanceRemoves expired sessions, email tokens and two-factor challenges, and optimizes the database.At start and then every hour

Tasks and restarts#

Long-running work goes through a task queue in the database. The API records the task and returns at once. A worker picks up the task and runs it.

When Stacklane stops, tasks that are still running are not finished. At the next start, Stacklane marks every task that was still running as failed with the error interrupted by a restart. It does not retry them automatically, because the action may be half done on the remote system. Tasks that were waiting and had not started yet run normally after the restart.

After a restart with interrupted tasks:

  1. Look for marked tasks interrupted by a restart as failed in the log.
  2. Check the affected servers, services or domains on the remote system.
  3. Start the action again if needed.

Usage metering and downtime#

The billing worker records usage for the current hour each time it runs. Hours in which Stacklane is not running at all are not recorded. Keep downtime short.

One process per database#

Run only one Stacklane process per database. Rate limits and cached platform settings are kept in the memory of the process.

Health checks#

stacklane check#

stacklane check checks the secret key, the database, the platform owner and the email configuration:

sudo -u stacklane stacklane check

It exits with status 0 when everything is fine and 1 when there is a problem, so you can use it in scripts. See Command line for all messages.

HTTP health endpoint#

GET /api/health returns 204 No Content when Stacklane can reach its database, and 503 Service Unavailable when it cannot. Use it for your monitoring or load balancer:

curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/api/health
204

What to monitor#

Troubleshooting#

The service does not start#

Look at the log first:

sudo journalctl -u stacklane -n 50 --no-pager
ErrorCause and fix
configuration file ... not foundThe file does not exist. Check the path in STACKLANE_CONFIG or --config.
invalid configuration in ...: unknown field ...A key name is wrong. See Configuration.
server.public_url must use http or httpsFix public_url.
security.access_token_minutes must be between 1 and 60Fix the value.
security.session_days must be between 1 and 365Fix the value.
secret key missing; run stacklane keygen firstRun sudo -u stacklane stacklane keygen. Check that stacklane can read the key file.
secret key file must contain 32 bytes encoded as 64 hex charactersThe key file is damaged. Restore it from your backup.
cannot open database at ...Check that the database is in /var/lib/stacklane and owned by stacklane.
mail.from is not a valid addressFix mail.from.
mail.smtp_url is invalidFix mail.smtp_url.
An error about the address being in useAnother program uses the port in server.listen.

Database owned by root#

If you ran a stacklane command as root on a new system, the database files can belong to root. Fix the owner:

sudo systemctl stop stacklane
sudo chown stacklane:stacklane /var/lib/stacklane/stacklane.db*
sudo systemctl start stacklane

Users are signed out after a few minutes#

The session refresh fails. Stacklane only accepts a refresh when the Origin header of the browser is the same as server.public_url. Check that public_url matches the address in the browser exactly, including https and the port. Check that your proxy does not change the Origin header.

Everyone gets "too many requests"#

Stacklane is behind a proxy, but server.behind_proxy is false. Every request then seems to come from the proxy, and all users share one rate limit. Set behind_proxy = true and restart.

Rate limits reset after a restart#

This is expected. Rate limits are kept in memory.

Emails do not arrive#

  1. Run sudo -u stacklane stacklane check. If it shows warn smtp not configured, set mail.smtp_url.
  2. Look for mail delivery failed in the log. The line shows the SMTP error.
  3. Check that the server can reach the SMTP server on the port in the URL.
  4. Check that mail.from uses an address your SMTP server allows you to send from.

The page shows "The web interface is not included in this build."#

The binary was built without the frontend. Build the frontend before the backend. See Development.

Requests fail with 503#

Stacklane answers with 503 Service Unavailable when more than 1024 requests are open at the same time, or when /api/health cannot reach the database. Check the log and the load on the server.

Requests fail with 413 or 408#

The request body was larger than 256 KiB (413), or the request took longer than 30 seconds (408).

Payments are not confirmed#

Payment providers send webhooks to <public_url>/api/billing/v1/webhooks/<provider>. Check that this address is reachable from the internet and that your proxy does not block it.

Tasks fail with "interrupted by a restart"#

Stacklane was stopped while the task was running. Check the resource on the remote system and start the action again. See Tasks and restarts.