Cloud
Stacklane creates cloud instances on Proxmox VE. Each region is one Proxmox cluster. Customers choose a region, an image (operating system) and a flavor (size), and Stacklane clones a template on the cluster. Only staff with the owner or admin role can manage the cloud settings.
Before you start#
You need:
- a Proxmox VE cluster that Stacklane can reach over HTTPS, usually on port 8006;
- an API token for that cluster;
- a storage where Stacklane can create disks;
- a network bridge for the instances;
- one cloud-init template VM per operating system you want to offer;
- a range of public IPv4 addresses with a gateway.
Proxmox API token#
Stacklane signs in with a Proxmox API token. It sends the header Authorization: PVEAPIToken=<token ID>=<secret>. The token ID has the form user@realm!tokenname, for example stacklane@pve!panel. Stacklane rejects a token ID without @ and !.
What the token must be allowed to do#
Give the token enough permissions for every call in this table, on the nodes, the storage, the bridge and the VMs that Stacklane manages.
| Stacklane action | Proxmox API calls |
|---|---|
| Test connection | GET /version, GET /nodes |
| Choose a node | GET /nodes (status and free memory) |
| Find the template | GET /cluster/resources?type=vm |
| Reserve a VM ID | GET /cluster/nextid |
| Create an instance | POST /nodes/{node}/qemu/{template}/clone (full clone to the region storage and target node), POST /nodes/{node}/qemu/{vmid}/config, PUT /nodes/{node}/qemu/{vmid}/resize |
| Power actions | POST /nodes/{node}/qemu/{vmid}/status/{start,stop,shutdown,reboot}, GET .../status/current |
| Wait for a task | GET /nodes/{node}/tasks/{upid}/status |
| Firewall | PUT .../firewall/options, GET, POST and DELETE .../firewall/rules |
| Monitoring graphs | GET /nodes/{node}/qemu/{vmid}/rrddata |
| Snapshots | POST .../snapshot, POST .../snapshot/{name}/rollback, DELETE .../snapshot/{name} |
| Volumes | GET and PUT /nodes/{node}/qemu/{vmid}/config (add and remove scsiN disks) |
| Delete an instance | POST .../status/stop, DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1 |
Stacklane waits up to 15 minutes for a Proxmox task. After that the task fails with "the operation timed out".
TLS#
Keep Verify the TLS certificate of the cluster on when the cluster has a valid certificate. Turn it off only when the cluster uses a self-signed certificate on a network you trust.
Templates#
Stacklane creates each instance as a full clone of a template. Prepare one template VM per image in each region:
- Install the operating system with cloud-init support.
- Install the QEMU guest agent. Stacklane turns on the guest agent option (
agent=1) on every instance. - Put the boot disk on
scsi0. Stacklane resizesscsi0to the disk size of the flavor. Proxmox cannot make a disk smaller, so keep the template disk as small as, or smaller than, your smallest flavor. - Add a cloud-init drive.
- Convert the VM to a template in Proxmox.
- Note the VM ID of the template.
The template can be on any node of the cluster. Stacklane finds it in the cluster resources.
When Stacklane creates an instance, it sets these options on the clone:
| Option | Value |
|---|---|
cores | The vCPUs of the flavor. |
memory | The memory of the flavor in MiB. |
ciuser | The default user of the image. |
cipassword | The password the customer entered, if any. |
sshkeys | The SSH keys the customer selected. |
net0 | virtio,bridge=<region bridge>,firewall=1 |
ipconfig0 | ip=<address>/<prefix>,gw=<gateway> from the IPv4 pool, or ip=dhcp when the pool has no free address. |
nameserver | The nameservers of the region. |
onboot | 1 |
agent | 1 |
Then Stacklane starts the instance.
Regions#
Open Regions in the Cloud group and click Add region.
| Field | Rules and meaning |
|---|---|
| Name | Required. Up to 60 characters. Customers see this name. |
| Slug | Required. Lowercase letters, digits and hyphens, up to 40 characters. Must be unique. |
| Country | Two-letter country code. Customers see it next to the region name. |
| API endpoint | The URL of the cluster, for example https://pve.example.net:8006. |
| API token ID | For example stacklane@pve!panel. |
| API token secret | Required for a new region. Leave it empty to keep the current secret. When you change the endpoint to another host, you must enter the secret again. |
| Storage | The Proxmox storage for instance disks and volumes, for example local-lvm. |
| Network bridge | The bridge for the instance network, for example vmbr0. |
| Preferred node | Optional. Leave it empty to place each instance on the online node with the most free memory. When you set a node, all instances go to that node. |
| Nameservers | IP addresses, separated by spaces, that instances use as DNS resolvers. Default: 1.1.1.1 9.9.9.9. |
| Verify the TLS certificate of the cluster | On by default. |
| Status | active or disabled. |
Click Save. Then click Test connection under Cluster health. Stacklane shows the Proxmox version and the nodes with their CPU and memory use. If the test fails, the error message from Proxmox is shown.
When a region is disabled, customers do not see it and cannot create new instances in it. Existing instances in the region keep working.
Image templates#
On the region page, under Image templates, enter the template VM ID for each image you offer in this cluster. The VM ID must be between 100 and 999,999,999. Leave the field empty to stop offering the image in this region.
Customers only see the images that have a template in the selected region.
IPv4 pool#
Under IPv4 pool, add the public addresses that Stacklane gives to instances.
-
In Addresses, enter single addresses or ranges, separated by commas or new lines. For example:
203.0.113.10-203.0.113.50 203.0.113.60, 203.0.113.61 -
Enter the Prefix length, from 8 to 32, for example
24. -
Enter the Gateway.
-
Click Add addresses.
You can add at most 4,096 addresses in one step. The page shows how many addresses are assigned. You can only remove an address that is not assigned to an instance.
Each new instance gets the first free address of the pool. When an instance is deleted, or its creation fails, the address becomes free again. When the pool is empty, the instance is created with DHCP and the portal shows no public address.
Images#
Open Images in the Cloud group. An image describes an operating system. You link it to a template per region on the region page.
| Field | Rules and meaning |
|---|---|
| Name | Required. Up to 60 characters, for example Ubuntu 24.04. |
| Slug | Required. Lowercase letters, digits and hyphens. Must be unique. |
| Family | Lowercase letters, digits and hyphens, for example ubuntu. |
| Version | Required. Up to 30 characters, for example 24.04. |
| Default user | The cloud-init user, for example ubuntu or debian. Lowercase letters, digits, _ and -, up to 32 characters. |
| Status | active or disabled. Customers only see active images. |
| Sort order | A number that sets the order in lists. |
Flavors#
Open Flavors in the Cloud group. A flavor is an instance size with a monthly price.
| Field | Rules |
|---|---|
| Name | Required. Up to 60 characters. |
| Slug | Required. Lowercase letters, digits and hyphens. Must be unique. |
| vCPUs | 1 to 256. |
| Memory (MiB) | 256 to 4,194,304. |
| Disk (GB) | 1 to 65,536. |
| Monthly price | 0 to 1,000,000.00, without VAT. |
| Currency | Must be the billing currency. See Billing. |
| Status | active or disabled. Customers can only choose active flavors. |
| Sort order | A number that sets the order in lists. |
Customers see the monthly price, the hourly price (monthly price ÷ 730) and the monthly price including their VAT rate. Usage is billed per hour. See Billing.
Limits and volume price#
Open Limits in the Cloud group. These limits apply to every organization.
| Field | Default | Range | Meaning |
|---|---|---|---|
| Maximum instances | 10 | 0 to 10,000 | Instances per organization. |
| Maximum vCPUs | 40 | 0 to 100,000 | Total vCPUs of all instances of an organization. |
| Maximum memory (MiB) | 81,920 | 0 to 1,000,000,000 | Total memory of all instances of an organization. |
| Snapshots per server | 3 | 0 to 50 | Snapshots per instance. |
| Volumes per server | 4 | 0 to 16 | Extra volumes per instance. |
| Volume price (cents per GB per month) | 10 | 0 to 100,000 | Without VAT, billed per hour. |
Every instance that is not deleted counts toward the quota, also an instance with an error. When a new instance would go above a limit, the customer gets the error quota_exceeded. Customers see their usage and limits when they create an instance.
Firewall, snapshots and volumes#
Customers manage these features themselves in the portal. This section explains what happens on the cluster.
Firewall#
When a customer saves firewall rules, Stacklane replaces all rules of the VM on Proxmox and turns on the VM firewall with these defaults:
- inbound policy:
DROP; - outbound policy:
ACCEPT.
So inbound traffic that matches no rule is blocked, and outbound traffic is allowed unless a rule blocks it. A customer can have up to 50 rules per instance. Each rule has a direction (in or out), an action (accept or drop), and optionally a protocol (tcp, udp or icmp), a port or port range (for TCP and UDP only), a source address or network, and a comment.
Snapshots#
A snapshot is a Proxmox VM snapshot. Customers can take a snapshot, restore it and delete it. The number of snapshots per instance is limited by Snapshots per server. Snapshots are not billed separately.
Volumes#
A volume is an extra disk on the region storage. Stacklane adds it as scsi1, scsi2 and so on. A volume is 1 to 4,096 GB. The number of volumes per instance is limited by Volumes per server. When a customer deletes a volume, Stacklane removes the disk from the VM and deletes it from the storage.
How provisioning works#
Changes to instances are not made during the API request. Stacklane puts a task in a queue and answers with 202 Accepted. A background worker runs the tasks one after the other.
- The customer creates an instance. Stacklane checks the quota, saves the instance with the status
creatingand adds aninstance.createtask. - The worker takes the task. It reserves an address from the pool, clones the template, sets the options, resizes the disk and starts the VM.
- On success the instance gets the status
running. On failure it gets the statuserror, and the address goes back to the pool.
Other actions work the same way. Each one is a task of its own kind:
| Task kind | Started by |
|---|---|
instance.create | Creating an instance. |
instance.power | Start, power off, shut down, reboot. Also used by Stacklane to power off instances of overdue accounts. |
instance.delete | Deleting an instance. |
instance.firewall | Saving firewall rules. |
instance.snapshot | Taking, restoring or deleting a snapshot. |
instance.volume | Adding or deleting a volume. |
A task has the state pending, running, finished or error. An instance can only run one task at a time. A second action while a task is pending or running returns 409 with the code busy.
When a task fails#
Stacklane does not retry failed tasks.
Find the error#
- The customer sees the failed task and its error message on the instance page.
- The service log contains a line
task failedwith the task kind and the error. See Operations for how to read the log. - Through the API,
GET /api/cloud/v1/taskslists the tasks of an organization with theirerrorfield.
Common errors#
| Error | Meaning | What to do |
|---|---|---|
cannot reach the hypervisor: ... | Stacklane cannot connect to the API endpoint. | Check the endpoint, the firewall between Stacklane and the cluster, and the TLS setting. Use Test connection. |
the hypervisor rejected the request: 401 ... or 403 ... | The token is wrong or misses a permission. | Check the token ID, the secret and the token's permissions. |
template <id> was not found in this region | No VM with that ID is marked as a template in the cluster. | Fix the template VM ID on the region page. |
image is not available in this region | The image has no template in the region. | Add the template VM ID on the region page. |
no node has enough free capacity | No online node has enough free memory, or the preferred node is offline. | Add capacity, or clear or change the preferred node. |
task failed on the hypervisor: ... | A Proxmox task ended with an error, for example a failed clone. | Look at the task log in Proxmox for the details. |
the operation timed out | A Proxmox task took more than 15 minutes. | Check the cluster and the storage. |
interrupted by a restart | Stacklane was restarted while the task was running. | Check the state of the VM in Proxmox and repeat the action. |
region credentials cannot be decrypted | The secret key of the server changed. | Restore the original secret key, or enter the token secret again. |
Repair the instance#
- Failed creation. After you fix the cause, the customer deletes the instance with the status
errorand creates it again. Deleting an instance that was never placed on a node only frees its address. - Failed power action, firewall or snapshot. Fix the cause and repeat the action.
- Failed snapshot. A snapshot that failed has the status
error. The customer can delete it. - Failed volume. A volume that failed has the status
error. The customer can delete it. - Failed deletion. The instance gets the status
error. Fix the cause and delete it again.
For testing without a Proxmox cluster, cloud.simulate = true in the configuration file uses a built-in simulator. See Configuration.
API endpoints#
All endpoints need the owner or admin staff role.
| Method | Path | Description |
|---|---|---|
GET, POST | /api/admin/cloud/regions | List or create regions. |
GET, PATCH | /api/admin/cloud/regions/{id} | Read or update a region. |
POST | /api/admin/cloud/regions/{id}/check | Test the connection to the cluster. |
GET, PUT | /api/admin/cloud/regions/{id}/images | List or set template VM IDs. Body for PUT: {"image_id": 1, "template_vmid": 9000}. Use null to remove. |
GET, POST | /api/admin/cloud/regions/{id}/addresses | List or add pool addresses. Body for POST: {"addresses": "...", "prefix": 24, "gateway": "203.0.113.1"}. |
DELETE | /api/admin/cloud/regions/{id}/addresses/{address} | Remove a free address by its ID. |
GET, POST | /api/admin/cloud/flavors | List or create flavors. |
PATCH | /api/admin/cloud/flavors/{id} | Update a flavor. |
GET, POST | /api/admin/cloud/images | List or create images. |
PATCH | /api/admin/cloud/images/{id} | Update an image. |
GET, PUT | /api/admin/cloud/settings | Read or save limits and the volume price. |