Proxmox automation at scale: what we learned

Proxmox VE is wonderful software and a challenging API. Automating it in a multi-tenant SaaS taught us a few things the docs don’t cover.

The API is cluster-aware, your code might not be

Every Proxmox call targets a node. If you cache “VM 101 lives on pve-01” and the admin live-migrates it to pve-02, the next call fails with “VM not found on node pve-01” — not a useful 404.

Our fix: never cache the (vmid, node) pair. Before any destructive operation, call /cluster/resources?type=vm and look up the current node. One extra API call per operation, zero silent failures.

async def get_vm_node(vmid: int) -> str:
    resources = await proxmox_client.get("/cluster/resources", type="vm")
    for r in resources:
        if r["vmid"] == vmid:
            return r["node"]
    raise VmNotFound(vmid)

Lesson: treat Proxmox as eventually consistent. The authoritative source is /cluster/resources, not your database.

Cloud-init is fiddly

Proxmox exposes cloud-init via the ide2 drive on each VM. The usual flow:

1. Clone from a cloud-image template
2. Set ciuser, cipassword, sshkeys, ipconfig0
3. Regenerate the cloud-init drive
4. Start the VM

The failure mode we hit repeatedly: cloud-init drive not regenerated. The VM starts with stale data from the template (wrong hostname, no SSH key, old network config). Symptoms range from “the VM is unreachable” to “the VM is someone else’s”.

The fix is calling PUT /nodes/{node}/qemu/{vmid}/config with an empty change right before start — Proxmox detects the ide2 drive is dirty and regenerates it. Undocumented, but it works. (Newer Proxmox versions do this automatically — check your version.)

Lesson: read the cloud-init log inside the VM at least once during your integration testing. If you see stale values, the drive wasn’t regenerated.

Bandwidth measurement lies

Proxmox exposes network RRD data: netin, netout bytes per second. For billing, you want bytes per billing cycle.

Don’t subtract current from last-hour — Proxmox resets counters on reboot, migration, and sometimes for no reason we could identify. Subtract naively and you get negative numbers, which make accountants unhappy.

Our approach: snapshot netin/netout every 5 minutes, detect decreases (reboot/migration), and only sum positive deltas. Store the running total per billing cycle in Postgres. It’s three tables and a Celery beat job.

def accumulate_bandwidth(vm: VpsInstance, sample: ProxmoxRRDSample):
    last = vm.last_bw_sample
    if not last or sample.netin < last.netin:  # reset detected
        vm.cycle_netin_bytes += sample.netin
    else:
        vm.cycle_netin_bytes += sample.netin - last.netin
    vm.last_bw_sample = sample

Lesson: your billing telemetry is adversarial. Assume the source is lying unless proven otherwise.

SSL pinning for API tokens

Proxmox’s web UI ships a self-signed cert by default. If you talk to Proxmox over HTTPS without verification, you’re one DNS trick from a full compromise. If you enable verification, every Proxmox admin has to install a “real” cert.

We split the difference: during the “Add Proxmox server” wizard, we fetch the cert, show the admin its SHA-256 fingerprint, and save it to the database. All subsequent calls verify against the pinned fingerprint. New cert? The admin has to re-approve. It’s what SSH has done for thirty years and it works.

The overall pattern

Every Proxmox automation we ship follows the same pattern:

1. Validate against /cluster/resources (current state, not cached)
2. Mutate with idempotent calls (regenerate cloud-init, always explicit)
3. Poll until the task finishes (/tasks/{upid}/status)
4. Verify the outcome (config actually changed, VM actually running)
5. Log everything in the ActivityLog with full actor + params

It’s verbose. It’s also the only thing that works under the real failure modes — migrations mid-operation, API flakes, tasks that hang, networks that partition.

If you run Proxmox at scale and have your own war stories, we’d love to swap notes. Get in touch.