05 - Linstor & Storage Deep Dive

LVM Global Filter

To prevent PVE from scanning DRBD and Linstor mapper devices as additional LVM physical volumes (which causes duplicate signature warnings and can confuse vgscan), the playbook injects a global filter into /etc/lvm/lvm.conf:

global_filter = [ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]

This is applied on all nodes before any VG or pool creation.

Disk Wipe (Optional)

When wipe_linstor_disks: true, the playbook runs the following against every raw block device target before creating LVM structures:

wipefs -a /dev/sdX       # removes filesystem and RAID signatures
sgdisk --zap-all /dev/sdX  # destroys GPT and MBR partition tables

This only affects targets defined as /dev/... paths. Existing thin pool targets (e.g. pve/data) are never touched. Set this flag only on first deployment — never on a running cluster.

Storage Pool Provisioning

The linstor_storage_pools list drives all storage provisioning. For each entry:

Raw device path (/dev/...): The playbook creates an LVM Volume Group and LVM Thin Pool on the specified device, then registers the resulting thin pool in Linstor.
Existing thin pool path (vg/lv, e.g. pve/data): LVM creation is skipped entirely. The existing thin pool is registered directly in Linstor without any modifications to the underlying storage.

After registering storage pools, the playbook creates one Linstor Resource Group per pool entry with a place-count equal to the total number of nodes in the inventory, then creates a Volume Group under it to make it available for VM disk provisioning.

local-lvm Removal

By default, a fresh Proxmox install registers local-lvm pointing at pve/data. If any storage pool in your configuration targets pve/data — either directly or as a raw device that will end up there — the playbook automatically removes local-lvm via pvesm remove local-lvm.

This prevents operators from accidentally provisioning VMs to the underlying thin pool outside of Linstor, which would cause thin-pool exhaustion invisible to DRBD. The local storage entry (ISO images, backups at /var/lib/vz) is always retained.

Proxmox Storage Backend

After Linstor storage is configured, each Resource Group is registered in the Proxmox storage subsystem:

pvesm add drbd ls-<rg_name> --controller <ha_vip> --resourcegroup <rg_name> --content images,rootdir

The controller address points to ha_vip so that VM operations always reach the active Linstor controller regardless of which node it is running on.

HA Controller via drbd-reactor

The Linstor controller database is stored on a DRBD-replicated volume (linstor_db) rather than locally, enabling automatic failover:

A 200M DRBD resource (linstor_db) is spawned across all nodes from the HA storage pool.
The standalone linstor-controller systemd service is disabled on all nodes.
A var-lib-linstor.mount systemd unit is deployed — it mounts the DRBD device at /var/lib/linstor when promoted.
/var/lib/linstor is marked immutable with chattr +i. If the DRBD device fails to mount, the immutable flag prevents any writes to the local filesystem — eliminating the risk of a controller database split-brain.
A drbd-reactor promoter config is deployed at /etc/drbd-reactor.d/linstor_db.toml. It manages:
- Promoting linstor_db to Primary on one node
- Mounting /var/lib/linstor
- Bringing up the floating VIP via ocf:heartbeat:IPaddr2
- Starting linstor-controller.service

Failover is automatic: if the active node loses quorum or crashes, drbd-reactor on another node promotes the resource and brings up all services within seconds.

Checking HA Status

drbd-reactorctl status linstor_db

Expected output on the active node:

Promoter: Resource linstor_db currently active on this node
● drbd-services@linstor_db.target
● ├─ drbd-promote@linstor_db.service
● ├─ ocf.rs@service_ip_linstor_db.service
● ├─ var-lib-linstor.mount
● └─ linstor-controller.service

Previous: Architecture & Phase Breakdown | Index | Next: Troubleshooting & FAQ

4.2 KiB Raw Blame History