85 lines
4.2 KiB
Markdown
85 lines
4.2 KiB
Markdown
# 05 - Linstor & Storage Deep Dive
|
|
|
|
## LVM Global Filter
|
|
|
|
To prevent PVE from scanning DRBD and Linstor mapper devices as additional LVM physical volumes (which causes duplicate signature warnings and can confuse `vgscan`), the playbook injects a global filter into `/etc/lvm/lvm.conf`:
|
|
|
|
```
|
|
global_filter = [ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]
|
|
```
|
|
|
|
This is applied on all nodes before any VG or pool creation.
|
|
|
|
## Disk Wipe (Optional)
|
|
|
|
When `wipe_linstor_disks: true`, the playbook runs the following against every raw block device target before creating LVM structures:
|
|
|
|
```bash
|
|
wipefs -a /dev/sdX # removes filesystem and RAID signatures
|
|
sgdisk --zap-all /dev/sdX # destroys GPT and MBR partition tables
|
|
```
|
|
|
|
This only affects targets defined as `/dev/...` paths. Existing thin pool targets (e.g. `pve/data`) are never touched. **Set this flag only on first deployment — never on a running cluster.**
|
|
|
|
## Storage Pool Provisioning
|
|
|
|
The `linstor_storage_pools` list drives all storage provisioning. For each entry:
|
|
|
|
1. **Raw device path (`/dev/...`):** The playbook creates an LVM Volume Group and LVM Thin Pool on the specified device, then registers the resulting thin pool in Linstor.
|
|
2. **Existing thin pool path (`vg/lv`, e.g. `pve/data`):** LVM creation is skipped entirely. The existing thin pool is registered directly in Linstor without any modifications to the underlying storage.
|
|
|
|
After registering storage pools, the playbook creates one Linstor Resource Group per pool entry with a `place-count` equal to the total number of nodes in the inventory, then creates a Volume Group under it to make it available for VM disk provisioning.
|
|
|
|
## local-lvm Removal
|
|
|
|
By default, a fresh Proxmox install registers `local-lvm` pointing at `pve/data`. If any storage pool in your configuration targets `pve/data` — either directly or as a raw device that will end up there — the playbook automatically removes `local-lvm` via `pvesm remove local-lvm`.
|
|
|
|
This prevents operators from accidentally provisioning VMs to the underlying thin pool outside of Linstor, which would cause thin-pool exhaustion invisible to DRBD. The `local` storage entry (ISO images, backups at `/var/lib/vz`) is always retained.
|
|
|
|
## Proxmox Storage Backend
|
|
|
|
After Linstor storage is configured, each Resource Group is registered in the Proxmox storage subsystem:
|
|
|
|
```bash
|
|
pvesm add drbd ls-<rg_name> --controller <ha_vip> --resourcegroup <rg_name> --content images,rootdir
|
|
```
|
|
|
|
The controller address points to `ha_vip` so that VM operations always reach the active Linstor controller regardless of which node it is running on.
|
|
|
|
## HA Controller via drbd-reactor
|
|
|
|
The Linstor controller database is stored on a DRBD-replicated volume (`linstor_db`) rather than locally, enabling automatic failover:
|
|
|
|
1. A 200M DRBD resource (`linstor_db`) is spawned across all nodes from the HA storage pool.
|
|
2. The standalone `linstor-controller` systemd service is disabled on all nodes.
|
|
3. A `var-lib-linstor.mount` systemd unit is deployed — it mounts the DRBD device at `/var/lib/linstor` when promoted.
|
|
4. `/var/lib/linstor` is marked immutable with `chattr +i`. If the DRBD device fails to mount, the immutable flag prevents any writes to the local filesystem — eliminating the risk of a controller database split-brain.
|
|
5. A `drbd-reactor` promoter config is deployed at `/etc/drbd-reactor.d/linstor_db.toml`. It manages:
|
|
- Promoting `linstor_db` to Primary on one node
|
|
- Mounting `/var/lib/linstor`
|
|
- Bringing up the floating VIP via `ocf:heartbeat:IPaddr2`
|
|
- Starting `linstor-controller.service`
|
|
|
|
Failover is automatic: if the active node loses quorum or crashes, `drbd-reactor` on another node promotes the resource and brings up all services within seconds.
|
|
|
|
### Checking HA Status
|
|
|
|
```bash
|
|
drbd-reactorctl status linstor_db
|
|
```
|
|
|
|
Expected output on the active node:
|
|
|
|
```
|
|
Promoter: Resource linstor_db currently active on this node
|
|
● drbd-services@linstor_db.target
|
|
● ├─ drbd-promote@linstor_db.service
|
|
● ├─ ocf.rs@service_ip_linstor_db.service
|
|
● ├─ var-lib-linstor.mount
|
|
● └─ linstor-controller.service
|
|
```
|
|
|
|
---
|
|
|
|
[Previous: Architecture & Phase Breakdown](04-architecture.md) | [Index](../README.md) | [Next: Troubleshooting & FAQ](06-troubleshooting.md)
|