Docs rewrite
This commit is contained in:
@@ -1,21 +1,84 @@
|
||||
# 05 - Linstor & Storage Implementation
|
||||
|
||||
This document details the configuration enforcement executed by the Linstor roles.
|
||||
# 05 - Linstor & Storage Deep Dive
|
||||
|
||||
## LVM Global Filter
|
||||
To prevent PVE from detecting double LVM signatures (raw block vs DRBD layer), the playbook injects the following global filter into `/etc/lvm/lvm.conf`:
|
||||
`global_filter =[ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]`
|
||||
|
||||
## Storage Provisioning
|
||||
The `community.general.lvg` task targets raw block devices strictly when the target string starts with `/dev/`. **The playbook does not invoke `wipefs`.** If residual partition tables exist on the target block device, the LVM module will purposefully fail to prevent data destruction.
|
||||
To prevent PVE from scanning DRBD and Linstor mapper devices as additional LVM physical volumes (which causes duplicate signature warnings and can confuse `vgscan`), the playbook injects a global filter into `/etc/lvm/lvm.conf`:
|
||||
|
||||
## Double Provisioning Protection (local-lvm)
|
||||
By default, Proxmox creates a local storage entry named `local-lvm` pointing to the `pve/data` thin-pool. If the playbook detects that you are reusing `pve/data` as a Linstor backend, it automatically executes `pvesm remove local-lvm`. This prevents users from accidentally deploying local VMs to the underlying pool, which causes silent thin-pool exhaustion and DRBD corruption. The `local` directory storage (`/var/lib/vz`) is intentionally retained for ISOs and backups.
|
||||
```
|
||||
global_filter = [ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]
|
||||
```
|
||||
|
||||
## HA Controller Topology
|
||||
Linstor Controller HA is handled via `drbd-reactor`.
|
||||
1. A 200M DRBD resource (`linstor_db`) is spawned across all nodes.
|
||||
2. The default `linstor-controller` systemd service is disabled.
|
||||
3. A `var-lib-linstor.mount` unit is deployed.
|
||||
4. The local `/var/lib/linstor` mount point is secured with the immutable flag (`chattr +i`). This guarantees that if the DRBD resource fails to mount, the local filesystem rejects writes, preventing database split-brains.
|
||||
5. `drbd-reactor` manages the floating VIP and promoter plugin logic.
|
||||
This is applied on all nodes before any VG or pool creation.
|
||||
|
||||
## Disk Wipe (Optional)
|
||||
|
||||
When `wipe_linstor_disks: true`, the playbook runs the following against every raw block device target before creating LVM structures:
|
||||
|
||||
```bash
|
||||
wipefs -a /dev/sdX # removes filesystem and RAID signatures
|
||||
sgdisk --zap-all /dev/sdX # destroys GPT and MBR partition tables
|
||||
```
|
||||
|
||||
This only affects targets defined as `/dev/...` paths. Existing thin pool targets (e.g. `pve/data`) are never touched. **Set this flag only on first deployment — never on a running cluster.**
|
||||
|
||||
## Storage Pool Provisioning
|
||||
|
||||
The `linstor_storage_pools` list drives all storage provisioning. For each entry:
|
||||
|
||||
1. **Raw device path (`/dev/...`):** The playbook creates an LVM Volume Group and LVM Thin Pool on the specified device, then registers the resulting thin pool in Linstor.
|
||||
2. **Existing thin pool path (`vg/lv`, e.g. `pve/data`):** LVM creation is skipped entirely. The existing thin pool is registered directly in Linstor without any modifications to the underlying storage.
|
||||
|
||||
After registering storage pools, the playbook creates one Linstor Resource Group per pool entry with a `place-count` equal to the total number of nodes in the inventory, then creates a Volume Group under it to make it available for VM disk provisioning.
|
||||
|
||||
## local-lvm Removal
|
||||
|
||||
By default, a fresh Proxmox install registers `local-lvm` pointing at `pve/data`. If any storage pool in your configuration targets `pve/data` — either directly or as a raw device that will end up there — the playbook automatically removes `local-lvm` via `pvesm remove local-lvm`.
|
||||
|
||||
This prevents operators from accidentally provisioning VMs to the underlying thin pool outside of Linstor, which would cause thin-pool exhaustion invisible to DRBD. The `local` storage entry (ISO images, backups at `/var/lib/vz`) is always retained.
|
||||
|
||||
## Proxmox Storage Backend
|
||||
|
||||
After Linstor storage is configured, each Resource Group is registered in the Proxmox storage subsystem:
|
||||
|
||||
```bash
|
||||
pvesm add drbd ls-<rg_name> --controller <ha_vip> --resourcegroup <rg_name> --content images,rootdir
|
||||
```
|
||||
|
||||
The controller address points to `ha_vip` so that VM operations always reach the active Linstor controller regardless of which node it is running on.
|
||||
|
||||
## HA Controller via drbd-reactor
|
||||
|
||||
The Linstor controller database is stored on a DRBD-replicated volume (`linstor_db`) rather than locally, enabling automatic failover:
|
||||
|
||||
1. A 200M DRBD resource (`linstor_db`) is spawned across all nodes from the HA storage pool.
|
||||
2. The standalone `linstor-controller` systemd service is disabled on all nodes.
|
||||
3. A `var-lib-linstor.mount` systemd unit is deployed — it mounts the DRBD device at `/var/lib/linstor` when promoted.
|
||||
4. `/var/lib/linstor` is marked immutable with `chattr +i`. If the DRBD device fails to mount, the immutable flag prevents any writes to the local filesystem — eliminating the risk of a controller database split-brain.
|
||||
5. A `drbd-reactor` promoter config is deployed at `/etc/drbd-reactor.d/linstor_db.toml`. It manages:
|
||||
- Promoting `linstor_db` to Primary on one node
|
||||
- Mounting `/var/lib/linstor`
|
||||
- Bringing up the floating VIP via `ocf:heartbeat:IPaddr2`
|
||||
- Starting `linstor-controller.service`
|
||||
|
||||
Failover is automatic: if the active node loses quorum or crashes, `drbd-reactor` on another node promotes the resource and brings up all services within seconds.
|
||||
|
||||
### Checking HA Status
|
||||
|
||||
```bash
|
||||
drbd-reactorctl status linstor_db
|
||||
```
|
||||
|
||||
Expected output on the active node:
|
||||
|
||||
```
|
||||
Promoter: Resource linstor_db currently active on this node
|
||||
● drbd-services@linstor_db.target
|
||||
● ├─ drbd-promote@linstor_db.service
|
||||
● ├─ ocf.rs@service_ip_linstor_db.service
|
||||
● ├─ var-lib-linstor.mount
|
||||
● └─ linstor-controller.service
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
[Previous: Architecture & Phase Breakdown](04-architecture.md) | [Index](../README.md) | [Next: Troubleshooting & FAQ](06-troubleshooting.md)
|
||||
|
||||
Reference in New Issue
Block a user