Mahahe/proxmox-post-install
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs rewrite

Browse Source
This commit is contained in:
Francesco Zimbolo
2026-04-24 12:58:14 +00:00
parent 9e294aea1a
commit 8477f5b592
7 changed files with 425 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
README.md
View File

@@ -1,33 +1,51 @@
# 🚀 Proxmox 9 & Linstor HCI Provisioning Pipeline # Proxmox 9 & Linstor HCI Provisioning Pipeline
👋 **Welcome!** Declarative Infrastructure as Code pipeline for bootstrapping bare-metal servers into production-ready **Proxmox VE 9 (Debian Trixie)** nodes — either as a 3-node HA cluster with a hyperconverged **Linstor/DRBD** storage stack, or as independent standalone nodes provisioned in parallel.
This repository contains our declarative Infrastructure as Code (IaC) pipeline. It is designed to automatically bootstrap bare-metal servers into production-ready **Proxmox VE 9 (Debian Trixie)** nodes. No local Ansible installation required. The repository ships with a VS Code DevContainer that pins all dependencies.
Whether you are building a 3-node Highly Available cluster with a hyperconverged **Linstor/DRBD** storage stack, or just prepping a dozen independent standalone nodes in parallel, this playbook handles the heavy lifting safely and predictably. ## Index
## ✨ Features 1. [Prerequisites & Node Preparation](docs/01-prerequisites.md)
2. [Configuring the Inventory](docs/02-configuration.md)
3. [Running the Playbooks](docs/03-execution.md)
4. [Architecture & Phase Breakdown](docs/04-architecture.md)
5. [Linstor & Storage Deep Dive](docs/05-linstor-storage.md)
6. [Troubleshooting & FAQ](docs/06-troubleshooting.md)
* **Zero-Touch Repositories:** Automatically configures modern DEB822 APT sources for Proxmox 9, handles Enterprise licensing per-node, and removes the UI subscription nag. ## Repository Layout
* **Smart Reboots:** Detects kernel upgrades (`/var/run/reboot-required`) and safely reboots nodes in the background before forming clusters.
* **HCI Storage:** Automatically provisions raw block devices into LVM-Thin pools, establishes Corosync clusters, and deploys a highly available Linstor Controller via `drbd-reactor`.
* **Flexible Topologies:** Easily toggle between full HCI Clusters, External SAN configurations, or completely independent Standalone nodes using simple variables.
## 🏁 Quick Start ```
ansible/
├── deploy.yml # Main playbook — full cluster provisioning
├── network.yml # Optional — configure p2p private bridge
├── ansible.cfg # Tuned defaults (pipelining, forks, remote_user)
├── inventories/
│ ├── _template/ # Copy this for each new customer/project
│ │ ├── hosts.ini
│ │ └── group_vars/all.yml
│ └── <customer>/ # One inventory per cluster
├── roles/
│ ├── p2p_network/ # Configures full-mesh p2p bridge
│ ├── linstor_repo/ # Adds LINBIT APT repository
│ ├── linstor_install/ # Installs DRBD, Linstor packages
│ ├── linstor_cluster/ # Registers nodes, configures /etc/hosts
│ ├── linstor_ha/ # HA controller via drbd-reactor
│ ├── linstor_storage/ # LVM thin pools, storage pool registration
│ └── proxmox_storage/ # Registers Linstor as Proxmox storage backend
└── docs/
```
To prevent the classic *"it works on my machine"* problem, this repository is completely self-contained. You do not need to install Ansible or any Python packages on your local computer. ## Quick Start
We use a **VS Code DevContainer** that automatically pre-configures your environment with all the correct Ansible versions, collections, and linters. ```bash
# 1. Open repo in VS Code DevContainer
# 2. Copy and configure an inventory
cp -r ansible/inventories/_template ansible/inventories/<customer>
⏩ **Ready to deploy? Head straight to [Step 01 - Prerequisites](docs/01-prerequisites.md)** to spin up the container and start your first deployment. # 3. (Optional) Configure p2p private network bridge
ansible-playbook ansible/network.yml -i ansible/inventories/<customer>/hosts.ini
## 📚 Documentation Directory # 4. Deploy
ansible-playbook ansible/deploy.yml -i ansible/inventories/<customer>/hosts.ini
If you are new to Ansible or this specific architecture, please review these guides in order. They are written specifically for our team to ensure safe and consistent deployments across all customer projects. ```
* 📋 [01 - Prerequisites & Node Preparation](docs/01-prerequisites.md)
* 🛠️[02 - Configuring the Cluster (Inventory & Vars)](docs/02-configuration.md)
* ⚡ [03 - Running the Playbook](docs/03-execution.md)
* 🏗️ [04 - Architecture & Phase Breakdown](docs/04-architecture.md)
* 💾 [05 - Linstor & Storage Deep Dive](docs/05-linstor-storage.md)
* 🚑 [06 - Troubleshooting & FAQ](docs/06-troubleshooting.md)

45
docs/01-prerequisites.md
View File

@@ -1,21 +1,44 @@
# 01 - Prerequisites & Node Preparation # 01 - Prerequisites & Node Preparation
This documentation outlines the required state for bare-metal servers before executing the Ansible provisioning pipeline. ## Node Requirements
## Node State Before running any playbook, each target node must meet the following conditions:
* **OS:** Proxmox VE 9.x freshly installed.
* **Authentication:** Root password known. *(Highly Recommended: Pre-deploy your local SSH public key to the nodes using `ssh-copy-id root@<ip>` before starting. This establishes passwordless authentication from your DevContainer/Control Node and prevents Ansible from prompting you for passwords).* - **OS:** Proxmox VE 9.x freshly installed (Debian Trixie base).
* **Network:** Static IPs assigned. If deploying Linstor, the management interface and Corosync/DRBD replication should ideally be physically separated. If utilizing a single NIC, QoS/traffic shaping must be handled at the switch level to prioritize Corosync UDP (5404/5405) over DRBD replication traffic. - **Network:** Static management IP assigned and reachable from the control node.
* **Storage:** Target block devices intended for Linstor raw LVM provisioning must contain no existing partition tables, LVM metadata, or filesystems. - **SSH:** Your public key deployed to `root` on each node. The DevContainer forwards your local SSH agent automatically — run `ssh-copy-id root@<ip>` from inside the container if not already done.
- **Storage:** If using raw block devices for Linstor (e.g. `/dev/sda`), the device must either be clean or you must set `wipe_linstor_disks: true` in `group_vars`. The playbook will not silently overwrite a disk with existing data unless explicitly told to.
## Network Planning (Cluster Deployments)
For a 3-node Linstor cluster, plan two separate networks:
| Network | Purpose | Variable |
|---|---|---|
| Management | Ansible SSH, Proxmox GUI | `ansible_host` |
| Private (p2p) | Corosync heartbeat, DRBD replication | `priv_ip` |
The private network is a full-mesh point-to-point topology: each node has two dedicated NICs, one direct link to each of the other two nodes, bridged together into an interface named `p2p`. If a dedicated private network is not available, `priv_ip` can be omitted and `ansible_host` is used as fallback for both Corosync and DRBD traffic.
## Execution Environment ## Execution Environment
Local Ansible installation is not required. This repository enforces the use of an isolated VS Code DevContainer.
1. Open this repository in Visual Studio Code. A local Ansible installation is not required. The repository ships with a VS Code DevContainer that pins all required Ansible versions and collections.
2. Ensure Docker/Podman and the "Dev Containers" extension are installed.
3. Select `Reopen in Container`. *(Note: VS Code automatically forwards your local SSH agent into the DevContainer).* 1. Open the repository in Visual Studio Code.
4. Open a terminal within VS Code. All paths referenced in this documentation originate from the `ansible/` directory. 2. Ensure Docker or Podman and the **Dev Containers** extension are installed.
3. Select **Reopen in Container** — VS Code automatically forwards your local SSH agent into the container.
4. Open a terminal inside the container. All commands in this documentation assume you are working from the `ansible/` directory:
```bash ```bash
cd ansible/ cd ansible/
``` ```
Verify connectivity before proceeding:
```bash
ansible all -i inventories/<customer>/hosts.ini -m ping
```
---
[Index](../README.md) | [Next: Configuring the Inventory](02-configuration.md)

118
docs/02-configuration.md
View File

@@ -1,42 +1,100 @@
# 02 - Configuring the Cluster (or Nodes) # 02 - Configuring the Inventory
Configurations are segmented per project/customer using isolated Ansible inventories. **Best Practice:** One customer (or one logical cluster) equals one inventory folder. Do not attempt to define multiple clusters within a single inventory to prevent cross-contamination (the "Blast Radius"). One inventory folder per customer or cluster. Never mix multiple clusters in a single inventory — variables are global within an inventory and will cross-contaminate.
## 1. Create the Environment ## 1. Create the Inventory
Copy the template directory for your specific project:
```bash ```bash
cp -r inventories/_template inventories/<customer_name> cp -r inventories/_template inventories/<customer>
mv inventories/<customer_name>/hosts.yml.example inventories/<customer_name>/hosts.yml
mv inventories/<customer_name>/group_vars/all.yml.example inventories/<customer_name>/group_vars/all.yml
``` ```
## 2. Configure `hosts.yml` (Topology & Licensing) The template contains two files to fill in:
Define the target PVE nodes. Ansible defaults to the `ansible_host` IP for Corosync/DRBD. If a dedicated backend network is available, define `priv_ip`.
**Licensing:** Proxmox licenses are socket/node-specific. Insert the `pve_enterprise_key` directly under each host. If a node lacks a key, omit the variable entirely; Ansible will fall back to the `pve-no-subscription` repository automatically for that specific node. ```
inventories/<customer>/
├── hosts.ini
└── group_vars/all.yml
```
## 2. Configure hosts.ini
Each line defines one node. The inventory alias (left-hand label) is used by Ansible internally and does not need to match the real system hostname — Proxmox and Linstor operations always use the actual hostname gathered from the node at runtime.
```ini
[proxmox]
pve-node-01 ansible_host=192.168.1.1 priv_ip=10.0.0.1 priv_nic_1=enp2s0 priv_nic_2=enp3s0
pve-node-02 ansible_host=192.168.1.2 priv_ip=10.0.0.2 priv_nic_1=enp2s0 priv_nic_2=enp3s0
pve-node-03 ansible_host=192.168.1.3 priv_ip=10.0.0.3 priv_nic_1=enp1s0 priv_nic_2=enp4s0
```
| Variable | Required | Description |
|---|---|---|
| `ansible_host` | Yes | Management IP — used for SSH and Proxmox GUI |
| `priv_ip` | No | Private IP for Corosync and DRBD replication. Falls back to `ansible_host` if omitted |
| `priv_nic_1` / `priv_nic_2` | No | The two NICs forming the p2p bridge. Required only when running `network.yml`. NIC names can differ per node — check with `ip link show` |
| `pve_enterprise_key` | No | Per-node Proxmox subscription key. If omitted, the node uses the no-subscription repository |
**Single-node example** (no cluster, no Linstor):
```ini
[proxmox]
pve-standalone ansible_host=10.0.0.50
```
## 3. Configure group_vars/all.yml
### Deployment Toggles
| Variable | Default | Description |
|---|---|---|
| `standalone_mode` | `false` | Skip all clustering and Linstor tasks, even with multiple nodes in inventory |
| `deploy_linstor` | `false` | Install and configure Linstor + DRBD. Requires 3+ nodes and `standalone_mode: false` |
| `disable_ipv6` | `false` | Disable IPv6 system-wide via sysctl. Useful when IPv6 is unrouted and causes connection timeouts |
| `wipe_linstor_disks` | `false` | Wipe partition tables and signatures from raw block devices before LVM creation. **Destructive — only set on first deployment** |
### Proxmox Cluster
```yaml ```yaml
all: pve_cluster_name: "my-cluster"
children: # pve_enterprise_key: "" # Optional global key; prefer per-host in hosts.ini
proxmox:
hosts:
pve-node-01:
ansible_host: 192.168.1.11
priv_ip: 10.0.0.11
pve_enterprise_key: "pve1c-111111111111"
pve-node-02:
ansible_host: 192.168.1.12
priv_ip: 10.0.0.12
# Omitted key: This node will use no-subscription
``` ```
## 3. Configure `group_vars/all.yml` ### Linstor HA Controller
* **Deployment Toggles:** ```yaml
* `deploy_linstor: true` (Provisions HCI/DRBD stack) ha_pool: "fast_pool_1" # Storage pool used for the linstor_db HA volume
* `standalone_mode: false` (Set to `true` if you are provisioning independent, unclustered servers). ha_vip: "10.0.0.10" # Floating VIP managed by drbd-reactor
* **HA Controller:** Define `ha_pool`, `ha_vip` (Virtual IP), and the CIDR prefix for the Linstor DB promoter. ha_vip_cidr: "24" # Prefix length — also used for the p2p bridge address
* **Storage Pools:** ```
* If mapping a raw disk (`/dev/sda`), the playbook invokes `community.general.lvg` to create the VG/Thin Pool.
* If mapping an existing path (`pve/data`), LVM tasks are skipped, and the target is registered directly in Linstor. ### Linstor Storage Pools
Each entry defines one Resource Group. All nodes in a pool share the same `pool_name`. Keys under `targets` must match the inventory alias from `hosts.ini`.
```yaml
linstor_storage_pools:
- pool_name: "fast_pool_1"
rg_name: "fast_pool_1"
vg_name: "pve"
thin_name: "data"
targets:
pve-node-01: "pve/data" # Register existing pve/data thin pool (no disk changes)
pve-node-02: "pve/data"
pve-node-03: "pve/data"
- pool_name: "slow_pool_1"
rg_name: "slow_pool_1"
vg_name: "sp1"
thin_name: "data"
targets:
pve-node-01: "/dev/sdb" # Create VG sp1 + thin pool on /dev/sdb
pve-node-02: "/dev/sdb"
pve-node-03: "/dev/sdb"
```
**Target path rules:**
- `/dev/...` — playbook creates the VG and LVM thin pool on that raw device. If `wipe_linstor_disks: true`, the disk is wiped first with `wipefs` and `sgdisk`.
- `vg_name/thin_name` (e.g. `pve/data`) — playbook registers the existing thin pool directly in Linstor without touching the underlying storage.
---
[Previous: Prerequisites](01-prerequisites.md) | [Index](../README.md) | [Next: Running the Playbooks](03-execution.md)

68
docs/03-execution.md
View File

@@ -1,35 +1,53 @@
# 03 - Execution # 03 - Running the Playbooks
Ensure you are operating within the `ansible/` directory inside the DevContainer. All commands assume you are inside the DevContainer, working from the `ansible/` directory.
## 1. Performance & Parallelism ## Step 1 — Verify Connectivity
This repository includes an `ansible.cfg` file configured for performance (`pipelining = True`) and high concurrency (`forks = 20`). If you are provisioning 10 independent servers concurrently in `standalone_mode`, Ansible will process them all simultaneously.
*(Warning: If your uplink bandwidth is limited, concurrent `apt dist-upgrade` tasks across 20 nodes may saturate the link. Lower the `forks` value if you experience APT timeouts).*
## 2. Test Connectivity
Verify SSH access to the nodes before triggering the pipeline.
**Authentication Flag (`-k`):** If you rely on root passwords, you must append `-k` (`--ask-pass`) to force Ansible to prompt you. If you have already deployed your SSH key to the nodes (e.g., via `ssh-copy-id`), omit the `-k` flag for seamless passwordless execution.
```bash ```bash
# With SSH Key deployed: ansible all -i inventories/<customer>/hosts.ini -m ping
ansible all -i inventories/<customer_name>/hosts.yml -m ping -u root
# Without SSH Key (prompts for password):
ansible all -i inventories/<customer_name>/hosts.yml -m ping -u root -k
``` ```
## 3. Execute the Playbook All nodes should return `pong`. If any fail with `Permission denied`, run `ssh-copy-id root@<ip>` for that node and retry.
Run the converged deployment playbook.
## Step 2 — (Optional) Configure the P2P Private Network
If you have dedicated NICs for Corosync and DRBD replication and have set `priv_nic_1`, `priv_nic_2`, and `priv_ip` in `hosts.ini`, run `network.yml` first to configure the `p2p` bridge on all nodes:
```bash ```bash
# With SSH Key deployed: ansible-playbook network.yml -i inventories/<customer>/hosts.ini
ansible-playbook -i inventories/<customer_name>/hosts.yml deploy.yml -u root
# Without SSH Key:
ansible-playbook -i inventories/<customer_name>/hosts.yml deploy.yml -u root -k
``` ```
## Expected Behavior This writes the bridge stanza into `/etc/network/interfaces` and applies it live with `ifreload -a` — no reboot required. The task is skipped on any node that does not have `priv_nic_1` and `priv_nic_2` defined.
* **Idempotency:** The playbook is declarative. If execution fails due to a transient network issue, re-running the exact command will bypass successful tasks and resume from the failure point.
* **State-Driven Reboots:** If APT installs core libraries or a new PVE kernel, it generates `/var/run/reboot-required`. Ansible detects this, initiates a graceful system reboot, drops the SSH connection, and continuously polls port 22 up to 10 minutes, seamlessly resuming execution once the servers are back online. ## Step 3 — Dry Run
Preview all changes before applying anything. Declarative tasks (file writes, package installs, sysctl, templates) are fully simulated and shown with diffs. Shell tasks (Linstor registrations, `pvecm`, disk wipes) are skipped in check mode — they will not be previewed but they also will not run.
```bash
ansible-playbook deploy.yml -i inventories/<customer>/hosts.ini --check --diff
```
## Step 4 — Deploy
```bash
ansible-playbook deploy.yml -i inventories/<customer>/hosts.ini
```
The playbook runs across all nodes in parallel (20 forks by default). Phases that require serial ordering — cluster join, Linstor node registration — are internally throttled or pinned to Node 1.
## Idempotency
The playbook is safe to re-run. Every task checks current state before acting. If a run fails partway through due to a transient issue (network blip, APT lock), re-running the same command resumes from the failure point — successful tasks are skipped.
## Reboots
If `apt dist-upgrade` installs a new kernel or critical library, `/var/run/reboot-required` is created. Ansible detects this, reboots the node gracefully, and polls SSH for up to 10 minutes before resuming. An explicit sync barrier ensures all nodes are back online before the clustering phase begins.
## Performance Notes
`ansible.cfg` is tuned with `pipelining = True` and `forks = 20`. For large standalone deployments (many independent nodes), all nodes are provisioned simultaneously. If uplink bandwidth is limited, lower `forks` in `ansible.cfg` to stagger `apt dist-upgrade` traffic.
---
[Previous: Configuring the Inventory](02-configuration.md) | [Index](../README.md) | [Next: Architecture & Phase Breakdown](04-architecture.md)

78
docs/04-architecture.md
View File

@@ -1,15 +1,73 @@
# 04 - Pipeline Architecture # 04 - Architecture & Phase Breakdown
The `deploy.yml` playbook orchestrates the initialization via sequential phases utilizing `include_role` blocks to guarantee procedural execution order. `deploy.yml` runs as a single play across all nodes, executing phases sequentially. Phases that only apply to cluster or Linstor deployments are gated by runtime conditions evaluated per-node.
## Phase 1: Repositories & Licensing (PVE 9 / Trixie) ## Conditions & Flags
Evaluates the `pve_enterprise_key` host variable per node. Complies with Debian Trixie standards by utilizing the `deb822_repository` module to write multi-line `.sources` configurations. Legacy `.list` files and unused Ceph repositories are aggressively purged to prevent APT cache duplication errors.
## Phase 2: System Updates & JS Patches | Condition | Meaning |
Forces a `dist-upgrade`. Implements a regex replacement targeting `proxmoxlib.js` to nullify the ExtJS subscription prompt. Installs required DRBD headers. Checks for `/var/run/reboot-required` and issues a state-driven graceful reboot if necessary. |---|---|
| `_skip_cluster` | True when inventory has only 1 node, or `standalone_mode: true`. Skips phases 3 and 4 entirely |
| `deploy_linstor` | Must be `true` for Linstor roles to run. Independent of node count |
| `priv_nic_1/2` + `priv_ip` defined | Required for Phase 0 (p2p bridge) to run |
## Phase 3: Proxmox Clustering (Skipped in Standalone Mode) ---
Generates 4096-bit RSA keys, constructs a full-mesh authorized_keys trust, and nullifies StrictHostKeyChecking for the defined subnets. Bootstraps Corosync via `pvecm create` on Node 01, followed by asynchronous `pvecm add` operations on worker nodes.
## Phase 4: Linstor/DRBD Configuration (Conditional) ## Phase 0 — P2P Private Network (Optional)
Wrapped in two feature flags (`deploy_linstor` and `not is_standalone`). Executes standard DRBD module loading, configures global LVM filters, sets up High Availability components, and injects backend parameters into `/etc/pve/storage.cfg`.
**Role:** `p2p_network`
**Condition:** `not _skip_cluster` and `priv_nic_1`, `priv_nic_2`, `priv_ip` all defined
Validates that both NICs exist on the node, then writes a bridge stanza into `/etc/network/interfaces` and applies it with `ifreload -a`. The bridge is named `p2p` and receives the node's `priv_ip` address. This phase is also available as a standalone playbook (`network.yml`) for reconfiguring the network independently of a full deployment.
---
## Phase 1 — Repositories & Licensing
Configures APT sources in DEB822 format (`.sources` files, not legacy `.list`). Behavior is per-node based on whether `pve_enterprise_key` is set:
- **With key:** Enterprise repository is added, no-subscription repository is removed, key is registered via `pvesubscription set`.
- **Without key:** No-subscription repository is added, enterprise repository is removed.
Legacy `.list` files for `pve-enterprise`, `pve-no-subscription`, and `ceph` are always deleted to prevent APT cache duplication. The LINBIT repository is added here if `deploy_linstor` is true.
Optionally disables IPv6 system-wide via sysctl (`disable_ipv6: true`) — useful when IPv6 is unrouted and causes TLS connection delays.
---
## Phase 2 — Updates & Nag Removal
Runs `apt dist-upgrade` asynchronously (up to 1 hour, polled every 30 seconds) to prevent SSH timeout during long upgrades. Applies a regex patch to `proxmoxlib.js` to suppress the subscription nag dialog. Installs DRBD kernel modules and Linstor packages if `deploy_linstor` is true.
After upgrades, checks for `/var/run/reboot-required` and reboots nodes that need it. An explicit sync barrier waits for all nodes to come back online before proceeding — this prevents `any_errors_fatal` from aborting the run if nodes reboot at different times.
---
## Phase 3 — Proxmox Clustering
**Condition:** `not _skip_cluster`
Generates 4096-bit RSA keys on all nodes, distributes public keys to build full-mesh root SSH trust, and disables `StrictHostKeyChecking` for the management and Corosync subnets.
Cluster formation is guarded by `pvecm status` — if the cluster already exists, formation tasks are skipped:
- **Node 1:** `pvecm create <name> --link0 <corosync_ip>`
- **Worker nodes:** `pvecm add <master_ip> --link0 <corosync_ip>` — throttled to one node at a time to prevent race conditions in the Corosync join sequence.
---
## Phase 4 — Linstor Configuration
**Condition:** `deploy_linstor` and `not _skip_cluster`
Executed via four roles in order:
| Role | Responsibility |
|---|---|
| `linstor_cluster` | Populates `/etc/hosts` with cluster node entries (private + public), registers all nodes in Linstor, configures fallback interfaces, waits for satellites to come Online |
| `linstor_storage` | Configures LVM global filter, optionally wipes disks, creates VGs and thin pools, registers storage pools and resource groups in Linstor |
| `linstor_ha` | Spawns the `linstor_db` DRBD resource, migrates the controller database onto it, deploys the `drbd-reactor` promoter config with floating VIP |
| `proxmox_storage` | Removes `local-lvm` if `pve/data` is managed by Linstor, registers each resource group as a Proxmox storage backend via `pvesm add drbd` |
---
[Previous: Running the Playbooks](03-execution.md) | [Index](../README.md) | [Next: Linstor & Storage Deep Dive](05-linstor-storage.md)

95
docs/05-linstor-storage.md
View File

@@ -1,21 +1,84 @@
# 05 - Linstor & Storage Implementation # 05 - Linstor & Storage Deep Dive
This document details the configuration enforcement executed by the Linstor roles.
## LVM Global Filter ## LVM Global Filter
To prevent PVE from detecting double LVM signatures (raw block vs DRBD layer), the playbook injects the following global filter into `/etc/lvm/lvm.conf`:
`global_filter =[ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]`
## Storage Provisioning To prevent PVE from scanning DRBD and Linstor mapper devices as additional LVM physical volumes (which causes duplicate signature warnings and can confuse `vgscan`), the playbook injects a global filter into `/etc/lvm/lvm.conf`:
The `community.general.lvg` task targets raw block devices strictly when the target string starts with `/dev/`. **The playbook does not invoke `wipefs`.** If residual partition tables exist on the target block device, the LVM module will purposefully fail to prevent data destruction.
## Double Provisioning Protection (local-lvm) ```
By default, Proxmox creates a local storage entry named `local-lvm` pointing to the `pve/data` thin-pool. If the playbook detects that you are reusing `pve/data` as a Linstor backend, it automatically executes `pvesm remove local-lvm`. This prevents users from accidentally deploying local VMs to the underlying pool, which causes silent thin-pool exhaustion and DRBD corruption. The `local` directory storage (`/var/lib/vz`) is intentionally retained for ISOs and backups. global_filter = [ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]
```
## HA Controller Topology This is applied on all nodes before any VG or pool creation.
Linstor Controller HA is handled via `drbd-reactor`.
1. A 200M DRBD resource (`linstor_db`) is spawned across all nodes. ## Disk Wipe (Optional)
2. The default `linstor-controller` systemd service is disabled.
3. A `var-lib-linstor.mount` unit is deployed. When `wipe_linstor_disks: true`, the playbook runs the following against every raw block device target before creating LVM structures:
4. The local `/var/lib/linstor` mount point is secured with the immutable flag (`chattr +i`). This guarantees that if the DRBD resource fails to mount, the local filesystem rejects writes, preventing database split-brains.
5. `drbd-reactor` manages the floating VIP and promoter plugin logic. ```bash
wipefs -a /dev/sdX # removes filesystem and RAID signatures
sgdisk --zap-all /dev/sdX # destroys GPT and MBR partition tables
```
This only affects targets defined as `/dev/...` paths. Existing thin pool targets (e.g. `pve/data`) are never touched. **Set this flag only on first deployment — never on a running cluster.**
## Storage Pool Provisioning
The `linstor_storage_pools` list drives all storage provisioning. For each entry:
1. **Raw device path (`/dev/...`):** The playbook creates an LVM Volume Group and LVM Thin Pool on the specified device, then registers the resulting thin pool in Linstor.
2. **Existing thin pool path (`vg/lv`, e.g. `pve/data`):** LVM creation is skipped entirely. The existing thin pool is registered directly in Linstor without any modifications to the underlying storage.
After registering storage pools, the playbook creates one Linstor Resource Group per pool entry with a `place-count` equal to the total number of nodes in the inventory, then creates a Volume Group under it to make it available for VM disk provisioning.
## local-lvm Removal
By default, a fresh Proxmox install registers `local-lvm` pointing at `pve/data`. If any storage pool in your configuration targets `pve/data` — either directly or as a raw device that will end up there — the playbook automatically removes `local-lvm` via `pvesm remove local-lvm`.
This prevents operators from accidentally provisioning VMs to the underlying thin pool outside of Linstor, which would cause thin-pool exhaustion invisible to DRBD. The `local` storage entry (ISO images, backups at `/var/lib/vz`) is always retained.
## Proxmox Storage Backend
After Linstor storage is configured, each Resource Group is registered in the Proxmox storage subsystem:
```bash
pvesm add drbd ls-<rg_name> --controller <ha_vip> --resourcegroup <rg_name> --content images,rootdir
```
The controller address points to `ha_vip` so that VM operations always reach the active Linstor controller regardless of which node it is running on.
## HA Controller via drbd-reactor
The Linstor controller database is stored on a DRBD-replicated volume (`linstor_db`) rather than locally, enabling automatic failover:
1. A 200M DRBD resource (`linstor_db`) is spawned across all nodes from the HA storage pool.
2. The standalone `linstor-controller` systemd service is disabled on all nodes.
3. A `var-lib-linstor.mount` systemd unit is deployed — it mounts the DRBD device at `/var/lib/linstor` when promoted.
4. `/var/lib/linstor` is marked immutable with `chattr +i`. If the DRBD device fails to mount, the immutable flag prevents any writes to the local filesystem — eliminating the risk of a controller database split-brain.
5. A `drbd-reactor` promoter config is deployed at `/etc/drbd-reactor.d/linstor_db.toml`. It manages:
- Promoting `linstor_db` to Primary on one node
- Mounting `/var/lib/linstor`
- Bringing up the floating VIP via `ocf:heartbeat:IPaddr2`
- Starting `linstor-controller.service`
Failover is automatic: if the active node loses quorum or crashes, `drbd-reactor` on another node promotes the resource and brings up all services within seconds.
### Checking HA Status
```bash
drbd-reactorctl status linstor_db
```
Expected output on the active node:
```
Promoter: Resource linstor_db currently active on this node
● drbd-services@linstor_db.target
● ├─ drbd-promote@linstor_db.service
● ├─ ocf.rs@service_ip_linstor_db.service
● ├─ var-lib-linstor.mount
● └─ linstor-controller.service
```
---
[Previous: Architecture & Phase Breakdown](04-architecture.md) | [Index](../README.md) | [Next: Troubleshooting & FAQ](06-troubleshooting.md)

91
docs/06-troubleshooting.md
View File

@@ -1,33 +1,81 @@
# 06 - Troubleshooting # 06 - Troubleshooting & FAQ
Common interventions for pipeline failures or hardware reuse. ## SSH: Host Key Verification Failed
## LVM Provisioning Failures (Dirty Disks) If a node was reinstalled, its host key changes and SSH refuses to connect:
If Ansible halts at the LVM Volume Group creation phase, the target block device likely retains partition data. Ensure you are targeting the correct device and wipe the signatures manually:
```
WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
```
Remove the stale entry and re-scan:
```bash ```bash
# Execute on the specific Proxmox node ssh-keygen -R <ip>
wipefs -a /dev/sdX ssh-keyscan <ip> >> ~/.ssh/known_hosts
``` ```
Rerun the playbook after clearing the disk.
## Corosync Quorum Loss ## APT: SSL Certificate Verification Failed
If execution hangs on `pvecm status` or clustering commands, Corosync has lost quorum. In a 3-node cluster, 2 nodes must be online. To force a single active node to regain quorum and unblock operations (for recovery only):
If `apt update` fails with `certificate verify failed`, check whether your firewall is performing SSL inspection on the APT repository domains. The playbook uses `http://` for the Proxmox no-subscription repository (which does not support HTTPS) and `https://` for the enterprise and LINBIT repositories.
If your FortiGate or similar is intercepting HTTPS traffic to these domains, add them to the SSL inspection exemption list:
- `*.proxmox.com`
- `*.linbit.com`
- `packages.linbit.com`
## APT: LINBIT Keyring Download Times Out
The LINBIT keyring is downloaded via Python's `urllib`. On nodes where IPv6 is configured but not routed, the download stalls while waiting for the IPv6 connection attempt to time out before falling back to IPv4. Set `disable_ipv6: true` in `group_vars` to avoid this.
## LVM: Volume Group Creation Fails (Dirty Disk)
If the `Ensure Volume Groups exist for raw disks` task fails, the target device likely has existing partition tables or filesystem signatures. Set `wipe_linstor_disks: true` in `group_vars` and re-run. The playbook will run `wipefs -a` and `sgdisk --zap-all` before attempting LVM creation.
**Never set `wipe_linstor_disks: true` on a cluster that already has data.**
## Linstor: Nodes Not Appearing as Online
Check node status from Node 1:
```bash
linstor node list
```
If nodes show as `OFFLINE` or `CONNECTING`, verify:
1. The satellite service is running: `systemctl status linstor-satellite`
2. The private network (if configured) is reachable: `ping <priv_ip>`
3. Firewall allows port 3366/tcp (Linstor satellite default)
## Linstor HA: Controller Not Promoting
If `drbd-reactorctl status linstor_db` shows `currently active on <unknown>` with all services inactive:
```bash
systemctl reload drbd-reactor
```
If the TOML config was deployed but `drbd-reactor` was not reloaded (e.g. the service was already running before the playbook ran), it will not pick up the new config until reloaded. The playbook handles this automatically on subsequent runs via a handler, but a manual reload fixes it immediately.
Also verify the DRBD resource has at least one node with a disk in `UpToDate` state:
```bash
drbdadm status linstor_db
```
## Corosync: Quorum Loss
If clustering tasks hang on `pvecm status` or nodes cannot join, Corosync has lost quorum. In a 3-node cluster, 2 nodes must be online. To force a degraded single-node cluster to regain quorum temporarily (recovery only):
```bash ```bash
pvecm expected 1 pvecm expected 1
``` ```
## Linstor HA VIP Unreachable
If the controller does not initialize, verify the promoter plugin state:
```bash
systemctl status drbd-reactor
journalctl -u drbd-reactor -n 50
```
Verify switch layer allows IP spoofing / MAC address floating for the defined `ha_vip`.
## Complete Cluster Reset ## Complete Cluster Reset
To obliterate Corosync configuration and reset the node to a standalone state:
To tear down Corosync and return a node to standalone state:
```bash ```bash
systemctl stop pve-cluster systemctl stop pve-cluster
pmxcfs -l pmxcfs -l
@@ -36,4 +84,9 @@ rm -rf /etc/corosync/*
killall pmxcfs killall pmxcfs
systemctl start pve-cluster systemctl start pve-cluster
``` ```
*Note: If DRBD configurations are active, an OS reinstall is significantly faster and safer than manual DRBD block device teardown.*
If DRBD resources are active on the node, an OS reinstall is faster and safer than manually tearing down DRBD metadata and LVM structures.
---
[Previous: Linstor & Storage Deep Dive](05-linstor-storage.md) | [Index](../README.md)