Added Senegal 191/25 Inventory

.devcontainer/devcontainer.json

@@ -1,19 +1,15 @@
-// For format details, see https://aka.ms/devcontainer.json. For config options, see the
+// For format details, see https://aka.ms/devcontainer.json.
-// README at: https://github.com/devcontainers/templates/tree/main/src/python
 {
-	"name": "Python 3",
+	"name": "Ansible - Proxmox Post-Install",
-	// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
 	"image": "mcr.microsoft.com/devcontainers/python:3-3.14-trixie",
-	// Features to add to the dev container. More info: https://containers.dev/features.
+
-	"features": {
+	// Do NOT use separate devcontainer features for ansible and ansible-lint.
-		"ghcr.io/devcontainers-extra/features/ansible": {},
+	// Both features use pipx and create isolated venvs, so ansible-lint ends up
-		"ghcr.io/hspaans/devcontainer-features/ansible-lint": {}
+	// with its own internal Ansible copy that cannot see externally installed
-	},
+	// collections — causing syntax-check failures even after collection install.
-	// Use 'forwardPorts' to make a list of ports inside the container available locally.
+	// Installing everything via pip into the shared base Python env solves this.
-	// "forwardPorts": [],
+	"postCreateCommand": "pip install --quiet ansible ansible-lint && ansible-galaxy collection install -r ansible/requirements.yaml",
-	// Use 'postCreateCommand' to run commands after the container is created.
+
-	// "postCreateCommand": "pip3 install --user -r requirements.txt",
-	// Configure tool-specific properties.
 	"customizations": {
 		"vscode": {
 			"settings": {
@@ -25,6 +21,4 @@
 			]
 		}
 	}
-	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
-	// "remoteUser": "root"
 }

README.md

@@ -1,24 +1,51 @@
-# Proxmox & Linstor HCI Automated Provisioning
+# Proxmox 9 & Linstor HCI Provisioning Pipeline
 
-This repository contains an Infrastructure as Code (IaC) pipeline to automatically bootstrap a Highly Available Proxmox Virtual Environment (PVE) cluster, with an optional High Availability Hyperconverged Linstor/DRBD storage stack.
+Declarative Infrastructure as Code pipeline for bootstrapping bare-metal servers into production-ready **Proxmox VE 9 (Debian Trixie)** nodes — either as a 3-node HA cluster with a hyperconverged **Linstor/DRBD** storage stack, or as independent standalone nodes provisioned in parallel.
 
-## 🚀 Quick Start (For Beginners)
+No local Ansible installation required. The repository ships with a VS Code DevContainer that pins all dependencies.
 
-You do not need to install Ansible locally. This repository includes a DevContainer pre-configured with all necessary tools, linters, and extensions.
+## Index
 
-1. Install [Visual Studio Code](https://code.visualstudio.com/) and [Docker](https://docs.docker.com/get-docker/) (or [Podman](https://podman.io/)).
+1. [Prerequisites & Node Preparation](docs/01-prerequisites.md)
-2. Install the VS Code [Dev Containers Extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).
+2. [Configuring the Inventory](docs/02-configuration.md)
-3. Clone this repository and open the folder in VS Code.
+3. [Running the Playbooks](docs/03-execution.md)
-4. Click **"Reopen in Container"** when prompted in the bottom right corner (or press `Ctrl+Shift+P` and type `Dev Containers: Reopen in Container`).
+4. [Architecture & Phase Breakdown](docs/04-architecture.md)
-5. Proceed to the [Documentation](docs/01-prerequisites.md) to configure your cluster variables and run your first deployment.
+5. [Linstor & Storage Deep Dive](docs/05-linstor-storage.md)
+6. [Troubleshooting & FAQ](docs/06-troubleshooting.md)
 
-## 📚 Documentation
+## Repository Layout
 
-Detailed documentation is available in the `docs/` directory. If you are new to Ansible or this stack, please read them in order:
+```
+ansible/
+├── deploy.yml                  # Main playbook — full cluster provisioning
+├── network.yml                 # Optional — configure p2p private bridge
+├── ansible.cfg                 # Tuned defaults (pipelining, forks, remote_user)
+├── inventories/
+│   ├── _template/              # Copy this for each new customer/project
+│   │   ├── hosts.ini
+│   │   └── group_vars/all.yml
+│   └── <customer>/             # One inventory per cluster
+├── roles/
+│   ├── p2p_network/            # Configures full-mesh p2p bridge
+│   ├── linstor_repo/           # Adds LINBIT APT repository
+│   ├── linstor_install/        # Installs DRBD, Linstor packages
+│   ├── linstor_cluster/        # Registers nodes, configures /etc/hosts
+│   ├── linstor_ha/             # HA controller via drbd-reactor
+│   ├── linstor_storage/        # LVM thin pools, storage pool registration
+│   └── proxmox_storage/        # Registers Linstor as Proxmox storage backend
+└── docs/
+```
 
-* [01 - Prerequisites & Node Preparation](docs/01-prerequisites.md)
+## Quick Start
-* [02 - Configuring the Cluster (Inventory & Vars)](docs/02-configuration.md)
+
-* [03 - Running the Playbook](docs/03-execution.md)
+```bash
-* [04 - Architecture & Phase Breakdown](docs/04-architecture.md)
+# 1. Open repo in VS Code DevContainer
-* [05 - Linstor & Storage Deep Dive](docs/05-linstor-storage.md)
+# 2. Copy and configure an inventory
-* [06 - Troubleshooting & FAQ](docs/06-troubleshooting.md)
+cp -r ansible/inventories/_template ansible/inventories/<customer>
+
+# 3. (Optional) Configure p2p private network bridge
+ansible-playbook ansible/network.yml -i ansible/inventories/<customer>/hosts.ini
+
+# 4. Deploy
+ansible-playbook ansible/deploy.yml -i ansible/inventories/<customer>/hosts.ini
+```

ansible/ansible.cfg

@@ -0,0 +1,15 @@
+[defaults]
+remote_user = root
+forks = 20
+pipelining = True
+result_format = yaml
+bin_ansible_callbacks = True
+callback_whitelist = profile_tasks
+
+# SSH connection timeout (opening the connection)
+timeout = 120
+
+[ssh_connection]
+ssh_args = -o ServerAliveInterval=60 -o ServerAliveCountMax=10 -o ControlMaster=auto -o ControlPersist=60s
+# Timeout for individual commands on remote host (apt upgrade, DRBD, etc.)
+command_timeout = 3600

ansible/deploy.yml

@@ -8,51 +8,109 @@
   vars:
     # Dynamically detect if we are in Single-Node or Cluster mode
     is_single_node: "{{ groups['proxmox'] | length == 1 }}"
-    
+    # standalone_mode can be set manually in group_vars to bypass cluster/linstor tasks
-    # Existing clustering variables
+    # even with multiple nodes in inventory (e.g. independent nodes for the same customer)
+    _skip_cluster: "{{ (is_single_node | bool) or (standalone_mode | default(false) | bool) }}"
+
+    # Corosync clustering IPs
     corosync_ip: "{{ priv_ip | default(ansible_host) }}"
     master_corosync_ip: "{{ hostvars[groups['proxmox'][0]]['priv_ip'] | default(hostvars[groups['proxmox'][0]]['ansible_host']) }}"
     is_master: "{{ inventory_hostname == groups['proxmox'][0] }}"
 
   tasks:
     # ==========================================
-    # PHASE 1: REPOSITORIES & LICENSING
+    # PHASE 0: P2P PRIVATE NETWORK
     # ==========================================
+    - name: Configure P2P Private Network Bridge
+      ansible.builtin.include_role:
+        name: p2p_network
+      when:
+        - not _skip_cluster | bool
+        - priv_nic_1 is defined
+        - priv_nic_2 is defined
+        - priv_ip is defined
+
+    # ==========================================
+    # PHASE 1: REPOSITORIES & LICENSING (PVE 9 / Trixie)
+    # ==========================================
+    - name: Disable IPv6 system-wide
+      ansible.posix.sysctl:
+        name: "{{ item }}"
+        value: "1"
+        state: present
+        sysctl_file: /etc/sysctl.d/99-disable-ipv6.conf
+        reload: true
+      loop:
+        - net.ipv6.conf.all.disable_ipv6
+        - net.ipv6.conf.default.disable_ipv6
+        - net.ipv6.conf.lo.disable_ipv6
+      when: disable_ipv6 | default(false) | bool
+
     - name: Apply Proxmox Enterprise Key
       ansible.builtin.command: pvesubscription set -k {{ pve_enterprise_key }}
-      when: pve_enterprise_key | length > 0
+      when: pve_enterprise_key | default('') | length > 0
       register: pve_key_result
       changed_when: "'successfully' in pve_key_result.stdout"
 
-    - name: Manage Proxmox Enterprise Repository
+    - name: Ensure Legacy .list Files are Removed (Prevent APT Duplicates)
-      ansible.builtin.lineinfile:
+      ansible.builtin.file:
-        path: /etc/apt/sources.list.d/pve-enterprise.list
+        path: "{{ item }}"
-        regexp: '^#?deb https://enterprise.proxmox.com/debian/pve'
+        state: absent
-        line: "{{ '' if pve_enterprise_key | length > 0 else '#' }}deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise"
+      loop:
-        state: present
+        - /etc/apt/sources.list.d/pve-enterprise.list
+        - /etc/apt/sources.list.d/pve-no-subscription.list
+        - /etc/apt/sources.list.d/ceph.list
 
-    - name: Manage Proxmox No-Subscription Repository
+    - name: Manage Proxmox Enterprise Repository (DEB822)
-      ansible.builtin.apt_repository:
+      ansible.builtin.deb822_repository:
-        repo: 'deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription'
+        name: pve-enterprise
-        state: "{{ 'absent' if pve_enterprise_key | length > 0 else 'present' }}"
+        types: deb
-        filename: pve-no-subscription
+        uris: https://enterprise.proxmox.com/debian/pve
+        suites: trixie
+        components: pve-enterprise
+        signed_by: /usr/share/keyrings/proxmox-archive-keyring.gpg
+        state: "{{ 'present' if pve_enterprise_key | default('') | length > 0 else 'absent' }}"
 
-    - name: Add Linstor Repositories (Skipped in Single Node)
+    - name: Manage Proxmox No-Subscription Repository (DEB822)
+      ansible.builtin.deb822_repository:
+        name: pve-no-subscription
+        types: deb
+        uris: http://download.proxmox.com/debian/pve
+        suites: trixie
+        components: pve-no-subscription
+        signed_by: /usr/share/keyrings/proxmox-archive-keyring.gpg
+        state: "{{ 'absent' if pve_enterprise_key | default('') | length > 0 else 'present' }}"
+
+    - name: Remove Ceph Enterprise Repository (Unused in Linstor setups)
+      ansible.builtin.file:
+        path: /etc/apt/sources.list.d/ceph.sources
+        state: absent
+      when: pve_enterprise_key | default('') | length == 0
+
+    - name: Add Linstor Repositories
       ansible.builtin.include_role:
         name: linstor_repo
-      when: 
+      when:
         - deploy_linstor | bool
-        - not is_single_node | bool
+        - not _skip_cluster | bool
 
     # ==========================================
     # PHASE 2: UPDATES & NAG REMOVAL
     # ==========================================
+
+    # NOTE: async/poll used here to prevent SSH timeout during long dist-upgrade.
+    # The task is submitted asynchronously (up to 1h) and polled every 30s.
     - name: Update APT cache and run dist-upgrade
       ansible.builtin.apt:
-        update_cache: yes
+        update_cache: true
         upgrade: dist
-        autoremove: yes
+        autoremove: true
-      register: apt_upgrade
+        lock_timeout: 300           # Wait up to 5 min for dpkg lock from other apt processes
+        dpkg_options: 'force-confnew,force-confdef'  # Non-interactive config file handling
+      environment:
+        DEBIAN_FRONTEND: noninteractive   # Prevent debconf from blocking on prompts
+      async: 3600           # Allow up to 1 hour for the full upgrade
+      poll: 30              # Check progress every 30 seconds
 
     - name: Remove Proxmox Subscription Nag (JS Patch)
       ansible.builtin.replace:
@@ -61,28 +119,45 @@
         replace: "void({ //\\1"
       notify: Restart pveproxy
 
-    - name: Install Linstor Packages (Skipped in Single Node)
+    - name: Install Linstor Packages
       ansible.builtin.include_role:
         name: linstor_install
-      when: 
+      when:
         - deploy_linstor | bool
-        - not is_single_node | bool
+        - not _skip_cluster | bool
 
-    - name: Reboot if kernel was updated
+    - name: Check if a reboot is required by APT
+      ansible.builtin.stat:
+        path: /var/run/reboot-required
+      register: reboot_required_file
+
+    - name: Reboot the node gracefully
       ansible.builtin.reboot:
-        msg: "Rebooting to apply new PVE Kernel updates"
+        msg: "Rebooting node to apply new kernel/system updates"
-      when: apt_upgrade.changed
+        reboot_timeout: 600
+      when: reboot_required_file.stat.exists
+
+    # Explicit barrier: ensure ALL nodes are back online before proceeding to Phase 3.
+    # Without this, any_errors_fatal would abort if one node reboots and another doesn't,
+    # causing them to arrive at Phase 3 at different times.
+    - name: Wait for all nodes to be reachable after (optional) reboot
+      ansible.builtin.wait_for_connection:
+        timeout: 300
+        sleep: 5
+
+    - name: Sync barrier — wait for all nodes before clustering phase
+      ansible.builtin.meta: clear_host_errors
 
     # ==========================================
     # PHASE 3: PROXMOX CLUSTERING
     # ==========================================
-    - name: PROXMOX CLUSTERING (Skipped in Single Node Mode)
+    - name: PROXMOX CLUSTERING (Skipped in Single Node / Standalone Mode)
-      when: not is_single_node | bool
+      when: not _skip_cluster | bool
       block:
         - name: Ensure SSH keypair exists for root
           ansible.builtin.user:
             name: root
-            generate_ssh_key: yes
+            generate_ssh_key: true
             ssh_key_bits: 4096
 
         - name: Fetch public SSH keys from all nodes
@@ -97,51 +172,63 @@
             state: present
           loop: "{{ ansible_play_hosts }}"
 
+        - name: Compute SSH wildcard host patterns for cluster subnets
+          ansible.builtin.set_fact:
+            _corosync_subnet: "{{ master_corosync_ip | regex_replace('\\.[0-9]+$', '.*') }}"
+            _mgmt_subnet: >-
+              {{ hostvars[groups['proxmox'][0]]['ansible_host']
+                 | regex_replace('\\.[0-9]+$', '.*') }}
+
         - name: Automatically accept host keys for cluster subnets
           ansible.builtin.blockinfile:
             path: /root/.ssh/config
-            create: yes
+            create: true
             mode: '0600'
             block: |
-              Host {{ master_corosync_ip | regex_replace('\.[0-9]+$', '.*') }} {{ hostvars[groups['proxmox'][0]]['ansible_host'] | regex_replace('\.[0-9]+$', '.*') }}
+              Host {{ _corosync_subnet }} {{ _mgmt_subnet }}
                 StrictHostKeyChecking no
                 UserKnownHostsFile=/dev/null
 
         - name: Check current Proxmox cluster status
           ansible.builtin.command: pvecm status
           register: pvecm_status
-          ignore_errors: yes
+          ignore_errors: true
           changed_when: false
 
         - name: Initialize the Proxmox Cluster (Master Node)
           ansible.builtin.command: pvecm create {{ pve_cluster_name }} --link0 {{ corosync_ip }}
-          when: 
+          changed_when: true
+          when:
             - pvecm_status.rc != 0
             - is_master | bool
 
         - name: Wait for Corosync to stabilize
           ansible.builtin.pause:
             seconds: 10
-          when: 
+          when:
             - pvecm_status.rc != 0
             - is_master | bool
 
-        - name: Join the Proxmox Cluster (Worker Nodes)
+        - name: Join the Proxmox Cluster (Worker Nodes — one at a time)
           ansible.builtin.command: >
             pvecm add {{ master_corosync_ip }} --link0 {{ corosync_ip }} --use_ssh yes
-          when: 
+          register: pvecm_join
+          changed_when: true
+          failed_when: pvecm_join.rc != 0 or 'TASK ERROR' in pvecm_join.stdout or 'TASK ERROR' in pvecm_join.stderr
+          when:
             - pvecm_status.rc != 0
             - not is_master | bool
-          async: 60
+          throttle: 1   # Prevent race condition: pvecm add does not support parallel joins
+          async: 120
           poll: 10
 
     # ==========================================
     # PHASE 4: LINSTOR CONFIGURATION
     # ==========================================
-    - name: LINSTOR CONFIGURATION (Skipped if deploy_linstor=false or Single Node)
+    - name: LINSTOR CONFIGURATION (Skipped if deploy_linstor=false or Single Node / Standalone)
-      when: 
+      when:
         - deploy_linstor | bool
-        - not is_single_node | bool
+        - not _skip_cluster | bool
       block:
         - name: Configure Linstor Cluster
           ansible.builtin.include_role:

ansible/group_vars/all.yml

@@ -1,37 +0,0 @@
----
-# Deployment Toggles
-deploy_linstor: false # Set to true to install and configure Linstor/DRBD
-
-# Proxmox Cluster Configuration
-pve_cluster_name: "prod-cluster"
-pve_enterprise_key: "" # Leave empty ("") to use no-subscription, or insert "PVE..." key
-
-# High Availability Controller Configuration
-ha_pool: "fast_pool_1"
-ha_vip: "10.191.125.4"
-ha_vip_cidr: "29"
-
-# Linstor Storage Pools Configuration
-linstor_storage_pools:
-  - pool_name: "fast_pool_1"
-    rg_name: "fast_pool_1"
-    vg_name: "pve"
-    thin_name: "data"
-    # TARGET RULES: 
-    # Use raw disk (/dev/sdb) to wipe/create LVM
-    # Use existing thin-pool path (pve/data) to register safely
-    targets:
-      pve-node-01: "pve/data" 
-      pve-node-02: "pve/data"
-      pve-node-03: "pve/data"
-  - pool_name: "slow_pool_1"
-    rg_name: "slow_pool_1"
-    vg_name: "sp1"
-    thin_name: "data"
-    # TARGET RULES: 
-    # Use raw disk (/dev/sdb) to wipe/create LVM
-    # Use existing thin-pool path (pve/data) to register safely
-    targets:
-      pve-node-01: "/dev/sda" 
-      pve-node-02: "/dev/sda"
-      pve-node-03: "/dev/sda"

ansible/inventories/_template/group_vars/all.yml

@@ -0,0 +1,70 @@
+---
+# ============================================================
+# Proxmox Post-Install - Group Variables Template
+# Copy this to your inventory folder as group_vars/all.yml
+# ============================================================
+
+# --- Deployment Toggles ---
+# standalone_mode: Set to true to skip ALL cluster and Linstor tasks,
+#   even when the inventory contains multiple nodes (useful for independent nodes per customer).
+standalone_mode: false
+
+# deploy_linstor: Set to true to install and configure Linstor + DRBD storage.
+#   Requires at least 3 nodes and standalone_mode: false.
+deploy_linstor: false
+
+# disable_ipv6: Set to true to disable IPv6 system-wide via sysctl.
+#   Useful when IPv6 is unrouted and causes connection timeouts (e.g. apt, curl).
+disable_ipv6: false
+
+# wipe_linstor_disks: Set to true to wipe partition tables and signatures from raw block devices
+#   before creating LVM VGs. Only affects targets defined as /dev/... paths.
+#   WARNING: destructive — only set on first deployment, never on a running cluster.
+wipe_linstor_disks: false
+
+# --- Proxmox Cluster ---
+pve_cluster_name: "example-cluster"
+# pve_enterprise_key: ""  # Optional: global key (prefer per-host key in hosts.ini)
+
+# --- Linstor / LINBIT Repository ---
+# Auto-calculated as: Debian major version - 4 (e.g. Debian 13 = proxmox-9).
+# Override manually if LINBIT publishes a differently-named repo for new Proxmox versions.
+# linbit_proxmox_version: 9
+
+# --- Linstor HA Controller ---
+# ha_pool: Name of the Linstor storage pool to use for the controller HA volume.
+ha_pool: "fast_pool_1"
+# ha_vip: Virtual IP that follows the active Linstor controller across nodes.
+ha_vip: "192.168.2.4"
+# ha_vip_cidr: Prefix length for the VIP address (e.g. 24 for /24).
+ha_vip_cidr: "24"
+
+# --- Linstor Storage Pools ---
+# Each entry defines one Resource Group, composed of one storage pool per node.
+# All pools in the same entry share the same pool_name across nodes.
+#
+# TARGET RULES:
+#   - Use a raw block device path (/dev/sdb) to auto-create VG + LVM Thin Pool.
+#   - Use an existing thin-pool path (vg_name/thin_name, e.g. "pve/data") to register
+#     an already-existing LVM thin pool WITHOUT touching existing data.
+#
+# Keys under "targets" must match the Ansible inventory_hostname of each node
+# (i.e. the left-hand label in hosts.ini, e.g. "pve-node-01").
+linstor_storage_pools:
+  - pool_name: "fast_pool_1"
+    rg_name: "fast_pool_1"
+    vg_name: "pve"
+    thin_name: "data"
+    targets:
+      pve-node-01: "pve/data"   # Register existing pve/data thin pool
+      pve-node-02: "pve/data"
+      pve-node-03: "pve/data"
+
+  - pool_name: "slow_pool_1"
+    rg_name: "slow_pool_1"
+    vg_name: "sp1"
+    thin_name: "data"
+    targets:
+      pve-node-01: "/dev/sdb"   # Format /dev/sdb as VG sp1 + thin pool data
+      pve-node-02: "/dev/sdb"
+      pve-node-03: "/dev/sdb"

ansible/inventories/_template/hosts.ini

@@ -0,0 +1,25 @@
+# ============================================================
+# Proxmox Cluster Inventory - INI Format
+# Copy this file to your inventory folder as "hosts.ini"
+# and adjust IPs, hostnames, and optional keys.
+# ============================================================
+
+[proxmox]
+# Format: <inventory_alias> ansible_host=<management_ip> [priv_ip=<cluster_ip>] [priv_nic_1=<nic>] [priv_nic_2=<nic>] [pve_enterprise_key=<key>]
+#
+# - inventory_alias : Arbitrary label used by Ansible. Does NOT need to match the real hostname.
+#                     The real system hostname (ansible_hostname) is used for Proxmox & Linstor nodes.
+# - ansible_host    : IP used by Ansible to SSH into the node (management/public IP).
+# - priv_ip         : IP used for Corosync cluster traffic and Linstor replication (private/dedicated NIC).
+#                     If omitted, ansible_host is used as fallback.
+# - priv_nic_1/2    : (Optional) The two NICs forming the full-mesh p2p bridge (named "p2p").
+#                     Required only when running network.yml to configure the private bridge.
+#                     NIC names differ per node — check with: ip link show
+# - pve_enterprise_key : (Optional) Per-node Proxmox enterprise subscription key.
+
+pve-node-01 ansible_host=192.168.1.1 priv_ip=192.168.2.1 priv_nic_1=enp2s0 priv_nic_2=enp3s0
+pve-node-02 ansible_host=192.168.1.2 priv_ip=192.168.2.2 priv_nic_1=enp2s0 priv_nic_2=enp3s0
+pve-node-03 ansible_host=192.168.1.3 priv_ip=192.168.2.3 priv_nic_1=enp2s0 priv_nic_2=enp3s0
+
+# Single-node example (no cluster, no Linstor):
+# pve-standalone ansible_host=10.0.0.50

ansible/inventories/senegal/group_vars/all.yml

@@ -0,0 +1,54 @@
+---
+# ============================================================
+# Proxmox Post-Install - Group Variables (Senegal)
+# ============================================================
+
+# --- Deployment Toggles ---
+standalone_mode: false
+deploy_linstor: true
+disable_ipv6: false
+wipe_linstor_disks: true
+
+# --- Proxmox Cluster ---
+pve_cluster_name: "senegal"
+# pve_enterprise_key: ""  # Optional: global key (prefer per-host key in hosts.ini)
+
+# --- Linstor / LINBIT Repository ---
+# Auto-calculated as: Debian major version - 4 (e.g. Debian 13 = proxmox-9).
+# Override manually if LINBIT publishes a differently-named repo for new Proxmox versions.
+# linbit_proxmox_version: 9
+
+# --- Linstor HA Controller ---
+ha_pool: "fast_pool_1"
+ha_vip: "10.191.125.4"
+ha_vip_cidr: "29"
+
+# --- Linstor Storage Pools ---
+# Each entry defines one Resource Group, composed of one storage pool per node.
+# All pools in the same entry share the same pool_name across nodes.
+#
+# TARGET RULES:
+#   - Use a raw block device path (/dev/sdb) to auto-create VG + LVM Thin Pool.
+#   - Use an existing thin-pool path (vg_name/thin_name, e.g. "pve/data") to register
+#     an already-existing LVM thin pool WITHOUT touching existing data.
+#
+# Keys under "targets" must match the Ansible inventory_hostname of each node
+# (i.e. the left-hand label in hosts.ini, e.g. "vtmis1-ve").
+linstor_storage_pools:
+  - pool_name: "fast_pool_1"
+    rg_name: "fast_pool_1"
+    vg_name: "pve"
+    thin_name: "data"
+    targets:
+      vtmis1-ve: "pve/data"
+      vtmis2-ve: "pve/data"
+      vtmis3-ve: "pve/data"
+
+  - pool_name: "slow_pool_1"
+    rg_name: "slow_pool_1"
+    vg_name: "sp1"
+    thin_name: "data"
+    targets:
+      vtmis1-ve: "/dev/sda"
+      vtmis2-ve: "/dev/sda"
+      vtmis3-ve: "/dev/sda"

ansible/inventories/senegal/hosts.ini

@@ -0,0 +1,25 @@
+# ============================================================
+# Proxmox Cluster Inventory - INI Format
+# Copy this file to your inventory folder as "hosts.ini"
+# and adjust IPs, hostnames, and optional keys.
+# ============================================================
+
+[proxmox]
+# Format: <inventory_alias> ansible_host=<management_ip> [priv_ip=<cluster_ip>] [priv_nic_1=<nic>] [priv_nic_2=<nic>] [pve_enterprise_key=<key>]
+#
+# - inventory_alias : Arbitrary label used by Ansible. Does NOT need to match the real hostname.
+#                     The real system hostname (ansible_hostname) is used for Proxmox & Linstor nodes.
+# - ansible_host    : IP used by Ansible to SSH into the node (management/public IP).
+# - priv_ip         : IP used for Corosync cluster traffic and Linstor replication (private/dedicated NIC).
+#                     If omitted, ansible_host is used as fallback.
+# - priv_nic_1/2    : (Optional) The two NICs forming the full-mesh p2p bridge (named "p2p").
+#                     Required only when running network.yml to configure the private bridge.
+#                     NIC names differ per node — check with: ip link show
+# - pve_enterprise_key : (Optional) Per-node Proxmox enterprise subscription key.
+
+vtmis1-ve ansible_host=10.191.25.21 priv_ip=10.191.125.1 priv_nic_1=nic2 priv_nic_2=nic3
+vtmis2-ve ansible_host=10.191.25.22 priv_ip=10.191.125.2 priv_nic_1=nic2 priv_nic_2=nic3
+vtmis3-ve ansible_host=10.191.25.23 priv_ip=10.191.125.3 priv_nic_1=nic2 priv_nic_2=nic3
+
+# Single-node example (no cluster, no Linstor):
+# pve-standalone ansible_host=10.0.0.50

ansible/inventory/hosts.yml

@@ -1,13 +0,0 @@
-all:
-  children:
-    proxmox:
-      hosts:
-        pve-node-01:
-          ansible_host: 10.191.25.21
-          priv_ip: 10.191.125.1
-        pve-node-02:
-          ansible_host: 10.191.25.22
-          priv_ip: 10.191.125.2
-        pve-node-03:
-          ansible_host: 10.191.25.23
-          priv_ip: 10.191.125.3

ansible/network.yml

@@ -0,0 +1,14 @@
+---
+- name: Configure P2P Private Network Bridge
+  hosts: proxmox
+  become: true
+  gather_facts: true
+
+  tasks:
+    - name: Configure p2p bridge
+      ansible.builtin.include_role:
+        name: p2p_network
+      when:
+        - priv_nic_1 is defined
+        - priv_nic_2 is defined
+        - priv_ip is defined

ansible/requirements.yaml

@@ -0,0 +1,4 @@
+---
+collections:
+  - name: ansible.posix
+  - name: community.general

ansible/roles/linstor_cluster/tasks/main.yml

@@ -1,5 +1,8 @@
 ---
 # tasks file for linstor_cluster
+# IMPORTANT: All Linstor node registrations use ansible_hostname (the real system hostname)
+# to ensure they match Proxmox node names, which are also derived from the system hostname.
+# The Ansible inventory_hostname (e.g. "pve1") may differ from the real hostname (e.g. "pve-node-01").
 
 - name: Safely manage LINSTOR cluster nodes in /etc/hosts
   ansible.builtin.blockinfile:
@@ -8,12 +11,12 @@
     block: |
       # Linstor Cluster Nodes (Primary: Private Network)
       {% for host in groups['proxmox'] %}
-      {{ hostvars[host].priv_ip }} {{ host }}
+      {{ hostvars[host]['priv_ip'] | default(hostvars[host]['ansible_host']) }} {{ hostvars[host]['ansible_facts']['hostname'] }}
       {% endfor %}
 
       # Linstor Cluster Nodes (Fallback: Public Network)
       {% for host in groups['proxmox'] %}
-      {{ hostvars[host].ansible_host }} {{ host }}-public
+      {{ hostvars[host]['ansible_host'] }} {{ hostvars[host]['ansible_facts']['hostname'] }}-public
       {% endfor %}
 
 - name: Ensure /etc/linstor directory exists
@@ -30,13 +33,17 @@
 
 # ---------------------------------------------------------
 # LINSTOR NODE REGISTRATION (Executed ONLY on Node 1)
+# Uses ansible_hostname (real system hostname) for Linstor node names
+# to guarantee compatibility with Proxmox node names.
 # ---------------------------------------------------------
 
 - name: Register nodes in LINSTOR
   ansible.builtin.shell: |
     set -o pipefail
-    if ! linstor node list | grep -q " {{ item }} "; then
+    NODE_NAME="{{ hostvars[item]['ansible_facts']['hostname'] }}"
-      linstor node create "{{ item }}" "{{ hostvars[item].priv_ip }}" --node-type Combined > /dev/null
+    NODE_IP="{{ hostvars[item]['priv_ip'] | default(hostvars[item]['ansible_host']) }}"
+    if ! linstor node list | grep -q " ${NODE_NAME} "; then
+      linstor node create "${NODE_NAME}" "${NODE_IP}" --node-type Combined > /dev/null
       echo "changed"
     fi
   args:
@@ -49,8 +56,10 @@
 - name: Configure Multipath Fallback Interfaces
   ansible.builtin.shell: |
     set -o pipefail
-    if ! linstor node interface list "{{ item }}" | grep -q " fallback "; then
+    NODE_NAME="{{ hostvars[item]['ansible_facts']['hostname'] }}"
-      linstor node interface create "{{ item }}" fallback "{{ hostvars[item].ansible_host }}" > /dev/null
+    NODE_PUBLIC_IP="{{ hostvars[item]['ansible_host'] }}"
+    if ! linstor node interface list "${NODE_NAME}" | grep -q " fallback "; then
+      linstor node interface create "${NODE_NAME}" fallback "${NODE_PUBLIC_IP}" > /dev/null
       echo "changed"
     fi
   args:
@@ -59,3 +68,22 @@
   changed_when: "'changed' in linstor_cluster_int_create.stdout"
   loop: "{{ groups['proxmox'] }}"
   when: inventory_hostname == groups['proxmox'][0]
+
+- name: Wait for all Linstor satellite nodes to appear as Online
+  ansible.builtin.shell: |
+    set -o pipefail
+    if linstor node list | sed 's/\x1b\[[0-9;]*m//g' | grep " {{ hostvars[item]['ansible_facts']['hostname'] }} " | grep -q "Online"; then
+      echo "online"
+    else
+      echo "not_ready"
+      exit 1
+    fi
+  args:
+    executable: /bin/bash
+  register: linstor_cluster_node_status
+  until: "'online' in linstor_cluster_node_status.stdout"
+  retries: 12
+  delay: 10
+  changed_when: false
+  loop: "{{ groups['proxmox'] }}"
+  when: inventory_hostname == groups['proxmox'][0]

ansible/roles/linstor_cluster/templates/linstor-client.conf.j2

@@ -1,3 +1,5 @@
 [global]
-# Ansible dynamically grabs all hosts in the 'proxmox' group and joins them with commas
+# Controllers list uses private IPs of all nodes (tries each in order).
-controllers={{ groups['proxmox'] | join(',') }}
+# After linstor_ha setup, the VIP becomes the active controller endpoint.
+# The drbd-reactor automatically manages controller failover via the VIP.
+controllers={% for host in groups['proxmox'] %}{{ hostvars[host]['priv_ip'] | default(hostvars[host]['ansible_host']) }}{% if not loop.last %},{% endif %}{% endfor %}

ansible/roles/linstor_ha/handlers/main.yml

@@ -0,0 +1,5 @@
+---
+- name: Reload drbd-reactor
+  ansible.builtin.systemd:
+    name: drbd-reactor
+    state: reloaded

ansible/roles/linstor_ha/tasks/main.yml

@@ -65,6 +65,7 @@
     src: linstor_db.toml.j2
     dest: /etc/drbd-reactor.d/linstor_db.toml
     mode: '0644'
+  notify: Reload drbd-reactor
 
 # Notice how the stat task is gone, replaced by 'creates' and 'removes'
 - name: Move existing local database to .orig safely

ansible/roles/linstor_repo/handlers/main.yml

@@ -0,0 +1,4 @@
+---
+- name: Update APT cache
+  ansible.builtin.apt:
+    update_cache: true

ansible/roles/linstor_repo/tasks/main.yml

@@ -1,24 +1,35 @@
 ---
 # tasks file for linstor_repo
+# Target: Proxmox 9 (Debian 13 / Trixie) with DRBD 9
+
+# linbit_proxmox_version defaults to: Debian major version minus 4
+# (Debian 13 = PVE 9, Debian 12 = PVE 8, etc.)
+# Override this variable in group_vars if LINBIT releases a new repo name.
 
 - name: Download LINBIT keyring package
   ansible.builtin.get_url:
     url: https://packages.linbit.com/public/linbit-keyring.deb
     dest: /tmp/linbit-keyring.deb
     mode: '0644'
+  check_mode: false
 
 - name: Install LINBIT keyring
   ansible.builtin.apt:
     deb: /tmp/linbit-keyring.deb
     state: present
 
-- name: Add LINBIT APT repository for Proxmox
+- name: Remove legacy LINBIT .list file (old APT format, prevents duplicates)
-  ansible.builtin.apt_repository:
+  ansible.builtin.file:
-    repo: >-
+    path: /etc/apt/sources.list.d/linbit.list
-      deb [signed-by=/etc/apt/trusted.gpg.d/linbit-keyring.gpg]
+    state: absent
-      https://packages.linbit.com/public/
+
-      proxmox-{{ ansible_facts['distribution_major_version'] | int - 4 }}
+- name: Add LINBIT APT repository for Proxmox (DEB822 format)
-      drbd-9
+  ansible.builtin.deb822_repository:
-    filename: linbit
+    name: linbit
+    types: deb
+    uris: https://packages.linbit.com/public/
+    suites: "proxmox-{{ linbit_proxmox_version | default(ansible_facts['distribution_major_version'] | int - 4) }}"
+    components: drbd-9
+    signed_by: /etc/apt/trusted.gpg.d/linbit-keyring.gpg
     state: present
-    update_cache: true
+  notify: Update APT cache

ansible/roles/linstor_storage/tasks/main.yml

@@ -1,22 +1,66 @@
 ---
 # tasks file for linstor_storage
+# NOTE: linstor_storage_pools[].targets keys are Ansible inventory_hostname values.
+# All Linstor registrations use ansible_hostname (real system hostname) instead,
+# to match Proxmox node names. The mapping is done per-host at registration time.
 
 - name: Configure LVM global_filter for DRBD safely
   ansible.builtin.lineinfile:
     path: /etc/lvm/lvm.conf
     insertafter: '^[ \t]*devices[ \t]*\{'
-    # This regex ensures we only ever have ONE DRBD filter line, replacing the old one if it exists
+    regexp: '^[ \t]*global_filter[ \t]*='
-    regexp: '^[ \t]*global_filter[ \t]*=.*r\|\^/dev/drbd\|'
     line: '        global_filter = [ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]'
     state: present
 
+- name: Remove duplicate global_filter entries from lvm.conf
+  ansible.builtin.shell: |
+    count=$(grep -c '^[[:space:]]*global_filter[[:space:]]*=' /etc/lvm/lvm.conf)
+    if [ "$count" -gt 1 ]; then
+      awk '/^[[:space:]]*global_filter[[:space:]]*=/{if(++n>1)next}1' /etc/lvm/lvm.conf > /tmp/lvm.conf.dedup
+      mv /tmp/lvm.conf.dedup /etc/lvm/lvm.conf
+      echo "changed: removed $((count - 1)) duplicate(s)"
+    fi
+  args:
+    executable: /bin/bash
+  register: lvm_dedup_result
+  changed_when: "'changed' in lvm_dedup_result.stdout"
+
+- name: Wipe existing signatures and partition tables from raw disks
+  ansible.builtin.shell: |
+    DEV="{{ item.targets[inventory_hostname] }}"
+    VG="{{ item.vg_name }}"
+
+    # Deactivate VG properly if LVM still knows about it
+    vgchange -an "$VG" 2>/dev/null || true
+
+    # Forcibly remove any stale dm-mapper devices for this VG (handles the case
+    # where the VG metadata was already wiped but kernel devices are still active)
+    dmsetup ls 2>/dev/null \
+      | awk -v vg="$VG" 'index($1, vg"-") == 1 { print $1 }' \
+      | xargs -r dmsetup remove --force 2>/dev/null || true
+
+    wipefs -a "$DEV"
+    sgdisk --zap-all "$DEV"
+    partprobe "$DEV" 2>/dev/null || true
+  args:
+    executable: /bin/bash
+  changed_when: true
+  loop: "{{ linstor_storage_pools }}"
+  when:
+    - wipe_linstor_disks | default(false) | bool
+    - item.targets[inventory_hostname] is defined
+    - item.targets[inventory_hostname].startswith('/dev/')
+
 - name: Ensure Volume Groups exist for raw disks
   community.general.lvg:
     vg: "{{ item.vg_name }}"
     pvs: "{{ item.targets[inventory_hostname] }}"
+    pv_options: "{{ '--force --force' if wipe_linstor_disks | default(false) | bool else '--force' }}"
   loop: "{{ linstor_storage_pools }}"
-  # Jinja2 magic: Only run this LVM task if the target string starts with '/dev/'
+  # Only run for pools that target this node AND use a raw block device
-  when: item.targets[inventory_hostname].startswith('/dev/')
+  when:
+    - item.targets[inventory_hostname] is defined
+    - item.targets[inventory_hostname].startswith('/dev/')
 
 - name: Ensure LVM Thin Pools exist for raw disks
   community.general.lvol:
@@ -26,25 +70,31 @@
     opts: "-T -Zn"
     shrink: false
   loop: "{{ linstor_storage_pools }}"
-  when: item.targets[inventory_hostname].startswith('/dev/')
+  when:
+    - item.targets[inventory_hostname] is defined
+    - item.targets[inventory_hostname].startswith('/dev/')
 
 # ---------------------------------------------------------
 # LINSTOR STORAGE REGISTRATION (Executed ONLY on Node 1)
+# Uses ansible_hostname (real system hostname) for Linstor node names
 # ---------------------------------------------------------
 
 - name: Register Storage Pools in LINSTOR
   ansible.builtin.shell: |
     set -o pipefail
 
-    # We use Jinja2 to write a custom bash loop for exactly the nodes in our inventory
+    # Iterate over all inventory hosts for this pool.
+    # Use ansible_hostname (real system hostname) as Linstor node name.
     {% for host in groups['proxmox'] %}
+    {% if item.targets[host] is defined %}
     {% set target = item.targets[host] %}
     {% set lvm_path = (item.vg_name + '/' + item.thin_name) if target.startswith('/dev/') else target %}
-
+    NODE_NAME="{{ hostvars[host]['ansible_facts']['hostname'] }}"
-    if ! linstor storage-pool list | grep -q " {{ host }} .* {{ item.pool_name }} "; then
+    if ! linstor storage-pool list | grep -q " ${NODE_NAME} .* {{ item.pool_name }} "; then
-      linstor storage-pool create lvmthin "{{ host }}" "{{ item.pool_name }}" "{{ lvm_path }}" > /dev/null
+      linstor storage-pool create lvmthin "${NODE_NAME}" "{{ item.pool_name }}" "{{ lvm_path }}" > /dev/null
       echo "changed"
     fi
+    {% endif %}
     {% endfor %}
   args:
     executable: /bin/bash
@@ -57,7 +107,9 @@
   ansible.builtin.shell: |
     set -o pipefail
     if ! linstor resource-group list | grep -q " {{ item.rg_name }} "; then
-      linstor resource-group create "{{ item.rg_name }}" --storage-pool "{{ item.pool_name }}" --place-count 3 > /dev/null
+      linstor resource-group create "{{ item.rg_name }}" \
+        --storage-pool "{{ item.pool_name }}" \
+        --place-count {{ groups['proxmox'] | length }} > /dev/null
       linstor volume-group create "{{ item.rg_name }}" > /dev/null
       echo "changed"
     fi

ansible/roles/p2p_network/handlers/main.yml

@@ -0,0 +1 @@
+---

ansible/roles/p2p_network/tasks/main.yml

@@ -0,0 +1,28 @@
+---
+- name: Validate p2p NICs exist on this node
+  ansible.builtin.assert:
+    that:
+      - priv_nic_1 in ansible_facts['interfaces']
+      - priv_nic_2 in ansible_facts['interfaces']
+    fail_msg: >-
+      One or both p2p NICs not found on {{ inventory_hostname }}.
+      Expected: {{ priv_nic_1 }}, {{ priv_nic_2 }}.
+      Available interfaces: {{ ansible_facts['interfaces'] | join(', ') }}
+
+- name: Configure p2p bridge in /etc/network/interfaces
+  ansible.builtin.blockinfile:
+    path: /etc/network/interfaces
+    marker: "# {mark} ANSIBLE MANAGED BLOCK - p2p bridge"
+    block: |
+      auto p2p
+      iface p2p inet static
+          address {{ priv_ip }}/{{ ha_vip_cidr }}
+          bridge-ports {{ priv_nic_1 }} {{ priv_nic_2 }}
+          bridge-stp on
+          bridge-fd 0
+  register: p2p_network_p2p_bridge_config
+
+- name: Apply network config  # noqa: no-handler
+  ansible.builtin.command: ifreload -a
+  changed_when: true
+  when: p2p_network_p2p_bridge_config is changed

docs/01-prerequisites.md

@@ -1,21 +1,44 @@
 # 01 - Prerequisites & Node Preparation
 
-This documentation outlines the required state for bare-metal servers before executing the Ansible provisioning pipeline.
+## Node Requirements
 
-## Node State
+Before running any playbook, each target node must meet the following conditions:
-* **OS:** Proxmox VE 8.x freshly installed.
+
-* **Authentication:** Root password known. SSH key exchange is handled automatically by Ansible.
+- **OS:** Proxmox VE 9.x freshly installed (Debian Trixie base).
-* **Network:** Static IPs assigned. If deploying Linstor, the management interface and Corosync/DRBD replication should ideally be physically separated. If utilizing a single NIC, QoS/traffic shaping must be handled at the switch level to prioritize Corosync UDP (5404/5405) over DRBD replication traffic.
+- **Network:** Static management IP assigned and reachable from the control node.
-* **Storage:** Target block devices intended for Linstor raw LVM provisioning must contain no existing partition tables, LVM metadata, or filesystems. 
+- **SSH:** Your public key deployed to `root` on each node. The DevContainer forwards your local SSH agent automatically — run `ssh-copy-id root@<ip>` from inside the container if not already done.
+- **Storage:** If using raw block devices for Linstor (e.g. `/dev/sda`), the device must either be clean or you must set `wipe_linstor_disks: true` in `group_vars`. The playbook will not silently overwrite a disk with existing data unless explicitly told to.
+
+## Network Planning (Cluster Deployments)
+
+For a 3-node Linstor cluster, plan two separate networks:
+
+| Network | Purpose | Variable |
+|---|---|---|
+| Management | Ansible SSH, Proxmox GUI | `ansible_host` |
+| Private (p2p) | Corosync heartbeat, DRBD replication | `priv_ip` |
+
+The private network is a full-mesh point-to-point topology: each node has two dedicated NICs, one direct link to each of the other two nodes, bridged together into an interface named `p2p`. If a dedicated private network is not available, `priv_ip` can be omitted and `ansible_host` is used as fallback for both Corosync and DRBD traffic.
 
 ## Execution Environment
-Local Ansible installation is not required. This repository enforces the use of an isolated VS Code DevContainer.
 
-1. Open this repository in Visual Studio Code.
+A local Ansible installation is not required. The repository ships with a VS Code DevContainer that pins all required Ansible versions and collections.
-2. Ensure Docker/Podman and the "Dev Containers" extension are installed.
+
-3. Select `Reopen in Container`.
+1. Open the repository in Visual Studio Code.
-4. Open a terminal within VS Code. All paths referenced in this documentation originate from the `ansible/` directory.
+2. Ensure Docker or Podman and the **Dev Containers** extension are installed.
+3. Select **Reopen in Container** — VS Code automatically forwards your local SSH agent into the container.
+4. Open a terminal inside the container. All commands in this documentation assume you are working from the `ansible/` directory:
 
 ```bash
 cd ansible/
-```
+```
+
+Verify connectivity before proceeding:
+
+```bash
+ansible all -i inventories/<customer>/hosts.ini -m ping
+```
+
+---
+
+[Index](../README.md) | [Next: Configuring the Inventory](02-configuration.md)

docs/02-configuration.md

@@ -1,34 +1,100 @@
-# 02 - Configuring the Cluster
+# 02 - Configuring the Inventory
 
-Cluster configurations are segmented per project/customer using Ansible inventories. Do not edit variables in the root directory.
+One inventory folder per customer or cluster. Never mix multiple clusters in a single inventory — variables are global within an inventory and will cross-contaminate.
 
-## 1. Create the Environment
+## 1. Create the Inventory
-Copy the template directory for your specific project:
 
 ```bash
-cp -r inventories/_template inventories/<customer_name>
+cp -r inventories/_template inventories/<customer>
-mv inventories/<customer_name>/hosts.yml.example inventories/<customer_name>/hosts.yml
-mv inventories/<customer_name>/group_vars/all.yml.example inventories/<customer_name>/group_vars/all.yml
 ```
 
-## 2. Configure `hosts.yml`
+The template contains two files to fill in:
-Define the target PVE nodes. Ansible defaults to the `ansible_host` IP for Corosync and DRBD replication. If a dedicated backend network is available, define `priv_ip`. Ansible uses conditional fallback logic automatically.
+
+```
+inventories/<customer>/
+├── hosts.ini
+└── group_vars/all.yml
+```
+
+## 2. Configure hosts.ini
+
+Each line defines one node. The inventory alias (left-hand label) is used by Ansible internally and does not need to match the real system hostname — Proxmox and Linstor operations always use the actual hostname gathered from the node at runtime.
+
+```ini
+[proxmox]
+pve-node-01 ansible_host=192.168.1.1 priv_ip=10.0.0.1 priv_nic_1=enp2s0 priv_nic_2=enp3s0
+pve-node-02 ansible_host=192.168.1.2 priv_ip=10.0.0.2 priv_nic_1=enp2s0 priv_nic_2=enp3s0
+pve-node-03 ansible_host=192.168.1.3 priv_ip=10.0.0.3 priv_nic_1=enp1s0 priv_nic_2=enp4s0
+```
+
+| Variable | Required | Description |
+|---|---|---|
+| `ansible_host` | Yes | Management IP — used for SSH and Proxmox GUI |
+| `priv_ip` | No | Private IP for Corosync and DRBD replication. Falls back to `ansible_host` if omitted |
+| `priv_nic_1` / `priv_nic_2` | No | The two NICs forming the p2p bridge. Required only when running `network.yml`. NIC names can differ per node — check with `ip link show` |
+| `pve_enterprise_key` | No | Per-node Proxmox subscription key. If omitted, the node uses the no-subscription repository |
+
+**Single-node example** (no cluster, no Linstor):
+```ini
+[proxmox]
+pve-standalone ansible_host=10.0.0.50
+```
+
+## 3. Configure group_vars/all.yml
+
+### Deployment Toggles
+
+| Variable | Default | Description |
+|---|---|---|
+| `standalone_mode` | `false` | Skip all clustering and Linstor tasks, even with multiple nodes in inventory |
+| `deploy_linstor` | `false` | Install and configure Linstor + DRBD. Requires 3+ nodes and `standalone_mode: false` |
+| `disable_ipv6` | `false` | Disable IPv6 system-wide via sysctl. Useful when IPv6 is unrouted and causes connection timeouts |
+| `wipe_linstor_disks` | `false` | Wipe partition tables and signatures from raw block devices before LVM creation. **Destructive — only set on first deployment** |
+
+### Proxmox Cluster
 
 ```yaml
-all:
+pve_cluster_name: "my-cluster"
-  children:
+# pve_enterprise_key: ""  # Optional global key; prefer per-host in hosts.ini
-    proxmox:
-      hosts:
-        pve-node-01:
-          ansible_host: 192.168.1.11
-          priv_ip: 10.0.0.11
 ```
 
-## 3. Configure `group_vars/all.yml`
+### Linstor HA Controller
 
-* **Deployment Toggles:** Set `deploy_linstor: true` for HCI setups, or `false` for traditional SAN integration.
+```yaml
-* **Enterprise Licensing:** Supply a key in `pve_enterprise_key`. If empty (`""`), the playbook applies the `pve-no-subscription` repository.
+ha_pool: "fast_pool_1"     # Storage pool used for the linstor_db HA volume
-* **HA Controller:** Define `ha_pool`, `ha_vip` (Virtual IP), and the CIDR prefix for the Linstor DB promoter.
+ha_vip: "10.0.0.10"        # Floating VIP managed by drbd-reactor
-* **Storage Pools:** 
+ha_vip_cidr: "24"          # Prefix length — also used for the p2p bridge address
-  * If mapping a raw disk (`/dev/sda`), the playbook invokes `community.general.lvg` to create the VG/Thin Pool.
+```
-  * If mapping an existing path (`pve/data`), LVM tasks are skipped, and the target is registered directly in Linstor.
+
+### Linstor Storage Pools
+
+Each entry defines one Resource Group. All nodes in a pool share the same `pool_name`. Keys under `targets` must match the inventory alias from `hosts.ini`.
+
+```yaml
+linstor_storage_pools:
+  - pool_name: "fast_pool_1"
+    rg_name: "fast_pool_1"
+    vg_name: "pve"
+    thin_name: "data"
+    targets:
+      pve-node-01: "pve/data"   # Register existing pve/data thin pool (no disk changes)
+      pve-node-02: "pve/data"
+      pve-node-03: "pve/data"
+
+  - pool_name: "slow_pool_1"
+    rg_name: "slow_pool_1"
+    vg_name: "sp1"
+    thin_name: "data"
+    targets:
+      pve-node-01: "/dev/sdb"   # Create VG sp1 + thin pool on /dev/sdb
+      pve-node-02: "/dev/sdb"
+      pve-node-03: "/dev/sdb"
+```
+
+**Target path rules:**
+- `/dev/...` — playbook creates the VG and LVM thin pool on that raw device. If `wipe_linstor_disks: true`, the disk is wiped first with `wipefs` and `sgdisk`.
+- `vg_name/thin_name` (e.g. `pve/data`) — playbook registers the existing thin pool directly in Linstor without touching the underlying storage.
+
+---
+
+[Previous: Prerequisites](01-prerequisites.md) | [Index](../README.md) | [Next: Running the Playbooks](03-execution.md)

docs/03-execution.md

@@ -1,21 +1,53 @@
-# 03 - Execution
+# 03 - Running the Playbooks
 
-Ensure you are operating within the `ansible/` directory inside the DevContainer.
+All commands assume you are inside the DevContainer, working from the `ansible/` directory.
 
-## 1. Test Connectivity
+## Step 1 — Verify Connectivity
-Verify SSH access to the nodes before triggering the pipeline. Because SSH keys are not yet distributed, Ansible must prompt for the root password (`-k`).
 
 ```bash
-ansible all -i inventories/<customer_name>/hosts.yml -m ping -u root -k
+ansible all -i inventories/<customer>/hosts.ini -m ping
 ```
 
-## 2. Execute the Playbook
+All nodes should return `pong`. If any fail with `Permission denied`, run `ssh-copy-id root@<ip>` for that node and retry.
-Run the converged deployment playbook. You will be prompted for the SSH password once.
+
+## Step 2 — (Optional) Configure the P2P Private Network
+
+If you have dedicated NICs for Corosync and DRBD replication and have set `priv_nic_1`, `priv_nic_2`, and `priv_ip` in `hosts.ini`, run `network.yml` first to configure the `p2p` bridge on all nodes:
 
 ```bash
-ansible-playbook -i inventories/<customer_name>/hosts.yml deploy.yml -u root -k
+ansible-playbook network.yml -i inventories/<customer>/hosts.ini
 ```
 
-## Expected Behavior
+This writes the bridge stanza into `/etc/network/interfaces` and applies it live with `ifreload -a` — no reboot required. The task is skipped on any node that does not have `priv_nic_1` and `priv_nic_2` defined.
-* **Idempotency:** The playbook is declarative. If execution fails due to a transient network issue, re-running the exact command will bypass successful tasks and resume from the failure point.
+
-* **Reboots:** If Phase 2 upgrades the running PVE kernel, the playbook forces a synchronized node reboot to ensure DRBD module compatibility against the active kernel headers. Ansible will pause and wait for the nodes to return online.
+## Step 3 — Dry Run
+
+Preview all changes before applying anything. Declarative tasks (file writes, package installs, sysctl, templates) are fully simulated and shown with diffs. Shell tasks (Linstor registrations, `pvecm`, disk wipes) are skipped in check mode — they will not be previewed but they also will not run.
+
+```bash
+ansible-playbook deploy.yml -i inventories/<customer>/hosts.ini --check --diff
+```
+
+## Step 4 — Deploy
+
+```bash
+ansible-playbook deploy.yml -i inventories/<customer>/hosts.ini
+```
+
+The playbook runs across all nodes in parallel (20 forks by default). Phases that require serial ordering — cluster join, Linstor node registration — are internally throttled or pinned to Node 1.
+
+## Idempotency
+
+The playbook is safe to re-run. Every task checks current state before acting. If a run fails partway through due to a transient issue (network blip, APT lock), re-running the same command resumes from the failure point — successful tasks are skipped.
+
+## Reboots
+
+If `apt dist-upgrade` installs a new kernel or critical library, `/var/run/reboot-required` is created. Ansible detects this, reboots the node gracefully, and polls SSH for up to 10 minutes before resuming. An explicit sync barrier ensures all nodes are back online before the clustering phase begins.
+
+## Performance Notes
+
+`ansible.cfg` is tuned with `pipelining = True` and `forks = 20`. For large standalone deployments (many independent nodes), all nodes are provisioned simultaneously. If uplink bandwidth is limited, lower `forks` in `ansible.cfg` to stagger `apt dist-upgrade` traffic.
+
+---
+
+[Previous: Configuring the Inventory](02-configuration.md) | [Index](../README.md) | [Next: Architecture & Phase Breakdown](04-architecture.md)

docs/04-architecture.md

@@ -1,15 +1,73 @@
-# 04 - Pipeline Architecture
+# 04 - Architecture & Phase Breakdown
 
-The `deploy.yml` playbook orchestrates the initialization via sequential phases utilizing `include_role` blocks to guarantee procedural execution order. 
+`deploy.yml` runs as a single play across all nodes, executing phases sequentially. Phases that only apply to cluster or Linstor deployments are gated by runtime conditions evaluated per-node.
 
-## Phase 1: Repositories & Licensing
+## Conditions & Flags
-Evaluates the `pve_enterprise_key` variable. Modifies `/etc/apt/sources.list.d/`, drops the ceph/enterprise configurations as needed, and applies Linstor repository keys.
 
-## Phase 2: System Updates & JS Patches
+| Condition | Meaning |
-Forces a `dist-upgrade`. Implements a regex replacement targeting `proxmoxlib.js` to nullify the ExtJS subscription prompt. Installs required DRBD headers and modules. Issues a conditional reboot.
+|---|---|
+| `_skip_cluster` | True when inventory has only 1 node, or `standalone_mode: true`. Skips phases 3 and 4 entirely |
+| `deploy_linstor` | Must be `true` for Linstor roles to run. Independent of node count |
+| `priv_nic_1/2` + `priv_ip` defined | Required for Phase 0 (p2p bridge) to run |
 
-## Phase 3: Proxmox Clustering
+---
-Generates 4096-bit RSA keys, constructs a full-mesh authorized_keys trust, and nullifies StrictHostKeyChecking for the defined subnets. Bootstraps Corosync via `pvecm create` on Node 01, followed by asynchronous `pvecm add` operations on worker nodes.
 
-## Phase 4: Linstor/DRBD Configuration (Conditional)
+## Phase 0 — P2P Private Network (Optional)
-Wrapped in a feature flag (`deploy_linstor`). Executes standard DRBD module loading, configures global LVM filters, sets up High Availability components, and injects backend parameters into `/etc/pve/storage.cfg`.
+
+**Role:** `p2p_network`
+**Condition:** `not _skip_cluster` and `priv_nic_1`, `priv_nic_2`, `priv_ip` all defined
+
+Validates that both NICs exist on the node, then writes a bridge stanza into `/etc/network/interfaces` and applies it with `ifreload -a`. The bridge is named `p2p` and receives the node's `priv_ip` address. This phase is also available as a standalone playbook (`network.yml`) for reconfiguring the network independently of a full deployment.
+
+---
+
+## Phase 1 — Repositories & Licensing
+
+Configures APT sources in DEB822 format (`.sources` files, not legacy `.list`). Behavior is per-node based on whether `pve_enterprise_key` is set:
+
+- **With key:** Enterprise repository is added, no-subscription repository is removed, key is registered via `pvesubscription set`.
+- **Without key:** No-subscription repository is added, enterprise repository is removed.
+
+Legacy `.list` files for `pve-enterprise`, `pve-no-subscription`, and `ceph` are always deleted to prevent APT cache duplication. The LINBIT repository is added here if `deploy_linstor` is true.
+
+Optionally disables IPv6 system-wide via sysctl (`disable_ipv6: true`) — useful when IPv6 is unrouted and causes TLS connection delays.
+
+---
+
+## Phase 2 — Updates & Nag Removal
+
+Runs `apt dist-upgrade` asynchronously (up to 1 hour, polled every 30 seconds) to prevent SSH timeout during long upgrades. Applies a regex patch to `proxmoxlib.js` to suppress the subscription nag dialog. Installs DRBD kernel modules and Linstor packages if `deploy_linstor` is true.
+
+After upgrades, checks for `/var/run/reboot-required` and reboots nodes that need it. An explicit sync barrier waits for all nodes to come back online before proceeding — this prevents `any_errors_fatal` from aborting the run if nodes reboot at different times.
+
+---
+
+## Phase 3 — Proxmox Clustering
+
+**Condition:** `not _skip_cluster`
+
+Generates 4096-bit RSA keys on all nodes, distributes public keys to build full-mesh root SSH trust, and disables `StrictHostKeyChecking` for the management and Corosync subnets.
+
+Cluster formation is guarded by `pvecm status` — if the cluster already exists, formation tasks are skipped:
+
+- **Node 1:** `pvecm create <name> --link0 <corosync_ip>`
+- **Worker nodes:** `pvecm add <master_ip> --link0 <corosync_ip>` — throttled to one node at a time to prevent race conditions in the Corosync join sequence.
+
+---
+
+## Phase 4 — Linstor Configuration
+
+**Condition:** `deploy_linstor` and `not _skip_cluster`
+
+Executed via four roles in order:
+
+| Role | Responsibility |
+|---|---|
+| `linstor_cluster` | Populates `/etc/hosts` with cluster node entries (private + public), registers all nodes in Linstor, configures fallback interfaces, waits for satellites to come Online |
+| `linstor_storage` | Configures LVM global filter, optionally wipes disks, creates VGs and thin pools, registers storage pools and resource groups in Linstor |
+| `linstor_ha` | Spawns the `linstor_db` DRBD resource, migrates the controller database onto it, deploys the `drbd-reactor` promoter config with floating VIP |
+| `proxmox_storage` | Removes `local-lvm` if `pve/data` is managed by Linstor, registers each resource group as a Proxmox storage backend via `pvesm add drbd` |
+
+---
+
+[Previous: Running the Playbooks](03-execution.md) | [Index](../README.md) | [Next: Linstor & Storage Deep Dive](05-linstor-storage.md)

docs/05-linstor-storage.md

@@ -1,18 +1,84 @@
-# 05 - Linstor & Storage Implementation
+# 05 - Linstor & Storage Deep Dive
-
-This document details the configuration enforcement executed by the Linstor roles.
 
 ## LVM Global Filter
-To prevent PVE from detecting double LVM signatures (raw block vs DRBD layer), the playbook injects the following global filter into `/etc/lvm/lvm.conf`. 
-`global_filter =[ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]`
 
-## Storage Provisioning
+To prevent PVE from scanning DRBD and Linstor mapper devices as additional LVM physical volumes (which causes duplicate signature warnings and can confuse `vgscan`), the playbook injects a global filter into `/etc/lvm/lvm.conf`:
-The `community.general.lvg` task targets raw block devices strictly when the target string starts with `/dev/`. **The playbook does not invoke `wipefs`.** If residual partition tables exist on the target block device, the LVM module will purposefully fail to prevent data destruction.
 
-## HA Controller Topology
+```
-Linstor Controller HA is handled via `drbd-reactor`.
+global_filter = [ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]
-1. A 200M DRBD resource (`linstor_db`) is spawned across all nodes.
+```
-2. The default `linstor-controller` systemd service is disabled.
+
-3. A `var-lib-linstor.mount` unit is deployed.
+This is applied on all nodes before any VG or pool creation.
-4. The local `/var/lib/linstor` mount point is secured with the immutable flag (`chattr +i`). This guarantees that if the DRBD resource fails to mount, the local filesystem rejects writes, preventing database split-brains.
+
-5. `drbd-reactor` manages the floating VIP and promoter plugin logic.
+## Disk Wipe (Optional)
+
+When `wipe_linstor_disks: true`, the playbook runs the following against every raw block device target before creating LVM structures:
+
+```bash
+wipefs -a /dev/sdX       # removes filesystem and RAID signatures
+sgdisk --zap-all /dev/sdX  # destroys GPT and MBR partition tables
+```
+
+This only affects targets defined as `/dev/...` paths. Existing thin pool targets (e.g. `pve/data`) are never touched. **Set this flag only on first deployment — never on a running cluster.**
+
+## Storage Pool Provisioning
+
+The `linstor_storage_pools` list drives all storage provisioning. For each entry:
+
+1. **Raw device path (`/dev/...`):** The playbook creates an LVM Volume Group and LVM Thin Pool on the specified device, then registers the resulting thin pool in Linstor.
+2. **Existing thin pool path (`vg/lv`, e.g. `pve/data`):** LVM creation is skipped entirely. The existing thin pool is registered directly in Linstor without any modifications to the underlying storage.
+
+After registering storage pools, the playbook creates one Linstor Resource Group per pool entry with a `place-count` equal to the total number of nodes in the inventory, then creates a Volume Group under it to make it available for VM disk provisioning.
+
+## local-lvm Removal
+
+By default, a fresh Proxmox install registers `local-lvm` pointing at `pve/data`. If any storage pool in your configuration targets `pve/data` — either directly or as a raw device that will end up there — the playbook automatically removes `local-lvm` via `pvesm remove local-lvm`.
+
+This prevents operators from accidentally provisioning VMs to the underlying thin pool outside of Linstor, which would cause thin-pool exhaustion invisible to DRBD. The `local` storage entry (ISO images, backups at `/var/lib/vz`) is always retained.
+
+## Proxmox Storage Backend
+
+After Linstor storage is configured, each Resource Group is registered in the Proxmox storage subsystem:
+
+```bash
+pvesm add drbd ls-<rg_name> --controller <ha_vip> --resourcegroup <rg_name> --content images,rootdir
+```
+
+The controller address points to `ha_vip` so that VM operations always reach the active Linstor controller regardless of which node it is running on.
+
+## HA Controller via drbd-reactor
+
+The Linstor controller database is stored on a DRBD-replicated volume (`linstor_db`) rather than locally, enabling automatic failover:
+
+1. A 200M DRBD resource (`linstor_db`) is spawned across all nodes from the HA storage pool.
+2. The standalone `linstor-controller` systemd service is disabled on all nodes.
+3. A `var-lib-linstor.mount` systemd unit is deployed — it mounts the DRBD device at `/var/lib/linstor` when promoted.
+4. `/var/lib/linstor` is marked immutable with `chattr +i`. If the DRBD device fails to mount, the immutable flag prevents any writes to the local filesystem — eliminating the risk of a controller database split-brain.
+5. A `drbd-reactor` promoter config is deployed at `/etc/drbd-reactor.d/linstor_db.toml`. It manages:
+   - Promoting `linstor_db` to Primary on one node
+   - Mounting `/var/lib/linstor`
+   - Bringing up the floating VIP via `ocf:heartbeat:IPaddr2`
+   - Starting `linstor-controller.service`
+
+Failover is automatic: if the active node loses quorum or crashes, `drbd-reactor` on another node promotes the resource and brings up all services within seconds.
+
+### Checking HA Status
+
+```bash
+drbd-reactorctl status linstor_db
+```
+
+Expected output on the active node:
+
+```
+Promoter: Resource linstor_db currently active on this node
+● drbd-services@linstor_db.target
+● ├─ drbd-promote@linstor_db.service
+● ├─ ocf.rs@service_ip_linstor_db.service
+● ├─ var-lib-linstor.mount
+● └─ linstor-controller.service
+```
+
+---
+
+[Previous: Architecture & Phase Breakdown](04-architecture.md) | [Index](../README.md) | [Next: Troubleshooting & FAQ](06-troubleshooting.md)

docs/06-troubleshooting.md

@@ -1,33 +1,81 @@
-# 06 - Troubleshooting
+# 06 - Troubleshooting & FAQ
 
-Common interventions for pipeline failures or hardware reuse.
+## SSH: Host Key Verification Failed
 
-## LVM Provisioning Failures (Dirty Disks)
+If a node was reinstalled, its host key changes and SSH refuses to connect:
-If Ansible halts at the LVM Volume Group creation phase, the target block device likely retains partition data. Ensure you are targeting the correct device and wipe the signatures manually:
+
+```
+WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
+```
+
+Remove the stale entry and re-scan:
 
 ```bash
-# Execute on the specific Proxmox node
+ssh-keygen -R <ip>
-wipefs -a /dev/sdX
+ssh-keyscan <ip> >> ~/.ssh/known_hosts
 ```
-Rerun the playbook after clearing the disk.
 
-## Corosync Quorum Loss
+## APT: SSL Certificate Verification Failed
-If execution hangs on `pvecm status` or clustering commands, Corosync has lost quorum. In a 3-node cluster, 2 nodes must be online. To force a single active node to regain quorum and unblock operations (for recovery only):
+
+If `apt update` fails with `certificate verify failed`, check whether your firewall is performing SSL inspection on the APT repository domains. The playbook uses `http://` for the Proxmox no-subscription repository (which does not support HTTPS) and `https://` for the enterprise and LINBIT repositories.
+
+If your FortiGate or similar is intercepting HTTPS traffic to these domains, add them to the SSL inspection exemption list:
+
+- `*.proxmox.com`
+- `*.linbit.com`
+- `packages.linbit.com`
+
+## APT: LINBIT Keyring Download Times Out
+
+The LINBIT keyring is downloaded via Python's `urllib`. On nodes where IPv6 is configured but not routed, the download stalls while waiting for the IPv6 connection attempt to time out before falling back to IPv4. Set `disable_ipv6: true` in `group_vars` to avoid this.
+
+## LVM: Volume Group Creation Fails (Dirty Disk)
+
+If the `Ensure Volume Groups exist for raw disks` task fails, the target device likely has existing partition tables or filesystem signatures. Set `wipe_linstor_disks: true` in `group_vars` and re-run. The playbook will run `wipefs -a` and `sgdisk --zap-all` before attempting LVM creation.
+
+**Never set `wipe_linstor_disks: true` on a cluster that already has data.**
+
+## Linstor: Nodes Not Appearing as Online
+
+Check node status from Node 1:
+
+```bash
+linstor node list
+```
+
+If nodes show as `OFFLINE` or `CONNECTING`, verify:
+1. The satellite service is running: `systemctl status linstor-satellite`
+2. The private network (if configured) is reachable: `ping <priv_ip>`
+3. Firewall allows port 3366/tcp (Linstor satellite default)
+
+## Linstor HA: Controller Not Promoting
+
+If `drbd-reactorctl status linstor_db` shows `currently active on <unknown>` with all services inactive:
+
+```bash
+systemctl reload drbd-reactor
+```
+
+If the TOML config was deployed but `drbd-reactor` was not reloaded (e.g. the service was already running before the playbook ran), it will not pick up the new config until reloaded. The playbook handles this automatically on subsequent runs via a handler, but a manual reload fixes it immediately.
+
+Also verify the DRBD resource has at least one node with a disk in `UpToDate` state:
+
+```bash
+drbdadm status linstor_db
+```
+
+## Corosync: Quorum Loss
+
+If clustering tasks hang on `pvecm status` or nodes cannot join, Corosync has lost quorum. In a 3-node cluster, 2 nodes must be online. To force a degraded single-node cluster to regain quorum temporarily (recovery only):
 
 ```bash
 pvecm expected 1
 ```
 
-## Linstor HA VIP Unreachable
-If the controller does not initialize, verify the promoter plugin state:
-```bash
-systemctl status drbd-reactor
-journalctl -u drbd-reactor -n 50
-```
-Verify switch layer allows IP spoofing / MAC address floating for the defined `ha_vip`.
-
 ## Complete Cluster Reset
-To obliterate Corosync configuration and reset the node to a standalone state:
+
+To tear down Corosync and return a node to standalone state:
+
 ```bash
 systemctl stop pve-cluster
 pmxcfs -l
@@ -36,4 +84,9 @@ rm -rf /etc/corosync/*
 killall pmxcfs
 systemctl start pve-cluster
 ```
-*Note: If DRBD configurations are active, an OS reinstall is significantly faster and safer than manual DRBD block device teardown.*
+
+If DRBD resources are active on the node, an OS reinstall is faster and safer than manually tearing down DRBD metadata and LVM structures.
+
+---
+
+[Previous: Linstor & Storage Deep Dive](05-linstor-storage.md) | [Index](../README.md)