From 983688a3884455254d2111fe6ff2fa525c71e12d Mon Sep 17 00:00:00 2001 From: Francesco Zimbolo Date: Thu, 23 Apr 2026 09:10:34 +0000 Subject: [PATCH] Updated docs --- README.md | 43 +++++++++++++++++++++++--------------- docs/01-prerequisites.md | 8 +++---- docs/02-configuration.md | 20 ++++++++++++------ docs/03-execution.md | 24 ++++++++++++++++----- docs/04-architecture.md | 10 ++++----- docs/05-linstor-storage.md | 5 ++++- 6 files changed, 72 insertions(+), 38 deletions(-) diff --git a/README.md b/README.md index f8cbf13..8ea3ff9 100644 --- a/README.md +++ b/README.md @@ -1,24 +1,33 @@ -# Proxmox & Linstor HCI Automated Provisioning +# 🚀 Proxmox 9 & Linstor HCI Provisioning Pipeline -This repository contains an Infrastructure as Code (IaC) pipeline to automatically bootstrap a Highly Available Proxmox Virtual Environment (PVE) cluster, with an optional High Availability Hyperconverged Linstor/DRBD storage stack. +👋 **Welcome!** -## 🚀 Quick Start (For Beginners) +This repository contains our declarative Infrastructure as Code (IaC) pipeline. It is designed to automatically bootstrap bare-metal servers into production-ready **Proxmox VE 9 (Debian Trixie)** nodes. -You do not need to install Ansible locally. This repository includes a DevContainer pre-configured with all necessary tools, linters, and extensions. +Whether you are building a 3-node Highly Available cluster with a hyperconverged **Linstor/DRBD** storage stack, or just prepping a dozen independent standalone nodes in parallel, this playbook handles the heavy lifting safely and predictably. -1. Install [Visual Studio Code](https://code.visualstudio.com/) and [Docker](https://docs.docker.com/get-docker/) (or [Podman](https://podman.io/)). -2. Install the VS Code [Dev Containers Extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers). -3. Clone this repository and open the folder in VS Code. -4. Click **"Reopen in Container"** when prompted in the bottom right corner (or press `Ctrl+Shift+P` and type `Dev Containers: Reopen in Container`). -5. Proceed to the [Documentation](docs/01-prerequisites.md) to configure your cluster variables and run your first deployment. +## ✨ Features -## 📚 Documentation +* **Zero-Touch Repositories:** Automatically configures modern DEB822 APT sources for Proxmox 9, handles Enterprise licensing per-node, and removes the UI subscription nag. +* **Smart Reboots:** Detects kernel upgrades (`/var/run/reboot-required`) and safely reboots nodes in the background before forming clusters. +* **HCI Storage:** Automatically provisions raw block devices into LVM-Thin pools, establishes Corosync clusters, and deploys a highly available Linstor Controller via `drbd-reactor`. +* **Flexible Topologies:** Easily toggle between full HCI Clusters, External SAN configurations, or completely independent Standalone nodes using simple variables. -Detailed documentation is available in the `docs/` directory. If you are new to Ansible or this stack, please read them in order: +## 🏁 Quick Start -* [01 - Prerequisites & Node Preparation](docs/01-prerequisites.md) -* [02 - Configuring the Cluster (Inventory & Vars)](docs/02-configuration.md) -* [03 - Running the Playbook](docs/03-execution.md) -* [04 - Architecture & Phase Breakdown](docs/04-architecture.md) -* [05 - Linstor & Storage Deep Dive](docs/05-linstor-storage.md) -* [06 - Troubleshooting & FAQ](docs/06-troubleshooting.md) +To prevent the classic *"it works on my machine"* problem, this repository is completely self-contained. You do not need to install Ansible or any Python packages on your local computer. + +We use a **VS Code DevContainer** that automatically pre-configures your environment with all the correct Ansible versions, collections, and linters. + +⏩ **Ready to deploy? Head straight to [Step 01 - Prerequisites](docs/01-prerequisites.md)** to spin up the container and start your first deployment. + +## 📚 Documentation Directory + +If you are new to Ansible or this specific architecture, please review these guides in order. They are written specifically for our team to ensure safe and consistent deployments across all customer projects. + +* 📋 [01 - Prerequisites & Node Preparation](docs/01-prerequisites.md) +* 🛠️[02 - Configuring the Cluster (Inventory & Vars)](docs/02-configuration.md) +* ⚡ [03 - Running the Playbook](docs/03-execution.md) +* 🏗️ [04 - Architecture & Phase Breakdown](docs/04-architecture.md) +* 💾 [05 - Linstor & Storage Deep Dive](docs/05-linstor-storage.md) +* 🚑 [06 - Troubleshooting & FAQ](docs/06-troubleshooting.md) diff --git a/docs/01-prerequisites.md b/docs/01-prerequisites.md index 8df27ed..19366b4 100644 --- a/docs/01-prerequisites.md +++ b/docs/01-prerequisites.md @@ -3,8 +3,8 @@ This documentation outlines the required state for bare-metal servers before executing the Ansible provisioning pipeline. ## Node State -* **OS:** Proxmox VE 8.x freshly installed. -* **Authentication:** Root password known. SSH key exchange is handled automatically by Ansible. +* **OS:** Proxmox VE 9.x freshly installed. +* **Authentication:** Root password known. *(Highly Recommended: Pre-deploy your local SSH public key to the nodes using `ssh-copy-id root@` before starting. This establishes passwordless authentication from your DevContainer/Control Node and prevents Ansible from prompting you for passwords).* * **Network:** Static IPs assigned. If deploying Linstor, the management interface and Corosync/DRBD replication should ideally be physically separated. If utilizing a single NIC, QoS/traffic shaping must be handled at the switch level to prioritize Corosync UDP (5404/5405) over DRBD replication traffic. * **Storage:** Target block devices intended for Linstor raw LVM provisioning must contain no existing partition tables, LVM metadata, or filesystems. @@ -13,9 +13,9 @@ Local Ansible installation is not required. This repository enforces the use of 1. Open this repository in Visual Studio Code. 2. Ensure Docker/Podman and the "Dev Containers" extension are installed. -3. Select `Reopen in Container`. +3. Select `Reopen in Container`. *(Note: VS Code automatically forwards your local SSH agent into the DevContainer).* 4. Open a terminal within VS Code. All paths referenced in this documentation originate from the `ansible/` directory. ```bash cd ansible/ -``` \ No newline at end of file +``` diff --git a/docs/02-configuration.md b/docs/02-configuration.md index 0d82e1e..650f79d 100644 --- a/docs/02-configuration.md +++ b/docs/02-configuration.md @@ -1,6 +1,6 @@ -# 02 - Configuring the Cluster +# 02 - Configuring the Cluster (or Nodes) -Cluster configurations are segmented per project/customer using Ansible inventories. Do not edit variables in the root directory. +Configurations are segmented per project/customer using isolated Ansible inventories. **Best Practice:** One customer (or one logical cluster) equals one inventory folder. Do not attempt to define multiple clusters within a single inventory to prevent cross-contamination (the "Blast Radius"). ## 1. Create the Environment Copy the template directory for your specific project: @@ -11,8 +11,10 @@ mv inventories//hosts.yml.example inventories//hos mv inventories//group_vars/all.yml.example inventories//group_vars/all.yml ``` -## 2. Configure `hosts.yml` -Define the target PVE nodes. Ansible defaults to the `ansible_host` IP for Corosync and DRBD replication. If a dedicated backend network is available, define `priv_ip`. Ansible uses conditional fallback logic automatically. +## 2. Configure `hosts.yml` (Topology & Licensing) +Define the target PVE nodes. Ansible defaults to the `ansible_host` IP for Corosync/DRBD. If a dedicated backend network is available, define `priv_ip`. + +**Licensing:** Proxmox licenses are socket/node-specific. Insert the `pve_enterprise_key` directly under each host. If a node lacks a key, omit the variable entirely; Ansible will fall back to the `pve-no-subscription` repository automatically for that specific node. ```yaml all: @@ -22,12 +24,18 @@ all: pve-node-01: ansible_host: 192.168.1.11 priv_ip: 10.0.0.11 + pve_enterprise_key: "pve1c-111111111111" + pve-node-02: + ansible_host: 192.168.1.12 + priv_ip: 10.0.0.12 + # Omitted key: This node will use no-subscription ``` ## 3. Configure `group_vars/all.yml` -* **Deployment Toggles:** Set `deploy_linstor: true` for HCI setups, or `false` for traditional SAN integration. -* **Enterprise Licensing:** Supply a key in `pve_enterprise_key`. If empty (`""`), the playbook applies the `pve-no-subscription` repository. +* **Deployment Toggles:** + * `deploy_linstor: true` (Provisions HCI/DRBD stack) + * `standalone_mode: false` (Set to `true` if you are provisioning independent, unclustered servers). * **HA Controller:** Define `ha_pool`, `ha_vip` (Virtual IP), and the CIDR prefix for the Linstor DB promoter. * **Storage Pools:** * If mapping a raw disk (`/dev/sda`), the playbook invokes `community.general.lvg` to create the VG/Thin Pool. diff --git a/docs/03-execution.md b/docs/03-execution.md index 8ecba50..d097322 100644 --- a/docs/03-execution.md +++ b/docs/03-execution.md @@ -2,20 +2,34 @@ Ensure you are operating within the `ansible/` directory inside the DevContainer. -## 1. Test Connectivity -Verify SSH access to the nodes before triggering the pipeline. Because SSH keys are not yet distributed, Ansible must prompt for the root password (`-k`). +## 1. Performance & Parallelism +This repository includes an `ansible.cfg` file configured for performance (`pipelining = True`) and high concurrency (`forks = 20`). If you are provisioning 10 independent servers concurrently in `standalone_mode`, Ansible will process them all simultaneously. +*(Warning: If your uplink bandwidth is limited, concurrent `apt dist-upgrade` tasks across 20 nodes may saturate the link. Lower the `forks` value if you experience APT timeouts).* + +## 2. Test Connectivity +Verify SSH access to the nodes before triggering the pipeline. + +**Authentication Flag (`-k`):** If you rely on root passwords, you must append `-k` (`--ask-pass`) to force Ansible to prompt you. If you have already deployed your SSH key to the nodes (e.g., via `ssh-copy-id`), omit the `-k` flag for seamless passwordless execution. ```bash +# With SSH Key deployed: +ansible all -i inventories//hosts.yml -m ping -u root + +# Without SSH Key (prompts for password): ansible all -i inventories//hosts.yml -m ping -u root -k ``` -## 2. Execute the Playbook -Run the converged deployment playbook. You will be prompted for the SSH password once. +## 3. Execute the Playbook +Run the converged deployment playbook. ```bash +# With SSH Key deployed: +ansible-playbook -i inventories//hosts.yml deploy.yml -u root + +# Without SSH Key: ansible-playbook -i inventories//hosts.yml deploy.yml -u root -k ``` ## Expected Behavior * **Idempotency:** The playbook is declarative. If execution fails due to a transient network issue, re-running the exact command will bypass successful tasks and resume from the failure point. -* **Reboots:** If Phase 2 upgrades the running PVE kernel, the playbook forces a synchronized node reboot to ensure DRBD module compatibility against the active kernel headers. Ansible will pause and wait for the nodes to return online. +* **State-Driven Reboots:** If APT installs core libraries or a new PVE kernel, it generates `/var/run/reboot-required`. Ansible detects this, initiates a graceful system reboot, drops the SSH connection, and continuously polls port 22 up to 10 minutes, seamlessly resuming execution once the servers are back online. diff --git a/docs/04-architecture.md b/docs/04-architecture.md index 9d16b24..bc8bfe9 100644 --- a/docs/04-architecture.md +++ b/docs/04-architecture.md @@ -2,14 +2,14 @@ The `deploy.yml` playbook orchestrates the initialization via sequential phases utilizing `include_role` blocks to guarantee procedural execution order. -## Phase 1: Repositories & Licensing -Evaluates the `pve_enterprise_key` variable. Modifies `/etc/apt/sources.list.d/`, drops the ceph/enterprise configurations as needed, and applies Linstor repository keys. +## Phase 1: Repositories & Licensing (PVE 9 / Trixie) +Evaluates the `pve_enterprise_key` host variable per node. Complies with Debian Trixie standards by utilizing the `deb822_repository` module to write multi-line `.sources` configurations. Legacy `.list` files and unused Ceph repositories are aggressively purged to prevent APT cache duplication errors. ## Phase 2: System Updates & JS Patches -Forces a `dist-upgrade`. Implements a regex replacement targeting `proxmoxlib.js` to nullify the ExtJS subscription prompt. Installs required DRBD headers and modules. Issues a conditional reboot. +Forces a `dist-upgrade`. Implements a regex replacement targeting `proxmoxlib.js` to nullify the ExtJS subscription prompt. Installs required DRBD headers. Checks for `/var/run/reboot-required` and issues a state-driven graceful reboot if necessary. -## Phase 3: Proxmox Clustering +## Phase 3: Proxmox Clustering (Skipped in Standalone Mode) Generates 4096-bit RSA keys, constructs a full-mesh authorized_keys trust, and nullifies StrictHostKeyChecking for the defined subnets. Bootstraps Corosync via `pvecm create` on Node 01, followed by asynchronous `pvecm add` operations on worker nodes. ## Phase 4: Linstor/DRBD Configuration (Conditional) -Wrapped in a feature flag (`deploy_linstor`). Executes standard DRBD module loading, configures global LVM filters, sets up High Availability components, and injects backend parameters into `/etc/pve/storage.cfg`. +Wrapped in two feature flags (`deploy_linstor` and `not is_standalone`). Executes standard DRBD module loading, configures global LVM filters, sets up High Availability components, and injects backend parameters into `/etc/pve/storage.cfg`. diff --git a/docs/05-linstor-storage.md b/docs/05-linstor-storage.md index 3274bd3..c2626fa 100644 --- a/docs/05-linstor-storage.md +++ b/docs/05-linstor-storage.md @@ -3,12 +3,15 @@ This document details the configuration enforcement executed by the Linstor roles. ## LVM Global Filter -To prevent PVE from detecting double LVM signatures (raw block vs DRBD layer), the playbook injects the following global filter into `/etc/lvm/lvm.conf`. +To prevent PVE from detecting double LVM signatures (raw block vs DRBD layer), the playbook injects the following global filter into `/etc/lvm/lvm.conf`: `global_filter =[ "r|^/dev/drbd|", "r|^/dev/mapper/[lL]instor|" ]` ## Storage Provisioning The `community.general.lvg` task targets raw block devices strictly when the target string starts with `/dev/`. **The playbook does not invoke `wipefs`.** If residual partition tables exist on the target block device, the LVM module will purposefully fail to prevent data destruction. +## Double Provisioning Protection (local-lvm) +By default, Proxmox creates a local storage entry named `local-lvm` pointing to the `pve/data` thin-pool. If the playbook detects that you are reusing `pve/data` as a Linstor backend, it automatically executes `pvesm remove local-lvm`. This prevents users from accidentally deploying local VMs to the underlying pool, which causes silent thin-pool exhaustion and DRBD corruption. The `local` directory storage (`/var/lib/vz`) is intentionally retained for ISOs and backups. + ## HA Controller Topology Linstor Controller HA is handled via `drbd-reactor`. 1. A 200M DRBD resource (`linstor_db`) is spawned across all nodes.