[CI] Refactor GitHub Actions workflows for build, documentation, and linting (#2260)

* Refactor GitHub Actions workflows for build, documentation, and linting - Consolidated build logic into reusable workflows (`task-build.yml` and `task-docs.yml`) to reduce duplication across multiple workflows. - Introduced `environments.json` to centralize the list of PlatformIO build environments, improving maintainability and clarity. - Updated `build.yml` and `build_and_docs_to_dev.yml` to utilize the new reusable workflows and environment definitions. - Enhanced `release.yml` to streamline the release process and integrate documentation generation. - Created reusable linting workflow (`task-lint.yml`) to standardize code formatting checks across the repository. - Simplified manual documentation workflow by leveraging the new reusable documentation workflow. - Improved artifact management and retention policies across workflows. - Updated dependencies and versions in workflows to ensure compatibility and performance. * CI/CD pipeline agnostic of Workflow Engine and integrated on github actions - Implemented ci.sh for orchestrating the complete build pipeline. - Created ci_00_config.sh for centralized configuration of build scripts. - Created ci_build_firmware.sh for building firmware for specified PlatformIO environments. - Created ci_prepare_artifacts.sh for preparing firmware artifacts for upload or deployment. - Created ci_set_version.sh for updating version tags in firmware configuration files. - Created ci_build.sh to orchestrate the complete build pipeline. - Created ci_qa.sh for code linting and formatting checks using clang-format. - Created ci_site.sh for building and deploying VuePress documentation with version management. - Implemented checks for required tools and dependencies in the new scripts. - Updated common_wu.py, compressFirmware.py, gen_wu.py, generate_board_docs.py, and prepare_deploy.sh with descriptive comments. Refactor CI/CD scripts for improved modularity and clarity - Consolidated build steps in task-build.yml to utilize ci.sh for version tagging, building, and artifact preparation. - Updated task-lint.yml to use ci.sh for code formatting checks instead of ci_qa.sh. - Enhanced CI_SCRIPTS.md documentation to reflect changes in script usage, command structure, and output organization. - Improved internal scripts for better error handling and logging. - Streamlined the output structure for build artifacts and documentation.
2026-02-15 14:58:58 -06:00
1 parent fd433c220e
commit 134c03362c
@@ -0,0 +1,575 @@
+# CI/CD Scripts Documentation
+
+This documentation describes the CI/CD scripts used to build OpenMQTTGateway firmware and documentation. These scripts work in GitHub Actions, locally, and in any CI/CD environment.
+
+- [Overview](#cicd-scripts-documentation)
+- [Quick Reference](#quick-reference)
+  - [Script Hierarchy](#script-hierarchy)
+  - [Script Description](#script-description)
+  - [Output Structure](#output-structure)
+- [Commands](#commands)
+  - [ci.sh - Main Entry Point](#commands)
+  - [ci.sh build - Build Firmware](#cish-build---build-firmware)
+  - [ci.sh site - Build Documentation](#cish-site---build-documentation)
+  - [ci.sh qa - Code Formatting Check](#cish-qa---code-formatting-check)
+- [Internal Scripts](#internal-scripts)
+  - [ci_set_version.sh](#ci_set_versionsh)
+  - [ci_build_firmware.sh](#ci_build_firmwaresh)
+  - [ci_prepare_artifacts.sh](#ci_prepare_artifactssh)
+  - [ci_00_config.sh](#ci_00_configsh)
+- [Python Helper Scripts](#python-helper-scripts)
+  - [generate_board_docs.py](#generate_board_docspy)
+  - [gen_wu.py](#gen_wupy)
+- [Environment Variables](#environment-variables)
+- [Exit Codes](#exit-codes)
+- [Environment Detection](#environment-detection)
+
+
+
+## Quick Reference
+
+### Script Hierarchy
+
+```
+ci.sh (dispatcher)
+├── build → ci_build.sh → ci_set_version.sh
+│                       → ci_build_firmware.sh
+│                       → ci_prepare_artifacts.sh
+├── site → ci_site.sh → generate_board_docs.py
+│                     → gen_wu.py
+├── qa → ci_qa.sh → clang-format
+└── all → qa + build + site (sequential)
+```
+
+### Script Description
+
+| Script | Purpose | Called By |
+|--------|---------|-----------|
+| `ci.sh` | Main command dispatcher | User/GitHub Actions |
+| `ci_build.sh` | Build firmware orchestrator | ci.sh build |
+| `ci_site.sh` | Documentation build orchestrator | ci.sh site |
+| `ci_qa.sh` | Code formatting checker | ci.sh qa |
+| `ci_set_version.sh` | Version injection in firmware | ci_build.sh |
+| `ci_build_firmware.sh` | PlatformIO build execution | ci_build.sh |
+| `ci_prepare_artifacts.sh` | Artifact packaging | ci_build.sh |
+| `ci_00_config.sh` | Shared configuration and functions | All scripts |
+
+
+### Output Structure
+
+Build outputs are organized in the project root:
+
+```
+.pio/build/<environment>/     # PlatformIO build outputs
+├── firmware.bin              # Main firmware binary
+├── bootloader.bin           # ESP32 bootloader
+└── partitions.bin           # ESP32 partition table
+
+generated/
+├── artifacts/               # Packaged firmware artifacts
+└── site/                   # Built documentation (VuePress output)
+
+scripts/
+├── latest_version.json     # Production version metadata
+└── latest_version_dev.json # Development version metadata
+```
+
+## Commands
+
+`ci.sh` is the main Entry Point. Command dispatcher that routes to specialized scripts.
+
+**Usage:**
+```bash
+./scripts/ci.sh <command> [OPTIONS]
+```
+
+**Commands:**
+- `build` - Build firmware for specified environment
+- `site` or `docs` - Build documentation website
+- `qa` or `lint` - Run code formatting checks
+- `all` or `pipeline` - Run complete pipeline (qa + build + site)
+
+**Examples:**
+```bash
+# Get Help
+./scripts/ci.sh build --help
+./scripts/ci.sh qa --help
+./scripts/ci.sh site --help
+
+# Build firmware
+./scripts/ci.sh build esp32dev-ble --mode dev
+./scripts/ci.sh build esp32dev-all-test --version v1.8.0 --deploy-ready
+
+# Build documentation
+./scripts/ci.sh site --mode prod
+./scripts/ci.sh site --mode dev --url-prefix /dev/
+
+# Check code formatting
+./scripts/ci.sh qa --check
+./scripts/ci.sh qa --fix
+
+# Run complete pipeline
+./scripts/ci.sh all esp32dev-ble --version v1.8.0
+./scripts/ci.sh all esp32dev-ble --no-site
+```
+
+**Options for `all` command:**
+- `--no-site` - Skip documentation build step
+- All options from `build` command are passed through
+
+---
+
+### ci.sh build - Build Firmware
+
+Orchestrates complete firmware build: version injection, compilation, artifact packaging.
+
+**Usage:**
+```bash
+./scripts/ci.sh build <environment> [OPTIONS]
+```
+
+**Required Arguments:**
+- `<environment>` - PlatformIO environment name (e.g., esp32dev-ble, nodemcuv2-rf)
+
+**Options:**
+- `--version <tag>` - Version string to inject (default: auto-generated from git)
+- `--mode <dev|prod>` - Build mode (default: prod)
+  - `dev` - Enables development OTA, sets DEVELOPMENTOTA=true
+  - `prod` - Standard production build
+- `--deploy-ready` - Package artifacts with deployment naming (env-firmware.bin)
+- `--output <dir>` - Output directory for artifacts (default: generated/artifacts)
+- `--skip-verification` - Skip build tools verification
+- `--clean` - Clean previous build before starting
+- `--verbose` - Enable verbose PlatformIO output
+- `--list-envs` - List all available PlatformIO environments
+- `--help` - Show help message
+
+**Execution Flow:**
+```
+ci.sh build esp32dev-ble --version v1.8.0 --mode prod --deploy-ready
+  │
+  ├─> ci_build.sh (orchestrator)
+  │     ├─> verify_build_tools() - Check python3, platformio, git
+  │     ├─> ci_set_version.sh v1.8.0 - Inject version in User_config.h
+  │     ├─> ci_build_firmware.sh esp32dev-ble - Execute PlatformIO build
+  │     └─> ci_prepare_artifacts.sh esp32dev-ble --deploy - Package binaries
+  │
+  └─> Outputs in generated/artifacts/
+        ├─ esp32dev-ble-firmware.bin
+        ├─ esp32dev-ble-bootloader.bin
+        └─ esp32dev-ble-partitions.bin
+```
+
+**Examples:**
+```bash
+# Development build
+./scripts/ci.sh build esp32dev-ble --mode dev
+
+# Production build with version
+./scripts/ci.sh build esp32dev-ble --version v1.8.0 --mode prod
+
+# Deploy-ready build
+./scripts/ci.sh build esp32dev-all-test --version v1.8.0 --deploy-ready
+
+# Clean build with verbose output
+./scripts/ci.sh build nodemcuv2-rf --clean --verbose
+
+# List available environments
+./scripts/ci.sh build --list-envs
+```
+
+**Environment Variables:**
+- `CI` - Set to 'true' in CI/CD environments
+- `BUILD_NUMBER` - Build number from CI/CD system
+- `GIT_COMMIT` - Git commit hash for auto-versioning
+- `PLATFORMIO_BUILD_FLAGS` - Additional PlatformIO flags (set by script when --mode dev)
+
+**Output Files:**
+- Standard mode: `firmware.bin`, `partitions.bin` in generated/artifacts/
+- Deploy mode: `<env>-firmware.bin`, `<env>-bootloader.bin`, `<env>-partitions.bin`
+
+---
+
+### ci.sh site - Build Documentation
+
+Builds VuePress documentation website with version management and WebUploader manifest generation.
+
+**Usage:**
+```bash
+./scripts/ci.sh site [OPTIONS]
+```
+
+**Options:**
+- `--mode <dev|prod>` - Documentation mode (default: prod)
+  - `dev` - Development documentation with watermark
+  - `prod` - Production documentation
+- `--version-source <release|custom>` - Version source (default: release)
+  - `release` - Use git tag as version
+  - `custom` - Use custom version string
+- `--custom-version <version>` - Custom version string (requires --version-source custom)
+- `--url-prefix <path>` - Base URL path (default: /)
+  - Example: `/dev/` for development subdirectory
+- `--no-webuploader` - Skip WebUploader manifest generation
+- `--webuploader-args <args>` - Additional arguments for gen_wu.py
+- `--preview` - Open browser after build (local development)
+- `--help` - Show help message
+
+**Execution Flow:**
+```
+ci.sh site --mode prod --version-source release
+  │
+  ├─> ci_site.sh (orchestrator)
+  │     ├─> check_requirements() - Verify node, npm, python3, pip3
+  │     ├─> install_dependencies() - npm install, pip3 install packages
+  │     ├─> download_common_config() - Fetch from theengs.io
+  │     ├─> get_version() - Extract from git tag or use custom
+  │     ├─> set_version() - Update VuePress config and JSON files
+  │     ├─> set_url_prefix() - Set base path in config
+  │     ├─> generate_board_docs.py - Auto-generate board documentation
+  │     ├─> npm run docs:build - Build VuePress site
+  │     └─> gen_wu.py - Generate WebUploader manifest
+  │
+  └─> Outputs in generated/site/
+        ├─ index.html
+        ├─ assets/
+        └─ [board documentation pages]
+```
+
+**Examples:**
+```bash
+# Production documentation
+./scripts/ci.sh site --mode prod
+
+# Development documentation with custom version
+./scripts/ci.sh site --mode dev --version-source custom --custom-version "DEVELOPMENT SHA:abc123"
+
+# Documentation for /dev/ subdirectory
+./scripts/ci.sh site --mode dev --url-prefix /dev/
+
+# Skip WebUploader manifest
+./scripts/ci.sh site --no-webuploader
+
+# Local preview
+./scripts/ci.sh site --preview
+```
+
+**Required Tools:**
+- Node.js (for VuePress)
+- npm (for package management)
+- Python 3 (for board docs generator)
+- pip3 (for Python dependencies: requests, pandas, markdown, pytablereader, tabulate)
+
+**Output Files:**
+- `generated/site/` - Complete static website
+- `scripts/latest_version.json` - Production version metadata (updated)
+- `scripts/latest_version_dev.json` - Development version metadata (updated)
+
+---
+
+### ci.sh qa - Code Formatting Check
+
+Checks and fixes code formatting using clang-format.
+
+**Usage:**
+```bash
+./scripts/ci.sh qa [OPTIONS]
+```
+
+**Options:**
+- `--check` - Check formatting without modifying files (default)
+- `--fix` - Automatically fix formatting issues
+- `--source <dir>` - Source directory to check (default: main)
+- `--extensions <list>` - File extensions to check, comma-separated (default: h,ino,cpp)
+- `--clang-format-version <ver>` - clang-format version to use (default: 9)
+- `--verbose` - Show detailed output for each file
+- `--help` - Show help message
+
+**Execution Flow:**
+```
+ci.sh qa --check --source main --extensions h,ino
+  │
+  ├─> ci_qa.sh (orchestrator)
+  │     ├─> check_clang_format() - Find clang-format-9 or clang-format
+  │     ├─> find_files() - Locate files matching extensions in source dir
+  │     └─> check_formatting() - Run clang-format --dry-run --Werror
+  │           └─> Report files with formatting issues
+  │
+  └─> Exit code: 0 (pass) or 1 (formatting issues found)
+```
+
+**Examples:**
+```bash
+# Check formatting (CI mode)
+./scripts/ci.sh qa --check
+
+# Fix formatting automatically
+./scripts/ci.sh qa --fix
+
+# Check specific directory
+./scripts/ci.sh qa --check --source lib
+
+# Check only .h and .ino files
+./scripts/ci.sh qa --check --extensions h,ino
+
+# Check with verbose output
+./scripts/ci.sh qa --check --verbose
+
+# Use different clang-format version
+./scripts/ci.sh qa --check --clang-format-version 11
+```
+
+**Required Tools:**
+- clang-format (version specified, default: 9)
+  - Install: `sudo apt-get install clang-format-9`
+
+**Output:**
+- Check mode: Lists files with formatting issues and shows diffs
+- Fix mode: Modifies files in-place and reports changes
+- Exit code 0: All files properly formatted
+- Exit code 1: Formatting issues found (in check mode)
+
+---
+
+## Internal Scripts
+
+These scripts are called by the main orchestrators. It can be invoked directly but is not raccomanded.
+
+### ci_set_version.sh
+
+Injects version string into firmware configuration files.
+
+**Called By:** `ci_build.sh`
+
+**Usage:**
+```bash
+./scripts/ci_set_version.sh <version> [--dev]
+```
+
+**Arguments:**
+- `<version>` - Version string to inject (e.g., v1.8.0 or abc123)
+- `--dev` - Development mode (updates latest_version_dev.json)
+
+**Files Modified:**
+- `main/User_config.h` - Replaces "version_tag" with actual version
+- `scripts/latest_version.json` - Production version metadata
+- `scripts/latest_version_dev.json` - Development version metadata (--dev mode)
+
+**Behavior:**
+- Creates .bak backup files before modification
+- Replaces all occurrences of "version_tag" string
+- Validates version string (must not be empty or "version_tag")
+- Cleans up backup files on success
+
+---
+
+### ci_build_firmware.sh
+
+Executes PlatformIO build for specified environment.
+
+**Called By:** `ci_build.sh`
+
+**Usage:**
+```bash
+./scripts/ci_build_firmware.sh <environment> [OPTIONS]
+```
+
+**Arguments:**
+- `<environment>` - PlatformIO environment name
+
+**Options:**
+- `--dev-ota` - Enable development OTA (sets PLATFORMIO_BUILD_FLAGS)
+- `--clean` - Clean before build
+- `--verbose` - Verbose PlatformIO output
+
+**Environment Variables Set:**
+- `PYTHONIOENCODING=utf-8`
+- `PYTHONUTF8=1`
+- `PLATFORMIO_BUILD_FLAGS="-DDEVELOPMENTOTA=true"` (when --dev-ota)
+
+**PlatformIO Command:**
+```bash
+platformio run -e <environment> [--verbose]
+```
+
+**Output Location:**
+- `.pio/build/<environment>/firmware.bin`
+- `.pio/build/<environment>/bootloader.bin` (ESP32 only)
+- `.pio/build/<environment>/partitions.bin` (ESP32 only)
+
+---
+
+### ci_prepare_artifacts.sh
+
+Packages firmware binaries from PlatformIO build directory.
+
+**Called By:** `ci_build.sh`
+
+**Usage:**
+```bash
+./scripts/ci_prepare_artifacts.sh <environment> [OPTIONS]
+```
+
+**Arguments:**
+- `<environment>` - PlatformIO environment name
+
+**Options:**
+- `--deploy` - Use deployment naming (prefix with environment name)
+- `--output <dir>` - Output directory (default: generated/artifacts)
+
+**Behavior:**
+
+Standard mode (no --deploy):
+- Copies: `firmware.bin`, `partitions.bin`
+- Does NOT copy: `bootloader.bin`
+
+Deploy mode (with --deploy):
+- Copies and renames:
+  - `firmware.bin` → `<env>-firmware.bin`
+  - `bootloader.bin` → `<env>-bootloader.bin`
+  - `partitions.bin` → `<env>-partitions.bin`
+
+**Source Location:**
+- `.pio/build/<environment>/`
+
+**Output Location:**
+- Specified by `--output` or default `generated/artifacts/`
+
+---
+
+### ci_00_config.sh
+
+Shared configuration and helper functions for all CI scripts.
+
+**Sourced By:** All ci_*.sh scripts
+
+**Provides:**
+- Color codes for terminal output (BLUE, GREEN, RED, YELLOW, NC)
+- Logging functions: `log_info()`, `log_warn()`, `log_error()`, `log_success()`
+- Path constants: `BUILD_DIR`, `ARTIFACTS_DIR`, `SITE_DIR`
+- Common utility functions
+
+**Constants Defined:**
+- `BUILD_DIR=".pio/build"` - PlatformIO build directory
+- `ARTIFACTS_DIR="generated/artifacts"` - Artifact output directory
+- `SITE_DIR="generated/site"` - Documentation output directory
+
+**Logging Functions:**
+```bash
+log_info "message"    # Blue [INFO] prefix
+log_warn "message"    # Yellow [WARN] prefix
+log_error "message"   # Red [ERROR] prefix
+log_success "message" # Green [SUCCESS] prefix
+```
+
+---
+
+## Python Helper Scripts
+
+Other scripts are present and used as internal scripts and it's used as retrocompatibility. Below the lists:
+ - `generate_board_docs.py`
+ - `gen_wu.py`
+
+### generate_board_docs.py
+
+Auto-generates board-specific documentation pages from platformio.ini.
+
+**Called By:** `ci_site.sh`
+
+**Usage:**
+```bash
+python3 ./scripts/generate_board_docs.py
+```
+
+**Input:**
+- `platformio.ini` - Board configurations
+- `environments.ini` - Additional environments
+
+**Output:**
+- Markdown files in `docs/` directory for each board configuration
+
+**Purpose:**
+- Creates documentation pages for each hardware board
+- Extracts configuration details from PlatformIO environment definitions
+- Formats technical specifications and pin mappings
+
+---
+
+### gen_wu.py
+
+Generates WebUploader manifest for OTA firmware updates.
+
+**Called By:** `ci_site.sh`
+
+**Usage:**
+```bash
+python3 ./scripts/gen_wu.py [--dev] [repository]
+```
+
+**Arguments:**
+- `--dev` - Generate development manifest
+- `repository` - GitHub repository name (e.g., 1technophile/OpenMQTTGateway)
+
+**Input:**
+- `.pio/build/<env>/firmware.bin` - Compiled firmware files
+- `scripts/latest_version.json` or `scripts/latest_version_dev.json`
+
+**Output:**
+- WebUploader manifest JSON file in `docs/.vuepress/public/`
+
+**Purpose:**
+- Creates manifest for web-based firmware updater
+- Lists available firmware files with metadata
+- Used by documentation site for OTA updates
+
+---
+
+## Environment Variables
+
+Scripts respect these environment variables:
+
+- `PYTHONIOENCODING=utf-8`: Python encoding
+- `PYTHONUTF8=1`: UTF-8 mode
+- `PLATFORMIO_BUILD_FLAGS`: Custom build flags
+- `ESP32_PLATFORM_VERSION`: Extracted automatically
+
+---
+
+## Exit Codes
+
+All scripts use standard exit codes:
+
+- `0` - Success
+- `1` - General error or failure
+- `2` - Missing required tools or dependencies
+
+Scripts use `set -euo pipefail` for strict error handling:
+- `-e` - Exit on error
+- `-u` - Exit on undefined variable
+- `-o pipefail` - Exit on pipe failure
+
+---
+
+## Environment Detection
+
+Scripts automatically detect if running in CI/CD:
+
+```bash
+if [[ "${CI:-false}" == "true" ]]; then
+    # Running in CI/CD
+    # Disable interactive prompts
+    # Use different output formatting
+fi
+```
+
+CI/CD environments typically set:
+- `CI=true`
+- `GITHUB_ACTIONS=true` (GitHub Actions)
+- `BUILD_NUMBER` (build number)
+- `GIT_COMMIT` (commit hash)
+
+
+---
+
+This documentation reflects the current implementation of CI/CD scripts. All scripts are located in `./scripts/` directory.
+
+For GitHub Actions workflow documentation, see `.github/workflows/README.md`.