chriz/OpenMQTT-Gateway
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity

[SITE] Renew the web board presentation and the ESP32 web upload + [SYS] Security checks (#2277)

Browse Source 
* Refactor GitHub Actions workflows for build, documentation, and linting

- Consolidated build logic into reusable workflows (`task-build.yml` and `task-docs.yml`) to reduce duplication across multiple workflows.
- Introduced `environments.json` to centralize the list of PlatformIO build environments, improving maintainability and clarity.
- Updated `build.yml` and `build_and_docs_to_dev.yml` to utilize the new reusable workflows and environment definitions.
- Enhanced `release.yml` to streamline the release process and integrate documentation generation.
- Created reusable linting workflow (`task-lint.yml`) to standardize code formatting checks across the repository.
- Simplified manual documentation workflow by leveraging the new reusable documentation workflow.
- Improved artifact management and retention policies across workflows.
- Updated dependencies and versions in workflows to ensure compatibility and performance.

CI/CD pipeline agnostic of Workflow Engine and integrated on github actions

- Implemented ci.sh for orchestrating the complete build pipeline.
- Created ci_00_config.sh for centralized configuration of build scripts.
- Created ci_build_firmware.sh for building firmware for specified PlatformIO environments.
- Created ci_prepare_artifacts.sh for preparing firmware artifacts for upload or deployment.
- Created ci_set_version.sh for updating version tags in firmware configuration files.
- Created ci_build.sh to orchestrate the complete build pipeline.
- Created ci_qa.sh for code linting and formatting checks using clang-format.
- Created ci_site.sh for building and deploying VuePress documentation with version management.
- Implemented checks for required tools and dependencies in the new scripts.
- Improved internal scripts for better error handling and logging.

UPDATE the web installer manifest generation and update documentation structure
- Enhanced ci_list-env.sh to list environments from a JSON file.
- Replaced  common_wu.py and gen_wu.py scripts with new npm scripts for site generation and previewing on docsgen/gen_wu.js
- Replaced  generate_board_docs.py with docsgen/generated_board_docs.js
- Added new npm scripts for integration of site generation on build phase.
- Created preview_site.js to serve locally generated site over HTTPS with improved error handling.
- Added new CI environments for CI builds in environments.json.
- Deleted lint.yml as part of workflow cleanup.
- Enhanced task-build.yml to include linting as a job and added support for specifying PlatformIO version.
- Improved task-docs.yml to handle versioning more effectively and added clean option.

Enhance documentation
- ADD CLEAR Mark of development version of site
- Updated README.md to include detailed workflow dependencies and relationships using mermaid diagrams.
- Improved development.md with a quick checklist for contributors and clarified the code style guide.
- Enhanced quick_start.md with tips for contributors and streamlined the workflow explanation.

LINT FIX
- Refined User_config.h for better formatting consistency.
- Adjusted blufi.cpp and gatewayBT.cpp for improved code readability and consistency in formatting.
- Updated gatewaySERIAL.cpp and mqttDiscovery.cpp to enhance logging error messages.
- Improved sensorDS1820.cpp for better logging of device information.

Add security scan workflows for vulnerability detection

Add SBOM generation and upload to release workflow; update security scan summary handling

Add shellcheck suppor + FIX shellcheck warning

Enhance documentation for CI/CD scripts and workflows, adding details for security scanning and SBOM generation processes

Fix formatting and alignment in BLE connection handling

Reviewed the full web board presentation and the ESP32 web upload. The project uses a modern pattern where data is divided from the presentation layer.

- Removed the `generate_board_docs` script.
- Updated the `gen_wu` script in order to generate `boards-info.json`: the fail that containe all information about the configuration
- Created and isolate the file `boards-info.js` to streamline the parsing of PlatformIO dependencies, modules, environments and improve the handling of library information.
- Introduced vuepress component `BoardEnvironmentTable.vue` that render `boards-info.json` as UI card component
- Introduced vuepress component `FlashEnvironmentSelector.vue` that render a selectred environment from  `boards-info.json` and provide esp-web-upload feature on it
- Introduced a new board page `board-selector.md` for improved firmware selection.
- Updated `web-install.md` to enhance the firmware upload process, including a new board environment table.
- Enhanced custom descriptions in `environments.ini` to include HTML links for better user guidance and board image link

Add CC1101 initialization improvements and logging enhancements
Add installation step for PlatformIO dependencies in documentation workflow

Remove ci_set_version.sh script and associated versioning functionality

* Fix comment provisined

Fix PlatformIO version input reference in documentation workflow

Remove outdated Squeezelite-ESP32 installer documentation
This commit is contained in:
Alessandro Staniscia authored and GitHub committed 2026-03-09 07:47:30 -05:00
1 parent 5f0966c2c1
commit 98481c5145
73 files changed
+5985 -2447
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/CI_SCRIPTS.md
+427 -212
View File
@@ -11,11 +11,13 @@ This documentation describes the CI/CD scripts used to build OpenMQTTGateway fir
- [ci.sh - Main Entry Point](#commands)
- [ci.sh build - Build Firmware](#cish-build---build-firmware)
- [ci.sh site - Build Documentation](#cish-site---build-documentation)
- [ci.sh qa - Code Formatting Check](#cish-qa---code-formatting-check)
- [ci.sh qa - Code Quality Checks](#cish-qa---code-quality-checks)
- [ci.sh security - Security Vulnerability Scan](#cish-security---security-vulnerability-scan)
- [Internal Scripts](#internal-scripts)
- [ci_set_version.sh](#ci_set_versionsh)
- [ci_list-env.sh](#ci_list-envsh)
- [ci_build_firmware.sh](#ci_build_firmwaresh)
- [ci_prepare_artifacts.sh](#ci_prepare_artifactssh)
- [ci_security.sh](#ci_securitysh)
- [ci_00_config.sh](#ci_00_configsh)
- [Python Helper Scripts](#python-helper-scripts)
- [generate_board_docs.py](#generate_board_docspy)
@@ -23,6 +25,7 @@ This documentation describes the CI/CD scripts used to build OpenMQTTGateway fir
- [Environment Variables](#environment-variables)
- [Exit Codes](#exit-codes)
- [Environment Detection](#environment-detection)
- [GitHub Actions Workflows Integration](#github-actions-workflows-integration)
@@ -32,27 +35,28 @@ This documentation describes the CI/CD scripts used to build OpenMQTTGateway fir
```
ci.sh (dispatcher)
├── build → ci_build.sh → ci_set_version.sh
│ → ci_build_firmware.sh
│ → ci_prepare_artifacts.sh
├── site → ci_site.sh → generate_board_docs.py
│ → gen_wu.py
├── qa → ci_qa.sh → clang-format
└── all → qa + build + site (sequential)
├── build → ci_build.sh → ci_build_firmware.sh
│ → ci_prepare_artifacts.sh (when --deploy-ready)
├── site → ci_site.sh
├── qa → ci_qa.sh
├── security → ci_security.sh (vulnerability scanning with Trivy)
├── list-env → ci_list-env.sh
└── all → qa + build (all envs with --mode) + site sequential
```
### Script Description
| Script | Purpose | Called By |
|--------|---------|-----------|
|--------|---------|----------|
| `ci.sh` | Main command dispatcher | User/GitHub Actions |
| `ci_build.sh` | Build firmware orchestrator | ci.sh build |
| `ci_site.sh` | Documentation build orchestrator | ci.sh site |
| `ci_qa.sh` | Code formatting checker | ci.sh qa |
| `ci_set_version.sh` | Version injection in firmware | ci_build.sh |
| `ci_00_config.sh` | Shared configuration loader | ci.sh |
| `ci_qa.sh` | Quality assurance and shellcheck | ci.sh qa |
| `ci_security.sh` | Security vulnerability scanning with Trivy | ci.sh security |
| `ci_list-env.sh` | List PlatformIO environments (fast JSON or full scan) | ci.sh list-env / direct |
| `ci_build_firmware.sh` | PlatformIO build execution | ci_build.sh |
| `ci_prepare_artifacts.sh` | Artifact packaging | ci_build.sh |
| `ci_00_config.sh` | Shared configuration and functions | All scripts |
| `ci_prepare_artifacts.sh` | Artifact packaging (when requested) | ci_build.sh |
### Output Structure
@@ -67,16 +71,14 @@ Build outputs are organized in the project root:
generated/
├── artifacts/ # Packaged firmware artifacts
└── site/ # Built documentation (VuePress output)
scripts/
├── latest_version.json # Production version metadata
└── latest_version_dev.json # Development version metadata
├── site/ # Built documentation (VuePress output)
└── reports/ # Security scan and SBOM reports
└── sbom/ # SBOM in CycloneDX and SPDX formats
```
## Commands
`ci.sh` is the main Entry Point. Command dispatcher that routes to specialized scripts.
`ci.sh` is the main entry point. It dispatches to the specialized scripts.
**Usage:**
```bash
@@ -84,115 +86,90 @@ scripts/
```
**Commands:**
- `build` - Build firmware for specified environment
- `site` or `docs` - Build documentation website
- `qa` or `lint` - Run code formatting checks
- `all` or `pipeline` - Run complete pipeline (qa + build + site)
- `build` - Build firmware for a PlatformIO environment (optionally prepare artifacts)
- `site` or `docs` - Build documentation site
- `qa` or `lint` - Run formatting and shellcheck checks
- `security` - Scan for security vulnerabilities using Trivy (filesystem, container images)
- `all` or `pipeline` - Run qa → build (all environments) → site with injected mode
- `list-env` - List available PlatformIO environments (JSON fast list or full ini scan)
**Examples:**
```bash
# Get Help
# Help per command
./scripts/ci.sh build --help
./scripts/ci.sh qa --help
./scripts/ci.sh site --help
./scripts/ci.sh qa --help
./scripts/ci.sh security --help
./scripts/ci.sh list-env --help
# Build firmware
./scripts/ci.sh build esp32dev-ble --mode dev
./scripts/ci.sh build esp32dev-all-test --version v1.8.0 --deploy-ready
./scripts/ci.sh build esp32dev-all-test --mode dev
./scripts/ci.sh build esp32dev-bt --version v1.8.0 --deploy-ready --output generated/artifacts
# Build documentation
./scripts/ci.sh site --mode prod
./scripts/ci.sh site --mode dev --url-prefix /dev/
# Build docs
./scripts/ci.sh site --mode prod --url-prefix /
./scripts/ci.sh site --mode dev --preview
# Check code formatting
# QA (formatting + shellcheck)
./scripts/ci.sh qa --check
./scripts/ci.sh qa --fix
./scripts/ci.sh qa --fix --verbose
# Run complete pipeline
./scripts/ci.sh all esp32dev-ble --version v1.8.0
./scripts/ci.sh all esp32dev-ble --no-site
# Security scanning
./scripts/ci.sh security --scan-type fs --severity HIGH,CRITICAL
./scripts/ci.sh security --scan-type fs --generate-sbom
# Full pipeline (qa + build all envs + site)
./scripts/ci.sh all --mode dev
./scripts/ci.sh all --mode prod --preview
```
**Options for `all` command:**
- `--no-site` - Skip documentation build step
- All options from `build` command are passed through
---
### ci.sh build - Build Firmware
Orchestrates complete firmware build: version injection, compilation, artifact packaging.
Runs the build pipeline (tool checks → PlatformIO build → optional artifact packaging).
**Usage:**
```bash
./scripts/ci.sh build <environment> [OPTIONS]
```
**Required Arguments:**
- `<environment>` - PlatformIO environment name (e.g., esp32dev-ble, nodemcuv2-rf)
**Required Argument:**
- `<environment>` PlatformIO environment name (e.g., esp32dev-ble)
**Options:**
- `--version <tag>` - Version string to inject (default: auto-generated from git)
- `--mode <dev|prod>` - Build mode (default: prod)
- `dev` - Enables development OTA, sets DEVELOPMENTOTA=true
- `prod` - Standard production build
- `--deploy-ready` - Package artifacts with deployment naming (env-firmware.bin)
- `--output <dir>` - Output directory for artifacts (default: generated/artifacts)
- `--skip-verification` - Skip build tools verification
- `--clean` - Clean previous build before starting
- `--verbose` - Enable verbose PlatformIO output
- `--list-envs` - List all available PlatformIO environments
- `--help` - Show help message
- `--mode <prod|dev>` Build mode (default: prod). `dev` enables OTA flags in the PlatformIO build.
- `--deploy-ready` Copy/rename build outputs and libs via ci_prepare_artifacts.sh.
- `--version [TAG]` Set version used for `OMG_VERSION` and artifact folder naming. If TAG is omitted the script auto-generates (CI: BUILD_NUMBER/GIT_COMMIT, local: timestamp).
- `--output <dir>` Output directory for packaged artifacts (only used when `--deploy-ready` is set, default `generated/artifacts`).
- `--skip-verification` Skip the tool availability checks.
- `--clean` Clean the PlatformIO environment before building.
- `--verbose` Verbose PlatformIO output.
- `--help` Show help.
**Execution Flow:**
```
ci.sh build esp32dev-ble --version v1.8.0 --mode prod --deploy-ready
│
├─> ci_build.sh (orchestrator)
│ ├─> verify_build_tools() - Check python3, platformio, git
│ ├─> ci_set_version.sh v1.8.0 - Inject version in User_config.h
│ ├─> ci_build_firmware.sh esp32dev-ble - Execute PlatformIO build
│ └─> ci_prepare_artifacts.sh esp32dev-ble --deploy - Package binaries
│
└─> Outputs in generated/artifacts/
├─ esp32dev-ble-firmware.bin
├─ esp32dev-ble-bootloader.bin
└─ esp32dev-ble-partitions.bin
```
**Behavior:**
- Tool check verifies python3, platformio, git (can be skipped).
- Builds via ci_build_firmware.sh (adds `--dev-ota` when `--mode dev`).
- Packaging runs only when `--deploy-ready` is provided; artifacts land under `generated/artifacts/firmware_build/` with env-prefixed filenames plus zipped libraries.
<!-- - Packaging runs only when `--deploy-ready` is provided; artifacts land under `generated/artifacts[/<version>]/firmware_build/` with env-prefixed filenames plus zipped libraries. -->
**Examples:**
```bash
# Development build
# Dev build with OTA flags
./scripts/ci.sh build esp32dev-ble --mode dev
# Production build with version
./scripts/ci.sh build esp32dev-ble --version v1.8.0 --mode prod
# Prod build with auto-version and deployable artifacts
./scripts/ci.sh build esp32dev-bt --version --deploy-ready --output generated/artifacts
# Deploy-ready build
./scripts/ci.sh build esp32dev-all-test --version v1.8.0 --deploy-ready
# Clean build with verbose output
# Clean + verbose build
./scripts/ci.sh build nodemcuv2-rf --clean --verbose
# List available environments
./scripts/ci.sh build --list-envs
```
**Environment Variables:**
- `CI` - Set to 'true' in CI/CD environments
- `BUILD_NUMBER` - Build number from CI/CD system
- `GIT_COMMIT` - Git commit hash for auto-versioning
- `PLATFORMIO_BUILD_FLAGS` - Additional PlatformIO flags (set by script when --mode dev)
**Output Files:**
- Standard mode: `firmware.bin`, `partitions.bin` in generated/artifacts/
- Deploy mode: `<env>-firmware.bin`, `<env>-bootloader.bin`, `<env>-partitions.bin`
---
### ci.sh site - Build Documentation
Builds VuePress documentation website with version management and WebUploader manifest generation.
Builds the VuePress documentation site.
**Usage:**
```bash
@@ -200,75 +177,36 @@ Builds VuePress documentation website with version management and WebUploader ma
```
**Options:**
- `--mode <dev|prod>` - Documentation mode (default: prod)
- `dev` - Development documentation with watermark
- `prod` - Production documentation
- `--version-source <release|custom>` - Version source (default: release)
- `release` - Use git tag as version
- `custom` - Use custom version string
- `--custom-version <version>` - Custom version string (requires --version-source custom)
- `--url-prefix <path>` - Base URL path (default: /)
- Example: `/dev/` for development subdirectory
- `--no-webuploader` - Skip WebUploader manifest generation
- `--webuploader-args <args>` - Additional arguments for gen_wu.py
- `--preview` - Open browser after build (local development)
- `--help` - Show help message
- `--mode <prod|dev>` Build mode (default: dev).
- `--url-prefix <path>` Base URL path for links (e.g., '/' for root, '/dev/' for dev) (default: /dev/).
- `--version <tag>` Version string written to meta.json (default: edge).
- `--preview` Start the local HTTPS preview server after building (https://localhost:8443).
- `--clean` Remove generated/site folder before building.
- `--insecure-curl` Allow curl to skip TLS verification when downloading common config.
- `--help` Show help.
**Execution Flow:**
```
ci.sh site --mode prod --version-source release
│
├─> ci_site.sh (orchestrator)
│ ├─> check_requirements() - Verify node, npm, python3, pip3
│ ├─> install_dependencies() - npm install, pip3 install packages
│ ├─> download_common_config() - Fetch from theengs.io
│ ├─> get_version() - Extract from git tag or use custom
│ ├─> set_version() - Update VuePress config and JSON files
│ ├─> set_url_prefix() - Set base path in config
│ ├─> generate_board_docs.py - Auto-generate board documentation
│ ├─> npm run docs:build - Build VuePress site
│ └─> gen_wu.py - Generate WebUploader manifest
│
└─> Outputs in generated/site/
├─ index.html
├─ assets/
└─ [board documentation pages]
```
**Behavior:**
- Checks for node, npm, openssl; installs npm deps; downloads commonConfig.js.
- Writes docs/.vuepress/meta.json with mode/url_prefix/version; builds via `npm run docs:build`.
- Preview mode runs `npm run site:preview`.
**Examples:**
```bash
# Production documentation
./scripts/ci.sh site --mode prod
# Production build
./scripts/ci.sh site --mode prod --url-prefix /
# Development documentation with custom version
./scripts/ci.sh site --mode dev --version-source custom --custom-version "DEVELOPMENT SHA:abc123"
# Development build with preview
./scripts/ci.sh site --mode dev --url-prefix /dev/ --version edge --preview
# Documentation for /dev/ subdirectory
./scripts/ci.sh site --mode dev --url-prefix /dev/
# Skip WebUploader manifest
./scripts/ci.sh site --no-webuploader
# Local preview
./scripts/ci.sh site --preview
# Clean then build with custom version
./scripts/ci.sh site --clean --version 1.8.0
```
**Required Tools:**
- Node.js (for VuePress)
- npm (for package management)
- Python 3 (for board docs generator)
- pip3 (for Python dependencies: requests, pandas, markdown, pytablereader, tabulate)
**Output Files:**
- `generated/site/` - Complete static website
- `scripts/latest_version.json` - Production version metadata (updated)
- `scripts/latest_version_dev.json` - Development version metadata (updated)
---
### ci.sh qa - Code Formatting Check
### ci.sh qa - Code Quality Checks
Checks and fixes code formatting using clang-format.
Checks and fixes code formatting using clang-format and runs shellcheck on shell scripts.
**Usage:**
```bash
@@ -276,13 +214,16 @@ Checks and fixes code formatting using clang-format.
```
**Options:**
- `--check` - Check formatting without modifying files (default)
- `--fix` - Automatically fix formatting issues
- `--source <dir>` - Source directory to check (default: main)
- `--extensions <list>` - File extensions to check, comma-separated (default: h,ino,cpp)
- `--clang-format-version <ver>` - clang-format version to use (default: 9)
- `--verbose` - Show detailed output for each file
- `--help` - Show help message
- `--check` Check formatting only (default)
- `--fix` Apply formatting in place
- `--format` Run only format checks
- `--shellcheck` Run shellcheck on shell scripts in scripts/ directory
- `--all` Future hook to run all QA checks (current implementation runs formatting + shellcheck)
- `--source <dir>` Source directory for formatting checks (default: main)
- `--extensions <list>` File extensions for formatting (comma-separated, default: h,ino,cpp)
- `--clang-format-version <ver>` clang-format version to use (default: 9)
- `--verbose` Verbose output
- `--help` Show help
**Execution Flow:**
```
@@ -290,27 +231,35 @@ ci.sh qa --check --source main --extensions h,ino
│
├─> ci_qa.sh (orchestrator)
│ ├─> check_clang_format() - Find clang-format-9 or clang-format
│ ├─> find_files() - Locate files matching extensions in source dir
│ └─> check_formatting() - Run clang-format --dry-run --Werror
│ └─> Report files with formatting issues
│ │ ├─> find_files() - Locate files matching extensions in source dir
│ │ └─> check_formatting() - Run clang-format --dry-run --Werror
│ │ └─> Report files with formatting issues
│ │
│ └─> shellcheck_check() - Find and scan shell scripts
│ ├─> find_shell_scripts() - Locate *.sh files in scripts/ directory
│ └─> run_shellcheck() - Run shellcheck on found scripts
│ └─> Report shell script issues
│
└─> Exit code: 0 (pass) or 1 (formatting issues found)
└─> Exit code: 0 (pass) or 1 (issues found)
```
**Examples:**
```bash
# Check formatting (CI mode)
# Check both formatting and shellcheck (default)
./scripts/ci.sh qa --check
# Fix formatting automatically
./scripts/ci.sh qa --fix
# Check specific directory
./scripts/ci.sh qa --check --source lib
# Check only formatting for specific directory
./scripts/ci.sh qa --check --format --source lib
# Check only .h and .ino files
./scripts/ci.sh qa --check --extensions h,ino
# Check only shellcheck for shell scripts
./scripts/ci.sh qa --check --shellcheck
# Check with verbose output
./scripts/ci.sh qa --check --verbose
@@ -321,46 +270,201 @@ ci.sh qa --check --source main --extensions h,ino
**Required Tools:**
- clang-format (version specified, default: 9)
- Install: `sudo apt-get install clang-format-9`
- shellcheck (for shell script linting)
- Install: `sudo apt-get install shellcheck`
**Output:**
- Check mode: Lists files with formatting issues and shows diffs
- Fix mode: Modifies files in-place and reports changes
- Exit code 0: All files properly formatted
- Exit code 1: Formatting issues found (in check mode)
- Check mode: Lists files with formatting issues and shell script errors, shows diffs
- Fix mode: Modifies formatting in-place and reports changes
- Exit code 0: All checks passed (proper formatting and no shellcheck errors)
- Exit code 1: Issues found (formatting or shellcheck violations)
---
### ci.sh security - Security Vulnerability Scan
Scans the project for security vulnerabilities using Trivy and generates Software Bill of Materials (SBOM).
**Usage:**
```bash
./scripts/ci.sh security [OPTIONS]
```
**Options:**
- `--scan-type <fs|config|image>` Type of scan (default: fs)
- `fs` - Filesystem scan (default, scans for vulnerabilities and misconfigurations)
- `config` - Configuration scan only
- `image` - Container image scan
- `--severity <levels>` Severity levels to report (comma-separated: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL) (default: HIGH,CRITICAL)
- `--scan-path <path>` Path to scan (default: current directory `.`)
- `--generate-sbom` Generate Software Bill of Materials in CycloneDX and SPDX formats (default: true)
- `--exit-code <0|1>` Exit code when vulnerabilities found (0=continue, 1=fail) (default: 0)
- `--upload-to-security-tab` Upload SARIF report to GitHub Security tab (GitHub Actions only, default: true)
- `--verbose` Verbose output
- `--help` Show help
**Behavior:**
- Installs Trivy if not present
- Scans filesystem or configuration for known vulnerabilities
- Generates multiple report formats: SARIF, JSON, table summary
- Creates SBOM in CycloneDX and SPDX formats when `--generate-sbom` is enabled
- Reports are saved to `generated/reports/` directory
- Summary is appended to GitHub job summary when running in GitHub Actions
- Uploads SARIF to GitHub Security tab for dashboard visibility
**Output Structure:**
```
generated/reports/
├── trivy-results.sarif # SARIF format (for GitHub Security tab)
├── trivy-results.json # JSON format (detailed results)
├── security-summary.md # Markdown summary
└── sbom/
├── sbom.cyclonedx.json # CycloneDX format
└── sbom.spdx.json # SPDX format
```
**Examples:**
```bash
# Scan filesystem for HIGH and CRITICAL vulnerabilities
./scripts/ci.sh security --scan-type fs --severity HIGH,CRITICAL
# Full scan with all severity levels and SBOM generation
./scripts/ci.sh security --scan-type fs --severity UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL --generate-sbom
# Scan specific path with verbose output
./scripts/ci.sh security --scan-path ./lib --verbose
# Configuration scan only
./scripts/ci.sh security --scan-type config
# Scan and fail on vulnerabilities
./scripts/ci.sh security --exit-code 1
```
**Required Tools:**
- Trivy (vulnerability scanner)
- Install: `wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -` && `echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list` && `sudo apt-get update && sudo apt-get install -y trivy`
**Output:**
- Detailed SARIF and JSON reports for integration with tools
- Human-readable Markdown summary with findings count
- SBOM artifacts for supply chain tracking
- GitHub Security tab integration when in GitHub Actions
- Exit code 0: Scan completed (vulnerabilities may have been found)
- Exit code 1: Scan failed or critical vulnerabilities found (only if `--exit-code 1`)
---
### ci.sh all - Complete Pipeline
Runs the complete CI/CD pipeline (qa → build all environments → site) with mode injected to all steps.
**Usage:**
```bash
./scripts/ci.sh all --mode <dev|prod> [--preview]
```
**Required Options:**
- `--mode <dev|prod>` Build mode (required). Injected to all pipeline steps:
- `qa`: Always runs `--check`
- `build`: Passes mode to all PlatformIO environments (dev enables OTA flags)
- `site`: Passes mode for documentation generation
**Optional Options:**
- `--preview` Start local HTTPS preview server at https://localhost:8443 after building the site
- `--help` Show help
**Behavior:**
- No environment argument needed; builds **all** available environments
- All three steps (qa, build, site) receive the same `--mode` value
- If any step fails, the pipeline aborts
- Site build warnings do not abort the pipeline (continues with success status)
**Execution Flow:**
```
ci.sh all --mode dev --preview
│
├─> Step 1: ci_qa.sh --check
│ └─> Exit on failure
│
├─> Step 2: ci_build.sh --mode dev (builds all environments)
│ └─> Exit on failure
│
└─> Step 3: ci_site.sh --mode dev --preview
├─> Build documentation
└─> Start preview server at https://localhost:8443
```
**Examples:**
```bash
# Complete pipeline in dev mode
./scripts/ci.sh all --mode dev
# Complete pipeline in prod mode with site preview
./scripts/ci.sh all --mode prod --preview
# Help for this command
./scripts/ci.sh all --help
```
**Output:**
- Step-by-step progress messages for each pipeline phase
- Final summary showing mode, preview status, duration, and overall status
- Exit code 0: All steps successful
- Exit code 1: Any step failed
---
### ci.sh list-env - List Environments
Lists PlatformIO environments available to build.
**Usage:**
```bash
./scripts/ci.sh list-env [--full]
```
**Options:**
- Default: read .github/workflows/environments.json (requires jq) and show curated list.
- `--full` Parse platformio.ini and environments.ini for an exhaustive list (skips *-test and *-all-).
- `--help` Show help.
**Examples:**
```bash
./scripts/ci.sh list-env
./scripts/ci.sh list-env --full
```
---
## Internal Scripts
These scripts are called by the main orchestrators. It can be invoked directly but is not raccomanded.
These scripts are called by the main orchestrators. They can be run directly for troubleshooting, but the preferred entrypoints are the ci.sh commands.
### ci_set_version.sh
### ci_list-env.sh
Injects version string into firmware configuration files.
Lists PlatformIO environments for OpenMQTTGateway.
**Called By:** `ci_build.sh`
**Called By:** `ci.sh list-env` or direct call
**Usage:**
```bash
./scripts/ci_set_version.sh <version> [--dev]
./scripts/ci_list-env.sh [--full]
```
**Arguments:**
- `<version>` - Version string to inject (e.g., v1.8.0 or abc123)
- `--dev` - Development mode (updates latest_version_dev.json)
**Options:**
- Default: read .github/workflows/environments.json for a curated list (needs jq)
- `--full` - Parse platformio.ini and environments.ini for all envs except *-test and *-all-
- `-h|--help` - Show help
**Files Modified:**
- `main/User_config.h` - Replaces "version_tag" with actual version
- `scripts/latest_version.json` - Production version metadata
- `scripts/latest_version_dev.json` - Development version metadata (--dev mode)
**Output:**
- Shows sorted environments in columns and prints the total count
**Behavior:**
- Creates .bak backup files before modification
- Replaces all occurrences of "version_tag" string
- Validates version string (must not be empty or "version_tag")
- Cleans up backup files on success
---
**Examples:**
```bash
./scripts/ci_list-env.sh
./scripts/ci_list-env.sh --full
```
### ci_build_firmware.sh
@@ -377,14 +481,17 @@ Executes PlatformIO build for specified environment.
- `<environment>` - PlatformIO environment name
**Options:**
- `--version <tag>` - Set OMG_VERSION for the build (passed through from ci_build.sh)
- `--dev-ota` - Enable development OTA (sets PLATFORMIO_BUILD_FLAGS)
- `--clean` - Clean before build
- `--verbose` - Verbose PlatformIO output
- `--no-verify` - Skip artifact verification after build
**Environment Variables Set:**
- `PYTHONIOENCODING=utf-8`
- `PYTHONUTF8=1`
- `PLATFORMIO_BUILD_FLAGS="-DDEVELOPMENTOTA=true"` (when --dev-ota)
- `OMG_VERSION` (when --version is provided)
**PlatformIO Command:**
```bash
@@ -400,7 +507,7 @@ platformio run -e <environment> [--verbose]
### ci_prepare_artifacts.sh
Packages firmware binaries from PlatformIO build directory.
Packages firmware binaries and libraries from a PlatformIO build directory; can also only create source archive when no environment is provided.
**Called By:** `ci_build.sh`
@@ -410,29 +517,19 @@ Packages firmware binaries from PlatformIO build directory.
```
**Arguments:**
- `<environment>` - PlatformIO environment name
- `<environment>` - PlatformIO environment name (optional; if omitted only source archive is created)
**Options:**
- `--deploy` - Use deployment naming (prefix with environment name)
- `--output <dir>` - Output directory (default: generated/artifacts)
- `--output <dir>` Output directory (default: generated/artifacts)
- `--version <tag>` Append a version subfolder inside the output directory
- `--clean` Clean output directory before writing
- `--help` Show help
**Behavior:**
Standard mode (no --deploy):
- Copies: `firmware.bin`, `partitions.bin`
- Does NOT copy: `bootloader.bin`
Deploy mode (with --deploy):
- Copies and renames:
- `firmware.bin` → `<env>-firmware.bin`
- `bootloader.bin` → `<env>-bootloader.bin`
- `partitions.bin` → `<env>-partitions.bin`
**Source Location:**
- `.pio/build/<environment>/`
**Output Location:**
- Specified by `--output` or default `generated/artifacts/`
- If `version` is provided, outputs go to `<output>/<version>/firmware_build/`; otherwise to `<output>/firmware_build/`.
- With an environment: copies and renames firmware.bin/partitions.bin/bootloader.bin/boot_app0.bin to `<env>-*.bin`, then zips libraries for that env into `*-libraries.tgz`.
- Without an environment: only creates `OpenMQTTGateway_sources.tgz` from `main` and `LICENSE.txt`.
- Lists the prepared artifacts and their sizes at the end.
---
@@ -452,6 +549,7 @@ Shared configuration and helper functions for all CI scripts.
- `BUILD_DIR=".pio/build"` - PlatformIO build directory
- `ARTIFACTS_DIR="generated/artifacts"` - Artifact output directory
- `SITE_DIR="generated/site"` - Documentation output directory
- `REPORTS_DIR="generated/reports"` - Security scan and quality reports directory
**Logging Functions:**
```bash
@@ -463,9 +561,74 @@ log_success "message" # Green [SUCCESS] prefix
---
### ci_security.sh
Performs security vulnerability scanning and Software Bill of Materials (SBOM) generation using Trivy.
**Called By:** `ci.sh security`
**Usage:**
```bash
./scripts/ci_security.sh [OPTIONS]
```
**Options:**
- `--scan-type <fs|config|image>` Type of scan (default: fs)
- `fs` - Filesystem scan (default)
- `config` - Configuration scan
- `image` - Container image scan
- `--severity <levels>` Severity levels (comma-separated: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL) (default: HIGH,CRITICAL)
- `--scan-path <path>` Path to scan (default: .)
- `--generate-sbom` Generate SBOM (default: true)
- `--exit-code <0|1>` Exit code behavior (0=continue, 1=fail) (default: 0)
- `--upload-to-security-tab` Upload SARIF to GitHub (default: true)
- `--verbose` Verbose output
- `--help` Show help
**Behavior:**
- Ensures Trivy is installed via package manager
- Creates `generated/reports/` directory structure
- Runs Trivy with specified parameters
- Generates SARIF, JSON, and table formats
- Creates SBOM in CycloneDX and SPDX formats (when enabled)
- Uploads SARIF to GitHub Security tab when `GITHUB_TOKEN` and `--upload-to-security-tab` are set
- Appends summary to GitHub job summary if in GitHub Actions
- Validates critical vulnerabilities and exits with code 1 if found and `--exit-code 1` is set
**Output Files:**
- `generated/reports/trivy-results.sarif` - SARIF format for GitHub integration
- `generated/reports/trivy-results.json` - Full JSON results
- `generated/reports/security-summary.md` - Human-readable summary
- `generated/reports/sbom/sbom.cyclonedx.json` - CycloneDX SBOM
- `generated/reports/sbom/sbom.spdx.json` - SPDX SBOM
**Exit Codes:**
- `0` - Success (vulnerabilities may have been found)
- `1` - Scan failed, critical vulnerabilities found (only if `--exit-code 1`), or missing dependencies
**Trivy Integration:**
- Scans for known vulnerabilities in dependencies
- Detects misconfigurations and insecure practices
- Generates compliant SBOM artifacts
- Provides detailed reporting in multiple formats
**Example:**
```bash
# Scan filesystem with severity filter
./scripts/ci_security.sh --scan-type fs --severity HIGH,CRITICAL
# Local scan with SBOM and JSON output
./scripts/ci_security.sh --scan-type fs --generate-sbom --verbose
# In GitHub Actions with security tab upload
./scripts/ci_security.sh --scan-type fs --severity HIGH,CRITICAL --upload-to-security-tab
```
---
## Python Helper Scripts
Other scripts are present and used as internal scripts and it's used as retrocompatibility. Below the lists:
Legacy helper scripts are kept for compatibility; they are not called by the current ci_site.sh flow. Below the list:
- `generate_board_docs.py`
- `gen_wu.py`
@@ -473,7 +636,7 @@ Other scripts are present and used as internal scripts and it's used as retrocom
Auto-generates board-specific documentation pages from platformio.ini.
**Called By:** `ci_site.sh`
**Called By:** Not invoked by current ci_site.sh (legacy helper)
**Usage:**
```bash
@@ -498,7 +661,7 @@ python3 ./scripts/generate_board_docs.py
Generates WebUploader manifest for OTA firmware updates.
**Called By:** `ci_site.sh`
**Called By:** Not invoked by current ci_site.sh (legacy helper)
**Usage:**
```bash
@@ -527,10 +690,10 @@ python3 ./scripts/gen_wu.py [--dev] [repository]
Scripts respect these environment variables:
- `PYTHONIOENCODING=utf-8`: Python encoding
- `PYTHONUTF8=1`: UTF-8 mode
- `PLATFORMIO_BUILD_FLAGS`: Custom build flags
- `ESP32_PLATFORM_VERSION`: Extracted automatically
- `CI`/`BUILD_NUMBER`/`GIT_COMMIT`: Used by ci_build.sh to auto-generate version when `--version` flag has no tag
- `PYTHONIOENCODING=utf-8`, `PYTHONUTF8=1`: Python encoding settings set by ci_build_firmware.sh
- `PLATFORMIO_BUILD_FLAGS`: Set to include development OTA flag when `--dev-ota` is used
- `OMG_VERSION`: Set when `--version` is passed to ci_build.sh/ci_build_firmware.sh
---
@@ -567,6 +730,58 @@ CI/CD environments typically set:
- `BUILD_NUMBER` (build number)
- `GIT_COMMIT` (commit hash)
---
## GitHub Actions Workflows Integration
The CI scripts integrate with GitHub Actions workflows in `.github/workflows/`:
### task-lint.yml
- Reusable workflow that runs `ci.sh qa --check`
- Installs clang-format and shellcheck
- Validates code formatting and shell script quality
- Can be called with custom source directory and file extensions
### task-build.yml
- Main build workflow orchestrator
- Calls task-lint.yml for code quality checks
- Calls task-security-scan.yml for vulnerability scanning
- Builds firmware for all or specified environments
- Prepares and uploads build artifacts
- Supports matrix builds for multiple environments
- Manages build artifact retention
### task-security-scan.yml
- Reusable security scanning workflow
- Installs Trivy vulnerability scanner
- Calls `ci_security.sh` with configurable parameters
- Generates SARIF, JSON, and SBOM reports
- Uploads SARIF to GitHub Security tab for code scanning dashboard
- Fails build on critical vulnerabilities when configured
- Uploads SBOM artifacts for supply chain tracking
### security-scan.yml
- Scheduled security scanning (runs weekly by default)
- Triggered manually with input parameters
- Allows filtering by severity level
- Configurable exit behavior (fail or continue)
- Optional SBOM generation and upload
**Workflow Dependencies:**
```
task-build.yml
├─> task-lint.yml (linting)
├─> task-security-scan.yml (security scanning)
└─> Build environment matrix (firmware compilation)
```
**Key Features:**
- Parallel linting and security scans
- Artifact retention policies
- GitHub Security tab integration
- Detailed build reports
- SBOM generation for compliance
- Support for custom build parameters
---