[SITE] Renew the web board presentation and the ESP32 web upload + [SYS] Security checks (#2277)
* Refactor GitHub Actions workflows for build, documentation, and linting - Consolidated build logic into reusable workflows (`task-build.yml` and `task-docs.yml`) to reduce duplication across multiple workflows. - Introduced `environments.json` to centralize the list of PlatformIO build environments, improving maintainability and clarity. - Updated `build.yml` and `build_and_docs_to_dev.yml` to utilize the new reusable workflows and environment definitions. - Enhanced `release.yml` to streamline the release process and integrate documentation generation. - Created reusable linting workflow (`task-lint.yml`) to standardize code formatting checks across the repository. - Simplified manual documentation workflow by leveraging the new reusable documentation workflow. - Improved artifact management and retention policies across workflows. - Updated dependencies and versions in workflows to ensure compatibility and performance. CI/CD pipeline agnostic of Workflow Engine and integrated on github actions - Implemented ci.sh for orchestrating the complete build pipeline. - Created ci_00_config.sh for centralized configuration of build scripts. - Created ci_build_firmware.sh for building firmware for specified PlatformIO environments. - Created ci_prepare_artifacts.sh for preparing firmware artifacts for upload or deployment. - Created ci_set_version.sh for updating version tags in firmware configuration files. - Created ci_build.sh to orchestrate the complete build pipeline. - Created ci_qa.sh for code linting and formatting checks using clang-format. - Created ci_site.sh for building and deploying VuePress documentation with version management. - Implemented checks for required tools and dependencies in the new scripts. - Improved internal scripts for better error handling and logging. UPDATE the web installer manifest generation and update documentation structure - Enhanced ci_list-env.sh to list environments from a JSON file. - Replaced common_wu.py and gen_wu.py scripts with new npm scripts for site generation and previewing on docsgen/gen_wu.js - Replaced generate_board_docs.py with docsgen/generated_board_docs.js - Added new npm scripts for integration of site generation on build phase. - Created preview_site.js to serve locally generated site over HTTPS with improved error handling. - Added new CI environments for CI builds in environments.json. - Deleted lint.yml as part of workflow cleanup. - Enhanced task-build.yml to include linting as a job and added support for specifying PlatformIO version. - Improved task-docs.yml to handle versioning more effectively and added clean option. Enhance documentation - ADD CLEAR Mark of development version of site - Updated README.md to include detailed workflow dependencies and relationships using mermaid diagrams. - Improved development.md with a quick checklist for contributors and clarified the code style guide. - Enhanced quick_start.md with tips for contributors and streamlined the workflow explanation. LINT FIX - Refined User_config.h for better formatting consistency. - Adjusted blufi.cpp and gatewayBT.cpp for improved code readability and consistency in formatting. - Updated gatewaySERIAL.cpp and mqttDiscovery.cpp to enhance logging error messages. - Improved sensorDS1820.cpp for better logging of device information. Add security scan workflows for vulnerability detection Add SBOM generation and upload to release workflow; update security scan summary handling Add shellcheck suppor + FIX shellcheck warning Enhance documentation for CI/CD scripts and workflows, adding details for security scanning and SBOM generation processes Fix formatting and alignment in BLE connection handling Reviewed the full web board presentation and the ESP32 web upload. The project uses a modern pattern where data is divided from the presentation layer. - Removed the `generate_board_docs` script. - Updated the `gen_wu` script in order to generate `boards-info.json`: the fail that containe all information about the configuration - Created and isolate the file `boards-info.js` to streamline the parsing of PlatformIO dependencies, modules, environments and improve the handling of library information. - Introduced vuepress component `BoardEnvironmentTable.vue` that render `boards-info.json` as UI card component - Introduced vuepress component `FlashEnvironmentSelector.vue` that render a selectred environment from `boards-info.json` and provide esp-web-upload feature on it - Introduced a new board page `board-selector.md` for improved firmware selection. - Updated `web-install.md` to enhance the firmware upload process, including a new board environment table. - Enhanced custom descriptions in `environments.ini` to include HTML links for better user guidance and board image link Add CC1101 initialization improvements and logging enhancements Add installation step for PlatformIO dependencies in documentation workflow Remove ci_set_version.sh script and associated versioning functionality * Fix comment provisined Fix PlatformIO version input reference in documentation workflow Remove outdated Squeezelite-ESP32 installer documentation
This commit is contained in:
Alessandro Staniscia authored and GitHub committed
1 parent
5f0966c2c1
commit
98481c5145
73 files changed
+5985
-2447
No files matched your search
+387
-282
@@ -1,8 +1,7 @@
|
||||
# GitHub Actions Workflows Documentation
|
||||
|
||||
This document provides a overview of all GitHub Actions workflows in the OpenMQTTGateway project.
|
||||
This document provides an overview of all GitHub Actions workflows in the OpenMQTTGateway project.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
@@ -14,7 +13,7 @@ Entry points triggered by user actions, schedules, or events:
|
||||
- `build_and_docs_to_dev.yml` - Daily development builds
|
||||
- `release.yml` - Production releases
|
||||
- `manual_docs.yml` - Documentation deployment
|
||||
- `lint.yml` - Code formatting check
|
||||
- `security-scan.yml` - Security vulnerability scanning
|
||||
- `stale.yml` - Issue management
|
||||
|
||||
### **Task Workflows** (Reusable components)
|
||||
@@ -22,8 +21,8 @@ Parameterized building blocks called by main workflows:
|
||||
- `task-build.yml` - Configurable firmware build
|
||||
- `task-docs.yml` - Configurable documentation build
|
||||
- `task-lint.yml` - Configurable code formatting check
|
||||
- `task-security-scan.yml` - Configurable security scanning
|
||||
|
||||
---
|
||||
|
||||
## Workflow Overview Table
|
||||
|
||||
@@ -33,11 +32,188 @@ Parameterized building blocks called by main workflows:
|
||||
| `build_and_docs_to_dev.yml` | Daily Cron, Manual | Development Builds + Docs | Firmware + Docs deployment |
|
||||
| `release.yml` | Release Published | Production Release | Release assets + Docs |
|
||||
| `manual_docs.yml` | Manual, Workflow Call | Documentation Only | GitHub Pages docs |
|
||||
| `lint.yml` | Push, Pull Request | Code Format Check | None |
|
||||
| `security-scan.yml` | Weekly Cron, Manual | Security Vulnerability Scanning | SARIF, SBOM reports |
|
||||
| `stale.yml` | Daily Cron | Issue Management | None |
|
||||
| **`task-build.yml`** | **Workflow Call** | **Reusable Build Logic** | **Configurable** |
|
||||
| **`task-docs.yml`** | **Workflow Call** | **Reusable Docs Logic** | **GitHub Pages** |
|
||||
| **`task-lint.yml`** | **Workflow Call** | **Reusable Lint Logic** | **None** |
|
||||
| **`task-security-scan.yml`** | **Workflow Call** | **Reusable Security Scan** | **SARIF, SBOM, Reports** |
|
||||
|
||||
|
||||
|
||||
|
||||
## Workflow Dependencies and Call Chain
|
||||
|
||||
```mermaid
|
||||
|
||||
flowchart TD
|
||||
%% Triggers
|
||||
subgraph triggers ["🎯 Triggers"]
|
||||
push["Push"]
|
||||
pr["Pull Request"]
|
||||
release["Release Published"]
|
||||
cron1["Cron: Daily 00:00 UTC"]
|
||||
manual["Manual Trigger"]
|
||||
cron3["Cron: Weekly Monday 02:00 UTC"]
|
||||
cron2["Cron: Daily 00:30 UTC"]
|
||||
|
||||
end
|
||||
|
||||
subgraph github_workflows ["📋 GitHub Workflows"]
|
||||
%% Main Workflows
|
||||
subgraph main ["📋 Main Workflows"]
|
||||
build["build.yml<br/>CI Build"]
|
||||
release_wf["release.yml<br/>Production Release"]
|
||||
build_dev["build_and_docs_to_dev.yml<br/>Dev Builds"]
|
||||
security_scan["security-scan.yml<br/>Security Scan"]
|
||||
stale["stale.yml<br/>Issue Management"]
|
||||
manual_docs["manual_docs.yml<br/>Docs Only"]
|
||||
end
|
||||
|
||||
%% Task Workflows
|
||||
subgraph tasks ["⚙️ Task Workflows"]
|
||||
task_build["task-build.yml<br/>Build Firmware"]
|
||||
task_lint["task-lint.yml<br/>Code Format"]
|
||||
task_security["task-security-scan.yml<br/>Security Scan"]
|
||||
task_docs["task-docs.yml<br/>Build & Deploy Docs"]
|
||||
|
||||
end
|
||||
end
|
||||
|
||||
subgraph ci_scripts ["🔧 CI Scripts"]
|
||||
%% CI Scripts Layer
|
||||
subgraph bash ["🔧 Orchestrator"]
|
||||
ci_main["ci.sh<br/>(main dispatcher)"]
|
||||
ci_build_script["ci_build.sh<br/>(build orchestrator)"]
|
||||
ci_site_script["ci_site.sh<br/>(docs orchestrator)"]
|
||||
ci_qa_script["ci_qa.sh<br/>(lint orchestrator)"]
|
||||
ci_security_script["ci_security.sh<br/>(security orchestrator)"]
|
||||
end
|
||||
|
||||
%% Sub-Scripts Layer
|
||||
subgraph sub_scripts ["⚙️ Workers"]
|
||||
ci_build_fw["ci_build_firmware.sh<br/>(PlatformIO build)"]
|
||||
ci_prep_art["ci_prepare_artifacts.sh<br/>(artifact packaging)"]
|
||||
gen_board["generate_board_docs<br/>(npm package)"]
|
||||
gen_wu["gen_wu<br/>(npm package)"]
|
||||
clang_fmt["clang-format<br/>(code formatter)"]
|
||||
trivy["Trivy<br/>(vulnerability scanner)"]
|
||||
end
|
||||
end
|
||||
|
||||
%% Trigger connections
|
||||
push -->|"all or subset<br/>(depends on branch)"| build
|
||||
pr -->|"all<br/>(always full)"| build
|
||||
|
||||
release --> release_wf
|
||||
manual --> manual_docs
|
||||
cron1 --> build_dev
|
||||
cron2 --> stale
|
||||
cron3 --> security_scan
|
||||
|
||||
%% Main workflow to task workflow connections
|
||||
build --> task_build
|
||||
build_dev --> task_build
|
||||
build_dev --> task_docs
|
||||
release_wf --> task_build
|
||||
release_wf --> task_docs
|
||||
manual_docs --> task_docs
|
||||
security_scan --> task_security
|
||||
|
||||
%% Task workflow internal dependencies
|
||||
task_build --> task_lint
|
||||
task_build --> task_security
|
||||
|
||||
%% Task workflows to CI scripts
|
||||
task_build -->|"ci.sh build ..."| ci_main
|
||||
task_docs -->|"ci.sh site ..."| ci_main
|
||||
task_lint -->|"ci.sh qa ..."| ci_main
|
||||
task_security -->|"ci.sh security ..."| ci_main
|
||||
|
||||
%% CI main dispatcher to orchestrators
|
||||
ci_main -->|"route: build"| ci_build_script
|
||||
ci_main -->|"route: site"| ci_site_script
|
||||
ci_main -->|"route: qa"| ci_qa_script
|
||||
ci_main -->|"route: security"| ci_security_script
|
||||
|
||||
%% Orchestrators to workers
|
||||
ci_build_script --> ci_build_fw
|
||||
ci_build_script --> ci_prep_art
|
||||
|
||||
ci_site_script --> gen_board
|
||||
ci_site_script --> gen_wu
|
||||
|
||||
ci_qa_script --> clang_fmt
|
||||
|
||||
ci_security_script --> trivy
|
||||
|
||||
%% Styling
|
||||
classDef triggerStyle fill:#e1f5ff,stroke:#0066cc,stroke-width:2px
|
||||
classDef mainStyle fill:#fff4e6,stroke:#ff9900,stroke-width:2px
|
||||
classDef taskStyle fill:#e6f7e6,stroke:#00aa00,stroke-width:2px
|
||||
classDef ciStyle fill:#ffe6f0,stroke:#cc0066,stroke-width:2px
|
||||
classDef subStyle fill:#f0e6ff,stroke:#9933ff,stroke-width:2px
|
||||
|
||||
class push,pr,release,manual,cron1,cron2,cron3 triggerStyle
|
||||
class build,release_wf,manual_docs,build_dev,stale,security_scan mainStyle
|
||||
class task_build,task_docs,task_lint,task_security taskStyle
|
||||
class ci_main,ci_build_script,ci_site_script,ci_qa_script,ci_security_script ciStyle
|
||||
class ci_build_fw,ci_prep_art,gen_board,gen_wu,clang_fmt,trivy subStyle
|
||||
|
||||
|
||||
style github_workflows stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:1.8px,fill:#fbfbfc
|
||||
style main stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fcfdff
|
||||
style tasks stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fcfdff
|
||||
|
||||
style ci_scripts stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:1.8px,fill:#fffaf5
|
||||
style bash stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fffaf5
|
||||
style sub_scripts stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fffaf5
|
||||
|
||||
style triggers fill:none,stroke:none
|
||||
|
||||
linkStyle 0,1,2,3,4,5,6 stroke:#0066cc,stroke-width:3px;
|
||||
linkStyle 7,8,9,10,11,12,13,14,15 stroke:orange, stroke-width:2px;
|
||||
linkStyle 16,17,18,19,20,21,22,23,24,25,26,27,28,29 stroke:red, stroke-width:2px;
|
||||
|
||||
```
|
||||
|
||||
### Workflow Relationships
|
||||
|
||||
**Main → Task Mapping**:
|
||||
- `build.yml` → calls `task-build.yml` (also contains inline documentation job)
|
||||
- `build_and_docs_to_dev.yml` → calls `task-build.yml` + `task-docs.yml`
|
||||
- `release.yml` → calls `task-build.yml` + `task-docs.yml`
|
||||
- `manual_docs.yml` → calls `task-docs.yml`
|
||||
- `security-scan.yml` → calls `task-security-scan.yml`
|
||||
- `stale.yml` → standalone (no dependencies)
|
||||
|
||||
**Task → CI Script Mapping**:
|
||||
- `task-build.yml` → `ci.sh build --version --mode --deploy-ready`
|
||||
- Routes to: `ci_build.sh` → `ci_build_firmware.sh`, `ci_prepare_artifacts.sh`
|
||||
- Output: `generated/artifacts/firmware_build/`
|
||||
- `task-docs.yml` → `ci.sh site --mode --version --url-prefix`
|
||||
- Routes to: `ci_site.sh` → `generate_board_docs` (npm), `gen_wu` (npm), VuePress
|
||||
- Output: `generated/site/`
|
||||
- `task-lint.yml` → `ci.sh qa --check --source --extensions --clang-format-version`
|
||||
- Routes to: `ci_qa.sh` → `clang-format`
|
||||
- `task-security-scan.yml` → `ci.sh security --scan-type --severity --generate-sbom`
|
||||
- Routes to: `ci_security.sh` → Trivy (vulnerability scanner)
|
||||
- Output: `generated/reports/` (SARIF, JSON, SBOM)
|
||||
|
||||
**Job Dependencies**:
|
||||
- `build_and_docs_to_dev.yml`: prepare → build (task) → deploy & documentation (task)
|
||||
- `release.yml`: prepare → build (task) → deploy → documentation (task)
|
||||
|
||||
**Script Execution Flow**:
|
||||
```
|
||||
GitHub Action (task-*.yml)
|
||||
↓
|
||||
./scripts/ci.sh <command> [OPTIONS] ← Main dispatcher
|
||||
↓
|
||||
./scripts/ci_<command>.sh ← Command orchestrator
|
||||
↓
|
||||
./scripts/ci_*.sh / *.py ← Worker scripts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -45,16 +221,19 @@ Parameterized building blocks called by main workflows:
|
||||
|
||||
### 1. `build.yml` - Continuous Integration Build
|
||||
|
||||
**Purpose**: Validates that code changes compile successfully across all supported hardware platforms.
|
||||
**Purpose**: Validates that code changes compile successfully with intelligent environment selection based on branch importance.
|
||||
|
||||
**Triggers**:
|
||||
- **Push**: Every commit pushed to any branch
|
||||
- **Pull Request**: Every PR creation or update
|
||||
|
||||
**What it does**:
|
||||
1. **Build job**: Calls `task-build.yml` with CI parameters
|
||||
- Builds firmware for **83 hardware environments** in parallel
|
||||
2. **Documentation job**: Inline job that validates docs build (doesn't deploy)
|
||||
1. **Determine build scope**: Selects environment list based on branch name
|
||||
- **Full build** (`all` environment): All PRs and Push on important branches (development, master, edge, stable, release/*, hotfix/*)
|
||||
- **Quick build** (`ci` subset environment): All PRs and Push on non-critical branches
|
||||
2. **Build job**: Calls `task-build.yml` with appropriate environment set
|
||||
- Builds firmware in parallel
|
||||
3. **Documentation job**: Inline job that validates docs build (doesn't deploy)
|
||||
- Downloads common config from theengs.io
|
||||
- Runs `npm install` and `npm run docs:build`
|
||||
- Uses Node.js 14.x
|
||||
@@ -64,15 +243,18 @@ Parameterized building blocks called by main workflows:
|
||||
- Python version: 3.13 (for build job)
|
||||
- Build strategy: Parallel matrix via task workflow
|
||||
- Artifact retention: 7 days
|
||||
- Development OTA: Disabled (`enable-dev-ota: false`)
|
||||
- Development OTA: Enabled (`enable-dev-ota: true`)
|
||||
- Environment selection logic:
|
||||
- **Full build** (`all`): All Pull Requests + branches: development, master, edge, stable, release/*, hotfix/*
|
||||
- **Quick build** (`ci` subset): All other feature branches
|
||||
|
||||
**Outputs**:
|
||||
- Firmware binaries for each environment (83 artifacts)
|
||||
- Firmware binaries for selected environments
|
||||
- No documentation deployment (validation only)
|
||||
|
||||
**Use Case**: Ensures no breaking changes before merge. Fast feedback for developers.
|
||||
**Use Case**: Ensures no breaking changes before merge. Fast feedback for feature branches (~10 min), comprehensive validation for PRs and critical branches (~40 min).
|
||||
|
||||
**Execution Context**: Runs for ALL contributors on ALL branches.
|
||||
**Execution Context**: Runs for ALL contributors on ALL branches with smart scaling based on branch importance.
|
||||
|
||||
---
|
||||
|
||||
@@ -86,38 +268,37 @@ Parameterized building blocks called by main workflows:
|
||||
|
||||
**What it does**:
|
||||
1. **Prepare job**: Generates 6-character short SHA
|
||||
2. **Build job**: Calls `task-build.yml` with development parameters
|
||||
- Builds firmware for **83 hardware environments** in parallel
|
||||
2. **Handle-firmwares job**: Calls `task-build.yml` with development parameters
|
||||
- Builds firmware for **all environments** in parallel
|
||||
- Enables development OTA updates with SHA commit version
|
||||
3. **Deploy job**: Prepares and uploads assets
|
||||
- Creates library dependency zips for each board
|
||||
- Generates source code zip
|
||||
- Removes test environment binaries
|
||||
4. **Documentation job**: Calls `task-docs.yml` with development parameters
|
||||
- Deploys to `/dev` subdirectory
|
||||
- Adds "DEVELOPMENT" watermark
|
||||
- Runs PageSpeed Insights
|
||||
- Artifact retention: 1 day
|
||||
3. **Handle-documentation job**: Calls `task-docs.yml` with development parameters
|
||||
- Deploys to `/dev` subdirectory on GitHub Pages
|
||||
- Uses short SHA as version identifier
|
||||
- Runs PageSpeed Insights on dev site
|
||||
|
||||
**Technical Details**:
|
||||
- **Calls**: `task-build.yml` + `task-docs.yml`
|
||||
- Python version: 3.13 (build), 3.11 (docs)
|
||||
- Node.js version: 16.x (docs)
|
||||
- Repository restriction: Hardcoded to `1technophile` owner only
|
||||
- Artifact retention: 1 day
|
||||
- Build flag: `enable-dev-ota: true` (passed to task-build.yml)
|
||||
- Version: 6-character short SHA (e.g., `abc123`)
|
||||
- Documentation URL prefix: `/dev/`
|
||||
- GitHub Pages destination: `dev` subdirectory
|
||||
- PageSpeed URL: `https://docs.openmqttgateway.com/dev/`
|
||||
|
||||
**Workflow Parameters**:
|
||||
- Build: `enable-dev-ota: true`, `version-tag: <short-sha>`, `artifact-retention-days: 1`
|
||||
- Docs: `mode: "dev"`, `version: <short-sha>`, `url-prefix: "/dev/"`, `destination-dir: "dev"`, `run-pagespeed: true`
|
||||
|
||||
**Outputs**:
|
||||
- Firmware binaries with `-firmware.bin` suffix
|
||||
- Firmware binaries with `-firmware.bin` suffix (1 day retention)
|
||||
- Bootloader and partition binaries
|
||||
- Library dependency zips per board
|
||||
- Source code zip
|
||||
- Documentation deployed to `docs.openmqttgateway.com/dev/`
|
||||
|
||||
**Version Labeling**:
|
||||
- Git SHA (6 chars) injected into firmware
|
||||
- Docs tagged: "DEVELOPMENT SHA:XXXXXX TEST ONLY"
|
||||
- Git SHA (6 chars) injected into firmware via `version-tag`
|
||||
- Docs display short SHA as version
|
||||
|
||||
**Use Case**: Daily bleeding-edge builds for early adopters and testing. Preview documentation changes.
|
||||
**Use Case**: Daily bleeding-edge builds for early adopters and testing. Preview documentation changes before production release.
|
||||
|
||||
**Execution Context**: Only runs on `1technophile` repository owner. Forks will skip this workflow automatically.
|
||||
|
||||
@@ -131,33 +312,34 @@ Parameterized building blocks called by main workflows:
|
||||
- **Release**: When a GitHub release is published (tagged)
|
||||
|
||||
**What it does**:
|
||||
1. **Prepare job**: Extracts version tag and release info
|
||||
1. **Prepare job**: Extracts version tag and release info from GitHub event
|
||||
2. **Build job**: Calls `task-build.yml` with production parameters
|
||||
- Builds firmware for **83 hardware environments** in parallel
|
||||
- Injects release tag version into firmware
|
||||
3. **Deploy job**: Prepares and uploads release assets
|
||||
- Downloads all build artifacts
|
||||
- Reorganizes for `prepare_deploy.sh` script
|
||||
- Creates library zips and source zip
|
||||
- Uploads to GitHub Release
|
||||
- Builds firmware for **all environments** in parallel
|
||||
- Injects release tag version into firmware
|
||||
- Artifact retention: 90 days
|
||||
3. **Deploy job**: Downloads and uploads release assets
|
||||
- Downloads all firmware artifacts from build job
|
||||
- Uploads binaries to GitHub Release
|
||||
4. **Documentation job**: Calls `task-docs.yml` for production docs
|
||||
- Deploys to root (`/`) of GitHub Pages
|
||||
- Uses release tag as version
|
||||
|
||||
**Technical Details**:
|
||||
- **Calls**: `task-build.yml` + `task-docs.yml`
|
||||
- Python version: 3.13 (build), 3.11 (docs)
|
||||
- Node.js version: 18.x
|
||||
- Build flag: Standard (no DEVELOPMENTOTA)
|
||||
- Artifact retention: 90 days
|
||||
- Uses `prepare_deploy.sh` script for asset preparation
|
||||
- Deploy uses `bgpat/release-asset-action` to attach assets to GitHub Release
|
||||
|
||||
**Workflow Parameters**:
|
||||
- Build: `enable-dev-ota: false`, `version-tag: <git-tag>`, `artifact-retention-days: 90`
|
||||
- Docs: `mode: "prod"`, `version: <git-tag>`, `url-prefix: "/"`, `destination-dir: "."`
|
||||
|
||||
**Outputs**:
|
||||
- Production firmware binaries attached to GitHub Release
|
||||
- Library zips per board
|
||||
- Source code zip
|
||||
- Production firmware binaries attached to GitHub Release (all prepared by `task-build.yml` with `--deploy-ready`)
|
||||
- Production documentation at `docs.openmqttgateway.com/`
|
||||
|
||||
**Version Labeling**:
|
||||
- Git tag (e.g., `v1.2.3`) injected into firmware and `latest_version.json`
|
||||
- Git tag (e.g., `v1.2.3`) injected into firmware
|
||||
|
||||
**Workflow Chain**:
|
||||
```
|
||||
@@ -184,10 +366,11 @@ prepare → build (task-build.yml) → deploy → documentation (task-docs.yml)
|
||||
|
||||
**Technical Details**:
|
||||
- **Calls**: `task-docs.yml`
|
||||
- Python version: 3.11
|
||||
- Node.js version: 14.x
|
||||
- Version source: Latest GitHub release tag
|
||||
- WebUploader manifest: Enabled
|
||||
- Mode: `prod`
|
||||
- Version: Uses latest release tag by default (or provided input)
|
||||
- URL prefix: `/`
|
||||
- Destination: Root of GitHub Pages
|
||||
- PageSpeed: Optional (disabled by default)
|
||||
|
||||
**Outputs**:
|
||||
- Production documentation at `docs.openmqttgateway.com/`
|
||||
@@ -199,34 +382,55 @@ prepare → build (task-build.yml) → deploy → documentation (task-docs.yml)
|
||||
|
||||
---
|
||||
|
||||
### 5. `lint.yml` - Code Format Validation
|
||||
### 5. `security-scan.yml` - Security Vulnerability Scanning
|
||||
|
||||
**Purpose**: Ensures code follows consistent formatting standards.
|
||||
**Purpose**: Scans the project for security vulnerabilities and generates Software Bill of Materials (SBOM) for supply chain security.
|
||||
|
||||
**Triggers**:
|
||||
- **Push**: Every commit pushed to any branch
|
||||
- **Pull Request**: Every PR creation or update
|
||||
- **Schedule**: Weekly on Monday at 02:00 UTC (`0 2 * * 1`)
|
||||
- **Manual**: Via workflow_dispatch button with input parameters
|
||||
|
||||
**Manual Trigger Inputs**:
|
||||
- `severity`: Severity levels to scan (choices: UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL; default: HIGH,CRITICAL)
|
||||
- Allows filtering results to specific severity levels
|
||||
|
||||
**What it does**:
|
||||
1. Calls `task-lint.yml` with specific parameters
|
||||
2. Checks formatting in `./main` directory only
|
||||
1. **Prepare job**: Sets up environment for scanning
|
||||
2. **Security scan job**: Calls `task-security-scan.yml` with parameters
|
||||
- Runs Trivy vulnerability scanner on filesystem
|
||||
- Filters results by severity level
|
||||
- Generates SARIF format for GitHub Security tab integration
|
||||
- Creates SBOM in CycloneDX and SPDX formats
|
||||
- Uploads findings to GitHub Security tab (code scanning dashboard)
|
||||
3. **Artifact upload job**: Stores generated reports and SBOM
|
||||
- SARIF results for GitHub integration
|
||||
- SBOM files for supply chain tracking
|
||||
- Retention: 90 days
|
||||
|
||||
**Technical Details**:
|
||||
- **Calls**: `task-lint.yml`
|
||||
- clang-format version: 9
|
||||
- File extensions: `.h`, `.ino` (not `.cpp`)
|
||||
- Source directory: `main` (single directory)
|
||||
- **Calls**: `task-security-scan.yml`
|
||||
- Vulnerability scanner: Trivy (vulnerability database updated automatically)
|
||||
- Report formats: SARIF (GitHub), JSON (detailed), Markdown (summary)
|
||||
- SBOM formats: CycloneDX and SPDX (standard formats)
|
||||
- Failure behavior: Does NOT fail the workflow on vulnerabilities (exit-code: 0)
|
||||
- GitHub Security tab: Auto-uploads SARIF for code scanning dashboard visibility
|
||||
|
||||
**Configuration**:
|
||||
```yaml
|
||||
source: 'main'
|
||||
extensions: 'h,ino'
|
||||
clang-format-version: '9'
|
||||
```
|
||||
**Workflow Parameters**:
|
||||
- `scan-type: "fs"` (filesystem scan)
|
||||
- `severity: "HIGH,CRITICAL"` (default, or manual input)
|
||||
- `generate-sbom: true` (always enabled)
|
||||
- `upload-to-security-tab: true` (GitHub Security integration)
|
||||
|
||||
**Use Case**: Maintains code quality and consistency. Prevents formatting debates in PRs.
|
||||
**Outputs**:
|
||||
- SARIF report: `generated/reports/trivy-results.sarif` (GitHub Security tab)
|
||||
- JSON report: `generated/reports/trivy-results.json` (detailed findings)
|
||||
- Markdown summary: `generated/reports/security-summary.md`
|
||||
- SBOM: `generated/reports/sbom/sbom.cyclonedx.json` + `sbom.spdx.json`
|
||||
- Artifacts retained for 90 days
|
||||
|
||||
**Execution Context**: Runs for ALL contributors on ALL branches.
|
||||
**Use Case**: Regular security audits, compliance tracking, vulnerability management, supply chain security.
|
||||
|
||||
**Execution Context**: Weekly automated scans + manual on-demand scanning for developers.
|
||||
|
||||
---
|
||||
|
||||
@@ -267,17 +471,19 @@ clang-format-version: '9'
|
||||
**Trigger**: `workflow_call` only (called by other workflows)
|
||||
|
||||
**Parameters**:
|
||||
- `python-version`: Python version (default: '3.13')
|
||||
- `enable-dev-ota`: Enable development OTA (default: false)
|
||||
- `version-tag`: Version to inject (default: 'unspecified')
|
||||
- `artifact-retention-days`: Artifact retention (default: 7)
|
||||
- `artifact-name-prefix`: Artifact name prefix (default: '')
|
||||
- `prepare-for-deploy`: Prepare for deployment (default: false)
|
||||
- `python-version`: Python version to use (default: '3.13')
|
||||
- `pio-version`: PlatformIO version to use (default: 'v6.1.18')
|
||||
- `environment-set`: Which set of environments to build: 'all' or 'ci' (default: 'all')
|
||||
- `enable-dev-ota`: Enable development OTA builds (default: false)
|
||||
- `version-tag`: Optional version tag to pass to ci.sh build - omitted if empty (default: '')
|
||||
- `artifact-retention-days`: Number of days to retain build artifacts (default: 7)
|
||||
|
||||
**What it does**:
|
||||
1. **Load environments**: Reads environment list from `environments.json`
|
||||
2. **Matrix build**: Builds all 83 environments in parallel
|
||||
3. **Build execution**: Calls unified `ci.sh build <environment> [OPTIONS]`:
|
||||
1. **Lint code**: Runs `task-lint.yml` to check code formatting (main directory, .h and .ino files)
|
||||
2. **Load environments**: Reads environment list from `environments.json` based on `environment-set` input (`all` or `ci`)
|
||||
3. **Install PlatformIO**: Uses `uv` to install the `pio-version` input (custom `pioarduino/platformio-core` fork)
|
||||
4. **Matrix build**: Builds selected environments in parallel, blocking on lint job completion
|
||||
5. **Build execution**: Calls unified `ci.sh build <environment> [OPTIONS]`:
|
||||
- `<environment>`: Target hardware (e.g., `esp32dev-ble`)
|
||||
- `--version <tag>`: Version to inject (SHA for dev, tag for prod)
|
||||
- `--mode <dev|prod>`: Build mode (enables/disables OTA)
|
||||
@@ -289,16 +495,17 @@ clang-format-version: '9'
|
||||
./scripts/ci.sh build esp32dev-ble --version v1.8.0 --mode prod --deploy-ready
|
||||
↓
|
||||
├─→ ci_build.sh (orchestrator)
|
||||
│ ├─→ ci_set_version.sh v1.8.0 [--dev]
|
||||
│ ├─→ ci_build_firmware.sh esp32dev-ble [--dev-ota]
|
||||
│ └─→ ci_prepare_artifacts.sh esp32dev-ble [--deploy] → outputs to generated/artifacts/
|
||||
```
|
||||
|
||||
**Technical Details**:
|
||||
- Runs on: Ubuntu latest
|
||||
- PlatformIO version: 6.1.18 (custom fork: `pioarduino/platformio-core`)
|
||||
- PlatformIO version: Configurable via `pio-version` input (default: v6.1.18, custom fork: `pioarduino/platformio-core`)
|
||||
- Python package manager: `uv` (astral-sh/setup-uv@v6)
|
||||
- Strategy: Matrix with fail-fast: false
|
||||
- Environment sets: `all` (complete set) or `ci` (subset for quick validation)
|
||||
- Strategy: Matrix with fail-fast: false (builds complete even if one environment fails)
|
||||
- Dependencies: Lint job must pass before build matrix starts
|
||||
- Main orchestrator: `ci.sh` → `ci_build.sh` → sub-scripts
|
||||
|
||||
**Callers**:
|
||||
@@ -315,36 +522,24 @@ clang-format-version: '9'
|
||||
**Trigger**: `workflow_call` only (called by other workflows)
|
||||
|
||||
**Parameters**:
|
||||
- `python-version`: Python version (default: '3.11')
|
||||
- `node-version`: Node.js version (default: '14.x')
|
||||
- `version-source`: Version source ('release', 'git-tag', 'custom')
|
||||
- `custom-version`: Custom version string (optional)
|
||||
- `base-path`: Base URL path (default: '/')
|
||||
- `destination-dir`: Deploy directory (default: '.')
|
||||
- `generate-webuploader`: Generate WebUploader manifest (default: true)
|
||||
- `webuploader-args`: WebUploader generation arguments (optional)
|
||||
- `run-pagespeed`: Run PageSpeed Insights (default: false)
|
||||
- `pagespeed-url`: URL for PageSpeed test (optional)
|
||||
- `mode`: Documentation mode (`prod` or `dev`, default: `prod`)
|
||||
- `version`: Version string for docs (default: `auto` → tag or short SHA depending on caller)
|
||||
- `url-prefix`: Base URL path (default: `/` for prod, `/dev/` for dev callers)
|
||||
- `destination-dir`: Deploy directory on GitHub Pages (default: `.`)
|
||||
- `run-pagespeed`: Run PageSpeed Insights after deploy (default: false)
|
||||
- `pagespeed-url`: URL to test with PageSpeed (default: `https://docs.openmqttgateway.com/`)
|
||||
|
||||
**What it does**:
|
||||
1. **Build documentation**: Calls unified `ci.sh site [OPTIONS]`:
|
||||
- `--mode <dev|prod>`: Documentation mode
|
||||
- `--custom-version <ver>`: Custom version string
|
||||
- `--version-source <release|custom>`: Version source
|
||||
- `--url-prefix <path>`: Base URL path (e.g., `/dev/`)
|
||||
- `--webuploader-args <args>`: WebUploader options
|
||||
- `--no-webuploader`: Skip manifest generation
|
||||
1. **Build documentation**: Calls unified `ci.sh site --mode <mode> --version <ver> --url-prefix <path>`
|
||||
2. **Deploy**: Publishes to GitHub Pages using `peaceiris/actions-gh-pages@v3`
|
||||
3. **PageSpeed test**: Optionally runs performance audit
|
||||
3. **PageSpeed test**: Optionally runs performance audit on the provided URL
|
||||
|
||||
**Command Flow**:
|
||||
```bash
|
||||
./scripts/ci.sh site --mode prod --version-source release --url-prefix /
|
||||
↓
|
||||
└─→ ci_site.sh (orchestrator)
|
||||
├─→ generate_board_docs.py (auto-generate board docs)
|
||||
├─→ npm run docs:build (VuePress compilation)
|
||||
└─→ gen_wu.py (WebUpdater manifest)
|
||||
./scripts/ci.sh site --mode prod --version v1.2.3 --url-prefix /
|
||||
↓
|
||||
└─→ ci_site.sh (orchestrator)
|
||||
└─→ npm run docs:build (VuePress compilation)
|
||||
```
|
||||
|
||||
**Callers**:
|
||||
@@ -356,15 +551,14 @@ clang-format-version: '9'
|
||||
|
||||
### 9. `task-lint.yml` - Reusable Lint Workflow
|
||||
|
||||
**Purpose**: Parameterized code formatting validation.
|
||||
**Purpose**: Parameterized code formatting validation for consistent code style.
|
||||
|
||||
**Trigger**: `workflow_call` only (called by other workflows)
|
||||
|
||||
**Parameters**:
|
||||
- `source`: Source directories to lint (default: './lib ./main')
|
||||
- `extensions`: File extensions to check (default: 'h,ino,cpp')
|
||||
- `clang-format-version`: clang-format version (default: '9')
|
||||
- `exclude-pattern`: Pattern to exclude (optional)
|
||||
- `source`: Source directory to lint (default: 'main')
|
||||
- `extensions`: File extensions to check, comma-separated (default: 'h,ino')
|
||||
- `clang-format-version`: clang-format version to use (default: '9')
|
||||
|
||||
**What it does**:
|
||||
1. Checks out code
|
||||
@@ -386,170 +580,77 @@ clang-format-version: '9'
|
||||
```
|
||||
|
||||
**Technical Details**:
|
||||
- Runs on: Ubuntu latest
|
||||
- Script: `ci_qa.sh` (custom formatting check script)
|
||||
- Install: `clang-format-$version` via apt-get
|
||||
- Default source: `main` (single directory)
|
||||
- Default extensions: `h,ino` (not cpp)
|
||||
- Formatter: `clang-format-$version` installed via apt-get
|
||||
- Default scope: `main` directory only (not lib)
|
||||
- Default file types: `.h` and `.ino` (not `.cpp`)
|
||||
- Strategy: Single sequential job (not parallelized)
|
||||
|
||||
**Callers**:
|
||||
- `lint.yml` (CI lint check)
|
||||
- `build.yml` (inline lint check before build)
|
||||
- Can be called by other workflows as needed
|
||||
|
||||
**Default Behavior**: If called without parameters, lints `main` directory for `.h` and `.ino` files.
|
||||
**Default Behavior**: If called without parameters, lints `main` directory for `.h` and `.ino` files only.
|
||||
|
||||
---
|
||||
|
||||
## Workflow Dependencies and Call Chain
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
%% Triggers
|
||||
subgraph triggers ["🎯 Triggers"]
|
||||
push["Push / Pull Request"]
|
||||
release["Release Published"]
|
||||
manual["Manual Trigger"]
|
||||
cron1["Cron: Daily 00:00 UTC"]
|
||||
cron2["Cron: Daily 00:30 UTC"]
|
||||
end
|
||||
### 10. `task-security-scan.yml` - Reusable Security Scan Workflow
|
||||
|
||||
subgraph github_workflows ["📋 GitHub Workflows"]
|
||||
%% Main Workflows
|
||||
subgraph main ["📋 Main Workflows"]
|
||||
lint["lint.yml<br/>Format Check"]
|
||||
|
||||
build["build.yml<br/>CI Build"]
|
||||
|
||||
release_wf["release.yml<br/>Production Release"]
|
||||
manual_docs["manual_docs.yml<br/>Docs Only"]
|
||||
build_dev["build_and_docs_to_dev.yml<br/>Dev Builds"]
|
||||
stale["stale.yml<br/>Issue Management"]
|
||||
end
|
||||
**Purpose**: Parameterized security vulnerability scanning and SBOM generation logic.
|
||||
|
||||
%% Task Workflows
|
||||
subgraph tasks ["⚙️ Task Workflows"]
|
||||
task_build["task-build.yml<br/>Build Firmware"]
|
||||
task_docs["task-docs.yml<br/>Build & Deploy Docs"]
|
||||
task_lint["task-lint.yml<br/>Code Format"]
|
||||
end
|
||||
end
|
||||
**Trigger**: `workflow_call` only (called by other workflows)
|
||||
|
||||
subgraph ci_scripts ["🔧 CI Scripts"]
|
||||
%% CI Scripts Layer
|
||||
subgraph bash ["🔧 Orchestrator"]
|
||||
ci_main["ci.sh<br/>(main dispatcher)"]
|
||||
ci_build_script["ci_build.sh<br/>(build orchestrator)"]
|
||||
ci_site_script["ci_site.sh<br/>(docs orchestrator)"]
|
||||
ci_qa_script["ci_qa.sh<br/>(lint orchestrator)"]
|
||||
end
|
||||
**Parameters**:
|
||||
- `scan-type`: Type of scan: 'fs' (filesystem), 'config' (configuration), or 'image' (container) (default: 'fs')
|
||||
- `severity`: Severity levels to report (comma-separated: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL) (default: 'HIGH,CRITICAL')
|
||||
- `scan-path`: Path to scan (default: '.')
|
||||
- `exit-code`: Exit code when vulnerabilities found (0=continue, 1=fail) (default: '0')
|
||||
- `upload-to-security-tab`: Upload SARIF to GitHub Security tab (default: true)
|
||||
- `generate-sbom`: Generate SBOM artifacts (default: true)
|
||||
|
||||
%% Sub-Scripts Layer
|
||||
subgraph sub_scripts ["⚙️ Workers"]
|
||||
ci_set_ver["ci_set_version.sh<br/>(version injection)"]
|
||||
ci_build_fw["ci_build_firmware.sh<br/>(PlatformIO build)"]
|
||||
ci_prep_art["ci_prepare_artifacts.sh<br/>(artifact packaging)"]
|
||||
gen_board["generate_board_docs.py<br/>(board docs)"]
|
||||
gen_wu["gen_wu.py<br/>(WebUpdater manifest)"]
|
||||
clang_fmt["clang-format<br/>(code formatter)"]
|
||||
end
|
||||
end
|
||||
**What it does**:
|
||||
1. **Install Trivy**: Retrieves and installs Trivy vulnerability scanner
|
||||
2. **Run security scan**: Calls unified `ci.sh security` with parameters:
|
||||
- `--scan-type <fs|config|image>`: Target type
|
||||
- `--severity <levels>`: Filter by severity
|
||||
- `--scan-path <path>`: Directory to scan
|
||||
- `--generate-sbom`: Generate SBOM in CycloneDX and SPDX formats
|
||||
- `--exit-code <0|1>`: Fail behavior on critical vulnerabilities
|
||||
- `--upload-to-security-tab`: Upload SARIF to GitHub
|
||||
3. **Upload artifacts**: Stores reports and SBOM for later download
|
||||
4. **GitHub Security integration**: SARIF automatically appears in Security tab
|
||||
|
||||
%% Trigger connections
|
||||
push --> lint
|
||||
push --> build
|
||||
|
||||
release --> release_wf
|
||||
manual --> manual_docs
|
||||
cron1 --> build_dev
|
||||
cron2 --> stale
|
||||
|
||||
%% Main workflow to task workflow connections
|
||||
build -->|calls| task_build
|
||||
lint -->|calls| task_lint
|
||||
build_dev -->|calls| task_build
|
||||
build_dev -->|calls| task_docs
|
||||
release_wf -->|calls| task_build
|
||||
release_wf -->|calls| task_docs
|
||||
manual_docs -->|calls| task_docs
|
||||
|
||||
%% Task workflows to CI scripts
|
||||
task_build -->|"ci.sh build<br/>--version --mode<br/>--deploy-ready"| ci_main
|
||||
task_docs -->|"ci.sh site<br/>--mode --url-prefix<br/>--version-source"| ci_main
|
||||
task_lint -->|"ci.sh qa<br/>--check --source<br/>--extensions"| ci_main
|
||||
|
||||
%% CI main dispatcher to orchestrators
|
||||
ci_main -->|"route: build"| ci_build_script
|
||||
ci_main -->|"route: site"| ci_site_script
|
||||
ci_main -->|"route: qa"| ci_qa_script
|
||||
|
||||
%% Orchestrators to workers
|
||||
ci_build_script --> ci_set_ver
|
||||
ci_build_script --> ci_build_fw
|
||||
ci_build_script --> ci_prep_art
|
||||
|
||||
ci_site_script --> gen_board
|
||||
ci_site_script --> gen_wu
|
||||
|
||||
ci_qa_script --> clang_fmt
|
||||
|
||||
%% Styling
|
||||
classDef triggerStyle fill:#e1f5ff,stroke:#0066cc,stroke-width:2px
|
||||
classDef mainStyle fill:#fff4e6,stroke:#ff9900,stroke-width:2px
|
||||
classDef taskStyle fill:#e6f7e6,stroke:#00aa00,stroke-width:2px
|
||||
classDef ciStyle fill:#ffe6f0,stroke:#cc0066,stroke-width:2px
|
||||
classDef subStyle fill:#f0e6ff,stroke:#9933ff,stroke-width:2px
|
||||
|
||||
class push,release,manual,cron1,cron2 triggerStyle
|
||||
class build,lint,release_wf,manual_docs,build_dev,stale mainStyle
|
||||
class task_build,task_docs,task_lint taskStyle
|
||||
class ci_main,ci_build_script,ci_site_script,ci_qa_script ciStyle
|
||||
class ci_set_ver,ci_build_fw,ci_prep_art,gen_board,gen_wu,clang_fmt subStyle
|
||||
|
||||
|
||||
style github_workflows stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:1.8px,fill:#fbfbfc
|
||||
style main stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fcfdff
|
||||
style tasks stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fcfdff
|
||||
|
||||
style ci_scripts stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:1.8px,fill:#fffaf5
|
||||
style bash stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fffaf5
|
||||
style sub_scripts stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fffaf5
|
||||
|
||||
style triggers fill:none,stroke:none
|
||||
|
||||
**Command Flow**:
|
||||
```bash
|
||||
./scripts/ci.sh security --scan-type fs --severity HIGH,CRITICAL --generate-sbom --upload-to-security-tab
|
||||
↓
|
||||
└─→ ci_security.sh (security orchestrator)
|
||||
└─→ Trivy (vulnerability scanner)
|
||||
├─→ Generate SARIF, JSON, summary
|
||||
├─→ Generate SBOM (CycloneDX, SPDX)
|
||||
└─→ Upload to GitHub Security tab
|
||||
```
|
||||
|
||||
### Workflow Relationships
|
||||
**Technical Details**:
|
||||
- Runs on: Ubuntu latest
|
||||
- Scanner: Trivy (latest version auto-installed)
|
||||
- Report formats: SARIF (GitHub integration), JSON (detailed), Markdown (summary)
|
||||
- SBOM formats: CycloneDX and SPDX (industry standards)
|
||||
- GitHub Security tab: Auto-uploads SARIF for code scanning dashboard
|
||||
- Strategy: Single sequential job (not parallelized)
|
||||
- Artifact retention: As configured by caller
|
||||
|
||||
**Main → Task Mapping**:
|
||||
- `build.yml` → calls `task-build.yml` (also contains inline documentation job)
|
||||
- `lint.yml` → calls `task-lint.yml`
|
||||
- `build_and_docs_to_dev.yml` → calls `task-build.yml` + `task-docs.yml` (with prepare/deploy jobs)
|
||||
- `release.yml` → calls `task-build.yml` + `task-docs.yml` (with prepare/deploy jobs)
|
||||
- `manual_docs.yml` → calls `task-docs.yml`
|
||||
- `stale.yml` → standalone (no dependencies)
|
||||
**Output Files**:
|
||||
- `generated/reports/trivy-results.sarif` - SARIF format (GitHub Security tab upload)
|
||||
- `generated/reports/trivy-results.json` - JSON format (detailed results)
|
||||
- `generated/reports/security-summary.md` - Human-readable summary
|
||||
- `generated/reports/sbom/sbom.cyclonedx.json` - CycloneDX SBOM
|
||||
- `generated/reports/sbom/sbom.spdx.json` - SPDX SBOM
|
||||
|
||||
**Task → CI Script Mapping**:
|
||||
- `task-build.yml` → `ci.sh build <env> --version --mode --deploy-ready`
|
||||
- Routes to: `ci_build.sh` → `ci_set_version.sh`, `ci_build_firmware.sh`, `ci_prepare_artifacts.sh`
|
||||
- Output: `generated/artifacts/` (default, can be overridden with `--output`)
|
||||
- `task-docs.yml` → `ci.sh site --mode --version-source --url-prefix --webuploader-args`
|
||||
- Routes to: `ci_site.sh` → `generate_board_docs.py`, `gen_wu.py`, VuePress
|
||||
- Output: `generated/site/`
|
||||
- `task-lint.yml` → `ci.sh qa --check --source --extensions --clang-format-version`
|
||||
- Routes to: `ci_qa.sh` → `clang-format`
|
||||
|
||||
**Job Dependencies**:
|
||||
- `build_and_docs_to_dev.yml`: prepare → build (task) → deploy & documentation (task)
|
||||
- `release.yml`: prepare → build (task) → deploy → documentation (task)
|
||||
|
||||
**Script Execution Flow**:
|
||||
```
|
||||
GitHub Action (task-*.yml)
|
||||
↓
|
||||
./scripts/ci.sh <command> [OPTIONS] ← Main dispatcher
|
||||
↓
|
||||
./scripts/ci_<command>.sh ← Command orchestrator
|
||||
↓
|
||||
./scripts/ci_*.sh / *.py ← Worker scripts
|
||||
```
|
||||
**Callers**:
|
||||
- `security-scan.yml` (weekly + manual scanning)
|
||||
- Can be called by other workflows for custom security workflows
|
||||
|
||||
---
|
||||
|
||||
@@ -562,25 +663,17 @@ All build environments are defined in `.github/workflows/environments.json`:
|
||||
```json
|
||||
{
|
||||
"environments": {
|
||||
"all": [ ...83 environments ],
|
||||
"metadata": {
|
||||
"totalCount": 83,
|
||||
"lastUpdated": "2024-12-25",
|
||||
"categories": {
|
||||
"esp32": 50,
|
||||
"esp8266": 20,
|
||||
"specialized": 13
|
||||
}
|
||||
"all": [ ...all environments ],
|
||||
"ci": [...a subset of environments]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- ✅ Single source of truth
|
||||
- ✅ Eliminates duplication across workflows
|
||||
- ✅ Easier to maintain and update
|
||||
- ✅ Consistent builds across CI/dev/release
|
||||
**Environment Sets**:
|
||||
- `all`: Complete production set for releases and comprehensive validation
|
||||
- `ci`: Representative subset for fast CI feedback
|
||||
|
||||
|
||||
### Environment Categories
|
||||
|
||||
@@ -653,24 +746,33 @@ All build environments are defined in `.github/workflows/environments.json`:
|
||||
- PlatformIO 6.1.18 from custom fork: `pioarduino/platformio-core`
|
||||
- Python package manager: `uv` for fast dependency installation
|
||||
- Orchestrator: `ci_build.sh`
|
||||
- Worker: `ci_set_version.sh` (version injection)
|
||||
- Worker: `ci_build_firmware.sh` (PlatformIO compilation)
|
||||
- Worker: `ci_prepare_artifacts.sh` (artifact packaging)
|
||||
|
||||
**Documentation System** (`ci.sh site`):
|
||||
- Documentation framework: VuePress
|
||||
- Orchestrator: `ci_site.sh`
|
||||
- Worker: `generate_board_docs.py` (auto-generate board pages)
|
||||
- Worker: `gen_wu.py` (WebUpdater manifest)
|
||||
- Worker: `generate_board_docs` (npm package - auto-generate board pages)
|
||||
- Worker: `gen_wu` (npm package - WebUpdater manifest)
|
||||
- External: Common config from theengs.io
|
||||
|
||||
**Code Quality** (`ci.sh qa`):
|
||||
- Orchestrator: `ci_qa.sh`
|
||||
- Worker: `clang-format` version 9
|
||||
- Worker: `shellcheck` for shell scripts
|
||||
- Default scope: `main` directory, `.h` and `.ino` files
|
||||
|
||||
**Security Scanning System** (`ci.sh security`):
|
||||
- Vulnerability scanner: Trivy (container vulnerability database)
|
||||
- Orchestrator: `ci_security.sh`
|
||||
- Worker: Trivy (filesystem, configuration, and container image scanning)
|
||||
- Report formats: SARIF (GitHub Security tab), JSON (detailed), Markdown (summary)
|
||||
- SBOM generation: CycloneDX and SPDX formats
|
||||
- GitHub Security integration: Auto-uploads SARIF for code scanning dashboard
|
||||
- Output: `generated/reports/` (security findings and artifacts)
|
||||
|
||||
**Configuration**:
|
||||
- Environment list: `.github/workflows/environments.json` (83 environments)
|
||||
- Environment list: `.github/workflows/environments.json`
|
||||
- Task workflows: `task-*.yml` (reusable GitHub Actions components)
|
||||
- Repository owner restriction: Hardcoded to `1technophile` for dev deployments
|
||||
- All scripts located in: `./scripts/`
|
||||
@@ -681,17 +783,20 @@ All build environments are defined in `.github/workflows/environments.json`:
|
||||
./scripts/ci.sh build esp32dev-ble --mode dev --version test
|
||||
|
||||
# Build documentation locally
|
||||
./scripts/ci.sh site --mode dev --preview
|
||||
./scripts/ci.sh site --mode dev --version test
|
||||
|
||||
# Check code format
|
||||
./scripts/ci.sh qa --check
|
||||
./scripts/ci.sh qa --check --source main --extensions h,ino --clang-format-version 9
|
||||
|
||||
# Security scanning
|
||||
./scripts/ci.sh security --scan-type fs --severity HIGH,CRITICAL --generate-sbom
|
||||
|
||||
# Run complete pipeline
|
||||
./scripts/ci.sh all esp32dev-ble --version v1.8.0
|
||||
./scripts/ci.sh all --mode dev
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Document Version**: 2.1
|
||||
**Last Updated**: December 29, 2025
|
||||
**Document Version**: 2.3
|
||||
**Last Updated**: 01/14/2026
|
||||
**Maintainer**: OpenMQTTGateway Development Team
|
||||
+34
-17
@@ -1,30 +1,47 @@
|
||||
name: Build
|
||||
# See details in .github/workflows/README.md (workflow docs)
|
||||
run-name: "CI build for ${{ github.ref_name }}"
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
determine-build-scope:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
environment-set: ${{ steps.decide.outputs.environment-set }}
|
||||
steps:
|
||||
- name: Decide environment set
|
||||
id: decide
|
||||
run: |
|
||||
# Full build for: development, master, edge, stable, release/*, hotfix/*, and all PRs
|
||||
if [[ "${{ github.event_name }}" == "pull_request" ]] || \
|
||||
[[ "${{ github.ref }}" == "refs/heads/development" ]] || \
|
||||
[[ "${{ github.ref }}" == "refs/heads/master" ]] || \
|
||||
[[ "${{ github.ref }}" == "refs/heads/edge" ]] || \
|
||||
[[ "${{ github.ref }}" == "refs/heads/stable" ]] || \
|
||||
[[ "${{ github.ref }}" == refs/heads/release/* ]] || \
|
||||
[[ "${{ github.ref }}" == refs/heads/hotfix/* ]]; then
|
||||
echo "environment-set=all" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "environment-set=ci" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
|
||||
build:
|
||||
needs: determine-build-scope
|
||||
name: Build firmware
|
||||
uses: ./.github/workflows/task-build.yml
|
||||
with:
|
||||
python-version: '3.13'
|
||||
enable-dev-ota: false
|
||||
enable-dev-ota: true
|
||||
artifact-retention-days: 7
|
||||
prepare-for-deploy: false
|
||||
environment-set: ${{ needs.determine-build-scope.outputs.environment-set }}
|
||||
|
||||
documentation:
|
||||
needs: build
|
||||
name: Build documentation
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "14.x"
|
||||
- name: Download Common Config
|
||||
run: |
|
||||
curl -o docs/.vuepress/public/commonConfig.js https://www.theengs.io/commonConfig.js
|
||||
- name: Install build dependencies
|
||||
run: npm install
|
||||
- name: Build documentation
|
||||
run: npm run docs:build
|
||||
uses: ./.github/workflows/task-docs.yml
|
||||
with:
|
||||
mode: "dev"
|
||||
run-pagespeed: false
|
||||
just-check: true
|
||||
|
||||
@@ -3,7 +3,7 @@ name: Build binaries, docs and publish to dev folder
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: '0 0 * * *'
|
||||
- cron: "0 0 * * *"
|
||||
|
||||
jobs:
|
||||
prepare:
|
||||
@@ -18,86 +18,26 @@ jobs:
|
||||
with:
|
||||
length: 6
|
||||
|
||||
build:
|
||||
handle-firmwares:
|
||||
needs: prepare
|
||||
name: Build development firmware
|
||||
name: Build and deploy development firmwares artefacts
|
||||
uses: ./.github/workflows/task-build.yml
|
||||
with:
|
||||
python-version: '3.13'
|
||||
enable-dev-ota: true
|
||||
version-tag: ${{ needs.prepare.outputs.short-sha }}
|
||||
artifact-retention-days: 1
|
||||
artifact-name-prefix: 'firmware-'
|
||||
prepare-for-deploy: true
|
||||
|
||||
deploy:
|
||||
needs: [prepare, build]
|
||||
runs-on: ubuntu-latest
|
||||
if: github.repository_owner == '1technophile'
|
||||
name: Deploy binaries and docs
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Download all firmware artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: firmware-*
|
||||
path: toDeploy
|
||||
merge-multiple: true
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
enable-cache: false
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
|
||||
|
||||
- name: Create library zips
|
||||
run: |
|
||||
# Install libraries for a representative environment to get libdeps
|
||||
platformio pkg install -e esp32dev-ble --no-save
|
||||
cd .pio/libdeps
|
||||
# Replace spaces with underscores in folder names
|
||||
find . -type d -name "* *" | while read FNAME; do mv "$FNAME" "${FNAME// /_}"; done
|
||||
# Zip libraries per board
|
||||
for i in */; do
|
||||
zip -r "${i%/}-libraries.zip" "$i"
|
||||
done
|
||||
mv *.zip ../../toDeploy/
|
||||
|
||||
- name: Prepare additional assets
|
||||
run: |
|
||||
cd toDeploy
|
||||
# Remove binaries for *-all*, *-test* env
|
||||
rm -f *-all*.bin *-test*.bin *-test*.zip || true
|
||||
cd ..
|
||||
# Zip source code
|
||||
zip -r toDeploy/OpenMQTTGateway_sources.zip main LICENSE.txt
|
||||
ls -lA toDeploy/
|
||||
|
||||
documentation:
|
||||
needs: [prepare, build]
|
||||
handle-documentation:
|
||||
needs: [prepare, handle-firmwares]
|
||||
name: Build and deploy development documentation
|
||||
uses: ./.github/workflows/task-docs.yml
|
||||
with:
|
||||
python-version: '3.11'
|
||||
node-version: '16.x'
|
||||
version-source: 'custom'
|
||||
custom-version: 'DEVELOPMENT SHA:${{ needs.prepare.outputs.short-sha }} TEST ONLY'
|
||||
base-path: '/dev/'
|
||||
destination-dir: 'dev'
|
||||
generate-webuploader: true
|
||||
webuploader-args: '--dev'
|
||||
mode: "dev"
|
||||
version: ${{ needs.prepare.outputs.short-sha }}
|
||||
url-prefix: "/dev/"
|
||||
destination-dir: "dev"
|
||||
run-pagespeed: true
|
||||
pagespeed-url: 'https://docs.openmqttgateway.com/dev/'
|
||||
secrets:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
APIKEY: ${{ secrets.APIKEY }}
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
pagespeed-apikey: ${{ secrets.APIKEY }}
|
||||
@@ -88,19 +88,22 @@
|
||||
"sonoff-basic",
|
||||
"sonoff-basic-rfr3"
|
||||
],
|
||||
"metadata": {
|
||||
"totalCount": 83,
|
||||
"categories": {
|
||||
"esp32": 42,
|
||||
"esp8266": 17,
|
||||
"specialized": 24
|
||||
},
|
||||
"notes": [
|
||||
"This unified list contains all environments from both build.yml and build_and_docs_to_dev.yml",
|
||||
"Previously build.yml had 85 environments and build_and_docs_to_dev.yml had 88",
|
||||
"After deduplication and proper ordering, the total is 83 unique environments",
|
||||
"Environments ending in -test or -all-test are typically excluded from production releases"
|
||||
]
|
||||
}
|
||||
"ci": [
|
||||
"esp32dev-all-test",
|
||||
"esp32dev-ble",
|
||||
"esp32dev-ir",
|
||||
"esp32dev-rf",
|
||||
"esp32dev-rtl_433",
|
||||
"esp32s3-dev-c1-ble",
|
||||
"esp32c3-dev-m1-ble",
|
||||
"heltec-ble",
|
||||
"lilygo-ble",
|
||||
"nodemcuv2-all-test",
|
||||
"nodemcuv2-ir",
|
||||
"nodemcuv2-rf",
|
||||
"theengs-bridge",
|
||||
"theengs-plug",
|
||||
"rfbridge"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -1,12 +0,0 @@
|
||||
name: Check Code Format
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
name: Lint code format
|
||||
uses: ./.github/workflows/task-lint.yml
|
||||
with:
|
||||
source: 'main'
|
||||
extensions: 'h,ino'
|
||||
clang-format-version: '9'
|
||||
@@ -9,13 +9,9 @@ jobs:
|
||||
name: Build and deploy production documentation
|
||||
uses: ./.github/workflows/task-docs.yml
|
||||
with:
|
||||
python-version: '3.11'
|
||||
node-version: '14.x'
|
||||
version-source: 'release'
|
||||
base-path: '/'
|
||||
destination-dir: '.'
|
||||
generate-webuploader: true
|
||||
webuploader-args: '${GITHUB_REPOSITORY}'
|
||||
run-pagespeed: false
|
||||
mode: "prod"
|
||||
version: "auto" # Will detect latest git tag
|
||||
url-prefix: "/"
|
||||
destination-dir: "."
|
||||
secrets:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
@@ -18,7 +18,7 @@ jobs:
|
||||
VERSION_TAG=${GITHUB_REF#refs/tags/}
|
||||
echo "version=${VERSION_TAG}" >> $GITHUB_OUTPUT
|
||||
echo "Extracted version: ${VERSION_TAG}"
|
||||
|
||||
|
||||
- name: Extract release info
|
||||
id: extract-release
|
||||
run: |
|
||||
@@ -33,63 +33,26 @@ jobs:
|
||||
name: Build release firmware
|
||||
uses: ./.github/workflows/task-build.yml
|
||||
with:
|
||||
python-version: '3.13'
|
||||
enable-dev-ota: false
|
||||
version-tag: ${{ needs.prepare.outputs.version-tag }}
|
||||
artifact-retention-days: 90
|
||||
artifact-name-prefix: 'firmware-'
|
||||
prepare-for-deploy: true
|
||||
|
||||
deploy:
|
||||
needs: [prepare, build]
|
||||
runs-on: ubuntu-latest
|
||||
name: Deploy release assets
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Download all firmware artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: firmware-*
|
||||
path: .pio/build
|
||||
merge-multiple: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
enable-cache: false
|
||||
|
||||
- name: Install platformio
|
||||
run: |
|
||||
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
|
||||
|
||||
- name: Reorganize artifacts for prepare_deploy.sh
|
||||
run: |
|
||||
# Move artifacts from download structure to expected .pio/build structure
|
||||
for env_dir in .pio/build/firmware-*; do
|
||||
if [ -d "$env_dir" ]; then
|
||||
env_name=$(basename "$env_dir" | sed 's/^firmware-//')
|
||||
mkdir -p ".pio/build/${env_name}"
|
||||
mv "$env_dir"/*.bin ".pio/build/${env_name}/" 2>/dev/null || true
|
||||
rm -rf "$env_dir"
|
||||
fi
|
||||
done
|
||||
|
||||
- name: Prepare Release Assets
|
||||
run: |
|
||||
sudo apt install rename
|
||||
./scripts/prepare_deploy.sh
|
||||
pattern: "*"
|
||||
path: generated/artifacts
|
||||
merge-multiple: true
|
||||
|
||||
- name: Upload Release Assets
|
||||
uses: bgpat/release-asset-action@03b0c30db1c4031ce3474740b0e4275cd7e126a3
|
||||
with:
|
||||
pattern: "toDeploy/*"
|
||||
pattern: "generated/artifacts/*"
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
release-url: ${{ needs.prepare.outputs.upload-url }}
|
||||
allow-overwrite: true
|
||||
@@ -99,13 +62,9 @@ jobs:
|
||||
name: Build and deploy release documentation
|
||||
uses: ./.github/workflows/task-docs.yml
|
||||
with:
|
||||
python-version: '3.11'
|
||||
node-version: '18.x'
|
||||
version-source: 'git-tag'
|
||||
base-path: '/'
|
||||
destination-dir: '.'
|
||||
generate-webuploader: true
|
||||
run-pagespeed: false
|
||||
mode: "prod"
|
||||
version: ${{ needs.prepare.outputs.version-tag }}
|
||||
url-prefix: "/"
|
||||
destination-dir: "."
|
||||
secrets:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
APIKEY: ${{ secrets.APIKEY }}
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
@@ -0,0 +1,44 @@
|
||||
name: Security Scan
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 2 * * 1' # Every Monday at 2 AM
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
severity:
|
||||
description: 'Severity levels to scan'
|
||||
required: false
|
||||
type: choice
|
||||
default: 'HIGH,CRITICAL'
|
||||
options:
|
||||
- 'CRITICAL'
|
||||
- 'HIGH,CRITICAL'
|
||||
- 'MEDIUM,HIGH,CRITICAL'
|
||||
- 'LOW,MEDIUM,HIGH,CRITICAL'
|
||||
scan-path:
|
||||
description: 'Path to scan (default: entire repo)'
|
||||
required: false
|
||||
type: string
|
||||
default: '.'
|
||||
exit-on-error:
|
||||
description: 'Fail if vulnerabilities found'
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
pull_request:
|
||||
paths:
|
||||
- 'main/**'
|
||||
- 'lib/**'
|
||||
- 'platformio.ini'
|
||||
- '.github/workflows/**'
|
||||
|
||||
jobs:
|
||||
security-scan:
|
||||
name: Scan for vulnerabilities
|
||||
uses: ./.github/workflows/task-security-scan.yml
|
||||
with:
|
||||
scan-type: 'fs'
|
||||
severity: ${{ inputs.severity || 'HIGH,CRITICAL' }}
|
||||
exit-code: ${{ inputs.exit-on-error && '1' || '0' }}
|
||||
upload-to-security-tab: true
|
||||
scan-path: ${{ inputs.scan-path || '.' }}
|
||||
@@ -4,37 +4,53 @@ on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
python-version:
|
||||
description: 'Python version to use'
|
||||
description: "Python version to use"
|
||||
required: false
|
||||
type: string
|
||||
default: '3.13'
|
||||
default: "3.13"
|
||||
pio-version:
|
||||
description: "PlatformIO version to use"
|
||||
required: false
|
||||
type: string
|
||||
default: "v6.1.18"
|
||||
environment-set:
|
||||
description: 'Which set of environments to build (all, ci)'
|
||||
required: false
|
||||
type: string
|
||||
default: 'all'
|
||||
enable-dev-ota:
|
||||
description: 'Enable development OTA builds'
|
||||
description: "Enable development OTA builds"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
version-tag:
|
||||
description: 'Version tag to inject into firmware (e.g., SHA or release tag)'
|
||||
description: "Optional version tag to pass to ci.sh build (omitted if empty)"
|
||||
required: false
|
||||
type: string
|
||||
default: 'unspecified'
|
||||
default: ""
|
||||
artifact-retention-days:
|
||||
description: 'Number of days to retain build artifacts'
|
||||
description: "Number of days to retain build artifacts"
|
||||
required: false
|
||||
type: number
|
||||
default: 7
|
||||
artifact-name-prefix:
|
||||
description: 'Prefix for artifact names (e.g., "firmware-" for dev builds)'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
prepare-for-deploy:
|
||||
description: 'Prepare firmware files for deployment with specific naming'
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
name: Lint code format
|
||||
uses: ./.github/workflows/task-lint.yml
|
||||
with:
|
||||
source: "main"
|
||||
extensions: "h,ino,cpp"
|
||||
clang-format-version: "9"
|
||||
|
||||
security-scan:
|
||||
name: Security vulnerability scan
|
||||
uses: ./.github/workflows/task-security-scan.yml
|
||||
with:
|
||||
severity: "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL"
|
||||
exit-code: "1"
|
||||
upload-to-security-tab: true
|
||||
|
||||
load-environments:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
@@ -43,11 +59,17 @@ jobs:
|
||||
- uses: actions/checkout@v4
|
||||
- id: set-matrix
|
||||
run: |
|
||||
ENVIRONMENTS=$(jq -c '.environments.all' .github/workflows/environments.json)
|
||||
ENV_SET="${{ inputs.environment-set }}"
|
||||
if [ "$ENV_SET" = "ci" ]; then
|
||||
ENVIRONMENTS=$(jq -c '.environments.ci' .github/workflows/environments.json)
|
||||
else
|
||||
ENVIRONMENTS=$(jq -c '.environments.all' .github/workflows/environments.json)
|
||||
fi
|
||||
echo "matrix=${ENVIRONMENTS}" >> $GITHUB_OUTPUT
|
||||
|
||||
|
||||
build:
|
||||
needs: load-environments
|
||||
needs: [lint, security-scan, load-environments]
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
@@ -70,35 +92,29 @@ jobs:
|
||||
|
||||
- name: Install PlatformIO dependencies
|
||||
run: |
|
||||
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
|
||||
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/${{ inputs.pio-version }}.zip
|
||||
|
||||
- name: Build firmware using ci.sh
|
||||
run: |
|
||||
BUILD_ARGS="${{ matrix.environments }}"
|
||||
|
||||
# Add version tag if specified
|
||||
if [ "${{ inputs.version-tag }}" != "unspecified" ]; then
|
||||
BUILD_ARGS="${{ matrix.environments }}"
|
||||
BUILD_ARGS="$BUILD_ARGS --deploy-ready"
|
||||
|
||||
# Optional version tag
|
||||
if [ -n "${{ inputs.version-tag }}" ]; then
|
||||
BUILD_ARGS="$BUILD_ARGS --version ${{ inputs.version-tag }}"
|
||||
fi
|
||||
|
||||
# Add mode flag (dev/prod)
|
||||
|
||||
# Mode
|
||||
if [ "${{ inputs.enable-dev-ota }}" = "true" ]; then
|
||||
BUILD_ARGS="$BUILD_ARGS --mode dev"
|
||||
else
|
||||
BUILD_ARGS="$BUILD_ARGS --mode prod"
|
||||
fi
|
||||
|
||||
# Add deploy flag if needed
|
||||
if [ "${{ inputs.prepare-for-deploy }}" = "true" ]; then
|
||||
BUILD_ARGS="$BUILD_ARGS --deploy-ready"
|
||||
fi
|
||||
|
||||
# Execute build (uses default generated/artifacts/ directory)
|
||||
./scripts/ci.sh build $BUILD_ARGS
|
||||
|
||||
- name: Upload firmware artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ${{ inputs.artifact-name-prefix }}${{ matrix.environments }}
|
||||
name: ${{ matrix.environments }}
|
||||
path: generated/artifacts/
|
||||
retention-days: ${{ inputs.artifact-retention-days }}
|
||||
@@ -3,61 +3,59 @@ name: Reusable Documentation Workflow
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
python-version:
|
||||
description: 'Python version to use'
|
||||
mode:
|
||||
description: 'Build mode: "prod" or "dev"'
|
||||
required: false
|
||||
type: string
|
||||
default: '3.11'
|
||||
node-version:
|
||||
description: 'Node.js version to use'
|
||||
default: "prod"
|
||||
version:
|
||||
description: 'Version string for documentation (use "auto" for automatic detection)'
|
||||
required: false
|
||||
type: string
|
||||
default: '14.x'
|
||||
version-source:
|
||||
description: 'Source of version: "release" (from GitHub release) or "custom" (provided)'
|
||||
required: false
|
||||
type: string
|
||||
default: 'release'
|
||||
custom-version:
|
||||
description: 'Custom version string when version-source is "custom"'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
base-path:
|
||||
description: 'Base path for documentation (e.g., "/" for prod, "/dev/" for dev)'
|
||||
required: false
|
||||
type: string
|
||||
default: '/'
|
||||
destination-dir:
|
||||
description: 'Destination directory for GitHub Pages deployment'
|
||||
required: false
|
||||
type: string
|
||||
default: '.'
|
||||
generate-webuploader:
|
||||
description: 'Generate WebUploader manifest'
|
||||
default: "auto"
|
||||
just-check:
|
||||
description: "Just create a quick build to check for errors"
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
webuploader-args:
|
||||
description: 'Additional arguments for gen_wu.py script'
|
||||
default: false
|
||||
url-prefix:
|
||||
description: 'URL prefix for docs (e.g., "/" for prod, "/dev/" for dev)'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
default: "/"
|
||||
run-pagespeed:
|
||||
description: 'Run PageSpeed Insights after deployment'
|
||||
description: "Run PageSpeed Insights after deploy"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
pagespeed-url:
|
||||
description: 'URL to test with PageSpeed Insights'
|
||||
description: "URL to test with PageSpeed"
|
||||
required: false
|
||||
type: string
|
||||
default: 'https://docs.openmqttgateway.com/'
|
||||
secrets:
|
||||
GITHUB_TOKEN:
|
||||
required: true
|
||||
APIKEY:
|
||||
default: "https://docs.openmqttgateway.com/"
|
||||
destination-dir:
|
||||
description: "GitHub Pages destination directory"
|
||||
required: false
|
||||
type: string
|
||||
default: "."
|
||||
python-version:
|
||||
description: "Python version to use"
|
||||
required: false
|
||||
type: string
|
||||
default: "3.13"
|
||||
pio-version:
|
||||
description: "PlatformIO version to use"
|
||||
required: false
|
||||
type: string
|
||||
default: "v6.1.18"
|
||||
secrets:
|
||||
github-token:
|
||||
description: "GitHub token for deploying to GitHub Pages"
|
||||
required: false
|
||||
pagespeed-apikey:
|
||||
description: "API key for PageSpeed Insights"
|
||||
required: false
|
||||
|
||||
|
||||
jobs:
|
||||
documentation:
|
||||
@@ -65,63 +63,84 @@ jobs:
|
||||
name: Create and deploy documentation
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: ${{ inputs.node-version }}
|
||||
fetch-depth: 0 # Fetch all history for git tags
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ inputs.python-version }}
|
||||
|
||||
- name: Build documentation site
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
enable-cache: false
|
||||
|
||||
- name: Install PlatformIO dependencies
|
||||
run: |
|
||||
# Build script arguments based on inputs
|
||||
ARGS=""
|
||||
|
||||
# Set version source and custom version
|
||||
if [ "${{ inputs.version-source }}" = "custom" ] && [ -n "${{ inputs.custom-version }}" ]; then
|
||||
ARGS="$ARGS --custom-version '${{ inputs.custom-version }}' --version-source custom"
|
||||
elif [ "${{ inputs.version-source }}" = "release" ]; then
|
||||
# Get latest release tag
|
||||
if command -v git >/dev/null 2>&1; then
|
||||
LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
|
||||
ARGS="$ARGS --custom-version '${LATEST_TAG}' --version-source release"
|
||||
uv pip install --system -U "https://github.com/pioarduino/platformio-core/archive/refs/tags/${{ inputs.pio-version }}.zip"
|
||||
|
||||
- name: Determine version
|
||||
id: version
|
||||
run: |
|
||||
VERSION="${{ inputs.version }}"
|
||||
MODE="${{ inputs.mode }}"
|
||||
|
||||
# If version is 'auto', determine it based on mode
|
||||
if [ "$VERSION" = "auto" ]; then
|
||||
if [ "$MODE" = "dev" ]; then
|
||||
# Dev mode: always use short SHA
|
||||
VERSION="$(git rev-parse --short HEAD)"
|
||||
echo "Dev mode - using SHA: $VERSION"
|
||||
else
|
||||
ARGS="$ARGS --version-source release"
|
||||
# Prod mode: try to get git tag
|
||||
if GIT_TAG=$(git describe --tags --exact-match 2>/dev/null); then
|
||||
VERSION="$GIT_TAG"
|
||||
echo "Using exact git tag: $VERSION"
|
||||
elif LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null); then
|
||||
VERSION="$LATEST_TAG"
|
||||
echo "Using latest tag: $VERSION"
|
||||
else
|
||||
VERSION="v0.0.0-unknown"
|
||||
echo "Warning: No tag found, using: $VERSION"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Set base path/URL prefix
|
||||
if [ "${{ inputs.base-path }}" != "/" ]; then
|
||||
ARGS="$ARGS --url-prefix '${{ inputs.base-path }}'"
|
||||
fi
|
||||
|
||||
# WebUploader generation
|
||||
if [ "${{ inputs.generate-webuploader }}" != "true" ]; then
|
||||
ARGS="$ARGS --no-webuploader"
|
||||
elif [ -n "${{ inputs.webuploader-args }}" ]; then
|
||||
ARGS="$ARGS --webuploader-args '${{ inputs.webuploader-args }}'"
|
||||
fi
|
||||
|
||||
# Run the build script
|
||||
eval "./scripts/ci_site.sh ${ARGS}"
|
||||
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
echo "Final version: $VERSION"
|
||||
|
||||
- name: Download all firmware artifacts
|
||||
if: ${{ inputs.mode == 'dev' }}
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: "*"
|
||||
path: generated/artifacts
|
||||
merge-multiple: true
|
||||
|
||||
- name: Build documentation site
|
||||
run: |
|
||||
./scripts/ci.sh site \
|
||||
--clean \
|
||||
--mode ${{ inputs.mode }} \
|
||||
--version ${{ steps.version.outputs.version }} \
|
||||
--url-prefix ${{ inputs.url-prefix }}
|
||||
|
||||
- name: Deploy to GitHub Pages
|
||||
uses: peaceiris/actions-gh-pages@v3
|
||||
if: ${{ inputs.just-check == false }}
|
||||
with:
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
github_token: ${{ secrets.github-token }}
|
||||
publish_dir: ./generated/site
|
||||
destination_dir: ${{ inputs.destination-dir }}
|
||||
cname: docs.openmqttgateway.com
|
||||
|
||||
- name: Run PageSpeed Insights
|
||||
if: inputs.run-pagespeed
|
||||
if: ${{ inputs.just-check == false && inputs.run-pagespeed == true }}
|
||||
uses: jakepartusch/psi-action@v1.3
|
||||
id: psi
|
||||
with:
|
||||
url: ${{ inputs.pagespeed-url }}
|
||||
threshold: 60
|
||||
key: ${{ secrets.APIKEY }}
|
||||
key: ${{ secrets.pagespeed-apikey }}
|
||||
@@ -4,20 +4,20 @@ on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
source:
|
||||
description: 'Source directory to lint (single directory)'
|
||||
description: "Source directory to lint"
|
||||
required: false
|
||||
type: string
|
||||
default: 'main'
|
||||
default: "main"
|
||||
extensions:
|
||||
description: 'File extensions to check (comma-separated)'
|
||||
description: "File extensions to check (comma-separated)"
|
||||
required: false
|
||||
type: string
|
||||
default: 'h,ino,cpp'
|
||||
default: "h,ino,cpp"
|
||||
clang-format-version:
|
||||
description: 'clang-format version to use'
|
||||
description: "clang-format version to use"
|
||||
required: false
|
||||
type: string
|
||||
default: '9'
|
||||
default: "9"
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
@@ -25,16 +25,16 @@ jobs:
|
||||
name: Check code format
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Install clang-format
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y clang-format
|
||||
|
||||
- name: Check code format with ci.sh qa
|
||||
sudo apt-get install -y clang-format shellcheck
|
||||
|
||||
- name: Check code format with ci.sh qa
|
||||
run: |
|
||||
./scripts/ci.sh qa \
|
||||
--check \
|
||||
--source "${{ inputs.source }}" \
|
||||
--extensions "${{ inputs.extensions }}" \
|
||||
--clang-format-version "${{ inputs.clang-format-version }}"
|
||||
--clang-format-version "${{ inputs.clang-format-version }}"
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
name: Reusable Security Scan Workflow
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
scan-type:
|
||||
description: 'Type of scan: fs (filesystem), config, or image'
|
||||
required: false
|
||||
type: string
|
||||
default: 'fs'
|
||||
severity:
|
||||
description: 'Severity levels to report (comma-separated: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL)'
|
||||
required: false
|
||||
type: string
|
||||
default: 'HIGH,CRITICAL'
|
||||
exit-code:
|
||||
description: 'Exit code when vulnerabilities found (0=continue, 1=fail)'
|
||||
required: false
|
||||
type: string
|
||||
default: '0'
|
||||
upload-to-security-tab:
|
||||
description: 'Upload SARIF to GitHub Security tab (GitHub-specific feature)'
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
scan-path:
|
||||
description: 'Path to scan (default: entire repository)'
|
||||
required: false
|
||||
type: string
|
||||
default: '.'
|
||||
generate-sbom:
|
||||
description: 'Generate Software Bill of Materials (SBOM) in CycloneDX and SPDX formats'
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
|
||||
jobs:
|
||||
security-scan:
|
||||
runs-on: ubuntu-latest
|
||||
name: Security vulnerability scan
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write # Required only for GitHub Security tab upload
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Install Trivy
|
||||
run: |
|
||||
set -euo pipefail
|
||||
curl -fsSL https://aquasecurity.github.io/trivy-repo/deb/public.key \
|
||||
| gpg --dearmor \
|
||||
| sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
|
||||
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" \
|
||||
| sudo tee /etc/apt/sources.list.d/trivy.list > /dev/null
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y trivy
|
||||
|
||||
- name: Run Security Scan using ci_security.sh
|
||||
run: |
|
||||
set -euo pipefail
|
||||
sbom_flag=""
|
||||
if [[ "${{ inputs.generate-sbom }}" == "true" ]]; then
|
||||
sbom_flag="--generate-sbom"
|
||||
fi
|
||||
|
||||
./scripts/ci_security.sh \
|
||||
--scan-type "${{ inputs.scan-type }}" \
|
||||
--scan-path "${{ inputs.scan-path }}" \
|
||||
--severity "${{ inputs.severity }}" \
|
||||
--exit-code "${{ inputs.exit-code }}" \
|
||||
${sbom_flag}
|
||||
|
||||
# GitHub-specific: Upload to Security tab
|
||||
- name: Upload SARIF to GitHub Security tab
|
||||
if: inputs.upload-to-security-tab == true && always()
|
||||
uses: github/codeql-action/upload-sarif@v4
|
||||
with:
|
||||
sarif_file: './generated/reports/trivy-results.sarif'
|
||||
category: 'trivy-security-scan'
|
||||
|
||||
# Agnostic: Upload all reports as artifacts (works on any CI)
|
||||
- name: Upload security reports as artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: security-scan-reports
|
||||
path: |
|
||||
./generated/reports/trivy-results.sarif
|
||||
./generated/reports/trivy-report.json
|
||||
./generated/reports/trivy-report.txt
|
||||
./generated/reports/security-summary.md
|
||||
./generated/reports/trivy-scan.log
|
||||
retention-days: 90
|
||||
|
||||
# Display summary in workflow output
|
||||
- name: Display Security Summary
|
||||
if: always()
|
||||
run: |
|
||||
cat ./generated/reports/security-summary.md >> $GITHUB_STEP_SUMMARY 2>/dev/null || echo "Summary not available"
|
||||
|
||||
#
|
||||
- name: Upload SBOM reports
|
||||
uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: sbom-reports
|
||||
path: |
|
||||
./generated/reports/sbom/
|
||||
retention-days: 90
|
||||
|
||||
# Optional: Fail the build if critical vulnerabilities found
|
||||
- name: Check for critical vulnerabilities
|
||||
if: inputs.exit-code == '1'
|
||||
run: |
|
||||
if [ -f ./generated/reports/trivy-results.sarif ]; then
|
||||
CRITICAL=$(jq '[.runs[].results[] | select(.level == "error")] | length' ./generated/reports/trivy-results.sarif 2>/dev/null || echo "0")
|
||||
if [ "$CRITICAL" -gt 0 ]; then
|
||||
echo "❌ Found ${CRITICAL} critical vulnerabilities!"
|
||||
echo "Review the security reports in artifacts."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
Reference in new issue
Block a user