[SITE] Renew the web board presentation and the ESP32 web upload + [SYS] Security checks (#2277)

* Refactor GitHub Actions workflows for build, documentation, and linting

- Consolidated build logic into reusable workflows (`task-build.yml` and `task-docs.yml`) to reduce duplication across multiple workflows.
- Introduced `environments.json` to centralize the list of PlatformIO build environments, improving maintainability and clarity.
- Updated `build.yml` and `build_and_docs_to_dev.yml` to utilize the new reusable workflows and environment definitions.
- Enhanced `release.yml` to streamline the release process and integrate documentation generation.
- Created reusable linting workflow (`task-lint.yml`) to standardize code formatting checks across the repository.
- Simplified manual documentation workflow by leveraging the new reusable documentation workflow.
- Improved artifact management and retention policies across workflows.
- Updated dependencies and versions in workflows to ensure compatibility and performance.

CI/CD pipeline agnostic of Workflow Engine and integrated on github actions

- Implemented ci.sh for orchestrating the complete build pipeline.
- Created ci_00_config.sh for centralized configuration of build scripts.
- Created ci_build_firmware.sh for building firmware for specified PlatformIO environments.
- Created ci_prepare_artifacts.sh for preparing firmware artifacts for upload or deployment.
- Created ci_set_version.sh for updating version tags in firmware configuration files.
- Created ci_build.sh to orchestrate the complete build pipeline.
- Created ci_qa.sh for code linting and formatting checks using clang-format.
- Created ci_site.sh for building and deploying VuePress documentation with version management.
- Implemented checks for required tools and dependencies in the new scripts.
- Improved internal scripts for better error handling and logging.

UPDATE the web installer manifest generation and update documentation structure
- Enhanced ci_list-env.sh to list environments from a JSON file.
- Replaced  common_wu.py and gen_wu.py scripts with new npm scripts for site generation and previewing on docsgen/gen_wu.js
- Replaced  generate_board_docs.py with docsgen/generated_board_docs.js
- Added new npm scripts for integration of site generation on build phase.
- Created preview_site.js to serve locally generated site over HTTPS with improved error handling.
- Added new CI environments for CI builds in environments.json.
- Deleted lint.yml as part of workflow cleanup.
- Enhanced task-build.yml to include linting as a job and added support for specifying PlatformIO version.
- Improved task-docs.yml to handle versioning more effectively and added clean option.

Enhance documentation
- ADD CLEAR Mark of development version of site
- Updated README.md to include detailed workflow dependencies and relationships using mermaid diagrams.
- Improved development.md with a quick checklist for contributors and clarified the code style guide.
- Enhanced quick_start.md with tips for contributors and streamlined the workflow explanation.

LINT FIX
- Refined User_config.h for better formatting consistency.
- Adjusted blufi.cpp and gatewayBT.cpp for improved code readability and consistency in formatting.
- Updated gatewaySERIAL.cpp and mqttDiscovery.cpp to enhance logging error messages.
- Improved sensorDS1820.cpp for better logging of device information.

Add security scan workflows for vulnerability detection

Add SBOM generation and upload to release workflow; update security scan summary handling

Add shellcheck suppor + FIX shellcheck warning

Enhance documentation for CI/CD scripts and workflows, adding details for security scanning and SBOM generation processes

Fix formatting and alignment in BLE connection handling

Reviewed the full web board presentation and the ESP32 web upload. The project uses a modern pattern where data is divided from the presentation layer.

- Removed the `generate_board_docs` script.
- Updated the `gen_wu` script in order to generate `boards-info.json`: the fail that containe all information about the configuration
- Created and isolate the file `boards-info.js` to streamline the parsing of PlatformIO dependencies, modules, environments and improve the handling of library information.
- Introduced vuepress component `BoardEnvironmentTable.vue` that render `boards-info.json` as UI card component
- Introduced vuepress component `FlashEnvironmentSelector.vue` that render a selectred environment from  `boards-info.json` and provide esp-web-upload feature on it
- Introduced a new board page `board-selector.md` for improved firmware selection.
- Updated `web-install.md` to enhance the firmware upload process, including a new board environment table.
- Enhanced custom descriptions in `environments.ini` to include HTML links for better user guidance and board image link

Add CC1101 initialization improvements and logging enhancements
Add installation step for PlatformIO dependencies in documentation workflow

Remove ci_set_version.sh script and associated versioning functionality

* Fix comment provisined

Fix PlatformIO version input reference in documentation workflow

Remove outdated Squeezelite-ESP32 installer documentation
This commit is contained in:
Alessandro Staniscia authored and GitHub committed 2026-03-09 07:47:30 -05:00
1 parent 5f0966c2c1
commit 98481c5145
73 files changed
+5947 -2409

No files matched your search

+390 -285
View File
@@ -1,8 +1,7 @@
# GitHub Actions Workflows Documentation
This document provides a overview of all GitHub Actions workflows in the OpenMQTTGateway project.
This document provides an overview of all GitHub Actions workflows in the OpenMQTTGateway project.
---
## Architecture Overview
@@ -14,7 +13,7 @@ Entry points triggered by user actions, schedules, or events:
- `build_and_docs_to_dev.yml` - Daily development builds
- `release.yml` - Production releases
- `manual_docs.yml` - Documentation deployment
- `lint.yml` - Code formatting check
- `security-scan.yml` - Security vulnerability scanning
- `stale.yml` - Issue management
### **Task Workflows** (Reusable components)
@@ -22,8 +21,8 @@ Parameterized building blocks called by main workflows:
- `task-build.yml` - Configurable firmware build
- `task-docs.yml` - Configurable documentation build
- `task-lint.yml` - Configurable code formatting check
- `task-security-scan.yml` - Configurable security scanning
---
## Workflow Overview Table
@@ -33,11 +32,188 @@ Parameterized building blocks called by main workflows:
| `build_and_docs_to_dev.yml` | Daily Cron, Manual | Development Builds + Docs | Firmware + Docs deployment |
| `release.yml` | Release Published | Production Release | Release assets + Docs |
| `manual_docs.yml` | Manual, Workflow Call | Documentation Only | GitHub Pages docs |
| `lint.yml` | Push, Pull Request | Code Format Check | None |
| `security-scan.yml` | Weekly Cron, Manual | Security Vulnerability Scanning | SARIF, SBOM reports |
| `stale.yml` | Daily Cron | Issue Management | None |
| **`task-build.yml`** | **Workflow Call** | **Reusable Build Logic** | **Configurable** |
| **`task-docs.yml`** | **Workflow Call** | **Reusable Docs Logic** | **GitHub Pages** |
| **`task-lint.yml`** | **Workflow Call** | **Reusable Lint Logic** | **None** |
| **`task-security-scan.yml`** | **Workflow Call** | **Reusable Security Scan** | **SARIF, SBOM, Reports** |
## Workflow Dependencies and Call Chain
```mermaid
flowchart TD
%% Triggers
subgraph triggers ["🎯 Triggers"]
push["Push"]
pr["Pull Request"]
release["Release Published"]
cron1["Cron: Daily 00:00 UTC"]
manual["Manual Trigger"]
cron3["Cron: Weekly Monday 02:00 UTC"]
cron2["Cron: Daily 00:30 UTC"]
end
subgraph github_workflows ["📋 GitHub Workflows"]
%% Main Workflows
subgraph main ["📋 Main Workflows"]
build["build.yml<br/>CI Build"]
release_wf["release.yml<br/>Production Release"]
build_dev["build_and_docs_to_dev.yml<br/>Dev Builds"]
security_scan["security-scan.yml<br/>Security Scan"]
stale["stale.yml<br/>Issue Management"]
manual_docs["manual_docs.yml<br/>Docs Only"]
end
%% Task Workflows
subgraph tasks ["⚙️ Task Workflows"]
task_build["task-build.yml<br/>Build Firmware"]
task_lint["task-lint.yml<br/>Code Format"]
task_security["task-security-scan.yml<br/>Security Scan"]
task_docs["task-docs.yml<br/>Build & Deploy Docs"]
end
end
subgraph ci_scripts ["🔧 CI Scripts"]
%% CI Scripts Layer
subgraph bash ["🔧 Orchestrator"]
ci_main["ci.sh<br/>(main dispatcher)"]
ci_build_script["ci_build.sh<br/>(build orchestrator)"]
ci_site_script["ci_site.sh<br/>(docs orchestrator)"]
ci_qa_script["ci_qa.sh<br/>(lint orchestrator)"]
ci_security_script["ci_security.sh<br/>(security orchestrator)"]
end
%% Sub-Scripts Layer
subgraph sub_scripts ["⚙️ Workers"]
ci_build_fw["ci_build_firmware.sh<br/>(PlatformIO build)"]
ci_prep_art["ci_prepare_artifacts.sh<br/>(artifact packaging)"]
gen_board["generate_board_docs<br/>(npm package)"]
gen_wu["gen_wu<br/>(npm package)"]
clang_fmt["clang-format<br/>(code formatter)"]
trivy["Trivy<br/>(vulnerability scanner)"]
end
end
%% Trigger connections
push -->|"all or subset<br/>(depends on branch)"| build
pr -->|"all<br/>(always full)"| build
release --> release_wf
manual --> manual_docs
cron1 --> build_dev
cron2 --> stale
cron3 --> security_scan
%% Main workflow to task workflow connections
build --> task_build
build_dev --> task_build
build_dev --> task_docs
release_wf --> task_build
release_wf --> task_docs
manual_docs --> task_docs
security_scan --> task_security
%% Task workflow internal dependencies
task_build --> task_lint
task_build --> task_security
%% Task workflows to CI scripts
task_build -->|"ci.sh build ..."| ci_main
task_docs -->|"ci.sh site ..."| ci_main
task_lint -->|"ci.sh qa ..."| ci_main
task_security -->|"ci.sh security ..."| ci_main
%% CI main dispatcher to orchestrators
ci_main -->|"route: build"| ci_build_script
ci_main -->|"route: site"| ci_site_script
ci_main -->|"route: qa"| ci_qa_script
ci_main -->|"route: security"| ci_security_script
%% Orchestrators to workers
ci_build_script --> ci_build_fw
ci_build_script --> ci_prep_art
ci_site_script --> gen_board
ci_site_script --> gen_wu
ci_qa_script --> clang_fmt
ci_security_script --> trivy
%% Styling
classDef triggerStyle fill:#e1f5ff,stroke:#0066cc,stroke-width:2px
classDef mainStyle fill:#fff4e6,stroke:#ff9900,stroke-width:2px
classDef taskStyle fill:#e6f7e6,stroke:#00aa00,stroke-width:2px
classDef ciStyle fill:#ffe6f0,stroke:#cc0066,stroke-width:2px
classDef subStyle fill:#f0e6ff,stroke:#9933ff,stroke-width:2px
class push,pr,release,manual,cron1,cron2,cron3 triggerStyle
class build,release_wf,manual_docs,build_dev,stale,security_scan mainStyle
class task_build,task_docs,task_lint,task_security taskStyle
class ci_main,ci_build_script,ci_site_script,ci_qa_script,ci_security_script ciStyle
class ci_build_fw,ci_prep_art,gen_board,gen_wu,clang_fmt,trivy subStyle
style github_workflows stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:1.8px,fill:#fbfbfc
style main stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fcfdff
style tasks stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fcfdff
style ci_scripts stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:1.8px,fill:#fffaf5
style bash stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fffaf5
style sub_scripts stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fffaf5
style triggers fill:none,stroke:none
linkStyle 0,1,2,3,4,5,6 stroke:#0066cc,stroke-width:3px;
linkStyle 7,8,9,10,11,12,13,14,15 stroke:orange, stroke-width:2px;
linkStyle 16,17,18,19,20,21,22,23,24,25,26,27,28,29 stroke:red, stroke-width:2px;
```
### Workflow Relationships
**Main → Task Mapping**:
- `build.yml` → calls `task-build.yml` (also contains inline documentation job)
- `build_and_docs_to_dev.yml` → calls `task-build.yml` + `task-docs.yml`
- `release.yml` → calls `task-build.yml` + `task-docs.yml`
- `manual_docs.yml` → calls `task-docs.yml`
- `security-scan.yml` → calls `task-security-scan.yml`
- `stale.yml` → standalone (no dependencies)
**Task → CI Script Mapping**:
- `task-build.yml` → `ci.sh build --version --mode --deploy-ready`
- Routes to: `ci_build.sh` → `ci_build_firmware.sh`, `ci_prepare_artifacts.sh`
- Output: `generated/artifacts/firmware_build/`
- `task-docs.yml` → `ci.sh site --mode --version --url-prefix`
- Routes to: `ci_site.sh` → `generate_board_docs` (npm), `gen_wu` (npm), VuePress
- Output: `generated/site/`
- `task-lint.yml` → `ci.sh qa --check --source --extensions --clang-format-version`
- Routes to: `ci_qa.sh` → `clang-format`
- `task-security-scan.yml` → `ci.sh security --scan-type --severity --generate-sbom`
- Routes to: `ci_security.sh` → Trivy (vulnerability scanner)
- Output: `generated/reports/` (SARIF, JSON, SBOM)
**Job Dependencies**:
- `build_and_docs_to_dev.yml`: prepare → build (task) → deploy & documentation (task)
- `release.yml`: prepare → build (task) → deploy → documentation (task)
**Script Execution Flow**:
```
GitHub Action (task-*.yml)
↓
./scripts/ci.sh <command> [OPTIONS] ← Main dispatcher
↓
./scripts/ci_<command>.sh ← Command orchestrator
↓
./scripts/ci_*.sh / *.py ← Worker scripts
```
---
@@ -45,16 +221,19 @@ Parameterized building blocks called by main workflows:
### 1. `build.yml` - Continuous Integration Build
**Purpose**: Validates that code changes compile successfully across all supported hardware platforms.
**Purpose**: Validates that code changes compile successfully with intelligent environment selection based on branch importance.
**Triggers**:
- **Push**: Every commit pushed to any branch
- **Pull Request**: Every PR creation or update
**What it does**:
1. **Build job**: Calls `task-build.yml` with CI parameters
- Builds firmware for **83 hardware environments** in parallel
2. **Documentation job**: Inline job that validates docs build (doesn't deploy)
1. **Determine build scope**: Selects environment list based on branch name
- **Full build** (`all` environment): All PRs and Push on important branches (development, master, edge, stable, release/*, hotfix/*)
- **Quick build** (`ci` subset environment): All PRs and Push on non-critical branches
2. **Build job**: Calls `task-build.yml` with appropriate environment set
- Builds firmware in parallel
3. **Documentation job**: Inline job that validates docs build (doesn't deploy)
- Downloads common config from theengs.io
- Runs `npm install` and `npm run docs:build`
- Uses Node.js 14.x
@@ -64,15 +243,18 @@ Parameterized building blocks called by main workflows:
- Python version: 3.13 (for build job)
- Build strategy: Parallel matrix via task workflow
- Artifact retention: 7 days
- Development OTA: Disabled (`enable-dev-ota: false`)
- Development OTA: Enabled (`enable-dev-ota: true`)
- Environment selection logic:
- **Full build** (`all`): All Pull Requests + branches: development, master, edge, stable, release/*, hotfix/*
- **Quick build** (`ci` subset): All other feature branches
**Outputs**:
- Firmware binaries for each environment (83 artifacts)
- Firmware binaries for selected environments
- No documentation deployment (validation only)
**Use Case**: Ensures no breaking changes before merge. Fast feedback for developers.
**Use Case**: Ensures no breaking changes before merge. Fast feedback for feature branches (~10 min), comprehensive validation for PRs and critical branches (~40 min).
**Execution Context**: Runs for ALL contributors on ALL branches.
**Execution Context**: Runs for ALL contributors on ALL branches with smart scaling based on branch importance.
---
@@ -86,38 +268,37 @@ Parameterized building blocks called by main workflows:
**What it does**:
1. **Prepare job**: Generates 6-character short SHA
2. **Build job**: Calls `task-build.yml` with development parameters
- Builds firmware for **83 hardware environments** in parallel
2. **Handle-firmwares job**: Calls `task-build.yml` with development parameters
- Builds firmware for **all environments** in parallel
- Enables development OTA updates with SHA commit version
3. **Deploy job**: Prepares and uploads assets
- Creates library dependency zips for each board
- Generates source code zip
- Removes test environment binaries
4. **Documentation job**: Calls `task-docs.yml` with development parameters
- Deploys to `/dev` subdirectory
- Adds "DEVELOPMENT" watermark
- Runs PageSpeed Insights
- Artifact retention: 1 day
3. **Handle-documentation job**: Calls `task-docs.yml` with development parameters
- Deploys to `/dev` subdirectory on GitHub Pages
- Uses short SHA as version identifier
- Runs PageSpeed Insights on dev site
**Technical Details**:
- **Calls**: `task-build.yml` + `task-docs.yml`
- Python version: 3.13 (build), 3.11 (docs)
- Node.js version: 16.x (docs)
- Repository restriction: Hardcoded to `1technophile` owner only
- Artifact retention: 1 day
- Build flag: `enable-dev-ota: true` (passed to task-build.yml)
- Version: 6-character short SHA (e.g., `abc123`)
- Documentation URL prefix: `/dev/`
- GitHub Pages destination: `dev` subdirectory
- PageSpeed URL: `https://docs.openmqttgateway.com/dev/`
**Workflow Parameters**:
- Build: `enable-dev-ota: true`, `version-tag: <short-sha>`, `artifact-retention-days: 1`
- Docs: `mode: "dev"`, `version: <short-sha>`, `url-prefix: "/dev/"`, `destination-dir: "dev"`, `run-pagespeed: true`
**Outputs**:
- Firmware binaries with `-firmware.bin` suffix
- Firmware binaries with `-firmware.bin` suffix (1 day retention)
- Bootloader and partition binaries
- Library dependency zips per board
- Source code zip
- Documentation deployed to `docs.openmqttgateway.com/dev/`
**Version Labeling**:
- Git SHA (6 chars) injected into firmware
- Docs tagged: "DEVELOPMENT SHA:XXXXXX TEST ONLY"
- Git SHA (6 chars) injected into firmware via `version-tag`
- Docs display short SHA as version
**Use Case**: Daily bleeding-edge builds for early adopters and testing. Preview documentation changes.
**Use Case**: Daily bleeding-edge builds for early adopters and testing. Preview documentation changes before production release.
**Execution Context**: Only runs on `1technophile` repository owner. Forks will skip this workflow automatically.
@@ -131,33 +312,34 @@ Parameterized building blocks called by main workflows:
- **Release**: When a GitHub release is published (tagged)
**What it does**:
1. **Prepare job**: Extracts version tag and release info
1. **Prepare job**: Extracts version tag and release info from GitHub event
2. **Build job**: Calls `task-build.yml` with production parameters
- Builds firmware for **83 hardware environments** in parallel
- Injects release tag version into firmware
3. **Deploy job**: Prepares and uploads release assets
- Downloads all build artifacts
- Reorganizes for `prepare_deploy.sh` script
- Creates library zips and source zip
- Uploads to GitHub Release
- Builds firmware for **all environments** in parallel
- Injects release tag version into firmware
- Artifact retention: 90 days
3. **Deploy job**: Downloads and uploads release assets
- Downloads all firmware artifacts from build job
- Uploads binaries to GitHub Release
4. **Documentation job**: Calls `task-docs.yml` for production docs
- Deploys to root (`/`) of GitHub Pages
- Uses release tag as version
**Technical Details**:
- **Calls**: `task-build.yml` + `task-docs.yml`
- Python version: 3.13 (build), 3.11 (docs)
- Node.js version: 18.x
- Build flag: Standard (no DEVELOPMENTOTA)
- Artifact retention: 90 days
- Uses `prepare_deploy.sh` script for asset preparation
- Deploy uses `bgpat/release-asset-action` to attach assets to GitHub Release
**Workflow Parameters**:
- Build: `enable-dev-ota: false`, `version-tag: <git-tag>`, `artifact-retention-days: 90`
- Docs: `mode: "prod"`, `version: <git-tag>`, `url-prefix: "/"`, `destination-dir: "."`
**Outputs**:
- Production firmware binaries attached to GitHub Release
- Library zips per board
- Source code zip
- Production firmware binaries attached to GitHub Release (all prepared by `task-build.yml` with `--deploy-ready`)
- Production documentation at `docs.openmqttgateway.com/`
**Version Labeling**:
- Git tag (e.g., `v1.2.3`) injected into firmware and `latest_version.json`
- Git tag (e.g., `v1.2.3`) injected into firmware
**Workflow Chain**:
```
@@ -184,10 +366,11 @@ prepare → build (task-build.yml) → deploy → documentation (task-docs.yml)
**Technical Details**:
- **Calls**: `task-docs.yml`
- Python version: 3.11
- Node.js version: 14.x
- Version source: Latest GitHub release tag
- WebUploader manifest: Enabled
- Mode: `prod`
- Version: Uses latest release tag by default (or provided input)
- URL prefix: `/`
- Destination: Root of GitHub Pages
- PageSpeed: Optional (disabled by default)
**Outputs**:
- Production documentation at `docs.openmqttgateway.com/`
@@ -199,34 +382,55 @@ prepare → build (task-build.yml) → deploy → documentation (task-docs.yml)
---
### 5. `lint.yml` - Code Format Validation
### 5. `security-scan.yml` - Security Vulnerability Scanning
**Purpose**: Ensures code follows consistent formatting standards.
**Purpose**: Scans the project for security vulnerabilities and generates Software Bill of Materials (SBOM) for supply chain security.
**Triggers**:
- **Push**: Every commit pushed to any branch
- **Pull Request**: Every PR creation or update
- **Schedule**: Weekly on Monday at 02:00 UTC (`0 2 * * 1`)
- **Manual**: Via workflow_dispatch button with input parameters
**Manual Trigger Inputs**:
- `severity`: Severity levels to scan (choices: UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL; default: HIGH,CRITICAL)
- Allows filtering results to specific severity levels
**What it does**:
1. Calls `task-lint.yml` with specific parameters
2. Checks formatting in `./main` directory only
1. **Prepare job**: Sets up environment for scanning
2. **Security scan job**: Calls `task-security-scan.yml` with parameters
- Runs Trivy vulnerability scanner on filesystem
- Filters results by severity level
- Generates SARIF format for GitHub Security tab integration
- Creates SBOM in CycloneDX and SPDX formats
- Uploads findings to GitHub Security tab (code scanning dashboard)
3. **Artifact upload job**: Stores generated reports and SBOM
- SARIF results for GitHub integration
- SBOM files for supply chain tracking
- Retention: 90 days
**Technical Details**:
- **Calls**: `task-lint.yml`
- clang-format version: 9
- File extensions: `.h`, `.ino` (not `.cpp`)
- Source directory: `main` (single directory)
- **Calls**: `task-security-scan.yml`
- Vulnerability scanner: Trivy (vulnerability database updated automatically)
- Report formats: SARIF (GitHub), JSON (detailed), Markdown (summary)
- SBOM formats: CycloneDX and SPDX (standard formats)
- Failure behavior: Does NOT fail the workflow on vulnerabilities (exit-code: 0)
- GitHub Security tab: Auto-uploads SARIF for code scanning dashboard visibility
**Configuration**:
```yaml
source: 'main'
extensions: 'h,ino'
clang-format-version: '9'
```
**Workflow Parameters**:
- `scan-type: "fs"` (filesystem scan)
- `severity: "HIGH,CRITICAL"` (default, or manual input)
- `generate-sbom: true` (always enabled)
- `upload-to-security-tab: true` (GitHub Security integration)
**Use Case**: Maintains code quality and consistency. Prevents formatting debates in PRs.
**Outputs**:
- SARIF report: `generated/reports/trivy-results.sarif` (GitHub Security tab)
- JSON report: `generated/reports/trivy-results.json` (detailed findings)
- Markdown summary: `generated/reports/security-summary.md`
- SBOM: `generated/reports/sbom/sbom.cyclonedx.json` + `sbom.spdx.json`
- Artifacts retained for 90 days
**Execution Context**: Runs for ALL contributors on ALL branches.
**Use Case**: Regular security audits, compliance tracking, vulnerability management, supply chain security.
**Execution Context**: Weekly automated scans + manual on-demand scanning for developers.
---
@@ -267,17 +471,19 @@ clang-format-version: '9'
**Trigger**: `workflow_call` only (called by other workflows)
**Parameters**:
- `python-version`: Python version (default: '3.13')
- `enable-dev-ota`: Enable development OTA (default: false)
- `version-tag`: Version to inject (default: 'unspecified')
- `artifact-retention-days`: Artifact retention (default: 7)
- `artifact-name-prefix`: Artifact name prefix (default: '')
- `prepare-for-deploy`: Prepare for deployment (default: false)
- `python-version`: Python version to use (default: '3.13')
- `pio-version`: PlatformIO version to use (default: 'v6.1.18')
- `environment-set`: Which set of environments to build: 'all' or 'ci' (default: 'all')
- `enable-dev-ota`: Enable development OTA builds (default: false)
- `version-tag`: Optional version tag to pass to ci.sh build - omitted if empty (default: '')
- `artifact-retention-days`: Number of days to retain build artifacts (default: 7)
**What it does**:
1. **Load environments**: Reads environment list from `environments.json`
2. **Matrix build**: Builds all 83 environments in parallel
3. **Build execution**: Calls unified `ci.sh build <environment> [OPTIONS]`:
1. **Lint code**: Runs `task-lint.yml` to check code formatting (main directory, .h and .ino files)
2. **Load environments**: Reads environment list from `environments.json` based on `environment-set` input (`all` or `ci`)
3. **Install PlatformIO**: Uses `uv` to install the `pio-version` input (custom `pioarduino/platformio-core` fork)
4. **Matrix build**: Builds selected environments in parallel, blocking on lint job completion
5. **Build execution**: Calls unified `ci.sh build <environment> [OPTIONS]`:
- `<environment>`: Target hardware (e.g., `esp32dev-ble`)
- `--version <tag>`: Version to inject (SHA for dev, tag for prod)
- `--mode <dev|prod>`: Build mode (enables/disables OTA)
@@ -289,16 +495,17 @@ clang-format-version: '9'
./scripts/ci.sh build esp32dev-ble --version v1.8.0 --mode prod --deploy-ready
↓
├─→ ci_build.sh (orchestrator)
│ ├─→ ci_set_version.sh v1.8.0 [--dev]
│ ├─→ ci_build_firmware.sh esp32dev-ble [--dev-ota]
│ └─→ ci_prepare_artifacts.sh esp32dev-ble [--deploy] → outputs to generated/artifacts/
```
**Technical Details**:
- Runs on: Ubuntu latest
- PlatformIO version: 6.1.18 (custom fork: `pioarduino/platformio-core`)
- PlatformIO version: Configurable via `pio-version` input (default: v6.1.18, custom fork: `pioarduino/platformio-core`)
- Python package manager: `uv` (astral-sh/setup-uv@v6)
- Strategy: Matrix with fail-fast: false
- Environment sets: `all` (complete set) or `ci` (subset for quick validation)
- Strategy: Matrix with fail-fast: false (builds complete even if one environment fails)
- Dependencies: Lint job must pass before build matrix starts
- Main orchestrator: `ci.sh` → `ci_build.sh` → sub-scripts
**Callers**:
@@ -315,36 +522,24 @@ clang-format-version: '9'
**Trigger**: `workflow_call` only (called by other workflows)
**Parameters**:
- `python-version`: Python version (default: '3.11')
- `node-version`: Node.js version (default: '14.x')
- `version-source`: Version source ('release', 'git-tag', 'custom')
- `custom-version`: Custom version string (optional)
- `base-path`: Base URL path (default: '/')
- `destination-dir`: Deploy directory (default: '.')
- `generate-webuploader`: Generate WebUploader manifest (default: true)
- `webuploader-args`: WebUploader generation arguments (optional)
- `run-pagespeed`: Run PageSpeed Insights (default: false)
- `pagespeed-url`: URL for PageSpeed test (optional)
- `mode`: Documentation mode (`prod` or `dev`, default: `prod`)
- `version`: Version string for docs (default: `auto` → tag or short SHA depending on caller)
- `url-prefix`: Base URL path (default: `/` for prod, `/dev/` for dev callers)
- `destination-dir`: Deploy directory on GitHub Pages (default: `.`)
- `run-pagespeed`: Run PageSpeed Insights after deploy (default: false)
- `pagespeed-url`: URL to test with PageSpeed (default: `https://docs.openmqttgateway.com/`)
**What it does**:
1. **Build documentation**: Calls unified `ci.sh site [OPTIONS]`:
- `--mode <dev|prod>`: Documentation mode
- `--custom-version <ver>`: Custom version string
- `--version-source <release|custom>`: Version source
- `--url-prefix <path>`: Base URL path (e.g., `/dev/`)
- `--webuploader-args <args>`: WebUploader options
- `--no-webuploader`: Skip manifest generation
1. **Build documentation**: Calls unified `ci.sh site --mode <mode> --version <ver> --url-prefix <path>`
2. **Deploy**: Publishes to GitHub Pages using `peaceiris/actions-gh-pages@v3`
3. **PageSpeed test**: Optionally runs performance audit
3. **PageSpeed test**: Optionally runs performance audit on the provided URL
**Command Flow**:
```bash
./scripts/ci.sh site --mode prod --version-source release --url-prefix /
↓
└─→ ci_site.sh (orchestrator)
├─→ generate_board_docs.py (auto-generate board docs)
├─→ npm run docs:build (VuePress compilation)
└─→ gen_wu.py (WebUpdater manifest)
./scripts/ci.sh site --mode prod --version v1.2.3 --url-prefix /
↓
└─→ ci_site.sh (orchestrator)
└─→ npm run docs:build (VuePress compilation)
```
**Callers**:
@@ -356,15 +551,14 @@ clang-format-version: '9'
### 9. `task-lint.yml` - Reusable Lint Workflow
**Purpose**: Parameterized code formatting validation.
**Purpose**: Parameterized code formatting validation for consistent code style.
**Trigger**: `workflow_call` only (called by other workflows)
**Parameters**:
- `source`: Source directories to lint (default: './lib ./main')
- `extensions`: File extensions to check (default: 'h,ino,cpp')
- `clang-format-version`: clang-format version (default: '9')
- `exclude-pattern`: Pattern to exclude (optional)
- `source`: Source directory to lint (default: 'main')
- `extensions`: File extensions to check, comma-separated (default: 'h,ino')
- `clang-format-version`: clang-format version to use (default: '9')
**What it does**:
1. Checks out code
@@ -386,170 +580,77 @@ clang-format-version: '9'
```
**Technical Details**:
- Runs on: Ubuntu latest
- Script: `ci_qa.sh` (custom formatting check script)
- Install: `clang-format-$version` via apt-get
- Default source: `main` (single directory)
- Default extensions: `h,ino` (not cpp)
- Formatter: `clang-format-$version` installed via apt-get
- Default scope: `main` directory only (not lib)
- Default file types: `.h` and `.ino` (not `.cpp`)
- Strategy: Single sequential job (not parallelized)
**Callers**:
- `lint.yml` (CI lint check)
- `build.yml` (inline lint check before build)
- Can be called by other workflows as needed
**Default Behavior**: If called without parameters, lints `main` directory for `.h` and `.ino` files.
---
## Workflow Dependencies and Call Chain
```mermaid
flowchart TD
%% Triggers
subgraph triggers ["🎯 Triggers"]
push["Push / Pull Request"]
release["Release Published"]
manual["Manual Trigger"]
cron1["Cron: Daily 00:00 UTC"]
cron2["Cron: Daily 00:30 UTC"]
end
subgraph github_workflows ["📋 GitHub Workflows"]
%% Main Workflows
subgraph main ["📋 Main Workflows"]
lint["lint.yml<br/>Format Check"]
build["build.yml<br/>CI Build"]
release_wf["release.yml<br/>Production Release"]
manual_docs["manual_docs.yml<br/>Docs Only"]
build_dev["build_and_docs_to_dev.yml<br/>Dev Builds"]
stale["stale.yml<br/>Issue Management"]
end
%% Task Workflows
subgraph tasks ["⚙️ Task Workflows"]
task_build["task-build.yml<br/>Build Firmware"]
task_docs["task-docs.yml<br/>Build & Deploy Docs"]
task_lint["task-lint.yml<br/>Code Format"]
end
end
subgraph ci_scripts ["🔧 CI Scripts"]
%% CI Scripts Layer
subgraph bash ["🔧 Orchestrator"]
ci_main["ci.sh<br/>(main dispatcher)"]
ci_build_script["ci_build.sh<br/>(build orchestrator)"]
ci_site_script["ci_site.sh<br/>(docs orchestrator)"]
ci_qa_script["ci_qa.sh<br/>(lint orchestrator)"]
end
%% Sub-Scripts Layer
subgraph sub_scripts ["⚙️ Workers"]
ci_set_ver["ci_set_version.sh<br/>(version injection)"]
ci_build_fw["ci_build_firmware.sh<br/>(PlatformIO build)"]
ci_prep_art["ci_prepare_artifacts.sh<br/>(artifact packaging)"]
gen_board["generate_board_docs.py<br/>(board docs)"]
gen_wu["gen_wu.py<br/>(WebUpdater manifest)"]
clang_fmt["clang-format<br/>(code formatter)"]
end
end
%% Trigger connections
push --> lint
push --> build
release --> release_wf
manual --> manual_docs
cron1 --> build_dev
cron2 --> stale
%% Main workflow to task workflow connections
build -->|calls| task_build
lint -->|calls| task_lint
build_dev -->|calls| task_build
build_dev -->|calls| task_docs
release_wf -->|calls| task_build
release_wf -->|calls| task_docs
manual_docs -->|calls| task_docs
%% Task workflows to CI scripts
task_build -->|"ci.sh build<br/>--version --mode<br/>--deploy-ready"| ci_main
task_docs -->|"ci.sh site<br/>--mode --url-prefix<br/>--version-source"| ci_main
task_lint -->|"ci.sh qa<br/>--check --source<br/>--extensions"| ci_main
%% CI main dispatcher to orchestrators
ci_main -->|"route: build"| ci_build_script
ci_main -->|"route: site"| ci_site_script
ci_main -->|"route: qa"| ci_qa_script
%% Orchestrators to workers
ci_build_script --> ci_set_ver
ci_build_script --> ci_build_fw
ci_build_script --> ci_prep_art
ci_site_script --> gen_board
ci_site_script --> gen_wu
ci_qa_script --> clang_fmt
%% Styling
classDef triggerStyle fill:#e1f5ff,stroke:#0066cc,stroke-width:2px
classDef mainStyle fill:#fff4e6,stroke:#ff9900,stroke-width:2px
classDef taskStyle fill:#e6f7e6,stroke:#00aa00,stroke-width:2px
classDef ciStyle fill:#ffe6f0,stroke:#cc0066,stroke-width:2px
classDef subStyle fill:#f0e6ff,stroke:#9933ff,stroke-width:2px
class push,release,manual,cron1,cron2 triggerStyle
class build,lint,release_wf,manual_docs,build_dev,stale mainStyle
class task_build,task_docs,task_lint taskStyle
class ci_main,ci_build_script,ci_site_script,ci_qa_script ciStyle
class ci_set_ver,ci_build_fw,ci_prep_art,gen_board,gen_wu,clang_fmt subStyle
**Default Behavior**: If called without parameters, lints `main` directory for `.h` and `.ino` files only.
style github_workflows stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:1.8px,fill:#fbfbfc
style main stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fcfdff
style tasks stroke:#6A7BD8,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fcfdff
style ci_scripts stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:1.8px,fill:#fffaf5
style bash stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fffaf5
style sub_scripts stroke:#FF9A3C,stroke-dasharray:6 4,stroke-width:0.6px,fill:#fffaf5
### 10. `task-security-scan.yml` - Reusable Security Scan Workflow
style triggers fill:none,stroke:none
**Purpose**: Parameterized security vulnerability scanning and SBOM generation logic.
**Trigger**: `workflow_call` only (called by other workflows)
**Parameters**:
- `scan-type`: Type of scan: 'fs' (filesystem), 'config' (configuration), or 'image' (container) (default: 'fs')
- `severity`: Severity levels to report (comma-separated: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL) (default: 'HIGH,CRITICAL')
- `scan-path`: Path to scan (default: '.')
- `exit-code`: Exit code when vulnerabilities found (0=continue, 1=fail) (default: '0')
- `upload-to-security-tab`: Upload SARIF to GitHub Security tab (default: true)
- `generate-sbom`: Generate SBOM artifacts (default: true)
**What it does**:
1. **Install Trivy**: Retrieves and installs Trivy vulnerability scanner
2. **Run security scan**: Calls unified `ci.sh security` with parameters:
- `--scan-type <fs|config|image>`: Target type
- `--severity <levels>`: Filter by severity
- `--scan-path <path>`: Directory to scan
- `--generate-sbom`: Generate SBOM in CycloneDX and SPDX formats
- `--exit-code <0|1>`: Fail behavior on critical vulnerabilities
- `--upload-to-security-tab`: Upload SARIF to GitHub
3. **Upload artifacts**: Stores reports and SBOM for later download
4. **GitHub Security integration**: SARIF automatically appears in Security tab
**Command Flow**:
```bash
./scripts/ci.sh security --scan-type fs --severity HIGH,CRITICAL --generate-sbom --upload-to-security-tab
↓
└─→ ci_security.sh (security orchestrator)
└─→ Trivy (vulnerability scanner)
├─→ Generate SARIF, JSON, summary
├─→ Generate SBOM (CycloneDX, SPDX)
└─→ Upload to GitHub Security tab
```
### Workflow Relationships
**Technical Details**:
- Runs on: Ubuntu latest
- Scanner: Trivy (latest version auto-installed)
- Report formats: SARIF (GitHub integration), JSON (detailed), Markdown (summary)
- SBOM formats: CycloneDX and SPDX (industry standards)
- GitHub Security tab: Auto-uploads SARIF for code scanning dashboard
- Strategy: Single sequential job (not parallelized)
- Artifact retention: As configured by caller
**Main → Task Mapping**:
- `build.yml` → calls `task-build.yml` (also contains inline documentation job)
- `lint.yml` → calls `task-lint.yml`
- `build_and_docs_to_dev.yml` → calls `task-build.yml` + `task-docs.yml` (with prepare/deploy jobs)
- `release.yml` → calls `task-build.yml` + `task-docs.yml` (with prepare/deploy jobs)
- `manual_docs.yml` → calls `task-docs.yml`
- `stale.yml` → standalone (no dependencies)
**Output Files**:
- `generated/reports/trivy-results.sarif` - SARIF format (GitHub Security tab upload)
- `generated/reports/trivy-results.json` - JSON format (detailed results)
- `generated/reports/security-summary.md` - Human-readable summary
- `generated/reports/sbom/sbom.cyclonedx.json` - CycloneDX SBOM
- `generated/reports/sbom/sbom.spdx.json` - SPDX SBOM
**Task → CI Script Mapping**:
- `task-build.yml` → `ci.sh build <env> --version --mode --deploy-ready`
- Routes to: `ci_build.sh` → `ci_set_version.sh`, `ci_build_firmware.sh`, `ci_prepare_artifacts.sh`
- Output: `generated/artifacts/` (default, can be overridden with `--output`)
- `task-docs.yml` → `ci.sh site --mode --version-source --url-prefix --webuploader-args`
- Routes to: `ci_site.sh` → `generate_board_docs.py`, `gen_wu.py`, VuePress
- Output: `generated/site/`
- `task-lint.yml` → `ci.sh qa --check --source --extensions --clang-format-version`
- Routes to: `ci_qa.sh` → `clang-format`
**Job Dependencies**:
- `build_and_docs_to_dev.yml`: prepare → build (task) → deploy & documentation (task)
- `release.yml`: prepare → build (task) → deploy → documentation (task)
**Script Execution Flow**:
```
GitHub Action (task-*.yml)
↓
./scripts/ci.sh <command> [OPTIONS] ← Main dispatcher
↓
./scripts/ci_<command>.sh ← Command orchestrator
↓
./scripts/ci_*.sh / *.py ← Worker scripts
```
**Callers**:
- `security-scan.yml` (weekly + manual scanning)
- Can be called by other workflows for custom security workflows
---
@@ -562,25 +663,17 @@ All build environments are defined in `.github/workflows/environments.json`:
```json
{
"environments": {
"all": [ ...83 environments ],
"metadata": {
"totalCount": 83,
"lastUpdated": "2024-12-25",
"categories": {
"esp32": 50,
"esp8266": 20,
"specialized": 13
}
"all": [ ...all environments ],
"ci": [...a subset of environments]
}
}
}
```
**Benefits**:
- ✅ Single source of truth
- ✅ Eliminates duplication across workflows
- ✅ Easier to maintain and update
- ✅ Consistent builds across CI/dev/release
**Environment Sets**:
- `all`: Complete production set for releases and comprehensive validation
- `ci`: Representative subset for fast CI feedback
### Environment Categories
@@ -653,24 +746,33 @@ All build environments are defined in `.github/workflows/environments.json`:
- PlatformIO 6.1.18 from custom fork: `pioarduino/platformio-core`
- Python package manager: `uv` for fast dependency installation
- Orchestrator: `ci_build.sh`
- Worker: `ci_set_version.sh` (version injection)
- Worker: `ci_build_firmware.sh` (PlatformIO compilation)
- Worker: `ci_prepare_artifacts.sh` (artifact packaging)
**Documentation System** (`ci.sh site`):
- Documentation framework: VuePress
- Orchestrator: `ci_site.sh`
- Worker: `generate_board_docs.py` (auto-generate board pages)
- Worker: `gen_wu.py` (WebUpdater manifest)
- Worker: `generate_board_docs` (npm package - auto-generate board pages)
- Worker: `gen_wu` (npm package - WebUpdater manifest)
- External: Common config from theengs.io
**Code Quality** (`ci.sh qa`):
- Orchestrator: `ci_qa.sh`
- Worker: `clang-format` version 9
- Worker: `shellcheck` for shell scripts
- Default scope: `main` directory, `.h` and `.ino` files
**Security Scanning System** (`ci.sh security`):
- Vulnerability scanner: Trivy (container vulnerability database)
- Orchestrator: `ci_security.sh`
- Worker: Trivy (filesystem, configuration, and container image scanning)
- Report formats: SARIF (GitHub Security tab), JSON (detailed), Markdown (summary)
- SBOM generation: CycloneDX and SPDX formats
- GitHub Security integration: Auto-uploads SARIF for code scanning dashboard
- Output: `generated/reports/` (security findings and artifacts)
**Configuration**:
- Environment list: `.github/workflows/environments.json` (83 environments)
- Environment list: `.github/workflows/environments.json`
- Task workflows: `task-*.yml` (reusable GitHub Actions components)
- Repository owner restriction: Hardcoded to `1technophile` for dev deployments
- All scripts located in: `./scripts/`
@@ -681,17 +783,20 @@ All build environments are defined in `.github/workflows/environments.json`:
./scripts/ci.sh build esp32dev-ble --mode dev --version test
# Build documentation locally
./scripts/ci.sh site --mode dev --preview
./scripts/ci.sh site --mode dev --version test
# Check code format
./scripts/ci.sh qa --check
./scripts/ci.sh qa --check --source main --extensions h,ino --clang-format-version 9
# Security scanning
./scripts/ci.sh security --scan-type fs --severity HIGH,CRITICAL --generate-sbom
# Run complete pipeline
./scripts/ci.sh all esp32dev-ble --version v1.8.0
./scripts/ci.sh all --mode dev
```
---
**Document Version**: 2.1
**Last Updated**: December 29, 2025
**Document Version**: 2.3
**Last Updated**: 01/14/2026
**Maintainer**: OpenMQTTGateway Development Team
+34 -17
View File
@@ -1,30 +1,47 @@
name: Build
# See details in .github/workflows/README.md (workflow docs)
run-name: "CI build for ${{ github.ref_name }}"
on: [push, pull_request]
jobs:
determine-build-scope:
runs-on: ubuntu-latest
outputs:
environment-set: ${{ steps.decide.outputs.environment-set }}
steps:
- name: Decide environment set
id: decide
run: |
# Full build for: development, master, edge, stable, release/*, hotfix/*, and all PRs
if [[ "${{ github.event_name }}" == "pull_request" ]] || \
[[ "${{ github.ref }}" == "refs/heads/development" ]] || \
[[ "${{ github.ref }}" == "refs/heads/master" ]] || \
[[ "${{ github.ref }}" == "refs/heads/edge" ]] || \
[[ "${{ github.ref }}" == "refs/heads/stable" ]] || \
[[ "${{ github.ref }}" == refs/heads/release/* ]] || \
[[ "${{ github.ref }}" == refs/heads/hotfix/* ]]; then
echo "environment-set=all" >> $GITHUB_OUTPUT
else
echo "environment-set=ci" >> $GITHUB_OUTPUT
fi
build:
needs: determine-build-scope
name: Build firmware
uses: ./.github/workflows/task-build.yml
with:
python-version: '3.13'
enable-dev-ota: false
enable-dev-ota: true
artifact-retention-days: 7
prepare-for-deploy: false
environment-set: ${{ needs.determine-build-scope.outputs.environment-set }}
documentation:
needs: build
name: Build documentation
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: "14.x"
- name: Download Common Config
run: |
curl -o docs/.vuepress/public/commonConfig.js https://www.theengs.io/commonConfig.js
- name: Install build dependencies
run: npm install
- name: Build documentation
run: npm run docs:build
uses: ./.github/workflows/task-docs.yml
with:
mode: "dev"
run-pagespeed: false
just-check: true
+11 -71
View File
@@ -3,7 +3,7 @@ name: Build binaries, docs and publish to dev folder
on:
workflow_dispatch:
schedule:
- cron: '0 0 * * *'
- cron: "0 0 * * *"
jobs:
prepare:
@@ -18,86 +18,26 @@ jobs:
with:
length: 6
build:
handle-firmwares:
needs: prepare
name: Build development firmware
name: Build and deploy development firmwares artefacts
uses: ./.github/workflows/task-build.yml
with:
python-version: '3.13'
enable-dev-ota: true
version-tag: ${{ needs.prepare.outputs.short-sha }}
artifact-retention-days: 1
artifact-name-prefix: 'firmware-'
prepare-for-deploy: true
deploy:
needs: [prepare, build]
runs-on: ubuntu-latest
if: github.repository_owner == '1technophile'
name: Deploy binaries and docs
steps:
- uses: actions/checkout@v4
- name: Download all firmware artifacts
uses: actions/download-artifact@v4
with:
pattern: firmware-*
path: toDeploy
merge-multiple: true
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
version: "latest"
enable-cache: false
- name: Install dependencies
run: |
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
- name: Create library zips
run: |
# Install libraries for a representative environment to get libdeps
platformio pkg install -e esp32dev-ble --no-save
cd .pio/libdeps
# Replace spaces with underscores in folder names
find . -type d -name "* *" | while read FNAME; do mv "$FNAME" "${FNAME// /_}"; done
# Zip libraries per board
for i in */; do
zip -r "${i%/}-libraries.zip" "$i"
done
mv *.zip ../../toDeploy/
- name: Prepare additional assets
run: |
cd toDeploy
# Remove binaries for *-all*, *-test* env
rm -f *-all*.bin *-test*.bin *-test*.zip || true
cd ..
# Zip source code
zip -r toDeploy/OpenMQTTGateway_sources.zip main LICENSE.txt
ls -lA toDeploy/
documentation:
needs: [prepare, build]
handle-documentation:
needs: [prepare, handle-firmwares]
name: Build and deploy development documentation
uses: ./.github/workflows/task-docs.yml
with:
python-version: '3.11'
node-version: '16.x'
version-source: 'custom'
custom-version: 'DEVELOPMENT SHA:${{ needs.prepare.outputs.short-sha }} TEST ONLY'
base-path: '/dev/'
destination-dir: 'dev'
generate-webuploader: true
webuploader-args: '--dev'
mode: "dev"
version: ${{ needs.prepare.outputs.short-sha }}
url-prefix: "/dev/"
destination-dir: "dev"
run-pagespeed: true
pagespeed-url: 'https://docs.openmqttgateway.com/dev/'
secrets:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
APIKEY: ${{ secrets.APIKEY }}
github-token: ${{ secrets.GITHUB_TOKEN }}
pagespeed-apikey: ${{ secrets.APIKEY }}
+17 -14
View File
@@ -88,19 +88,22 @@
"sonoff-basic",
"sonoff-basic-rfr3"
],
"metadata": {
"totalCount": 83,
"categories": {
"esp32": 42,
"esp8266": 17,
"specialized": 24
},
"notes": [
"This unified list contains all environments from both build.yml and build_and_docs_to_dev.yml",
"Previously build.yml had 85 environments and build_and_docs_to_dev.yml had 88",
"After deduplication and proper ordering, the total is 83 unique environments",
"Environments ending in -test or -all-test are typically excluded from production releases"
]
}
"ci": [
"esp32dev-all-test",
"esp32dev-ble",
"esp32dev-ir",
"esp32dev-rf",
"esp32dev-rtl_433",
"esp32s3-dev-c1-ble",
"esp32c3-dev-m1-ble",
"heltec-ble",
"lilygo-ble",
"nodemcuv2-all-test",
"nodemcuv2-ir",
"nodemcuv2-rf",
"theengs-bridge",
"theengs-plug",
"rfbridge"
]
}
}
-12
View File
@@ -1,12 +0,0 @@
name: Check Code Format
on: [push, pull_request]
jobs:
lint:
name: Lint code format
uses: ./.github/workflows/task-lint.yml
with:
source: 'main'
extensions: 'h,ino'
clang-format-version: '9'
+5 -9
View File
@@ -9,13 +9,9 @@ jobs:
name: Build and deploy production documentation
uses: ./.github/workflows/task-docs.yml
with:
python-version: '3.11'
node-version: '14.x'
version-source: 'release'
base-path: '/'
destination-dir: '.'
generate-webuploader: true
webuploader-args: '${GITHUB_REPOSITORY}'
run-pagespeed: false
mode: "prod"
version: "auto" # Will detect latest git tag
url-prefix: "/"
destination-dir: "."
secrets:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
github-token: ${{ secrets.GITHUB_TOKEN }}
+9 -50
View File
@@ -33,63 +33,26 @@ jobs:
name: Build release firmware
uses: ./.github/workflows/task-build.yml
with:
python-version: '3.13'
enable-dev-ota: false
version-tag: ${{ needs.prepare.outputs.version-tag }}
artifact-retention-days: 90
artifact-name-prefix: 'firmware-'
prepare-for-deploy: true
deploy:
needs: [prepare, build]
runs-on: ubuntu-latest
name: Deploy release assets
steps:
- uses: actions/checkout@v4
- name: Download all firmware artifacts
uses: actions/download-artifact@v4
with:
pattern: firmware-*
path: .pio/build
merge-multiple: false
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
version: "latest"
enable-cache: false
- name: Install platformio
run: |
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
- name: Reorganize artifacts for prepare_deploy.sh
run: |
# Move artifacts from download structure to expected .pio/build structure
for env_dir in .pio/build/firmware-*; do
if [ -d "$env_dir" ]; then
env_name=$(basename "$env_dir" | sed 's/^firmware-//')
mkdir -p ".pio/build/${env_name}"
mv "$env_dir"/*.bin ".pio/build/${env_name}/" 2>/dev/null || true
rm -rf "$env_dir"
fi
done
- name: Prepare Release Assets
run: |
sudo apt install rename
./scripts/prepare_deploy.sh
pattern: "*"
path: generated/artifacts
merge-multiple: true
- name: Upload Release Assets
uses: bgpat/release-asset-action@03b0c30db1c4031ce3474740b0e4275cd7e126a3
with:
pattern: "toDeploy/*"
pattern: "generated/artifacts/*"
github-token: ${{ secrets.GITHUB_TOKEN }}
release-url: ${{ needs.prepare.outputs.upload-url }}
allow-overwrite: true
@@ -99,13 +62,9 @@ jobs:
name: Build and deploy release documentation
uses: ./.github/workflows/task-docs.yml
with:
python-version: '3.11'
node-version: '18.x'
version-source: 'git-tag'
base-path: '/'
destination-dir: '.'
generate-webuploader: true
run-pagespeed: false
mode: "prod"
version: ${{ needs.prepare.outputs.version-tag }}
url-prefix: "/"
destination-dir: "."
secrets:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
APIKEY: ${{ secrets.APIKEY }}
github-token: ${{ secrets.GITHUB_TOKEN }}
+44
View File
@@ -0,0 +1,44 @@
name: Security Scan
on:
schedule:
- cron: '0 2 * * 1' # Every Monday at 2 AM
workflow_dispatch:
inputs:
severity:
description: 'Severity levels to scan'
required: false
type: choice
default: 'HIGH,CRITICAL'
options:
- 'CRITICAL'
- 'HIGH,CRITICAL'
- 'MEDIUM,HIGH,CRITICAL'
- 'LOW,MEDIUM,HIGH,CRITICAL'
scan-path:
description: 'Path to scan (default: entire repo)'
required: false
type: string
default: '.'
exit-on-error:
description: 'Fail if vulnerabilities found'
required: false
type: boolean
default: false
pull_request:
paths:
- 'main/**'
- 'lib/**'
- 'platformio.ini'
- '.github/workflows/**'
jobs:
security-scan:
name: Scan for vulnerabilities
uses: ./.github/workflows/task-security-scan.yml
with:
scan-type: 'fs'
severity: ${{ inputs.severity || 'HIGH,CRITICAL' }}
exit-code: ${{ inputs.exit-on-error && '1' || '0' }}
upload-to-security-tab: true
scan-path: ${{ inputs.scan-path || '.' }}
+46 -30
View File
@@ -4,37 +4,53 @@ on:
workflow_call:
inputs:
python-version:
description: 'Python version to use'
description: "Python version to use"
required: false
type: string
default: '3.13'
default: "3.13"
pio-version:
description: "PlatformIO version to use"
required: false
type: string
default: "v6.1.18"
environment-set:
description: 'Which set of environments to build (all, ci)'
required: false
type: string
default: 'all'
enable-dev-ota:
description: 'Enable development OTA builds'
description: "Enable development OTA builds"
required: false
type: boolean
default: false
version-tag:
description: 'Version tag to inject into firmware (e.g., SHA or release tag)'
description: "Optional version tag to pass to ci.sh build (omitted if empty)"
required: false
type: string
default: 'unspecified'
default: ""
artifact-retention-days:
description: 'Number of days to retain build artifacts'
description: "Number of days to retain build artifacts"
required: false
type: number
default: 7
artifact-name-prefix:
description: 'Prefix for artifact names (e.g., "firmware-" for dev builds)'
required: false
type: string
default: ''
prepare-for-deploy:
description: 'Prepare firmware files for deployment with specific naming'
required: false
type: boolean
default: false
jobs:
lint:
name: Lint code format
uses: ./.github/workflows/task-lint.yml
with:
source: "main"
extensions: "h,ino,cpp"
clang-format-version: "9"
security-scan:
name: Security vulnerability scan
uses: ./.github/workflows/task-security-scan.yml
with:
severity: "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL"
exit-code: "1"
upload-to-security-tab: true
load-environments:
runs-on: ubuntu-latest
outputs:
@@ -43,11 +59,17 @@ jobs:
- uses: actions/checkout@v4
- id: set-matrix
run: |
ENVIRONMENTS=$(jq -c '.environments.all' .github/workflows/environments.json)
ENV_SET="${{ inputs.environment-set }}"
if [ "$ENV_SET" = "ci" ]; then
ENVIRONMENTS=$(jq -c '.environments.ci' .github/workflows/environments.json)
else
ENVIRONMENTS=$(jq -c '.environments.all' .github/workflows/environments.json)
fi
echo "matrix=${ENVIRONMENTS}" >> $GITHUB_OUTPUT
build:
needs: load-environments
needs: [lint, security-scan, load-environments]
runs-on: ubuntu-latest
strategy:
fail-fast: false
@@ -70,35 +92,29 @@ jobs:
- name: Install PlatformIO dependencies
run: |
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/${{ inputs.pio-version }}.zip
- name: Build firmware using ci.sh
run: |
BUILD_ARGS="${{ matrix.environments }}"
BUILD_ARGS="$BUILD_ARGS --deploy-ready"
# Add version tag if specified
if [ "${{ inputs.version-tag }}" != "unspecified" ]; then
# Optional version tag
if [ -n "${{ inputs.version-tag }}" ]; then
BUILD_ARGS="$BUILD_ARGS --version ${{ inputs.version-tag }}"
fi
# Add mode flag (dev/prod)
# Mode
if [ "${{ inputs.enable-dev-ota }}" = "true" ]; then
BUILD_ARGS="$BUILD_ARGS --mode dev"
else
BUILD_ARGS="$BUILD_ARGS --mode prod"
fi
# Add deploy flag if needed
if [ "${{ inputs.prepare-for-deploy }}" = "true" ]; then
BUILD_ARGS="$BUILD_ARGS --deploy-ready"
fi
# Execute build (uses default generated/artifacts/ directory)
./scripts/ci.sh build $BUILD_ARGS
- name: Upload firmware artifacts
uses: actions/upload-artifact@v4
with:
name: ${{ inputs.artifact-name-prefix }}${{ matrix.environments }}
name: ${{ matrix.environments }}
path: generated/artifacts/
retention-days: ${{ inputs.artifact-retention-days }}
+90 -71
View File
@@ -3,61 +3,59 @@ name: Reusable Documentation Workflow
on:
workflow_call:
inputs:
python-version:
description: 'Python version to use'
mode:
description: 'Build mode: "prod" or "dev"'
required: false
type: string
default: '3.11'
node-version:
description: 'Node.js version to use'
default: "prod"
version:
description: 'Version string for documentation (use "auto" for automatic detection)'
required: false
type: string
default: '14.x'
version-source:
description: 'Source of version: "release" (from GitHub release) or "custom" (provided)'
required: false
type: string
default: 'release'
custom-version:
description: 'Custom version string when version-source is "custom"'
required: false
type: string
default: ''
base-path:
description: 'Base path for documentation (e.g., "/" for prod, "/dev/" for dev)'
required: false
type: string
default: '/'
destination-dir:
description: 'Destination directory for GitHub Pages deployment'
required: false
type: string
default: '.'
generate-webuploader:
description: 'Generate WebUploader manifest'
default: "auto"
just-check:
description: "Just create a quick build to check for errors"
required: false
type: boolean
default: true
webuploader-args:
description: 'Additional arguments for gen_wu.py script'
default: false
url-prefix:
description: 'URL prefix for docs (e.g., "/" for prod, "/dev/" for dev)'
required: false
type: string
default: ''
default: "/"
run-pagespeed:
description: 'Run PageSpeed Insights after deployment'
description: "Run PageSpeed Insights after deploy"
required: false
type: boolean
default: false
pagespeed-url:
description: 'URL to test with PageSpeed Insights'
description: "URL to test with PageSpeed"
required: false
type: string
default: 'https://docs.openmqttgateway.com/'
secrets:
GITHUB_TOKEN:
required: true
APIKEY:
default: "https://docs.openmqttgateway.com/"
destination-dir:
description: "GitHub Pages destination directory"
required: false
type: string
default: "."
python-version:
description: "Python version to use"
required: false
type: string
default: "3.13"
pio-version:
description: "PlatformIO version to use"
required: false
type: string
default: "v6.1.18"
secrets:
github-token:
description: "GitHub token for deploying to GitHub Pages"
required: false
pagespeed-apikey:
description: "API key for PageSpeed Insights"
required: false
jobs:
documentation:
@@ -65,63 +63,84 @@ jobs:
name: Create and deploy documentation
steps:
- uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ inputs.node-version }}
fetch-depth: 0 # Fetch all history for git tags
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ inputs.python-version }}
- name: Build documentation site
run: |
# Build script arguments based on inputs
ARGS=""
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
version: "latest"
enable-cache: false
# Set version source and custom version
if [ "${{ inputs.version-source }}" = "custom" ] && [ -n "${{ inputs.custom-version }}" ]; then
ARGS="$ARGS --custom-version '${{ inputs.custom-version }}' --version-source custom"
elif [ "${{ inputs.version-source }}" = "release" ]; then
# Get latest release tag
if command -v git >/dev/null 2>&1; then
LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
ARGS="$ARGS --custom-version '${LATEST_TAG}' --version-source release"
- name: Install PlatformIO dependencies
run: |
uv pip install --system -U "https://github.com/pioarduino/platformio-core/archive/refs/tags/${{ inputs.pio-version }}.zip"
- name: Determine version
id: version
run: |
VERSION="${{ inputs.version }}"
MODE="${{ inputs.mode }}"
# If version is 'auto', determine it based on mode
if [ "$VERSION" = "auto" ]; then
if [ "$MODE" = "dev" ]; then
# Dev mode: always use short SHA
VERSION="$(git rev-parse --short HEAD)"
echo "Dev mode - using SHA: $VERSION"
else
ARGS="$ARGS --version-source release"
# Prod mode: try to get git tag
if GIT_TAG=$(git describe --tags --exact-match 2>/dev/null); then
VERSION="$GIT_TAG"
echo "Using exact git tag: $VERSION"
elif LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null); then
VERSION="$LATEST_TAG"
echo "Using latest tag: $VERSION"
else
VERSION="v0.0.0-unknown"
echo "Warning: No tag found, using: $VERSION"
fi
fi
fi
# Set base path/URL prefix
if [ "${{ inputs.base-path }}" != "/" ]; then
ARGS="$ARGS --url-prefix '${{ inputs.base-path }}'"
fi
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "Final version: $VERSION"
# WebUploader generation
if [ "${{ inputs.generate-webuploader }}" != "true" ]; then
ARGS="$ARGS --no-webuploader"
elif [ -n "${{ inputs.webuploader-args }}" ]; then
ARGS="$ARGS --webuploader-args '${{ inputs.webuploader-args }}'"
fi
- name: Download all firmware artifacts
if: ${{ inputs.mode == 'dev' }}
uses: actions/download-artifact@v4
with:
pattern: "*"
path: generated/artifacts
merge-multiple: true
# Run the build script
eval "./scripts/ci_site.sh ${ARGS}"
- name: Build documentation site
run: |
./scripts/ci.sh site \
--clean \
--mode ${{ inputs.mode }} \
--version ${{ steps.version.outputs.version }} \
--url-prefix ${{ inputs.url-prefix }}
- name: Deploy to GitHub Pages
uses: peaceiris/actions-gh-pages@v3
if: ${{ inputs.just-check == false }}
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
github_token: ${{ secrets.github-token }}
publish_dir: ./generated/site
destination_dir: ${{ inputs.destination-dir }}
cname: docs.openmqttgateway.com
- name: Run PageSpeed Insights
if: inputs.run-pagespeed
if: ${{ inputs.just-check == false && inputs.run-pagespeed == true }}
uses: jakepartusch/psi-action@v1.3
id: psi
with:
url: ${{ inputs.pagespeed-url }}
threshold: 60
key: ${{ secrets.APIKEY }}
key: ${{ secrets.pagespeed-apikey }}
+8 -8
View File
@@ -4,20 +4,20 @@ on:
workflow_call:
inputs:
source:
description: 'Source directory to lint (single directory)'
description: "Source directory to lint"
required: false
type: string
default: 'main'
default: "main"
extensions:
description: 'File extensions to check (comma-separated)'
description: "File extensions to check (comma-separated)"
required: false
type: string
default: 'h,ino,cpp'
default: "h,ino,cpp"
clang-format-version:
description: 'clang-format version to use'
description: "clang-format version to use"
required: false
type: string
default: '9'
default: "9"
jobs:
lint:
@@ -25,11 +25,10 @@ jobs:
name: Check code format
steps:
- uses: actions/checkout@v4
- name: Install clang-format
run: |
sudo apt-get update
sudo apt-get install -y clang-format
sudo apt-get install -y clang-format shellcheck
- name: Check code format with ci.sh qa
run: |
@@ -38,3 +37,4 @@ jobs:
--source "${{ inputs.source }}" \
--extensions "${{ inputs.extensions }}" \
--clang-format-version "${{ inputs.clang-format-version }}"
+124
View File
@@ -0,0 +1,124 @@
name: Reusable Security Scan Workflow
on:
workflow_call:
inputs:
scan-type:
description: 'Type of scan: fs (filesystem), config, or image'
required: false
type: string
default: 'fs'
severity:
description: 'Severity levels to report (comma-separated: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL)'
required: false
type: string
default: 'HIGH,CRITICAL'
exit-code:
description: 'Exit code when vulnerabilities found (0=continue, 1=fail)'
required: false
type: string
default: '0'
upload-to-security-tab:
description: 'Upload SARIF to GitHub Security tab (GitHub-specific feature)'
required: false
type: boolean
default: true
scan-path:
description: 'Path to scan (default: entire repository)'
required: false
type: string
default: '.'
generate-sbom:
description: 'Generate Software Bill of Materials (SBOM) in CycloneDX and SPDX formats'
required: false
type: boolean
default: true
jobs:
security-scan:
runs-on: ubuntu-latest
name: Security vulnerability scan
permissions:
contents: read
security-events: write # Required only for GitHub Security tab upload
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Install Trivy
run: |
set -euo pipefail
curl -fsSL https://aquasecurity.github.io/trivy-repo/deb/public.key \
| gpg --dearmor \
| sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" \
| sudo tee /etc/apt/sources.list.d/trivy.list > /dev/null
sudo apt-get update
sudo apt-get install -y trivy
- name: Run Security Scan using ci_security.sh
run: |
set -euo pipefail
sbom_flag=""
if [[ "${{ inputs.generate-sbom }}" == "true" ]]; then
sbom_flag="--generate-sbom"
fi
./scripts/ci_security.sh \
--scan-type "${{ inputs.scan-type }}" \
--scan-path "${{ inputs.scan-path }}" \
--severity "${{ inputs.severity }}" \
--exit-code "${{ inputs.exit-code }}" \
${sbom_flag}
# GitHub-specific: Upload to Security tab
- name: Upload SARIF to GitHub Security tab
if: inputs.upload-to-security-tab == true && always()
uses: github/codeql-action/upload-sarif@v4
with:
sarif_file: './generated/reports/trivy-results.sarif'
category: 'trivy-security-scan'
# Agnostic: Upload all reports as artifacts (works on any CI)
- name: Upload security reports as artifacts
uses: actions/upload-artifact@v4
if: always()
with:
name: security-scan-reports
path: |
./generated/reports/trivy-results.sarif
./generated/reports/trivy-report.json
./generated/reports/trivy-report.txt
./generated/reports/security-summary.md
./generated/reports/trivy-scan.log
retention-days: 90
# Display summary in workflow output
- name: Display Security Summary
if: always()
run: |
cat ./generated/reports/security-summary.md >> $GITHUB_STEP_SUMMARY 2>/dev/null || echo "Summary not available"
#
- name: Upload SBOM reports
uses: actions/upload-artifact@v4
if: always()
with:
name: sbom-reports
path: |
./generated/reports/sbom/
retention-days: 90
# Optional: Fail the build if critical vulnerabilities found
- name: Check for critical vulnerabilities
if: inputs.exit-code == '1'
run: |
if [ -f ./generated/reports/trivy-results.sarif ]; then
CRITICAL=$(jq '[.runs[].results[] | select(.level == "error")] | length' ./generated/reports/trivy-results.sarif 2>/dev/null || echo "0")
if [ "$CRITICAL" -gt 0 ]; then
echo "❌ Found ${CRITICAL} critical vulnerabilities!"
echo "Review the security reports in artifacts."
exit 1
fi
fi
+18 -5
View File
@@ -25,11 +25,24 @@ managed_components
.github/workflows/_docs
# CI/CD Generated outputs
docs/.vuepress/dist/
docs/.vuepress/components/
docs/.vuepress/public/
^docs/.vuepress/public/img/
^docs/.vuepress/public/*.png
^docs/.vuepress/public/*.txt
docs/.vuepress/public/boards-info.json
docs/.vuepress/public/commonConfig.js
docs/.vuepress/public/firmware_build
# Generated site/firmware/reports files
generated/
# Generated ssl for testing
.ssl
# Generated JSON files from templates
scripts/latest_version.json
scripts/latest_version_dev.json
# Generated configuration files
docs/.vuepress/meta.json
# Generated full documentation pages
docs/prerequisites/board-full.md
docs/upload/web-install-full.md
+1
View File
@@ -0,0 +1 @@
disable=SC1091
@@ -0,0 +1,440 @@
<template>
<div class="board-environment-list">
<div v-if="loading" class="custom-block tip">
<p class="custom-block-title">
<span class="loading-spinner" aria-label="Loading"></span>
Loading supported environments...
</p>
</div>
<div v-if="error" class="custom-block danger">
<p class="custom-block-title">Error</p>
<p>{{ error }}</p>
</div>
<div v-if="!loading && !error" class="boards-grid">
<article
v-for="board in boards"
:key="board.environment"
class="board-card"
@click="openSelector(board.environment)">
<div class="board-card__image">
<img
:src="getBoardImageUrl(board)"
:alt="board.environment"
loading="lazy">
</div>
<div class="board-card__content">
<div class="board-card__header">
<h3 class="board-card__title">
<code>{{ board.environment }}</code>
</h3>
<span v-if="board.microcontroller" class="board-card__chip">
{{ board.microcontroller }}
</span>
</div>
<p v-if="board.description" class="board-card__description" v-html="board.description">
</p>
<!-- Modules Section with expand/collapse -->
<div v-if="Array.isArray(board.modules) && board.modules.length" class="board-card__libraries">
<div class="modules-label-row">
<span class="libraries-label">Modules:</span>
<span
v-if="board.modules.length > 2"
class="expand-icon-btn"
@click.stop="toggleModules(board.environment)"
:title="expandedModules[board.environment] ? 'Hide' : 'Show all'">
<svg :class="{rotated: expandedModules[board.environment]}" width="19" height="19" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><polyline points="9 6 15 12 9 18"/></svg>
</span>
</div>
<transition name="expand-fade">
<div class="libraries-badges"
:style="expandedModules[board.environment] ? '' : 'max-height: 2.2em; overflow: hidden;'">
<span
v-for="(mod, index) in board.modules"
:key="index"
class="lib-badge">
{{ mod }}
</span>
</div>
</transition>
</div>
<!-- Libraries Section with expand/collapse -->
<div v-if="Array.isArray(board.libraries) && board.libraries.length" class="board-card__libraries">
<div class="modules-label-row">
<span class="libraries-label">Libraries:</span>
<span
v-if="board.libraries.length > 2"
class="expand-icon-btn"
@click.stop="toggleLibraries(board.environment)"
:title="expandedLibraries[board.environment] ? 'Hide' : 'Show all'">
<svg :class="{rotated: expandedLibraries[board.environment]}" width="19" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><polyline points="9 6 15 12 9 18"/></svg>
</span>
</div>
<transition name="expand-fade">
<div class="libraries-badges"
:style="expandedLibraries[board.environment] ? '' : 'max-height: 2.2em; overflow: hidden;'">
<span
v-for="(lib, index) in board.libraries"
:key="index"
class="lib-badge">
{{ lib }}
</span>
</div>
</transition>
</div>
<div class="board-card__action">
<span class="action-text">More info →</span>
</div>
</div>
</article>
</div>
</div>
</template>
<script>
export default {
name: 'BoardEnvironmentTable',
props: {
boardsUrl: {
type: String,
default: '/boards-info.json'
},
selectorPath: {
type: String,
default: '/upload/board-selector.html'
}
},
data() {
return {
boards: [],
loading: false,
error: null,
expandedModules: {},
expandedLibraries: {}
};
},
computed: {
resolvedBoardsUrl() {
return this.buildUrl(this.boardsUrl);
},
resolvedSelectorUrl() {
return this.buildUrl(this.selectorPath);
}
},
methods: {
buildUrl(path) {
const base = this.$site?.base || '/';
const cleanBase = base.endsWith('/') ? base.slice(0, -1) : base;
const cleanPath = path.startsWith('/') ? path : `/${path}`;
return `${cleanBase}${cleanPath}`;
},
async loadBoards() {
this.loading = true;
this.error = null;
try {
const response = await fetch(this.resolvedBoardsUrl);
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
const data = await response.json();
if (!Array.isArray(data)) {
throw new Error('boards-info.json must be an array');
}
this.boards = data
.filter(board => board && typeof board.environment === 'string')
.sort((a, b) => a.environment.localeCompare(b.environment));
} catch (err) {
console.error('Failed to load boards-info:', err);
this.error = err.message || 'Unable to load board information';
} finally {
this.loading = false;
}
},
toggleModules(env) {
this.$set(this.expandedModules, env, !this.expandedModules[env]);
},
toggleLibraries(env) {
this.$set(this.expandedLibraries, env, !this.expandedLibraries[env]);
},
openSelector(environment) {
if (!environment) return;
const url = `${this.resolvedSelectorUrl}?env=${encodeURIComponent(environment)}`;
window.location.href = url;
},
getBoardImageUrl(board) {
const customImg = board.customImg || board.CustomImg;
if (customImg) {
// If it's an absolute URL, use it as-is
if (customImg.startsWith('http')) {
return customImg;
}
// Otherwise, build the URL with site base
return this.buildUrl(customImg);
}
return this.buildUrl('/img/microcontroller.gif');
}
},
mounted() {
this.loadBoards();
}
};
</script>
<style scoped>
.board-environment-list {
margin: 2rem 0;
}
/* Grid Layout */
.boards-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(320px, 1fr));
gap: 1.5rem;
margin-top: 1.5rem;
}
/* Card Design */
.board-card {
background: #ffffff;
border: 1px solid var(--border-color, #eaecef);
border-radius: 8px;
overflow: hidden;
cursor: pointer;
transition: all 0.3s ease;
display: flex;
flex-direction: column;
box-shadow: 0 2px 4px rgba(0, 0, 0, 0.05);
}
.board-card:hover {
transform: translateY(-4px);
box-shadow: 0 8px 16px rgba(0, 0, 0, 0.1);
border-color: var(--accent-color, #3eaf7c);
}
/* Image Section */
.board-card__image {
background: var(--code-bg-color, #f6f8fa);
padding: 1.5rem;
display: flex;
align-items: center;
justify-content: center;
min-height: 120px;
border-bottom: 1px solid var(--border-color, #eaecef);
}
.board-card__image img {
max-width: 100px;
max-height: 100px;
object-fit: contain;
}
/* Content Section */
.board-card__content {
padding: 1.25rem;
flex: 1;
display: flex;
flex-direction: column;
gap: 0.75rem;
}
/* Header */
.board-card__header {
display: flex;
justify-content: space-between;
align-items: flex-start;
gap: 0.75rem;
margin-bottom: 0.25rem;
}
.board-card__title {
margin: 0;
font-size: 1rem;
font-weight: 600;
line-height: 1.4;
flex: 1;
}
.board-card__title code {
background: transparent;
color: var(--accent-color, #3eaf7c);
font-size: 0.9em;
padding: 0;
font-weight: 600;
}
.board-card__chip {
background: var(--code-bg-color, #f6f8fa);
color: var(--text-color-secondary, #666);
padding: 0.3rem 0.8rem;
border-radius: 4px;
font-size: 0.75rem;
font-weight: 600;
white-space: nowrap;
flex-shrink: 0;
border: 1px solid var(--border-color, #eaecef);
}
/* Description */
.board-card__description {
margin: 0;
font-size: 0.9rem;
line-height: 1.5;
color: var(--text-color-secondary, #666);
}
/* Libraries Section */
.board-card__libraries {
margin-top: 0.5rem;
padding-top: 0.25rem;
border-top: 1px solid var(--border-color, #eaecef);
}
.modules-label-row {
display: flex;
align-items: center;
gap: 0.2em;
margin-bottom: 0.5rem;
}
.libraries-label {
font-size: 0.8rem;
font-weight: 600;
color: var(--text-color-secondary, #666);
display: inline-block;
}
.libraries-badges {
display: flex;
flex-wrap: wrap;
gap: 0.4rem;
}
.lib-badge {
display: inline-block;
background: var(--code-bg-color, #f6f8fa);
color: var(--text-color, #2c3e50);
padding: 0.2rem 0.6rem;
border-radius: 4px;
font-size: 0.75rem;
font-weight: 500;
border: 1px solid var(--border-color, #eaecef);
}
/* Action Footer */
.board-card__action {
padding-top: 0.75rem;
margin-top: 0.5rem;
border-top: 1px solid var(--border-color, #eaecef);
}
.action-text {
color: var(--accent-color, #3eaf7c);
font-size: 0.85rem;
font-weight: 600;
display: flex;
align-items: center;
gap: 0.25rem;
}
.board-card:hover .action-text {
text-decoration: underline;
}
/* Loading Spinner */
.loading-spinner {
display: inline-block;
width: 16px;
height: 16px;
min-width: 16px;
border: 2px solid #cfd8dc;
border-top-color: var(--accent-color, #3eaf7c);
border-radius: 50%;
animation: spin 0.8s linear infinite;
vertical-align: middle;
margin-right: 0.5rem;
}
@keyframes spin {
to {
transform: rotate(360deg);
}
}
/* Responsive */
@media (max-width: 768px) {
.boards-grid {
grid-template-columns: 1fr;
gap: 1rem;
}
.board-card__image {
min-height: 100px;
padding: 1rem;
}
.board-card__image img {
max-width: 80px;
max-height: 80px;
}
}
@media (max-width: 480px) {
.board-card__header {
flex-direction: column;
align-items: flex-start;
}
.board-card__chip {
align-self: flex-start;
}
}
/* Espansione animata per badge */
.expand-fade-enter-active, .expand-fade-leave-active {
transition: max-height 0.3s cubic-bezier(0.4, 0, 0.2, 1), opacity 0.3s;
}
.expand-fade-enter, .expand-fade-leave-to {
opacity: 0;
max-height: 0;
}
.expand-fade-enter-to, .expand-fade-leave {
opacity: 1;
max-height: 500px;
}
.expand-icon-btn {
display: inline-flex;
align-items: center;
justify-content: center;
background: none;
border: none;
color: var(--accent-color, #3eaf7c);
cursor: pointer;
margin-top: 0.3em;
margin-left: 0.2em;
padding: 0.1em 0.2em;
border-radius: 50%;
transition: background 0.2s;
}
.expand-icon-btn:hover {
background: rgba(62, 175, 124, 0.08);
}
.expand-icon-btn svg {
transition: transform 0.25s cubic-bezier(0.4,0,0.2,1);
vertical-align: middle;
}
.expand-icon-btn svg.rotated {
transform: rotate(90deg);
}
</style>
@@ -0,0 +1,686 @@
<template>
<div class="flash-selector">
<!-- Loading State -->
<div v-if="loading" class="custom-block tip">
<p class="custom-block-title">
<span class="loading-spinner"></span>
Loading environments...
</p>
</div>
<!-- Error State -->
<div v-if="error" class="custom-block danger">
<p class="custom-block-title">Error</p>
<p>{{ error }}</p>
</div>
<!-- Board Details (when selected) -->
<transition name="fade">
<div v-if="currentEnvironment && !loading" class="board-details">
<!-- Hero Section - Clean and Focused -->
<div class="board-hero">
<div class="board-hero__content">
<!-- Inline Board Switcher -->
<h2>
<div class="board-switcher">
<label for="envSelect" class="switcher-label">Environment:</label>
<div class="switcher-select-wrapper">
<select
id="envSelect"
v-model="selectedEnvironment"
class="switcher-select">
<option
v-for="env in environments"
:key="env.environment"
:value="env.environment">
{{ env.environment }}
</option>
</select>
<span class="switcher-icon">⚙️</span>
</div>
</div>
</h2>
<div v-if="currentEnvironment.microcontroller" class="board-hero__meta">
<span class="meta-label">MCU:</span>
<span class="meta-value">{{ currentEnvironment.microcontroller }}</span>
</div>
<p v-if="currentEnvironment.description" class="board-hero__description">
<span class="meta-label">Description:</span> <span v-html="currentEnvironment.description"></span>
</p>
<!-- Primary Action -->
<div class="hero-action">
<esp-web-install-button
v-if="manifestUrl"
:manifest="manifestUrl"
aria-label="Flash firmware to device">
</esp-web-install-button>
<button v-else class="connect-button connect-button--disabled" disabled>
CONNECT
</button>
</div>
</div>
<!-- Image Optional -->
<div v-if="boardImageUrl && !imageError" class="board-hero__image">
<img
:src="boardImageUrl"
:alt="currentEnvironment.environment"
loading="lazy"
@error="handleImageError">
</div>
</div>
<!-- Technical Details - Collapsible -->
<details class="tech-details" open>
<summary class="tech-details__summary">
<span class="tech-details__icon">📋</span>
<span class="tech-details__title">Technical Information</span>
<span class="tech-details__toggle">▼</span>
</summary>
<div class="tech-details__content">
<div class="tech-grid">
<!-- Libraries -->
<div v-if="currentEnvironment.libraries && currentEnvironment.libraries.length" class="tech-section">
<h4 class="tech-section__title">Libraries</h4>
<div class="tech-section__content">
<div class="chip-group">
<span v-for="(lib, index) in currentEnvironment.libraries" :key="index" class="chip">
{{ lib }}
</span>
</div>
</div>
</div>
<!-- Gateway Modules -->
<div v-if="currentEnvironment.modules && currentEnvironment.modules.length" class="tech-section">
<h4 class="tech-section__title">Gateway Modules</h4>
<div class="tech-section__content">
<div class="chip-group">
<span v-for="(mod, index) in currentEnvironment.modules" :key="index" class="chip">
{{ mod }}
</span>
</div>
</div>
</div>
<!-- Hardware -->
<div v-if="currentEnvironment.hardware" class="tech-section">
<h4 class="tech-section__title">Hardware</h4>
<div class="tech-section__content" >
<span class="chip" v-html="currentEnvironment.hardware"></span>
</div>
</div>
<!-- Partitions -->
<div v-if="currentEnvironment.partitions" class="tech-section">
<h4 class="tech-section__title">Partitions</h4>
<div class="tech-section__content">
<code class="code-inline">{{ currentEnvironment.partitions }}</code>
</div>
</div>
<!-- Build Options -->
<div v-if="currentEnvironment.options" class="tech-section tech-section--full">
<h4 class="tech-section__title">Build Options</h4>
<div class="tech-section__content">
<pre class="code-block">{{ formatOptions(currentEnvironment.options) }}</pre>
</div>
</div>
</div>
</div>
</details>
</div>
</transition>
<!-- Borad Detail not selected message -->
<transition name="fade">
<div v-if="!currentEnvironment && !loading && !error" class="custom-block info">
<p class="custom-block-title">Need to select one environment...</p>
</div>
</transition>
</div>
</template>
<script>
export default {
name: 'FlashEnvironmentSelector',
props: {
boardsUrl: {
type: String,
default: '/boards-info.json'
},
firmwareBaseUrl: {
type: String,
default: '/firmware_build/'
}
},
data() {
return {
environments: [],
selectedEnvironment: '',
loading: true,
error: null,
imageError: false
};
},
computed: {
currentEnvironment() {
return this.environments.find(env => env.environment === this.selectedEnvironment);
},
resolvedBoardsUrl() {
return this.buildUrl(this.boardsUrl);
},
resolvedFirmwareBaseUrl() {
return this.buildUrl(this.firmwareBaseUrl);
},
manifestUrl() {
if (!this.selectedEnvironment) return null;
return `${this.resolvedFirmwareBaseUrl}${this.selectedEnvironment}.manifest.json`;
},
boardImageUrl() {
if (!this.currentEnvironment) return null;
const customImg = this.currentEnvironment.customImg || this.currentEnvironment.CustomImg;
if (customImg && !this.imageError) {
// If it's an absolute URL, use it as-is
if (customImg.startsWith('http')) {
return customImg;
}
// Otherwise, build the URL with site base
return this.buildUrl(customImg);
}
return this.buildUrl('/img/microcontroller.gif');
}
},
watch: {
selectedEnvironment(newValue) {
this.imageError = false;
this.error = null;
}
},
mounted() {
const script = document.createElement('script');
script.type="module";
script.src="https://unpkg.com/esp-web-tools@10/dist/web/install-button.js?module";
document.head.appendChild(script);
this.loadEnvironments();
},
methods: {
buildUrl(path) {
// Concatenate this.$site.base with the provided path
const base = this.$site?.base || '/';
// Remove trailing slash from base and leading slash from path to avoid double slashes
const cleanBase = base.endsWith('/') ? base.slice(0, -1) : base;
const cleanPath = path.startsWith('/') ? path : `/${path}`;
return `${cleanBase}${cleanPath}`;
},
getQueryParam(param) {
// Extract query parameter from URL
const params = new URLSearchParams(window.location.search);
return params.get(param);
},
async loadEnvironments() {
this.loading = true;
this.error = null;
try {
const response = await fetch(this.resolvedBoardsUrl);
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
const data = await response.json();
if (!Array.isArray(data)) {
throw new Error('Invalid data format: expected an array');
}
const validEnvironments = data.filter(env =>
env &&
typeof env === 'object' &&
env.environment &&
typeof env.environment === 'string'
);
if (validEnvironments.length === 0) {
throw new Error('No valid environments found in data');
}
this.environments = validEnvironments;
// Check for env parameter in query string and auto-select
const envParam = this.getQueryParam('env');
if (envParam) {
const matchingEnv = validEnvironments.find(env => env.environment === envParam);
if (matchingEnv) {
this.selectedEnvironment = envParam;
} else {
console.warn(`Environment '${envParam}' not found in available environments`);
}
}
} catch (error) {
console.error('Failed to load environments:', error);
this.error = error.message || 'Failed to load environments';
} finally {
this.loading = false;
}
},
formatOptions(optionsText) {
if (!optionsText) return '';
let optionsTextMap=[];
if (typeof optionsText === 'string'){
optionsTextMap = optionsText.split('\n')
}else if (Array.isArray(optionsText)){
optionsTextMap = optionsText;
}
return optionsTextMap.map(line => line.trim())
.filter(line => line.length > 0)
.join('\n');
},
handleImageError(event) {
this.imageError = true;
}
}
};
</script>
<style scoped>
/* Main Container */
.flash-selector {
max-width: 1200px;
margin: 2rem auto;
padding: 0 1rem;
}
/* Board Details */
cursor: pointer;
/* Board Details Container */
.board-details {
margin-top: 2rem;
}
/* Hero Section - Material Design 3 */
.board-hero {
display: grid;
grid-template-columns: 1fr auto;
gap: 24px;
background: #ffffff;
border-radius: 12px;
padding: 32px;
margin-bottom: 16px;
box-shadow: 0 1px 2px 0 rgba(0, 0, 0, 0.05);
border: 1px solid rgba(0, 0, 0, 0.08);
}
.board-hero__content {
display: flex;
flex-direction: column;
gap: 16px;
}
.board-hero__title {
margin: 0 0 8px 0;
font-size: 1.75rem;
font-weight: 400;
color: var(--accent-color, #3eaf7c);
font-family: 'Monaco', 'Courier New', monospace;
line-height: 1.2;
letter-spacing: -0.01em;
}
/* Board Switcher - Material Chip Style */
.board-switcher {
display: flex;
align-items: center;
gap: 8px;
margin: 0;
}
.switcher-label {
font-size: 0.75rem;
text-transform: uppercase;
font-weight: 500;
letter-spacing: 0.1em;
color: rgba(0, 0, 0, 0.6);
}
.switcher-select-wrapper {
position: relative;
display: inline-flex;
align-items: center;
}
.switcher-select {
appearance: none;
background: rgba(0, 0, 0, 0.08);
color: rgba(0, 0, 0, 0.87);
border: none;
padding: 6px 32px 6px 12px;
border-radius: 16px;
font-size: 0.875rem;
font-weight: 500;
font-family: 'Monaco', 'Courier New', monospace;
cursor: pointer;
transition: background 0.2s cubic-bezier(0.4, 0, 0.2, 1);
height: 32px;
min-width: 180px;
}
.switcher-select:hover {
background: rgba(0, 0, 0, 0.12);
}
.switcher-select:focus {
outline: 2px solid var(--accent-color, #3eaf7c);
outline-offset: 2px;
}
.switcher-icon {
position: absolute;
right: 8px;
pointer-events: none;
font-size: 1rem;
opacity: 0.6;
}
.board-hero__meta {
display: flex;
align-items: baseline;
gap: 8px;
margin: 0;
}
.meta-label {
font-size: 0.75rem;
text-transform: uppercase;
font-weight: 500;
letter-spacing: 0.1em;
color: rgba(0, 0, 0, 0.6);
}
.meta-value {
font-size: 0.875rem;
font-weight: 500;
color: rgba(0, 0, 0, 0.87);
font-family: 'Monaco', 'Courier New', monospace;
}
.board-hero__description {
margin: 0;
font-size: 1rem;
line-height: 1.5;
color: rgba(0, 0, 0, 0.87);
max-width: 600px;
}
.board-hero__image {
display: flex;
align-items: flex-start;
justify-content: center;
min-width: 180px;
max-width: 200px;
}
.board-hero__image img {
max-width: 100%;
max-height: 180px;
object-fit: contain;
}
/* Primary Action - Material Filled Button */
.hero-action {
margin-top: 8px;
}
.connect-button {
background: #1976d2;
color: #ffffff;
border: none;
padding: 10px 24px;
border-radius: 20px;
font-size: 0.875rem;
font-weight: 500;
cursor: pointer;
transition: box-shadow 0.28s cubic-bezier(0.4, 0, 0.2, 1);
box-shadow: 0 1px 2px 0 rgba(0, 0, 0, 0.3), 0 1px 3px 1px rgba(0, 0, 0, 0.15);
text-transform: uppercase;
letter-spacing: 0.125em;
height: 40px;
min-width: 120px;
}
.connect-button:hover:not(:disabled) {
box-shadow: 0 1px 2px 0 rgba(0, 0, 0, 0.3), 0 2px 6px 2px rgba(0, 0, 0, 0.15);
}
.connect-button:active:not(:disabled) {
box-shadow: 0 1px 2px 0 rgba(0, 0, 0, 0.3), 0 1px 3px 1px rgba(0, 0, 0, 0.15);
}
.connect-button--disabled {
background: rgba(0, 0, 0, 0.12);
color: rgba(0, 0, 0, 0.38);
cursor: not-allowed;
box-shadow: none;
}
/* Technical Details - Material Surface */
.tech-details {
background: #ffffff;
border-radius: 12px;
overflow: hidden;
box-shadow: 0 1px 2px 0 rgba(0, 0, 0, 0.05);
border: 1px solid rgba(0, 0, 0, 0.08);
}
.tech-details__summary {
display: flex;
align-items: center;
gap: 12px;
padding: 16px 24px;
cursor: pointer;
user-select: none;
background: rgba(0, 0, 0, 0.02);
transition: background 0.2s cubic-bezier(0.4, 0, 0.2, 1);
list-style: none;
}
.tech-details__summary::-webkit-details-marker {
display: none;
}
.tech-details__summary:hover {
background: rgba(0, 0, 0, 0.04);
}
.tech-details__icon {
font-size: 1.25rem;
opacity: 0.6;
}
.tech-details__title {
font-size: 1rem;
font-weight: 500;
color: rgba(0, 0, 0, 0.87);
flex: 1;
letter-spacing: 0.01em;
}
.tech-details__toggle {
font-size: 0.875rem;
transition: transform 0.2s cubic-bezier(0.4, 0, 0.2, 1);
opacity: 0.6;
}
.tech-details[open] .tech-details__toggle {
transform: rotate(180deg);
}
.tech-details__content {
padding: 24px;
}
/* Tech Grid */
.tech-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(280px, 1fr));
gap: 24px;
}
.tech-section {
display: flex;
flex-direction: column;
gap: 12px;
}
.tech-section--full {
grid-column: 1 / -1;
}
.tech-section__title {
margin: 0;
font-size: 0.75rem;
text-transform: uppercase;
font-weight: 500;
letter-spacing: 0.1em;
color: rgba(0, 0, 0, 0.6);
}
.tech-section__content {
font-size: 0.875rem;
line-height: 1.5;
color: rgba(0, 0, 0, 0.87);
}
/* Chips - Material Design */
.chip-group {
display: flex;
flex-wrap: wrap;
gap: 8px;
}
.chip {
background: rgba(0, 0, 0, 0.08);
color: rgba(0, 0, 0, 0.87);
padding: 6px 12px;
border-radius: 8px;
font-size: 0.875rem;
font-weight: 400;
border: none;
height: 32px;
display: inline-flex;
align-items: center;
}
/* Code Styles - Material Surface */
.code-inline {
background: rgba(0, 0, 0, 0.05);
padding: 2px 6px;
border-radius: 4px;
font-size: 0.875rem;
font-family: 'Monaco', 'Courier New', monospace;
color: rgba(0, 0, 0, 0.87);
}
.code-block {
background: rgba(0, 0, 0, 0.03);
padding: 16px;
border-radius: 8px;
font-size: 0.8125rem;
line-height: 1.5;
overflow-x: auto;
margin: 0;
border: 1px solid rgba(0, 0, 0, 0.08);
white-space: pre-wrap;
word-break: break-word;
font-family: 'Monaco', 'Courier New', monospace;
color: rgba(0, 0, 0, 0.87);
}
/* Loading Spinner */
.loading-spinner {
display: inline-block;
width: 18px;
height: 18px;
border: 2px solid rgba(62, 175, 124, 0.3);
border-top-color: var(--accent-color, #3eaf7c);
border-radius: 50%;
animation: spin 0.8s linear infinite;
vertical-align: middle;
margin-right: 0.5rem;
}
@keyframes spin {
to { transform: rotate(360deg); }
}
/* Fade Transition */
.fade-enter-active, .fade-leave-active {
transition: opacity 0.3s ease;
}
.fade-enter-from, .fade-leave-to {
opacity: 0;
}
/* Accessibility */
.visually-hidden {
position: absolute;
width: 1px;
height: 1px;
margin: -1px;
padding: 0;
overflow: hidden;
clip: rect(0, 0, 0, 0);
border: 0;
}
/* Responsive Design */
@media (max-width: 968px) {
.board-hero {
grid-template-columns: 1fr;
}
.board-hero__image {
order: -1;
max-width: 150px;
margin: 0 auto;
}
.tech-grid {
grid-template-columns: 1fr;
}
}
@media (max-width: 640px) {
.selector-card {
padding: 1.5rem;
}
.board-hero {
padding: 1.5rem;
}
.board-hero__title {
font-size: 1.5rem;
}
.connect-button {
width: 100%;
padding: 0.9rem 2rem;
}
}
</style>
+32 -14
View File
@@ -1,20 +1,35 @@
let meta = require('./defaults.json');
try {
meta_overload = require('./meta.json');
meta = { ...meta, ...meta_overload };
} catch (e) {
console.warn('meta.json not found or not valid. Using default configuration.');
}
const fs = require('fs');
const path = require('path');
const commonConfigPath = path.resolve(__dirname, 'public/commonConfig.js');
if (!fs.existsSync(commonConfigPath)) {
throw new Error(`commonConfig.js not found in ${commonConfigPath}.\nPlease download from https://www.theengs.io/commonConfig.js or create this file before you build the documentation.`);
}
const commonConfig = require('./public/commonConfig');
module.exports = {
...commonConfig,
title: 'Theengs OpenMQTTGateway version_tag',
base: '/',
title: `${meta.title} - ${meta.version}`,
base: meta.url_prefix,
dest: meta.dest, // default is generated/site
description: 'One gateway, many technologies: MQTT gateway for ESP8266 or ESP32 with bidirectional 433mhz/315mhz/868mhz, Infrared communications, BLE, LoRa, beacons detection, mi flora / mi jia / LYWSD02/ Mi Scale compatibility, SMS & LORA.',
head: [
...commonConfig.head,
['script', {type: 'module', src: 'https://unpkg.com/esp-web-tools@9.4.3/dist/web/install-button.js?module'}]
],
head: [...commonConfig.head],
themeConfig: {
repo: '1technophile/OpenMQTTGateway',
docsDir: 'docs',
...commonConfig.themeConfig,
repo: meta.theme_config_repo,
docsDir: 'docs',
mode: meta.mode,
sidebar: [
['/','0 - What is it for 🏠'],
['/', '0 - What is it for 🏠'],
{
title: '1 - Prerequisites🧭', // required
//collapsable: true, // optional, defaults to true
@@ -42,9 +57,10 @@ module.exports = {
},
{
title: '3 - Upload ➡️', // required
path: '/upload/',
sidebarDepth: 1, // optional, defaults to 1
children: [
'upload/web-install',
['upload/web-install', "(Option 1) Upload from the web"],
'upload/binaries',
'upload/builds',
'upload/gitpod',
@@ -86,23 +102,25 @@ module.exports = {
},
{
title: '6 - Participate 💻', // required
path: '/participate/',
sidebarDepth: 1, // optional, defaults to 1
children: [
'participate/quick_start',
'participate/support',
'participate/development',
'participate/adding-protocols',
'participate/community',
['https://github.com/1technophile/OpenMQTTGateway/blob/development/LICENSE.txt','License']
[meta.url_license_file, 'License']
]
}
]
]
},
plugins: {
'sitemap': {
hostname: 'https://docs.openmqttgateway.com',
hostname: meta.hostname,
urls: [
'https://decoder.theengs.io/devices/devices.html',
'https://community.openmqttgateway.com/',
meta.url_community_forum,
'https://shop.theengs.io/',
'https://shop.theengs.io/products/theengs-plug-smart-plug-ble-gateway-and-energy-consumption',
'https://shop.theengs.io/products/theengs-bridge-esp32-ble-mqtt-gateway-with-ethernet-and-external-antenna',
+12
View File
@@ -0,0 +1,12 @@
{
"title": "Theengs OpenMQTTGateway",
"version": "edge",
"url_prefix": "/",
"dest": "generated/site",
"mode": "prod",
"url_license_file": "https://github.com/1technophile/OpenMQTTGateway/blob/development/LICENSE.txt",
"url_community_forum": "https://community.openmqttgateway.com",
"hostname": "https://docs.openmqttgateway.com",
"repo": "1technophile/OpenMQTTGateway",
"theme_config_repo": "1technophile/OpenMQTTGateway"
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 324 KiB

+10
View File
@@ -1,3 +1,13 @@
<ClientOnly>
<span v-if="$site.themeConfig.mode === 'dev'">
::: warning Development Version
This is the edge version of the documentation. It is under active development and may contain bugs, incomplete features, or breaking changes. Use it at your own risk.
:::
</span>
</ClientOnly>
OpenMQTTGateway aims to unify various technologies and protocols into a single firmware. This reduces the need for multiple physical bridges and streamlines diverse technologies under the widely-used [MQTT](http://mqtt.org/) protocol.
<div style="text-align: center;">
Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

+46
View File
@@ -0,0 +1,46 @@
# Participate in OpenMQTTGateway
OpenMQTTGateway is community-driven: every feature, bug fix, and doc improvement comes from contributors like you. Whether you want to add support for a new sensor, fix a bug, improve documentation, or just help others—there's a place for you here.
## Choose your path
### 🚀 New to the project? Start here
[**Quick Start Guide**](./quick_start.md) walks you through setting up your development environment, building firmware, and previewing the docs. It covers PlatformIO, VS Code, terminal workflows, and troubleshooting. Perfect if you're new to ESP32/ESP8266 development or just want to get your hands dirty quickly.
**What you'll learn:** Installing tools · Building firmware · Flashing boards · Running tests · Working with the docs
---
### 💻 Ready to contribute code?
[**Development Contributions**](./development.md) covers the contribution workflow: naming conventions, code quality checks, testing on multiple boards, and opening pull requests. This is your checklist before submitting code.
**What you'll find:** PR workflow · Naming rules (ZgatewayXXX, ZsensorYYY) · QA automation · DCO · CI/CD integration
---
### 🔌 Adding a new protocol or device?
[**Adding Protocols**](./adding-protocols.md) explains how to integrate RF, IR, or BLE devices. Most protocol work happens upstream (RCSwitch, Pilight, IRRemoteESP8266, Theengs Decoder), and this guide shows you how to wire it into OMG.
**What you'll learn:** Upstream contribution · Protocol integration · Testing decoders · Payload validation
---
### 💬 Join the community
[**Community Participation**](./community.md) is where you ask questions, share your projects, and help others. The forum is the first stop for feature requests, debugging help, and general IoT/home automation discussion.
**What you can do:** Ask questions · Share builds · Propose features · Help newcomers
---
### ❤️ Support the project
[**Supporting the Project**](./support.md) lists all the ways you can help beyond code: answering forum questions, improving docs, purchasing Theengs products, sponsoring developers, or creating content about your OMG setup.
**How you can help:** Forum support · Documentation · Sponsorship · Content creation
---
::: tip Not sure where to start?
- **First time here?** → [Quick Start](./quick_start.md)
- **Have a fix or feature ready?** → [Development contributions](./development.md)
- **Need help or want to discuss?** → [Community](./community.md)
:::
+14 -4
View File
@@ -1,12 +1,22 @@
# Adding protocols
Adding your device protocol to OpenMQTTGateway enables it to increase interoperability and to create new use cases with your device. Below you will find some guidance to do that.
Adding your device protocol to OpenMQTTGateway increases interoperability and creates new use cases. Use the steps below to pick the right path and upstream your changes.
::: tip Before you open a PR
Read the [Development contributions guide](./development.md) for naming rules, QA, and PR steps. It keeps your protocol addition reviewable.
:::
## RF or IR
For adding RF and IR protocols to OpenMQTTGateway the best way is to do a pull request to [RCSwitch](https://github.com/1technophile/rc-switch), [Pilight](https://github.com/pilight/pilight) for RF, and [IRRemoteESP8266](https://github.com/crankyoldgit/IRremoteESP8266) for IR.
1. Implement or extend the protocol in the upstream libs first: [RCSwitch](https://github.com/1technophile/rc-switch) or [Pilight](https://github.com/pilight/pilight) for RF, and [IRRemoteESP8266](https://github.com/crankyoldgit/IRremoteESP8266) for IR.
2. Once merged (or while your PR is pending), wire it in OMG as a new decoder/mapping.
3. Add a short test sniff/build locally with the relevant environment (for example `esp32dev-all-test`).
## BLE
For BLE message decoding OpenMQTTGateway uses the [Theengs Decoder](https://decoder.theengs.io/) library. New device decoder pull requests can be submitted directly to the [GitHub repository](https://github.com/theengs/decoder).
1. For BLE message decoding, OpenMQTTGateway uses [Theengs Decoder](https://decoder.theengs.io/). Submit new device decoders directly to the [GitHub repository](https://github.com/theengs/decoder).
2. After the decoder exists, ensure the device is reported correctly through OMG (advertising devices are supported without connection).
3. Build locally for a BLE-enabled environment and validate the payload.
Currently we support the reading of advertizing BLE devices, advertizing means that the BLE device broadcasts regularly its sensor data without the need of a BLE connection.
Notes:
- We support reading **advertising** BLE devices (they broadcast regularly without a connection).
- Keep payloads compact and follow existing JSON fields for consistency.
+7 -2
View File
@@ -1,4 +1,9 @@
# Community participation
The first entry step for participating to OMG is sharing and discussing with the [community](https://community.openmqttgateway.com), by sharing your experiences and answering to other questions you are giving back what the others gave to you.
The first entry step for participating to OMG is sharing and discussing with the [community](https://community.openmqttgateway.com). By sharing your experiences and answering questions you give back what others gave to you.
You can use the forum to ask questions, post answers, suggest features and discuss about home automation or Internet of things in a more general way.
Use the forum to:
- Ask questions about hardware, builds, MQTT, BLE/RF/IR, or docs.
- Share projects, logs, and payload samples that help others.
- Suggest features or confirm if something already exists.
When you have a change to contribute, follow the [Development contributions guide](./development.md) for the PR flow.
+22 -15
View File
@@ -1,30 +1,37 @@
# Development contributions
We like pull requests from everyone. By participating in this project, you
agree to follow the code of conduct below
We like pull requests from everyone and ask you to follow the code of conduct below.
[code of conduct](https://github.com/1technophile/OpenMQTTGateway/blob/master/CODE_OF_CONDUCT.md)
* [Code of conduct](https://github.com/1technophile/OpenMQTTGateway/blob/master/CODE_OF_CONDUCT.md)
[code style guide](https://google.github.io/styleguide/cppguide.html#Formatting)
* [Code style guide](https://google.github.io/styleguide/cppguide.html#Formatting)
So as to format automatically your document you have to add the "clang-Format" extension to VSCode, once done, you can format the code by doing a right click into the code file window and clicking "Format document".
If you need a step-by-step install and build guide, read the [Quick Start to Develop OpenMQTTGateway](./quick_start.md). It explains the tools, PlatformIO, and docs workflow. Use this page as a fast checklist once you know the flow.
Fork the [development branch](https://github.com/1technophile/OpenMQTTGateway/tree/development), then clone the repo
**Quick checklist**
1. Fork the [development branch](https://github.com/1technophile/OpenMQTTGateway/tree/development) and clone the repo.
2. Make your changes and follow the naming rules:
* New gateway: `ZgatewayXXX` where `XXX` is the protocol name.
* New sensor: `ZsensorYYY` where `YYY` is the sensor type.
* New actuator: `ZactuatorZZZ` where `ZZZ` is the actuator type.
3. Review your code and compile for ESP32 and ESP8266.
4. Test on your hardware.
5. Open a pull request, verify the GitHub Actions CI build, and request a review.
Make your modification,
* If you want to add a new gateway, name it `ZgatewayXXX`, `XXX` replaced by your gateway communication type, can be more than three letters
* If you want to add a new sensor, name it `ZsensorYYY`, `YYY` replaced by your sensor type, can be more than three letters
* If you want to add a new actuator, name it `ZactuatorZZZ`, `ZZZ` replaced by your actuator type, can be more than three letters
## Automated Testing and CI
Review your code, compile it for ESP32 and ESP8266
Your pull request will be automatically tested by GitHub Actions. If you want to run the same checks locally before pushing, you can use the `ci.sh` script in the scripts folder.
Test it locally on your hardware config
**Need help with the ci.sh commands?** See [section 5 in Quick Start](./quick_start.md#5-firmware-development-from-the-terminal-with-cish) for detailed examples of running QA checks and builds from the terminal.
Emit a pull request
For more details on how CI works and available scripts, see the [CI documentation](https://github.com/1technophile/OpenMQTTGateway/blob/development/scripts) in the scripts folder.
Verify the GitHub Actions CI compilation results
For a comprehensive overview of all GitHub Actions workflows used in this project, check the [Workflows README](https://github.com/1technophile/OpenMQTTGateway/tree/development/.github/workflows).
## Code Quality
To format your code automatically, add the "clang-Format" extension to VSCode, then right click in the file and choose "Format document".
Request for review
We may suggest some changes, improvements or alternatives.
+840
View File
@@ -0,0 +1,840 @@
---
title: Quick Start to Develop OpenMQTTGateway
permalink: /participate/quick_start.html
sidebarDepth: 2
lang: en-US
---
# Quick Start to Develop OpenMQTTGateway
This document helps a new contributor. Maybe this is your first time with:
- Git
- PlatformIO
- ESP32 or ESP8266
It is normal to feel a bit lost at the beginning. This guide walks with you, step by step.
::: tip Already experienced?
If you only need the contribution rules and PR flow, jump to [Development contributions](./development.md). For community help or non-code support, see [Community participation](./community.md) and [Supporting the project](./support.md).
:::
In this document you will learn how to:
- Prepare the development environment on Windows or Linux.
- Build and flash the firmware (ESP32 / ESP8266 and other boards).
- Build and preview the documentation website in the docs folder.
At the end of each step you see a **Check**. Use it to confirm that the previous step is correct before you go to the next one. If a check fails, do not worry: read the step again, fix it, and try one more time.
## 1. Understand this project
OpenMQTTGateway is a firmware project. It runs on ESP32, ESP8266 and other boards. It connects many protocols (Bluetooth, RF, IR, LoRa, sensors) to MQTT.
The repository also contains the documentation website. The docs use VuePress and live in the docs folder.
Very simple view of the repository:
- main: C++ source code for the firmware.
- lib: reusable libraries.
- scripts: helper scripts for build, test, CI.
- docs: documentation website.
You will mainly work on:
- Firmware code: files under main and sometimes lib.
- Docs site: files under docs and docs/.vuepress.
**Check:** open the repository in a file explorer and confirm you see at least main, lib, scripts and docs folders. If not, read section 3 again.
## 2. Prepare your computer
You can use Windows or Linux. The steps are very similar. The examples use a terminal. On Windows you can use **Git Bash** or **WSL**; on Linux you can use your normal shell.
Not everybody needs to install the same tools. It depends on what you want to do:
- **Only firmware (with VS Code or terminal):** you mainly need Git, PlatformIO, VS Code (if you like GUI) and a bash shell.
- **Only documentation website:** you mainly need Git, Node.js and npm.
- **Both firmware and docs:** you need everything.
The next subsections explain this.
### 2.1 Common tools (for everyone)
Install these tools first. They are useful for **all** types of work:
- Git
- Visual Studio Code (recommended, but you can also use another editor)
- A bash shell
- On Windows: Git Bash or WSL (Ubuntu is a good choice).
On Windows you can download:
- Git from the official site: <https://git-scm.com/downloads>
- Visual Studio Code from <https://code.visualstudio.com/>
- Git Bash comes with "Git for Windows".
On Linux you can install these with your package manager (for example apt on Ubuntu). For more help you can read the Git book: <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>.
**Check (common tools):** open a terminal (Git Bash, WSL, or Linux shell) and run:
```bash
git --version
```
If this command fails, install Git again and then close and reopen the terminal.
### 2.2 Tools for firmware development
For firmware builds you need **PlatformIO**. PlatformIO itself will install the right compiler and frameworks for your boards.
You have two main options:
1. Use **PlatformIO inside VS Code** (recommended for most users).
2. Use **PlatformIO CLI in the terminal** (good for advanced or CI usage).
#### 2.2.1 PlatformIO inside VS Code (recommended)
If you only plan to build firmware from VS Code, you usually do **not** need to install Python or PlatformIO CLI manually. The PlatformIO extension will download what it needs.
- Just install VS Code and the PlatformIO extension (see section 2.3).
- When you first open the project, PlatformIO will set up its own tools.
**Check:** after installing the extension and opening the project, the VS Code status bar shows something like "PlatformIO: Ready".
#### 2.2.2 PlatformIO command line (CLI) with Python (for terminal builds)
If you want to use the bash scripts (ci.sh, ci_build_firmware.sh, etc.) or build from a plain terminal, you need Python and PlatformIO CLI.
Install these extra tools:
- Python 3.10 or newer
- PlatformIO CLI (pip package)
PlatformIO is the main build system for this project. We will use it from VS Code and from the terminal. If you want to read more, the official docs are here: <https://docs.platformio.org/>.
On Windows you can download Python from: <https://www.python.org/downloads/>
On Linux you can install Python with your package manager.
Then install PlatformIO CLI globally with Python:
```bash
python3 -m pip install -U platformio
```
**Check (firmware tools):**
```bash
python3 --version
platformio --version
```
If you see version numbers, the install is good. If the platformio command is not found, check that Python added the scripts folder (for example ~/.local/bin on Linux) to your PATH, then open a new terminal and try again.
### 2.3 Tools for documentation website (Node.js and npm)
::: warning Note
This section is **only needed** if you want to **build or preview the site on your own computer**.
:::
If you want to build or edit the **documentation website**, you also need Node.js and npm. You do **not** need these tools just to build firmware.
Install:
- Node.js 18 or newer
- npm (usually comes with Node.js)
On Windows you can get Node.js from: <https://nodejs.org/en/download>
On Linux you can install Node.js with your package manager or by following the docs: <https://nodejs.org/en/download/package-manager>.
**Check (docs tools):**
```bash
node --version
npm --version
```
If one of these commands fails, reinstall Node.js, close the terminal, and try again.
### 2.4 PlatformIO extension in Visual Studio Code
Now add PlatformIO inside VS Code. You can also see pictures of these steps in many ESP32 + PlatformIO tutorials, for example on the PlatformIO site: <https://platformio.org/install/ide?install=vscode>.
Steps (based on common PlatformIO + VS Code ESP32 guides, adapted for this project):
1. Open Visual Studio Code.
2. Click the Extensions icon on the left.
3. In the search box, type "PlatformIO IDE".
4. Click **Install** on the "PlatformIO IDE" extension.
When the install finishes, a new icon appears on the left sidebar. It looks like an ant or an alien head. This opens the PlatformIO view.
**Check:**
- In VS Code, you see the PlatformIO icon on the left.
- If you click it, PlatformIO opens without errors.
If you do not see the icon, restart VS Code and wait a few seconds.
**Friendly tip:** the first install can take several minutes because PlatformIO downloads tools for many boards. It happens only once. You can get a coffee while it works.
## 3. Get the source code
Now download this repository to your computer from GitHub.
1. Open a terminal (Git Bash, WSL or Linux shell).
2. Choose a parent folder where you want to keep your projects.
3. Run:
```bash
git clone https://github.com/1technophile/OpenMQTTGateway.git
cd OpenMQTTGateway
```
**Check:**
- Run:
```bash
git status
```
- The output should say "On branch" and "nothing to commit".
- In your file explorer or with:
```bash
ls
```
you should see main, lib, scripts, docs and other files.
If the clone fails, check your internet connection or a possible proxy. Then try again. If you are new to Git, the GitHub "Hello World" guide can also help: <https://docs.github.com/en/get-started/start-your-journey/hello-world>.
## 4. Firmware development with Visual Studio Code and PlatformIO
This section explains how to build and flash the firmware using Visual Studio Code with the PlatformIO extension.
If you know the classic Arduino IDE, you can think of PlatformIO as a more powerful alternative. It manages many boards, libraries, and environments for you. The official introduction is here: <https://docs.platformio.org/en/latest/what-is-platformio.html>.
You do **not** create a new PlatformIO project. This repository is already a PlatformIO project. You just open it.
### 4.1 Open the project in VS Code
1. Start Visual Studio Code.
2. Click **File → Open Folder…**.
3. Select the OpenMQTTGateway folder you cloned in section 3.
4. Click **Open**.
VS Code will load the folder. PlatformIO will detect the platformio.ini file and start to prepare its toolchain (compiler, libraries, board support). You do not need to install these by hand.
**Check:**
- In the bottom status bar you see messages like "PlatformIO: Installing" and then "PlatformIO: Ready".
- You can also open a terminal inside VS Code (View → Terminal) and run:
```bash
pio --version
```
You should see the PlatformIO version.
If PlatformIO stays stuck on "Installing" for a long time, close VS Code, reopen it and wait again. Check that your internet connection is working.
### 4.2 Look at the PlatformIO project structure
PlatformIO projects usually have folders like (you will see similar names in many tutorials):
- src: main source code.
- lib: libraries.
- include: headers.
- platformio.ini: configuration for boards and environments.
In OpenMQTTGateway the structure is a bit different, because it is a large project:
- main: main C++ files for the firmware.
- lib: libraries used by the firmware.
- platformio.ini and environments.ini: list many PlatformIO environments.
- test: tests for PlatformIO unit test framework.
**Check:** open platformio.ini in VS Code and scroll. You should see several [env:...] sections. This means PlatformIO understands the project.
If you want to know more about this file, see the PlatformIO docs about configuration: <https://docs.platformio.org/en/latest/projectconf/index.html>.
### 4.3 Choose a PlatformIO environment
An "environment" in PlatformIO is a build target. It defines the board, framework and options. You can think of it as a named profile for a specific device.
To select the environment for your board:
1. Press Ctrl+Shift+P (on macOS use Cmd+Shift+P).
2. Type "PlatformIO: Switch Project Environment" and press Enter.
3. A list of environments appears (for example esp32dev-all-test, theengs-bridge and many others).
4. Choose an environment that matches your board. For a generic ESP32 DevKit you can start with **esp32dev-all-test**.
**Check:**
- Look at the bottom blue bar. You should see the selected environment name.
If you chose the wrong environment, repeat the steps and pick another one.
### 4.4 Build the firmware in VS Code
Now compile the firmware.
1. In VS Code, open the PlatformIO view (left sidebar icon).
2. In the PlatformIO toolbar (bottom of the window), click the **check** (✓) icon. This runs the "Build" task.
PlatformIO now compiles the code for the selected environment. This can take several minutes the first time.
**Check:**
- In the Terminal panel, the build ends with the word "SUCCESS".
- On disk, a file like:
```
.pio/build/<your-env-name>/firmware.bin
```
exists.
If the build fails, read the error at the end of the log. Take your time; errors are part of the normal developer life.
Common problems:
- Missing tools: check that platformio works from the terminal (section 2.2).
- Wrong environment: try with esp32dev-all-test first.
Fix the problem, then run the build again.
### 4.5 Upload the firmware to the board
After a successful build you can flash the firmware (upload the program to the board).
1. Connect your ESP board to the computer with a good USB cable.
2. In the PlatformIO toolbar click the **right arrow** (→) icon. This runs the "Upload" task.
PlatformIO selects a serial port automatically in many cases.
**Check:**
- The upload log ends with success, and no "Failed to connect" error.
- The board reboots after upload.
If the upload fails:
- Check the USB cable (some cables are power-only and do not carry data).
- On some boards you must press and hold the BOOT button during upload.
- Make sure no other program is using the same serial port (for example another serial monitor).
- You can set the serial port manually in platformio.ini with the upload_port option.
### 4.6 Open the Serial Monitor
The Serial Monitor lets you see log messages from the firmware.
1. In the PlatformIO toolbar click the **plug** icon. This opens the Serial Monitor.
2. Set the baud rate to 115200 if it is not already set.
**Check:**
- After reset, you see text from the board, for example boot messages and MQTT logs.
If the text is unreadable, check that the baud rate matches the value in platformio.ini (monitor_speed). In this project the default is usually 115200.
If you want to learn more about Serial Monitor problems, the PlatformIO docs have a short page: <https://docs.platformio.org/en/latest/core/userguide/device/cmd_monitor.html>.
### 4.7 Run PlatformIO tests (optional)
You can run unit tests for the project.
1. Open a terminal inside VS Code (View → Terminal).
2. In the project root, run:
```bash
pio test -e test
```
**Check:**
- The output ends with "SUCCESS". If tests fail, read which test failed and open the file under test to understand why.
## 5. Firmware development from the terminal with ci.sh
You can also work only from the terminal. This is useful if you prefer command line, use remote development, or want to reproduce the same steps as the CI (Continuous Integration, the automatic build that runs on GitHub).
All helper scripts live in the scripts folder. The main entry point is:
- scripts/ci.sh
This script calls other scripts:
- scripts/ci_build.sh
- scripts/ci_build_firmware.sh
- scripts/ci_prepare_artifacts.sh
- scripts/ci_site.sh
- scripts/ci_qa.sh
- scripts/ci_list-env.sh
The scripts are written for bash. On Windows always use **Git Bash** or **WSL** for them, not plain PowerShell or cmd.
If you are new to bash, a gentle introduction is here: <https://ubuntu.com/tutorials/command-line-for-beginners>.
### 5.1 Check that you are in the right place
1. Open a bash terminal.
2. Go to your clone:
```bash
cd path/to/OpenMQTTGateway
```
**Check:** run:
```bash
ls scripts main docs
```
You should see these folders listed. If not, you are in the wrong directory.
### 5.2 List available firmware environments
You can ask the helper scripts which PlatformIO environments exist.
```bash
./scripts/ci.sh list-env
```
**Check:**
- The command prints many names in columns (for example esp32dev-all-test, theengs-bridge, rfbridge-direct, and so on).
If the command fails with "command not found" or similar, confirm that bash is used and that the file has execute permission (on Linux you may need chmod +x scripts/ci.sh).
### 5.3 Quick build of one firmware (simple mode)
Use ci.sh with the build command to build one firmware from the terminal.
Example for a development build on esp32dev-all-test:
```bash
./scripts/ci.sh build esp32dev-all-test --mode dev --verbose
```
What this does:
- Checks that Python, PlatformIO and git are installed.
- Calls ci_build_firmware.sh to run platformio run -e esp32dev-all-test.
- Optionally prepares artifacts if you add deployment options.
Useful flags:
- --mode dev or --mode prod: choose development or production mode. For local work, **dev** is usually enough.
- --clean: remove old build files before building.
- --verbose: show more logs from PlatformIO.
**Check:**
- The command ends with a "Build Summary" block and "Status: SUCCESS".
- The folder .pio/build/esp32dev-all-test contains firmware.bin.
If the build fails, read the error line near the bottom. Fix missing tools (see section 2) or wrong environment name, then run the command again.
### 5.4 Use ci_build_firmware.sh directly (advanced control)
Sometimes you want to work closer to PlatformIO.
You can call scripts/ci_build.sh directly when you want more control or when you debug a CI issue:
```bash
./scripts/ci_build.sh esp32dev-all-test --dev-ota --clean --verbose
```
This script:
- Validates the environment name.
- Sets environment variables for the build (for example OMG_VERSION, OTA flags).
- Runs platformio run for the chosen environment.
- Verifies that firmware.bin and other files exist.
**Check:**
- At the end you see a "Build Summary" for that environment.
- The directory .pio/build/esp32dev-all-test contains .bin and .elf files.
If there are no artifacts, check that PlatformIO is installed and that the environment name matches one from ci_list-env.sh.
### 5.5 Prepare artifacts for sharing or flashing tools
After a successful build, you can create a clean set of files ready for release or for use with external flashing tools (for example esptool.py or web uploaders).
Use scripts/ci_prepare_artifacts.sh through ci_build.sh, or call it directly.
Example (after a build) with ci_build.sh wrapper:
```bash
./scripts/ci_build.sh esp32dev-all-test --mode prod --deploy-ready
```
Example direct call to ci_prepare_artifacts.sh:
```bash
./scripts/ci_prepare_artifacts.sh esp32dev-all-test --clean
```
This script:
- Copies firmware.bin, partitions.bin, bootloader.bin and other files from .pio/build/<env> into the output folder.
- Renames them with the environment name, for example esp32dev-all-test-firmware.bin.
- Creates archives for the libraries used by that environment.
**Check:**
- The folder generated/toDeploy (or your custom output) exists.
- It contains files like <env>-firmware.bin and one or more *-libraries.tgz archives.
If the script says the build directory is missing, run a firmware build first (section 5.3 or 5.4).
### 5.6 Run QA checks from the terminal
Before you send a pull request, it is good to check the code style.
Run:
```bash
./scripts/ci.sh qa --check
```
This uses scripts/ci_qa.sh to run clang-format checks on files under main by default.
**Check:**
- The summary says "Status: SUCCESS".
If there are formatting problems, you can auto-fix them:
```bash
./scripts/ci.sh qa --fix
```
After this, review the changes with git diff, then commit them.
### 5.7 Run the full pipeline (QA + firmware build + docs)
For a full local run, similar to CI, use:
```bash
./scripts/ci.sh all esp32dev-all-test --mode prod
```
This will:
- Run QA checks.
- Build the firmware.
- Build the documentation site in production mode (unless you add --no-site).
**Check:**
- At the end you see a "Complete Pipeline Summary" with "Status: SUCCESS".
If QA fails, fix style issues first. If the build fails, fix code or environment problems, then run again.
::: tip Next Steps
Once your code builds successfully, check the [Development contributions guide](./development.md) for:
- Code naming rules (ZgatewayXXX, ZsensorYYY, etc.)
- Code style and quality requirements
- How to open a pull request
- Contributing workflow
:::
## 6. Work on the documentation website (from the terminal)
The documentation website uses VuePress and Node.js. The site source is in the docs folder.
::: warning Note
This section is **only needed** if you want to **build or preview the site on your own computer**.
If you make a **very small change** in a markdown (.md) file (for example fix a typo or add one sentence), you can:
- Edit the file in VS Code.
- Push your change and open a pull request.
In that case it is **not mandatory** to install Node.js, npm, or to run `./scripts/ci.sh site`. The CI on GitHub will build the site for you and show problems if there are any.
If you want to see your doc changes locally before you push (recommended for bigger edits), then follow the steps below.
:::
### 6.1 Install Node dependencies
From the project root:
```bash
cd OpenMQTTGateway # only if you are not already inside
npm install
```
This command reads package.json and installs all Node modules needed for the docs.
**Check:**
- The command finishes without error.
- A node_modules folder exists in the project root.
If install is slow, your internet may be the reason. If it fails, try again later or clear the npm cache with:
```bash
npm cache clean --force
```
### 6.2 Build the docs site with ci.sh
If you want to check that everything still builds well after bigger doc changes, you can build the site locally.
The recommended way is:
```bash
./scripts/ci.sh site --mode prod --url-prefix / --version 1.8.0
```
Important options:
- `--mode dev` or `--mode prod`: development or production build (default: dev).
- `--url-prefix PATH`: base URL path for links, e.g. `/` for root or `/dev/` for dev (default: /dev/).
- `--version TAG`: version string written into docs/.vuepress/meta.json (default: edge).
- `--preview`: if added, starts a local HTTPS preview server after build.
- `--clean`: remove generated/site folder before build.
- `--insecure-curl`: allow curl to skip TLS verification if needed.
The script (scripts/ci_site.sh) will:
- Check that node, npm and openssl are available.
- Download a shared configuration file for the site.
- Create docs/.vuepress/meta.json with site info.
- Run npm run docs:build to build the site.
**Check:**
- The summary at the end says "Site Build Summary" and "Status: SUCCESS".
- The folder docs/.vuepress/dist exists and contains HTML files.
If openssl or node is missing, go back to section 2 and install them.
### 6.3 Preview the docs site locally
To preview the site in your browser:
```bash
./scripts/ci.sh site --mode dev --url-prefix /dev/ --version edge --preview
```
The script will start a local HTTPS server.
**Check:**
- The log prints a line similar to: "Preview server running at https://localhost:8443/dev/".
- Open that URL in your browser. You should see the OpenMQTTGateway documentation.
To stop the preview, go back to the terminal and press Ctrl+C.
### 6.4 Work on docs with plain npm commands
If you prefer not to use the ci.sh wrapper, you can work directly with npm and VuePress.
From the project root:
```bash
npm run docs:dev
```
This runs VuePress in development mode with hot reload. When you change a .md file under docs, the browser reloads.
For a production build:
```bash
npm run docs:build
```
**Check:**
- For docs:dev, the terminal prints a local URL like http://localhost:8080. Open it in the browser and see the docs.
- For docs:build, the folder docs/.vuepress/dist is created.
If docs:build fails with an OpenSSL error on new Node versions, set:
```bash
export NODE_OPTIONS="--openssl-legacy-provider"
```
then run the command again. The ci_site.sh script already does this for you.
### 6.5 Where to edit docs
- All documentation pages are markdown (.md) files under docs.
- This file itself is in docs/participate/quick_start.md.
To add or edit docs:
1. Open the docs folder in VS Code.
2. Change or create markdown files.
3. Run npm run docs:dev to check how the page looks.
**Check:** after a change and a page refresh, your new text appears on the site.
## 7. Typical workflows for contributors
This section gives some example "day to day" flows.
### 7.1 Quick firmware change with VS Code
1. Open the project folder in VS Code.
2. Choose the right environment (section 4.3).
3. Edit code in main or lib.
4. Build (section 4.4).
5. Upload (section 4.5).
6. Watch logs in Serial Monitor (section 4.6).
**Check:** your change has the expected effect on the real device.
### 7.2 Firmware change using terminal only
1. Open a bash terminal.
2. Go to the project root.
3. Build firmware with:
```bash
./scripts/ci.sh build <env> --mode dev --verbose
```
4. Flash firmware using PlatformIO CLI or another flash tool with the generated firmware file.
**Check:** the device boots with your new firmware and behaves as expected.
### 7.3 Change documentation and preview
1. Edit markdown files under docs.
2. In a terminal, run npm run docs:dev or ./scripts/ci.sh site --mode dev --preview.
3. Open the local URL in your browser.
**Check:** your text appears on the page and looks correct.
## 8. Troubleshooting
This list gives quick help for common problems.
- **Problem:** platformio command not found.
- **Fix:** install it with python3 -m pip install -U platformio and open a new terminal.
- **Problem:** Build fails for all environments.
- **Fix:** run ./scripts/ci.sh list-env to confirm you use a valid env. Check that you did not change platformio.ini or environments.ini in a wrong way.
- **Problem:** Build fails with missing libraries.
- **Fix:** run a clean build (add --clean). If you work only with VS Code, press the trash icon or use the clean target from PlatformIO.
- **Problem:** Upload fails ("Failed to connect to ESP32").
- **Fix:** choose the right serial port, check the cable, try holding BOOT during upload.
- **Problem:** Serial output is garbled.
- **Fix:** set Serial Monitor baud to 115200 and confirm monitor_speed in platformio.ini is the same.
- **Problem:** npm install is very slow or fails.
- **Fix:** check your internet connection. If needed run npm cache clean --force and try again.
- **Problem:** Site build fails with an OpenSSL error.
- **Fix:** export NODE_OPTIONS="--openssl-legacy-provider" before running npm run docs:build, or use ./scripts/ci.sh site which already sets this.
If you still have problems after these steps, you can open an issue on the project GitHub page. Include:
- Your operating system (Windows or Linux, version).
- What you tried to do.
- The exact command you ran.
- The error message from the end of the log.
This information helps maintainers reproduce and fix the problem.
## 9. Glossary (simple words)
This glossary explains some words used in this guide.
- **Firmware**: the program that runs inside your ESP32 or ESP8266 board.
- **Repository (repo)**: the project folder stored on GitHub and on your computer.
- **Commit**: a saved set of changes in Git, with a message.
- **Branch**: a line of development in Git, like a separate copy where you work.
- **Pull Request (PR)**: a request to merge your branch into the main project on GitHub.
- **Environment (env)**: a PlatformIO configuration for a specific board and options.
- **CI (Continuous Integration)**: automatic scripts that build and test the project on every change.
- **Serial Monitor**: a window that shows text messages sent by your board over USB.
- **MQTT**: a lightweight network protocol used to send messages between devices.
- **VuePress**: a static site generator used to build this documentation.
If any word in this guide is not clear, you can search it on the web or ask in the project community. Many people had the same question before you.
## 10. External links and further reading
Here is a list of useful links related to tools used in this project:
- **OpenMQTTGateway project**: <https://github.com/1technophile/OpenMQTTGateway>
- **PlatformIO main site**: <https://platformio.org/>
- **PlatformIO installation for VS Code**: <https://platformio.org/install/ide?install=vscode>
- **PlatformIO documentation**: <https://docs.platformio.org/>
- **Visual Studio Code documentation**: <https://code.visualstudio.com/docs>
- **Git official site**: <https://git-scm.com/>
- **Pro Git book (free)**: <https://git-scm.com/book/en/v2>
- **GitHub getting started**: <https://docs.github.com/en/get-started>
- **Node.js documentation**: <https://nodejs.org/en/docs>
- **npm documentation**: <https://docs.npmjs.com/>
- **VuePress documentation** (v1, similar to what this project uses): <https://v1.vuepress.vuejs.org/>
- **MQTT introduction (HiveMQ)**: <https://www.hivemq.com/mqtt-essentials/>
You do **not** need to read all of them now. Keep this list as a bookmark and come back when you are curious or stuck.
## 11. Final words and friendly advice
Contributing to an open source project is a journey. The first steps (install tools, learn Git, learn PlatformIO) can feel slow. This is normal.
Some final tips:
- Change one thing at a time and test often.
- Keep your changes small; they are easier to review.
- Write clear commit messages.
- When something breaks, read the last 10–20 lines of the log first.
- Do not be afraid to ask questions; everyone started as a beginner.
You are now ready to work on both firmware and documentation for OpenMQTTGateway. Take your time, follow the checks after each step, and you will build confidence with the toolchain and the project. Step by step, you will become faster and more comfortable.
If you are new and want to "play" a bit before you change real logic, here are some safe exercises:
1. **Practice the build chain only.**
- Clone the repo, open it in VS Code, select `esp32dev-all-test`, build, and upload to your board.
- Check that you can see logs in the Serial Monitor.
2. **Make a tiny log change.**
- Find a `LOG` or `Serial` message in a gateway or sensor file under `main`.
- Change the text slightly (for example add a word), rebuild, upload, and verify you see the new message.
3. **Edit a small doc page.**
- Fix a typo or add one sentence of clarification in a markdown file under `docs`.
- Run `./scripts/ci.sh site --mode prod --preview` and confirm your change appears in the browser.
4. **Run QA locally.**
- Run `./scripts/ci.sh qa --check`. If there are issues, run `./scripts/ci.sh qa --fix` and see how files are changed.
5. **Open a small PR.**
- For example, only the doc change or a very small code improvement. This lets you learn the review process without much stress.
These small steps build confidence. After that you can move to bigger things: adding new sensors, improving MQTT payloads, or extending the web documentation.
+16 -9
View File
@@ -1,15 +1,22 @@
# Supporting the project
If you like the project and/or used it please consider supporting it! It can be done in different ways:
* Purchase the [Theengs mobile application](https://app.theengs.io)
* Purchase the [Theengs plug](https://shop.theengs.io)
* Helping other users in the [community](https://community.openmqttgateway.com)
* [Contribute](development) to the [code](https://github.com/1technophile/OpenMQTTGateway) or the [documentation](https://docs.openmqttgateway.com)
* Buy devices, boards or parts from the [compatible web site](https://compatible.openmqttgateway.com), the devices and parts linked use affiliated links.
* Donate or sponsor the project [developers](https://github.com/1technophile/OpenMQTTGateway/graphs/contributors)
* Make a video or a blog article about what you have done with [OpenMQTTGateway](https://docs.openmqttgateway.com) and share it.
If you like the project, you can help in different ways. Choose what fits you best.
Support open-source development through sponsorship and gain exclusive access to our private forum. Your questions, issues, and feature requests will receive priority attention, plus you'll gain insider access to our roadmap.
## Give time and knowledge
- Help other users in the [community](https://community.openmqttgateway.com).
- Share logs, payload samples, or how-to posts that unblock others.
## Contribute code or docs
- Follow the [Development contributions guide](./development.md) to open a PR.
- Improve docs or examples if you spot unclear steps.
## Financial support
- Purchase the [Theengs mobile application](https://app.theengs.io) or the [Theengs plug](https://shop.theengs.io).
- Buy devices or parts via the [compatible web site](https://compatible.openmqttgateway.com) (affiliate links help the project).
- Sponsor the project [developers](https://github.com/1technophile/OpenMQTTGateway/graphs/contributors).
- Make a video or blog article about what you built with [OpenMQTTGateway](https://docs.openmqttgateway.com) and share it.
Support open-source development through sponsorship and gain exclusive access to our private forum. Your questions, issues, and feature requests receive priority attention, plus you'll see the roadmap early.
<div style="text-align: center;">
<iframe src="https://github.com/sponsors/theengs/button" title="Sponsor Theengs" height="32" width="228" style="border: 0; border-radius: 6px;"></iframe>
+11 -1
View File
@@ -1,3 +1,7 @@
---
pageClass: table-generated-page
---
# Boards
OpenMQTTGateway is not closed to one board or type of board, by using the power of the Arduino framework and libraries that are cross compatibles it let you many choice of hardware, from an ESP8266 to an ESP32.
@@ -43,4 +47,10 @@ The plug is available in North America only, other regions are planned.
Choosing your board depends heavily on the technologies you want to use with it.
To have a good overview of the compatibilities per board you can refer to the compatible modules attributes of each [board](https://compatible.openmqttgateway.com/index.php/boards/).
The choice between these boards will depend on your knowledge and your requirements in terms of reliability, situation, modules wanted and devices you already have. The table below present those (auto-generated)
The choice between these boards will depend on your knowledge and your requirements in terms of reliability, situation, modules wanted and devices you already have. Use the table below to explore the latest environments.
<BoardEnvironmentTable
boardsUrl="/boards-info.json"
selectorPath="/upload/board-selector.html"
/>
-4
View File
@@ -1,4 +0,0 @@
# Ignore everything in this directory
*
# Except this file
!.gitignore
+17 -1
View File
@@ -52,7 +52,21 @@ With SRX882S receiver connect the CS pin to 3.3V
|ESP32|**D27**|D12|D18|**3V3**|D23|D19|D5|GND
To use the CC1101 module, `ZradioCC1101` must be uncomment in the `User_config.h` or added to the `build_flags`.
More information about the [CC1101 wiring](https://github.com/LSatan/SmartRC-CC1101-Driver-Lib#wiring). ( Please note that with OMG we are recommending CC1101 GDO2 to be connected to ESP32 D27 and GDO0 to be connected to D12, this is different than the LSatan diagram. This is due to the ESP32 using D2 as part of the boot process. )
More information about the [CC1101 wiring](https://github.com/LSatan/SmartRC-CC1101-Driver-Lib#wiring).
:::tip Please note that with OMG we are recommending CC1101 GDO2 to be connected to ESP32 D27 and GDO0 to be connected to D12, this is different than the LSatan diagram. This is due to the ESP32 using D2 as part of the boot process.
If you want to use custom SPI pins for the CC1101 module, you can define the following variables in your `User_config.h` or as `build_flags` in `platformio.ini`:
- `RF_CC1101_SCK`: SPI clock pin (SCK)
- `RF_CC1101_SCK`: SPI clock pin (SCK)
- `RF_CC1101_MISO`: SPI MISO pin (Master In Slave Out)
- `RF_CC1101_MOSI`: SPI MOSI pin (Master Out Slave In)
- `RF_CC1101_CS`: SPI chip select pin (CSN)
When **all** these variables are defined, OpenMQTTGateway will use your custom pinout for the CC1101 connection. This is useful if your board does not use the default pins or if you want to avoid conflicts with other devices.
:::
## ESP32 Hardware setup
![Addon_RF](../img/OpenMQTTgateway_ESP32_Addon_RF.png)
@@ -73,3 +87,5 @@ The RF processing can be achieved after the modification by either RF, RF2 or Pi
## WIFI RF GATEWAY Hardware setup
This board doesn't require any hardware modifications.
+79
View File
@@ -0,0 +1,79 @@
# Upload
OpenMQTTGateway provides several installation approaches suited for different use cases. The quickest path uses pre-built binaries that work with standard configurations. For custom parameters or specific gateway module combinations, you can build firmware using your development environment. Alternatively, you can flash devices directly from your web browser without installing any software.
<p align="center">
<img src="../img/upload.png" alt="Firmware upload process">
</p>
## Choosing Your Upload Method
### Web-Based Installation
The [web installation](web-install.md) method represents the easiest way to get started with OpenMQTTGateway. You can flash your device directly from your browser without downloading files or installing development tools. This method works on Windows, macOS, and Linux with Chrome, Edge, or Opera browsers.
The web installer displays available environments for different boards and gateway configurations. Select your board type, connect via USB, and click the connect button. The installer handles everything automatically, including erasing old firmware and writing the new one. The web installer uses [ESP Web Tools](https://esphome.github.io/esp-web-tools/) technology to communicate directly with your ESP device.
### Ready-to-Go Binary Installation
[Downloading and installing pre-built binaries](binaries.md) offers control over the flashing process using desktop tools. Download the binary files for your board from the [GitHub releases page](https://github.com/1technophile/OpenMQTTGateway/releases).
For ESP32 devices, you need the firmware binary, bootloader, and boot application partition files written to specific memory addresses. Windows users can use the ESP32 Flash Download Tool from Espressif. On Linux and macOS, the esptool.py command-line utility provides a straightforward upload method.
This method works well for standard configurations without modifying source code. After flashing, you can still configure WiFi, MQTT broker settings, and basic parameters through the configuration portal.
### Custom Build and Upload
[Building from source](builds.md) becomes necessary when you need specific pin assignments, custom MQTT topics, or particular gateway module combinations not available in pre-built binaries.
[PlatformIO](https://platformio.org/) provides the recommended build environment. After downloading the [source code from GitHub](https://github.com/1technophile/OpenMQTTGateway), you will find a `platformio.ini` file defining build environments for various hardware combinations.
The configuration system uses a layering approach where default values from [`User_config.h`](https://github.com/1technophile/OpenMQTTGateway/blob/development/main/User_config.h) and [`config_XX.h`](https://github.com/1technophile/OpenMQTTGateway/tree/development/main) files can be overridden by build flags in `platformio.ini` or `environments.ini`. You can embed WiFi credentials and MQTT settings at build time for automatic connection on first boot.
### Browser-Based Building with Gitpod
For those who want to build custom firmware without setting up a local development environment, [Gitpod](gitpod.md) offers a cloud-based solution. By clicking on the [Gitpod link](https://gitpod.io#https://github.com/1technophile/OpenMQTTGateway/tree/development), you get a complete development environment running in your browser with PlatformIO already installed and configured.
After the automatic initial build completes, modify the environment configuration by editing `environments.ini` and run build commands in the browser terminal. Download the generated firmware files and flash them using the binary installation method.
## Configuring Network and MQTT Settings
After flashing firmware, configure network connectivity and MQTT broker settings using either runtime or build-time approaches.
### Runtime Configuration Portal
When you power on a freshly flashed device, it creates a WiFi access point named OpenMQTTGateway or starting with OMG_. You can find detailed information about the [configuration portal here](portal.md).
Connecting to this access point opens a portal where you configure your WiFi network, MQTT broker details, and optional security settings including TLS encryption and certificates. For devices with Ethernet, access the portal through the LAN IP address and configure WiFi as fallback connectivity.
The portal accepts broker IP addresses or hostnames with mDNS support like homeassistant.local. Set a gateway password to protect future configuration changes, OTA updates, and web interface access.
### Build-Time Configuration
Alternatively, embed network and MQTT settings directly in firmware during the build process. Set parameters in `User_config.h` or add them as build flags in your PlatformIO environment definition. Store sensitive information in a separate `_env.ini` file excluded from version control.
## Advanced Configuration Options
Beyond basic connectivity, OpenMQTTGateway supports several [advanced features](advanced-configuration.md) that enhance security and integration capabilities.
### Secure MQTT Connections
For deployments over the internet or public networks, enable TLS encryption to secure communication between the gateway and MQTT broker. Configure your broker with a valid certificate and obtain the Certificate Authority certificate. The gateway can verify server identity against this certificate or connect with encryption without validation.
Provide the CA certificate at build time in `default_server_cert.h` or paste it into the configuration portal. The gateway supports both self-signed certificates and those from public certificate authorities.
### Home Assistant Auto-Discovery
When you use [Home Assistant](https://www.home-assistant.io/) as your home automation platform, OpenMQTTGateway automatically creates device entries and sensors through Home Assistant's MQTT discovery protocol, enabled by default in all standard builds.
Enable discovery in your Home Assistant MQTT integration settings and create a dedicated MQTT user. The gateway registers itself as a device and creates sensor entities automatically, appearing in Configuration → Devices section.
### Topic Customization
The gateway publishes messages to MQTT topics following the format `home/OpenMQTTGateway/GATEWAYtoMQTT`. Enable the `valueAsATopic` feature to append received values to the topic path, making topic-based filtering easier and avoiding warnings in certain controllers.
## Next Steps
After successfully uploading firmware and configuring your gateway, you can proceed to configure specific gateway modules for [RF](../setitup/rf.md), [IR](../setitup/ir.md), [Bluetooth](../setitup/ble.md), [LoRa](../setitup/lora.md), or other protocols you want to use. Each module has configuration options adjustable through MQTT commands or the web interface without rebuilding firmware.
The [troubleshooting section](troubleshoot.md) covers common issues, but if you encounter problems not addressed here, the [OpenMQTTGateway community forum](https://community.openmqttgateway.com) provides an active place to ask questions and share solutions with other users.
+10
View File
@@ -0,0 +1,10 @@
---
pageClass: table-generated-page
---
<FlashEnvironmentSelector
boardsUrl="/boards-info.json"
firmwareBaseUrl="firmware_build/" />
+9 -11
View File
@@ -2,7 +2,14 @@
pageClass: table-generated-page
---
# (Option 1) Upload from the web
## Select your firmware
To upload firmware to your ESP device directly from this page, first connect your ESP device to a USB port on your computer. Then, choose the appropriate firmware from the available options. Next, click the **Connect** button and select the USB port where your ESP is plugged in. Wait for the upload process to finish. After the upload completes, you can set up your [WiFi and MQTT credentials](portal.md).
<BoardEnvironmentTable
boardsUrl="/boards-info.json"
selectorPath="/upload/board-selector.html"
/>
::: tip Running on a tablet or phone
If you want to use the BLE decoding capabilities of OpenMQTTGateway with a tablet or smartphone you can use [Theengs App](https://app.theengs.io/).
@@ -15,16 +22,8 @@ The correct driver to then select in the popup of this web install is
`/dev/cu.wchusbserialXXXXXXXXXXX`
:::
You can upload the firmware to your ESP device directly from here.
1. Plug in your ESP to a USB port.
2. Select the firmware in the box below.
3. Click the install button and choose the port that the ESP is connected to.
4. Wait until the process is complete.
5. Once completed you can configure your [WiFi and MQTT credentials](portal.md)
<web-uploader/>
Upload powered by [ESP Web Tools](https://esphome.github.io/esp-web-tools/)
## Using OpenMQTTGateway ?
Support open-source development through sponsorship and gain exclusive access to our private forum. Your questions, issues, and feature requests will receive priority attention, plus you'll gain insider access to our roadmap.
@@ -33,5 +32,4 @@ Support open-source development through sponsorship and gain exclusive access to
<iframe src="https://github.com/sponsors/theengs/button" title="Sponsor Theengs" height="32" width="228" style="border: 0; border-radius: 6px;"></iframe>
</div>
## Environments characteristics
The auto-generated table below describes the libraries and the modules of each board configuration.
+232
View File
@@ -0,0 +1,232 @@
'use strict';
/**
* Universal parser for PlatformIO dependencies.
* Formats URLs and registry strings into a consistent "Registry Style":
* Name @ Version (provider:user)
*/
function smartFormat(dep) {
if (!dep) return "";
if (typeof dep !== 'string') return dep;
const cleanDep = dep.trim();
// Configuration for Git providers with specific regex for archives, releases, and git repos
const providers = [
{
id: 'gh',
name: 'github',
// Captures: 1. Author, 2. Repo, 3. Version from path (releases), 4. Version from filename/branch
regex: /github\.com\/([^/]+)\/([^/.]+)(?:\/(?:archive|releases\/download\/([^/]+)|tree)\/)?([^/]+)?(?:\.zip|\.git)?$/i
},
{
id: 'gl',
name: 'gitlab',
// Captures: 1. Author, 2. Repo, 3. Version from path, 4. Version from filename
regex: /gitlab\.com\/([^/]+)\/([^/.]+)(?:\/(?:-\/)?(?:archive|releases)\/([^/]+))?\/([^/]+)?(?:\.zip|\.git)?$/i
},
{
id: 'bb',
name: 'bitbucket',
// Captures: 1. Author, 2. Repo, 3. Version from path, 4. Version from filename
regex: /bitbucket\.org\/([^/]+)\/([^/.]+)(?:\/(?:get|downloads)\/([^/]+))?\/([^/]+)?(?:\.zip|\.git)?$/i
}
];
// 1. Try to match against Git providers (GitHub, GitLab, Bitbucket)
for (const p of providers) {
const match = cleanDep.match(p.regex);
if (match) {
let [_, author, repo, pathVer, fileVer] = match;
// Prioritize version from path (typical in releases) over filename
let version = pathVer || fileVer || "latest";
// Clean up version string: remove extensions and 'v' prefix
version = version
.replace(/\.(zip|git|tar\.gz)$/i, '')
.replace(/^v(\d)/i, '$1'); // Removes 'v' only if followed by a number
// Avoid redundancy if the version string is identical to the repo name
if (version.toLowerCase() === repo.toLowerCase()) {
version = "latest";
}
return `${repo} @ ${version} (${p.id}:${author})`;
}
}
// 2. Fallback for Standard PlatformIO Registry format (e.g., owner/lib @ ^1.0.0)
if (cleanDep.includes('/') || cleanDep.includes('@')) {
const parts = cleanDep.split('@');
const fullName = parts[0].trim(); // Includes owner/name
// Clean up version if present
let version = "latest";
if (parts[1]) {
version = parts[1].trim().replace(/^[\^~=]/, '');
}
// Separate owner and library name for consistent formatting
if (fullName.includes('/')) {
const [owner, libName] = fullName.split('/');
return `${libName.trim()} @ ${version} (pio:${owner.trim()})`;
}
return `${fullName} @ ${version}`;
}
// 3. Return original string if no patterns match
return cleanDep;
}
function rowConfigFromPlatformIO() {
const { execSync } = require('child_process');
try {
const jsonConfig = execSync('pio project config --json-output').toString();
const config = JSON.parse(jsonConfig);
return config;
} catch (error) {
console.error("Make sure PlatformIO Core is installed and in PATH");
throw error;
}
}
function cleanValue(v) {
if (typeof v !== 'string') return v;
return v
.replace(/{/g, '')
.replace(/}/g, '')
.replace(/\$/g, '')
.replace(/env:/g, '')
.replace(/'/g, '')
.replace(/-D/g, '');
}
function convertJsonToSections(jsonConfig) {
const sections = {};
jsonConfig.forEach(([sectionName, configArray]) => {
sections[sectionName] = {};
configArray.forEach(([key, value]) => {
sections[sectionName][key] = value;
});
});
return sections;
}
function cleanLibraries(raw) {
if (!raw) return [];
if (typeof raw === 'string') {
raw = raw.split(',')
}
return raw.map((dep) => smartFormat(dep));
}
function extractModulesFromFlags(flags) {
if (!flags) return [];
let flagArray = [];
if (Array.isArray(flags)) {
flagArray = flags;
} else if (typeof flags === 'string') {
flagArray = flags.split(',').map(s => s.trim()).filter(s => s.length > 0);
} else {
return [];
}
const modules = [];
flagArray.forEach((flag) => {
// Match -DZmoduleName, allowing surrounding quotes
const match = flag.match(/^['" ]*-DZ([^=]+)/);
if (match) {
const moduleName = match[1];
// Additional constraint: must contain 'gateway', 'sensor', or 'actuator'
if (moduleName.includes('gateway') || moduleName.includes('sensor') || moduleName.includes('actuator')) {
modules.push(moduleName);
}
}
//if MQTT_BROKER_MODE = true then modules.push("MQTT Broker Mode");
const brokerMatch = flag.match(/^['" ]*-DMQTT_BROKER_MODE(?:=([^'"\s]+))?/);
if (brokerMatch) {
const value = brokerMatch[1];
// Add only if not explicitly set to false (case insensitive)
if (!value || value.toLowerCase() !== 'false') {
modules.push("MQTT Broker Mode");
}
}
});
return modules;
}
function collectBoardsInformations(sections, { includeTests = false } = {}) {
const rows = [];
Object.entries(sections).forEach(([section, items]) => {
if (!section.includes('env:')) return;
if (!includeTests && section.includes('-test')) return;
const env = section.replace('env:', '');
let uc = '';
let hardware = '';
let description = '';
let modules = [];
let platform = '';
let partitions = '';
let libraries = [];
let options = [];
let customImg = '';
Object.entries(items).forEach(([k, raw]) => {
const v = cleanValue(raw);
if (k === 'board') uc = v;
if (k === 'platform') platform = smartFormat(v);
if (k === 'board_build.partitions') partitions = v;
if (k === 'custom_description') description = v;
if (k === 'custom_hardware') hardware = v;
if (k === 'custom_img') customImg = v;
if (k === 'lib_deps') {
libraries = cleanLibraries(raw);
}
if (k === 'build_flags') {
options = v;
modules = extractModulesFromFlags(v);
}
});
rows.push({
Environment: env,
uC: uc,
Hardware: hardware,
Description: description,
Modules: modules,
Platform: platform,
Partitions: partitions,
Libraries: libraries,
Options: options,
CustomImg: customImg
});
});
rows.sort((a, b) => a.Environment.localeCompare(b.Environment, 'en', { sensitivity: 'base' }));
return rows;
}
function loadBoardsInfo(options = {}) {
const { includeTests = false } = options;
const config = rowConfigFromPlatformIO();
const sections = convertJsonToSections(config);
return collectBoardsInformations(sections, { includeTests });
}
function ensureDir(dir) {
const fs = require('fs');
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
}
module.exports = {
loadBoardsInfo,
ensureDir
};
+77
View File
@@ -0,0 +1,77 @@
// Common templates and constants for web installer manifest generation
// Used by: scripts/gen_wu.js
const mf_temp32 = (vars) => `{
"name": "OpenMQTTGateway",
"new_install_prompt_erase": true,
"builds": [
{
"chipFamily": "ESP32",
"improv": false,
"parts": [
{ "path": "${vars.cp}${vars.bl}", "offset": 4096 },
{ "path": "${vars.cp}${vars.part}", "offset": 32768 },
{ "path": "${vars.cp}${vars.boot}", "offset": 57344 },
{ "path": "${vars.cp}${vars.bin}", "offset": 65536 }
]
}
]
}`;
const mf_temp32c3 = (vars) => `{
"name": "OpenMQTTGateway",
"new_install_prompt_erase": true,
"builds": [
{
"chipFamily": "ESP32-C3",
"improv": false,
"parts": [
{ "path": "${vars.cp}${vars.bl}", "offset": 0 },
{ "path": "${vars.cp}${vars.part}", "offset": 32768 },
{ "path": "${vars.cp}${vars.boot}", "offset": 57344 },
{ "path": "${vars.cp}${vars.bin}", "offset": 65536 }
]
}
]
}`;
const mf_temp32s3 = (vars) => `{
"name": "OpenMQTTGateway",
"new_install_prompt_erase": true,
"builds": [
{
"chipFamily": "ESP32-S3",
"improv": false,
"parts": [
{ "path": "${vars.cp}${vars.bl}", "offset": 0 },
{ "path": "${vars.cp}${vars.part}", "offset": 32768 },
{ "path": "${vars.cp}${vars.boot}", "offset": 57344 },
{ "path": "${vars.cp}${vars.bin}", "offset": 65536 }
]
}
]
}`;
const mf_temp8266 = (vars) => `{
"name": "OpenMQTTGateway",
"new_install_prompt_erase": true,
"builds": [
{
"chipFamily": "ESP8266",
"parts": [{ "path": "${vars.cp}${vars.bin}", "offset": 0 }]
}
]
}`;
const cors_proxy = ''; // 'https://cors.bridged.cc/'
const esp32_boot = 'https://github.com/espressif/arduino-esp32/raw/2.0.7/tools/partitions/boot_app0.bin';
module.exports = {
mf_temp32,
mf_temp32c3,
mf_temp32s3,
mf_temp8266,
cors_proxy,
esp32_boot
};
+349
View File
@@ -0,0 +1,349 @@
#!/usr/bin/env node
// Creates web installer manifests for ESP Web Tools firmware installation
const fs = require('fs');
const path = require('path');
const https = require('https');
const { loadBoardsInfo, ensureDir } = require('./boards-info');
const {
mf_temp32,
mf_temp32c3,
mf_temp32s3,
mf_temp8266,
cors_proxy,
esp32_boot
} = require('./common_wu.js');
// ============================================================================
// Directory Configuration
// ============================================================================
// Base directories
const ROOT_DIR = path.join(__dirname, '..');
const DOCS_DIR = path.join(ROOT_DIR, 'docs');
const VUEPRESS_DIR = path.join(DOCS_DIR, '.vuepress');
const PUBLIC_DIR = path.join(VUEPRESS_DIR, 'public');
const COMPONENTS_DIR = path.join(VUEPRESS_DIR, 'components');
const ARTIFACTS_DIR = path.join(ROOT_DIR, 'generated', 'artifacts');
// Feature-specific directories
const FIRMWARE_SRC_DIR = path.join(ARTIFACTS_DIR, 'firmware_build');
const FIRMWARE_BUILD_DIR = path.join(PUBLIC_DIR, 'firmware_build');
const BOARDS_INFO_FILE = path.join(PUBLIC_DIR, 'boards-info.json');
// Configuration files
const DEFAULTS_CONFIG_PATH = path.join(VUEPRESS_DIR, 'defaults.json');
const META_CONFIG_PATH = path.join(VUEPRESS_DIR, 'meta.json');
let meta = require(DEFAULTS_CONFIG_PATH);
try {
const meta_overload = require(META_CONFIG_PATH);
meta = { ...meta, ...meta_overload };
} catch (e) {
console.warn('meta.json not found or not valid. Using default configuration.');
}
// Parse command line arguments
const args = process.argv.slice(2);
const dev = args.includes('--dev') || meta.mode === 'dev';
const repo = meta.repo || '1technophile/OpenMQTTGateway';
const firmwareManifestFolder = dev ? `/dev/firmware_build/` : `/firmware_build/`;
// ============================================================================
// Utility Functions
// ============================================================================
function ensureFirmwareArtifacts() {
if (!fs.existsSync(FIRMWARE_SRC_DIR)) {
throw new Error(`Missing firmware artifacts in ${FIRMWARE_SRC_DIR}. Run "ci.sh build ..." first to populate this folder.`);
}
console.log(`Found firmware artifacts in: ${FIRMWARE_SRC_DIR}`);
}
// Replace version_tag in template and write to destination
function renderVersionTemplate(templatePath, outputPath, version) {
if (!fs.existsSync(templatePath)) {
throw new Error(`Template not found: ${templatePath}`);
}
const content = fs.readFileSync(templatePath, 'utf8').replace(/version_tag/g, version);
fs.writeFileSync(outputPath, content);
console.log(`Generated version file from template: ${outputPath}`);
}
/**
* Download file from URL
*/
function downloadFile(url) {
return new Promise((resolve, reject) => {
https.get(url, (response) => {
// Handle redirects
if (response.statusCode === 302 || response.statusCode === 301) {
return downloadFile(response.headers.location).then(resolve).catch(reject);
}
if (response.statusCode !== 200) {
return reject(new Error(`Failed to download: ${response.statusCode}`));
}
const chunks = [];
response.on('data', (chunk) => chunks.push(chunk));
response.on('end', () => resolve(Buffer.concat(chunks)));
response.on('error', reject);
}).on('error', reject);
});
}
/**
* Fetch JSON from URL
*/
function fetchJson(url) {
return new Promise((resolve, reject) => {
https.get(url, { headers: { 'User-Agent': 'OpenMQTTGateway-Script' } }, (response) => {
if (response.statusCode !== 200) {
return reject(new Error(`Failed to fetch: ${response.statusCode}`));
}
let data = '';
response.on('data', (chunk) => data += chunk);
response.on('end', () => {
try {
resolve(JSON.parse(data));
} catch (e) {
reject(e);
}
});
response.on('error', reject);
}).on('error', reject);
});
}
/**
* Download and save asset
*/
async function downloadAsset(asset, destPath) {
const buffer = await downloadFile(asset.browser_download_url);
const filename = asset.browser_download_url.split('/').pop();
fs.writeFileSync(path.join(destPath, filename), buffer);
console.log(`Downloaded asset: ${filename} to ${destPath}`);
}
/**
* Create manifest and Vue option for a firmware
* Partition path uses filename only (matches Python; split is redundant but harmless)
*/
function createManifest(name, templateFn) {
const fw = name.split('-firmware')[0];
const man_file = fw + '.manifest.json';
const fwp_name = fw + '-partitions.bin';
const fwb_name = fw + '-bootloader.bin';
// Use filename to mirror Python behavior (no directories present today)
const partPath = fwp_name.split('/').pop();
const mani_str = templateFn({
cp: cors_proxy,
part: firmwareManifestFolder + partPath,
bin: firmwareManifestFolder + name,
bl: firmwareManifestFolder + fwb_name,
boot: firmwareManifestFolder + esp32_boot.split('/').pop()
});
const outPath = path.join(FIRMWARE_BUILD_DIR, man_file);
fs.writeFileSync(outPath, mani_str);
console.log(`Created manifest for ${fw}: ${outPath}`);
}
/**
* Create manifest for ESP8266
* Python adds manif_folder when writing to file, not in return
*/
function createManifest8266(name) {
const fw = name.split('-firmware')[0];
const man_file = fw + '.manifest.json';
const mani_str = mf_temp8266({
cp: cors_proxy,
bin: firmwareManifestFolder + name
});
const outPath = path.join(FIRMWARE_BUILD_DIR, man_file);
fs.writeFileSync(outPath, mani_str);
console.log(`Created manifest for ${fw} (ESP8266): ${outPath}`);
}
/**
* Device type matchers
*/
const ESP32_NAMES = ['esp32', 'ttgo', 'heltec', 'thingpulse', 'theengs', 'lilygo', 'shelly', 'tinypico'];
const ESP8266_NAMES = ['nodemcu', 'sonoff', 'rf-wifi-gateway', 'manual-wifi-test', 'rfbridge'];
const deviceMatchers = {
esp32: (name) => name.includes('firmware.bin') &&
!name.includes('esp32c3') && !name.includes('esp32s3') &&
ESP32_NAMES.some(key => name.includes(key)),
esp32c3: (name) => name.includes('firmware.bin') && name.includes('esp32c3'),
esp32s3: (name) => name.includes('firmware.bin') && name.includes('esp32s3'),
esp8266: (name) => name.includes('firmware.bin') &&
ESP8266_NAMES.some(key => name.includes(key))
};
/**
* Setup dev environment
*/
async function setupDevEnvironment() {
console.log('DEV mode: preparing web upload files...');
ensureFirmwareArtifacts();
// Generate OTA latest version definition from template
const tpl = path.join(__dirname, 'latest_version_dev.json.tpl');
renderVersionTemplate(tpl, path.join(FIRMWARE_BUILD_DIR, 'latest_version_dev.json'), meta.version);
// Copy the binaries from FIRMWARE_SRC_DIR to FIRMWARE_BUILD_DIR
const files = fs.readdirSync(FIRMWARE_SRC_DIR);
let copied = 0;
for (const name of files) {
if (name.includes('.bin')) {
fs.copyFileSync(
path.join(FIRMWARE_SRC_DIR, name),
path.join(FIRMWARE_BUILD_DIR, name)
);
copied++;
console.log(`Copied binary: ${name}`);
}
}
console.log(`Copied ${copied} firmware binaries to ${FIRMWARE_BUILD_DIR}`);
}
/**
* Setup release environment
*/
async function setupReleaseEnvironment() {
console.log('RELEASE mode: downloading and preparing web upload files...');
// Generate OTA latest version definition from template
const tpl = path.join(__dirname, 'latest_version.json.tpl');
renderVersionTemplate(tpl, path.join(FIRMWARE_BUILD_DIR, 'latest_version.json'), meta.version);
const releaseUrl = `https://api.github.com/repos/${repo}/releases/latest`;
console.log(`Fetching latest release info from: ${releaseUrl}`);
const rel_data = await fetchJson(releaseUrl);
if (!rel_data.assets) {
console.error('No assets found in the latest release!');
process.exit(1);
}
// Download all assets
let downloaded = 0;
for (const asset of rel_data.assets) {
const name = asset.name;
if (name.includes('firmware.bin') ||
name.includes('partitions.bin') ||
name.includes('bootloader.bin')) {
await downloadAsset(asset, FIRMWARE_BUILD_DIR);
downloaded++;
}
}
console.log(`Downloaded ${downloaded} firmware assets to ${FIRMWARE_BUILD_DIR}`);
}
/**
* Process firmware files and generate manifests
*/
function processFirmwareFiles(files) {
let manifestCount = 0;
for (const name of files) {
if (deviceMatchers.esp32(name)) {
createManifest(name, mf_temp32);
manifestCount++;
}
if (deviceMatchers.esp32c3(name)) {
createManifest(name, mf_temp32c3);
manifestCount++;
}
if (deviceMatchers.esp32s3(name)) {
createManifest(name, mf_temp32s3);
manifestCount++;
}
if (deviceMatchers.esp8266(name)) {
createManifest8266(name);
manifestCount++;
}
}
console.log(`Generated ${manifestCount} manifest files in ${FIRMWARE_BUILD_DIR}`);
}
/**
* Main execution function
*/
// ===================== OpenMQTTGateway Web Uploader Manifest Generator =====================
// ===================== MAIN SCRIPT STARTS HERE =====================
async function main() {
console.log('================================================================================');
console.log(' OpenMQTTGateway Web Uploader Manifest Generator - START');
console.log('================================================================================');
// === [1] Load and generate boards info ===
console.log('\n[1/4] Generating boards-info.json ...');
const boardsInfo = loadBoardsInfo({ verbose: 0 });
const boardsJson = boardsInfo.map((row) => ({
environment: row.Environment,
hardware: row.Hardware,
description: row.Description,
microcontroller: row.uC,
modules: row.Modules.filter(Boolean),
platform: row.Platform,
partitions: row.Partitions,
libraries: row.Libraries.filter(Boolean),
options: row.Options,
customImg: row.CustomImg
}));
ensureDir(path.dirname(BOARDS_INFO_FILE));
fs.writeFileSync(BOARDS_INFO_FILE, JSON.stringify(boardsJson, null, 2), 'utf8');
console.log(`Generated boards-info.json with ${boardsJson.length} boards: ${BOARDS_INFO_FILE}`);
// === [2] Ensure output directory ===
console.log('\n[2/4] Ensuring output directory ...');
ensureDir(FIRMWARE_BUILD_DIR);
console.log(`Ensured output directory exists: ${FIRMWARE_BUILD_DIR}`);
// === [3] Setup environment (dev or release) ===
console.log('\n[3/4] Preparing firmware files ...');
try {
if (dev) {
await setupDevEnvironment();
} else {
await setupReleaseEnvironment();
}
} catch (error) {
console.error(`Error setting up environment: ${error.message}`);
process.exit(1);
}
// === [4] Download boot binary and generate manifests ===
console.log('\n[4/4] Downloading boot binary and generating manifests ...');
console.log(`Downloading boot binary: ${esp32_boot}`);
const boot_bin = await downloadFile(esp32_boot);
const boot_filename = esp32_boot.split('/').pop();
fs.writeFileSync(path.join(FIRMWARE_BUILD_DIR, boot_filename), boot_bin);
console.log(`Saved boot binary as: ${boot_filename}`);
const files = fs.readdirSync(FIRMWARE_BUILD_DIR).sort();
console.log(`Processing firmware files in ${FIRMWARE_BUILD_DIR}...`);
processFirmwareFiles(files);
console.log('\n================================================================================');
console.log(' OpenMQTTGateway Web Uploader Manifest Generator - END');
console.log('================================================================================');
}
// ===================== MAIN SCRIPT ENDS HERE =====================
// Run main function
main().catch(error => {
console.error('Fatal error:', error);
process.exit(1);
});
File renamed without changes.
File renamed without changes.
+12 -9
View File
@@ -46,7 +46,7 @@ build_flags =
board_build.flash_mode = dout
board_build.ldscript = eagle.flash.1m64.ld ;this frees more space for firmware uplad via OTA.
;extra_scripts = scripts/compressFirmware.py ;uncomment this to compress the firmware. This helps updating e.g. Sonoff RF Bridge via OTA flash by saving space for the uploaded firmware.
custom_description = RF gateway for the Sonoff RF Bridge requiring direct hack, relying on ESPilight library, [tutorial](https://1technophile.blogspot.com/2019/04/sonoff-rf-bridge-pilight-or-how-to.html).
custom_description = 'RF gateway for the Sonoff RF Bridge requiring direct hack, relying on ESPilight library, <a href="https://1technophile.blogspot.com/2019/04/sonoff-rf-bridge-pilight-or-how-to.html" target="_blank">tutorial</a>.'
custom_hardware = RFBridge v1
[env:theengs-bridge]
@@ -84,7 +84,7 @@ build_flags =
'-DGATEWAY_MODEL="TBRIDGE01"'
;'-DBOARD_HAS_PSRAM'
;'-mfix-esp32-psram-cache-issue'
custom_description = BLE gateway with external antenna and Ethernet/WiFi connectivity, [user guide](https://tbridge01.theengs.io/)
custom_description = 'BLE gateway with external antenna and Ethernet/WiFi connectivity, <a href="https://tbridge01.theengs.io/" target="_blank">user guide</a>'
custom_hardware = Theengs Bridge gateway ethernet
[env:theengs-bridge-v11]
@@ -123,8 +123,9 @@ build_flags =
;'-DBOARD_HAS_PSRAM'
;'-DSELF_TEST=true'
;'-mfix-esp32-psram-cache-issue'
custom_description = BLE gateway with external antenna and Ethernet/WiFi connectivity, [user guide](https://tbridge02.theengs.io/)
custom_description = 'BLE gateway with external antenna and Ethernet/WiFi connectivity, <a href="https://tbridge02.theengs.io/" target="_blank">user guide</a>'
custom_hardware = Theengs Bridge gateway ethernet
custom_img = img/Theengs-Bridge-ble-gateway.png
; DISCLAIMER: This is the default environment for Theengs Plug.
; Any modifications to this configuration are not covered by warranty.
@@ -181,8 +182,9 @@ build_flags =
'-DGATEWAY_MANUFACTURER="Theengs"'
'-DGATEWAY_MODEL="TPLUG01"'
'-UZwebUI="WebUI"' ; Disable WebUI
custom_description = Smart Plug, BLE Gateway and energy monitoring, [user guide](https://tplug01.theengs.io/)
custom_description = 'Smart Plug, BLE Gateway and energy monitoring, <a href="https://tplug01.theengs.io/" target="_blank">user guide</a>'
custom_hardware = Theengs Plug
custom_img = img/Theengs-Plug-OpenMQTTGateway.png
[env:esp32dev-all-test]
platform = ${com.esp32_platform}
@@ -517,7 +519,7 @@ build_flags =
; '-DMQTT_SERVER="192.168.1.17"'
; '-Dwifi_ssid="WIFI_SSID"'
; '-Dwifi_password="WIFI_PASSWORD"'
custom_description = Suitable for low power with BLE gateway, [tutorial](https://1technophile.blogspot.com/2021/04/low-power-esp32-ble-gateway.html)
custom_description = 'Suitable for low power with BLE gateway, <a href="https://1technophile.blogspot.com/2021/04/low-power-esp32-ble-gateway.html" target="_blank">tutorial</a>'
custom_hardware = LOLIN 32 Lite
[env:esp32-olimex-gtw-ble-eth]
@@ -1581,7 +1583,8 @@ build_flags =
'-DGateway_Name="OMG_AVATTO_IR"'
board_build.flash_mode = dout
custom_description = IR gateway bi directional
custom_hardware = Avatto Bakey IR first version, [tutorial](https://1technophile.blogspot.com/2020/07/avatto-s06-ir-gateway-compatible-with.html)
custom_hardware = 'Avatto Bakey IR first version,<a href="https://1technophile.blogspot.com/2020/07/avatto-s06-ir-gateway-compatible-with.html" target="_blank">tutorial</a>'
custom_img = https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWdLlg_I1Fxg8PKa0LNCwj3fDtSCVn50Zqima9QZvJfPpIyr3Rh7cvg_WPzRkZpzP4_Tu9inXfo3e6CbLLLpZzo5yGOUX_zFT0CnopCtyuEgJyHHJLP8ctfm1UyHP3KZJGzRbZul4F9JBX/
[env:nodemcuv2-rf]
platform = ${com.esp8266_platform}
@@ -1664,7 +1667,7 @@ build_flags =
'-DRF_RECEIVER_GPIO=5'
board_build.flash_mode = dout
custom_description = RF gateway for USB stick using RCSwitch
custom_hardware = RF Wifi USB stick, [tutorial](https://1technophile.blogspot.com/2019/09/hack-of-rf-wifi-gateway-usb-stick.html)
custom_hardware = 'RF Wifi USB stick, <a href="https://1technophile.blogspot.com/2019/09/hack-of-rf-wifi-gateway-usb-stick.html" target="_blank">tutorial</a>'
[env:nodemcuv2-rf2]
platform = ${com.esp8266_platform}
@@ -1773,8 +1776,8 @@ build_flags =
'-DZgatewayRF="RF"'
'-DGateway_Name="OMG_SONOFF_BASIC_RFR3"'
board_build.flash_mode = dout
custom_description = Wifi relay and RF receiver using RCSwitch library, [tutorial](https://1technophile.blogspot.com/2019/08/new-sonoff-rfr3-as-433tomqtt-gateway.html)
custom_hardware = Sonoff Basic RFR3, [tutorial](https://1technophile.blogspot.com/2019/08/new-sonoff-rfr3-as-433tomqtt-gateway.html)
custom_description = 'Wifi relay and RF receiver using RCSwitch library, <a href="https://1technophile.blogspot.com/2019/08/new-sonoff-rfr3-as-433tomqtt-gateway.html" target="_blank">tutorial</a>'
custom_hardware = 'Sonoff Basic RFR3, <a href="https://1technophile.blogspot.com/2019/08/new-sonoff-rfr3-as-433tomqtt-gateway.html" target="_blank">tutorial</a>'
[env:esp32dev-ble-datatest]
platform = ${com.esp32_platform}
@@ -1,16 +1,16 @@
#include <Adafruit_Sensor.h>
#include <DHT.h>
#include <DHT_U.h>
#include <LoRa.h>
#include <SPI.h>
#include <WiFi.h>
#include <Wire.h>
#include <stdio.h>
#include <Adafruit_Sensor.h>
#include <DHT.h>
#include <DHT_U.h>
#include "SSD1306.h"
//LoRa pins
#define SCK 5 // GPIO5 -- SX1278's SCK
#define SCK 5 // GPIO5 -- SX1278's SCK
#define MISO 19 // GPIO19 -- SX1278's MISnO
#define MOSI 27 // GPIO27 -- SX1278's MOSI
#define SS 18 // GPIO18 -- SX1278's CS
@@ -23,14 +23,14 @@
#define OLED_SCL 22
//#define OLED_RST 16
#define LED 25
#define LED 25
#define DHT_PIN 14
#define DHT_TYPE DHT22
#define CNT_PIN 34
#define CNT_PIN 34
#define BAT_PIN 35
#define BAT_PIN 35
const int MAX_ANALOG_VAL = 4095;
const float MAX_BATTERY_VOLTAGE = 4.2;
@@ -38,7 +38,7 @@ const float MAX_BATTERY_VOLTAGE = 4.2;
int cntpkt = 0;
unsigned long counter = 0;
unsigned long last_interrupt_time = 0;
int pulse_seen = 0; // pulse detected by interrupt function
int pulse_seen = 0; // pulse detected by interrupt function
int bounce_delay_ms = 100;
SSD1306 display(0x3c, OLED_SDA, OLED_SCL);
@@ -46,9 +46,9 @@ String rssi = "RSSI --";
String packSize = "--";
String packet;
DHT_Unified dht(DHT_PIN, DHT_TYPE); //Inizializza oggetto chiamato "dht", parametri: pin a cui è connesso il sensore, tipo di dht 11/22
DHT_Unified dht(DHT_PIN, DHT_TYPE); // Initialize the "dht" object with sensor pin and DHT type (11/22)
void ICACHE_RAM_ATTR bounceCheck ();
void ICACHE_RAM_ATTR bounceCheck();
void setup() {
pinMode(LED, OUTPUT);
@@ -78,29 +78,47 @@ void setup() {
pinMode(BAT_PIN, INPUT);
pinMode(CNT_PIN, INPUT_PULLUP);
attachInterrupt (CNT_PIN, bounceCheck, RISING);
attachInterrupt(CNT_PIN, bounceCheck, RISING);
dht.begin();
sensor_t sensor;
dht.temperature().getSensor(&sensor);
Serial.println(F("------------------------------------"));
Serial.println(F("Temperature Sensor"));
Serial.print (F("Sensor Type: ")); Serial.println(sensor.name);
Serial.print (F("Driver Ver: ")); Serial.println(sensor.version);
Serial.print (F("Unique ID: ")); Serial.println(sensor.sensor_id);
Serial.print (F("Max Value: ")); Serial.print(sensor.max_value); Serial.println(F("°C"));
Serial.print (F("Min Value: ")); Serial.print(sensor.min_value); Serial.println(F("°C"));
Serial.print (F("Resolution: ")); Serial.print(sensor.resolution); Serial.println(F("°C"));
Serial.print(F("Sensor Type: "));
Serial.println(sensor.name);
Serial.print(F("Driver Ver: "));
Serial.println(sensor.version);
Serial.print(F("Unique ID: "));
Serial.println(sensor.sensor_id);
Serial.print(F("Max Value: "));
Serial.print(sensor.max_value);
Serial.println(F("°C"));
Serial.print(F("Min Value: "));
Serial.print(sensor.min_value);
Serial.println(F("°C"));
Serial.print(F("Resolution: "));
Serial.print(sensor.resolution);
Serial.println(F("°C"));
Serial.println(F("------------------------------------"));
// Print humidity sensor details.
dht.humidity().getSensor(&sensor);
Serial.println(F("Humidity Sensor"));
Serial.print (F("Sensor Type: ")); Serial.println(sensor.name);
Serial.print (F("Driver Ver: ")); Serial.println(sensor.version);
Serial.print (F("Unique ID: ")); Serial.println(sensor.sensor_id);
Serial.print (F("Max Value: ")); Serial.print(sensor.max_value); Serial.println(F("%"));
Serial.print (F("Min Value: ")); Serial.print(sensor.min_value); Serial.println(F("%"));
Serial.print (F("Resolution: ")); Serial.print(sensor.resolution); Serial.println(F("%"));
Serial.print(F("Sensor Type: "));
Serial.println(sensor.name);
Serial.print(F("Driver Ver: "));
Serial.println(sensor.version);
Serial.print(F("Unique ID: "));
Serial.println(sensor.sensor_id);
Serial.print(F("Max Value: "));
Serial.print(sensor.max_value);
Serial.println(F("%"));
Serial.print(F("Min Value: "));
Serial.print(sensor.min_value);
Serial.println(F("%"));
Serial.print(F("Resolution: "));
Serial.print(sensor.resolution);
Serial.println(F("%"));
Serial.println(F("------------------------------------"));
delay(1500);
@@ -117,15 +135,14 @@ void loop() {
display.drawString(90, 0, String(cntpkt));
String NodeId = WiFi.macAddress();
// float temp = dht.readTemperature();
// float hum = dht.readHumidity();
// float temp = dht.readTemperature();
// float hum = dht.readHumidity();
float battery = getBattery();
dht.temperature().getEvent(&event1);
if (isnan(event1.temperature)) {
Serial.println(F("Error reading temperature!"));
}
else {
} else {
Serial.print(F("Temperature: "));
Serial.print(event1.temperature);
Serial.println(F("°C"));
@@ -134,8 +151,7 @@ void loop() {
dht.humidity().getEvent(&event2);
if (isnan(event2.relative_humidity)) {
Serial.println(F("Error reading humidity!"));
}
else {
} else {
Serial.print(F("Humidity: "));
Serial.print(event2.relative_humidity);
Serial.println(F("%"));
@@ -153,7 +169,7 @@ void loop() {
display.drawString(0, 15, String(NodeId));
display.drawString(0, 30, "count: " + String(counter));
display.drawString(0, 45, "battery: " + String(battery)+ " %");
display.drawString(0, 45, "battery: " + String(battery) + " %");
display.display();
delay(5000);
@@ -167,10 +183,10 @@ void loop() {
delay(6000);
}
void ICACHE_RAM_ATTR bounceCheck (){
unsigned long interrupt_time = millis();
if (interrupt_time - last_interrupt_time > bounce_delay_ms) counter++; // void loop() then notes pulse == 1 and takes action
last_interrupt_time = interrupt_time;
void ICACHE_RAM_ATTR bounceCheck() {
unsigned long interrupt_time = millis();
if (interrupt_time - last_interrupt_time > bounce_delay_ms) counter++; // void loop() then notes pulse == 1 and takes action
last_interrupt_time = interrupt_time;
}
float getBattery(void) {
@@ -179,6 +195,5 @@ float getBattery(void) {
float voltageLevel = (rawValue / 4095.0) * 2 * 1.1 * 3.3; // calculate voltage level
float perc = voltageLevel / MAX_BATTERY_VOLTAGE * 100;
return(perc);
return (perc);
}
+14 -15
View File
@@ -22,14 +22,14 @@
#define OLED_SCL 22
//#define OLED_RST 16
#define LED 25
#define LED 25
#define DHT_PIN 12
#define DHT_TYPE DHT22
#define CNT_PIN 34
#define CNT_PIN 34
#define BAT_PIN 35
#define BAT_PIN 35
const int MAX_ANALOG_VAL = 4095;
const float MAX_BATTERY_VOLTAGE = 4.2;
@@ -37,7 +37,7 @@ const float MAX_BATTERY_VOLTAGE = 4.2;
int cntpkt = 0;
unsigned long counter = 0;
unsigned long last_interrupt_time = 0;
int pulse_seen = 0; // pulse detected by interrupt function
int pulse_seen = 0; // pulse detected by interrupt function
int bounce_delay_ms = 100;
SSD1306 display(0x3c, OLED_SDA, OLED_SCL);
@@ -45,9 +45,9 @@ String rssi = "RSSI --";
String packSize = "--";
String packet;
DHT dht(DHT_PIN, DHT_TYPE); //Inizializza oggetto chiamato "dht", parametri: pin a cui è connesso il sensore, tipo di dht 11/22
DHT dht(DHT_PIN, DHT_TYPE); // Initialize the "dht" object with sensor pin and DHT type (11/22)
void ICACHE_RAM_ATTR bounceCheck ();
void ICACHE_RAM_ATTR bounceCheck();
void setup() {
pinMode(LED, OUTPUT);
@@ -77,7 +77,7 @@ void setup() {
pinMode(BAT_PIN, INPUT);
pinMode(CNT_PIN, INPUT_PULLUP);
attachInterrupt (CNT_PIN, bounceCheck, RISING);
attachInterrupt(CNT_PIN, bounceCheck, RISING);
delay(1500);
}
@@ -97,7 +97,7 @@ void loop() {
// send packet
LoRa.beginPacket();
// Build json string to send
//per test rimuovere
// Remove this line for test
String msg = "{\"model\":\"ESP32CNT\",\"id\":\"" + NodeId + "\",\"count\":\"" + String(counter) + "\",\"tempc\":\"" + String(temp) + "\",\"hum\":\"" + String(hum) + "\",\"batt\":\"" + String(battery) + "\"}";
// Send json string
@@ -108,7 +108,7 @@ void loop() {
display.drawString(0, 15, String(NodeId));
display.drawString(0, 30, "count: " + String(counter));
display.drawString(0, 45, "battery: " + String(battery)+ " %");
display.drawString(0, 45, "battery: " + String(battery) + " %");
display.display();
delay(5000);
@@ -121,10 +121,10 @@ void loop() {
delay(60000); // wait for 60 seconds
}
void ICACHE_RAM_ATTR bounceCheck (){
unsigned long interrupt_time = millis();
if (interrupt_time - last_interrupt_time > bounce_delay_ms) counter++; // void loop() then notes pulse == 1 and takes action
last_interrupt_time = interrupt_time;
void ICACHE_RAM_ATTR bounceCheck() {
unsigned long interrupt_time = millis();
if (interrupt_time - last_interrupt_time > bounce_delay_ms) counter++; // void loop() then notes pulse == 1 and takes action
last_interrupt_time = interrupt_time;
}
float getBattery(void) {
@@ -133,6 +133,5 @@ float getBattery(void) {
float voltageLevel = (rawValue / 4095.0) * 2 * 1.1 * 3.3; // calculate voltage level
float perc = voltageLevel / MAX_BATTERY_VOLTAGE * 100;
return(perc);
return (perc);
}
+13
View File
@@ -92,6 +92,19 @@ extern bool cmpToMainTopic(const char*, const char*);
extern bool pub(const char*, const char*, bool);
extern bool pub(const char*, const char*);
// Float-specific overload to log with the correct format specifier.
inline void Config_update(JsonObject& data, const char* key, float& var) {
if (data.containsKey(key)) {
float newVal = data[key].as<float>();
if (var != newVal) {
var = newVal;
THEENGS_LOG_NOTICE(F("Config %s changed to: %F" CR), key, newVal);
} else {
THEENGS_LOG_NOTICE(F("Config %s unchanged, currently: %F" CR), key, newVal);
}
}
}
template <typename T>
void Config_update(JsonObject& data, const char* key, T& var) {
if (data.containsKey(key)) {
+1 -1
View File
@@ -29,7 +29,7 @@
#define user_config_h
/*-------------------VERSION----------------------*/
#ifndef OMG_VERSION
# define OMG_VERSION "version_tag"
# define OMG_VERSION "edge"
#endif
/*-------------CONFIGURE WIFIMANAGER-------------(only ESP8266 & SONOFF RFBridge)*/
+4 -4
View File
@@ -49,9 +49,9 @@ static NimBLEOta* pNimBLEOta;
static NimBLECharacteristic* pCommandCharacteristic;
static NimBLECharacteristic* pRecvFwCharacteristic;
#ifndef BLUFI_MFG_ID
# define BLUFI_MFG_ID 0xFFFF // Default Manufacturer ID if not defined
#endif
# ifndef BLUFI_MFG_ID
# define BLUFI_MFG_ID 0xFFFF // Default Manufacturer ID if not defined
# endif
struct pkt_info {
uint8_t* pkt;
@@ -201,7 +201,7 @@ void restart_connection_timer() {}
void stop_connection_timer() {}
# endif
void set_blufi_mfg_data () {
void set_blufi_mfg_data() {
if (!NimBLEDevice::isInitialized() || !NimBLEDevice::getAdvertising()->isAdvertising()) {
THEENGS_LOG_NOTICE(F("Unable to set advertising data" CR));
return;
+52 -14
View File
@@ -40,6 +40,7 @@ extern rtl_433_ESP rtl_433;
# endif
int currentReceiver = ACTIVE_NONE;
boolean isDriverEnabled = false;
extern void enableActiveReceiver();
extern void disableCurrentReceiver();
@@ -53,43 +54,63 @@ public:
ZCommonRFWrapper() : RFReceiver() {}
void enable() override { enableActiveReceiver(); }
void disable() override { disableCurrentReceiver(); }
int getReceiverID() const override { return currentReceiver; }
};
ZCommonRFWrapper iRFReceiver;
RFConfiguration iRFConfig(iRFReceiver);
//TODO review
// Initialize the CC1101 and tune the RX frequency.
// - Uses truncated exponential backoff to avoid tight retry loops
// - Validates the configured frequency before touching the radio
// - Emits explicit logs for success/failure so watchdog resets are traceable
void initCC1101() {
# ifdef ZradioCC1101 //receiving with CC1101
// Loop on getCC1101() until it returns true and break after 10 attempts
# ifdef ZradioCC1101 // receiving with CC1101
const float freqMhz = iRFConfig.getFrequency();
if (!iRFConfig.validFrequency(freqMhz)) {
THEENGS_LOG_ERROR(F("C1101 invalid frequency: %F MHz" CR), freqMhz);
return;
}
int delayMS = 16;
int delayMaxMS = 500;
for (int i = 0; i < 10; i++) {
# if defined(RF_MODULE_SCK) && defined(RF_MODULE_MISO) && \
defined(RF_MODULE_MOSI) && defined(RF_MODULE_CS)
ELECHOUSE_cc1101.setSpiPin(RF_MODULE_SCK, RF_MODULE_MISO, RF_MODULE_MOSI, RF_MODULE_CS);
bool connected = false;
for (int attempt = 1; attempt <= 10; attempt++) {
# if defined(RF_CC1101_SCK) && defined(RF_CC1101_MISO) && \
defined(RF_CC1101_MOSI) && defined(RF_CC1101_CS)
THEENGS_LOG_TRACE(F("initCC1101 with custom SPI pins, SCK=%d, MISO=%d, MOSI=%d, CS=%d" CR), RF_CC1101_SCK, RF_CC1101_MISO, RF_CC1101_MOSI, RF_CC1101_CS);
ELECHOUSE_cc1101.setSpiPin(RF_CC1101_SCK, RF_CC1101_MISO, RF_CC1101_MOSI, RF_CC1101_CS);
# endif
if (ELECHOUSE_cc1101.getCC1101()) {
THEENGS_LOG_NOTICE(F("C1101 spi Connection OK" CR));
connected = true;
THEENGS_LOG_NOTICE(F("C1101 SPI connection OK on attempt %d" CR), attempt);
ELECHOUSE_cc1101.Init();
ELECHOUSE_cc1101.SetRx(iRFConfig.getFrequency());
ELECHOUSE_cc1101.SetRx(freqMhz);
THEENGS_LOG_NOTICE(F("C1101 tuned RX to %F MHz" CR), freqMhz);
break;
} else {
THEENGS_LOG_ERROR(F("C1101 spi Connection Error" CR));
delay(delayMS);
}
THEENGS_LOG_ERROR(F("C1101 SPI connection error (attempt %d), retrying" CR), attempt);
delay(delayMS);
// truncated exponential backoff
delayMS = delayMS * 2;
if (delayMS > delayMaxMS) delayMS = delayMaxMS;
}
if (!connected) {
THEENGS_LOG_ERROR(F("C1101 init failed after retries, radio left disabled" CR));
}
# else
THEENGS_LOG_TRACE(F("initCC1101 skipped: ZradioCC1101 not enabled" CR));
# endif
}
void setupCommonRF() {
iRFConfig.reInit();
iRFConfig.loadFromStorage();
iRFConfig.loadFromStorage(true);
}
# if !defined(ZgatewayRFM69) && !defined(ZactuatorSomfy)
@@ -120,28 +141,37 @@ bool validReceiver(int receiver) {
# endif
void disableCurrentReceiver() {
if (!isDriverEnabled) {
THEENGS_LOG_TRACE(F("Receiver %d is already disabled" CR), currentReceiver);
return;
}
THEENGS_LOG_TRACE(F("disableCurrentReceiver: %d" CR), currentReceiver);
switch (currentReceiver) {
case ACTIVE_NONE:
isDriverEnabled = false;
break;
# ifdef ZgatewayPilight
case ACTIVE_PILIGHT:
disablePilightReceive();
isDriverEnabled = false;
break;
# endif
# ifdef ZgatewayRF
case ACTIVE_RF:
disableRFReceive();
isDriverEnabled = false;
break;
# endif
# ifdef ZgatewayRTL_433
case ACTIVE_RTL:
disableRTLreceive();
isDriverEnabled = false;
break;
# endif
# ifdef ZgatewayRF2
case ACTIVE_RF2:
disableRF2Receive();
isDriverEnabled = false;
break;
# endif
default:
@@ -150,6 +180,10 @@ void disableCurrentReceiver() {
}
void enableActiveReceiver() {
if (isDriverEnabled) {
THEENGS_LOG_TRACE(F("Receiver %d is already enabled" CR), currentReceiver);
return;
}
THEENGS_LOG_TRACE(F("enableActiveReceiver: %d" CR), iRFConfig.getActiveReceiver());
switch (iRFConfig.getActiveReceiver()) {
# ifdef ZgatewayPilight
@@ -157,6 +191,7 @@ void enableActiveReceiver() {
initCC1101();
enablePilightReceive();
currentReceiver = ACTIVE_PILIGHT;
isDriverEnabled = true;
break;
# endif
# ifdef ZgatewayRF
@@ -164,6 +199,7 @@ void enableActiveReceiver() {
initCC1101();
enableRFReceive(iRFConfig.getFrequency(), RF_RECEIVER_GPIO, RF_EMITTER_GPIO);
currentReceiver = ACTIVE_RF;
isDriverEnabled = true;
break;
# endif
# ifdef ZgatewayRTL_433
@@ -171,6 +207,7 @@ void enableActiveReceiver() {
initCC1101();
enableRTLreceive();
currentReceiver = ACTIVE_RTL;
isDriverEnabled = true;
break;
# endif
# ifdef ZgatewayRF2
@@ -178,6 +215,7 @@ void enableActiveReceiver() {
initCC1101();
enableRF2Receive();
currentReceiver = ACTIVE_RF2;
isDriverEnabled = true;
break;
# endif
case ACTIVE_RECERROR:
+1 -1
View File
@@ -190,7 +190,7 @@ extern RFConfiguration iRFConfig;
/*-------------------RF frequency----------------------*/
//Match frequency to the hardware version of the radio used.
#ifndef RF_FREQUENCY
# define RF_FREQUENCY 433.92
# define RF_FREQUENCY 433.92f
#endif
/**
+8 -8
View File
@@ -337,20 +337,20 @@ void BM2_connect::publishData() {
static const unsigned char BM6_AES_KEY[16] = {
108, // l
101, // e
97, // a
97, // a
103, // g
101, // e
110, // n
100, // d
255, // 0xff
254, // 0xfe
48, // 0
49, // 1
48, // 0
48, // 0
48, // 0
48, // 0
57, // 9
48, // 0
49, // 1
48, // 0
48, // 0
48, // 0
48, // 0
57, // 9
};
void BM6_connect::notifyCB(NimBLERemoteCharacteristic* pChar, uint8_t* pData, size_t length, bool isNotify) {
+2 -2
View File
@@ -1025,10 +1025,10 @@ void launchBTDiscovery(bool overrideDiscovery) {
// This should not happen if JSON_MSG_BUFFER is large enough for
// the Theengs json properties
THEENGS_LOG_ERROR(F("JSON deserialization of Theengs properties overflowed (error %s), buffer capacity: %u. Program might crash. Properties json: %s" CR),
error.c_str(), jsonBuffer.capacity(), properties.c_str());
error.c_str(), jsonBuffer.capacity(), properties.c_str());
} else {
THEENGS_LOG_ERROR(F("JSON deserialization of Theengs properties errored: %" CR),
error.c_str());
error.c_str());
}
}
for (JsonPair prop : jsonBuffer["properties"].as<JsonObject>()) {
+2 -2
View File
@@ -402,7 +402,7 @@ void enableRFReceive(
float rfFrequency = iRFConfig.getFrequency(),
int rfReceiverGPIO = RF_RECEIVER_GPIO,
int rfEmitterGPIO = RF_EMITTER_GPIO) {
THEENGS_LOG_NOTICE(F("[RF] Enable RF Receiver: %fMhz, RF_EMITTER_GPIO: %d, RF_RECEIVER_GPIO: %d" CR), rfFrequency, rfEmitterGPIO, rfReceiverGPIO);
THEENGS_LOG_NOTICE(F("[RF] Enable RF Receiver: %F Mhz, RF_EMITTER_GPIO: %d, RF_RECEIVER_GPIO: %d" CR), rfFrequency, rfEmitterGPIO, rfReceiverGPIO);
# ifdef RF_DISABLE_TRANSMIT
mySwitch.disableTransmit();
@@ -410,7 +410,7 @@ void enableRFReceive(
mySwitch.enableTransmit(rfEmitterGPIO);
# endif
mySwitch.setRepeatTransmit(rfEmitterGPIO);
mySwitch.setRepeatTransmit(RF_EMITTER_REPEAT);
mySwitch.enableReceive(rfReceiverGPIO);
THEENGS_LOG_TRACE(F("[RF] Setup command topic: %s%s%s\n Setup done" CR), (const char*)mqtt_topic, (const char*)gateway_name, (const char*)subjectMQTTtoRF);
+1 -1
View File
@@ -316,7 +316,7 @@ void sendMQTTfromNestedJson(JsonVariant obj, char* topic, int level, int maxLeve
topic[topicLength] = '\0';
} else {
THEENGS_LOG_ERROR(F("Nested key '%s' at level %d does not fit within max topic length of %d, skipping"),
key, level, mqtt_topic_max_size);
key, level, mqtt_topic_max_size);
}
}
+2 -2
View File
@@ -96,8 +96,8 @@ char gateway_name[parameters_size + 1] = Gateway_Name;
unsigned long lastDiscovery = 0;
#if BLEDecryptor
char ble_aes[parameters_size] = BLE_AES;
StaticJsonDocument<JSON_BLE_AES_CUSTOM_KEYS> ble_aes_keys;
char ble_aes[parameters_size] = BLE_AES;
StaticJsonDocument<JSON_BLE_AES_CUSTOM_KEYS> ble_aes_keys;
#endif
#if !MQTT_BROKER_MODE
+1 -1
View File
@@ -600,7 +600,7 @@ void createDiscovery(const char* sensor_type,
}
}
if (diagnostic_entity) { // entity_category
if (diagnostic_entity) { // entity_category
sensor["ent_cat"] = "diagnostic";
}
+19 -22
View File
@@ -65,6 +65,7 @@ void RFConfiguration::reInit() {
activeReceiver = ACTIVE_RECEIVER;
rssiThreshold = 0;
newOokThreshold = 0;
THEENGS_LOG_TRACE(F("RFConfiguration reInit: frequency=%F, activeReceiver=%d" CR), frequency, activeReceiver);
}
/**
@@ -82,13 +83,13 @@ void RFConfiguration::eraseStorage() {
preferences.begin(Gateway_Short_Name, false);
if (preferences.isKey("RFConfig")) {
int result = preferences.remove("RFConfig");
Log.notice(F("RF config erase result: %d" CR), result);
THEENGS_LOG_NOTICE(F("RF config erase result: %d" CR), result);
} else {
Log.notice(F("RF config not found" CR));
THEENGS_LOG_NOTICE(F("RF config not found" CR));
}
preferences.end();
#else
Log.warning(F("RF Config Erase not support with this board" CR));
THEENGS_LOG_WARNING(F("RF Config Erase not support with this board" CR));
#endif
}
@@ -120,9 +121,9 @@ void RFConfiguration::saveOnStorage() {
preferences.begin(Gateway_Short_Name, false);
int result = preferences.putString("RFConfig", conf);
preferences.end();
Log.notice(F("RF Config_save: %s, result: %d" CR), conf.c_str(), result);
THEENGS_LOG_NOTICE(F("RF Config_save: %s, result: %d" CR), conf.c_str(), result);
#else
Log.warning(F("RF Config_save not support with this board" CR));
THEENGS_LOG_WARNING(F("RF Config_save not support with this board" CR));
#endif
}
@@ -149,24 +150,25 @@ void RFConfiguration::loadFromStorage(bool reinitReceiver) {
auto error = deserializeJson(jsonBuffer, preferences.getString("RFConfig", "{}"));
preferences.end();
if (error) {
Log.error(F("RF Config deserialization failed: %s, buffer capacity: %u" CR), error.c_str(), jsonBuffer.capacity());
THEENGS_LOG_ERROR(F("RF Config deserialization failed: %s, buffer capacity: %u" CR), error.c_str(), jsonBuffer.capacity());
return;
}
if (jsonBuffer.isNull()) {
Log.warning(F("RF Config is null" CR));
THEENGS_LOG_WARNING(F("RF Config is null" CR));
return;
}
JsonObject jo = jsonBuffer.as<JsonObject>();
fromJson(jo);
Log.notice(F("RF Config loaded" CR));
THEENGS_LOG_NOTICE(F("RF Config loaded" CR));
} else {
preferences.end();
Log.notice(F("RF Config not found using default" CR));
THEENGS_LOG_NOTICE(F("RF Config not found using default" CR));
}
#endif
// Disable and re-enable the receiver to ensure proper initialization
if (reinitReceiver) {
iRFReceiver.disable();
delay(100);
iRFReceiver.enable();
}
}
@@ -209,10 +211,10 @@ void RFConfiguration::loadFromMessage(JsonObject& RFdata) {
if (RFdata.containsKey("erase") && RFdata["erase"].as<bool>()) {
eraseStorage();
Log.notice(F("RF Config erased" CR));
THEENGS_LOG_NOTICE(F("RF Config erased" CR));
} else if (RFdata.containsKey("save") && RFdata["save"].as<bool>()) {
saveOnStorage();
Log.notice(F("RF Config saved" CR));
THEENGS_LOG_NOTICE(F("RF Config saved" CR));
}
}
@@ -245,17 +247,17 @@ void RFConfiguration::fromJson(JsonObject& RFdata) {
if (RFdata.containsKey("frequency") && validFrequency(RFdata["frequency"])) {
Config_update(RFdata, "frequency", frequency);
Log.notice(F("RF Receive mhz: %F" CR), frequency);
THEENGS_LOG_NOTICE(F("RF Receive mhz: %F" CR), frequency);
success = true;
}
if (RFdata.containsKey("active")) {
Config_update(RFdata, "active", activeReceiver);
Log.notice(F("RF receiver active: %d" CR), activeReceiver);
THEENGS_LOG_NOTICE(F("RF receiver active: %d" CR), activeReceiver);
success = true;
}
#ifdef ZgatewayRTL_433
if (RFdata.containsKey("rssithreshold")) {
Log.notice(F("RTL_433 RSSI Threshold : %d " CR), rssiThreshold);
THEENGS_LOG_NOTICE(F("RTL_433 RSSI Threshold : %d " CR), rssiThreshold);
Config_update(RFdata, "rssithreshold", rssiThreshold);
rtl_433.setRSSIThreshold(rssiThreshold);
success = true;
@@ -263,18 +265,18 @@ void RFConfiguration::fromJson(JsonObject& RFdata) {
# if defined(RF_SX1276) || defined(RF_SX1278)
if (RFdata.containsKey("ookthreshold")) {
Config_update(RFdata, "ookthreshold", newOokThreshold);
Log.notice(F("RTL_433 ookThreshold %d" CR), newOokThreshold);
THEENGS_LOG_NOTICE(F("RTL_433 ookThreshold %d" CR), newOokThreshold);
rtl_433.setOOKThreshold(newOokThreshold);
success = true;
}
# endif
if (RFdata.containsKey("status")) {
Log.notice(F("RF get status:" CR));
THEENGS_LOG_NOTICE(F("RF get status:" CR));
rtl_433.getStatus();
success = true;
}
if (!success) {
Log.error(F("MQTTtoRF Fail json" CR));
THEENGS_LOG_ERROR(F("MQTTtoRF Fail json" CR));
}
#endif
}
@@ -307,11 +309,6 @@ void RFConfiguration::toJson(JsonObject& RFdata) {
RFdata["rssithreshold"] = rssiThreshold;
RFdata["ookthreshold"] = newOokThreshold;
RFdata["active"] = activeReceiver;
// Add white-list vector to the JSON object
JsonArray whiteListArray = RFdata.createNestedArray("white-list");
// Add black-list vector to the JSON object
JsonArray blackListArray = RFdata.createNestedArray("black-list");
}
/**
+4 -4
View File
@@ -92,10 +92,10 @@ void setupZsensorDS1820() {
}
ds1820_resolution[ds1820_count] = ds1820.getResolution(ds1820_address);
THEENGS_LOG_TRACE(F("DS1820: Device %d, Type: %s, Address: %s, Resolution: %d" CR),
ds1820_count,
(char*)ds1820_type[ds1820_count].c_str(),
(char*)ds1820_addr[ds1820_count].c_str(),
ds1820_resolution[ds1820_count]);
ds1820_count,
(char*)ds1820_type[ds1820_count].c_str(),
(char*)ds1820_addr[ds1820_count].c_str(),
ds1820_resolution[ds1820_count]);
ds1820_count++;
}
}
+369 -261
View File
File diff suppressed because it is too large. Load diff
+10 -2
View File
@@ -1,12 +1,20 @@
{
"scripts": {
"docs:dev": "vuepress dev docs",
"docs:build": "vuepress build docs"
"gen:ssl": "node scripts/ensure_ssl_certs.js",
"gen:metadata": "node docsgen/gen_wu.js",
"gen:site:only": "vuepress build docs",
"docs:dev": "npm run gen:metadata && vuepress dev docs",
"docs:build": "npm run gen:metadata && npm run gen:site:only",
"site:preview": " npm run gen:ssl && node scripts/preview_site.js",
"site:preview:full": "npm run docs:build && npm run gen:ssl && node scripts/preview_site.js"
},
"devDependencies": {
"vuepress": "^1.9.10"
},
"dependencies": {
"ini": "^4.1.1",
"markdown-table": "^3.0.3",
"mime-types": "^2.1.35",
"vuepress-plugin-sitemap": "^2.3.1"
}
}
+427 -212
View File
@@ -11,11 +11,13 @@ This documentation describes the CI/CD scripts used to build OpenMQTTGateway fir
- [ci.sh - Main Entry Point](#commands)
- [ci.sh build - Build Firmware](#cish-build---build-firmware)
- [ci.sh site - Build Documentation](#cish-site---build-documentation)
- [ci.sh qa - Code Formatting Check](#cish-qa---code-formatting-check)
- [ci.sh qa - Code Quality Checks](#cish-qa---code-quality-checks)
- [ci.sh security - Security Vulnerability Scan](#cish-security---security-vulnerability-scan)
- [Internal Scripts](#internal-scripts)
- [ci_set_version.sh](#ci_set_versionsh)
- [ci_list-env.sh](#ci_list-envsh)
- [ci_build_firmware.sh](#ci_build_firmwaresh)
- [ci_prepare_artifacts.sh](#ci_prepare_artifactssh)
- [ci_security.sh](#ci_securitysh)
- [ci_00_config.sh](#ci_00_configsh)
- [Python Helper Scripts](#python-helper-scripts)
- [generate_board_docs.py](#generate_board_docspy)
@@ -23,6 +25,7 @@ This documentation describes the CI/CD scripts used to build OpenMQTTGateway fir
- [Environment Variables](#environment-variables)
- [Exit Codes](#exit-codes)
- [Environment Detection](#environment-detection)
- [GitHub Actions Workflows Integration](#github-actions-workflows-integration)
@@ -32,27 +35,28 @@ This documentation describes the CI/CD scripts used to build OpenMQTTGateway fir
```
ci.sh (dispatcher)
├── build → ci_build.sh → ci_set_version.sh
│ → ci_build_firmware.sh
│ → ci_prepare_artifacts.sh
├── site → ci_site.sh → generate_board_docs.py
│ → gen_wu.py
├── qa → ci_qa.sh → clang-format
└── all → qa + build + site (sequential)
├── build → ci_build.sh → ci_build_firmware.sh
│ → ci_prepare_artifacts.sh (when --deploy-ready)
├── site → ci_site.sh
├── qa → ci_qa.sh
├── security → ci_security.sh (vulnerability scanning with Trivy)
├── list-env → ci_list-env.sh
└── all → qa + build (all envs with --mode) + site sequential
```
### Script Description
| Script | Purpose | Called By |
|--------|---------|-----------|
|--------|---------|----------|
| `ci.sh` | Main command dispatcher | User/GitHub Actions |
| `ci_build.sh` | Build firmware orchestrator | ci.sh build |
| `ci_site.sh` | Documentation build orchestrator | ci.sh site |
| `ci_qa.sh` | Code formatting checker | ci.sh qa |
| `ci_set_version.sh` | Version injection in firmware | ci_build.sh |
| `ci_00_config.sh` | Shared configuration loader | ci.sh |
| `ci_qa.sh` | Quality assurance and shellcheck | ci.sh qa |
| `ci_security.sh` | Security vulnerability scanning with Trivy | ci.sh security |
| `ci_list-env.sh` | List PlatformIO environments (fast JSON or full scan) | ci.sh list-env / direct |
| `ci_build_firmware.sh` | PlatformIO build execution | ci_build.sh |
| `ci_prepare_artifacts.sh` | Artifact packaging | ci_build.sh |
| `ci_00_config.sh` | Shared configuration and functions | All scripts |
| `ci_prepare_artifacts.sh` | Artifact packaging (when requested) | ci_build.sh |
### Output Structure
@@ -67,16 +71,14 @@ Build outputs are organized in the project root:
generated/
├── artifacts/ # Packaged firmware artifacts
└── site/ # Built documentation (VuePress output)
scripts/
├── latest_version.json # Production version metadata
└── latest_version_dev.json # Development version metadata
├── site/ # Built documentation (VuePress output)
└── reports/ # Security scan and SBOM reports
└── sbom/ # SBOM in CycloneDX and SPDX formats
```
## Commands
`ci.sh` is the main Entry Point. Command dispatcher that routes to specialized scripts.
`ci.sh` is the main entry point. It dispatches to the specialized scripts.
**Usage:**
```bash
@@ -84,115 +86,90 @@ scripts/
```
**Commands:**
- `build` - Build firmware for specified environment
- `site` or `docs` - Build documentation website
- `qa` or `lint` - Run code formatting checks
- `all` or `pipeline` - Run complete pipeline (qa + build + site)
- `build` - Build firmware for a PlatformIO environment (optionally prepare artifacts)
- `site` or `docs` - Build documentation site
- `qa` or `lint` - Run formatting and shellcheck checks
- `security` - Scan for security vulnerabilities using Trivy (filesystem, container images)
- `all` or `pipeline` - Run qa → build (all environments) → site with injected mode
- `list-env` - List available PlatformIO environments (JSON fast list or full ini scan)
**Examples:**
```bash
# Get Help
# Help per command
./scripts/ci.sh build --help
./scripts/ci.sh qa --help
./scripts/ci.sh site --help
./scripts/ci.sh qa --help
./scripts/ci.sh security --help
./scripts/ci.sh list-env --help
# Build firmware
./scripts/ci.sh build esp32dev-ble --mode dev
./scripts/ci.sh build esp32dev-all-test --version v1.8.0 --deploy-ready
./scripts/ci.sh build esp32dev-all-test --mode dev
./scripts/ci.sh build esp32dev-bt --version v1.8.0 --deploy-ready --output generated/artifacts
# Build documentation
./scripts/ci.sh site --mode prod
./scripts/ci.sh site --mode dev --url-prefix /dev/
# Build docs
./scripts/ci.sh site --mode prod --url-prefix /
./scripts/ci.sh site --mode dev --preview
# Check code formatting
# QA (formatting + shellcheck)
./scripts/ci.sh qa --check
./scripts/ci.sh qa --fix
./scripts/ci.sh qa --fix --verbose
# Run complete pipeline
./scripts/ci.sh all esp32dev-ble --version v1.8.0
./scripts/ci.sh all esp32dev-ble --no-site
# Security scanning
./scripts/ci.sh security --scan-type fs --severity HIGH,CRITICAL
./scripts/ci.sh security --scan-type fs --generate-sbom
# Full pipeline (qa + build all envs + site)
./scripts/ci.sh all --mode dev
./scripts/ci.sh all --mode prod --preview
```
**Options for `all` command:**
- `--no-site` - Skip documentation build step
- All options from `build` command are passed through
---
### ci.sh build - Build Firmware
Orchestrates complete firmware build: version injection, compilation, artifact packaging.
Runs the build pipeline (tool checks → PlatformIO build → optional artifact packaging).
**Usage:**
```bash
./scripts/ci.sh build <environment> [OPTIONS]
```
**Required Arguments:**
- `<environment>` - PlatformIO environment name (e.g., esp32dev-ble, nodemcuv2-rf)
**Required Argument:**
- `<environment>` PlatformIO environment name (e.g., esp32dev-ble)
**Options:**
- `--version <tag>` - Version string to inject (default: auto-generated from git)
- `--mode <dev|prod>` - Build mode (default: prod)
- `dev` - Enables development OTA, sets DEVELOPMENTOTA=true
- `prod` - Standard production build
- `--deploy-ready` - Package artifacts with deployment naming (env-firmware.bin)
- `--output <dir>` - Output directory for artifacts (default: generated/artifacts)
- `--skip-verification` - Skip build tools verification
- `--clean` - Clean previous build before starting
- `--verbose` - Enable verbose PlatformIO output
- `--list-envs` - List all available PlatformIO environments
- `--help` - Show help message
- `--mode <prod|dev>` Build mode (default: prod). `dev` enables OTA flags in the PlatformIO build.
- `--deploy-ready` Copy/rename build outputs and libs via ci_prepare_artifacts.sh.
- `--version [TAG]` Set version used for `OMG_VERSION` and artifact folder naming. If TAG is omitted the script auto-generates (CI: BUILD_NUMBER/GIT_COMMIT, local: timestamp).
- `--output <dir>` Output directory for packaged artifacts (only used when `--deploy-ready` is set, default `generated/artifacts`).
- `--skip-verification` Skip the tool availability checks.
- `--clean` Clean the PlatformIO environment before building.
- `--verbose` Verbose PlatformIO output.
- `--help` Show help.
**Execution Flow:**
```
ci.sh build esp32dev-ble --version v1.8.0 --mode prod --deploy-ready
│
├─> ci_build.sh (orchestrator)
│ ├─> verify_build_tools() - Check python3, platformio, git
│ ├─> ci_set_version.sh v1.8.0 - Inject version in User_config.h
│ ├─> ci_build_firmware.sh esp32dev-ble - Execute PlatformIO build
│ └─> ci_prepare_artifacts.sh esp32dev-ble --deploy - Package binaries
│
└─> Outputs in generated/artifacts/
├─ esp32dev-ble-firmware.bin
├─ esp32dev-ble-bootloader.bin
└─ esp32dev-ble-partitions.bin
```
**Behavior:**
- Tool check verifies python3, platformio, git (can be skipped).
- Builds via ci_build_firmware.sh (adds `--dev-ota` when `--mode dev`).
- Packaging runs only when `--deploy-ready` is provided; artifacts land under `generated/artifacts/firmware_build/` with env-prefixed filenames plus zipped libraries.
<!-- - Packaging runs only when `--deploy-ready` is provided; artifacts land under `generated/artifacts[/<version>]/firmware_build/` with env-prefixed filenames plus zipped libraries. -->
**Examples:**
```bash
# Development build
# Dev build with OTA flags
./scripts/ci.sh build esp32dev-ble --mode dev
# Production build with version
./scripts/ci.sh build esp32dev-ble --version v1.8.0 --mode prod
# Prod build with auto-version and deployable artifacts
./scripts/ci.sh build esp32dev-bt --version --deploy-ready --output generated/artifacts
# Deploy-ready build
./scripts/ci.sh build esp32dev-all-test --version v1.8.0 --deploy-ready
# Clean build with verbose output
# Clean + verbose build
./scripts/ci.sh build nodemcuv2-rf --clean --verbose
# List available environments
./scripts/ci.sh build --list-envs
```
**Environment Variables:**
- `CI` - Set to 'true' in CI/CD environments
- `BUILD_NUMBER` - Build number from CI/CD system
- `GIT_COMMIT` - Git commit hash for auto-versioning
- `PLATFORMIO_BUILD_FLAGS` - Additional PlatformIO flags (set by script when --mode dev)
**Output Files:**
- Standard mode: `firmware.bin`, `partitions.bin` in generated/artifacts/
- Deploy mode: `<env>-firmware.bin`, `<env>-bootloader.bin`, `<env>-partitions.bin`
---
### ci.sh site - Build Documentation
Builds VuePress documentation website with version management and WebUploader manifest generation.
Builds the VuePress documentation site.
**Usage:**
```bash
@@ -200,75 +177,36 @@ Builds VuePress documentation website with version management and WebUploader ma
```
**Options:**
- `--mode <dev|prod>` - Documentation mode (default: prod)
- `dev` - Development documentation with watermark
- `prod` - Production documentation
- `--version-source <release|custom>` - Version source (default: release)
- `release` - Use git tag as version
- `custom` - Use custom version string
- `--custom-version <version>` - Custom version string (requires --version-source custom)
- `--url-prefix <path>` - Base URL path (default: /)
- Example: `/dev/` for development subdirectory
- `--no-webuploader` - Skip WebUploader manifest generation
- `--webuploader-args <args>` - Additional arguments for gen_wu.py
- `--preview` - Open browser after build (local development)
- `--help` - Show help message
- `--mode <prod|dev>` Build mode (default: dev).
- `--url-prefix <path>` Base URL path for links (e.g., '/' for root, '/dev/' for dev) (default: /dev/).
- `--version <tag>` Version string written to meta.json (default: edge).
- `--preview` Start the local HTTPS preview server after building (https://localhost:8443).
- `--clean` Remove generated/site folder before building.
- `--insecure-curl` Allow curl to skip TLS verification when downloading common config.
- `--help` Show help.
**Execution Flow:**
```
ci.sh site --mode prod --version-source release
│
├─> ci_site.sh (orchestrator)
│ ├─> check_requirements() - Verify node, npm, python3, pip3
│ ├─> install_dependencies() - npm install, pip3 install packages
│ ├─> download_common_config() - Fetch from theengs.io
│ ├─> get_version() - Extract from git tag or use custom
│ ├─> set_version() - Update VuePress config and JSON files
│ ├─> set_url_prefix() - Set base path in config
│ ├─> generate_board_docs.py - Auto-generate board documentation
│ ├─> npm run docs:build - Build VuePress site
│ └─> gen_wu.py - Generate WebUploader manifest
│
└─> Outputs in generated/site/
├─ index.html
├─ assets/
└─ [board documentation pages]
```
**Behavior:**
- Checks for node, npm, openssl; installs npm deps; downloads commonConfig.js.
- Writes docs/.vuepress/meta.json with mode/url_prefix/version; builds via `npm run docs:build`.
- Preview mode runs `npm run site:preview`.
**Examples:**
```bash
# Production documentation
./scripts/ci.sh site --mode prod
# Production build
./scripts/ci.sh site --mode prod --url-prefix /
# Development documentation with custom version
./scripts/ci.sh site --mode dev --version-source custom --custom-version "DEVELOPMENT SHA:abc123"
# Development build with preview
./scripts/ci.sh site --mode dev --url-prefix /dev/ --version edge --preview
# Documentation for /dev/ subdirectory
./scripts/ci.sh site --mode dev --url-prefix /dev/
# Skip WebUploader manifest
./scripts/ci.sh site --no-webuploader
# Local preview
./scripts/ci.sh site --preview
# Clean then build with custom version
./scripts/ci.sh site --clean --version 1.8.0
```
**Required Tools:**
- Node.js (for VuePress)
- npm (for package management)
- Python 3 (for board docs generator)
- pip3 (for Python dependencies: requests, pandas, markdown, pytablereader, tabulate)
**Output Files:**
- `generated/site/` - Complete static website
- `scripts/latest_version.json` - Production version metadata (updated)
- `scripts/latest_version_dev.json` - Development version metadata (updated)
---
### ci.sh qa - Code Formatting Check
### ci.sh qa - Code Quality Checks
Checks and fixes code formatting using clang-format.
Checks and fixes code formatting using clang-format and runs shellcheck on shell scripts.
**Usage:**
```bash
@@ -276,13 +214,16 @@ Checks and fixes code formatting using clang-format.
```
**Options:**
- `--check` - Check formatting without modifying files (default)
- `--fix` - Automatically fix formatting issues
- `--source <dir>` - Source directory to check (default: main)
- `--extensions <list>` - File extensions to check, comma-separated (default: h,ino,cpp)
- `--clang-format-version <ver>` - clang-format version to use (default: 9)
- `--verbose` - Show detailed output for each file
- `--help` - Show help message
- `--check` Check formatting only (default)
- `--fix` Apply formatting in place
- `--format` Run only format checks
- `--shellcheck` Run shellcheck on shell scripts in scripts/ directory
- `--all` Future hook to run all QA checks (current implementation runs formatting + shellcheck)
- `--source <dir>` Source directory for formatting checks (default: main)
- `--extensions <list>` File extensions for formatting (comma-separated, default: h,ino,cpp)
- `--clang-format-version <ver>` clang-format version to use (default: 9)
- `--verbose` Verbose output
- `--help` Show help
**Execution Flow:**
```
@@ -290,27 +231,35 @@ ci.sh qa --check --source main --extensions h,ino
│
├─> ci_qa.sh (orchestrator)
│ ├─> check_clang_format() - Find clang-format-9 or clang-format
│ ├─> find_files() - Locate files matching extensions in source dir
│ └─> check_formatting() - Run clang-format --dry-run --Werror
│ └─> Report files with formatting issues
│ │ ├─> find_files() - Locate files matching extensions in source dir
│ │ └─> check_formatting() - Run clang-format --dry-run --Werror
│ │ └─> Report files with formatting issues
│ │
│ └─> shellcheck_check() - Find and scan shell scripts
│ ├─> find_shell_scripts() - Locate *.sh files in scripts/ directory
│ └─> run_shellcheck() - Run shellcheck on found scripts
│ └─> Report shell script issues
│
└─> Exit code: 0 (pass) or 1 (formatting issues found)
└─> Exit code: 0 (pass) or 1 (issues found)
```
**Examples:**
```bash
# Check formatting (CI mode)
# Check both formatting and shellcheck (default)
./scripts/ci.sh qa --check
# Fix formatting automatically
./scripts/ci.sh qa --fix
# Check specific directory
./scripts/ci.sh qa --check --source lib
# Check only formatting for specific directory
./scripts/ci.sh qa --check --format --source lib
# Check only .h and .ino files
./scripts/ci.sh qa --check --extensions h,ino
# Check only shellcheck for shell scripts
./scripts/ci.sh qa --check --shellcheck
# Check with verbose output
./scripts/ci.sh qa --check --verbose
@@ -321,46 +270,201 @@ ci.sh qa --check --source main --extensions h,ino
**Required Tools:**
- clang-format (version specified, default: 9)
- Install: `sudo apt-get install clang-format-9`
- shellcheck (for shell script linting)
- Install: `sudo apt-get install shellcheck`
**Output:**
- Check mode: Lists files with formatting issues and shows diffs
- Fix mode: Modifies files in-place and reports changes
- Exit code 0: All files properly formatted
- Exit code 1: Formatting issues found (in check mode)
- Check mode: Lists files with formatting issues and shell script errors, shows diffs
- Fix mode: Modifies formatting in-place and reports changes
- Exit code 0: All checks passed (proper formatting and no shellcheck errors)
- Exit code 1: Issues found (formatting or shellcheck violations)
---
### ci.sh security - Security Vulnerability Scan
Scans the project for security vulnerabilities using Trivy and generates Software Bill of Materials (SBOM).
**Usage:**
```bash
./scripts/ci.sh security [OPTIONS]
```
**Options:**
- `--scan-type <fs|config|image>` Type of scan (default: fs)
- `fs` - Filesystem scan (default, scans for vulnerabilities and misconfigurations)
- `config` - Configuration scan only
- `image` - Container image scan
- `--severity <levels>` Severity levels to report (comma-separated: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL) (default: HIGH,CRITICAL)
- `--scan-path <path>` Path to scan (default: current directory `.`)
- `--generate-sbom` Generate Software Bill of Materials in CycloneDX and SPDX formats (default: true)
- `--exit-code <0|1>` Exit code when vulnerabilities found (0=continue, 1=fail) (default: 0)
- `--upload-to-security-tab` Upload SARIF report to GitHub Security tab (GitHub Actions only, default: true)
- `--verbose` Verbose output
- `--help` Show help
**Behavior:**
- Installs Trivy if not present
- Scans filesystem or configuration for known vulnerabilities
- Generates multiple report formats: SARIF, JSON, table summary
- Creates SBOM in CycloneDX and SPDX formats when `--generate-sbom` is enabled
- Reports are saved to `generated/reports/` directory
- Summary is appended to GitHub job summary when running in GitHub Actions
- Uploads SARIF to GitHub Security tab for dashboard visibility
**Output Structure:**
```
generated/reports/
├── trivy-results.sarif # SARIF format (for GitHub Security tab)
├── trivy-results.json # JSON format (detailed results)
├── security-summary.md # Markdown summary
└── sbom/
├── sbom.cyclonedx.json # CycloneDX format
└── sbom.spdx.json # SPDX format
```
**Examples:**
```bash
# Scan filesystem for HIGH and CRITICAL vulnerabilities
./scripts/ci.sh security --scan-type fs --severity HIGH,CRITICAL
# Full scan with all severity levels and SBOM generation
./scripts/ci.sh security --scan-type fs --severity UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL --generate-sbom
# Scan specific path with verbose output
./scripts/ci.sh security --scan-path ./lib --verbose
# Configuration scan only
./scripts/ci.sh security --scan-type config
# Scan and fail on vulnerabilities
./scripts/ci.sh security --exit-code 1
```
**Required Tools:**
- Trivy (vulnerability scanner)
- Install: `wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -` && `echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list` && `sudo apt-get update && sudo apt-get install -y trivy`
**Output:**
- Detailed SARIF and JSON reports for integration with tools
- Human-readable Markdown summary with findings count
- SBOM artifacts for supply chain tracking
- GitHub Security tab integration when in GitHub Actions
- Exit code 0: Scan completed (vulnerabilities may have been found)
- Exit code 1: Scan failed or critical vulnerabilities found (only if `--exit-code 1`)
---
### ci.sh all - Complete Pipeline
Runs the complete CI/CD pipeline (qa → build all environments → site) with mode injected to all steps.
**Usage:**
```bash
./scripts/ci.sh all --mode <dev|prod> [--preview]
```
**Required Options:**
- `--mode <dev|prod>` Build mode (required). Injected to all pipeline steps:
- `qa`: Always runs `--check`
- `build`: Passes mode to all PlatformIO environments (dev enables OTA flags)
- `site`: Passes mode for documentation generation
**Optional Options:**
- `--preview` Start local HTTPS preview server at https://localhost:8443 after building the site
- `--help` Show help
**Behavior:**
- No environment argument needed; builds **all** available environments
- All three steps (qa, build, site) receive the same `--mode` value
- If any step fails, the pipeline aborts
- Site build warnings do not abort the pipeline (continues with success status)
**Execution Flow:**
```
ci.sh all --mode dev --preview
│
├─> Step 1: ci_qa.sh --check
│ └─> Exit on failure
│
├─> Step 2: ci_build.sh --mode dev (builds all environments)
│ └─> Exit on failure
│
└─> Step 3: ci_site.sh --mode dev --preview
├─> Build documentation
└─> Start preview server at https://localhost:8443
```
**Examples:**
```bash
# Complete pipeline in dev mode
./scripts/ci.sh all --mode dev
# Complete pipeline in prod mode with site preview
./scripts/ci.sh all --mode prod --preview
# Help for this command
./scripts/ci.sh all --help
```
**Output:**
- Step-by-step progress messages for each pipeline phase
- Final summary showing mode, preview status, duration, and overall status
- Exit code 0: All steps successful
- Exit code 1: Any step failed
---
### ci.sh list-env - List Environments
Lists PlatformIO environments available to build.
**Usage:**
```bash
./scripts/ci.sh list-env [--full]
```
**Options:**
- Default: read .github/workflows/environments.json (requires jq) and show curated list.
- `--full` Parse platformio.ini and environments.ini for an exhaustive list (skips *-test and *-all-).
- `--help` Show help.
**Examples:**
```bash
./scripts/ci.sh list-env
./scripts/ci.sh list-env --full
```
---
## Internal Scripts
These scripts are called by the main orchestrators. It can be invoked directly but is not raccomanded.
These scripts are called by the main orchestrators. They can be run directly for troubleshooting, but the preferred entrypoints are the ci.sh commands.
### ci_set_version.sh
### ci_list-env.sh
Injects version string into firmware configuration files.
Lists PlatformIO environments for OpenMQTTGateway.
**Called By:** `ci_build.sh`
**Called By:** `ci.sh list-env` or direct call
**Usage:**
```bash
./scripts/ci_set_version.sh <version> [--dev]
./scripts/ci_list-env.sh [--full]
```
**Arguments:**
- `<version>` - Version string to inject (e.g., v1.8.0 or abc123)
- `--dev` - Development mode (updates latest_version_dev.json)
**Options:**
- Default: read .github/workflows/environments.json for a curated list (needs jq)
- `--full` - Parse platformio.ini and environments.ini for all envs except *-test and *-all-
- `-h|--help` - Show help
**Files Modified:**
- `main/User_config.h` - Replaces "version_tag" with actual version
- `scripts/latest_version.json` - Production version metadata
- `scripts/latest_version_dev.json` - Development version metadata (--dev mode)
**Output:**
- Shows sorted environments in columns and prints the total count
**Behavior:**
- Creates .bak backup files before modification
- Replaces all occurrences of "version_tag" string
- Validates version string (must not be empty or "version_tag")
- Cleans up backup files on success
---
**Examples:**
```bash
./scripts/ci_list-env.sh
./scripts/ci_list-env.sh --full
```
### ci_build_firmware.sh
@@ -377,14 +481,17 @@ Executes PlatformIO build for specified environment.
- `<environment>` - PlatformIO environment name
**Options:**
- `--version <tag>` - Set OMG_VERSION for the build (passed through from ci_build.sh)
- `--dev-ota` - Enable development OTA (sets PLATFORMIO_BUILD_FLAGS)
- `--clean` - Clean before build
- `--verbose` - Verbose PlatformIO output
- `--no-verify` - Skip artifact verification after build
**Environment Variables Set:**
- `PYTHONIOENCODING=utf-8`
- `PYTHONUTF8=1`
- `PLATFORMIO_BUILD_FLAGS="-DDEVELOPMENTOTA=true"` (when --dev-ota)
- `OMG_VERSION` (when --version is provided)
**PlatformIO Command:**
```bash
@@ -400,7 +507,7 @@ platformio run -e <environment> [--verbose]
### ci_prepare_artifacts.sh
Packages firmware binaries from PlatformIO build directory.
Packages firmware binaries and libraries from a PlatformIO build directory; can also only create source archive when no environment is provided.
**Called By:** `ci_build.sh`
@@ -410,29 +517,19 @@ Packages firmware binaries from PlatformIO build directory.
```
**Arguments:**
- `<environment>` - PlatformIO environment name
- `<environment>` - PlatformIO environment name (optional; if omitted only source archive is created)
**Options:**
- `--deploy` - Use deployment naming (prefix with environment name)
- `--output <dir>` - Output directory (default: generated/artifacts)
- `--output <dir>` Output directory (default: generated/artifacts)
- `--version <tag>` Append a version subfolder inside the output directory
- `--clean` Clean output directory before writing
- `--help` Show help
**Behavior:**
Standard mode (no --deploy):
- Copies: `firmware.bin`, `partitions.bin`
- Does NOT copy: `bootloader.bin`
Deploy mode (with --deploy):
- Copies and renames:
- `firmware.bin` → `<env>-firmware.bin`
- `bootloader.bin` → `<env>-bootloader.bin`
- `partitions.bin` → `<env>-partitions.bin`
**Source Location:**
- `.pio/build/<environment>/`
**Output Location:**
- Specified by `--output` or default `generated/artifacts/`
- If `version` is provided, outputs go to `<output>/<version>/firmware_build/`; otherwise to `<output>/firmware_build/`.
- With an environment: copies and renames firmware.bin/partitions.bin/bootloader.bin/boot_app0.bin to `<env>-*.bin`, then zips libraries for that env into `*-libraries.tgz`.
- Without an environment: only creates `OpenMQTTGateway_sources.tgz` from `main` and `LICENSE.txt`.
- Lists the prepared artifacts and their sizes at the end.
---
@@ -452,6 +549,7 @@ Shared configuration and helper functions for all CI scripts.
- `BUILD_DIR=".pio/build"` - PlatformIO build directory
- `ARTIFACTS_DIR="generated/artifacts"` - Artifact output directory
- `SITE_DIR="generated/site"` - Documentation output directory
- `REPORTS_DIR="generated/reports"` - Security scan and quality reports directory
**Logging Functions:**
```bash
@@ -463,9 +561,74 @@ log_success "message" # Green [SUCCESS] prefix
---
### ci_security.sh
Performs security vulnerability scanning and Software Bill of Materials (SBOM) generation using Trivy.
**Called By:** `ci.sh security`
**Usage:**
```bash
./scripts/ci_security.sh [OPTIONS]
```
**Options:**
- `--scan-type <fs|config|image>` Type of scan (default: fs)
- `fs` - Filesystem scan (default)
- `config` - Configuration scan
- `image` - Container image scan
- `--severity <levels>` Severity levels (comma-separated: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL) (default: HIGH,CRITICAL)
- `--scan-path <path>` Path to scan (default: .)
- `--generate-sbom` Generate SBOM (default: true)
- `--exit-code <0|1>` Exit code behavior (0=continue, 1=fail) (default: 0)
- `--upload-to-security-tab` Upload SARIF to GitHub (default: true)
- `--verbose` Verbose output
- `--help` Show help
**Behavior:**
- Ensures Trivy is installed via package manager
- Creates `generated/reports/` directory structure
- Runs Trivy with specified parameters
- Generates SARIF, JSON, and table formats
- Creates SBOM in CycloneDX and SPDX formats (when enabled)
- Uploads SARIF to GitHub Security tab when `GITHUB_TOKEN` and `--upload-to-security-tab` are set
- Appends summary to GitHub job summary if in GitHub Actions
- Validates critical vulnerabilities and exits with code 1 if found and `--exit-code 1` is set
**Output Files:**
- `generated/reports/trivy-results.sarif` - SARIF format for GitHub integration
- `generated/reports/trivy-results.json` - Full JSON results
- `generated/reports/security-summary.md` - Human-readable summary
- `generated/reports/sbom/sbom.cyclonedx.json` - CycloneDX SBOM
- `generated/reports/sbom/sbom.spdx.json` - SPDX SBOM
**Exit Codes:**
- `0` - Success (vulnerabilities may have been found)
- `1` - Scan failed, critical vulnerabilities found (only if `--exit-code 1`), or missing dependencies
**Trivy Integration:**
- Scans for known vulnerabilities in dependencies
- Detects misconfigurations and insecure practices
- Generates compliant SBOM artifacts
- Provides detailed reporting in multiple formats
**Example:**
```bash
# Scan filesystem with severity filter
./scripts/ci_security.sh --scan-type fs --severity HIGH,CRITICAL
# Local scan with SBOM and JSON output
./scripts/ci_security.sh --scan-type fs --generate-sbom --verbose
# In GitHub Actions with security tab upload
./scripts/ci_security.sh --scan-type fs --severity HIGH,CRITICAL --upload-to-security-tab
```
---
## Python Helper Scripts
Other scripts are present and used as internal scripts and it's used as retrocompatibility. Below the lists:
Legacy helper scripts are kept for compatibility; they are not called by the current ci_site.sh flow. Below the list:
- `generate_board_docs.py`
- `gen_wu.py`
@@ -473,7 +636,7 @@ Other scripts are present and used as internal scripts and it's used as retrocom
Auto-generates board-specific documentation pages from platformio.ini.
**Called By:** `ci_site.sh`
**Called By:** Not invoked by current ci_site.sh (legacy helper)
**Usage:**
```bash
@@ -498,7 +661,7 @@ python3 ./scripts/generate_board_docs.py
Generates WebUploader manifest for OTA firmware updates.
**Called By:** `ci_site.sh`
**Called By:** Not invoked by current ci_site.sh (legacy helper)
**Usage:**
```bash
@@ -527,10 +690,10 @@ python3 ./scripts/gen_wu.py [--dev] [repository]
Scripts respect these environment variables:
- `PYTHONIOENCODING=utf-8`: Python encoding
- `PYTHONUTF8=1`: UTF-8 mode
- `PLATFORMIO_BUILD_FLAGS`: Custom build flags
- `ESP32_PLATFORM_VERSION`: Extracted automatically
- `CI`/`BUILD_NUMBER`/`GIT_COMMIT`: Used by ci_build.sh to auto-generate version when `--version` flag has no tag
- `PYTHONIOENCODING=utf-8`, `PYTHONUTF8=1`: Python encoding settings set by ci_build_firmware.sh
- `PLATFORMIO_BUILD_FLAGS`: Set to include development OTA flag when `--dev-ota` is used
- `OMG_VERSION`: Set when `--version` is passed to ci_build.sh/ci_build_firmware.sh
---
@@ -567,6 +730,58 @@ CI/CD environments typically set:
- `BUILD_NUMBER` (build number)
- `GIT_COMMIT` (commit hash)
---
## GitHub Actions Workflows Integration
The CI scripts integrate with GitHub Actions workflows in `.github/workflows/`:
### task-lint.yml
- Reusable workflow that runs `ci.sh qa --check`
- Installs clang-format and shellcheck
- Validates code formatting and shell script quality
- Can be called with custom source directory and file extensions
### task-build.yml
- Main build workflow orchestrator
- Calls task-lint.yml for code quality checks
- Calls task-security-scan.yml for vulnerability scanning
- Builds firmware for all or specified environments
- Prepares and uploads build artifacts
- Supports matrix builds for multiple environments
- Manages build artifact retention
### task-security-scan.yml
- Reusable security scanning workflow
- Installs Trivy vulnerability scanner
- Calls `ci_security.sh` with configurable parameters
- Generates SARIF, JSON, and SBOM reports
- Uploads SARIF to GitHub Security tab for code scanning dashboard
- Fails build on critical vulnerabilities when configured
- Uploads SBOM artifacts for supply chain tracking
### security-scan.yml
- Scheduled security scanning (runs weekly by default)
- Triggered manually with input parameters
- Allows filtering by severity level
- Configurable exit behavior (fail or continue)
- Optional SBOM generation and upload
**Workflow Dependencies:**
```
task-build.yml
├─> task-lint.yml (linting)
├─> task-security-scan.yml (security scanning)
└─> Build environment matrix (firmware compilation)
```
**Key Features:**
- Parallel linting and security scans
- Artifact retention policies
- GitHub Security tab integration
- Detailed build reports
- SBOM generation for compliance
- Support for custom build parameters
---
+192 -38
View File
@@ -6,8 +6,9 @@
set -euo pipefail
# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_DIR
# Load shared configuration (colors, logging functions, paths)
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
@@ -36,7 +37,9 @@ Commands:
build Build firmware for specified environment
site Build and deploy documentation/website
qa Run quality assurance checks (linting, formatting)
security Run security vulnerability scan using Trivy
all Run complete pipeline (qa + build + site)
list-env List available environments for building firmware
Examples:
# Build firmware
@@ -51,69 +54,206 @@ Examples:
$0 qa --check
$0 qa --fix
# Run complete pipeline
$0 all esp32dev-bt --version v1.8.0
# Run security scan
$0 security
$0 security --scan-type config --exit-code 1
$0 security --severity UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
# Run complete pipeline (all envs, mode required)
$0 all --mode dev
$0 all --mode prod --preview
$0 all --mode dev -e esp32dev-bt
# List available environments
$0 list-env
Get help for specific commands:
$0 build --help
$0 site --help
$0 qa --help
$0 security --help
$0 list-env --help
EOF
exit 0
}
# Function to run build pipeline
run_build_pipeline() {
log_info "Executing build pipeline..."
"${SCRIPT_DIR}/ci_build.sh" "$@"
# Function to get list of environments to build
# Uses ci_list-env.sh (no parameters) and returns one env per line
get_environments() {
# Run the curated list script and normalize output
"${SCRIPT_DIR}/ci_list-env.sh" \
| sed -r 's/\x1B\[[0-9;]*[mK]//g' \
| tr '\t ' '\n' \
| sed '/^\s*$/d' \
| grep -E '^[A-Za-z0-9._-]+$' \
| sort -u
}
# Function to run site pipeline
run_site_pipeline() {
log_info "Executing site pipeline..."
"${SCRIPT_DIR}/ci_site.sh" "$@"
}
# Function to run QA pipeline
run_qa_pipeline() {
log_info "Executing QA pipeline..."
"${SCRIPT_DIR}/ci_qa.sh" "$@"
}
# Function to run complete pipeline
# Function to run complete pipeline using the underlying scripts
# Usage: run_all_pipeline --mode <dev|prod> [--preview]
run_all_pipeline() {
local start_time
start_time=$(date +%s)
log_info "Starting complete CI/CD pipeline..."
local mode=""
local preview=false
local env_override=""
local version=""
local do_clean=false
# Parse arguments
while [[ $# -gt 0 ]]; do
case "$1" in
--mode)
if [[ -z "${2:-}" ]]; then
log_error "--mode requires an argument (dev or prod)"
return 1
fi
mode="$2"
if [[ "$mode" != "dev" && "$mode" != "prod" ]]; then
log_error "Invalid mode: $mode. Must be 'dev' or 'prod'"
return 1
fi
shift 2
;;
--clean)
do_clean=true
shift
;;
--preview)
preview=true
shift
;;
-e|--env)
if [[ -z "${2:-}" ]]; then
log_error "-e|--env requires an environment name"
return 1
fi
env_override="$2"
shift 2
;;
-v|--version)
if [[ -z "${2:-}" ]]; then
log_error "-v|--version requires a version string"
return 1
fi
version="$2"
shift 2
;;
--help|-h)
echo "Usage: $0 all --mode <dev|prod> [--preview]"
echo ""
echo "Options:"
echo " --mode <dev|prod> Build mode (required)"
echo " --preview Optional Show site in local at http://localhost:8443"
echo " -e, --env <name> Optional Build only the specified environment"
return 0
;;
*)
log_error "Unknown option: $1"
return 1
;;
esac
done
# Validate mode is provided
if [[ -z "$mode" ]]; then
log_error "--mode is required. Usage: $0 all --mode <dev|prod> [--preview]"
return 1
fi
log_info "Starting complete CI/CD pipeline (mode: $mode, preview: $preview)..."
echo ""
# Step 1: Quality Assurance
log_info "═══ Step 1/3: Quality Assurance ═══"
run_qa_pipeline --check || {
log_info "RUN: ---> ${SCRIPT_DIR}/ci_qa.sh --check"
if ! "${SCRIPT_DIR}/ci_qa.sh" --check; then
log_error "QA checks failed. Pipeline aborted."
return 1
}
fi
echo ""
# Step 2: Build Firmware
log_info "═══ Step 2/3: Build Firmware ═══"
run_build_pipeline "$@" || {
log_error "Build failed. Pipeline aborted."
# Get list of environments
local -a environments
if [[ -n "$env_override" ]]; then
log_info "Using single environment override: $env_override"
environments=("$env_override")
else
mapfile -t environments < <(get_environments)
fi
if [[ ${#environments[@]} -eq 0 ]]; then
log_error "No environments found to build"
return 1
}
fi
log_info "Found ${#environments[@]} environments to build"
echo ""
# Step 3: Build Site (only if not in --no-site mode)
if [[ ! " $* " =~ " --no-site " ]]; then
log_info "═══ Step 3/3: Build Documentation ═══"
run_site_pipeline --mode prod || {
log_warn "Site build failed, but continuing..."
}
else
log_info "Skipping site build (--no-site flag)"
local build_count=0
local failed_builds=()
local build_args=()
if [[ -n "$version" ]]; then
build_args+=("--version" "$version")
log_info "Using version override: $version"
fi
build_args+=("--mode" "$mode")
build_args+=("--deploy-ready")
if [[ "$do_clean" == true ]]; then
build_args+=("--clean")
fi
for env in "${environments[@]}"; do
((++build_count))
log_info "[$build_count/${#environments[@]}] Building: $env"
set +e
log_info "RUN: ---> ${SCRIPT_DIR}/ci_build.sh" "$env" "${build_args[@]}"
"${SCRIPT_DIR}/ci_build.sh" "$env" "${build_args[@]}"
local rc=$?
set -e
if [[ $rc -ne 0 ]]; then
log_error "Build failed for environment: $env"
failed_builds+=("$env")
fi
done
echo ""
if [[ ${#failed_builds[@]} -gt 0 ]]; then
log_error "Build failed for ${#failed_builds[@]} environment(s):"
printf ' - %s\n' "${failed_builds[@]}"
return 1
fi
log_success "All environments built successfully (${#environments[@]} total)"
echo ""
# Step 3: Build Site
log_info "═══ Step 3/3: Build Documentation ═══"
local site_args=("--mode" "$mode")
if [[ "$preview" == true ]]; then
site_args+=("--preview")
fi
if [[ -n "$version" ]]; then
site_args+=("--version" "$version")
fi
if [[ "$do_clean" == true ]]; then
site_args+=("--clean")
fi
log_info "RUN: --->${SCRIPT_DIR}/ci_site.sh" "${site_args[@]}"
if ! "${SCRIPT_DIR}/ci_site.sh" "${site_args[@]}"; then
log_warn "Site build failed, but continuing..."
fi
echo ""
local end_time
end_time=$(date +%s)
@@ -123,6 +263,8 @@ run_all_pipeline() {
echo "╔════════════════════════════════════════╗"
echo "║ Complete Pipeline Summary ║"
echo "╚════════════════════════════════════════╝"
echo " Mode: $mode"
echo " Preview: $preview"
echo " Total Duration: ${duration}s"
echo " Status: SUCCESS ✓"
echo "╚════════════════════════════════════════╝"
@@ -151,14 +293,26 @@ main() {
# Route to appropriate pipeline
case "$command" in
build)
run_build_pipeline "$@"
log_info "Executing build pipeline..."
"${SCRIPT_DIR}/ci_build.sh" "$@"
;;
site|docs)
run_site_pipeline "$@"
log_info "Executing site pipeline..."
"${SCRIPT_DIR}/ci_site.sh" "$@"
;;
qa|lint)
run_qa_pipeline "$@"
log_info "Executing QA pipeline..."
"${SCRIPT_DIR}/ci_qa.sh" "$@"
;;
security)
log_info "Executing security scan..."
"${SCRIPT_DIR}/ci_security.sh" "$@"
;;
list-env)
log_info "Executing list-env pipeline..."
"${SCRIPT_DIR}/ci_list-env.sh" "$@"
;;
all|pipeline)
run_all_pipeline "$@"
;;
+6 -33
View File
@@ -1,39 +1,16 @@
#!/bin/bash
# shellcheck disable=SC2034
# Build Scripts Configuration
# Used by: All build scripts for centralized configuration
# Python Configuration
PYTHON_VERSION="3.13"
PLATFORMIO_VERSION="https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip"
# Centralized Output Directory Structure
# All CI/CD generated files go under generated/
GENERATED_BASE_DIR="generated"
ARTIFACTS_DIR="${GENERATED_BASE_DIR}/artifacts"
SITE_DIR="${GENERATED_BASE_DIR}/site"
REPORTS_DIR="${GENERATED_BASE_DIR}/reports"
ARTIFACTS_DIR="generated/artifacts"
SITE_DIR="generated/site"
REPORTS_DIR="generated/reports"
# PlatformIO Directory Configuration
BUILD_DIR=".pio/build"
SCRIPTS_DIR="scripts"
# Build Configuration
DEFAULT_ENVIRONMENT="esp32dev-all-test"
ENABLE_VERBOSE_BUILD="false"
ENABLE_BUILD_CACHE="true"
# Artifact Configuration
ARTIFACT_RETENTION_DAYS="7"
CREATE_MANIFEST="true"
COMPRESS_ARTIFACTS="false"
# Version Configuration
VERSION_FILE_PROD="scripts/latest_version.json"
VERSION_FILE_DEV="scripts/latest_version_dev.json"
USER_CONFIG_FILE="main/User_config.h"
# Logging Configuration
ENABLE_COLOR_OUTPUT="true"
LOG_LEVEL="INFO" # DEBUG, INFO, WARN, ERROR
# ============================================================================
# Colors - ANSI color codes for terminal output
@@ -50,9 +27,5 @@ readonly NC='\033[0m' # No Color
log_info() { echo -e "${GREEN}[INFO]${NC} $*" >&2; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $*" >&2; }
log_error() { echo -e "${RED}[ERROR]${NC} $*" >&2; }
log_success() { echo -e "${GREEN}[SUCCESS]${NC} $*" >&2; }
log_step() { echo -e "${BLUE}[STEP]${NC} $*" >&2; }
# Advanced Options
ENABLE_CCACHE="false"
CCACHE_DIR=".ccache"
MAX_BUILD_JOBS="4"
+48 -73
View File
@@ -6,8 +6,10 @@
set -euo pipefail
# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
readonly SCRIPT_DIR
readonly PROJECT_ROOT
# Load shared configuration (colors, logging functions, paths)
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
@@ -71,7 +73,6 @@ verify_build_tools() {
log_info "Verifying required build tools..."
local missing_tools=()
local version_mismatch=()
# Check Python
if ! command_exists python3; then
@@ -130,39 +131,6 @@ cleanup_on_error() {
find . -name "*.bak" -type f -exec bash -c 'mv "$1" "${1%.bak}"' _ {} \; 2>/dev/null || true
}
# Function to list available environments
list_environments() {
log_info "Available PlatformIO environments:"
echo ""
local env_files=("${PROJECT_ROOT}/platformio.ini" "${PROJECT_ROOT}/environments.ini")
local envs=()
for file in "${env_files[@]}"; do
if [[ -f "$file" ]]; then
while IFS= read -r line; do
if [[ "$line" =~ ^\[env:([^\]]+)\] ]]; then
local env_name="${BASH_REMATCH[1]}"
# Skip test environments
if [[ ! "$env_name" =~ -test$ && ! "$env_name" =~ -all- ]]; then
envs+=("$env_name")
fi
fi
done < "$file"
fi
done
# Sort and display unique environments
if [[ ${#envs[@]} -gt 0 ]]; then
printf '%s\n' "${envs[@]}" | sort -u | column -c 80
else
log_warn "No environments found in configuration files"
fi
echo ""
log_info "Total: $(printf '%s\n' "${envs[@]}" | sort -u | wc -l) environments"
}
# Show usage
usage() {
cat << EOF
@@ -174,15 +142,14 @@ Arguments:
environment PlatformIO environment name
Options:
--version [TAG] Set version tag (if TAG omitted, auto-generated)
--mode MODE Build mode: 'prod' or 'dev' [default: prod]
'dev' enables OTA and development version
'dev' enables OTA and development features
--deploy-ready Prepare for deployment (renamed artifacts)
--version [TAG] Set version tag (if TAG omitted, auto-generated)
--output DIR Output directory for artifacts [default: generated/artifacts/]
--skip-verification Skip build tools verification
--clean Clean build before starting
--verbose Enable verbose output
--list-envs List all available PlatformIO environments
--help Show this help message
Environment Variables:
@@ -191,9 +158,6 @@ Environment Variables:
GIT_COMMIT Git commit hash for versioning
Examples:
# List available environments
$0 --list-envs
# Local development build
$0 esp32dev-all-test --mode dev
@@ -212,7 +176,7 @@ main() {
local version=""
local set_version=false
local mode=""
local deploy=false
local prepare_for_deploy=false
local output_dir=""
local skip_verification=false
local clean=false
@@ -246,7 +210,7 @@ main() {
shift 2
;;
--deploy-ready)
deploy=true
prepare_for_deploy=true
shift
;;
--output)
@@ -270,10 +234,6 @@ main() {
verbose=true
shift
;;
--list-envs)
list_environments
exit 0
;;
--help|-h)
usage
exit 0
@@ -290,12 +250,6 @@ main() {
esac
done
# Set default mode if not specified
if [[ -z "$mode" ]]; then
mode="prod"
log_info "Mode not specified, defaulting to production"
fi
# Validate environment
if [[ -z "$environment" ]]; then
log_error "Environment name is required"
@@ -303,6 +257,13 @@ main() {
exit 1
fi
# Set default mode if not specified
if [[ -z "$mode" ]]; then
mode="prod"
log_info "Mode not specified, defaulting to production"
fi
# Auto-generate version if --version flag is set but no tag provided
if [[ "$set_version" == "true" && -z "$version" ]]; then
if [[ "${CI:-false}" == "true" ]]; then
@@ -338,19 +299,8 @@ main() {
echo ""
fi
# Step 2: Set version (only if --version flag was provided)
if [[ "$set_version" == "true" ]]; then
log_step "2/4 Setting version: $version"
if [[ "$mode" == "dev" ]]; then
"${SCRIPT_DIR}/ci_set_version.sh" "$version" --dev || exit 1
else
"${SCRIPT_DIR}/ci_set_version.sh" "$version" --prod || exit 1
fi
echo ""
else
log_info "Skipping version setting (use --version to set version)"
echo ""
fi
# Step 2: Set version
# not required
# Step 3: Build firmware
log_step "3/4 Building firmware for: $environment"
@@ -358,19 +308,44 @@ main() {
[[ "$mode" == "dev" ]] && build_opts+=(--dev-ota)
[[ "$clean" == "true" ]] && build_opts+=(--clean)
[[ "$verbose" == "true" ]] && build_opts+=(--verbose)
[[ "$set_version" == "true" ]] && build_opts+=(--version "$version")
"${SCRIPT_DIR}/ci_build_firmware.sh" "$environment" "${build_opts[@]}" || exit 1
echo ""
# Step 4: Prepare artifacts
log_step "4/4 Preparing artifacts..."
local artifact_opts=()
[[ "$deploy" == "true" ]] && artifact_opts+=(--deploy) || artifact_opts+=(--standard)
artifact_opts+=(--manifest)
[[ -n "$output_dir" ]] && artifact_opts+=(--output "$output_dir")
"${SCRIPT_DIR}/ci_prepare_artifacts.sh" "$environment" "${artifact_opts[@]}" || exit 1
echo ""
if [[ "$prepare_for_deploy" == "true" ]]; then
log_info "Preparing artifacts for deployment"
local artifact_opts=()
[[ "$clean" == "true" ]] && artifact_opts+=(--clean)
[[ -n "$output_dir" ]] && artifact_opts+=(--output "$output_dir")
[[ "$set_version" == "true" ]] && artifact_opts+=(--version "$version")
"${SCRIPT_DIR}/ci_prepare_artifacts.sh" "$environment" "${artifact_opts[@]}" || exit 1
echo ""
# Check if site folder exists and copy built files to avoid rebuilding the site
if [[ "$mode" == "dev" ]]; then
local site_dir="${PROJECT_ROOT}/generated/site/dev"
local artifacts_dir="${output_dir:-${PROJECT_ROOT}/generated/artifacts/firmware_build}"
if [[ -d "$site_dir" ]]; then
log_info "Site folder exists, copying built firmware files to site/dev..."
# Copy firmware files for the current environment
for file in "${artifacts_dir}/${environment}"-*.bin "${artifacts_dir}/${environment}"-*.tgz; do
if [[ -f "$file" ]]; then
cp -v "$file" "$site_dir/" || log_warn "Failed to copy $(basename "$file")"
fi
done
log_info "✓ Firmware files copied to site/dev (no site rebuild needed)"
else
log_warn "Site folder not found at: $site_dir"
log_info "Run 'ci.sh site ' to generate it"
fi
fi
fi
# Print summary
print_summary "$environment" "$version" "$start_time"
+28 -15
View File
@@ -6,8 +6,10 @@
set -euo pipefail
# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
readonly SCRIPT_DIR
readonly PROJECT_ROOT
# Load shared configuration (colors, logging functions, paths)
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
@@ -41,18 +43,7 @@ validate_environment() {
log_info "Building environment: $env"
}
# Function to setup build environment variables
setup_build_env() {
local enable_dev_ota="${1:-false}"
export PYTHONIOENCODING=utf-8
export PYTHONUTF8=1
if [[ "$enable_dev_ota" == "true" ]]; then
export PLATFORMIO_BUILD_FLAGS='"-DDEVELOPMENTOTA=true"'
log_info "Development OTA enabled"
fi
}
# Function to check PlatformIO availability
check_platformio() {
@@ -78,6 +69,18 @@ run_build() {
local env="$1"
local clean="${2:-false}"
local verbose="${3:-false}"
local enable_dev_ota="${4:-false}"
local version="${5:-edge}"
if [[ "$enable_dev_ota" == "true" ]]; then
log_info "Development OTA enabled"
export PLATFORMIO_BUILD_FLAGS='"-DDEVELOPMENTOTA=true"'
fi
if [[ -n "$version" ]]; then
log_info "Setting firmware version to: $version"
export PLATFORMIO_BUILD_FLAGS="${PLATFORMIO_BUILD_FLAGS} -DOMG_VERSION=\\\"${version}\\\""
fi
log_build "Starting build for environment: $env"
@@ -95,6 +98,9 @@ run_build() {
local start_time
start_time=$(date +%s)
log_info "PlatformIO Build Flags: $PLATFORMIO_BUILD_FLAGS"
log_info "Executing: $build_cmd"
if eval "$build_cmd"; then
local end_time
end_time=$(date +%s)
@@ -184,6 +190,7 @@ Options:
--clean Clean build artifacts before building
--verbose Enable verbose build output
--no-verify Skip artifact verification
--version <ver> Set firmware version (default: edge)
--help Show this help message
Examples:
@@ -201,10 +208,15 @@ main() {
local clean_build_flag=false
local verbose=false
local verify=true
local version="edge"
# Parse arguments
while [[ $# -gt 0 ]]; do
case "$1" in
--version)
version="$2"
shift 2
;;
--dev-ota)
enable_dev_ota=true
shift
@@ -254,7 +266,8 @@ main() {
validate_environment "$environment" || exit 1
# Setup build environment
setup_build_env "$enable_dev_ota"
export PYTHONIOENCODING=utf-8
export PYTHONUTF8=1
# Clean if requested
if [[ "$clean_build_flag" == "true" ]]; then
@@ -262,7 +275,7 @@ main() {
fi
# Run build
run_build "$environment" "$clean_build_flag" "$verbose" || exit 1
run_build "$environment" "$clean_build_flag" "$verbose" "$enable_dev_ota" "$version" || exit 1
# Verify artifacts
if [[ "$verify" == "true" ]]; then
+125
View File
@@ -0,0 +1,125 @@
#!/bin/bash
# Lists all available PlatformIO environments for OpenMQTTGateway
# Usage: ./scripts/ci_list.sh [--full]
set -euo pipefail
# Constants
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
readonly SCRIPT_DIR
readonly PROJECT_ROOT
# Load shared configuration (colors, logging functions, paths)
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
source "${SCRIPT_DIR}/ci_00_config.sh"
else
echo "ERROR: ci_00_config.sh not found" >&2
exit 1
fi
## use .github/workflows/environments.json to list environments
list_environments_from_Json() {
local json_file="${PROJECT_ROOT}/.github/workflows/environments.json"
if [[ ! -f "$json_file" ]]; then
log_error "JSON file not found: $json_file"
return 1
fi
if ! command -v jq >/dev/null 2>&1; then
log_error "jq is required to read $json_file"
return 1
fi
log_info "Available PlatformIO environments from JSON:"
echo ""
local envs=()
while IFS= read -r env; do
# Skip test environments
if [[ ! "$env" =~ -test$ && ! "$env" =~ -all- ]]; then
envs+=("$env")
fi
done < <(jq -r '.environments.all[]? // empty' "$json_file")
# Sort and display unique environments
if [[ ${#envs[@]} -gt 0 ]]; then
printf '%s\n' "${envs[@]}" | sort -u | column -c 80
else
log_warn "No environments found in JSON file"
fi
echo ""
log_info "Total: $(printf '%s\n' "${envs[@]}" | sort -u | wc -l) environments"
}
list_environments() {
log_info "Available PlatformIO environments:"
echo ""
local env_files=("${PROJECT_ROOT}/platformio.ini" "${PROJECT_ROOT}/environments.ini")
local envs=()
for file in "${env_files[@]}"; do
if [[ -f "$file" ]]; then
while IFS= read -r line; do
if [[ "$line" =~ ^\[env:([^\]]+)\] ]]; then
local env_name="${BASH_REMATCH[1]}"
envs+=("$env_name")
fi
done < "$file"
fi
done
# Sort and display unique environments
if [[ ${#envs[@]} -gt 0 ]]; then
printf '%s\n' "${envs[@]}" | sort -u | column -c 80
else
log_warn "No environments found in configuration files"
fi
echo ""
log_info "Total: $(printf '%s\n' "${envs[@]}" | sort -u | wc -l) environments"
}
# Main execution
usage() {
cat <<EOF
Usage: ${0##*/} [--full]
List available PlatformIO environments for OpenMQTTGateway.
Options:
--full Parse platformio.ini and environments.ini for an exhaustive list
(slower, includes all defined envs except *-test / *-all-)
(default) Use .github/workflows/environments.json (fast, curated list)
Examples:
${0##*/} # quick, curated list
${0##*/} --full # exhaustive list from config files
EOF
}
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
USE_FULL=false
while [[ $# -gt 0 ]]; do
case "$1" in
--full)
USE_FULL=true
shift
;;
-h|--help)
usage
exit 0
;;
*)
log_warn "Unknown option: $1"
usage
exit 1
;;
esac
done
if [[ "$USE_FULL" == true ]]; then
list_environments
else
list_environments_from_Json
fi
fi
+101 -127
View File
@@ -1,4 +1,5 @@
#!/bin/bash
# shellcheck disable=SC2015
# Prepares firmware artifacts for upload or deployment
# Used by: CI/CD pipelines for artifact packaging
# Usage: ./prepare_artifacts.sh <environment> [OPTIONS]
@@ -6,8 +7,10 @@
set -euo pipefail
# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
readonly SCRIPT_DIR
readonly PROJECT_ROOT
# Load shared configuration (colors, logging functions, paths)
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
@@ -22,17 +25,22 @@ BUILD_DIR="${PROJECT_ROOT}/${BUILD_DIR}"
DEFAULT_OUTPUT_DIR="${PROJECT_ROOT}/${ARTIFACTS_DIR}"
# Function to create output directory
create_output_dir() {
prepare_output_dir() {
local output_dir="$1"
local clean_flag="${2:-false}"
if [[ -d "$output_dir" ]]; then
log_warn "Output directory already exists: $output_dir"
log_info "Cleaning existing artifacts..."
rm -rf "$output_dir"
if [[ "$clean_flag" == "true" ]]; then
log_warn "Cleaning and recreating output directory: $output_dir"
rm -rf "$output_dir"
mkdir -p "$output_dir"
else
log_warn "Output directory already exists and will be reused: $output_dir"
fi
else
mkdir -p "$output_dir"
log_info "Created output directory: $output_dir"
fi
mkdir -p "$output_dir"
log_info "Created output directory: $output_dir"
}
# Function to copy artifact with optional renaming
@@ -58,42 +66,14 @@ copy_artifact() {
fi
}
# Function to prepare standard artifacts (no renaming)
prepare_standard_artifacts() {
local env="$1"
local output_dir="$2"
local env_dir="${BUILD_DIR}/${env}"
log_info "Preparing STANDARD artifacts for: $env"
local copied=0
# Copy firmware.bin (required)
if copy_artifact "${env_dir}/firmware.bin" "${output_dir}/firmware.bin" "firmware"; then
((copied++))
fi
# Copy partitions.bin (optional)
copy_artifact "${env_dir}/partitions.bin" "${output_dir}/partitions.bin" "partitions" && ((copied++)) || true
# Note: bootloader.bin is NOT copied in standard mode (only needed for deployment)
if [[ $copied -eq 0 ]]; then
log_error "No artifacts were copied"
return 1
fi
log_info "Copied ${copied} artifact(s) in standard mode"
}
# Function to prepare deployment artifacts (with renaming)
prepare_deployment_artifacts() {
prepare_artifacts() {
local env="$1"
local output_dir="$2"
local env_dir="${BUILD_DIR}/${env}"
log_info "Preparing DEPLOYMENT artifacts for: $env"
log_info "Preparing firmware directory for: $env"
local copied=0
# Copy and rename firmware.bin
@@ -118,54 +98,57 @@ prepare_deployment_artifacts() {
log_info "Copied ${copied} artifact(s) in deployment mode"
}
# Function to create manifest file
create_manifest() {
prepare_libraries() {
local env="$1"
local output_dir="$2"
local manifest="${output_dir}/manifest.txt"
local env_dir="${BUILD_DIR}/${env}"
log_info "Creating artifact manifest..."
# Process libraries: create temp copy with renamed folders, zip, preserve originals
log_info "Processing libraries for environment: $env"
TEMP_LIBDEPS=$(mktemp -p "$output_dir" -d) || { echo "Failed to create temp directory"; return 1; }
{
echo "Environment: $env"
echo "Build Date: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
echo "Build Host: $(hostname)"
echo ""
echo "Artifacts:"
cp -r .pio/libdeps/"$env" "$TEMP_LIBDEPS/" || { log_error "Failed to copy libdeps for $env"; return 1; }
find "$output_dir" -type f -name "*.bin" | sort | while read -r file; do
local size
size=$(stat -f%z "$file" 2>/dev/null || stat -c%s "$file" 2>/dev/null)
local size_kb=$((size / 1024))
local md5sum_val
md5sum_val=$(md5sum "$file" 2>/dev/null | cut -d' ' -f1 || md5 -q "$file" 2>/dev/null)
echo " - $(basename "$file"): ${size_kb} KB (MD5: ${md5sum_val})"
done
} > "$manifest"
(
cd "$TEMP_LIBDEPS"
log_step "Replace space by _ in folder names (temp copy only)"
find . -type d -name "* *" | while read -r FNAME; do
mv "$FNAME" "${FNAME// /_}"
done
log_info "Manifest created: $manifest"
log_step "Zipping libraries per board"
for i in */; do
tar -czf "${i%/}-libraries.tgz" "$i" > /dev/null
done
mv ./*.tgz "${output_dir}"
)
rm -rf "$TEMP_LIBDEPS"
log_info "✓ Created library archives in: $output_dir"
}
# Function to compress artifacts (optional)
compress_artifacts() {
prepare_sources() {
local output_dir="$1"
local archive_name="$2"
log_info "Compressing artifacts..."
log_info "Preparing source code archive"
local archive="${output_dir}/${archive_name}.tar.gz"
if tar -czf "$archive" -C "$output_dir" .; then
local size
size=$(stat -f%z "$archive" 2>/dev/null || stat -c%s "$archive" 2>/dev/null)
local size_kb=$((size / 1024))
log_info "Archive created: ${archive_name}.tar.gz (${size_kb} KB)"
# Create and move sources tar.gz (newly generated, safe to move)
if tar -czf "${output_dir}/OpenMQTTGateway_sources.tgz" main LICENSE.txt > /dev/null; then
log_info "✓ Created source archive: OpenMQTTGateway_sources.tgz"
else
log_error "Failed to create archive"
log_error "Failed to create source archive"
return 1
fi
}
# Function to list artifacts
list_artifacts() {
local output_dir="$1"
@@ -194,17 +177,14 @@ usage() {
cat << EOF
Usage: $0 <environment> [OPTIONS]
Prepare firmware artifacts for upload or deployment.
Prepare artifacts for upload or deployment.
Arguments:
environment PlatformIO environment name
environment PlatformIO environment name, if omitted will be created source archive only.
Options:
--deploy Prepare for deployment (rename with environment prefix)
--standard Prepare standard artifacts (no renaming) [default]
--clean Clean existing output directory before preparing artifacts
--output DIR Output directory [default: generated/artifacts/]
--manifest Create manifest file with artifact metadata
--compress Compress artifacts into tar.gz archive
--help Show this help message
Examples:
@@ -218,32 +198,27 @@ EOF
# Main execution
main() {
local environment=""
local mode="standard"
local output_dir="$DEFAULT_OUTPUT_DIR"
local create_manifest_flag=false
local compress_flag=false
local clean_flag=false
#local version="" ## WILL BE USED WHEN THE VERSION ITSELF AFFECTS THE ARTIFACTS NAMING
# Parse arguments
while [[ $# -gt 0 ]]; do
case "$1" in
--deploy)
mode="deploy"
shift
;;
--standard)
mode="standard"
shift
;;
--output)
output_dir="$2"
shift 2
;;
--manifest)
create_manifest_flag=true
shift
-v|--version)
if [[ -z "${2:-}" ]]; then
log_error "-v|--version requires a version string"
return 1
fi
#version="$2"
shift 2
;;
--compress)
compress_flag=true
--clean)
clean_flag=true
shift
;;
--help|-h)
@@ -262,48 +237,47 @@ main() {
esac
done
# Validate inputs
if [[ -z "$environment" ]]; then
log_error "Environment name is required"
usage
exit 1
fi
# Change to project root
cd "$PROJECT_ROOT"
# Check if build directory exists
if [[ ! -d "${BUILD_DIR}/${environment}" ]]; then
log_error "Build directory not found for environment: $environment"
log_error "Run build_firmware.sh first"
exit 1
fi
# TODO FOR NEXT STEP MULTI RELEASE: TAG, RC, edge
#if [[ -n "$version" ]]; then
# # Sanitize version string for directory name
# safe_version=$(echo "$version" | sed 's/[^a-zA-Z0-9._-]/_/g')
# output_dir="${output_dir}/${safe_version}"
#fi
# Create output directory
create_output_dir "$output_dir"
# Prepare artifacts based on mode
case "$mode" in
standard)
prepare_standard_artifacts "$environment" "$output_dir" || exit 1
;;
deploy)
prepare_deployment_artifacts "$environment" "$output_dir" || exit 1
;;
*)
log_error "Unknown mode: $mode"
# Validate inputs
if [[ -z "$environment" ]]; then
log_info "No environment specified, only preparing source archive"
# Create output directory
prepare_output_dir "$output_dir" "$clean_flag"
# Prepare source code archive
prepare_sources "$output_dir" || exit 1
else
# Check if build directory exists
if [[ ! -d "${BUILD_DIR}/${environment}" ]]; then
log_error "Build directory not found for environment: $environment"
log_error "Run build_firmware.sh first"
exit 1
;;
esac
fi
# Create manifest if requested
if [[ "$create_manifest_flag" == "true" ]]; then
create_manifest "$environment" "$output_dir"
fi
#normalize output directory path
#output_dir="${output_dir}/firmware-${environment}"
output_dir="${output_dir}/firmware_build"
# Compress if requested
if [[ "$compress_flag" == "true" ]]; then
compress_artifacts "$output_dir" "$environment"
# Create output directory
prepare_output_dir "$output_dir" "$clean_flag"
# Prepare artifacts based on mode
prepare_artifacts "$environment" "$output_dir" || exit 1
# Prepare libraries
prepare_libraries "$environment" "$output_dir" || exit 1
fi
# Show summary
+70 -5
View File
@@ -6,8 +6,10 @@
set -euo pipefail
# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
readonly SCRIPT_DIR
readonly PROJECT_ROOT
# Load shared configuration
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
@@ -18,7 +20,6 @@ else
fi
# Default values
CHECK_MODE=true
FIX_MODE=false
FORMAT_ONLY=false
SOURCE_DIR="main"
@@ -202,6 +203,63 @@ fix_formatting() {
fi
}
# Function to run shellcheck on scripts
run_shellcheck() {
log_info "Checking shell scripts with ShellCheck..."
# Check if shellcheck is installed
if ! command -v shellcheck >/dev/null 2>&1; then
log_warn "ShellCheck not found, skipping shell script checks"
log_info "To install ShellCheck:"
log_info " Ubuntu/Debian: sudo apt-get install shellcheck"
log_info " macOS: brew install shellcheck"
return 0 # Don't fail, just skip
fi
log_info "✓ ShellCheck found: $(shellcheck --version | head -n2 | tail -n1)"
# Find all .sh files in scripts directory
local scripts_dir="${PROJECT_ROOT}/scripts"
if [[ ! -d "$scripts_dir" ]]; then
log_warn "Scripts directory not found: $scripts_dir"
return 0
fi
local shell_files
shell_files=$(find "$scripts_dir" -type f -name "*.sh" 2>/dev/null)
if [[ -z "$shell_files" ]]; then
log_warn "No shell scripts found in $scripts_dir"
return 0
fi
local file_count
file_count=$(echo "$shell_files" | wc -l)
log_info "Found $file_count shell script(s) to check"
# Run shellcheck on all files at once to allow cross-file analysis
log_info "Running ShellCheck on scripts directory..."
local shellcheck_output
# shellcheck disable=SC2086
shellcheck_output=$(shellcheck -f gcc $shell_files 2>&1)
local shellcheck_result=$?
if [[ $shellcheck_result -ne 0 ]]; then
echo ""
log_warn "⚠ ShellCheck found issues:"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "$shellcheck_output"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo ""
log_error "Please fix the ShellCheck warnings/errors"
return 1
fi
log_success "✓ All $file_count shell script(s) passed ShellCheck"
return 0
}
# Function to run all QA checks
run_all_checks() {
log_info "Running all quality assurance checks..."
@@ -215,6 +273,13 @@ run_all_checks() {
fi
echo ""
# ShellCheck
log_info "═══ Shell Scripts Check ═══"
if ! run_shellcheck; then
all_passed=false
fi
echo ""
# Future: Add more checks here
# - cppcheck static analysis
# - code complexity metrics
@@ -237,6 +302,7 @@ run_format_check() {
local clang_format_cmd
clang_format_cmd=$(check_clang_format "$CLANG_FORMAT_VERSION")
# shellcheck disable=SC2181
if [[ $? -ne 0 ]] || [[ -z "$clang_format_cmd" ]]; then
log_error "clang-format not found"
log_error "Please install clang-format:"
@@ -258,6 +324,7 @@ run_format_check() {
local files
files=$(find_files "$SOURCE_DIR" "$EXTENSIONS")
# shellcheck disable=SC2181
if [[ $? -ne 0 ]] || [[ -z "$files" ]]; then
log_error "Source directory not found: ${PROJECT_ROOT}/${SOURCE_DIR}"
return 1
@@ -317,13 +384,11 @@ parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--check)
CHECK_MODE=true
FIX_MODE=false
shift
;;
--fix)
FIX_MODE=true
CHECK_MODE=false
shift
;;
--format)
+428
View File
@@ -0,0 +1,428 @@
#!/bin/bash
# OpenMQTTGateway Security Scan Script
# Runs Trivy vulnerability scanner and generates reports
# Based on task-security-scan.yml workflow
# Usage: ./scripts/ci_security.sh [OPTIONS]
set -euo pipefail
# Constants
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
readonly SCRIPT_DIR
readonly PROJECT_ROOT
# Load shared configuration (colors, logging functions, paths)
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
source "${SCRIPT_DIR}/ci_00_config.sh"
else
echo "ERROR: ci_00_config.sh not found" >&2
exit 1
fi
# Default values
SCAN_TYPE="fs"
SEVERITY="HIGH,CRITICAL"
EXIT_CODE_ON_VULN="0"
SCAN_PATH="."
EXCLUDE_PATHS=""
GENERATE_SBOM=false # Off by default
OUTPUT_DIR="${PROJECT_ROOT}/${REPORTS_DIR}"
# Show usage
usage() {
cat << EOF
Usage: $0 [OPTIONS]
Security vulnerability scanning using Trivy
Options:
--scan-type <type> Type of scan: fs (filesystem), config, or image
Default: fs
--severity <levels> Severity levels to report (comma-separated)
Options: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
Default: HIGH,CRITICAL
--scan-path <path> Path to scan (default: entire repository)
Default: .
--exit-code <code> Exit code when vulnerabilities found
0 = continue (don't fail)
1 = fail build
Default: 0
--exclude <paths> Paths to exclude from scan (comma-separated)
Example: node_modules,test,docs
Default: (none)
--generate-sbom Generate Software Bill of Materials (SBOM)
Creates CycloneDX and SPDX JSON formats
Default: off
--output-dir <dir> Directory to save reports
Default: ./generated/reports
-h, --help Show this help message
Examples:
# Scan filesystem for HIGH and CRITICAL issues
$0
# Scan config files and fail if vulnerabilities found
$0 --scan-type config --exit-code 1
# Scan specific path for all severity levels
$0 --scan-path ./main --severity UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
# Scan with excluded paths
$0 --exclude node_modules,test,docs
# Scan with multiple options
$0 --scan-path ./main --exclude test --severity HIGH,CRITICAL --exit-code 1
# Scan with SBOM generation
$0 --generate-sbom
# Complete scan with SBOM and custom severity
$0 --severity UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL --generate-sbom
EOF
exit 0
}
# Function to check if Trivy is installed
check_trivy() {
if ! command -v trivy &> /dev/null; then
log_error "Trivy is not installed. Install it from: https://github.com/aquasecurity/trivy"
return 1
fi
log_success "Trivy found: $(trivy --version)"
}
# Function to check if jq is installed (for JSON parsing)
check_jq() {
if ! command -v jq &> /dev/null; then
log_warn "jq is not installed. Some report features will be limited."
return 1
fi
return 0
}
# Function to run Trivy scan with specified format
# Usage: run_trivy_scan <format> <filename> [exit_code]
run_trivy_scan() {
local format="$1"
local filename="$2"
local exit_code="${3:-0}" # Default to 0 for non-critical scans
local output_file="${OUTPUT_DIR}/${filename}"
log_info "Running Trivy $SCAN_TYPE scan ($format format)..."
mkdir -p "$OUTPUT_DIR"
# Build Trivy command with all enabled scanners
local trivy_cmd=(
trivy "$SCAN_TYPE" "$SCAN_PATH"
--format "$format"
--output "$output_file"
--severity "$SEVERITY"
--exit-code "$exit_code"
)
# Add exclude paths if provided
if [[ -n "$EXCLUDE_PATHS" ]]; then
IFS=',' read -r -a exclude_dirs <<< "$EXCLUDE_PATHS"
for skip_dir in "${exclude_dirs[@]}"; do
if [[ -n "$skip_dir" ]]; then
trivy_cmd+=(--skip-dirs "$skip_dir")
fi
done
fi
log_info "Executing Trivy ${SCAN_TYPE} scan"
if "${trivy_cmd[@]}" 2>&1 | tee -a "${OUTPUT_DIR}/trivy-scan.log"; then
log_success "${format^^} report generated: $output_file"
return 0
else
local rc=$?
if [[ $rc -eq 1 && "$exit_code" == "1" ]]; then
log_error "Vulnerabilities found (exit code: $rc)"
return $rc
fi
log_success "Scan completed with exit code: $rc"
return 0
fi
}
# Function to create summary report
create_summary_report() {
log_info "Creating security summary report..."
local summary_file="${OUTPUT_DIR}/security-summary.md"
{
echo "# 🔒 Security Scan Results"
echo ""
echo "**Scan Type**: $SCAN_TYPE"
echo "**Path Scanned**: $SCAN_PATH"
echo "**Severity Filter**: $SEVERITY"
echo "**Scan Date**: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
echo ""
} > "$summary_file"
# Parse SARIF if it exists
local sarif_file="${OUTPUT_DIR}/trivy-results.sarif"
if [[ -f "$sarif_file" ]]; then
log_info "Parsing SARIF results..."
if check_jq; then
# Count vulnerabilities by severity
local vuln_count
vuln_count=$(jq '[.runs[].results[]] | length' "$sarif_file" 2>/dev/null || echo "0")
local critical high medium low
critical=$(jq '[.runs[].results[] | select(.level == "error")] | length' "$sarif_file" 2>/dev/null || echo "0")
high=$(jq '[.runs[].results[] | select(.level == "warning")] | length' "$sarif_file" 2>/dev/null || echo "0")
medium=$(jq '[.runs[].results[] | select(.level == "note")] | length' "$sarif_file" 2>/dev/null || echo "0")
low=$(jq '[.runs[].results[] | select(.level == "none")] | length' "$sarif_file" 2>/dev/null || echo "0")
{
echo "## Vulnerability Summary"
echo ""
echo "**Total Vulnerabilities**: ${vuln_count}"
echo ""
echo "- 🔴 **Critical**: ${critical}"
echo "- 🟠 **High**: ${high}"
echo "- 🟡 **Medium**: ${medium}"
echo "- 🟢 **Low**: ${low}"
echo ""
} >> "$summary_file"
# List vulnerability details
if [[ "$vuln_count" -gt 0 ]]; then
{
echo "## Vulnerability Details"
echo ""
} >> "$summary_file"
jq -r '.runs[].results[] |
"### \(.level | ascii_upcase): \(.ruleId)\n" +
"**Location**: \(.locations[0].physicalLocation.artifactLocation.uri // "N/A")\n" +
"**Message**: \(.message.text)\n" +
(if .properties.CVE then "**CVE**: \(.properties.CVE)\n" else "" end) +
(if .properties.cvss then "**CVSS Score**: \(.properties.cvss)\n" else "" end) +
""' "$sarif_file" >> "$summary_file" 2>/dev/null || \
echo "Could not parse vulnerability details" >> "$summary_file"
fi
else
echo "⚠️ jq not available for detailed SARIF parsing" >> "$summary_file"
fi
else
echo "⚠️ SARIF file not found" >> "$summary_file"
fi
# Add available report formats
{
echo "## Report Formats Available"
echo ""
if [[ -f "${OUTPUT_DIR}/trivy-results.sarif" ]]; then
echo "- ✅ \`trivy-results.sarif\` - SARIF format (for GitHub/IDE integration)"
fi
if [[ -f "${OUTPUT_DIR}/trivy-report.json" ]]; then
echo "- ✅ \`trivy-report.json\` - JSON format (for automation)"
fi
if [[ -f "${OUTPUT_DIR}/trivy-report.txt" ]]; then
echo "- ✅ \`trivy-report.txt\` - Human-readable table"
fi
echo ""
} >> "$summary_file"
# Add footer
{
echo "---"
echo ""
echo "Generated by OpenMQTTGateway CI/CD Security Scan"
echo "For more information, see: https://github.com/aquasecurity/trivy"
} >> "$summary_file"
log_success "Summary report generated: $summary_file"
}
# Function to check for critical vulnerabilities and fail if needed
check_critical_vulnerabilities() {
if [[ "$EXIT_CODE_ON_VULN" != "1" ]]; then
return 0
fi
local sarif_file="${OUTPUT_DIR}/trivy-results.sarif"
if [[ ! -f "$sarif_file" ]]; then
log_warn "SARIF file not found, skipping critical check"
return 0
fi
if check_jq; then
local critical
critical=$(jq '[.runs[].results[] | select(.level == "error")] | length' "$sarif_file" 2>/dev/null || echo "0")
if [[ "$critical" -gt 0 ]]; then
log_error "❌ Found ${critical} critical vulnerabilities!"
log_error "Review the security reports in: $OUTPUT_DIR"
return 1
fi
fi
return 0
}
# Main execution
main() {
# Parse arguments
while [[ $# -gt 0 ]]; do
case "$1" in
--scan-type)
if [[ -z "${2:-}" ]]; then
log_error "--scan-type requires an argument"
return 1
fi
SCAN_TYPE="$2"
shift 2
;;
--severity)
if [[ -z "${2:-}" ]]; then
log_error "--severity requires an argument"
return 1
fi
SEVERITY="$2"
shift 2
;;
--scan-path)
if [[ -z "${2:-}" ]]; then
log_error "--scan-path requires an argument"
return 1
fi
SCAN_PATH="$2"
shift 2
;;
--exit-code)
if [[ -z "${2:-}" ]]; then
log_error "--exit-code requires an argument"
return 1
fi
EXIT_CODE_ON_VULN="$2"
shift 2
;;
--exclude)
if [[ -z "${2:-}" ]]; then
log_error "--exclude requires an argument"
return 1
fi
EXCLUDE_PATHS="$2"
shift 2
;;
--generate-sbom)
GENERATE_SBOM=true
shift
;;
--output-dir)
if [[ -z "${2:-}" ]]; then
log_error "--output-dir requires an argument"
return 1
fi
OUTPUT_DIR="$2"
shift 2
;;
-h|--help)
usage
;;
*)
log_error "Unknown option: $1"
echo ""
usage
;;
esac
done
# Pre-flight checks
log_info "Running security scan checks..."
if ! check_trivy; then
log_error "Required tool check failed"
return 1
fi
check_jq || true
echo ""
log_info "═══ Security Scan Configuration ═══"
echo " Scan Type: $SCAN_TYPE"
echo " Path: $SCAN_PATH"
echo " Severity: $SEVERITY"
echo " Exit on Vulnerabilities: $EXIT_CODE_ON_VULN"
echo " Excluded Paths: ${EXCLUDE_PATHS:-none}"
echo " Generate SBOM: ${GENERATE_SBOM:-false}"
echo " Output Directory: $OUTPUT_DIR"
echo ""
# Run scans
if ! run_trivy_scan "sarif" "trivy-results.sarif" "$EXIT_CODE_ON_VULN"; then
if [[ "$EXIT_CODE_ON_VULN" == "1" ]]; then
return 1
fi
fi
run_trivy_scan "json" "trivy-report.json" "0"
run_trivy_scan "table" "trivy-report.txt" "0"
# Create summary
create_summary_report
# Generate SBOM if requested
if [[ "$GENERATE_SBOM" == "true" ]]; then
log_info "Generating Software Bill of Materials (SBOM)..."
local sbom_dir="${OUTPUT_DIR}/sbom"
mkdir -p "$sbom_dir"
# Generate CycloneDX SBOM
log_info "Generating CycloneDX SBOM..."
if trivy "$SCAN_TYPE" "$SCAN_PATH" \
--format cyclonedx \
--output "$sbom_dir/sbom-cyclonedx.json" \
--exit-code 0 2>&1 | grep -i "generated\|error" || true; then
if [[ -f "$sbom_dir/sbom-cyclonedx.json" ]]; then
log_success "CycloneDX SBOM generated: $sbom_dir/sbom-cyclonedx.json"
fi
fi
# Generate SPDX SBOM
log_info "Generating SPDX SBOM..."
if trivy "$SCAN_TYPE" "$SCAN_PATH" \
--format spdx-json \
--output "$sbom_dir/sbom-spdx.json" \
--exit-code 0 2>&1 | grep -i "generated\|error" || true; then
if [[ -f "$sbom_dir/sbom-spdx.json" ]]; then
log_success "SPDX SBOM generated: $sbom_dir/sbom-spdx.json"
fi
fi
if [[ -f "$sbom_dir/sbom-cyclonedx.json" ]] || [[ -f "$sbom_dir/sbom-spdx.json" ]]; then
log_success "SBOM generation completed"
echo ""
fi
fi
# Check for critical vulnerabilities
if ! check_critical_vulnerabilities; then
return 1
fi
echo ""
log_success "Security scan completed successfully!"
log_info "Reports saved to: $OUTPUT_DIR"
return 0
}
# Execute main function
main "$@"
-197
View File
@@ -1,197 +0,0 @@
#!/bin/bash
# Updates version tags in firmware configuration and JSON files
# Used by: CI/CD pipelines for versioning builds
# Usage: ./set_version.sh <version_tag> [--dev]
set -euo pipefail
# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
readonly USER_CONFIG="${PROJECT_ROOT}/main/User_config.h"
readonly LATEST_VERSION_PROD="${SCRIPT_DIR}/latest_version.json"
readonly LATEST_VERSION_DEV="${SCRIPT_DIR}/latest_version_dev.json"
# Load shared configuration (colors, logging functions, paths)
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
source "${SCRIPT_DIR}/ci_00_config.sh"
else
echo "ERROR: ci_00_config.sh not found" >&2
exit 1
fi
# Function to validate version tag
validate_version() {
local version="$1"
if [[ -z "$version" ]] || [[ "$version" == "version_tag" ]]; then
log_error "Invalid version tag: '$version'"
return 1
fi
log_info "Version tag validated: $version"
}
# Function to backup files
backup_file() {
local file="$1"
if [[ -f "$file" ]]; then
cp "$file" "${file}.bak"
log_info "Backed up: $file"
fi
}
# Function to replace version in file
replace_version() {
local file="$1"
local version="$2"
if [[ ! -f "$file" ]]; then
log_warn "File not found: $file (skipping)"
return 0
fi
# Backup before modification
backup_file "$file"
# Replace version_tag placeholder
if sed -i "s/version_tag/${version}/g" "$file"; then
log_info "Updated version in: $file"
else
log_error "Failed to update version in: $file"
return 1
fi
}
# Function to set production version
set_production_version() {
local version="$1"
log_info "Setting PRODUCTION version: $version"
replace_version "$USER_CONFIG" "$version" || return 1
replace_version "$VERSION_JSON" "$version" || return 1
}
# Function to set development version
set_development_version() {
local version="$1"
log_info "Setting DEVELOPMENT version: $version"
replace_version "$USER_CONFIG" "$version" || return 1
replace_version "$VERSION_DEV_JSON" "$version" || return 1
}
# Function to restore backups
restore_backups() {
log_warn "Restoring backups..."
for file in "$USER_CONFIG" "$VERSION_JSON" "$VERSION_DEV_JSON"; do
if [[ -f "${file}.bak" ]]; then
mv "${file}.bak" "$file"
log_info "Restored: $file"
fi
done
}
# Function to clean backups
clean_backups() {
for file in "$USER_CONFIG" "$VERSION_JSON" "$VERSION_DEV_JSON"; do
if [[ -f "${file}.bak" ]]; then
rm "${file}.bak"
fi
done
}
# Show usage
usage() {
cat << EOF
Usage: $0 <version_tag> [OPTIONS]
Update version tags in firmware configuration files.
Arguments:
version_tag Version string to inject (e.g., v1.2.3, abc123, dev-20230101)
Options:
--dev Use development version files (latest_version_dev.json)
--prod Use production version files (latest_version.json) [default]
--help Show this help message
Examples:
$0 v1.2.3 # Production release
$0 abc123 --dev # Development build
$0 \${{ github.sha }} --dev # CI/CD with commit SHA
EOF
}
# Main execution
main() {
local version=""
local is_dev=false
# Parse arguments
while [[ $# -gt 0 ]]; do
case "$1" in
--dev)
is_dev=true
shift
;;
--prod)
is_dev=false
shift
;;
--help|-h)
usage
exit 0
;;
-*)
log_error "Unknown option: $1"
usage
exit 1
;;
*)
version="$1"
shift
;;
esac
done
# Validate version
if [[ -z "$version" ]]; then
log_error "Version tag is required"
usage
exit 1
fi
validate_version "$version" || exit 1
# Change to project root
cd "$PROJECT_ROOT"
# Set version with error handling
if [[ "$is_dev" == true ]]; then
set_development_version "$version" || {
restore_backups
exit 1
}
else
set_production_version "$version" || {
restore_backups
exit 1
}
fi
# Clean up backups on success
clean_backups
log_info "Version update completed successfully"
}
# Run main if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
+159 -308
View File
@@ -1,16 +1,19 @@
#!/bin/bash
# CI/CD Site/Documentation Builder
# Builds and deploys VuePress documentation with version management
# Usage: ./scripts/ci_site.sh [OPTIONS]
# This script builds and deploys VuePress documentation with version management.
# Usage: ./scripts/ci_site.sh [OPTIONS] - Specify options for the script execution.
set -euo pipefail
# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
readonly DOCS_DIR="${PROJECT_ROOT}/docs"
readonly VUEPRESS_CONFIG="${DOCS_DIR}/.vuepress/config.js"
readonly VUEPRESS_BUILD_DIR="${DOCS_DIR}/.vuepress/dist"
# Resolve the folder containing this script so relative paths work
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
DOCS_DIR="${PROJECT_ROOT}/docs"
readonly SCRIPT_DIR
readonly PROJECT_ROOT
readonly DOCS_DIR
# Load shared configuration
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
@@ -20,20 +23,14 @@ else
exit 1
fi
# Final output directory for site
# Final output directory for the site
readonly SITE_OUTPUT_DIR="${PROJECT_ROOT}/${SITE_DIR}"
# Default values
MODE="prod"
URL_PREFIX="/"
CUSTOM_VERSION=""
VERSION_SOURCE="release"
PREVIEW=false
GENERATE_WEBUPLOADER=true
WEBUPLOADER_ARGS=""
DEPLOY_DIR="."
RUN_PAGESPEED=false
PAGESPEED_URL="https://docs.openmqttgateway.com/"
MODE="prod" # Set the default mode to production
CURL_INSECURE=false # Allow curl to skip TLS verification (use only when needed)
CLEAN=false # Clean generated/site before build
LEGACY_OPENSSL=false # Use --openssl-legacy-provider for older Node.js versions
# Function to check required tools
check_requirements() {
@@ -42,7 +39,7 @@ check_requirements() {
local missing_tools=()
if ! command -v node >/dev/null 2>&1; then
missing_tools+=("node")
missing_tools+=("node") # Node.js is required
else
local node_version
node_version=$(node --version)
@@ -50,27 +47,19 @@ check_requirements() {
fi
if ! command -v npm >/dev/null 2>&1; then
missing_tools+=("npm")
missing_tools+=("npm") # npm is required
else
local npm_version
npm_version=$(npm --version)
log_info "✓ npm ${npm_version} found"
fi
if ! command -v python3 >/dev/null 2>&1; then
missing_tools+=("python3")
if ! command -v openssl >/dev/null 2>&1; then
missing_tools+=("openssl") # OpenSSL is required
else
local python_version
python_version=$(python3 --version | grep -oP '\d+\.\d+' || echo "unknown")
log_info "✓ Python ${python_version} found"
fi
if ! command -v pip3 >/dev/null 2>&1; then
missing_tools+=("pip3")
else
local pip_version
pip_version=$(pip3 --version | grep -oP '\d+\.\d+' || echo "unknown")
log_info "✓ pip ${pip_version} found"
local openssl_version
openssl_version=$(openssl version | grep -oP '\d+\.\d+\.\d+' || echo "unknown")
log_info "✓ OpenSSL ${openssl_version} found"
fi
if [[ ${#missing_tools[@]} -gt 0 ]]; then
@@ -86,21 +75,6 @@ check_requirements() {
install_dependencies() {
log_info "Installing dependencies..."
# Upgrade pip first (if pip module is available)
if python3 -m pip --version >/dev/null 2>&1; then
log_info "Upgrading pip..."
python3 -m pip install --upgrade pip --quiet || {
log_warn "Failed to upgrade pip, continuing with existing version..."
}
fi
# Install Python dependencies (without --user if in virtualenv)
log_info "Installing Python dependencies..."
pip3 install requests pandas markdown pytablereader tabulate || {
log_error "Failed to install Python dependencies"
return 1
}
# Install Node dependencies
log_info "Installing Node.js dependencies..."
cd "${PROJECT_ROOT}"
@@ -118,118 +92,49 @@ download_common_config() {
local config_url="https://www.theengs.io/commonConfig.js"
local config_dest="${DOCS_DIR}/.vuepress/public/commonConfig.js"
local curl_opts="-sSf"
# Optionally disable TLS verification if explicitly requested
if [[ "${CURL_INSECURE}" == "true" ]]; then
curl_opts+="k"
log_warn "curl is running with --insecure; TLS verification is disabled"
fi
mkdir -p "$(dirname "$config_dest")"
if curl -sSf -o "$config_dest" "$config_url"; then
if curl ${curl_opts} -o "$config_dest" "$config_url"; then
log_info "✓ Common config downloaded"
else
log_warn "Failed to download common config, continuing anyway..."
fi
}
# Function to get version
get_version() {
local version=""
if [[ "$VERSION_SOURCE" == "custom" && -n "$CUSTOM_VERSION" ]]; then
version="$CUSTOM_VERSION"
log_info "Using custom version: $version"
elif [[ "$VERSION_SOURCE" == "release" ]]; then
# Try to get latest git tag (simulating GitHub release)
if command -v git >/dev/null 2>&1 && [[ -d "${PROJECT_ROOT}/.git" ]]; then
version=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
log_info "Using release version from git: $version"
else
version="development"
log_warn "Git not available, using version: $version"
fi
else
# Auto-detect from git
if command -v git >/dev/null 2>&1 && [[ -d "${PROJECT_ROOT}/.git" ]]; then
version=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
log_info "Using git version: $version"
else
version="development"
log_warn "Git not available, using version: $version"
fi
fi
echo "$version"
}
# Function to set version in config files
set_version() {
local version="$1"
log_info "Setting version: $version"
# Update VuePress config
if [[ -f "$VUEPRESS_CONFIG" ]]; then
sed -i "s|version_tag|${version}|g" "$VUEPRESS_CONFIG"
fi
# Update version JSON file based on version source
if [[ "$VERSION_SOURCE" == "custom" ]]; then
# Custom version updates dev file
local version_file="${SCRIPT_DIR}/latest_version_dev.json"
if [[ -f "$version_file" ]]; then
sed -i "s|version_tag|${version}|g" "$version_file"
fi
else
# Release version updates production file
local version_file="${SCRIPT_DIR}/latest_version.json"
if [[ -f "$version_file" ]]; then
sed -i "s|version_tag|${version}|g" "$version_file"
fi
fi
}
# Function to set URL prefix (base path)
set_url_prefix() {
create_configuration_files() {
local url_prefix="$1"
local version="$2"
local mode="$3"
local dest_line=""
if [[ "$url_prefix" != "/" ]]; then
log_info "Setting URL prefix: $url_prefix"
sed -i "s|base: '/'|base: '${url_prefix}'|g" "$VUEPRESS_CONFIG"
if [[ "$mode" == "dev" ]]; then
dest_line=" \"dest\": \"generated/site/dev\","
fi
# download common config
download_common_config
## Create a meta.json file on config folder
local meta_file="${DOCS_DIR}/.vuepress/meta.json"
cat > "$meta_file" <<EOF
{
"title": "Theengs OpenMQTTGateway",
"version": "${version}",
"url_prefix": "${url_prefix}",
${dest_line}
"mode": "${mode}"
}
# Function to generate board documentation
generate_board_docs() {
log_info "Generating board documentation..."
local generator="${SCRIPT_DIR}/generate_board_docs.py"
if [[ -f "$generator" ]]; then
cd "${PROJECT_ROOT}"
python3 "$generator" || {
log_warn "Board documentation generation failed, continuing..."
}
else
log_warn "Board documentation generator not found, skipping..."
fi
}
# Function to generate WebUploader manifest
generate_webuploader() {
if [[ "$GENERATE_WEBUPLOADER" != true ]]; then
log_info "Skipping WebUploader generation"
return 0
fi
log_info "Generating WebUploader manifest..."
local generator="${SCRIPT_DIR}/gen_wu.py"
if [[ -f "$generator" ]]; then
cd "${PROJECT_ROOT}"
python3 "$generator" $WEBUPLOADER_ARGS || {
log_warn "WebUploader generation failed, continuing..."
}
else
log_warn "WebUploader generator not found, skipping..."
fi
EOF
log_info "✓ Generated meta.json at ${meta_file}"
}
# Function to build documentation
@@ -238,45 +143,44 @@ build_docs() {
cd "${PROJECT_ROOT}"
# Set Node options for compatibility with newer Node.js versions
export NODE_OPTIONS="--openssl-legacy-provider"
# Set Node options for compatibility with older Node.js versions (if requested)
if [[ "$LEGACY_OPENSSL" == true ]]; then
log_info "Using --openssl-legacy-provider for older Node.js compatibility"
export NODE_OPTIONS="--openssl-legacy-provider"
fi
npm run docs:build || {
log_error "Documentation build failed"
return 1
}
if [[ -d "$VUEPRESS_BUILD_DIR" ]]; then
log_info "✓ Documentation built successfully"
# Copy to centralized output directory
log_info "Copying site to: $SITE_OUTPUT_DIR"
mkdir -p "$SITE_OUTPUT_DIR"
rm -rf "${SITE_OUTPUT_DIR}"/*
cp -r "${VUEPRESS_BUILD_DIR}"/* "${SITE_OUTPUT_DIR}/"
log_info " VuePress output: $VUEPRESS_BUILD_DIR"
log_info " Final output: $SITE_OUTPUT_DIR"
else
log_error "Build output directory not found: $VUEPRESS_BUILD_DIR"
return 1
fi
log_info "✓ Documentation built successfully"
}
# Function to preview documentation
preview_docs() {
log_info "Starting documentation preview..."
log_info "Starting local preview server..."
if [[ ! -d "$SITE_OUTPUT_DIR" ]]; then
log_error "Build output not found. Run build first."
cd "${PROJECT_ROOT}"
npm run site:preview || {
log_error "Failed to start preview server"
return 1
}
log_info "Preview server running at https://localhost:8443"
log_info "Press Ctrl+C to stop the preview server"
}
# Function to clean generated site folder
clean_build() {
local site_folder="${PROJECT_ROOT}/generated/site"
if [[ -d "$site_folder" ]]; then
log_info "Cleaning ${site_folder}..."
rm -rf "$site_folder"
log_info "✓ Cleaned ${site_folder}"
else
log_info "Nothing to clean: ${site_folder} does not exist"
fi
log_info "Preview server starting at: http://localhost:8080"
log_info "Press Ctrl+C to stop"
cd "$SITE_OUTPUT_DIR"
python3 -m http.server 8080
}
# Show usage
@@ -284,175 +188,133 @@ usage() {
cat << EOF
Usage: $0 [OPTIONS]
Build and deploy OpenMQTTGateway documentation site using VuePress.
This script handles the complete documentation build pipeline including:
- Installing Node.js and Python dependencies
- Downloading shared configuration from theengs.io
- Version management (from git tags or custom version)
- Generating board documentation from environments
- Creating WebUploader manifest for firmware updates
- Building VuePress static site
- Optional local preview or deployment to GitHub Pages
Build and deploy OpenMQTTGateway documentation with VuePress.
Generates site structure and meta.json file.
OPTIONS:
--mode MODE
Build mode: 'prod' or 'dev' [default: prod]
- prod: Production documentation with release version
- dev: Development documentation with custom version tag
Build mode: 'prod' (production) or 'dev' (development) [default: dev]
--url-prefix PATH
URL prefix for documentation routing (VuePress base path) [default: /]
Controls how URLs are generated in the built site.
Examples:
--url-prefix / # Production site at root URL
--url-prefix /dev/ # Development site at /dev/ URL path
Note: Should match --deploy-dir for correct link generation
Base URL path for links (e.g., '/' for root, '/dev/' for dev) [default: /dev/]
--custom-version TAG
Override version tag displayed in documentation
Example: --custom-version "v1.8.0-beta"
Note: Automatically sets --version-source to 'custom'
--version-source SOURCE
Source for version information: 'release' or 'custom' [default: release]
- release: Use latest git tag from repository
- custom: Use version from --custom-version parameter
--version VERSION
Version string written to meta.json (e.g., '1.8.0', 'edge') [default: edge]
--preview
Start local HTTP server to preview built documentation
Server will run at http://localhost:8080
Press Ctrl+C to stop the preview server
Start local HTTPS preview server after build (https://localhost:8443)
--deploy-dir DIR
Deployment directory on GitHub Pages [default: .]
Controls where files are copied in the gh-pages branch.
Examples:
--deploy-dir . # Deploy to root of gh-pages branch
--deploy-dir dev # Deploy to dev/ folder in gh-pages
Note: Should match --url-prefix for correct site structure
--clean
Remove generated/site folder before build
--no-webuploader
Skip WebUploader manifest generation
By default, generates manifest for web-based firmware updates
--insecure-curl
Allow curl to skip TLS verification (use only if cert errors block download)
--webuploader-args ARGS
Additional arguments passed to gen_wu.py script
Example: --webuploader-args "--dev"
--run-pagespeed
Run Google PageSpeed Insights after deployment
Requires APIKEY to be configured in workflow
--pagespeed-url URL
URL to test with PageSpeed Insights
Default: https://docs.openmqttgateway.com/
--legacy-openssl
Use --openssl-legacy-provider for older Node.js versions (pre-17)
--help
Show this help message and exit
Show this message and exit
EXAMPLES:
# Production build
$0 --mode prod --url-prefix / --version 1.8.0
# Build production documentation with latest release version
$0 --mode prod
# Development build with preview
$0 --mode dev --url-prefix /dev/ --version edge --preview
# Build and preview development documentation locally
$0 --mode dev --url-prefix /dev/ --preview
# Build with custom version tag
$0 --custom-version "v1.8.0-beta" --version-source custom
# Build without WebUploader manifest
$0 --mode prod --no-webuploader
# Build for dev environment with custom WebUploader args
$0 --mode dev --url-prefix /dev/ --deploy-dir dev --webuploader-args "--dev"
WORKFLOW:
1. Check requirements (Node.js, npm, Python)
2. Install dependencies (npm packages, Python libraries)
3. Download common configuration from theengs.io
4. Determine version (from git tags or custom)
5. Generate board documentation from PlatformIO environments
6. Generate WebUploader manifest (optional)
7. Build VuePress static site
8. Preview (optional)
OUTPUT:
Built documentation will be in: generated/site/
# Clean and rebuild
$0 --clean --mode prod --version 1.8.0
EOF
exit 0
}
# Main execution
main() {
local version="edge"
local mode="dev"
local url_prefix=""
local url_prefix_set=false
local do_preview=false
# Parse command line arguments
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--mode)
MODE="$2"
if [[ $# -lt 2 ]]; then
log_error "--mode requires an argument: 'prod' or 'dev'"
usage
exit 1
fi
if [[ "$2" != "prod" && "$2" != "dev" ]]; then
log_error "Invalid mode: $2. Must be 'prod' or 'dev'"
usage
exit 1
fi
mode="$2"
shift 2
;;
--url-prefix|--base-path)
URL_PREFIX="$2"
--url-prefix)
url_prefix="$2"
url_prefix_set=true
shift 2
;;
--deploy-dir|--destination-dir)
DEPLOY_DIR="$2"
shift 2
;;
--custom-version)
CUSTOM_VERSION="$2"
VERSION_SOURCE="custom"
shift 2
;;
--version-source)
VERSION_SOURCE="$2"
--version)
version="$2"
shift 2
;;
--preview)
PREVIEW=true
do_preview=true
shift
;;
--no-webuploader)
GENERATE_WEBUPLOADER=false
--clean)
CLEAN=true
shift
;;
--webuploader-args)
WEBUPLOADER_ARGS="$2"
shift 2
;;
--run-pagespeed)
RUN_PAGESPEED=true
--insecure-curl)
CURL_INSECURE=true
shift
;;
--pagespeed-url)
PAGESPEED_URL="$2"
shift 2
--legacy-openssl)
LEGACY_OPENSSL=true
shift
;;
--help)
usage
exit 0
;;
*)
log_error "Unknown option: $1"
usage
exit 0
;;
esac
done
}
# Main execution
main() {
# Set url_prefix based on mode if not explicitly provided
if [[ "$url_prefix_set" == false ]]; then
if [[ "$mode" == "prod" ]]; then
url_prefix="/"
else
url_prefix="/dev/"
fi
fi
local start_time
start_time=$(date +%s)
parse_args "$@"
log_info "Starting site build pipeline..."
log_info "Mode: $MODE"
log_info "URL prefix: $URL_PREFIX"
log_info "Deploy directory: $DEPLOY_DIR"
log_info "Mode: $mode"
log_info "URL prefix: $url_prefix"
# Clean if requested
if [[ "$CLEAN" == "true" ]]; then
clean_build
fi
# Check requirements
check_requirements || exit 1
@@ -460,28 +322,14 @@ main() {
# Install dependencies
install_dependencies || exit 1
# Download common config
download_common_config
# Get and set version
local version
version=$(get_version)
set_version "$version"
# Set URL prefix
set_url_prefix "$URL_PREFIX"
# Generate board documentation
generate_board_docs
# Generate WebUploader manifest
generate_webuploader
# Create configuration files (meta.json)
create_configuration_files "$url_prefix" "$version" "$mode" || exit 1
# Build documentation
build_docs || exit 1
# Preview if requested
if [[ "$PREVIEW" == true ]]; then
if [[ "$do_preview" == true ]]; then
preview_docs
fi
@@ -501,5 +349,8 @@ main() {
echo "╚════════════════════════════════════════╝"
}
# Execute main function
main "$@"
# Run main if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi
-116
View File
@@ -1,116 +0,0 @@
# Common templates and constants for web installer manifest generation
# Used by: scripts/gen_wu.py
import string
mf_temp32 = string.Template('''{
"name": "OpenMQTTGateway",
"new_install_prompt_erase": true,
"builds": [
{
"chipFamily": "ESP32",
"improv": false,
"parts": [
{ "path": "$cp$bl", "offset": 4096 },
{ "path": "$cp$part", "offset": 32768 },
{ "path": "$cp$boot", "offset": 57344 },
{ "path": "$cp$bin", "offset": 65536 }
]
}
]
}''')
mf_temp32c3 = string.Template('''{
"name": "OpenMQTTGateway",
"new_install_prompt_erase": true,
"builds": [
{
"chipFamily": "ESP32-C3",
"improv": false,
"parts": [
{ "path": "$cp$bl", "offset": 0 },
{ "path": "$cp$part", "offset": 32768 },
{ "path": "$cp$boot", "offset": 57344 },
{ "path": "$cp$bin", "offset": 65536 }
]
}
]
}''')
mf_temp32s3 = string.Template('''{
"name": "OpenMQTTGateway",
"new_install_prompt_erase": true,
"builds": [
{
"chipFamily": "ESP32-S3",
"improv": false,
"parts": [
{ "path": "$cp$bl", "offset": 0 },
{ "path": "$cp$part", "offset": 32768 },
{ "path": "$cp$boot", "offset": 57344 },
{ "path": "$cp$bin", "offset": 65536 }
]
}
]
}''')
mf_temp8266 = string.Template('''{
"name": "OpenMQTTGateway",
"new_install_prompt_erase": true,
"builds": [
{
"chipFamily": "ESP8266",
"parts": [{ "path": "$cp$bin", "offset": 0 }]
}
]
}''')
wu_temp_opt = string.Template('''
<option
value="$mff"
>
$mfn
</option>
''')
wu_temp_p1 = '''<template>
<div align="center">
<select>
<optgroup label="ESP32">'''
wu_temp_p2 = '''
</optgroup>
<optgroup label="ESP32C3">'''
wu_temp_p3 = '''
</optgroup>
<optgroup label="ESP32S3">'''
wu_temp_p4 = '''
</optgroup>
<optgroup label="ESP8266">'''
wu_temp_end = '''
</optgroup>
</select><br><br>
<esp-web-install-button erase-first></esp-web-install-button>
</div>
</template>
<script>
export default {
mounted () {
const espWebInstallButton = document.querySelector("esp-web-install-button");
espWebInstallButton.addEventListener("state-changed", (ev) => { console.log(ev.detail) });
const selectFW = document.querySelector("select");
espWebInstallButton.manifest = selectFW.value;
selectFW.addEventListener("change", () => {
espWebInstallButton.manifest = selectFW.value;
});
}
}
</script>'''
manif_path = 'docs/.vuepress/public/firmware_build/'
vue_path = 'docs/.vuepress/components/'
cors_proxy = '' # 'https://cors.bridged.cc/'
esp32_boot = 'https://github.com/espressif/arduino-esp32/raw/2.0.7/tools/partitions/boot_app0.bin'
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env node
'use strict';
const fs = require('fs');
const path = require('path');
function ensureSSLCerts() {
const sslDir = path.join(process.cwd(), '.ssl');
const keyFile = path.join(sslDir, 'key.pem');
const certFile = path.join(sslDir, 'cert.pem');
if (fs.existsSync(keyFile) && fs.existsSync(certFile)) {
return;
}
const { execSync } = require('child_process');
fs.mkdirSync(sslDir, { recursive: true });
try {
execSync(`openssl req -new -x509 -keyout "${keyFile}" -out "${certFile}" -days 365 -nodes -subj "/C=US/ST=State/L=City/O=OpenMQTTGateway/CN=localhost" 2>/dev/null`, {
stdio: 'inherit'
});
console.log('✓ SSL certificate generated');
} catch (err) {
console.error('Failed to generate SSL certificate. Ensure openssl is installed.');
process.exit(1);
}
}
ensureSSLCerts();
-149
View File
@@ -1,149 +0,0 @@
# Creates web installer manifests for ESP Web Tools firmware installation
# Used by: .github/workflows/task-docs.yml
import os
import requests
import json
import argparse
import shutil
from common_wu import mf_temp32, mf_temp32c3, mf_temp8266, wu_temp_opt, wu_temp_p1, wu_temp_p2, wu_temp_p3, wu_temp_p4, wu_temp_end, vue_path, manif_path, cors_proxy, esp32_boot, mf_temp32s3
parser = argparse.ArgumentParser()
parser.add_argument('--dev', action='store_true')
parser.add_argument('repo', nargs='?', default='1technophile/OpenMQTTGateway')
args = parser.parse_args()
repo = args.repo
dev = args.dev
bin_path = 'toDeploy/'
manif_folder = "/firmware_build/"
if not os.path.exists(manif_path):
os.makedirs(manif_path)
if dev:
print('Generate Web Upload in dev mode')
manif_folder = "/dev" + manif_folder
# copy OTA latest version definition
shutil.copy("scripts/latest_version_dev.json",
manif_path + "latest_version_dev.json")
# copy the binaries frombin_path to manif_path
for name in os.listdir(bin_path):
if '.bin' in name:
shutil.copyfile(bin_path + name, (manif_path + name))
else:
print('Generate Web Upload in release mode')
# copy OTA latest version definition
shutil.copy("scripts/latest_version.json", manif_path + "latest_version.json")
release = requests.get('https://api.github.com/repos/' +
repo + '/releases/latest')
rel_data = json.loads(release.text)
if 'assets' in rel_data:
assets = rel_data['assets']
# Download assets into manif_path
for item in range(len(assets)):
name = assets[item]['name']
if 'firmware.bin' in name:
fw_bin = requests.get(assets[item]['browser_download_url'])
filename = assets[item]['browser_download_url'].split('/')[-1]
with open(manif_path + filename, 'wb') as output_file:
output_file.write(fw_bin.content)
print('Downloaded: ' + filename)
if 'partitions.bin' in name:
part_bin = requests.get(assets[item]['browser_download_url'])
filename = assets[item]['browser_download_url'].split('/')[-1]
with open(manif_path + filename, 'wb') as output_file:
output_file.write(part_bin.content)
print('Downloaded: ' + filename)
if 'bootloader.bin' in name:
bl_bin = requests.get(assets[item]['browser_download_url'])
filename = assets[item]['browser_download_url'].split('/')[-1]
with open(manif_path + filename, 'wb') as output_file:
output_file.write(bl_bin.content)
print('Downloaded: ' + filename)
else:
print('Assets not found')
os._exit(1)
if not os.path.exists(vue_path):
os.makedirs(vue_path)
boot_bin = requests.get(esp32_boot)
filename = esp32_boot.split('/')[-1]
with open(manif_path + filename, 'wb') as output_file:
output_file.write(boot_bin.content)
wu_file = open(vue_path + 'web-uploader.vue', 'w')
wu_file.write(wu_temp_p1)
for name in sorted(os.listdir(manif_path)):
if 'firmware.bin' in name and ('esp32c3' not in name ) and ('esp32s3' not in name ) and ('esp32' in name or 'ttgo' in name or 'heltec' in name or 'thingpulse' in name or 'theengs' in name or 'lilygo' in name or 'shelly' in name or 'tinypico' in name):
fw = name.split('-firmware')[0]
man_file = fw + '.manifest.json'
fwp_name = name.split('-firmware')[0] + '-partitions.bin'
fwb_name = name.split('-firmware')[0] + '-bootloader.bin'
mani_str = mf_temp32.substitute({'cp': cors_proxy, 'part': manif_folder + fwp_name.split('/')[-1], 'bin': manif_folder + name, 'bl': manif_folder + fwb_name, 'boot': manif_folder + esp32_boot.split('/')[-1]})
with open(manif_path + man_file, 'w') as nf:
nf.write(mani_str)
wu_file.write(wu_temp_opt.substitute(
{'mff': manif_folder + man_file, 'mfn': fw}))
print('Created: ' + man_file)
wu_file.write(wu_temp_p2)
for name in sorted(os.listdir(manif_path)):
if 'firmware.bin' in name and ('esp32c3' in name ):
fw = name.split('-firmware')[0]
man_file = fw + '.manifest.json'
fwp_name = name.split('-firmware')[0] + '-partitions.bin'
fwb_name = name.split('-firmware')[0] + '-bootloader.bin'
mani_str = mf_temp32c3.substitute({'cp': cors_proxy, 'part': manif_folder + fwp_name, 'bin': manif_folder + name, 'bl': manif_folder + fwb_name, 'boot': manif_folder + esp32_boot.split('/')[-1]})
with open(manif_path + man_file, 'w') as nf:
nf.write(mani_str)
wu_file.write(wu_temp_opt.substitute(
{'mff': manif_folder + man_file, 'mfn': fw}))
print('Created: ' + man_file)
wu_file.write(wu_temp_p3)
for name in sorted(os.listdir(manif_path)):
if 'firmware.bin' in name and ('esp32s3' in name ):
fw = name.split('-firmware')[0]
man_file = fw + '.manifest.json'
fwp_name = name.split('-firmware')[0] + '-partitions.bin'
fwb_name = name.split('-firmware')[0] + '-bootloader.bin'
mani_str = mf_temp32s3.substitute({'cp': cors_proxy, 'part': manif_folder + fwp_name, 'bin': manif_folder + name, 'bl': manif_folder + fwb_name, 'boot': manif_folder + esp32_boot.split('/')[-1]})
with open(manif_path + man_file, 'w') as nf:
nf.write(mani_str)
wu_file.write(wu_temp_opt.substitute(
{'mff': manif_folder + man_file, 'mfn': fw}))
print('Created: ' + man_file)
wu_file.write(wu_temp_p4)
for name in sorted(os.listdir(manif_path)):
if 'firmware.bin' in name and ('nodemcu' in name or 'sonoff' in name or 'rf-wifi-gateway' in name or 'manual-wifi-test' in name or 'rfbridge' in name):
fw = name.split('-firmware')[0]
man_file = fw + '.manifest.json'
mani_str = mf_temp8266.substitute(
{'cp': cors_proxy, 'bin': manif_folder + name})
with open(manif_path + man_file, 'w') as nf:
nf.write(mani_str)
wu_file.write(
manif_folder + wu_temp_opt.substitute({'mff': manif_folder + man_file, 'mfn': fw}))
print('Created: ' + man_file)
wu_file.write(wu_temp_end)
wu_file.close()
-90
View File
@@ -1,90 +0,0 @@
# Generates board documentation table from platformio.ini environments
# Used by: .github/workflows/task-docs.yml
import pytablereader as ptr
import pandas as pd
import os
import re
import configparser
conf = configparser.ConfigParser()
# Init the table with the columns
table_init = pd.DataFrame(columns=['Environment', 'uC', 'Hardware', 'Description', 'Modules', 'Platform',
'Partitions', 'Libraries', 'Options'])
table = table_init
# Parse platformio.ini to retrieve boards information
conf.read('environments.ini')
for each_section in conf.sections():
if ("env:" in each_section and "-test" not in each_section):
env = each_section.replace("env:", "")
uc = ""
board = ""
hardware = ""
description = ""
modules = ""
platform = ""
partitions = ""
libraries = ""
options = ""
for (k, v) in conf.items(each_section):
v = v.replace('{', '').replace('}', '').replace('$', '').replace(
"env:", '').replace('\'', '').replace("-D", "")
if (k == "board"):
uc = v
if (k == "platform"):
platform = v
if (k == "board_build.partitions"):
partitions = v
if (k == "lib_deps"):
libraries = v
libraries = libraries.replace(
"\ncom-esp.lib_deps\n", "").replace(
"\ncom-arduino.lib_deps\n", "").replace("libraries.", "")
if (k == "build_flags"):
options = v
for o in options.split('\n'):
if ("gateway" in o or "sensor" in o or "actuator" in o):
if (modules != ""):
modules = modules + "\n"
modules = modules + o[1:o.rfind("=\"")]
options = options.replace(
"com-esp.build_flags\n", "")
if (k == "custom_description"):
description = v
if (k == "custom_hardware"):
hardware = v
table.loc[len(table.index)] = [env, uc, hardware, description, modules, platform,
partitions, libraries, options]
# Sort rows per Environment name
table.sort_values(by=['Environment'], inplace=True,
key=lambda col: col.str.lower())
# Produce individual file
for ind in table.index:
table_extract = table.iloc[ind]
print(table_extract)
file = open("docs/prerequisites/boards/" +
table.iloc[ind]["Environment"] + ".md", 'w')
table_extract = table_extract.rename_axis("Board index")
table_md = table_extract.to_markdown()
n = file.write(table_md)
file.close()
# Produce list file
# Add link to the file from the environment and replace /n with ,
for ind in table.index:
table['Environment'][ind] = "[" + table['Environment'][ind] + \
"](../prerequisites/boards/" + table['Environment'][ind] + ")"
table = table.replace("\n", ", ", regex=True)
table = table.drop(["Partitions", "Hardware", "Platform", "Options","Modules"], axis=1)
table = table.reset_index(drop=True)
print(table)
# Convert to Markdown and save per Model_Id
table_md = table.to_markdown()
file = open("docs/prerequisites/board.md", 'a')
n = file.write("# Supported\n" + table_md)
file = open("docs/upload/web-install.md", 'a')
n = file.write(table_md)
file.close()
-28
View File
@@ -1,28 +0,0 @@
#!/bin/bash
# Prepares firmware binaries and libraries for GitHub release deployment
# Used by: .github/workflows/release.yml
set -e
echo "renaming bin files with the environment name"
rename -v 's:/:-:g' .pio/build/*/*.bin
mkdir toDeploy
rename 's/.pio-build-//' .*.bin
(
cd .pio/libdeps
echo "replace space by _ in folder names"
find . -type d -name "* *" | while read FNAME; do mv "$FNAME" "${FNAME// /_}"; done
echo "zipping libraries per board"
for i in */
do
zip -r "${i%/}-libraries.zip" "$i"
done
ls -lA
mv *.zip ../../toDeploy
)
# remove binaries for *-all*, *-test* env and only zip containing *-test*
rm -f *-all*.bin *-test*.bin *-test*.zip
echo "zipping code and licence"
zip -r OpenMQTTGateway_sources.zip main LICENSE.txt
mv *.zip toDeploy
mv *.bin toDeploy
ls -lA toDeploy
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env node
'use strict';
// Minimal HTTPS static server for generated/site only.
const fs = require('fs');
const path = require('path');
const https = require('https');
const mime = require('mime-types');
const repoRoot = path.resolve(__dirname, '..');
const siteRoot = path.join(repoRoot, 'generated', 'site');
const port = parseInt(process.argv[2] || 8443, 10);
const keyFile = process.argv[3] || path.join(repoRoot, '.ssl', 'key.pem');
const certFile = process.argv[4] || path.join(repoRoot, '.ssl', 'cert.pem');
if (!fs.existsSync(siteRoot)) {
console.error(`Error: folder not found: ${siteRoot}`);
process.exit(1);
}
if (!fs.existsSync(keyFile) || !fs.existsSync(certFile)) {
console.error(`Error: SSL certificates not found at ${keyFile} or ${certFile}`);
console.error('Use scripts/ensure_ssl_certs.js to generate them.');
process.exit(1);
}
const server = https.createServer({
key: fs.readFileSync(keyFile),
cert: fs.readFileSync(certFile)
}, (req, res) => {
const reqPath = decodeURIComponent(req.url.split('?')[0]);
const safePath = reqPath.endsWith('/') ? `${reqPath}index.html` : reqPath;
const filePath = path.join(siteRoot, safePath);
if (!filePath.startsWith(siteRoot)) {
res.writeHead(403);
return res.end('Forbidden');
}
fs.stat(filePath, (err, stats) => {
if (err || !stats.isFile()) {
res.writeHead(404);
return res.end('Not Found');
}
const type = mime.contentType(path.extname(filePath)) || 'application/octet-stream';
res.writeHead(200, { 'Content-Type': type });
fs.createReadStream(filePath).pipe(res);
});
});
server.listen(port, '0.0.0.0', () => {
console.log(`Serving generated/site over HTTPS at https://localhost:${port}/`);
console.log(`Serving generated/site over HTTPS at https://<host-ip>:${port}/`);
console.log(`Note: remember that if you are testing DEV site you should go on https://<host-ip>:${port}/dev/`);
});
process.on('SIGINT', () => {
server.close(() => process.exit());
});