[SITE] Renew the web board presentation and the ESP32 web upload + [SYS] Security checks (#2277)
* Refactor GitHub Actions workflows for build, documentation, and linting - Consolidated build logic into reusable workflows (`task-build.yml` and `task-docs.yml`) to reduce duplication across multiple workflows. - Introduced `environments.json` to centralize the list of PlatformIO build environments, improving maintainability and clarity. - Updated `build.yml` and `build_and_docs_to_dev.yml` to utilize the new reusable workflows and environment definitions. - Enhanced `release.yml` to streamline the release process and integrate documentation generation. - Created reusable linting workflow (`task-lint.yml`) to standardize code formatting checks across the repository. - Simplified manual documentation workflow by leveraging the new reusable documentation workflow. - Improved artifact management and retention policies across workflows. - Updated dependencies and versions in workflows to ensure compatibility and performance. CI/CD pipeline agnostic of Workflow Engine and integrated on github actions - Implemented ci.sh for orchestrating the complete build pipeline. - Created ci_00_config.sh for centralized configuration of build scripts. - Created ci_build_firmware.sh for building firmware for specified PlatformIO environments. - Created ci_prepare_artifacts.sh for preparing firmware artifacts for upload or deployment. - Created ci_set_version.sh for updating version tags in firmware configuration files. - Created ci_build.sh to orchestrate the complete build pipeline. - Created ci_qa.sh for code linting and formatting checks using clang-format. - Created ci_site.sh for building and deploying VuePress documentation with version management. - Implemented checks for required tools and dependencies in the new scripts. - Improved internal scripts for better error handling and logging. UPDATE the web installer manifest generation and update documentation structure - Enhanced ci_list-env.sh to list environments from a JSON file. - Replaced common_wu.py and gen_wu.py scripts with new npm scripts for site generation and previewing on docsgen/gen_wu.js - Replaced generate_board_docs.py with docsgen/generated_board_docs.js - Added new npm scripts for integration of site generation on build phase. - Created preview_site.js to serve locally generated site over HTTPS with improved error handling. - Added new CI environments for CI builds in environments.json. - Deleted lint.yml as part of workflow cleanup. - Enhanced task-build.yml to include linting as a job and added support for specifying PlatformIO version. - Improved task-docs.yml to handle versioning more effectively and added clean option. Enhance documentation - ADD CLEAR Mark of development version of site - Updated README.md to include detailed workflow dependencies and relationships using mermaid diagrams. - Improved development.md with a quick checklist for contributors and clarified the code style guide. - Enhanced quick_start.md with tips for contributors and streamlined the workflow explanation. LINT FIX - Refined User_config.h for better formatting consistency. - Adjusted blufi.cpp and gatewayBT.cpp for improved code readability and consistency in formatting. - Updated gatewaySERIAL.cpp and mqttDiscovery.cpp to enhance logging error messages. - Improved sensorDS1820.cpp for better logging of device information. Add security scan workflows for vulnerability detection Add SBOM generation and upload to release workflow; update security scan summary handling Add shellcheck suppor + FIX shellcheck warning Enhance documentation for CI/CD scripts and workflows, adding details for security scanning and SBOM generation processes Fix formatting and alignment in BLE connection handling Reviewed the full web board presentation and the ESP32 web upload. The project uses a modern pattern where data is divided from the presentation layer. - Removed the `generate_board_docs` script. - Updated the `gen_wu` script in order to generate `boards-info.json`: the fail that containe all information about the configuration - Created and isolate the file `boards-info.js` to streamline the parsing of PlatformIO dependencies, modules, environments and improve the handling of library information. - Introduced vuepress component `BoardEnvironmentTable.vue` that render `boards-info.json` as UI card component - Introduced vuepress component `FlashEnvironmentSelector.vue` that render a selectred environment from `boards-info.json` and provide esp-web-upload feature on it - Introduced a new board page `board-selector.md` for improved firmware selection. - Updated `web-install.md` to enhance the firmware upload process, including a new board environment table. - Enhanced custom descriptions in `environments.ini` to include HTML links for better user guidance and board image link Add CC1101 initialization improvements and logging enhancements Add installation step for PlatformIO dependencies in documentation workflow Remove ci_set_version.sh script and associated versioning functionality * Fix comment provisined Fix PlatformIO version input reference in documentation workflow Remove outdated Squeezelite-ESP32 installer documentation
This commit is contained in:
Alessandro Staniscia authored and GitHub committed
1 parent
5f0966c2c1
commit
98481c5145
73 files changed
+5985
-2447
No files matched your search
+427
-212
@@ -11,11 +11,13 @@ This documentation describes the CI/CD scripts used to build OpenMQTTGateway fir
|
||||
- [ci.sh - Main Entry Point](#commands)
|
||||
- [ci.sh build - Build Firmware](#cish-build---build-firmware)
|
||||
- [ci.sh site - Build Documentation](#cish-site---build-documentation)
|
||||
- [ci.sh qa - Code Formatting Check](#cish-qa---code-formatting-check)
|
||||
- [ci.sh qa - Code Quality Checks](#cish-qa---code-quality-checks)
|
||||
- [ci.sh security - Security Vulnerability Scan](#cish-security---security-vulnerability-scan)
|
||||
- [Internal Scripts](#internal-scripts)
|
||||
- [ci_set_version.sh](#ci_set_versionsh)
|
||||
- [ci_list-env.sh](#ci_list-envsh)
|
||||
- [ci_build_firmware.sh](#ci_build_firmwaresh)
|
||||
- [ci_prepare_artifacts.sh](#ci_prepare_artifactssh)
|
||||
- [ci_security.sh](#ci_securitysh)
|
||||
- [ci_00_config.sh](#ci_00_configsh)
|
||||
- [Python Helper Scripts](#python-helper-scripts)
|
||||
- [generate_board_docs.py](#generate_board_docspy)
|
||||
@@ -23,6 +25,7 @@ This documentation describes the CI/CD scripts used to build OpenMQTTGateway fir
|
||||
- [Environment Variables](#environment-variables)
|
||||
- [Exit Codes](#exit-codes)
|
||||
- [Environment Detection](#environment-detection)
|
||||
- [GitHub Actions Workflows Integration](#github-actions-workflows-integration)
|
||||
|
||||
|
||||
|
||||
@@ -32,27 +35,28 @@ This documentation describes the CI/CD scripts used to build OpenMQTTGateway fir
|
||||
|
||||
```
|
||||
ci.sh (dispatcher)
|
||||
├── build → ci_build.sh → ci_set_version.sh
|
||||
│ → ci_build_firmware.sh
|
||||
│ → ci_prepare_artifacts.sh
|
||||
├── site → ci_site.sh → generate_board_docs.py
|
||||
│ → gen_wu.py
|
||||
├── qa → ci_qa.sh → clang-format
|
||||
└── all → qa + build + site (sequential)
|
||||
├── build → ci_build.sh → ci_build_firmware.sh
|
||||
│ → ci_prepare_artifacts.sh (when --deploy-ready)
|
||||
├── site → ci_site.sh
|
||||
├── qa → ci_qa.sh
|
||||
├── security → ci_security.sh (vulnerability scanning with Trivy)
|
||||
├── list-env → ci_list-env.sh
|
||||
└── all → qa + build (all envs with --mode) + site sequential
|
||||
```
|
||||
|
||||
### Script Description
|
||||
|
||||
| Script | Purpose | Called By |
|
||||
|--------|---------|-----------|
|
||||
|--------|---------|----------|
|
||||
| `ci.sh` | Main command dispatcher | User/GitHub Actions |
|
||||
| `ci_build.sh` | Build firmware orchestrator | ci.sh build |
|
||||
| `ci_site.sh` | Documentation build orchestrator | ci.sh site |
|
||||
| `ci_qa.sh` | Code formatting checker | ci.sh qa |
|
||||
| `ci_set_version.sh` | Version injection in firmware | ci_build.sh |
|
||||
| `ci_00_config.sh` | Shared configuration loader | ci.sh |
|
||||
| `ci_qa.sh` | Quality assurance and shellcheck | ci.sh qa |
|
||||
| `ci_security.sh` | Security vulnerability scanning with Trivy | ci.sh security |
|
||||
| `ci_list-env.sh` | List PlatformIO environments (fast JSON or full scan) | ci.sh list-env / direct |
|
||||
| `ci_build_firmware.sh` | PlatformIO build execution | ci_build.sh |
|
||||
| `ci_prepare_artifacts.sh` | Artifact packaging | ci_build.sh |
|
||||
| `ci_00_config.sh` | Shared configuration and functions | All scripts |
|
||||
| `ci_prepare_artifacts.sh` | Artifact packaging (when requested) | ci_build.sh |
|
||||
|
||||
|
||||
### Output Structure
|
||||
@@ -67,16 +71,14 @@ Build outputs are organized in the project root:
|
||||
|
||||
generated/
|
||||
├── artifacts/ # Packaged firmware artifacts
|
||||
└── site/ # Built documentation (VuePress output)
|
||||
|
||||
scripts/
|
||||
├── latest_version.json # Production version metadata
|
||||
└── latest_version_dev.json # Development version metadata
|
||||
├── site/ # Built documentation (VuePress output)
|
||||
└── reports/ # Security scan and SBOM reports
|
||||
└── sbom/ # SBOM in CycloneDX and SPDX formats
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
`ci.sh` is the main Entry Point. Command dispatcher that routes to specialized scripts.
|
||||
`ci.sh` is the main entry point. It dispatches to the specialized scripts.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
@@ -84,115 +86,90 @@ scripts/
|
||||
```
|
||||
|
||||
**Commands:**
|
||||
- `build` - Build firmware for specified environment
|
||||
- `site` or `docs` - Build documentation website
|
||||
- `qa` or `lint` - Run code formatting checks
|
||||
- `all` or `pipeline` - Run complete pipeline (qa + build + site)
|
||||
- `build` - Build firmware for a PlatformIO environment (optionally prepare artifacts)
|
||||
- `site` or `docs` - Build documentation site
|
||||
- `qa` or `lint` - Run formatting and shellcheck checks
|
||||
- `security` - Scan for security vulnerabilities using Trivy (filesystem, container images)
|
||||
- `all` or `pipeline` - Run qa → build (all environments) → site with injected mode
|
||||
- `list-env` - List available PlatformIO environments (JSON fast list or full ini scan)
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Get Help
|
||||
# Help per command
|
||||
./scripts/ci.sh build --help
|
||||
./scripts/ci.sh qa --help
|
||||
./scripts/ci.sh site --help
|
||||
./scripts/ci.sh qa --help
|
||||
./scripts/ci.sh security --help
|
||||
./scripts/ci.sh list-env --help
|
||||
|
||||
# Build firmware
|
||||
./scripts/ci.sh build esp32dev-ble --mode dev
|
||||
./scripts/ci.sh build esp32dev-all-test --version v1.8.0 --deploy-ready
|
||||
./scripts/ci.sh build esp32dev-all-test --mode dev
|
||||
./scripts/ci.sh build esp32dev-bt --version v1.8.0 --deploy-ready --output generated/artifacts
|
||||
|
||||
# Build documentation
|
||||
./scripts/ci.sh site --mode prod
|
||||
./scripts/ci.sh site --mode dev --url-prefix /dev/
|
||||
# Build docs
|
||||
./scripts/ci.sh site --mode prod --url-prefix /
|
||||
./scripts/ci.sh site --mode dev --preview
|
||||
|
||||
# Check code formatting
|
||||
# QA (formatting + shellcheck)
|
||||
./scripts/ci.sh qa --check
|
||||
./scripts/ci.sh qa --fix
|
||||
./scripts/ci.sh qa --fix --verbose
|
||||
|
||||
# Run complete pipeline
|
||||
./scripts/ci.sh all esp32dev-ble --version v1.8.0
|
||||
./scripts/ci.sh all esp32dev-ble --no-site
|
||||
# Security scanning
|
||||
./scripts/ci.sh security --scan-type fs --severity HIGH,CRITICAL
|
||||
./scripts/ci.sh security --scan-type fs --generate-sbom
|
||||
|
||||
# Full pipeline (qa + build all envs + site)
|
||||
./scripts/ci.sh all --mode dev
|
||||
./scripts/ci.sh all --mode prod --preview
|
||||
```
|
||||
|
||||
**Options for `all` command:**
|
||||
- `--no-site` - Skip documentation build step
|
||||
- All options from `build` command are passed through
|
||||
|
||||
---
|
||||
|
||||
### ci.sh build - Build Firmware
|
||||
|
||||
Orchestrates complete firmware build: version injection, compilation, artifact packaging.
|
||||
Runs the build pipeline (tool checks → PlatformIO build → optional artifact packaging).
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci.sh build <environment> [OPTIONS]
|
||||
```
|
||||
|
||||
**Required Arguments:**
|
||||
- `<environment>` - PlatformIO environment name (e.g., esp32dev-ble, nodemcuv2-rf)
|
||||
**Required Argument:**
|
||||
- `<environment>` PlatformIO environment name (e.g., esp32dev-ble)
|
||||
|
||||
**Options:**
|
||||
- `--version <tag>` - Version string to inject (default: auto-generated from git)
|
||||
- `--mode <dev|prod>` - Build mode (default: prod)
|
||||
- `dev` - Enables development OTA, sets DEVELOPMENTOTA=true
|
||||
- `prod` - Standard production build
|
||||
- `--deploy-ready` - Package artifacts with deployment naming (env-firmware.bin)
|
||||
- `--output <dir>` - Output directory for artifacts (default: generated/artifacts)
|
||||
- `--skip-verification` - Skip build tools verification
|
||||
- `--clean` - Clean previous build before starting
|
||||
- `--verbose` - Enable verbose PlatformIO output
|
||||
- `--list-envs` - List all available PlatformIO environments
|
||||
- `--help` - Show help message
|
||||
- `--mode <prod|dev>` Build mode (default: prod). `dev` enables OTA flags in the PlatformIO build.
|
||||
- `--deploy-ready` Copy/rename build outputs and libs via ci_prepare_artifacts.sh.
|
||||
- `--version [TAG]` Set version used for `OMG_VERSION` and artifact folder naming. If TAG is omitted the script auto-generates (CI: BUILD_NUMBER/GIT_COMMIT, local: timestamp).
|
||||
- `--output <dir>` Output directory for packaged artifacts (only used when `--deploy-ready` is set, default `generated/artifacts`).
|
||||
- `--skip-verification` Skip the tool availability checks.
|
||||
- `--clean` Clean the PlatformIO environment before building.
|
||||
- `--verbose` Verbose PlatformIO output.
|
||||
- `--help` Show help.
|
||||
|
||||
**Execution Flow:**
|
||||
```
|
||||
ci.sh build esp32dev-ble --version v1.8.0 --mode prod --deploy-ready
|
||||
│
|
||||
├─> ci_build.sh (orchestrator)
|
||||
│ ├─> verify_build_tools() - Check python3, platformio, git
|
||||
│ ├─> ci_set_version.sh v1.8.0 - Inject version in User_config.h
|
||||
│ ├─> ci_build_firmware.sh esp32dev-ble - Execute PlatformIO build
|
||||
│ └─> ci_prepare_artifacts.sh esp32dev-ble --deploy - Package binaries
|
||||
│
|
||||
└─> Outputs in generated/artifacts/
|
||||
├─ esp32dev-ble-firmware.bin
|
||||
├─ esp32dev-ble-bootloader.bin
|
||||
└─ esp32dev-ble-partitions.bin
|
||||
```
|
||||
**Behavior:**
|
||||
- Tool check verifies python3, platformio, git (can be skipped).
|
||||
- Builds via ci_build_firmware.sh (adds `--dev-ota` when `--mode dev`).
|
||||
- Packaging runs only when `--deploy-ready` is provided; artifacts land under `generated/artifacts/firmware_build/` with env-prefixed filenames plus zipped libraries.
|
||||
<!-- - Packaging runs only when `--deploy-ready` is provided; artifacts land under `generated/artifacts[/<version>]/firmware_build/` with env-prefixed filenames plus zipped libraries. -->
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Development build
|
||||
# Dev build with OTA flags
|
||||
./scripts/ci.sh build esp32dev-ble --mode dev
|
||||
|
||||
# Production build with version
|
||||
./scripts/ci.sh build esp32dev-ble --version v1.8.0 --mode prod
|
||||
# Prod build with auto-version and deployable artifacts
|
||||
./scripts/ci.sh build esp32dev-bt --version --deploy-ready --output generated/artifacts
|
||||
|
||||
# Deploy-ready build
|
||||
./scripts/ci.sh build esp32dev-all-test --version v1.8.0 --deploy-ready
|
||||
|
||||
# Clean build with verbose output
|
||||
# Clean + verbose build
|
||||
./scripts/ci.sh build nodemcuv2-rf --clean --verbose
|
||||
|
||||
# List available environments
|
||||
./scripts/ci.sh build --list-envs
|
||||
```
|
||||
|
||||
**Environment Variables:**
|
||||
- `CI` - Set to 'true' in CI/CD environments
|
||||
- `BUILD_NUMBER` - Build number from CI/CD system
|
||||
- `GIT_COMMIT` - Git commit hash for auto-versioning
|
||||
- `PLATFORMIO_BUILD_FLAGS` - Additional PlatformIO flags (set by script when --mode dev)
|
||||
|
||||
**Output Files:**
|
||||
- Standard mode: `firmware.bin`, `partitions.bin` in generated/artifacts/
|
||||
- Deploy mode: `<env>-firmware.bin`, `<env>-bootloader.bin`, `<env>-partitions.bin`
|
||||
|
||||
---
|
||||
|
||||
### ci.sh site - Build Documentation
|
||||
|
||||
Builds VuePress documentation website with version management and WebUploader manifest generation.
|
||||
Builds the VuePress documentation site.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
@@ -200,75 +177,36 @@ Builds VuePress documentation website with version management and WebUploader ma
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `--mode <dev|prod>` - Documentation mode (default: prod)
|
||||
- `dev` - Development documentation with watermark
|
||||
- `prod` - Production documentation
|
||||
- `--version-source <release|custom>` - Version source (default: release)
|
||||
- `release` - Use git tag as version
|
||||
- `custom` - Use custom version string
|
||||
- `--custom-version <version>` - Custom version string (requires --version-source custom)
|
||||
- `--url-prefix <path>` - Base URL path (default: /)
|
||||
- Example: `/dev/` for development subdirectory
|
||||
- `--no-webuploader` - Skip WebUploader manifest generation
|
||||
- `--webuploader-args <args>` - Additional arguments for gen_wu.py
|
||||
- `--preview` - Open browser after build (local development)
|
||||
- `--help` - Show help message
|
||||
- `--mode <prod|dev>` Build mode (default: dev).
|
||||
- `--url-prefix <path>` Base URL path for links (e.g., '/' for root, '/dev/' for dev) (default: /dev/).
|
||||
- `--version <tag>` Version string written to meta.json (default: edge).
|
||||
- `--preview` Start the local HTTPS preview server after building (https://localhost:8443).
|
||||
- `--clean` Remove generated/site folder before building.
|
||||
- `--insecure-curl` Allow curl to skip TLS verification when downloading common config.
|
||||
- `--help` Show help.
|
||||
|
||||
**Execution Flow:**
|
||||
```
|
||||
ci.sh site --mode prod --version-source release
|
||||
│
|
||||
├─> ci_site.sh (orchestrator)
|
||||
│ ├─> check_requirements() - Verify node, npm, python3, pip3
|
||||
│ ├─> install_dependencies() - npm install, pip3 install packages
|
||||
│ ├─> download_common_config() - Fetch from theengs.io
|
||||
│ ├─> get_version() - Extract from git tag or use custom
|
||||
│ ├─> set_version() - Update VuePress config and JSON files
|
||||
│ ├─> set_url_prefix() - Set base path in config
|
||||
│ ├─> generate_board_docs.py - Auto-generate board documentation
|
||||
│ ├─> npm run docs:build - Build VuePress site
|
||||
│ └─> gen_wu.py - Generate WebUploader manifest
|
||||
│
|
||||
└─> Outputs in generated/site/
|
||||
├─ index.html
|
||||
├─ assets/
|
||||
└─ [board documentation pages]
|
||||
```
|
||||
**Behavior:**
|
||||
- Checks for node, npm, openssl; installs npm deps; downloads commonConfig.js.
|
||||
- Writes docs/.vuepress/meta.json with mode/url_prefix/version; builds via `npm run docs:build`.
|
||||
- Preview mode runs `npm run site:preview`.
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Production documentation
|
||||
./scripts/ci.sh site --mode prod
|
||||
# Production build
|
||||
./scripts/ci.sh site --mode prod --url-prefix /
|
||||
|
||||
# Development documentation with custom version
|
||||
./scripts/ci.sh site --mode dev --version-source custom --custom-version "DEVELOPMENT SHA:abc123"
|
||||
# Development build with preview
|
||||
./scripts/ci.sh site --mode dev --url-prefix /dev/ --version edge --preview
|
||||
|
||||
# Documentation for /dev/ subdirectory
|
||||
./scripts/ci.sh site --mode dev --url-prefix /dev/
|
||||
|
||||
# Skip WebUploader manifest
|
||||
./scripts/ci.sh site --no-webuploader
|
||||
|
||||
# Local preview
|
||||
./scripts/ci.sh site --preview
|
||||
# Clean then build with custom version
|
||||
./scripts/ci.sh site --clean --version 1.8.0
|
||||
```
|
||||
|
||||
**Required Tools:**
|
||||
- Node.js (for VuePress)
|
||||
- npm (for package management)
|
||||
- Python 3 (for board docs generator)
|
||||
- pip3 (for Python dependencies: requests, pandas, markdown, pytablereader, tabulate)
|
||||
|
||||
**Output Files:**
|
||||
- `generated/site/` - Complete static website
|
||||
- `scripts/latest_version.json` - Production version metadata (updated)
|
||||
- `scripts/latest_version_dev.json` - Development version metadata (updated)
|
||||
|
||||
---
|
||||
|
||||
### ci.sh qa - Code Formatting Check
|
||||
### ci.sh qa - Code Quality Checks
|
||||
|
||||
Checks and fixes code formatting using clang-format.
|
||||
Checks and fixes code formatting using clang-format and runs shellcheck on shell scripts.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
@@ -276,13 +214,16 @@ Checks and fixes code formatting using clang-format.
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `--check` - Check formatting without modifying files (default)
|
||||
- `--fix` - Automatically fix formatting issues
|
||||
- `--source <dir>` - Source directory to check (default: main)
|
||||
- `--extensions <list>` - File extensions to check, comma-separated (default: h,ino,cpp)
|
||||
- `--clang-format-version <ver>` - clang-format version to use (default: 9)
|
||||
- `--verbose` - Show detailed output for each file
|
||||
- `--help` - Show help message
|
||||
- `--check` Check formatting only (default)
|
||||
- `--fix` Apply formatting in place
|
||||
- `--format` Run only format checks
|
||||
- `--shellcheck` Run shellcheck on shell scripts in scripts/ directory
|
||||
- `--all` Future hook to run all QA checks (current implementation runs formatting + shellcheck)
|
||||
- `--source <dir>` Source directory for formatting checks (default: main)
|
||||
- `--extensions <list>` File extensions for formatting (comma-separated, default: h,ino,cpp)
|
||||
- `--clang-format-version <ver>` clang-format version to use (default: 9)
|
||||
- `--verbose` Verbose output
|
||||
- `--help` Show help
|
||||
|
||||
**Execution Flow:**
|
||||
```
|
||||
@@ -290,27 +231,35 @@ ci.sh qa --check --source main --extensions h,ino
|
||||
│
|
||||
├─> ci_qa.sh (orchestrator)
|
||||
│ ├─> check_clang_format() - Find clang-format-9 or clang-format
|
||||
│ ├─> find_files() - Locate files matching extensions in source dir
|
||||
│ └─> check_formatting() - Run clang-format --dry-run --Werror
|
||||
│ └─> Report files with formatting issues
|
||||
│ │ ├─> find_files() - Locate files matching extensions in source dir
|
||||
│ │ └─> check_formatting() - Run clang-format --dry-run --Werror
|
||||
│ │ └─> Report files with formatting issues
|
||||
│ │
|
||||
│ └─> shellcheck_check() - Find and scan shell scripts
|
||||
│ ├─> find_shell_scripts() - Locate *.sh files in scripts/ directory
|
||||
│ └─> run_shellcheck() - Run shellcheck on found scripts
|
||||
│ └─> Report shell script issues
|
||||
│
|
||||
└─> Exit code: 0 (pass) or 1 (formatting issues found)
|
||||
└─> Exit code: 0 (pass) or 1 (issues found)
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Check formatting (CI mode)
|
||||
# Check both formatting and shellcheck (default)
|
||||
./scripts/ci.sh qa --check
|
||||
|
||||
# Fix formatting automatically
|
||||
./scripts/ci.sh qa --fix
|
||||
|
||||
# Check specific directory
|
||||
./scripts/ci.sh qa --check --source lib
|
||||
# Check only formatting for specific directory
|
||||
./scripts/ci.sh qa --check --format --source lib
|
||||
|
||||
# Check only .h and .ino files
|
||||
./scripts/ci.sh qa --check --extensions h,ino
|
||||
|
||||
# Check only shellcheck for shell scripts
|
||||
./scripts/ci.sh qa --check --shellcheck
|
||||
|
||||
# Check with verbose output
|
||||
./scripts/ci.sh qa --check --verbose
|
||||
|
||||
@@ -321,46 +270,201 @@ ci.sh qa --check --source main --extensions h,ino
|
||||
**Required Tools:**
|
||||
- clang-format (version specified, default: 9)
|
||||
- Install: `sudo apt-get install clang-format-9`
|
||||
- shellcheck (for shell script linting)
|
||||
- Install: `sudo apt-get install shellcheck`
|
||||
|
||||
**Output:**
|
||||
- Check mode: Lists files with formatting issues and shows diffs
|
||||
- Fix mode: Modifies files in-place and reports changes
|
||||
- Exit code 0: All files properly formatted
|
||||
- Exit code 1: Formatting issues found (in check mode)
|
||||
- Check mode: Lists files with formatting issues and shell script errors, shows diffs
|
||||
- Fix mode: Modifies formatting in-place and reports changes
|
||||
- Exit code 0: All checks passed (proper formatting and no shellcheck errors)
|
||||
- Exit code 1: Issues found (formatting or shellcheck violations)
|
||||
|
||||
---
|
||||
|
||||
### ci.sh security - Security Vulnerability Scan
|
||||
|
||||
Scans the project for security vulnerabilities using Trivy and generates Software Bill of Materials (SBOM).
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci.sh security [OPTIONS]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `--scan-type <fs|config|image>` Type of scan (default: fs)
|
||||
- `fs` - Filesystem scan (default, scans for vulnerabilities and misconfigurations)
|
||||
- `config` - Configuration scan only
|
||||
- `image` - Container image scan
|
||||
- `--severity <levels>` Severity levels to report (comma-separated: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL) (default: HIGH,CRITICAL)
|
||||
- `--scan-path <path>` Path to scan (default: current directory `.`)
|
||||
- `--generate-sbom` Generate Software Bill of Materials in CycloneDX and SPDX formats (default: true)
|
||||
- `--exit-code <0|1>` Exit code when vulnerabilities found (0=continue, 1=fail) (default: 0)
|
||||
- `--upload-to-security-tab` Upload SARIF report to GitHub Security tab (GitHub Actions only, default: true)
|
||||
- `--verbose` Verbose output
|
||||
- `--help` Show help
|
||||
|
||||
**Behavior:**
|
||||
- Installs Trivy if not present
|
||||
- Scans filesystem or configuration for known vulnerabilities
|
||||
- Generates multiple report formats: SARIF, JSON, table summary
|
||||
- Creates SBOM in CycloneDX and SPDX formats when `--generate-sbom` is enabled
|
||||
- Reports are saved to `generated/reports/` directory
|
||||
- Summary is appended to GitHub job summary when running in GitHub Actions
|
||||
- Uploads SARIF to GitHub Security tab for dashboard visibility
|
||||
|
||||
**Output Structure:**
|
||||
```
|
||||
generated/reports/
|
||||
├── trivy-results.sarif # SARIF format (for GitHub Security tab)
|
||||
├── trivy-results.json # JSON format (detailed results)
|
||||
├── security-summary.md # Markdown summary
|
||||
└── sbom/
|
||||
├── sbom.cyclonedx.json # CycloneDX format
|
||||
└── sbom.spdx.json # SPDX format
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Scan filesystem for HIGH and CRITICAL vulnerabilities
|
||||
./scripts/ci.sh security --scan-type fs --severity HIGH,CRITICAL
|
||||
|
||||
# Full scan with all severity levels and SBOM generation
|
||||
./scripts/ci.sh security --scan-type fs --severity UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL --generate-sbom
|
||||
|
||||
# Scan specific path with verbose output
|
||||
./scripts/ci.sh security --scan-path ./lib --verbose
|
||||
|
||||
# Configuration scan only
|
||||
./scripts/ci.sh security --scan-type config
|
||||
|
||||
# Scan and fail on vulnerabilities
|
||||
./scripts/ci.sh security --exit-code 1
|
||||
```
|
||||
|
||||
**Required Tools:**
|
||||
- Trivy (vulnerability scanner)
|
||||
- Install: `wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -` && `echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list` && `sudo apt-get update && sudo apt-get install -y trivy`
|
||||
|
||||
**Output:**
|
||||
- Detailed SARIF and JSON reports for integration with tools
|
||||
- Human-readable Markdown summary with findings count
|
||||
- SBOM artifacts for supply chain tracking
|
||||
- GitHub Security tab integration when in GitHub Actions
|
||||
- Exit code 0: Scan completed (vulnerabilities may have been found)
|
||||
- Exit code 1: Scan failed or critical vulnerabilities found (only if `--exit-code 1`)
|
||||
|
||||
---
|
||||
|
||||
### ci.sh all - Complete Pipeline
|
||||
|
||||
Runs the complete CI/CD pipeline (qa → build all environments → site) with mode injected to all steps.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci.sh all --mode <dev|prod> [--preview]
|
||||
```
|
||||
|
||||
**Required Options:**
|
||||
- `--mode <dev|prod>` Build mode (required). Injected to all pipeline steps:
|
||||
- `qa`: Always runs `--check`
|
||||
- `build`: Passes mode to all PlatformIO environments (dev enables OTA flags)
|
||||
- `site`: Passes mode for documentation generation
|
||||
|
||||
**Optional Options:**
|
||||
- `--preview` Start local HTTPS preview server at https://localhost:8443 after building the site
|
||||
- `--help` Show help
|
||||
|
||||
**Behavior:**
|
||||
- No environment argument needed; builds **all** available environments
|
||||
- All three steps (qa, build, site) receive the same `--mode` value
|
||||
- If any step fails, the pipeline aborts
|
||||
- Site build warnings do not abort the pipeline (continues with success status)
|
||||
|
||||
**Execution Flow:**
|
||||
```
|
||||
ci.sh all --mode dev --preview
|
||||
│
|
||||
├─> Step 1: ci_qa.sh --check
|
||||
│ └─> Exit on failure
|
||||
│
|
||||
├─> Step 2: ci_build.sh --mode dev (builds all environments)
|
||||
│ └─> Exit on failure
|
||||
│
|
||||
└─> Step 3: ci_site.sh --mode dev --preview
|
||||
├─> Build documentation
|
||||
└─> Start preview server at https://localhost:8443
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Complete pipeline in dev mode
|
||||
./scripts/ci.sh all --mode dev
|
||||
|
||||
# Complete pipeline in prod mode with site preview
|
||||
./scripts/ci.sh all --mode prod --preview
|
||||
|
||||
# Help for this command
|
||||
./scripts/ci.sh all --help
|
||||
```
|
||||
|
||||
**Output:**
|
||||
- Step-by-step progress messages for each pipeline phase
|
||||
- Final summary showing mode, preview status, duration, and overall status
|
||||
- Exit code 0: All steps successful
|
||||
- Exit code 1: Any step failed
|
||||
|
||||
---
|
||||
|
||||
### ci.sh list-env - List Environments
|
||||
|
||||
Lists PlatformIO environments available to build.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci.sh list-env [--full]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- Default: read .github/workflows/environments.json (requires jq) and show curated list.
|
||||
- `--full` Parse platformio.ini and environments.ini for an exhaustive list (skips *-test and *-all-).
|
||||
- `--help` Show help.
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
./scripts/ci.sh list-env
|
||||
./scripts/ci.sh list-env --full
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Internal Scripts
|
||||
|
||||
These scripts are called by the main orchestrators. It can be invoked directly but is not raccomanded.
|
||||
These scripts are called by the main orchestrators. They can be run directly for troubleshooting, but the preferred entrypoints are the ci.sh commands.
|
||||
|
||||
### ci_set_version.sh
|
||||
### ci_list-env.sh
|
||||
|
||||
Injects version string into firmware configuration files.
|
||||
Lists PlatformIO environments for OpenMQTTGateway.
|
||||
|
||||
**Called By:** `ci_build.sh`
|
||||
**Called By:** `ci.sh list-env` or direct call
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci_set_version.sh <version> [--dev]
|
||||
./scripts/ci_list-env.sh [--full]
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
- `<version>` - Version string to inject (e.g., v1.8.0 or abc123)
|
||||
- `--dev` - Development mode (updates latest_version_dev.json)
|
||||
**Options:**
|
||||
- Default: read .github/workflows/environments.json for a curated list (needs jq)
|
||||
- `--full` - Parse platformio.ini and environments.ini for all envs except *-test and *-all-
|
||||
- `-h|--help` - Show help
|
||||
|
||||
**Files Modified:**
|
||||
- `main/User_config.h` - Replaces "version_tag" with actual version
|
||||
- `scripts/latest_version.json` - Production version metadata
|
||||
- `scripts/latest_version_dev.json` - Development version metadata (--dev mode)
|
||||
**Output:**
|
||||
- Shows sorted environments in columns and prints the total count
|
||||
|
||||
**Behavior:**
|
||||
- Creates .bak backup files before modification
|
||||
- Replaces all occurrences of "version_tag" string
|
||||
- Validates version string (must not be empty or "version_tag")
|
||||
- Cleans up backup files on success
|
||||
|
||||
---
|
||||
**Examples:**
|
||||
```bash
|
||||
./scripts/ci_list-env.sh
|
||||
./scripts/ci_list-env.sh --full
|
||||
```
|
||||
|
||||
### ci_build_firmware.sh
|
||||
|
||||
@@ -377,14 +481,17 @@ Executes PlatformIO build for specified environment.
|
||||
- `<environment>` - PlatformIO environment name
|
||||
|
||||
**Options:**
|
||||
- `--version <tag>` - Set OMG_VERSION for the build (passed through from ci_build.sh)
|
||||
- `--dev-ota` - Enable development OTA (sets PLATFORMIO_BUILD_FLAGS)
|
||||
- `--clean` - Clean before build
|
||||
- `--verbose` - Verbose PlatformIO output
|
||||
- `--no-verify` - Skip artifact verification after build
|
||||
|
||||
**Environment Variables Set:**
|
||||
- `PYTHONIOENCODING=utf-8`
|
||||
- `PYTHONUTF8=1`
|
||||
- `PLATFORMIO_BUILD_FLAGS="-DDEVELOPMENTOTA=true"` (when --dev-ota)
|
||||
- `OMG_VERSION` (when --version is provided)
|
||||
|
||||
**PlatformIO Command:**
|
||||
```bash
|
||||
@@ -400,7 +507,7 @@ platformio run -e <environment> [--verbose]
|
||||
|
||||
### ci_prepare_artifacts.sh
|
||||
|
||||
Packages firmware binaries from PlatformIO build directory.
|
||||
Packages firmware binaries and libraries from a PlatformIO build directory; can also only create source archive when no environment is provided.
|
||||
|
||||
**Called By:** `ci_build.sh`
|
||||
|
||||
@@ -410,29 +517,19 @@ Packages firmware binaries from PlatformIO build directory.
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
- `<environment>` - PlatformIO environment name
|
||||
- `<environment>` - PlatformIO environment name (optional; if omitted only source archive is created)
|
||||
|
||||
**Options:**
|
||||
- `--deploy` - Use deployment naming (prefix with environment name)
|
||||
- `--output <dir>` - Output directory (default: generated/artifacts)
|
||||
- `--output <dir>` Output directory (default: generated/artifacts)
|
||||
- `--version <tag>` Append a version subfolder inside the output directory
|
||||
- `--clean` Clean output directory before writing
|
||||
- `--help` Show help
|
||||
|
||||
**Behavior:**
|
||||
|
||||
Standard mode (no --deploy):
|
||||
- Copies: `firmware.bin`, `partitions.bin`
|
||||
- Does NOT copy: `bootloader.bin`
|
||||
|
||||
Deploy mode (with --deploy):
|
||||
- Copies and renames:
|
||||
- `firmware.bin` → `<env>-firmware.bin`
|
||||
- `bootloader.bin` → `<env>-bootloader.bin`
|
||||
- `partitions.bin` → `<env>-partitions.bin`
|
||||
|
||||
**Source Location:**
|
||||
- `.pio/build/<environment>/`
|
||||
|
||||
**Output Location:**
|
||||
- Specified by `--output` or default `generated/artifacts/`
|
||||
- If `version` is provided, outputs go to `<output>/<version>/firmware_build/`; otherwise to `<output>/firmware_build/`.
|
||||
- With an environment: copies and renames firmware.bin/partitions.bin/bootloader.bin/boot_app0.bin to `<env>-*.bin`, then zips libraries for that env into `*-libraries.tgz`.
|
||||
- Without an environment: only creates `OpenMQTTGateway_sources.tgz` from `main` and `LICENSE.txt`.
|
||||
- Lists the prepared artifacts and their sizes at the end.
|
||||
|
||||
---
|
||||
|
||||
@@ -452,6 +549,7 @@ Shared configuration and helper functions for all CI scripts.
|
||||
- `BUILD_DIR=".pio/build"` - PlatformIO build directory
|
||||
- `ARTIFACTS_DIR="generated/artifacts"` - Artifact output directory
|
||||
- `SITE_DIR="generated/site"` - Documentation output directory
|
||||
- `REPORTS_DIR="generated/reports"` - Security scan and quality reports directory
|
||||
|
||||
**Logging Functions:**
|
||||
```bash
|
||||
@@ -463,9 +561,74 @@ log_success "message" # Green [SUCCESS] prefix
|
||||
|
||||
---
|
||||
|
||||
### ci_security.sh
|
||||
|
||||
Performs security vulnerability scanning and Software Bill of Materials (SBOM) generation using Trivy.
|
||||
|
||||
**Called By:** `ci.sh security`
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./scripts/ci_security.sh [OPTIONS]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `--scan-type <fs|config|image>` Type of scan (default: fs)
|
||||
- `fs` - Filesystem scan (default)
|
||||
- `config` - Configuration scan
|
||||
- `image` - Container image scan
|
||||
- `--severity <levels>` Severity levels (comma-separated: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL) (default: HIGH,CRITICAL)
|
||||
- `--scan-path <path>` Path to scan (default: .)
|
||||
- `--generate-sbom` Generate SBOM (default: true)
|
||||
- `--exit-code <0|1>` Exit code behavior (0=continue, 1=fail) (default: 0)
|
||||
- `--upload-to-security-tab` Upload SARIF to GitHub (default: true)
|
||||
- `--verbose` Verbose output
|
||||
- `--help` Show help
|
||||
|
||||
**Behavior:**
|
||||
- Ensures Trivy is installed via package manager
|
||||
- Creates `generated/reports/` directory structure
|
||||
- Runs Trivy with specified parameters
|
||||
- Generates SARIF, JSON, and table formats
|
||||
- Creates SBOM in CycloneDX and SPDX formats (when enabled)
|
||||
- Uploads SARIF to GitHub Security tab when `GITHUB_TOKEN` and `--upload-to-security-tab` are set
|
||||
- Appends summary to GitHub job summary if in GitHub Actions
|
||||
- Validates critical vulnerabilities and exits with code 1 if found and `--exit-code 1` is set
|
||||
|
||||
**Output Files:**
|
||||
- `generated/reports/trivy-results.sarif` - SARIF format for GitHub integration
|
||||
- `generated/reports/trivy-results.json` - Full JSON results
|
||||
- `generated/reports/security-summary.md` - Human-readable summary
|
||||
- `generated/reports/sbom/sbom.cyclonedx.json` - CycloneDX SBOM
|
||||
- `generated/reports/sbom/sbom.spdx.json` - SPDX SBOM
|
||||
|
||||
**Exit Codes:**
|
||||
- `0` - Success (vulnerabilities may have been found)
|
||||
- `1` - Scan failed, critical vulnerabilities found (only if `--exit-code 1`), or missing dependencies
|
||||
|
||||
**Trivy Integration:**
|
||||
- Scans for known vulnerabilities in dependencies
|
||||
- Detects misconfigurations and insecure practices
|
||||
- Generates compliant SBOM artifacts
|
||||
- Provides detailed reporting in multiple formats
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
# Scan filesystem with severity filter
|
||||
./scripts/ci_security.sh --scan-type fs --severity HIGH,CRITICAL
|
||||
|
||||
# Local scan with SBOM and JSON output
|
||||
./scripts/ci_security.sh --scan-type fs --generate-sbom --verbose
|
||||
|
||||
# In GitHub Actions with security tab upload
|
||||
./scripts/ci_security.sh --scan-type fs --severity HIGH,CRITICAL --upload-to-security-tab
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Python Helper Scripts
|
||||
|
||||
Other scripts are present and used as internal scripts and it's used as retrocompatibility. Below the lists:
|
||||
Legacy helper scripts are kept for compatibility; they are not called by the current ci_site.sh flow. Below the list:
|
||||
- `generate_board_docs.py`
|
||||
- `gen_wu.py`
|
||||
|
||||
@@ -473,7 +636,7 @@ Other scripts are present and used as internal scripts and it's used as retrocom
|
||||
|
||||
Auto-generates board-specific documentation pages from platformio.ini.
|
||||
|
||||
**Called By:** `ci_site.sh`
|
||||
**Called By:** Not invoked by current ci_site.sh (legacy helper)
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
@@ -498,7 +661,7 @@ python3 ./scripts/generate_board_docs.py
|
||||
|
||||
Generates WebUploader manifest for OTA firmware updates.
|
||||
|
||||
**Called By:** `ci_site.sh`
|
||||
**Called By:** Not invoked by current ci_site.sh (legacy helper)
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
@@ -527,10 +690,10 @@ python3 ./scripts/gen_wu.py [--dev] [repository]
|
||||
|
||||
Scripts respect these environment variables:
|
||||
|
||||
- `PYTHONIOENCODING=utf-8`: Python encoding
|
||||
- `PYTHONUTF8=1`: UTF-8 mode
|
||||
- `PLATFORMIO_BUILD_FLAGS`: Custom build flags
|
||||
- `ESP32_PLATFORM_VERSION`: Extracted automatically
|
||||
- `CI`/`BUILD_NUMBER`/`GIT_COMMIT`: Used by ci_build.sh to auto-generate version when `--version` flag has no tag
|
||||
- `PYTHONIOENCODING=utf-8`, `PYTHONUTF8=1`: Python encoding settings set by ci_build_firmware.sh
|
||||
- `PLATFORMIO_BUILD_FLAGS`: Set to include development OTA flag when `--dev-ota` is used
|
||||
- `OMG_VERSION`: Set when `--version` is passed to ci_build.sh/ci_build_firmware.sh
|
||||
|
||||
---
|
||||
|
||||
@@ -567,6 +730,58 @@ CI/CD environments typically set:
|
||||
- `BUILD_NUMBER` (build number)
|
||||
- `GIT_COMMIT` (commit hash)
|
||||
|
||||
---
|
||||
|
||||
## GitHub Actions Workflows Integration
|
||||
|
||||
The CI scripts integrate with GitHub Actions workflows in `.github/workflows/`:
|
||||
|
||||
### task-lint.yml
|
||||
- Reusable workflow that runs `ci.sh qa --check`
|
||||
- Installs clang-format and shellcheck
|
||||
- Validates code formatting and shell script quality
|
||||
- Can be called with custom source directory and file extensions
|
||||
|
||||
### task-build.yml
|
||||
- Main build workflow orchestrator
|
||||
- Calls task-lint.yml for code quality checks
|
||||
- Calls task-security-scan.yml for vulnerability scanning
|
||||
- Builds firmware for all or specified environments
|
||||
- Prepares and uploads build artifacts
|
||||
- Supports matrix builds for multiple environments
|
||||
- Manages build artifact retention
|
||||
|
||||
### task-security-scan.yml
|
||||
- Reusable security scanning workflow
|
||||
- Installs Trivy vulnerability scanner
|
||||
- Calls `ci_security.sh` with configurable parameters
|
||||
- Generates SARIF, JSON, and SBOM reports
|
||||
- Uploads SARIF to GitHub Security tab for code scanning dashboard
|
||||
- Fails build on critical vulnerabilities when configured
|
||||
- Uploads SBOM artifacts for supply chain tracking
|
||||
|
||||
### security-scan.yml
|
||||
- Scheduled security scanning (runs weekly by default)
|
||||
- Triggered manually with input parameters
|
||||
- Allows filtering by severity level
|
||||
- Configurable exit behavior (fail or continue)
|
||||
- Optional SBOM generation and upload
|
||||
|
||||
**Workflow Dependencies:**
|
||||
```
|
||||
task-build.yml
|
||||
├─> task-lint.yml (linting)
|
||||
├─> task-security-scan.yml (security scanning)
|
||||
└─> Build environment matrix (firmware compilation)
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- Parallel linting and security scans
|
||||
- Artifact retention policies
|
||||
- GitHub Security tab integration
|
||||
- Detailed build reports
|
||||
- SBOM generation for compliance
|
||||
- Support for custom build parameters
|
||||
|
||||
---
|
||||
|
||||
|
||||
+197
-43
@@ -6,8 +6,9 @@
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly SCRIPT_DIR
|
||||
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
@@ -36,7 +37,9 @@ Commands:
|
||||
build Build firmware for specified environment
|
||||
site Build and deploy documentation/website
|
||||
qa Run quality assurance checks (linting, formatting)
|
||||
security Run security vulnerability scan using Trivy
|
||||
all Run complete pipeline (qa + build + site)
|
||||
list-env List available environments for building firmware
|
||||
|
||||
Examples:
|
||||
# Build firmware
|
||||
@@ -51,78 +54,217 @@ Examples:
|
||||
$0 qa --check
|
||||
$0 qa --fix
|
||||
|
||||
# Run complete pipeline
|
||||
$0 all esp32dev-bt --version v1.8.0
|
||||
# Run security scan
|
||||
$0 security
|
||||
$0 security --scan-type config --exit-code 1
|
||||
$0 security --severity UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
|
||||
|
||||
# Run complete pipeline (all envs, mode required)
|
||||
$0 all --mode dev
|
||||
$0 all --mode prod --preview
|
||||
$0 all --mode dev -e esp32dev-bt
|
||||
|
||||
# List available environments
|
||||
$0 list-env
|
||||
|
||||
Get help for specific commands:
|
||||
$0 build --help
|
||||
$0 site --help
|
||||
$0 qa --help
|
||||
$0 security --help
|
||||
$0 list-env --help
|
||||
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
# Function to run build pipeline
|
||||
run_build_pipeline() {
|
||||
log_info "Executing build pipeline..."
|
||||
"${SCRIPT_DIR}/ci_build.sh" "$@"
|
||||
|
||||
|
||||
# Function to get list of environments to build
|
||||
# Uses ci_list-env.sh (no parameters) and returns one env per line
|
||||
get_environments() {
|
||||
# Run the curated list script and normalize output
|
||||
"${SCRIPT_DIR}/ci_list-env.sh" \
|
||||
| sed -r 's/\x1B\[[0-9;]*[mK]//g' \
|
||||
| tr '\t ' '\n' \
|
||||
| sed '/^\s*$/d' \
|
||||
| grep -E '^[A-Za-z0-9._-]+$' \
|
||||
| sort -u
|
||||
}
|
||||
|
||||
# Function to run site pipeline
|
||||
run_site_pipeline() {
|
||||
log_info "Executing site pipeline..."
|
||||
"${SCRIPT_DIR}/ci_site.sh" "$@"
|
||||
}
|
||||
|
||||
# Function to run QA pipeline
|
||||
run_qa_pipeline() {
|
||||
log_info "Executing QA pipeline..."
|
||||
"${SCRIPT_DIR}/ci_qa.sh" "$@"
|
||||
}
|
||||
|
||||
# Function to run complete pipeline
|
||||
# Function to run complete pipeline using the underlying scripts
|
||||
# Usage: run_all_pipeline --mode <dev|prod> [--preview]
|
||||
run_all_pipeline() {
|
||||
local start_time
|
||||
start_time=$(date +%s)
|
||||
|
||||
log_info "Starting complete CI/CD pipeline..."
|
||||
|
||||
local mode=""
|
||||
local preview=false
|
||||
local env_override=""
|
||||
local version=""
|
||||
local do_clean=false
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--mode)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
log_error "--mode requires an argument (dev or prod)"
|
||||
return 1
|
||||
fi
|
||||
mode="$2"
|
||||
if [[ "$mode" != "dev" && "$mode" != "prod" ]]; then
|
||||
log_error "Invalid mode: $mode. Must be 'dev' or 'prod'"
|
||||
return 1
|
||||
fi
|
||||
shift 2
|
||||
;;
|
||||
--clean)
|
||||
do_clean=true
|
||||
shift
|
||||
;;
|
||||
--preview)
|
||||
preview=true
|
||||
shift
|
||||
;;
|
||||
-e|--env)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
log_error "-e|--env requires an environment name"
|
||||
return 1
|
||||
fi
|
||||
env_override="$2"
|
||||
shift 2
|
||||
;;
|
||||
-v|--version)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
log_error "-v|--version requires a version string"
|
||||
return 1
|
||||
fi
|
||||
version="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: $0 all --mode <dev|prod> [--preview]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --mode <dev|prod> Build mode (required)"
|
||||
echo " --preview Optional Show site in local at http://localhost:8443"
|
||||
echo " -e, --env <name> Optional Build only the specified environment"
|
||||
return 0
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown option: $1"
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Validate mode is provided
|
||||
if [[ -z "$mode" ]]; then
|
||||
log_error "--mode is required. Usage: $0 all --mode <dev|prod> [--preview]"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Starting complete CI/CD pipeline (mode: $mode, preview: $preview)..."
|
||||
echo ""
|
||||
|
||||
|
||||
# Step 1: Quality Assurance
|
||||
log_info "═══ Step 1/3: Quality Assurance ═══"
|
||||
run_qa_pipeline --check || {
|
||||
log_info "RUN: ---> ${SCRIPT_DIR}/ci_qa.sh --check"
|
||||
if ! "${SCRIPT_DIR}/ci_qa.sh" --check; then
|
||||
log_error "QA checks failed. Pipeline aborted."
|
||||
return 1
|
||||
}
|
||||
fi
|
||||
echo ""
|
||||
|
||||
|
||||
# Step 2: Build Firmware
|
||||
log_info "═══ Step 2/3: Build Firmware ═══"
|
||||
run_build_pipeline "$@" || {
|
||||
log_error "Build failed. Pipeline aborted."
|
||||
return 1
|
||||
}
|
||||
echo ""
|
||||
|
||||
# Step 3: Build Site (only if not in --no-site mode)
|
||||
if [[ ! " $* " =~ " --no-site " ]]; then
|
||||
log_info "═══ Step 3/3: Build Documentation ═══"
|
||||
run_site_pipeline --mode prod || {
|
||||
log_warn "Site build failed, but continuing..."
|
||||
}
|
||||
# Get list of environments
|
||||
local -a environments
|
||||
if [[ -n "$env_override" ]]; then
|
||||
log_info "Using single environment override: $env_override"
|
||||
environments=("$env_override")
|
||||
else
|
||||
log_info "Skipping site build (--no-site flag)"
|
||||
mapfile -t environments < <(get_environments)
|
||||
fi
|
||||
|
||||
if [[ ${#environments[@]} -eq 0 ]]; then
|
||||
log_error "No environments found to build"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Found ${#environments[@]} environments to build"
|
||||
echo ""
|
||||
|
||||
local build_count=0
|
||||
local failed_builds=()
|
||||
local build_args=()
|
||||
|
||||
if [[ -n "$version" ]]; then
|
||||
build_args+=("--version" "$version")
|
||||
log_info "Using version override: $version"
|
||||
fi
|
||||
build_args+=("--mode" "$mode")
|
||||
build_args+=("--deploy-ready")
|
||||
if [[ "$do_clean" == true ]]; then
|
||||
build_args+=("--clean")
|
||||
fi
|
||||
|
||||
for env in "${environments[@]}"; do
|
||||
((++build_count))
|
||||
log_info "[$build_count/${#environments[@]}] Building: $env"
|
||||
|
||||
set +e
|
||||
log_info "RUN: ---> ${SCRIPT_DIR}/ci_build.sh" "$env" "${build_args[@]}"
|
||||
"${SCRIPT_DIR}/ci_build.sh" "$env" "${build_args[@]}"
|
||||
local rc=$?
|
||||
set -e
|
||||
if [[ $rc -ne 0 ]]; then
|
||||
log_error "Build failed for environment: $env"
|
||||
failed_builds+=("$env")
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
if [[ ${#failed_builds[@]} -gt 0 ]]; then
|
||||
log_error "Build failed for ${#failed_builds[@]} environment(s):"
|
||||
printf ' - %s\n' "${failed_builds[@]}"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_success "All environments built successfully (${#environments[@]} total)"
|
||||
echo ""
|
||||
|
||||
# Step 3: Build Site
|
||||
log_info "═══ Step 3/3: Build Documentation ═══"
|
||||
local site_args=("--mode" "$mode")
|
||||
if [[ "$preview" == true ]]; then
|
||||
site_args+=("--preview")
|
||||
fi
|
||||
if [[ -n "$version" ]]; then
|
||||
site_args+=("--version" "$version")
|
||||
fi
|
||||
if [[ "$do_clean" == true ]]; then
|
||||
site_args+=("--clean")
|
||||
fi
|
||||
|
||||
log_info "RUN: --->${SCRIPT_DIR}/ci_site.sh" "${site_args[@]}"
|
||||
if ! "${SCRIPT_DIR}/ci_site.sh" "${site_args[@]}"; then
|
||||
log_warn "Site build failed, but continuing..."
|
||||
fi
|
||||
echo ""
|
||||
|
||||
local end_time
|
||||
end_time=$(date +%s)
|
||||
local duration=$((end_time - start_time))
|
||||
|
||||
|
||||
echo ""
|
||||
echo "╔════════════════════════════════════════╗"
|
||||
echo "║ Complete Pipeline Summary ║"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
echo " Mode: $mode"
|
||||
echo " Preview: $preview"
|
||||
echo " Total Duration: ${duration}s"
|
||||
echo " Status: SUCCESS ✓"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
@@ -151,14 +293,26 @@ main() {
|
||||
# Route to appropriate pipeline
|
||||
case "$command" in
|
||||
build)
|
||||
run_build_pipeline "$@"
|
||||
log_info "Executing build pipeline..."
|
||||
"${SCRIPT_DIR}/ci_build.sh" "$@"
|
||||
;;
|
||||
site|docs)
|
||||
run_site_pipeline "$@"
|
||||
log_info "Executing site pipeline..."
|
||||
"${SCRIPT_DIR}/ci_site.sh" "$@"
|
||||
;;
|
||||
qa|lint)
|
||||
run_qa_pipeline "$@"
|
||||
log_info "Executing QA pipeline..."
|
||||
"${SCRIPT_DIR}/ci_qa.sh" "$@"
|
||||
;;
|
||||
security)
|
||||
log_info "Executing security scan..."
|
||||
"${SCRIPT_DIR}/ci_security.sh" "$@"
|
||||
;;
|
||||
list-env)
|
||||
log_info "Executing list-env pipeline..."
|
||||
"${SCRIPT_DIR}/ci_list-env.sh" "$@"
|
||||
;;
|
||||
|
||||
all|pipeline)
|
||||
run_all_pipeline "$@"
|
||||
;;
|
||||
|
||||
+6
-33
@@ -1,39 +1,16 @@
|
||||
#!/bin/bash
|
||||
# shellcheck disable=SC2034
|
||||
# Build Scripts Configuration
|
||||
# Used by: All build scripts for centralized configuration
|
||||
|
||||
# Python Configuration
|
||||
PYTHON_VERSION="3.13"
|
||||
PLATFORMIO_VERSION="https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip"
|
||||
|
||||
# Centralized Output Directory Structure
|
||||
# All CI/CD generated files go under generated/
|
||||
GENERATED_BASE_DIR="generated"
|
||||
ARTIFACTS_DIR="${GENERATED_BASE_DIR}/artifacts"
|
||||
SITE_DIR="${GENERATED_BASE_DIR}/site"
|
||||
REPORTS_DIR="${GENERATED_BASE_DIR}/reports"
|
||||
ARTIFACTS_DIR="generated/artifacts"
|
||||
SITE_DIR="generated/site"
|
||||
REPORTS_DIR="generated/reports"
|
||||
|
||||
# PlatformIO Directory Configuration
|
||||
BUILD_DIR=".pio/build"
|
||||
SCRIPTS_DIR="scripts"
|
||||
|
||||
# Build Configuration
|
||||
DEFAULT_ENVIRONMENT="esp32dev-all-test"
|
||||
ENABLE_VERBOSE_BUILD="false"
|
||||
ENABLE_BUILD_CACHE="true"
|
||||
|
||||
# Artifact Configuration
|
||||
ARTIFACT_RETENTION_DAYS="7"
|
||||
CREATE_MANIFEST="true"
|
||||
COMPRESS_ARTIFACTS="false"
|
||||
|
||||
# Version Configuration
|
||||
VERSION_FILE_PROD="scripts/latest_version.json"
|
||||
VERSION_FILE_DEV="scripts/latest_version_dev.json"
|
||||
USER_CONFIG_FILE="main/User_config.h"
|
||||
|
||||
# Logging Configuration
|
||||
ENABLE_COLOR_OUTPUT="true"
|
||||
LOG_LEVEL="INFO" # DEBUG, INFO, WARN, ERROR
|
||||
|
||||
# ============================================================================
|
||||
# Colors - ANSI color codes for terminal output
|
||||
@@ -50,9 +27,5 @@ readonly NC='\033[0m' # No Color
|
||||
log_info() { echo -e "${GREEN}[INFO]${NC} $*" >&2; }
|
||||
log_warn() { echo -e "${YELLOW}[WARN]${NC} $*" >&2; }
|
||||
log_error() { echo -e "${RED}[ERROR]${NC} $*" >&2; }
|
||||
log_success() { echo -e "${GREEN}[SUCCESS]${NC} $*" >&2; }
|
||||
log_step() { echo -e "${BLUE}[STEP]${NC} $*" >&2; }
|
||||
|
||||
# Advanced Options
|
||||
ENABLE_CCACHE="false"
|
||||
CCACHE_DIR=".ccache"
|
||||
MAX_BUILD_JOBS="4"
|
||||
+50
-75
@@ -6,8 +6,10 @@
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
readonly SCRIPT_DIR
|
||||
readonly PROJECT_ROOT
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
@@ -71,7 +73,6 @@ verify_build_tools() {
|
||||
log_info "Verifying required build tools..."
|
||||
|
||||
local missing_tools=()
|
||||
local version_mismatch=()
|
||||
|
||||
# Check Python
|
||||
if ! command_exists python3; then
|
||||
@@ -130,39 +131,6 @@ cleanup_on_error() {
|
||||
find . -name "*.bak" -type f -exec bash -c 'mv "$1" "${1%.bak}"' _ {} \; 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Function to list available environments
|
||||
list_environments() {
|
||||
log_info "Available PlatformIO environments:"
|
||||
echo ""
|
||||
|
||||
local env_files=("${PROJECT_ROOT}/platformio.ini" "${PROJECT_ROOT}/environments.ini")
|
||||
local envs=()
|
||||
|
||||
for file in "${env_files[@]}"; do
|
||||
if [[ -f "$file" ]]; then
|
||||
while IFS= read -r line; do
|
||||
if [[ "$line" =~ ^\[env:([^\]]+)\] ]]; then
|
||||
local env_name="${BASH_REMATCH[1]}"
|
||||
# Skip test environments
|
||||
if [[ ! "$env_name" =~ -test$ && ! "$env_name" =~ -all- ]]; then
|
||||
envs+=("$env_name")
|
||||
fi
|
||||
fi
|
||||
done < "$file"
|
||||
fi
|
||||
done
|
||||
|
||||
# Sort and display unique environments
|
||||
if [[ ${#envs[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${envs[@]}" | sort -u | column -c 80
|
||||
else
|
||||
log_warn "No environments found in configuration files"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
log_info "Total: $(printf '%s\n' "${envs[@]}" | sort -u | wc -l) environments"
|
||||
}
|
||||
|
||||
# Show usage
|
||||
usage() {
|
||||
cat << EOF
|
||||
@@ -174,15 +142,14 @@ Arguments:
|
||||
environment PlatformIO environment name
|
||||
|
||||
Options:
|
||||
--version [TAG] Set version tag (if TAG omitted, auto-generated)
|
||||
--mode MODE Build mode: 'prod' or 'dev' [default: prod]
|
||||
'dev' enables OTA and development version
|
||||
'dev' enables OTA and development features
|
||||
--deploy-ready Prepare for deployment (renamed artifacts)
|
||||
--version [TAG] Set version tag (if TAG omitted, auto-generated)
|
||||
--output DIR Output directory for artifacts [default: generated/artifacts/]
|
||||
--skip-verification Skip build tools verification
|
||||
--clean Clean build before starting
|
||||
--verbose Enable verbose output
|
||||
--list-envs List all available PlatformIO environments
|
||||
--help Show this help message
|
||||
|
||||
Environment Variables:
|
||||
@@ -191,9 +158,6 @@ Environment Variables:
|
||||
GIT_COMMIT Git commit hash for versioning
|
||||
|
||||
Examples:
|
||||
# List available environments
|
||||
$0 --list-envs
|
||||
|
||||
# Local development build
|
||||
$0 esp32dev-all-test --mode dev
|
||||
|
||||
@@ -212,7 +176,7 @@ main() {
|
||||
local version=""
|
||||
local set_version=false
|
||||
local mode=""
|
||||
local deploy=false
|
||||
local prepare_for_deploy=false
|
||||
local output_dir=""
|
||||
local skip_verification=false
|
||||
local clean=false
|
||||
@@ -246,7 +210,7 @@ main() {
|
||||
shift 2
|
||||
;;
|
||||
--deploy-ready)
|
||||
deploy=true
|
||||
prepare_for_deploy=true
|
||||
shift
|
||||
;;
|
||||
--output)
|
||||
@@ -270,10 +234,6 @@ main() {
|
||||
verbose=true
|
||||
shift
|
||||
;;
|
||||
--list-envs)
|
||||
list_environments
|
||||
exit 0
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
@@ -290,12 +250,6 @@ main() {
|
||||
esac
|
||||
done
|
||||
|
||||
# Set default mode if not specified
|
||||
if [[ -z "$mode" ]]; then
|
||||
mode="prod"
|
||||
log_info "Mode not specified, defaulting to production"
|
||||
fi
|
||||
|
||||
# Validate environment
|
||||
if [[ -z "$environment" ]]; then
|
||||
log_error "Environment name is required"
|
||||
@@ -303,6 +257,13 @@ main() {
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Set default mode if not specified
|
||||
if [[ -z "$mode" ]]; then
|
||||
mode="prod"
|
||||
log_info "Mode not specified, defaulting to production"
|
||||
fi
|
||||
|
||||
|
||||
# Auto-generate version if --version flag is set but no tag provided
|
||||
if [[ "$set_version" == "true" && -z "$version" ]]; then
|
||||
if [[ "${CI:-false}" == "true" ]]; then
|
||||
@@ -338,39 +299,53 @@ main() {
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Step 2: Set version (only if --version flag was provided)
|
||||
if [[ "$set_version" == "true" ]]; then
|
||||
log_step "2/4 Setting version: $version"
|
||||
if [[ "$mode" == "dev" ]]; then
|
||||
"${SCRIPT_DIR}/ci_set_version.sh" "$version" --dev || exit 1
|
||||
else
|
||||
"${SCRIPT_DIR}/ci_set_version.sh" "$version" --prod || exit 1
|
||||
fi
|
||||
echo ""
|
||||
else
|
||||
log_info "Skipping version setting (use --version to set version)"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# Step 2: Set version
|
||||
# not required
|
||||
|
||||
# Step 3: Build firmware
|
||||
log_step "3/4 Building firmware for: $environment"
|
||||
local build_opts=()
|
||||
[[ "$mode" == "dev" ]] && build_opts+=(--dev-ota)
|
||||
[[ "$clean" == "true" ]] && build_opts+=(--clean)
|
||||
[[ "$verbose" == "true" ]] && build_opts+=(--verbose)
|
||||
[[ "$set_version" == "true" ]] && build_opts+=(--version "$version")
|
||||
|
||||
"${SCRIPT_DIR}/ci_build_firmware.sh" "$environment" "${build_opts[@]}" || exit 1
|
||||
echo ""
|
||||
|
||||
# Step 4: Prepare artifacts
|
||||
log_step "4/4 Preparing artifacts..."
|
||||
local artifact_opts=()
|
||||
[[ "$deploy" == "true" ]] && artifact_opts+=(--deploy) || artifact_opts+=(--standard)
|
||||
artifact_opts+=(--manifest)
|
||||
[[ -n "$output_dir" ]] && artifact_opts+=(--output "$output_dir")
|
||||
|
||||
"${SCRIPT_DIR}/ci_prepare_artifacts.sh" "$environment" "${artifact_opts[@]}" || exit 1
|
||||
echo ""
|
||||
|
||||
if [[ "$prepare_for_deploy" == "true" ]]; then
|
||||
log_info "Preparing artifacts for deployment"
|
||||
local artifact_opts=()
|
||||
[[ "$clean" == "true" ]] && artifact_opts+=(--clean)
|
||||
[[ -n "$output_dir" ]] && artifact_opts+=(--output "$output_dir")
|
||||
[[ "$set_version" == "true" ]] && artifact_opts+=(--version "$version")
|
||||
"${SCRIPT_DIR}/ci_prepare_artifacts.sh" "$environment" "${artifact_opts[@]}" || exit 1
|
||||
echo ""
|
||||
# Check if site folder exists and copy built files to avoid rebuilding the site
|
||||
if [[ "$mode" == "dev" ]]; then
|
||||
local site_dir="${PROJECT_ROOT}/generated/site/dev"
|
||||
local artifacts_dir="${output_dir:-${PROJECT_ROOT}/generated/artifacts/firmware_build}"
|
||||
|
||||
if [[ -d "$site_dir" ]]; then
|
||||
log_info "Site folder exists, copying built firmware files to site/dev..."
|
||||
|
||||
# Copy firmware files for the current environment
|
||||
for file in "${artifacts_dir}/${environment}"-*.bin "${artifacts_dir}/${environment}"-*.tgz; do
|
||||
if [[ -f "$file" ]]; then
|
||||
cp -v "$file" "$site_dir/" || log_warn "Failed to copy $(basename "$file")"
|
||||
fi
|
||||
done
|
||||
|
||||
log_info "✓ Firmware files copied to site/dev (no site rebuild needed)"
|
||||
else
|
||||
log_warn "Site folder not found at: $site_dir"
|
||||
log_info "Run 'ci.sh site ' to generate it"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Print summary
|
||||
print_summary "$environment" "$version" "$start_time"
|
||||
|
||||
@@ -6,8 +6,10 @@
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
readonly SCRIPT_DIR
|
||||
readonly PROJECT_ROOT
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
@@ -41,18 +43,7 @@ validate_environment() {
|
||||
log_info "Building environment: $env"
|
||||
}
|
||||
|
||||
# Function to setup build environment variables
|
||||
setup_build_env() {
|
||||
local enable_dev_ota="${1:-false}"
|
||||
|
||||
export PYTHONIOENCODING=utf-8
|
||||
export PYTHONUTF8=1
|
||||
|
||||
if [[ "$enable_dev_ota" == "true" ]]; then
|
||||
export PLATFORMIO_BUILD_FLAGS='"-DDEVELOPMENTOTA=true"'
|
||||
log_info "Development OTA enabled"
|
||||
fi
|
||||
}
|
||||
|
||||
|
||||
# Function to check PlatformIO availability
|
||||
check_platformio() {
|
||||
@@ -78,6 +69,18 @@ run_build() {
|
||||
local env="$1"
|
||||
local clean="${2:-false}"
|
||||
local verbose="${3:-false}"
|
||||
local enable_dev_ota="${4:-false}"
|
||||
local version="${5:-edge}"
|
||||
|
||||
if [[ "$enable_dev_ota" == "true" ]]; then
|
||||
log_info "Development OTA enabled"
|
||||
export PLATFORMIO_BUILD_FLAGS='"-DDEVELOPMENTOTA=true"'
|
||||
fi
|
||||
|
||||
if [[ -n "$version" ]]; then
|
||||
log_info "Setting firmware version to: $version"
|
||||
export PLATFORMIO_BUILD_FLAGS="${PLATFORMIO_BUILD_FLAGS} -DOMG_VERSION=\\\"${version}\\\""
|
||||
fi
|
||||
|
||||
log_build "Starting build for environment: $env"
|
||||
|
||||
@@ -95,6 +98,9 @@ run_build() {
|
||||
local start_time
|
||||
start_time=$(date +%s)
|
||||
|
||||
log_info "PlatformIO Build Flags: $PLATFORMIO_BUILD_FLAGS"
|
||||
log_info "Executing: $build_cmd"
|
||||
|
||||
if eval "$build_cmd"; then
|
||||
local end_time
|
||||
end_time=$(date +%s)
|
||||
@@ -184,6 +190,7 @@ Options:
|
||||
--clean Clean build artifacts before building
|
||||
--verbose Enable verbose build output
|
||||
--no-verify Skip artifact verification
|
||||
--version <ver> Set firmware version (default: edge)
|
||||
--help Show this help message
|
||||
|
||||
Examples:
|
||||
@@ -201,10 +208,15 @@ main() {
|
||||
local clean_build_flag=false
|
||||
local verbose=false
|
||||
local verify=true
|
||||
local version="edge"
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--version)
|
||||
version="$2"
|
||||
shift 2
|
||||
;;
|
||||
--dev-ota)
|
||||
enable_dev_ota=true
|
||||
shift
|
||||
@@ -254,7 +266,8 @@ main() {
|
||||
validate_environment "$environment" || exit 1
|
||||
|
||||
# Setup build environment
|
||||
setup_build_env "$enable_dev_ota"
|
||||
export PYTHONIOENCODING=utf-8
|
||||
export PYTHONUTF8=1
|
||||
|
||||
# Clean if requested
|
||||
if [[ "$clean_build_flag" == "true" ]]; then
|
||||
@@ -262,7 +275,7 @@ main() {
|
||||
fi
|
||||
|
||||
# Run build
|
||||
run_build "$environment" "$clean_build_flag" "$verbose" || exit 1
|
||||
run_build "$environment" "$clean_build_flag" "$verbose" "$enable_dev_ota" "$version" || exit 1
|
||||
|
||||
# Verify artifacts
|
||||
if [[ "$verify" == "true" ]]; then
|
||||
|
||||
Executable
+125
@@ -0,0 +1,125 @@
|
||||
#!/bin/bash
|
||||
# Lists all available PlatformIO environments for OpenMQTTGateway
|
||||
# Usage: ./scripts/ci_list.sh [--full]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
readonly SCRIPT_DIR
|
||||
readonly PROJECT_ROOT
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
source "${SCRIPT_DIR}/ci_00_config.sh"
|
||||
else
|
||||
echo "ERROR: ci_00_config.sh not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
## use .github/workflows/environments.json to list environments
|
||||
list_environments_from_Json() {
|
||||
local json_file="${PROJECT_ROOT}/.github/workflows/environments.json"
|
||||
if [[ ! -f "$json_file" ]]; then
|
||||
log_error "JSON file not found: $json_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if ! command -v jq >/dev/null 2>&1; then
|
||||
log_error "jq is required to read $json_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Available PlatformIO environments from JSON:"
|
||||
echo ""
|
||||
local envs=()
|
||||
while IFS= read -r env; do
|
||||
# Skip test environments
|
||||
if [[ ! "$env" =~ -test$ && ! "$env" =~ -all- ]]; then
|
||||
envs+=("$env")
|
||||
fi
|
||||
done < <(jq -r '.environments.all[]? // empty' "$json_file")
|
||||
|
||||
# Sort and display unique environments
|
||||
if [[ ${#envs[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${envs[@]}" | sort -u | column -c 80
|
||||
else
|
||||
log_warn "No environments found in JSON file"
|
||||
fi
|
||||
echo ""
|
||||
log_info "Total: $(printf '%s\n' "${envs[@]}" | sort -u | wc -l) environments"
|
||||
|
||||
}
|
||||
|
||||
|
||||
list_environments() {
|
||||
log_info "Available PlatformIO environments:"
|
||||
echo ""
|
||||
local env_files=("${PROJECT_ROOT}/platformio.ini" "${PROJECT_ROOT}/environments.ini")
|
||||
local envs=()
|
||||
for file in "${env_files[@]}"; do
|
||||
if [[ -f "$file" ]]; then
|
||||
while IFS= read -r line; do
|
||||
if [[ "$line" =~ ^\[env:([^\]]+)\] ]]; then
|
||||
local env_name="${BASH_REMATCH[1]}"
|
||||
envs+=("$env_name")
|
||||
fi
|
||||
done < "$file"
|
||||
fi
|
||||
done
|
||||
# Sort and display unique environments
|
||||
if [[ ${#envs[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${envs[@]}" | sort -u | column -c 80
|
||||
else
|
||||
log_warn "No environments found in configuration files"
|
||||
fi
|
||||
echo ""
|
||||
log_info "Total: $(printf '%s\n' "${envs[@]}" | sort -u | wc -l) environments"
|
||||
}
|
||||
|
||||
# Main execution
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: ${0##*/} [--full]
|
||||
|
||||
List available PlatformIO environments for OpenMQTTGateway.
|
||||
|
||||
Options:
|
||||
--full Parse platformio.ini and environments.ini for an exhaustive list
|
||||
(slower, includes all defined envs except *-test / *-all-)
|
||||
(default) Use .github/workflows/environments.json (fast, curated list)
|
||||
|
||||
Examples:
|
||||
${0##*/} # quick, curated list
|
||||
${0##*/} --full # exhaustive list from config files
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
USE_FULL=false
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--full)
|
||||
USE_FULL=true
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_warn "Unknown option: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ "$USE_FULL" == true ]]; then
|
||||
list_environments
|
||||
else
|
||||
list_environments_from_Json
|
||||
fi
|
||||
fi
|
||||
+108
-134
@@ -1,4 +1,5 @@
|
||||
#!/bin/bash
|
||||
# shellcheck disable=SC2015
|
||||
# Prepares firmware artifacts for upload or deployment
|
||||
# Used by: CI/CD pipelines for artifact packaging
|
||||
# Usage: ./prepare_artifacts.sh <environment> [OPTIONS]
|
||||
@@ -6,8 +7,10 @@
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
readonly SCRIPT_DIR
|
||||
readonly PROJECT_ROOT
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
@@ -22,17 +25,22 @@ BUILD_DIR="${PROJECT_ROOT}/${BUILD_DIR}"
|
||||
DEFAULT_OUTPUT_DIR="${PROJECT_ROOT}/${ARTIFACTS_DIR}"
|
||||
|
||||
# Function to create output directory
|
||||
create_output_dir() {
|
||||
prepare_output_dir() {
|
||||
local output_dir="$1"
|
||||
local clean_flag="${2:-false}"
|
||||
|
||||
if [[ -d "$output_dir" ]]; then
|
||||
log_warn "Output directory already exists: $output_dir"
|
||||
log_info "Cleaning existing artifacts..."
|
||||
rm -rf "$output_dir"
|
||||
if [[ "$clean_flag" == "true" ]]; then
|
||||
log_warn "Cleaning and recreating output directory: $output_dir"
|
||||
rm -rf "$output_dir"
|
||||
mkdir -p "$output_dir"
|
||||
else
|
||||
log_warn "Output directory already exists and will be reused: $output_dir"
|
||||
fi
|
||||
else
|
||||
mkdir -p "$output_dir"
|
||||
log_info "Created output directory: $output_dir"
|
||||
fi
|
||||
|
||||
mkdir -p "$output_dir"
|
||||
log_info "Created output directory: $output_dir"
|
||||
}
|
||||
|
||||
# Function to copy artifact with optional renaming
|
||||
@@ -58,42 +66,14 @@ copy_artifact() {
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to prepare standard artifacts (no renaming)
|
||||
prepare_standard_artifacts() {
|
||||
local env="$1"
|
||||
local output_dir="$2"
|
||||
local env_dir="${BUILD_DIR}/${env}"
|
||||
|
||||
log_info "Preparing STANDARD artifacts for: $env"
|
||||
|
||||
local copied=0
|
||||
|
||||
# Copy firmware.bin (required)
|
||||
if copy_artifact "${env_dir}/firmware.bin" "${output_dir}/firmware.bin" "firmware"; then
|
||||
((copied++))
|
||||
fi
|
||||
|
||||
# Copy partitions.bin (optional)
|
||||
copy_artifact "${env_dir}/partitions.bin" "${output_dir}/partitions.bin" "partitions" && ((copied++)) || true
|
||||
|
||||
# Note: bootloader.bin is NOT copied in standard mode (only needed for deployment)
|
||||
|
||||
if [[ $copied -eq 0 ]]; then
|
||||
log_error "No artifacts were copied"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Copied ${copied} artifact(s) in standard mode"
|
||||
}
|
||||
|
||||
# Function to prepare deployment artifacts (with renaming)
|
||||
prepare_deployment_artifacts() {
|
||||
prepare_artifacts() {
|
||||
local env="$1"
|
||||
local output_dir="$2"
|
||||
local env_dir="${BUILD_DIR}/${env}"
|
||||
|
||||
log_info "Preparing DEPLOYMENT artifacts for: $env"
|
||||
|
||||
log_info "Preparing firmware directory for: $env"
|
||||
local copied=0
|
||||
|
||||
# Copy and rename firmware.bin
|
||||
@@ -118,54 +98,57 @@ prepare_deployment_artifacts() {
|
||||
log_info "Copied ${copied} artifact(s) in deployment mode"
|
||||
}
|
||||
|
||||
# Function to create manifest file
|
||||
create_manifest() {
|
||||
prepare_libraries() {
|
||||
local env="$1"
|
||||
local output_dir="$2"
|
||||
local manifest="${output_dir}/manifest.txt"
|
||||
local env_dir="${BUILD_DIR}/${env}"
|
||||
|
||||
# Process libraries: create temp copy with renamed folders, zip, preserve originals
|
||||
log_info "Processing libraries for environment: $env"
|
||||
TEMP_LIBDEPS=$(mktemp -p "$output_dir" -d) || { echo "Failed to create temp directory"; return 1; }
|
||||
|
||||
cp -r .pio/libdeps/"$env" "$TEMP_LIBDEPS/" || { log_error "Failed to copy libdeps for $env"; return 1; }
|
||||
|
||||
(
|
||||
cd "$TEMP_LIBDEPS"
|
||||
log_step "Replace space by _ in folder names (temp copy only)"
|
||||
find . -type d -name "* *" | while read -r FNAME; do
|
||||
mv "$FNAME" "${FNAME// /_}"
|
||||
done
|
||||
|
||||
log_info "Creating artifact manifest..."
|
||||
log_step "Zipping libraries per board"
|
||||
for i in */; do
|
||||
tar -czf "${i%/}-libraries.tgz" "$i" > /dev/null
|
||||
done
|
||||
|
||||
{
|
||||
echo "Environment: $env"
|
||||
echo "Build Date: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
|
||||
echo "Build Host: $(hostname)"
|
||||
echo ""
|
||||
echo "Artifacts:"
|
||||
|
||||
find "$output_dir" -type f -name "*.bin" | sort | while read -r file; do
|
||||
local size
|
||||
size=$(stat -f%z "$file" 2>/dev/null || stat -c%s "$file" 2>/dev/null)
|
||||
local size_kb=$((size / 1024))
|
||||
local md5sum_val
|
||||
md5sum_val=$(md5sum "$file" 2>/dev/null | cut -d' ' -f1 || md5 -q "$file" 2>/dev/null)
|
||||
echo " - $(basename "$file"): ${size_kb} KB (MD5: ${md5sum_val})"
|
||||
done
|
||||
} > "$manifest"
|
||||
|
||||
log_info "Manifest created: $manifest"
|
||||
mv ./*.tgz "${output_dir}"
|
||||
)
|
||||
|
||||
rm -rf "$TEMP_LIBDEPS"
|
||||
log_info "✓ Created library archives in: $output_dir"
|
||||
}
|
||||
|
||||
# Function to compress artifacts (optional)
|
||||
compress_artifacts() {
|
||||
prepare_sources() {
|
||||
local output_dir="$1"
|
||||
local archive_name="$2"
|
||||
|
||||
log_info "Compressing artifacts..."
|
||||
log_info "Preparing source code archive"
|
||||
|
||||
local archive="${output_dir}/${archive_name}.tar.gz"
|
||||
|
||||
if tar -czf "$archive" -C "$output_dir" .; then
|
||||
local size
|
||||
size=$(stat -f%z "$archive" 2>/dev/null || stat -c%s "$archive" 2>/dev/null)
|
||||
local size_kb=$((size / 1024))
|
||||
log_info "Archive created: ${archive_name}.tar.gz (${size_kb} KB)"
|
||||
# Create and move sources tar.gz (newly generated, safe to move)
|
||||
if tar -czf "${output_dir}/OpenMQTTGateway_sources.tgz" main LICENSE.txt > /dev/null; then
|
||||
log_info "✓ Created source archive: OpenMQTTGateway_sources.tgz"
|
||||
else
|
||||
log_error "Failed to create archive"
|
||||
log_error "Failed to create source archive"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
# Function to list artifacts
|
||||
list_artifacts() {
|
||||
local output_dir="$1"
|
||||
@@ -194,17 +177,14 @@ usage() {
|
||||
cat << EOF
|
||||
Usage: $0 <environment> [OPTIONS]
|
||||
|
||||
Prepare firmware artifacts for upload or deployment.
|
||||
Prepare artifacts for upload or deployment.
|
||||
|
||||
Arguments:
|
||||
environment PlatformIO environment name
|
||||
environment PlatformIO environment name, if omitted will be created source archive only.
|
||||
|
||||
Options:
|
||||
--deploy Prepare for deployment (rename with environment prefix)
|
||||
--standard Prepare standard artifacts (no renaming) [default]
|
||||
--clean Clean existing output directory before preparing artifacts
|
||||
--output DIR Output directory [default: generated/artifacts/]
|
||||
--manifest Create manifest file with artifact metadata
|
||||
--compress Compress artifacts into tar.gz archive
|
||||
--help Show this help message
|
||||
|
||||
Examples:
|
||||
@@ -218,32 +198,27 @@ EOF
|
||||
# Main execution
|
||||
main() {
|
||||
local environment=""
|
||||
local mode="standard"
|
||||
local output_dir="$DEFAULT_OUTPUT_DIR"
|
||||
local create_manifest_flag=false
|
||||
local compress_flag=false
|
||||
local clean_flag=false
|
||||
#local version="" ## WILL BE USED WHEN THE VERSION ITSELF AFFECTS THE ARTIFACTS NAMING
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--deploy)
|
||||
mode="deploy"
|
||||
shift
|
||||
;;
|
||||
--standard)
|
||||
mode="standard"
|
||||
shift
|
||||
;;
|
||||
--output)
|
||||
output_dir="$2"
|
||||
shift 2
|
||||
;;
|
||||
--manifest)
|
||||
create_manifest_flag=true
|
||||
shift
|
||||
-v|--version)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
log_error "-v|--version requires a version string"
|
||||
return 1
|
||||
fi
|
||||
#version="$2"
|
||||
shift 2
|
||||
;;
|
||||
--compress)
|
||||
compress_flag=true
|
||||
--clean)
|
||||
clean_flag=true
|
||||
shift
|
||||
;;
|
||||
--help|-h)
|
||||
@@ -262,48 +237,47 @@ main() {
|
||||
esac
|
||||
done
|
||||
|
||||
# Validate inputs
|
||||
if [[ -z "$environment" ]]; then
|
||||
log_error "Environment name is required"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Change to project root
|
||||
cd "$PROJECT_ROOT"
|
||||
|
||||
# Check if build directory exists
|
||||
if [[ ! -d "${BUILD_DIR}/${environment}" ]]; then
|
||||
log_error "Build directory not found for environment: $environment"
|
||||
log_error "Run build_firmware.sh first"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create output directory
|
||||
create_output_dir "$output_dir"
|
||||
|
||||
# Prepare artifacts based on mode
|
||||
case "$mode" in
|
||||
standard)
|
||||
prepare_standard_artifacts "$environment" "$output_dir" || exit 1
|
||||
;;
|
||||
deploy)
|
||||
prepare_deployment_artifacts "$environment" "$output_dir" || exit 1
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown mode: $mode"
|
||||
|
||||
# TODO FOR NEXT STEP MULTI RELEASE: TAG, RC, edge
|
||||
#if [[ -n "$version" ]]; then
|
||||
# # Sanitize version string for directory name
|
||||
# safe_version=$(echo "$version" | sed 's/[^a-zA-Z0-9._-]/_/g')
|
||||
# output_dir="${output_dir}/${safe_version}"
|
||||
#fi
|
||||
|
||||
|
||||
# Validate inputs
|
||||
if [[ -z "$environment" ]]; then
|
||||
log_info "No environment specified, only preparing source archive"
|
||||
|
||||
# Create output directory
|
||||
prepare_output_dir "$output_dir" "$clean_flag"
|
||||
|
||||
# Prepare source code archive
|
||||
prepare_sources "$output_dir" || exit 1
|
||||
else
|
||||
# Check if build directory exists
|
||||
if [[ ! -d "${BUILD_DIR}/${environment}" ]]; then
|
||||
log_error "Build directory not found for environment: $environment"
|
||||
log_error "Run build_firmware.sh first"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# Create manifest if requested
|
||||
if [[ "$create_manifest_flag" == "true" ]]; then
|
||||
create_manifest "$environment" "$output_dir"
|
||||
fi
|
||||
|
||||
# Compress if requested
|
||||
if [[ "$compress_flag" == "true" ]]; then
|
||||
compress_artifacts "$output_dir" "$environment"
|
||||
fi
|
||||
|
||||
#normalize output directory path
|
||||
#output_dir="${output_dir}/firmware-${environment}"
|
||||
output_dir="${output_dir}/firmware_build"
|
||||
|
||||
|
||||
# Create output directory
|
||||
prepare_output_dir "$output_dir" "$clean_flag"
|
||||
|
||||
# Prepare artifacts based on mode
|
||||
prepare_artifacts "$environment" "$output_dir" || exit 1
|
||||
|
||||
# Prepare libraries
|
||||
prepare_libraries "$environment" "$output_dir" || exit 1
|
||||
fi
|
||||
|
||||
# Show summary
|
||||
|
||||
+70
-5
@@ -6,8 +6,10 @@
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
readonly SCRIPT_DIR
|
||||
readonly PROJECT_ROOT
|
||||
|
||||
# Load shared configuration
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
@@ -18,7 +20,6 @@ else
|
||||
fi
|
||||
|
||||
# Default values
|
||||
CHECK_MODE=true
|
||||
FIX_MODE=false
|
||||
FORMAT_ONLY=false
|
||||
SOURCE_DIR="main"
|
||||
@@ -202,6 +203,63 @@ fix_formatting() {
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to run shellcheck on scripts
|
||||
run_shellcheck() {
|
||||
log_info "Checking shell scripts with ShellCheck..."
|
||||
|
||||
# Check if shellcheck is installed
|
||||
if ! command -v shellcheck >/dev/null 2>&1; then
|
||||
log_warn "ShellCheck not found, skipping shell script checks"
|
||||
log_info "To install ShellCheck:"
|
||||
log_info " Ubuntu/Debian: sudo apt-get install shellcheck"
|
||||
log_info " macOS: brew install shellcheck"
|
||||
return 0 # Don't fail, just skip
|
||||
fi
|
||||
|
||||
log_info "✓ ShellCheck found: $(shellcheck --version | head -n2 | tail -n1)"
|
||||
|
||||
# Find all .sh files in scripts directory
|
||||
local scripts_dir="${PROJECT_ROOT}/scripts"
|
||||
if [[ ! -d "$scripts_dir" ]]; then
|
||||
log_warn "Scripts directory not found: $scripts_dir"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local shell_files
|
||||
shell_files=$(find "$scripts_dir" -type f -name "*.sh" 2>/dev/null)
|
||||
|
||||
if [[ -z "$shell_files" ]]; then
|
||||
log_warn "No shell scripts found in $scripts_dir"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local file_count
|
||||
file_count=$(echo "$shell_files" | wc -l)
|
||||
log_info "Found $file_count shell script(s) to check"
|
||||
|
||||
# Run shellcheck on all files at once to allow cross-file analysis
|
||||
log_info "Running ShellCheck on scripts directory..."
|
||||
|
||||
local shellcheck_output
|
||||
# shellcheck disable=SC2086
|
||||
shellcheck_output=$(shellcheck -f gcc $shell_files 2>&1)
|
||||
local shellcheck_result=$?
|
||||
|
||||
if [[ $shellcheck_result -ne 0 ]]; then
|
||||
echo ""
|
||||
log_warn "⚠ ShellCheck found issues:"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "$shellcheck_output"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
log_error "Please fix the ShellCheck warnings/errors"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_success "✓ All $file_count shell script(s) passed ShellCheck"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Function to run all QA checks
|
||||
run_all_checks() {
|
||||
log_info "Running all quality assurance checks..."
|
||||
@@ -215,6 +273,13 @@ run_all_checks() {
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# ShellCheck
|
||||
log_info "═══ Shell Scripts Check ═══"
|
||||
if ! run_shellcheck; then
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Future: Add more checks here
|
||||
# - cppcheck static analysis
|
||||
# - code complexity metrics
|
||||
@@ -237,6 +302,7 @@ run_format_check() {
|
||||
local clang_format_cmd
|
||||
clang_format_cmd=$(check_clang_format "$CLANG_FORMAT_VERSION")
|
||||
|
||||
# shellcheck disable=SC2181
|
||||
if [[ $? -ne 0 ]] || [[ -z "$clang_format_cmd" ]]; then
|
||||
log_error "clang-format not found"
|
||||
log_error "Please install clang-format:"
|
||||
@@ -258,6 +324,7 @@ run_format_check() {
|
||||
local files
|
||||
files=$(find_files "$SOURCE_DIR" "$EXTENSIONS")
|
||||
|
||||
# shellcheck disable=SC2181
|
||||
if [[ $? -ne 0 ]] || [[ -z "$files" ]]; then
|
||||
log_error "Source directory not found: ${PROJECT_ROOT}/${SOURCE_DIR}"
|
||||
return 1
|
||||
@@ -317,13 +384,11 @@ parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--check)
|
||||
CHECK_MODE=true
|
||||
FIX_MODE=false
|
||||
shift
|
||||
;;
|
||||
--fix)
|
||||
FIX_MODE=true
|
||||
CHECK_MODE=false
|
||||
shift
|
||||
;;
|
||||
--format)
|
||||
|
||||
Executable
+428
@@ -0,0 +1,428 @@
|
||||
#!/bin/bash
|
||||
# OpenMQTTGateway Security Scan Script
|
||||
# Runs Trivy vulnerability scanner and generates reports
|
||||
# Based on task-security-scan.yml workflow
|
||||
# Usage: ./scripts/ci_security.sh [OPTIONS]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
readonly SCRIPT_DIR
|
||||
readonly PROJECT_ROOT
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
source "${SCRIPT_DIR}/ci_00_config.sh"
|
||||
else
|
||||
echo "ERROR: ci_00_config.sh not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Default values
|
||||
SCAN_TYPE="fs"
|
||||
SEVERITY="HIGH,CRITICAL"
|
||||
EXIT_CODE_ON_VULN="0"
|
||||
SCAN_PATH="."
|
||||
EXCLUDE_PATHS=""
|
||||
GENERATE_SBOM=false # Off by default
|
||||
|
||||
OUTPUT_DIR="${PROJECT_ROOT}/${REPORTS_DIR}"
|
||||
|
||||
# Show usage
|
||||
usage() {
|
||||
cat << EOF
|
||||
Usage: $0 [OPTIONS]
|
||||
|
||||
Security vulnerability scanning using Trivy
|
||||
|
||||
Options:
|
||||
--scan-type <type> Type of scan: fs (filesystem), config, or image
|
||||
Default: fs
|
||||
|
||||
--severity <levels> Severity levels to report (comma-separated)
|
||||
Options: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
|
||||
Default: HIGH,CRITICAL
|
||||
|
||||
--scan-path <path> Path to scan (default: entire repository)
|
||||
Default: .
|
||||
|
||||
--exit-code <code> Exit code when vulnerabilities found
|
||||
0 = continue (don't fail)
|
||||
1 = fail build
|
||||
Default: 0
|
||||
|
||||
--exclude <paths> Paths to exclude from scan (comma-separated)
|
||||
Example: node_modules,test,docs
|
||||
Default: (none)
|
||||
|
||||
--generate-sbom Generate Software Bill of Materials (SBOM)
|
||||
Creates CycloneDX and SPDX JSON formats
|
||||
Default: off
|
||||
|
||||
--output-dir <dir> Directory to save reports
|
||||
Default: ./generated/reports
|
||||
|
||||
-h, --help Show this help message
|
||||
|
||||
Examples:
|
||||
# Scan filesystem for HIGH and CRITICAL issues
|
||||
$0
|
||||
|
||||
# Scan config files and fail if vulnerabilities found
|
||||
$0 --scan-type config --exit-code 1
|
||||
|
||||
# Scan specific path for all severity levels
|
||||
$0 --scan-path ./main --severity UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
|
||||
|
||||
# Scan with excluded paths
|
||||
$0 --exclude node_modules,test,docs
|
||||
|
||||
# Scan with multiple options
|
||||
$0 --scan-path ./main --exclude test --severity HIGH,CRITICAL --exit-code 1
|
||||
|
||||
# Scan with SBOM generation
|
||||
$0 --generate-sbom
|
||||
|
||||
# Complete scan with SBOM and custom severity
|
||||
$0 --severity UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL --generate-sbom
|
||||
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
# Function to check if Trivy is installed
|
||||
check_trivy() {
|
||||
if ! command -v trivy &> /dev/null; then
|
||||
log_error "Trivy is not installed. Install it from: https://github.com/aquasecurity/trivy"
|
||||
return 1
|
||||
fi
|
||||
log_success "Trivy found: $(trivy --version)"
|
||||
}
|
||||
|
||||
# Function to check if jq is installed (for JSON parsing)
|
||||
check_jq() {
|
||||
if ! command -v jq &> /dev/null; then
|
||||
log_warn "jq is not installed. Some report features will be limited."
|
||||
return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# Function to run Trivy scan with specified format
|
||||
# Usage: run_trivy_scan <format> <filename> [exit_code]
|
||||
run_trivy_scan() {
|
||||
local format="$1"
|
||||
local filename="$2"
|
||||
local exit_code="${3:-0}" # Default to 0 for non-critical scans
|
||||
local output_file="${OUTPUT_DIR}/${filename}"
|
||||
|
||||
log_info "Running Trivy $SCAN_TYPE scan ($format format)..."
|
||||
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
# Build Trivy command with all enabled scanners
|
||||
local trivy_cmd=(
|
||||
trivy "$SCAN_TYPE" "$SCAN_PATH"
|
||||
--format "$format"
|
||||
--output "$output_file"
|
||||
--severity "$SEVERITY"
|
||||
--exit-code "$exit_code"
|
||||
)
|
||||
|
||||
# Add exclude paths if provided
|
||||
if [[ -n "$EXCLUDE_PATHS" ]]; then
|
||||
IFS=',' read -r -a exclude_dirs <<< "$EXCLUDE_PATHS"
|
||||
for skip_dir in "${exclude_dirs[@]}"; do
|
||||
if [[ -n "$skip_dir" ]]; then
|
||||
trivy_cmd+=(--skip-dirs "$skip_dir")
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
log_info "Executing Trivy ${SCAN_TYPE} scan"
|
||||
|
||||
if "${trivy_cmd[@]}" 2>&1 | tee -a "${OUTPUT_DIR}/trivy-scan.log"; then
|
||||
log_success "${format^^} report generated: $output_file"
|
||||
return 0
|
||||
else
|
||||
local rc=$?
|
||||
if [[ $rc -eq 1 && "$exit_code" == "1" ]]; then
|
||||
log_error "Vulnerabilities found (exit code: $rc)"
|
||||
return $rc
|
||||
fi
|
||||
log_success "Scan completed with exit code: $rc"
|
||||
return 0
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to create summary report
|
||||
create_summary_report() {
|
||||
log_info "Creating security summary report..."
|
||||
|
||||
local summary_file="${OUTPUT_DIR}/security-summary.md"
|
||||
|
||||
{
|
||||
echo "# 🔒 Security Scan Results"
|
||||
echo ""
|
||||
echo "**Scan Type**: $SCAN_TYPE"
|
||||
echo "**Path Scanned**: $SCAN_PATH"
|
||||
echo "**Severity Filter**: $SEVERITY"
|
||||
echo "**Scan Date**: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
|
||||
echo ""
|
||||
} > "$summary_file"
|
||||
|
||||
# Parse SARIF if it exists
|
||||
local sarif_file="${OUTPUT_DIR}/trivy-results.sarif"
|
||||
if [[ -f "$sarif_file" ]]; then
|
||||
log_info "Parsing SARIF results..."
|
||||
|
||||
if check_jq; then
|
||||
# Count vulnerabilities by severity
|
||||
local vuln_count
|
||||
vuln_count=$(jq '[.runs[].results[]] | length' "$sarif_file" 2>/dev/null || echo "0")
|
||||
|
||||
local critical high medium low
|
||||
critical=$(jq '[.runs[].results[] | select(.level == "error")] | length' "$sarif_file" 2>/dev/null || echo "0")
|
||||
high=$(jq '[.runs[].results[] | select(.level == "warning")] | length' "$sarif_file" 2>/dev/null || echo "0")
|
||||
medium=$(jq '[.runs[].results[] | select(.level == "note")] | length' "$sarif_file" 2>/dev/null || echo "0")
|
||||
low=$(jq '[.runs[].results[] | select(.level == "none")] | length' "$sarif_file" 2>/dev/null || echo "0")
|
||||
|
||||
{
|
||||
echo "## Vulnerability Summary"
|
||||
echo ""
|
||||
echo "**Total Vulnerabilities**: ${vuln_count}"
|
||||
echo ""
|
||||
echo "- 🔴 **Critical**: ${critical}"
|
||||
echo "- 🟠 **High**: ${high}"
|
||||
echo "- 🟡 **Medium**: ${medium}"
|
||||
echo "- 🟢 **Low**: ${low}"
|
||||
echo ""
|
||||
} >> "$summary_file"
|
||||
|
||||
# List vulnerability details
|
||||
if [[ "$vuln_count" -gt 0 ]]; then
|
||||
{
|
||||
echo "## Vulnerability Details"
|
||||
echo ""
|
||||
} >> "$summary_file"
|
||||
|
||||
jq -r '.runs[].results[] |
|
||||
"### \(.level | ascii_upcase): \(.ruleId)\n" +
|
||||
"**Location**: \(.locations[0].physicalLocation.artifactLocation.uri // "N/A")\n" +
|
||||
"**Message**: \(.message.text)\n" +
|
||||
(if .properties.CVE then "**CVE**: \(.properties.CVE)\n" else "" end) +
|
||||
(if .properties.cvss then "**CVSS Score**: \(.properties.cvss)\n" else "" end) +
|
||||
""' "$sarif_file" >> "$summary_file" 2>/dev/null || \
|
||||
echo "Could not parse vulnerability details" >> "$summary_file"
|
||||
fi
|
||||
else
|
||||
echo "⚠️ jq not available for detailed SARIF parsing" >> "$summary_file"
|
||||
fi
|
||||
else
|
||||
echo "⚠️ SARIF file not found" >> "$summary_file"
|
||||
fi
|
||||
|
||||
# Add available report formats
|
||||
{
|
||||
echo "## Report Formats Available"
|
||||
echo ""
|
||||
if [[ -f "${OUTPUT_DIR}/trivy-results.sarif" ]]; then
|
||||
echo "- ✅ \`trivy-results.sarif\` - SARIF format (for GitHub/IDE integration)"
|
||||
fi
|
||||
if [[ -f "${OUTPUT_DIR}/trivy-report.json" ]]; then
|
||||
echo "- ✅ \`trivy-report.json\` - JSON format (for automation)"
|
||||
fi
|
||||
if [[ -f "${OUTPUT_DIR}/trivy-report.txt" ]]; then
|
||||
echo "- ✅ \`trivy-report.txt\` - Human-readable table"
|
||||
fi
|
||||
echo ""
|
||||
} >> "$summary_file"
|
||||
|
||||
# Add footer
|
||||
{
|
||||
echo "---"
|
||||
echo ""
|
||||
echo "Generated by OpenMQTTGateway CI/CD Security Scan"
|
||||
echo "For more information, see: https://github.com/aquasecurity/trivy"
|
||||
} >> "$summary_file"
|
||||
|
||||
log_success "Summary report generated: $summary_file"
|
||||
}
|
||||
|
||||
# Function to check for critical vulnerabilities and fail if needed
|
||||
check_critical_vulnerabilities() {
|
||||
if [[ "$EXIT_CODE_ON_VULN" != "1" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
local sarif_file="${OUTPUT_DIR}/trivy-results.sarif"
|
||||
|
||||
if [[ ! -f "$sarif_file" ]]; then
|
||||
log_warn "SARIF file not found, skipping critical check"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if check_jq; then
|
||||
local critical
|
||||
critical=$(jq '[.runs[].results[] | select(.level == "error")] | length' "$sarif_file" 2>/dev/null || echo "0")
|
||||
|
||||
if [[ "$critical" -gt 0 ]]; then
|
||||
log_error "❌ Found ${critical} critical vulnerabilities!"
|
||||
log_error "Review the security reports in: $OUTPUT_DIR"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--scan-type)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
log_error "--scan-type requires an argument"
|
||||
return 1
|
||||
fi
|
||||
SCAN_TYPE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--severity)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
log_error "--severity requires an argument"
|
||||
return 1
|
||||
fi
|
||||
SEVERITY="$2"
|
||||
shift 2
|
||||
;;
|
||||
--scan-path)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
log_error "--scan-path requires an argument"
|
||||
return 1
|
||||
fi
|
||||
SCAN_PATH="$2"
|
||||
shift 2
|
||||
;;
|
||||
--exit-code)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
log_error "--exit-code requires an argument"
|
||||
return 1
|
||||
fi
|
||||
EXIT_CODE_ON_VULN="$2"
|
||||
shift 2
|
||||
;;
|
||||
--exclude)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
log_error "--exclude requires an argument"
|
||||
return 1
|
||||
fi
|
||||
EXCLUDE_PATHS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--generate-sbom)
|
||||
GENERATE_SBOM=true
|
||||
shift
|
||||
;;
|
||||
--output-dir)
|
||||
if [[ -z "${2:-}" ]]; then
|
||||
log_error "--output-dir requires an argument"
|
||||
return 1
|
||||
fi
|
||||
OUTPUT_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown option: $1"
|
||||
echo ""
|
||||
usage
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Pre-flight checks
|
||||
log_info "Running security scan checks..."
|
||||
if ! check_trivy; then
|
||||
log_error "Required tool check failed"
|
||||
return 1
|
||||
fi
|
||||
check_jq || true
|
||||
|
||||
echo ""
|
||||
log_info "═══ Security Scan Configuration ═══"
|
||||
echo " Scan Type: $SCAN_TYPE"
|
||||
echo " Path: $SCAN_PATH"
|
||||
echo " Severity: $SEVERITY"
|
||||
echo " Exit on Vulnerabilities: $EXIT_CODE_ON_VULN"
|
||||
echo " Excluded Paths: ${EXCLUDE_PATHS:-none}"
|
||||
echo " Generate SBOM: ${GENERATE_SBOM:-false}"
|
||||
echo " Output Directory: $OUTPUT_DIR"
|
||||
echo ""
|
||||
|
||||
# Run scans
|
||||
if ! run_trivy_scan "sarif" "trivy-results.sarif" "$EXIT_CODE_ON_VULN"; then
|
||||
if [[ "$EXIT_CODE_ON_VULN" == "1" ]]; then
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
run_trivy_scan "json" "trivy-report.json" "0"
|
||||
run_trivy_scan "table" "trivy-report.txt" "0"
|
||||
|
||||
# Create summary
|
||||
create_summary_report
|
||||
|
||||
# Generate SBOM if requested
|
||||
if [[ "$GENERATE_SBOM" == "true" ]]; then
|
||||
log_info "Generating Software Bill of Materials (SBOM)..."
|
||||
|
||||
local sbom_dir="${OUTPUT_DIR}/sbom"
|
||||
mkdir -p "$sbom_dir"
|
||||
|
||||
# Generate CycloneDX SBOM
|
||||
log_info "Generating CycloneDX SBOM..."
|
||||
if trivy "$SCAN_TYPE" "$SCAN_PATH" \
|
||||
--format cyclonedx \
|
||||
--output "$sbom_dir/sbom-cyclonedx.json" \
|
||||
--exit-code 0 2>&1 | grep -i "generated\|error" || true; then
|
||||
if [[ -f "$sbom_dir/sbom-cyclonedx.json" ]]; then
|
||||
log_success "CycloneDX SBOM generated: $sbom_dir/sbom-cyclonedx.json"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Generate SPDX SBOM
|
||||
log_info "Generating SPDX SBOM..."
|
||||
if trivy "$SCAN_TYPE" "$SCAN_PATH" \
|
||||
--format spdx-json \
|
||||
--output "$sbom_dir/sbom-spdx.json" \
|
||||
--exit-code 0 2>&1 | grep -i "generated\|error" || true; then
|
||||
if [[ -f "$sbom_dir/sbom-spdx.json" ]]; then
|
||||
log_success "SPDX SBOM generated: $sbom_dir/sbom-spdx.json"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ -f "$sbom_dir/sbom-cyclonedx.json" ]] || [[ -f "$sbom_dir/sbom-spdx.json" ]]; then
|
||||
log_success "SBOM generation completed"
|
||||
echo ""
|
||||
fi
|
||||
fi
|
||||
# Check for critical vulnerabilities
|
||||
if ! check_critical_vulnerabilities; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
log_success "Security scan completed successfully!"
|
||||
log_info "Reports saved to: $OUTPUT_DIR"
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# Execute main function
|
||||
main "$@"
|
||||
@@ -1,197 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Updates version tags in firmware configuration and JSON files
|
||||
# Used by: CI/CD pipelines for versioning builds
|
||||
# Usage: ./set_version.sh <version_tag> [--dev]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
readonly USER_CONFIG="${PROJECT_ROOT}/main/User_config.h"
|
||||
readonly LATEST_VERSION_PROD="${SCRIPT_DIR}/latest_version.json"
|
||||
readonly LATEST_VERSION_DEV="${SCRIPT_DIR}/latest_version_dev.json"
|
||||
|
||||
# Load shared configuration (colors, logging functions, paths)
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
source "${SCRIPT_DIR}/ci_00_config.sh"
|
||||
else
|
||||
echo "ERROR: ci_00_config.sh not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Function to validate version tag
|
||||
validate_version() {
|
||||
local version="$1"
|
||||
|
||||
if [[ -z "$version" ]] || [[ "$version" == "version_tag" ]]; then
|
||||
log_error "Invalid version tag: '$version'"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Version tag validated: $version"
|
||||
}
|
||||
|
||||
# Function to backup files
|
||||
backup_file() {
|
||||
local file="$1"
|
||||
|
||||
if [[ -f "$file" ]]; then
|
||||
cp "$file" "${file}.bak"
|
||||
log_info "Backed up: $file"
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to replace version in file
|
||||
replace_version() {
|
||||
local file="$1"
|
||||
local version="$2"
|
||||
|
||||
if [[ ! -f "$file" ]]; then
|
||||
log_warn "File not found: $file (skipping)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Backup before modification
|
||||
backup_file "$file"
|
||||
|
||||
# Replace version_tag placeholder
|
||||
if sed -i "s/version_tag/${version}/g" "$file"; then
|
||||
log_info "Updated version in: $file"
|
||||
else
|
||||
log_error "Failed to update version in: $file"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to set production version
|
||||
set_production_version() {
|
||||
local version="$1"
|
||||
|
||||
log_info "Setting PRODUCTION version: $version"
|
||||
|
||||
replace_version "$USER_CONFIG" "$version" || return 1
|
||||
replace_version "$VERSION_JSON" "$version" || return 1
|
||||
}
|
||||
|
||||
# Function to set development version
|
||||
set_development_version() {
|
||||
local version="$1"
|
||||
|
||||
log_info "Setting DEVELOPMENT version: $version"
|
||||
|
||||
replace_version "$USER_CONFIG" "$version" || return 1
|
||||
replace_version "$VERSION_DEV_JSON" "$version" || return 1
|
||||
}
|
||||
|
||||
# Function to restore backups
|
||||
restore_backups() {
|
||||
log_warn "Restoring backups..."
|
||||
|
||||
for file in "$USER_CONFIG" "$VERSION_JSON" "$VERSION_DEV_JSON"; do
|
||||
if [[ -f "${file}.bak" ]]; then
|
||||
mv "${file}.bak" "$file"
|
||||
log_info "Restored: $file"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# Function to clean backups
|
||||
clean_backups() {
|
||||
for file in "$USER_CONFIG" "$VERSION_JSON" "$VERSION_DEV_JSON"; do
|
||||
if [[ -f "${file}.bak" ]]; then
|
||||
rm "${file}.bak"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# Show usage
|
||||
usage() {
|
||||
cat << EOF
|
||||
Usage: $0 <version_tag> [OPTIONS]
|
||||
|
||||
Update version tags in firmware configuration files.
|
||||
|
||||
Arguments:
|
||||
version_tag Version string to inject (e.g., v1.2.3, abc123, dev-20230101)
|
||||
|
||||
Options:
|
||||
--dev Use development version files (latest_version_dev.json)
|
||||
--prod Use production version files (latest_version.json) [default]
|
||||
--help Show this help message
|
||||
|
||||
Examples:
|
||||
$0 v1.2.3 # Production release
|
||||
$0 abc123 --dev # Development build
|
||||
$0 \${{ github.sha }} --dev # CI/CD with commit SHA
|
||||
|
||||
EOF
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
local version=""
|
||||
local is_dev=false
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--dev)
|
||||
is_dev=true
|
||||
shift
|
||||
;;
|
||||
--prod)
|
||||
is_dev=false
|
||||
shift
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
-*)
|
||||
log_error "Unknown option: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
version="$1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Validate version
|
||||
if [[ -z "$version" ]]; then
|
||||
log_error "Version tag is required"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
validate_version "$version" || exit 1
|
||||
|
||||
# Change to project root
|
||||
cd "$PROJECT_ROOT"
|
||||
|
||||
# Set version with error handling
|
||||
if [[ "$is_dev" == true ]]; then
|
||||
set_development_version "$version" || {
|
||||
restore_backups
|
||||
exit 1
|
||||
}
|
||||
else
|
||||
set_production_version "$version" || {
|
||||
restore_backups
|
||||
exit 1
|
||||
}
|
||||
fi
|
||||
|
||||
# Clean up backups on success
|
||||
clean_backups
|
||||
|
||||
log_info "Version update completed successfully"
|
||||
}
|
||||
|
||||
# Run main if executed directly
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
+161
-310
@@ -1,16 +1,19 @@
|
||||
#!/bin/bash
|
||||
# CI/CD Site/Documentation Builder
|
||||
# Builds and deploys VuePress documentation with version management
|
||||
# Usage: ./scripts/ci_site.sh [OPTIONS]
|
||||
# This script builds and deploys VuePress documentation with version management.
|
||||
# Usage: ./scripts/ci_site.sh [OPTIONS] - Specify options for the script execution.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Constants
|
||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
readonly DOCS_DIR="${PROJECT_ROOT}/docs"
|
||||
readonly VUEPRESS_CONFIG="${DOCS_DIR}/.vuepress/config.js"
|
||||
readonly VUEPRESS_BUILD_DIR="${DOCS_DIR}/.vuepress/dist"
|
||||
# Resolve the folder containing this script so relative paths work
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
DOCS_DIR="${PROJECT_ROOT}/docs"
|
||||
readonly SCRIPT_DIR
|
||||
readonly PROJECT_ROOT
|
||||
readonly DOCS_DIR
|
||||
|
||||
|
||||
# Load shared configuration
|
||||
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
|
||||
@@ -20,29 +23,23 @@ else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Final output directory for site
|
||||
# Final output directory for the site
|
||||
readonly SITE_OUTPUT_DIR="${PROJECT_ROOT}/${SITE_DIR}"
|
||||
|
||||
# Default values
|
||||
MODE="prod"
|
||||
URL_PREFIX="/"
|
||||
CUSTOM_VERSION=""
|
||||
VERSION_SOURCE="release"
|
||||
PREVIEW=false
|
||||
GENERATE_WEBUPLOADER=true
|
||||
WEBUPLOADER_ARGS=""
|
||||
DEPLOY_DIR="."
|
||||
RUN_PAGESPEED=false
|
||||
PAGESPEED_URL="https://docs.openmqttgateway.com/"
|
||||
MODE="prod" # Set the default mode to production
|
||||
CURL_INSECURE=false # Allow curl to skip TLS verification (use only when needed)
|
||||
CLEAN=false # Clean generated/site before build
|
||||
LEGACY_OPENSSL=false # Use --openssl-legacy-provider for older Node.js versions
|
||||
|
||||
# Function to check required tools
|
||||
check_requirements() {
|
||||
log_info "Checking required tools..."
|
||||
|
||||
local missing_tools=()
|
||||
|
||||
|
||||
if ! command -v node >/dev/null 2>&1; then
|
||||
missing_tools+=("node")
|
||||
missing_tools+=("node") # Node.js is required
|
||||
else
|
||||
local node_version
|
||||
node_version=$(node --version)
|
||||
@@ -50,27 +47,19 @@ check_requirements() {
|
||||
fi
|
||||
|
||||
if ! command -v npm >/dev/null 2>&1; then
|
||||
missing_tools+=("npm")
|
||||
missing_tools+=("npm") # npm is required
|
||||
else
|
||||
local npm_version
|
||||
npm_version=$(npm --version)
|
||||
log_info "✓ npm ${npm_version} found"
|
||||
fi
|
||||
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
missing_tools+=("python3")
|
||||
if ! command -v openssl >/dev/null 2>&1; then
|
||||
missing_tools+=("openssl") # OpenSSL is required
|
||||
else
|
||||
local python_version
|
||||
python_version=$(python3 --version | grep -oP '\d+\.\d+' || echo "unknown")
|
||||
log_info "✓ Python ${python_version} found"
|
||||
fi
|
||||
|
||||
if ! command -v pip3 >/dev/null 2>&1; then
|
||||
missing_tools+=("pip3")
|
||||
else
|
||||
local pip_version
|
||||
pip_version=$(pip3 --version | grep -oP '\d+\.\d+' || echo "unknown")
|
||||
log_info "✓ pip ${pip_version} found"
|
||||
local openssl_version
|
||||
openssl_version=$(openssl version | grep -oP '\d+\.\d+\.\d+' || echo "unknown")
|
||||
log_info "✓ OpenSSL ${openssl_version} found"
|
||||
fi
|
||||
|
||||
if [[ ${#missing_tools[@]} -gt 0 ]]; then
|
||||
@@ -86,21 +75,6 @@ check_requirements() {
|
||||
install_dependencies() {
|
||||
log_info "Installing dependencies..."
|
||||
|
||||
# Upgrade pip first (if pip module is available)
|
||||
if python3 -m pip --version >/dev/null 2>&1; then
|
||||
log_info "Upgrading pip..."
|
||||
python3 -m pip install --upgrade pip --quiet || {
|
||||
log_warn "Failed to upgrade pip, continuing with existing version..."
|
||||
}
|
||||
fi
|
||||
|
||||
# Install Python dependencies (without --user if in virtualenv)
|
||||
log_info "Installing Python dependencies..."
|
||||
pip3 install requests pandas markdown pytablereader tabulate || {
|
||||
log_error "Failed to install Python dependencies"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Install Node dependencies
|
||||
log_info "Installing Node.js dependencies..."
|
||||
cd "${PROJECT_ROOT}"
|
||||
@@ -118,118 +92,49 @@ download_common_config() {
|
||||
|
||||
local config_url="https://www.theengs.io/commonConfig.js"
|
||||
local config_dest="${DOCS_DIR}/.vuepress/public/commonConfig.js"
|
||||
local curl_opts="-sSf"
|
||||
|
||||
# Optionally disable TLS verification if explicitly requested
|
||||
if [[ "${CURL_INSECURE}" == "true" ]]; then
|
||||
curl_opts+="k"
|
||||
log_warn "curl is running with --insecure; TLS verification is disabled"
|
||||
fi
|
||||
|
||||
mkdir -p "$(dirname "$config_dest")"
|
||||
|
||||
if curl -sSf -o "$config_dest" "$config_url"; then
|
||||
if curl ${curl_opts} -o "$config_dest" "$config_url"; then
|
||||
log_info "✓ Common config downloaded"
|
||||
else
|
||||
log_warn "Failed to download common config, continuing anyway..."
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to get version
|
||||
get_version() {
|
||||
local version=""
|
||||
|
||||
if [[ "$VERSION_SOURCE" == "custom" && -n "$CUSTOM_VERSION" ]]; then
|
||||
version="$CUSTOM_VERSION"
|
||||
log_info "Using custom version: $version"
|
||||
elif [[ "$VERSION_SOURCE" == "release" ]]; then
|
||||
# Try to get latest git tag (simulating GitHub release)
|
||||
if command -v git >/dev/null 2>&1 && [[ -d "${PROJECT_ROOT}/.git" ]]; then
|
||||
version=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
|
||||
log_info "Using release version from git: $version"
|
||||
else
|
||||
version="development"
|
||||
log_warn "Git not available, using version: $version"
|
||||
fi
|
||||
else
|
||||
# Auto-detect from git
|
||||
if command -v git >/dev/null 2>&1 && [[ -d "${PROJECT_ROOT}/.git" ]]; then
|
||||
version=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
|
||||
log_info "Using git version: $version"
|
||||
else
|
||||
version="development"
|
||||
log_warn "Git not available, using version: $version"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "$version"
|
||||
}
|
||||
|
||||
# Function to set version in config files
|
||||
set_version() {
|
||||
local version="$1"
|
||||
|
||||
log_info "Setting version: $version"
|
||||
|
||||
# Update VuePress config
|
||||
if [[ -f "$VUEPRESS_CONFIG" ]]; then
|
||||
sed -i "s|version_tag|${version}|g" "$VUEPRESS_CONFIG"
|
||||
fi
|
||||
|
||||
# Update version JSON file based on version source
|
||||
if [[ "$VERSION_SOURCE" == "custom" ]]; then
|
||||
# Custom version updates dev file
|
||||
local version_file="${SCRIPT_DIR}/latest_version_dev.json"
|
||||
if [[ -f "$version_file" ]]; then
|
||||
sed -i "s|version_tag|${version}|g" "$version_file"
|
||||
fi
|
||||
else
|
||||
# Release version updates production file
|
||||
local version_file="${SCRIPT_DIR}/latest_version.json"
|
||||
if [[ -f "$version_file" ]]; then
|
||||
sed -i "s|version_tag|${version}|g" "$version_file"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to set URL prefix (base path)
|
||||
set_url_prefix() {
|
||||
create_configuration_files() {
|
||||
local url_prefix="$1"
|
||||
|
||||
if [[ "$url_prefix" != "/" ]]; then
|
||||
log_info "Setting URL prefix: $url_prefix"
|
||||
sed -i "s|base: '/'|base: '${url_prefix}'|g" "$VUEPRESS_CONFIG"
|
||||
fi
|
||||
}
|
||||
local version="$2"
|
||||
local mode="$3"
|
||||
local dest_line=""
|
||||
|
||||
# Function to generate board documentation
|
||||
generate_board_docs() {
|
||||
log_info "Generating board documentation..."
|
||||
|
||||
local generator="${SCRIPT_DIR}/generate_board_docs.py"
|
||||
|
||||
if [[ -f "$generator" ]]; then
|
||||
cd "${PROJECT_ROOT}"
|
||||
python3 "$generator" || {
|
||||
log_warn "Board documentation generation failed, continuing..."
|
||||
}
|
||||
else
|
||||
log_warn "Board documentation generator not found, skipping..."
|
||||
if [[ "$mode" == "dev" ]]; then
|
||||
dest_line=" \"dest\": \"generated/site/dev\","
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to generate WebUploader manifest
|
||||
generate_webuploader() {
|
||||
if [[ "$GENERATE_WEBUPLOADER" != true ]]; then
|
||||
log_info "Skipping WebUploader generation"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log_info "Generating WebUploader manifest..."
|
||||
|
||||
local generator="${SCRIPT_DIR}/gen_wu.py"
|
||||
|
||||
if [[ -f "$generator" ]]; then
|
||||
cd "${PROJECT_ROOT}"
|
||||
python3 "$generator" $WEBUPLOADER_ARGS || {
|
||||
log_warn "WebUploader generation failed, continuing..."
|
||||
}
|
||||
else
|
||||
log_warn "WebUploader generator not found, skipping..."
|
||||
fi
|
||||
# download common config
|
||||
download_common_config
|
||||
|
||||
|
||||
## Create a meta.json file on config folder
|
||||
local meta_file="${DOCS_DIR}/.vuepress/meta.json"
|
||||
cat > "$meta_file" <<EOF
|
||||
{
|
||||
"title": "Theengs OpenMQTTGateway",
|
||||
"version": "${version}",
|
||||
"url_prefix": "${url_prefix}",
|
||||
${dest_line}
|
||||
"mode": "${mode}"
|
||||
}
|
||||
EOF
|
||||
log_info "✓ Generated meta.json at ${meta_file}"
|
||||
}
|
||||
|
||||
# Function to build documentation
|
||||
@@ -238,45 +143,44 @@ build_docs() {
|
||||
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
# Set Node options for compatibility with newer Node.js versions
|
||||
export NODE_OPTIONS="--openssl-legacy-provider"
|
||||
# Set Node options for compatibility with older Node.js versions (if requested)
|
||||
if [[ "$LEGACY_OPENSSL" == true ]]; then
|
||||
log_info "Using --openssl-legacy-provider for older Node.js compatibility"
|
||||
export NODE_OPTIONS="--openssl-legacy-provider"
|
||||
fi
|
||||
|
||||
npm run docs:build || {
|
||||
log_error "Documentation build failed"
|
||||
return 1
|
||||
}
|
||||
|
||||
if [[ -d "$VUEPRESS_BUILD_DIR" ]]; then
|
||||
log_info "✓ Documentation built successfully"
|
||||
|
||||
# Copy to centralized output directory
|
||||
log_info "Copying site to: $SITE_OUTPUT_DIR"
|
||||
mkdir -p "$SITE_OUTPUT_DIR"
|
||||
rm -rf "${SITE_OUTPUT_DIR}"/*
|
||||
cp -r "${VUEPRESS_BUILD_DIR}"/* "${SITE_OUTPUT_DIR}/"
|
||||
|
||||
log_info " VuePress output: $VUEPRESS_BUILD_DIR"
|
||||
log_info " Final output: $SITE_OUTPUT_DIR"
|
||||
else
|
||||
log_error "Build output directory not found: $VUEPRESS_BUILD_DIR"
|
||||
return 1
|
||||
fi
|
||||
log_info "✓ Documentation built successfully"
|
||||
}
|
||||
|
||||
# Function to preview documentation
|
||||
preview_docs() {
|
||||
log_info "Starting documentation preview..."
|
||||
log_info "Starting local preview server..."
|
||||
|
||||
if [[ ! -d "$SITE_OUTPUT_DIR" ]]; then
|
||||
log_error "Build output not found. Run build first."
|
||||
cd "${PROJECT_ROOT}"
|
||||
|
||||
npm run site:preview || {
|
||||
log_error "Failed to start preview server"
|
||||
return 1
|
||||
}
|
||||
|
||||
log_info "Preview server running at https://localhost:8443"
|
||||
log_info "Press Ctrl+C to stop the preview server"
|
||||
}
|
||||
|
||||
# Function to clean generated site folder
|
||||
clean_build() {
|
||||
local site_folder="${PROJECT_ROOT}/generated/site"
|
||||
if [[ -d "$site_folder" ]]; then
|
||||
log_info "Cleaning ${site_folder}..."
|
||||
rm -rf "$site_folder"
|
||||
log_info "✓ Cleaned ${site_folder}"
|
||||
else
|
||||
log_info "Nothing to clean: ${site_folder} does not exist"
|
||||
fi
|
||||
|
||||
log_info "Preview server starting at: http://localhost:8080"
|
||||
log_info "Press Ctrl+C to stop"
|
||||
|
||||
cd "$SITE_OUTPUT_DIR"
|
||||
python3 -m http.server 8080
|
||||
}
|
||||
|
||||
# Show usage
|
||||
@@ -284,204 +188,148 @@ usage() {
|
||||
cat << EOF
|
||||
Usage: $0 [OPTIONS]
|
||||
|
||||
Build and deploy OpenMQTTGateway documentation site using VuePress.
|
||||
|
||||
This script handles the complete documentation build pipeline including:
|
||||
- Installing Node.js and Python dependencies
|
||||
- Downloading shared configuration from theengs.io
|
||||
- Version management (from git tags or custom version)
|
||||
- Generating board documentation from environments
|
||||
- Creating WebUploader manifest for firmware updates
|
||||
- Building VuePress static site
|
||||
- Optional local preview or deployment to GitHub Pages
|
||||
Build and deploy OpenMQTTGateway documentation with VuePress.
|
||||
Generates site structure and meta.json file.
|
||||
|
||||
OPTIONS:
|
||||
--mode MODE
|
||||
Build mode: 'prod' or 'dev' [default: prod]
|
||||
- prod: Production documentation with release version
|
||||
- dev: Development documentation with custom version tag
|
||||
Build mode: 'prod' (production) or 'dev' (development) [default: dev]
|
||||
|
||||
--url-prefix PATH
|
||||
URL prefix for documentation routing (VuePress base path) [default: /]
|
||||
Controls how URLs are generated in the built site.
|
||||
Examples:
|
||||
--url-prefix / # Production site at root URL
|
||||
--url-prefix /dev/ # Development site at /dev/ URL path
|
||||
Note: Should match --deploy-dir for correct link generation
|
||||
Base URL path for links (e.g., '/' for root, '/dev/' for dev) [default: /dev/]
|
||||
|
||||
--custom-version TAG
|
||||
Override version tag displayed in documentation
|
||||
Example: --custom-version "v1.8.0-beta"
|
||||
Note: Automatically sets --version-source to 'custom'
|
||||
|
||||
--version-source SOURCE
|
||||
Source for version information: 'release' or 'custom' [default: release]
|
||||
- release: Use latest git tag from repository
|
||||
- custom: Use version from --custom-version parameter
|
||||
--version VERSION
|
||||
Version string written to meta.json (e.g., '1.8.0', 'edge') [default: edge]
|
||||
|
||||
--preview
|
||||
Start local HTTP server to preview built documentation
|
||||
Server will run at http://localhost:8080
|
||||
Press Ctrl+C to stop the preview server
|
||||
Start local HTTPS preview server after build (https://localhost:8443)
|
||||
|
||||
--deploy-dir DIR
|
||||
Deployment directory on GitHub Pages [default: .]
|
||||
Controls where files are copied in the gh-pages branch.
|
||||
Examples:
|
||||
--deploy-dir . # Deploy to root of gh-pages branch
|
||||
--deploy-dir dev # Deploy to dev/ folder in gh-pages
|
||||
Note: Should match --url-prefix for correct site structure
|
||||
--clean
|
||||
Remove generated/site folder before build
|
||||
|
||||
--no-webuploader
|
||||
Skip WebUploader manifest generation
|
||||
By default, generates manifest for web-based firmware updates
|
||||
--insecure-curl
|
||||
Allow curl to skip TLS verification (use only if cert errors block download)
|
||||
|
||||
--webuploader-args ARGS
|
||||
Additional arguments passed to gen_wu.py script
|
||||
Example: --webuploader-args "--dev"
|
||||
|
||||
--run-pagespeed
|
||||
Run Google PageSpeed Insights after deployment
|
||||
Requires APIKEY to be configured in workflow
|
||||
|
||||
--pagespeed-url URL
|
||||
URL to test with PageSpeed Insights
|
||||
Default: https://docs.openmqttgateway.com/
|
||||
--legacy-openssl
|
||||
Use --openssl-legacy-provider for older Node.js versions (pre-17)
|
||||
|
||||
--help
|
||||
Show this help message and exit
|
||||
Show this message and exit
|
||||
|
||||
EXAMPLES:
|
||||
# Production build
|
||||
$0 --mode prod --url-prefix / --version 1.8.0
|
||||
|
||||
# Build production documentation with latest release version
|
||||
$0 --mode prod
|
||||
# Development build with preview
|
||||
$0 --mode dev --url-prefix /dev/ --version edge --preview
|
||||
|
||||
# Build and preview development documentation locally
|
||||
$0 --mode dev --url-prefix /dev/ --preview
|
||||
|
||||
# Build with custom version tag
|
||||
$0 --custom-version "v1.8.0-beta" --version-source custom
|
||||
|
||||
# Build without WebUploader manifest
|
||||
$0 --mode prod --no-webuploader
|
||||
|
||||
# Build for dev environment with custom WebUploader args
|
||||
$0 --mode dev --url-prefix /dev/ --deploy-dir dev --webuploader-args "--dev"
|
||||
|
||||
WORKFLOW:
|
||||
1. Check requirements (Node.js, npm, Python)
|
||||
2. Install dependencies (npm packages, Python libraries)
|
||||
3. Download common configuration from theengs.io
|
||||
4. Determine version (from git tags or custom)
|
||||
5. Generate board documentation from PlatformIO environments
|
||||
6. Generate WebUploader manifest (optional)
|
||||
7. Build VuePress static site
|
||||
8. Preview (optional)
|
||||
|
||||
OUTPUT:
|
||||
Built documentation will be in: generated/site/
|
||||
# Clean and rebuild
|
||||
$0 --clean --mode prod --version 1.8.0
|
||||
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
local version="edge"
|
||||
local mode="dev"
|
||||
local url_prefix=""
|
||||
local url_prefix_set=false
|
||||
local do_preview=false
|
||||
|
||||
# Parse command line arguments
|
||||
parse_args() {
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--mode)
|
||||
MODE="$2"
|
||||
if [[ $# -lt 2 ]]; then
|
||||
log_error "--mode requires an argument: 'prod' or 'dev'"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
if [[ "$2" != "prod" && "$2" != "dev" ]]; then
|
||||
log_error "Invalid mode: $2. Must be 'prod' or 'dev'"
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
mode="$2"
|
||||
shift 2
|
||||
;;
|
||||
--url-prefix|--base-path)
|
||||
URL_PREFIX="$2"
|
||||
--url-prefix)
|
||||
url_prefix="$2"
|
||||
url_prefix_set=true
|
||||
shift 2
|
||||
;;
|
||||
--deploy-dir|--destination-dir)
|
||||
DEPLOY_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
--custom-version)
|
||||
CUSTOM_VERSION="$2"
|
||||
VERSION_SOURCE="custom"
|
||||
shift 2
|
||||
;;
|
||||
--version-source)
|
||||
VERSION_SOURCE="$2"
|
||||
--version)
|
||||
version="$2"
|
||||
shift 2
|
||||
;;
|
||||
--preview)
|
||||
PREVIEW=true
|
||||
do_preview=true
|
||||
shift
|
||||
;;
|
||||
|
||||
--no-webuploader)
|
||||
GENERATE_WEBUPLOADER=false
|
||||
--clean)
|
||||
CLEAN=true
|
||||
shift
|
||||
;;
|
||||
--webuploader-args)
|
||||
WEBUPLOADER_ARGS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--run-pagespeed)
|
||||
RUN_PAGESPEED=true
|
||||
--insecure-curl)
|
||||
CURL_INSECURE=true
|
||||
shift
|
||||
;;
|
||||
--pagespeed-url)
|
||||
PAGESPEED_URL="$2"
|
||||
shift 2
|
||||
--legacy-openssl)
|
||||
LEGACY_OPENSSL=true
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown option: $1"
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
# Set url_prefix based on mode if not explicitly provided
|
||||
if [[ "$url_prefix_set" == false ]]; then
|
||||
if [[ "$mode" == "prod" ]]; then
|
||||
url_prefix="/"
|
||||
else
|
||||
url_prefix="/dev/"
|
||||
fi
|
||||
fi
|
||||
|
||||
local start_time
|
||||
start_time=$(date +%s)
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
log_info "Starting site build pipeline..."
|
||||
log_info "Mode: $MODE"
|
||||
log_info "URL prefix: $URL_PREFIX"
|
||||
log_info "Deploy directory: $DEPLOY_DIR"
|
||||
log_info "Mode: $mode"
|
||||
log_info "URL prefix: $url_prefix"
|
||||
|
||||
# Clean if requested
|
||||
if [[ "$CLEAN" == "true" ]]; then
|
||||
clean_build
|
||||
fi
|
||||
|
||||
# Check requirements
|
||||
check_requirements || exit 1
|
||||
|
||||
# Install dependencies
|
||||
install_dependencies || exit 1
|
||||
|
||||
# Download common config
|
||||
download_common_config
|
||||
|
||||
# Get and set version
|
||||
local version
|
||||
version=$(get_version)
|
||||
set_version "$version"
|
||||
|
||||
# Set URL prefix
|
||||
set_url_prefix "$URL_PREFIX"
|
||||
|
||||
# Generate board documentation
|
||||
generate_board_docs
|
||||
|
||||
# Generate WebUploader manifest
|
||||
generate_webuploader
|
||||
|
||||
# Create configuration files (meta.json)
|
||||
create_configuration_files "$url_prefix" "$version" "$mode" || exit 1
|
||||
|
||||
# Build documentation
|
||||
build_docs || exit 1
|
||||
|
||||
# Preview if requested
|
||||
if [[ "$PREVIEW" == true ]]; then
|
||||
if [[ "$do_preview" == true ]]; then
|
||||
preview_docs
|
||||
fi
|
||||
|
||||
@@ -501,5 +349,8 @@ main() {
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
}
|
||||
|
||||
# Execute main function
|
||||
main "$@"
|
||||
|
||||
# Run main if executed directly
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
@@ -1,116 +0,0 @@
|
||||
# Common templates and constants for web installer manifest generation
|
||||
# Used by: scripts/gen_wu.py
|
||||
import string
|
||||
|
||||
mf_temp32 = string.Template('''{
|
||||
"name": "OpenMQTTGateway",
|
||||
"new_install_prompt_erase": true,
|
||||
"builds": [
|
||||
{
|
||||
"chipFamily": "ESP32",
|
||||
"improv": false,
|
||||
"parts": [
|
||||
{ "path": "$cp$bl", "offset": 4096 },
|
||||
{ "path": "$cp$part", "offset": 32768 },
|
||||
{ "path": "$cp$boot", "offset": 57344 },
|
||||
{ "path": "$cp$bin", "offset": 65536 }
|
||||
]
|
||||
}
|
||||
]
|
||||
}''')
|
||||
|
||||
mf_temp32c3 = string.Template('''{
|
||||
"name": "OpenMQTTGateway",
|
||||
"new_install_prompt_erase": true,
|
||||
"builds": [
|
||||
{
|
||||
"chipFamily": "ESP32-C3",
|
||||
"improv": false,
|
||||
"parts": [
|
||||
{ "path": "$cp$bl", "offset": 0 },
|
||||
{ "path": "$cp$part", "offset": 32768 },
|
||||
{ "path": "$cp$boot", "offset": 57344 },
|
||||
{ "path": "$cp$bin", "offset": 65536 }
|
||||
]
|
||||
}
|
||||
]
|
||||
}''')
|
||||
|
||||
mf_temp32s3 = string.Template('''{
|
||||
"name": "OpenMQTTGateway",
|
||||
"new_install_prompt_erase": true,
|
||||
"builds": [
|
||||
{
|
||||
"chipFamily": "ESP32-S3",
|
||||
"improv": false,
|
||||
"parts": [
|
||||
{ "path": "$cp$bl", "offset": 0 },
|
||||
{ "path": "$cp$part", "offset": 32768 },
|
||||
{ "path": "$cp$boot", "offset": 57344 },
|
||||
{ "path": "$cp$bin", "offset": 65536 }
|
||||
]
|
||||
}
|
||||
]
|
||||
}''')
|
||||
|
||||
mf_temp8266 = string.Template('''{
|
||||
"name": "OpenMQTTGateway",
|
||||
"new_install_prompt_erase": true,
|
||||
"builds": [
|
||||
{
|
||||
"chipFamily": "ESP8266",
|
||||
"parts": [{ "path": "$cp$bin", "offset": 0 }]
|
||||
}
|
||||
]
|
||||
}''')
|
||||
|
||||
wu_temp_opt = string.Template('''
|
||||
<option
|
||||
value="$mff"
|
||||
>
|
||||
$mfn
|
||||
</option>
|
||||
''')
|
||||
|
||||
wu_temp_p1 = '''<template>
|
||||
<div align="center">
|
||||
<select>
|
||||
<optgroup label="ESP32">'''
|
||||
|
||||
wu_temp_p2 = '''
|
||||
</optgroup>
|
||||
<optgroup label="ESP32C3">'''
|
||||
|
||||
wu_temp_p3 = '''
|
||||
</optgroup>
|
||||
<optgroup label="ESP32S3">'''
|
||||
|
||||
wu_temp_p4 = '''
|
||||
</optgroup>
|
||||
<optgroup label="ESP8266">'''
|
||||
|
||||
wu_temp_end = '''
|
||||
</optgroup>
|
||||
</select><br><br>
|
||||
<esp-web-install-button erase-first></esp-web-install-button>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script>
|
||||
export default {
|
||||
mounted () {
|
||||
const espWebInstallButton = document.querySelector("esp-web-install-button");
|
||||
espWebInstallButton.addEventListener("state-changed", (ev) => { console.log(ev.detail) });
|
||||
const selectFW = document.querySelector("select");
|
||||
espWebInstallButton.manifest = selectFW.value;
|
||||
selectFW.addEventListener("change", () => {
|
||||
espWebInstallButton.manifest = selectFW.value;
|
||||
});
|
||||
}
|
||||
}
|
||||
</script>'''
|
||||
|
||||
manif_path = 'docs/.vuepress/public/firmware_build/'
|
||||
vue_path = 'docs/.vuepress/components/'
|
||||
cors_proxy = '' # 'https://cors.bridged.cc/'
|
||||
esp32_boot = 'https://github.com/espressif/arduino-esp32/raw/2.0.7/tools/partitions/boot_app0.bin'
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
function ensureSSLCerts() {
|
||||
const sslDir = path.join(process.cwd(), '.ssl');
|
||||
const keyFile = path.join(sslDir, 'key.pem');
|
||||
const certFile = path.join(sslDir, 'cert.pem');
|
||||
|
||||
if (fs.existsSync(keyFile) && fs.existsSync(certFile)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const { execSync } = require('child_process');
|
||||
|
||||
fs.mkdirSync(sslDir, { recursive: true });
|
||||
|
||||
try {
|
||||
execSync(`openssl req -new -x509 -keyout "${keyFile}" -out "${certFile}" -days 365 -nodes -subj "/C=US/ST=State/L=City/O=OpenMQTTGateway/CN=localhost" 2>/dev/null`, {
|
||||
stdio: 'inherit'
|
||||
});
|
||||
console.log('✓ SSL certificate generated');
|
||||
} catch (err) {
|
||||
console.error('Failed to generate SSL certificate. Ensure openssl is installed.');
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
ensureSSLCerts();
|
||||
@@ -1,149 +0,0 @@
|
||||
# Creates web installer manifests for ESP Web Tools firmware installation
|
||||
# Used by: .github/workflows/task-docs.yml
|
||||
import os
|
||||
import requests
|
||||
import json
|
||||
import argparse
|
||||
import shutil
|
||||
|
||||
from common_wu import mf_temp32, mf_temp32c3, mf_temp8266, wu_temp_opt, wu_temp_p1, wu_temp_p2, wu_temp_p3, wu_temp_p4, wu_temp_end, vue_path, manif_path, cors_proxy, esp32_boot, mf_temp32s3
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--dev', action='store_true')
|
||||
parser.add_argument('repo', nargs='?', default='1technophile/OpenMQTTGateway')
|
||||
args = parser.parse_args()
|
||||
repo = args.repo
|
||||
dev = args.dev
|
||||
|
||||
bin_path = 'toDeploy/'
|
||||
manif_folder = "/firmware_build/"
|
||||
|
||||
if not os.path.exists(manif_path):
|
||||
os.makedirs(manif_path)
|
||||
|
||||
if dev:
|
||||
print('Generate Web Upload in dev mode')
|
||||
manif_folder = "/dev" + manif_folder
|
||||
# copy OTA latest version definition
|
||||
shutil.copy("scripts/latest_version_dev.json",
|
||||
manif_path + "latest_version_dev.json")
|
||||
# copy the binaries frombin_path to manif_path
|
||||
for name in os.listdir(bin_path):
|
||||
if '.bin' in name:
|
||||
shutil.copyfile(bin_path + name, (manif_path + name))
|
||||
else:
|
||||
print('Generate Web Upload in release mode')
|
||||
# copy OTA latest version definition
|
||||
shutil.copy("scripts/latest_version.json", manif_path + "latest_version.json")
|
||||
release = requests.get('https://api.github.com/repos/' +
|
||||
repo + '/releases/latest')
|
||||
rel_data = json.loads(release.text)
|
||||
if 'assets' in rel_data:
|
||||
assets = rel_data['assets']
|
||||
# Download assets into manif_path
|
||||
for item in range(len(assets)):
|
||||
name = assets[item]['name']
|
||||
if 'firmware.bin' in name:
|
||||
fw_bin = requests.get(assets[item]['browser_download_url'])
|
||||
filename = assets[item]['browser_download_url'].split('/')[-1]
|
||||
with open(manif_path + filename, 'wb') as output_file:
|
||||
output_file.write(fw_bin.content)
|
||||
print('Downloaded: ' + filename)
|
||||
if 'partitions.bin' in name:
|
||||
part_bin = requests.get(assets[item]['browser_download_url'])
|
||||
filename = assets[item]['browser_download_url'].split('/')[-1]
|
||||
with open(manif_path + filename, 'wb') as output_file:
|
||||
output_file.write(part_bin.content)
|
||||
print('Downloaded: ' + filename)
|
||||
if 'bootloader.bin' in name:
|
||||
bl_bin = requests.get(assets[item]['browser_download_url'])
|
||||
filename = assets[item]['browser_download_url'].split('/')[-1]
|
||||
with open(manif_path + filename, 'wb') as output_file:
|
||||
output_file.write(bl_bin.content)
|
||||
print('Downloaded: ' + filename)
|
||||
else:
|
||||
print('Assets not found')
|
||||
os._exit(1)
|
||||
|
||||
if not os.path.exists(vue_path):
|
||||
os.makedirs(vue_path)
|
||||
|
||||
boot_bin = requests.get(esp32_boot)
|
||||
filename = esp32_boot.split('/')[-1]
|
||||
with open(manif_path + filename, 'wb') as output_file:
|
||||
output_file.write(boot_bin.content)
|
||||
|
||||
wu_file = open(vue_path + 'web-uploader.vue', 'w')
|
||||
wu_file.write(wu_temp_p1)
|
||||
|
||||
for name in sorted(os.listdir(manif_path)):
|
||||
if 'firmware.bin' in name and ('esp32c3' not in name ) and ('esp32s3' not in name ) and ('esp32' in name or 'ttgo' in name or 'heltec' in name or 'thingpulse' in name or 'theengs' in name or 'lilygo' in name or 'shelly' in name or 'tinypico' in name):
|
||||
fw = name.split('-firmware')[0]
|
||||
man_file = fw + '.manifest.json'
|
||||
fwp_name = name.split('-firmware')[0] + '-partitions.bin'
|
||||
fwb_name = name.split('-firmware')[0] + '-bootloader.bin'
|
||||
mani_str = mf_temp32.substitute({'cp': cors_proxy, 'part': manif_folder + fwp_name.split('/')[-1], 'bin': manif_folder + name, 'bl': manif_folder + fwb_name, 'boot': manif_folder + esp32_boot.split('/')[-1]})
|
||||
|
||||
with open(manif_path + man_file, 'w') as nf:
|
||||
nf.write(mani_str)
|
||||
|
||||
wu_file.write(wu_temp_opt.substitute(
|
||||
{'mff': manif_folder + man_file, 'mfn': fw}))
|
||||
|
||||
print('Created: ' + man_file)
|
||||
|
||||
wu_file.write(wu_temp_p2)
|
||||
|
||||
for name in sorted(os.listdir(manif_path)):
|
||||
if 'firmware.bin' in name and ('esp32c3' in name ):
|
||||
fw = name.split('-firmware')[0]
|
||||
man_file = fw + '.manifest.json'
|
||||
fwp_name = name.split('-firmware')[0] + '-partitions.bin'
|
||||
fwb_name = name.split('-firmware')[0] + '-bootloader.bin'
|
||||
mani_str = mf_temp32c3.substitute({'cp': cors_proxy, 'part': manif_folder + fwp_name, 'bin': manif_folder + name, 'bl': manif_folder + fwb_name, 'boot': manif_folder + esp32_boot.split('/')[-1]})
|
||||
|
||||
with open(manif_path + man_file, 'w') as nf:
|
||||
nf.write(mani_str)
|
||||
|
||||
wu_file.write(wu_temp_opt.substitute(
|
||||
{'mff': manif_folder + man_file, 'mfn': fw}))
|
||||
|
||||
print('Created: ' + man_file)
|
||||
|
||||
wu_file.write(wu_temp_p3)
|
||||
|
||||
for name in sorted(os.listdir(manif_path)):
|
||||
if 'firmware.bin' in name and ('esp32s3' in name ):
|
||||
fw = name.split('-firmware')[0]
|
||||
man_file = fw + '.manifest.json'
|
||||
fwp_name = name.split('-firmware')[0] + '-partitions.bin'
|
||||
fwb_name = name.split('-firmware')[0] + '-bootloader.bin'
|
||||
mani_str = mf_temp32s3.substitute({'cp': cors_proxy, 'part': manif_folder + fwp_name, 'bin': manif_folder + name, 'bl': manif_folder + fwb_name, 'boot': manif_folder + esp32_boot.split('/')[-1]})
|
||||
|
||||
with open(manif_path + man_file, 'w') as nf:
|
||||
nf.write(mani_str)
|
||||
|
||||
wu_file.write(wu_temp_opt.substitute(
|
||||
{'mff': manif_folder + man_file, 'mfn': fw}))
|
||||
|
||||
print('Created: ' + man_file)
|
||||
|
||||
wu_file.write(wu_temp_p4)
|
||||
|
||||
for name in sorted(os.listdir(manif_path)):
|
||||
if 'firmware.bin' in name and ('nodemcu' in name or 'sonoff' in name or 'rf-wifi-gateway' in name or 'manual-wifi-test' in name or 'rfbridge' in name):
|
||||
fw = name.split('-firmware')[0]
|
||||
man_file = fw + '.manifest.json'
|
||||
mani_str = mf_temp8266.substitute(
|
||||
{'cp': cors_proxy, 'bin': manif_folder + name})
|
||||
|
||||
with open(manif_path + man_file, 'w') as nf:
|
||||
nf.write(mani_str)
|
||||
|
||||
wu_file.write(
|
||||
manif_folder + wu_temp_opt.substitute({'mff': manif_folder + man_file, 'mfn': fw}))
|
||||
|
||||
print('Created: ' + man_file)
|
||||
|
||||
wu_file.write(wu_temp_end)
|
||||
wu_file.close()
|
||||
@@ -1,90 +0,0 @@
|
||||
# Generates board documentation table from platformio.ini environments
|
||||
# Used by: .github/workflows/task-docs.yml
|
||||
import pytablereader as ptr
|
||||
import pandas as pd
|
||||
import os
|
||||
import re
|
||||
import configparser
|
||||
conf = configparser.ConfigParser()
|
||||
|
||||
# Init the table with the columns
|
||||
table_init = pd.DataFrame(columns=['Environment', 'uC', 'Hardware', 'Description', 'Modules', 'Platform',
|
||||
'Partitions', 'Libraries', 'Options'])
|
||||
table = table_init
|
||||
|
||||
# Parse platformio.ini to retrieve boards information
|
||||
conf.read('environments.ini')
|
||||
for each_section in conf.sections():
|
||||
if ("env:" in each_section and "-test" not in each_section):
|
||||
env = each_section.replace("env:", "")
|
||||
uc = ""
|
||||
board = ""
|
||||
hardware = ""
|
||||
description = ""
|
||||
modules = ""
|
||||
platform = ""
|
||||
partitions = ""
|
||||
libraries = ""
|
||||
options = ""
|
||||
for (k, v) in conf.items(each_section):
|
||||
v = v.replace('{', '').replace('}', '').replace('$', '').replace(
|
||||
"env:", '').replace('\'', '').replace("-D", "")
|
||||
if (k == "board"):
|
||||
uc = v
|
||||
if (k == "platform"):
|
||||
platform = v
|
||||
if (k == "board_build.partitions"):
|
||||
partitions = v
|
||||
if (k == "lib_deps"):
|
||||
libraries = v
|
||||
libraries = libraries.replace(
|
||||
"\ncom-esp.lib_deps\n", "").replace(
|
||||
"\ncom-arduino.lib_deps\n", "").replace("libraries.", "")
|
||||
if (k == "build_flags"):
|
||||
options = v
|
||||
for o in options.split('\n'):
|
||||
if ("gateway" in o or "sensor" in o or "actuator" in o):
|
||||
if (modules != ""):
|
||||
modules = modules + "\n"
|
||||
modules = modules + o[1:o.rfind("=\"")]
|
||||
options = options.replace(
|
||||
"com-esp.build_flags\n", "")
|
||||
if (k == "custom_description"):
|
||||
description = v
|
||||
if (k == "custom_hardware"):
|
||||
hardware = v
|
||||
table.loc[len(table.index)] = [env, uc, hardware, description, modules, platform,
|
||||
partitions, libraries, options]
|
||||
|
||||
# Sort rows per Environment name
|
||||
table.sort_values(by=['Environment'], inplace=True,
|
||||
key=lambda col: col.str.lower())
|
||||
|
||||
# Produce individual file
|
||||
for ind in table.index:
|
||||
table_extract = table.iloc[ind]
|
||||
print(table_extract)
|
||||
file = open("docs/prerequisites/boards/" +
|
||||
table.iloc[ind]["Environment"] + ".md", 'w')
|
||||
table_extract = table_extract.rename_axis("Board index")
|
||||
table_md = table_extract.to_markdown()
|
||||
n = file.write(table_md)
|
||||
file.close()
|
||||
|
||||
# Produce list file
|
||||
# Add link to the file from the environment and replace /n with ,
|
||||
for ind in table.index:
|
||||
table['Environment'][ind] = "[" + table['Environment'][ind] + \
|
||||
"](../prerequisites/boards/" + table['Environment'][ind] + ")"
|
||||
|
||||
table = table.replace("\n", ", ", regex=True)
|
||||
table = table.drop(["Partitions", "Hardware", "Platform", "Options","Modules"], axis=1)
|
||||
table = table.reset_index(drop=True)
|
||||
print(table)
|
||||
# Convert to Markdown and save per Model_Id
|
||||
table_md = table.to_markdown()
|
||||
file = open("docs/prerequisites/board.md", 'a')
|
||||
n = file.write("# Supported\n" + table_md)
|
||||
file = open("docs/upload/web-install.md", 'a')
|
||||
n = file.write(table_md)
|
||||
file.close()
|
||||
@@ -1,5 +0,0 @@
|
||||
{
|
||||
"latest_version": "version_tag",
|
||||
"title": "OpenMQTTGateway",
|
||||
"release_url": "https://github.com/1technophile/OpenMQTTGateway/releases/tag/version_tag"
|
||||
}
|
||||
@@ -1,4 +0,0 @@
|
||||
{
|
||||
"latest_version": "version_tag",
|
||||
"title": "OpenMQTTGateway Nightly (TEST ONLY)"
|
||||
}
|
||||
@@ -1,28 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Prepares firmware binaries and libraries for GitHub release deployment
|
||||
# Used by: .github/workflows/release.yml
|
||||
set -e
|
||||
echo "renaming bin files with the environment name"
|
||||
rename -v 's:/:-:g' .pio/build/*/*.bin
|
||||
mkdir toDeploy
|
||||
rename 's/.pio-build-//' .*.bin
|
||||
(
|
||||
cd .pio/libdeps
|
||||
echo "replace space by _ in folder names"
|
||||
find . -type d -name "* *" | while read FNAME; do mv "$FNAME" "${FNAME// /_}"; done
|
||||
echo "zipping libraries per board"
|
||||
for i in */
|
||||
do
|
||||
zip -r "${i%/}-libraries.zip" "$i"
|
||||
done
|
||||
ls -lA
|
||||
mv *.zip ../../toDeploy
|
||||
)
|
||||
# remove binaries for *-all*, *-test* env and only zip containing *-test*
|
||||
rm -f *-all*.bin *-test*.bin *-test*.zip
|
||||
echo "zipping code and licence"
|
||||
zip -r OpenMQTTGateway_sources.zip main LICENSE.txt
|
||||
mv *.zip toDeploy
|
||||
mv *.bin toDeploy
|
||||
|
||||
ls -lA toDeploy
|
||||
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
// Minimal HTTPS static server for generated/site only.
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const https = require('https');
|
||||
const mime = require('mime-types');
|
||||
|
||||
const repoRoot = path.resolve(__dirname, '..');
|
||||
const siteRoot = path.join(repoRoot, 'generated', 'site');
|
||||
const port = parseInt(process.argv[2] || 8443, 10);
|
||||
const keyFile = process.argv[3] || path.join(repoRoot, '.ssl', 'key.pem');
|
||||
const certFile = process.argv[4] || path.join(repoRoot, '.ssl', 'cert.pem');
|
||||
|
||||
if (!fs.existsSync(siteRoot)) {
|
||||
console.error(`Error: folder not found: ${siteRoot}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!fs.existsSync(keyFile) || !fs.existsSync(certFile)) {
|
||||
console.error(`Error: SSL certificates not found at ${keyFile} or ${certFile}`);
|
||||
console.error('Use scripts/ensure_ssl_certs.js to generate them.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const server = https.createServer({
|
||||
key: fs.readFileSync(keyFile),
|
||||
cert: fs.readFileSync(certFile)
|
||||
}, (req, res) => {
|
||||
const reqPath = decodeURIComponent(req.url.split('?')[0]);
|
||||
const safePath = reqPath.endsWith('/') ? `${reqPath}index.html` : reqPath;
|
||||
const filePath = path.join(siteRoot, safePath);
|
||||
|
||||
if (!filePath.startsWith(siteRoot)) {
|
||||
res.writeHead(403);
|
||||
return res.end('Forbidden');
|
||||
}
|
||||
|
||||
fs.stat(filePath, (err, stats) => {
|
||||
if (err || !stats.isFile()) {
|
||||
res.writeHead(404);
|
||||
return res.end('Not Found');
|
||||
}
|
||||
const type = mime.contentType(path.extname(filePath)) || 'application/octet-stream';
|
||||
res.writeHead(200, { 'Content-Type': type });
|
||||
fs.createReadStream(filePath).pipe(res);
|
||||
});
|
||||
});
|
||||
|
||||
server.listen(port, '0.0.0.0', () => {
|
||||
console.log(`Serving generated/site over HTTPS at https://localhost:${port}/`);
|
||||
console.log(`Serving generated/site over HTTPS at https://<host-ip>:${port}/`);
|
||||
console.log(`Note: remember that if you are testing DEV site you should go on https://<host-ip>:${port}/dev/`);
|
||||
});
|
||||
|
||||
process.on('SIGINT', () => {
|
||||
server.close(() => process.exit());
|
||||
});
|
||||
Reference in new issue
Block a user