chriz/OpenMQTT-Gateway
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity

[SITE] Renew the web board presentation and the ESP32 web upload + [SYS] Security checks (#2277)

Browse Source 
* Refactor GitHub Actions workflows for build, documentation, and linting

- Consolidated build logic into reusable workflows (`task-build.yml` and `task-docs.yml`) to reduce duplication across multiple workflows.
- Introduced `environments.json` to centralize the list of PlatformIO build environments, improving maintainability and clarity.
- Updated `build.yml` and `build_and_docs_to_dev.yml` to utilize the new reusable workflows and environment definitions.
- Enhanced `release.yml` to streamline the release process and integrate documentation generation.
- Created reusable linting workflow (`task-lint.yml`) to standardize code formatting checks across the repository.
- Simplified manual documentation workflow by leveraging the new reusable documentation workflow.
- Improved artifact management and retention policies across workflows.
- Updated dependencies and versions in workflows to ensure compatibility and performance.

CI/CD pipeline agnostic of Workflow Engine and integrated on github actions

- Implemented ci.sh for orchestrating the complete build pipeline.
- Created ci_00_config.sh for centralized configuration of build scripts.
- Created ci_build_firmware.sh for building firmware for specified PlatformIO environments.
- Created ci_prepare_artifacts.sh for preparing firmware artifacts for upload or deployment.
- Created ci_set_version.sh for updating version tags in firmware configuration files.
- Created ci_build.sh to orchestrate the complete build pipeline.
- Created ci_qa.sh for code linting and formatting checks using clang-format.
- Created ci_site.sh for building and deploying VuePress documentation with version management.
- Implemented checks for required tools and dependencies in the new scripts.
- Improved internal scripts for better error handling and logging.

UPDATE the web installer manifest generation and update documentation structure
- Enhanced ci_list-env.sh to list environments from a JSON file.
- Replaced  common_wu.py and gen_wu.py scripts with new npm scripts for site generation and previewing on docsgen/gen_wu.js
- Replaced  generate_board_docs.py with docsgen/generated_board_docs.js
- Added new npm scripts for integration of site generation on build phase.
- Created preview_site.js to serve locally generated site over HTTPS with improved error handling.
- Added new CI environments for CI builds in environments.json.
- Deleted lint.yml as part of workflow cleanup.
- Enhanced task-build.yml to include linting as a job and added support for specifying PlatformIO version.
- Improved task-docs.yml to handle versioning more effectively and added clean option.

Enhance documentation
- ADD CLEAR Mark of development version of site
- Updated README.md to include detailed workflow dependencies and relationships using mermaid diagrams.
- Improved development.md with a quick checklist for contributors and clarified the code style guide.
- Enhanced quick_start.md with tips for contributors and streamlined the workflow explanation.

LINT FIX
- Refined User_config.h for better formatting consistency.
- Adjusted blufi.cpp and gatewayBT.cpp for improved code readability and consistency in formatting.
- Updated gatewaySERIAL.cpp and mqttDiscovery.cpp to enhance logging error messages.
- Improved sensorDS1820.cpp for better logging of device information.

Add security scan workflows for vulnerability detection

Add SBOM generation and upload to release workflow; update security scan summary handling

Add shellcheck suppor + FIX shellcheck warning

Enhance documentation for CI/CD scripts and workflows, adding details for security scanning and SBOM generation processes

Fix formatting and alignment in BLE connection handling

Reviewed the full web board presentation and the ESP32 web upload. The project uses a modern pattern where data is divided from the presentation layer.

- Removed the `generate_board_docs` script.
- Updated the `gen_wu` script in order to generate `boards-info.json`: the fail that containe all information about the configuration
- Created and isolate the file `boards-info.js` to streamline the parsing of PlatformIO dependencies, modules, environments and improve the handling of library information.
- Introduced vuepress component `BoardEnvironmentTable.vue` that render `boards-info.json` as UI card component
- Introduced vuepress component `FlashEnvironmentSelector.vue` that render a selectred environment from  `boards-info.json` and provide esp-web-upload feature on it
- Introduced a new board page `board-selector.md` for improved firmware selection.
- Updated `web-install.md` to enhance the firmware upload process, including a new board environment table.
- Enhanced custom descriptions in `environments.ini` to include HTML links for better user guidance and board image link

Add CC1101 initialization improvements and logging enhancements
Add installation step for PlatformIO dependencies in documentation workflow

Remove ci_set_version.sh script and associated versioning functionality

* Fix comment provisined

Fix PlatformIO version input reference in documentation workflow

Remove outdated Squeezelite-ESP32 installer documentation
This commit is contained in:
Alessandro Staniscia authored and GitHub committed 2026-03-09 07:47:30 -05:00
1 parent 5f0966c2c1
commit 98481c5145
73 files changed
+5985 -2447
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/ci_site.sh
+161 -310
View File
@@ -1,16 +1,19 @@
#!/bin/bash
# CI/CD Site/Documentation Builder
# Builds and deploys VuePress documentation with version management
# Usage: ./scripts/ci_site.sh [OPTIONS]
# This script builds and deploys VuePress documentation with version management.
# Usage: ./scripts/ci_site.sh [OPTIONS] - Specify options for the script execution.
set -euo pipefail
# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
readonly DOCS_DIR="${PROJECT_ROOT}/docs"
readonly VUEPRESS_CONFIG="${DOCS_DIR}/.vuepress/config.js"
readonly VUEPRESS_BUILD_DIR="${DOCS_DIR}/.vuepress/dist"
# Resolve the folder containing this script so relative paths work
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
DOCS_DIR="${PROJECT_ROOT}/docs"
readonly SCRIPT_DIR
readonly PROJECT_ROOT
readonly DOCS_DIR
# Load shared configuration
if [[ -f "${SCRIPT_DIR}/ci_00_config.sh" ]]; then
@@ -20,29 +23,23 @@ else
exit 1
fi
# Final output directory for site
# Final output directory for the site
readonly SITE_OUTPUT_DIR="${PROJECT_ROOT}/${SITE_DIR}"
# Default values
MODE="prod"
URL_PREFIX="/"
CUSTOM_VERSION=""
VERSION_SOURCE="release"
PREVIEW=false
GENERATE_WEBUPLOADER=true
WEBUPLOADER_ARGS=""
DEPLOY_DIR="."
RUN_PAGESPEED=false
PAGESPEED_URL="https://docs.openmqttgateway.com/"
MODE="prod" # Set the default mode to production
CURL_INSECURE=false # Allow curl to skip TLS verification (use only when needed)
CLEAN=false # Clean generated/site before build
LEGACY_OPENSSL=false # Use --openssl-legacy-provider for older Node.js versions
# Function to check required tools
check_requirements() {
log_info "Checking required tools..."
local missing_tools=()
if ! command -v node >/dev/null 2>&1; then
missing_tools+=("node")
missing_tools+=("node") # Node.js is required
else
local node_version
node_version=$(node --version)
@@ -50,27 +47,19 @@ check_requirements() {
fi
if ! command -v npm >/dev/null 2>&1; then
missing_tools+=("npm")
missing_tools+=("npm") # npm is required
else
local npm_version
npm_version=$(npm --version)
log_info "✓ npm ${npm_version} found"
fi
if ! command -v python3 >/dev/null 2>&1; then
missing_tools+=("python3")
if ! command -v openssl >/dev/null 2>&1; then
missing_tools+=("openssl") # OpenSSL is required
else
local python_version
python_version=$(python3 --version | grep -oP '\d+\.\d+' || echo "unknown")
log_info "✓ Python ${python_version} found"
fi
if ! command -v pip3 >/dev/null 2>&1; then
missing_tools+=("pip3")
else
local pip_version
pip_version=$(pip3 --version | grep -oP '\d+\.\d+' || echo "unknown")
log_info "✓ pip ${pip_version} found"
local openssl_version
openssl_version=$(openssl version | grep -oP '\d+\.\d+\.\d+' || echo "unknown")
log_info "✓ OpenSSL ${openssl_version} found"
fi
if [[ ${#missing_tools[@]} -gt 0 ]]; then
@@ -86,21 +75,6 @@ check_requirements() {
install_dependencies() {
log_info "Installing dependencies..."
# Upgrade pip first (if pip module is available)
if python3 -m pip --version >/dev/null 2>&1; then
log_info "Upgrading pip..."
python3 -m pip install --upgrade pip --quiet || {
log_warn "Failed to upgrade pip, continuing with existing version..."
}
fi
# Install Python dependencies (without --user if in virtualenv)
log_info "Installing Python dependencies..."
pip3 install requests pandas markdown pytablereader tabulate || {
log_error "Failed to install Python dependencies"
return 1
}
# Install Node dependencies
log_info "Installing Node.js dependencies..."
cd "${PROJECT_ROOT}"
@@ -118,118 +92,49 @@ download_common_config() {
local config_url="https://www.theengs.io/commonConfig.js"
local config_dest="${DOCS_DIR}/.vuepress/public/commonConfig.js"
local curl_opts="-sSf"
# Optionally disable TLS verification if explicitly requested
if [[ "${CURL_INSECURE}" == "true" ]]; then
curl_opts+="k"
log_warn "curl is running with --insecure; TLS verification is disabled"
fi
mkdir -p "$(dirname "$config_dest")"
if curl -sSf -o "$config_dest" "$config_url"; then
if curl ${curl_opts} -o "$config_dest" "$config_url"; then
log_info "✓ Common config downloaded"
else
log_warn "Failed to download common config, continuing anyway..."
fi
}
# Function to get version
get_version() {
local version=""
if [[ "$VERSION_SOURCE" == "custom" && -n "$CUSTOM_VERSION" ]]; then
version="$CUSTOM_VERSION"
log_info "Using custom version: $version"
elif [[ "$VERSION_SOURCE" == "release" ]]; then
# Try to get latest git tag (simulating GitHub release)
if command -v git >/dev/null 2>&1 && [[ -d "${PROJECT_ROOT}/.git" ]]; then
version=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
log_info "Using release version from git: $version"
else
version="development"
log_warn "Git not available, using version: $version"
fi
else
# Auto-detect from git
if command -v git >/dev/null 2>&1 && [[ -d "${PROJECT_ROOT}/.git" ]]; then
version=$(git describe --tags --abbrev=0 2>/dev/null || echo "development")
log_info "Using git version: $version"
else
version="development"
log_warn "Git not available, using version: $version"
fi
fi
echo "$version"
}
# Function to set version in config files
set_version() {
local version="$1"
log_info "Setting version: $version"
# Update VuePress config
if [[ -f "$VUEPRESS_CONFIG" ]]; then
sed -i "s|version_tag|${version}|g" "$VUEPRESS_CONFIG"
fi
# Update version JSON file based on version source
if [[ "$VERSION_SOURCE" == "custom" ]]; then
# Custom version updates dev file
local version_file="${SCRIPT_DIR}/latest_version_dev.json"
if [[ -f "$version_file" ]]; then
sed -i "s|version_tag|${version}|g" "$version_file"
fi
else
# Release version updates production file
local version_file="${SCRIPT_DIR}/latest_version.json"
if [[ -f "$version_file" ]]; then
sed -i "s|version_tag|${version}|g" "$version_file"
fi
fi
}
# Function to set URL prefix (base path)
set_url_prefix() {
create_configuration_files() {
local url_prefix="$1"
if [[ "$url_prefix" != "/" ]]; then
log_info "Setting URL prefix: $url_prefix"
sed -i "s|base: '/'|base: '${url_prefix}'|g" "$VUEPRESS_CONFIG"
fi
}
local version="$2"
local mode="$3"
local dest_line=""
# Function to generate board documentation
generate_board_docs() {
log_info "Generating board documentation..."
local generator="${SCRIPT_DIR}/generate_board_docs.py"
if [[ -f "$generator" ]]; then
cd "${PROJECT_ROOT}"
python3 "$generator" || {
log_warn "Board documentation generation failed, continuing..."
}
else
log_warn "Board documentation generator not found, skipping..."
if [[ "$mode" == "dev" ]]; then
dest_line=" \"dest\": \"generated/site/dev\","
fi
}
# Function to generate WebUploader manifest
generate_webuploader() {
if [[ "$GENERATE_WEBUPLOADER" != true ]]; then
log_info "Skipping WebUploader generation"
return 0
fi
log_info "Generating WebUploader manifest..."
local generator="${SCRIPT_DIR}/gen_wu.py"
if [[ -f "$generator" ]]; then
cd "${PROJECT_ROOT}"
python3 "$generator" $WEBUPLOADER_ARGS || {
log_warn "WebUploader generation failed, continuing..."
}
else
log_warn "WebUploader generator not found, skipping..."
fi
# download common config
download_common_config
## Create a meta.json file on config folder
local meta_file="${DOCS_DIR}/.vuepress/meta.json"
cat > "$meta_file" <<EOF
{
"title": "Theengs OpenMQTTGateway",
"version": "${version}",
"url_prefix": "${url_prefix}",
${dest_line}
"mode": "${mode}"
}
EOF
log_info "✓ Generated meta.json at ${meta_file}"
}
# Function to build documentation
@@ -238,45 +143,44 @@ build_docs() {
cd "${PROJECT_ROOT}"
# Set Node options for compatibility with newer Node.js versions
export NODE_OPTIONS="--openssl-legacy-provider"
# Set Node options for compatibility with older Node.js versions (if requested)
if [[ "$LEGACY_OPENSSL" == true ]]; then
log_info "Using --openssl-legacy-provider for older Node.js compatibility"
export NODE_OPTIONS="--openssl-legacy-provider"
fi
npm run docs:build || {
log_error "Documentation build failed"
return 1
}
if [[ -d "$VUEPRESS_BUILD_DIR" ]]; then
log_info "✓ Documentation built successfully"
# Copy to centralized output directory
log_info "Copying site to: $SITE_OUTPUT_DIR"
mkdir -p "$SITE_OUTPUT_DIR"
rm -rf "${SITE_OUTPUT_DIR}"/*
cp -r "${VUEPRESS_BUILD_DIR}"/* "${SITE_OUTPUT_DIR}/"
log_info " VuePress output: $VUEPRESS_BUILD_DIR"
log_info " Final output: $SITE_OUTPUT_DIR"
else
log_error "Build output directory not found: $VUEPRESS_BUILD_DIR"
return 1
fi
log_info "✓ Documentation built successfully"
}
# Function to preview documentation
preview_docs() {
log_info "Starting documentation preview..."
log_info "Starting local preview server..."
if [[ ! -d "$SITE_OUTPUT_DIR" ]]; then
log_error "Build output not found. Run build first."
cd "${PROJECT_ROOT}"
npm run site:preview || {
log_error "Failed to start preview server"
return 1
}
log_info "Preview server running at https://localhost:8443"
log_info "Press Ctrl+C to stop the preview server"
}
# Function to clean generated site folder
clean_build() {
local site_folder="${PROJECT_ROOT}/generated/site"
if [[ -d "$site_folder" ]]; then
log_info "Cleaning ${site_folder}..."
rm -rf "$site_folder"
log_info "✓ Cleaned ${site_folder}"
else
log_info "Nothing to clean: ${site_folder} does not exist"
fi
log_info "Preview server starting at: http://localhost:8080"
log_info "Press Ctrl+C to stop"
cd "$SITE_OUTPUT_DIR"
python3 -m http.server 8080
}
# Show usage
@@ -284,204 +188,148 @@ usage() {
cat << EOF
Usage: $0 [OPTIONS]
Build and deploy OpenMQTTGateway documentation site using VuePress.
This script handles the complete documentation build pipeline including:
- Installing Node.js and Python dependencies
- Downloading shared configuration from theengs.io
- Version management (from git tags or custom version)
- Generating board documentation from environments
- Creating WebUploader manifest for firmware updates
- Building VuePress static site
- Optional local preview or deployment to GitHub Pages
Build and deploy OpenMQTTGateway documentation with VuePress.
Generates site structure and meta.json file.
OPTIONS:
--mode MODE
Build mode: 'prod' or 'dev' [default: prod]
- prod: Production documentation with release version
- dev: Development documentation with custom version tag
Build mode: 'prod' (production) or 'dev' (development) [default: dev]
--url-prefix PATH
URL prefix for documentation routing (VuePress base path) [default: /]
Controls how URLs are generated in the built site.
Examples:
--url-prefix / # Production site at root URL
--url-prefix /dev/ # Development site at /dev/ URL path
Note: Should match --deploy-dir for correct link generation
Base URL path for links (e.g., '/' for root, '/dev/' for dev) [default: /dev/]
--custom-version TAG
Override version tag displayed in documentation
Example: --custom-version "v1.8.0-beta"
Note: Automatically sets --version-source to 'custom'
--version-source SOURCE
Source for version information: 'release' or 'custom' [default: release]
- release: Use latest git tag from repository
- custom: Use version from --custom-version parameter
--version VERSION
Version string written to meta.json (e.g., '1.8.0', 'edge') [default: edge]
--preview
Start local HTTP server to preview built documentation
Server will run at http://localhost:8080
Press Ctrl+C to stop the preview server
Start local HTTPS preview server after build (https://localhost:8443)
--deploy-dir DIR
Deployment directory on GitHub Pages [default: .]
Controls where files are copied in the gh-pages branch.
Examples:
--deploy-dir . # Deploy to root of gh-pages branch
--deploy-dir dev # Deploy to dev/ folder in gh-pages
Note: Should match --url-prefix for correct site structure
--clean
Remove generated/site folder before build
--no-webuploader
Skip WebUploader manifest generation
By default, generates manifest for web-based firmware updates
--insecure-curl
Allow curl to skip TLS verification (use only if cert errors block download)
--webuploader-args ARGS
Additional arguments passed to gen_wu.py script
Example: --webuploader-args "--dev"
--run-pagespeed
Run Google PageSpeed Insights after deployment
Requires APIKEY to be configured in workflow
--pagespeed-url URL
URL to test with PageSpeed Insights
Default: https://docs.openmqttgateway.com/
--legacy-openssl
Use --openssl-legacy-provider for older Node.js versions (pre-17)
--help
Show this help message and exit
Show this message and exit
EXAMPLES:
# Production build
$0 --mode prod --url-prefix / --version 1.8.0
# Build production documentation with latest release version
$0 --mode prod
# Development build with preview
$0 --mode dev --url-prefix /dev/ --version edge --preview
# Build and preview development documentation locally
$0 --mode dev --url-prefix /dev/ --preview
# Build with custom version tag
$0 --custom-version "v1.8.0-beta" --version-source custom
# Build without WebUploader manifest
$0 --mode prod --no-webuploader
# Build for dev environment with custom WebUploader args
$0 --mode dev --url-prefix /dev/ --deploy-dir dev --webuploader-args "--dev"
WORKFLOW:
1. Check requirements (Node.js, npm, Python)
2. Install dependencies (npm packages, Python libraries)
3. Download common configuration from theengs.io
4. Determine version (from git tags or custom)
5. Generate board documentation from PlatformIO environments
6. Generate WebUploader manifest (optional)
7. Build VuePress static site
8. Preview (optional)
OUTPUT:
Built documentation will be in: generated/site/
# Clean and rebuild
$0 --clean --mode prod --version 1.8.0
EOF
exit 0
}
# Main execution
main() {
local version="edge"
local mode="dev"
local url_prefix=""
local url_prefix_set=false
local do_preview=false
# Parse command line arguments
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--mode)
MODE="$2"
if [[ $# -lt 2 ]]; then
log_error "--mode requires an argument: 'prod' or 'dev'"
usage
exit 1
fi
if [[ "$2" != "prod" && "$2" != "dev" ]]; then
log_error "Invalid mode: $2. Must be 'prod' or 'dev'"
usage
exit 1
fi
mode="$2"
shift 2
;;
--url-prefix|--base-path)
URL_PREFIX="$2"
--url-prefix)
url_prefix="$2"
url_prefix_set=true
shift 2
;;
--deploy-dir|--destination-dir)
DEPLOY_DIR="$2"
shift 2
;;
--custom-version)
CUSTOM_VERSION="$2"
VERSION_SOURCE="custom"
shift 2
;;
--version-source)
VERSION_SOURCE="$2"
--version)
version="$2"
shift 2
;;
--preview)
PREVIEW=true
do_preview=true
shift
;;
--no-webuploader)
GENERATE_WEBUPLOADER=false
--clean)
CLEAN=true
shift
;;
--webuploader-args)
WEBUPLOADER_ARGS="$2"
shift 2
;;
--run-pagespeed)
RUN_PAGESPEED=true
--insecure-curl)
CURL_INSECURE=true
shift
;;
--pagespeed-url)
PAGESPEED_URL="$2"
shift 2
--legacy-openssl)
LEGACY_OPENSSL=true
shift
;;
--help)
usage
exit 0
;;
*)
log_error "Unknown option: $1"
usage
exit 0
;;
esac
done
}
# Main execution
main() {
# Set url_prefix based on mode if not explicitly provided
if [[ "$url_prefix_set" == false ]]; then
if [[ "$mode" == "prod" ]]; then
url_prefix="/"
else
url_prefix="/dev/"
fi
fi
local start_time
start_time=$(date +%s)
parse_args "$@"
log_info "Starting site build pipeline..."
log_info "Mode: $MODE"
log_info "URL prefix: $URL_PREFIX"
log_info "Deploy directory: $DEPLOY_DIR"
log_info "Mode: $mode"
log_info "URL prefix: $url_prefix"
# Clean if requested
if [[ "$CLEAN" == "true" ]]; then
clean_build
fi
# Check requirements
check_requirements || exit 1
# Install dependencies
install_dependencies || exit 1
# Download common config
download_common_config
# Get and set version
local version
version=$(get_version)
set_version "$version"
# Set URL prefix
set_url_prefix "$URL_PREFIX"
# Generate board documentation
generate_board_docs
# Generate WebUploader manifest
generate_webuploader
# Create configuration files (meta.json)
create_configuration_files "$url_prefix" "$version" "$mode" || exit 1
# Build documentation
build_docs || exit 1
# Preview if requested
if [[ "$PREVIEW" == true ]]; then
if [[ "$do_preview" == true ]]; then
preview_docs
fi
@@ -501,5 +349,8 @@ main() {
echo "╚════════════════════════════════════════╝"
}
# Execute main function
main "$@"
# Run main if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi