chriz/OpenMQTT-Gateway
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity

[CI] Refactor GitHub Actions workflows for build, documentation, and linting (#2260)

Browse Source 
* Refactor GitHub Actions workflows for build, documentation, and linting

- Consolidated build logic into reusable workflows (`task-build.yml` and `task-docs.yml`) to reduce duplication across multiple workflows.
- Introduced `environments.json` to centralize the list of PlatformIO build environments, improving maintainability and clarity.
- Updated `build.yml` and `build_and_docs_to_dev.yml` to utilize the new reusable workflows and environment definitions.
- Enhanced `release.yml` to streamline the release process and integrate documentation generation.
- Created reusable linting workflow (`task-lint.yml`) to standardize code formatting checks across the repository.
- Simplified manual documentation workflow by leveraging the new reusable documentation workflow.
- Improved artifact management and retention policies across workflows.
- Updated dependencies and versions in workflows to ensure compatibility and performance.

* CI/CD pipeline agnostic of Workflow Engine and integrated on github actions

- Implemented ci.sh for orchestrating the complete build pipeline.
- Created ci_00_config.sh for centralized configuration of build scripts.
- Created ci_build_firmware.sh for building firmware for specified PlatformIO environments.
- Created ci_prepare_artifacts.sh for preparing firmware artifacts for upload or deployment.
- Created ci_set_version.sh for updating version tags in firmware configuration files.
- Created ci_build.sh to orchestrate the complete build pipeline.
- Created ci_qa.sh for code linting and formatting checks using clang-format.
- Created ci_site.sh for building and deploying VuePress documentation with version management.
- Implemented checks for required tools and dependencies in the new scripts.
- Updated common_wu.py, compressFirmware.py, gen_wu.py, generate_board_docs.py, and prepare_deploy.sh with descriptive comments.

Refactor CI/CD scripts for improved modularity and clarity

- Consolidated build steps in task-build.yml to utilize ci.sh for version tagging, building, and artifact preparation.
- Updated task-lint.yml to use ci.sh for code formatting checks instead of ci_qa.sh.
- Enhanced CI_SCRIPTS.md documentation to reflect changes in script usage, command structure, and output organization.
- Improved internal scripts for better error handling and logging.
- Streamlined the output structure for build artifacts and documentation.
This commit is contained in:
Alessandro Staniscia authored and GitHub committed 2026-02-15 14:58:58 -06:00
1 parent fd433c220e
commit 134c03362c
27 files changed
+4151 -346
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/build_and_docs_to_dev.yml
+43 -161
View File
@@ -1,173 +1,65 @@
name: Build binaries, docs and publish to dev folder
on:
workflow_dispatch:
schedule:
- cron: '0 0 * * *'
jobs:
build:
strategy:
fail-fast: false
matrix:
environments:
- "rfbridge"
- "rfbridge-direct"
- "theengs-bridge"
- "theengs-bridge-v11"
- "theengs-plug"
- "esp32dev-all-test"
- "esp32dev-rf"
- "esp32dev-pilight"
- "esp32dev-pilight-cc1101"
- "esp32dev-somfy-cc1101"
- "esp32dev-pilight-somfy-cc1101"
- "esp32dev-weatherstation"
- "esp32dev-gf-sun-inverter"
- "esp32dev-ir"
- "esp32dev-ble"
- "esp32dev-ble-broker"
- "esp32dev-ble-mqtt-undecoded"
- "esp32dev-ble-aws"
- "esp32feather-ble"
- "esp32-lolin32lite-ble"
- "esp32-olimex-gtw-ble-eth"
- "esp32-olimex-gtw-ble-poe"
- "esp32-olimex-gtw-ble-poe-iso"
- "esp32-wt32-eth01-ble-eth"
- "esp32-olimex-gtw-ble-wifi"
- "esp32-m5stick-ble"
- "esp32-m5stack-ble"
- "esp32-m5tough-ble"
- "esp32-m5stick-c-ble"
- "esp32-m5stick-cp-ble"
- "esp32-m5atom-matrix"
- "esp32-m5atom-lite"
- "esp32doitv1-aithinker-r01-sx1278"
- "esp32dev-rtl_433"
- "esp32dev-rtl_433-fsk"
- "heltec-rtl_433"
- "heltec-rtl_433-fsk"
- "heltec-ble"
- "lilygo-rtl_433"
- "lilygo-rtl_433-fsk"
- "lilygo-t3-s3-rtl_433"
- "lilygo-t3-s3-rtl_433-fsk"
- "lilygo-ble"
- "esp32dev-multi_receiver"
- "esp32dev-multi_receiver-pilight"
- "tinypico-ble"
- "ttgo-lora32-v1"
- "ttgo-lora32-v21"
- "ttgo-t-beam"
- "heltec-wifi-lora-32"
- "shelly-plus1"
- "nodemcuv2-all-test"
- "nodemcuv2-fastled-test"
- "nodemcuv2-2g"
- "nodemcuv2-ir"
- "nodemcuv2-serial"
- "avatto-bakeey-ir"
- "nodemcuv2-rf"
- "nodemcuv2-rf-cc1101"
- "nodemcuv2-somfy-cc1101"
- "manual-wifi-test"
- "rf-wifi-gateway"
- "nodemcuv2-rf2"
- "nodemcuv2-rf2-cc1101"
- "nodemcuv2-pilight"
- "nodemcuv2-weatherstation"
- "sonoff-basic"
- "sonoff-basic-rfr3"
- "esp32dev-ble-datatest"
- "esp32s3-dev-c1-ble"
- "esp32s3-m5stack-stamps3"
- "esp32s3-atomS3U"
- "esp32c3-dev-m1-ble"
- "airm2m_core_esp32c3"
- "esp32c3-dev-c2-ble"
- "esp32c3-dev-c2-ble-no-serial"
- "esp32c3_lolin_mini"
- "esp32c3_lolin_mini_with_serial"
- "esp32c3-m5stamp"
- "esp32c3u-m5stamp"
- "thingpulse-espgateway"
- "esp32dev-ble-idf"
prepare:
runs-on: ubuntu-latest
if: github.repository_owner == '1technophile'
name: Build ${{ matrix.environments }}
outputs:
short-sha: ${{ steps.short-sha.outputs.sha }}
steps:
- uses: actions/checkout@v4
- uses: benjlevesque/short-sha@v2.1
id: short-sha
with:
length: 6
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
version: "latest"
enable-cache: false
- name: Install dependencies
run: |
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
- name: Set sha tag
run: |
sed -i "s/version_tag/${{ steps.short-sha.outputs.sha }}/g" main/User_config.h scripts/latest_version_dev.json
- name: Run PlatformIO
env:
PYTHONIOENCODING: utf-8
PYTHONUTF8: '1'
run: |
export PLATFORMIO_BUILD_FLAGS="'-DDEVELOPMENTOTA=true'"
platformio run -e ${{ matrix.environments }}
- name: Prepare firmware artifacts
run: |
mkdir -p firmware
cp .pio/build/${{ matrix.environments }}/firmware.bin firmware/${{ matrix.environments }}-firmware.bin
if [ -f .pio/build/${{ matrix.environments }}/partitions.bin ]; then
cp .pio/build/${{ matrix.environments }}/partitions.bin firmware/${{ matrix.environments }}-partitions.bin
fi
if [ -f .pio/build/${{ matrix.environments }}/bootloader.bin ]; then
cp .pio/build/${{ matrix.environments }}/bootloader.bin firmware/${{ matrix.environments }}-bootloader.bin
fi
- name: Upload firmware
uses: actions/upload-artifact@v4
with:
name: firmware-${{ matrix.environments }}
path: firmware/
retention-days: 1
build:
needs: prepare
name: Build development firmware
uses: ./.github/workflows/task-build.yml
with:
python-version: '3.13'
enable-dev-ota: true
version-tag: ${{ needs.prepare.outputs.short-sha }}
artifact-retention-days: 1
artifact-name-prefix: 'firmware-'
prepare-for-deploy: true
deploy:
needs: build
needs: [prepare, build]
runs-on: ubuntu-latest
if: github.repository_owner == '1technophile'
name: Deploy binaries and docs
steps:
- uses: actions/checkout@v4
- uses: benjlevesque/short-sha@v2.1
id: short-sha
with:
length: 6
- name: Download all firmware artifacts
uses: actions/download-artifact@v4
with:
pattern: firmware-*
path: toDeploy
merge-multiple: true
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
version: "latest"
enable-cache: false
- name: Install dependencies
run: |
uv pip install --system -U https://github.com/pioarduino/platformio-core/archive/refs/tags/v6.1.18.zip
- name: Create library zips
run: |
# Install libraries for a representative environment to get libdeps
@@ -180,6 +72,7 @@ jobs:
zip -r "${i%/}-libraries.zip" "$i"
done
mv *.zip ../../toDeploy/
- name: Prepare additional assets
run: |
cd toDeploy
@@ -189,33 +82,22 @@ jobs:
# Zip source code
zip -r toDeploy/OpenMQTTGateway_sources.zip main LICENSE.txt
ls -lA toDeploy/
- name: Set sha tag for docs
run: |
sed -i "s/version_tag/DEVELOPMENT SHA:${{ steps.short-sha.outputs.sha }} TEST ONLY/g" docs/.vuepress/config.js
sed -i "s|base: '/'|base: '/dev/'|g" docs/.vuepress/config.js
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: "16.x"
- name: Download Common Config
run: |
curl -o docs/.vuepress/public/commonConfig.js https://www.theengs.io/commonConfig.js
- name: Build documentation
run: |
python ./scripts/gen_wu.py --dev
npm install
npm run docs:build
- name: Deploy to GitHub Pages
uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./docs/.vuepress/dist
destination_dir: dev
cname: docs.openmqttgateway.com
- name: Running Page Speed Insights
uses: jakepartusch/psi-action@v1.3
id: psi
with:
url: "https://docs.openmqttgateway.com/dev/"
threshold: 60
key: ${{ secrets.APIKEY }}
documentation:
needs: [prepare, build]
name: Build and deploy development documentation
uses: ./.github/workflows/task-docs.yml
with:
python-version: '3.11'
node-version: '16.x'
version-source: 'custom'
custom-version: 'DEVELOPMENT SHA:${{ needs.prepare.outputs.short-sha }} TEST ONLY'
base-path: '/dev/'
destination-dir: 'dev'
generate-webuploader: true
webuploader-args: '--dev'
run-pagespeed: true
pagespeed-url: 'https://docs.openmqttgateway.com/dev/'
secrets:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
APIKEY: ${{ secrets.APIKEY }}