OpenWrt Fastbuild Actions


Github

OpenWrt Fastbuild Actions (OFA) is a continuous integration framework for building your own OpenWrt firmware online. It uses Github Actions as the building virtual machine and leverages the advantages of docker for incremental building.

Github Actions (GA) is a continuous integration infrastructure provided by Github. For public repositories, it provides free running duration with maximum of 6 hours for each job, making the building of OpenWrt firmware online possible. According to its Terms of Service, this kind of usage is allowed.

Combined with a docker repository, OFA enables you to build a firmware and store the intermediate building files into the repository, which can be loaded and reused in the next building. Such strategy can largely accelerates the building process. A typically 2 hours of fresh building can be compressed to be within 30 minutes.

Here is what a building looks like

Multiple building profiles

OFA allows coexistence of multiple profiles, which enables managing and building different targets at the same time. Within each profile, you can setup its own source repository, specific tag/branch/commit, enabled packages and patch files. Each profile is built in separate GA runners, with independent triggers. The output files and builders are also stored separatedly.

Flexible pre-building customization

Many aspects of the building process can be customized in OFA’s source tree, including

  • OpenWrt’s source repo and revision (tag/branch/commit).
  • Package feeds and individual package repos you want to be built.
  • Building configuration. (The .config file)
  • Files to be directly copied to the source root.
  • Patches to be applied to the source root.
  • Shell scripts to be ruuned at the source root.
  • The docker repo used for storing building state.

Their structure is organized as

user
├── default                         # Default profile settings
│   └── ...
├── target1                         # First target
│   ├── config.diff                 # From .config
│   ├── custom.sh                   # Scripts to be executed before compiling
│   ├── files                       # Files to be copied to OpenWrt buildroot dir, arranged in same structure
│   │   └── somefile
│   └── settings.ini                # Settings
└── target2
    ├── config.diff
    ├── custom.sh
    ├── packages.txt
    ├── patches                     # Patches to be applied onto OpenWrt buildroot dir
    │   └── 001-somepatch.patch
    └── settings.ini

These customization options cover most building stages, making it easier to construct a robust building pipeline. Here is the structure of profiles I’m using.

user
├── default
│   ├── files
│   │   └── package
│   │       ├── kernel
│   │       │   └── linux
│   │       │       └── modules
│   │       │           └── netconsole.mk
│   │       ├── openwrt-packages
│   │       │   ├── libJudy
│   │       │   │   ├── Makefile
│   │       │   │   └── patches
│   │       │   │       ├── 100-host-compile-JudyTablesGen.patch
│   │       │   │       └── 300-makefile-nodoc-notest.patch
│   │       │   └── netconsole
│   │       │       ├── Makefile
│   │       │       └── files
│   │       │           ├── netconsole.config
│   │       │           └── netconsole.init
│   │       └── system
│   │           └── fstools
│   │               └── 0010-fstools-block-make-extroot-mount-preparation-more-robust.patch
│   └── patches
│       ├── 000-download-max-time.patch
│       ├── 002-netdata-with-dbengine.patch
│       ├── 003-netdata-init-with-TZ.patch
│       └── 006-autossh-init.patch
├── wdr4310v1
│   ├── config.diff
│   ├── custom.sh
│   ├── files
│   │   └── package
│   │       ├── firmware
│   │       │   └── wireless-regdb
│   │       │       └── patches
│   │       │           └── 601-reghack.patch
│   │       ├── kernel
│   │       │   └── mac80211
│   │       │       └── patches
│   │       │           └── ath
│   │       │               └── 499-ath9k_reghack.patch
│   │       └── network
│   │           └── services
│   │               └── hostapd
│   │                   └── patches
│   │                       └── 399-reghack.patch
│   ├── packages.txt
│   └── settings.ini
└── x86_64
    ├── config.diff
    ├── custom.sh
    ├── packages.txt
    ├── patches
    │   ├── 005-openclash-clashr.patch.disable
    │   └── 007-transmission-init.patch
    └── settings.ini

Buildtime customization

Some temporary options are not suitable to be fixed. For example, you may want to temporarily trigger a complete re-building instead of the default incremental task. However, if specified via modifying profile configurations, the option controling this behavior needs to be changed back.

OFA supports building-specific options (modifying configuration files is profile-specific) that are only used once. With them, you can control the behavior of source code updating, feed & package updating, linking & creating of builders, full/package-only building toggling and failure processing. These options can be provided in the payload of triggering events or the commit messages of OFA’s source tree.

Moreover, buildtime configuring, such as executing make menuconfig, can be enabled via the debug or menuconfig option. The shell of builder VM can then be connected before and after compiling. For details, refer to Debugging.

Various triggering methods

A push event will automatically trigger the building of targets whose configurations have been changed within this push. It is also possible to schedule building tasks with cron. To trigger building manually, one of the tricks is to monitor the star button, so that a building can be triggered when you starred your own repo.

There are advanced methods. Github provides APIs for dispatching events to repos, such as the repository dispatch event and the deployment event, which can be the source for triggering Github Actions. I have written a simple userscript running in browsers for calling them. With this method you are also allowed to specify building options in JSON. See tete1030/github-repo-dispatcher for details.

Debugging

OFA provides rich features to ease the process of debugging.

For detecting problems as quickly as possible, it has a parallel pipeline dedicated to running through all steps without actually building anything. Without it, many times you may waste hours of waiting before knowing that something you have changed causes a failure.

When an error happens, OFA automatically detects it and falls back to the nearest debug checkpoint, where you can connect to the VM. You can also specify the debug option to make OFA pauses at each debug checkpoint.

With tete1030/safe-debugger-actions and tmate.io, accessing the builder VM can be achieved via SSH command or in a web page. An encrypted message containing the connection info will be shown in the log pannel. Or the connection info can also be sent to your Slack directly without encryption.


Author: Texot
Reprint policy: All articles in this blog are used except for special statements CC BY 4.0 reprint polocy. If reproduced, please indicate source Texot !
 Current
OpenWrt Fastbuild Actions OpenWrt Fastbuild Actions
Github OpenWrt Fastbuild Actions (OFA) is a continuous integration framework for building your own OpenWrt firmware onli
2020-02-13
Next 
Safe Action Debugger Safe Action Debugger
To make life easier in the development of OpenWrt Fastbuild Actions, I built a module called Safe Action Debugger (SAD)
2020-02-06
  TOC