v31.1-roots.1 · Commit 8: [PATCH 08/17] chore(roots): integrate v31 identity and maintenance tooling
ci/README.md
Before: ci/README.md · After: ci/README.md
File metadata
index 0b6a34460ca1640f927ff527c68d99824e14fd03..bf73cd69b8ae71792c9fb45665148ecc27fbfd31 100644
Download original patch · Release notes
Diff rows 1–176 of 176. Line numbers refer to the original before/after files.
+ Added− RemovedUnmarked lines are context.
Scroll the code area horizontally to read long lines. Before is on the left; after is on the right. Unified view fits narrow screens more easily.
ci/README.md
CHANGED
|
@@ -20,30 +20,11 @@ requires `bash`, `docker`, and `python3` to be installed. To run on different ar
|
|
| 20 |
sudo apt install bash docker.io python3 qemu-user-static
|
| 21 |
```
|
| 22 |
|
| 23 |
-
For some sanitizer builds, the kernel's address-space layout randomization
|
| 24 |
-
(ASLR) entropy can cause sanitizer shadow memory mappings to fail. When running
|
| 25 |
-
the CI locally you may need to reduce that entropy by running:
|
| 26 |
|
| 27 |
```
|
| 28 |
-
sudo sysctl -w vm.mmap_rnd_bits=28
|
| 29 |
-
```
|
| 30 |
-
|
| 31 |
-
To run a test that requires emulating a CPU architecture different from the
|
| 32 |
-
host, we may rely on the container environment recognizing foreign executables
|
| 33 |
-
and automatically running them using `qemu`. The following sets us up to do so
|
| 34 |
-
(also works for `podman`):
|
| 35 |
-
|
| 36 |
-
```
|
| 37 |
-
docker run --rm --privileged docker.io/multiarch/qemu-user-static --reset -p yes
|
| 38 |
-
```
|
| 39 |
-
|
| 40 |
-
It is recommended to run the CI system in a clean environment. The `env -i`
|
| 41 |
-
command below ensures that *only* specified environment variables are propagated
|
| 42 |
-
into the local CI.
|
| 43 |
-
To run the test stage with a specific configuration:
|
| 44 |
-
|
| 45 |
-
```
|
| 46 |
-
env -i HOME="$HOME" PATH="$PATH" USER="$USER" FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh
|
| 47 |
```
|
| 48 |
|
| 49 |
## Configurations
|
|
@@ -62,40 +43,112 @@ It is also possible to force a specific configuration without modifying the
|
|
| 62 |
file. For example,
|
| 63 |
|
| 64 |
```
|
| 65 |
-
env -i HOME="$HOME" PATH="$PATH" USER="$USER" MAKEJOBS="-j1" FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh
|
| 66 |
```
|
| 67 |
|
| 68 |
The files starting with `0n` (`n` greater than 0) are the scripts that are run
|
| 69 |
in order.
|
| 70 |
|
| 71 |
-
## Cache
|
| 72 |
-
|
| 73 |
-
In order to avoid rebuilding all dependencies for each build, the binaries are
|
| 74 |
-
cached and reused when possible. Changes in the dependency-generator will
|
| 75 |
-
trigger cache-invalidation and rebuilds as necessary.
|
| 76 |
-
|
| 77 |
-
## Configuring a repository for CI
|
| 78 |
-
|
| 79 |
-
### Primary repository
|
| 80 |
-
|
| 81 |
-
To configure the primary repository, follow these steps:
|
| 82 |
-
|
| 83 |
-
1. Register with [WarpBuild](https://www.warpbuild.com/) and purchase runners.
|
| 84 |
-
2. Install the WarpBuild GitHub app against the GitHub organization.
|
| 85 |
-
3. Enable organisation-level runners to be used in public repositories:
|
| 86 |
-
1. `Org settings -> Actions -> Runner Groups -> Default -> Allow public repos`
|
| 87 |
-
4. Permit the following actions to run:
|
| 88 |
-
1. actions/cache/restore@\*
|
| 89 |
-
1. actions/cache/save@\*
|
| 90 |
-
1. actions/github-script@\*
|
| 91 |
-
1. docker/setup-buildx-action@\*
|
| 92 |
-
1. warpbuilds/cache/restore@\*
|
| 93 |
-
1. warpbuilds/cache/save@\*
|
| 94 |
-
|
| 95 |
-
### Forked repositories
|
| 96 |
-
|
| 97 |
-
When used in a fork the CI will run on GitHub's free hosted runners by default.
|
| 98 |
-
In this case, GitHub's cache size limitations may cause caches to be frequently evicted and missed, but the workflows will run (slowly).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
|
| 100 |
-
It is also possible to use your own WarpBuild Runners in your own fork with an appropriate patch to the `REPO_USE_WARP_RUNNERS` variable in ../.github/workflows/ci.yml
|
| 101 |
-
NB that WarpBuild Runners only work at an organisation level, therefore in order to use your own WarpBuild Runners, *the fork must be within your own organisation*.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
sudo apt install bash docker.io python3 qemu-user-static
|
| 21 |
```
|
| 22 |
|
| 23 |
+
It is recommended to run the ci system in a clean env. To run the test stage
|
| 24 |
+
with a specific configuration,
|
|
|
|
| 25 |
|
| 26 |
```
|
| 27 |
+
env -i HOME="$HOME" PATH="$PATH" USER="$USER" bash -c 'FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh'
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
```
|
| 29 |
|
| 30 |
## Configurations
|
|
|
|
| 43 |
file. For example,
|
| 44 |
|
| 45 |
```
|
| 46 |
+
env -i HOME="$HOME" PATH="$PATH" USER="$USER" bash -c 'MAKEJOBS="-j1" FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh'
|
| 47 |
```
|
| 48 |
|
| 49 |
The files starting with `0n` (`n` greater than 0) are the scripts that are run
|
| 50 |
in order.
|
| 51 |
|
| 52 |
+
## GitHub Actions policy
|
| 53 |
+
|
| 54 |
+
Bitcoin Roots uses a smaller, risk-selected pull-request matrix rather than
|
| 55 |
+
running every Bitcoin Core CI configuration on every change:
|
| 56 |
+
|
| 57 |
+
| Workflow | Events | Purpose |
|
| 58 |
+
| --- | --- | --- |
|
| 59 |
+
| `.github/workflows/ci.yml` | pull request | Required lint plus path-selected build and test coverage. |
|
| 60 |
+
| `.github/workflows/nightly.yml` | schedule, manual dispatch, reusable call | Expensive sanitizer, fuzz, compatibility, and platform assurance. |
|
| 61 |
+
| `.github/workflows/release.yml` | Roots release tags, manual dispatch | Validate a canonical release tag and produce the five release packages and portable patch. |
|
| 62 |
+
|
| 63 |
+
These repository workflows are kept in every canonical Roots branch and release
|
| 64 |
+
tag so a PR tests the code it proposes. They are hosting automation, however,
|
| 65 |
+
so `ci/release/create-patch-series.sh` excludes `.github/**` from the
|
| 66 |
+
downloadable patch applied on top of Bitcoin Core. Source tests and CI support
|
| 67 |
+
under `test/**`, `src/**/test/**`, and `ci/**` remain in that patch.
|
| 68 |
+
|
| 69 |
+
### Fast PR and development coverage
|
| 70 |
+
|
| 71 |
+
`ci/change-classifier.py` retains additive categories and labels. Complete file
|
| 72 |
+
lists select affected groups; malformed, truncated, empty and unknown input
|
| 73 |
+
selects broad **fast** coverage. Lint/classifier checks always run. The stable
|
| 74 |
+
`required result` check rejects failure, cancellation or skipping of a selected
|
| 75 |
+
job. Verified pure promotions may reuse exact successful canonical CI; explicit
|
| 76 |
+
`ci:*` labels still request fresh coverage.
|
| 77 |
+
|
| 78 |
+
| Change | Automatic fast coverage beyond lint |
|
| 79 |
+
| --- | --- |
|
| 80 |
+
| Documentation only | No compiled job. |
|
| 81 |
+
| Branding or GUI | Actual Qt/resource build and GUI unit tests. |
|
| 82 |
+
| Wallet, general C/C++, critical/build/unknown | Headless ASan/UBSan unit tests, affected Roots functional regressions, bounded libFuzzer smoke. |
|
| 83 |
+
|
| 84 |
+
Fast functional selection and fuzz budgets are implemented by
|
| 85 |
+
`ci/test/roots-regressions.py`, invoked through the existing
|
| 86 |
+
`ci/test_run_all.sh` / `ci/test/03_test_script.sh` entry points. Known wallet,
|
| 87 |
+
network, runtime, policy and mining paths select their acceptance groups;
|
| 88 |
+
mixed paths take the union. Unknown input selects all 16 Roots functional
|
| 89 |
+
regressions and 9 fuzz targets. Fast functional execution has a 900-second
|
| 90 |
+
process-group timeout. Each selected libFuzzer target has a 5-second campaign,
|
| 91 |
+
5-second per-input limit and 15-second outer timeout; input counts are reported
|
| 92 |
+
results, never a campaign quota. Fixed seed 1 makes smoke commands reproducible.
|
| 93 |
+
Fast fuzz does not download or replay the full external corpus.
|
| 94 |
+
|
| 95 |
+
Developer commands using an already configured/built profile:
|
| 96 |
+
|
| 97 |
+
```sh
|
| 98 |
+
python3 ci/test/roots-regressions.py plan --cadence fast \
|
| 99 |
+
--files-json '["src/wallet/spend.cpp"]'
|
| 100 |
+
python3 ci/test/roots-regressions.py functional --cadence fast \
|
| 101 |
+
--files-json '["src/wallet/spend.cpp"]' --build-dir build \
|
| 102 |
+
--artifacts /tmp/roots-fast-functional -- -j2 --quiet
|
| 103 |
+
python3 ci/test/roots-regressions.py fuzz --cadence fast \
|
| 104 |
+
--files-json '["src/policy/policy.cpp"]' --build-dir build_fuzz \
|
| 105 |
+
--artifacts /tmp/roots-fast-fuzz
|
| 106 |
+
python3 -m unittest discover -s ci/test
|
| 107 |
+
```
|
| 108 |
|
| 109 |
+
Fuzz execution requires the native libFuzzer build (`--preset libfuzzer` or
|
| 110 |
+
`00_setup_env_native_fuzz.sh`), not a plain standalone corpus-replay binary.
|
| 111 |
+
Selection/result JSON and per-command logs are temporary failure artifacts,
|
| 112 |
+
not a second source of configuration or a release-state ledger. Timed-out owned
|
| 113 |
+
process groups are terminated; unowned node processes are untouched.
|
| 114 |
+
|
| 115 |
+
### Nightly/manual assurance and caches
|
| 116 |
+
|
| 117 |
+
`nightly.yml` keeps cron `23 2 * * *` (UTC), `workflow_dispatch` and explicit
|
| 118 |
+
label-triggered reusable calls. Ordinary PRs do not run the broad platform
|
| 119 |
+
matrix. Nightly/manual coverage includes full ASan/USDT/Qt, TSan, MSan,
|
| 120 |
+
clang-tidy/dependency boundaries, i686 no-IPC, ARM32, native kernel/chainstate,
|
| 121 |
+
previous-release extended compatibility, extended functional tests, Windows and
|
| 122 |
+
both macOS architecture smokes, and native Linux/macOS/Windows full fuzz corpora.
|
| 123 |
+
Selected empty matrices are skipped before strategy expansion. The schedule
|
| 124 |
+
skips only after the same SHA's preceding completed scheduled run succeeded;
|
| 125 |
+
failed or unavailable/malformed history retries conservatively.
|
| 126 |
+
|
| 127 |
+
| Execution | Explicit budget |
|
| 128 |
+
| --- | --- |
|
| 129 |
+
| PR classifier/result and lint | 10 / 20 minutes. |
|
| 130 |
+
| PR native ASan and fuzz builds plus bounded regressions | 60 minutes each. |
|
| 131 |
+
| PR GUI build/unit checks | 120 minutes. |
|
| 132 |
+
| Nightly ASan | 180 minutes; full applicable native suite. |
|
| 133 |
+
| Nightly TSan/MSan/i686/Linux full corpus | 240 minutes each. |
|
| 134 |
+
| Previous-release extended compatibility | 300 minutes. |
|
| 135 |
+
| Extended functional/tidy/native platform fuzz | 360 minutes. |
|
| 136 |
+
| Nightly selected Roots fuzz campaigns | 60 seconds/target, 75-second outer timeout, after full Linux corpus replay. |
|
| 137 |
+
|
| 138 |
+
Direct `--cadence nightly` functional execution has a 7200-second outer budget;
|
| 139 |
+
full/extended hosted suites use their existing native runners and job budgets.
|
| 140 |
+
Labels `ci:sanitizers`, `ci:fuzz`, `ci:platforms` and `ci:full` explicitly request
|
| 141 |
+
reusable heavy coverage; `ci:compat` requests fresh fast compatibility
|
| 142 |
+
regressions. Manual suite selection can request previous-release/extended
|
| 143 |
+
coverage independently. Failure artifacts retain native test logs, CTest
|
| 144 |
+
output and fuzz crashes/results. Job timeouts are bounds, not measured cold
|
| 145 |
+
build predictions. Local selector/fixture validation does not establish a
|
| 146 |
+
hosted job pass or Windows/macOS/32-bit execution.
|
| 147 |
+
|
| 148 |
+
Compiler/dependency caches remain read-only to PR code; trusted default-branch
|
| 149 |
+
nightly/manual compiler cache saves retain the source behavior. Native v31
|
| 150 |
+
cache-action paths, compiler profiles, no-IPC i686 profile and VS2026 static
|
| 151 |
+
preset replace obsolete v30 names. Windows GUI workflows request the optional
|
| 152 |
+
vcpkg `miniupnpc` feature when enabling `WITH_MINIUPNPC`; fuzz builds explicitly
|
| 153 |
+
disable ZMQ to match their wallet-only dependency manifest. Release signing
|
| 154 |
+
secrets never enter these test containers.
|