Weekly rebuild + Dependabot - #99
Conversation
Add a gated weekly security rebuild, scope Dependabot to security updates for vm/go.mod, and add a smoke test the rebuild must pass before it ships. Most CVEs on this image come from the Go toolchain baked into the service binary rather than a dependency manifest, so they are fixed by rebuilding rather than by bumping anything. The rebuild only republishes when it actually clears a CVE, so a no-op never surfaces as an update in Docker Desktop.
Dependency bumps are the only way to clear CVEs a rebuild cannot, but nothing verified them before a merge. Run make smoke-test on every PR so a bump is tested on its branch rather than after it lands.
424c8cb to
be43c27
Compare
Builds are now always --pull --no-cache, in the Makefile rather than at each call site: a cached base layer reproduces the old image and clears no CVE, and that property matters for every build, not just the weekly rebuild's. IMAGE and TAG were already overridable, so the rebuild's separate buildx call and script invocation collapse into make smoke-test IMAGE=dde-candidate TAG=scan, leaving scripts/smoke-test.sh with a single caller. build-extension already guards the binary download, so that step goes too.
no-cache on build-push-docker.yml put every push to main and every tag through an uncached QEMU arm64 build, on the path you wait on when shipping a security fix. pull: true invalidates the downstream layers whenever a base digest moves, which is what the freshness actually depends on, and the weekly rebuild covers the apk upgrade layer that only no-cache can re-run. Restores the gha cache: without it the runner has nothing to import, so dropping no-cache alone would have left the build cold anyway.
Pushing the bump and the tag first meant a failed publish left main claiming TAG?=x.y.z for an image that was never pushed. The next run reads TAG from the Makefile and scans that tag, so the scan failed and the job died every week after, with nothing watching for CVEs in the meantime. Publishing first fails safe: the working-tree edits go with the runner and main is untouched. The reverse failure -- published, then the push to main rejected because main moved -- leaves an unreferenced image and republishes the same version next week.
| build-extension: ## Build service image to be deployed as a desktop extension | ||
| ls binaries/linux/localstack-* > /dev/null 2>&1 || ./downloadBinaries.sh | ||
| docker build --tag=$(IMAGE):$(TAG) . | ||
| docker build --pull --no-cache --tag=$(IMAGE):$(TAG) . |
There was a problem hiding this comment.
Added --pull to force pull images if there were rebuilt with same tag and --no-cache to force apk upgrade to re-run even when the Alpine digest hasn't moved.
IMAGE was hardcoded in the job env, so testing the publish path meant editing the workflow to point somewhere safe -- easy to get wrong on a file whose whole job is pushing to production. Now a fork can override it at dispatch. The || fallback keeps the scheduled run working: inputs is null on a schedule trigger, so the default only applies to workflow_dispatch.
carole-lavillonniere
changed the title
carole-lavillonniere
changed the title
Pive01
left a comment
Pive01
left a comment
There was a problem hiding this comment.
Overall LGTM great job!
I left though some comments that we might want to address, mainly into categories as:
- Why we use hashes instead of stable releases? Is it a standard security practice? Does it really matter for actions and images though that are not dependency of the final docker image?
- Scripts I imagine do the job but are quite hard to follow, would be great if they could be a bit more human readable (or maybe I'm just bad). But not a blocker
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 |
There was a problem hiding this comment.
Question: why the specific hash?
|
|
||
| - name: Scan published image (before) | ||
| # --ignore-unfixed matches what the secops pipeline files, so an unfixable advisory can't loop here forever. | ||
| uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 |
There was a problem hiding this comment.
Question: why the hash and not a stable version (live v1 or something)
There was a problem hiding this comment.
Using a hash is recommended by GH for security reasons.
Secure actions matter even if they're not in the final artifact because they get access to the job's credentials, they can potentially push any image to our docker repo or any code to our github repo etc.
Pinning an action to a full-length commit SHA is currently the only way to use an action as an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a backdoor to the action's repository
https://docs.github.com/en/actions/reference/security/secure-use#using-third-party-actions
Added the github actions to dependabot though, otherwise it's counterproductive and they end up never being bumped 6eb1c2c
|
|
||
| - name: Build and smoke-test candidate (amd64, for scanning) | ||
| if: steps.before.outputs.proceed == 'true' | ||
| run: make smoke-test IMAGE=dde-candidate TAG=scan |
There was a problem hiding this comment.
Question: where does this magic string dde-candidate comes from?
There was a problem hiding this comment.
Yeah that was confusing. Is it better now with some renaming 8286d0c?
| CURRENT=$(sed -n 's/^TAG?=\(.*\)$/\1/p' Makefile) | ||
| [ -n "$CURRENT" ] || { echo "could not read TAG from Makefile" >&2; exit 1; } | ||
|
|
||
| # Security rebuilds only move the patch component; feature releases set year.month by hand. |
There was a problem hiding this comment.
Question: aside that it's a bit hard to follow what is going on, if the patch is in a new month shouldn't we still change the version in order to follow the calendar versioning? Not sure if is what the script is doing
There was a problem hiding this comment.
Good point! I did not realise it was using calendar versioning. How does 2d63822 look?
There was a problem hiding this comment.
Looks good, thanks for addressing 💪
| if: steps.delta.outputs.cleared == 'true' && !inputs.dry_run | ||
| run: | | ||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" |
There was a problem hiding this comment.
should this maybe be the localstack bot instead?
There was a problem hiding this comment.
Maybe it works already but you might need to set some credentials somewhere (?)
Let me know if you need me to add some secrets
There was a problem hiding this comment.
No new credentials should be needed because this is just metadata, however I changed to use the user's id 88328844+localstack-bot@users.noreply.github.com instead so github can link it better to the actual profile 8227bca
Bumping only the patch meant the date part froze at whatever the last hand-made feature release set, so a security release in December 2026 would still have shipped as 2026.8.x. Take year.month from the current date and bump the patch only within the same month; the first release of a new month restarts at 0. Also drops the CHANGELOG's claim that the project follows semver, which it has not for as long as versions have looked like 2026.8.0.
IMAGE meant the published Docker Hub image in the job env, and then line 83 reused the same name for the throwaway local build, which is what made dde-candidate:scan hard to place. Now PUBLISHED_IMAGE is the thing on Docker Hub and localstack-docker-desktop:candidate is the local build being evaluated, with the scan artefacts named after whichever image they describe.
Matches the identity the rest of the org releases under (openapi, localstack-cli, localstack-sdk-python, localstack-pro all set these four vars). git honours the GIT_AUTHOR_*/GIT_COMMITTER_* environment directly, so the git config calls go. Attribution only: the push still uses GITHUB_TOKEN, which the publish step relies on, since pushes made with it do not trigger workflows. Pushing the tag as the bot with a PAT would fire build-push-docker.yml and publish the tag twice.
b81d2e8 to
9cc0691
Compare
A commit SHA is immutable, which is the point, but it also means nothing tells you a newer release exists. Dependabot updates the pin and the trailing version comment together, so the pins stay current without being unpinned. Unlike the gomod entry this one wants version updates, so no limit of 0; grouped into a single PR so the pins arrive as one review rather than one per action.
9cc0691 to
6eb1c2c
Compare
Pive01
left a comment
Pive01
left a comment
There was a problem hiding this comment.
LGTM! Thanks for your effort 🚀
The plain localstack-bot@users.noreply.github.com form copied from openapi's release.yml can render as an unlinked author: the account (id 88328844) was created in 2021, and GitHub links post-2017 accounts by the ID-prefixed address. localstack-cli's homebrew.yml already commits with that form. Cosmetic — attribution only, no change to how the push authenticates.
2 participants
Closes COSY-926
Goal
This PR is automating a weekly rebuild and introducing dependabot to open PRs bumping go library patches.
The goal is to reduce the amount of manual work needed to remediate CVEs.
Changes
Testing
Tested by forking the repo (since the new workflow does not exist on main it cannot be run here) and performing a dry run there.