ci: disable kaniko registry cache push #59

Merged

gofix merged 1 commit from fix/disable-kaniko-cache-push into master

2026-05-03 17:24:00 +00:00

gofix commented

2026-05-03 17:22:17 +00:00

Owner

Why

The v2.1.8+rs.1 retry build hung for 75+ minutes on Pushing image to code.rly.best/gofix/portal-tunnel-rs-cache-arm64:<hash>. The kaniko process consumed only 0:49 of CPU during that window, had a single open socket, and made no network progress. Forgejo's task log gets wiped on cancel so we cannot tell whether the registry rejected the layer or kaniko's HTTP client deadlocked. Either way the cache push step is the bottleneck.

Builds were not consistently hitting the cache between runs anyway — the only successes were the 5m17s cold-cache run and a 1m11s warm-cache run, both well within what a no-cache build can do. Removing --cache=true lets the build complete in the time the actual work needs (cargo build inside QEMU emulation), without the cache-write deadlock.

This unblocks the v2.1.8+rs.1 release. Reintroducing a working cache (different builder, or different cache backend) is follow-up.

Validation

cargo fmt --check, cargo test --locked, cargo clippy --locked --all-targets -- -D warnings all clean
workflow YAML parses

Follow-up

After merge: re-cut v2.1.8+rs.1 (currently in draft) — same SHA, just retry the tag push so CI runs without the cache flag.

## Why The v2.1.8+rs.1 retry build hung for 75+ minutes on `Pushing image to code.rly.best/gofix/portal-tunnel-rs-cache-arm64:<hash>`. The kaniko process consumed only 0:49 of CPU during that window, had a single open socket, and made no network progress. Forgejo's task log gets wiped on cancel so we cannot tell whether the registry rejected the layer or kaniko's HTTP client deadlocked. Either way the cache push step is the bottleneck. Builds were not consistently hitting the cache between runs anyway — the only successes were the 5m17s cold-cache run and a 1m11s warm-cache run, both well within what a no-cache build can do. Removing `--cache=true` lets the build complete in the time the actual work needs (cargo build inside QEMU emulation), without the cache-write deadlock. This unblocks the v2.1.8+rs.1 release. Reintroducing a working cache (different builder, or different cache backend) is follow-up. ## Validation - `cargo fmt --check`, `cargo test --locked`, `cargo clippy --locked --all-targets -- -D warnings` all clean - workflow YAML parses ## Follow-up After merge: re-cut `v2.1.8+rs.1` (currently in draft) — same SHA, just retry the tag push so CI runs without the cache flag.

gofix added 1 commit

2026-05-03 17:22:17 +00:00

ci: disable kaniko registry cache push

Rust CI / Format, lint, and test (pull_request) Successful in 1m17s

Details

d06ab93e6f

Kaniko's `--cache=true --cache-repo=...` writes every intermediate
layer (including the multi-GB cargo target dir from RUN cache mounts
that Kaniko does not recognise as ephemeral) to a separate cache repo
on each build. The push to that repo started hanging indefinitely on
the v2.1.8+rs.1 retry — kaniko sat on `Pushing image to ...
portal-tunnel-rs-cache-arm64:<hash>` for 75+ minutes with zero
progress, no network activity, no CPU. Forgejo's task log gets wiped
on cancel, so we still cannot tell whether the registry is rejecting
the layer or kaniko's HTTP client is deadlocked.

Drop the cache flags. We lose intermediate layer reuse across builds,
but builds were not consistently hitting the cache anyway and the
cache-push step was the slowest part of the pipeline. Builds become
~5-10 min for the QEMU-emulated arm64 path with no cache, vs >75 min
hung. Reintroducing a working cache mechanism (e.g. with a different
builder, or against a cache backend that handles the layer sizes) is
follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

gofix merged commit f14226019b into master

2026-05-03 17:24:00 +00:00

gofix referenced this pull request from a commit

2026-05-03 17:24:01 +00:00

ci: disable kaniko registry cache push (#59)

gofix referenced this pull request

2026-05-03 17:39:10 +00:00

fix: switch final stage to debian-slim so file caps survive Kaniko #60