Skip to content

Add a GitHub Actions chapter to the Services and Tools page - #308

Merged
cottage14 merged 2 commits into
apache:masterfrom
potiuk:gha-shared-queues-guidance
Sep 11, 2026
Merged

Add a GitHub Actions chapter to the Services and Tools page#308
cottage14 merged 2 commits into
apache:masterfrom
potiuk:gha-shared-queues-guidance

Conversation

@potiuk

@potiuk potiuk commented Sep 10, 2026

Copy link
Copy Markdown
Member

Why

GitHub Actions runners are a shared pool, but nothing on the site explained
that queue time is a community concern, why the queues pile up, or what a PMC
can actually do about it. The pile-up on 8-9 September 2026 made the gap
obvious, so this documents both the general causes and that episode.

What

services.md — new GitHub Actions chapter (between Build services and
Product naming, with a ToC entry):

  • Why queues build up: many queued workflows (AI-generated contributions likely
    a significant contributor), build misconfigurations, and local spikes from
    cherry-picked or re-run jobs creating noisy-neighbour effects.
  • The volume of security updates we all now see: everything gets scanned, and
    everyone — Infra included — rushes to apply what the scanners find.
  • A worked example of the 8-9 September 2026 queues and the factors behind
    them, linking the tool that came out of the checks
    (utils: add org-wide queued/running Actions job snapshot infrastructure-actions#1259).
  • Five mitigations: optimise PR workflows (paths: filters, dynamic matrices,
    Yetus, Develocity), review workflows for anomalies such as retired macos-13
    labels, group dependency updates, use runs-on: ubuntu-slim for short jobs,
    and explore self-hosted runners.

The GitHub Actions Policy pointer and the Dependabot / dependency-management
section moved into the chapter, so the standalone Dependency management
section is gone (its content is unchanged, just relocated) and the duplicate
Dependabot entry under Build services is dropped.

dependabot.md — new Group security updates section:

A groups block defaults to applies-to: version-updates, so a project that
has grouped its version updates still gets one pull request per security alert,
each with a full CI run behind it. The section shows the
applies-to: security-updates configuration, notes the all-or-nothing
trade-off of grouped PRs, and the full example now gives every ecosystem both a
version group and a security group.

Cross-links into the new chapter from the GitHub Actions Policy page, the
GitHub Actions and Secrets page, build-supported-services.md#gha, and the
documentation index.

Notes for reviewers

  • applies-to with version-updates / security-updates was verified against
    GitHub's live dependabot.yml configuration reference; no repository-level
    setting is needed.
  • ubuntu-slim is documented here for the first time on the site. If there is a
    canonical Infra page or wiki entry for the ASF runner labels, it would be
    worth linking too.
  • Mitigation 5 suggests talking to Infra about self-hosted runners, while
    self-hosted-runners.html says Infra does not recommend them for security
    reasons. The chapter links that page so a reader sees both, but say the word
    if you want the wording aligned.
  • The More information line under build-supported-services.md#gha said the
    notes were on "a Confluence wiki page" while linking github-actions-secrets.html;
    that sentence became a list, so the mislabel is fixed in passing.

🤖 Generated with Claude Code

https://claude.ai/code/session_0126n4c75RhhntsoyCN28BJP

@cottage14 cottage14 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some text notes:

dependabot.md

line 87: change text to
'Every project repository is now scanned by GitHub's Dependabot alerts and by the scanners that downstream users and their employers run against ASF project releases, so we now see many more security updates than we used to. Projects rush to fix what the scanners report as quickly as they can.

That response has a cost: by default, a 'groups' block applies only to version updates, so security updates open one pull request per alert, each with a full CI run behind it. This can add hundreds of builds into the shared GitHub Actions queue.

Line 89: change text to:
A group with applies-to: security-updates set collects all outstanding security fixes for that group into one pull request:

line 113: remove 'worth understanding'

services.md

line 172-182: change text to:
Dependency updates are a growing share of that load. Every project repository is now scanned by GitHub's Dependabot alerts and by the scanners that downstream users and their employers run against ASF project releases, so we now see many more security updates than we used to. Projects rush to fix what the scanners report as quickly as they can.

That response has a cost: by default, a 'groups' block applies only to version updates, so security updates open one pull request per alert, each with a full CI run behind it. This can add hundreds of builds into the shared GitHub Actions queue.

NOTE: the material in lines 174-181 is suitable for a blog post, but not for this page. We assert the problem; we do not have to provide proof of it.

line 185: change text to: "Here are ways you and your PMC can help mitigate this issue:"

NOTE: putting in time-dependent comments "while Infra does this..." increases the maintenance task in keeping the page current

Line 187: In the last sentence, change text to "Contact Infra via a Jira ticket or by email to users@infra for advice on the best approach for your project."

line 188: after 'valid jobs' replace the m-dash with a period and capitalize "The"

line 189: remove commas from the first sentence.

line 190j: replace 'faster' with 'more quickly'

@ppkarwasz ppkarwasz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is probably worth mentioning that PMCs can greatly reduce the churn of Dependabot PRs and the runner time these use by:

  • Pinning actions from GitHub-owned organizations (actions, github) to major version tags (v2, v3, ...) not SHA1. Yes, the teams that write actions and those that administer the GitHub runners are likely distinct, but the likelihood those a breached is not enough, IMHO, to test setup-foo every week on all your repos.
  • Use reusable workflows. The likelihood an action upgrade causes failure in only one of your repos is small. No need to test third-party action updates on all of them. And to tools that complain that workflows on foo-bar don't pin reusable workflows in foo-parent, you can explain that there is no trust boundary between your repos.
  • If apache/infrastructure-asfyaml#122 were to be approved, we would not even need to upgrade our codeql-action every other week. GitHub would do it for us.

Comment thread content/pages/build-supported-services.md
@ppkarwasz

Copy link
Copy Markdown
Member

🤖 Generated with Claude Code

Note for Claude: GitHub doesn't use the same engine to show Markdown files in a repo and issue/pull request descriptions. New lines in the latter are compulsory line breaks, so Claude should not wrap PR descriptions.

Document why the shared GitHub Actions runner queues build up and what a
project can do about it: the 8-9 September 2026 queue pile-up as a worked
example, plus mitigations (PR workflow optimisation, checking for retired
runner labels, grouping dependency updates, ubuntu-slim runners, and
self-hosted runners).

Move the GitHub Actions Policy pointer and the Dependabot / dependency
management section into the new chapter, and cross-link the chapter from
the policy, secrets, build services and documentation index pages.

Also add a "Group security updates" section to the Dependabot page. A
`groups` block defaults to applies-to: version-updates, so security
updates keep opening one pull request per alert -- and one CI run with it
-- unless a group sets applies-to: security-updates.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_0126n4c75RhhntsoyCN28BJP
@potiuk
potiuk force-pushed the gha-shared-queues-guidance branch from 529b1fa to f6e6acb Compare September 10, 2026 17:32
Reviewer edits from @cottage14:

- Rework the dependency-update paragraphs on both `services.md` and
  `dependabot.md` to the reviewer's wording: assert the problem rather
  than argue it, and drop the emphatic register.
- Remove the "Example: the queues of 8-9 September 2026" section. The
  material belongs in a blog post, not on a reference page.
- Drop the "while Infra is working on a multi-tenant experiment" clause
  from the mitigations intro; time-dependent text ages the page.
- Point at a Jira ticket or `users@infra` explicitly instead of "feel
  free to reach out to Infra".
- Assorted line edits: em-dash to a full stop, comma removal, "faster"
  to "more quickly", "trade-off worth understanding" to "trade-off".

From @ppkarwasz's review:

- Add a "Pin first-party actions to a major version tag" mitigation.
  The policy requires a git hash only for external actions; the
  `apache/*`, `github/*` and `actions/*` namespaces are exempt, so
  tag-pinning those removes a large share of Dependabot traffic.
- Add a "Reuse workflows across repositories" mitigation: shared CI in
  reusable workflows means an action bump is tested once rather than
  once per repository.
- Link the INFRA "GitHub Actions Recommended Practices" wiki page from
  `build-supported-services.md`, noting it needs a committer or Member
  login, using the same URL form as 4cb161f.

Also make the reference to the shared queue from within `services.md`
an in-page anchor rather than a link back to the same document.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_01RtBfpombmwwbMDPD1JcUkR
@potiuk
potiuk force-pushed the gha-shared-queues-guidance branch from f6e6acb to ff068b7 Compare September 10, 2026 17:35
@potiuk
potiuk requested a review from cottage14 September 10, 2026 17:35
@potiuk

potiuk commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

@cottage14 All of the line edits are in, and the branch is rebased on top of 4cb161f:

  • dependabot.md lines 87 and 89 and services.md 172-182 now use your wording.
  • The "Example: the queues of 8-9 September 2026" section is gone. Agreed on the reasoning — we assert the problem, we don't have to prove it.
  • The mitigations intro is now "Here are ways you and your PMC can help mitigate this issue:", with the multi-tenant-experiment clause dropped for the maintenance reason you gave.
  • Jira ticket / users@infra for the contact sentence; em-dash → full stop and capitalised "The"; commas removed; "faster" → "more quickly"; "trade-off worth understanding" → "trade-off".

One deviation to flag: the link in your suggested text was services-html#github-actions, so I used services.html#github-actions on dependabot.md, and inside services.md itself it's an in-page anchor.

Your 4cb161f note on the policy page and the wiki link I added to build-supported-services.md now use the same URL, so they're consistent.

@potiuk

potiuk commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

@ppkarwasz Three things:

Pinning actions/* and github/* to major tags — added as a mitigation bullet. I first read this as conflicting with the policy's SHA1 requirement, but the policy is narrower than that: it says you MAY use actions in the apache/*, github/* and actions/* namespaces without restrictions, and the MUST pin-to-a-hash rule applies only to external actions. So what you're describing is already permitted, it just wasn't written anywhere a PMC would come across it. The bullet restates the external-action requirement alongside it so it doesn't read as a general licence to stop SHA-pinning.

Reusable workflows — added, on the argument you make: shared CI in one repository means an action or dependency bump is reviewed and tested once instead of once per repository.

apache/infrastructure-asfyaml#122 — agreed on the effect it would have, but it isn't merged, so I'd rather not forward-reference it. Once it lands, this is the right page for it and I'll add it.

On the hard-wrapped PR description: that's my own preference, not an agent artifact. The description here is the commit message, and commit messages should be hard-wrapped — that's what git expects and what every git tool assumes. When GitHub auto-populates a PR description from a commit body it carries that wrapping through verbatim, so a wrapped PR body is what you get by default from a properly wrapped commit anyway.

I also keep it for accessibility reasons I learned the hard way working with a blind programmer on an 80-character braille display: a hard-wrapped paragraph is predictable there in a way one long viewport-reflowed line is not. Very wide screens have the mirror-image problem — an unwrapped paragraph becomes one enormous line. It's cheap to maintain (:set textwidth=80 and gqip reflows a paragraph in vi), and I think it still earns its keep even now that screens are far wider than 80 columns.

@ppkarwasz

Copy link
Copy Markdown
Member

I also keep it for accessibility reasons I learned the hard way working with a blind programmer on an 80-character braille display: a hard-wrapped paragraph is predictable there in a way one long viewport-reflowed line is not. Very wide screens have the mirror-image problem — an unwrapped paragraph becomes one enormous line. It's cheap to maintain (:set textwidth=80 and gqip reflows a paragraph in vi), and I think it still earns its keep even now that screens are far wider than 80 columns.

Point taken! Should this convention also apply to the *.md files then? In that case line breaks really don't matter, so you can wrap on 80 columns or apply semantic line breaks.

@sebbASF

sebbASF commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Having very long lines in files where line breaks are not significant (e.g. .md, .html) makes it harder for humans to review changes. Maintenance can also be harder.

@ppkarwasz

Copy link
Copy Markdown
Member

This is why I have recently started using semantic line breaks, so a one word change in a paragraph doesn't reflow the whole thing.

@potiuk

potiuk commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

The distinction I make is that line breaks are really good for short documents or snippets which needs to be reviewed commented on understood as a whole. Or code. The long paragraphs and lots of text in longer documents which are supposed to be just right as documents or books are better of course to be using just long lines that can automatically wrap depending on like what's the form of you are reading it.

@potiuk

potiuk commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

Speaking of semantic line breaks I actually do not like it. Especially the markdown variant of it were semantic line breaks are double spaces at the end of the line because you actually can't see in many editors that they are there. I got used to the fact that marked down joins paragraphs which are not separated by empty lines and separate them with empty lines.

For me it just works.

@cottage14 cottage14 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, this looks ready to commit

@cottage14
cottage14 merged commit 26aa25c into apache:master Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants