Investigate performance improvements for Repology

[jamietanna]

Similar to recent work with Maven Central, #44057 notes that Mend-hosted Renovate sees rate limiting from Repology.

We've raised repology/repology-rs#552, but we will have a number of things we could investigate to improve the performance of Renovate's Repology integration

[jamietanna]

Via Claude Opus 5:

I investigated the datasource end-to-end, including probing the live API and running the datasource locally against Repology with the real throttle/cache in place. Here's what I found.

## How it works today

Every lookup does up to two Renovate HTTP requests, sequentially:

1. `GET tools/project-by?...&name_type=binname&...` — a **302** to `/api/v1/project/<project>`, which got follows inside the same request
2. same again with `name_type=srcname` if step 1 didn't yield a match

Result is cached by `(registryUrl, repoName, pkgName)` for 60 min (`repology/index.ts:171-185`). `repology.org` is capped at concurrency 1 + 2000 ms throttle (`lib/util/http/rate-limits.ts:13,47`).

**Measured baseline** (6 realistic distro packages, cold cache): 10 requests, 2 wrong `null` results, 2 lookups downloading the same project payload twice. With throttling enabled, 5 packages = 14.3 s wall clock / 42.8 s queued.

## Findings, ranked by upstream load removed

### A. Negative results are never cached
`_queryPackage` returns `undefined` for not-found, and `withCache` explicitly refuses to store `undefined` (`lib/util/cache/package/with-cache.ts:132`). Verified across two runs with a persistent cache dir: the successful lookup served from cache (0 requests), while `alpine_3_22/does-not-exist-xyz` cost 2 requests again. `debian_13/build-essential` from discussion #44057 is genuinely absent from Repology (`/api/v1/project/build-essential` returns `[]`) — so it burns 2 requests every run, forever. Fix: return `null` instead of `undefined`.

### B. `binnames` is unsupported — the Debian/Ubuntu/Fedora families are broken *and* pay double
`schema.ts` only reads singular `binname`. Repology returns `binnames` (array) instead for **69 of the 149 repos** in the `nginx` project — every `debian_*`, `ubuntu_*`, `fedora_*`, `rocky_*`, `almalinux_*`, `centos_stream_*`, `kali_rolling`, `raspbian_*`, `opensuse_tumbleweed`, `openmandriva_*`, `mageia_*`. `findPackageInResponse` only survives this by the `repoPackages.length === 1` short-circuit; Debian/Ubuntu split `trixie/main` + `trixie-security/main` into 2 entries, so the filter fails.

Measured consequences: `debian_13/nginx-common` and `ubuntu_24_04/libssl-dev` return `null` (they exist), and the failed binname pass falls through to srcname, which re-downloads the *same* project — `fedora_42/gcc` and `ubuntu_24_04/git` both fetched their project twice in one lookup (gcc is 2.3 MB JSON / 176 KB gzip, ×2). Adding `binnames` to the schema and the match roughly halves requests for the biggest distro families and removes an uncacheable-null path.

### C. No project-level cache — the expensive part is never shared
The cache key is `(repo, pkg)`, but the costly artifact is the *project* payload, which is shared across packages and repos. Because got follows the 302 internally, Renovate never sees the project name, so `nginx` + `nginx-full`, or the binname and srcname passes, each pay for it separately. Repology's ToS states **"these redirects do not count towards API RPS limit"** — the resolve step is free, only `/api/v1/` counts.

The fix is to split into two cached steps: `(repo, nameType, pkgName) → projectName` (tiny, long TTL, cacheable negatives) and `projectName → packages` (shared, 60 min). That needs `followRedirect: false` plumbed into `HttpOptions` — arbitrary got options already pass through (only `baseUrl`/`cacheProvider`/`readOnly` are stripped, `http.ts:282`), so it's a type addition plus reading `location`. Pair it with scoping the throttle to `https://repology.org/api/` so the free resolver request isn't 2 s-throttled.

### D. TTL is aligned to the hourly spike the maintainer complained about
Issue #32292 (from the Repology maintainer) shows hourly request peaks attributed to Renovate. 60 min TTL + hourly cron ≈ every run hits upstream. Two levers: raise the soft TTL (distro versions move on the order of days) and/or add TTL jitter — there is no jitter/splay mechanism anywhere in `lib/util` today. `cacheTtlOverride['datasource-repology']` is the self-hosted-only workaround that exists now.

### E. `withCache` has no `fallback: true`
~20 other datasources pass it; Repology — the one most likely to fail or block us (#44057, #38036, #21730) — does not. Enabling it serves stale data within `cacheHardTtlMinutes` (default 7 days) on 429/5xx instead of failing the lookup and re-hitting on retry.

### F. The 403 path rethrows a handled error
In `_queryPackage`'s catch, when `tools/project-by` is unsupported (403) and the direct-API fallback finds nothing, control falls to `throw err` — a 403 surfaces as a lookup failure rather than "not found", and nothing is cached, so it repeats every run. The "this repo doesn't support the resolver" fact also isn't cached per repo, so every package in such a repo pays 403 + API.

### G. Retries multiply a bad situation
got retry limit is 2 outside tests, so a blocked/timing-out Repology (exactly the #44057 symptom) triples our request count against a host that's already refusing us.

## Things that look like levers but aren't

- **HTTP cache provider / etag revalidation**: Repology sends `cache-control: max-age=3600` but no `etag` or `last-modified`, so conditional requests are impossible — no gain over `withCache`.
- **Compression**: already in place; `nginx` is 379 KB raw → 21 KB gzip on the wire.
- **Server-side narrowing**: `?inrepo=` on `/api/v1/project/<name>` is ignored (still 379,156 bytes), and `/api/v1/projects/?inrepo=&search=` is *bigger* (572 KB) and still returns every repo's packages.
- **Not caching, but the actual cause of the block**: the ToS requires bulk clients to send a UA with a link to their source repo *and an accessible issue tracker*. Renovate's default (`Renovate/{{renovateVersion}} (https://github.com/renovatebot/renovate)`, `lib/config/options/index.ts:117`) complies; the Mend-hosted override `Mend-hosted Renovate/x (https://**redacted**@mend.io)` does not — which lines up with #44057. Worth noting that renovatebot/renovate restricts issue creation to maintainers, which may not satisfy "accessible issue tracker".

[jamietanna]

Related: #27802