test(file-provider): add an end-to-end benchmark harness

[juliusvaart]

Summary

Performance claims about this package have so far been made from figures taken on one person's account, which reproduce nowhere, or from a mocked server, which measures the mock. This adds a harness that measures the package's own code against a real Nextcloud over real HTTP, and that can run the same measurement on an unmodified baseline revision and print the difference.

docker compose -f Benchmarks/docker-compose.yml up -d
Benchmarks/provision.sh
Benchmarks/compare.sh upstream/stable-34.0

compare.sh builds a detached worktree at the baseline ref, copies only this directory into it, builds both revisions in release, runs both against the same server and prints a table per scenario. The baseline stays the baseline: nothing crosses into that worktree except the measuring instrument.

Everything below the File Provider observer is production code — real NextcloudKit, real WebDAV, real XML parsing, the package's own Enumerator and ingestion, and a real Realm database. The observers themselves are stand-ins, because NSFileProviderChangeObserver and NSFileProviderEnumerationObserver are supplied by the framework inside a running extension and cannot be obtained outside one; the harness implements what the framework does, draining moreComing batches until the enumerator reports the last one, and timestamps what arrives. CountingRemoteInterface replaces nothing: it forwards every call to the real interface and records how many requests were issued and how many overlapped.

Four scenarios, reporting durations and counts side by side. A duration alone cannot separate code that got smarter from a server that was warmer; the counts are what a claim can be held to, and they do not move between machines.

working-set-scan one full working-set walk, nothing changed remotely
working-set-scan-change the same walk with one file genuinely changed
item-enumeration-cold first enumeration of a wide directory
item-enumeration-repeat a second enumeration of the same unchanged directory

The reference server is Nextcloud 31 in a container, on SQLite and without Redis: the benchmarks measure the client, and a heavier server stack only adds variance. provision.sh writes the fixture into the data directory and indexes it with occ files:scan, which is far faster than several thousand WebDAV uploads and leaves the same server-side state. NFPK_BENCH_URL, NFPK_BENCH_USER and NFPK_BENCH_PASSWORD point the harness at any other instance.

This commit changes no production code. What the harness cannot see — the framework's own scheduling and retry behaviour — is documented in the doc:PerformanceBenchmarks article, together with the log lines to count for it from an installed build.

Assisted-by: Claude Code:claude-opus-5

TODO

• Confirm the methodology is what you want before the eight follow-up PRs go up
• Pin the reference server image by digest — nextcloud:31-apache is a floating tag, so two runs months apart are not strictly the same server
• Decide whether this can run in CI at all: GitHub macOS runners have no Docker, and the package is macOS-only so a Linux job cannot build it. Local/manual for now
• Verify the harness against server 32 and 33 (only 31 exercised so far)

Checklist

• Sign-off message is added to all commits.
• The commit messages are following The Conventional Commits specification.
• The commit history is clean with no merge commits.
  • How to rebase your commits
  • How to squash commits with rebase
• Uploaded screenshots from before and after for UI changes. n/a — no UI changes
• Test(s) added to branch, with before/after results included in PR description, for performance improvements where applicable.
• Documentation has been updated or is not required.
• Backports requested where applicable (critical bugfixes).
• Labels added where applicable (bug/enhancement).
• Milestone added for target version.

AI (if applicable)

• The content of this PR was partly or fully generated using AI
  • I have read the guidelines for AI-assisted contributions.
  • I have read Nextcloud's AI Contribution Policy.

[i2h3]

Performance claims about this package have so far been made from figures taken on one person's account

👀 What performance claims where?

Besides: I am currently working on a full coverage end-to-end testing for the file provider which includes performance measurements, so this might be redundant.

[juliusvaart]

Hi @i2h3,

The performance stats are in my closed pull request (with too many changes in one): #10728 see below.

I'm using my dev-build from the other pull request with our server (6.2TB in 481.757 files on Hetzner Storage Share) and it's blazing fast.

Performance stats

Metric	Before	After	Conditions
Dataless transitions	30,117 over 630 dirs	0	7-min window, same account
Error lines in log	15,229	2 (unrelated)	7-min window
Notification-rate faults	421 → 863 per second	none	flagged by macOS
Metadata rows written per full scan	1,998 (for 7 real changes)	11	one full scan each
Full scan wall clock	98 s	51 s	~2,000 materialised items
Full scan wall clock	305 s	160 s	2,566 materialised items
Scan for a push-named change	305 s (full walk)	17 s (5 containers)	2,566 materialised items
Discovery → report	4 m 34 s	1.3 s (first batch)	during a full scan
Web-UI file visible in Finder	~5 min	~6 s	end to end

[juliusvaart]

@i2h3 for my other PR's I rely on this PR, i can rework them to use the current setup?

[i2h3]

@juliusvaart Please ensure this pull request is based on master (unless it really is just meant to be integrated into the specific major release branch and not any future releases as well) and opened to be integrated into master.

[i2h3]

I will get back to this topic later.

[juliusvaart]

Now based on master

[juliusvaart]

@i2h3 to add, I'm trying to work towards a version that enumerates everything in the background to get the same performance as iCloud itself has. Usable on the full range of Nextcloud server configurations and with large setups like our own.

My current working version (we're now testing in our company) is not there yet. But as I said, it's really fast and reliable and responsive with push from NC-server. Browsing deep directory structures is very fast.

[juliusvaart]

And you can also build a version from my macos/vf/all branch if you want to try it out.