Skip to content

feat(benchmarking): record sweperf in-sandbox execution time - #1892

Open
Haowei Cai (Roy) (roycaihw) wants to merge 1 commit into
agent-substrate:mainfrom
roycaihw:feat/sweperf-record-inner-duration
Open

Haowei Cai (Roy) (roycaihw) wants to merge 1 commit into
agent-substrate:mainfrom
roycaihw:feat/sweperf-record-inner-duration

Conversation

@roycaihw

Copy link
Copy Markdown

Related: #1694

Workload_Cycle_<n> is timed on the client: POST /execute, then polling
GET /status?job_id= every 200 ms. That time includes routing, request
handling and poll granularity, so it can't be compared with the same trace run
outside Substrate, and it can't separate sandbox overhead from harness overhead.

replay.py already reports execution_duration_ms (gke-labs/sweperf c30c0d6).
This PR parses it and records it under the same name with request type
replay, next to the existing http entry:

Type Name Meaning
http Workload_Cycle_3 client-observed (unchanged)
replay Workload_Cycle_3 time spent running the trace steps inside the sandbox

The stats CSV/JSONL then carry both http_Workload_Cycle_<n> and
replay_Workload_Cycle_<n>. Servers older than c30c0d6 don't send the field,
and nothing extra is recorded for them. Note that locust's "Aggregated" row now
includes the replay entries as well.

pollJobCompletion now returns the final status (it returned only error
before); its caller and tests are updated.

  • Tests pass (go test ./internal/benchmarking/boomer/sweperf/; new cases
    cover a raw replay.py reply, a missing field, and negative, zero and
    fractional values)
  • Appropriate changes to documentation are included in the PR (doc comments)

The sweperf boomer times each cycle as Workload_Cycle_<n> from the client
side: POST /execute, then polling GET /status?job_id= every 200ms until the
job completes. That number includes routing, request handling and poll
granularity, so it cannot be compared with a run of the same trace outside
Substrate, and cannot separate sandbox overhead from harness overhead.

replay.py already reports the time it spent running the trace steps as
execution_duration_ms in the /status reply (gke-labs/sweperf c30c0d6).
Parse it, and record it under the same name with request type "replay",
next to the existing "http" entry, so locust shows both side by side and
the stats CSV/JSONL carry http_Workload_Cycle_<n> and
replay_Workload_Cycle_<n>.

Servers built before c30c0d6 do not send the field; nothing extra is
recorded for them.
@roycaihw

Copy link
Copy Markdown
Author

/cc Nishanth Kotla (@Nishanth29)

@Nishanth29

Copy link
Copy Markdown
Contributor

Thanks Roy! #1848 already parses execution_duration_ms and records it as a single aggregated CycleCEL row, so these two may conflict.

As discussed offline , we're starting with the aggregate and will add the per-cycle split in a follow-up once #1848 lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants