Skip to content

T4.1: Ranking schema, rubric v1 and ranking runs - #147

Open
MichaelHiddenMB wants to merge 2 commits into
candiedoperation:masterfrom
MichaelHiddenMB:t4-1-ranking-schema
Open

MichaelHiddenMB wants to merge 2 commits into
candiedoperation:masterfrom
MichaelHiddenMB:t4-1-ranking-schema

Conversation

@MichaelHiddenMB

Copy link
Copy Markdown

What this adds

  • horizon_ranking schema (calculation layer): rubric_versions, ranking_runs, project_review_settings, ranked_members, metric_values, member_scores, calibration_notes. Plus the shared horizon_api.publications pointer table. Everything is CREATE ... IF NOT EXISTS, nothing is dropped or renamed, and there are no grants (T7.1 owns those).
  • Rubric v1 at horizon/pipeline/ranking/rubrics/v1.yaml, loaded and validated with pydantic. The loader rejects bucket weights or metric shares that don't sum to 100, unknown units, blank labels and duplicate keys. A version is locked once stored: if the file changes under the same version, the next run fails with "rubric v1 changed; create a new version instead".
  • Two Dagster assets in a new 08_ranking group:
    • ranking_run opens a run for a semester. It stores the rubric version, the role mapping version, the latest complete feature run per project repo (source_runs) and the published projects run. It marks the run failed if any repo has no complete run in the 8 days before as_of.
    • ranked_members snapshots who is ranked in which role. Tech leads are ranked club-wide and engineers per project. A member with both roles is ranked as tech lead. PMs, designers and project leads are skipped, as are archived projects. window_end is capped at as_of.

Rubric changes from the T4.1 card

  • Delivery is verified completion only (100%) for both roles. On-time rate is removed because there are no SOW due dates yet.
  • Engineer consistency is active_week_ratio 60% + longest_inactive_stretch 40% (days, lower is better). end_loading is removed for engineers. Tech lead team_end_loading stays.
  • review_turnaround_hours is the time from a reviewer being assigned to a PR to that reviewer's review.
  • Some teams don't have engineers review PRs. A row in horizon_ranking.project_review_settings with engineers_review_prs = false drops the engineer review_collaboration bucket for that team, and its 15% is split equally over the other five buckets (33/23/18/13/13). Each run records this in ranked_members.excluded_buckets, so old runs keep the weights they used. A project with no row counts as reviews on.
  • Event attendance and weekly report regularity from the weights doc are intentionally left out until there's data for them. Their shares went to the remaining metric in each bucket.

For later Track 4 work: T4.5 should score with RoleRubric.effective_weights(excluded_buckets). T4.2 and T4.3 can skip the dropped review metrics for those engineers.

Depends on T1.1

The assets read horizon_gt_features.projects, project_members and project_repositories using T1.1's column names, but don't create them. Until T1.1 lands, ranking_run fails with a clear "T1.1 tables missing" message and writes nothing. All of the SQL that touches those tables is in pipeline/ranking/store.py.

Questions for Yash

  1. ranked_members.project_name comes from projects.friendly_name. Is that the right column?
  2. display_name falls back to username, then the Authentik pk, when it's blank. OK?
  3. role_title joins a member's distinct role titles with ", ". Should it use one specific subteam's title instead?

Testing

  • ./nx run horizon-dag:check passes (build, ruff, 69 tests). That covers the rubric loader, the weight split, role selection, window capping and source freshness.
  • schema.sql applied twice in a row without errors on a throwaway local Postgres 18, creating 8 tables in horizon_ranking and horizon_api. I haven't repeated this on the Docker local-debug Postgres yet.
  • Before the rubric changes, I smoke-tested both assets against stand-in T1.1 tables: a normal run, a retried run reusing its row, a stale source failing the run, a changed rubric being rejected, and missing T1.1 tables.

Add the horizon_ranking calculation schema (rubric_versions, ranking_runs, ranked_members, metric_values, member_scores, calibration_notes) and the shared horizon_api.publications table to schema.sql. Every statement is idempotent.

Add pipeline/ranking with the v1 rubric YAML and a pydantic loader. The loader rejects bucket weights or metric shares that don't sum to 100, and refuses a changed file under an existing rubric version.

Add the 08_ranking Dagster assets. ranking_run opens a run with source provenance and fails it when a project repository has no complete feature run in the 8 days before as_of. ranked_members snapshots tech leads (club-wide) and engineers (per project) from the T1.1 roster tables.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant