Repository navigation
T4.1: Ranking schema, rubric v1 and ranking runs - #147
Open
MichaelHiddenMB wants to merge 2 commits into
Open
MichaelHiddenMB wants to merge 2 commits into
MichaelHiddenMB wants to merge 2 commits into
Conversation
Add the horizon_ranking calculation schema (rubric_versions, ranking_runs, ranked_members, metric_values, member_scores, calibration_notes) and the shared horizon_api.publications table to schema.sql. Every statement is idempotent. Add pipeline/ranking with the v1 rubric YAML and a pydantic loader. The loader rejects bucket weights or metric shares that don't sum to 100, and refuses a changed file under an existing rubric version. Add the 08_ranking Dagster assets. ranking_run opens a run with source provenance and fails it when a project repository has no complete feature run in the 8 days before as_of. ranked_members snapshots tech leads (club-wide) and engineers (per project) from the T1.1 roster tables.
…y and consistency
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
horizon_rankingschema (calculation layer):rubric_versions,ranking_runs,project_review_settings,ranked_members,metric_values,member_scores,calibration_notes. Plus the sharedhorizon_api.publicationspointer table. Everything isCREATE ... IF NOT EXISTS, nothing is dropped or renamed, and there are no grants (T7.1 owns those).horizon/pipeline/ranking/rubrics/v1.yaml, loaded and validated with pydantic. The loader rejects bucket weights or metric shares that don't sum to 100, unknown units, blank labels and duplicate keys. A version is locked once stored: if the file changes under the same version, the next run fails with "rubric v1 changed; create a new version instead".08_rankinggroup:ranking_runopens a run for a semester. It stores the rubric version, the role mapping version, the latest complete feature run per project repo (source_runs) and the published projects run. It marks the runfailedif any repo has no complete run in the 8 days beforeas_of.ranked_memberssnapshots who is ranked in which role. Tech leads are ranked club-wide and engineers per project. A member with both roles is ranked as tech lead. PMs, designers and project leads are skipped, as are archived projects.window_endis capped atas_of.Rubric changes from the T4.1 card
active_week_ratio60% +longest_inactive_stretch40% (days, lower is better).end_loadingis removed for engineers. Tech leadteam_end_loadingstays.review_turnaround_hoursis the time from a reviewer being assigned to a PR to that reviewer's review.horizon_ranking.project_review_settingswithengineers_review_prs = falsedrops the engineerreview_collaborationbucket for that team, and its 15% is split equally over the other five buckets (33/23/18/13/13). Each run records this inranked_members.excluded_buckets, so old runs keep the weights they used. A project with no row counts as reviews on.For later Track 4 work: T4.5 should score with
RoleRubric.effective_weights(excluded_buckets). T4.2 and T4.3 can skip the dropped review metrics for those engineers.Depends on T1.1
The assets read
horizon_gt_features.projects,project_membersandproject_repositoriesusing T1.1's column names, but don't create them. Until T1.1 lands,ranking_runfails with a clear "T1.1 tables missing" message and writes nothing. All of the SQL that touches those tables is inpipeline/ranking/store.py.Questions for Yash
ranked_members.project_namecomes fromprojects.friendly_name. Is that the right column?display_namefalls back tousername, then the Authentik pk, when it's blank. OK?role_titlejoins a member's distinct role titles with ", ". Should it use one specific subteam's title instead?Testing
./nx run horizon-dag:checkpasses (build, ruff, 69 tests). That covers the rubric loader, the weight split, role selection, window capping and source freshness.schema.sqlapplied twice in a row without errors on a throwaway local Postgres 18, creating 8 tables inhorizon_rankingandhorizon_api. I haven't repeated this on the Docker local-debug Postgres yet.