How the “impact” scores are built
Intuition: we rank engineers by a blend of
shipped work (diff + paths + labels),
collaboration (help after PR open + review/help signals),
and delivery completion (closing language + testing collateral + merge CI success when available).
Leaderboard score:
full_stack_score = combined_score + delivery_score + collaboration_score
1) Shipping / “what they landed”
shipping_scoresums per-PR:size_component × path_multiplier × label_multipliersize_component = sqrt(log1p(additions+deletions)) × min(1, log1p(changed_files)/4)(sublinear in churn)path_multiplieruses per-filepath_weight(filename)averaged over touched files.label_multiplierstarts at 1 and boosts if label names contain:bug/fix,security,breaking,priority(orp0/p1).
2) Review leverage
- With
--fetch-reviews, we addreview_scorefrom distinct reviewer–PR pairs. - Base per pair:
0.35. - If the review state is
APPROVED: +0.65. Otherwise, nonstandard states get0.25. - De-duped per (reviewer, PR).
3) Delivery completion (“end-to-end-ish”)
issue closure: PR body/title containsclose/fix/resolve+#NNN.linked_issue_refs: counts those referenced#NNN.tests/e2e collateral: PR touches paths that look like tests (requires--fetch-files).merge CI success: completed check-runs on the merge commit whereconclusion == success(requires--fetch-merge-checks).
4) Collaboration after PR open
- With
--fetch-collaboration, we look at comments after PR created_at on PRs you merged. - Counts of non-author, non-bot issue-thread comments and diff line comments become
collaboration_scorevialog1pand caps. - We also include breadth signals (how many distinct partners), plus small request/assignee signals from the PR payload.
Timeline columns (
tl_*): derived from merge dates only (UTC calendar days). They don’t affect the leaderboard score; they’re intended for scheduling (“streaks”, “gaps”, “how recent”).
Scheduling band:
performance_band is High/Medium/Low from tertiles on full_stack_score within the computed window (not an absolute “quality” measure).
PR / Issue contributions (by everyone involved)
What this view shows: for a selected merged PR (or an issue linked via close/fix/resolve #NNN),
we display a participant breakdown:
PR author shipping proxy, review leverage (if fetched), and collaboration comments after PR open (if fetched).
—
| Login | Total | Ship | Review | Comments | Flags |
|---|
Possible improvements (next iteration)
1) Make it more real-time
- Add a lightweight ingestion layer: GitHub webhooks for PR events (opened, synchronized, review submitted, closed/merged).
- Store raw events + snapshots, then recompute metrics on a schedule (e.g., every hour) or incrementally.
- For dashboards, update a single JSON blob and re-render the static UI (or serve via a small API).
2) Speed up fetching
- Cache per-PR responses (pull details, files, reviews) keyed by PR number + last updated time.
- Use selective fetching: only fetch expensive fields (files/reviews/collaboration) for PRs that are likely to matter (e.g., large diffs, certain paths/labels, close-keyword presence).
- Parallelize requests with rate-limit awareness, or reduce round-trips with GraphQL batching.
- Why we used 30 days / smaller windows: GitHub search returns up to 1000 results per query, and per-PR fetches multiply API calls.
3) Clustering: score-based → rule-based + ramp-up
- Today’s clustering is score-derived (heuristic). With PostHog-specific business context, we can make it more rule-based (e.g., core product areas vs infra, ownership of key workflows).
- On longer windows, estimate “when they joined the project” (e.g., first PR in that area) and compute time-to-first-impact / ramp-up speed.
- Add labels for special roles: on-call ownership, incident follow-ups, release gating, or core review rotations.
Build time (for context): about
1:08:38 of focused work on the assignment itself.
The full GitHub data pull through completion of merged-PR fetching (search slices,
per-PR payloads, optional files/reviews/collaboration/checks) took on the order of ~14 hours wall
clock, including rate limits and checkpoints.