Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7df019e
feat(db): add contributor aggregation tables
Notoriousbrain Aug 11, 2025
78bebcb
feat(cron): add a authorized cron function
Notoriousbrain Aug 11, 2025
3e403d1
feat(providers/github): add GraphQL contributions client (Commit 3/n …
Notoriousbrain Aug 11, 2025
1e2b352
feat(providers/gitlab): add Events client with timeout, Retry-After, …
Notoriousbrain Aug 12, 2025
f165138
perf(leaderboard): optimize aggregator — single totals recompute + co…
Notoriousbrain Aug 12, 2025
87598d2
fix(leaderboard/redis): robust ZRANGE parsing + add lb-sync-one scrip…
Notoriousbrain Aug 12, 2025
7391d12
feat(leaderboard): optimized cron-protected backfill & refresh routes…
Notoriousbrain Aug 13, 2025
17763f0
feat(leaderboard): public read API with Redis Top-N and DB fallback (…
Notoriousbrain Aug 18, 2025
8fbde31
fix(leaderboard/aggregator): race-free concurrency + single-pass day …
Notoriousbrain Aug 18, 2025
cd497b7
fix(leaderboard): make Redis read resilient; graceful DB fallback on …
Notoriousbrain Aug 18, 2025
f30a9cc
feat(leaderboard): add Redis-backed profiles + API and hydrate UI
Notoriousbrain Aug 18, 2025
e0474cc
fix(export): guard user map writes to satisfy TS and avoid undefined …
Notoriousbrain Aug 18, 2025
8102979
feat(cron): add the cron job for fetching contributions
Notoriousbrain Aug 18, 2025
6f0a77f
feat(leaderboard): fetch the contributions and update them by cron
Notoriousbrain Aug 29, 2025
466e3e2
feat(leaderboard): add the final leaderboard
Notoriousbrain Aug 30, 2025
b301527
merge with latest code
Notoriousbrain Aug 30, 2025
1b4c2da
fix(bugs): change according to coderabbit
Notoriousbrain Aug 31, 2025
d9a9c57
some other change
Notoriousbrain Aug 31, 2025
a359c9a
merge conflict except the snapshot.json
Notoriousbrain Sep 1, 2025
ba5e908
solve the migration file issue
Notoriousbrain Sep 1, 2025
18742b4
replace all unnecassary migration file with one
Notoriousbrain Sep 1, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions apps/web/app/(public)/leaderboard/page.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
import LeaderboardClient from '@/components/leaderboard/leaderboard-client';

export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';

type SearchParams = Promise<Record<string, string | string[] | undefined>>;

function normalizeWindow(v: unknown): '30d' | '365d' {
if (v === '30d') return '30d';
if (v === '365d') return '365d';
return '365d';
}

export default async function LeaderboardPage({ searchParams }: { searchParams?: SearchParams }) {
const sp = await searchParams;
const winParam = sp && 'window' in sp
? Array.isArray(sp.window) ? sp.window[0] : sp.window
: undefined;
const initialWindow = normalizeWindow(winParam);

return (
<div className="mx-auto max-w-6xl px-4 py-10">
<div className="mt-12 mb-6">
<h1 className="text-3xl font-bold tracking-tight">Global Leaderboard</h1>
<p className="text-muted-foreground">
Top contributors across GitHub and GitLab based on open source contributions.
</p>
</div>
<LeaderboardClient initialWindow={initialWindow} />
</div>
);
}
150 changes: 150 additions & 0 deletions apps/web/app/api/internal/leaderboard/backfill/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';

import { NextRequest } from 'next/server';
import { z } from 'zod/v4';
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix Zod import path.

zod/v4 is not a published entry; import from 'zod'. This will otherwise fail to build.

-import { z } from 'zod/v4';
+import { z } from 'zod';
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
import { z } from 'zod/v4';
import { z } from 'zod';
🤖 Prompt for AI Agents
In apps/web/app/api/internal/leaderboard/backfill/route.ts around line 5, the
import uses the unpublished path 'zod/v4' which breaks builds; update the import
to use the published package by replacing the module path 'zod/v4' with 'zod' so
the file imports z from the correct package.


import { isCronAuthorized } from '@workspace/env/verify-cron';
import { env } from '@workspace/env/server';

import { backfillLockKey, withLock, acquireLock, releaseLock } from '@workspace/api/redis/locks';
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Incorrect import path: file is redis/lock, not redis/locks.

This will break resolution.

-import { backfillLockKey, withLock, acquireLock, releaseLock } from '@workspace/api/redis/locks';
+import { backfillLockKey, withLock, acquireLock, releaseLock } from '@workspace/api/redis/lock';
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
import { backfillLockKey, withLock, acquireLock, releaseLock } from '@workspace/api/redis/locks';
// apps/web/app/api/internal/leaderboard/backfill/route.ts
import { backfillLockKey, withLock, acquireLock, releaseLock } from '@workspace/api/redis/lock';
🤖 Prompt for AI Agents
In apps/web/app/api/internal/leaderboard/backfill/route.ts around line 10 the
import path is incorrect: it imports from '@workspace/api/redis/locks' but the
module lives at '@workspace/api/redis/lock'; update the import statement to use
the correct path ('@workspace/api/redis/lock') so the named exports
backfillLockKey, withLock, acquireLock, releaseLock resolve correctly and the
module can be loaded.

import { setUserMetaFromProviders } from '@workspace/api/leaderboard/use-meta';
import { refreshUserRollups } from '@workspace/api/leaderboard/aggregator';
import { syncUserLeaderboards } from '@workspace/api/leaderboard/redis';
import { db } from '@workspace/db';

function ymd(d = new Date()) {
const y = d.getUTCFullYear();
const m = String(d.getUTCMonth() + 1).padStart(2, '0');
const dd = String(d.getUTCDate()).padStart(2, '0');
return `${y}-${m}-${dd}`;
}

const Body = z
.object({
userId: z
.string()
.min(1)
.transform((s) => s.trim()),
githubLogin: z
.string()
.min(1)
.transform((s) => s.trim())
.optional(),
gitlabUsername: z
.string()
.min(1)
.transform((s) => s.trim())
.optional(),

days: z.number().int().min(1).max(365).optional(),
concurrency: z.number().int().min(1).max(8).optional(),
})
.refine((b) => !!b.githubLogin || !!b.gitlabUsername, {
message: 'At least one of githubLogin or gitlabUsername is required.',
});

export async function POST(req: NextRequest) {
const auth = req.headers.get('authorization');
if (!isCronAuthorized(auth)) return new Response('Unauthorized', { status: 401 });

const json = await req.json().catch(() => ({}));
const parsed = Body.safeParse(json);
if (!parsed.success) {
return new Response(`Bad Request: ${parsed.error.message}`, { status: 400 });
}
const body = parsed.data;

const providers = (
[
...(body.githubLogin ? (['github'] as const) : []),
...(body.gitlabUsername ? (['gitlab'] as const) : []),
] as Array<'github' | 'gitlab'>
).sort();

const githubToken = env.GITHUB_TOKEN;
const gitlabToken = env.GITLAB_TOKEN;
const gitlabBaseUrl = 'https://gitlab.com';

if (providers.includes('github') && !githubToken) {
return new Response('Bad Request: github requested but GITHUB_TOKEN not set', { status: 400 });
}
if (providers.includes('gitlab') && !gitlabToken) {
return new Response('Bad Request: gitlab requested but GITLAB_TOKEN not set', { status: 400 });
}

const ttlSec = 5 * 60;
const todayStr = ymd(new Date());

async function runOnce() {
const wrote = await refreshUserRollups(
{ db },
{
userId: body.userId,
githubLogin: body.githubLogin,
gitlabUsername: body.gitlabUsername,
githubToken,
gitlabToken,
gitlabBaseUrl,
},
);

await syncUserLeaderboards(db, body.userId);

await setUserMetaFromProviders(body.userId, body.githubLogin, body.gitlabUsername);

return wrote;
}
Comment on lines +79 to +97
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Two-provider flow only runs the aggregation once.

When both providers are present, runOnce is invoked a single time, so one provider never backfills. Run each provider sequentially under the two-key lock.

-  async function runOnce() {
-    const wrote = await refreshUserRollups(
+  async function runOnce(p: 'github' | 'gitlab') {
+    const wrote = await refreshUserRollups(
       { db },
       {
         userId: body.userId,
-        githubLogin: body.githubLogin,
-        gitlabUsername: body.gitlabUsername,
-        githubToken,
-        gitlabToken,
+        githubLogin: p === 'github' ? body.githubLogin : undefined,
+        gitlabUsername: p === 'gitlab' ? body.gitlabUsername : undefined,
+        githubToken: p === 'github' ? githubToken : undefined,
+        gitlabToken: p === 'gitlab' ? gitlabToken : undefined,
         gitlabBaseUrl,
       },
     );
 
-    await syncUserLeaderboards(db, body.userId);
-
-    await setUserMetaFromProviders(body.userId, body.githubLogin, body.gitlabUsername);
+    await syncUserLeaderboards(db, body.userId);
+    await setUserMetaFromProviders(body.userId, body.githubLogin, body.gitlabUsername);
 
     return wrote;
   }
@@
-      return await withLock(k1, ttlSec, async () => {
-        const t2 = await acquireLock(k2, ttlSec);
+      return await withLock(k1, ttlSec, async () => {
+        const t2 = await acquireLock(k2, ttlSec);
         if (!t2) throw new Error(`LOCK_CONFLICT:${p2}`);
         try {
-          const out = await runOnce();
+          const out1 = await runOnce(p1);
+          const out2 = await runOnce(p2);
           return Response.json({
             ok: true,
             userId: body.userId,
             providers,
             mode: 'rollups',
             snapshotDate: todayStr,
-            wrote: out.wrote,
-            providerUsed: out.provider,
+            wrote: out2.wrote, // preserve existing shape
+            providerUsed: 'both',
+            wroteByProvider: { [p1]: out1.wrote, [p2]: out2.wrote }, // additive field
           });
         } finally {
           await releaseLock(k2, t2);
         }
       });

Alternative (if multi-provider is not supported by design): enforce exclusivity at validation time and error when both are provided.

Also applies to: 99-122


try {
if (providers.length === 2) {
const [p1, p2] = providers;
const k1 = backfillLockKey(p1 as 'github' | 'gitlab', body.userId);
const k2 = backfillLockKey(p2 as 'github' | 'gitlab', body.userId);

return await withLock(k1, ttlSec, async () => {
const t2 = await acquireLock(k2, ttlSec);
if (!t2) throw new Error(`LOCK_CONFLICT:${p2}`);
try {
const out = await runOnce();
return Response.json({
ok: true,
userId: body.userId,
providers,
mode: 'rollups',
snapshotDate: todayStr,
wrote: out.wrote,
providerUsed: out.provider,
});
} finally {
await releaseLock(k2, t2);
}
});
}

const p = providers[0]!;
const key = backfillLockKey(p, body.userId);

return await withLock(key, ttlSec, async () => {
const out = await runOnce();
return Response.json({
ok: true,
userId: body.userId,
providers,
mode: 'rollups',
snapshotDate: todayStr,
wrote: out.wrote,
providerUsed: out.provider,
});
});
} catch (err: unknown) {
const id = Math.random().toString(36).slice(2, 8);
console.error(`[rollup-backfill:${id}]`, err);
const msg = String(err instanceof Error ? err.message : err);
if (msg.startsWith('LOCK_CONFLICT')) {
const p = msg.split(':')[1] || 'unknown';
return new Response(`Conflict: backfill already running for ${p}`, { status: 409 });
}
return new Response(`Internal Error (ref ${id})`, { status: 500 });
}
}
167 changes: 167 additions & 0 deletions apps/web/app/api/internal/leaderboard/cron/daily/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
export const revalidate = 0;

import { isCronAuthorized } from '@workspace/env/verify-cron';
import { env } from '@workspace/env/server';
import { NextRequest } from 'next/server';
import { z } from 'zod/v4';

import { syncUserLeaderboards } from '@workspace/api/leaderboard/redis';
import { refreshUserRollups } from '@workspace/api/leaderboard/aggregator';
import { redis } from '@workspace/api/redis/client';
import { db } from '@workspace/db';

const USER_SET = 'lb:users';
const META = (id: string) => `lb:user:${id}`;

const Query = z.object({
limit: z.coerce.number().int().min(1).max(5000).default(1000),
concurrency: z.coerce.number().int().min(1).max(8).default(4),
dry: z.union([z.literal('1'), z.literal('true'), z.literal('0'), z.literal('false')]).optional(),
});

function ymd(d = new Date()) {
const y = d.getUTCFullYear();
const m = String(d.getUTCMonth() + 1).padStart(2, '0');
const dd = String(d.getUTCDate()).padStart(2, '0');
return `${y}-${m}-${dd}`;
}

export async function GET(req: NextRequest) {
const ok =
isCronAuthorized(req.headers.get('authorization')) || !!req.headers.get('x-vercel-cron');
if (!ok) return new Response('Unauthorized', { status: 401 });

const parsed = Query.safeParse(Object.fromEntries(req.nextUrl.searchParams));
if (!parsed.success) {
return new Response(`Bad Request: ${parsed.error.message}`, { status: 400 });
}
const { limit, concurrency, dry } = parsed.data;
const isDry = dry === '1' || dry === 'true';

const snapshotDate = ymd(new Date());

try {
await redis.ping();

const allIdsRaw = await redis.smembers(USER_SET);
const userIds = (Array.isArray(allIdsRaw) ? allIdsRaw : []).map(String).slice(0, limit);

if (userIds.length === 0) {
return Response.json({
ok: true,
scanned: 0,
processed: 0,
skipped: 0,
errors: [],
snapshotDate,
note: `No user IDs in Redis set "${USER_SET}". Seed it via backfill.`,
});
}

const pipe = redis.pipeline();
for (const id of userIds) pipe.hgetall(META(id));
const rawResults = await pipe.exec();

const metaRows = rawResults.map((r) => {
const val = Array.isArray(r) ? r[1] : r;
return val && typeof val === 'object' ? (val as Record<string, unknown>) : {};
});

const asTrimmedString = (v: unknown): string | undefined => {
if (typeof v === 'string') return v.trim();
if (v == null) return undefined;
if (typeof v === 'number' || typeof v === 'boolean') return String(v).trim();
if (Array.isArray(v)) return v.length ? String(v[0]).trim() : undefined;
return undefined;
};

if (isDry) {
const preview = userIds.map((id, i) => {
const m = metaRows[i] || {};
const githubLogin = asTrimmedString(m.githubLogin) ?? null;
const gitlabUsername = asTrimmedString(m.gitlabUsername) ?? null;
return { userId: id, githubLogin, gitlabUsername };
});
return Response.json({
ok: true,
dryRun: true,
scanned: userIds.length,
sample: preview.slice(0, 10),
snapshotDate,
});
}

const workers = Math.max(1, Math.min(concurrency, 8));
let idx = 0;
let processed = 0;
let skipped = 0;
const errors: Array<{ userId: string; error: string }> = [];

const tasks = Array.from({ length: workers }, async () => {
while (true) {
const i = idx++;
if (i >= userIds.length) break;

const userId = userIds[i]!;
const m = metaRows[i] || {};
const githubLogin = asTrimmedString(m.githubLogin);
const gitlabUsername = asTrimmedString(m.gitlabUsername);

if (!githubLogin && !gitlabUsername) {
skipped++;
continue;
}

if (githubLogin && !env.GITHUB_TOKEN) {
errors.push({ userId, error: 'Missing GITHUB_TOKEN' });
skipped++;
continue;
}
if (gitlabUsername && !env.GITLAB_TOKEN) {
errors.push({ userId, error: 'Missing GITLAB_TOKEN' });
skipped++;
continue;
}

try {
await refreshUserRollups(
{ db },
{
userId,
githubLogin,
gitlabUsername,
githubToken: env.GITHUB_TOKEN,
gitlabToken: env.GITLAB_TOKEN,
gitlabBaseUrl: env.GITLAB_ISSUER || 'https://gitlab.com',
},
);

await syncUserLeaderboards(db, userId);
processed++;
} catch (err) {
errors.push({ userId, error: String(err instanceof Error ? err.message : err) });
}
}
});

await Promise.all(tasks);

return Response.json({
ok: true,
scanned: userIds.length,
processed,
skipped,
errors,
snapshotDate,
});
} catch (err: unknown) {
const msg = String(err instanceof Error ? `${err.name}: ${err.message}` : err);
if (env.VERCEL_ENV !== 'production') {
console.error('[cron/daily-rollups] fatal:', err);
return new Response(`Internal Error: ${msg}`, { status: 500 });
}
return new Response('Internal Error', { status: 500 });
}
}
Loading