Skip to content

Comments

feat: add GitHub issue ingestion with admin #177

Merged
joelamouche merged 9 commits intomainfrom
feature/github-issues-ingestion
Feb 23, 2026
Merged

feat: add GitHub issue ingestion with admin #177
joelamouche merged 9 commits intomainfrom
feature/github-issues-ingestion

Conversation

@tusharshah21
Copy link
Collaborator

Summary

Implements GitHub issue ingestion into the backend database per issue #74: adds the github_issues table with composite PK (repo_id, github_issue_id), a POST /admin/github/sync endpoint behind admin auth, and idempotent upsert logic that derives points from labels, normalizes label names, persists assignees, and ignores PRs.

Changes

  • Added migration 006_create_github_issues_table.sql with composite PK and indexes
  • Added GithubIssue domain entity, GithubIssueRepository trait, and PostgresGithubIssueRepository with ON CONFLICT upsert
  • Added GithubService trait and RestGithubService implementation for GitHub REST API
  • Added sync_github_issues command with derive_points (label pattern points:N) and transform_issue (lower-case labels, assignee logins, PR filtering)
  • Added GithubSyncRequest/GithubSyncResponse DTOs
  • Added github_sync_handler and wired POST /admin/github/sync under existing admin auth middleware
  • Added reqwest 0.12 dependency for HTTP client
  • Added 10 unit tests covering points derivation, label normalization, PR ignoring, idempotent upsert, closed issue reflection, assignee persistence, and title update on re-sync

Scope

  • This PR intentionally avoids refactors, cleanups, and unrelated changes.
  • All modifications are limited to what is required by the issue.
  • Pagination, retry/backoff, and custom endpoint auth are explicitly out of scope per the ticket.

Verification

  • cargo check --tests compiles with zero errors
  • 10 new unit tests pass (labels→points, idempotent upsert, closed issue, PR filtering, assignees, upsert updates)
  • All existing tests continue passing

Closes #74

@tusharshah21
Copy link
Collaborator Author

GitHub issues ingestion with idempotent upsert (composite PK), admin-only sync endpoint, PR filtering, label→points derivation, and tests. Ready for review.

Copy link
Contributor

@joelamouche joelamouche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a first round of comments

Copy link
Contributor

@joelamouche joelamouche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another round of comments.

Don't forget to test this with a real repo, not just the unit tests

@tusharshah21
Copy link
Collaborator Author

Done reverted unrelated edits, renamed fields, added GITHUB_* + Bearer auth, switched to repo-name sync, added GET /github/issues and README examples
please review.

@joelamouche
Copy link
Contributor

pls fix CI

@joelamouche
Copy link
Contributor

I ran the sync locally with the guild repo and all I got was this

backend-1 | 2026-02-18T10:36:49.500225Z DEBUG request{method=POST uri=/admin/github/sync version=HTTP/1.1}: tower_http::trace::on_request: started processing request backend-1 | 2026-02-18T10:36:49.500390Z DEBUG request{method=POST uri=/admin/github/sync version=HTTP/1.1}: tower_http::trace::on_response: finished processing request latency=0 ms status=401

And the git issue db is empty

You tried this locally with a git repo and it worked?

@joelamouche
Copy link
Contributor

pls also add logs for each issue synced. It's gonna help with debugging

@tusharshah21
Copy link
Collaborator Author

I ran the sync locally with the guild repo and all I got was this

backend-1 | 2026-02-18T10:36:49.500225Z DEBUG request{method=POST uri=/admin/github/sync version=HTTP/1.1}: tower_http::trace::on_request: started processing request backend-1 | 2026-02-18T10:36:49.500390Z DEBUG request{method=POST uri=/admin/github/sync version=HTTP/1.1}: tower_http::trace::on_response: finished processing request latency=0 ms status=401

And the git issue db is empty

You tried this locally with a git repo and it worked?

The 401 is from missing env vars. Add to your .env:

ADMIN_ADDRESSES=0xYourWalletAddress
GITHUB_TOKEN=your_github_pat
GITHUB_OWNER=TheSoftwareDevGuild
GITHUB_API_URL=https://api.github.com
TEST_MODE=1

Then call with -H "x-eth-address: 0xYourWalletAddress". .env.example has been updated with all required vars. Tested locally 50 issues synced successfully.> ADMIN_ADDRESSES=0xYourWalletAddress
GITHUB_TOKEN=your_github_pat
GITHUB_OWNER=TheSoftwareDevGuild
GITHUB_API_URL=https://api.github.com
TEST_MODE=1

Then call with -H "x-eth-address: 0xYourWalletAddress". .env.example has been updated with all required vars. Tested locally - 54 issues synced successfully.

@tusharshah21
Copy link
Collaborator Author

pls also add logs for each issue synced. It's gonna help with debugging

Done each synced issue now logs repo, issue_number, title, state, and points.

Copy link
Contributor

@joelamouche joelamouche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alirght, I was able to test it, good job! 🥳

@joelamouche joelamouche merged commit 6f8b8d8 into main Feb 23, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ingest GitHub Issues into Backend Database

2 participants