Skip to content

Migrate extraction pipeline code from the local repo to the GitHub #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Dv1t merged 103 commits into from
Dec 12, 2025
Merged

Migrate extraction pipeline code from the local repo to the GitHub #5

Dv1t merged 103 commits into from
Dec 12, 2025

Conversation

@AxisAlexNT
Copy link
Collaborator

@AxisAlexNT AxisAlexNT commented Dec 12, 2025

  1. Extraction pipeline code migrated from the internal repo;
  2. Added README into extraction submodule;
  3. Added license headers to the Python source code files.

@AxisAlexNT
First new outlines-partial
cbf7538
@AxisAlexNT
Qwen does not work great for this task right now
fd9843d
@AxisAlexNT
Added multiple options support into pipeline
50845f6
@AxisAlexNT
Before launching on all texts
2d238de
@AxisAlexNT
Launching on massive data
71b76b8
@AxisAlexNT
Reduce timeout
3c07822
@AxisAlexNT
Will now start with micro-C steps
235980d
@AxisAlexNT
Add all schemas
d9fa7d1
@AxisAlexNT
Update schema to try pass Ollama limitations
0866a2e
@AxisAlexNT
Will retry with the smaller outputs
a58254c
@AxisAlexNT
Looks better for schema C
319b296
@AxisAlexNT
Attempt to test schema C with longer context
2ed1f6d
@AxisAlexNT
Tried to update schema C more strictly specifying types
6b8da65
@AxisAlexNT
C5 schema created
b5d2dbe
@AxisAlexNT
C5 schema works, but not ideal
ffd5bc6
@AxisAlexNT
Added descriptions to all fields in C5
8c40a1c
@AxisAlexNT
Temporarily switch off tools
5bd0b1b
@AxisAlexNT
Try to run all schemas on all models
1b4444b
@AxisAlexNT
Adding Marker-PDF
8dbe352
@AxisAlexNT
Updated marker-pdf script to stream progress
56aafdb
@AxisAlexNT
Reordered models for bench
62ed81f
@AxisAlexNT
Added all models and Ollama base URL to config, but requires a lot of…
ad14cb2 
… VRAM
@AxisAlexNT
Removed excessive models
5585682
@AxisAlexNT
Update prompts for Chinese models to ask results in English
fa6803a
@AxisAlexNT
Slightly updated schemas and added full run
f05cbf5
@AxisAlexNT
Schem supports per-step timeouts
9e6b257
@AxisAlexNT
Split B schema into B and B_types
9108abe
@AxisAlexNT
Frozen requirements and environment
5ed7f4c
@AxisAlexNT
Added support for API_TOKEN at Vast.AI
3b37c41
@AxisAlexNT
A lot of articles were parsed
428b3e4
@AxisAlexNT
Prompt slightly changed
116d5b7
@AxisAlexNT
Fixed errors in tuples
c939969
@AxisAlexNT
f-string bug fixed
b1c0b4d
@AxisAlexNT
Bugfix messages
6ed6443
@AxisAlexNT
Update params to include more models
6e045fd
@AxisAlexNT
Start run for all articles
bd4e8c9
@AxisAlexNT
Re-run try faster
3ffa95c
@AxisAlexNT
Will pass the whole article text and lower the temperature for the fi…
a668f07 
…x transform
@AxisAlexNT
Validator bug fixed
75c7a1d
@AxisAlexNT
Return back to snippets
915aa22
@AxisAlexNT
Try to easy-fix an object
bdeafa9
@AxisAlexNT
Easy validation fixed
2bef036
@AxisAlexNT
Easy-fix sould be even more applicable
98c6ba3
@AxisAlexNT
Try adding full article, not snippets
83d05af
@AxisAlexNT
Try making simple fixer even more robust
9772942
@AxisAlexNT
Remove non-strict seq pass
8a68061
@AxisAlexNT
Seems that easy fixer should take first mapping
dfcab0d
@AxisAlexNT
Add other non-qwen-1m models
51d2309
@AxisAlexNT
Add more passes
3e3d409
@AxisAlexNT
Intermediate hyb_db perfmetrics added
72d686b
@AxisAlexNT
Upgraded solution for continuation and performance metrics
e99266a
@AxisAlexNT
Added more models and steps for benchmark
5ac2aaf
@AxisAlexNT
Changed input directory to benchmarks in markdown
5f4d279
@AxisAlexNT
Merge PROBEst history before migrating selected extraction files
f22c21b
@AxisAlexNT
Add MIT license header to Python files
778ecc2
@AxisAlexNT
Added README for the extraction module
02b61ba
@AxisAlexNT AxisAlexNT requested review from Dv1t and dsmutin December 12, 2025 13:29
@AxisAlexNT AxisAlexNT self-assigned this Dec 12, 2025
@Dv1t Dv1t merged commit 35a83b0 into main Dec 12, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Dv1t Dv1t Awaiting requested review from Dv1t

@dsmutin dsmutin Awaiting requested review from dsmutin

Assignees

@AxisAlexNT AxisAlexNT

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@AxisAlexNT @Dv1t