Adaption due to Understat dynamic data load #3
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi, this is the following PR. I heavily relied on LLMs, but worked for me fine. Up for a critical review:
Problem
Understat changed their website to load data dynamically via JavaScript, so the datesData and shotsData variables are no longer in the initial HTML. This caused understat-db ingest to fail with a ValueError when trying to extract JSON from script tags.
Solution
Switched to direct API endpoints instead of parsing HTML:
Matches:
getLeagueData/{league}/{season}— returns JSON withdates,teams, andplayersShots:
getMatchData/{match_id}— returns JSON withshots,rosters, andtmplChanges
Updated matches() to call the API endpoint directly
Updated shots() to call the API endpoint directly
Added required HTTP headers (Referer, User-Agent, Accept, X-Requested-With) for the API
Improved error handling in extract_json() with clearer messages
Kept fallback to the old HTML parsing method for backwards compatibility
Testing
✅ matches() works (tested with Bundesliga 2025)
✅ shots() works (tested with match ID 30224)
✅ understat-db ingest command now works correctly
Backwards Compatibility
The old HTML parsing method is still available as a fallback for cases where the API might not be accessible, ensuring backwards compatibility.