Skip to content

Conversation

@aaronjae22
Copy link
Collaborator

The analytics page was crashing in production due to memory overflow when processing large amounts of data. Server was running out of memory trying to load everything at once, causing 500 errors in production when accessing the analytics page.

This PR fixes that by optimizing how we handle data processing. It restructured how we fetch and process data to be more memory-efficient while keeping the exact same functionality and output.

  • Process QueryLogs in chunks using iterator() instead of loading everything at once
  • Let the database handle counting (instead of Python doing it)
  • Only fetch the fields we actually need for sentiment analysis
  • Clean up dictionary operations for better memory usage

Closes #146

@aaronjae22 aaronjae22 requested review from alexbainter and lisad June 2, 2025 14:17
@aaronjae22 aaronjae22 merged commit 6f19764 into main Jun 2, 2025
4 checks passed
@aaronjae22 aaronjae22 self-assigned this Jun 2, 2025
@aaronjae22 aaronjae22 added the bug Something isn't working label Jun 2, 2025
@aaronjae22
Copy link
Collaborator Author

Working as expected now
Screen Capture Result Jun 2 2025

@aaronjae22 aaronjae22 deleted the fix-analytics branch June 2, 2025 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Analytics page errors May 5

3 participants