Skip to content

Comments

[capitolweb] scraper and parser refactor#27

Open
will-horning wants to merge 6 commits intomasterfrom
will_separate_scraper_and_parser
Open

[capitolweb] scraper and parser refactor#27
will-horning wants to merge 6 commits intomasterfrom
will_separate_scraper_and_parser

Conversation

@will-horning
Copy link

This PR changes a bunch of things in the parser and scraper:

  1. I removed the celery stuff, that would only really be useful if we were doing more large scale (and by necessity distributed) pre-compute jobs. Both scraper and parser are run through custom manage.py commands.
  2. I separated the worker app into two apps: scraper and parser.
  3. Added some thin test coverage for both scraper and parser.
  4. Refactored the big parser function into a class that lazily computes each field using django's cached_property decorator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant