New hub and worker architecture #165
Conversation
src/hypofuzz/hypofuzz.py
Outdated
| # TODO actually defer starting up targets here, based on worker lifetime | ||
| # and startup cost estimators here | ||
| for nodeid in candidates: | ||
| self.add_target(nodeid) |
There was a problem hiding this comment.
I think I'd like to move this responsibility into the hub process; it can slowly 'feed in' extra targets to amortize startup cost, etc. That way only one process needs to read from the database to get good performance, and we get a pretty nice separation of concerns too.
There was a problem hiding this comment.
good call, I think. My only wonder is if the workers have access to expensive-to-serialize information that means they could make better decisions than the hub, but I don't think so. It's just estimator state that we're already sharing anyway.
Zac-HD
left a comment
Zac-HD
left a comment
There was a problem hiding this comment.
Looks solid, though there are obviously some substantial followup PRs to come before it really pays off 🙂
For this PR, I'd love to have some basic tests to act as sanity-checks when we later start complicating things, and then merge.
2 participants
Split from #154. The allocation behavior is exactly the same as master, and several things here are no-ops. I figured splitting out the architecture makes it easier to review that, and then we can review the estimators separately.