fix: performance issues around insertion & averages#17
Conversation
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
src/lib.rs
Outdated
| fn flush(&mut self) -> Result<(), diesel::result::Error> { | ||
| use crate::schema::metrics::dsl::metrics; | ||
| // trace!("Flushing {} records", self.queue.len()); | ||
| if self.queue.is_empty() { | ||
| self.last_flush = Instant::now(); | ||
| return Ok(()); | ||
| } | ||
| let mut drain_buffer: Vec<NewMetric> = self.queue.drain(..).collect(); | ||
| let db = &mut self.db; | ||
| let queue = self.queue.drain(..); | ||
| db.transaction::<_, diesel::result::Error, _>(|db| { | ||
| for rec in queue { | ||
| insert_into(metrics).values(&rec).execute(db)?; | ||
| } | ||
| insert_into(metrics).values(&drain_buffer).execute(db)?; | ||
| Ok(()) | ||
| })?; |
There was a problem hiding this comment.
The new flush drains the entire queue into a Vec and performs one insert_into(metrics).values(&drain_buffer) call. SQLite’s default SQLITE_MAX_VARIABLE_NUMBER is 999, so a queue larger than about 333 records (3 columns per row) will cause Error: too many SQL variables and the flushed metrics are dropped because the queue was drained before the error surfaced. The previous per-row insert avoided this limit. Consider chunking the insert into smaller batches or reducing the flush size to stay under SQLite’s parameter cap.
Useful? React with 👍 / 👎.
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
hopefully reduces some overhead in metrics logging in particular