Open
Conversation
…r-probing-hash-join-clean # Conflicts: # src/execution/radix_partitioned_hashtable.cpp
… into linear-probing-hash-join-clean # Conflicts: # .sanitizer-thread-suppressions.txt
…r-probing-hash-join-clean # Conflicts: # src/execution/radix_partitioned_hashtable.cpp
gropaul
pushed a commit
that referenced
this pull request
Feb 3, 2025
Fixes duckdblabs/duckdb-internal#3922 The failing query ```SQL SET order_by_non_integer_literal=true; SELECT DISTINCT ON ( 'string' ) 'string', GROUP BY CUBE ( 'string', ), 'string' IN ( SELECT 'string' ), HAVING 'string' IN ( SELECT 'string'); ``` The Plan generated before optimization is below. During optimization there is an attempt to convert the mark join into a semi. Before this conversion takes place, we usually check to make sure the mark join is not used in any operators above the mark join to prevent plan verification errors. Up until this point, only logical projections were checked for mark joins. Turns out this query is planned in such a way that the mark join is in one of the expressions of the aggregate operator. This was not checked, so the mark to semi conversion would take place. The fix is to modify the filter pushdown optimization so that it stores table indexes from logical aggregate operators. ``` ┌───────────────────────────┐ │ PROJECTION #1 │ │ ──────────────────── │ │ Expressions: #[2.0] │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ FILTER │ │ ──────────────────── │ │ Expressions: #[2.1] │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ AGGREGATE #2, #3, #4 │ │ ──────────────────── │ │ Groups: │ │ 'string' │ │ #[14.0] │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ COMPARISON_JOIN │ │ ──────────────────── │ │ Join Type: MARK │ │ ├──────────────┐ │ Conditions: │ │ │ ('string' = #[8.0]) │ │ └─────────────┬─────────────┘ │ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ DUMMY_SCAN #0 ││ PROJECTION #8 │ │ ──────────────────── ││ ──────────────────── │ │ ││ Expressions: 'string' │ └───────────────────────────┘└─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ DUMMY_SCAN #7 │ │ ──────────────────── │ └───────────────────────────┘ ```
gropaul
pushed a commit
that referenced
this pull request
Feb 18, 2025
We had two users crash with the following backtrace:
```
frame #0: 0x0000ffffab2571ec
frame #1: 0x0000aaaaac00c5fc duckling`duckdb::InternalException::InternalException(this=<unavailable>, msg=<unavailable>) at exception.cpp:328:2
frame #2: 0x0000aaaaac1ee418 duckling`duckdb::optional_ptr<duckdb::OptimisticDataWriter, true>::CheckValid(this=<unavailable>) const at optional_ptr.hpp:34:11
frame #3: 0x0000aaaaac1eea8c duckling`duckdb::MergeCollectionTask::Execute(duckdb::PhysicalBatchInsert const&, duckdb::ClientContext&, duckdb::GlobalSinkState&, duckdb::LocalSinkState&) [inlined] duckdb::optional_ptr<duckdb::OptimisticDataWriter, true>::operator*(this=<unavailable>) at optional_ptr.hpp:43:3
frame #4: 0x0000aaaaac1eea84 duckling`duckdb::MergeCollectionTask::Execute(this=0x0000aaaaf1b06150, op=<unavailable>, context=0x0000aaaba820d8d0, gstate_p=0x0000aaab06880f00, lstate_p=<unavailable>) at physical_batch_insert.cpp:219:90
frame #5: 0x0000aaaaac1d2e10 duckling`duckdb::PhysicalBatchInsert::Sink(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSinkInput&) const [inlined] duckdb::PhysicalBatchInsert::ExecuteTask(this=0x0000aaaafa62ab40, context=<unavailable>, gstate_p=0x0000aaab06880f00, lstate_p=0x0000aab12d442960) const at physical_batch_insert.cpp:425:8
frame #6: 0x0000aaaaac1d2dd8 duckling`duckdb::PhysicalBatchInsert::Sink(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSinkInput&) const [inlined] duckdb::PhysicalBatchInsert::ExecuteTasks(this=0x0000aaaafa62ab40, context=<unavailable>, gstate_p=0x0000aaab06880f00, lstate_p=0x0000aab12d442960) const at physical_batch_insert.cpp:431:9
frame #7: 0x0000aaaaac1d2dd8 duckling`duckdb::PhysicalBatchInsert::Sink(this=0x0000aaaafa62ab40, context=0x0000aab2fffd7cb0, chunk=<unavailable>, input=<unavailable>) const at physical_batch_insert.cpp:494:4
frame #8: 0x0000aaaaac353158 duckling`duckdb::PipelineExecutor::ExecutePushInternal(duckdb::DataChunk&, duckdb::ExecutionBudget&, unsigned long) [inlined] duckdb::PipelineExecutor::Sink(this=0x0000aab2fffd7c00, chunk=0x0000aab2fffd7d30, input=0x0000fffec0aba8d8) at pipeline_executor.cpp:521:24
frame #9: 0x0000aaaaac353130 duckling`duckdb::PipelineExecutor::ExecutePushInternal(this=0x0000aab2fffd7c00, input=0x0000aab2fffd7d30, chunk_budget=0x0000fffec0aba980, initial_idx=0) at pipeline_executor.cpp:332:23
frame #10: 0x0000aaaaac34f7b4 duckling`duckdb::PipelineExecutor::Execute(this=0x0000aab2fffd7c00, max_chunks=<unavailable>) at pipeline_executor.cpp:201:13
frame #11: 0x0000aaaaac34f258 duckling`duckdb::PipelineTask::ExecuteTask(duckdb::TaskExecutionMode) [inlined] duckdb::PipelineExecutor::Execute(this=<unavailable>) at pipeline_executor.cpp:278:9
frame #12: 0x0000aaaaac34f250 duckling`duckdb::PipelineTask::ExecuteTask(this=0x0000aab16dafd630, mode=<unavailable>) at pipeline.cpp:51:33
frame #13: 0x0000aaaaac348298 duckling`duckdb::ExecutorTask::Execute(this=0x0000aab16dafd630, mode=<unavailable>) at executor_task.cpp:49:11
frame #14: 0x0000aaaaac356600 duckling`duckdb::TaskScheduler::ExecuteForever(this=0x0000aaaaf0105560, marker=0x0000aaaaf00ee578) at task_scheduler.cpp:189:32
frame #15: 0x0000ffffab0a31fc
frame #16: 0x0000ffffab2ad5c8
```
Core dump analysis showed that the assertion `D_ASSERT(lstate.writer);`
in `MergeCollectionTask::Execute` (i.e. it is crashing because
`lstate.writer` is NULLPTR) was not satisfied when
`PhysicalBatchInsert::Sink` was processing merge tasks from (other)
pipeline executors.
My suspicion is that this is only likely to happen for heavily
concurrent workloads (applicable to the two users which crashed). The
patch submitted as part of this PR has addressed the issue for these
users.
gropaul
pushed a commit
that referenced
this pull request
Nov 7, 2025
…al get function is an unnest (duckdb#19467) The issue is caused by a very specific plan where a logical get (that has an unnest function) is the direct child of a logical comparison join. Normally for logical gets, we only add the one table index to the set of table indexes that can be joined on. However, if the logical get has an unnest function and there are children, the table indexes of those children can also be joined on. The Join Order Optimizer did not consider this case, which also turns out to be extremely rare. Here is the plan before the join order optimizer gets to it. I tried to write another test case but struggled for about 30 min. ``` ┌───────────────────────────┐ │ PROJECTION #23 │ │ ──────────────────── │ │ Expressions: │ │ #[0.0] (VARCHAR[]) │ │ #[0.1] (VARCHAR) │ │ #[0.2] (INTEGER) │ │ #[0.3] (VARCHAR) │ │ #[0.4] (VARCHAR) │ │ #[8.0] (VARCHAR) │ │ #[17.0] (JSON) │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ DELIM_JOIN │ │ ──────────────────── │ │ Join Type: INNER │ │ ├──────────────┐ │ Conditions: │ │ │ (#[29.0] IS NOT DISTINCT │ │ │ FROM #[2.1]) │ │ └─────────────┬─────────────┘ │ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ WINDOW #29 ││ COMPARISON_JOIN │ │ ──────────────────── ││ ──────────────────── │ │ Expressions: ││ Join Type: INNER │ │ ROW_NUMBER() OVER (ROWS ││ │ │ BETWEEN UNBOUNDED ││ Conditions: │ │ PRECEDING AND CURRENT ROW)││ (#[2.1] IS NOT DISTINCT ├──────────────┐ │ ││ FROM #[17.1]) │ │ │ ││ (#[2.2] IS NOT DISTINCT │ │ │ ││ FROM #[17.2]) │ │ │ ││ (#[2.3] IS NOT DISTINCT │ │ │ ││ FROM #[17.3]) │ │ └─────────────┬─────────────┘└─────────────┬─────────────┘ │ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ SEQ_SCAN #0 ││ UNNEST #8 ││ PROJECTION #17 │ │ ──────────────────── ││ ──────────────────── ││ ──────────────────── │ │ Table: xxxxxxxxxxxxxx ││ ││ Expressions: │ │ Type: Sequential Scan ││ ││ #[16.1] (JSON) │ │ ││ ││ #[10.1] (BIGINT) │ │ ││ ││ #[10.2] (VARCHAR) │ │ ││ ││ #[10.3] (VARCHAR[]) │ └───────────────────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ PROJECTION #2 ││ JSON_EACH #16 │ │ ──────────────────── ││ ──────────────────── │ │ Expressions: ││ │ │ #[30.2] (VARCHAR[]) ││ │ │ #[30.0] (BIGINT) ││ │ │ #[30.1] (VARCHAR) ││ │ │ #[30.2] (VARCHAR[]) ││ │ └─────────────┬─────────────┘└─────────────┬─────────────┘ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ DELIM_GET #30 ││ PROJECTION #10 │ │ ──────────────────── ││ ──────────────────── │ │ ││ Expressions: │ │ ││ #[31.1] (VARCHAR) │ │ ││ #[31.0] (BIGINT) │ │ ││ #[31.1] (VARCHAR) │ │ ││ #[31.2] (VARCHAR[]) │ └───────────────────────────┘└─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ DELIM_GET #31 │ │ ──────────────────── │ └───────────────────────────┘ ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.