refactor(query): stream style block writer for hash join spill #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

bohutang wants to merge 3,722 commits into main from refactor/stream_writer

The diff you're trying to view is too large. We only load the first 3000 changed files.

.cargo/audit.toml

-Original file line number
+Diff line change
@@ Expand Up / @@ -3,20 +3,10 @@ ignore = [ @@
         # rio: v0.9.4 https://rustsec.org/advisories/RUSTSEC-2020-0021
         # allows a use-after-free buffer access when a future is leaked
         "RUSTSEC-2020-0021",
-        # time: Potential segfault in the time crate
-        # We are not affected by this CVE.
-        # And there is no actions we can take, waiting for upstream.
-        "RUSTSEC-2020-0071",
-        # hyper
-        # Lenient `hyper` header parsing of `Content-Length` could allow request smuggling
-        "RUSTSEC-2021-0078",
-        # hyper
-        # Integer overflow in `hyper`'s parsing of the `Transfer-Encoding` header leads to data loss
-        "RUSTSEC-2021-0079",
-        # ansi_term
-        # ansi_term is Unmaintained
-        "RUSTSEC-2021-0139",
-        # buf_redux
-        # buf_redux is Unmaintained
-        "RUSTSEC-2023-0028"
+        # rsa
+        # Marvin Attack: potential key recovery through timing sidechannels
+        "RUSTSEC-2023-0071",
+        # atty
+        # Potential unaligned read
+        "RUSTSEC-2021-0145",
     ]

.config/version-schema.json

-Original file line number
+Diff line change
@@ -0,0 +1,12 @@
+    {
+      "$schema": "http://json-schema.org/draft-07/schema#",
+      "type": "object",
+      "properties": {
+        "dependencies": {
+          "type": "object",
+          "properties": {
+            "databend-common-version": false
+          }
+        }
+      }
+    }

.config/workspace-schema.json

-Original file line number
+Diff line change
@@ -0,0 +1,57 @@
+    {
+      "$schema": "http://json-schema.org/draft-07/schema#",
+      "type": "object",
+      "properties": {
+        "dependencies": {
+          "type": "object",
+          "additionalProperties": {
+            "type": "object",
+            "properties": {
+              "workspace": {
+                "type": "boolean",
+                "enum": [
+                  true
+                ]
+              }
+            },
+            "required": [
+              "workspace"
+            ]
+          }
+        },
+        "dev-dependencies": {
+          "type": "object",
+          "additionalProperties": {
+            "type": "object",
+            "properties": {
+              "workspace": {
+                "type": "boolean",
+                "enum": [
+                  true
+                ]
+              }
+            },
+            "required": [
+              "workspace"
+            ]
+          }
+        },
+        "build-dependencies": {
+          "type": "object",
+          "additionalProperties": {
+            "type": "object",
+            "properties": {
+              "workspace": {
+                "type": "boolean",
+                "enum": [
+                  true
+                ]
+              }
+            },
+            "required": [
+              "workspace"
+            ]
+          }
+        }
+      }
+    }

.gitattributes

-Original file line number
+Diff line change
@@ -1,3 +1,2 @@
     docker/** -text
-    scripts/** -text
-    website/databend/** -text
+    scripts/** -text

.github/CODEOWNERS

-Original file line number
+Diff line change
@@ Expand Up / @@ -39,10 +39,6 @@ @@
     /src/common/http/                  @datafuselabs/databend-api-reviewer
     /src/query/service/src/servers/    @datafuselabs/databend-api-reviewer
-    # Website
-    /website/              @datafuselabs/databend-document-reviewer
-    /docs/                 @datafuselabs/databend-document-reviewer
     # Tool
     /docker/               @datafuselabs/databend-tool-reviewer
     /scripts/              @datafuselabs/databend-tool-reviewer
@@ Expand Down @@

.github/ISSUE_TEMPLATE/20_bug_report.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -8,17 +8,17 @@ body:
  
          value: |

            Thank you very much for submitting feedback to Databend to help Databend develop better.

            If it is an idea or help wanted, please go to: [Discussion](https://github.com/datafuselabs/databend/discussions)

            If it is an idea or help wanted, please go to: [Discussion](https://github.com/databendlabs/databend/discussions)

      - type: checkboxes

        attributes:

          label: Search before asking

          description: >

            Please make sure to search in the [issues](https://github.com/datafuselabs/databend/issues) first to see

            Please make sure to search in the [issues](https://github.com/databendlabs/databend/issues) first to see

            whether the same issue was reported already.

          options:

            - label: >

                I had searched in the [issues](https://github.com/datafuselabs/databend/issues) and found no similar

                I had searched in the [issues](https://github.com/databendlabs/databend/issues) and found no similar

                issues.

              required: true

.github/ISSUE_TEMPLATE/config.yml

-Original file line number
+Diff line change
@@ -1,5 +1,5 @@
     blank_issues_enabled: true
     contact_links:
       - name: Question
-        url: https://github.com/datafuselabs/databend/discussions/categories/q-a
+        url: https://github.com/databendlabs/databend/discussions/categories/q-a
         about: Please ask and answer questions on the discussions Q&A category.

.github/PULL_REQUEST_TEMPLATE.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,7 +1,26 @@
  
    I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

    I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

    ## Summary

    Summary about this PR

    <!--

    Briefly describe what this PR aims to solve. Include background context that will help reviewers understand the purpose of the PR.

    - Closes #issue

    - fixes: #[Link the issue here]

    -->

    ## Tests

    - [ ] Unit Test

    - [ ] Logic Test

    - [ ] Benchmark Test

    - [ ] No Test - _Explain why_

    ## Type of change

    - [ ] Bug Fix (non-breaking change which fixes an issue)

    - [ ] New Feature (non-breaking change which adds functionality)

    - [ ] Breaking Change (fix or feature that could cause existing functionality not to work as expected)

    - [ ] Documentation Update

    - [ ] Refactoring

    - [ ] Performance Improvement

    - [ ] Other (please describe):

.github/actions/artifact_download/action.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -8,15 +8,15 @@ inputs:
  
        description: ""

        required: true

      category:

        description: "default/hdfs"

        description: "default/hdfs/udf"

        required: false

        default: default

      path:

        description: "default to ./target/${BUILD_PROFILE}/"

        required: false

        default: ""

      artifacts:

        description: "Artifacts to download, only works with s3"

        description: "Artifacts to download, only works with s3/gcs"

        required: false

        default: "meta,query"

    outputs:

    @@ -42,7 +42,7 @@ runs:
  
            fi

            echo "path=${path}" >> $GITHUB_OUTPUT

        - uses: actions/download-artifact@v3

        - uses: actions/download-artifact@v4

          if: env.RUNNER_PROVIDER == 'github'

          with:

            name: ${{ env.BUILD_PROFILE }}-${{ inputs.sha }}-${{ inputs.target }}-${{ inputs.category }}

    @@ -54,11 +54,7 @@ runs:
  
          run: |

            artifacts="${{ inputs.artifacts }}"

            for artifact in ${artifacts//,/ }; do

              if [[ "${artifact}" == "open-sharing" ]]; then

                chmod +x ${{ steps.info.outputs.path }}/open-sharing || true

              else

                chmod +x ${{ steps.info.outputs.path }}/databend-$artifact

              fi

              chmod +x ${{ steps.info.outputs.path }}/databend-$artifact

            done

        - name: Download artifact from s3

    @@ -67,32 +63,21 @@ runs:
  
          run: |

            artifacts="${{ inputs.artifacts }}"

            for artifact in ${artifacts//,/ }; do

              if [[ "${artifact}" == "open-sharing" ]]; then

                echo "==> fetching open-sharing ..."

                aws s3 cp s3://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/open-sharing ${{ steps.info.outputs.path }}/open-sharing --no-progress || true

                chmod +x ${{ steps.info.outputs.path }}/open-sharing || true

              else

                echo "==> fetching databend-$artifact ..."

                aws s3 cp s3://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/databend-$artifact ${{ steps.info.outputs.path }}/databend-$artifact --no-progress

                chmod +x ${{ steps.info.outputs.path }}/databend-$artifact

              fi

              echo "==> fetching databend-$artifact ..."

              aws s3 cp s3://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/databend-$artifact ${{ steps.info.outputs.path }}/databend-$artifact --no-progress

              chmod +x ${{ steps.info.outputs.path }}/databend-$artifact

            done

        - name: Download artifact from gcs

          if: env.RUNNER_PROVIDER == 'gcp'

          shell: bash

          run: |

            gcloud config set storage/parallel_composite_upload_enabled False

            artifacts="${{ inputs.artifacts }}"

            for artifact in ${artifacts//,/ }; do

              if [[ "${artifact}" == "open-sharing" ]]; then

                echo "==> fetching open-sharing ..."

                gcloud storage cp gs://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/open-sharing ${{ steps.info.outputs.path }}/open-sharing || true

                chmod +x ${{ steps.info.outputs.path }}/open-sharing || true

              else

                echo "==> fetching databend-$artifact ..."

                gcloud storage cp gs://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/databend-$artifact ${{ steps.info.outputs.path }}/databend-$artifact

                chmod +x ${{ steps.info.outputs.path }}/databend-$artifact

              fi

              echo "==> fetching databend-$artifact ..."

              gcloud storage cp gs://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/databend-$artifact ${{ steps.info.outputs.path }}/databend-$artifact

              chmod +x ${{ steps.info.outputs.path }}/databend-$artifact

            done

        - name: Check Disk Space

.github/actions/artifact_failure/action.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -19,9 +19,12 @@ runs: @@
                 docker logs "$line" > .databend/docker/"$line".log
             done
+            touch nohup.out
+            cp nohup.out .databend/nohup.out
             mkdir -p target
             find -type d -name .databend -print0 | xargs -0 tar -zcf target/failure-${{ inputs.name }}.tar.gz
-        - uses: actions/upload-artifact@v3
+        - uses: actions/upload-artifact@v4
           with:
             name: ${{ inputs.name }}
             path: target/failure-${{ inputs.name }}.tar.gz

.github/actions/artifact_meta_container_failure/action.yml

-Original file line number
+Diff line change
@@ -0,0 +1,23 @@
+    name: "Upload meta containers failure Artifacts"
+    description: "Upload meta containers Artifacts"
+    inputs:
+      name:
+        description: ""
+        required: true
+    runs:
+      using: "composite"
+      steps:
+        - name: pack failure artifacts
+          shell: bash
+          run: |
+            mkdir -p logs
+            for name in "test-databend-meta-0" "test-databend-meta-1" "test-databend-meta-2" "databend-metaverifier"
+            do
+              echo "cat logs of $name"
+              kubectl exec -i $name -n databend -- /cat-logs.sh > logs/$name.log
+            done
+            tar -zcf logs/failure-${{ inputs.name }}.tar.gz logs/*
+        - uses: actions/upload-artifact@v4
+          with:
+            name: ${{ inputs.name }}
+            path: logs/failure-${{ inputs.name }}.tar.gz

.github/actions/artifact_upload/action.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -32,32 +32,31 @@ runs: @@
         - name: Upload artifact to github
           if: env.RUNNER_PROVIDER == 'github'
-          uses: actions/upload-artifact@v3
+          uses: actions/upload-artifact@v4
           with:
             name: ${{ env.BUILD_PROFILE }}-${{ inputs.sha }}-${{ inputs.target }}-${{ inputs.category }}
             path: ${{ steps.info.outputs.path }}/databend-*
-        - name: Upload open-sharing to github
-          if: env.RUNNER_PROVIDER == 'github' && inputs.category == 'default'
-          uses: actions/upload-artifact@v3
-          with:
-            name: ${{ env.BUILD_PROFILE }}-${{ inputs.sha }}-${{ inputs.target }}-${{ inputs.category }}
-            path: ${{ steps.info.outputs.path }}/open-sharing
         - name: Upload artifacts to s3
           if: env.RUNNER_PROVIDER == 'aws'
           shell: bash
           run: |
+            echo "### artifacts 🚀" >> $GITHUB_STEP_SUMMARY
             aws s3 sync ./target/${{ inputs.target }}/${{ env.BUILD_PROFILE }}/ \
               s3://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/ \
               --exclude "*" --include "databend-*" --no-progress
-            aws s3 cp ./target/${{ inputs.target }}/${{ env.BUILD_PROFILE }}/open-sharing s3://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/ --no-progress || true
+            artifacts="meta,query,query.debug"
+            for artifact in ${artifacts//,/ }; do
+              if [[ -f ./target/${{ inputs.target }}/${{ env.BUILD_PROFILE }}/databend-${artifact} ]]; then
+                url=$(aws s3 presign s3://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/databend-${artifact} --expires-in 21600)
+                echo "- [${artifact}](${url})" >> $GITHUB_STEP_SUMMARY
+              fi
+            done
         - name: Upload artifact to gcs
           if: env.RUNNER_PROVIDER == 'gcp'
           shell: bash
           run: |
+            gcloud config set storage/parallel_composite_upload_enabled False
             gcloud storage cp --quiet --recursive ./target/${{ inputs.target }}/${{ env.BUILD_PROFILE }}/databend-* \
               gs://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/
-            gcloud storage cp --quiet ./target/${{ inputs.target }}/${{ env.BUILD_PROFILE }}/open-sharing \
-              gs://databend-ci/${{ env.BUILD_PROFILE }}/${{ inputs.sha }}/${{ inputs.target }}/${{ inputs.category }}/ || true

.github/actions/benchmark_cloud/action.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -41,14 +41,30 @@ runs:
  
            sudo apt-get update -yq

            sudo apt-get install -yq python3

        - name: Prepare

          working-directory: benchmark/clickbench

          shell: bash

          id: prepare

          env:

            BENDSQL_DSN: "databend://${{ inputs.cloud_user }}:${{ inputs.cloud_password }}@${{ inputs.cloud_gateway }}:443"

          run: |

            if [[ "${{ inputs.dataset }}" == "load" ]]; then

              echo "database=load_test_${{ inputs.run_id }}" >> $GITHUB_OUTPUT

              echo "tries=1" >> $GITHUB_OUTPUT

            else

              echo "database=clickbench" >> $GITHUB_OUTPUT

              echo "tries=3" >> $GITHUB_OUTPUT

            fi

        - name: Run Benchmark

          working-directory: benchmark/clickbench

          env:

            BENCHMARK_ID: ${{ inputs.run_id }}

            BENCHMARK_DATASET: ${{ inputs.dataset }}

            BENCHMARK_SIZE: ${{ inputs.size }}

            BENCHMARK_VERSION: ${{ inputs.version }}

            BENCHMARK_DATABASE: clickbench

            BENCHMARK_DATABASE: ${{ steps.prepare.outputs.database }}

            BENCHMARK_TRIES: ${{ steps.prepare.outputs.tries }}

            CLOUD_USER: ${{ inputs.cloud_user }}

            CLOUD_PASSWORD: ${{ inputs.cloud_password }}

            CLOUD_GATEWAY: ${{ inputs.cloud_gateway }}

    @@ -76,9 +92,9 @@ runs:
  
            mv result.json result-${{ inputs.dataset }}-cloud-${{ inputs.size }}.json

        - name: Upload artifact

          uses: actions/upload-artifact@v3

          uses: actions/upload-artifact@v4

          with:

            name: benchmark-${{ inputs.dataset }}

            name: benchmark-${{ inputs.dataset }}-${{ inputs.size }}

            path: benchmark/clickbench/result-${{ inputs.dataset }}-cloud-${{ inputs.size }}.json

        - name: Remove warehouse

.github/actions/benchmark_local/action.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -54,7 +54,7 @@ runs:
  
            mv result.json result-${{ inputs.dataset }}-local.json

        - name: Upload artifact

          uses: actions/upload-artifact@v3

          uses: actions/upload-artifact@v4

          with:

            name: benchmark-${{ inputs.dataset }}

            name: benchmark-${{ inputs.dataset }}-local

            path: benchmark/clickbench/result-${{ inputs.dataset }}-local.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(query): stream style block writer for hash join spill #40

Uh oh!

Diff view

Diff view

Uh oh!

There are no files selected for viewing

Uh oh!

Uh oh!

refactor(query): stream style block writer for hash join spill #40

Are you sure you want to change the base?

Uh oh!

refactor(query): stream style block writer for hash join spill #40

Uh oh!

Diff view

Diff view

Uh oh!

There are no files selected for viewing

Uh oh!

Uh oh!