Skip to content

Added functionality similar to parquet-tools to print info of parquet… #65

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
subkanthi wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Added functionality similar to parquet-tools to print info of parquet… #65

subkanthi wants to merge 6 commits into from
+387 −1

Conversation

@subkanthi
Copy link
Collaborator

@subkanthi subkanthi commented Oct 15, 2025

… files.

@subkanthi subkanthi linked an issue Oct 15, 2025 that may be closed by this pull request
Provide convenient introspection of Parquet files #59
Open
@subkanthi
Copy link
Collaborator Author

subkanthi commented Oct 29, 2025

Testing:
local file

 describe-parquet iris.parquet -a
2025-10-28 19:00:00 [2119751-100] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/config
---
summary:
  rows: 150
  rowGroups: 1
  compressedSize: 1885
  uncompressedSize: 0
  createdBy: "DuckDB"
  columnCount: 5
columns:
- name: "sepal.length"
  type: "DOUBLE"
  repetition: "OPTIONAL"
- name: "sepal.width"
  type: "DOUBLE"
  repetition: "OPTIONAL"
- name: "petal.length"
  type: "DOUBLE"
  repetition: "OPTIONAL"
- name: "petal.width"
  type: "DOUBLE"
  repetition: "OPTIONAL"
- name: "variety"
  type: "BINARY"
  repetition: "OPTIONAL"
  logicalType: "STRING"
rowGroups:

@subkanthi
Copy link
Collaborator Author

subkanthi commented Nov 7, 2025

remote file

describe-parquet   https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-01.parquet
2025-11-06 20:34:28 [12843412-49] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/config
---
summary:
  rows: 3475226
  rowGroups: 4
  compressedSize: 59138125
  uncompressedSize: 92602414
  createdBy: "parquet-cpp-arrow version 16.1.0"
  columnCount: 20

@subkanthi
Copy link
Collaborator Author

subkanthi commented Nov 7, 2025 

 describe-parquet -a   s3://aws-public-blockchain/v1.0/btc/transactions/date=2025-01-01/part-00000-33e8d075-2099-409b-a806-68dd17217d39-c000.snappy.parquet
2025-11-06 20:41:15 [12843412-78] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/config
---
summary:
  rows: 292213
  rowGroups: 5
  compressedSize: 539840873
  uncompressedSize: 717049859
  createdBy: "parquet-mr version 1.10.1 (build 65f31597b18a0f2718a129fd2d69af0168952c55)"
  columnCount: 19
columns:
- name: "hash"
  type: "BINARY"
  repetition: "OPTIONAL"
  logicalType: "STRING"
- name: "version"
  type: "INT64"
  repetition: "OPTIONAL"
- name: "size"
- ```

@subkanthi subkanthi marked this pull request as ready for review November 7, 2025 02:42
@subkanthi subkanthi requested a review from shyiko November 7, 2025 02:42
@subkanthi subkanthi requested a review from xieandrew January 16, 2026 18:48
xieandrew
xieandrew reviewed Jan 22, 2026
View reviewed changes
Copy link
Collaborator

@xieandrew xieandrew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the commands and everything seems to work. Just a few comments.

ice/src/main/java/com/altinity/ice/cli/internal/cmd/DescribeParquet.java
Option... options)
throws IOException {

Lazy<software.amazon.awssdk.services.s3.S3Client> s3ClientLazy =
Copy link
Collaborator

@xieandrew xieandrew Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be imported

ice/src/main/java/com/altinity/ice/cli/internal/cmd/DescribeParquet.java
}

public static void run(InputFile inputFile, boolean json, Option... options) throws IOException {

Copy link
Collaborator

@xieandrew xieandrew Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Empty line here could be removed

ice/src/test/java/com/altinity/ice/cli/internal/cmd/DescribeParquetTest.java
System.setOut(new PrintStream(outContent));

try {
InMemoryCatalog catalog = new InMemoryCatalog();
Copy link
Collaborator

@xieandrew xieandrew Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

InMemoryCatalog implements Closeable, should this be in a try with resources statement combined with the previous line?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xieandrew xieandrew xieandrew left review comments

@shyiko shyiko Awaiting requested review from shyiko

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Provide convenient introspection of Parquet files

3 participants

@subkanthi @xieandrew