dev: add make notebook

[kevinjqliu]

Rationale for this change

Add 2 new Make command (make notebook) to spin up a jupyter notebook; make notebook-infra spins up jupyter notebook along with integration test infrastructure.

Pyiceberg Example Notebook

Pyiceberg example notebook (notebooks/pyiceberg_example.ipynb) is based on the https://py.iceberg.apache.org/#getting-started-with-pyiceberg page and doesn't require additional test infra.

Spark Example Notebook

Spark integration example notebook (notebooks/spark_integration_example.ipynb) is based on https://iceberg.apache.org/docs/nightly/spark-getting-started/ and requires integration test infrastructure (Spark, IRC, S3)

With spark connect (#2491) and our testing setup, we can quickly spin up a local env with make test-integration-exec which includes:

• spark
• iceberg rest catalog
• hive metastore
• minio

In the jupyter notebook, connect to spark easily

from pyspark.sql import SparkSession

# Create SparkSession against the remote Spark Connect server
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
spark.sql("SHOW CATALOGS").show()

Are these changes tested?

Yes, run both make notebook and make notebook-infra locally and run the example notebooks

Are there any user-facing changes?

[Fokko]

Should we move this into a poetry dependency group? Similar to the docs.

[Fokko]

With spark connect (#2491) and our testing setup, we can quickly spin up a local env with

I agree, and that's great, but should we also spin up the resources as part of this effort? We could even inject a notebook that imports Spark-connect, etc (which won't be installed from a fresh install? I think this is a dev dependency, we probably want to double check there to avoid scaring newcomers to the project).

[jayceslesar]

Bonus idea: what if make notebook or some other CLI entry point spun up pyspark + catalog configured via pyiceberg.yaml so users could immediately start querying their data?

[kevinjqliu]

We could even inject a notebook that imports Spark-connect

We could do getting started as a notebook! https://py.iceberg.apache.org/#getting-started-with-pyiceberg

[kevinjqliu]

Bonus idea: what if make notebook or some other CLI entry point spun up pyspark + catalog configured via pyiceberg.yaml so users could immediately start querying their data?

yea we could do that. the integration test setup gives us 2 different catalogs (rest and hms)

[Fokko]

@kevinjqliu I would keep it simple, and go with the preferred catalog; REST :)

[kevinjqliu]

adding the make install pre-req here because otherwise dev/provision.py will fail

[kevinjqliu]

Render:

[kevinjqliu]

Render:

[Fokko]

We can also add a .pre-commit linter for notebooks: https://github.com/nbQA-dev/nbQA

[Fokko]

Looks great! Should we also add a section to the docs?

[Fokko]

We can do this, but then we have to make sure that they are not bundled in the release. The notebooks do contain code.

[kevinjqliu]

yea agreed. i double check the artifacts, the new notebooks/ dir is not included. Similar to how the mkdocs/ dir is not included.

Feels like this can be a potential footgun where a folder is included in the artifact but RAT check is ignored in .rat-excludes. I think we can add a CI check to prevent this. I'll track this as a separate issue

[kevinjqliu]

#2888

[kevinjqliu]

Added to the frontpage, https://py.iceberg.apache.org/

[kevinjqliu]

Added a new section to https://py.iceberg.apache.org/contributing/

[kevinjqliu]

Added some docs and linter for notebook

[geruh]

LGTM! This is awesome Kevin! Was able to get it up and running locally with no issues

[kevinjqliu]

Merging this! We can add more improvements as follow up