Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,11 @@ jobs:

- name: Install dependencies
run: |
uv pip install --system -e ".[dev]"
uv sync --all-groups

- name: Run tests
run: |
pytest
uv run pytest

test-package-install:
name: Test Package Installation
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish-stac.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Publish STAC
name: Publish STAC to GH-Pages

on:
# Run daily at 6 AM UTC
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/staging.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,15 +36,15 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: stac-catalog
path: build-artifact
path: public_releases

deploy:
name: Deploy
runs-on: ubuntu-latest
needs: build
environment:
name: staging
url: https://staging.overturemaps.org/${{github.repository}}/pr/${{ github.event.number }}/index.html
url: https://staging.overturemaps.org/${{ github.event.repository.name }}/pr/${{ github.event.number }}/catalog.json

steps:
- name: Configure AWS credentials 🔐
Expand All @@ -56,11 +56,11 @@ jobs:
- name: Download artifacts 📥
uses: actions/download-artifact@v4
with:
name: build-artifact
name: stac-catalog

- name: Copy to S3
run: |
aws s3 cp --recursive . s3://overture-managed-staging-usw2/gh-pages/${{ github.repository }}/pr/${{ github.event.number }}/
aws s3 sync --delete . s3://overture-managed-staging-usw2/gh-pages/${{ github.event.repository.name }}/pr/${{ github.event.number }}/

- name: Bust the Cache
run: aws cloudfront create-invalidation --distribution-id E1KP2IN0H2RGGT --paths "/${{ github.repository }}/pr/${{ github.event.number }}/*"
run: aws cloudfront create-invalidation --distribution-id E1KP2IN0H2RGGT --paths "/${{ github.event.repository.name }}/pr/${{ github.event.number }}/*"
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -77,3 +77,5 @@ output/
*.swp
*.swo
*~

tests/data
41 changes: 12 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,47 +4,30 @@
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Generate STAC (SpatioTemporal Asset Catalog) catalogs for all public Overture Maps releases.
Generate STAC catalogs for all public Overture Maps releases.

See it in action here:
<https://radiantearth.github.io/stac-browser/#/external/labs.overturemaps.org/stac/catalog.json?.language=en>
**[Browse the catalog](https://radiantearth.github.io/stac-browser/#/external/labs.overturemaps.org/stac/catalog.json?.language=en)**

### Installing/Updating Dependencies
## Setup

```bash
# Install package in editable mode with dev dependencies
uv pip install -e ".[dev]"

# Install just the package (no dev dependencies)
uv pip install -e .

# Update dependencies
uv pip install --upgrade -e ".[dev]"

# Add a new dependency (manually edit pyproject.toml, then):
uv pip install -e ".[dev]"
uv sync
```

### Running the Application
## Usage

```bash
# Run the STAC generator (parallel mode with 4 workers by default)
gen-stac --output ./public_releases

# Run in debug mode (generates only 1 item per collection)
gen-stac --output ./public_releases --debug
gen-stac --output ./releases

# Control parallelization
gen-stac --output ./public_releases --workers 8 # Use 8 parallel workers
gen-stac --output ./public_releases --no-parallel # Disable parallelization
# Debug mode (2 items per collection)
gen-stac --output ./releases --debug

# Recommended for production (balance speed and resource usage)
gen-stac --output ./public_releases --workers 4
# Custom worker count (default: 4)
gen-stac --output ./releases --workers 8
```

### Before Committing
## Development

```bash
# Run the full CI check locally
ruff format . && ruff check . && pytest
uv run ruff format . && uv run ruff check . && uv run pytest
```
10 changes: 7 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "overture-stac"
version = "1.0.2"
version = "1.0.3"
description = "Generate STAC catalogs for Overture Maps Releases"
authors = [
{name = "Overture Maps Foundation"}
Expand All @@ -15,7 +15,7 @@ dependencies = [
"pyyaml>=6.0.2",
]

[project.optional-dependencies]
[dependency-groups]
dev = [
"pytest>=7.4.0",
"ruff>=0.1.0",
Expand All @@ -36,11 +36,15 @@ testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = ["--verbose"]
addopts = ["--verbose", "-W", "default"]
markers = [
"integration: marks tests as integration tests (connects to real services)",
"slow: marks tests as slow (deselect with '-m \"not slow\"')",
]
filterwarnings = [
"default",
"ignore::DeprecationWarning:pystac.*",
]

[tool.ruff]
line-length = 88
Expand Down
24 changes: 9 additions & 15 deletions src/overture_stac/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,6 @@ def main():
help="Debug flag to only generate 1 item per collection",
)

parser.add_argument(
"--no-parallel",
dest="parallel",
action="store_false",
help="Disable parallel processing (default: parallel enabled)",
)

parser.add_argument(
"--workers",
type=int,
Expand All @@ -47,15 +40,16 @@ def main():
args = parser.parse_args()

filesystem = fs.S3FileSystem(anonymous=True, region="us-west-2")
public_releases = filesystem.get_file_info(
available_releases = filesystem.get_file_info(
fs.FileSelector("overturemaps-us-west-2/release")
)

# TODO: These should be stored elsewhere, but for now we'll hardcode them here
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If schema versions are available in new STAC catalogs, do we need to maintain this mapping? We could read from the published catalog and only require schema for new releases.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about this — but then we'd always need to be passing in the new schema version.

I don't think that any information surfaced from STAC should be anything new — it should just be a convenient index into the Overture Release and artifacts. I think we need to tie the schema version to the data somewhere else in the pipeline so that it's discoverable.

schema_version_mapping = {
schema_version_mapping: dict[str, str] = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may also need to change this when instantiating OvertureRelease

Suggested change
schema_version_mapping: dict[str, str] = {
SCHEMA_VERSION_MAPPING: dict[str, str] = {

"2026-03-18.0": "TBD",
"2026-02-18.0": "1.15.0",
"2026-01-21.0": "1.15.0",
"2025-12-17.0": "1.15.0",
"2025-11-19.0": "1.14.0",
}

overture_releases_catalog = pystac.Catalog(
Expand All @@ -67,11 +61,13 @@ def main():
output.mkdir(parents=True, exist_ok=True)

for idx, release_info in enumerate(
sorted(public_releases, key=lambda p: p.path, reverse=True)
sorted(available_releases, key=lambda p: p.path, reverse=True)
):
release = release_info.path.split("/")[-1]

title = f"{release} Overture Release" if idx > 0 else "Latest Overture Release"
title: str = (
f"{release} Overture Release" if idx > 0 else "Latest Overture Release"
)

this_release = OvertureRelease(
release=release,
Expand All @@ -80,9 +76,7 @@ def main():
debug=args.debug,
)

this_release.build_release_catalog(
title=title, parallel=args.parallel, max_workers=args.workers
)
this_release.build_release_catalog(title=title, max_workers=args.workers)

child = overture_releases_catalog.add_child(
child=this_release.release_catalog, title=title
Expand Down
Loading