0.8.0

Version 0.8.0 (from 2024-10-04)

• Added module zappend.contrib that contributes functions to zappend's core functionality.

• Added experimental function zappend.contrib.write_levels() that generates datasets using the multi-level dataset format as specified by xcube.
  It resembles the store.write_data(cube, "<name>.levels", ...) method provided by the xcube filesystem data stores ("file", "s3", "memory", etc.). The zappend version may be used for potentially very large datasets in terms of dimension sizes or for datasets with very large number of chunks. It is considerably slower than the xcube version (which basically uses xarray.to_zarr() for each resolution level), but should run robustly with stable memory consumption. The function requires xcube package to be installed. (#19)

• Updated documentation

New Contributors

• @clarasb made their first contribution in #100

Full Changelog: v0.7.1...v0.8.0

0.8.0.dev1

Full Changelog: v0.8.0.dev0...v0.8.0.dev1

0.8.0.dev0

What's Changed

• Added experimental function zappend.levels.write_levels() that generates datasets using the multi-level dataset format as specified by xcube. It resembles the store.write_data(cube, "<name>.levels", ...) method provided by the xcube filesystem data stores ("file", "s3", "memory", etc.). The zappend version may be used for potentially very large datasets in terms of dimension sizes or for datasets with very large number of chunks. It is considerably slower than the xcube version (which basically uses xarray.to_zarr() for each resolution level), but should run robustly with stable memory consumption. The function requires xcube package to be installed. (#19)

• Update documentation by @clarasb in #100

New Contributors

• @clarasb made their first contribution in #100

Full Changelog: v0.7.1...v0.8.0.dev0

0.7.1

Version 0.7.1

• The function zappend.api.zappend() now returns the number of slices processed. (#93)
• Moved all project configuration to pyproject.toml and removed setup.cfg and requirements files. (#88)
• Added new section How do I ... to the documentation. (#66)
• Fixed link to slice sources in documentation main page.
• Fixed broken CI. (#97)

Full Changelog: v0.7.0...v0.7.1

0.7.0

Enhancements

• Made writing custom slice sources easier and more flexible: (#82)

  • Slice items can now be a contextlib.AbstractContextManager so custom slice functions can now be used with @contextlib.contextmanager.
  • Introduced SliceSource.close() so contextlib.closing()
    is applicable. Deprecated SliceSource.dispose().
  • Introduced new optional configuration setting slice_source_kwargs that contains keyword-arguments passed to a configured slice_source together with each slice item.
  • Introduced optional configuration setting extra that holds additional configuration not validated by default. Intended use is by a slice_source that expects an argument named ctx and therefore can access the configuration.

• Improved readability of the configuration reference by using setting categories and applied logical ordering of settings within categories (#85)

Full Changelog: v0.6.0...v0.7.0

0.6.0

Enhancements

• Added configuration setting force_new, which forces creation of a new
  target dataset. An existing target dataset (and its lock) will be
  permanently deleted before appending of slice datasets begins. (#72)

• Chunk sizes can now be null for a given dimension. In this case the actual
  chunk size used is the size of the array's shape in that dimension. (#77)

API Changes

• Simplified writing of custom slice sources for users. The configuration setting
  slice_source can now be a SliceSource class or any function that returns a
  slice item: a local file path or URI, an xarray.Dataset,
  a SliceSource object.
  Dropped concept of slice factories entirely, including functions
  to_slice_factory() and to_slice_factories(). (#78)

• Extracted Config class out of Context and made available via new
  Context.config: Config property. The change concerns any usages of the
  ctx: Context argument passed to user slice factories. (#74)

Full Changelog: v0.5.1...v0.6.0

0.5.1

Fixed rollback for situations where writing to Zarr fails shortly after the Zarr directory has been created. [#69]
In this case the error message was TypeError: Transaction._delete_dir() missing 1 required positional argument: 'target_path'.

Full Changelog: v0.5.0...v0.5.1

0.5.0

Enhancements

• The configuration setting attrs can now be used to define dynamically
  computed dataset attributes using the syntax {{ expression }}. [#60]

  Example:

  permit_eval: true
  attrs:
    title: HROC Ocean Colour Monthly Composite
    time_coverage_start: {{ lower_bound(ds.time) }}
    time_coverage_end: {{ upper_bound(ds.time) }}

• Introduced new configuration setting attrs_update_mode that controls
  how dataset attributes are updated. [#59]

• Simplified logging to console. You can now set configuration setting
  logging to a log level which will implicitly enable console logging with
  given log level. [#64]

• Added a section in the notebook examples/zappend-demo.ipynb
  that demonstrates transaction rollbacks.

• Added CLI option --traceback. [#57]

• Added a section in the notebook examples/zappend-demo.ipynb
  that demonstrates transaction rollbacks.

Fixes

• Fixed issue where a NetCDF package was missing to run the
  demo Notebook examples/zappend-demo.ipynb in
  Binder. [#47]

Full Changelog: v0.4.1...v0.5.0

0.4.1

Fixes

• Global metadata attributes of target dataset is no longer empty. [#56]

• If the target parent directory did not exist, an exception was raised reporting that the lock file to be written does not exist. Changed this to report that the target parent directory does not exist. [#55]

Enhancements

• Added missing documentation of the append_step setting in the configuration reference.

Full Changelog: v0.4.0...v0.4.1

0.4.0

Enhancements

• A new configuration setting append_step can be used to validate
  the step sizes between the labels of a coordinate variable associated with
  the append dimension. Its value can be a number for numerical labels
  or a time delta value of the form 8h (8 hours) or 2D (two days) for
  date/time labels. The value can also be negative. [#21]

• The configuration setting append_step can take the special values
  "+" and "-" which are used to verify that the labels are monotonically
  increasing and decreasing, respectively. [#20]

• It is now possible to reference environment variables
  in configuration files using the syntax ${ENV_VAR}. [#36]

• Added a demo Notebook exmaples/zappend-demo.ipynb and linked
  it by a binder badge in README.md. [#47]

Fixes

• When slice_source was given as class or function and passed
  to the zappend() function either as configuration entry or as keyword
  argument, a ValidationError was accidentally raised. [#49]

• Fixed an issue where an absolute lock file path was computed if the target
  Zarr path was relative in the local filesystem, and had no parent directory.
  [#45]

Full Changelog: v0.3.0...v0.4.0