Release Notes

v0.9.1 - 2022-09-08

• Persist Pangeo Forge execution context metadata in target datasets. This information, which includes the pangeo-forge-recipes version as well as recipe and input hashes, attaches execution provenance to the dataset itself. GH PR 359

• File pattern support for .tiffs. GH PR 393

• Improved HDFReferenceRecipe by passing target_options to the MultiZarrToZarr class.

• Typo fixes, documentation updates, and project health improvements. GH PR 364, GH PR 365, GH PR 366, GH PR 388, GH PR 396, GH PR 394, GH PR 398, GH PR 407.

• Fixed XarrayZarrRecipe’s finalize_target() stage by using bulk delete APIs for consolidating Zarr coordinates.

v0.9 - 2022-05-11

• Breaking changes: Deprecated XarrayZarrRecipe manual stage methods. Manual execution can be performed with any of the executors described in Recipe Execution. Also deprecated FilePattern(..., is_opendap=True) kwarg, which is superseded by FilePattern(..., file_type="opendap"). GH PR 362

• Added serialization module along with BaseRecipe.sha256 and FilePattern.sha256 methods. Collectively, this provides for generation of deterministic hashes for both recipe and file pattern instances. Checking these hashes against those from a prior version of the recipe can be used to determine whether or not a particular recipe instance in a Python module (which may contain arbitrary numbers of recipe instances) has changed since the last time the instances in that module were executed. The file pattern hashes are based on a blockchain built cumulatively from all of the index:filepath pairs yielded by the pattern’s self.items() method. As such, in cases where a new pattern is intended to append to an existing dataset which was built from a prior version of that pattern, the pattern hash can be used to determine the index from which to begin appending. This is demonstrated in the tests. GH PR 349

• Created new Prefect executor which wraps the Dask executor in a single Task. This should mitigate problems related to large numbers of Prefect Tasks (GH issue 347). See Recipe Execution for details.

• Implemented feature to cap cached filename lengths at 255 bytes on local filesystems, to accomodate the POSIX filename length limit. Cached filename lengths are not truncated on any other filesystem. GH PR 353

v0.8.3 - 2022-04-19

• Added .file_type attribute to pangeo_forge_recipes.patterns.FilePattern. This attribute will eventually supercede .is_opendap, which will be deprecated in 0.9.0. Until then, FilePattern(..., is_opendap=True) is supported as equivalent to FilePattern(..., file_type="opendap"). GH PR 322

v0.8.2 - 2022-02-23

• Removed click from dependencies and removed cli entrypoint.

v0.8.1 - 2022-02-23

• Fixed dependency issue with pip installation.

• Fixed bug where recipes would fail if the target chunks exceeded the full array length. GH issue 279

v0.8.0 - 2022-02-17

• Breaking change: Replace recipe classes’ storage attibutes with .storage_config of type pangeo_forge_recipes.storage.StorageConfig. GH PR 288

• Add setup_logging convenience function. GH PR 287

v0.7.0 - 2022-02-14 ❤️

• Apache Beam executor added. GH issue 169. By Alex Merose.

• Dask executor updates. GH PR 260 GH PR 261

• Index type update. GH PR 257

• Fix incompatibility with fsspec>=2021.11.1. GH PR 247

v0.6.1 - 2021-10-25

• Major internal refactor of executors. GH PR 219. Began deprecation cycle for recipe methods (e.g. recipe.prepare_target()) in favor of module functions.

• Addition of open_input_with_fsspec_reference option on pangeo_forge_recipes.recipes.XarrayZarrRecipe, permitting the bypassing of h5py when opening inputs. GH PR 218

v0.6.0 - 2021-09-02

• Added pangeo_forge_recipes.recipes.HDFReferenceRecipe class to create virtual Zarrs from collections of NetCDF / HDF5 files. GH PR 174

• Limit output from logging. GH PR 175

• Change documentation structure. GH PR 178

• Move fsspec_open_kwargs and is_opendap parameters out of pangeo_forge_recipes.recipes.XarrayZarrRecipe and into pangeo_forge_recipes.patterns.FilePattern. Add query_string_secrets as attribute of pangeo_forge_recipes.patterns.FilePattern. GH PR 167

v0.5.0 - 2021-07-11

• Added subset_inputs option to pangeo_forge_recipes.recipes.XarrayZarrRecipe. GH issue 93, GH PR 166

• Fixed file opening to eliminate HDF errors related to closed files. GH issue 170, GH PR 171

• Changed default behavior of executors so that the cache_input loop is always run, regardless of the value of cache_inputs. GH PR 168

v0.4.0 - 2021-06-25

• Fixed issue with recipe serialilzation causing high memory usage of Dask schedulers and workers when executing recipes with Prefect or Dask GH PR 160.

• Added new methods .to_dask(), to_prefect(), and .to_function() for converting a recipe to one of the Dask, Prefect, or Python execution plans. The previous method, recpie.to_pipelines() is now deprecated.

v0.3.4 - 2021-05-25

• Added copy_pruned method to pangeo_forge_recipes.recipes.XarrayZarrRecipe to facilitate testing.

• Internal refactor of storage module.

v0.3.3 - 2021-05-10

Many feature enhancements. Non-backwards compatible changes to core API. Package renamed from pangeo_forge to pangeo_forge_recipes.

There were problems with packaging for the 0.3.0-0.3.2 releases.

v0.2.0 - 2021-04-26

First release since major Spring 2021 overhaul. This release depends on Xarray v0.17.1, which has not yet been released as of the date of this release.

v0.1.0 - 2020-10-22

First release.