Release Notes
Contents
Release Notes#
v0.9.4 - 2023-01-06#
Fixed bug in
0.9.3
which broke all recipe object imports ifscipy
(an optional dependency) was missing from environment. GH PR 466
v0.9.3 - 2023-01-03#
Warning
Yanked due to presence of a bug. Please use 0.9.4
instead.
Bugfix to allow opening of zarr files. This fix allows using Zarr stores as an input source for recipes. GH PR 462
Add netcdf3 support for opening source files with kerchunk. Resolves a long-standing issue wherein netcdf3 source files could not be loaded lazily, which effectively blocked the use of large netcdf3 files as recipe sources. GH PR 383
Fix zarr reference bug GH PR 455
Add
dataset_type
class attribute for recipe classes GH PR 437
v0.9.2 - 2022-10-26#
Bump peter-evans/slash-command-dispatch from 2 to 3 #434 (@dependabot)
Bump actions/setup-python from 3 to 4 #433 (@dependabot)
Bump LouisBrunner/checks-action from 1.2.0 to 1.5.0 #432 (@dependabot)
Bump codecov/codecov-action from 2.0.2 to 3.1.1 #431 (@dependabot)
Fix CI #430 (@andersy005)
Avoid serialization blues by computing + caching the hash. #429 (@alxmrs)
[pre-commit.ci] pre-commit autoupdate #426 (@pre-commit-ci)
Cancel Redundant CI jobs #424 (@andersy005)
Remove pre-commit action in favor of pre-commit.ci #423 (@andersy005)
Add SciPy video and slides to what_is_pangeo_forge.md #416 (@rabernat)
Updated release_notes.md in preparation for a tag & release. #413 (@alxmrs)
[pre-commit.ci] pre-commit autoupdate #381 (@pre-commit-ci)
Contributors to this release#
(GitHub contributors page for this release)
@alxmrs | @andersy005 | @dependabot | @derekocallaghan | @martindurant | @pre-commit-ci | @rabernat | @yuvipanda
v0.9.1 - 2022-09-08#
Persist Pangeo Forge execution context metadata in target datasets. This information, which includes the
pangeo-forge-recipes
version as well as recipe and input hashes, attaches execution provenance to the dataset itself. GH PR 359File pattern support for
.tiff
s. GH PR 393Improved
HDFReferenceRecipe
by passingtarget_options
to theMultiZarrToZarr
class.Typo fixes, documentation updates, and project health improvements. GH PR 364, GH PR 365, GH PR 366, GH PR 388, GH PR 396, GH PR 394, GH PR 398, GH PR 407.
Fixed
XarrayZarrRecipe
’sfinalize_target()
stage by using bulk delete APIs for consolidating Zarr coordinates.
v0.9 - 2022-05-11#
Breaking changes: Deprecated
XarrayZarrRecipe
manual stage methods. Manual execution can be performed with any of the executors described in Recipe Execution. Also deprecatedFilePattern(..., is_opendap=True)
kwarg, which is superseded byFilePattern(..., file_type="opendap")
. GH PR 362Added
serialization
module along withBaseRecipe.sha256
andFilePattern.sha256
methods. Collectively, this provides for generation of deterministic hashes for both recipe and file pattern instances. Checking these hashes against those from a prior version of the recipe can be used to determine whether or not a particular recipe instance in a Python module (which may contain arbitrary numbers of recipe instances) has changed since the last time the instances in that module were executed. The file pattern hashes are based on a blockchain built cumulatively from all of the index:filepath pairs yielded by the pattern’sself.items()
method. As such, in cases where a new pattern is intended to append to an existing dataset which was built from a prior version of that pattern, the pattern hash can be used to determine the index from which to begin appending. This is demonstrated in the tests. GH PR 349Created new Prefect executor which wraps the Dask executor in a single Task. This should mitigate problems related to large numbers of Prefect Tasks (GH issue 347). See Recipe Execution for details.
Implemented feature to cap cached filename lengths at 255 bytes on local filesystems, to accomodate the POSIX filename length limit. Cached filename lengths are not truncated on any other filesystem. GH PR 353
v0.8.3 - 2022-04-19#
Added
.file_type
attribute topangeo_forge_recipes.patterns.FilePattern
. This attribute will eventually supercede.is_opendap
, which will be deprecated in0.9.0
. Until then,FilePattern(..., is_opendap=True)
is supported as equivalent toFilePattern(..., file_type="opendap")
. GH PR 322
v0.8.2 - 2022-02-23#
Removed click from dependencies and removed cli entrypoint.
v0.8.1 - 2022-02-23#
Fixed dependency issue with pip installation.
Fixed bug where recipes would fail if the target chunks exceeded the full array length. GH issue 279
v0.8.0 - 2022-02-17#
Breaking change: Replace recipe classes’ storage attibutes with
.storage_config
of typepangeo_forge_recipes.storage.StorageConfig
. GH PR 288Add
setup_logging
convenience function. GH PR 287
v0.7.0 - 2022-02-14 ❤️#
Apache Beam executor added. GH issue 169. By Alex Merose.
Index type update. GH PR 257
Fix incompatibility with
fsspec>=2021.11.1
. GH PR 247
v0.6.1 - 2021-10-25#
Major internal refactor of executors. GH PR 219. Began deprecation cycle for recipe methods (e.g.
recipe.prepare_target()
) in favor of module functions.Addition of
open_input_with_fsspec_reference
option onpangeo_forge_recipes.recipes.XarrayZarrRecipe
, permitting the bypassing of h5py when opening inputs. GH PR 218
v0.6.0 - 2021-09-02#
Added
pangeo_forge_recipes.recipes.HDFReferenceRecipe
class to create virtual Zarrs from collections of NetCDF / HDF5 files. GH PR 174Limit output from logging. GH PR 175
Change documentation structure. GH PR 178
Move
fsspec_open_kwargs
andis_opendap
parameters out ofpangeo_forge_recipes.recipes.XarrayZarrRecipe
and intopangeo_forge_recipes.patterns.FilePattern
. Addquery_string_secrets
as attribute ofpangeo_forge_recipes.patterns.FilePattern
. GH PR 167
v0.5.0 - 2021-07-11#
Added
subset_inputs
option topangeo_forge_recipes.recipes.XarrayZarrRecipe
. GH issue 93, GH PR 166Fixed file opening to eliminate HDF errors related to closed files. GH issue 170, GH PR 171
Changed default behavior of executors so that the
cache_input
loop is always run, regardless of the value ofcache_inputs
. GH PR 168
v0.4.0 - 2021-06-25#
Fixed issue with recipe serialilzation causing high memory usage of Dask schedulers and workers when executing recipes with Prefect or Dask GH PR 160.
Added new methods
.to_dask()
,to_prefect()
, and.to_function()
for converting a recipe to one of the Dask, Prefect, or Python execution plans. The previous method,recpie.to_pipelines()
is now deprecated.
v0.3.4 - 2021-05-25#
Added
copy_pruned
method topangeo_forge_recipes.recipes.XarrayZarrRecipe
to facilitate testing.Internal refactor of storage module.
v0.3.3 - 2021-05-10#
Many feature enhancements.
Non-backwards compatible changes to core API.
Package renamed from pangeo_forge
to pangeo_forge_recipes
.
There were problems with packaging for the 0.3.0-0.3.2 releases.
v0.2.0 - 2021-04-26#
First release since major Spring 2021 overhaul. This release depends on Xarray v0.17.1, which has not yet been released as of the date of this release.
v0.1.0 - 2020-10-22#
First release.