Transforms

Once you have a file pattern for your source data, it’s time to define a set of transforms to apply to the data, which may include:

• Standard transforms from Apache Beam’s Python transform catalog

• pangeo-forge-recipes core transforms, such as Openers and Writers

• Third-party extensions from the Pangeo Forge Ecosystem

• Your own transforms, such as custom Preprocessors

Hint

Please refer to the Generic sequence and Common styles for discussion of how transforms are commonly connected together; Example recipes provides representative examples.

⚙️ Deploy-time configurable keyword arguments

Keyword arguments designated by the gear emoji ⚙️ below are deploy-time configurable. They should therefore not be provided in your recipe file. Instead, values for these arguments are specified in a per-deployment Configuration file. The values provided in the configuration file will be injected into your recipe by the Command Line Interface.

Openers

Once you’ve created a file pattern for your source data, you’ll need to open it somehow. Pangeo Forge currently provides the following openers:

• pangeo_forge_recipes.transforms.OpenURLWithFSSpec

  • ⚙️ cache - Deploy-time configurable keyword argument

• pangeo_forge_recipes.transforms.OpenWithXarray

• pangeo_forge_recipes.transforms.OpenWithKerchunk

Preprocessors

Before writing out your analysis-ready, cloud-optimized (ARCO) dataset, it’s possible you may want to preprocess the data. A custom Apache Beam PTransform can be written for this purpose and included in your recipe.

# TODO: Add preprocessor example.

Writers

• pangeo_forge_recipes.transforms.StoreToZarr

  • ⚙️ target_root - Deploy-time configurable keyword argument

• pangeo_forge_recipes.transforms.WriteCombinedReference

  • ⚙️ target_root - Deploy-time configurable keyword argument

What’s next

Once your recipe is defined, you’re ready to move on to Deployment.