Transforms#
Once you have a file pattern for your source data, it’s time to define a set of transforms to apply to the data, which may include:
Standard transforms from Apache Beam’s Python transform catalog
pangeo-forge-recipes
core transforms, such as Openers and WritersThird-party extensions from the Pangeo Forge Ecosystem
Your own transforms, such as custom Preprocessors
Hint
Please refer to the Generic sequence and Common styles for discussion of how transforms are commonly connected together; Example recipes provides representative examples.
⚙️ Deploy-time configurable keyword arguments
Keyword arguments designated by the gear emoji ⚙️ below are deploy-time configurable. They should therefore not be provided in your recipe file. Instead, values for these arguments are specified in a per-deployment Configuration file. The values provided in the configuration file will be injected into your recipe by the Command Line Interface.
Openers#
Once you’ve created a file pattern for your source data, you’ll need to open it somehow. Pangeo Forge currently provides the following openers:
Preprocessors#
Before writing out your analysis-ready, cloud-optimized (ARCO) dataset, it’s possible
you may want to preprocess the data. A custom Apache Beam PTransform
can be written
for this purpose and included in your recipe.
# TODO: Add preprocessor example.
Writers#
What’s next#
Once your recipe is defined, you’re ready to move on to Deployment.