Core Concepts#

When making a Recipe Contribution to Pangeo Forge Cloud, familiarity with the following core concepts is recommended.

Catalog#

The analysis-ready, cloud optimized (ARCO) datasets produced by Recipes executed on Pangeo Forge Cloud are registered in the Pangeo Forge Catalog:

Datasets in this catalog are built from recipes stored in Feedstocks.

Feedstocks#

Once a Recipe Contribution passes the required checks, it is automatically moved into a new standalone GitHub repository within the pangeo-forge Github organization. These repositories, which store production recipes, are known as Feedstocks (a concept borrowed directly from Conda Forge).

A current listing of all Pangeo Forge Feedstocks is available at:

All datasets in the Catalog are built from recipes stored in a Feedstock repo. As such, Feedstocks serve as a record of the exact provenance of all ARCO data produced by Pangeo Forge Cloud.

If corrections or improvements are required on a dataset in the Catalog, these can be implementeded via Pull Requests against the associated Feedstock repo. Each time a Pull Request is merged into the default branch of a Feedstock repo, a new production build of the recipes in that Feedstock is initiated.

Each build of the recipes in a Feedstock repo is tracked by a Recipe Run.

Recipe Runs#

Recipe Runs are simple metadata objects used by Pangeo Forge Cloud to track information about a particular recipe build.

A current listing of all Recipe Runs is available at:

Every Recipe Run records metadata about a specific recipe execution job on a Bakery.

Bakeries#

Bakeries are the cloud compute infrastructure that execute recipes contributed to Pangeo Forge Cloud. Bakeries use Prefect to orchestrate the various stages of recipe execution. Each Bakery is coupled to one or more cloud storage buckets where the ARCO data produced by recipes is stored.

A listing of current Bakery deployments is available at:

Bakery implementation templates are availble for all three major cloud providers:

Bakery operation entails non-trivial financial and maintenance cost, and is therefore typically undertaken by businesses, organizations, and/or institutions with dedicated funding for this purpose. If you are interested in deploying your own bakery, please consult the above-listed repos for more information.