Command Line Interface

Command Line Interface#

Prerequisites#

Using the CLI to deploy a recipe assumes the following prerequisites:

  1. The CLI is installed.

  2. The recipe file resides with a feedstock.

  3. A Configuration file is available.

With these prerequistes complete, the CLI can be invoked to deploy a recipe.

Configuration file#

Deployment requires a configuration file which can be provided as a Python or JSON file, e.g.:

# mypy: disable-error-code="name-defined"

c.Bake.bakery_class = "pangeo_forge_runner.bakery.local.LocalDirectBakery"

c.TargetStorage.fsspec_class = "fsspec.implementations.local.LocalFileSystem"
c.TargetStorage.root_path = "./target"

c.InputCacheStorage.fsspec_class = "fsspec.implementations.local.LocalFileSystem"
c.InputCacheStorage.root_path = "./cache"

{
  "Bake": {
    "bakery_class": "pangeo_forge_runner.bakery.local.LocalDirectBakery"
  },
  "TargetStorage": {
    "fsspec_class": "fsspec.implementations.local.LocalFileSystem",
    "root_path": "./target"
  },
  "InputCacheStorage": {
    "fsspec_class": "fsspec.implementations.local.LocalFileSystem",
    "root_path": "./cache"
  }
}

This file centralizes configuration of:

For a full listing of available configuration, see: Configuration Reference.

Invocation#

A recipe in your feedstock can be deployed with the following command:

pangeo-forge-runner bake --repo=$REPO -f=$CONFIG_FILE --Bake.recipe_id=$RECIPE_ID --Bake.job_name=$JOB_NAME --prune

Where the variables have the following values assigned to them:

  • REPO: Path to the feedstock repo. This can be a local path or a URL to a GitHub repo.

  • CONFIG_FILE: Local path to the deployment Configuration file.

  • RECIPE_ID: The id of the recipe you’d like to run as it appears in your feedstock’s meta.yaml.

  • JOB_NAME: A unique name for this deployment.

Tip

The --prune flag limits the deployment to the first two increments of the dataset’s concat dimension. This is useful for testing your deployment on a limited subset of data. To deploy a full production run, simply omit this flag.

Contents