fovus pipeline create

fovus pipeline create

Create pipeline.

Local workflow host creates a pipeline record only. You run the workflow on your workstation; PIPELINE_DIRECTORY is optional and is not used.

Remote workflow host creates the pipeline and starts execution on Fovus-hosted compute (configuration required). Provide PIPELINE_DIRECTORY as your local project folder; the folder is created if it does not exist yet. Files are uploaded to remote storage and the pipeline id is saved for that directory after a successful create. If a pipeline id is already stored for the directory, you must confirm before it is replaced unless you pass –override or run with –silence (non-interactive), which replaces it by default.

Use -c/–config for a pipeline config JSON. CLI options override config values.

Usage

fovus pipeline create [OPTIONS] [PIPELINE_DIRECTORY]

Options

--pipeline-directory <legacy_pipeline_directory>

Deprecated: use the PIPELINE_DIRECTORY positional argument instead.

-c, --config <config_path>

Path to pipeline config JSON. Can specify workflowHost and all create arguments; CLI options override config.

--workflow-host <workflow_host>

Where the workflow runs: local (on your workstation) or remote (on Fovus-hosted compute). Default: remote, or from config workflowHost if -c is used.

Options:

local | remote

--workflow-manager <workflow_manager>

Engine or language that orchestrates and manages job submissions during a pipeline. Use custom when you orchestrate the pipeline yourself on Fovus.

Options:

nextflow | mini_wdl | custom

--name <name>

The pipeline name. Overrides config if provided.

--project-name <project_name>

The project name associated with your pipeline. If omitted, the default project will be used. Use ‘None’ to specify no project.

--auto-delete-days <auto_delete_days>

Number of days after which the pipeline will be permanently dBeleted. Only applicable when workflow host is remote.

--setup-command <setup_command>

Setup command for remote execution (e.g. git clone). Overrides config if provided.

--run-command <run_command>

Run command for remote execution (e.g. nextflow run nf-core/rnaseq). Overrides config if provided.

--workflow-manager-version <WORKFLOW_MANAGER_VERSION>

Optional workflow manager version to set on the pipeline.

--storage-connectors <storage_connectors_tuple>

The name(s) of the storage connectors this pipeline may access. A storage connector gives both the pipeline host (the machine running pipeline orchestration, e.g. Nextflow) and the pipeline’s jobs access to a storage location outside Fovus storage, set up for your cost center in your workspace settings. This is what lets pipeline steps like a Nextflow publishDir write to an external bucket. Names may contain only letters, digits, and hyphens. Whether a connector exists and whether you may use it is decided by Fovus when the pipeline is created. Defaults to no connectors. Only applies to remote workflow host; ignored (with a warning) for local. Repeat the option for multiple connectors.

--include-paths <include_paths>

Include only these paths when uploading pipeline directory (remote workflow host). Relative paths inside PIPELINE_DIRECTORY. Use * and ? for wildcards. Only one of –include-paths or –exclude-paths.

--exclude-paths <exclude_paths>

Exclude these paths when uploading pipeline directory (remote workflow host). Relative paths inside PIPELINE_DIRECTORY. Use * and ? for wildcards. Only one of –include-paths or –exclude-paths.

--capacity-reservation <capacity_reservations>

Run this pipeline on reserved capacity. Provide a capacity reservation name or reservation ID, optionally followed by =NODE_COUNT (e.g. my-reservation=3). Several reservations may be given as a comma-separated list; write a literal comma in a name as ‘,,’. A name that ends in =<number> is read as a count, so give that reservation by ID. Without a count, every machine still available in the reservation is committed. Reservations of different machine types, and reservations in different regions, may be combined: tasks are reallocated across the queues of every reservation given. The pipeline holds the nodes from creation until it is deleted – across every stage, between them, and while it is stopped, so that a stopped pipeline can be resumed – and every job the pipeline creates runs on them. Remote workflow host only.

--override

Replace the existing pipeline id for the pipeline directory without prompting (remote workflow host only).

Arguments

PIPELINE_DIRECTORY

Optional argument