🔗 Pipelines

Pipelines group related backup and copy jobs into a single workflow that runs on a schedule.

Why Pipelines?

A typical backup scenario involves multiple steps: back up local files, then copy snapshots to remote storage. These steps are logically related and should run together — not as independent scheduled tasks with manually offset cron times.

Pipeline — the scenario you schedule and run (e.g. "nightly documents backup: local → remote").

Job — one atomic operation inside a pipeline (a backup or copy step).

When a pipeline's cron triggers, all its jobs run sequentially in config order.

Configuration Structure

pipelines:
  - name: string          # Required: Unique pipeline name
    cron: string          # Optional: Cron expression for scheduling
    healthcheck_url: string  # Optional: Healthchecks.io ping URL
    hooks:                # Optional: Lifecycle hooks for the pipeline
      before: []
      success: []
      failure: []
    jobs:                 # Required: List of jobs to run
      - type: backup
        name: string
        # ... backup fields
      - type: copy
        name: string
        # ... copy fields

Required Fields

`name`

Unique identifier for the pipeline. Used in CLI and logs.

name: documents-nightly
name: photos-weekly

`jobs`

List of jobs to execute when the pipeline runs. Job names must be unique within a pipeline.

See Backup Job and Copy Job for job configuration details.

Optional Fields

`cron`

Cron expression for scheduling the entire pipeline.

Format: minute hour day month weekday

Examples:

cron: "0 2 * * *"      # Daily at 2:00 AM
cron: "0 */6 * * *"    # Every 6 hours
cron: "30 3 * * 0"     # Weekly on Sunday at 3:30 AM

To use scheduling:

Set cron expression on the pipeline
Add to system crontab: */5 * * * * crestic cron --config /path/to/crestic.yaml --healthcheck
Crestic tracks per-pipeline state, so system cron can run every 5-30 minutes

See Cron Command for more details.

`healthcheck_url`

Optional Healthchecks.io ping URL for this pipeline. Match the check schedule to the pipeline cron. Pings are sent only when --healthcheck is passed.

healthcheck_url: https://hc-ping.com/01234567-89ab-cdef-0123-456789abcdef

`hooks`

Optional lifecycle hooks that wrap the entire pipeline run (before, success, failure). See Hooks for details and environment variables (CRESTIC_PIPELINE_NAME, CRESTIC_ERROR).

hooks:
  before:
    - echo "Starting pipeline..."
  success:
    - echo "Pipeline done"
  failure:
    - echo "Pipeline failed: $CRESTIC_ERROR" >&2

Complete Example

pipelines:
  - name: documents-nightly
    cron: "0 2 * * *"
    healthcheck_url: https://hc-ping.com/01234567-89ab-cdef-0123-456789abcdef
    hooks:
      before:
        - echo "Starting pipeline..."
      failure:
        - echo "Pipeline failed: $CRESTIC_ERROR" >&2
    jobs:
      - type: backup
        name: local-backup
        from:
          - /home/user/Documents
        to: local-repo
        options:
          tag: [documents]
 
      - type: copy
        name: offsite-copy
        from: local-repo
        to: remote-repo
        options:
          tag: [documents]
 
repositories:
  local-repo:
    path: /backup/restic
    password_command: "pass show restic/local"
  remote-repo:
    path: rclone:backblaze:backup/restic
    password_command: "pass show restic/remote"

Running Pipelines

Run Entire Pipeline

crestic backup --pipeline documents-nightly

Run Specific Job

Jobs are referenced by qualified name: pipeline/job

crestic backup --job documents-nightly/local-backup
crestic backup --job documents-nightly/offsite-copy

Run Multiple Jobs

crestic backup --job documents-nightly/local-backup,photos-weekly/backup

Run All Jobs from All Pipelines

crestic backup --all

Migration from Flat Jobs

The top-level jobs: key has been replaced by pipelines: with nested jobs.

Before:

jobs:
  - type: backup
    name: documents-backup
    cron: "0 2 * * *"
    from: [/home/user/Documents]
    to: local-repo
 
  - type: copy
    name: documents-copy-to-remote
    cron: "0 3 * * *"
    from: local-repo
    to: remote-repo

After:

pipelines:
  - name: documents-nightly
    cron: "0 2 * * *"
    jobs:
      - type: backup
        name: local-backup
        from: [/home/user/Documents]
        to: local-repo
 
      - type: copy
        name: offsite-copy
        from: local-repo
        to: remote-repo

Key changes:

cron moves from individual jobs to the pipeline
Jobs are nested under pipelines
CLI uses --pipeline or --job pipeline/job instead of --job job-name
CRESTIC_JOB_NAME in hooks is now the qualified name (documents-nightly/local-backup)