run-analyses Guide

This guide used to document a run-validation command. That command no longer exists — it was replaced by run-analyses (cli/run_analyses.py), which has a different config layout and CLI. This page now documents the current tool; the title and URL are unchanged.

Overview

run-analyses is the master orchestrator. It calls each step’s own CLI entry point as a subprocess, reusing one YAML config per step (config/run_analyses.yaml maps step names to those per-step config files).

Use it when you want to run several validation or scenario steps for one area/experiment in one command. For individual analysis types — and for anything that needs --source/--model/--scenario (NSe’s workflow) — run the dedicated scripts directly (e.g. argo-profiles, tidal-analysis); run-analyses does not forward those flags, only --area, --experiment, --years, and --analyses-dir.

Config structure

# config/run_analyses.yaml — maps step names to their per-step YAML config

validations:
  gridded_2d:          config/gridded_2d_validation.yaml
  gridded_3d:          config/gridded_3d_validation.yaml
  tidal:               config/tidal_analysis.yaml
  fixed_platform:      config/fixed_platform.yaml
  station_timeseries:  config/station_timeseries.yaml
  cruise_ctd:          config/cruise_ctd_profiles.yaml
  argo:                config/argo_profiles.yaml
  glodap:              config/glodap_profiles.yaml
  wod:                 config/wod_profiles.yaml
  ices:                config/ices_profiles.yaml

scenarios:
  single_scenario:     config/single_scenario.yaml
  multi_scenario:      config/multi_scenario.yaml

Per-step extra flags can be added with a dict instead of a plain path:

validations:
  tidal:
    config: config/tidal_analysis.yaml
    args:   [--layers, surface, bottom]   # step-specific extra flags

Omit any step that does not apply to this project.

Usage

# Run all validation steps
run-analyses validations --area NS --experiment Baseline

# Override years across every step
run-analyses validations --area NS --experiment Baseline --years 2010 2022

# Run a single step (useful for debugging or re-running one analysis)
run-analyses validations --area NS --experiment Baseline --step tidal

# Skip one step, run the rest
run-analyses validations --area NS --experiment Baseline --no-tidal

# Scenario steps
run-analyses scenarios --area NS --experiment Baseline

# List all recognised step names
run-analyses --list-steps

# Preview without running anything
run-analyses validations --area NS --experiment Baseline --dryrun

--analyses-dir overrides output.analyses_dir in every step’s config; as with the individual scripts, there is no ./analyses fallback — see Tidal Analysis: analyses_dir has no default.

Step names

Validation steps (run order):

Scenario steps (run order): single_scenario (single-scenario), multi_scenario (multi-scenario).

Each step can be skipped with --no-<step-name-with-dashes>, e.g. --no-gridded-2d, --no-fixed-platform.

Error handling

Each step runs independently. If one step fails it is logged and the orchestrator continues with the remaining steps, unless --stop-on-error is given. Failed steps are listed in the final summary. Exit code is non-zero if any step failed.

Partial runs with –step

--step is useful when:

• A single step failed and needs to be re-run after fixing the config.
• You want to add a new analysis type without re-running everything.
• You are debugging a specific script.

# Re-run only tidal analysis after fixing the tidal config
run-analyses validations --area NS --experiment Baseline --step tidal

Relationship to individual scripts

run-analyses delegates to the same individual scripts that you can call directly, as a subprocess per step. The per-step config files (e.g. config/argo_profiles.yaml) are passed through unchanged, with --area, --experiment, --years, and --analyses-dir overridden from the orchestrator’s own command line.

For --source/--model/--scenario (NSe’s per-source workflow, --no-model, per-step --dataset-name, or any other script-specific flag not listed under “Step names” above — run the individual scripts directly instead of going through the orchestrator.