Inspection and Visualization

Note

Output streams. The inspection commands (status, validate, dag) write their result — the table, --format json, or the Mermaid text — to stdout, while all framework logs go to stderr. So the deliverable is always clean and pipeable, e.g. neurodags status pipeline.yml --format json 2>/dev/null yields a single valid JSON document. Increase verbosity with --log-level DEBUG (still on stderr).

Dry Run Mode

Inspect planned computations without executing any nodes. Returns a dataframe describing what would run and whether cached outputs already exist.

from neurodags.loaders import load_configuration
from neurodags.orchestrators import iterate_derivative_pipeline, run_pipeline

config = load_configuration("pipeline.yml")

# Dry-run all derivatives in DerivativeList
plan = run_pipeline(config, dry_run=True)
print(plan)

# Or dry-run a single derivative
plan = iterate_derivative_pipeline(config, "BandPower", dry_run=True)
print(plan)

CLI equivalent:

neurodags dry-run pipeline.yml --derivative BandPower --output dry_run_results.csv

The dry-run plan shows each step, whether its output is cached, and where it would write. Use this to:

  • Verify the DAG structure before a long run

  • Identify which files need recomputation

  • Debug path resolution issues

Failure Behavior

When a node fails during execution:

  1. The error is logged with full traceback, derivative name, step id, and file path.

  2. A .error marker file is written at the expected output location (e.g. sub-01@MyDerivative.error). Its content is a human-readable summary of the failure — useful for post-hoc inspection without digging through logs.

  3. By default (raise_on_error=False) execution continues to the next file; all failures are collected in the log.

  4. The failed file will be retried on the next run — the .error file is excluded from the cached-artifact check, so NeuroDAGs treats the output as missing and re-runs it.

To inspect which files failed after a run, look for *.error files in the derivatives directory:

find derivatives/ -name "*.error"

Each .error file contains:

Derivative 'MyDerivative' step id=2 node='my_node' failed:
<exception message>

Automatic Cleanup of Stale Error Markers

When a file that previously failed is re-run and succeeds, NeuroDAGs automatically deletes the stale .error marker. This keeps the derivatives directory clean and ensures dry-run plans accurately reflect current state — a file that succeeded on re-run will not appear as has_error_marker: true on subsequent runs.

Skipping Previously Failed Files

To skip files that already have a .error marker (and avoid retrying known failures), use skip_errors=True:

run_pipeline(config, skip_errors=True)

CLI:

neurodags run pipeline.yml --skip-errors
neurodags dry-run pipeline.yml --skip-errors   # shows which files would be skipped

Skipped files are logged and reported in the dry-run plan with has_error_marker: true. To retry a skipped file, delete its .error marker:

rm derivatives/sub-01/ses-SE0/sub-01_ses-SE0_task-rest.vhdr@MyDerivative.error

Pipeline Status

neurodags status gives a quick at-a-glance summary of how many files are done, missing (not yet computed), or errored — without having to open a CSV.

neurodags status pipeline.yml

Example output:

config: /abs/path/pipeline.yml
files:  42

Derivative               total   done  missing  errored
───────────────────────────────────────────────────────
Alpha                       42     30       10        2
Beta                        42     25       15        2
───────────────────────────────────────────────────────
Total                       84     55       25        4

2 error(s) found.  Run with --list-errors for details.
25 derivative(s) missing.  Run with --list-missing for details.
Use --format indices to get the file indices for --only-index resubmission.

Status definitions:

Status

Meaning

done

Output file(s) exist and no .error marker present

missing

No output file and no .error marker — not yet computed

errored

A .error marker file exists from a previous failed run

Flags:

# show only specific derivatives
neurodags status pipeline.yml --derivative Alpha --derivative Beta

# print the file index and path of each errored file
neurodags status pipeline.yml --list-errors

# print the file index and path of each missing file
neurodags status pipeline.yml --list-missing

# combine both
neurodags status pipeline.yml --list-errors --list-missing

# parallelize the underlying dry-run (same semantics as 'run')
neurodags status pipeline.yml --n-jobs 4
neurodags status pipeline.yml --n-jobs -1   # all cores

Each listed file is prefixed with its global file index — the same index that neurodags run --only-index and the SLURM array templates consume — and files are sorted by index within each derivative. With --list-errors the .error marker file path is shown alongside each failed input, making it easy to inspect or delete:

Errored files:
  [Alpha]
    [idx 0] /data/sub-01/ses-01/sub-01_task-rest.vhdr
      error file: /derivatives/sub-01/ses-01/sub-01_task-rest.vhdr@Alpha.error

Missing files:
  [Alpha]
    [idx 7] /data/sub-02/ses-01/sub-02_task-rest.vhdr
    [idx 12] /data/sub-05/ses-01/sub-05_task-rest.vhdr

The index is a position in the discovered input list (datasets in config order, files in discovery order). It identifies the same file only while the dataset configuration and the set of input files are unchanged — regenerate it right before resubmitting, don’t store it. If an index cannot be determined it is shown as [idx ?].

Exit codes: 0 only when all derivatives are complete (no missing, no errored); 1 when any derivative is missing or has a .error marker. This makes status usable in shell scripts and CI dependency chains:

# fail fast if anything is incomplete or errored
neurodags status pipeline.yml || sbatch resubmit.sh

Machine-readable output: --format json emits a structured JSON object instead of the table:

neurodags status pipeline.yml --format json

{
  "config": "pipeline.yml",
  "n_files": 42,
  "derivatives": {
    "Alpha": {"total": 42, "done": 30, "skipped": 0, "missing": 10, "errored": 2}
  },
  "grand_total": {"total": 42, "done": 30, "skipped": 0, "missing": 10, "errored": 2},
  "complete": false,
  "missing_indices": [3, 7, 12, 15, 18, 21, 25, 30, 33, 40],
  "errored_indices": [0, 9]
}

missing_indices / errored_indices are the sorted, de-duplicated file indices with at least one missing / errored derivative (across the selected derivatives; narrow with --derivative). A file can appear in both lists if it is missing for one derivative and errored for another.

Add --include-files (only valid with --format json) for a per-file breakdown: a flat files list with one record per (source file, derivative) pair, sorted by (index, derivative):

neurodags status pipeline.yml --format json --include-files

  "files": [
    {"index": 0, "dataset": "my_dataset", "file_path": "/data/sub-01/ses-01/sub-01_task-rest.vhdr",
     "derivative": "Alpha", "status": "errored",
     "error_path": "/derivatives/sub-01/ses-01/sub-01_task-rest.vhdr@Alpha.error"},
    {"index": 1, "dataset": "my_dataset", "file_path": "/data/sub-02/ses-01/sub-02_task-rest.vhdr",
     "derivative": "Alpha", "status": "done", "error_path": null}
  ]

status is one of done, skipped, missing, errored, no-save, unknown. When no input files are found, --format json still emits a valid document (zero counts, empty index lists, "complete": true).

Index-only output: --format indices prints nothing but the sorted, space-separated file indices that are missing or errored — ready to paste into --only-index or sbatch --array. Narrow it with --list-missing (missing only) or --list-errors (errored only); with neither or both flags it prints the union. A complete pipeline prints an empty line.

neurodags status pipeline.yml --format indices                  # e.g. "0 3 7 9 12"
neurodags status pipeline.yml --format indices --list-errors    # errored only: "0 9"
neurodags status pipeline.yml --derivative Alpha --format indices

The exit code keeps its meaning in every format: 1 whenever something is missing or errored, i.e. exactly when the list is non-empty. Under set -e, capturing it with $(...) would abort the script precisely when there is something to resubmit, so add --exit-zero:

IDX=$(neurodags status pipeline.yml --format indices --exit-zero)
[ -n "$IDX" ] && neurodags run pipeline.yml --only-index $IDX

With --exit-zero, status exits 0 whenever the report was produced, but real failures (missing or invalid config, unset environment variable, unreadable -d file) still exit non-zero, so set -e still stops on a broken config. Avoid the older || true guard: it also swallows those real failures, leaving IDX silently empty. See HPC / SLURM Array Jobs for resubmitting through SLURM.

DAG Visualization (Mermaid)

NeuroDAGs can render pipeline and derivative graphs as interactive Mermaid diagrams saved to standalone HTML files.

Pipeline-level overview — one node per derivative, edges show inter-derivative dependencies:

import yaml
from neurodags.mermaid import pipeline_to_html

with open("pipeline.yml") as f:
    config = yaml.safe_load(f)

pipeline_to_html(config, output_path="pipeline_dag.html", auto_open=True)

Derivative-level detail — every computation node and data reference inside one derivative:

from neurodags.mermaid import derivative_to_html

derivative_to_html(
    config["DerivativeDefinitions"]["BandPower"],
    "BandPower",
    output_path="bandpower_dag.html",
    auto_open=True,
)

Node shapes used in derivative diagrams:

Shape

Meaning

Circle (((...)))

SourceFile — raw input file

Cylinder [(...)]

Upstream derivative artifact (cached on disk)

Rectangle [...]

Computation node

To get the raw Mermaid string (e.g. for embedding in Jupyter or custom HTML):

from neurodags.mermaid import pipeline_to_mermaid, derivative_to_mermaid

print(pipeline_to_mermaid(config))
print(derivative_to_mermaid(config["DerivativeDefinitions"]["BandPower"], "BandPower"))

Layout engines:

HTML output uses the ELK layout engine by default — orthogonal edge routing with active crossing minimisation, significantly cleaner than curved edges for dense pipelines. ELK loads its bundle from the CDN so internet access is required when opening the HTML. Use --layout dagre (or layout="dagre") for offline use; dagre renders with right-angle step edges and has no CDN dependency.

# ELK layout via Python API
pipeline_to_html(config, output_path="pipeline_dag.html", layout="elk")
derivative_to_html(deriv_def, "BandPower", output_path="bp.html", layout="elk")

CLI:

neurodags dag pipeline.yml
neurodags dag pipeline.yml --html pipeline_dag.html
neurodags dag pipeline.yml --derivative BandPower --html bandpower_dag.html
neurodags dag pipeline.yml --html pipeline_dag.html --layout elk   # ELK for dense graphs

See the DAG Visualization with Mermaid example for a complete walkthrough.

Interactive File Explorer

Built-in Dash-Plotly explorer for .fif (MNE) and .nc (NetCDF/xarray) files:

neurodags view path/to/file.fif
neurodags view path/to/file.nc

# Alternative module entry point
python -m neurodags.visualization path/to/file.fif
python -m neurodags.visualization path/to/file.nc

Features:

  • Variable selector — when the .nc file is a multi-variable Dataset, a dropdown lets you switch between variables interactively without restarting

  • Dimension-aware UI — per-variable dropdowns to slice along any dimension

  • Plot types: Line, Scatter, Bar, Heatmap

  • Works with both xr.DataArray and xr.Dataset files produced by NeuroDAGs