Inspection and Visualization

Dry Run Mode

Inspect planned computations without executing any nodes. Returns a dataframe describing what would run and whether cached outputs already exist.

from neurodags.loaders import load_configuration
from neurodags.orchestrators import iterate_derivative_pipeline, run_pipeline

config = load_configuration("pipeline.yml")

# Dry-run all derivatives in DerivativeList
plan = run_pipeline(config, dry_run=True)
print(plan)

# Or dry-run a single derivative
plan = iterate_derivative_pipeline(config, "BandPower", dry_run=True)
print(plan)

CLI equivalent:

neurodags dry-run pipeline.yml --derivative BandPower --output dry_run_results.csv

The dry-run plan shows each step, whether its output is cached, and where it would write. Use this to:

  • Verify the DAG structure before a long run

  • Identify which files need recomputation

  • Debug path resolution issues

Failure Behavior

When a node fails during execution:

  1. The error is logged with full traceback, derivative name, step id, and file path.

  2. A .error marker file is written at the expected output location (e.g. sub-01@MyDerivative.error). Its content is a human-readable summary of the failure — useful for post-hoc inspection without digging through logs.

  3. By default (raise_on_error=False) execution continues to the next file; all failures are collected in the log.

  4. The failed file will be retried on the next run — the .error file is excluded from the cached-artifact check, so NeuroDAGs treats the output as missing and re-runs it.

To inspect which files failed after a run, look for *.error files in the derivatives directory:

find derivatives/ -name "*.error"

Each .error file contains:

Derivative 'MyDerivative' step id=2 node='my_node' failed:
<exception message>

Automatic Cleanup of Stale Error Markers

When a file that previously failed is re-run and succeeds, NeuroDAGs automatically deletes the stale .error marker. This keeps the derivatives directory clean and ensures dry-run plans accurately reflect current state — a file that succeeded on re-run will not appear as has_error_marker: true on subsequent runs.

Skipping Previously Failed Files

To skip files that already have a .error marker (and avoid retrying known failures), use skip_errors=True:

run_pipeline(config, skip_errors=True)

CLI:

neurodags run pipeline.yml --skip-errors
neurodags dry-run pipeline.yml --skip-errors   # shows which files would be skipped

Skipped files are logged and reported in the dry-run plan with has_error_marker: true. To retry a skipped file, delete its .error marker:

rm derivatives/sub-01/ses-SE0/sub-01_ses-SE0_task-rest.vhdr@MyDerivative.error

Pipeline Status

neurodags status gives a quick at-a-glance summary of how many files are done, missing (not yet computed), or errored — without having to open a CSV.

neurodags status pipeline.yml

Example output:

config: /abs/path/pipeline.yml
files:  42

Derivative               total   done  missing  errored
───────────────────────────────────────────────────────
Alpha                       42     30       10        2
Beta                        42     25       15        2
───────────────────────────────────────────────────────
Total                       84     55       25        4

2 error(s) found.  Run with --list-errors for details.

Status definitions:

Status

Meaning

done

Output file(s) exist and no .error marker present

missing

No output file and no .error marker — not yet computed

errored

A .error marker file exists from a previous failed run

Flags:

# show only specific derivatives
neurodags status pipeline.yml --derivative Alpha --derivative Beta

# print the file path of each errored file
neurodags status pipeline.yml --list-errors

# print the file path of each missing file
neurodags status pipeline.yml --list-missing

# combine both
neurodags status pipeline.yml --list-errors --list-missing

# parallelize the underlying dry-run (same semantics as 'run')
neurodags status pipeline.yml --n-jobs 4
neurodags status pipeline.yml --n-jobs -1   # all cores

With --list-errors the .error marker file path is shown alongside each failed input, making it easy to inspect or delete:

Errored files:
  [Alpha]
    /data/sub-01/ses-01/sub-01_task-rest.vhdr
      error file: /derivatives/sub-01/ses-01/sub-01_task-rest.vhdr@Alpha.error

Exit codes: 0 only when all derivatives are complete (no missing, no errored); 1 when any derivative is missing or has a .error marker. This makes status usable in shell scripts and CI dependency chains:

# fail fast if anything is incomplete or errored
neurodags status pipeline.yml || sbatch resubmit.sh

Machine-readable output: --format json emits a structured JSON object instead of the table:

neurodags status pipeline.yml --format json

{
  "config": "pipeline.yml",
  "n_files": 42,
  "derivatives": {
    "Alpha": {"total": 42, "done": 30, "missing": 10, "errored": 2}
  },
  "grand_total": {"total": 42, "done": 30, "missing": 10, "errored": 2},
  "complete": false
}

DAG Visualization (Mermaid)

NeuroDAGs can render pipeline and derivative graphs as interactive Mermaid diagrams saved to standalone HTML files.

Pipeline-level overview — one node per derivative, edges show inter-derivative dependencies:

import yaml
from neurodags.mermaid import pipeline_to_html

with open("pipeline.yml") as f:
    config = yaml.safe_load(f)

pipeline_to_html(config, output_path="pipeline_dag.html", auto_open=True)

Derivative-level detail — every computation node and data reference inside one derivative:

from neurodags.mermaid import derivative_to_html

derivative_to_html(
    config["DerivativeDefinitions"]["BandPower"],
    "BandPower",
    output_path="bandpower_dag.html",
    auto_open=True,
)

Node shapes used in derivative diagrams:

Shape

Meaning

Circle (((...)))

SourceFile — raw input file

Cylinder [(...)]

Upstream derivative artifact (cached on disk)

Rectangle [...]

Computation node

To get the raw Mermaid string (e.g. for embedding in Jupyter or custom HTML):

from neurodags.mermaid import pipeline_to_mermaid, derivative_to_mermaid

print(pipeline_to_mermaid(config))
print(derivative_to_mermaid(config["DerivativeDefinitions"]["BandPower"], "BandPower"))

Layout engines:

HTML output uses the ELK layout engine by default — orthogonal edge routing with active crossing minimisation, significantly cleaner than curved edges for dense pipelines. ELK loads its bundle from the CDN so internet access is required when opening the HTML. Use --layout dagre (or layout="dagre") for offline use; dagre renders with right-angle step edges and has no CDN dependency.

# ELK layout via Python API
pipeline_to_html(config, output_path="pipeline_dag.html", layout="elk")
derivative_to_html(deriv_def, "BandPower", output_path="bp.html", layout="elk")

CLI:

neurodags dag pipeline.yml
neurodags dag pipeline.yml --html pipeline_dag.html
neurodags dag pipeline.yml --derivative BandPower --html bandpower_dag.html
neurodags dag pipeline.yml --html pipeline_dag.html --layout elk   # ELK for dense graphs

See the DAG Visualization with Mermaid example for a complete walkthrough.

Interactive File Explorer

Built-in Dash-Plotly explorer for .fif (MNE) and .nc (NetCDF/xarray) files:

neurodags view path/to/file.fif
neurodags view path/to/file.nc

# Alternative module entry point
python -m neurodags.visualization path/to/file.fif
python -m neurodags.visualization path/to/file.nc

Features:

  • Variable selector — when the .nc file is a multi-variable Dataset, a dropdown lets you switch between variables interactively without restarting

  • Dimension-aware UI — per-variable dropdowns to slice along any dimension

  • Plot types: Line, Scatter, Bar, Heatmap

  • Works with both xr.DataArray and xr.Dataset files produced by NeuroDAGs