Note

Go to the end to download the full example code.

Quickstart: Synthetic EEG Pipeline

This example walks through a complete NeuroDAGs pipeline using synthetically generated EEG data — no real dataset required.

We will:

  1. Generate a synthetic multi-subject BrainVision dataset.

  2. Define a pipeline in Python (preprocessing → spectral → band power).

  3. Inspect the plan with a dry run.

  4. Execute the pipeline.

  5. Assemble results into a dataframe.

  6. Plot band power across subjects.

Setup

Standard imports and a temporary working directory.

import tempfile
from pathlib import Path

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import yaml

from neurodags.datasets import generate_dummy_dataset
from neurodags.orchestrators import (
    build_derivative_dataframe,
    iterate_derivative_pipeline,
    run_pipeline,
)

WORKDIR = Path(tempfile.mkdtemp(prefix="neurodags_quickstart_"))
DATA_DIR = WORKDIR / "rawdata"
OUT_DIR = WORKDIR / "derivatives"
OUT_DIR.mkdir(parents=True, exist_ok=True)

print(f"Working directory: {WORKDIR}")

Working directory: /tmp/neurodags_quickstart_85xyz1s4

Step 1 — Generate Synthetic Dataset

generate_dummy_dataset() creates BrainVision trios (.vhdr / .vmrk / .eeg) using 1/f^α (pink) noise to mimic realistic EEG spectral characteristics.

We generate 3 subjects × 1 session at 200 Hz, 30 seconds each.

generate_dummy_dataset(
    data_params={
        "DATASET": "quickstart",
        "PATTERN": "sub-%subject%/ses-%session%/sub-%subject%_ses-%session%_task-rest",
        "NSUBS": 3,
        "NSESSIONS": 1,
        "NTASKS": 1,
        "NACQS": 1,
        "NRUNS": 1,
        "PREFIXES": {
            "subject": "S",
            "session": "SE",
            "task": "T",
            "acquisition": "A",
            "run": "R",
        },
        "ROOT": str(DATA_DIR),
    },
    generation_args={
        "NCHANNELS": 8,
        "SFREQ": 200.0,
        "STOP": 30.0,
        "NUMEVENTS": 10,
        "random_state": 0,
    },
)

source_files = sorted(DATA_DIR.rglob("*.vhdr"))
print(f"Generated {len(source_files)} source file(s):")
for f in source_files:
    print(f"  {f.relative_to(WORKDIR)}")

Creating RawArray with float64 data, n_channels=8, n_times=6000
    Range : 0 ... 5999 =      0.000 ...    29.995 secs
Ready.
/home/runner/work/neurodags/neurodags/src/neurodags/datasets.py:703: RuntimeWarning: Encountered data in 'double' format. Converting to float32.
  export_raw(fname=str(vhdr_path), raw=raw, fmt="brainvision", overwrite=True)
Generated 3 source file(s):
  rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr
  rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr
  rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr

Step 2 — Datasets config as YAML

In real workflows, save this string to datasets.yml and point load_configuration at that file for version-controlled, reproducible runs. Paths are injected from Python so the notebook remains runnable.

DATASETS_YAML = f"""\
quickstart:
  name: Quickstart
  file_pattern: "{DATA_DIR / '**' / '*.vhdr'}"
  derivatives_path: "{OUT_DIR}"
"""

datasets = yaml.safe_load(DATASETS_YAML)
print("Datasets:", list(datasets))

Datasets: ['quickstart']

Step 3 — Pipeline config as YAML

The pipeline below is defined entirely in YAML — the format used in pipeline.yml files checked into version control.

This pipeline has three derivatives:

  • BasicPrep: band-pass filter → 2-second epochs.

  • Spectrum: Welch PSD on each epoch.

  • BandPower: relative power in δ, θ, α, β bands, averaged across epochs (save: false — computed but not written to disk; for_dataframe: true — included in the aggregated dataframe).

PIPELINE_YAML = """\
mount_point: null

DerivativeDefinitions:

  BasicPrep:
    overwrite: false
    nodes:
      - id: 0
        derivative: SourceFile
      - id: 1
        node: basic_preprocessing
        args:
          mne_object: id.0
          filter_args: {l_freq: 1.0, h_freq: 80.0}
          epoch_config: {duration: 2.0, overlap: 0.0}

  Spectrum:
    overwrite: false
    nodes:
      - id: 0
        derivative: BasicPrep.fif
      - id: 1
        node: mne_spectrum_array
        args:
          meeg: id.0
          method: welch
          method_kwargs: {n_per_seg: 200}

  BandPower:
    save: false
    for_dataframe: true
    nodes:
      - id: 0
        derivative: Spectrum.nc
      - id: 1
        node: extract_data_var
        args: {dataset_like: id.0, data_var: spectrum}
      - id: 2
        node: bandpower
        args:
          psd_like: id.1
          relative: true
          bands:
            delta: [1.0,  4.0]
            theta: [4.0,  8.0]
            alpha: [8.0, 13.0]
            beta:  [13.0, 30.0]
      - id: 3
        node: aggregate_across_dimension
        args: {xarray_data: id.2, dim: epochs, operation: mean}

DerivativeList:
  - BasicPrep
  - Spectrum
  - BandPower
"""

pipeline_config = yaml.safe_load(PIPELINE_YAML)
pipeline_config["datasets"] = datasets  # inject dynamic dataset paths

print("Pipeline defined with derivatives:", pipeline_config["DerivativeList"])

Pipeline defined with derivatives: ['BasicPrep', 'Spectrum', 'BandPower']

Step 4 — Dry Run

Inspect the execution plan for BasicPrep without running any computation. The returned dataframe shows which outputs are cached and which would be computed.

plan = iterate_derivative_pipeline(pipeline_config, "BasicPrep", dry_run=True)
# 'plan' column contains per-step dicts — expand for display
steps = []
for _, row in plan.iterrows():
    for step in row["plan"]:
        steps.append({"file": row["file_path"].split("/")[-1], **step})
print(pd.DataFrame(steps)[["file", "id", "kind", "cached"]].to_string(index=False))

{"dataset": "quickstart", "file_count": 3, "event": "Found files in dataset", "level": "info", "timestamp": "2026-07-14T03:01:56.229803Z"}
{"total_datasets": 1, "total_files": 3, "event": "File discovery complete", "level": "info", "timestamp": "2026-07-14T03:01:56.229960Z"}
{"total_files": 3, "event": "Starting derivative processing", "level": "info", "timestamp": "2026-07-14T03:01:56.230107Z"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.230175Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.230236Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.230295Z"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.230831Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.230935Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.231000Z"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.231490Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.231566Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.231629Z"}
{"index": 0, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr", "derivative": "BasicPrep", "event": "Processed file successfully", "level": "info", "timestamp": "2026-07-14T03:01:56.232150Z"}
{"index": 0, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr", "derivative": "BasicPrep", "file": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr", "reference_base": "/tmp/neurodags_quickstart_85xyz1s4/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr", "overwrite": false, "plan": [{"id": "final", "kind": "derivative_output", "name": "BasicPrep", "prefix": "/tmp/neurodags_quickstart_85xyz1s4/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep", "cached": false, "paths": [], "has_error_marker": false, "error_path": null, "has_skip_marker": false, "skip_path": null}, {"id": 0, "kind": "source", "name": "SourceFile", "path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr"}], "event": "Dry run:", "level": "info", "timestamp": "2026-07-14T03:01:56.232233Z"}
{"index": 1, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr", "derivative": "BasicPrep", "event": "Processed file successfully", "level": "info", "timestamp": "2026-07-14T03:01:56.232305Z"}
{"index": 1, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr", "derivative": "BasicPrep", "file": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr", "reference_base": "/tmp/neurodags_quickstart_85xyz1s4/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr", "overwrite": false, "plan": [{"id": "final", "kind": "derivative_output", "name": "BasicPrep", "prefix": "/tmp/neurodags_quickstart_85xyz1s4/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep", "cached": false, "paths": [], "has_error_marker": false, "error_path": null, "has_skip_marker": false, "skip_path": null}, {"id": 0, "kind": "source", "name": "SourceFile", "path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr"}], "event": "Dry run:", "level": "info", "timestamp": "2026-07-14T03:01:56.232358Z"}
{"index": 2, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr", "derivative": "BasicPrep", "event": "Processed file successfully", "level": "info", "timestamp": "2026-07-14T03:01:56.232436Z"}
{"index": 2, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr", "derivative": "BasicPrep", "file": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr", "reference_base": "/tmp/neurodags_quickstart_85xyz1s4/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr", "overwrite": false, "plan": [{"id": "final", "kind": "derivative_output", "name": "BasicPrep", "prefix": "/tmp/neurodags_quickstart_85xyz1s4/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep", "cached": false, "paths": [], "has_error_marker": false, "error_path": null, "has_skip_marker": false, "skip_path": null}, {"id": 0, "kind": "source", "name": "SourceFile", "path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr"}], "event": "Dry run:", "level": "info", "timestamp": "2026-07-14T03:01:56.232489Z"}
{"total_files": 3, "event": "Completed derivative processing", "level": "info", "timestamp": "2026-07-14T03:01:56.232548Z"}
                         file    id              kind cached
sub-S0_ses-SE0_task-rest.vhdr final derivative_output  False
sub-S0_ses-SE0_task-rest.vhdr     0            source    NaN
sub-S1_ses-SE0_task-rest.vhdr final derivative_output  False
sub-S1_ses-SE0_task-rest.vhdr     0            source    NaN
sub-S2_ses-SE0_task-rest.vhdr final derivative_output  False
sub-S2_ses-SE0_task-rest.vhdr     0            source    NaN

Step 5 — Execute the Pipeline

run_pipeline runs all derivatives in DerivativeList, sorted by dependency order. Already-cached outputs are skipped automatically.

run_pipeline(pipeline_config, raise_on_error=True)

# List produced files
produced = sorted(OUT_DIR.rglob("*@*.fif")) + sorted(OUT_DIR.rglob("*@*.nc"))
print(f"\nProduced {len(produced)} derivative file(s):")
for f in produced:
    print(f"  {f.relative_to(WORKDIR)}")

{"order": ["BasicPrep", "Spectrum", "BandPower"], "event": "Derivative execution order", "level": "info", "timestamp": "2026-07-14T03:01:56.236740Z"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.236853Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.236949Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.237013Z"}
{"dataset": "quickstart", "file_count": 3, "event": "Found files in dataset", "level": "info", "timestamp": "2026-07-14T03:01:56.237721Z"}
{"total_datasets": 1, "total_files": 3, "event": "File discovery complete", "level": "info", "timestamp": "2026-07-14T03:01:56.237797Z"}
{"total_files": 3, "event": "Starting derivative processing", "level": "info", "timestamp": "2026-07-14T03:01:56.237937Z"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.238004Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.238065Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.238123Z"}
Not setting metadata
15 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 15 events and 400 original time points ...
0 bad epochs dropped
/home/runner/work/neurodags/neurodags/src/neurodags/nodes/preprocessing.py:143: RuntimeWarning: This filename (/tmp/neurodags_quickstart_85xyz1s4/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep.fif) does not conform to MNE naming conventions. All epochs files should end with -epo.fif, -epo.fif.gz, _epo.fif or _epo.fif.gz
  ".fif": Artifact(item=mne_object, writer=lambda path: mne_object.save(path, overwrite=True))
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.464711Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.464859Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.464958Z"}
Not setting metadata
15 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 15 events and 400 original time points ...
0 bad epochs dropped
/home/runner/work/neurodags/neurodags/src/neurodags/nodes/preprocessing.py:143: RuntimeWarning: This filename (/tmp/neurodags_quickstart_85xyz1s4/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep.fif) does not conform to MNE naming conventions. All epochs files should end with -epo.fif, -epo.fif.gz, _epo.fif or _epo.fif.gz
  ".fif": Artifact(item=mne_object, writer=lambda path: mne_object.save(path, overwrite=True))
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.484737Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.484830Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.484924Z"}
Not setting metadata
15 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 15 events and 400 original time points ...
0 bad epochs dropped
/home/runner/work/neurodags/neurodags/src/neurodags/nodes/preprocessing.py:143: RuntimeWarning: This filename (/tmp/neurodags_quickstart_85xyz1s4/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep.fif) does not conform to MNE naming conventions. All epochs files should end with -epo.fif, -epo.fif.gz, _epo.fif or _epo.fif.gz
  ".fif": Artifact(item=mne_object, writer=lambda path: mne_object.save(path, overwrite=True))
{"index": 0, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr", "derivative": "BasicPrep", "event": "Processed file successfully", "level": "info", "timestamp": "2026-07-14T03:01:56.503560Z"}
{"index": 1, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr", "derivative": "BasicPrep", "event": "Processed file successfully", "level": "info", "timestamp": "2026-07-14T03:01:56.503644Z"}
{"index": 2, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr", "derivative": "BasicPrep", "event": "Processed file successfully", "level": "info", "timestamp": "2026-07-14T03:01:56.503704Z"}
{"total_files": 3, "event": "Completed derivative processing", "level": "info", "timestamp": "2026-07-14T03:01:56.503765Z"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.503840Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.503930Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.503996Z"}
{"dataset": "quickstart", "file_count": 3, "event": "Found files in dataset", "level": "info", "timestamp": "2026-07-14T03:01:56.504752Z"}
{"total_datasets": 1, "total_files": 3, "event": "File discovery complete", "level": "info", "timestamp": "2026-07-14T03:01:56.504828Z"}
{"total_files": 3, "event": "Starting derivative processing", "level": "info", "timestamp": "2026-07-14T03:01:56.504972Z"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.505037Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.505098Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.505156Z"}
Effective window size : 1.280 (s)
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.677489Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.677623Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.677696Z"}
Effective window size : 1.280 (s)
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.694274Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.694371Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.694444Z"}
Effective window size : 1.280 (s)
{"index": 0, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr", "derivative": "Spectrum", "event": "Processed file successfully", "level": "info", "timestamp": "2026-07-14T03:01:56.709485Z"}
{"index": 1, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr", "derivative": "Spectrum", "event": "Processed file successfully", "level": "info", "timestamp": "2026-07-14T03:01:56.709570Z"}
{"index": 2, "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr", "derivative": "Spectrum", "event": "Processed file successfully", "level": "info", "timestamp": "2026-07-14T03:01:56.709637Z"}
{"total_files": 3, "event": "Completed derivative processing", "level": "info", "timestamp": "2026-07-14T03:01:56.709695Z"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.709770Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.709837Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.709926Z"}
{"dataset": "quickstart", "file_count": 3, "event": "Found files in dataset", "level": "info", "timestamp": "2026-07-14T03:01:56.710704Z"}
{"total_datasets": 1, "total_files": 3, "event": "File discovery complete", "level": "info", "timestamp": "2026-07-14T03:01:56.710784Z"}
{"derivative": "BandPower", "event": "Derivative is marked with save=False; skipping execution.", "level": "info", "timestamp": "2026-07-14T03:01:56.710861Z"}

Produced 6 derivative file(s):
  derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep.fif
  derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep.fif
  derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep.fif
  derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@Spectrum.nc
  derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@Spectrum.nc
  derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@Spectrum.nc

Step 6 — Assemble Dataframe

build_derivative_dataframe() collects every for_dataframe=True derivative into a single dataframe.

output_format="wide" gives one row per file with derivative columns.

df = build_derivative_dataframe(pipeline_config, output_format="wide")

# Extract readable subject labels from the file path
df["subject"] = df["file_path"].apply(
    lambda p: next(
        (part for part in Path(p).parts if part.startswith("sub-")),
        Path(p).stem.split("_")[0],
    )
)

print(f"DataFrame shape: {df.shape}")
print(df.head())

{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.712432Z"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.712542Z"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.712609Z"}
{"missing_derivatives": ["BasicPrep", "Spectrum"], "event": "Some requested derivatives are either undefined or flagged out of dataframe collection.", "level": "warning", "timestamp": "2026-07-14T03:01:56.712675Z"}
{"dataset": "quickstart", "file_count": 3, "event": "Found files in dataset", "level": "info", "timestamp": "2026-07-14T03:01:56.713429Z"}
{"total_datasets": 1, "total_files": 3, "event": "File discovery complete", "level": "info", "timestamp": "2026-07-14T03:01:56.713508Z"}
{"dataset": "quickstart", "file_count": 3, "event": "Found files in dataset", "level": "info", "timestamp": "2026-07-14T03:01:56.714143Z"}
{"total_datasets": 1, "total_files": 3, "event": "File discovery complete", "level": "info", "timestamp": "2026-07-14T03:01:56.714219Z"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.714458Z", "logger": "neurodags.derivatives"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.714573Z", "logger": "neurodags.derivatives"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.714655Z", "logger": "neurodags.derivatives"}
{"total": 3, "event": "Collecting dataframe row", "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr", "index": 0, "level": "info", "timestamp": "2026-07-14T03:01:56.714769Z", "logger": "neurodags.orchestrators"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.745793Z", "logger": "neurodags.derivatives"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.745939Z", "logger": "neurodags.derivatives"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.746030Z", "logger": "neurodags.derivatives"}
{"total": 3, "event": "Collecting dataframe row", "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr", "index": 1, "level": "info", "timestamp": "2026-07-14T03:01:56.746131Z", "logger": "neurodags.orchestrators"}
{"event": "Overriding existing derivative registration for 'BasicPrep'", "level": "info", "timestamp": "2026-07-14T03:01:56.762057Z", "logger": "neurodags.derivatives"}
{"event": "Overriding existing derivative registration for 'Spectrum'", "level": "info", "timestamp": "2026-07-14T03:01:56.762176Z", "logger": "neurodags.derivatives"}
{"event": "Overriding existing derivative registration for 'BandPower'", "level": "info", "timestamp": "2026-07-14T03:01:56.762264Z", "logger": "neurodags.derivatives"}
{"total": 3, "event": "Collecting dataframe row", "dataset": "quickstart", "file_path": "/tmp/neurodags_quickstart_85xyz1s4/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr", "index": 2, "level": "info", "timestamp": "2026-07-14T03:01:56.762364Z", "logger": "neurodags.orchestrators"}
DataFrame shape: (3, 36)
   index     dataset  ... BandPower.nc@freqbands-beta_spaces-EEG007  subject
0      0  quickstart  ...                                  0.165119   sub-S0
1      1  quickstart  ...                                  0.165119   sub-S1
2      2  quickstart  ...                                  0.165119   sub-S2

[3 rows x 36 columns]

Step 7 — Visualise Band Power

Group by subject and plot mean relative band power per frequency band.

band_cols = [c for c in df.columns if any(b in c for b in ["delta", "theta", "alpha", "beta"])]

if band_cols:
    # Melt to long form for plotting
    df_long = df[["subject", *band_cols]].melt(
        id_vars="subject", var_name="band_channel", value_name="relative_power"
    )
    # Extract band name from column label
    df_long["band"] = df_long["band_channel"].apply(
        lambda x: next((b for b in ["delta", "theta", "alpha", "beta"] if b in x), None)
    )
    band_means = df_long.groupby(["subject", "band"])["relative_power"].mean().reset_index()

    bands = ["delta", "theta", "alpha", "beta"]
    band_means = band_means[band_means["band"].isin(bands)]

    subjects = sorted(band_means["subject"].unique())
    x = np.arange(len(bands))
    width = 0.8 / len(subjects)

    fig, ax = plt.subplots(figsize=(8, 4))
    for i, sub in enumerate(subjects):
        vals = [
            band_means.loc[
                (band_means["subject"] == sub) & (band_means["band"] == b), "relative_power"
            ].mean()
            for b in bands
        ]
        ax.bar(x + i * width, vals, width=width, label=sub)

    ax.set_xticks(x + width * (len(subjects) - 1) / 2)
    ax.set_xticklabels(bands)
    ax.set_ylabel("Relative Power")
    ax.set_title("Mean Relative Band Power per Subject")
    ax.legend(title="Subject")
    plt.tight_layout()
    plt.savefig(WORKDIR / "band_power.png", dpi=100)
    plt.show()
    print(f"Plot saved to {WORKDIR / 'band_power.png'}")
else:
    print("No band power columns found in dataframe.")
Mean Relative Band Power per Subject
Plot saved to /tmp/neurodags_quickstart_85xyz1s4/band_power.png

What’s Next

  • Swap generate_dummy_dataset for real BIDS data by pointing file_pattern at your raw EEG files.

  • Save PIPELINE_YAML / DATASETS_YAML to pipeline.yml and datasets.yml for version-controlled, reproducible workflows.

  • Run the same workflow from the CLI with commands such as neurodags validate pipeline.yml, neurodags dry-run pipeline.yml --derivative BasicPrep, and neurodags run pipeline.yml.

  • Add custom nodes via new_definitions: my_nodes.py.

  • Scale up: set n_jobs=-1 for file-level parallelism via joblib.

  • Inspect any .nc file interactively with the built-in Dash explorer:

    neurodags view path/to/file.nc

Total running time of the script: (0 minutes 9.734 seconds)

Gallery generated by Sphinx-Gallery