Note

Go to the end to download the full example code.

Quickstart: Synthetic EEG Pipeline

This example walks through a complete NeuroDAGs pipeline using synthetically generated EEG data — no real dataset required.

We will:

  1. Generate a synthetic multi-subject BrainVision dataset.

  2. Define a pipeline in Python (preprocessing → spectral → band power).

  3. Inspect the plan with a dry run.

  4. Execute the pipeline.

  5. Assemble results into a dataframe.

  6. Plot band power across subjects.

Setup

Standard imports and a temporary working directory.

import tempfile
from pathlib import Path

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import yaml

from neurodags.datasets import generate_dummy_dataset
from neurodags.orchestrators import build_derivative_dataframe, iterate_derivative_pipeline, run_pipeline

WORKDIR = Path(tempfile.mkdtemp(prefix="neurodags_quickstart_"))
DATA_DIR = WORKDIR / "rawdata"
OUT_DIR = WORKDIR / "derivatives"
OUT_DIR.mkdir(parents=True, exist_ok=True)

print(f"Working directory: {WORKDIR}")

2026-06-04 04:58:18 [debug    ] Registered derivative          name=CheckLineFrequency
2026-06-04 04:58:18 [debug    ] Registered derivative          name=BasicPrep1
2026-06-04 04:58:18 [debug    ] Registered derivative          name=InterDerivativeDependence
2026-06-04 04:58:18 [debug    ] Registered derivative          name=SpectrumArrayWelch
2026-06-04 04:58:18 [debug    ] Registered derivative          name=SpectrumArrayMultitaper
2026-06-04 04:58:18 [debug    ] Registered derivative          name=MEEGMetadata
/home/runner/work/neurodags/neurodags/src/neurodags/nodes/spectral.py:14: DeprecationWarning:
The `fooof` package is being deprecated and replaced by the `specparam` (spectral parameterization) package.
This version of `fooof` (1.1) is fully functional, but will not be further updated.
New projects are recommended to update to using `specparam` (see Changelog for details).
  from fooof import FOOOF
Working directory: /tmp/neurodags_quickstart_ns00zpwm

Step 1 — Generate Synthetic Dataset

generate_dummy_dataset() creates BrainVision trios (.vhdr / .vmrk / .eeg) using 1/f^α (pink) noise to mimic realistic EEG spectral characteristics.

We generate 3 subjects × 1 session at 200 Hz, 30 seconds each.

generate_dummy_dataset(
    data_params={
        "DATASET": "quickstart",
        "PATTERN": "sub-%subject%/ses-%session%/sub-%subject%_ses-%session%_task-rest",
        "NSUBS": 3,
        "NSESSIONS": 1,
        "NTASKS": 1,
        "NACQS": 1,
        "NRUNS": 1,
        "PREFIXES": {
            "subject": "S",
            "session": "SE",
            "task": "T",
            "acquisition": "A",
            "run": "R",
        },
        "ROOT": str(DATA_DIR),
    },
    generation_args={
        "NCHANNELS": 8,
        "SFREQ": 200.0,
        "STOP": 30.0,
        "NUMEVENTS": 10,
        "random_state": 0,
    },
)

source_files = sorted(DATA_DIR.rglob("*.vhdr"))
print(f"Generated {len(source_files)} source file(s):")
for f in source_files:
    print(f"  {f.relative_to(WORKDIR)}")

Creating RawArray with float64 data, n_channels=8, n_times=6000
    Range : 0 ... 5999 =      0.000 ...    29.995 secs
Ready.
/home/runner/work/neurodags/neurodags/src/neurodags/datasets.py:703: RuntimeWarning: Encountered data in 'double' format. Converting to float32.
  export_raw(fname=str(vhdr_path), raw=raw, fmt="brainvision", overwrite=True)
2026-06-04 04:58:24 [debug    ] get_num_digits: called         method=safe n=3
2026-06-04 04:58:24 [debug    ] get_num_digits: computed (safe) digits=1 n=3
2026-06-04 04:58:24 [debug    ] get_num_digits: called         method=safe n=1
2026-06-04 04:58:24 [debug    ] get_num_digits: computed (safe) digits=1 n=1
2026-06-04 04:58:24 [debug    ] get_num_digits: called         method=safe n=1
2026-06-04 04:58:24 [debug    ] get_num_digits: computed (safe) digits=1 n=1
2026-06-04 04:58:24 [debug    ] get_num_digits: called         method=safe n=1
2026-06-04 04:58:24 [debug    ] get_num_digits: computed (safe) digits=1 n=1
2026-06-04 04:58:24 [debug    ] get_num_digits: called         method=safe n=1
2026-06-04 04:58:24 [debug    ] get_num_digits: computed (safe) digits=1 n=1
Generated 3 source file(s):
  rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr
  rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr
  rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr

Step 2 — Datasets config as YAML

In real workflows, save this string to datasets.yml and point load_configuration at that file for version-controlled, reproducible runs. Paths are injected from Python so the notebook remains runnable.

DATASETS_YAML = f"""\
quickstart:
  name: Quickstart
  file_pattern: "{DATA_DIR / '**' / '*.vhdr'}"
  derivatives_path: "{OUT_DIR}"
"""

datasets = yaml.safe_load(DATASETS_YAML)
print("Datasets:", list(datasets))

Datasets: ['quickstart']

Step 3 — Pipeline config as YAML

The pipeline below is defined entirely in YAML — the format used in pipeline.yml files checked into version control.

This pipeline has three derivatives:

  • BasicPrep: band-pass filter → 2-second epochs.

  • Spectrum: Welch PSD on each epoch.

  • BandPower: relative power in δ, θ, α, β bands, averaged across epochs (save: false — computed but not written to disk; for_dataframe: true — included in the aggregated dataframe).

PIPELINE_YAML = """\
mount_point: null

DerivativeDefinitions:

  BasicPrep:
    overwrite: false
    nodes:
      - id: 0
        derivative: SourceFile
      - id: 1
        node: basic_preprocessing
        args:
          mne_object: id.0
          filter_args: {l_freq: 1.0, h_freq: 80.0}
          epoch_config: {duration: 2.0, overlap: 0.0}

  Spectrum:
    overwrite: false
    nodes:
      - id: 0
        derivative: BasicPrep.fif
      - id: 1
        node: mne_spectrum_array
        args:
          meeg: id.0
          method: welch
          method_kwargs: {n_per_seg: 200}

  BandPower:
    save: false
    for_dataframe: true
    nodes:
      - id: 0
        derivative: Spectrum.nc
      - id: 1
        node: extract_data_var
        args: {dataset_like: id.0, data_var: spectrum}
      - id: 2
        node: bandpower
        args:
          psd_like: id.1
          relative: true
          bands:
            delta: [1.0,  4.0]
            theta: [4.0,  8.0]
            alpha: [8.0, 13.0]
            beta:  [13.0, 30.0]
      - id: 3
        node: aggregate_across_dimension
        args: {xarray_data: id.2, dim: epochs, operation: mean}

DerivativeList:
  - BasicPrep
  - Spectrum
  - BandPower
"""

pipeline_config = yaml.safe_load(PIPELINE_YAML)
pipeline_config["datasets"] = datasets  # inject dynamic dataset paths

print("Pipeline defined with derivatives:", pipeline_config["DerivativeList"])

Pipeline defined with derivatives: ['BasicPrep', 'Spectrum', 'BandPower']

Step 4 — Dry Run

Inspect the execution plan for BasicPrep without running any computation. The returned dataframe shows which outputs are cached and which would be computed.

plan = iterate_derivative_pipeline(pipeline_config, "BasicPrep", dry_run=True)
# 'plan' column contains per-step dicts — expand for display
steps = []
for _, row in plan.iterrows():
    for step in row["plan"]:
        steps.append({"file": row["file_path"].split("/")[-1], **step})
print(pd.DataFrame(steps)[["file", "id", "kind", "cached"]].to_string(index=False))

2026-06-04 04:58:24 [debug    ] iterate_call_pipeline: called  datasets_configuration=None pipeline_configuration={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:24 [debug    ] Registered derivative          name=BasicPrep
2026-06-04 04:58:24 [debug    ] Registered derivative          name=Spectrum
2026-06-04 04:58:24 [debug    ] Registered derivative          name=BandPower
2026-06-04 04:58:24 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:24 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:24 [debug    ] get_all_files_from_pipeline_configuration: called pipeline_input={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:24 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:24 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:24 [debug    ] get_all_files_across_datasets: processing dataset dataset=quickstart
2026-06-04 04:58:24 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:24 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:24 [debug    ] get_all_files_across_datasets: resolved pattern dataset=quickstart pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:24 [debug    ] get_files_from_pattern: called pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr recursive=True
2026-06-04 04:58:24 [debug    ] get_files_from_pattern: found files count=3
2026-06-04 04:58:24 [debug    ] find_unique_root: called       mode=maximal n_paths=3 strict=True style=auto
2026-06-04 04:58:24 [debug    ] find_unique_root: inferred style inferred=posix
2026-06-04 04:58:24 [debug    ] find_unique_root: normalized paths sample sample=['/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr']
2026-06-04 04:58:24 [debug    ] find_unique_root: prefix info  common_prefix=/tmp/neurodags_quickstart_ns00zpwm/rawdata prefix_len=4
2026-06-04 04:58:24 [debug    ] find_unique_root: candidate check depth=1 unique=True
2026-06-04 04:58:24 [debug    ] find_unique_root: candidate check depth=2 unique=True
2026-06-04 04:58:24 [debug    ] find_unique_root: candidate check depth=3 unique=True
2026-06-04 04:58:24 [debug    ] find_unique_root: candidate check depth=4 unique=True
2026-06-04 04:58:24 [debug    ] find_unique_root: selected maximal root root=/tmp/neurodags_quickstart_ns00zpwm/rawdata
2026-06-04 04:58:24 [debug    ] get_all_files_across_datasets: common root for dataset common_root=/tmp/neurodags_quickstart_ns00zpwm/rawdata dataset=quickstart
2026-06-04 04:58:24 [info     ] Found files in dataset         dataset=quickstart file_count=3
2026-06-04 04:58:24 [debug    ] get_all_files_across_datasets: dataset summary dataset=quickstart file_count=3
2026-06-04 04:58:24 [info     ] File discovery complete        total_datasets=1 total_files=3
2026-06-04 04:58:24 [debug    ] get_all_files_from_pipeline_configuration: completed total_datasets=1 total_files=3
2026-06-04 04:58:24 [debug    ] Processing file                dataset=quickstart file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr index=0
2026-06-04 04:58:24 [debug    ] Processing file                dataset=quickstart file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr index=1
2026-06-04 04:58:24 [debug    ] Processing file                dataset=quickstart file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr index=2
2026-06-04 04:58:24 [info     ] Starting derivative processing total_files=3
2026-06-04 04:58:24 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:24 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/derivatives
2026-06-04 04:58:24 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:24 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/derivatives
2026-06-04 04:58:24 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:24 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/derivatives
2026-06-04 04:58:24 [info     ] Processed file successfully    dataset=quickstart derivative=BasicPrep file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr index=0
2026-06-04 04:58:24 [info     ] Dry run:                       dataset=quickstart derivative=BasicPrep file=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr index=0 overwrite=False plan=[{'id': 'final', 'kind': 'derivative_output', 'name': 'BasicPrep', 'prefix': '/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep', 'cached': False, 'paths': [], 'has_error_marker': False, 'error_path': None, 'has_skip_marker': False, 'skip_path': None}, {'id': 0, 'kind': 'source', 'name': 'SourceFile', 'path': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr'}] reference_base=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr
2026-06-04 04:58:24 [info     ] Processed file successfully    dataset=quickstart derivative=BasicPrep file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr index=1
2026-06-04 04:58:24 [info     ] Dry run:                       dataset=quickstart derivative=BasicPrep file=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr index=1 overwrite=False plan=[{'id': 'final', 'kind': 'derivative_output', 'name': 'BasicPrep', 'prefix': '/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep', 'cached': False, 'paths': [], 'has_error_marker': False, 'error_path': None, 'has_skip_marker': False, 'skip_path': None}, {'id': 0, 'kind': 'source', 'name': 'SourceFile', 'path': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr'}] reference_base=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr
2026-06-04 04:58:24 [info     ] Processed file successfully    dataset=quickstart derivative=BasicPrep file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr index=2
2026-06-04 04:58:24 [info     ] Dry run:                       dataset=quickstart derivative=BasicPrep file=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr index=2 overwrite=False plan=[{'id': 'final', 'kind': 'derivative_output', 'name': 'BasicPrep', 'prefix': '/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep', 'cached': False, 'paths': [], 'has_error_marker': False, 'error_path': None, 'has_skip_marker': False, 'skip_path': None}, {'id': 0, 'kind': 'source', 'name': 'SourceFile', 'path': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr'}] reference_base=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr
2026-06-04 04:58:24 [info     ] Completed derivative processing total_files=3
                         file    id              kind cached
sub-S1_ses-SE0_task-rest.vhdr final derivative_output  False
sub-S1_ses-SE0_task-rest.vhdr     0            source    NaN
sub-S2_ses-SE0_task-rest.vhdr final derivative_output  False
sub-S2_ses-SE0_task-rest.vhdr     0            source    NaN
sub-S0_ses-SE0_task-rest.vhdr final derivative_output  False
sub-S0_ses-SE0_task-rest.vhdr     0            source    NaN

Step 5 — Execute the Pipeline

run_pipeline runs all derivatives in DerivativeList, sorted by dependency order. Already-cached outputs are skipped automatically.

run_pipeline(pipeline_config, raise_on_error=True)

# List produced files
produced = sorted(OUT_DIR.rglob("*@*.fif")) + sorted(OUT_DIR.rglob("*@*.nc"))
print(f"\nProduced {len(produced)} derivative file(s):")
for f in produced:
    print(f"  {f.relative_to(WORKDIR)}")

2026-06-04 04:58:24 [info     ] Derivative execution order     order=['BasicPrep', 'Spectrum', 'BandPower']
2026-06-04 04:58:24 [debug    ] iterate_call_pipeline: called  datasets_configuration=None pipeline_configuration={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:24 [info     ] Overriding existing derivative registration for 'BasicPrep'
2026-06-04 04:58:24 [debug    ] Registered derivative          name=BasicPrep
2026-06-04 04:58:24 [info     ] Overriding existing derivative registration for 'Spectrum'
2026-06-04 04:58:24 [debug    ] Registered derivative          name=Spectrum
2026-06-04 04:58:24 [info     ] Overriding existing derivative registration for 'BandPower'
2026-06-04 04:58:24 [debug    ] Registered derivative          name=BandPower
2026-06-04 04:58:24 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:24 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:24 [debug    ] get_all_files_from_pipeline_configuration: called pipeline_input={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:24 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:24 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:24 [debug    ] get_all_files_across_datasets: processing dataset dataset=quickstart
2026-06-04 04:58:24 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:24 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:24 [debug    ] get_all_files_across_datasets: resolved pattern dataset=quickstart pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:24 [debug    ] get_files_from_pattern: called pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr recursive=True
2026-06-04 04:58:24 [debug    ] get_files_from_pattern: found files count=3
2026-06-04 04:58:24 [debug    ] find_unique_root: called       mode=maximal n_paths=3 strict=True style=auto
2026-06-04 04:58:24 [debug    ] find_unique_root: inferred style inferred=posix
2026-06-04 04:58:24 [debug    ] find_unique_root: normalized paths sample sample=['/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr']
2026-06-04 04:58:24 [debug    ] find_unique_root: prefix info  common_prefix=/tmp/neurodags_quickstart_ns00zpwm/rawdata prefix_len=4
2026-06-04 04:58:24 [debug    ] find_unique_root: candidate check depth=1 unique=True
2026-06-04 04:58:24 [debug    ] find_unique_root: candidate check depth=2 unique=True
2026-06-04 04:58:24 [debug    ] find_unique_root: candidate check depth=3 unique=True
2026-06-04 04:58:24 [debug    ] find_unique_root: candidate check depth=4 unique=True
2026-06-04 04:58:24 [debug    ] find_unique_root: selected maximal root root=/tmp/neurodags_quickstart_ns00zpwm/rawdata
2026-06-04 04:58:24 [debug    ] get_all_files_across_datasets: common root for dataset common_root=/tmp/neurodags_quickstart_ns00zpwm/rawdata dataset=quickstart
2026-06-04 04:58:24 [info     ] Found files in dataset         dataset=quickstart file_count=3
2026-06-04 04:58:24 [debug    ] get_all_files_across_datasets: dataset summary dataset=quickstart file_count=3
2026-06-04 04:58:24 [info     ] File discovery complete        total_datasets=1 total_files=3
2026-06-04 04:58:24 [debug    ] get_all_files_from_pipeline_configuration: completed total_datasets=1 total_files=3
2026-06-04 04:58:24 [debug    ] Processing file                dataset=quickstart file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr index=0
2026-06-04 04:58:24 [debug    ] Processing file                dataset=quickstart file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr index=1
2026-06-04 04:58:24 [debug    ] Processing file                dataset=quickstart file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr index=2
2026-06-04 04:58:24 [info     ] Starting derivative processing total_files=3
2026-06-04 04:58:24 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:24 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/derivatives
2026-06-04 04:58:24 [debug    ] Execute node                   derivative=BasicPrep id=1 kwargs={'mne_object': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}} node=basic_preprocessing
2026-06-04 04:58:24 [debug    ] Attempting to load MEEG file   file=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr kwargs={'preload': True, 'verbose': 'error'}
2026-06-04 04:58:24 [debug    ] Loaded MEEG file as Raw        file=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr
2026-06-04 04:58:24 [debug    ] MNEReport: loaded MNE object from file input=<RawBrainVision | sub-S1_ses-SE0_task-rest.eeg, 8 x 6000 (30.0 s), ~388 KiB, data loaded>
2026-06-04 04:58:24 [debug    ] Filter Applied                 filter_args={'l_freq': 1.0, 'h_freq': 80.0}
Not setting metadata
15 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 15 events and 400 original time points ...
0 bad epochs dropped
2026-06-04 04:58:24 [debug    ] EPOCH SEGMENTATION with make_fixed_length_epochs epoch_config={'duration': 2.0, 'overlap': 0.0}
2026-06-04 04:58:24 [debug    ] Processed artifact             file=Artifact(item=<Epochs | 15 events (all good), 0 – 1.995 s (baseline off), ~388 KiB, data loaded,
 '1': 15>, writer=<function basic_preprocessing.<locals>.<lambda> at 0x7f59384bfac0>) name=.fif
/home/runner/work/neurodags/neurodags/src/neurodags/nodes/preprocessing.py:143: RuntimeWarning: This filename (/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep.fif) does not conform to MNE naming conventions. All epochs files should end with -epo.fif, -epo.fif.gz, _epo.fif or _epo.fif.gz
  ".fif": Artifact(item=mne_object, writer=lambda path: mne_object.save(path, overwrite=True))
2026-06-04 04:58:25 [debug    ] Saved artifact                 file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep.fif
2026-06-04 04:58:25 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:25 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/derivatives
2026-06-04 04:58:25 [debug    ] Execute node                   derivative=BasicPrep id=1 kwargs={'mne_object': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}} node=basic_preprocessing
2026-06-04 04:58:25 [debug    ] Attempting to load MEEG file   file=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr kwargs={'preload': True, 'verbose': 'error'}
2026-06-04 04:58:25 [debug    ] Loaded MEEG file as Raw        file=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr
2026-06-04 04:58:25 [debug    ] MNEReport: loaded MNE object from file input=<RawBrainVision | sub-S2_ses-SE0_task-rest.eeg, 8 x 6000 (30.0 s), ~388 KiB, data loaded>
2026-06-04 04:58:25 [debug    ] Filter Applied                 filter_args={'l_freq': 1.0, 'h_freq': 80.0}
Not setting metadata
15 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 15 events and 400 original time points ...
0 bad epochs dropped
2026-06-04 04:58:25 [debug    ] EPOCH SEGMENTATION with make_fixed_length_epochs epoch_config={'duration': 2.0, 'overlap': 0.0}
2026-06-04 04:58:25 [debug    ] Processed artifact             file=Artifact(item=<Epochs | 15 events (all good), 0 – 1.995 s (baseline off), ~388 KiB, data loaded,
 '1': 15>, writer=<function basic_preprocessing.<locals>.<lambda> at 0x7f59384bfac0>) name=.fif
/home/runner/work/neurodags/neurodags/src/neurodags/nodes/preprocessing.py:143: RuntimeWarning: This filename (/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep.fif) does not conform to MNE naming conventions. All epochs files should end with -epo.fif, -epo.fif.gz, _epo.fif or _epo.fif.gz
  ".fif": Artifact(item=mne_object, writer=lambda path: mne_object.save(path, overwrite=True))
2026-06-04 04:58:25 [debug    ] Saved artifact                 file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep.fif
2026-06-04 04:58:25 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:25 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/derivatives
2026-06-04 04:58:25 [debug    ] Execute node                   derivative=BasicPrep id=1 kwargs={'mne_object': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}} node=basic_preprocessing
2026-06-04 04:58:25 [debug    ] Attempting to load MEEG file   file=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr kwargs={'preload': True, 'verbose': 'error'}
2026-06-04 04:58:25 [debug    ] Loaded MEEG file as Raw        file=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr
2026-06-04 04:58:25 [debug    ] MNEReport: loaded MNE object from file input=<RawBrainVision | sub-S0_ses-SE0_task-rest.eeg, 8 x 6000 (30.0 s), ~388 KiB, data loaded>
2026-06-04 04:58:25 [debug    ] Filter Applied                 filter_args={'l_freq': 1.0, 'h_freq': 80.0}
Not setting metadata
15 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 15 events and 400 original time points ...
0 bad epochs dropped
2026-06-04 04:58:25 [debug    ] EPOCH SEGMENTATION with make_fixed_length_epochs epoch_config={'duration': 2.0, 'overlap': 0.0}
2026-06-04 04:58:25 [debug    ] Processed artifact             file=Artifact(item=<Epochs | 15 events (all good), 0 – 1.995 s (baseline off), ~388 KiB, data loaded,
 '1': 15>, writer=<function basic_preprocessing.<locals>.<lambda> at 0x7f5938305ab0>) name=.fif
/home/runner/work/neurodags/neurodags/src/neurodags/nodes/preprocessing.py:143: RuntimeWarning: This filename (/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep.fif) does not conform to MNE naming conventions. All epochs files should end with -epo.fif, -epo.fif.gz, _epo.fif or _epo.fif.gz
  ".fif": Artifact(item=mne_object, writer=lambda path: mne_object.save(path, overwrite=True))
2026-06-04 04:58:25 [debug    ] Saved artifact                 file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep.fif
2026-06-04 04:58:25 [info     ] Processed file successfully    dataset=quickstart derivative=BasicPrep file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr index=0
2026-06-04 04:58:25 [info     ] Processed file successfully    dataset=quickstart derivative=BasicPrep file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr index=1
2026-06-04 04:58:25 [info     ] Processed file successfully    dataset=quickstart derivative=BasicPrep file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr index=2
2026-06-04 04:58:25 [info     ] Completed derivative processing total_files=3
2026-06-04 04:58:25 [debug    ] iterate_call_pipeline: called  datasets_configuration=None pipeline_configuration={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:25 [info     ] Overriding existing derivative registration for 'BasicPrep'
2026-06-04 04:58:25 [debug    ] Registered derivative          name=BasicPrep
2026-06-04 04:58:25 [info     ] Overriding existing derivative registration for 'Spectrum'
2026-06-04 04:58:25 [debug    ] Registered derivative          name=Spectrum
2026-06-04 04:58:25 [info     ] Overriding existing derivative registration for 'BandPower'
2026-06-04 04:58:25 [debug    ] Registered derivative          name=BandPower
2026-06-04 04:58:25 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:25 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:25 [debug    ] get_all_files_from_pipeline_configuration: called pipeline_input={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:25 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:25 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: processing dataset dataset=quickstart
2026-06-04 04:58:25 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:25 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: resolved pattern dataset=quickstart pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:25 [debug    ] get_files_from_pattern: called pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr recursive=True
2026-06-04 04:58:25 [debug    ] get_files_from_pattern: found files count=3
2026-06-04 04:58:25 [debug    ] find_unique_root: called       mode=maximal n_paths=3 strict=True style=auto
2026-06-04 04:58:25 [debug    ] find_unique_root: inferred style inferred=posix
2026-06-04 04:58:25 [debug    ] find_unique_root: normalized paths sample sample=['/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr']
2026-06-04 04:58:25 [debug    ] find_unique_root: prefix info  common_prefix=/tmp/neurodags_quickstart_ns00zpwm/rawdata prefix_len=4
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=1 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=2 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=3 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=4 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: selected maximal root root=/tmp/neurodags_quickstart_ns00zpwm/rawdata
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: common root for dataset common_root=/tmp/neurodags_quickstart_ns00zpwm/rawdata dataset=quickstart
2026-06-04 04:58:25 [info     ] Found files in dataset         dataset=quickstart file_count=3
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: dataset summary dataset=quickstart file_count=3
2026-06-04 04:58:25 [info     ] File discovery complete        total_datasets=1 total_files=3
2026-06-04 04:58:25 [debug    ] get_all_files_from_pipeline_configuration: completed total_datasets=1 total_files=3
2026-06-04 04:58:25 [debug    ] Processing file                dataset=quickstart file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr index=0
2026-06-04 04:58:25 [debug    ] Processing file                dataset=quickstart file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr index=1
2026-06-04 04:58:25 [debug    ] Processing file                dataset=quickstart file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr index=2
2026-06-04 04:58:25 [info     ] Starting derivative processing total_files=3
2026-06-04 04:58:25 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:25 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/derivatives
2026-06-04 04:58:25 [debug    ] Using cached derivative        child_derivative=BasicPrep.fif derivative=Spectrum id=0 paths=['/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep.fif']
2026-06-04 04:58:25 [debug    ] Execute node                   derivative=Spectrum id=1 kwargs={'meeg': '/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep.fif', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}} node=mne_spectrum_array
2026-06-04 04:58:25 [debug    ] Attempting to load MEEG file   file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep.fif kwargs={'preload': True, 'verbose': 'error'}
2026-06-04 04:58:25 [debug    ] Loaded MEEG file as Epochs     file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep.fif
2026-06-04 04:58:25 [debug    ] MNEReport: loaded MNE object from file input=<EpochsFIF | 15 events (all good), 0 – 1.995 s (baseline off), ~389 KiB, data loaded,
 '1': 15>
Effective window size : 1.280 (s)
2026-06-04 04:58:25 [debug    ] Processed artifact             file=Artifact(item=<xarray.Dataset> Size: 125kB
Dimensions:      (epochs: 15, spaces: 8, frequencies: 129)
Coordinates:
  * epochs       (epochs) int64 120B 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
  * spaces       (spaces) <U6 192B 'EEG000' 'EEG001' ... 'EEG006' 'EEG007'
  * frequencies  (frequencies) float64 1kB 0.0 0.7812 1.562 ... 99.22 100.0
Data variables:
    spectrum     (epochs, spaces, frequencies) float64 124kB 0.004506 ... 2.2...
Attributes:
    metadata:  {\n  "method": "welch",\n  "method_kwargs": {\n    "n_per_seg"..., writer=<function mne_spectrum_array.<locals>.<lambda> at 0x7f59381e5ab0>) name=.nc
<frozen importlib._bootstrap>:241: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject
2026-06-04 04:58:25 [debug    ] Saved artifact                 file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@Spectrum.nc
2026-06-04 04:58:25 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:25 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/derivatives
2026-06-04 04:58:25 [debug    ] Using cached derivative        child_derivative=BasicPrep.fif derivative=Spectrum id=0 paths=['/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep.fif']
2026-06-04 04:58:25 [debug    ] Execute node                   derivative=Spectrum id=1 kwargs={'meeg': '/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep.fif', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}} node=mne_spectrum_array
2026-06-04 04:58:25 [debug    ] Attempting to load MEEG file   file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep.fif kwargs={'preload': True, 'verbose': 'error'}
2026-06-04 04:58:25 [debug    ] Loaded MEEG file as Epochs     file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep.fif
2026-06-04 04:58:25 [debug    ] MNEReport: loaded MNE object from file input=<EpochsFIF | 15 events (all good), 0 – 1.995 s (baseline off), ~389 KiB, data loaded,
 '1': 15>
Effective window size : 1.280 (s)
2026-06-04 04:58:25 [debug    ] Processed artifact             file=Artifact(item=<xarray.Dataset> Size: 125kB
Dimensions:      (epochs: 15, spaces: 8, frequencies: 129)
Coordinates:
  * epochs       (epochs) int64 120B 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
  * spaces       (spaces) <U6 192B 'EEG000' 'EEG001' ... 'EEG006' 'EEG007'
  * frequencies  (frequencies) float64 1kB 0.0 0.7812 1.562 ... 99.22 100.0
Data variables:
    spectrum     (epochs, spaces, frequencies) float64 124kB 0.004506 ... 2.2...
Attributes:
    metadata:  {\n  "method": "welch",\n  "method_kwargs": {\n    "n_per_seg"..., writer=<function mne_spectrum_array.<locals>.<lambda> at 0x7f593805d120>) name=.nc
2026-06-04 04:58:25 [debug    ] Saved artifact                 file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@Spectrum.nc
2026-06-04 04:58:25 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:25 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/derivatives
2026-06-04 04:58:25 [debug    ] Using cached derivative        child_derivative=BasicPrep.fif derivative=Spectrum id=0 paths=['/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep.fif']
2026-06-04 04:58:25 [debug    ] Execute node                   derivative=Spectrum id=1 kwargs={'meeg': '/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep.fif', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}} node=mne_spectrum_array
2026-06-04 04:58:25 [debug    ] Attempting to load MEEG file   file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep.fif kwargs={'preload': True, 'verbose': 'error'}
2026-06-04 04:58:25 [debug    ] Loaded MEEG file as Epochs     file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep.fif
2026-06-04 04:58:25 [debug    ] MNEReport: loaded MNE object from file input=<EpochsFIF | 15 events (all good), 0 – 1.995 s (baseline off), ~389 KiB, data loaded,
 '1': 15>
Effective window size : 1.280 (s)
2026-06-04 04:58:25 [debug    ] Processed artifact             file=Artifact(item=<xarray.Dataset> Size: 125kB
Dimensions:      (epochs: 15, spaces: 8, frequencies: 129)
Coordinates:
  * epochs       (epochs) int64 120B 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
  * spaces       (spaces) <U6 192B 'EEG000' 'EEG001' ... 'EEG006' 'EEG007'
  * frequencies  (frequencies) float64 1kB 0.0 0.7812 1.562 ... 99.22 100.0
Data variables:
    spectrum     (epochs, spaces, frequencies) float64 124kB 0.004506 ... 2.2...
Attributes:
    metadata:  {\n  "method": "welch",\n  "method_kwargs": {\n    "n_per_seg"..., writer=<function mne_spectrum_array.<locals>.<lambda> at 0x7f593805d480>) name=.nc
2026-06-04 04:58:25 [debug    ] Saved artifact                 file=/tmp/neurodags_quickstart_ns00zpwm/derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@Spectrum.nc
2026-06-04 04:58:25 [info     ] Processed file successfully    dataset=quickstart derivative=Spectrum file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr index=0
2026-06-04 04:58:25 [info     ] Processed file successfully    dataset=quickstart derivative=Spectrum file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr index=1
2026-06-04 04:58:25 [info     ] Processed file successfully    dataset=quickstart derivative=Spectrum file_path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr index=2
2026-06-04 04:58:25 [info     ] Completed derivative processing total_files=3
2026-06-04 04:58:25 [debug    ] iterate_call_pipeline: called  datasets_configuration=None pipeline_configuration={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:25 [info     ] Overriding existing derivative registration for 'BasicPrep'
2026-06-04 04:58:25 [debug    ] Registered derivative          name=BasicPrep
2026-06-04 04:58:25 [info     ] Overriding existing derivative registration for 'Spectrum'
2026-06-04 04:58:25 [debug    ] Registered derivative          name=Spectrum
2026-06-04 04:58:25 [info     ] Overriding existing derivative registration for 'BandPower'
2026-06-04 04:58:25 [debug    ] Registered derivative          name=BandPower
2026-06-04 04:58:25 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:25 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:25 [debug    ] get_all_files_from_pipeline_configuration: called pipeline_input={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:25 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:25 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: processing dataset dataset=quickstart
2026-06-04 04:58:25 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:25 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: resolved pattern dataset=quickstart pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:25 [debug    ] get_files_from_pattern: called pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr recursive=True
2026-06-04 04:58:25 [debug    ] get_files_from_pattern: found files count=3
2026-06-04 04:58:25 [debug    ] find_unique_root: called       mode=maximal n_paths=3 strict=True style=auto
2026-06-04 04:58:25 [debug    ] find_unique_root: inferred style inferred=posix
2026-06-04 04:58:25 [debug    ] find_unique_root: normalized paths sample sample=['/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr']
2026-06-04 04:58:25 [debug    ] find_unique_root: prefix info  common_prefix=/tmp/neurodags_quickstart_ns00zpwm/rawdata prefix_len=4
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=1 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=2 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=3 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=4 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: selected maximal root root=/tmp/neurodags_quickstart_ns00zpwm/rawdata
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: common root for dataset common_root=/tmp/neurodags_quickstart_ns00zpwm/rawdata dataset=quickstart
2026-06-04 04:58:25 [info     ] Found files in dataset         dataset=quickstart file_count=3
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: dataset summary dataset=quickstart file_count=3
2026-06-04 04:58:25 [info     ] File discovery complete        total_datasets=1 total_files=3
2026-06-04 04:58:25 [debug    ] get_all_files_from_pipeline_configuration: completed total_datasets=1 total_files=3
2026-06-04 04:58:25 [info     ] Derivative is marked with save=False; skipping execution. derivative=BandPower

Produced 6 derivative file(s):
  derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@BasicPrep.fif
  derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@BasicPrep.fif
  derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@BasicPrep.fif
  derivatives/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr@Spectrum.nc
  derivatives/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr@Spectrum.nc
  derivatives/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr@Spectrum.nc

Step 6 — Assemble Dataframe

build_derivative_dataframe() collects every for_dataframe=True derivative into a single dataframe.

output_format="wide" gives one row per file with derivative columns.

df = build_derivative_dataframe(pipeline_config, output_format="wide")

# Extract readable subject labels from the file path
df["subject"] = df["file_path"].apply(
    lambda p: next(
        (part for part in Path(p).parts if part.startswith("sub-")),
        Path(p).stem.split("_")[0],
    )
)

print(f"DataFrame shape: {df.shape}")
print(df.head())

2026-06-04 04:58:25 [debug    ] build_derivative_dataframe: called datasets_configuration=None pipeline_configuration={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:25 [info     ] Overriding existing derivative registration for 'BasicPrep'
2026-06-04 04:58:25 [debug    ] Registered derivative          name=BasicPrep
2026-06-04 04:58:25 [info     ] Overriding existing derivative registration for 'Spectrum'
2026-06-04 04:58:25 [debug    ] Registered derivative          name=Spectrum
2026-06-04 04:58:25 [info     ] Overriding existing derivative registration for 'BandPower'
2026-06-04 04:58:25 [debug    ] Registered derivative          name=BandPower
2026-06-04 04:58:25 [warning  ] Some requested derivatives are either undefined or flagged out of dataframe collection. missing_derivatives=['BasicPrep', 'Spectrum']
2026-06-04 04:58:25 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:25 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:25 [debug    ] get_all_files_from_pipeline_configuration: called pipeline_input={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:25 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:25 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: processing dataset dataset=quickstart
2026-06-04 04:58:25 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:25 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: resolved pattern dataset=quickstart pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:25 [debug    ] get_files_from_pattern: called pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr recursive=True
2026-06-04 04:58:25 [debug    ] get_files_from_pattern: found files count=3
2026-06-04 04:58:25 [debug    ] find_unique_root: called       mode=maximal n_paths=3 strict=True style=auto
2026-06-04 04:58:25 [debug    ] find_unique_root: inferred style inferred=posix
2026-06-04 04:58:25 [debug    ] find_unique_root: normalized paths sample sample=['/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr']
2026-06-04 04:58:25 [debug    ] find_unique_root: prefix info  common_prefix=/tmp/neurodags_quickstart_ns00zpwm/rawdata prefix_len=4
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=1 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=2 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=3 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=4 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: selected maximal root root=/tmp/neurodags_quickstart_ns00zpwm/rawdata
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: common root for dataset common_root=/tmp/neurodags_quickstart_ns00zpwm/rawdata dataset=quickstart
2026-06-04 04:58:25 [info     ] Found files in dataset         dataset=quickstart file_count=3
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: dataset summary dataset=quickstart file_count=3
2026-06-04 04:58:25 [info     ] File discovery complete        total_datasets=1 total_files=3
2026-06-04 04:58:25 [debug    ] get_all_files_from_pipeline_configuration: completed total_datasets=1 total_files=3
2026-06-04 04:58:25 [debug    ] get_all_files_from_pipeline_configuration: called pipeline_input={'mount_point': None, 'DerivativeDefinitions': {'BasicPrep': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'SourceFile'}, {'id': 1, 'node': 'basic_preprocessing', 'args': {'mne_object': 'id.0', 'filter_args': {'l_freq': 1.0, 'h_freq': 80.0}, 'epoch_config': {'duration': 2.0, 'overlap': 0.0}}}]}, 'Spectrum': {'overwrite': False, 'nodes': [{'id': 0, 'derivative': 'BasicPrep.fif'}, {'id': 1, 'node': 'mne_spectrum_array', 'args': {'meeg': 'id.0', 'method': 'welch', 'method_kwargs': {'n_per_seg': 200}}}]}, 'BandPower': {'save': False, 'for_dataframe': True, 'nodes': [{'id': 0, 'derivative': 'Spectrum.nc'}, {'id': 1, 'node': 'extract_data_var', 'args': {'dataset_like': 'id.0', 'data_var': 'spectrum'}}, {'id': 2, 'node': 'bandpower', 'args': {'psd_like': 'id.1', 'relative': True, 'bands': {'delta': [1.0, 4.0], 'theta': [4.0, 8.0], 'alpha': [8.0, 13.0], 'beta': [13.0, 30.0]}}}, {'id': 3, 'node': 'aggregate_across_dimension', 'args': {'xarray_data': 'id.2', 'dim': 'epochs', 'operation': 'mean'}}]}}, 'DerivativeList': ['BasicPrep', 'Spectrum', 'BandPower'], 'datasets': {'quickstart': {'name': 'Quickstart', 'file_pattern': '/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr', 'derivatives_path': '/tmp/neurodags_quickstart_ns00zpwm/derivatives'}}}
2026-06-04 04:58:25 [debug    ] Loading YAML rules             arg_type=dict
2026-06-04 04:58:25 [debug    ] Loading YAML from in-memory mapping action=deepcopy keys=4
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: processing dataset dataset=quickstart
2026-06-04 04:58:25 [debug    ] get_path: called               mount_point=None path_type=str
2026-06-04 04:58:25 [debug    ] get_path: returning direct path path=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: resolved pattern dataset=quickstart pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr
2026-06-04 04:58:25 [debug    ] get_files_from_pattern: called pattern=/tmp/neurodags_quickstart_ns00zpwm/rawdata/**/*.vhdr recursive=True
2026-06-04 04:58:25 [debug    ] get_files_from_pattern: found files count=3
2026-06-04 04:58:25 [debug    ] find_unique_root: called       mode=maximal n_paths=3 strict=True style=auto
2026-06-04 04:58:25 [debug    ] find_unique_root: inferred style inferred=posix
2026-06-04 04:58:25 [debug    ] find_unique_root: normalized paths sample sample=['/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr']
2026-06-04 04:58:25 [debug    ] find_unique_root: prefix info  common_prefix=/tmp/neurodags_quickstart_ns00zpwm/rawdata prefix_len=4
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=1 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=2 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=3 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: candidate check depth=4 unique=True
2026-06-04 04:58:25 [debug    ] find_unique_root: selected maximal root root=/tmp/neurodags_quickstart_ns00zpwm/rawdata
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: common root for dataset common_root=/tmp/neurodags_quickstart_ns00zpwm/rawdata dataset=quickstart
2026-06-04 04:58:25 [info     ] Found files in dataset         dataset=quickstart file_count=3
2026-06-04 04:58:25 [debug    ] get_all_files_across_datasets: dataset summary dataset=quickstart file_count=3
2026-06-04 04:58:25 [info     ] File discovery complete        total_datasets=1 total_files=3
2026-06-04 04:58:25 [debug    ] get_all_files_from_pipeline_configuration: completed total_datasets=1 total_files=3
2026-06-04 04:58:25 [debug    ] build_derivative_dataframe: enumerated files per_dataset={'quickstart': ['/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr', '/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr']} total_files=3
{"total": 3, "event": "Collecting dataframe row", "file_path": "/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S1/ses-SE0/sub-S1_ses-SE0_task-rest.vhdr", "dataset": "quickstart", "index": 0, "level": "info", "timestamp": "2026-06-04T04:58:25.386047Z", "logger": "neurodags.orchestrators"}
{"total": 3, "event": "Collecting dataframe row", "file_path": "/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S2/ses-SE0/sub-S2_ses-SE0_task-rest.vhdr", "dataset": "quickstart", "index": 1, "level": "info", "timestamp": "2026-06-04T04:58:25.412567Z", "logger": "neurodags.orchestrators"}
{"total": 3, "event": "Collecting dataframe row", "file_path": "/tmp/neurodags_quickstart_ns00zpwm/rawdata/sub-S0/ses-SE0/sub-S0_ses-SE0_task-rest.vhdr", "dataset": "quickstart", "index": 2, "level": "info", "timestamp": "2026-06-04T04:58:25.426645Z", "logger": "neurodags.orchestrators"}
DataFrame shape: (3, 36)
   index     dataset  ... BandPower.nc@freqbands-beta_spaces-EEG007  subject
0      0  quickstart  ...                                  0.165119   sub-S1
1      1  quickstart  ...                                  0.165119   sub-S2
2      2  quickstart  ...                                  0.165119   sub-S0

[3 rows x 36 columns]

Step 7 — Visualise Band Power

Group by subject and plot mean relative band power per frequency band.

band_cols = [c for c in df.columns if any(b in c for b in ["delta", "theta", "alpha", "beta"])]

if band_cols:
    # Melt to long form for plotting
    df_long = df[["subject", *band_cols]].melt(
        id_vars="subject", var_name="band_channel", value_name="relative_power"
    )
    # Extract band name from column label
    df_long["band"] = df_long["band_channel"].apply(
        lambda x: next((b for b in ["delta", "theta", "alpha", "beta"] if b in x), None)
    )
    band_means = df_long.groupby(["subject", "band"])["relative_power"].mean().reset_index()

    bands = ["delta", "theta", "alpha", "beta"]
    band_means = band_means[band_means["band"].isin(bands)]

    subjects = sorted(band_means["subject"].unique())
    x = np.arange(len(bands))
    width = 0.8 / len(subjects)

    fig, ax = plt.subplots(figsize=(8, 4))
    for i, sub in enumerate(subjects):
        vals = [
            band_means.loc[
                (band_means["subject"] == sub) & (band_means["band"] == b), "relative_power"
            ].mean()
            for b in bands
        ]
        ax.bar(x + i * width, vals, width=width, label=sub)

    ax.set_xticks(x + width * (len(subjects) - 1) / 2)
    ax.set_xticklabels(bands)
    ax.set_ylabel("Relative Power")
    ax.set_title("Mean Relative Band Power per Subject")
    ax.legend(title="Subject")
    plt.tight_layout()
    plt.savefig(WORKDIR / "band_power.png", dpi=100)
    plt.show()
    print(f"Plot saved to {WORKDIR / 'band_power.png'}")
else:
    print("No band power columns found in dataframe.")
Mean Relative Band Power per Subject
Plot saved to /tmp/neurodags_quickstart_ns00zpwm/band_power.png

What’s Next

  • Swap generate_dummy_dataset for real BIDS data by pointing file_pattern at your raw EEG files.

  • Save PIPELINE_YAML / DATASETS_YAML to pipeline.yml and datasets.yml for version-controlled, reproducible workflows.

  • Run the same workflow from the CLI with commands such as neurodags validate pipeline.yml, neurodags dry-run pipeline.yml --derivative BasicPrep, and neurodags run pipeline.yml.

  • Add custom nodes via new_definitions: my_nodes.py.

  • Scale up: set n_jobs=-1 for file-level parallelism via joblib.

  • Inspect any .nc file interactively with the built-in Dash explorer:

    neurodags view path/to/file.nc

Total running time of the script: (0 minutes 8.761 seconds)

Gallery generated by Sphinx-Gallery