Closure Operators Migration Guide

This guide shows how to migrate from Nextflow’s index-based groupTuple, join, and combine operators to the closure-based groupTupleBy, joinBy, and combineBy operators provided by nf-bids.

Quick reference

Index-based Closure-based Key benefit

groupTuple(by: 0)

groupTupleBy { it.field }

Semantic grouping; works with maps

join(by: [0])

joinBy(right) { it.field }

No tuple restructuring needed

combine(right, by: 0)

combineBy(right) { it.field }

Self-documenting, semantic fields

Installation

nextflow.config
plugins { id 'nf-bids@0.3.0' }
Workflow script
include { groupTupleBy } from 'plugin/nf-bids'
include { joinBy }       from 'plugin/nf-bids'
include { combineBy }    from 'plugin/nf-bids'

groupTuple → groupTupleBy

Single-field grouping

// Before: requires tuple, opaque index
channel.of(['sub-01', 'file1.nii'], ['sub-01', 'file2.nii'])
       .groupTuple(by: 0)
// → ['sub-01', ['file1.nii', 'file2.nii']]

// After: map, semantic field
channel.of([subject: 'sub-01', file: 'file1.nii'],
           [subject: 'sub-01', file: 'file2.nii'])
       .groupTupleBy { it.subject }
// → ['sub-01', [[subject:'sub-01', file:'file1.nii'],
//               [subject:'sub-01', file:'file2.nii']]]
groupTupleBy keeps the full item in the list; groupTuple decomposes it. Downstream code must use items.collect { it.file } instead of files[0].

Composite key

// Before
channel.map { [it.subject + '_' + it.session, it] }
       .groupTuple(by: 0)

// After
channel.groupTupleBy { [it.subject, it.session] }
// Closure returns List → wrapped in CompositeKey for correct equality

Sorting within groups

// Before
channel.groupTuple(by: 0, sort: { a, b -> a[1] <=> b[1] })

// After  (closures receive full items, not decomposed elements)
channel.groupTupleBy({ it.subject }, [sort: { it.run }])

Size-based streaming

// Before
channel.groupTuple(by: 0, size: 2, remainder: false)

// After
channel.groupTupleBy({ it.type }, [size: 2, remainder: false])

join → joinBy

Simple join

// Before: must restructure both channels first
left  = left_ch.map  { [it.id, it] }
right = right_ch.map { [it.id, it] }
left.join(right).view { key, l, r -> ... }

// After: no restructuring
left_ch.joinBy(right_ch) { it.id }
       .view { fused -> ... }
joinBy now emits fused items (without the key). Keep extraction logic in the closure and consume a single fused item downstream.

Different key fields

// Before
scans.map { [it.scan_id, it] }
     .join(metadata.map { [it.participant_id, it] })

// After
scans.joinBy(metadata, { it.scan_id }, { it.participant_id })

Outer join

// Before
left.map { [it.id, it] }
    .join(right.map { [it.id, it] }, remainder: true)

// After
left.joinBy(right, { it.id }, [remainder: true])

combine → combineBy

Cartesian product with fused outputs

// Built-in: combine with positional key
subjects.combine(sessions, by: 0)

// Closure-based: same matching semantics, fused output payload
subjects.combineBy(sessions, { it.subject })
// Emits fused combined items (key omitted)

Filtered combinations

// Before
scans.combine(protocols)
     .filter { scanId, scanMod, protoName, protoMod -> scanMod == protoMod }

// After
scans.combineBy(protocols, { it.modality })
     .filter { fused -> fused.modality != null }

Common pitfalls

joinBy output is a fused item

// ❌ WRONG — expecting three tuple values
left.joinBy(right) { it.id }.view { key, l, r -> println l }

// ✅ CORRECT
left.joinBy(right) { it.id }.view { fused -> println fused }

Items are preserved as maps

// groupTuple decomposes the tuple:
channel.of(['A', 1, 'x']).groupTuple(by: 0)
// → ['A', [1], ['x']]   ← separate lists per element

// groupTupleBy preserves items:
channel.of([group: 'A', val: 1]).groupTupleBy { it.group }
// → ['A', [[group:'A', val:1]]]   ← items as-is in a single list
// Access: items.collect { it.val }

Null keys are silently dropped

// Items where the closure returns null are skipped with a trace log
.groupTupleBy { it.session }          // null session → item dropped

// Provide a default to keep all items
.groupTupleBy { it.session ?: 'no-session' }

When to stay on index-based operators

Closure-based operators add a small overhead (~10–30 ms for typical BIDS datasets). Prefer the built-in operators when:

  • Items are simple fixed-position tuples with no semantic fields.

  • Maximum throughput is required and the dataset exceeds 100 k items.

  • Integrating with existing code that already produces the right tuple shape.

Related pages