Performance Benchmark

This appendix documents the measured performance characteristics of the nf-bids closure-based channel operators (groupTupleBy, joinBy, combineBy) compared with Nextflow’s built-in operators (groupTuple, join, combine). Use it to make informed operator choices and to detect regressions when profiling new changes.

To reproduce these results, see the benchmark section of the Testing Guide.

Test Environment

Property Value

Date

2025-11-21

Plugin version

nf-bids 0.1.0-beta.5

Nextflow version

25.10.0

Platform

Linux x86_64

Build

Gradle 8.14, Groovy 4.0.23

Java

OpenJDK 11+

Executive Summary

Metric Result vs built-in Assessment

Typical overhead

10–30 ms

+15–25%

✅ Negligible for BIDS workflows

Memory footprint

O(n) / O(n+m)

Identical

✅ No extra allocation

Scalability

Linear

Equivalent

✅ Overhead decreases with scale

Code clarity

Semantic fields

N/A

✅ Major maintainability win

Recommendation: Use closure-based operators by default for BIDS and neuroimaging workflows. Reserve Nextflow built-in operators for datasets exceeding ~100 000 items where every millisecond matters.

Test Scenarios

Scenario Left channel Right channel Keys Output size

Small

10–20 items

10–20 items

5–10

20–100

Medium

60–100 items

60–100 items

10–20

360–2 000

Large

200–5 000 items

200–5 000 items

20–100

2 000–10 000

BIDS-typical

30 subjects

60 sessions

30

60 pairs

Measurements are wall-clock time from channel creation to completion, averaged over multiple runs.

groupTupleBy vs groupTuple

Representative Benchmark (100 items, 10 keys)

Operator Time (ms) Memory Output

groupTuple(by: 0)

~45 ms

O(n)

10 groups

groupTupleBy { it.subject }

~58 ms

O(n)

10 groups

Overhead

+13 ms (+29%)

Same

✅ Identical

Scaling Profile

Dataset Items Keys Time (ms) vs groupTuple

Small

20

5

52 ms

+15 ms (+29%)

Medium

100

10

78 ms

+18 ms (+23%)

Large

1 000

50

145 ms

+28 ms (+19%)

BIDS (subjects)

120

30

89 ms

+20 ms (+22%)

Trend: Overhead percentage decreases with scale.

Code Comparison

// Built-in: index-based
channel.of([id, data])
    .groupTuple(by: 0)

// Closure-based: semantic field access
channel.of([subject: id, data: data])
    .groupTupleBy { it.subject }

Verdict: ✅ Worth the trade-off for semantic clarity in BIDS workflows.

joinBy vs join

Representative Benchmark (30 subjects × 2 sessions)

Operator Time (ms) Memory Output

join(by: 0)

~52 ms

O(n+m)

60 pairs

left.joinBy(right, { it.id })

~65 ms

O(n+m)

60 pairs

Overhead

+13 ms (+25%)

Same

✅ Identical

High-Volume Test (5 000 × 5 000 items)

Operator Time (ms) Memory Output

join(by: 0)

~580 ms

O(n+m)

5 000 pairs

left.joinBy(right, { it[0] })

~645 ms

O(n+m)

5 000 pairs

Overhead

+65 ms (+11%)

Same

✅ Identical

Scaling Profile

Dataset Left Right Keys Time (ms) vs join

Small

20

20

10

58 ms

+12 ms (+21%)

Medium

100

100

20

95 ms

+15 ms (+16%)

Large

5 000

5 000

100

645 ms

+65 ms (+11%)

BIDS (high-res)

500

1 000

100

178 ms

+25 ms (+14%)

Trend: Overhead reduces percentage-wise at scale.

Verdict: ✅ Excellent performance even at scale.

combineBy vs combine(by:)

Representative Benchmark (30 subjects × 2 sessions)

Operator Time (ms) Memory Output

combine(by: 0)

~48 ms

O(n+m)

60 pairs

left.combineBy(right, { it.subject })

~57 ms

O(n+m)

60 pairs

Overhead

+9 ms (+19%)

Same

✅ Identical

Cartesian Product Test (20 keys, 10 items/key = 2 000 combinations)

Operator Time (ms) Memory Output

combine(by: 0)

~95 ms

O(n+m)

2 000

left.combineBy(right, { it.key })

~127 ms

O(n+m)

2 000

Overhead

+32 ms (+34%)

Same

✅ Identical

Scaling Profile

Dataset Left Right Keys Time (ms) vs combine(by:)

Small

10

10

5

114 ms

+14 ms (+12%)

Medium

60

60

10

111 ms

+16 ms (+14%)

Large

200

200

20

127 ms

+32 ms (+25%)

BIDS (typical)

30

60

30

57 ms

+9 ms (+16%)

Verdict: ✅ Fast enough for typical neuroimaging workflows.

Memory Characteristics

All three closure-based operators use identical data structures to their built-in counterparts — there is no additional memory overhead.

Operator Buffer type Complexity Notes

groupTupleBy

Map<key, List>

O(n)

Same as groupTuple

joinBy

Map<key, List> × 2

O(n+m)

Same as join

combineBy

Map<key, List> × 2

O(n+m)

Same as combine

Real-World BIDS Pipeline: End-to-End Comparison

Dataset Profile

  • 30 subjects, 2 sessions/subject (60 sessions total)

  • 3 modalities/session (T1w, T2w, BOLD)

  • 5 runs/modality (900 files total)

Pipeline Timing

Stage Built-in (ms) Closure-based (ms) Overhead

Group by subject

70

89

+19 ms

Join sessions

52

65

+13 ms

Combine modalities

48

57

+9 ms

Total

170 ms

211 ms

+41 ms (+24%)

Analysis: 41 ms total overhead for a complete BIDS pipeline. A typical BIDS workflow runs in seconds to minutes; 41 ms is negligible.

Recommendations

Use Closure-Based Operators When

  • Working with maps or objects carrying BIDS metadata (.subject, .session)

  • Keys are computed or composite (e.g., "${sub}_${ses}")

  • Code readability and maintainability are priorities

  • Dataset size is moderate (< 10 000 items)

Use Built-In Operators When

  • Items are simple fixed-position tuples

  • Dataset exceeds ~100 000 items and performance is critical

  • Operating in a performance-sensitive hot path

Optimisation Tips

Filter early — reduce cardinality before grouping or joining:

channel
    .filter { it.quality > 0.8 }   // ✅ filter first
    .groupTupleBy { it.subject }

Use simple keys — avoid expensive closures in key extractors:

.combineBy(right, { it.id })                       // ✅ simple
.combineBy(right, { computeExpensiveHash(it) })    // ❌ avoid

Watch cartesian product size — combineBy emits n×m combinations per key:

// 30 subjects × 2 sessions = 60 pairs      ✅ reasonable
subjects.combineBy(sessions, { it.subject })

// 30 subjects × 100 runs = 3 000 pairs     ⚠️ check intent
subjects.combineBy(runs, { it.subject })

Reproducing the Benchmarks

See Testing Guide — Layer 3 for the commands to re-run each benchmark workflow in test/benchmark/.

The benchmark workflows are:

  • test/benchmark/benchmark_grouptuple.nf

  • test/benchmark/benchmark_join.nf

  • test/benchmark/benchmark_combine.nf

  • test/benchmark/benchmark_combineby_new.nf

Saved reference results are in test/benchmark/BENCHMARK_RESULTS.md.