Rivulet Benchmarks¶

This project contains comprehensive performance benchmarks for Rivulet using BenchmarkDotNet.

Running Benchmarks¶

Run All Benchmarks¶

cd tests\Rivulet.Benchmarks
dotnet run -c Release

Note: The project targets .NET 9.0 for building, but BenchmarkDotNet automatically runs all benchmarks on both .NET 8.0 and .NET 9.0 as specified by the [SimpleJob] attributes in each benchmark class.

Run Specific Benchmark Suite¶

dotnet run -c Release -- --filter "*CoreOperatorsBenchmarks*"
dotnet run -c Release -- --filter "*BatchingBenchmarks*"
dotnet run -c Release -- --filter "*ErrorHandlingBenchmarks*"
dotnet run -c Release -- --filter "*AdvancedFeaturesBenchmarks*"
dotnet run -c Release -- --filter "*ConcurrencyScalingBenchmarks*"

Quick Run (Short Job)¶

For faster results with fewer iterations:

dotnet run -c Release -- --job short

Export Results¶

# Export to various formats
dotnet run -c Release -- --exporters json,html,markdown

Benchmark Suites¶

1. CoreOperatorsBenchmarks¶

Measures the performance of core parallel operators:

• SelectParallelAsync (CPU-bound light operations)
• SelectParallelAsync (I/O simulation with Task.Delay)
• SelectParallelAsync (Ordered output)
• SelectParallelStreamAsync (Streaming results)
• ForEachParallelAsync (Side effects without result collection)
• Baseline (Sequential processing)
• Baseline (Task.WhenAll unbounded)

Configuration: 1,000 items, MaxDegreeOfParallelism = 8-32

Comparison: Shows performance vs sequential processing and unbounded Task.WhenAll.

2. BatchingBenchmarks¶

Evaluates batch processing performance with different batch sizes:

• BatchParallelAsync (Batch size 100, 500, 1000)
• BatchParallelStreamAsync (Streaming batches)
• Baseline (Individual item processing)

Configuration: 10,000 items, MaxDegreeOfParallelism = 4

Analysis: Demonstrates optimal batch sizing for different workload types.

3. ErrorHandlingBenchmarks¶

Measures the overhead of error handling and retry mechanisms:

• No retries (Success path - baseline)
• With retries (10% transient failures)
• ErrorMode.FailFast
• ErrorMode.BestEffort
• Backoff.Exponential
• Backoff.ExponentialJitter

Configuration: 500 items, MaxDegreeOfParallelism = 8

Insights: Quantifies the performance impact of retry policies and error modes.

4. AdvancedFeaturesBenchmarks¶

Compares the overhead of advanced resilience features:

• No advanced features (baseline)
• With CircuitBreaker
• With RateLimit (Token bucket)
• With AdaptiveConcurrency
• With Progress tracking
• With Metrics tracking
• With all features combined

Configuration: 500 items, MaxDegreeOfParallelism = 16

Purpose: Helps understand the performance cost of production-grade features.

5. ConcurrencyScalingBenchmarks¶

Analyzes how performance scales with different MaxDegreeOfParallelism values:

• MaxDegreeOfParallelism: 1, 2, 4, 8 (baseline), 16, 32, 64, 128

Configuration: 1,000 items with 1ms I/O simulation per item

Goal: Find the optimal concurrency level for different workload characteristics.

Benchmark Configuration¶

All benchmarks run on both .NET 8.0 and .NET 9.0 to compare performance across runtime versions.

Common Settings¶

• MemoryDiagnoser: Enabled to track allocations
• MarkdownExporter: Generates markdown-formatted results
• Job: SimpleJob for both Net80 and Net90

Customizing Runs¶

# Run with specific runtime only
dotnet run -c Release -- --runtimes net8.0

# Run with specific number of iterations
dotnet run -c Release -- --iterationCount 10

# Run with memory profiling
dotnet run -c Release -- --memory

# Run with thread pool sizing diagnostics
dotnet run -c Release -- --envVars COMPlus_ThreadPool_ForceMinWorkerThreads:100

Interpreting Results¶

Key Metrics¶

• Mean: Average execution time
• StdDev: Standard deviation (variability)
• Median: Middle value (less affected by outliers)
• Allocated: Memory allocated per operation
• Gen0/Gen1/Gen2: Garbage collection counts

Performance Goals¶

• I/O-Bound Operations: Rivulet should significantly outperform sequential processing
• CPU-Bound Operations: Performance should scale linearly with cores up to hardware limits
• Memory: Should have minimal allocations compared to Task.WhenAll
• Overhead: Advanced features should add <10% overhead when not triggered

Baseline Comparisons¶

• Sequential Processing: Shows maximum possible speedup
• Task.WhenAll (unbounded): Shows cost of bounded concurrency vs unbounded

Example Results¶

// .NET 8.0 vs .NET 9.0 - SelectParallelAsync I/O simulation (1000 items, 1ms delay each)

|                 Method |  Runtime |     Mean | Allocated |
|----------------------- |--------- |---------:|----------:|
|    SelectParallelAsync | .NET 8.0 |  498 ms  |   ~1 MB   |
|    SelectParallelAsync | .NET 9.0 |  492 ms  |   ~0.9 MB | (Faster!)
| Sequential Processing  | .NET 8.0 | 1004 ms  |   ~0.5 MB |
|         Task.WhenAll   | .NET 8.0 |   45 ms  |   ~5 MB   | (Unbounded!)

CI/CD Integration¶

To run benchmarks in CI/CD pipelines:

# GitHub Actions example
- name: Run Benchmarks
  run: |
    cd tests/Rivulet.Benchmarks
    dotnet run -c Release -- --filter "*CoreOperatorsBenchmarks*" --job short --exporters json

Contributing Benchmarks¶

When adding new benchmarks:

1. Focus: Each benchmark should measure one specific aspect
2. Baseline: Include a baseline for comparison
3. Configuration: Use realistic workload sizes
4. Documentation: Add comments explaining what is being measured
5. Naming: Use descriptive names following the existing pattern

Resources¶

• BenchmarkDotNet Documentation
• .NET Performance Best Practices
• Rivulet.Core Main README

Benchmark Results Summary¶

Test Environment: - BenchmarkDotNet: v0.14.0 - OS: Windows 11 22H2 - CPU: AMD Ryzen 9 9950X (32 logical, 16 physical cores) - RAM: 64GB DDR5 6000MHz

1. Advanced Features Overhead¶

Measures the performance impact of production-grade features. Configuration: 500 items, 2ms delay each, MaxDegreeOfParallelism = 16

Key Insights: - ✅ Circuit breaker, rate limiting, progress, and metrics add <1% overhead - ⚠️ Adaptive concurrency adds ~3.5x overhead due to continuous sampling and adjustment - ✅ Combined features still maintain minimal overhead (~1%) - ✅ .NET 9.0 shows slightly better performance with combined features

2. Batching Performance¶

Evaluates batch processing efficiency with different batch sizes. Configuration: 5,000 items, MaxDegreeOfParallelism = 4

Key Insights: - ✅ 78-157x speedup - Batch processing is dramatically faster than individual item processing - ✅ Larger batches = better performance - Batch size 500-1000 provides ~2x improvement over batch size 100 - ✅ Minimal memory overhead - Batching uses 97-98% less memory than individual processing - ✅ Stream vs non-stream - BatchParallelStreamAsync has similar performance with slightly higher allocations - ✅ .NET 9.0 consistency - Negligible difference between .NET 8.0 and .NET 9.0 for batch operations

3. Concurrency Scaling¶

Analyzes performance scaling with different concurrency levels. Configuration: 1,000 items, 1ms delay per item

Key Insights: - ✅ Perfect linear scaling - Performance doubles with each doubling of concurrency (1→2→4→8) - ✅ Efficient up to 128 threads - Maintains near-linear scaling even at high concurrency - ✅ Minimal memory growth - Allocations grow only ~15% from concurrency 1 to 128 - ✅ I/O-bound workloads excel - 125x speedup for I/O-heavy operations - ✅ .NET 8.0 vs 9.0 parity - Virtually identical performance across both runtimes

4. Core Operators Performance¶

Measures core parallel operator performance across different workload types. Configuration: 1,000 items

Key Insights: - ✅ 31x speedup for I/O operations - Dramatically outperforms sequential processing for I/O-bound workloads - ✅ Near Task.WhenAll performance - Rivulet provides controlled concurrency with minimal overhead vs unbounded - ✅ Ordered output cost - Maintaining order adds ~27% memory overhead but same execution time - ✅ .NET 9.0 improvements - StreamAsync and ForEachAsync show 15-20% better performance on .NET 9.0 - ✅ Memory efficient - CPU-bound operations use only 2x baseline memory despite parallelization

5. Error Handling and Retry Performance¶

Measures the overhead of error handling mechanisms and retry policies. Configuration: 500 items, MaxDegreeOfParallelism = 8

Key Insights: - ✅ Minimal overhead when no errors - FailFast and BestEffort add <3% overhead on success path - ✅ Jitter reduces contention - ExponentialJitter is 4x faster than standard Exponential backoff - ✅ Memory efficient retries - Retry mechanism uses 50% less memory than baseline - ✅ .NET 9.0 performance win - 22% faster execution on success path (714μs → 554μs) - ⚠️ Retry cost - With 10% transient failures, expect ~150x slowdown (due to delays between retries)

Note: Benchmark results vary based on hardware, OS, and system load. Always run benchmarks on the same machine with minimal background processes for consistent comparisons.