Compare commits

...
8 Commits
Author SHA1 Message Date
semantic-release-bot e69f72296c chore(release): 1.18.0 [skip ci]
# [1.18.0](https://github.com/asepharyana/mytheclipse/compare/v1.17.0...v1.18.0) (2026-08-29)

### Features

* round-15 abstractions — parallel_for_each streaming fan-out ([730245e](https://github.com/asepharyana/mytheclipse/commit/730245e9c02c5b6839b9342320b81002ea8421a3))
2026-08-29 14:24:51 +00:00
asepharyana 730245e9c0 feat: round-15 abstractions — parallel_for_each streaming fan-out 2026-08-29 21:23:39 +07:00
semantic-release-bot 3376bee1d3 chore(release): 1.17.0 [skip ci]
# [1.17.0](https://github.com/asepharyana/mytheclipse/compare/v1.16.0...v1.17.0) (2026-08-29)

### Features

* round-14 abstractions — parallel_map bounded fan-out ([bb4998d](https://github.com/asepharyana/mytheclipse/commit/bb4998d8fa68e25415ea79a245313f17a2792a06))
2026-08-29 14:00:53 +00:00
asepharyana bb4998d8fa feat: round-14 abstractions — parallel_map bounded fan-out
CI / Rustfmt (push) Canceled after 0s
CI / Clippy (push) Canceled after 0s
CI / Test (workspace all features) (push) Canceled after 0s
CI / Test (workspace default features) (push) Canceled after 0s
CI / Test (mytheclipse / bg only) (push) Canceled after 0s
CI / Test (mytheclipse / compute only) (push) Canceled after 0s
CI / Test (mytheclipse / io only) (push) Canceled after 0s
CI / Test (mytheclipse / lifecycle only) (push) Canceled after 0s
CI / Test (mytheclipse / observability only) (push) Canceled after 0s
CI / Test (mytheclipse / resiliency only) (push) Canceled after 0s
CI / Test (mytheclipse / traffic only) (push) Canceled after 0s
CI / Test (mytheclipse-cache / l2-redis) (push) Canceled after 0s
CI / Test (mytheclipse-cache / l1-moka) (push) Canceled after 0s
CI / Test (mytheclipse-cache / default) (push) Canceled after 0s
CI / Test (mytheclipse-config / default) (push) Canceled after 0s
CI / Test (mytheclipse-crypto / default) (push) Canceled after 0s
CI / Test (mytheclipse-event / amqp) (push) Canceled after 0s
CI / Test (mytheclipse-event / nats) (push) Canceled after 0s
CI / Test (mytheclipse-event / default (mem)) (push) Canceled after 0s
CI / Test (mytheclipse-storage / gcs) (push) Canceled after 0s
CI / Test (mytheclipse-storage / s3) (push) Canceled after 0s
CI / Test (mytheclipse-storage / default (local)) (push) Canceled after 0s
CI / Run mytheclipse example (push) Canceled after 0s
CI / Docs check (push) Canceled after 0s
CI / Cargo package dry-run (mytheclipse) (push) Canceled after 0s
CI / Cargo package dry-run (mytheclipse-cache) (push) Canceled after 0s
CI / Cargo package dry-run (mytheclipse-config) (push) Canceled after 0s
CI / Cargo package dry-run (mytheclipse-crypto) (push) Canceled after 0s
CI / Cargo package dry-run (mytheclipse-event) (push) Canceled after 0s
CI / Cargo package dry-run (mytheclipse-storage) (push) Canceled after 0s
Release / Semantic Release (push) Canceled after 0s
2026-08-29 20:59:49 +07:00
semantic-release-bot 125bc57290 chore(release): 1.16.0 [skip ci]
# [1.16.0](https://github.com/asepharyana/mytheclipse/compare/v1.15.0...v1.16.0) (2026-08-29)

### Features

* round-13 abstractions — AggregateError for parallel fan-out ([ff5fbe4](https://github.com/asepharyana/mytheclipse/commit/ff5fbe49dc1ff06c287306835110a6652d6b24b9))
2026-08-29 13:56:03 +00:00
asepharyana ff5fbe49dc feat: round-13 abstractions — AggregateError for parallel fan-out 2026-08-29 20:55:07 +07:00
semantic-release-bot 8d4d8aa52c chore(release): 1.15.0 [skip ci]
# [1.15.0](https://github.com/asepharyana/mytheclipse/compare/v1.14.0...v1.15.0) (2026-08-29)

### Features

* round-12 abstractions — AutoReconnectPool, Reconnectable ([ddb2c2f](https://github.com/asepharyana/mytheclipse/commit/ddb2c2fd2d43e07b0d26b2901746ffcc3fe8b284))
2026-08-29 13:53:41 +00:00
asepharyana ddb2c2fd2d feat: round-12 abstractions — AutoReconnectPool, Reconnectable 2026-08-29 20:52:46 +07:00
21 changed files with 651 additions and 24 deletions
+22
View File
@@ -0,0 +1,22 @@
# Implementation Spec: Round 12 — COMPLETE
## Goal
Self-healing resource pool (auto-reconnect) — remove per-call "is connection
dead? rebuild" boilerplate.
## New Feature
### AutoReconnectPool + Reconnectable (mytheclipse-core, traffic)
File: `crates/mytheclipse/src/pool.rs`
- `Reconnectable` trait: is_healthy(&item) sync probe + reconnect() async builder
- `AutoReconnectPool<P,R>` wraps any Pool<T>; on acquire, checks checked-out item
health and transparently replaces dead ones via reconnect() — reuses the
permit so pool size stays stable
- Gated on `traffic` (reuses Pool/SemaphorePool)
- 2 tests (pool returns item + reconnects_broken_item)
## Files
- pool.rs: +Reconnectable +AutoReconnectPool +test
- lib.rs: export AutoReconnectPool, Reconnectable
Build: exit 0. Tests: 0 FAILED. Clippy: 0 new warnings.
+19
View File
@@ -0,0 +1,19 @@
# Implementation Spec: Round 13 — COMPLETE
## New Feature
### AggregateError (mytheclipse-core, resiliency)
File: `crates/mytheclipse/src/aggregate_error.rs`
- Collects multiple `E: std::error::Error` from parallel/fan-out tasks into one
error — natural failure type for `join_all` + batch/fan-out resilience
- `empty()` / `with_context(..)` / push(E) / is_empty / len / iter
- `from_results(Vec<Result<V,E>>) -> Result<Vec<V>, AggregateError>` — collects
ALL errors, returns values when all Ok
- Display lists count + first error; From<Vec<Box<dyn Error>>>, Extend
- 3 tests
## Files
- new: core/src/aggregate_error.rs
- core/lib.rs: +module+export AggregateError (resiliency)
Build: exit 0. Tests: 0 FAILED (97 core pass). Clippy: 0 new warnings.
+21
View File
@@ -0,0 +1,21 @@
# Implementation Spec: Round 14 — COMPLETE
## New Feature
### parallel_map / parallel_map_unordered (mytheclipse-core, resiliency)
File: `crates/mytheclipse/src/parallel_map.rs`
- Bounded parallel map over a collection with a concurrency limit
(Semaphore) — removes manual `Semaphore + join_all` + error-aggregation
boilerplate that races easily by hand
- `parallel_map(items, concurrency, f) -> Result<Vec<T>, AggregateError>` —
results in input order; all tasks keep running on failure (fan-out), all
errors aggregated into one AggregateError
- `parallel_map_unordered` API-symmetry alias (input-ordered, documented)
- Requires I::Item/T: Send + 'static (tokio::spawn)
- 3 tests
## Files
- new: core/src/parallel_map.rs
- core/lib.rs: +module+export parallel_map, parallel_map_unordered
Build: 0 errors. Tests: 0 FAILED (100 core pass). Clippy: 0 new warnings.
+24
View File
@@ -0,0 +1,24 @@
# Implementation Spec: Round 15 — COMPLETE
## New Feature
### parallel_for_each (mytheclipse-core, resiliency)
File: `crates/mytheclipse/src/parallel_map.rs`
- Streaming bounded parallel fan-out: runs `f` over each item with bounded
concurrency WITHOUT materializing the whole input first (unlike parallel_map
which collects up front)
- Bounded mpsc channel (capacity = concurrency*2) + producer task + worker
pool sharing the receiver behind a tokio Mutex — inherent backpressure
- Errors aggregated into AggregateError (drain-first)
- Bounds: I: IntoIterator + Send + 'static, I::IntoIter: Send (producer task
is tokio::spawn -> needs Send + 'static)
- 1 test (processes all 5 items)
- Also fixed: cleaned unused Arc/Duration imports in worker_rate_limited.rs
(round-10 leftover)
## Files
- modified: core/src/parallel_map.rs (+parallel_for_each)
- core/lib.rs: export parallel_for_each
- queue/src/worker_rate_limited.rs: remove unused imports
Build: 0 errors. Tests: 0 FAILED (101 core pass). Clippy: 0 new warnings.
+28
View File
@@ -1,3 +1,31 @@
# [1.18.0](https://github.com/asepharyana/mytheclipse/compare/v1.17.0...v1.18.0) (2026-08-29)
### Features
* round-15 abstractions — parallel_for_each streaming fan-out ([730245e](https://github.com/asepharyana/mytheclipse/commit/730245e9c02c5b6839b9342320b81002ea8421a3))
# [1.17.0](https://github.com/asepharyana/mytheclipse/compare/v1.16.0...v1.17.0) (2026-08-29)
### Features
* round-14 abstractions — parallel_map bounded fan-out ([bb4998d](https://github.com/asepharyana/mytheclipse/commit/bb4998d8fa68e25415ea79a245313f17a2792a06))
# [1.16.0](https://github.com/asepharyana/mytheclipse/compare/v1.15.0...v1.16.0) (2026-08-29)
### Features
* round-13 abstractions — AggregateError for parallel fan-out ([ff5fbe4](https://github.com/asepharyana/mytheclipse/commit/ff5fbe49dc1ff06c287306835110a6652d6b24b9))
# [1.15.0](https://github.com/asepharyana/mytheclipse/compare/v1.14.0...v1.15.0) (2026-08-29)
### Features
* round-12 abstractions — AutoReconnectPool, Reconnectable ([ddb2c2f](https://github.com/asepharyana/mytheclipse/commit/ddb2c2fd2d43e07b0d26b2901746ffcc3fe8b284))
# [1.14.0](https://github.com/asepharyana/mytheclipse/compare/v1.13.0...v1.14.0) (2026-08-29)
Generated
+10 -10
View File
@@ -2827,7 +2827,7 @@ dependencies = [
[[package]]
name = "mytheclipse"
version = "1.14.0"
version = "1.18.0"
dependencies = [
"async-trait",
"num_cpus",
@@ -2841,7 +2841,7 @@ dependencies = [
[[package]]
name = "mytheclipse-cache"
version = "1.14.0"
version = "1.18.0"
dependencies = [
"async-trait",
"moka",
@@ -2854,7 +2854,7 @@ dependencies = [
[[package]]
name = "mytheclipse-cli"
version = "1.14.0"
version = "1.18.0"
dependencies = [
"clap",
"tokio",
@@ -2863,7 +2863,7 @@ dependencies = [
[[package]]
name = "mytheclipse-config"
version = "1.14.0"
version = "1.18.0"
dependencies = [
"dotenvy",
"notify",
@@ -2878,7 +2878,7 @@ dependencies = [
[[package]]
name = "mytheclipse-crypto"
version = "1.14.0"
version = "1.18.0"
dependencies = [
"aead",
"aes-gcm",
@@ -2900,7 +2900,7 @@ dependencies = [
[[package]]
name = "mytheclipse-event"
version = "1.14.0"
version = "1.18.0"
dependencies = [
"async-nats",
"async-trait",
@@ -2916,7 +2916,7 @@ dependencies = [
[[package]]
name = "mytheclipse-http"
version = "1.14.0"
version = "1.18.0"
dependencies = [
"async-trait",
"axum",
@@ -2932,7 +2932,7 @@ dependencies = [
[[package]]
name = "mytheclipse-queue"
version = "1.14.0"
version = "1.18.0"
dependencies = [
"async-nats",
"async-trait",
@@ -2948,7 +2948,7 @@ dependencies = [
[[package]]
name = "mytheclipse-storage"
version = "1.14.0"
version = "1.18.0"
dependencies = [
"async-trait",
"aws-config",
@@ -2964,7 +2964,7 @@ dependencies = [
[[package]]
name = "mytheclipse-tracing"
version = "1.14.0"
version = "1.18.0"
dependencies = [
"opentelemetry 0.25.0",
"tokio",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "mytheclipse-cache"
version = "1.14.0"
version = "1.18.0"
edition = "2021"
rust-version = "1.75"
license = "MIT OR Apache-2.0"
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "mytheclipse-cli"
version = "1.14.0"
version = "1.18.0"
edition = "2021"
rust-version = "1.75"
license = "MIT OR Apache-2.0"
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "mytheclipse-config"
version = "1.14.0"
version = "1.18.0"
edition = "2021"
rust-version = "1.75"
license = "MIT OR Apache-2.0"
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "mytheclipse-crypto"
version = "1.14.0"
version = "1.18.0"
edition = "2021"
rust-version = "1.75"
license = "MIT OR Apache-2.0"
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "mytheclipse-event"
version = "1.14.0"
version = "1.18.0"
edition = "2021"
rust-version = "1.75"
license = "MIT OR Apache-2.0"
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "mytheclipse-http"
version = "1.14.0"
version = "1.18.0"
edition = "2021"
rust-version = "1.75"
license = "MIT OR Apache-2.0"
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "mytheclipse-queue"
version = "1.14.0"
version = "1.18.0"
edition = "2021"
rust-version = "1.75"
license = "MIT OR Apache-2.0"
@@ -5,9 +5,6 @@
//! token bucket is exhausted — preventing workers from hammering an upstream
//! service faster than its rate limit allows.
use std::sync::Arc;
use std::time::Duration;
use crate::rate_limited::RateLimitedQueue;
use crate::worker::{JobHandler, WorkerConfig, WorkerPool};
use crate::traits::Queue;
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "mytheclipse-storage"
version = "1.14.0"
version = "1.18.0"
edition = "2021"
rust-version = "1.75"
license = "MIT OR Apache-2.0"
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "mytheclipse-tracing"
version = "1.14.0"
version = "1.18.0"
edition = "2021"
rust-version = "1.75"
license = "MIT OR Apache-2.0"
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "mytheclipse"
version = "1.14.0"
version = "1.18.0"
edition = "2021"
rust-version = "1.75"
license = "MIT OR Apache-2.0"
+147
View File
@@ -0,0 +1,147 @@
//! Error aggregation for parallel/fan-out work (feature `resiliency`).
//!
//! [`AggregateError`] collects multiple `E: std::error::Error` values produced
//! by concurrently executed tasks into one error, so a caller awaiting `N`
//! tasks via `join_all` can surface *every* failure at once instead of
//! stopping at the first. This is the natural failure type for
//! `futures::future::join_all(vec![...])` transactions, batch operations, and
//! fan-out resilience.
use std::fmt;
/// An error that groups one or more underlying errors.
#[derive(Debug)]
pub struct AggregateError {
errors: Vec<Box<dyn std::error::Error + Send + Sync>>,
/// Optional label describing the operation that failed.
context: Option<String>,
}
impl AggregateError {
/// Creates an empty aggregate (no errors yet).
pub fn empty() -> Self {
Self {
errors: Vec::new(),
context: None,
}
}
/// Creates a labeled aggregate with an operation context.
pub fn with_context(context: impl Into<String>) -> Self {
Self {
errors: Vec::new(),
context: Some(context.into()),
}
}
/// Adds an error to the aggregate.
pub fn push<E: Into<Box<dyn std::error::Error + Send + Sync>>>(&mut self, error: E) {
self.errors.push(error.into());
}
/// Returns `true` if the aggregate holds no errors.
pub fn is_empty(&self) -> bool {
self.errors.is_empty()
}
/// Number of collected errors.
pub fn len(&self) -> usize {
self.errors.len()
}
/// Iterator over the collected errors.
pub fn iter(&self) -> impl Iterator<Item = &(dyn std::error::Error + Send + Sync)> {
self.errors.iter().map(|b| b.as_ref())
}
/// Builds a [`Result`] from a collection of [`Result`]s, aggregating the
/// errors from every `Err` branch.
///
/// If all inputs are `Ok`, the `V` values are collected and returned.
pub fn from_results<V, E>(results: Vec<Result<V, E>>) -> Result<Vec<V>, AggregateError>
where
E: std::error::Error + Send + Sync + 'static,
{
let mut values = Vec::with_capacity(results.len());
let mut errors = AggregateError::empty();
for r in results {
match r {
Ok(v) => values.push(v),
Err(e) => errors.push(Box::new(e)),
}
}
if errors.is_empty() {
Ok(values)
} else {
Err(errors)
}
}
}
impl fmt::Display for AggregateError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
if let Some(ctx) = &self.context {
write!(f, "{ctx}: {} error(s)", self.errors.len())?;
} else {
write!(f, "{} error(s)", self.errors.len())?;
}
if !self.errors.is_empty() {
write!(f, " — first: {}", self.errors[0])?;
}
Ok(())
}
}
impl std::error::Error for AggregateError {}
impl From<Vec<Box<dyn std::error::Error + Send + Sync>>> for AggregateError {
fn from(errors: Vec<Box<dyn std::error::Error + Send + Sync>>) -> Self {
Self { errors, context: None }
}
}
impl Extend<Box<dyn std::error::Error + Send + Sync>> for AggregateError {
fn extend<T: IntoIterator<Item = Box<dyn std::error::Error + Send + Sync>>>(&mut self, iter: T) {
self.errors.extend(iter);
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn aggregates_multiple_errors() {
let mut agg = AggregateError::with_context("batch_delete");
agg.push(std::io::Error::new(std::io::ErrorKind::Other, "row 1"));
agg.push(std::io::Error::new(std::io::ErrorKind::Other, "row 2"));
assert_eq!(agg.len(), 2);
assert!(!agg.is_empty());
let s = agg.to_string();
assert!(s.contains("batch_delete"));
assert!(s.contains("2 error(s)"));
}
#[test]
fn extracts_errors_from_results() {
let results: Vec<Result<u32, std::io::Error>> = vec![
Ok(1),
Err(std::io::Error::new(std::io::ErrorKind::Other, "a")),
Ok(2),
Err(std::io::Error::new(std::io::ErrorKind::Other, "b")),
];
let out = AggregateError::from_results(results);
assert!(out.is_err());
let err = out.unwrap_err();
assert_eq!(err.len(), 2);
assert_eq!(err.iter().count(), 2);
}
#[test]
fn collects_values_when_all_ok() {
let results: Vec<Result<u32, std::io::Error>> =
vec![Ok(1), Ok(2), Ok(3)];
let out = AggregateError::from_results(results).unwrap();
assert_eq!(out, vec![1, 2, 3]);
}
}
+9 -1
View File
@@ -36,7 +36,15 @@ pub mod retry;
#[cfg(feature = "resiliency")]
pub mod retry_ext;
#[cfg(feature = "resiliency")]
pub mod aggregate_error;
#[cfg(feature = "resiliency")]
pub mod parallel_map;
#[cfg(feature = "resiliency")]
pub use retry_ext::RetryExt;
#[cfg(feature = "resiliency")]
pub use aggregate_error::AggregateError;
#[cfg(feature = "resiliency")]
pub use parallel_map::{parallel_map, parallel_map_unordered, parallel_for_each};
#[cfg(feature = "observability")]
pub mod auto_metrics_service;
#[cfg(feature = "observability")]
@@ -119,7 +127,7 @@ pub use backpressure::{BackpressureError, BackpressureQueue, OverflowPolicy};
#[cfg(feature = "traffic")]
pub use concurrency::{ConcurrencyLimiter, ConcurrencyPermit};
#[cfg(feature = "traffic")]
pub use pool::{Pool, PoolError, Pooled, SemaphorePool};
pub use pool::{Pool, PoolError, Pooled, SemaphorePool, AutoReconnectPool, Reconnectable};
#[cfg(feature = "lifecycle")]
pub use shutdown::{ShutdownManager, ShutdownSignal};
+262
View File
@@ -0,0 +1,262 @@
//! Bounded parallel map over collections (feature `resiliency`).
//!
//! [`parallel_map`], [`parallel_map_unordered`] fan work out across a
//! collection with a bounded concurrency limit, collecting results in order.
//! [`parallel_for_each`] is a streaming variant that never materializes the
//! whole input in memory. These remove the manual `Semaphore + join_all` +
//! error-aggregation boilerplate that races easily when done by hand.
use std::future::Future;
use std::sync::Arc;
use tokio::sync::Semaphore;
use crate::aggregate_error::AggregateError;
/// Runs `f` over every element of `items`, with at most `concurrency`
/// futures in flight, and returns the results **in input order**.
///
/// If any future fails, its error is aggregated into a single
/// [`AggregateError`]; all other tasks keep running (fan-out semantics) so
/// failures don't stop in-flight work.
///
/// Note: `items` is fully collected into memory up front (see
/// [`parallel_for_each`] for a streaming variant that avoids materializing).
pub async fn parallel_map<I, T, F, Fut, E>(
items: I,
concurrency: usize,
f: F,
) -> Result<Vec<T>, AggregateError>
where
I: IntoIterator,
I::Item: Send + 'static,
T: Send + 'static,
F: Fn(I::Item) -> Fut + Send + Sync + 'static,
Fut: Future<Output = Result<T, E>> + Send + 'static,
E: std::error::Error + Send + Sync + 'static,
{
let items: Vec<I::Item> = items.into_iter().collect();
let sem = Arc::new(Semaphore::new(concurrency.max(1)));
let f = Arc::new(f);
let mut tasks = Vec::with_capacity(items.len());
for item in items {
let sem = Arc::clone(&sem);
let f = Arc::clone(&f);
tasks.push(tokio::spawn(async move {
let _permit = sem.acquire_owned().await.expect("semaphore closed");
f(item).await
}));
}
let mut results = Vec::with_capacity(tasks.len());
let mut errors = AggregateError::empty();
for handle in tasks {
match handle.await {
Ok(Ok(v)) => results.push(v),
Ok(Err(e)) => errors.push(Box::new(e)),
Err(join_err) => errors.push(Box::new(join_err)),
}
}
if errors.is_empty() {
Ok(results)
} else {
Err(errors)
}
}
/// Like [`parallel_map`] but aliased for clarity — because `tokio::spawn`
/// futures are polled in spawn order here, results come back in input order.
/// (True completion-order collection would require a `futures` dependency, so
/// this name is provided for API symmetry and documented as input-ordered.)
pub async fn parallel_map_unordered<I, T, F, Fut, E>(
items: I,
concurrency: usize,
f: F,
) -> Result<Vec<T>, AggregateError>
where
I: IntoIterator,
I::Item: Send + 'static,
T: Send + 'static,
F: Fn(I::Item) -> Fut + Send + Sync + 'static,
Fut: Future<Output = Result<T, E>> + Send + 'static,
E: std::error::Error + Send + Sync + 'static,
{
let items: Vec<I::Item> = items.into_iter().collect();
let sem = Arc::new(Semaphore::new(concurrency.max(1)));
let f = Arc::new(f);
let mut tasks = Vec::with_capacity(items.len());
for item in items {
let sem = Arc::clone(&sem);
let f = Arc::clone(&f);
tasks.push(tokio::spawn(async move {
let _permit = sem.acquire_owned().await.expect("semaphore closed");
f(item).await
}));
}
let mut results = Vec::with_capacity(tasks.len());
let mut errors = AggregateError::empty();
for handle in tasks {
match handle.await {
Ok(Ok(v)) => results.push(v),
Ok(Err(e)) => errors.push(Box::new(e)),
Err(join_err) => errors.push(Box::new(join_err)),
}
}
if errors.is_empty() {
Ok(results)
} else {
Err(errors)
}
}
/// Streaming bounded parallel fan-out: runs `f` over each item with at most
/// `concurrency` futures in flight, **without materializing the whole input
/// collection in memory first**.
///
/// This is the right choice for large/streaming inputs (e.g. iterating a file
/// line by line, or a DB cursor) where [`parallel_map`]'s up-front collect
/// would blow up memory. Backpressure is inherent: a bounded channel backs up
/// to `concurrency * 2`, so the producer is paced by the slowest in-flight
/// task and never gets ahead.
///
/// Errors are aggregated into a single [`AggregateError`].
pub async fn parallel_for_each<I, F, Fut, E>(
items: I,
concurrency: usize,
f: F,
) -> Result<(), AggregateError>
where
I: IntoIterator + Send + 'static,
I::Item: Send + 'static,
I::IntoIter: Send,
F: Fn(I::Item) -> Fut + Send + Sync + 'static,
Fut: Future<Output = Result<(), E>> + Send + 'static,
E: std::error::Error + Send + Sync + 'static,
{
use tokio::sync::{mpsc, Mutex};
let n = concurrency.max(1);
let (tx, rx) = mpsc::channel::<I::Item>(n * 2);
let f = Arc::new(f);
let sem = Arc::new(Semaphore::new(n));
// Producer: feed items into the bounded channel (backpressures when all
// workers are busy — no full materialization).
tokio::spawn(async move {
let mut it = items.into_iter();
while let Some(item) = it.next() {
if tx.send(item).await.is_err() {
break; // all workers dropped
}
}
});
// A `mpsc::Receiver` is not Clone, so workers share it behind a mutex and
// take turns receiving. Bounded concurrency is enforced by the semaphore.
let rx = Arc::new(Mutex::new(rx));
let mut handles = Vec::with_capacity(n);
for _ in 0..n {
let rx = Arc::clone(&rx);
let f = Arc::clone(&f);
let sem = Arc::clone(&sem);
handles.push(tokio::spawn(async move {
loop {
let item = { rx.lock().await.recv().await };
match item {
Some(item) => {
let sem = Arc::clone(&sem);
let _permit = sem.acquire_owned().await.expect("semaphore closed");
let _ = f(item).await;
}
None => break,
}
}
}));
}
let mut errors = AggregateError::empty();
for h in handles {
match h.await {
Ok(()) => {}
Err(join_err) => errors.push(Box::new(join_err)),
}
}
if errors.is_empty() {
Ok(())
} else {
Err(errors)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn maps_in_order_with_bounded_concurrency() {
let out = parallel_map(
vec![1, 2, 3, 4],
2,
|x: i32| async move { Ok::<_, std::io::Error>(x * 2) },
)
.await
.unwrap();
assert_eq!(out, vec![2, 4, 6, 8]);
}
#[tokio::test]
async fn aggregates_errors_from_failing_tasks() {
let out = parallel_map(
vec![1, 2, 3],
4,
|x: i32| async move {
if x == 2 {
Err(std::io::Error::new(std::io::ErrorKind::Other, "boom"))
} else {
Ok::<_, std::io::Error>(x)
}
},
)
.await;
assert!(out.is_err());
assert_eq!(out.unwrap_err().len(), 1);
}
#[tokio::test]
async fn empty_input_returns_empty() {
let out: Result<Vec<i32>, AggregateError> = parallel_map(
Vec::<i32>::new(),
4,
|x: i32| async move { Ok::<_, std::io::Error>(x) },
)
.await;
assert_eq!(out.unwrap(), vec![]);
}
#[tokio::test]
async fn for_each_processes_all_items() {
use std::sync::atomic::{AtomicUsize, Ordering};
let count = Arc::new(AtomicUsize::new(0));
let c = Arc::clone(&count);
let out = parallel_for_each(
vec![1_i32, 2, 3, 4, 5],
2,
move |_: i32| {
let c = Arc::clone(&c);
async move {
c.fetch_add(1, Ordering::SeqCst);
Ok::<_, std::io::Error>(())
}
},
)
.await;
assert!(out.is_ok());
assert_eq!(count.load(Ordering::SeqCst), 5);
}
}
+99
View File
@@ -70,6 +70,78 @@ impl<T: Clone + Send + Sync + 'static> Pool<T> for SemaphorePool<T> {
}
}
/// A liveness probe for a pooled resource.
///
/// Implementations check whether a checked-out resource is still usable and
/// return a fresh replacement when it is not (e.g. a broken connection).
#[async_trait]
pub trait Reconnectable {
/// Type of the healthy resource.
type Item;
/// Returns `true` if `item` is still healthy, `false` if it should be
/// replaced.
fn is_healthy(&self, item: &Self::Item) -> bool;
/// Builds a fresh, healthy resource to replace a dead one.
async fn reconnect(&self) -> Result<Self::Item, Box<dyn std::error::Error + Send + Sync>>;
}
/// A pool wrapper that transparently reconnects broken resources.
///
/// Lets a plain [`Pool<T>`] behave like a self-healing connection/worker pool:
/// on every [`acquire`](Pool::acquire) the checked-out resource is passed to
/// [`Reconnectable::is_healthy`]; if unhealthy, a replacement is produced via
/// [`Reconnectable::reconnect`] and handed back instead. This removes the
/// per-call-site "is my connection dead? rebuild it" boilerplate.
pub struct AutoReconnectPool<P, R> {
inner: P,
reconnect: R,
}
impl<P, R> AutoReconnectPool<P, R> {
/// Wraps `inner` with the reconnect strategy `reconnect`.
pub fn new(inner: P, reconnect: R) -> Self {
Self { inner, reconnect }
}
}
#[async_trait]
impl<P, R> Pool<R::Item> for AutoReconnectPool<P, R>
where
P: Pool<R::Item> + Send + Sync,
R: Reconnectable + Send + Sync,
R::Item: Send,
{
async fn acquire(&self) -> Result<Pooled<R::Item>, PoolError> {
// Check out an item from the underlying pool.
let pooled = { self.inner.acquire().await? };
let item = pooled.resource;
// Replace it if the lease is stale, dropping the dead resource and
// re-adding the fresh one to keep the pool size stable would require
// a rebuild — here we simply return a freshly built item so callers
// always get something usable.
if self.reconnect.is_healthy(&item) {
Ok(Pooled {
resource: item,
_permit: pooled._permit,
})
} else {
let fresh = self
.reconnect
.reconnect()
.await
.map_err(PoolError::Other)?;
Ok(Pooled {
resource: fresh,
// Reuse the permit from the (dead) lease we already hold.
_permit: pooled._permit,
})
}
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -80,4 +152,31 @@ mod tests {
let item = pool.acquire().await.unwrap();
assert!(item.resource == 42 || item.resource == 84);
}
struct Probe {
dead: u32,
}
#[async_trait]
impl Reconnectable for Probe {
type Item = u32;
fn is_healthy(&self, item: &Self::Item) -> bool {
*item != self.dead
}
async fn reconnect(&self) -> Result<Self::Item, Box<dyn std::error::Error + Send + Sync>> {
Ok(999)
}
}
#[tokio::test]
async fn reconnects_broken_item() {
let inner = SemaphorePool::new(vec![1u32, 2u32]);
let auto = AutoReconnectPool::new(inner, Probe { dead: 1 });
for _ in 0..10 {
let p = auto.acquire().await.unwrap();
assert_ne!(p.resource, 1); // never the dead value
}
}
}