Struct ProviderCascade 

pub struct ProviderCascade { /* private fields */ }

A configurable chain of LLM providers that tries primary → secondary → tertiary on failure.

Unlike the binary LocalWithFallback routing decision, ProviderCascade supports an arbitrary number of providers and maintains per-provider circuit breakers, latency telemetry, and Prometheus counters.

Clone behaviour

ProviderCascade is cheap to clone — all clones share the same underlying Arc<Mutex<Vec<CascadeEntry>>> and its contained CircuitBreaker / atomic state.

Examples

use std::time::Duration;
use tokio_prompt_orchestrator::config::WorkerKind;
use tokio_prompt_orchestrator::routing::cascade::{CascadeEntry, CircuitBreakerConfig, ProviderCascade};

let cascade = ProviderCascade::new(vec![
    CascadeEntry::new(WorkerKind::LlamaCpp, CircuitBreakerConfig::default()),
    CascadeEntry::new(WorkerKind::OpenAi,   CircuitBreakerConfig::default()),
]);
assert_eq!(cascade.len(), 2);

Implementations

impl ProviderCascade

pub fn new(entries: Vec<CascadeEntry>) -> Self

Create a new ProviderCascade from an ordered list of cascade entries.

The first entry is the primary provider; subsequent entries are tried in order on failure.

Panics

This function does not panic.

pub fn len(&self) -> usize

Return the number of providers in the cascade.

Panics

This function does not panic.

pub fn is_empty(&self) -> bool

Return true if no providers are configured.

Panics

This function does not panic.

pub async fn call<F, Fut, T, E>( &self, f: F, ) -> Result<CascadeResult<T>, CascadeError<E>>

where F: FnMut(WorkerKind) -> Fut, Fut: Future<Output = Result<T, E>>, E: Debug,

Try each provider in order, returning the first successful result.

Open circuit breakers are skipped. For each attempted provider, Prometheus counters cascade_attempts_total and cascade_failures_total are updated. Per-entry latency and success rate atomics are also updated.

Arguments

• f — An async factory that receives a WorkerKind and returns a Future<Output = Result<T, E>>. It will be called once per attempted provider.

Errors

Returns CascadeError::EmptyCascade when no entries are configured, or CascadeError::AllFailed when every provider either had an open breaker or returned an error.

Panics

This function does not panic.

pub fn snapshot(&self) -> Vec<CascadeEntry>

Return a point-in-time snapshot of all cascade entries.

Useful for health-check endpoints and dashboards.

Panics

This function does not panic.

Trait Implementations

impl Clone for ProviderCascade

fn clone(&self) -> ProviderCascade

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ProviderCascade

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.