Module failover 

Provider Failover Chain

Wraps multiple ModelWorker implementations in a priority-ordered failover chain. On failure the chain automatically retries with the next provider, skipping any provider that the health monitor currently marks as unusable.

Example

use std::sync::Arc;
use tokio_prompt_orchestrator::{EchoWorker, ModelWorker};
use tokio_prompt_orchestrator::provider_health::ProviderHealthMonitor;
use tokio_prompt_orchestrator::failover::FailoverChain;

#[tokio::main]
async fn main() {
    let health = ProviderHealthMonitor::new(50);
    let chain = FailoverChain::new(health)
        .add_worker("primary", Arc::new(EchoWorker::new()))
        .add_worker("backup", Arc::new(EchoWorker::new()))
        .with_max_attempts(3);

    let (provider_used, response) = chain.infer("Hello!").await.unwrap();
    println!("Answered by {}: {}", provider_used, response);
}

Structs

FailoverChain

A priority-ordered chain of model workers with automatic failover.