Struct LoadBalancedWorker 

pub struct LoadBalancedWorker { /* private fields */ }

A worker pool that distributes inference requests across multiple backends.

Wraps any number of Arc<dyn ModelWorker> instances and routes requests using a configurable LoadBalanceStrategy. All strategies are thread-safe; the pool is safe to share across Tokio tasks via Arc.

Example

use std::sync::Arc;

let pool = LoadBalancedWorker::round_robin(vec![
    Arc::new(AnthropicWorker::new("claude-sonnet-4-6")?) as Arc<dyn tokio_prompt_orchestrator::ModelWorker>,
    Arc::new(OpenAiWorker::new("gpt-4o")?) as Arc<dyn tokio_prompt_orchestrator::ModelWorker>,
]);

Implementations

impl LoadBalancedWorker

pub fn round_robin(workers: Vec<Arc<dyn ModelWorker>>) -> Self

Create a round-robin pool.

Panics

Panics if workers is empty.

pub fn replicate(worker: Arc<dyn ModelWorker>, n: usize) -> Self

Create a round-robin pool by replicating a single worker n times.

All pool slots share the same underlying Arc; this is useful for concurrency control rather than load distribution across different backends.

Panics

Panics if n is zero.

pub fn least_loaded(workers: Vec<Arc<dyn ModelWorker>>) -> Self

Create a least-loaded pool.

Panics

Panics if workers is empty.

pub fn with_names(self, names: Vec<String>) -> Self

Attach human-readable names to each worker slot for metrics labels.

If names is shorter than the pool, unnamed workers default to "unknown".

pub fn len(&self) -> usize

Return the number of workers in the pool.

pub fn is_empty(&self) -> bool

Return true if the pool is empty (should never be true after construction).

Trait Implementations

impl ModelWorker for LoadBalancedWorker

fn infer<'life0, 'life1, 'async_trait>( &'life0 self, prompt: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<Vec<String>, OrchestratorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform inference on the given prompt.

fn infer_stream<'life0, 'life1, 'async_trait>( &'life0 self, prompt: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<TokenStream, OrchestratorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Stream inference tokens as they arrive from the provider.