Struct LoadBalancer 

pub struct LoadBalancer { /* private fields */ }

Thread-safe weighted round-robin load balancer for model endpoints.

Example

use tokio_prompt_orchestrator::load_balancer::{
    BalancerConfig, LoadBalancer, ModelEndpoint,
};
use std::time::Duration;

let config = BalancerConfig {
    endpoints: vec![
        ModelEndpoint {
            id: "ep-a".to_string(),
            url: "http://a".to_string(),
            weight: 2,
            max_rps: 100.0,
            healthy: true,
            latency_p99_ms: 0.0,
        },
        ModelEndpoint {
            id: "ep-b".to_string(),
            url: "http://b".to_string(),
            weight: 1,
            max_rps: 50.0,
            healthy: true,
            latency_p99_ms: 0.0,
        },
    ],
    health_check_interval: Duration::from_secs(30),
    failover: true,
};

let lb = LoadBalancer::new(config);
let ep = lb.select();
assert!(ep.is_some());

Implementations

impl LoadBalancer

pub fn new(config: BalancerConfig) -> Self

Create a new load balancer from the given configuration.

pub fn select(&self) -> Option<ModelEndpoint>

Select the next endpoint using weighted round-robin.

Returns None when:

• No endpoints are configured.
• All endpoints are unhealthy and failover = true.

pub fn mark_success(&self, id: &str, latency_ms: f64)

Record a successful response from endpoint id with the given latency.

Resets the consecutive-failure counter and marks the endpoint healthy.

pub fn mark_failure(&self, id: &str)

Record a failure from endpoint id.

After 3 consecutive failures the endpoint is marked unhealthy.

pub fn stats(&self) -> LoadBalancerStats

Return a snapshot of current balancer statistics.

pub fn failover_enabled(&self) -> bool

Return the configured failover setting.

Trait Implementations

impl Clone for LoadBalancer

fn clone(&self) -> LoadBalancer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.