Module load_balancer 

Multi-Model Load Balancer

Weighted round-robin selection of model endpoints with health tracking and automatic failover.

Selection Algorithm

Uses the smooth weighted round-robin algorithm (Nginx-style): each endpoint’s current_weight is incremented by its weight on every call to LoadBalancer::select; the endpoint with the highest current_weight is chosen and then has total_weight subtracted from its current_weight.

Health Tracking

• After 3 consecutive failures an endpoint is marked unhealthy.
• 1 success recovers an endpoint.
• When failover = true unhealthy endpoints are skipped during selection.

Structs

BalancerConfig

Configuration for the LoadBalancer.

EndpointStats

Per-endpoint statistics tracked by the LoadBalancer.

LoadBalancer

Thread-safe weighted round-robin load balancer for model endpoints.

LoadBalancerStats

Aggregate statistics for the entire balancer.

ModelEndpoint

A single model inference endpoint.