Module smart_router 

Cost-Aware Smart Router

Routes prompts to the cheapest provider meeting the request’s SLA:

• Latency budget (max acceptable latency in ms)
• Quality floor (minimum acceptable quality score)
• Cost ceiling (maximum USD per request)

Algorithm

1. Filter providers: remove unhealthy and those exceeding the latency budget.
2. Filter providers: remove those below the quality floor.
3. Filter providers: remove those exceeding the cost ceiling.
4. If prefer_quality is set, select the highest-quality remaining provider.
5. Otherwise select the cheapest provider that meets all constraints.
6. The remaining providers (sorted cheapest-first) become the fallback list.

Pricing is configurable and can be hot-reloaded from a TOML string at runtime via SmartRouter::load_pricing_toml.

Structs

ModelPricing

Per-token pricing and capability metadata for one provider/model combination.

RoutingDecision

The result of a routing call: primary provider plus ordered fallbacks.

RoutingRequirements

SLA constraints for a single routing request.

SmartRouter

Routes incoming requests to the cheapest (or highest-quality) provider that satisfies all SLA constraints.