Struct AdaptiveTimeoutManager 

pub struct AdaptiveTimeoutManager { /* private fields */ }

Thread-safe, per-model adaptive timeout manager.

Uses a [DashMap] so that multiple tokio tasks can record outcomes and query timeouts concurrently without a global lock.

Implementations

impl AdaptiveTimeoutManager

pub fn new() -> Self

Create a new manager with an empty model registry.

pub fn record_outcome(&self, model: &str, duration_ms: u64, success: bool)

Record the outcome of one request for model.

This updates the rolling sample window and the EMA.

pub fn get_timeout(&self, model: &str) -> Duration

Compute the recommended timeout for model.

Returns p95 * 1.5, clamped to [5 s, 120 s]. Falls back to 30 s when fewer than two samples have been collected.

pub fn adjust_for_load(&self, model: &str, queue_depth: usize) -> Duration

Adjust the base timeout by the current queue depth.

Scales get_timeout by sqrt(queue_depth / 10).max(1.0) so that a backlogged queue gets proportionally more time.

pub fn model_summary(&self, model: &str) -> Option<TimeoutSummary>

Return a TimeoutSummary snapshot for model, or None if the model has never been seen.

Trait Implementations

impl Default for AdaptiveTimeoutManager

fn default() -> Self

Returns the “default value” for a type.