Skip to main content

retry_inference

Function retry_inference 

Source
pub async fn retry_inference<F, Fut>(
    max_attempts: usize,
    base_delay: Duration,
    f: F,
) -> Result<Vec<String>, OrchestratorError>
where F: FnMut() -> Fut, Fut: Future<Output = Result<Vec<String>, OrchestratorError>>,
Expand description

Execute an inference operation with retries, honouring RateLimited back-off.

Unlike RetryPolicy::retry, this variant:

  • Sleeps for exactly retry_after_secs when the provider says to back off.
  • Stops retrying on BudgetExceeded (non-transient).
  • Falls back to exponential backoff for all other errors.
use std::time::Duration;
use tokio_prompt_orchestrator::enhanced::retry::retry_inference;
use tokio_prompt_orchestrator::{OrchestratorError, OpenAiWorker, ModelWorker};
let tokens = retry_inference(3, Duration::from_millis(200), || async {
    worker.infer("hello").await
}).await?;