Skip to main content

ModelWorker

tokio_prompt_orchestrator::worker

Trait ModelWorker 

Source 
pub trait ModelWorker: Send + Sync {
    // Required method
    fn infer<'life0, 'life1, 'async_trait>(
        &'life0 self,
        prompt: &'life1 str,
    ) -> Pin<Box<dyn Future<Output = Result<Vec<String>, OrchestratorError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait;

    // Provided method
    fn infer_stream<'life0, 'life1, 'async_trait>(
        &'life0 self,
        prompt: &'life1 str,
    ) -> Pin<Box<dyn Future<Output = Result<TokenStream, OrchestratorError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait { ... }
}
Expand description

Trait for model inference workers

Implementations must be thread-safe (Send + Sync) for use across tasks. The trait is object-safe to allow dynamic dispatch via Arc<dyn ModelWorker>.

§Resilience

Worker implementations do not include retry logic. Retries are handled by the pipeline’s inference stage (see stages::inference_stage). If you call a worker directly outside of the pipeline, you are responsible for implementing appropriate retry, timeout, and backoff logic.

Required Methods§

Source

fn infer<'life0, 'life1, 'async_trait>( &'life0 self, prompt: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<Vec<String>, OrchestratorError>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform inference on the given prompt.

Returns tokens as a vector of strings. For streaming implementations, this should be the final token set.

An empty token vector is valid but callers should treat it as an empty response.

Provided Methods§

Source

fn infer_stream<'life0, 'life1, 'async_trait>( &'life0 self, prompt: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<TokenStream, OrchestratorError>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Stream inference tokens as they arrive from the provider.

The default implementation calls infer and yields each token in order, so workers that don’t override this still work with streaming consumers. Override for true SSE/chunked streaming from the provider.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§