Struct VllmWorker 

pub struct VllmWorker { /* private fields */ }

vLLM inference server worker

Connects to a vLLM server instance. Server URL can be set via VLLM_URL environment variable or defaults to http://localhost:8000

Example

use tokio_prompt_orchestrator::VllmWorker;
use std::sync::Arc;

let worker = Arc::new(
    VllmWorker::new()
        .with_url("http://localhost:8000")
        .with_max_tokens(1024)
);

Implementations

impl VllmWorker

pub fn new() -> Self

Create a new VllmWorker pointing at a vLLM inference server.

Reads the server URL from the VLLM_URL environment variable. If the variable is not set the worker falls back to http://localhost:8000. Default settings: 512 max tokens, temperature 0.7, top_p 0.95, 60 s timeout.

Environment Variables

Variable	Required	Description
VLLM_URL	No	vLLM server base URL (default: http://localhost:8000)

Errors

This constructor never returns an error. Network errors are deferred until ModelWorker::infer is called.

Examples

use tokio_prompt_orchestrator::VllmWorker;
use std::sync::Arc;

// Optionally set VLLM_URL=http://gpu-host:8000 in the environment.
let worker = Arc::new(
    VllmWorker::new()
        .with_max_tokens(1024)
        .with_temperature(0.5),
);

pub fn with_url(self, url: impl Into<String>) -> Self

Set server URL

pub fn with_max_tokens(self, max_tokens: u32) -> Self

Set maximum tokens to generate

pub fn with_temperature(self, temperature: f32) -> Self

Set temperature (0.0 – 2.0).

Values outside [0.0, 2.0] are clamped and a WARN-level log line is emitted. Most vLLM-compatible servers reject values outside this range.

pub fn with_top_p(self, top_p: f32) -> Self

Set top_p sampling parameter

pub fn with_timeout(self, timeout: Duration) -> Self

Set request timeout

Trait Implementations

impl Default for VllmWorker

fn default() -> Self

Returns the “default value” for a type.

impl ModelWorker for VllmWorker

fn infer<'life0, 'life1, 'async_trait>( &'life0 self, prompt: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<Vec<String>, OrchestratorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform inference on the given prompt.

fn infer_stream<'life0, 'life1, 'async_trait>( &'life0 self, prompt: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<TokenStream, OrchestratorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Stream inference tokens as they arrive from the provider.