Struct LlamaCppWorker 

pub struct LlamaCppWorker { /* private fields */ }

llama.cpp HTTP server worker

Connects to a llama.cpp server instance. Server URL can be set via LLAMA_CPP_URL environment variable or defaults to http://localhost:8080

Example

use tokio_prompt_orchestrator::LlamaCppWorker;
use std::sync::Arc;

let worker = Arc::new(
    LlamaCppWorker::new()
        .with_url("http://localhost:8080")
        .with_max_tokens(512)
);

Implementations

impl LlamaCppWorker

pub fn new() -> Self

Create a new LlamaCppWorker pointing at a llama.cpp HTTP server.

Reads the server URL from the LLAMA_CPP_URL environment variable. If the variable is not set the worker falls back to http://localhost:8080. Default settings: 256 max tokens, temperature 0.8, 30 s timeout.

Environment Variables

Variable	Required	Description
LLAMA_CPP_URL	No	llama.cpp server base URL (default: http://localhost:8080)

Errors

This constructor never returns an error. Network errors are deferred until ModelWorker::infer is called.

Examples

use tokio_prompt_orchestrator::LlamaCppWorker;
use std::sync::Arc;

// Optionally set LLAMA_CPP_URL=http://gpu-host:8080 in the environment.
let worker = Arc::new(
    LlamaCppWorker::new()
        .with_max_tokens(512)
        .with_temperature(0.8),
);

pub fn with_url(self, url: impl Into<String>) -> Self

Set server URL

pub fn with_max_tokens(self, max_tokens: i32) -> Self

Set maximum tokens to generate

pub fn with_temperature(self, temperature: f32) -> Self

Set temperature (0.0 – 2.0).

Values outside [0.0, 2.0] are clamped and a WARN-level log line is emitted. Most llama.cpp-compatible servers reject values outside this range.

pub fn with_timeout(self, timeout: Duration) -> Self

Set request timeout

Trait Implementations

impl Default for LlamaCppWorker

fn default() -> Self

Returns the “default value” for a type.

impl ModelWorker for LlamaCppWorker

fn infer<'life0, 'life1, 'async_trait>( &'life0 self, prompt: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<Vec<String>, OrchestratorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform inference on the given prompt.

fn infer_stream<'life0, 'life1, 'async_trait>( &'life0 self, prompt: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<TokenStream, OrchestratorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Stream inference tokens as they arrive from the provider.