Module multi_pipeline 

Multi-pipeline routing with prompt classification.

Manages a fleet of named pipeline instances and routes each incoming PromptRequest to the best-fit pipeline based on prompt characteristics: length, complexity signals, explicit priority hints, and learned latency.

Concept

A single LLM orchestrator often needs to handle very different workloads simultaneously:

• Fast path — short FAQ-style questions, single-turn, low latency required
• Reasoning path — multi-step problems, chain-of-thought, higher quality
• Code path — specialised coding model, longer context window
• Batch path — offline background jobs, throughput over latency

Rather than forcing all traffic through one pipeline, MultiPipelineRouter dispatches to the right pipeline and falls back gracefully when a pipeline is at capacity.

Example

use std::sync::Arc;
use tokio_prompt_orchestrator::{EchoWorker, PromptRequest, SessionId};
use tokio_prompt_orchestrator::multi_pipeline::{
    MultiPipelineRouter, PipelineDescriptor, PromptClass,
};
use std::collections::HashMap;

let router = MultiPipelineRouter::builder()
    .add_pipeline(PipelineDescriptor::new("fast", PromptClass::Faq, Arc::new(EchoWorker::new())))
    .add_pipeline(PipelineDescriptor::new("reasoning", PromptClass::Reasoning, Arc::new(EchoWorker::new())))
    .build();

let request = PromptRequest {
    session: SessionId::new("s1"),
    request_id: "r1".to_string(),
    input: "What is 2+2?".to_string(),
    meta: HashMap::new(),
    deadline: None,
};

let class = router.classify(&request);
router.route(request).await.unwrap();

Structs

HeuristicClassifier

Heuristic prompt classifier.

MultiPipelineRouter

Routes incoming prompts across multiple named pipeline instances.

MultiPipelineRouterBuilder

Builder for MultiPipelineRouter.

PipelineDescriptor

Descriptor for a single named pipeline instance.

PipelineRoutingStats

Snapshot of routing stats for one pipeline.

Enums

PromptClass

Broad classification of a prompt’s intent and resource requirements.