Module cache 

Caching Layer

Provides response caching with Redis backend (optional) or in-memory fallback.

Usage

use tokio_prompt_orchestrator::enhanced::CacheLayer;
let cache = CacheLayer::new_memory(1000); // 1000 entries

// Check cache
if let Some(result) = cache.get("prompt_hash").await {
    println!("{result}"); // use the cached result
}

// ... do inference ...
let result = "inference result".to_string();

// Store result
cache.set("prompt_hash", result, 3600).await; // TTL: 1 hour

Structs

CacheLayer

A two-backend response cache: in-memory (default) or Redis (feature caching).

CacheStats

Cache statistics

Functions

cache_key

Generate a stable cache key from a prompt string.