Struct CacheLayer 

pub struct CacheLayer { /* private fields */ }

A two-backend response cache: in-memory (default) or Redis (feature caching).

Use CacheLayer::new_memory for single-process deployments or testing. Use CacheLayer::new_redis (feature caching) when multiple pipeline instances need a shared cache.

Eviction

The in-memory backend evicts the entry with the nearest expiry when the max_entries limit is reached (O(n) scan). For large caches consider a dedicated LRU crate.

Entries with ttl_secs == 0 are silently ignored by CacheLayer::set.

Thread safety

CacheLayer is Clone + Send + Sync. All clones share the same Arc<MemoryCache>.

Examples

use tokio_prompt_orchestrator::enhanced::CacheLayer;

let cache = CacheLayer::new_memory(1000);
cache.set("prompt:abc", "response text", 3600).await;

if let Some(hit) = cache.get("prompt:abc").await {
    println!("cache hit: {hit}");
}

Implementations

impl CacheLayer

pub fn new_memory(max_entries: usize) -> Self

Create an in-memory CacheLayer with a maximum entry count.

When max_entries is reached, the entry nearest to expiry is evicted before inserting the new value.

Arguments

• max_entries — Soft cap on the number of cached entries. Use 0 to disable the cap (entries accumulate until evicted by TTL checks on get).

Examples

use tokio_prompt_orchestrator::enhanced::CacheLayer;

let cache = CacheLayer::new_memory(500);

pub async fn get(&self, key: &str) -> Option<String>

Get cached value if exists and not expired

Panics

This function does not panic.

pub async fn get_checked(&self, key: &str) -> Result<Option<String>, String>

Get a cached value, distinguishing cache misses from backend errors.

Returns:

• Ok(Some(value)) — cache hit
• Ok(None) — cache miss
• Err(msg) — backend error (treat as miss but log it)

pub async fn set( &self, key: impl Into<String>, value: impl Into<String>, ttl_secs: u64, )

Set cached value with TTL in seconds

Panics

This function does not panic.

pub async fn delete(&self, key: &str)

Delete cached value

Panics

This function does not panic.

pub async fn clear(&self)

Clear all cached values

Panics

This function does not panic.

pub fn stats(&self) -> CacheStats

Get cache statistics

Panics

This function does not panic.

Trait Implementations

impl Clone for CacheLayer

fn clone(&self) -> CacheLayer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.