Struct PromptCache 

pub struct PromptCache { /* private fields */ }

Thread-safe, content-addressed LRU prompt cache.

Clone-cheap: all clones share the same underlying data.

Implementations

impl PromptCache

pub fn new(config: CacheConfig) -> Self

Create a new cache with the given configuration.

pub fn get(&self, model_id: &str, prompt: &str) -> Option<Vec<String>>

Look up a cached response for (model_id, prompt).

Returns None on a cache miss or if the entry has expired. Updates the LRU order and hit counter on a hit.

Prompts longer than max_prompt_len are always treated as misses.

pub fn insert( &self, model_id: &str, prompt: &str, response: Vec<String>, ttl: Option<Duration>, )

Insert a response for (model_id, prompt).

• ttl: when None, the cache’s default_ttl is used.
• Prompts longer than max_prompt_len are silently dropped.
• If the cache is full the LRU entry is evicted first.

pub fn evict_expired(&self) -> usize

Evict all entries whose TTL has elapsed.

Returns the number of entries removed.

pub fn stats(&self) -> CacheStats

Return a snapshot of current cache statistics.

pub fn flush(&self)

Remove all entries from the cache.

Trait Implementations

impl Clone for PromptCache

fn clone(&self) -> PromptCache

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.