Struct SemanticDeduplicator 

pub struct SemanticDeduplicator { /* private fields */ }

Near-duplicate request detector using SimHash locality-sensitive hashing.

Thread-safe and lock-free on the hot path via DashMap.

Parameters

• similarity_threshold — Maximum Hamming distance for two prompts to be considered duplicates. Typical values: 0 (exact), 3 (paraphrases), 6 (loose semantic equivalence).
• window — How long fingerprints are kept before expiry.

Implementations

impl SemanticDeduplicator

pub fn new(similarity_threshold: u32, window: Duration) -> Self

Create a new SemanticDeduplicator.

Arguments

• similarity_threshold — Hamming-distance threshold. Prompts whose fingerprints differ by ≤ this many bits are treated as duplicates. Use 0 for exact-content dedup only.
• window — TTL for cached fingerprints.

pub fn is_novel(&self, text: &str) -> bool

Returns true if text has not been seen recently (or is not a near-duplicate of a recently seen prompt).

This is a read-only check; call register to record a novel request.

pub fn register(&self, text: &str)

Record text as a recently processed request.

Call this after confirming a request is novel (via is_novel) and sending it to the LLM.

pub fn check_and_register(&self, text: &str) -> bool

Check if text is novel and, if so, register it atomically.

Returns true if the request was novel and has been registered. Returns false if it matched a cached near-duplicate.

pub fn cached_count(&self) -> usize

Number of fingerprints currently cached (including possibly expired ones not yet evicted).

pub fn evict_expired(&self)

Manually evict all expired entries. Called automatically on every register and check_and_register call.

Trait Implementations

impl Clone for SemanticDeduplicator

fn clone(&self) -> SemanticDeduplicator

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.