Module semantic_dedup 

Semantic Request Deduplication

Unlike the exact-match Deduplicator (which only catches byte-identical requests), SemanticDeduplicator uses SimHash — a locality-sensitive hashing technique — to detect near-duplicate prompts. Two prompts that are paraphrases of the same question produce SimHash fingerprints that differ in very few bits, and are coalesced to avoid redundant LLM calls.

How SimHash Works

1. The input is split into overlapping 3-gram token windows.
2. Each token is hashed with FNV-1a into a 64-bit value.
3. For each of the 64 bit positions, we accumulate +1 if the token’s hash has that bit set, −1 otherwise.
4. The SimHash fingerprint is the sign vector of the 64 accumulators.
5. Two fingerprints whose Hamming distance ≤ similarity_threshold bits are considered semantically equivalent.

A threshold of 3 bits catches common paraphrases (punctuation changes, synonym substitution). A threshold of 0 bits is exact-content matching.

Example

use tokio_prompt_orchestrator::enhanced::SemanticDeduplicator;
use std::time::Duration;

let dedup = SemanticDeduplicator::new(3, Duration::from_secs(300));

// First request is always novel.
assert!(dedup.is_novel("What is the capital of France?"));
dedup.register("What is the capital of France?");

// Near-duplicate (punctuation change) is caught.
assert!(!dedup.is_novel("What is the capital of France"));

// Distinct question remains novel.
assert!(dedup.is_novel("What is the capital of Germany?"));

Structs

SemanticDeduplicator

Near-duplicate request detector using SimHash locality-sensitive hashing.

SimHashFingerprint

A 64-bit SimHash fingerprint of a text string.