Module token_counter 

Multi-Model Token Counting with BPE Approximation

Provides heuristic token counting for all major LLM families without requiring a full tokeniser dependency. Counts are BPE approximations tuned per-family and are accurate to within ~5–10 % for typical English prose and code.

Quick Start

use tokio_prompt_orchestrator::token_counter::{TokenCounter, TokenizerFamily};

let counter = TokenCounter::new(TokenizerFamily::Claude);
let tc = counter.count("Hello, world! This is a test.");
println!("tokens: {}", tc.total_tokens);

Structs

BpeApproxTokenizer

BPE-approximation tokeniser parameterised by family.

TokenCount

Result of a token-count operation.

TokenCounter

High-level token counter for a specific model/family combination.

Enums

CountMethod

How the token count was derived.

TokenizerFamily

The tokeniser family that governs counting heuristics.