Struct TokenCounter 

pub struct TokenCounter {
    pub tokenizer: BpeApproxTokenizer,
    pub model_contexts: HashMap<String, usize>,
}

High-level token counter for a specific model/family combination.

Fields

tokenizer: BpeApproxTokenizer

Underlying BPE approximation tokeniser.

model_contexts: HashMap<String, usize>

Per-model context window sizes (tokens).

Implementations

impl TokenCounter

pub fn new(family: TokenizerFamily) -> Self

Create a counter for the given tokeniser family, pre-populated with context windows for all major models.

pub fn count(&self, text: &str) -> TokenCount

Count tokens in a single text string.

pub fn count_messages(&self, messages: &[(String, String)]) -> TokenCount

Count tokens across a list of (role, content) message pairs.

Each message incurs a ~4-token overhead for the role marker and formatting tokens used by chat APIs.

pub fn count_with_system( &self, system: &str, messages: &[(String, String)], ) -> TokenCount

Count tokens for a system prompt plus a message list.

pub fn remaining_context(&self, model: &str, used: usize) -> Option<usize>

How many tokens remain in model’s context window after used tokens.

Returns None if the model is not in the built-in table.

pub fn batch_count(&self, texts: &[&str]) -> Vec<TokenCount>

Count tokens for each text in texts, returning one TokenCount per entry.

pub fn model_context_window(model: &str) -> Option<usize>

Look up the context window (in tokens) for a known model.

Returns None for unrecognised model identifiers.