Struct BpeApproxTokenizer 

pub struct BpeApproxTokenizer {
    pub family: TokenizerFamily,
}

BPE-approximation tokeniser parameterised by family.

Fields

family: TokenizerFamily

The tokeniser family controlling the chars-per-token ratio.

Implementations

impl BpeApproxTokenizer

pub fn count_tokens(text: &str, family: &TokenizerFamily) -> usize

Approximate token count for text using family-specific heuristics.

The heuristic:

1. Compute a base count from chars / chars_per_token.
2. Add +1 token per punctuation run (commas, periods, etc.).
3. Adjust for runs of digits (numbers tokenise more densely).
4. Adjust for leading/trailing whitespace overhead.

pub fn estimate_output_tokens(input_tokens: usize, task_hint: &str) -> usize

Estimate output tokens given input token count and a task hint.

The ratios are empirical defaults; callers should tune for their workload.

pub fn split_into_chunks( text: &str, max_tokens: usize, overlap: usize, family: &TokenizerFamily, ) -> Vec<String>

Split text into chunks of at most max_tokens tokens, with overlap tokens of context carried forward between chunks.

Chunks are split at sentence boundaries (., !, ?) where possible to preserve semantic coherence.

pub fn fits_in_context( text: &str, model: &str, context_window: usize, family: &TokenizerFamily, ) -> bool

Return true if text fits within context_window tokens for model.

Trait Implementations

impl Clone for BpeApproxTokenizer

fn clone(&self) -> BpeApproxTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BpeApproxTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.