pub fn estimate_tokens(text: &str) -> u64Expand description
Estimate the number of tokens in text.
Uses the βlen / 4β heuristic: one token β 4 UTF-8 bytes in English
prose. Over-estimates by ~10β15 % on code; ~5 % on English. Never
returns 0 for non-empty input (minimum 1 token).