Struct BatchConfig 

pub struct BatchConfig {
    pub max_batch_size: usize,
    pub max_wait_ms: u64,
    pub group_by_prefix_len: usize,
}

Configuration for SmartBatcher.

Fields

max_batch_size: usize

Maximum number of requests in a single batch.

Defaults to 8. Must be ≥ 1.

max_wait_ms: u64

Maximum time (milliseconds) to wait for a batch to fill before flushing with whatever is available.

Defaults to 50 ms.

group_by_prefix_len: usize

Length (in bytes) of the prompt prefix used for similarity grouping.

When > 0, requests are bucketed by their first group_by_prefix_len bytes. This improves prefix-cache hit rate on compatible servers. Set to 0 to disable grouping (FIFO ordering).

Defaults to 0.

Trait Implementations

impl Clone for BatchConfig

fn clone(&self) -> BatchConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BatchConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for BatchConfig

fn default() -> Self

Returns the “default value” for a type.