Struct SmartBatcher 

pub struct SmartBatcher { /* private fields */ }

An adaptive batcher that groups PromptRequests into micro-batches.

See the module documentation for usage details.

Implementations

impl SmartBatcher

pub fn new(config: BatchConfig) -> Self

Create a new batcher with the given BatchConfig.

pub async fn submit(&self, request: PromptRequest) -> bool

Submit a single PromptRequest to the staging buffer.

Returns true if submitting this request immediately triggered a full-batch flush (i.e. poll_ready will return Some on the next call).

pub async fn poll_ready(&self) -> Option<Vec<PromptRequest>>

Poll for a ready batch.

Returns Some(batch) when either:

• Any bucket has reached BatchConfig::max_batch_size, or
• Any bucket’s age exceeds BatchConfig::max_wait_ms.

Returns None when no batch is ready yet. The caller should call this in a loop with a short sleep (e.g. 1–5 ms) between iterations.

pub async fn flush_all(&self) -> Option<Vec<PromptRequest>>

Force-flush all pending requests regardless of batch size or age.

Returns all requests across all buckets as a single flat batch, or None if nothing is pending. Useful for graceful shutdown.

pub async fn pending_count(&self) -> usize

Number of requests currently sitting in staging buckets.

pub fn stats(&self) -> BatcherStats

Snapshot of batcher statistics.

Trait Implementations

impl Clone for SmartBatcher

fn clone(&self) -> SmartBatcher

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.