Struct EvalHarness 

pub struct EvalHarness { /* private fields */ }

Evaluation harness for comparing prompt strategies.

Implementations

impl EvalHarness

pub fn new() -> Self

Create a new harness with EvalMetric::ExactMatch as the primary metric.

pub fn with_metric(metric: EvalMetric) -> Self

Create a harness with a custom primary metric.

pub fn add_case(&mut self, case: EvalCase)

Register a new evaluation case.

pub fn run_eval( &self, strategy_name: &str, responses: Vec<(String, u64, f64)>, ) -> EvalReport

Run an evaluation for strategy_name.

responses must be in the same order as cases were added. Each element is (response_text, latency_ms, cost_usd).

Panics

Does not panic — if responses is shorter than cases, remaining cases are skipped.

pub fn score(&self, response: &str, case: &EvalCase, metric: &EvalMetric) -> f64

Score a single response against a case using the supplied metric.

pub fn compare_strategies(reports: &[EvalReport]) -> Vec<(&EvalReport, f64)>

Rank strategies by composite score: pass_rate * 0.6 - avg_cost_usd_norm * 0.2 - avg_latency_norm * 0.2.

Returns references to reports in descending composite score order.

pub fn filter_by_tag(&self, tag: &str) -> Vec<&EvalCase>

Return all cases tagged with tag.

pub fn difficulty_breakdown(&self, report: &EvalReport) -> HashMap<u8, f64>

Compute per-difficulty pass rate from a report.

Keys are difficulty levels (1–5); values are pass rates (0.0 – 1.0).

Trait Implementations

impl Default for EvalHarness

fn default() -> Self

Returns the “default value” for a type.