TestHarness¶

Execution runner for experiments with logging and result aggregation.

Overview¶

TestHarness manages experiment runs with proper timeout handling, logging, and result aggregation. It is attached to Context during test execution.

Properties	Type	Description
timeout	float	Max time per experiment
log_dir	str	Directory for execution logs
experiments	list[Experiment]	Queue of experiments to run
results	list[ExperimentResult]	Collected results

Methods	Description
run_experiment	Run single experiment with timeout and logging
run_test	Run test batch with parallel execution
run	Run all queued experiments
export_results	Export results for analysis

Protocol Definition¶

from typing import Protocol

class TestHarness(Protocol):
    """Execution runner for experiments."""

    timeout: float
    log_dir: str
    experiments: list[Experiment]
    results: list[ExperimentResult]

    def run_experiment(self, experiment: Experiment) -> ExperimentResult:
        """Run single experiment with timeout and logging."""
        ...

    def run_test(self, test: Test) -> TestResult:
        """Run test batch with parallel execution."""
        ...

    def run(self) -> list[ExperimentResult]:
        """Run all queued experiments."""
        ...

    def export_results(self, path: str) -> None:
        """Export results for analysis."""
        ...

Methods¶

run_experiment(experiment) -> ExperimentResult¶

Runs a single experiment with timeout and error handling.

run_test(test) -> TestResult¶

Runs a test batch, potentially in parallel.

run() -> list[ExperimentResult]¶

Runs all queued experiments and returns results.

export_results(path)¶

Exports results to JSON/CSV for analysis.