Collaborate on GitHub

List your open branches: git branch -a --list 'eval/*' 'loaders/*'. Full map: .github/CODEOWNERS.

1 · Set up & smoke-test (offline, zero deps)

# from the eval/ directory
pip install -e ".[claudecode,hf]"
python tests/test_smoke.py                 # offline checks, stdlib-only
python -m memeval.cli run --benchmark longmemeval --model echo \
    --no-memory --path tests/fixtures/longmemeval.json

2 · Run the five benchmarks via the Claude Code CLI (built-in vs our plugin memory)

The primary local path — drives the Claude Code CLI on your subscription (API keys stripped, no API billing), comparing Claude Code's built-in CLAUDE.md memory against our OKF-backed plugin memory. Cross-platform (macOS · Linux · Windows · WSL).

# from the eval/ directory · subscription auth, no API key
python -m memeval.claudecode.run_bench --benchmark all --mode all \
    --model claude-haiku-4-5 --results ../results.json

Each run appends to the aggregate ledger and saves a per-benchmark, versioned file results/{vX.Y}/{bench-name}-{timestamp}.json (vX.Y = the memory-system version, starts v0.1; bump 0.1 per memory change + run). Per-developer guide: eval/memeval/claudecode/README.md.

3 · Run a dataset through the OpenCode memory harness (real code)

from memeval.agent import run_agent, AgentResult
from memeval.results import append_result
from memeval.schema import Benchmark
from cookbook_memory import CookbookMemory   # Keith's harness: implements MemoryStore

class OpenCodeAgent:                          # satisfies AgentAdapter
    name = "opencode+claude-haiku-4-5"
    price_in, price_out = 0.80, 4.0           # $/Mtok (verify)
    def solve(self, task, ctx):
        # OpenCode runs its loop; every step touches the SHARED store:
        for _ in range(MAX_STEPS):
            hits = ctx.retrieve(subquery)             # -> Keith's memory harness
            edit = ctx.generate(build_prompt(task, hits))
            ctx.remember(f"tried: {summary(edit)}")     # written to the shared store
            if tests_pass(edit): break
        return AgentResult(prediction=patch, patch=patch, success=tests_pass(patch))

store = CookbookMemory(backends=["markdown", "sqlite", "graph"])   # the real harness
rr = run_agent(Benchmark.SWE_BENCH_CL, OpenCodeAgent(), memory=True,
               store=store, path_or_id="data/swe_bench_cl.json")
append_result(rr, "results.json", run_id="brent-swecl-haiku-mem")   # -> Results page

The store= is the integration point: pass any MemoryStore (Keith's harness, or a single backend). Memory-off baseline = same code with memory=False. Coding solve-rate needs a CODE grader= (apply patch + run tests) — until then CODE accuracy is left ungraded.

4 · Log the run so it shows on the Results page

# one command: run + append to the ledger the Results page renders
python -m memeval.results run --benchmark longmemeval --model claude-haiku-4-5 \
    --memory --results ../results.json --out runs/lme_haiku_mem.jsonl
git add ../results.json && git commit -m "results: longmemeval haiku+mem" # via PR

Then it appears on the Results page after merge. Each captain logs their benchmark on their own API key (see the plan).

5 · Or launch it from GitHub Actions (no local setup)

Any collaborator with write access can run a benchmark from the browser: Actions → "Benchmark run" → Run workflow. Defaults are free + offline (EchoModel on the bundled fixture, no secrets); choose a real model + dataset (and a budget) for a paid run.

• Inputs: benchmark · model · memory on/off · dataset (blank = fixture) · dev-slice / limit · budget_usd · open-PR.
• It uploads the trajectory + results.json as artifacts and (by default) opens a PR adding the run to the Results page — review & merge to publish.
• budget_usd defaults to $200 per run; set 0 to disable the cap. echo runs are free and need no key.