Evals

Two separate eval systems, each checking something different. Knowledge evals test the memory pipeline: whether remember() extracts facts correctly and query() returns them, using programmatic checks. Training evals test a fine-tuned model: whether its weights actually learned the dataset, scored by an LLM judge. One checks what Gligabao stores and retrieves; the other checks what a model has absorbed.

How an eval runs

The same five steps for both eval systems.

Diagram of an eval run in five steps: define (an eval is a set of cases), run (an executor runs each case), judge (checks or an LLM judge give pass or fail), record (runs saved in the database with counts and cost), review (admin UI and experiments)

Two eval systems

Each one checks something different.

Knowledge evals

End-to-end checks of memory: remember() then query() against the real LLM, verifying the temporal facts that were stored and the answer that came back.

  • Evals: an eval is a set of cases, each one JSON: the texts to remember (with their reference_time), a question, and what must hold. Built-in cases cover a simple fact, a job change, a 60-person document split into ~20 chunks, web search and timeseries ingestion.
  • Programmatic checks, no judge: expect_edges asserts a fact exists between two entities with the right valid_at / invalid_at and whether it was closed; expect_contains asserts the answer mentions what it should. Further checks cover chunks, web searches and stored series.
  • Contradictions tested: "Alice works at Acme", then "Alice started at Globex": the Acme fact must be closed at that date, the Globex one live, and "Where does she work now?" must answer Globex.
  • Eval mode: with serve --eval the server adds /v1/gb/eval/remember and /v1/gb/eval/query, which return an ATIF trajectory of the call: every prompt, raw model output, tool step, token count and cost.
  • Compare models and reasoning per stage: choose the model and thinking level of extract_graph, resolve_edges and others, change one stage per run, and see which facts each model got wrong and what it cost.
  • Experiments: every run is saved in a separate eval database, with its own datasets, results and trajectories, never reset, so runs can be compared later in the admin UI.

Training evals

Checks of a fine-tuned model: it is asked the questions of its own training dataset and an LLM judge scores how closely its answers agree with the dataset's.

  • One click: press Eval on a trained model in the admin UI, or call POST /v1/gb/training/models/{id}/evals. It runs in the background and can be cancelled, keeping what was done.
  • Scored 0 to 5: 4 or 5 is correct, 2 or 3 partial, 0 or 1 incorrect. An eval reports the mean score, the share of correct answers, the counts and the judge's cost.
  • One question per prompt: the model was trained on single turns, and small Gemma 4 models let questions bleed into each other, so it is asked one question at a time, as in training.
  • Judged in batches: the judge grades independent items, so ten pairs go in one call (batch_size 1 to 25, judge_model and limit are optional).
  • Live view: progress streams to the admin UI answer by answer, and every question, reference, response, score and reason is stored.
  • Dataset change detected: the dataset's SHA-256 is recorded, so the UI flags a model whose dataset changed since training.

Run them

Knowledge evals run from the command line, training evals from the admin UI or the API. Experiments can be compared in the admin UI.

terminal
# knowledge evals: run an eval, varying the model of one stage
$ ./run_evals.sh --eval chunking -v
$ ./run_evals.sh -experiment extract-models -models extract_graph=claude-sonnet-4-5

# training evals: judge a fine-tuned model on its first 50 training questions
$ curl -X POST localhost:7878/v1/gb/training/models/1/evals -d '{"limit":50,"batch_size":10}'