ZenRay

2026

Abstract

Trace retrieval and ML pipelines to see what went in, what came out, and where candidates were dropped.

View source Visit project

1   Background and motivation

When a retrieval pipeline produces a poor answer, the final prompt reveals only the documents that survived. It does not explain whether a useful candidate disappeared during filtering, reranking, or deduplication. I built ZenRay to retain that history, so debugging can follow an item through the pipeline rather than infer its fate from the final output.

2   Instrumentation model

The Python SDK represents a pipeline call as a run containing steps. Decorators create run and step contexts around existing functions, while helper calls record scores and drop reasons for individual candidates. A step can capture input and output counts, score distributions, kept and dropped samples, and attached artifacts such as prompts or responses.

Candidate identifiers connect these records across stages. This makes questions such as where a particular document disappeared answerable from the trace. Nested step contexts also preserve structure when one stage calls another, instead of flattening every operation into an undifferentiated sequence of log messages.

Tracing where candidates leave a retrieval pipeline Retrieval, reranking and filtering pass candidates left to right. Each stage reports its inputs, outputs and drop reasons down a collector into a FastAPI trace store, which a React dashboard queries. Retrieve Rerank Filter LLM answer Trace store inputs, outputs, and the drop reason recorded per stage Dashboard top-k reranked kept @trace decorators query
Figure 1.Candidates move left to right, but every stage also reports what it received, what it returned, and why it dropped the rest. That side-channel is what makes "where did the good candidates go" answerable after the run instead of during it.

3   Collection and storage

The SDK queues trace data and sends batches to a FastAPI ingestion endpoint using an API key. The server places ingestion work on a Redis queue; a worker writes structured run and step records to Postgres and larger candidate or artifact payloads to object storage. A React dashboard queries these records to display runs, individual steps, and candidate histories.

This separation gives collection and inspection different responsibilities. Application instrumentation creates the evidence, ingestion transfers it, and the dashboard presents it. Bounded queues, retries, and explicit flush behaviour are part of that path, so tracing has operational costs that need to be understood alongside the pipeline being observed.

4   What a trace can establish

A trace can show that a candidate received a low score or was removed by a rule, but it does not by itself prove whether that decision was correct. Its value is making the decision inspectable and comparable between runs. The stored detail is also limited by what was instrumented and by sampling caps, so a missing artifact should not automatically be interpreted as an event that never happened.