- 0
- Engine calls
- $0.0000
- Search cost
- $0.0000
- Total cost
- 0.0s
- Elapsed time
- Waiting for the question.
Final answer
Following the evidence.
Read full answer
Not graded: a live question has no answer key here.
Session pending
Checked search on Telem
Backstop asks one engine through Telem, checks the results against your agent's goal, and pays for another engine only when the first one missed.
Against Telem's default routing, on held-out test questions never used for tuning. See the results
Live race
Ask a question and watch two search setups answer it at the same time through Telem. Both sides use the same agent model, prompt and tools; only the search step differs. Every count and cost comes from the run's event log, checker included.
Required for live races on this server. Kept only in this page until you close or reload it.
Final answer
Following the evidence.
Not graded: a live question has no answer key here.
Session pending
How it works
Telem's default routing calls several engines before any results exist. Backstop asks one, checks what came back against the goal, and spends another search only on a miss.
Agent asks
search(query, goal)
The agent says what it needs, not only what to type.
Telem searches
The first engine is routed by question type.
One engine at a time, with the goal and the Telem session attached.
Checker
Rescue on a miss
At most two rescues per search, then Not found.
Answer
Telem sessions
Every run is grouped in one Telem session, so each search and rescue shows up together in the Telem console.
Tavily Extract
When a snippet is thin, Extract reads the page with the goal as its query and hands evidence chunks to the checker.
Audit trail
Every search, check and rescue is recorded as an event in the run's log, so each answer traces back to what was searched and why.
Results
None of these test questions were used to tune the checker or build the rescue table. Total cost counts search, page reads, the checker and the agent model.
Loading results
Three setups, same test questions
Accuracy bars run from 0 to 100%. The other bars are scaled to the largest value; shorter is better.
| Setup | Accuracyhigher is better | Cost per questiontotal, all models | Engine callsper question | Median timeper question |
|---|
Developers
BackstopTelem keeps Telem's response shape and adds status, verdict, why, trace, cost and a rank reason on every result. Every other AsyncTelem method passes through.
from telem import AsyncTelem
from backstop import BackstopTelem
client = AsyncTelem()
client = BackstopTelem()
resp = await client.search(
"What was Stripe's valuation in its 2023 round?",
goal="find the valuation figure",
)
print(resp.status, resp.results[0].url)
curl -s https://backstop-pied.vercel.app/v1/search \
-H "Authorization: Bearer $BACKSTOP_KEY" \
-H "Content-Type: application/json" \
-d '{"query": "What was Stripe'\''s valuation in its 2023 round?",
"goal": "find the valuation figure"}'
git clone https://github.com/oliverrowebardeen/backstop && cd backstop
pip install -e . # Python 3.11+; TELEM_API_KEY, ANTHROPIC_API_KEY and OPENAI_API_KEY in .env