Introduction
EvalKit is Syntropylabs’ SDK and platform for tracing, evaluating, and testing AI agents and LLM applications.
Drop the SDK into your app, call init(), and every LLM call, tool invocation, HTTP request, and database query is captured as a trace automatically. EvalKit pairs that tracing with LLM-as-judge evaluation, dataset-driven batch testing, and synthetic-user scenario simulation — the full loop from instrumentation to quality signal.
Distributed Tracing
Every LLM call, tool use, HTTP request, and DB query, captured as a span in a full waterfall.
LLM Evaluation
Run LLM-as-judge and statistical evaluators on any trace, dataset, or simulation run.
Zero-Config Auto-Instrument
Patches OpenAI, Anthropic, HTTP clients, and DB drivers at import time — no manual wiring.
Scenario Simulation
Generate synthetic users from your agent’s own prompt and tools, then run them against your real agent code.
syntropylabs-evalkit — pip install syntropylabs-evalkit and npm install syntropylabs-evalkit. The Python import name stays evalkit.Who it’s for
- AI engineers debugging multi-step LLM pipelines in production
- Product teams who want quality signals without building an eval harness from scratch
- ML engineers running model comparisons, regression tests, or offline eval suites
- Founders and PMs who need cost and usage visibility across models, sessions, and devices
How the pieces fit together
| Layer | What it does |
|---|---|
SDK (evalkit.init()) | Runs inside your app. Auto-instruments providers/frameworks/HTTP/DB, exports spans in the background. |
| Trace ingest | Receives spans over HTTPS, keyed by a per-project subscription key (tk_live_…). |
| Evaluation | LLM-as-judge and statistical rules score traces — on demand (offline) or automatically as they arrive (online). |
| Simulation | Generates multi-turn synthetic-user scenarios and plays them against your real agent entrypoint. |
| Dashboard | Tracing waterfalls, analytics, evaluator collections, model catalog, and simulation results. |
Next steps
- Quickstart — install, initialize, and see your first trace in under 5 minutes
- Core Concepts — the vocabulary used throughout these docs
- Python SDK / TypeScript SDK — full SDK reference
- Integrations — every supported provider, framework, and datastore