FAQ
Short answers to the questions people ask most often about EvalKit.
What is EvalKit?
EvalKit is Syntropylabs’ SDK for tracing, evaluating, and testing AI agents and LLM applications. It’s available for Python and TypeScript/Node.js.
How do I install the EvalKit SDK?
For Python, run "pip install syntropylabs-evalkit" (the import name stays "evalkit"). For TypeScript/Node.js, run "npm install syntropylabs-evalkit".
What gets traced automatically?
After calling evalkit.init(), every major LLM provider (OpenAI, Anthropic, Bedrock, Google, Cohere, Mistral, Groq), HTTP client, and database driver is traced automatically — including tool calls the model makes — along with every function in your own app’s source tree.
Can I bring my own LLM API key (BYOK)?
Yes. Scenario generation and evaluation are bring-your-own-key: pass provider, model and api_key (Python) or apiKey (TypeScript) to use your own key instead of the hosted default. Any provider in the catalog works, not just OpenAI, Anthropic and Google. Runs launched from the dashboard don’t need this — they use the provider connection saved on your organization.
Where do provider API keys live in the dashboard?
On the organization, not on each model. Connect a provider once in Settings → Providers and every project in that organization can use every model from it — no re-pasting the same key per model or per project. A project can still point a model at a different connection when it needs to bill to a separate account. Keys are encrypted at rest and are never returned to the browser.
Which providers are supported, and how current is the model list?
Around 48 providers: the frontier labs (OpenAI, Anthropic, Google, xAI, Mistral, Cohere, DeepSeek, Moonshot/Kimi, Z.ai/GLM, MiniMax, Qwen, Perplexity), the clouds (AWS Bedrock, Vertex AI, Azure OpenAI, Databricks, watsonx, SageMaker), the aggregators and fast-inference hosts (OpenRouter, Together, Fireworks, Groq, Cerebras, SambaNova, DeepInfra, Nebius), self-hosted OpenAI-compatible servers (Ollama, vLLM, LM Studio, or any custom base URL), and voice/media vendors (ElevenLabs, Deepgram, AssemblyAI, Cartesia, Replicate, fal.ai). Bedrock supports all four auth styles — IAM keys, a Bedrock API key, assume-role, or a custom runtime endpoint. Model lists, context windows and prices re-sync from LiteLLM’s price sheet every 12 hours, so a new model is usually selectable the day it ships.
What’s the difference between offline and online evaluation?
Offline evaluation is triggered manually — by selecting traces in the dashboard, or by calling evalkit.evaluate() in code. Online evaluation automatically evaluates every new trace in a Trace Project at a configured polling interval (1–60 minutes).
How does scenario simulation work?
EvalKit generates realistic, multi-turn synthetic user scenarios from your agent’s system prompt and tools, plays them against your real agent turn by turn via simulate_user (Python) or simulateUser (TypeScript), and scores the results.
Where do I view traces and simulation results?
In the dashboard: Tracing shows the full waterfall view of spans for each trace, and Simulations shows scenario scores, per-turn traces, tool trajectory, and LLM-judge ratings for each run.
Do I need to self-host anything to use EvalKit?
No — the SDK defaults to the hosted trace-ingest and control-plane endpoints. If you do want to self-host, override base_url/baseUrl and api_url/apiUrl at init() time; see the Configuration reference.
What languages does the SDK support?
Python (syntropylabs-evalkit, imported as evalkit) and TypeScript/Node.js (syntropylabs-evalkit). Both cover the same concepts — tracing, evaluation, and scenario simulation.