Tracing
Shows what happened.
- Agent started
- Retriever ran
- Tool called
- Answer generated
- Response delivered
Everything ran as planned.
AI finished. Still failed.
Tracing shows what executed. Witdem shows whether the run achieved its product goal, what it cost, and why.
Open Source · Apache 2.0 · Self hosted
Witdem
Shows whether the goal was achieved.
Product goal not achieved.
The run did not deliver the intended result.
- Fits into your existing AI stack.
- Analyzes live traffic and historical traces.
- Support from integration through production.
Your trace is green. Your customer still didn't get the right outcome.
- LLM call
- 200 OK
- No Product Goal result
Define what success means.
You define what success means. Witdem evaluates the run against that definition.
01 Define success
Write the product goal.
Name the result you actually want in .witdem/witdem.yaml.
02 Run the system
Run the application as you do today.
Witdem does not replace the pipeline. The workflow executes as it already does.
03 Evaluate
Witdem evaluates the run.
It checks the run against the definition you wrote. Runtime success is not the goal.
The job is a non-empty answer. Artifact validity is the goal.
version: 1
service:
name: boolean-answer-agent
runtime: langgraph
telemetry:
capture_content: false
contracts:
answer:
artifact:
name: Agent answer
valid:
non_empty: $.answer
decision:
name: Answer validity
expected: true
observed: $.witdem.artifact_valid
product_goal:
name: Useful answer returned
achieved: $.witdem.artifact_validversion: 1
service:
name: classification-agent
runtime: langgraph
telemetry:
capture_content: false
contracts:
qualification:
artifact:
name: Company profile
valid:
non_empty: $.profile
decision:
name: Qualification decision
outcomes:
accepted: The company qualifies.
rejected: The company does not qualify.
escalated: Human review is required.
expected: $.expected_decision
observed: $.decision
product_goal:
name: Correct company qualification
achieved:
all:
- $.witdem.artifact_valid
- $.witdem.decision_correctversion: 1
service:
name: grounded-rag
runtime: haystack
telemetry:
capture_content: false
contracts:
grounded_answer:
artifact:
name: Grounded answer
valid:
non_empty: $.answer
decision:
name: Grounding decision
expected: true
observed: $.grounded
product_goal:
name: Successful grounded answer
achieved:
all:
- $.witdem.artifact_valid
- $.witdem.decision_correct
evidence_sufficient: $.grounded
evaluations:
- name: Groundedness
score: $.groundedness_score
target: 0.85
direction: higher_is_betterNo guessing
Mappings are declared. The SDK does not generate them, and it does not infer business success from a green span.
No silent judge
Witdem does not pick a model to score your output. If you have a quality score, you return it. The contract reads it.
No hidden business logic
The file is explicit, versionable, and owned by the application. Change the meaning of success in YAML, not in a dashboard toggle.
Now you can show both. The run, and whether it did the job.
- LLM call
- 200 OK
- Execution completed
- Product Goal achieved
Know what your AI costs. And why.
Witdem connects spend to workflow, task, run, execution path, provider, model, and product goal.
Demonstration data. A conceptual example, not a claimed customer metric.
Cost without context is just a number.
Execution overview
-
Total runs
1,842,910 ↗ 12.4% vs last 30 days -
Total cost
$1,843,210 ↗ 8.1% vs last 30 days -
Total tokens
18.4B ↗ 6.2% vs last 30 days -
Average cost per run
$1.00 ↗ 3.4% vs last 30 days -
Average latency
4.8s ↗ 1.1% vs last 30 days
Cost over time
$1,843,210 Total cost
Top workflows by cost
- Support Agent 42%
- Recipe Assistant 28%
- Invoice Parser 18%
- Other 12%
Top models by cost
Outcome summary
Assured
Achieved87.4%Needs attention
Achieved6.3%Not achieved
Goal failed6.3%
Goal achievement rate 93.7%
Top cost drivers
Where spend concentrates on the executed path.
| Driver | Runs | Cost | Share |
|---|---|---|---|
| Retry path | 38,416 | $642,180 | 35% |
| Long context | 82,200 | $479,235 | 26% |
| Tool loops | 41,800 | $239,617 | 13% |
| High-cost model route | 60,400 | $202,753 | 11% |
| Other | 120,400 | $279,425 | 15% |
Illustration
Where the spend sits
$1,121,415 on retries and long context
- Retries that still miss the goal
- Context that is longer than the job needs
- Tool loops that add cost without changing the outcome
Cost versus speed
Support Agent. Same product goal. Different cost and time.
| Model | Runs | Avg. cost | Avg. latency | Product goal |
|---|---|---|---|---|
| openai · gpt-4.1 | 84,200 | $0.81 | 8.4s | Achieved |
| anthropic · claude-sonnet | 61,100 | $0.54 | 6.2s | Achieved |
| ollama · llama3.1 | 20,400 | Not measured | 11.1s | Achieved |
Cost by business unit
See what your AI workflows actually did and what they cost.
Fits your existing AI stack.
Witdem works with the frameworks, models, and telemetry standards your AI workflows already use.
Your existing AI stack Witdem meaning
Agent frameworks
Pipelines, agents and retrieval.
Graphs, nodes, tools and streaming.
Chains, tools and retrievers.
Agents, tools and handoffs.
Agent steps, models and tools.
Models, providers & telemetry
Messages API and Claude Agent SDK.
Model routing, usage and cost.
Multi-provider model routing.
Instrument your own application steps.
Standard OTLP telemetry.
Not a new orchestrator. Not a new runtime.
Witdem adds the meaning behind your traces to the stack you already have.
The principle
Built around evidence, not assumptions.
Witdem reports what your system actually observed. It does not fill gaps with guesses.
-
01
The contract
Your application defines what success means.
Witdem evaluates the run against that definition.
-
02
What was observed
Execution paths come from the run itself.
Including branches, loops and concurrency.
-
03
Missing stays missing
Unknown cost, tokens or other measurements remain unknown.
Witdem does not silently turn missing data into zero.
You already have an agent. Add three things.
Witdem does not replace your pipeline. You install a library, wrap the function you already call, and write one file that names a good result. Then you run the application exactly as you do today.
-
01
Install the SDK
One package. Pick the extra that matches your stack.
pip install "witdem-sdk[haystack]"Haystack extra. Stable.
-
02
Wrap the entrypoint once
Existing
runandinvokecalls stay as they are.from witdem_sdk.integrations.haystack import instrument pipeline = instrument(build_pipeline()) result = pipeline.run(data) -
03
Write what success means
Create
.witdem/witdem.yaml. That is the contract. Without it, Witdem can show the trace. It cannot tell you if the product goal was met.
See Witdem in action.
From one line of setup and two lines of instrumentation to a measured run, its execution graph, business outcome, cost, and contract evidence.
You run agents at scale.
Know what they actually deliver.
- You already run AI workflows at scale. Witdem helps you see whether they actually work.
- It integrates into an existing stack, analyzes live traffic, and can backfill historical traces.
- Customer Success stays until the contract is in your repo and the team uses the dashboard.
Built for production. Designed for scale.
Near real time analysis
Analyze very large trace volumes with enterprise infrastructure.
Enterprise feature
Backfill existing traces
Bring historical telemetry into Witdem.
Enterprise feature
Integration
Integrate Witdem into your existing AI infrastructure.
Enterprise feature
Customer Success
Get support from integration through operation.
Enterprise feature