OpenAI · Anthropic · Gemini · Bedrock · LangChain

You know how to monitor models. LLMs need different instrumentation.

Peekr is LLM observability built for ML engineers — cost-per-query, full trace waterfalls, hallucination scores, and compliance guardrails. Two lines of Python. No proxy.

Start observing free →See full feature set

10k spans/month free · no credit card · MIT SDK license

2

Lines of code

0

Proxied requests

10k

Free spans/month

6+

LLM SDKs auto-instrumented

Why LLM monitoring is different

Traditional ML monitoring doesn't apply here.

📊

Tokens, not features

Before: Traditional ML: monitor feature drift, prediction confidence, data skew

With LLMs: LLMs: monitor token counts, prompt templates, context window usage, and cost-per-query

🔎

Trace over metric

Before: Traditional ML: aggregate accuracy metrics over a batch

With LLMs: LLMs: inspect the exact input/output of every call — you need the full trace, not averages

💰

Cost is a first-class signal

Before: Traditional ML: GPU/compute cost billed separately from model inference

With LLMs: LLMs: every API call has a per-token cost. Cost spikes are often the first sign of a prompt bug

🧪

Hallucination is not accuracy

Before: Traditional ML: compare prediction vs ground-truth label

With LLMs: LLMs: claims must be verdicted as supported / contradicted / unsupported — sentence-level evaluation

What you get

Four signals. Every LLM call.

Cost per query

Token spend per call — broken down by model, operation, and user

Alert when cost grows >20% week-over-week

Cost dashboards →

Latency trace

Waterfall of every span: LLM call, tool, retrieval, eval

Detect which step is the bottleneck — usually not the model

Trace explorer →

Hallucination score

Claim-level verdicts: supported / contradicted / unsupported

Alert when unsupported claim rate rises above threshold

Eval setup →

Compliance violations

Guardrail hits by pack (HIPAA, FDCPA, FINRA, GDPR)

Immediate block + audit log entry on every violation

Guardrails →

Setup

Two lines. Every LLM call traced.

No wrappers. No monkey-patching. No proxy. Peekr hooks into the SDK at the transport layer.

1

Install

pip install peekr

One package. No proxy agent. No architecture change.

2

Instrument

import peekr

peekr.instrument(
    exporter=peekr.HTTPExporter(
        endpoint="https://peekr.starkspherelabs.com",
        api_key="pk_live_…",
    ),
)

# Every openai.chat.completions.create(), anthropic.messages.create(),
# and google.generativeai call is now traced automatically.

Auto-instruments OpenAI, Anthropic, Gemini, Bedrock, LangChain, CrewAI, and LlamaIndex.

3

Ship — traces appear immediately

# Peekr dashboard shows per-trace:
Trace #4821   42ms   1,820 tok   $0.0018
  └─ openai.chat   38ms   1,820 tok
       prompt:    "Summarize: {{doc}}"
       tokens_in: 1,680
       tokens_out: 140
       cost:      $0.0018
       eval:      { Hallucination: 0.92 ✓ }

No sampling. Every call logged. Retention configurable.

More resources

Everything your LLM stack needs.

LLM Observability guide

What to measure and how to instrument from scratch.

AI Agent Observability

Trace multi-step agents: tool calls, sub-agents, and long-running workflows.

Team Observability Platform

Shared dashboards, per-tenant cost tracking, RBAC, and alerting.

LLM Guardrails

17 compliance packs enforced in-process on every call.

Debug AI Agents

Trace waterfalls and auto-detected performance patterns.

LLM Observability landscape

How Peekr compares to LangSmith, Helicone, Arize, and more.

Trace your first LLM call in 5 minutes.

10k spans/month free. No credit card. MIT SDK license. Works with the LLM provider you already use.

Start free →Talk to us

Auto-instruments OpenAI · Anthropic · Gemini · Bedrock · LangChain · CrewAI · LlamaIndex