mirror of https://github.com/BerriAI/litellm.git synced 2026-09-06 08:16:43 +00:00

feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through callbacks (#31134 )

* feat(proxy): add logging_endpoints package init

* feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through the success/failure callback fan-out

* feat(proxy): register callback_logs_router

* test(proxy): add logging_endpoints test package init

* test(proxy): cover /v1/callbacks/logs replay, admin guard, and partial-failure handling

* refactor(proxy): move callback-logs request/response models to litellm/types/proxy

* refactor(proxy): wrap callback-logs replay in CallbackLogsReplayer class with payload logging

* test(proxy): update callback-logs tests for class-based replayer and separated types

* fix(proxy): cover /v1/callbacks/ in backend component allowlist

The new /v1/callbacks/logs route was dropped by both component
allowlists, failing test_gateway_plus_backend_covers_full_app. It's an
admin-only spend-logging route, so it belongs on the backend (control
plane) alongside the existing /callbacks family.

* refactor(proxy): use builtin dict/list generics in callback-logs endpoint

Switch Dict/List from typing to builtin dict/list to satisfy the ruff
strict-rule budget (UP006).

* refactor(proxy): use builtin dict/list generics in callback-logs types

UP006: builtin generics over typing.Dict/List.

* chore(ui): regenerate schema.d.ts for /v1/callbacks/logs

Run npm run gen:api to add the CallbackLogRecord/CallbackLogsRequest/
CallbackLogsResponse types and the /v1/callbacks/logs path, keeping the
dashboard types in sync with the proxy OpenAPI spec.

* fix(proxy): force stream=False when replaying callback logs

A replayed StandardLoggingPayload is a terminal, fully-aggregated event —
the producer (e.g. the rust realtime gateway) already collected the whole
session before POSTing. Marking the rebuilt Logging object as streaming made
async_success_handler wait for a complete_streaming_response that never
arrives, so the spend log was never written. Realtime sessions now land in
LiteLLM_SpendLogs.

* feat(litellm-rust): CustomLogger callback layer posting to /v1/callbacks/logs

integrations/ mirrors litellm/integrations/: a sync, typed CustomLogger trait
(base contract), a typed StandardLoggingPayload, and LiteLLMPythonProxyAPILogger
— the first concrete logger, owning a bounded channel + background worker that
batches and POSTs to the Python proxy's /v1/callbacks/logs.

* feat(litellm-rust): RealTimeStreaming per-session log collector

1:1 with Python's RealTimeStreaming: observe() accumulates O(1) usage/model/id
per event (never buffers frames); log_messages() builds one StandardLoggingPayload
on session close and fans out to the CustomLogger callbacks. request_id == the
OpenAI realtime session id (sess_…), with the gateway id as fallback.

* feat(litellm-rust): wire realtime logging into the splice (lock-free observe)

The collector is owned on the splice task and observed via a synchronous &mut
callback threaded through providers::realtime::realtime() — no Arc/Mutex/atomic
on the per-frame hot path. On session close the bridge flushes one payload.
AppState carries the registered loggers; main spawns the proxy logger.

* docs(litellm-rust): ai-gateway realtime logging architecture

* docs(litellm-rust): document request-log egress to the LiteLLM control plane

Add a 'Request logging' guide to the ai-gateway README: how to point the gateway
at a LiteLLM proxy via LITELLM_PROXY_BASE_URL (+ LITELLM_MASTER_KEY for the
admin-only /v1/callbacks/logs POST), and the non-blocking / one-payload-per-session
behavior.

* feat(litellm-rust): make log-egress tunables env-overridable

Channel capacity, batch size, and flush interval now read from
LITELLM_LOG_CHANNEL_CAPACITY / LITELLM_LOG_BATCH_SIZE / LITELLM_LOG_FLUSH_INTERVAL_MS,
falling back to the DEFAULT_* consts on missing/invalid/non-positive values.
Grouped behind an EgressTunables::from_env() read once at logger construction.

* docs(litellm-rust): document log-egress tuning env vars

* docs(litellm-rust): require constants in a crate-level constants.rs

Mirror of Python's litellm/constants.py rule — magic numbers and fixed strings
go in src/constants.rs, not inline in feature modules; env-overridable tunables
keep their DEFAULT_* value there.

* refactor(litellm-rust): move ai-gateway constants into constants.rs

Per the new rule: the log-egress defaults (proxy base, ingest path, channel
capacity, batch size, flush interval) and the realtime provider default move to
crates/ai-gateway/src/constants.rs; modules import from it.

* ci: run logging_endpoints tests in the proxy-infra coverage shard

tests/test_litellm/proxy/logging_endpoints wasn't in any coverage-uploading
job, so callback_logs_endpoints.py showed only import-level coverage (~35%) on
codecov/patch despite being ~98% covered locally. Add it to proxy-infra's
test-path so the test is exercised under --cov.

* fix(litellm-rust): hash the master key before logging — never send the raw credential

Greptile/Veria P1: user_api_key_hash was the plaintext LITELLM_MASTER_KEY, which
fans out to spend logs and every callback (Langfuse/Datadog) and could be
recovered from logs. SHA-256 it (auth::hash_token, matching the proxy's
hash_token); the field is named *_hash and the proxy stores it verbatim when it
isn't sk-prefixed, so the DB value is identical with zero plaintext exposure.

* fix(litellm-rust): observe realtime logging on upstream events only

Greptile P1: observe ran on the client->upstream arm too, so an authenticated
client could send a fabricated response.done and inflate its own spend log.
session.created/response.done are server->client events; observe the upstream
arm only.

* feat(proxy): bound callback-logs batch + return per-record failures

Greptile P2: cap /v1/callbacks/logs at MAX_CALLBACK_LOG_RECORDS (default 1000,
env-overridable) so one POST can't trigger an unbounded callback/DB fan-out; and
return per-record {index, error} failures so a caller (the rust gateway) can
distinguish a transient callback error from a structurally bad payload.

* chore(ui): regenerate schema.d.ts for CallbackLogFailure / failures field

* fix(constants): make MAX_CALLBACK_LOG_RECORDS a plain constant

It doesn't need to be env-configurable (only the rust egress tunables are). As an
os.getenv var it tripped tests/documentation_tests/test_env_keys.py, which requires
every env key to be documented in the (separate-repo) config_settings.md. Plain
constant → not scanned → code-quality + documentation checks pass.

* docs(litellm-rust): trim ai-gateway ARCHITECTURE.md to one diagram + notes

* docs(litellm-rust): tighten the README request-logging section

* docs(litellm-rust): ARCHITECTURE.md is just the diagram (gateway = inference, spend = callback)

* docs(litellm-rust): drop em-dashes from the request-logging section

---------

Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>

2026-06-24 15:25:10 -07:00

4.6 KiB

Raw Blame History

CLAUDE.md

This file defines the rules for Rust work in LiteLLM.

Crates (exactly three — see AGENTS.md)

litellm-core describes work; litellm-ai-gateway executes it; litellm-python-bridge exposes it to the Python SDK. A crate is a layer, not a route — add modules, not crates.

Core Boundary

litellm-core is the pure translation layer; the litellm-ai-gateway host executes work.

Route-level Rust structure mirrors LiteLLM's Python responsibilities:

core/src/<route>/ owns the route contract, shared types, and provider template traits. For OCR, this means core/src/ocr.
core/src/providers/<provider>/<route>/transformation.rs owns the provider-specific transform. For Mistral OCR, this means core/src/providers/mistral/ocr/transformation.rs.
Network execution lives in the host crate ai-gateway (ai-gateway/src/io/), never inside core.

Allowed in core:

Pure request transforms
Pure response transforms
Pure stream chunk normalization
Shared data types and validation errors
Deterministic token/cost helper logic

Not allowed in core:

Network calls
Environment variable or secret reads
Filesystem access
Database or cache access
Provider SDK signing or auth flows
Logging callbacks, spend writes, or custom callbacks
Global mutable runtime state

Python owns rollout state and fallback while Rust is being introduced. Rust paths must be off by default until parity tests prove equivalence with Python.

Production Bar

Rust code in this workspace is held to a strict parity and robustness bar from the first PR:

Correctness parity is proven with tests. Do not rely on README claims or manual inspection for a port that mirrors Python behavior.
Every provider transform must have unit tests for supported-parameter filtering, request body shape, response normalization, missing/null fields, and bad-input errors.
When Rust is exposed through Python, add Python tests that prove disabled, enabled, and unavailable-bridge fallback behavior.
Avoid panics on user/provider input. Return typed errors and let the host map them to Python exceptions or HTTP responses.
OCR handles documents that often contain personal data. Do not log document contents, base64 payloads, provider response bodies, or secrets.
Error messages must be useful but data-minimized. Truncate or sanitize any upstream body before it crosses a host boundary.
Treat empty or whitespace-only credentials, URLs, and config values as absent at the host/config resolution layer.
Preserve Python output shape intentionally. If a field is always serialized as null for Python parity, leave a short comment explaining that parity choice.

Host I/O Rules

These rules apply when adding future crates or modules that execute network I/O, such as ai-gateway, router hosts, or standalone servers:

Set connect and full-request timeouts. No unbounded waits.
Reuse HTTP clients; do not construct clients per request.
Prefer rustls TLS for portable Python wheels and Linux images unless there is a documented reason not to.
Add request IDs and structured tracing at the host layer, without logging OCR document contents or secrets.
Do not echo raw upstream response bodies to callers. Sanitize and bound them.
Avoid expect/unwrap in server startup and request paths unless the panic is impossible by construction and documented.

Constants

Magic numbers and fixed strings go in a crate-level constants.rs, never hardcoded inline — the Rust mirror of Python's litellm/constants.py.

Each crate that needs them has src/constants.rs (declared mod constants;); import from it (use crate::constants::...). Don't scatter const values at the top of feature modules.
An env-overridable tunable still lives in constants.rs as its DEFAULT_* value; the env read (with fallback to that default) happens at the host/config resolution layer, not in core/providers.
Exception: a value that is purely local to one function and has no meaning elsewhere may stay inline, but prefer constants.rs when in doubt.

Checks

Run these before pushing Rust changes. The same checks run in GitHub Actions for changes under litellm-rust/.

cd litellm-rust
cargo fmt --check
# the ai-gateway binary + server code is behind the `server` feature
cargo clippy -p litellm-ai-gateway --all-targets --features server -- -D warnings
cargo clippy -p litellm-core -p litellm-python-bridge --all-targets -- -D warnings
cargo test --workspace

When a Rust path is exposed through Python, add Python parity tests that compare the existing Python output with the Rust-backed output.

4.6 KiB Raw Blame History