mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
4efce809d0
* feat(proxy): add logging_endpoints package init
* feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through the success/failure callback fan-out
* feat(proxy): register callback_logs_router
* test(proxy): add logging_endpoints test package init
* test(proxy): cover /v1/callbacks/logs replay, admin guard, and partial-failure handling
* refactor(proxy): move callback-logs request/response models to litellm/types/proxy
* refactor(proxy): wrap callback-logs replay in CallbackLogsReplayer class with payload logging
* test(proxy): update callback-logs tests for class-based replayer and separated types
* fix(proxy): cover /v1/callbacks/ in backend component allowlist
The new /v1/callbacks/logs route was dropped by both component
allowlists, failing test_gateway_plus_backend_covers_full_app. It's an
admin-only spend-logging route, so it belongs on the backend (control
plane) alongside the existing /callbacks family.
* refactor(proxy): use builtin dict/list generics in callback-logs endpoint
Switch Dict/List from typing to builtin dict/list to satisfy the ruff
strict-rule budget (UP006).
* refactor(proxy): use builtin dict/list generics in callback-logs types
UP006: builtin generics over typing.Dict/List.
* chore(ui): regenerate schema.d.ts for /v1/callbacks/logs
Run npm run gen:api to add the CallbackLogRecord/CallbackLogsRequest/
CallbackLogsResponse types and the /v1/callbacks/logs path, keeping the
dashboard types in sync with the proxy OpenAPI spec.
* fix(proxy): force stream=False when replaying callback logs
A replayed StandardLoggingPayload is a terminal, fully-aggregated event —
the producer (e.g. the rust realtime gateway) already collected the whole
session before POSTing. Marking the rebuilt Logging object as streaming made
async_success_handler wait for a complete_streaming_response that never
arrives, so the spend log was never written. Realtime sessions now land in
LiteLLM_SpendLogs.
* feat(litellm-rust): CustomLogger callback layer posting to /v1/callbacks/logs
integrations/ mirrors litellm/integrations/: a sync, typed CustomLogger trait
(base contract), a typed StandardLoggingPayload, and LiteLLMPythonProxyAPILogger
— the first concrete logger, owning a bounded channel + background worker that
batches and POSTs to the Python proxy's /v1/callbacks/logs.
* feat(litellm-rust): RealTimeStreaming per-session log collector
1:1 with Python's RealTimeStreaming: observe() accumulates O(1) usage/model/id
per event (never buffers frames); log_messages() builds one StandardLoggingPayload
on session close and fans out to the CustomLogger callbacks. request_id == the
OpenAI realtime session id (sess_…), with the gateway id as fallback.
* feat(litellm-rust): wire realtime logging into the splice (lock-free observe)
The collector is owned on the splice task and observed via a synchronous &mut
callback threaded through providers::realtime::realtime() — no Arc/Mutex/atomic
on the per-frame hot path. On session close the bridge flushes one payload.
AppState carries the registered loggers; main spawns the proxy logger.
* docs(litellm-rust): ai-gateway realtime logging architecture
* docs(litellm-rust): document request-log egress to the LiteLLM control plane
Add a 'Request logging' guide to the ai-gateway README: how to point the gateway
at a LiteLLM proxy via LITELLM_PROXY_BASE_URL (+ LITELLM_MASTER_KEY for the
admin-only /v1/callbacks/logs POST), and the non-blocking / one-payload-per-session
behavior.
* feat(litellm-rust): make log-egress tunables env-overridable
Channel capacity, batch size, and flush interval now read from
LITELLM_LOG_CHANNEL_CAPACITY / LITELLM_LOG_BATCH_SIZE / LITELLM_LOG_FLUSH_INTERVAL_MS,
falling back to the DEFAULT_* consts on missing/invalid/non-positive values.
Grouped behind an EgressTunables::from_env() read once at logger construction.
* docs(litellm-rust): document log-egress tuning env vars
* docs(litellm-rust): require constants in a crate-level constants.rs
Mirror of Python's litellm/constants.py rule — magic numbers and fixed strings
go in src/constants.rs, not inline in feature modules; env-overridable tunables
keep their DEFAULT_* value there.
* refactor(litellm-rust): move ai-gateway constants into constants.rs
Per the new rule: the log-egress defaults (proxy base, ingest path, channel
capacity, batch size, flush interval) and the realtime provider default move to
crates/ai-gateway/src/constants.rs; modules import from it.
* ci: run logging_endpoints tests in the proxy-infra coverage shard
tests/test_litellm/proxy/logging_endpoints wasn't in any coverage-uploading
job, so callback_logs_endpoints.py showed only import-level coverage (~35%) on
codecov/patch despite being ~98% covered locally. Add it to proxy-infra's
test-path so the test is exercised under --cov.
* fix(litellm-rust): hash the master key before logging — never send the raw credential
Greptile/Veria P1: user_api_key_hash was the plaintext LITELLM_MASTER_KEY, which
fans out to spend logs and every callback (Langfuse/Datadog) and could be
recovered from logs. SHA-256 it (auth::hash_token, matching the proxy's
hash_token); the field is named *_hash and the proxy stores it verbatim when it
isn't sk-prefixed, so the DB value is identical with zero plaintext exposure.
* fix(litellm-rust): observe realtime logging on upstream events only
Greptile P1: observe ran on the client->upstream arm too, so an authenticated
client could send a fabricated response.done and inflate its own spend log.
session.created/response.done are server->client events; observe the upstream
arm only.
* feat(proxy): bound callback-logs batch + return per-record failures
Greptile P2: cap /v1/callbacks/logs at MAX_CALLBACK_LOG_RECORDS (default 1000,
env-overridable) so one POST can't trigger an unbounded callback/DB fan-out; and
return per-record {index, error} failures so a caller (the rust gateway) can
distinguish a transient callback error from a structurally bad payload.
* chore(ui): regenerate schema.d.ts for CallbackLogFailure / failures field
* fix(constants): make MAX_CALLBACK_LOG_RECORDS a plain constant
It doesn't need to be env-configurable (only the rust egress tunables are). As an
os.getenv var it tripped tests/documentation_tests/test_env_keys.py, which requires
every env key to be documented in the (separate-repo) config_settings.md. Plain
constant → not scanned → code-quality + documentation checks pass.
* docs(litellm-rust): trim ai-gateway ARCHITECTURE.md to one diagram + notes
* docs(litellm-rust): tighten the README request-logging section
* docs(litellm-rust): ARCHITECTURE.md is just the diagram (gateway = inference, spend = callback)
* docs(litellm-rust): drop em-dashes from the request-logging section
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
109 lines
4.6 KiB
Markdown
109 lines
4.6 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file defines the rules for Rust work in LiteLLM.
|
|
|
|
## Crates (exactly three — see AGENTS.md)
|
|
|
|
`litellm-core` describes work; `litellm-ai-gateway` executes it; `litellm-python-bridge`
|
|
exposes it to the Python SDK. A crate is a **layer**, not a route — add modules, not crates.
|
|
|
|
## Core Boundary
|
|
|
|
`litellm-core` is the pure translation layer; the `litellm-ai-gateway` host executes work.
|
|
|
|
Route-level Rust structure mirrors LiteLLM's Python responsibilities:
|
|
- `core/src/<route>/` owns the route contract, shared types, and provider
|
|
template traits. For OCR, this means `core/src/ocr`.
|
|
- `core/src/providers/<provider>/<route>/transformation.rs` owns the
|
|
provider-specific transform. For Mistral OCR, this means
|
|
`core/src/providers/mistral/ocr/transformation.rs`.
|
|
- Network execution lives in the host crate `ai-gateway` (`ai-gateway/src/io/`),
|
|
never inside `core`.
|
|
|
|
Allowed in `core`:
|
|
- Pure request transforms
|
|
- Pure response transforms
|
|
- Pure stream chunk normalization
|
|
- Shared data types and validation errors
|
|
- Deterministic token/cost helper logic
|
|
|
|
Not allowed in `core`:
|
|
- Network calls
|
|
- Environment variable or secret reads
|
|
- Filesystem access
|
|
- Database or cache access
|
|
- Provider SDK signing or auth flows
|
|
- Logging callbacks, spend writes, or custom callbacks
|
|
- Global mutable runtime state
|
|
|
|
Python owns rollout state and fallback while Rust is being introduced. Rust
|
|
paths must be off by default until parity tests prove equivalence with Python.
|
|
|
|
## Production Bar
|
|
|
|
Rust code in this workspace is held to a strict parity and robustness bar from
|
|
the first PR:
|
|
|
|
- Correctness parity is proven with tests. Do not rely on README claims or
|
|
manual inspection for a port that mirrors Python behavior.
|
|
- Every provider transform must have unit tests for supported-parameter
|
|
filtering, request body shape, response normalization, missing/null fields,
|
|
and bad-input errors.
|
|
- When Rust is exposed through Python, add Python tests that prove disabled,
|
|
enabled, and unavailable-bridge fallback behavior.
|
|
- Avoid panics on user/provider input. Return typed errors and let the host map
|
|
them to Python exceptions or HTTP responses.
|
|
- OCR handles documents that often contain personal data. Do not log document
|
|
contents, base64 payloads, provider response bodies, or secrets.
|
|
- Error messages must be useful but data-minimized. Truncate or sanitize any
|
|
upstream body before it crosses a host boundary.
|
|
- Treat empty or whitespace-only credentials, URLs, and config values as absent
|
|
at the host/config resolution layer.
|
|
- Preserve Python output shape intentionally. If a field is always serialized as
|
|
`null` for Python parity, leave a short comment explaining that parity choice.
|
|
|
|
## Host I/O Rules
|
|
|
|
These rules apply when adding future crates or modules that execute network I/O,
|
|
such as `ai-gateway`, router hosts, or standalone servers:
|
|
|
|
- Set connect and full-request timeouts. No unbounded waits.
|
|
- Reuse HTTP clients; do not construct clients per request.
|
|
- Prefer rustls TLS for portable Python wheels and Linux images unless there is
|
|
a documented reason not to.
|
|
- Add request IDs and structured tracing at the host layer, without logging OCR
|
|
document contents or secrets.
|
|
- Do not echo raw upstream response bodies to callers. Sanitize and bound them.
|
|
- Avoid `expect`/`unwrap` in server startup and request paths unless the panic is
|
|
impossible by construction and documented.
|
|
|
|
## Constants
|
|
|
|
Magic numbers and fixed strings go in a crate-level `constants.rs`, never
|
|
hardcoded inline — the Rust mirror of Python's `litellm/constants.py`.
|
|
|
|
- Each crate that needs them has `src/constants.rs` (declared `mod constants;`);
|
|
import from it (`use crate::constants::...`). Don't scatter `const` values at
|
|
the top of feature modules.
|
|
- An env-overridable tunable still lives in `constants.rs` as its `DEFAULT_*`
|
|
value; the env read (with fallback to that default) happens at the host/config
|
|
resolution layer, not in `core`/`providers`.
|
|
- Exception: a value that is purely local to one function and has no meaning
|
|
elsewhere may stay inline, but prefer `constants.rs` when in doubt.
|
|
|
|
## Checks
|
|
|
|
Run these before pushing Rust changes. The same checks run in GitHub Actions
|
|
for changes under `litellm-rust/`.
|
|
|
|
```bash
|
|
cd litellm-rust
|
|
cargo fmt --check
|
|
# the ai-gateway binary + server code is behind the `server` feature
|
|
cargo clippy -p litellm-ai-gateway --all-targets --features server -- -D warnings
|
|
cargo clippy -p litellm-core -p litellm-python-bridge --all-targets -- -D warnings
|
|
cargo test --workspace
|
|
```
|
|
|
|
When a Rust path is exposed through Python, add Python parity tests that compare
|
|
the existing Python output with the Rust-backed output.
|