Commit graph

205 commits

Author SHA1 Message Date
joshua-berri
9e81aa3f40
feat(shadow-eval): skip redundant reverse evaluations on opt-in (#52192)
* feat(shadow-eval): let reverse jobs skip samples the baseline group served

A reverse shadow eval judges the router's response against baseline_model.
When the router itself sent the request to the baseline's model group, the
comparison only produces ties and still pays for a shadow and a judge call.

Reverse jobs can now opt in with skip_same_model. A sampled request whose
served group (router alias resolved, fallback hops respected) is the
baseline's own group is counted in a new same_model funnel stage instead of
being judged. It makes no provider calls, writes no attempt row, and leaves
max_turns and the eval budget untouched. When the served group is unknown
the sample is judged as before. The job detail reports
same_model_skipped_count even when nothing has been judged yet.

The flag defaults to false and is rejected on forward jobs, so existing
jobs and requests behave exactly as before. Adds two columns through an
additive, schema-only migration.

* test(shadow-eval): drive same-model skip tests off explicit completion signals

Judged cases now wait on the attempt write the injected prisma reports,
then await the pipeline task that made it. Skip cases drain only the tasks
the hook created, through a task factory scoped to the hook call, before
asserting that no provider call, attempt row, or spend was recorded. This
replaces a whole-loop task diff and the assertion on the number of tasks.

Adds a skip case with tier parameter overrides, which the policy still
treats as the same group, and a case showing a lost sampling roll counts as
not_sampled before same_model. Drops the class docstring claim that a same
group comparison only produces ties.

* style(tests): remove extra shadow evaluation test spacing

* fix(ui): send skip_same_model false from the shadow eval start form

The generated request type now requires skip_same_model, which broke the
dashboard typecheck and builds. The form sends false, the API default, so
jobs started from the UI behave exactly as before.

* fix(migrations): bound shadow evaluation schema lock waits

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 21:38:00 -08:00
joshua-berri
43d6b45cb7
fix(bedrock)!: split the OpenAI breakpoint and Bedrock cachePoint model flags (#52180)
* fix(bedrock): split the OpenAI breakpoint and Bedrock cachePoint model flags

supports_prompt_cache_breakpoint meant both "uses the OpenAI prompt_cache_breakpoint
format" and "Bedrock Converse takes cachePoint blocks", so an OpenAI-format opt-out on a
Bedrock Claude deployment silently turned off its prompt caching.

Bedrock now reads only supports_bedrock_converse_cache_point, then supports_prompt_caching
as before. The OpenAI path reads supports_openai_prompt_cache_breakpoint and keeps the old
key as a deprecated alias. The Kimi K3 catalog rows move to the new Bedrock key, and a
Bedrock deployment still on the old key logs a one-time migration warning.

Fixes #52140

* fix(bedrock): expose the split prompt cache flags in get_model_info and word the migration warning for unlisted models

* test(bedrock): mark the system prompt directly in the routed cache point regression

* fix(bedrock): warn about the old breakpoint flag even when a catalog cache point flag decides

* fix(cost-map): keep the old Kimi K3 breakpoint flag for clients that read the remote catalog

* test(bedrock): keep the old breakpoint flag on catalog cache point opt-outs for older clients

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 18:50:32 -08:00
Mateo Wang
8fadc09434
feat(focus): upgrade FOCUS export to the 1.5 Cost and Usage schema (#52105)
* feat(focus): upgrade cost and usage export to FOCUS 1.5

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(focus): identify the combined daily request SKU meter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 15:43:15 -08:00
joshua-berri
5792794691
feat(alerts): report observed Anthropic cache misses as advisories (#51973)
* feat(alerts): open a prompt cache alert when a proven Anthropic prefix stops reading and writing the cache

Behind general_settings.prompt_cache_alerts, which is off by default, a
logger watches native Anthropic chat completions that send exactly one
explicit five-minute cache breakpoint. A cache write the provider reported
for that prefix proves it. Three later requests that each report zero
cache reads and writes, all started after the proof and within its five
minutes, save one alert for the deployment and configuration through the
existing alert repository, so a dismissed or resolved alert is never
reopened.

Only the provider's raw response counts. A missing or contradictory usage
is unknown, never zero. Streaming requests, failures and unknown usage end
the streak, and so does a caching request whose prefix can't be compared.
A read or a new write resets it. The prefix is a keyed hash, and the
configuration covers the model, endpoint and a keyed digest of the API key,
so nothing from the prompt or credentials is kept.

Streaks live in the proxy's shared Redis, one Lua script per update, and
requests are ordered by their own start and end so replays and overlaps
can't add misses. Without Redis nothing is tracked. A Redis or database
error never affects the request, and evidence lost to Redis ends its
scope's streak before anything else counts.

* fix(alerts): reset the whole deployment for unassessable prompt cache evidence and quiet a saved configuration

Streamed attempts log an empty API key, so their evidence could not reach
the streak of the assessed configuration. Streams, failures, missing usage
and prefixes that can't be compared now reset every streak on their
deployment. Assessed traffic keeps its credential, model and endpoint
specific configuration.

After an alert is saved, its configuration stays quiet in Redis for a day,
so later misses don't write the same incident again. The quiet window is
only set once the save succeeds, and a failed save can trigger again.

Evidence lost to Redis now resets its deployment from a loop instead of
recursion, and a lost reset older than any proof is dropped. Asyncio
timeouts count as store failures on Python 3.10, and the module no longer
uses StrEnum. The proxy imports and registers the logger only when
prompt_cache_alerts is on, and removes it from the callback lists when
the lifespan ends.

* fix(alerts): keep the proxy lifespan a context manager and reset deployments seen while Redis is reloading

Registering prompt cache alerts had moved the asynccontextmanager decorator
from the proxy lifespan onto the new helper, which broke every proxy start.
The decorator is back on _proxy_lifespan, and a lifespan test now checks
that registration runs on entry and its cleanup on exit.

A config reload can leave the shared Redis getter empty for a moment.
Evidence seen then is remembered like evidence lost to a Redis error, so
its deployment is reset before anything else reaches Redis once it is back.

* test(alerts): let a failed completion raise its mapped error before asserting the deployment reset

* fix(alerts): describe prompt cache alerts as advisory and type their boundaries

Only callbacks that reach the logger and Redis are observed, so delayed, lost or unlogged requests can fall between the misses. The docstring now says an alert does not guarantee consecutive misses or an outage, and that a hit delivered after the stored proof expired leaves the prefix unproven until a new write. Tests pin both behaviors. The logger no longer calls the partially typed CustomLogger constructor, whose two attributes every reader already defaults, and registration reads the typed general settings view.

* fix(alerts): keep retained prompt cache misses alive as long as a refreshed proof

A proof or hit that returned before already stored misses but arrived after them extended the proof key while the misses kept their older expiry, so they could vanish early and an observed streak would be lost. Both refresh branches now give the retained misses the proof's expiry.

* fix(alerts): serialize each message's fields once in the prompt cache prefix

Every block's identity carried its message's other fields, so a large field on a message with many blocks made the canonical prefix grow with fields times blocks. The message's fields are now one segment ahead of its blocks, keeping role, metadata, block order and fail-closed redaction in the prefix while its size stays linear in the request.

* test(proxy): keep a leaked worker config out of the prompt cache lifespan test

An earlier test leaves WORKER_CONFIG in the environment, and the lifespan then loads its config path. The registration test now clears it with the database URLs.

* test(alerts): clean each prompt cache deployment tag in its own comprehension

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 15:36:07 -08:00
Agung Besti
c482013a8a
fix(guardrails): log requests that a guardrail blocks on /apply_guardrail (#51014)
* fix(guardrails): record the guardrail verdict on a request dict that starts empty

The Bedrock guardrail and the shared `log_guardrail_information` wrapper both picked the request dict with `or {}`. An empty dict is falsy, so a caller that passed one had the verdict written to a throwaway dict. /apply_guardrail passes an empty dict for a request that carries no messages and no metadata, so nothing downstream could see what the guardrail decided. The verdict now goes to the dict the caller passed, and only a missing dict falls back to a new one.

* fix(guardrails): log requests that a guardrail blocks on /apply_guardrail

A request a guardrail blocked on /apply_guardrail was missing from the logs. The endpoint skipped the failure handlers for an HTTPException, the spend log failure hook wrote no row because the route is not an LLM API route, and the guardrail's verdict stayed on a request dict no logger reads.

The endpoint now runs the failure handlers for every exception and copies the verdict onto the logged request, so allowed and blocked requests both carry it along with the guardrail's cost. It marks the failure of a request it had set up with a sentinel, and the spend log failure hook writes its row for that mark. A request rejected before setup, such as one with a bad key or an unknown guardrail, is still not logged. Guardrail records a caller sends in metadata are dropped for both response formats so a verdict cannot be forged in the logs.

* fix(guardrails): keep Presidio and Gray Swan verdicts on empty request data

Presidio's text-only path and Gray Swan's API-failure path replaced an empty request dict with a throwaway one, so the verdict never reached the logged request. The apply_guardrail logging tests now run the real request setup and read what a logging callback receives.

* test(guardrails): cover the endpoint failure row, the sync decorator, and an unreachable Gray Swan without real sockets

* fix(logging): redact prompt text in guardrail records when message logging is off

* fix(logging): redact the failure payload when message logging is off

* fix(logging): keep guardrail details for other callbacks when one opts out of message logging

* fix(logging): redact failure events per callback and keep guardrail compression stats

A callback that turns off message logging now gets the same redacted copy on failure events as on success events, so a blocked /apply_guardrail call no longer hands it the raw prompt and verdict

Guardrail redaction keeps the numeric compression stats (tokens saved, compression ratio) that spend tracking reads, through one helper shared with the spend-log sanitizer

* fix(policy): test each batch input on its own copy of the request data

The guardrail logging decorator now records a verdict on an empty request dict instead of dropping it. The policy test endpoint shared one request dict across every input and merged it into the agent call, so each later input and agent request carried the earlier inputs' verdicts. Each input now gets its own copy, which keeps the caller's request data unchanged

* test(policy): look up the counting guardrail through the real registry

The batch isolation test now registers its guardrail as a litellm callback, so the endpoint finds it through the real guardrail registry instead of a mocked one, and its apply_guardrail override carries the same concrete types as the base class

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-16 15:18:08 -08:00
Shivam Rawat
1b62821ba9
fix(otel v2): stop legacy langfuse loggers sending to refused key/team hosts (#51344)
* fix(otel v2): stop the legacy langfuse_otel logger exporting to refused key/team hosts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(otel v2): assert the refused-host regression through public config and header outputs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel v2): refuse key/team langfuse hosts in the langfuse callback and prompt fetch, and fail closed on unparseable hosts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel v2): check the langfuse SDK clients against the http URL they actually call

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel v2): skip allowlist entries that won't parse and read the V2 flag only for unlisted hosts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel v2): a malformed key/team langfuse host drops only its own destination

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(otel v2): read the V2 flag before the allowlist and cache the allowlist and flag checks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 11:34:20 -08:00
Nate Armstrong
c60a320b05
perf(token_counter): memoize token counts of long texts per tokenizer (#51904)
* perf(token_counter): memoize token counts of long texts per tokenizer

* test(prompt_caching): count a prompt no other test has counted in the loop lag checks

* fix(token_counter): key the count memo on the string hash instead of blake2b

* test(router): embed distinct texts so the batches are counted in full

* test: time token counts over text no earlier count has memoized
2026-11-16 19:21:55 +00:00
Yassin Kortam
90886f4f5e
feat(control_plane): ocsf audit events, sink protocol, queued delivery and hash chain (#49660)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 17:36:24 +00:00
Yassin Kortam
5b722c7c6f
feat(control_plane): otel server and dataplane client spans with w3c propagation (#49659)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 17:36:24 +00:00
Nate Armstrong
8d200a4f98
feat(messages): serve bedrock invoke messages on rust only (#51330) 2026-11-15 16:58:31 -08:00
Nate Armstrong
09ce4f50ed
perf(auth): read token model fields from the class instead of deprecated __fields__ (#51901)
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
unit / root (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / enterprise-managed-files (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / rust-bridge (push) Waiting to run
unit / assert-shard-coverage (push) Waiting to run
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* perf(auth): read token model fields from the class instead of deprecated __fields__

* test(s3): assert 403 retry backoff on the logged wait instead of wall-clock gaps
2026-11-15 07:07:49 +00:00
Nate Armstrong
3bcd77954e
refactor(s3-log): move the rust-backed s3 logger from s3_v3 to s3_v2 (#51893) 2026-11-14 22:24:23 -08:00
Shivam Rawat
a1ea22e996
feat(otel v2): proxy-level default langfuse_span_scope for key and team destinations (#51726)
* feat(otel v2): proxy-level default langfuse_span_scope for key and team destinations

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel): regenerate dashboard API documentation types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel): preserve tenant destinations with malformed otel settings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(otel): cover explicit narrow scope and non-Langfuse defaults

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Read only the prefixed Langfuse destination scope environment variable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 15:45:11 -08:00
yuneng-jiang
da55893656
ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864)
* test(unit): reset module state only when a test changed it

The proxy conftest deep-copied every snapshotted litellm attribute before each
test, and the copy of litellm.model_cost alone was a median 0.12 s for each of
56,031 proxy tests on the runner. The reset now copies an attribute back only
when it no longer equals the snapshot.

The enterprise and MCP server conftests reloaded litellm before every test,
3.27 s and 0.21 s per test. They now use the same reset.

* test(unit): remove six flaky tests

They failed on main without a code change behind them between 2026-11-05 and
2026-11-14: the vector store hook SDK fallback, the concurrent user config TPM
count, three Prometheus metrics server port tests and the OpenCode list cost
billing test.

* ci: run four heavy unit shards on 16-core runners and keep two smoke jobs off pull requests

proxy-feature-endpoints, proxy-endpoints, root and llms-providers ran 1.9 to 2.2
times faster with 8 workers on a 16-core runner in a benchmark run.

windows-sdk and proxy-build-from-pip still run on pushes to main and manual
runs. proxy-build-from-pip installs a published release and never runs the
pull request's code.

* test: move 109 tests that start a process or open a socket into tests/integration

tests/unit is in-process by its own rules, and these tests spawn a Python
interpreter, a shell or a local TCP or HTTP server. They took 921 s of unit
shard time per run, 396 s of it in the MCP client transport tests. Each one now
lives in a *_real_io.py file under the integration directory for its domain and
runs with that directory's group.

The tests themselves are unchanged apart from one repository-root path in the
CLI test.

* test(unit): copy back only the dict entries a test changed

A test module that registers models at import leaves litellm.model_cost four
entries larger than the snapshot before every test in its directory, so the
reset still deep-copied the whole price map for each of them. It now rebuilds a
changed dict from the live entries that still match and copies only the rest.

* ci(smoke): keep the two jobs that skip on pull requests out of the smoke gate

A skipped job passing the gate is what test_unit_passed_gate guards against,
so the gate goes back to requiring success from every job it needs and
windows-sdk and proxy-build-from-pip stop being part of it. They still run
and report on pushes to main and manual runs.
2026-11-14 15:41:30 -08:00
Nate Armstrong
8e92894ff2
feat(s3-log): replace the python s3_v2 logger with a rust callback (#51539)
The s3_v2 S3Logger is ported to a Rust callback, callbacks-s3-log, behind
the CustomLoggerV2 contract, with the bridge constructor NativeS3Log and the
Python glue in litellm/integrations/s3_v3.py. It is now the only
implementation: the Python logger is deleted, s3_v2.py re-exports the
Rust-backed S3Logger, and LITELLM_NATIVE_S3_V2 is removed. The callback keeps
its name, list position and settings, so no config changes.

The original s3_v2 tests are replicated against the Rust callback as
<original>_rust.py, and Python's outputs for the differential suite are
recorded in tests/unit/integrations/s3_v2_golden/python_s3_v2.json.
2026-11-14 23:03:58 +00:00
yucheng-berri
580203cce6
fix(otel): send the Langfuse environment, release and version on OTel v2 (#51385)
* fix(otel): send the Langfuse environment, release and version on OTel v2

* fix(otel): keep the Langfuse environment on the Langfuse exporter only

* fix(otel): annotate the exporter resource attributes as a read-only mapping

* fix(otel): shed the Langfuse release and version first at the attribute limit

* fix(otel): keep base attributes when the Langfuse release and version do not fit

- the model span sheds release and version before the identity fields
- the request root takes release and version only when they fit under its attribute limit
- an exporter routed to a tenant's credentials drops the proxy's Langfuse environment
- a langfuse.environment the proxy's resource already names wins over LANGFUSE_TRACING_ENVIRONMENT

* test(otel): annotate the new attribute-limit and routing tests
2026-11-14 11:40:35 -08:00
paveln83
9fba44d300
fix(openai): cap and bill web search rounds on Responses-pinned chat completions (#51802)
* fix(openai): cap web search agentic loops with max_agentic_loops on OpenAI chat completions

The OpenAI chat dispatcher ran every custom logger's chat agentic loop
without a depth check, and the web search follow-up never carried the
loop depth, so max_agentic_loops was ignored on openai/ deployments.
The dispatcher now skips loggers the shared chat completion loop in
litellm.acompletion already runs, so those go through its depth, cap,
and repeated-call checks once. Loggers that only implement the
chat-specific hooks still run in the OpenAI dispatcher

* fix(openai): honor max_agentic_loops when chat completions bridge to Responses

A gpt-5.4+ chat completion with function tools on api.openai.com is sent to
/v1/responses by the Responses bridge. The Responses handler then ran its own
agentic loop on that call, which never saw the request's max_agentic_loops or
depth, and the shared chat completion loop in litellm.acompletion ran again on
what it returned, so every chat follow-up restarted the Responses loop at depth
0. With max_agentic_loops=1, web search made 8 Responses calls and 7 searches

The async bridge now marks its Responses call as part of a chat completion
turn, and the Responses handler skips its own loop for such calls. The shared
chat loop owns the whole turn, so its depth, cap, and repeated-call checks run
once

* fix(responses): keep search follow-ups on the Responses API for responses/ pinned models

When an async chat completion crosses the Responses bridge, the shared chat
loop now owns the turn and builds each web search follow-up from the model
recorded on the logging object. responses() overwrote that record with the
stripped model, so a model pinned with the responses/ prefix (for example
openai/responses/<model> on a Responses-only gateway) sent its follow-up to
chat completions, and the turn ended on the unexecuted tool call

responses() now leaves the record alone when the chat loop owns the turn, the
same way completion() skips its own record when the reverse bridge owns it

* fix(agentic-loop): bill every round of a non-streamed Responses-bridged chat loop once

A bridged chat request logs its first round before the chat loop runs, so a
follow-up run as an internal sub-call was never billed. Each follow-up of a
non-streamed bridged parent now writes its own spend row, and a failed
follow-up no longer re-bills rounds that already wrote one

* fix(agentic-loop): keep streamed Responses-bridged web search from billing its final round twice

* fix(openai): bill every bridged search round in one row after guardrails

On the proxy a non-streamed Responses-pinned chat request now parks one
spend row that sums every round the loop served, so the row and the
x-litellm-response-cost header show the whole request and nothing is
written before post-call guardrails pass. A streamed request keeps round
one's row and folds every later round into the replayed final row.

The bridge now skips only Responses hooks that run their own chat loop,
so hooks that only handle the Responses API still run.

* fix(openai): price each bridged search round at its own size

Each round of a bridged chat request is now priced on its own usage and the
combined proxy row adds those prices, so summed usage never crosses a
long-context price tier. A streamed replay keeps that total and its cost
breakdown

* fix(agentic-loop): run the chat hook of bridged callbacks that define both agentic hooks

The bridge skipped the Responses hook of a callback that runs its own chat
loop, and the shared loop only asked the general gate, so a custom callback
with both hooks stopped intercepting bridged chat. Ask its chat gate after
the shared loop. Price tier tests now use test-owned gpt-5.6 prices.

* fix(agentic-loop): route bridged chat to the chat hook only for callbacks that opt in

A custom callback that defined both agentic hooks lost its Responses hook on
chat completions bridged to the Responses API, so a chat hook that declined
left the request unhandled, and one that claimed it changed which hook
answered compared with main.

Callbacks now opt in with wants_chat_completion_hooks_on_bridged_chat. Web
search interception sets it, so its loop still runs once on the chat side.
Every other callback keeps main's routing through its Responses hook.

* fix(websearch_interception): keep main's Responses loop for a bridged web_search_options call

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 14:36:10 -05:00
giladk1221
9d827093c5
feat(github_copilot): support GitHub App authentication with PEM upload (#51427)
* feat(github_copilot): support GitHub App installation authentication

* Add GitHub App authentication to the Copilot credential form

* Explain secure private-key setup beside Copilot App fields

* Store uploaded Copilot App keys through encrypted named credentials

* fix(github_copilot): handle App validation errors and credential replacement

* fix(ui): make GitHub App field help accessible

* fix(github_copilot): preserve saved App key references on edit

* fix(github_copilot): trust litellm's CA settings when minting App tokens and hide App keys from /health

* fix(github_copilot): keep device-login headers, mint App tokens off the shared thread pool, cache App responses

Device-login requests keep the merge base header behavior, so caller headers still replace the Copilot defaults. Only App requests merge and pin their identity headers.

App token refreshes are single-flight per cache key with a shared future. Only the refreshing caller uses a thread, so waiting callers no longer fill the default executor that every provider call shares.

A cached App token is served before the key secret is read. Uploaded keys still rotate the token when their contents change.

App deployments are deployment-owned, so they share the response cache like shared device-login deployments.

App credentials are validated even when credential_info names no provider.

* fix(github_copilot): finish an App token refresh even when the caller that started it is cancelled

The caller that owns a refresh used to await asyncio.to_thread, so cancelling it while the job was still queued cancelled the job, left the in-flight future unfinished, and every later request for that App waited on it. The refresh now runs in the executor without being tied to the caller, and every caller, the owner included, awaits the shared future

* fix(github_copilot): refuse deployments naming an unloaded credential, keep App key references, accept numeric App IDs

A Copilot deployment whose litellm_credential_name is not loaded now answers 400 instead of falling back to the shared device login. The router registry keeps github_copilot_app_private_key_ref values as references, a YAML App ID can be an integer, the server-owned field error names Copilot auth for Copilot fields, and the credential edit modal shows the stored provider again

* fix(github_copilot): report a deployment naming an unloaded credential as unhealthy

The /health check now answers unhealthy with the same 400 the request path returns for a Copilot deployment whose litellm_credential_name is not loaded, instead of running it through the shared device login

* fix(github_copilot): keep the base fallback for an unloaded credential and report the App host as api_base

A Copilot deployment that names an unloaded credential uses the shared device login again, as on base.
A deployment that sets github_copilot_auth_type: github_app still refuses before any GitHub call.
get_api_base now returns the Copilot host an App or per-user session calls instead of an admin api_base it ignores.

* fix(github_copilot): count a GitHub rejection of the App installation token as a Prometheus deployment failure

* fix(github_copilot): keep per-user session api_base reporting as on base and name Copilot in the non-admin 403

Only an App deployment reports its App host as api_base before a configured api_base.
Per-user OAuth sessions report the same api_base they did on base.
The 403 for a non-admin attaching App credentials now names GitHub Copilot authentication.
The token-dir fixture in the App api_base test is applied with usefixtures.

* fix(github_copilot): report the configured api_base instead of raising when GitHub App settings are incomplete

* test(github_copilot): type the incomplete App settings test parameter

---------

Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 10:33:20 -05:00
paveln83
3ad164382b
fix(responses): send cache_control as a prompt_cache_breakpoint on the chat-to-Responses bridge (#51783)
* fix(responses bridge): carry cache_control as a prompt_cache_breakpoint to deployments that read them

A chat request with Anthropic-style cache_control on a system or user part got its mark renamed to
prompt_cache_breakpoint on the plain chat path, but the chat-to-Responses bridge (taken once a
function tool and reasoning are on, and always for bedrock_mantle GPT) dropped it, so explicit-mode
caching never wrote. The bridge now renames the marks before converting the messages when the
served deployment reads breakpoints, using the same gate the Responses config uses for native
/v1/responses input. Deployments that do not read breakpoints keep today's behavior.

The rename also leaves unmarked content parts untouched instead of validating them, so a part with
non-string keys no longer raises on either path.

* fix(responses): keep leading system messages in client order once one goes to input

A marked first system string became a list and moved to input while the
next unmarked string stayed in instructions, so the model read them in
reverse. The leading run that joins instructions now stops at the first
system message whose content is not a string

* fix(responses): leave Bedrock Mantle and unmarked system order as they were on the bridge

Bedrock Mantle already caches the growing prefix on its own, so turning
cache_control into breakpoints there added write charges. The client-order
rule for leading system messages now applies only when a mark turns a system
string into a list, so unmarked chats send the same request as before.

* test(responses): cover a marked user turn in the unmarked system split test

A cache_control mark on the user message must not switch unmarked leading
system messages to the client-order rule. The split test now runs with the
user turn plain and marked, so dropping the role check from the system
string test fails it.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 05:00:49 -05:00
quietjules
9ba216d521
fix(vertex_ai): reuse one Google token across RAG Engine searches and cold bursts (#51239)
* fix(vertex_ai): reuse one Google token across vector store searches

* fix(vertex_ai): load Vertex credentials once when a cold worker gets a burst

* test(vertex_ai): count Google token fetches for RAG Engine searches through the proxy

* test(vertex_ai): type the fixtures and token callback in the concurrent search test

* fix(vertex_ai): lock Google token loads per credential and reload a key Google rejects

A shared RAG Engine config made one store's slow token fetch hold up every
other store, and kept a rotated key file's deleted key until a restart.
First loads and expired refreshes now take a lock per credentials and project,
refreshes take a lock per credentials object, and any RefreshError reloads the
credentials once.

* fix(vertex_ai): share one Google token load or failure per credential and reload rejected keys on async calls

* test(vertex_ai): check async callers sharing an expired credential refresh it once

* fix(vertex_ai): reload Google credentials only when a refresh is rejected for good

A retryable refresh error such as a 503 now raises without reloading the key, so an outage
does not double google-auth's retries. Credentials that cannot be weak-referenced share the
config's refresh lock again, as on main. The burst test for a rejected key moves to VertexBase
and holds the token response until the other callers have started

* fix(vertex_ai): reload Google credentials only when Google answers invalid_grant

google-auth leaves its retryable flag False on metadata server, impersonation, and workload identity outages, and on a 502, so the previous check reloaded the key file on those outages and doubled the token tries. Now only an invalid_grant answer or "Reauthentication is needed" reloads

The sync reload also refills the cache under the key the caller asked with, so a call without vertex_project keeps one cached copy, as the async reload already did

* fix(vertex_ai): pick up a swapped Google key file on the next call

The credential cache key now carries the key file's inode, mtime and size, for a
vertex_credentials path and for GOOGLE_APPLICATION_CREDENTIALS. A key file that is replaced
or rewritten, such as a rotated Kubernetes secret, is loaded on the next call instead of
after Google rejects the old key or the proxy restarts

* fix(vertex_ai): keep the earlier key while a replaced key file does not load

A key file caught halfway through a non-atomic write, or replaced by one that does not load,
now keeps the credentials loaded from its earlier version in use and logs a warning, as the
cache did before key files were re-read. The next complete file is loaded on the next call.
The pub/sub test no longer writes a key file it only reaches through the seeded cache

* fix(vertex_ai): try a replaced key file again when its first token fetch fails

The earlier key now stands in only when the replaced file does not load. A token fetch that fails on the new key raises like any first load, so the next call tries the new key again instead of keeping the old one

* fix(vertex): raise when an async reload keeps an earlier key Google rejected

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 03:12:09 -05:00
Nate Armstrong
cd2e4b6b60
feat(custom-logger-v2): add the contract for rust callbacks and bind the proxy request scope (#51526)
* feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0)

* feat(custom-logger-v2): add the rust contract for native callbacks and bind the proxy request scope

* fix(custom-logger-v2): add field projections and keep the request scope across copies

* fix(custom-logger-v2): pass the request scope to the async failure attempt_ended line

* fix(custom-logger-v2): add v2 callbacks from litellm.callbacks to the success and failure lists

_safe_add_callback_to_list dropped CustomLoggerV2 instances, so a V2 callback registered through litellm.callbacks got provider events but never log_prepare or attempt_ended. The manager now adds each V2 instance once by identity, and proxy startup fans V2 callbacks out like CustomLogger ones.

* fix(custom-logger-v2): contain a failing v2 callback on logging hooks

A V2 log_prepare writer that raised, or returned Abort or Respond, ended the line and escaped async_success_handler, so attempt_ended delivery, sync-for-async scheduling and the Redis flush were skipped. On hooks off the request path a V2 entry's exception or outcome is now counted and logged like a legacy logger's exception, and the line goes on. ContractViolation and BaseException still propagate, and legacy entries are unchanged.

* fix(custom-logger-v2): run blocking and awaited v2 callbacks on synchronous logging lines

run_plan_sync raised ContractViolation for a V2 log_prepare writer or any awaited V2 subscriber, the logging plane swallowed it, and every legacy logger at that hook was skipped. Legacy entries now always run in order as before. A V2 writer or awaited subscriber runs inline in list order through asyncio.run when the thread has no running event loop, so sync SDK calls apply its patch and finish it before the handler returns. On a thread with a running loop it cannot block, so it is delivered off-path and its patch is not applied. Lines without such entries take the unchanged legacy path.

* fix(custom-logger-v2): map a v2 abort to its http error at every proxy hook point

Only pre_call turned a V2 Abort into an HTTP error. during_call, post_call_success, post_mcp_call, list_models and the deferred stream hook raised V2Outcome, which reached the client as a 500, and post_call_failure, which never raised before, raised it too. Abort now raises the exception it carries or an HTTPException with its status, message and headers at those points, and at post_call_failure it becomes the transformed error through keep_first. Respond is still not wired and keeps raising V2Outcome. The pre_call routing test is updated to the HTTPException raise_v2_abort already makes.

* fix(custom-logger-v2): apply no-log to v2 logging subscribers without a request scope

The proxy request scope already carries no-log and caller opt-outs, so a V2 callback that honors no-log is skipped on proxy requests. SDK calls have no scope, and the logging plane ran V2 subscribers with an empty selection, so a no-log call was still delivered to them. Without a bound scope the logging lines now open one with no_log set when the call's litellm_params carry no-log, unless global_disable_no_log_param is on. Callbacks that do not honor no-log, and so every writer or abortable, are unaffected.

* fix(custom-logger-v2): deliver a failed attempt to v2 subscribers once per logging object

Callers such as MCP tool errors and Responses streaming run both failure handlers without for_async_request, so a V2 attempt_ended subscriber, now present in both failure lists, got the failure twice. The first failure handler to include V2 subscribers claims the delivery on the Logging object and the other skips them. Legacy callbacks are unchanged.

* refactor(custom-logger-v2): settle a synchronous v2 run through the shared gather and use aware times in the new tests

Drops the blind except the sync line used to capture a V2 callback's exception, reusing _gather_started, and gives the new logging tests timezone-aware times, so the strict ruff counts do not grow.

* fix(custom_logger_v2): await a single-entry v2 stage in the caller's context

A native handler in its own stage ran inside an asyncio.gather child task, so a ContextVar it set was lost when the stage ended, while a legacy callback's stays visible to later callbacks and the rest of the request. A stage with one v2 entry is now awaited directly, with gather's exception handling kept. Multi-entry parallel stages still gather.

* fix(custom-logger-v2): raise a native callback panic as a runtime error

* fix(custom-logger-v2): run a synchronous v2 line on a private event loop

* fix(custom-logger-v2): deliver an mcp tool failure to v2 subscribers from the async failure handler

* fix(custom-logger-v2): raise an abort's carried exception at the deployment hooks and keep the original failure at post_deployment_failure

* fix(custom-logger-v2): apply caller opt-outs to v2 logging lines without a bound request scope

* fix(custom-logger-v2): dedupe v2 callbacks in the event lists by logger key so a native built twice is delivered once

* fix(custom-logger-v2): log an unregistered abort error kind and keep the carried traceback at post_call_failure

* fix(callbacks): reject a selection name or alias that is not lowercase

* fix(custom-logger-v2): run v2 deployment callbacks on rust bridge deployment hooks

* fix(router): type the fallback event kwargs handed to the v2 dispatch

* feat(custom_logger_v2): type the contract fields as records and mark derived-only fields

Fields that the host derives from a hook's arguments now carry pydantic records on the Python side and matching serde records in litellm-callbacks, with records.json as the shared sample both sides check. Twelve new fields take bits 30 to 41, a write to a derived-only field is a manifest error, and pre_call can write budget_reservation and rate_limit_statuses in place. The payload reference field and the PayloadHost trait give Rust a handle to the logging payload instead of a copy

* feat(custom_logger_v2): derive typed fields from hook arguments and always build call facts

A slot table maps each typed field to a function over the hook's arguments. A derived slot runs only when a subscriber reads it, once per hook invocation, and off-path readers settle their reads before the hook returns. Auth fields come from the auth object only, budget_plan is claimed once, and budget_reservation and rate_limit_statuses write through to their carriers so the next pre_call subscriber sees them. Call facts are now built on unscoped logging lines and at post_call_success, post_call_failure and response_headers

* feat(callbacks-python): hand the logging payload to native callbacks by reference

NativeCallback stashes the shallow payload copy it is given and passes Rust a PayloadRef. The Python host renders a handle with a safe_dumps compatible encoder, on the caller's event loop while the hook runs and on the blocking pool after it, and dropping the last reference releases the stash entry. The encoder and its JSON helpers move here from the S3 logging bridge so every port shares one copy

* feat(custom_logger_v2): accept every value python accepts for typed members and serve logged key identity

Typed members keep the bool, int, float or str python holds through a Scalar
type on both sides, so ids, limits and priorities no longer lose values that
lax pydantic refused. A member the contract cannot carry is named in rejected
so a port can reproduce what python does with it.

Logging hooks get logged_key_identity, built from the payload metadata the
python loggers read on every route, while key_identity stays on request hooks
and keeps reading the auth object.

* fix(custom_logger_v2): read the live budget reservation dict and carry non-finite floats both ways

budget_reservation read the dict budget_reservation.py builds with a KeyError
on callback_bound. Undeclared keys are now ignored, callback_bound is a member,
entries keep a non-mapping item as null so capacity's model access group check
stays exact, and costs read as the float budgets computes with float(x or 0.0).

Float members cross as the raw json marker when they are not finite, and a
native verdict returns through json_to_py, so an infinite or NaN cost survives
a read and a write instead of turning into null.

* feat(callbacks): carry a python value as its str() text and truthiness in scalar members

Scalar gains a Stringified variant with the str() text and bool() of a value
that is not a bool, int within i128, float or str, so a port can reproduce
the label, path or log text python formats and the falsiness of an empty
container without confusing it with a real string. Budget reservations say
whether python held the entries as a list.

* feat(custom_logger_v2): carry non-scalar values and any iterable tags or entries across the boundary

A Scalar member now carries a list, dict, set, bytes, object or an int past
i128 as its str() text and truthiness instead of naming it in rejected, which
stays for values whose str() or bool() raises. request_tags carries any
iterable as the items prometheus iterates, and budget reservation entries
carry what budgets iterates with entries_is_list for capacity's list check,
never written back to the shared dict.

* feat(custom_logger_v2): say whether a stringified value hashes and whether it is an int

A Stringified scalar now carries hashable, from trying hash() at the
boundary, so a port can raise the TypeError capacity's priority lookup and
the limiter's descriptor identities raise for a list, dict or set and not
for a tuple or object. It also carries integer, so an int past i128 is told
apart from any other value whose text is digits.

* fix(custom_logger_v2): wrap a non-list guardrail mode as one mode like prometheus does

Prometheus treats any guardrail_mode that is not a list as a single mode, so a
tuple mode holds no str mode and a missing mode is one None mode. The
derivation unpacked tuples and rejected other non-str modes instead.

* feat(callbacks): let a native handler await typed host calls the python caller serves

* test(callbacks-python): drive a native handler's host calls through the python dispatcher

* test(custom_logger_v2): route a native logger with a host through the execution driver

* fix(custom_logger_v2): type the wrapped guardrail modes for pyright

* feat(custom_logger_v2): carry the python type name and hash error text of a stringified value

* feat(custom_logger_v2): serve the model group the logging handlers compute and key remaining limits by it

* feat(custom_logger_v2): tell a budget reservation that is not a mapping apart from none

* feat(custom_logger_v2): name malformed team model preferences in rejected with the text capacity logs

* feat(custom_logger_v2): serve a logging payload that is not a mapping apart from a missing one

* fix(callbacks-python): render null keys like a python dict merge, adding a missing key last

* feat(custom_logger_v2): carry an explicit output cap past i64 as a scalar

* feat(custom_logger_v2): serve the custom metadata labels prometheus derives from the logged payload

* feat(custom_logger_v2): serve the input sequence length bucket prometheus picks at attempt end

* feat(custom_logger_v2): carry the start and end times prometheus reads from the call details

* feat(custom_logger_v2): serve the limiter's body tags, session id, resolved agent and agent policy limits at pre_call

* fix(callbacks): export the limiter request record

* test(custom_logger_v2): give the type name cases stable ids and sort the record exports

* fix(custom_logger_v2): type the new derivations and tests for pyright

* fix(custom_logger_v2): drop team model preferences and serve the fallback_on_saturation handoff main's capacity callback reads

* style(callbacks): rustfmt the record exports

* refactor(custom_logger_v2): rename native callbacks to rust callbacks

native.py becomes rust.py, with native_logger, NativeLogger, NativeHandle,
NativeState and HostedNative renamed to their rust forms. The pyo3 classes
NativeCallback and NativeRequestState become RustCallback and
RustRequestState, and the tests, fixtures and messages follow.

* refactor(custom_logger_v2): rename slots to field sources

slots.py becomes field_sources.py: Slot is FieldSource, SlotFields is
SourcedFields, and with_slots, NO_SLOTS and slot_table_problem follow. A
table that only derives is named DERIVED_*_FIELDS, and a table that can
write back keeps SOURCES (PRE_CALL_AUTH_SOURCES, PRE_CALL_SOURCES). The
include_v2 comment moves down to the functions that take it.

* refactor(litellm_logging): place the v2 members of Logging beside what they serve

v2_scope, v2_auth and v2_failure_delivered join the other per-request
class defaults, v2_logging_scope and its opt-out helper follow
should_run_callback, v2_call_facts sits by the standard logging payload
builder, and claim_v2_failure sits above failure_handler.
bind_v2_request_scope checks for a Logging object directly instead of two
runtime protocols.

* refactor(proxy): rename v2_fields to v2_proxy_fields and say why it is separate

* refactor(custom_logger_v2): drop the logged_payload_not_a_mapping field

A logged payload that is present but not a mapping now looks the same to a
rust callback as a missing one, a parity gap accepted for a case that does
not occur. metadata_labels, input_sequence_length and limiter_request move
down to bits 44-46, on both sides of the contract.

* feat(callbacks): type a rust callback's request state

Callback gains an associated State type: request_state returns it and
handle and handle_hosted receive Option<&State> beside the input, which no
longer carries the state. ErasedCallback is the object-safe form the python
wrapper stores; it carries the state type-erased and downcasts it once,
returning CallbackError::StateType rather than panicking on a mismatch.
handle_hosted's default names its unused host _host. The record sample
count follows the dropped logged_payload_not_a_mapping field.

* docs(callbacks): say what PayloadRef, PayloadHost and RenderOptions.null_keys are

* refactor(callbacks-python): render payloads with python-compat's json writers

pyjson.rs reimplemented py_encode_basestring_ascii and json's floatstr.
python-compat's json module now exposes write_string, generic over a String
or byte sink, and float_text, and the payload renderer uses them.

* docs(callbacks): note why Respond, Reroute, call_ended and flush are not wired yet

* docs(custom_logger_v2): add a README for the package

* fix(custom_logger_v2): import the field sources tests from field_sources

* test(callbacks-python): stub the litellm package in each payload test

* fix(custom_logger_v2): type the disabled callbacks lookup and the v2 dedupe check
2026-11-14 02:33:27 +00:00
quietjules
f5ef14308b
fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved (#50516)
* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved

Converse used to drop any service_tier outside default, flex, and priority
with no error, rewrite auto to default, and drop reserved. Now reserved and
standard_only map to Bedrock tiers, auto sends no tier, and any other value
answers 400 unless drop_params is on. Nova batch rows honor the row's
drop_params.

* fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows

* test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs

* fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params

* fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params

* fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase

* fix(bedrock): mark the service tier block field read-only

* fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params

* test(prometheus): sort the service tier label test imports

* fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400

* fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova

* fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk

The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one

* test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name

The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it

* refactor(bedrock): drop the passthrough's own service tier header reader that main re-added

The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:33:57 +00:00
Nate Armstrong
20b6e855bd
feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0) (#51214) 2026-11-13 22:08:05 +00:00
ishaan-j-berri
46716fad56
fix(proxy): alert on customer org, customer team, and customer member budgets (#51579)
* fix(proxy): alert on customer org, customer team, and customer member budgets

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): scope customer alert dispatch and Redis tracing

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(alerts): escape customer identities in Slack budgets

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): dispatch customer budget events through custom loggers

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): read committed customer budget spend from writer

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): retain customer budget claims for the full period

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 09:16:33 -08:00
joshua-berri
e2dfd3dad1
fix(langfuse): reserve span capacity for failure usage attributes (#51538)
* fix(langfuse): reserve span capacity for failure usage attributes

* fix(langfuse): type pending failure attribute keys

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-12 21:14:14 -08:00
yujonglee
2be96aed13
refactor(redis): delete the Python RedisBatch now that every request op is native (#51383)
The request scope keeps only the native batches. RedisBatch, its ops, the cluster fallback, the Python post-call
deadline and the DualCache post-call helpers had no production caller left, and the pre-call SET, DEL and miss
tracking that UserApiKeyCache never reached go with them. The mixed-families pipeline metadata had no producer left
2026-11-13 03:22:28 +00:00
tin-berri
9633be87ea
feat(ui): show customer organization usage and budget utilization (#51483)
* feat(ui): show customer organization usage by date range

* feat(customers): align usage reads with daily spend contract

* fix(customers): combine spend display and resolve CI failures

* test(vector-stores): isolate search warnings from SDK verbosity

* fix(ui): cap customer usage date ranges at 400 days
2026-11-12 17:25:41 -08:00
ishaan-j-berri
df8b3b498e
feat(proxy): add customer hierarchy usage analytics (#51495) 2026-11-13 00:32:13 +00:00
joshua-berri
a7052ea8a8
fix(cost): preserve Bedrock cross-region pricing for Messages (#51479)
* fix(cost): bill Bedrock cross-region profiles at the profile rate on /v1/messages

Bedrock reports the region-stripped model id for a cross-region inference profile,
and the cost selector preferred that served id, so streaming and non-streaming
Messages spend (input, output, cache write and cache read) used the global row.
A served id carrying a different cross-region prefix keeps its own price.

Fixes #46286

* test(langfuse): derive the Bedrock profile export cost from the profile's model info

The regional us. deployment is now billed at its profile rate, so the expected
Langfuse cost comes from that profile's rates and the mock usage instead of a
hardcoded global price.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-12 15:54:56 -08:00
Kerry Lu
7130a5b4fd
feat(dynamic_rate_limiter_v3): prometheus share, borrowed and throttle metrics per team and model (#51417)
* feat(dynamic_rate_limiter_v3): Prometheus share, borrowed and throttle metrics per team and model

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): name the BLE001 suppressions on the metrics guards

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): count only final refusals, publish the served model's share, join the priority label

Review round 1: a steered alternate's refusal no longer counts as a throttle when the preferred model then admits
the request, a router fallback's settlement publishes the fallback target's own share under its model group, the
gauges carry the same priority label the 429 carries, the gauges aggregate with livemostrecent so a new window
replaces an old worker's value, the share read is skipped when no Prometheus logger is configured, the unit tests
restore the process registry, and the integration test polls for the settled totals.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(prometheus): type the gauge setter and the label factory so the type gate holds

_set_labeled_gauge takes a _MetricLike and sets the series only when it is a real Gauge, prometheus_label_factory
declares the dict[str, str | None] it returns, the priority resolver goes back to its original shape, and the
label test tolerates the now visible None values.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(prometheus): keep the label test diff to the two None-tolerant assertions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 15:28:51 -08:00
joshua-berri
dc1284d3d2
fix(otel): distinguish missing usage from prompt estimates on chat failures (#51443)
* fix(otel): distinguish unknown timeout usage from prompt estimates

* fix(otel): handle expected tokenizer failures and cover parent spans

* fix(otel): validate failure usage inputs at typed boundaries

* fix(otel): type Logging.messages for the failed prompt token estimate

Declare Logging.messages as Sequence[object] | None at its single __init__
assignment and drop the list TypeAdapter, so the failure estimate reads a typed
value without new unknown-type diagnostics.

* fix(otel): skip the failure prompt estimate for prompts with attachments

Only estimate the failed prompt when every message is plain text, so files,
documents and images never reach PDF parsing or media decoding on the failure
path. Those requests still report unknown provider usage without an estimate.

* fix(otel): count the failed prompt off the event loop for async requests

Async failure logging now awaits the existing offload_token_count limiter for the
optional prompt estimate, and the sync half of an async request no longer counts
inline, so a slow tokenizer load cannot stall other requests.

* fix(otel): only count the failed prompt for loggers that receive failures

The estimate check now reads general callbacks, the global failure lists through
a typed get_failure_callbacks accessor, and the dynamic failure lists, so a logger
registered only for success never triggers failure token counting.

* fix(otel): return an immutable failure callback snapshot

* fix(otel): keep malformed content blocks from breaking failure logging

Only compare string block types against the text block table, and run the
text-only check inside the estimate's narrow try, so a malformed prompt leaves
usage unknown without an estimate instead of interrupting failure callbacks.

* fix(otel): stop the token counter from debug-logging raw prompts

The failure estimate counts the original prompt, so the counter's debug line
could write prompt text to application logs while message logging was off.
Drop that line and assert the prompt stays out of debug logs on both routes.

* test(otel): pin text-only eligibility and failure callback categories

Cover the public has_only_text_content contract for malformed and non-text
messages, and check get_callbacks_by_type puts sync and async failure
registrations in the failure categories through the public manager APIs.

* fix(otel): keep failure logging when the prompt estimate cannot run

Return no estimate when the offloaded count hits a worker or tokenizer
RuntimeError or OSError, so failure callbacks still run, while cancellation
still propagates. Treat SDK Message objects like their dict form when checking
whether a prompt is text only.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-12 13:48:07 -08:00
joshua-berri
d723545320
fix(otel): report the provider model in experimental GenAI traces (#51430)
* fix(otel): report the provider's response model under gen_ai_latest_experimental

With OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, gen_ai.response.model now uses
the provider_response_model kept on the response hidden params, then the deployment model when
the response only echoes the client alias. Legacy mode and litellm.model_group keep the alias.

Fixes #51029

* fix(otel): type the semconv response model inputs and attach hidden params in tests

Bind the request model once through as_str so the deployment fallback no longer reads untyped
kwargs. Non-streamed fixtures now attach provider_response_model with set_hidden_params, since
ModelResponse(hidden_params=...) does not keep it on the pydantic private attribute.

* fix(otel): read the request model as object before narrowing it

Binding kwargs.get("model") to a Final[object] stops basedpyright from reporting an unknown
argument passed to as_str.

* fix(otel): validate the untyped request model through a TypeAdapter boundary

TypeAdapter(object).validate_python accepts the untyped kwargs value and returns a known object,
the existing litellm/utils.py and router.py pattern, so as_str no longer receives an Unknown.

* test(otel): stream the aliased response through a real Logging object

Greptile flagged the MagicMock logging_obj as a double on our own code. The streamed fixture now
builds a real Logging with a fixed start time, like test_streaming_handler.py.

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-12 10:45:56 -08:00
moe-berri
d2711ac559
feat(alerts): add management controls and opt-in daily summaries (#51418)
* feat(alerts): add alert controls and daily notification summaries

* fix(alerts): reserve daily notifications before delivery
2026-11-12 10:06:07 -08:00
zoveki
22252073e7
feat(bedrock): search knowledge bases by image and return image hits (#51063)
* feat(bedrock): search knowledge bases by image and return image hits

vector_stores.search and asearch take a query_image (base64 jpeg, png, gif or webp), and query is optional when it is sent. Bedrock sends it as an IMAGE retrievalQuery, and a caller's own retrievalQuery in extra_body still wins

Bedrock IMAGE hits now come back as image content items instead of being dropped. The media type comes from the source file mime type metadata, then the data URI label, then application/octet-stream

A request with both query and query_image, with neither, or with a malformed image gets a 400. So does query_image on a provider without image search, unless drop_params is on. The proxy accepts query_image in place of query on /v1/vector_stores/{id}/search

Spend logs redact image data when prompts are not stored. Emulated file_search skips image-only hits, RAG rerank maps its indices to the text documents it ranked, and the dashboard tester renders image hits

* fix(vector-stores): keep image hits to image searches and reject a caller retrievalQuery

Text searches on a multimodal Bedrock knowledge base return the same text hits as before, so chat, RAG and logging payloads stay unchanged. Image hits come back only when the caller sent query_image

A caller retrievalQuery now answers 400 pointing at query_image, or is dropped under drop_params, instead of replacing the built query unchecked

A file_search chat tool that carries query_image or rewrite_query searches again instead of failing, and a non-string query next to an image answers 400 instead of 500. query_image is keyword-only on search and asearch

* fix(vector-stores): redact query images and keep store search params out of image checks

* fix(vector-stores): redact images nested in extra_body and keep a store's extra_body retrievalQuery out of searches

* fix(vector_stores): keep a caller's image query through store and router params

A deployment's retrievalQuery no longer reaches a routed search, a caller's query fields survive a store's own extra_body, and an extra_body image stops before the provider. file_search forwards rewrite_query, both retrieval query spellings are redacted, and the cost error stops echoing the response when message logging is off

* fix(cost): leave image search responses out of the cost error text

* fix(vector_stores): accept line-wrapped base64 in query_image data

Bedrock takes inlineContent wrapped with \n or \r\n line breaks, as plain base64 writes it, so the check ignores only those before decoding and the data goes to Bedrock as sent

* fix(vector_stores): drop rewrite_query false from the chat file_search tool

False asks for the default search every store already runs, and MongoDB
rejects the field outright, so forwarding it made a MongoDB chat lose the
store's context for a no-op flag. True is still forwarded.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-12 00:19:34 +00:00
Yassin Kortam
2c44f26288
chore: remove cookbook and preserve live assets (#50944) 2026-11-11 15:38:27 -08:00
mrinal-berri
5e74515c29
feat(otel): let a team or key OTel v2 destination add its own request metadata keys (#50813)
* feat(otel): let a team or key OTel v2 destination add its own request metadata keys

A team or key callback can now set baggage_metadata_keys in its callback_vars,
a comma-separated list of requester_metadata.<key> paths. The tenant fan-out
adds them as litellm.metadata.<key> to that destination's spans only, on top of
the global LITELLM_OTEL_BAGGAGE_METADATA_KEYS; the operator's exporter and other
destinations never see them. Values are bound when each span ends, an attribute
the span already carries wins, and names of proxy identity fields are rejected
when saved, along with lists over 20 keys.

Resolves LIT-9545

* fix(otel): refuse request headers in a destination's baggage_metadata_keys

The proxy copies the request headers into requester_metadata.headers with
credential headers masked, but a header a gateway in front of it added
(an IAP or OIDC assertion, an internal token) stays as sent. A team or key
admin listing requester_metadata.headers.<name> could export it to an
endpoint the team controls. Entries under requester_metadata.headers are
now rejected when saved, so exporting a header stays an operator decision
in LITELLM_OTEL_BAGGAGE_METADATA_KEYS. Also drops an unused import.

* fix(otel): hold destination metadata to the configured span limits

A destination's extra litellm.metadata.* attributes were added to a span
view built from a plain mapping, so they skipped the limits the SDK
applies in set_attribute: a 1 MiB caller value survived an operator's
OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT. Values are now cut to the length
limit, and keys past the attribute-count limit are dropped and counted
instead of evicting an attribute the span already carries.

The per-destination views are built in one Final tuple ahead of the
forwarding loop, since a Final cannot be assigned inside the loop.

* fix(otel): apply destination metadata after the destination view, within the provider's limits

The span limits were checked against the span as it ended, before the
destination view strips prompt content and backend-only attributes, so a
model-call span full of llm.input_messages.* could drop the team's keys on
a no_content destination that had room. The values are now bound when the
span ends and added as the last step before export, counted against what
that destination actually sends, so their dropped count also survives the
view. The limits come from the tracer provider the fan-out is attached to,
read from the live span at start, instead of a separate env-built SpanLimits.

* refactor(otel): consolidate span limits, destination validators and metadata key parsing

Reads a span's limits in one guarded place that the emitter and the fan-out
share, replaces the two parallel lists of per-destination callback-var
validators with one mapping, splits and strips a baggage_metadata_keys list
once, drops a filter the post-view merge already enforces, and folds the
repeated fan-out test setup into one helper. No behavior change.

* ci: retrigger checks stranded by the GitHub Actions outage
2026-11-11 13:57:41 -08:00
yujonglee
11af2eb22d
refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822)
* refactor(rust_bridge): route every Rust-to-Python call through host modules

Rust now reaches Python only through litellm/rust_bridge/host/ (plus the
execution driver), and no Python module outside host/ imports it. Library
calls that reproduce Python behavior (httpx and OSError error text,
huggingface_hub downloads, the random generator, os.environ) go through
host functions too. The rule is written into host/AGENTS.md

The live litellm.model_cost table, its version and every Python writer move
to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only
the functions Rust calls. The cost map fetch passes use_http_settings instead
of handing Rust an http_settings() snapshot

* refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust

The Redis bridge asked Python for all of os.environ to read one key. CPython's
os.environ writes through to the process environment, so Rust reads it with
ProcessEnvironment like the rest of the module. Also drops doc comments that
restated their code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(rust): raise the native extension size cap to 55 MB

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:57:54 +00:00
Joseph Barker
2c9068dbab
fix(guardrails/rubrik): moderate text-completion prompts and forward rac_agent_id (#49912)
* fix(guardrails/rubrik): moderate text-completion prompts and forward rac_agent_id

_moderate_prompt synthesized a user message from texts for /v1/completions
requests but built the payload from structured_messages, so the webhook
received an empty messages list and the prompt went unmoderated. Pass the
resolved messages into _build_prompt_moderation_payload.

Forward metadata.rac_agent_id to the before_prompt and after_completion
webhooks so agent-scoped policies apply. Only that key is forwarded; the
rest of request_data metadata carries non-serializable objects and inbound
headers

* fix(guardrails/rubrik): also read rac_agent_id from litellm_metadata

/v1/messages, /v1/responses and other LITELLM_METADATA_ROUTES keep proxy
metadata under litellm_metadata, so a caller on those routes may set
rac_agent_id there. Read metadata first, then litellm_metadata, forwarding
only a non-empty string

* fix(guardrails/rubrik): take rac_agent_id from key or team metadata only

The Rubrik webhooks use rac_agent_id to choose which agent-scoped
policies apply. Forwarding it from the request body let a caller pick any
agent's policies, including one a policy exempts. Read it from the
caller's admin-configured key metadata, falling back to team metadata,
via CustomGuardrail._get_admin_metadata. The proxy writes those from the
authenticated key and strips client-supplied user_api_key_* fields from
both metadata dicts, so a caller cannot set or override the value

---------

Co-authored-by: yucheng-berri <yucheng@berri.ai>
2026-11-10 16:30:41 -08:00
taxfree
bc3a69ce35
fix(otel): serialize pydantic Message objects in langfuse_otel, weave_otel and langtrace (#46845)
langfuse_otel, weave_otel and langtrace now encode litellm.Message objects, and any value JSON cannot encode, when they write the prompt, so spans keep their token usage and model parameters. Under LITELLM_OTEL_V2 the payload builder dumps Message objects instead of dropping them
2026-11-10 14:50:39 -08:00
Mateo Wang
9fb29c7e38
fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions (#50467)
* fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions

An allowed_tools tool_choice in the flat Responses layout ({"type": "allowed_tools", "mode": ..., "tools": [...]})
answered 400 on litellm.completion, acompletion, Router and the proxy's chat and /v1/responses-to-chat routes. One
parser now reads both layouts into the nested Chat layout, and the existing Responses bridge flattens it back for
/v1/responses, so either layout reaches OpenAI in the shape each endpoint expects

* fix(tool_choice): drop an allowed_tools choice under drop_params and refuse conflicting flat entries

An allowed_tools tool_choice a provider can't enforce, or a malformed one, now drops under drop_params and answers 400 without it, the same rule LiteLLM uses for other unsupported values. That covers providers outside OpenAI and Azure, malformed choices on every provider, and Azure api versions older than 2023-12-01

A flat allowed_tools entry that carries both name and a nested function or custom key answers 400, since the two names can disagree, matching what flattening already refuses

* fix(tool-choice): keep the 400 when LiteLLM runs the tools, report auto after a drop

Under drop_params an allowed_tools choice a provider can't enforce was dropped, so on MCP gateway tools and in
run_tool_loop and arun_tool_loop LiteLLM could run a tool the allow list excluded. Those paths now keep the 400 they
answered before the flat layout was accepted, and a malformed choice stays a 400 on MCP gateway tools.

/v1/responses served through chat echoed a choice it had dropped, on sync, async and streamed responses. It now
reports auto, and a choice the provider received is still echoed as sent.

* fix(tool-choice): refuse an excluded tool right before LiteLLM runs it

Checking the provider up front missed a model alias, an old Azure api_version,
a litellm_proxy hop, and SDK fallbacks. The tool loop and the MCP gateway now
check the allowed_tools list right before they run a tool call, so an excluded
call answers 400 and never runs.

Under drop_params the choice now drops only when the caller runs every tool,
so a provider-run MCP server or built-in keeps the 400. The Azure checks read
the request's api_version, which also fixes the /v1/responses echo there.

* fix(tool-choice): drop the choice under drop_params and hand an excluded MCP call back unrun

The drop now applies to every provider that cannot take an allowed_tools
tool_choice, whatever the request's tools hold. LiteLLM never runs a call the
choice excludes: on MCP paths the whole round goes back to the caller in a 200,
the same shape as require_approval, and the tool loop keeps its 400 because it
only returns final content. A malformed choice stays a 400 before any request
on the tool loop and on chat with MCP gateway tools.

* fix(tool-choice): refuse an allowed_tools entry whose name is not a string

An mcp or other non-function entry with a list or dict name passed the
parser and crashed when the allow list was hashed. The parser now treats
it as malformed on both layouts, so it answers the same 400 as an empty
allow list, or drops under drop_params.

* fix(tool-choice): honor server-level and null-name mcp entries and stop hooks running excluded calls

* fix(tool-choice): match named mcp entries by server label and let built-in entries allow their interception call

* fix(tool-choice): leave a malformed allowed_tools choice unrewritten in the interception hooks

* fix(tool-choice): resolve mcp entries like the gateway and keep the choice on every hook path

Under drop_params a malformed allowed_tools choice now acts as if it was never sent, and only a choice that parses is stored and enforced. Without drop_params it answers 400 before any provider request, on /v1/responses with gateway tools before MCP listing

An mcp entry resolves its label the way the gateway picks servers to list (server name in any case, alias, access group, toolset), and a named entry matches the tool's own name or the gateway's prefixed name. The gateway rewrites mcp entries into function entries in the caller's layout and sends "none" when nothing is left

The web search follow-up keeps an allowed_tools choice, compression's litellm_content_retrieve is never excluded and joins the provider-facing choice, and the native /v1/responses hook loop applies the exclusion check and stores the choice

* fix(allowed_tools): stop web search and code interpreter follow-ups forcing another call

A kept allowed_tools choice now goes to the follow-up in mode auto, so a
required choice no longer loops to the agentic loop limit. The MCP label
scope dataclass uses a default_factory so the module imports on Python 3.11

* fix(allowed_tools): stop the Responses web search follow-up forcing another call

The Responses-shaped follow-up after a web search left tool_choice out of its patch, so the caller's required choice went out again through the chat to Responses bridge and looped to the loop limit. It now carries a kept allowed_tools choice in mode auto, like the chat follow-up

* fix(mcp): scope each MCP entry to its own allowed_tools and keep the allowlist on follow-ups

A label now covers only the tools its own allowed_tools lists, so a second label on the same server no longer widens it. Responses MCP follow-ups, streamed or not, keep an allowed_tools choice in auto mode instead of dropping it. The new Responses tests fake only the upstream MCP listing, so LiteLLM's own discovery runs

* fix(mcp): match an entry's allowed_tools against the tool name the gateway lists

A server with a display name override lists the display name, and the listing
filter matches allowed_tools against it, but the label scope only checked the
original name, so no spelling worked. The scope now also accepts the listed
name and its unprefixed part, like the listing filter. Tests now cover the
whole gateway and toolset entries and a display-named tool.

* fix(mcp): match an entry's allowed_tools only against a server's own prefix, not the first hyphen

* fix(mcp): match a tool no server resolves only by its exact listed name in an entry's allowed_tools

* fix(mcp): refuse a tool whose display name only looks like an allowed name, and let the chat follow-up answer

An MCP entry's allowed_tools now matches a tool only by its exact listed name or a spelling routing
accepts for its real name, so a display name that carries the server's own prefix no longer passes for
another tool. The chat MCP follow-up keeps an allowed_tools choice in auto mode, so the model can answer
with text after the tool ran.

* fix(mcp): match a toolset label's tools by exact native name

A toolset label served any tool whose name a routing spelling of a granted name matched, so a toolset granting the literal name "server-tool" also served the real tool "tool" that only another toolset grants. It now serves a tool only when the toolset grants its exact native name, the same check tool_is_granted uses when the gateway lists tools
2026-11-10 13:04:06 -08:00
norkelbyte
68e78bc53a
fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted (#50730)
* fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted

* fix(responses): forward and log a finished websocket turn when its early count fails or is cancelled

A usage payload the parser rejects no longer stops the relay before the terminal event reaches the client, and a client that leaves while the turn is being counted still gets the turn logged. A cancelled early count now clears its counted-tokens marker so the logging callback counts the turn in full.

* fix(responses): finish a turn's early token count when the client leaves

A client that left while a finished turn was being counted cancelled the count halfway. The router's count could be written with no marker, so the logging callback counted the turn a second time. The count now runs to the end before the turn is logged, and the failure path goes back to clearing only on errors

* fix(router): count one router request per Responses WebSocket turn

A socket whose finished turns each counted their own request in router
usage no longer adds one more request when it closes, so N turns count N
requests, as N HTTP calls do. A socket with no counted turn still counts 1

* fix(proxy): keep router usage markers out of pass-through client metadata and ClickHouse rows

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 04:13:51 +00:00
yujonglee
523b3efea2
refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501)
* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust

* fix(model-cost): keep main's compat helpers pure and move network tests to integration

ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 02:37:35 +00:00
zoveki
abe3b11f7a
fix(proxy): show committed spend in the remaining budget gauges (#49199)
* fix(proxy): report live key and team spend to budget gauges within the key cache TTL

* fix(prometheus): carry live spend for the user, org and customer budget gauges too

The remaining budget gauges for users, orgs and customers read the same cached rows as the key and
team gauges, so they read high for about a minute after a charge. Auth now carries the live counter
for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded.
The x-litellm-key-spend header and logging spend fields keep their current values.

* fix(prometheus): show committed spend in the remaining budget gauges

Reads twin counters that take only charged cost, so a worst-case budget hold
never shows as spent and a request that finishes last cannot set a gauge back.
The twins exist only while the Prometheus logger is registered, and every path
that resets a spend counter drops them. Failed requests now set the org gauge
from the carried org row and committed spend, not the cached org row.

* fix(prometheus): count a charge the cached customer row already holds once

* fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets

Auth now reads committed spend through the admission MGET and names the counters a charge
may move, so the master key and unbudgeted entities get no twin. Organization spend edits
and the global spend reset drop their twins, and a long request no longer reads as charged.

* fix(prometheus): read the committed org spend on failed requests without the auth object

* test(prometheus): cover a failed request with no carried org

* fix(prometheus): move the committed counters for charges with no auth on the row

A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them.

* fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes

Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a
charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed
counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys
and teams in bounded pages.

* fix(prometheus): count a charged failure's own cost in the org gauge

A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request

* fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog

* fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete

Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins

* fix(prometheus): keep committed spend values out of logger dumps of pass-through rows

CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values

* fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster

The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files

* fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-09 20:05:37 +00:00
yucheng-berri
b83a1f0608
fix(otel): name the routed model on auto-router LLM spans (#50498)
On an auto-router request the LLM span reported the name the caller asked
for (the router or an alias of it) as gen_ai.request.model (OTel v2) and
gen_ai.response.model (v1 and v2). Both now name the model group the router
picked, read from the request's routing_decision. In v2 the span name and the
vendor keys built from the request model (Langfuse, OpenInference, Langtrace,
Weave) follow. Requests that were not auto-routed are unchanged.
2026-11-07 15:52:00 -08:00
mrinal-berri
e3387b6fc5
fix(otel): seed request metadata baggage for team-only OTel v2 destinations (#50373)
* fix(otel): seed request metadata baggage for team-only OTel v2 destinations

A team newrelic (or other preset) destination exported spans without the
operator-allowlisted litellm.metadata.* keys unless a proxy-wide otel callback
was listed, because only the OTel logger's async_pre_call_hook seeded
requester_metadata into Baggage and that hook runs only for loggers in
litellm.callbacks. ProxyLogging.pre_call_hook now re-seeds identity with the
request data once requester_metadata exists.

Resolves LIT-9783

* fix(otel): seed only request metadata from the proxy pre-call hook

The pre-call re-seed also passed the request model, which by then can be an
alias target, mcp-tool-call or an internal embedding model, so it rewrote
gen_ai.request.model on the server span for V2 deployments without otel listed.
Seed only the request metadata and keep the model auth seeded. Adds a unit test
for the pre-call seed and the guardrails_only skip, drops the explanatory
comments, and requires one server span plus the model span in the integration
test.

* test(otel): annotate new test variables Final and build test config without mutation

Addresses review: the new unit and integration tests now follow AGENTS.md
(LIT010 Final on every variable, no in-place mutation, read-only parameter
types, one for-clause per comprehension).

* test(otel): cover key-level callbacks and assert tenant spans carry no credentials

Parametrizes the tenant metadata test over team and key callbacks, and adds a
test that sends credential headers and a planted New Relic key with an
allowlist naming credential headers, then asserts no exported span carries the
master key, the virtual key, the team's New Relic key or any credential header
value, and that every export used only the team's own New Relic key.
2026-11-06 14:47:48 -08:00
yujonglee
2b4485dee5
fix(guardrails): check a model-level post_call response once across the proxy and deployment layers (#50300)
#40154 made the deployment post_call hook really run apply_guardrail guardrails, so the proxy
checked the same response a second time (LIT-10481). The deployment hook now records guardrail +
response hash on the request and on the response, and the proxy skips a guardrail only on a
matching record. The hash covers every field except ids, usage, model and other metadata set
between layers, so a response rewritten between the layers (OCR included) is still checked.
Guardrails with their own post_call hook keep their pre-#40154 behavior

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 07:34:06 +09:00
Mateo Wang
3f38730d2c
refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986)
* refactor(typing): clear basedpyright Any errors (wave 1)

* refactor(typing): clear basedpyright Any errors (wave 2)

* refactor(typing): clear basedpyright Any errors (wave 3)

* refactor(typing): clear basedpyright Any errors (wave 4)

* refactor(typing): clear basedpyright Any errors (wave 5)

* refactor(typing): clear basedpyright Any errors (wave 6)

* fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves

* fix(logging): keep callbacks running when litellm_params is a params model

should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request

Adds behavior tests for the changed lines codecov listed as not run

* fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter

The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call.

* fix: bill regional uplift multipliers float() accepts, bytes included

* fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type
2026-11-06 13:31:15 -08:00
JeongCheolLee
febbe38326
fix(anthropic)!: do not surface server_tool_use as client tool_calls (#48385)
* fix(anthropic): do not surface server_tool_use as client tool_calls

Anthropic server-executed tools (web search, tool search, code execution)
were returned in chat completions tool_calls with srvtoolu_ ids, so
OpenAI-protocol agent clients treated them as pending work and re-invoked
the model until their recursion limit.

Move them to provider_specific_fields.server_tool_calls on both the
non-streaming and streaming paths. History replay, the /v1/responses
bridge and the /v1/messages streaming guardrail read that field, so
multi-turn reconstruction, web_search_call output items and the
guardrail's fail-closed check on server tool blocks keep working.

* fix(responses): return Claude web_fetch calls as web_search_call open_page items

* fix(responses): keep Claude server web calls without a result as hosted items

A non-streamed Claude turn that calls web_search or web_fetch next to a
client tool can return the server call before its result. Build an
in_progress web_search_call for it, so the Responses bridge no longer
returns it as a client function_call

* fix(anthropic): keep pending server calls native and hide server calls from client tool paths

Replaying a server call that has no result yet now sends it back as
server_tool_use, so Anthropic accepts the next turn. The tool permission
guardrail checks server calls again and rewrite mode removes a denied one.
The /v1/messages adapter keeps Claude's server blocks, and the Responses
bridge never returns a server call as a client function_call.

* fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls

Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls,
so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to
output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage
and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending
Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own
server_tool_use instead of running again

* fix(anthropic): let guardrails and loggers read Claude's server tool calls again

Guardrails, tool policies, tracing loggers and the Responses guardrail
translation read only client tool calls after server calls moved to
provider_specific_fields. They now read server calls first, then client
calls, through one shared helper, and the Responses guardrail treats
Claude's web_search_call items as web_search or web_fetch calls.
Non-streaming splits server calls by block type, as streaming does.

* fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing

A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy,
streamed or not. Redaction with message logging off hides server call arguments and hosted web
call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming
guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls.

* fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge

Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use.

* fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses

Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy.

* test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions

* test(guardrails): type the fake Pillar client like its Protocol

* fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool

Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or
web_search still needs that call and its thinking replayed, or Claude loses the fetch result

* fix(guardrails): scan replayed Claude server calls and hosted web calls

Request guardrails now see and rewrite server_tool_calls replayed in chat
messages. Cato and Purview scan Claude web_search and web_fetch calls on
Responses output, and a function tool call with an extra custom key is no
longer dropped from Cisco, Purview, Repello and PANW scans.

* fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks

* fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows

* test: annotate the new server tool call test locals with Final and type their parameters

* test: type the new guardrail test helpers with litellm's tool call types instead of Any

* test: narrow the remaining new test helper parameters from bare dict and object

* fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs

Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls

* fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera

* fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-06 03:54:34 -08:00
ishaan-j-berri
da2ecbc5bb
feat: email affected owners before model deprecations (#50172)
* feat: opt-in model deprecation emails for affected owners

* fix: annotate deprecation sender initialization

* fix: clarify deprecation emails and use consistent settings toggles

* fix: exclude former team admins from deprecation emails

* feat: clarify provider deprecations and model migration steps
2026-11-05 16:14:12 +08:00
zoveki
784754ca48
fix(azure)!: answer 401 when Entra rejects the deployment's credential (#49863)
* fix(azure): answer 401 when Entra rejects the deployment's credential

A ClientAuthenticationError from the Entra token provider (wrong, expired or rotated client secret) used to surface as a retryable 500 carrying the app's client id and Entra trace ids. Map it to a 401 AuthenticationError that keeps the AADSTS code and first sentence with GUIDs redacted, on both the direct token fetch and the provider handed to the OpenAI SDK client. Network failures stay 500.

* fix(azure): keep network failures to Entra a retryable 500

azure-identity wraps a transport failure in ClientAuthenticationError, so the
401 mapping now skips errors whose cause chain holds ServiceRequestError or
ServiceResponseError

* fix(azure): cover async token providers and Entra 5xx, and scope caller-sent Entra credentials

* fix(azure): map only AADSTS-coded Entra rejections to 401 and keep caller credential failures out of cooldown

An Entra ClientAuthenticationError now answers 401 only when its message carries an AADSTS code, its HTTP
status is not 429 or 5xx, and the code is not the AADSTS50196 throttle, so a chained credential's outage or
throttle stays a retryable error. Caller-sent Entra fields no longer change the deployment id; a 401 from a
value the caller sent skips the shared deployment's cooldown instead. The Azure AI Foundry agent token fetch
maps the same rejection, and the A2A endpoint answers it with HTTP 401.

* fix(router): count router default_litellm_params as configured Entra credentials when deciding cooldown

* fix(azure): keep transient Entra codes retryable and scope 401 handling to Entra rejections

ClientSecretCredential drops Entra's HTTP status, so throttling and transient
codes (AADSTS50033, 50196, 90006, 90024, 90033, 90055) now stay retryable
instead of mapping to 401.

The A2A endpoint answers 401 only for an AuthenticationError caused by an
Entra token rejection, and the router skips cooldown for a caller-sent Entra
field only on Azure-family deployments. The azure/ Decisions route and the
Bing grounding search tool now map an environment-credential rejection to the
same 401.

* fix(azure): map only known Entra credential rejections to 401 and cover the OIDC token exchange

Unlisted AADSTS codes keep the retryable 500, the router skips the 401 cooldown only when an Entra rejection caused it, and a federated token Entra rejects answers the same 401 on Azure chat, Responses and A2A agents

* fix(azure): answer 401 for Entra rejections on the /azure passthrough and token counting

The fallback hop skips cooldown when Entra rejected a credential the caller sent,
Prometheus counts a deployment's own Entra rejection as a deployment failure again,
and an OIDC exchange error keeps Entra's headers so Retry-After reaches the router.

* fix(azure): keep a no-op caller Entra field from skipping cooldown and redact the passthrough 401 message

A caller-sent Entra field now skips the deployment's cooldown only when it
differs from the deployment, its named credential, the router defaults, the
AZURE_* environment variables and the default scope. An empty value counts as
unset, so a deployment whose own secret is wrong still cools down when the
caller's field changes nothing about the token request.

The /azure passthrough's Entra 401 message goes through the same internal
hostname redaction as other client-facing errors.

* fix(router): compare a caller's Entra field with the value the deployment uses

A caller-sent Entra field skipped the cooldown only when it matched no
configured source, so a value equal to an environment default the deployment
overrides (a stale AZURE_CLIENT_SECRET, say) counted as the deployment's own
and let a caller's rejected credential cool down a working deployment.

The check now resolves the value the deployment would use without the caller,
in order from the router defaults, the deployment, its named credential, the
AZURE_* environment variables and the default scope, and skips the cooldown
only when the caller's value differs from it.

* fix(azure): follow the settings merge for empty Entra fields and keep non-rejection OIDC exchange errors unchanged

* fix(azure): count only the failed attempt's Entra rejection and ignore a non-callable token provider

* fix(azure): keep the streaming Entra rejection inside the failed attempt

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-04 20:44:55 +00:00