mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-18 09:45:23 +00:00
205 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9e81aa3f40
|
feat(shadow-eval): skip redundant reverse evaluations on opt-in (#52192)
* feat(shadow-eval): let reverse jobs skip samples the baseline group served A reverse shadow eval judges the router's response against baseline_model. When the router itself sent the request to the baseline's model group, the comparison only produces ties and still pays for a shadow and a judge call. Reverse jobs can now opt in with skip_same_model. A sampled request whose served group (router alias resolved, fallback hops respected) is the baseline's own group is counted in a new same_model funnel stage instead of being judged. It makes no provider calls, writes no attempt row, and leaves max_turns and the eval budget untouched. When the served group is unknown the sample is judged as before. The job detail reports same_model_skipped_count even when nothing has been judged yet. The flag defaults to false and is rejected on forward jobs, so existing jobs and requests behave exactly as before. Adds two columns through an additive, schema-only migration. * test(shadow-eval): drive same-model skip tests off explicit completion signals Judged cases now wait on the attempt write the injected prisma reports, then await the pipeline task that made it. Skip cases drain only the tasks the hook created, through a task factory scoped to the hook call, before asserting that no provider call, attempt row, or spend was recorded. This replaces a whole-loop task diff and the assertion on the number of tasks. Adds a skip case with tier parameter overrides, which the policy still treats as the same group, and a case showing a lost sampling roll counts as not_sampled before same_model. Drops the class docstring claim that a same group comparison only produces ties. * style(tests): remove extra shadow evaluation test spacing * fix(ui): send skip_same_model false from the shadow eval start form The generated request type now requires skip_same_model, which broke the dashboard typecheck and builds. The form sends false, the API default, so jobs started from the UI behave exactly as before. * fix(migrations): bound shadow evaluation schema lock waits --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
43d6b45cb7
|
fix(bedrock)!: split the OpenAI breakpoint and Bedrock cachePoint model flags (#52180)
* fix(bedrock): split the OpenAI breakpoint and Bedrock cachePoint model flags supports_prompt_cache_breakpoint meant both "uses the OpenAI prompt_cache_breakpoint format" and "Bedrock Converse takes cachePoint blocks", so an OpenAI-format opt-out on a Bedrock Claude deployment silently turned off its prompt caching. Bedrock now reads only supports_bedrock_converse_cache_point, then supports_prompt_caching as before. The OpenAI path reads supports_openai_prompt_cache_breakpoint and keeps the old key as a deprecated alias. The Kimi K3 catalog rows move to the new Bedrock key, and a Bedrock deployment still on the old key logs a one-time migration warning. Fixes #52140 * fix(bedrock): expose the split prompt cache flags in get_model_info and word the migration warning for unlisted models * test(bedrock): mark the system prompt directly in the routed cache point regression * fix(bedrock): warn about the old breakpoint flag even when a catalog cache point flag decides * fix(cost-map): keep the old Kimi K3 breakpoint flag for clients that read the remote catalog * test(bedrock): keep the old breakpoint flag on catalog cache point opt-outs for older clients --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
8fadc09434
|
feat(focus): upgrade FOCUS export to the 1.5 Cost and Usage schema (#52105)
* feat(focus): upgrade cost and usage export to FOCUS 1.5 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(focus): identify the combined daily request SKU meter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
5792794691
|
feat(alerts): report observed Anthropic cache misses as advisories (#51973)
* feat(alerts): open a prompt cache alert when a proven Anthropic prefix stops reading and writing the cache Behind general_settings.prompt_cache_alerts, which is off by default, a logger watches native Anthropic chat completions that send exactly one explicit five-minute cache breakpoint. A cache write the provider reported for that prefix proves it. Three later requests that each report zero cache reads and writes, all started after the proof and within its five minutes, save one alert for the deployment and configuration through the existing alert repository, so a dismissed or resolved alert is never reopened. Only the provider's raw response counts. A missing or contradictory usage is unknown, never zero. Streaming requests, failures and unknown usage end the streak, and so does a caching request whose prefix can't be compared. A read or a new write resets it. The prefix is a keyed hash, and the configuration covers the model, endpoint and a keyed digest of the API key, so nothing from the prompt or credentials is kept. Streaks live in the proxy's shared Redis, one Lua script per update, and requests are ordered by their own start and end so replays and overlaps can't add misses. Without Redis nothing is tracked. A Redis or database error never affects the request, and evidence lost to Redis ends its scope's streak before anything else counts. * fix(alerts): reset the whole deployment for unassessable prompt cache evidence and quiet a saved configuration Streamed attempts log an empty API key, so their evidence could not reach the streak of the assessed configuration. Streams, failures, missing usage and prefixes that can't be compared now reset every streak on their deployment. Assessed traffic keeps its credential, model and endpoint specific configuration. After an alert is saved, its configuration stays quiet in Redis for a day, so later misses don't write the same incident again. The quiet window is only set once the save succeeds, and a failed save can trigger again. Evidence lost to Redis now resets its deployment from a loop instead of recursion, and a lost reset older than any proof is dropped. Asyncio timeouts count as store failures on Python 3.10, and the module no longer uses StrEnum. The proxy imports and registers the logger only when prompt_cache_alerts is on, and removes it from the callback lists when the lifespan ends. * fix(alerts): keep the proxy lifespan a context manager and reset deployments seen while Redis is reloading Registering prompt cache alerts had moved the asynccontextmanager decorator from the proxy lifespan onto the new helper, which broke every proxy start. The decorator is back on _proxy_lifespan, and a lifespan test now checks that registration runs on entry and its cleanup on exit. A config reload can leave the shared Redis getter empty for a moment. Evidence seen then is remembered like evidence lost to a Redis error, so its deployment is reset before anything else reaches Redis once it is back. * test(alerts): let a failed completion raise its mapped error before asserting the deployment reset * fix(alerts): describe prompt cache alerts as advisory and type their boundaries Only callbacks that reach the logger and Redis are observed, so delayed, lost or unlogged requests can fall between the misses. The docstring now says an alert does not guarantee consecutive misses or an outage, and that a hit delivered after the stored proof expired leaves the prefix unproven until a new write. Tests pin both behaviors. The logger no longer calls the partially typed CustomLogger constructor, whose two attributes every reader already defaults, and registration reads the typed general settings view. * fix(alerts): keep retained prompt cache misses alive as long as a refreshed proof A proof or hit that returned before already stored misses but arrived after them extended the proof key while the misses kept their older expiry, so they could vanish early and an observed streak would be lost. Both refresh branches now give the retained misses the proof's expiry. * fix(alerts): serialize each message's fields once in the prompt cache prefix Every block's identity carried its message's other fields, so a large field on a message with many blocks made the canonical prefix grow with fields times blocks. The message's fields are now one segment ahead of its blocks, keeping role, metadata, block order and fail-closed redaction in the prefix while its size stays linear in the request. * test(proxy): keep a leaked worker config out of the prompt cache lifespan test An earlier test leaves WORKER_CONFIG in the environment, and the lifespan then loads its config path. The registration test now clears it with the database URLs. * test(alerts): clean each prompt cache deployment tag in its own comprehension --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
c482013a8a
|
fix(guardrails): log requests that a guardrail blocks on /apply_guardrail (#51014)
* fix(guardrails): record the guardrail verdict on a request dict that starts empty
The Bedrock guardrail and the shared `log_guardrail_information` wrapper both picked the request dict with `or {}`. An empty dict is falsy, so a caller that passed one had the verdict written to a throwaway dict. /apply_guardrail passes an empty dict for a request that carries no messages and no metadata, so nothing downstream could see what the guardrail decided. The verdict now goes to the dict the caller passed, and only a missing dict falls back to a new one.
* fix(guardrails): log requests that a guardrail blocks on /apply_guardrail
A request a guardrail blocked on /apply_guardrail was missing from the logs. The endpoint skipped the failure handlers for an HTTPException, the spend log failure hook wrote no row because the route is not an LLM API route, and the guardrail's verdict stayed on a request dict no logger reads.
The endpoint now runs the failure handlers for every exception and copies the verdict onto the logged request, so allowed and blocked requests both carry it along with the guardrail's cost. It marks the failure of a request it had set up with a sentinel, and the spend log failure hook writes its row for that mark. A request rejected before setup, such as one with a bad key or an unknown guardrail, is still not logged. Guardrail records a caller sends in metadata are dropped for both response formats so a verdict cannot be forged in the logs.
* fix(guardrails): keep Presidio and Gray Swan verdicts on empty request data
Presidio's text-only path and Gray Swan's API-failure path replaced an empty request dict with a throwaway one, so the verdict never reached the logged request. The apply_guardrail logging tests now run the real request setup and read what a logging callback receives.
* test(guardrails): cover the endpoint failure row, the sync decorator, and an unreachable Gray Swan without real sockets
* fix(logging): redact prompt text in guardrail records when message logging is off
* fix(logging): redact the failure payload when message logging is off
* fix(logging): keep guardrail details for other callbacks when one opts out of message logging
* fix(logging): redact failure events per callback and keep guardrail compression stats
A callback that turns off message logging now gets the same redacted copy on failure events as on success events, so a blocked /apply_guardrail call no longer hands it the raw prompt and verdict
Guardrail redaction keeps the numeric compression stats (tokens saved, compression ratio) that spend tracking reads, through one helper shared with the spend-log sanitizer
* fix(policy): test each batch input on its own copy of the request data
The guardrail logging decorator now records a verdict on an empty request dict instead of dropping it. The policy test endpoint shared one request dict across every input and merged it into the agent call, so each later input and agent request carried the earlier inputs' verdicts. Each input now gets its own copy, which keeps the caller's request data unchanged
* test(policy): look up the counting guardrail through the real registry
The batch isolation test now registers its guardrail as a litellm callback, so the endpoint finds it through the real guardrail registry instead of a mocked one, and its apply_guardrail override carries the same concrete types as the base class
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
|
1b62821ba9
|
fix(otel v2): stop legacy langfuse loggers sending to refused key/team hosts (#51344)
* fix(otel v2): stop the legacy langfuse_otel logger exporting to refused key/team hosts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel v2): assert the refused-host regression through public config and header outputs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): refuse key/team langfuse hosts in the langfuse callback and prompt fetch, and fail closed on unparseable hosts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): check the langfuse SDK clients against the http URL they actually call Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): skip allowlist entries that won't parse and read the V2 flag only for unlisted hosts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): a malformed key/team langfuse host drops only its own destination Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(otel v2): read the V2 flag before the allowlist and cache the allowlist and flag checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
c60a320b05
|
perf(token_counter): memoize token counts of long texts per tokenizer (#51904)
* perf(token_counter): memoize token counts of long texts per tokenizer * test(prompt_caching): count a prompt no other test has counted in the loop lag checks * fix(token_counter): key the count memo on the string hash instead of blake2b * test(router): embed distinct texts so the batches are counted in full * test: time token counts over text no earlier count has memoized |
||
|
|
90886f4f5e
|
feat(control_plane): ocsf audit events, sink protocol, queued delivery and hash chain (#49660)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
5b722c7c6f
|
feat(control_plane): otel server and dataplane client spans with w3c propagation (#49659)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
8d200a4f98
|
feat(messages): serve bedrock invoke messages on rust only (#51330) | ||
|
|
09ce4f50ed
|
perf(auth): read token model fields from the class instead of deprecated __fields__ (#51901)
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
unit / root (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / enterprise-managed-files (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / rust-bridge (push) Waiting to run
unit / assert-shard-coverage (push) Waiting to run
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* perf(auth): read token model fields from the class instead of deprecated __fields__ * test(s3): assert 403 retry backoff on the logged wait instead of wall-clock gaps |
||
|
|
3bcd77954e
|
refactor(s3-log): move the rust-backed s3 logger from s3_v3 to s3_v2 (#51893) | ||
|
|
a1ea22e996
|
feat(otel v2): proxy-level default langfuse_span_scope for key and team destinations (#51726)
* feat(otel v2): proxy-level default langfuse_span_scope for key and team destinations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): regenerate dashboard API documentation types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): preserve tenant destinations with malformed otel settings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): cover explicit narrow scope and non-Langfuse defaults Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Read only the prefixed Langfuse destination scope environment variable Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
da55893656
|
ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864)
* test(unit): reset module state only when a test changed it The proxy conftest deep-copied every snapshotted litellm attribute before each test, and the copy of litellm.model_cost alone was a median 0.12 s for each of 56,031 proxy tests on the runner. The reset now copies an attribute back only when it no longer equals the snapshot. The enterprise and MCP server conftests reloaded litellm before every test, 3.27 s and 0.21 s per test. They now use the same reset. * test(unit): remove six flaky tests They failed on main without a code change behind them between 2026-11-05 and 2026-11-14: the vector store hook SDK fallback, the concurrent user config TPM count, three Prometheus metrics server port tests and the OpenCode list cost billing test. * ci: run four heavy unit shards on 16-core runners and keep two smoke jobs off pull requests proxy-feature-endpoints, proxy-endpoints, root and llms-providers ran 1.9 to 2.2 times faster with 8 workers on a 16-core runner in a benchmark run. windows-sdk and proxy-build-from-pip still run on pushes to main and manual runs. proxy-build-from-pip installs a published release and never runs the pull request's code. * test: move 109 tests that start a process or open a socket into tests/integration tests/unit is in-process by its own rules, and these tests spawn a Python interpreter, a shell or a local TCP or HTTP server. They took 921 s of unit shard time per run, 396 s of it in the MCP client transport tests. Each one now lives in a *_real_io.py file under the integration directory for its domain and runs with that directory's group. The tests themselves are unchanged apart from one repository-root path in the CLI test. * test(unit): copy back only the dict entries a test changed A test module that registers models at import leaves litellm.model_cost four entries larger than the snapshot before every test in its directory, so the reset still deep-copied the whole price map for each of them. It now rebuilds a changed dict from the live entries that still match and copies only the rest. * ci(smoke): keep the two jobs that skip on pull requests out of the smoke gate A skipped job passing the gate is what test_unit_passed_gate guards against, so the gate goes back to requiring success from every job it needs and windows-sdk and proxy-build-from-pip stop being part of it. They still run and report on pushes to main and manual runs. |
||
|
|
8e92894ff2
|
feat(s3-log): replace the python s3_v2 logger with a rust callback (#51539)
The s3_v2 S3Logger is ported to a Rust callback, callbacks-s3-log, behind the CustomLoggerV2 contract, with the bridge constructor NativeS3Log and the Python glue in litellm/integrations/s3_v3.py. It is now the only implementation: the Python logger is deleted, s3_v2.py re-exports the Rust-backed S3Logger, and LITELLM_NATIVE_S3_V2 is removed. The callback keeps its name, list position and settings, so no config changes. The original s3_v2 tests are replicated against the Rust callback as <original>_rust.py, and Python's outputs for the differential suite are recorded in tests/unit/integrations/s3_v2_golden/python_s3_v2.json. |
||
|
|
580203cce6
|
fix(otel): send the Langfuse environment, release and version on OTel v2 (#51385)
* fix(otel): send the Langfuse environment, release and version on OTel v2 * fix(otel): keep the Langfuse environment on the Langfuse exporter only * fix(otel): annotate the exporter resource attributes as a read-only mapping * fix(otel): shed the Langfuse release and version first at the attribute limit * fix(otel): keep base attributes when the Langfuse release and version do not fit - the model span sheds release and version before the identity fields - the request root takes release and version only when they fit under its attribute limit - an exporter routed to a tenant's credentials drops the proxy's Langfuse environment - a langfuse.environment the proxy's resource already names wins over LANGFUSE_TRACING_ENVIRONMENT * test(otel): annotate the new attribute-limit and routing tests |
||
|
|
9fba44d300
|
fix(openai): cap and bill web search rounds on Responses-pinned chat completions (#51802)
* fix(openai): cap web search agentic loops with max_agentic_loops on OpenAI chat completions The OpenAI chat dispatcher ran every custom logger's chat agentic loop without a depth check, and the web search follow-up never carried the loop depth, so max_agentic_loops was ignored on openai/ deployments. The dispatcher now skips loggers the shared chat completion loop in litellm.acompletion already runs, so those go through its depth, cap, and repeated-call checks once. Loggers that only implement the chat-specific hooks still run in the OpenAI dispatcher * fix(openai): honor max_agentic_loops when chat completions bridge to Responses A gpt-5.4+ chat completion with function tools on api.openai.com is sent to /v1/responses by the Responses bridge. The Responses handler then ran its own agentic loop on that call, which never saw the request's max_agentic_loops or depth, and the shared chat completion loop in litellm.acompletion ran again on what it returned, so every chat follow-up restarted the Responses loop at depth 0. With max_agentic_loops=1, web search made 8 Responses calls and 7 searches The async bridge now marks its Responses call as part of a chat completion turn, and the Responses handler skips its own loop for such calls. The shared chat loop owns the whole turn, so its depth, cap, and repeated-call checks run once * fix(responses): keep search follow-ups on the Responses API for responses/ pinned models When an async chat completion crosses the Responses bridge, the shared chat loop now owns the turn and builds each web search follow-up from the model recorded on the logging object. responses() overwrote that record with the stripped model, so a model pinned with the responses/ prefix (for example openai/responses/<model> on a Responses-only gateway) sent its follow-up to chat completions, and the turn ended on the unexecuted tool call responses() now leaves the record alone when the chat loop owns the turn, the same way completion() skips its own record when the reverse bridge owns it * fix(agentic-loop): bill every round of a non-streamed Responses-bridged chat loop once A bridged chat request logs its first round before the chat loop runs, so a follow-up run as an internal sub-call was never billed. Each follow-up of a non-streamed bridged parent now writes its own spend row, and a failed follow-up no longer re-bills rounds that already wrote one * fix(agentic-loop): keep streamed Responses-bridged web search from billing its final round twice * fix(openai): bill every bridged search round in one row after guardrails On the proxy a non-streamed Responses-pinned chat request now parks one spend row that sums every round the loop served, so the row and the x-litellm-response-cost header show the whole request and nothing is written before post-call guardrails pass. A streamed request keeps round one's row and folds every later round into the replayed final row. The bridge now skips only Responses hooks that run their own chat loop, so hooks that only handle the Responses API still run. * fix(openai): price each bridged search round at its own size Each round of a bridged chat request is now priced on its own usage and the combined proxy row adds those prices, so summed usage never crosses a long-context price tier. A streamed replay keeps that total and its cost breakdown * fix(agentic-loop): run the chat hook of bridged callbacks that define both agentic hooks The bridge skipped the Responses hook of a callback that runs its own chat loop, and the shared loop only asked the general gate, so a custom callback with both hooks stopped intercepting bridged chat. Ask its chat gate after the shared loop. Price tier tests now use test-owned gpt-5.6 prices. * fix(agentic-loop): route bridged chat to the chat hook only for callbacks that opt in A custom callback that defined both agentic hooks lost its Responses hook on chat completions bridged to the Responses API, so a chat hook that declined left the request unhandled, and one that claimed it changed which hook answered compared with main. Callbacks now opt in with wants_chat_completion_hooks_on_bridged_chat. Web search interception sets it, so its loop still runs once on the chat side. Every other callback keeps main's routing through its Responses hook. * fix(websearch_interception): keep main's Responses loop for a bridged web_search_options call --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
9d827093c5
|
feat(github_copilot): support GitHub App authentication with PEM upload (#51427)
* feat(github_copilot): support GitHub App installation authentication * Add GitHub App authentication to the Copilot credential form * Explain secure private-key setup beside Copilot App fields * Store uploaded Copilot App keys through encrypted named credentials * fix(github_copilot): handle App validation errors and credential replacement * fix(ui): make GitHub App field help accessible * fix(github_copilot): preserve saved App key references on edit * fix(github_copilot): trust litellm's CA settings when minting App tokens and hide App keys from /health * fix(github_copilot): keep device-login headers, mint App tokens off the shared thread pool, cache App responses Device-login requests keep the merge base header behavior, so caller headers still replace the Copilot defaults. Only App requests merge and pin their identity headers. App token refreshes are single-flight per cache key with a shared future. Only the refreshing caller uses a thread, so waiting callers no longer fill the default executor that every provider call shares. A cached App token is served before the key secret is read. Uploaded keys still rotate the token when their contents change. App deployments are deployment-owned, so they share the response cache like shared device-login deployments. App credentials are validated even when credential_info names no provider. * fix(github_copilot): finish an App token refresh even when the caller that started it is cancelled The caller that owns a refresh used to await asyncio.to_thread, so cancelling it while the job was still queued cancelled the job, left the in-flight future unfinished, and every later request for that App waited on it. The refresh now runs in the executor without being tied to the caller, and every caller, the owner included, awaits the shared future * fix(github_copilot): refuse deployments naming an unloaded credential, keep App key references, accept numeric App IDs A Copilot deployment whose litellm_credential_name is not loaded now answers 400 instead of falling back to the shared device login. The router registry keeps github_copilot_app_private_key_ref values as references, a YAML App ID can be an integer, the server-owned field error names Copilot auth for Copilot fields, and the credential edit modal shows the stored provider again * fix(github_copilot): report a deployment naming an unloaded credential as unhealthy The /health check now answers unhealthy with the same 400 the request path returns for a Copilot deployment whose litellm_credential_name is not loaded, instead of running it through the shared device login * fix(github_copilot): keep the base fallback for an unloaded credential and report the App host as api_base A Copilot deployment that names an unloaded credential uses the shared device login again, as on base. A deployment that sets github_copilot_auth_type: github_app still refuses before any GitHub call. get_api_base now returns the Copilot host an App or per-user session calls instead of an admin api_base it ignores. * fix(github_copilot): count a GitHub rejection of the App installation token as a Prometheus deployment failure * fix(github_copilot): keep per-user session api_base reporting as on base and name Copilot in the non-admin 403 Only an App deployment reports its App host as api_base before a configured api_base. Per-user OAuth sessions report the same api_base they did on base. The 403 for a non-admin attaching App credentials now names GitHub Copilot authentication. The token-dir fixture in the App api_base test is applied with usefixtures. * fix(github_copilot): report the configured api_base instead of raising when GitHub App settings are incomplete * test(github_copilot): type the incomplete App settings test parameter --------- Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
3ad164382b
|
fix(responses): send cache_control as a prompt_cache_breakpoint on the chat-to-Responses bridge (#51783)
* fix(responses bridge): carry cache_control as a prompt_cache_breakpoint to deployments that read them A chat request with Anthropic-style cache_control on a system or user part got its mark renamed to prompt_cache_breakpoint on the plain chat path, but the chat-to-Responses bridge (taken once a function tool and reasoning are on, and always for bedrock_mantle GPT) dropped it, so explicit-mode caching never wrote. The bridge now renames the marks before converting the messages when the served deployment reads breakpoints, using the same gate the Responses config uses for native /v1/responses input. Deployments that do not read breakpoints keep today's behavior. The rename also leaves unmarked content parts untouched instead of validating them, so a part with non-string keys no longer raises on either path. * fix(responses): keep leading system messages in client order once one goes to input A marked first system string became a list and moved to input while the next unmarked string stayed in instructions, so the model read them in reverse. The leading run that joins instructions now stops at the first system message whose content is not a string * fix(responses): leave Bedrock Mantle and unmarked system order as they were on the bridge Bedrock Mantle already caches the growing prefix on its own, so turning cache_control into breakpoints there added write charges. The client-order rule for leading system messages now applies only when a mark turns a system string into a list, so unmarked chats send the same request as before. * test(responses): cover a marked user turn in the unmarked system split test A cache_control mark on the user message must not switch unmarked leading system messages to the client-order rule. The split test now runs with the user turn plain and marked, so dropping the role check from the system string test fails it. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
9ba216d521
|
fix(vertex_ai): reuse one Google token across RAG Engine searches and cold bursts (#51239)
* fix(vertex_ai): reuse one Google token across vector store searches * fix(vertex_ai): load Vertex credentials once when a cold worker gets a burst * test(vertex_ai): count Google token fetches for RAG Engine searches through the proxy * test(vertex_ai): type the fixtures and token callback in the concurrent search test * fix(vertex_ai): lock Google token loads per credential and reload a key Google rejects A shared RAG Engine config made one store's slow token fetch hold up every other store, and kept a rotated key file's deleted key until a restart. First loads and expired refreshes now take a lock per credentials and project, refreshes take a lock per credentials object, and any RefreshError reloads the credentials once. * fix(vertex_ai): share one Google token load or failure per credential and reload rejected keys on async calls * test(vertex_ai): check async callers sharing an expired credential refresh it once * fix(vertex_ai): reload Google credentials only when a refresh is rejected for good A retryable refresh error such as a 503 now raises without reloading the key, so an outage does not double google-auth's retries. Credentials that cannot be weak-referenced share the config's refresh lock again, as on main. The burst test for a rejected key moves to VertexBase and holds the token response until the other callers have started * fix(vertex_ai): reload Google credentials only when Google answers invalid_grant google-auth leaves its retryable flag False on metadata server, impersonation, and workload identity outages, and on a 502, so the previous check reloaded the key file on those outages and doubled the token tries. Now only an invalid_grant answer or "Reauthentication is needed" reloads The sync reload also refills the cache under the key the caller asked with, so a call without vertex_project keeps one cached copy, as the async reload already did * fix(vertex_ai): pick up a swapped Google key file on the next call The credential cache key now carries the key file's inode, mtime and size, for a vertex_credentials path and for GOOGLE_APPLICATION_CREDENTIALS. A key file that is replaced or rewritten, such as a rotated Kubernetes secret, is loaded on the next call instead of after Google rejects the old key or the proxy restarts * fix(vertex_ai): keep the earlier key while a replaced key file does not load A key file caught halfway through a non-atomic write, or replaced by one that does not load, now keeps the credentials loaded from its earlier version in use and logs a warning, as the cache did before key files were re-read. The next complete file is loaded on the next call. The pub/sub test no longer writes a key file it only reaches through the seeded cache * fix(vertex_ai): try a replaced key file again when its first token fetch fails The earlier key now stands in only when the replaced file does not load. A token fetch that fails on the new key raises like any first load, so the next call tries the new key again instead of keeping the old one * fix(vertex): raise when an async reload keeps an earlier key Google rejected --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
cd2e4b6b60
|
feat(custom-logger-v2): add the contract for rust callbacks and bind the proxy request scope (#51526)
* feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0) * feat(custom-logger-v2): add the rust contract for native callbacks and bind the proxy request scope * fix(custom-logger-v2): add field projections and keep the request scope across copies * fix(custom-logger-v2): pass the request scope to the async failure attempt_ended line * fix(custom-logger-v2): add v2 callbacks from litellm.callbacks to the success and failure lists _safe_add_callback_to_list dropped CustomLoggerV2 instances, so a V2 callback registered through litellm.callbacks got provider events but never log_prepare or attempt_ended. The manager now adds each V2 instance once by identity, and proxy startup fans V2 callbacks out like CustomLogger ones. * fix(custom-logger-v2): contain a failing v2 callback on logging hooks A V2 log_prepare writer that raised, or returned Abort or Respond, ended the line and escaped async_success_handler, so attempt_ended delivery, sync-for-async scheduling and the Redis flush were skipped. On hooks off the request path a V2 entry's exception or outcome is now counted and logged like a legacy logger's exception, and the line goes on. ContractViolation and BaseException still propagate, and legacy entries are unchanged. * fix(custom-logger-v2): run blocking and awaited v2 callbacks on synchronous logging lines run_plan_sync raised ContractViolation for a V2 log_prepare writer or any awaited V2 subscriber, the logging plane swallowed it, and every legacy logger at that hook was skipped. Legacy entries now always run in order as before. A V2 writer or awaited subscriber runs inline in list order through asyncio.run when the thread has no running event loop, so sync SDK calls apply its patch and finish it before the handler returns. On a thread with a running loop it cannot block, so it is delivered off-path and its patch is not applied. Lines without such entries take the unchanged legacy path. * fix(custom-logger-v2): map a v2 abort to its http error at every proxy hook point Only pre_call turned a V2 Abort into an HTTP error. during_call, post_call_success, post_mcp_call, list_models and the deferred stream hook raised V2Outcome, which reached the client as a 500, and post_call_failure, which never raised before, raised it too. Abort now raises the exception it carries or an HTTPException with its status, message and headers at those points, and at post_call_failure it becomes the transformed error through keep_first. Respond is still not wired and keeps raising V2Outcome. The pre_call routing test is updated to the HTTPException raise_v2_abort already makes. * fix(custom-logger-v2): apply no-log to v2 logging subscribers without a request scope The proxy request scope already carries no-log and caller opt-outs, so a V2 callback that honors no-log is skipped on proxy requests. SDK calls have no scope, and the logging plane ran V2 subscribers with an empty selection, so a no-log call was still delivered to them. Without a bound scope the logging lines now open one with no_log set when the call's litellm_params carry no-log, unless global_disable_no_log_param is on. Callbacks that do not honor no-log, and so every writer or abortable, are unaffected. * fix(custom-logger-v2): deliver a failed attempt to v2 subscribers once per logging object Callers such as MCP tool errors and Responses streaming run both failure handlers without for_async_request, so a V2 attempt_ended subscriber, now present in both failure lists, got the failure twice. The first failure handler to include V2 subscribers claims the delivery on the Logging object and the other skips them. Legacy callbacks are unchanged. * refactor(custom-logger-v2): settle a synchronous v2 run through the shared gather and use aware times in the new tests Drops the blind except the sync line used to capture a V2 callback's exception, reusing _gather_started, and gives the new logging tests timezone-aware times, so the strict ruff counts do not grow. * fix(custom_logger_v2): await a single-entry v2 stage in the caller's context A native handler in its own stage ran inside an asyncio.gather child task, so a ContextVar it set was lost when the stage ended, while a legacy callback's stays visible to later callbacks and the rest of the request. A stage with one v2 entry is now awaited directly, with gather's exception handling kept. Multi-entry parallel stages still gather. * fix(custom-logger-v2): raise a native callback panic as a runtime error * fix(custom-logger-v2): run a synchronous v2 line on a private event loop * fix(custom-logger-v2): deliver an mcp tool failure to v2 subscribers from the async failure handler * fix(custom-logger-v2): raise an abort's carried exception at the deployment hooks and keep the original failure at post_deployment_failure * fix(custom-logger-v2): apply caller opt-outs to v2 logging lines without a bound request scope * fix(custom-logger-v2): dedupe v2 callbacks in the event lists by logger key so a native built twice is delivered once * fix(custom-logger-v2): log an unregistered abort error kind and keep the carried traceback at post_call_failure * fix(callbacks): reject a selection name or alias that is not lowercase * fix(custom-logger-v2): run v2 deployment callbacks on rust bridge deployment hooks * fix(router): type the fallback event kwargs handed to the v2 dispatch * feat(custom_logger_v2): type the contract fields as records and mark derived-only fields Fields that the host derives from a hook's arguments now carry pydantic records on the Python side and matching serde records in litellm-callbacks, with records.json as the shared sample both sides check. Twelve new fields take bits 30 to 41, a write to a derived-only field is a manifest error, and pre_call can write budget_reservation and rate_limit_statuses in place. The payload reference field and the PayloadHost trait give Rust a handle to the logging payload instead of a copy * feat(custom_logger_v2): derive typed fields from hook arguments and always build call facts A slot table maps each typed field to a function over the hook's arguments. A derived slot runs only when a subscriber reads it, once per hook invocation, and off-path readers settle their reads before the hook returns. Auth fields come from the auth object only, budget_plan is claimed once, and budget_reservation and rate_limit_statuses write through to their carriers so the next pre_call subscriber sees them. Call facts are now built on unscoped logging lines and at post_call_success, post_call_failure and response_headers * feat(callbacks-python): hand the logging payload to native callbacks by reference NativeCallback stashes the shallow payload copy it is given and passes Rust a PayloadRef. The Python host renders a handle with a safe_dumps compatible encoder, on the caller's event loop while the hook runs and on the blocking pool after it, and dropping the last reference releases the stash entry. The encoder and its JSON helpers move here from the S3 logging bridge so every port shares one copy * feat(custom_logger_v2): accept every value python accepts for typed members and serve logged key identity Typed members keep the bool, int, float or str python holds through a Scalar type on both sides, so ids, limits and priorities no longer lose values that lax pydantic refused. A member the contract cannot carry is named in rejected so a port can reproduce what python does with it. Logging hooks get logged_key_identity, built from the payload metadata the python loggers read on every route, while key_identity stays on request hooks and keeps reading the auth object. * fix(custom_logger_v2): read the live budget reservation dict and carry non-finite floats both ways budget_reservation read the dict budget_reservation.py builds with a KeyError on callback_bound. Undeclared keys are now ignored, callback_bound is a member, entries keep a non-mapping item as null so capacity's model access group check stays exact, and costs read as the float budgets computes with float(x or 0.0). Float members cross as the raw json marker when they are not finite, and a native verdict returns through json_to_py, so an infinite or NaN cost survives a read and a write instead of turning into null. * feat(callbacks): carry a python value as its str() text and truthiness in scalar members Scalar gains a Stringified variant with the str() text and bool() of a value that is not a bool, int within i128, float or str, so a port can reproduce the label, path or log text python formats and the falsiness of an empty container without confusing it with a real string. Budget reservations say whether python held the entries as a list. * feat(custom_logger_v2): carry non-scalar values and any iterable tags or entries across the boundary A Scalar member now carries a list, dict, set, bytes, object or an int past i128 as its str() text and truthiness instead of naming it in rejected, which stays for values whose str() or bool() raises. request_tags carries any iterable as the items prometheus iterates, and budget reservation entries carry what budgets iterates with entries_is_list for capacity's list check, never written back to the shared dict. * feat(custom_logger_v2): say whether a stringified value hashes and whether it is an int A Stringified scalar now carries hashable, from trying hash() at the boundary, so a port can raise the TypeError capacity's priority lookup and the limiter's descriptor identities raise for a list, dict or set and not for a tuple or object. It also carries integer, so an int past i128 is told apart from any other value whose text is digits. * fix(custom_logger_v2): wrap a non-list guardrail mode as one mode like prometheus does Prometheus treats any guardrail_mode that is not a list as a single mode, so a tuple mode holds no str mode and a missing mode is one None mode. The derivation unpacked tuples and rejected other non-str modes instead. * feat(callbacks): let a native handler await typed host calls the python caller serves * test(callbacks-python): drive a native handler's host calls through the python dispatcher * test(custom_logger_v2): route a native logger with a host through the execution driver * fix(custom_logger_v2): type the wrapped guardrail modes for pyright * feat(custom_logger_v2): carry the python type name and hash error text of a stringified value * feat(custom_logger_v2): serve the model group the logging handlers compute and key remaining limits by it * feat(custom_logger_v2): tell a budget reservation that is not a mapping apart from none * feat(custom_logger_v2): name malformed team model preferences in rejected with the text capacity logs * feat(custom_logger_v2): serve a logging payload that is not a mapping apart from a missing one * fix(callbacks-python): render null keys like a python dict merge, adding a missing key last * feat(custom_logger_v2): carry an explicit output cap past i64 as a scalar * feat(custom_logger_v2): serve the custom metadata labels prometheus derives from the logged payload * feat(custom_logger_v2): serve the input sequence length bucket prometheus picks at attempt end * feat(custom_logger_v2): carry the start and end times prometheus reads from the call details * feat(custom_logger_v2): serve the limiter's body tags, session id, resolved agent and agent policy limits at pre_call * fix(callbacks): export the limiter request record * test(custom_logger_v2): give the type name cases stable ids and sort the record exports * fix(custom_logger_v2): type the new derivations and tests for pyright * fix(custom_logger_v2): drop team model preferences and serve the fallback_on_saturation handoff main's capacity callback reads * style(callbacks): rustfmt the record exports * refactor(custom_logger_v2): rename native callbacks to rust callbacks native.py becomes rust.py, with native_logger, NativeLogger, NativeHandle, NativeState and HostedNative renamed to their rust forms. The pyo3 classes NativeCallback and NativeRequestState become RustCallback and RustRequestState, and the tests, fixtures and messages follow. * refactor(custom_logger_v2): rename slots to field sources slots.py becomes field_sources.py: Slot is FieldSource, SlotFields is SourcedFields, and with_slots, NO_SLOTS and slot_table_problem follow. A table that only derives is named DERIVED_*_FIELDS, and a table that can write back keeps SOURCES (PRE_CALL_AUTH_SOURCES, PRE_CALL_SOURCES). The include_v2 comment moves down to the functions that take it. * refactor(litellm_logging): place the v2 members of Logging beside what they serve v2_scope, v2_auth and v2_failure_delivered join the other per-request class defaults, v2_logging_scope and its opt-out helper follow should_run_callback, v2_call_facts sits by the standard logging payload builder, and claim_v2_failure sits above failure_handler. bind_v2_request_scope checks for a Logging object directly instead of two runtime protocols. * refactor(proxy): rename v2_fields to v2_proxy_fields and say why it is separate * refactor(custom_logger_v2): drop the logged_payload_not_a_mapping field A logged payload that is present but not a mapping now looks the same to a rust callback as a missing one, a parity gap accepted for a case that does not occur. metadata_labels, input_sequence_length and limiter_request move down to bits 44-46, on both sides of the contract. * feat(callbacks): type a rust callback's request state Callback gains an associated State type: request_state returns it and handle and handle_hosted receive Option<&State> beside the input, which no longer carries the state. ErasedCallback is the object-safe form the python wrapper stores; it carries the state type-erased and downcasts it once, returning CallbackError::StateType rather than panicking on a mismatch. handle_hosted's default names its unused host _host. The record sample count follows the dropped logged_payload_not_a_mapping field. * docs(callbacks): say what PayloadRef, PayloadHost and RenderOptions.null_keys are * refactor(callbacks-python): render payloads with python-compat's json writers pyjson.rs reimplemented py_encode_basestring_ascii and json's floatstr. python-compat's json module now exposes write_string, generic over a String or byte sink, and float_text, and the payload renderer uses them. * docs(callbacks): note why Respond, Reroute, call_ended and flush are not wired yet * docs(custom_logger_v2): add a README for the package * fix(custom_logger_v2): import the field sources tests from field_sources * test(callbacks-python): stub the litellm package in each payload test * fix(custom_logger_v2): type the disabled callbacks lookup and the v2 dedupe check |
||
|
|
f5ef14308b
|
fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved (#50516)
* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved Converse used to drop any service_tier outside default, flex, and priority with no error, rewrite auto to default, and drop reserved. Now reserved and standard_only map to Bedrock tiers, auto sends no tier, and any other value answers 400 unless drop_params is on. Nova batch rows honor the row's drop_params. * fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows * test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs * fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params * fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params * fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase * fix(bedrock): mark the service tier block field read-only * fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params * test(prometheus): sort the service tier label test imports * fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400 * fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova * fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one * test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it * refactor(bedrock): drop the passthrough's own service tier header reader that main re-added The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
20b6e855bd
|
feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0) (#51214) | ||
|
|
46716fad56
|
fix(proxy): alert on customer org, customer team, and customer member budgets (#51579)
* fix(proxy): alert on customer org, customer team, and customer member budgets Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): scope customer alert dispatch and Redis tracing Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(alerts): escape customer identities in Slack budgets Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): dispatch customer budget events through custom loggers Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): read committed customer budget spend from writer Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): retain customer budget claims for the full period Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> |
||
|
|
e2dfd3dad1
|
fix(langfuse): reserve span capacity for failure usage attributes (#51538)
* fix(langfuse): reserve span capacity for failure usage attributes * fix(langfuse): type pending failure attribute keys --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
2be96aed13
|
refactor(redis): delete the Python RedisBatch now that every request op is native (#51383)
The request scope keeps only the native batches. RedisBatch, its ops, the cluster fallback, the Python post-call deadline and the DualCache post-call helpers had no production caller left, and the pre-call SET, DEL and miss tracking that UserApiKeyCache never reached go with them. The mixed-families pipeline metadata had no producer left |
||
|
|
9633be87ea
|
feat(ui): show customer organization usage and budget utilization (#51483)
* feat(ui): show customer organization usage by date range * feat(customers): align usage reads with daily spend contract * fix(customers): combine spend display and resolve CI failures * test(vector-stores): isolate search warnings from SDK verbosity * fix(ui): cap customer usage date ranges at 400 days |
||
|
|
df8b3b498e
|
feat(proxy): add customer hierarchy usage analytics (#51495) | ||
|
|
a7052ea8a8
|
fix(cost): preserve Bedrock cross-region pricing for Messages (#51479)
* fix(cost): bill Bedrock cross-region profiles at the profile rate on /v1/messages Bedrock reports the region-stripped model id for a cross-region inference profile, and the cost selector preferred that served id, so streaming and non-streaming Messages spend (input, output, cache write and cache read) used the global row. A served id carrying a different cross-region prefix keeps its own price. Fixes #46286 * test(langfuse): derive the Bedrock profile export cost from the profile's model info The regional us. deployment is now billed at its profile rate, so the expected Langfuse cost comes from that profile's rates and the mock usage instead of a hardcoded global price. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
7130a5b4fd
|
feat(dynamic_rate_limiter_v3): prometheus share, borrowed and throttle metrics per team and model (#51417)
* feat(dynamic_rate_limiter_v3): Prometheus share, borrowed and throttle metrics per team and model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): name the BLE001 suppressions on the metrics guards Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): count only final refusals, publish the served model's share, join the priority label Review round 1: a steered alternate's refusal no longer counts as a throttle when the preferred model then admits the request, a router fallback's settlement publishes the fallback target's own share under its model group, the gauges carry the same priority label the 429 carries, the gauges aggregate with livemostrecent so a new window replaces an old worker's value, the share read is skipped when no Prometheus logger is configured, the unit tests restore the process registry, and the integration test polls for the settled totals. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prometheus): type the gauge setter and the label factory so the type gate holds _set_labeled_gauge takes a _MetricLike and sets the series only when it is a real Gauge, prometheus_label_factory declares the dict[str, str | None] it returns, the priority resolver goes back to its original shape, and the label test tolerates the now visible None values. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(prometheus): keep the label test diff to the two None-tolerant assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
dc1284d3d2
|
fix(otel): distinguish missing usage from prompt estimates on chat failures (#51443)
* fix(otel): distinguish unknown timeout usage from prompt estimates * fix(otel): handle expected tokenizer failures and cover parent spans * fix(otel): validate failure usage inputs at typed boundaries * fix(otel): type Logging.messages for the failed prompt token estimate Declare Logging.messages as Sequence[object] | None at its single __init__ assignment and drop the list TypeAdapter, so the failure estimate reads a typed value without new unknown-type diagnostics. * fix(otel): skip the failure prompt estimate for prompts with attachments Only estimate the failed prompt when every message is plain text, so files, documents and images never reach PDF parsing or media decoding on the failure path. Those requests still report unknown provider usage without an estimate. * fix(otel): count the failed prompt off the event loop for async requests Async failure logging now awaits the existing offload_token_count limiter for the optional prompt estimate, and the sync half of an async request no longer counts inline, so a slow tokenizer load cannot stall other requests. * fix(otel): only count the failed prompt for loggers that receive failures The estimate check now reads general callbacks, the global failure lists through a typed get_failure_callbacks accessor, and the dynamic failure lists, so a logger registered only for success never triggers failure token counting. * fix(otel): return an immutable failure callback snapshot * fix(otel): keep malformed content blocks from breaking failure logging Only compare string block types against the text block table, and run the text-only check inside the estimate's narrow try, so a malformed prompt leaves usage unknown without an estimate instead of interrupting failure callbacks. * fix(otel): stop the token counter from debug-logging raw prompts The failure estimate counts the original prompt, so the counter's debug line could write prompt text to application logs while message logging was off. Drop that line and assert the prompt stays out of debug logs on both routes. * test(otel): pin text-only eligibility and failure callback categories Cover the public has_only_text_content contract for malformed and non-text messages, and check get_callbacks_by_type puts sync and async failure registrations in the failure categories through the public manager APIs. * fix(otel): keep failure logging when the prompt estimate cannot run Return no estimate when the offloaded count hits a worker or tokenizer RuntimeError or OSError, so failure callbacks still run, while cancellation still propagates. Treat SDK Message objects like their dict form when checking whether a prompt is text only. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
d723545320
|
fix(otel): report the provider model in experimental GenAI traces (#51430)
* fix(otel): report the provider's response model under gen_ai_latest_experimental With OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, gen_ai.response.model now uses the provider_response_model kept on the response hidden params, then the deployment model when the response only echoes the client alias. Legacy mode and litellm.model_group keep the alias. Fixes #51029 * fix(otel): type the semconv response model inputs and attach hidden params in tests Bind the request model once through as_str so the deployment fallback no longer reads untyped kwargs. Non-streamed fixtures now attach provider_response_model with set_hidden_params, since ModelResponse(hidden_params=...) does not keep it on the pydantic private attribute. * fix(otel): read the request model as object before narrowing it Binding kwargs.get("model") to a Final[object] stops basedpyright from reporting an unknown argument passed to as_str. * fix(otel): validate the untyped request model through a TypeAdapter boundary TypeAdapter(object).validate_python accepts the untyped kwargs value and returns a known object, the existing litellm/utils.py and router.py pattern, so as_str no longer receives an Unknown. * test(otel): stream the aliased response through a real Logging object Greptile flagged the MagicMock logging_obj as a double on our own code. The streamed fixture now builds a real Logging with a fixed start time, like test_streaming_handler.py. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
d2711ac559
|
feat(alerts): add management controls and opt-in daily summaries (#51418)
* feat(alerts): add alert controls and daily notification summaries * fix(alerts): reserve daily notifications before delivery |
||
|
|
22252073e7
|
feat(bedrock): search knowledge bases by image and return image hits (#51063)
* feat(bedrock): search knowledge bases by image and return image hits
vector_stores.search and asearch take a query_image (base64 jpeg, png, gif or webp), and query is optional when it is sent. Bedrock sends it as an IMAGE retrievalQuery, and a caller's own retrievalQuery in extra_body still wins
Bedrock IMAGE hits now come back as image content items instead of being dropped. The media type comes from the source file mime type metadata, then the data URI label, then application/octet-stream
A request with both query and query_image, with neither, or with a malformed image gets a 400. So does query_image on a provider without image search, unless drop_params is on. The proxy accepts query_image in place of query on /v1/vector_stores/{id}/search
Spend logs redact image data when prompts are not stored. Emulated file_search skips image-only hits, RAG rerank maps its indices to the text documents it ranked, and the dashboard tester renders image hits
* fix(vector-stores): keep image hits to image searches and reject a caller retrievalQuery
Text searches on a multimodal Bedrock knowledge base return the same text hits as before, so chat, RAG and logging payloads stay unchanged. Image hits come back only when the caller sent query_image
A caller retrievalQuery now answers 400 pointing at query_image, or is dropped under drop_params, instead of replacing the built query unchecked
A file_search chat tool that carries query_image or rewrite_query searches again instead of failing, and a non-string query next to an image answers 400 instead of 500. query_image is keyword-only on search and asearch
* fix(vector-stores): redact query images and keep store search params out of image checks
* fix(vector-stores): redact images nested in extra_body and keep a store's extra_body retrievalQuery out of searches
* fix(vector_stores): keep a caller's image query through store and router params
A deployment's retrievalQuery no longer reaches a routed search, a caller's query fields survive a store's own extra_body, and an extra_body image stops before the provider. file_search forwards rewrite_query, both retrieval query spellings are redacted, and the cost error stops echoing the response when message logging is off
* fix(cost): leave image search responses out of the cost error text
* fix(vector_stores): accept line-wrapped base64 in query_image data
Bedrock takes inlineContent wrapped with \n or \r\n line breaks, as plain base64 writes it, so the check ignores only those before decoding and the data goes to Bedrock as sent
* fix(vector_stores): drop rewrite_query false from the chat file_search tool
False asks for the default search every store already runs, and MongoDB
rejects the field outright, so forwarding it made a MongoDB chat lose the
store's context for a no-op flag. True is still forwarded.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
|
2c44f26288
|
chore: remove cookbook and preserve live assets (#50944) | ||
|
|
5e74515c29
|
feat(otel): let a team or key OTel v2 destination add its own request metadata keys (#50813)
* feat(otel): let a team or key OTel v2 destination add its own request metadata keys A team or key callback can now set baggage_metadata_keys in its callback_vars, a comma-separated list of requester_metadata.<key> paths. The tenant fan-out adds them as litellm.metadata.<key> to that destination's spans only, on top of the global LITELLM_OTEL_BAGGAGE_METADATA_KEYS; the operator's exporter and other destinations never see them. Values are bound when each span ends, an attribute the span already carries wins, and names of proxy identity fields are rejected when saved, along with lists over 20 keys. Resolves LIT-9545 * fix(otel): refuse request headers in a destination's baggage_metadata_keys The proxy copies the request headers into requester_metadata.headers with credential headers masked, but a header a gateway in front of it added (an IAP or OIDC assertion, an internal token) stays as sent. A team or key admin listing requester_metadata.headers.<name> could export it to an endpoint the team controls. Entries under requester_metadata.headers are now rejected when saved, so exporting a header stays an operator decision in LITELLM_OTEL_BAGGAGE_METADATA_KEYS. Also drops an unused import. * fix(otel): hold destination metadata to the configured span limits A destination's extra litellm.metadata.* attributes were added to a span view built from a plain mapping, so they skipped the limits the SDK applies in set_attribute: a 1 MiB caller value survived an operator's OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT. Values are now cut to the length limit, and keys past the attribute-count limit are dropped and counted instead of evicting an attribute the span already carries. The per-destination views are built in one Final tuple ahead of the forwarding loop, since a Final cannot be assigned inside the loop. * fix(otel): apply destination metadata after the destination view, within the provider's limits The span limits were checked against the span as it ended, before the destination view strips prompt content and backend-only attributes, so a model-call span full of llm.input_messages.* could drop the team's keys on a no_content destination that had room. The values are now bound when the span ends and added as the last step before export, counted against what that destination actually sends, so their dropped count also survives the view. The limits come from the tracer provider the fan-out is attached to, read from the live span at start, instead of a separate env-built SpanLimits. * refactor(otel): consolidate span limits, destination validators and metadata key parsing Reads a span's limits in one guarded place that the emitter and the fan-out share, replaces the two parallel lists of per-destination callback-var validators with one mapping, splits and strips a baggage_metadata_keys list once, drops a filter the post-view merge already enforces, and folds the repeated fan-out test setup into one helper. No behavior change. * ci: retrigger checks stranded by the GitHub Actions outage |
||
|
|
11af2eb22d
|
refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822)
* refactor(rust_bridge): route every Rust-to-Python call through host modules Rust now reaches Python only through litellm/rust_bridge/host/ (plus the execution driver), and no Python module outside host/ imports it. Library calls that reproduce Python behavior (httpx and OSError error text, huggingface_hub downloads, the random generator, os.environ) go through host functions too. The rule is written into host/AGENTS.md The live litellm.model_cost table, its version and every Python writer move to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only the functions Rust calls. The cost map fetch passes use_http_settings instead of handing Rust an http_settings() snapshot * refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust The Redis bridge asked Python for all of os.environ to read one key. CPython's os.environ writes through to the process environment, so Rust reads it with ProcessEnvironment like the rest of the module. Also drops doc comments that restated their code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(rust): raise the native extension size cap to 55 MB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2c9068dbab
|
fix(guardrails/rubrik): moderate text-completion prompts and forward rac_agent_id (#49912)
* fix(guardrails/rubrik): moderate text-completion prompts and forward rac_agent_id _moderate_prompt synthesized a user message from texts for /v1/completions requests but built the payload from structured_messages, so the webhook received an empty messages list and the prompt went unmoderated. Pass the resolved messages into _build_prompt_moderation_payload. Forward metadata.rac_agent_id to the before_prompt and after_completion webhooks so agent-scoped policies apply. Only that key is forwarded; the rest of request_data metadata carries non-serializable objects and inbound headers * fix(guardrails/rubrik): also read rac_agent_id from litellm_metadata /v1/messages, /v1/responses and other LITELLM_METADATA_ROUTES keep proxy metadata under litellm_metadata, so a caller on those routes may set rac_agent_id there. Read metadata first, then litellm_metadata, forwarding only a non-empty string * fix(guardrails/rubrik): take rac_agent_id from key or team metadata only The Rubrik webhooks use rac_agent_id to choose which agent-scoped policies apply. Forwarding it from the request body let a caller pick any agent's policies, including one a policy exempts. Read it from the caller's admin-configured key metadata, falling back to team metadata, via CustomGuardrail._get_admin_metadata. The proxy writes those from the authenticated key and strips client-supplied user_api_key_* fields from both metadata dicts, so a caller cannot set or override the value --------- Co-authored-by: yucheng-berri <yucheng@berri.ai> |
||
|
|
bc3a69ce35
|
fix(otel): serialize pydantic Message objects in langfuse_otel, weave_otel and langtrace (#46845)
langfuse_otel, weave_otel and langtrace now encode litellm.Message objects, and any value JSON cannot encode, when they write the prompt, so spans keep their token usage and model parameters. Under LITELLM_OTEL_V2 the payload builder dumps Message objects instead of dropping them |
||
|
|
9fb29c7e38
|
fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions (#50467)
* fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions
An allowed_tools tool_choice in the flat Responses layout ({"type": "allowed_tools", "mode": ..., "tools": [...]})
answered 400 on litellm.completion, acompletion, Router and the proxy's chat and /v1/responses-to-chat routes. One
parser now reads both layouts into the nested Chat layout, and the existing Responses bridge flattens it back for
/v1/responses, so either layout reaches OpenAI in the shape each endpoint expects
* fix(tool_choice): drop an allowed_tools choice under drop_params and refuse conflicting flat entries
An allowed_tools tool_choice a provider can't enforce, or a malformed one, now drops under drop_params and answers 400 without it, the same rule LiteLLM uses for other unsupported values. That covers providers outside OpenAI and Azure, malformed choices on every provider, and Azure api versions older than 2023-12-01
A flat allowed_tools entry that carries both name and a nested function or custom key answers 400, since the two names can disagree, matching what flattening already refuses
* fix(tool-choice): keep the 400 when LiteLLM runs the tools, report auto after a drop
Under drop_params an allowed_tools choice a provider can't enforce was dropped, so on MCP gateway tools and in
run_tool_loop and arun_tool_loop LiteLLM could run a tool the allow list excluded. Those paths now keep the 400 they
answered before the flat layout was accepted, and a malformed choice stays a 400 on MCP gateway tools.
/v1/responses served through chat echoed a choice it had dropped, on sync, async and streamed responses. It now
reports auto, and a choice the provider received is still echoed as sent.
* fix(tool-choice): refuse an excluded tool right before LiteLLM runs it
Checking the provider up front missed a model alias, an old Azure api_version,
a litellm_proxy hop, and SDK fallbacks. The tool loop and the MCP gateway now
check the allowed_tools list right before they run a tool call, so an excluded
call answers 400 and never runs.
Under drop_params the choice now drops only when the caller runs every tool,
so a provider-run MCP server or built-in keeps the 400. The Azure checks read
the request's api_version, which also fixes the /v1/responses echo there.
* fix(tool-choice): drop the choice under drop_params and hand an excluded MCP call back unrun
The drop now applies to every provider that cannot take an allowed_tools
tool_choice, whatever the request's tools hold. LiteLLM never runs a call the
choice excludes: on MCP paths the whole round goes back to the caller in a 200,
the same shape as require_approval, and the tool loop keeps its 400 because it
only returns final content. A malformed choice stays a 400 before any request
on the tool loop and on chat with MCP gateway tools.
* fix(tool-choice): refuse an allowed_tools entry whose name is not a string
An mcp or other non-function entry with a list or dict name passed the
parser and crashed when the allow list was hashed. The parser now treats
it as malformed on both layouts, so it answers the same 400 as an empty
allow list, or drops under drop_params.
* fix(tool-choice): honor server-level and null-name mcp entries and stop hooks running excluded calls
* fix(tool-choice): match named mcp entries by server label and let built-in entries allow their interception call
* fix(tool-choice): leave a malformed allowed_tools choice unrewritten in the interception hooks
* fix(tool-choice): resolve mcp entries like the gateway and keep the choice on every hook path
Under drop_params a malformed allowed_tools choice now acts as if it was never sent, and only a choice that parses is stored and enforced. Without drop_params it answers 400 before any provider request, on /v1/responses with gateway tools before MCP listing
An mcp entry resolves its label the way the gateway picks servers to list (server name in any case, alias, access group, toolset), and a named entry matches the tool's own name or the gateway's prefixed name. The gateway rewrites mcp entries into function entries in the caller's layout and sends "none" when nothing is left
The web search follow-up keeps an allowed_tools choice, compression's litellm_content_retrieve is never excluded and joins the provider-facing choice, and the native /v1/responses hook loop applies the exclusion check and stores the choice
* fix(allowed_tools): stop web search and code interpreter follow-ups forcing another call
A kept allowed_tools choice now goes to the follow-up in mode auto, so a
required choice no longer loops to the agentic loop limit. The MCP label
scope dataclass uses a default_factory so the module imports on Python 3.11
* fix(allowed_tools): stop the Responses web search follow-up forcing another call
The Responses-shaped follow-up after a web search left tool_choice out of its patch, so the caller's required choice went out again through the chat to Responses bridge and looped to the loop limit. It now carries a kept allowed_tools choice in mode auto, like the chat follow-up
* fix(mcp): scope each MCP entry to its own allowed_tools and keep the allowlist on follow-ups
A label now covers only the tools its own allowed_tools lists, so a second label on the same server no longer widens it. Responses MCP follow-ups, streamed or not, keep an allowed_tools choice in auto mode instead of dropping it. The new Responses tests fake only the upstream MCP listing, so LiteLLM's own discovery runs
* fix(mcp): match an entry's allowed_tools against the tool name the gateway lists
A server with a display name override lists the display name, and the listing
filter matches allowed_tools against it, but the label scope only checked the
original name, so no spelling worked. The scope now also accepts the listed
name and its unprefixed part, like the listing filter. Tests now cover the
whole gateway and toolset entries and a display-named tool.
* fix(mcp): match an entry's allowed_tools only against a server's own prefix, not the first hyphen
* fix(mcp): match a tool no server resolves only by its exact listed name in an entry's allowed_tools
* fix(mcp): refuse a tool whose display name only looks like an allowed name, and let the chat follow-up answer
An MCP entry's allowed_tools now matches a tool only by its exact listed name or a spelling routing
accepts for its real name, so a display name that carries the server's own prefix no longer passes for
another tool. The chat MCP follow-up keeps an allowed_tools choice in auto mode, so the model can answer
with text after the tool ran.
* fix(mcp): match a toolset label's tools by exact native name
A toolset label served any tool whose name a routing spelling of a granted name matched, so a toolset granting the literal name "server-tool" also served the real tool "tool" that only another toolset grants. It now serves a tool only when the toolset grants its exact native name, the same check tool_is_granted uses when the gateway lists tools
|
||
|
|
68e78bc53a
|
fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted (#50730)
* fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted * fix(responses): forward and log a finished websocket turn when its early count fails or is cancelled A usage payload the parser rejects no longer stops the relay before the terminal event reaches the client, and a client that leaves while the turn is being counted still gets the turn logged. A cancelled early count now clears its counted-tokens marker so the logging callback counts the turn in full. * fix(responses): finish a turn's early token count when the client leaves A client that left while a finished turn was being counted cancelled the count halfway. The router's count could be written with no marker, so the logging callback counted the turn a second time. The count now runs to the end before the turn is logged, and the failure path goes back to clearing only on errors * fix(router): count one router request per Responses WebSocket turn A socket whose finished turns each counted their own request in router usage no longer adds one more request when it closes, so N turns count N requests, as N HTTP calls do. A socket with no counted turn still counts 1 * fix(proxy): keep router usage markers out of pass-through client metadata and ClickHouse rows --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
523b3efea2
|
refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501)
* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
abe3b11f7a
|
fix(proxy): show committed spend in the remaining budget gauges (#49199)
* fix(proxy): report live key and team spend to budget gauges within the key cache TTL * fix(prometheus): carry live spend for the user, org and customer budget gauges too The remaining budget gauges for users, orgs and customers read the same cached rows as the key and team gauges, so they read high for about a minute after a charge. Auth now carries the live counter for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded. The x-litellm-key-spend header and logging spend fields keep their current values. * fix(prometheus): show committed spend in the remaining budget gauges Reads twin counters that take only charged cost, so a worst-case budget hold never shows as spent and a request that finishes last cannot set a gauge back. The twins exist only while the Prometheus logger is registered, and every path that resets a spend counter drops them. Failed requests now set the org gauge from the carried org row and committed spend, not the cached org row. * fix(prometheus): count a charge the cached customer row already holds once * fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets Auth now reads committed spend through the admission MGET and names the counters a charge may move, so the master key and unbudgeted entities get no twin. Organization spend edits and the global spend reset drop their twins, and a long request no longer reads as charged. * fix(prometheus): read the committed org spend on failed requests without the auth object * test(prometheus): cover a failed request with no carried org * fix(prometheus): move the committed counters for charges with no auth on the row A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them. * fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys and teams in bounded pages. * fix(prometheus): count a charged failure's own cost in the org gauge A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request * fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog * fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins * fix(prometheus): keep committed spend values out of logger dumps of pass-through rows CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values * fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files * fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
b83a1f0608
|
fix(otel): name the routed model on auto-router LLM spans (#50498)
On an auto-router request the LLM span reported the name the caller asked for (the router or an alias of it) as gen_ai.request.model (OTel v2) and gen_ai.response.model (v1 and v2). Both now name the model group the router picked, read from the request's routing_decision. In v2 the span name and the vendor keys built from the request model (Langfuse, OpenInference, Langtrace, Weave) follow. Requests that were not auto-routed are unchanged. |
||
|
|
e3387b6fc5
|
fix(otel): seed request metadata baggage for team-only OTel v2 destinations (#50373)
* fix(otel): seed request metadata baggage for team-only OTel v2 destinations A team newrelic (or other preset) destination exported spans without the operator-allowlisted litellm.metadata.* keys unless a proxy-wide otel callback was listed, because only the OTel logger's async_pre_call_hook seeded requester_metadata into Baggage and that hook runs only for loggers in litellm.callbacks. ProxyLogging.pre_call_hook now re-seeds identity with the request data once requester_metadata exists. Resolves LIT-9783 * fix(otel): seed only request metadata from the proxy pre-call hook The pre-call re-seed also passed the request model, which by then can be an alias target, mcp-tool-call or an internal embedding model, so it rewrote gen_ai.request.model on the server span for V2 deployments without otel listed. Seed only the request metadata and keep the model auth seeded. Adds a unit test for the pre-call seed and the guardrails_only skip, drops the explanatory comments, and requires one server span plus the model span in the integration test. * test(otel): annotate new test variables Final and build test config without mutation Addresses review: the new unit and integration tests now follow AGENTS.md (LIT010 Final on every variable, no in-place mutation, read-only parameter types, one for-clause per comprehension). * test(otel): cover key-level callbacks and assert tenant spans carry no credentials Parametrizes the tenant metadata test over team and key callbacks, and adds a test that sends credential headers and a planted New Relic key with an allowlist naming credential headers, then asserts no exported span carries the master key, the virtual key, the team's New Relic key or any credential header value, and that every export used only the team's own New Relic key. |
||
|
|
2b4485dee5
|
fix(guardrails): check a model-level post_call response once across the proxy and deployment layers (#50300)
#40154 made the deployment post_call hook really run apply_guardrail guardrails, so the proxy checked the same response a second time (LIT-10481). The deployment hook now records guardrail + response hash on the request and on the response, and the proxy skips a guardrail only on a matching record. The hash covers every field except ids, usage, model and other metadata set between layers, so a response rewritten between the layers (OCR included) is still checked. Guardrails with their own post_call hook keep their pre-#40154 behavior Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3f38730d2c
|
refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986)
* refactor(typing): clear basedpyright Any errors (wave 1) * refactor(typing): clear basedpyright Any errors (wave 2) * refactor(typing): clear basedpyright Any errors (wave 3) * refactor(typing): clear basedpyright Any errors (wave 4) * refactor(typing): clear basedpyright Any errors (wave 5) * refactor(typing): clear basedpyright Any errors (wave 6) * fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves * fix(logging): keep callbacks running when litellm_params is a params model should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request Adds behavior tests for the changed lines codecov listed as not run * fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call. * fix: bill regional uplift multipliers float() accepts, bytes included * fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type |
||
|
|
febbe38326
|
fix(anthropic)!: do not surface server_tool_use as client tool_calls (#48385)
* fix(anthropic): do not surface server_tool_use as client tool_calls Anthropic server-executed tools (web search, tool search, code execution) were returned in chat completions tool_calls with srvtoolu_ ids, so OpenAI-protocol agent clients treated them as pending work and re-invoked the model until their recursion limit. Move them to provider_specific_fields.server_tool_calls on both the non-streaming and streaming paths. History replay, the /v1/responses bridge and the /v1/messages streaming guardrail read that field, so multi-turn reconstruction, web_search_call output items and the guardrail's fail-closed check on server tool blocks keep working. * fix(responses): return Claude web_fetch calls as web_search_call open_page items * fix(responses): keep Claude server web calls without a result as hosted items A non-streamed Claude turn that calls web_search or web_fetch next to a client tool can return the server call before its result. Build an in_progress web_search_call for it, so the Responses bridge no longer returns it as a client function_call * fix(anthropic): keep pending server calls native and hide server calls from client tool paths Replaying a server call that has no result yet now sends it back as server_tool_use, so Anthropic accepts the next turn. The tool permission guardrail checks server calls again and rewrite mode removes a denied one. The /v1/messages adapter keeps Claude's server blocks, and the Responses bridge never returns a server call as a client function_call. * fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls, so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own server_tool_use instead of running again * fix(anthropic): let guardrails and loggers read Claude's server tool calls again Guardrails, tool policies, tracing loggers and the Responses guardrail translation read only client tool calls after server calls moved to provider_specific_fields. They now read server calls first, then client calls, through one shared helper, and the Responses guardrail treats Claude's web_search_call items as web_search or web_fetch calls. Non-streaming splits server calls by block type, as streaming does. * fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy, streamed or not. Redaction with message logging off hides server call arguments and hosted web call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls. * fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use. * fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy. * test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions * test(guardrails): type the fake Pillar client like its Protocol * fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or web_search still needs that call and its thinking replayed, or Claude loses the fetch result * fix(guardrails): scan replayed Claude server calls and hosted web calls Request guardrails now see and rewrite server_tool_calls replayed in chat messages. Cato and Purview scan Claude web_search and web_fetch calls on Responses output, and a function tool call with an extra custom key is no longer dropped from Cisco, Purview, Repello and PANW scans. * fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks * fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows * test: annotate the new server tool call test locals with Final and type their parameters * test: type the new guardrail test helpers with litellm's tool call types instead of Any * test: narrow the remaining new test helper parameters from bare dict and object * fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls * fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera * fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
da2ecbc5bb
|
feat: email affected owners before model deprecations (#50172)
* feat: opt-in model deprecation emails for affected owners * fix: annotate deprecation sender initialization * fix: clarify deprecation emails and use consistent settings toggles * fix: exclude former team admins from deprecation emails * feat: clarify provider deprecations and model migration steps |
||
|
|
784754ca48
|
fix(azure)!: answer 401 when Entra rejects the deployment's credential (#49863)
* fix(azure): answer 401 when Entra rejects the deployment's credential A ClientAuthenticationError from the Entra token provider (wrong, expired or rotated client secret) used to surface as a retryable 500 carrying the app's client id and Entra trace ids. Map it to a 401 AuthenticationError that keeps the AADSTS code and first sentence with GUIDs redacted, on both the direct token fetch and the provider handed to the OpenAI SDK client. Network failures stay 500. * fix(azure): keep network failures to Entra a retryable 500 azure-identity wraps a transport failure in ClientAuthenticationError, so the 401 mapping now skips errors whose cause chain holds ServiceRequestError or ServiceResponseError * fix(azure): cover async token providers and Entra 5xx, and scope caller-sent Entra credentials * fix(azure): map only AADSTS-coded Entra rejections to 401 and keep caller credential failures out of cooldown An Entra ClientAuthenticationError now answers 401 only when its message carries an AADSTS code, its HTTP status is not 429 or 5xx, and the code is not the AADSTS50196 throttle, so a chained credential's outage or throttle stays a retryable error. Caller-sent Entra fields no longer change the deployment id; a 401 from a value the caller sent skips the shared deployment's cooldown instead. The Azure AI Foundry agent token fetch maps the same rejection, and the A2A endpoint answers it with HTTP 401. * fix(router): count router default_litellm_params as configured Entra credentials when deciding cooldown * fix(azure): keep transient Entra codes retryable and scope 401 handling to Entra rejections ClientSecretCredential drops Entra's HTTP status, so throttling and transient codes (AADSTS50033, 50196, 90006, 90024, 90033, 90055) now stay retryable instead of mapping to 401. The A2A endpoint answers 401 only for an AuthenticationError caused by an Entra token rejection, and the router skips cooldown for a caller-sent Entra field only on Azure-family deployments. The azure/ Decisions route and the Bing grounding search tool now map an environment-credential rejection to the same 401. * fix(azure): map only known Entra credential rejections to 401 and cover the OIDC token exchange Unlisted AADSTS codes keep the retryable 500, the router skips the 401 cooldown only when an Entra rejection caused it, and a federated token Entra rejects answers the same 401 on Azure chat, Responses and A2A agents * fix(azure): answer 401 for Entra rejections on the /azure passthrough and token counting The fallback hop skips cooldown when Entra rejected a credential the caller sent, Prometheus counts a deployment's own Entra rejection as a deployment failure again, and an OIDC exchange error keeps Entra's headers so Retry-After reaches the router. * fix(azure): keep a no-op caller Entra field from skipping cooldown and redact the passthrough 401 message A caller-sent Entra field now skips the deployment's cooldown only when it differs from the deployment, its named credential, the router defaults, the AZURE_* environment variables and the default scope. An empty value counts as unset, so a deployment whose own secret is wrong still cools down when the caller's field changes nothing about the token request. The /azure passthrough's Entra 401 message goes through the same internal hostname redaction as other client-facing errors. * fix(router): compare a caller's Entra field with the value the deployment uses A caller-sent Entra field skipped the cooldown only when it matched no configured source, so a value equal to an environment default the deployment overrides (a stale AZURE_CLIENT_SECRET, say) counted as the deployment's own and let a caller's rejected credential cool down a working deployment. The check now resolves the value the deployment would use without the caller, in order from the router defaults, the deployment, its named credential, the AZURE_* environment variables and the default scope, and skips the cooldown only when the caller's value differs from it. * fix(azure): follow the settings merge for empty Entra fields and keep non-rejection OIDC exchange errors unchanged * fix(azure): count only the failed attempt's Entra rejection and ignore a non-callable token provider * fix(azure): keep the streaming Entra rejection inside the failed attempt --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |