The anthropic_messages route wraps client-side BadRequestError as
AnthropicError (a BaseLLMException subclass) with status_code=400, so
"except BadRequestError" missed those cells and they fell through to the
generic Exception arm, returning 500 instead of the expected 400.
Replace the isinstance-on-BadRequestError check with a tiny classifier
that prefers BadRequestError membership, then falls back to the exception's
status_code attribute (set by every BaseLLMException subclass), then 500.
Apply to both _call_chat and _call_messages for consistency.
Fixes the 13 CircleCI llm_translation_testing failures on
bedrock_invoke_messages cells where the effort was disabled / invalid /
empty / xhigh-on-unsupported / max-on-unsupported.
Two CI failures, both pre-existing in different ways:
1. reasoning_effort_grid: all 33 bedrock_invoke_messages cells failed with
AttributeError("module 'litellm' has no attribute 'messages'"). litellm
exposes the async Anthropic Messages entrypoint as litellm.anthropic_messages
(via "from .llms.anthropic.experimental_pass_through.messages.handler
import *" in litellm/__init__.py), not litellm.messages.acreate. Swap
the call.
2. tests/test_litellm/interactions/test_openapi_compliance.py::TestResponseCompliance::test_interaction_response_fields
asserts the live Google spec contains "steps". Google's spec has churned
through "outputs" -> "steps" -> neither, and presently carries neither.
The test broke on main as soon as upstream dropped "steps"; pulling the
key off the assert list realigns the test with the live schema. Re-add
the per-turn output field once upstream stabilizes on a name.
The openapi-compliance fix doesn't belong to this PR conceptually but is
included here per request to unblock CI before the morning.
The Content-Length header check in _process_image_response rejects the
image before the body is streamed, so the mock body never needs to be
materialized. Use an empty body instead of b"x" * 100MB (addresses
greptile/cursor review feedback).
Mock the image fetch instead of downloading a 50MB+ image from
upload.wikimedia.org. The runner was intermittently rate-limited
(HTTP 429), so the code raised "Unable to fetch image ... Status
code: 429" and the size-limit assertions failed even though
pytest.raises(litellm.ImageFetchError) still matched.
Mirror the established LargeImageClient pattern in
tests/test_litellm/litellm_core_utils/test_image_handling.py: stub
litellm.module_level_client with a response whose Content-Length
exceeds the 50MB limit and bypass SSRF validation, so the
size-limit rejection path is exercised deterministically with no
external network dependency.
test_completion_fireworks_ai and test_completion_cost_fireworks_ai
made real Fireworks calls and broke whenever Fireworks rotated its
serverless catalog (no externally-verifiable model list exists).
They also asserted nothing — just printed.
Mock the HTTP post and assert real behavior instead: the request is
built with the right model/messages and the OpenAI-compatible
response parses back; the cost path yields a non-zero cost against
the local cost map. No network, no model dependency, stronger than
the old smoke checks.
The live deepseek-v3p1 call kept hitting Fireworks NOT_FOUND because
Fireworks rotates its serverless catalog and no externally-verifiable
list exists. The [False] branch also never sent an image, so it only
proved the model responded.
Mock the HTTP post (mirrors test_global_disable_flag_with_transform_
messages_helper) and assert the real behavior: #transform=inline is
appended to the PDF URL unless disabled. No network, no model
dependency, and stronger coverage than the old live test.
Fireworks removed llama-v3p3-70b-instruct from serverless, so every
live test using it now fails with NotFoundError ("Model not found,
inaccessible, and/or not deployed").
Swap the 6 references (3 files) to the currently-served
accounts/fireworks/models/deepseek-v3p1 — the canonical model in
Fireworks' current docs examples and present in LiteLLM's cost map.
test_get_model_params_fireworks_ai is a pure pricing-heuristic test
(no network) asserting the >16b branch, so it uses llama-v3p1-70b-
instruct instead to keep the "fireworks-ai-above-16b" assertion and
branch coverage intact.
Google restructured the live spec at ai.google.dev/static/api/
interactions.openapi.json: the output-only fields (notably the
`steps` array, formerly `outputs`) moved off the request schema
`CreateModelInteractionParams` onto a dedicated `Interaction`
response schema. The response-side tests still read from the
request schema, so `test_interaction_response_fields` failed with
"Output field 'steps' not in spec".
Point `test_interaction_response_fields` and `test_status_enum_values`
at the `Interaction` schema (the semantic response object; all output
fields incl. `steps` present there). Request-side tests keep using
`CreateModelInteractionParams` (all request fields verified still
present). 13/13 pass against the current live spec.
- Drop the "v4" suffix throughout: it referred to the QA sweep iteration,
not this test suite. There's only one regression suite, so just call it
reasoning_effort_grid.
- Move tests/test_litellm/reasoning_effort_grid_v4/ -> tests/llm_translation/
reasoning_effort_grid/. Two reasons:
1. The parent tests/test_litellm/conftest.py installs an autouse fixture
(isolate_host_aws_config) that clears every AWS_* env var before each
test, which would silently skip every Bedrock cell.
2. tests/llm_translation/conftest.py already wires up the Redis-backed
VCR persister and auto-applies @pytest.mark.vcr to every collected
item via apply_vcr_auto_marker_to_items. Living under that conftest
means the suite gets cassette replay for free -- first CI run with
provider creds records 231 cassettes, every subsequent run replays
them with no live spend.
- Trim the suite's own conftest down to just the wire_capture fixture; the
inherited llm_translation conftest covers the VCR plumbing.
- Drop the dedicated reasoning_effort_grid_v4_e2e CircleCI job. The existing
llm_translation_testing job globs tests/llm_translation/**/test_*.py, so
the suite is gated by an existing job with no new wiring.
The runtime _validate_effort_for_model allows effort='max' for any
Claude 4.6 model (opus or sonnet), and model_prices_and_context_window
sets supports_max_reasoning_effort: true for claude-sonnet-4-6. The
grid spec previously gave sonnet-4-6 entries _CAPS_NONE, so expected()
returned status=400 for effort='max', which mismatched the runtime's
status=200 and caused 6 cells (one per route) to fail.
Rename _CAPS_OPUS_4_6 to _CAPS_4_6 (since the cap set is shared by
opus and sonnet 4.6) and assign it to all sonnet-4-6 entries.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- Remove unused vertex_credentials_path fixture (and now-unused os import)
from conftest.py.
- Parse Bedrock Converse complete_input_dict (logged as a JSON string by
converse_handler.py) before passing to _assert_cell, so dict accessors
work uniformly across routes.
- Extend _BUDGET_TOKENS with xhigh and max entries so the budget-mode
branch in expected() cannot KeyError if a future budget model gains
the matching cap.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Encode the 231-cell QA sweep (21 provider x model combos x 11 effort
values) from #27039 / #27074 as an automated CircleCI-gated regression
suite. Each cell hits the real provider endpoint, captures the outgoing
wire body via a pre-call CustomLogger, and asserts:
- thinking.type, output_config.effort, thinking.budget_tokens, max_tokens
in the captured request body (regression signal for silent drops/strips
in any provider transformation)
- HTTP status (200 vs BadRequestError -> 400) returned by litellm
(regression signal for clean-error vs leaked-500 mappings)
The matrix is encoded as a small rule set keyed by (model_mode, effort)
plus per-model xhigh/max capability overrides, then expanded across the
five chat-completion routes (Anthropic direct, Azure AI Foundry, Vertex
AI, Bedrock Converse, Bedrock Invoke /chat) and the Bedrock Invoke
/v1/messages route. Cells skip at runtime when the route's provider env
vars are absent, so PR builds without credentials no-op gracefully.
Wired into CircleCI as the reasoning_effort_grid_v4_e2e job behind the
existing main / litellm_* branch filter.
- Introduce `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` opt-in that switches OTEL traces to conform with the OpenTelemetry GenAI semantic conventions specification
- Extract all semconv behavior into a new `OTELGenAISemconvMixin` class in `gen_ai_semconv.py`, mixed into `OpenTelemetry` to keep concerns separated
- In semconv mode, span name follows `{operation} {model}` pattern (e.g. `chat gpt-4`) and span kind is set to `CLIENT` instead of legacy `litellm_request`
- Replace `gen_ai.system` with `gen_ai.provider.name` and drop `llm.is_streaming` in semconv mode; add `gen_ai.request.{frequency_penalty,presence_penalty,top_k,seed,stop_sequences,stream,choice.count}` and `gen_ai.usage.cache_{creation,read}.input_tokens` attributes
- Replace per-message `gen_ai.content.prompt` / per-choice `gen_ai.content.completion` log events with a single consolidated `gen_ai.client.inference.operation.details` event; omit `gen_ai.input/output.messages` when content capture is disabled
- Suppress the non-standard `raw_gen_ai_request` child span entirely in semconv mode
- Support both programmatic (`OpenTelemetryConfig.semconv_stability_opt_in` field) and environment variable activation; the two sources are unioned so either or both can enable the opt-in
- Extract OTEL SDK `LogRecord` / `SeverityNumber` version-compatibility shim into a reusable `_otel_log_types()` static method to deduplicate the `< 1.39.0` / `>= 1.39.0` import branching
- Add 30+ unit tests covering opt-in gating, span naming, attribute emission/omission rules, stop sequence normalization, cache token attributes, and the consolidated event lifecycle
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
test_keepalive_timeout_flag and test_timeout_worker_healthcheck_flag
were the only run_server tests in test_proxy_cli.py that neither
stripped DATABASE_URL/DIRECT_URL nor mocked the prisma DB path. When a
DATABASE_URL is present (CI/env leak), run_server --local enters the DB
block and blocks in the un-timeout'd subprocess.run(["prisma"]) at
proxy_cli.py:987 plus the ProxyExtrasDBManager migrate-deploy retry
loops, ~370s per test on the CI runner. --dist=loadscope pins both to
one xdist worker, so the proxy-infra job appears stuck at 99% and hits
the 20-min timeout.
Apply the same isolation every other run_server test in this file
already uses: mock PrismaManager.setup_database +
should_update_prisma_schema and strip DATABASE_URL/DIRECT_URL. Full
module drops from 31.7s to 2.9s locally; both tests fall off the slow
list.
* test: modernize models used in CircleCI e2e test suites
Replaces obsolete models (gpt-4o, gpt-4o-mini, gpt-3.5-turbo,
claude-3-5-sonnet-20240620, claude-sonnet-4-20250514) with current
equivalents across the e2e_openai_endpoints and
proxy_e2e_anthropic_messages_tests CircleCI jobs.
- gpt-4o -> gpt-5.5 (responses API e2e tests)
- gpt-4o-mini -> gpt-5-mini (websocket responses, oai_misc_config)
- gpt-4o-mini-2024-07-18 -> gpt-4.1-mini-2025-04-14 (fine-tuning,
still actively fine-tunable)
- gpt-4 / gpt-3.5-turbo target_model_names example -> gpt-5.5 /
gpt-5-mini
- bedrock claude-3-5-sonnet-20240620 batch entry -> haiku-4-5-20251001
(also aligning oai_misc_config model_name with what
test_bedrock_batches_api.py actually requests)
- bedrock claude-sonnet-4-20250514 (deprecated, retires 2026-06-15)
-> claude-sonnet-4-5-20250929
* test: point bedrock-claude-sonnet-4 alias at Sonnet 4.6, not 4.5
Greptile/Cursor flagged that after the previous commit, the
bedrock-claude-sonnet-4 alias collided with bedrock-claude-sonnet-4.5
(both pointed to claude-sonnet-4-5-20250929). Rename to
bedrock-claude-sonnet-4.6 and point it at the Sonnet 4.6 Bedrock ID
(us.anthropic.claude-sonnet-4-6, already in the litellm model
registry) so the alias name matches the underlying model version.
* test: modernize models across remaining CI-mounted configs & tests
Expands the modernization sweep to all CircleCI-mounted proxy configs
and to test directories where the model literal is a fixture/route key
(not the test's subject).
Config changes:
- proxy_server_config.yaml: bump gpt-3.5-turbo / gpt-3.5-turbo-1106 /
gpt-4o / gemini-1.5-flash / dall-e-3 underlying models; rename
gpt-3.5-turbo-end-user-test alias to gpt-5-mini-end-user-test; bump
text-embedding-ada-002 underlying to text-embedding-3-small. User-
facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, etc.)
preserved for backward compatibility with tests.
- simple_config.yaml, otel_test_config.yaml, spend_tracking_config.yaml:
bump gpt-3.5-turbo underlying to gpt-5-mini.
- pass_through_config.yaml: claude-3-5-sonnet / claude-3-7-sonnet /
claude-3-haiku entries replaced with claude-sonnet-4-5 / claude-
haiku-4-5 / claude-opus-4-7.
- oai_misc_config.yaml: align alias name with the gpt-5-mini rename.
Test changes (proactive: claude-sonnet-4-20250514 / claude-opus-4-
20250514 retire 2026-06-15):
- tests/llm_translation/test_anthropic_completion.py: bump 3 references
+ paired Vertex AI ID to claude-sonnet-4-5.
- tests/llm_translation/test_optional_params.py: bump 2 references.
- tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py
and test_bedrock_anthropic_messages_test.py: bump router fixtures
using the deprecated model IDs.
- tests/pass_through_unit_tests/base_anthropic_messages_tool_search_test.py:
modernize docstring examples.
- tests/test_end_users.py: update references to renamed alias.
* test: modernize placeholder model literals in router_unit_tests
Mass replace_all on fixture/placeholder model literals across the
router_unit_tests/ suite (model name is a routing key / label, not the
test subject). Sub-agent sweep so far — additional commits will follow
for logging_callback_tests/, enterprise/, top-level tests/test_*.py,
and other CI-mounted dirs.
Mappings applied:
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 / claude-3-opus-20240229 /
claude-3-haiku-20240307 / claude-3-5-sonnet-20240620 ->
claude-sonnet-4-5-20250929 / claude-opus-4-7 /
claude-haiku-4-5-20251001 as appropriate
Explicitly preserved:
- gpt-4o-mini-* variants (transcribe, tts, etc.) where they're current
- gpt-4-turbo / gpt-4-vision-preview / gpt-4-0613 (subject literals)
- JSONL batch body literals
- Mock LLM response model fields (must match upstream)
- Fake/mock identifiers
* test: modernize placeholder model literals across remaining CI suites
Sub-agent sweep across logging_callback_tests/, guardrails_tests/,
enterprise/, pass_through_unit_tests/, otel_tests/,
llm_responses_api_testing/, batches_tests/, spend_tracking_tests/,
litellm_utils_tests/, unified_google_tests/, and a few top-level
tests/test_*.py files where the model literal is a fixture or
placeholder (router model_list, mock standard logging payload, mock
callback data) rather than the test's subject.
Mappings applied (see scope notes below):
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5.5 (corrected from initial gpt-5 — bare gpt-5
is not a valid OpenAI alias; only gpt-5.5 / gpt-5.4 / gpt-5.2-codex
/ gpt-5-mini exist)
- gpt-4o-mini (bare) -> gpt-5-mini
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 -> claude-sonnet-4-5-20250929
- claude-3-opus-20240229 -> claude-opus-4-7
- claude-3-haiku-20240307 -> claude-haiku-4-5-20251001
- claude-3-5-sonnet-20240620/20241022 -> claude-sonnet-4-5-20250929
- claude-3-7-sonnet-20250219 -> claude-sonnet-4-6
- gemini-1.5-flash -> gemini-2.5-flash
- gemini-1.5-pro -> gemini-2.5-pro
Explicitly preserved (not modernized):
- llm_translation/ tests where model is the SUBJECT (provider-specific
translation/transformation logic). Only the deprecated 20250514
references were already bumped in a prior commit.
- Cost-calc / tokenizer subject tests in test_utils.py (skip-ranges
documented by the sub-agent).
- Bedrock model IDs in test_health_check.py path-stripping tests.
- JSONL batch request bodies and mock LLM response bodies (must match
upstream literal).
- Langfuse expected-request-body JSON fixtures (cost values are exact-
match-asserted; changing the model would shift response_cost).
- gpt-3.5-turbo-instruct (text-completion endpoint; no modern OpenAI
equivalent).
- Top-level tests calling the proxy through user-facing aliases
(gpt-3.5-turbo, gpt-4, text-embedding-ada-002, dall-e-3) — aliases
in proxy_server_config.yaml stay; only the underlying model was
bumped.
- tests/test_gpt5_azure_temperature_support.py (the test's whole point
is model-name handling).
- Fake / mock / openai/fake identifiers.
Notable side fixes:
- test_spend_accuracy_tests.py: UPSTREAM_MODEL now matches what
spend_tracking_config.yaml's proxy actually routes to (gpt-5-mini),
resolving a latent inconsistency.
- proxy_server_config.yaml: bare `gpt-5` alias renamed to `gpt-5.5`
(bare gpt-5 is not a valid OpenAI alias).
- test_batches_logging_unit_tests.py: explicit_models list entries
kept distinct (gpt-5-mini + gpt-5.5) after bulk rename.
* test: fix CI failures from model modernization sweep
CI surfaced 4 categories of regression from the bulk modernization:
1. Azure deployment names are customer-specific. Reverted:
- tests/litellm_utils_tests/test_health_check.py: azure/text-
embedding-3-small -> azure/text-embedding-ada-002 (the CI Azure
account does not have a text-embedding-3-small deployment).
- tests/logging_callback_tests/test_custom_callback_router.py:
same revert for two router fixtures driving aembedding.
2. gpt-5 family does not accept temperature != 1. Tests that pass a
custom temperature swapped from gpt-5-mini to gpt-4.1-mini (modern
non-reasoning OpenAI mini that still accepts temperature/logprobs):
- tests/logging_callback_tests/test_datadog.py
- tests/logging_callback_tests/test_langsmith_unit_test.py
- tests/logging_callback_tests/test_otel_logging.py
3. proxy_server_config.yaml's gpt-3.5-turbo-large alias was routing to
gpt-5.5 (a reasoning model that rejects logprobs). The proxy test
tests/test_openai_endpoints.py::test_chat_completion_streaming
exercises logprobs/top_logprobs through that alias. Bumped the
underlying model to gpt-4.1 (non-reasoning, still modern).
4. tests/logging_callback_tests/test_gcs_pub_sub.py asserts against a
pinned JSON fixture (gcs_pub_sub_body/spend_logs_payload.json) with
hardcoded model="gpt-4o" and a model-specific spend value. Reverted
the litellm.acompletion calls in the test to model="gpt-4o" so the
fixture's exact-match assertions still hold.
5. tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py:
anthropic.messages.create routing to openai/gpt-5-mini returned an
empty content[0] with max_tokens=100 (reasoning-token consumption).
Swapped to openai/gpt-4.1-mini.
* test: fix Assistants API model + 2 cursor[bot] review nits
1. pass_through_unit_tests/test_custom_logger_passthrough.py: gpt-5.5
isn't accepted by the /v1/assistants endpoint
("unsupported_model"). Switch to gpt-4.1-mini (modern, Assistants-
API-supported, non-reasoning).
2. example_config_yaml/pass_through_config.yaml: the previous sweep
bumped the claude-3-7-sonnet alias to claude-opus-4-7, which is a
tier change (Sonnet -> Opus). Map to claude-sonnet-4-6 to keep the
Sonnet tier intact. (Cursor bugbot review.)
3. example_config_yaml/simple_config.yaml: model_name was left as
gpt-3.5-turbo while the underlying was bumped to gpt-5-mini, which
muddles the "simple" example. Make both sides gpt-5-mini so the
most basic example is a straight 1:1 mapping again. (Cursor bugbot
review.)
* fix: revert gpt-4/gpt-3.5-turbo alias underlying to non-reasoning models
tests/test_openai_endpoints.py::test_completion calls the proxy alias
"gpt-4" with temperature=0, and other tests call gpt-3.5-turbo with
custom temperature / logprobs / the legacy /v1/completions endpoint.
The earlier modernization mapped both aliases to gpt-5.5 / gpt-5-mini,
which are reasoning models that reject temperature != 1 and don't
expose /v1/completions. Map the aliases to gpt-4.1 / gpt-4.1-mini
(modern non-reasoning OpenAI models) instead — keeps user-facing
aliases preserved while picking a current underlying that still
supports the parameters/endpoints the tests exercise.
Adds test_update_config_env_var_round_trip_not_double_encrypted, which
drives the real /config/update handler: first write plaintext, then
re-POST the stored ciphertext (the Admin UI round-trip) and assert the
value is not stacked with a second encryption layer and untouched keys
stay byte-identical. Verified to fail against the pre-fix handler and
pass after. Also tightens the unit test to exactly three ciphertext
re-feeds.
A single decrypt-then-encrypt chokepoint (_encrypt_env_variables_for_db)
now backs both update_config and save_config. Re-submitting a value the
Admin UI read back from /get/config/callbacks as ciphertext no longer
stacks a second encryption layer, which previously decrypted to garbage
and silently broke the callback. The chokepoint decrypts with the pure
_decrypt_db_variables (no os.environ mutation on the write path) and
encrypts exactly once; update_config merges only the sent keys so
untouched env vars keep their stored ciphertext byte-for-byte.
* fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint (#27943)
* docs: add one-line docstring to _disable_debugging (#27894)
Squash-merged by litellm-agent from oss-agent-shin's PR.
* Add jp. Bedrock cross-region inference profile for claude-sonnet-4-6 (#27831)
Squash-merged by litellm-agent from Cyberfilo's PR.
* Sanitize empty text content blocks on /v1/messages (#27832)
Squash-merged by litellm-agent from Cyberfilo's PR.
* fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint
The bedrock-mantle gateway (Claude Mythos Preview) serves the Anthropic
Messages API at /anthropic/v1/messages; /v1/messages returns 404 Not
Found. Both AmazonMantleConfig (chat/completions caller route) and
AmazonMantleMessagesConfig (anthropic-messages caller route) hardcoded
the wrong path, so every Mantle request 404'd before reaching the model.
Per the Anthropic docs: "[Claude in Amazon Bedrock] uses the Messages
API at /anthropic/v1/messages with SSE streaming."
https://platform.claude.com/docs/en/api/claude-on-amazon-bedrock
Confirmed independently against the live endpoint:
/v1/chat/completions -> 200 OK
/v1/messages -> 404 Not Found (what litellm used)
/anthropic/v1/messages -> 200 OK (Claude only)
Adds a regression test asserting both Mantle configs build the
/anthropic/v1/messages path, and updates the existing assertions that
encoded the wrong path.
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
* fix: sanitize empty text blocks in sync anthropic_messages_handler path
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com>
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Feat: Add Weighted-Routing Failover
* test(router): cover weighted failover helper functions
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): align weighted failover deployment list type with mypy
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): address greptile review on weighted failover
- Narrow exception swallowing in `_maybe_run_weighted_failover` to
`openai.APIError` so model failures defer to the regular fallback
while programming bugs (AttributeError/KeyError/TypeError) surface.
- Note async-only limitation of `enable_weighted_failover` in the
Router constructor docstring.
- Make the weighted distribution test less flaky (1000 iterations,
looser bound) and make the non-simple-shuffle test deterministic by
failing both deployments instead of relying on the latency strategy's
first pick.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): ensure weighted failover metadata persists in kwargs
The previous `kwargs.setdefault(metadata_variable_name, {}) or {}` returned
a brand-new dict whenever the existing metadata was falsy (empty dict or
None), so writes to `_failover_excluded_ids` never made it back into
`kwargs`. Multi-hop weighted failover then re-selected previously failed
deployments and exhausted `max_fallbacks` prematurely.
Explicitly assign a fresh dict into kwargs when metadata is missing so
mutations are visible to subsequent failover hops.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(router): regression for weighted failover metadata persistence
Asserts kwargs["metadata"]["_failover_excluded_ids"] is populated after
_maybe_run_weighted_failover, proving the metadata dict written by the
helper is the same object that lives in kwargs (no disconnected copy).
Pairs with the prior fix that replaced `setdefault(..., {}) or {}` with
an explicit get/assign so writes survive across hops.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): harden weighted failover error/state handling
- Catch RouterRateLimitError (ValueError) alongside openai.APIError in
_maybe_run_weighted_failover so an exhausted intra-group retry falls
through to the regular cross-group fallback path instead of bubbling
out and bypassing configured fallbacks.
- Stop mutating the shared input_kwargs dict; build a local copy with
the weighted-failover keys so the entry (with _excluded_deployment_ids)
cannot leak into later fallback paths reading the same dict.
- _get_excluded_filtered_deployments now returns an empty list when the
exclusion filter removes every healthy deployment, instead of falling
back to the original list. The original-list behavior risked re-picking
the just-failed deployment; callers already handle the empty case by
raising their no-deployments error, which weighted failover now catches
and converts into a normal cross-group fallback.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(router): fall through to rpm/tpm when total weight is zero
When the weight metric's total is zero (e.g. after weighted-failover
exclusion leaves only zero-weight backups), continue to the next metric
(rpm/tpm) instead of returning a uniform random pick immediately. This
lets rpm/tpm still drive routing when present, and only falls back to
the uniform random pick at the end if no metric provides a positive
total weight.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(router): skip weighted failover when remaining deployments are all in cooldown
_maybe_run_weighted_failover was computing 'remaining' from all_deployments
(every deployment in the model group, including those in cooldown). This meant
that when all non-excluded deployments were in cooldown the method still invoked
run_async_fallback unnecessarily, which propagated into async_get_healthy_deployments,
found no eligible deployments, and raised RouterRateLimitError — only safely
caught thanks to the earlier exception-broadening fix.
The fix: before computing 'remaining', fetch the current cooldown set via
_async_get_cooldown_deployments and subtract it from all_ids. This allows
_maybe_run_weighted_failover to return None immediately (skipping the
run_async_fallback call entirely) when every non-failed deployment is in cooldown,
letting the caller fall through to the correct cross-group fallback path without
the wasteful extra round-trip.
Tests added:
- unit: _maybe_run_weighted_failover returns None without calling run_async_fallback
when all remaining deployments are in cooldown
- unit: _maybe_run_weighted_failover still calls run_async_fallback when at least
one healthy (non-cooldown) deployment is available
- integration: end-to-end fallthrough to cross-group fallback when remaining
deployments are in cooldown
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(vertex-ai): fix zero cost/usage on completed Vertex AI batch jobs
Vertex batch jobs recorded 0 spend and 0 tokens after PR #25627 added
automatic transformation of GCS predictions.jsonl to OpenAI format.
Two bugs fixed:
1. batch_utils.py: the Vertex-specific cost/usage reader
(calculate_vertex_ai_batch_cost_and_usage) was always invoked and
reads raw usageMetadata fields that no longer exist in the
OpenAI-shaped output. Now the reader is only used when
disable_vertex_batch_output_transformation=True; otherwise the
generic path handles the already-transformed OpenAI-shaped content.
2. cost_calculator.py: batch_cost_calculator skipped the global
litellm.get_model_info() lookup when a model_info dict was passed
in, even when that dict had no pricing fields (e.g. deployment
metadata with only id/db_model). It now falls back to the global
pricing table when the provided model_info has no pricing data.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Update litellm/cost_calculator.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(cost-calculator): use not-any guard for pricing fallback in batch_cost_calculator
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(cost-calculator): treat explicit zero batch pricing as set in model_info
The fallback to litellm.get_model_info() used truthy checks on pricing
fields, so 0.0 was treated as missing and replaced by global rates.
Use `is not None` like elsewhere in cost calculation. Add regression test.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(managed_batches): convert raw output_file_id to managed ID in CheckBatchCost poller
CheckBatchCost bypasses async_post_call_success_hook, causing raw provider
output_file_ids to be persisted in LiteLLM_ManagedObjectTable. This fix converts
output_file_id and error_file_id to managed base64 IDs before the DB write.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(check_batch_cost): persist managed file before mutating response and propagate team_id
- Move setattr after store_unified_file_id so the response only receives the
managed ID once the DB record is successfully written. Avoids serializing
an orphaned managed ID into file_object when the store call fails.
- Populate team_id on the minimal UserAPIKeyAuth from job.team_id so the
managed file record is created with the correct team ownership, allowing
other team members to access the batch output file via /files/{id}/content.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(managed_batches): extend test to cover error_file_id conversion
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix managed file test
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(proxy): fix vector store retrieve/list/update/delete routing without model
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): remove unchecked query-param injection in vector store management endpoints
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(proxy): use subset assertion for vector store route test to allow extra kwargs like shared_session
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
_build_mcp_server_table omitted delegate_auth_to_upstream, so GET /v1/mcp/server always returned the default false while the registry kept the DB value.
Co-authored-by: Cursor <cursoragent@cursor.com>
Log verbose_logger.warning when loading oauth2 interactive servers with
available_on_public_internet=false and delegate_auth_to_upstream=true
(config + DB). Dashboard Alert for the same combo. CLAUDE note for
operators. Tests for log and M2M skip.
Remove available_on_public_internet gating from delegate-auth-to-upstream
paths so oauth2 + delegate_auth_to_upstream interactive servers behave
the same when marked internal. Keeps M2M exclusion. Updates tests.
pytest-cov treats --cov=<module-name> as a Python package and emits XML
paths relative to the package root, stripping the litellm/ prefix
(`proxy/proxy_server.py` instead of `litellm/proxy/proxy_server.py`).
Codecov's auto-prefix heuristic then drops every file whose basename is
ambiguous in the repo — `proxy_server.py` (3 copies under enterprise/),
`router.py` (2 copies), `utils.py` (20+), `main.py` (20+), `constants.py`
(2). The 11 highest-fix-rate hotspots have never appeared in Codecov.
Switching to --cov=./litellm treats the argument as a path, which makes
coverage.xml emit repo-relative paths (`litellm/proxy/proxy_server.py`).
Each path is unambiguous, so Codecov resolves all files correctly.
Verified locally: rerunning a single proxy_unit_tests test with
--cov=./litellm produced `filename="litellm/proxy/proxy_server.py"`,
`filename="litellm/router.py"`, and `filename="litellm/types/router.py"`
as distinct entries — exactly the disambiguation Codecov needs.
Touches every workflow that uploads coverage: the two reusable GHA
workflows (_test-unit-base.yml, _test-unit-services-base.yml),
test-mcp.yml, and all 14 invocations in .circleci/config.yml.
Per-file `vi.mock("@tremor/react", ...)` factories fully replace the
setup-level mock from `tests/setupTests.ts`, so the global Button/Tooltip
overrides are lost in any file that re-mocks `@tremor/react`. Without
them, the real Tremor `<Button>` leaks through and its internal
`useTooltip(300)` schedules a native 300ms `setTimeout` on pointer
events. When the test environment is torn down before the timer fires,
the trailing `setState` calls `getCurrentEventPriority`, which reads
`window.event` against a destroyed jsdom -> "window is not defined"
flake observed on CI.
Patches the 7 leaky test files to re-supply `Button` (bare `<button>`)
and `Tooltip` (Fragment) overrides matching `setupTests.ts`. Also drops
a dead `afterEach` workaround in `user_edit_view.test.tsx` (the
fake-timer dance it ran could not drain a real timer scheduled before
the swap) and corrects a misleading comment in `MakeMCPPublicForm.test.tsx`.
This reverts commit e25a988a3f.
The `fixes: ["::litellm/"]` rule turned out to be applied *after* Codecov's
auto-resolution, not before. Files with unique basenames (which were
auto-resolving correctly to `litellm/<path>`) got an extra `litellm/`
prepended, producing `litellm/litellm/<path>` storage. Files with
ambiguous basenames (the actual target of the fix) continued to be
dropped because the auto-resolution still failed for them.
Net result on the verification run: 1375 files now stored under
unresolvable `litellm/litellm/...` paths, and the 11 originally-missing
hotspots are still missing. Reverting before piling on further changes.
Audit of .github/workflows/ via gh run history shows the following have
either never run or have been dormant for 10+ weeks. CI coverage that
still matters is preserved on CircleCI (e.g. llm_translation_testing).
Removed workflows:
- test-litellm.yml — workflow_dispatch only, last run 2026-02-12 (cancelled);
CCI local_testing_part1/2 covers the same tests
- llm-translation-testing.yml — last run 2025-07-10; replaced by CCI
llm_translation_testing job (run_llm_translation_tests.py kept for the
make test-llm-translation target)
- run_observatory_tests.yml — last run 2026-03-03 (cancelled)
- scan_duplicate_issues.yml — last run 2026-03-02 (failure)
- publish_to_pypi.yml — never run
- read_pyproject_version.yml — fires on every push to main but its echoed
version output is not consumed by any downstream step
Removed orphan files (no callers in workflows, CCI, or Makefile):
- .github/workflows/README.md — documented only publish_to_pypi.yml
- .github/workflows/update_release.py + results_stats.csv
- .github/actions/helm-oci-chart-releaser/
pytest-cov runs with --cov=litellm, which makes coverage.xml store paths
relative to the package root (e.g. `proxy/proxy_server.py` instead of
`litellm/proxy/proxy_server.py`). Codecov auto-resolves these only when
the basename is unique in the repo. Files like proxy_server.py, router.py,
utils.py, main.py, and constants.py — which have duplicates under
enterprise/ or other subpackages — get silently dropped during ingest.
The `fixes: ["::litellm/"]` rule prepends `litellm/` to every uploaded
path so they resolve unambiguously. Confirmed against multiple recent
coverage.xml artifacts that no uploader currently emits paths already
prefixed with `litellm/`, so the rule is safe to apply universally.
This restores Codecov visibility for the highest-fix-rate hotspots:
proxy_server.py, router.py, proxy/utils.py, litellm_logging.py,
constants.py, key_management_endpoints.py, utils.py, main.py,
user_api_key_auth.py, team_endpoints.py, and litellm_pre_call_utils.py.
* feat(custom_logger): add async_post_agentic_loop_response_hook
Lets a CustomLogger shape the response returned by the agentic-loop
follow-up call without bypassing the loop's safety / observability
machinery (depth tracking, fingerprinting, etc.). Default returns the
response unchanged.
Used by websearch_interception to inject Anthropic-native
web_search_tool_result blocks when the originating client requested a
native web_search_* tool.
* feat(llm_http_handler): call post-agentic-loop hook on the originating callback
In _execute_anthropic_agentic_plan, after anthropic_messages.acreate
returns, call the originating callback's
async_post_agentic_loop_response_hook so it can mutate the final
response (e.g. inject native tool_result blocks). Pass the callback
through from _call_agentic_completion_hooks.
Exceptions in the post-hook are caught and logged so a buggy callback
can't kill the request.
* feat(websearch_interception): add is_anthropic_native_web_search_tool
Identifies tools the Anthropic-native clients (Claude Desktop, the
Anthropic SDK, the Anthropic Console) use to request native search:
type starts with "web_search_" (e.g. web_search_20250305). Rejects the
LiteLLM standard tool, the OpenAI-function variant, the bare
"WebSearch" legacy name, and the bare "web_search" Claude Code shape.
This lets us decide per-request whether the client expects
web_search_tool_result content blocks in the response, without
renaming any existing constants or touching native-provider skip
logic.
* feat(websearch_interception): add build_web_search_tool_result_block
Produces the Anthropic-native web_search_tool_result content block
from a structured SearchResponse. Anthropic-native clients use this
block to populate citations / source links — the existing text-blob
flatten path only feeds readable evidence to the model and discards
the structure, so this builder gives us the missing piece.
Shape matches https://docs.anthropic.com/en/api/web-search-tool —
web_search_result items carry url, title, page_age, encrypted_content
(empty string when the search provider doesn't supply one).
* feat(websearch_interception): emit native web_search_tool_result blocks
When the originating client request carried a native Anthropic
web_search_* tool, the final response now also carries
web_search_tool_result content blocks alongside the model's text
answer — so Claude Desktop / Anthropic SDK clients can populate the
citations panel and replay conversation history with structured search
evidence.
Wiring:
- Pre-request hooks (both deployment + Anthropic path) set a flag on
kwargs when they see a native web_search_* tool, so the signal
survives the conversion-to-litellm_web_search step regardless of
which hook fires first.
- _execute_search now returns (text, SearchResponse) so the structured
results aren't lost when the text is flattened for the follow-up
model call.
- _build_anthropic_request_patch returns the parallel list of
SearchResponse objects.
- async_build_agentic_loop_plan pre-builds the web_search_tool_result
blocks (one per tool_use_id) and stashes them on plan.metadata when
the flag is set.
- async_post_agentic_loop_response_hook reads the metadata and
prepends the blocks to response.content.
- _execute_agentic_loop mirrors the injection for the legacy path so
both paths behave identically.
Clients that send the LiteLLM standard tool keep the existing
text-only behavior — no regression.
* test(websearch_interception): cover native web_search_tool_result emission
18 tests across:
- detector branches (native vs litellm-standard, OpenAI-function shape,
Claude Desktop builtin WebSearch, bare web_search, missing type)
- block-builder shape (results, none, empty)
- pre-request hook flag-setting (native sets, standard does not)
- async_build_agentic_loop_plan attaches blocks to plan.metadata when
the flag is present, leaves metadata untouched when absent
- post-hook injection into dict and object responses
- legacy _execute_agentic_loop mirrors the injection so both paths
return the same shape
* test(websearch_short_circuit): keep _execute_search mocks in sync with new tuple return
* test(websearch_thinking_constraint): keep _execute_search mocks in sync with new tuple return
* feat(websearch_interception): emit native blocks from try_short_circuit_search
The agentic-loop post-hook only fires when the model returns a tool_use
block. Cowork / Claude Desktop on Bedrock actually make TWO requests
per user turn: the main /v1/messages with their builtin tool, and a
separate standalone /v1/messages whose only tool is
web_search_20250305. That second request hits try_short_circuit_search
— no agentic loop, no post-hook — and was returning text-only, leaving
the citations panel empty.
When the short-circuit input carries a native web_search_* tool, build
a synthetic server_tool_use + web_search_tool_result pair (using the
structured SearchResponse already returned by _execute_search) so the
client gets the native shape it expects. The legacy text block is
preserved so non-native short-circuit callers (Claude Code,
github_copilot, etc.) see the same payload as before.
Failure path still emits the native block pair (with empty results)
plus the text-error block, so the client gets a well-formed response
rather than a malformed half-shape.
* test(websearch_native_blocks): cover short-circuit native-block emission
Three new cases on top of the existing 18:
- native web_search_20250305 short-circuit → [server_tool_use,
web_search_tool_result, text], ids paired, urls/titles carried.
- litellm_web_search short-circuit → text-only (no regression).
- native short-circuit on search failure → still emits the native
block pair (empty results) plus the text-error block, so the client
never sees a malformed half-shape.
* test(websearch_short_circuit): index assertions by block type, not by position
Native short-circuit responses now have [server_tool_use,
web_search_tool_result, text] when the input carries
web_search_20250305 — find the text block by type rather than relying
on content[0].
* fix(websearch_interception): gate legacy WebSearch name on schema absence
Clients like Cowork / Claude Desktop ship a client-side tool named
"WebSearch" with a full input_schema — they handle it themselves and
expect to make a separate native web_search_20250305 sub-request for
the actual search.
Today is_web_search_tool matches the bare name regardless of other
fields, which hijacks the client's tool server-side. The agentic loop
fires on the main request, the model never gets to emit the
client-side tool_use, and the separate native sub-request (where
citation data flows) is never made. Net: citations panel empty.
Real Anthropic client tools always carry input_schema (the API rejects
them otherwise), so a bare {name: "WebSearch"} with no schema is the
only thing that could be a legacy interception marker. Gate the match
on schema absence: legacy callers (if any) keep working, real
client-side WebSearch tools pass through untouched.
* fix(websearch_interception): drop "WebSearch" from response-detection lists
Post-conversion the model always sees ``litellm_web_search``, so the
"WebSearch" entry in the response-side tool_use detection lists was
dead at best. If a model ever did return ``tool_use(name="WebSearch")``
it would now (incorrectly) hijack the client's own ``WebSearch`` tool
again — same Cowork problem we just fixed on the input side. Drop it.
* test(websearch_native_blocks): cover the WebSearch legacy-name schema gate
Three new cases:
- {name: "WebSearch"} (bare interception marker) → still matched
- {name: "WebSearch", input_schema: {...}} (Cowork client tool) →
passes through untouched
- {name: "WebSearch", description: "..."} (no schema) → still matched
on the assumption it's a legacy marker rather than a malformed real
client tool.
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* fix(rate-limit): stop v3 limiter from leaking internal stash to provider body
PR #27001 (atomic TPM rate limit) introduced a reservation flow that
writes four LiteLLM-internal keys onto the request data dict:
_litellm_rate_limit_descriptors
_litellm_tpm_reserved_tokens
_litellm_tpm_reserved_model
_litellm_tpm_reserved_scopes
_litellm_tpm_reservation_released
These keys are forwarded as request body params to the upstream provider,
which rejects them as unknown fields:
OpenAI -> 400 'Unknown parameter: _litellm_rate_limit_descriptors'
(mapped by litellm to RateLimitError / 429, hiding the bug
behind a misleading 'throttling_error' code)
Anthropic -> 400 '_litellm_rate_limit_descriptors: Extra inputs are
not permitted'
Net effect: every chat completion against any real provider fails the
moment a virtual key has any tpm_limit / rpm_limit set — i.e. v3-enforced
key-level TPM/RPM limits are broken end-to-end. The v3 RPM/TPM check
itself still runs (raises 429 on over-limit), but the success path
poisons the upstream body.
Reproduced on litellm_internal_staging HEAD (410ce761dc) against
gpt-4o-mini and claude-haiku-4-5 with a 1-RPM/1-TPM key — first request
fails with the provider's unknown-field error.
Fix: the stash is metadata only.
- Add RATE_LIMIT_DESCRIPTORS_KEY constant and a _LITELLM_STASH_KEYS
registry so we have a single source of truth for stash keys.
- New helper _stash_value_in_metadata_channels writes to
data['metadata'] / data['litellm_metadata'] without touching the
top level.
- _stash_reservation_in_data and the descriptor stash now route
through that helper. _mark_reservation_released stops writing
top-level.
- _lookup_stashed_value also checks kwargs['metadata'] /
kwargs['litellm_metadata'] (raw request_data shape) in addition to
kwargs['litellm_params']['metadata'] (completion kwargs shape).
- async_post_call_failure_hook now reads descriptors via the unified
metadata lookup instead of request_data.get(top-level).
- Defense in depth: async_pre_call_hook strips any stash key that
somehow surfaced at the top level (stale cache, future refactor,
test fixture) before returning.
Tests:
- New regression test asserts no _litellm_* stash key is present at
the top level of data after async_pre_call_hook, and that the
metadata channel still carries the reservation + descriptors so
success / failure reconciliation works.
- Existing test_tpm_concurrent.py tests that asserted top-level
presence are updated to read from data['metadata'] — the location
is an implementation detail; the spec is that post-call callbacks
can resolve the stash.
Verified end-to-end against OpenAI gpt-4o-mini and Anthropic
claude-haiku-4-5 via /v1/chat/completions on a low-rpm key:
- With limits not exceeded: HTTP 200, valid completion response,
no leaked fields in body.
- With RPM exceeded: HTTP 429 from v3 enforcement
('Rate limit exceeded ... Limit type: requests').
- With TPM exceeded: HTTP 429 from v3 enforcement
('Rate limit exceeded ... Limit type: tokens').
Full v3 hook test suite passes (171 tests).
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore(rate-limit): use RATE_LIMIT_DESCRIPTORS_KEY constant in test, trim noisy comments
Address greptile P2: test fixture now uses the imported constant.
Drop comments that re-explain what well-named identifiers already convey.
* fix(rate-limit): reject caller-supplied stash values to prevent TPM-refund abuse
Strip _LITELLM_STASH_KEYS from data top-level and both metadata channels at
the start of async_pre_call_hook. Without this, an authenticated caller can
inject _litellm_rate_limit_descriptors plus _litellm_tpm_reserved_tokens in
body metadata, trigger a proxy-side rejection, and cause
async_post_call_failure_hook to refund TPM counters against attacker-named
scopes (e.g. another tenant's api_key).
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
