Store search tool allowlists only on object permissions, wire auth/management/UI flows to object_permission.search_tools, and remove legacy team-metadata search credential code and tests.
Made-with: Cursor
Greptile review on #26756 (P2): if `attempt_db_reconnect` itself raises
(e.g. lock cancellation, timer error, unexpected internal failure), the
original `httpx.ReadError` / transport error was lost — `failure_handler`
and `db_exceptions` alerts then logged the reconnect exception instead of
the actual DB transport problem, masking the root cause.
Wrap the reconnect call in a try/except. On reconnect failure, re-raise
the *original* `first_exc` and chain the reconnect error as `__cause__`
so it remains visible for debuggability without becoming the primary
exception observers see.
Adds `test_call_with_db_reconnect_retry_preserves_original_error_when_reconnect_raises`
asserting (a) the propagated exception is the original transport error
and (b) the reconnect exception is attached as `__cause__`.
Two related fixes layered on top of the existing reconnect plumbing:
1. Restore reconnect-and-retry on `PrismaClient.get_generic_data` (issue
#25143). 1.83.x lost the transport-reconnect-and-retry-once branch that
1.82.6 had on this method, so transient `httpx.ReadError` flaps now
surface immediately as `db_exceptions` alerts. `_update_config_from_db`
fans out four concurrent `get_generic_data` reads, so a single transport
blip used to mark four alerts and a stale config window.
Adds `call_with_db_reconnect_retry` to `litellm/proxy/db/exception_handler.py`
— a single canonical "try DB read, on transport error reconnect once and
retry once" wrapper. Mirrors the inline pattern in
`auth_checks._fetch_key_object_from_db_with_reconnect` so we have one
implementation rather than three drifting copies, and gives future read
paths a clean opt-in.
2. Fix the `_engine_confirmed_dead` flag-reset bug in
`_run_reconnect_cycle`. The flag was cleared before `_do_heavy_reconnect()`
ran, so any failure inside the heavy reconnect (timeout, missing
DATABASE_URL, recreate failure) left the flag False — and the next
attempt could silently demote to the lightweight path even though the
engine was genuinely dead. Move the reset into the success branch so the
flag stays True across heavy-reconnect failures and the next attempt
re-enters the heavy branch.
Tests:
- `tests/test_litellm/proxy/db/test_exception_handler_reconnect_retry.py`
(new) — 9 tests covering the helper's contract: happy path, retry on
transport error, no retry on data-layer errors, propagation when reconnect
fails, propagation after second transport error, `hasattr` guard for
partial mocks, fresh-coroutine-per-call invariant, explicit timeout
override, default timeouts read off the prisma_client.
- `tests/test_litellm/proxy/db/test_prisma_self_heal.py` — adds:
- `test_get_generic_data_retries_on_transport_error_for_config_table`
- `test_get_generic_data_propagates_when_reconnect_fails`
- `test_engine_confirmed_dead_persists_across_failed_heavy_reconnect`
(regression test for the flag-reset bug).
All 16 self-heal tests + 9 helper tests + 535 auth/exception-handler tests
pass locally.
- server.py: drop the redundant server_id append in
_get_filtered_mcp_servers_from_mcp_server_names. iter_known_server_prefixes
already yields server_id unconditionally, so the manual append (and its
misleading comment) was a no-op duplicate.
- utils.py: rewrite the SHORT_MCP_TOOL_PREFIX docstring to accurately
describe the collision behaviour. The previous wording said collisions
were 'cosmetic only', but a natural-hash collision IS a routing-correctness
issue, which is precisely why we already added _assign_unique_short_prefix
to rehash deterministically. The new comment cross-references that path.
- utils.py: restrict the first character of the short prefix to [A-Za-z]
via a 52-char alphabet for position 0 only. The remaining two positions
still use the full base62 alphabet. This keeps prefixes valid identifiers
on every backend and gives 52*62*62 = 199_888 distinct prefixes (still
comfortably more than any realistic deployment).
- tests: add coverage proving the first character of the prefix is always
alphabetic across many server_ids and rehash attempts.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The test was flaking on unrelated asyncio ERROR records (e.g. "Unclosed
client session" from background tasks in other tests). Restrict the
assertion to records emitted by LiteLLM loggers so the test only fails
on errors actually produced by the code under test.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Two MCP servers can natural-hash to the same three-character base62
prefix. With 62**3 = 238_328 slots the birthday bound is ~488 servers
for 50% collision probability, so a single proxy hosting more than
~100 MCP servers has a non-trivial chance of seeing a collision in
practice — and a collision means tool names from two different servers
share a routing key, causing silent mis-routing.
Mitigation:
- compute_short_server_prefix(server_id, attempt=N) folds an attempt
counter into the SHA-256 seed, so rehashes are deterministic and
produce a fresh three-char prefix space per attempt.
- New MCPServer.short_prefix field caches the resolved (post-dedup)
prefix on the model so it stays stable across the process lifetime.
- MCPServerManager._assign_unique_short_prefix walks attempts 0..N
until it finds a prefix not already used by another server in the
combined registry. Logs an INFO line when a rehash happens so
operators have a breadcrumb if it ever does.
- Wired into every registration path: load_servers_from_config,
add_server, update_server, reload_servers_from_database. The
database reload path also carries the previously-resolved prefix
forward so reloads don't churn it.
- get_server_prefix prefers the cached short_prefix when set, so the
resolved value (not the raw natural hash) is used everywhere.
- iter_known_server_prefixes yields the cached short_prefix too, so
reverse-lookup tolerance covers the rehashed form.
No-op when LITELLM_USE_SHORT_MCP_TOOL_PREFIX is disabled — the field
stays None and behaviour is unchanged.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds LITELLM_USE_SHORT_MCP_TOOL_PREFIX. When enabled, tool / prompt /
resource / resource-template names emitted from MCP servers are prefixed
with a deterministic three-character base62 ID derived from the server's
server_id (SHA-256 → base62) instead of the (potentially long)
alias / server_name. This keeps namespaced tool names well under the
60-character upper bound enforced by some model APIs while still letting
us distinguish MCP-routed tools from local tools.
Behavioural notes:
- Default off — when the env var is unset, the long-prefix behaviour
is unchanged. The plan is to flip the default in a future release
and remove the gate after a deprecation window.
- Prefix derivation is deterministic, so it is stable across processes,
workers and restarts without any persistence layer.
- Reverse-lookup is tolerant: _create_prefixed_tools registers every
known prefix form (alias / server_name / server_id / short ID) in
the routing map and _get_mcp_server_from_tool_name resolves any of
them. Old clients holding cached long-prefixed names continue to
route correctly even after the flag is enabled.
- _get_allowed_mcp_servers_from_mcp_server_names accepts the short
prefix in /mcp/{server_name}-style URLs.
- The OpenAPI tool-listing path now filters by the active server
prefix instead of server.name so spec-backed servers benefit too.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Unify cost calc in success_handler dict and typed branches
* Trim verbose comments and docstrings
---------
Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>
Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local>
Claude 3.5 Sonnet v2 reached EOL on Bedrock 2026-03-01, returning the same
404 EOL error as 3.7 Sonnet. Sonnet 4.5 supports both InvokeModel and
Converse APIs on Bedrock, so use the same model for both routes.
AWS Bedrock has reached end-of-life for `claude-3-7-sonnet-20250219-v1:0`,
returning 404s with "This model version has reached the end of its life."
Update test references to `claude-sonnet-4-5-20250929-v1:0` (same capability
surface: thinking, tools, prompt caching, PDF input, vision, computer use).
The bedrock/invoke pass-through tests stay on Sonnet 3.5 since Sonnet 4.5
is converse-only on Bedrock.
- Tighten _is_anthropic_document_data_uri to match the mimes Anthropic
actually accepts as base64 `document` source ({application/pdf,
text/plain}). The previous application/* + text/* prefix match would
route e.g. data:application/json URIs through the document path,
producing blocks the Anthropic API rejects. Unsupported mimes now
stay on the existing image code path (same failure mode as before the
fix — no regression, just stops introducing a new one).
- On the Bedrock tool-result `type: "file"` branch, accept either
file_data or file_id and raise BadRequestError on both-None, mirroring
the user-message _process_file_message pattern. Previously a file
block with only file_id was silently dropped.
- Consolidate the six new PDF tool-result tests under tests/test_litellm/
only (the PR template's required location and where the unit-test CI
workflow runs with coverage). The duplicate copies under
tests/llm_translation/ added drift risk with no additional coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Updated instances of DualCache to UserApiKeyCache across multiple files to enhance cache handling for user API keys.
- Adjusted cache retrieval and storage methods to ensure proper serialization and deserialization of cached objects.
- Introduced a new UserApiKeyCache class to streamline caching logic and improve type safety.
- Updated relevant tests to reflect changes in caching behavior and ensure compatibility with the new cache implementation.
Duplicate the three Bedrock and three Anthropic tool-result tests into
tests/test_litellm/ so they're picked up by `make test-unit` (and its
coverage report). The originals in tests/llm_translation/ stay — they
run under integration and remain the canonical translation-suite
regression cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The parent OpenAIGPTConfig already handles reasoning->reasoning_content
for non-streaming via _extract_reasoning_content. The override was dead
code giving false confidence. Streaming fix in chunk_parser is the only
change needed for chat completions.
Addresses Agent Shin review feedback on #26595
OpenAI Chat Completions `{type: "file", file: {file_data: "data:application/pdf;..."}}`
content blocks inside tool messages were silently dropped when translated to
Bedrock Converse and direct Anthropic. Additionally, PDFs sent via `image_url`
data URIs were either dropped (Bedrock) or wrapped as `type: "image"` and
rejected by the API (Anthropic).
- _convert_to_bedrock_tool_call_result: add `type: "file"` branch; pass through
document blocks produced by BedrockImageProcessor for PDF `image_url` URIs.
Single choke point covers both sync and async converse paths.
- convert_to_anthropic_tool_result: add `type: "file"` branch delegating to
`anthropic_process_openai_file_message`; branch `image_url` on data-URI mime
type so non-image mimes route through the file helper to produce document
blocks.
- AnthropicMessagesToolResultParam.content union extended to accept
`AnthropicMessagesDocumentParam` alongside text and image.
- Add 6 tests (3 Bedrock + 3 Anthropic) covering file-PDF, image_url-PDF, and
image_url-PNG regression.
Fixes#24641
Supersedes #24646 with an OpenAI-native approach and test coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to the prior commit. process_items only converted empty
`items: {}` to `{"type": "object"}`. But anyOf branches like
`{"type": "array"}` (no items field at all) were untouched, so after
convert_anyof_null_to_nullable stripped the null branch and added
nullable, the array branch was sent to Vertex as
`{"type": "array", "nullable": true}` — which Vertex rejects with
INVALID_ARGUMENT (`any_of[0].items: missing field`).
Make process_items synthesize `items: {"type": "object"}` for any
`type == "array"` schema where items is missing or empty.
Also:
- Convert test_gemini_tool_calling_working_demo to a hermetic mock
test asserting items is present on the array branch in the sent
body. Was previously a real-network call to Vertex and was the
test the user reported still failing in CI.
- Add unit test test_build_vertex_schema_array_branch_missing_items_in_anyof
covering the missing-items shape directly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cache provider config lookups for Vertex Anthropic messages so repeated requests reuse the same config object and preserve credential cache state. Add a regression test to catch any future loss of config reuse.
Made-with: Cursor
* fix(caching): preserve prompt_tokens_details through embedding cache round-trip
The embedding caching layer was dropping prompt_tokens_details (including
image_count) because CachedEmbedding had no field for usage metadata and
the cache retrieval code reconstructed Usage without it. This caused
inconsistent responses where the first call returned image_count but
cached responses did not, breaking cost tracking for multimodal embeddings.
Add prompt_tokens_details to CachedEmbedding, persist per-item details
during cache storage, aggregate them on retrieval, and merge them in
combine_usage() for partial cache hits.
* style: apply Black formatting to caching files
* fix(caching): address Greptile review — cyclic import, guarded construction, nested dict merge
Move PromptTokensDetailsWrapper to inline import to resolve CodeQL cyclic
import warning. Guard PromptTokensDetailsWrapper construction with
try/except to handle unexpected cached keys. Add recursive dict merging
in _merge_prompt_tokens_details for nested fields like
cache_creation_token_details.
Drop accidental dashboard export path renames in _experimental/out, remove committed local proxy log, and remove the unintended ad-hoc test file so the feature commit only contains intentional source changes.
Made-with: Cursor
Treat search tools like models by adding team/key allowed_search_tools controls, enforcing search tool authorization checks, and moving credential ownership to search tool config only to avoid exposing secrets in team metadata.
Made-with: Cursor
Adds transform_response to OVHCloudChatConfig to normalise the new
easoning field to
easoning_content in non-streaming responses,
matching the existing streaming fix in chunk_parser.
Addresses maintainer feedback on #26595
Allow search requests to resolve provider credentials from request metadata, team metadata, and default team settings with clear precedence, and expose this flow in proxy docs/UI with regression tests.
Made-with: Cursor
Route vector store search `extra_body` into provider transformers and handle Bedrock `retrievalConfiguration` explicitly so only intended provider-specific fields are forwarded.
Made-with: Cursor
- Replace isinstance(item, dict) with hasattr(item, 'get') so Pydantic
model instances (ResponseOutputMessage, ResponseFunctionToolCall) are
accepted alongside plain dicts (P1)
- Use 'is not None' guards instead of or-chain for system_instructions
coalescing to prevent falsy values (e.g. []) falling through to the
wrong kwarg (P2)
- Emit per-tool-call span attributes (gen_ai.completion.N.function_call.*)
for Responses API function_call items, matching the choices branch
parity with _tool_calls_kv_pair (P2)
- Add 4 new tests: Pydantic-like objects, falsy fallthrough guard,
per-tool-call attribute emission, multiple tool call indexing
Previously the normalize_callback_names call only ran when the existing
litellm_settings DB row already had a success_callback key. On the very
first write (no row yet, or row missing the key), incoming mixed-case
values like ["SQS", "sQs"] persisted as-is. delete_callback (lowercase
lookup) then could not find them, and a follow-up /config/update would
union normalized incoming with mixed-case stored entries, producing
duplicates.
Always normalize incoming success_callback before merging, and dedupe
both the standalone first-write case and the union-with-existing case.
Adds test_success_callback_normalized_on_first_write covering the
no-existing-row path; the existing union test still passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
convert_anyof_null_to_nullable was stripping the items field from array
branches inside anyOf when a sibling null branch was present, leaving
{"type": "array"} without items. Vertex requires items whenever
type == "array" (even inside anyOf) and rejects the call with
INVALID_ARGUMENT.
Leave the (possibly empty) items in place so the downstream process_items
step can convert {} to {"type": "object"}, which is what Vertex wants.
Also:
- Update test_build_vertex_schema expected output, which was codifying
the broken shape.
- Convert test_gemini_tool_calling_not_working to a hermetic mock test
that asserts the request body sent to Vertex includes items inside
the callbacks anyOf array branch. The previous form made a real
network call and was flaky in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
