The parent OpenAIGPTConfig already handles reasoning->reasoning_content
for non-streaming via _extract_reasoning_content. The override was dead
code giving false confidence. Streaming fix in chunk_parser is the only
change needed for chat completions.
Addresses Agent Shin review feedback on #26595
OpenAI Chat Completions `{type: "file", file: {file_data: "data:application/pdf;..."}}`
content blocks inside tool messages were silently dropped when translated to
Bedrock Converse and direct Anthropic. Additionally, PDFs sent via `image_url`
data URIs were either dropped (Bedrock) or wrapped as `type: "image"` and
rejected by the API (Anthropic).
- _convert_to_bedrock_tool_call_result: add `type: "file"` branch; pass through
document blocks produced by BedrockImageProcessor for PDF `image_url` URIs.
Single choke point covers both sync and async converse paths.
- convert_to_anthropic_tool_result: add `type: "file"` branch delegating to
`anthropic_process_openai_file_message`; branch `image_url` on data-URI mime
type so non-image mimes route through the file helper to produce document
blocks.
- AnthropicMessagesToolResultParam.content union extended to accept
`AnthropicMessagesDocumentParam` alongside text and image.
- Add 6 tests (3 Bedrock + 3 Anthropic) covering file-PDF, image_url-PDF, and
image_url-PNG regression.
Fixes#24641
Supersedes #24646 with an OpenAI-native approach and test coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to the prior commit. process_items only converted empty
`items: {}` to `{"type": "object"}`. But anyOf branches like
`{"type": "array"}` (no items field at all) were untouched, so after
convert_anyof_null_to_nullable stripped the null branch and added
nullable, the array branch was sent to Vertex as
`{"type": "array", "nullable": true}` — which Vertex rejects with
INVALID_ARGUMENT (`any_of[0].items: missing field`).
Make process_items synthesize `items: {"type": "object"}` for any
`type == "array"` schema where items is missing or empty.
Also:
- Convert test_gemini_tool_calling_working_demo to a hermetic mock
test asserting items is present on the array branch in the sent
body. Was previously a real-network call to Vertex and was the
test the user reported still failing in CI.
- Add unit test test_build_vertex_schema_array_branch_missing_items_in_anyof
covering the missing-items shape directly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cache provider config lookups for Vertex Anthropic messages so repeated requests reuse the same config object and preserve credential cache state. Add a regression test to catch any future loss of config reuse.
Made-with: Cursor
* fix(caching): preserve prompt_tokens_details through embedding cache round-trip
The embedding caching layer was dropping prompt_tokens_details (including
image_count) because CachedEmbedding had no field for usage metadata and
the cache retrieval code reconstructed Usage without it. This caused
inconsistent responses where the first call returned image_count but
cached responses did not, breaking cost tracking for multimodal embeddings.
Add prompt_tokens_details to CachedEmbedding, persist per-item details
during cache storage, aggregate them on retrieval, and merge them in
combine_usage() for partial cache hits.
* style: apply Black formatting to caching files
* fix(caching): address Greptile review — cyclic import, guarded construction, nested dict merge
Move PromptTokensDetailsWrapper to inline import to resolve CodeQL cyclic
import warning. Guard PromptTokensDetailsWrapper construction with
try/except to handle unexpected cached keys. Add recursive dict merging
in _merge_prompt_tokens_details for nested fields like
cache_creation_token_details.
Drop accidental dashboard export path renames in _experimental/out, remove committed local proxy log, and remove the unintended ad-hoc test file so the feature commit only contains intentional source changes.
Made-with: Cursor
Treat search tools like models by adding team/key allowed_search_tools controls, enforcing search tool authorization checks, and moving credential ownership to search tool config only to avoid exposing secrets in team metadata.
Made-with: Cursor
Adds transform_response to OVHCloudChatConfig to normalise the new
easoning field to
easoning_content in non-streaming responses,
matching the existing streaming fix in chunk_parser.
Addresses maintainer feedback on #26595
Allow search requests to resolve provider credentials from request metadata, team metadata, and default team settings with clear precedence, and expose this flow in proxy docs/UI with regression tests.
Made-with: Cursor
Route vector store search `extra_body` into provider transformers and handle Bedrock `retrievalConfiguration` explicitly so only intended provider-specific fields are forwarded.
Made-with: Cursor
- Replace isinstance(item, dict) with hasattr(item, 'get') so Pydantic
model instances (ResponseOutputMessage, ResponseFunctionToolCall) are
accepted alongside plain dicts (P1)
- Use 'is not None' guards instead of or-chain for system_instructions
coalescing to prevent falsy values (e.g. []) falling through to the
wrong kwarg (P2)
- Emit per-tool-call span attributes (gen_ai.completion.N.function_call.*)
for Responses API function_call items, matching the choices branch
parity with _tool_calls_kv_pair (P2)
- Add 4 new tests: Pydantic-like objects, falsy fallthrough guard,
per-tool-call attribute emission, multiple tool call indexing
Build the tool_call part dict separately with an explicit type
annotation so mypy can track the type, avoiding the
'Unsupported target for indexed assignment' error on
tool_call["parts"][0]["id"].
Previously the normalize_callback_names call only ran when the existing
litellm_settings DB row already had a success_callback key. On the very
first write (no row yet, or row missing the key), incoming mixed-case
values like ["SQS", "sQs"] persisted as-is. delete_callback (lowercase
lookup) then could not find them, and a follow-up /config/update would
union normalized incoming with mixed-case stored entries, producing
duplicates.
Always normalize incoming success_callback before merging, and dedupe
both the standalone first-write case and the union-with-existing case.
Adds test_success_callback_normalized_on_first_write covering the
no-existing-row path; the existing union test still passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
convert_anyof_null_to_nullable was stripping the items field from array
branches inside anyOf when a sibling null branch was present, leaving
{"type": "array"} without items. Vertex requires items whenever
type == "array" (even inside anyOf) and rejects the call with
INVALID_ARGUMENT.
Leave the (possibly empty) items in place so the downstream process_items
step can convert {} to {"type": "object"}, which is what Vertex wants.
Also:
- Update test_build_vertex_schema expected output, which was codifying
the broken shape.
- Convert test_gemini_tool_calling_not_working to a hermetic mock test
that asserts the request body sent to Vertex includes items inside
the callbacks anyOf array branch. The previous form made a real
network call and was flaky in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes#25840
The OTel integration's set_attributes() method never populates
gen_ai.output.messages, gen_ai.system_instructions, or
gen_ai.response.finish_reasons for /v1/responses calls because
ResponsesAPIResponse uses 'output' instead of 'choices' and the
system prompt arrives as 'instructions' instead of 'system_instructions'.
Changes:
- Add elif branch for response_obj.get('output') to extract response
text from Responses API output items (type='message'/output_text)
and tool calls (type='function_call')
- Coalesce system_instructions/instructions/system kwargs so the
system prompt is captured for Responses API, Anthropic Messages
API, and Vertex AI Gemini paths
- Handle plain-string system prompts without unnecessary wrapping
- Extract response_obj.get('status') as finish reason for Responses API
- Add _transform_responses_api_output_to_otel() method
vi.clearAllMocks does not reset mockImplementation, so the error-notification
test was inadvertently relying on a deleteField stub set up in earlier tests
and would time out when run in isolation.
Previously, useStoreRequestInSpendLogs and useDeleteProxyConfigField
did not refresh the proxyConfig cache on success, so the Logging
Settings form continued to render the pre-save values until React
Query refetched on its own. Wire both hooks to invalidate
proxyConfigKeys on success so any active observer (currently the
Logging Settings page) repulls fresh data.
Export proxyConfigKeys for cross-hook reuse.
The endpoint loaded the full merged YAML+DB config and re-saved every
top-level section to LiteLLM_Config rows via save_config(), so a UI toggle
of one field persisted unrelated YAML state to DB as a side effect. It
also rejected every request when store_model_in_db was False — including
the request that would flip the flag to True (chicken-and-egg).
Replace save_config with targeted per-section upserts: read the existing
litellm_config row, merge in the request, upsert just that row. Sections
the caller did not send are not touched. Drop the blanket
store_model_in_db guard — the endpoint already requires prisma_client,
and the startup-side override at proxy_server.py:6491 picks up
general_settings.store_model_in_db=True from the DB on next restart.
Switch the spend-logs save flow from mutateAsync + try/catch to
mutate + callbacks. Errors now surface through a single onError path
(no more double toast on failure), and the delete-then-update sequencing
runs through onSettled instead of awaited promises. handleFormSubmit is
no longer async.
Tighten the corresponding test to assert exactly one error toast fires.
Adds a CLI flag (`--timeout_worker_healthcheck`, env `TIMEOUT_WORKER_HEALTHCHECK`)
that forwards to uvicorn's `timeout_worker_healthcheck` Config kwarg (added in
uvicorn 0.37.0). Lets operators raise the supervisor's worker-ping timeout above
the default 5s when triaging workers being killed and respawned under load.
The helper introspects `uvicorn.Config.__init__` and only sets the kwarg if
supported, otherwise prints a warning - so the existing uvicorn>=0.32.1,<1.0.0
floor pin is unaffected. Gunicorn and Hypercorn paths are unchanged (the uvicorn
supervisor isn't running there); the value is also not passed to the helper at
all on those paths so the "uvicorn too old" warning never fires spuriously.
* Use auth key name if there are no app id in in headers or in extra_data
* use key alias instead of key name
* Fix
* last priority key alias
* Fix
* Add tests
* [Feat] Day-0 support for GPT-5.5 and GPT-5.5 Pro (#26449)
* feat(openai): day-0 support for GPT-5.5 and GPT-5.5 Pro
Add pricing + capability entries for the new GPT-5.5 family launched by
OpenAI on 2026-04-24:
- gpt-5.5 / gpt-5.5-2026-04-23 (chat): $5/$30/$0.50 per 1M
input/output/cached input
- gpt-5.5-pro / gpt-5.5-pro-2026-04-23 (responses-only): $60/$360/$6
per 1M input/output/cached input
Other fees (long-context >272k, flex, batches, priority, cache
discounts) follow the same ratios as GPT-5.4, with context window
retained at 1.05M input / 128K output.
No transformation / classifier code changes are required:
OpenAIGPT5Config.is_model_gpt_5_4_plus_model() already matches 5.5+ via
numeric version parsing, and model registration is driven from the
JSON. The existing responses-API bridge for tools + reasoning_effort
(litellm/main.py:970) already covers gpt-5.5-pro.
Tests:
- GPT5_MODELS regression list now covers gpt-5.5-pro and dated variants
- New test_generic_cost_per_token_gpt55_pro cost-calc test
- Updated test_generic_cost_per_token_gpt55 for long-context fields
* fix(openai): mirror reasoning_effort flags onto gpt-5.5 dated variants
gpt-5.5-2026-04-23 and gpt-5.5-pro-2026-04-23 were missing the
supports_none_reasoning_effort, supports_xhigh_reasoning_effort, and
supports_minimal_reasoning_effort flags that their non-dated
counterparts define. Reasoning-effort routing in OpenAIGPT5Config is
fully capability-driven from these JSON flags — since an absent flag
is treated as False for opt-in levels (xhigh), users pinning to a
dated snapshot would silently lose xhigh support and diverge from the
base alias on logprobs + flexible temperature handling.
Copy the flags onto both dated variants so every dated snapshot
inherits the base model's reasoning-effort capability profile.
Adds a parametrized regression test that asserts
supports_{none,minimal,xhigh}_reasoning_effort parity between each
dated variant and its non-dated counterpart, preventing future drift
when new snapshots are added.
* [Feat] Add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants) (#26361)
* feat(azure): add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants)
Azure variants of OpenAI's GPT-5.5 family. Microsoft has not yet
shipped GPT-5.5 on Azure OpenAI (latest GA on the Foundry models page
is GPT-5.4 as of 2026-04-24), but adding the entries day-0 mirrors the
established precedent for azure/gpt-5.4* (which were in the cost map
before the Azure rollout) so cost tracking and capability flags work
the moment customers deploy.
Schema follows the existing azure/gpt-5.4* shape:
- Same base/long-context pricing as openai/gpt-5.5*: $5/$30 chat,
$60/$360 pro per 1M, with priority tier 2x base
- Azure variants drop the flex/batches keys (Azure has no flex tier)
but keep priority pricing, matching gpt-5.4* precedent
- mode=chat for the thinking model, mode=responses for pro
reasoning_effort capability flags mirror the OpenAI variants exactly
since Azure proxies the same API contract: minimal rejection on both
chat and pro, low/none rejection on pro. Once #26456 (which sets
supports_low_reasoning_effort + minimal=false on openai/gpt-5.5*)
lands, OpenAI and Azure flag profiles align.
Tests pin entry presence + pricing for all four Azure variants and
verify the live-API-derived reasoning_effort flags.
* test: register supports_low_reasoning_effort in cost-map JSON schema
azure/gpt-5.5-pro and azure/gpt-5.5-pro-2026-04-23 added in this branch
carry supports_low_reasoning_effort=false. The strict
'additionalProperties: false' schema in
test_aaamodel_prices_and_context_window_json_is_valid rejected the new
key. Register it alongside the other supports_*_reasoning_effort
entries.
Note: the runtime side of this flag (code that reads it) lands in
#26456. Until that PR merges the flag is inert for both Azure and
OpenAI pro entries, but having the schema accept it lets cost-map
tests pass on either merge order.
* Use sanitize deep copy style to replace deepcopy usage
* Added test checking error is not happening anymore
* Added warning log when json copy failed
* Reduce to one change
* Fix spaces
---------
Co-authored-by: Ido Lavi <ido@noma.security>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: TomAlon <tom@noma.security>
Replaces falsy or with explicit is not None check so that a valid
seconds=0.0 value is not silently dropped during field migration.
Addresses Greptile review feedback on #26595
OVHCloud is deprecating two response fields on 2026-05-11:
- reasoning_content replaced by reasoning (LLM reasoning models)
- duration replaced by seconds (Speech-to-Text models)
Adds backward-compatible support for both field names during the
transition window, preferring the new field when present and falling
back to the legacy field.
Fixes#26586
