mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-16 04:29:31 +00:00
54519 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yujonglee
|
7d79ccb7cc
|
test(rust-bridge): select native routes per test without LITELLM_RUST (#46620)
* feat(rust-bridge): scope catalog rules with a contextvar Co-Authored-By: Yujong Lee <yujong@berri.ai> * test(rust-bridge): select native integration routes with scoped rules Co-Authored-By: Yujong Lee <yujong@berri.ai> * test(rust-bridge): run the Rust extension suite without LITELLM_RUST Co-Authored-By: Yujong Lee <yujong@berri.ai> * test(rust-bridge): use scoped catalog rules in dispatch tests Co-Authored-By: Yujong Lee <yujong@berri.ai> * refactor(rust-bridge): keep catalog rules a static global and override them once per test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Mateo Wang
|
8fb6fee8c5
|
test(integration): expect signed empty thinking blocks to survive /v1/messages replay after #41362 (#46686)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
08b8bb7896
|
test(cost): add best-guess Claude and GPT-6 cache-write cases (#46687) | ||
|
devin-ai-integration[bot]
|
0a680f6821
|
test(translation): add claude-sonnet-5-5 anthropic messages service_tier case (#46678) | ||
|
devin-ai-integration[bot]
|
53cd1d4a98
|
fix(model_prices): registry audit 2026-10-15, OpenAI GPT-4 series shutdown dates, Azure gpt-chat-latest retirement, xAI voice text input, Sail DeepSeek V4.1 Flash prices (#46675) | ||
|
Taylor Hawkes
|
fd84e97f49
|
feat(providers): add Inference APIs as OpenAI-compatible provider (#41669)
* feat(providers): add Inference APIs as OpenAI-compatible provider Registers https://api.inferenceapis.com/v1 as a JSON-configured provider (chat completions and Responses API). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(providers): list Inference APIs in Add Model and the provider docs index, add request-level tests - provider_create_fields.json entry so the Add Model dropdown test passes - provider_endpoints_support.json (+ backup) entry so the provider docs check passes - tests now capture the outgoing HTTP request (URL, auth header, body) instead of only inspecting registrations - drop a redundant comment Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(providers): inject a recording HTTP session instead of patching httpx.Client Request-level tests now set litellm.client_session to an httpx.Client with a MockTransport, so nothing is monkeypatched on a class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4 * chore(providers): drop redundant comment on the inferenceapis entry Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4 * feat(inferenceapis): add cost map rows for the 12 chat models and test the responses route Prices and context windows come from https://inferenceapis.com/pricing and GET /v1/models. Moves the provider test to tests/unit and adds the interactions flag the endpoint matrix now carries. * feat(inferenceapis): price the vendor's chat model aliases and mark /v1/messages supported * fix(inferenceapis): serve /v1/messages natively and keep audio routes off the provider * fix(json providers): send provider-specific params in extra_body and report context overflow as ContextWindowExceededError * fix(exception mapping): match the Inference APIs overflow phrase instead of any context_length_exceeded substring * fix(exception mapping): map context overflow to ContextWindowExceededError for Inference APIs only Providers without their own mapper keep reporting an overflow as BadRequestError, since the router raises a ContextWindowExceededError instead of trying generic fallbacks when context_window_fallbacks has no entry for the group. Adds the integration cells for the provider, its fallbacks, and the extra_body and overflow families. * test(inferenceapis): hold upstream replies across the worker kill in the chaos cell A reply the killed worker sent before SIGKILL can lose its queued spend row, which the product never promised to keep. Holding the upstream until after the kill means only the surviving worker answers, so every 200 must log exactly once. * test(inferenceapis): make the chaos cell kill the quieter worker and always release held replies The survivor now must hold at least one in-flight request, so the burst check cannot pass as 0 == 0, and the held vendor is released on exit so a failed assertion cannot stall the wire server teardown --------- Co-authored-by: TaylorHawkes <857858+TaylorHawkes@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
구동하
|
a9e45a24d3
|
fix(streaming): keep provider response headers on SDK streaming errors (#41528)
* fix(streaming): keep provider headers on MidStreamFallbackError A 429 raised while opening a stream is wrapped in MidStreamFallbackError, which dropped the litellm_response_headers exception_type() attaches to the wrapped error. retry-after and x-request-id reached callers on the non-streaming path but not on the streaming one, so retry backoff, provider request id logging and proxy error headers lost them. * refactor(streaming): drop explanatory comment from MidStreamFallbackError header copy * test(streaming): cover sync and async header carry on MidStreamFallbackError Parametrizes the provider-header test over the sync and async stream wrappers, and drops the hasattr test, which pinned how the attribute is stored rather than what any caller observes * test(streaming): add integration cells for stream-open provider headers Cover the SDK stream openers (gemini, vertex_ai, codestral, predibase and the eager providers), the Router stream retry and fallback paths, and the proxy on both the router and the config-less serving paths, including a mixed burst, a worker SIGKILL and a SIGTERM restart. Add a helper that boots a config-less proxy owned by the test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
17f621e116
|
fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836)
* fix(vertex_ai): route Model Garden models through the generateContent adapter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vertex_ai): annotate generateContent route test parameters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): type litellm_params in get_vertex_ai_model_route * fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions Streamed Model Garden calls through the generateContent adapter now request stream usage where the deployment supports it, keep cached-token details, and look up the provider-relative cost key, so they bill like the non-stream call. The adapter forwards router metadata, passes upstream errors through with their status, countTokens falls back to local counting for non-Gemini routes, and the native Gemini config is limited to the Gemini routes. * fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing Streamed generateContent through the chat adapter now ends with the upstream usage, honors always_include_stream_usage false and the caller's stream_options, counts one failure per failed call, and routes Vertex deployments by their base_model like chat does * fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage * test(google_genai): expect the trailing usage chunk on bridged OpenAI streams The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone. * fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk * fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent * test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire * test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
08a24d391a
|
fix(count_tokens): count system and tools on Vertex Gemini, fall back to Mantle on Bedrock 404 (#46430)
* fix: align token counting with inference context Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve Vertex Gemini count message shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: forward Google countTokens tools in shapes every provider counter accepts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: add package marker for Vertex count-token unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): ignore deployment system_instruction when counting tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): forward system and tools on Google AI Studio, keep strict on Anthropic tools Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(count_tokens): ruff format gemini count handler Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): return Mapping from count request body builders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): truncate long Anthropic tool names on Vertex count, update stale strict test expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): simplify Gemini count context and tests Put the Vertex Gemini count branch back in the same shape as main and the partner model branch, keeping only the system and tools context, the provider error mapping and the gemini tokenizer default Share one error response helper between the Vertex and AI Studio callers, drop the unread kind fields and the extra validation of map_function output, and say Gemini instead of Vertex AI in the error that AI Studio users also see Tighten the tests by removing a stray parametrize on a helper, sharing the SDK count args, asserting the exact forwarded system and tools, and folding the Anthropic tool shape tests into one table * fix(count_tokens): reuse inference tool translation for Vertex Gemini count Vertex and AI Studio count now convert tools with the same functions inference uses, translate_anthropic_tools_to_openai and the Responses tool converter, before VertexGeminiConfig.map_function. Unnamed and Anthropic hosted tools now count the same declarations inference sends instead of being dropped Tools that cannot be converted, such as a non list tools value, a non object tool or a schema that is not a JSON object, now return a 400 before any request instead of being counted without them --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
18831b818c
|
fix: keep model capability flags out of provider requests (#46647)
* fix: keep model capability flags out of provider requests A supports_* flag or another model map field set on a deployment's litellm_params was forwarded to the provider as a request field, so Bedrock answered 400 Extra inputs are not permitted on chat traffic and on the background health check alike. The shared owned-kwarg predicate now treats every ProviderSpecificModelInfo field as litellm-owned. Builds on the health-check-only approach in #40952 by leilei3167. * fix: treat every supports_ model map flag as a capability field and warn when one sits in litellm_params * fix(router): skip declared litellm params in the capability flag warning and warn for runtime-added deployments * fix(router): strip model capability flags from the router's deployments instead of every request * fix(router): keep capability flags out of health probes and read them from the loaded catalog * perf(router): read the catalog's supports_ field set once per cost map load Cache the loaded catalog's field names when the cost map snapshot is taken so the capability-flag check stops walking every catalog entry per deployment, and add integration cells that drive the flagged and control deployments through chat, streaming, messages, responses, embeddings, health and model management against a vendor that rejects the flags |
||
|
Mateo Wang
|
56dad9077d
|
fix(decisions): answer 400 on a malformed provider_specific_header instead of dropping it (#46619)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
606ecf56de
|
fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals (#46633)
* fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove an OpenAI-format box answer reaches the caller as a box Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): give the box wire test a deployment that advertises supports_box_questions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Sushit Lal Pakrashy
|
958cba7016
|
fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results (#38284)
* fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results
* fix(vertex_ai): use object instead of Any in schema-ref traversal
* fix(vertex_ai): use immutable tuple traversal to satisfy type-discipline gate
* test(vertex_ai): add direct contract test for _contains_json_schema_ref
* fix(vertex_ai): type the $ref walk over JsonValue and drop the private-helper test
* fix(vertex_ai): wrap a tool result only when a $ref holds a non-empty string
Gemini rejects {"$ref": "<name>"} in function_response.response but accepts null, empty,
numeric, boolean, object, and list values, so those tool results stay structured
* perf(vertex_ai): skip the $ref walk when the tool result text cannot spell the key
* test(vertex_ai): cover a $ref on the tool result's root object
* docs(vertex_ai): cite where the Gemini $ref rule was observed
* fix(vertex_ai): keep a $ref under a list of lists structured
Gemini accepts a $ref reached through a list nested directly in a list,
so those tool results keep their structured form. Adds the audit's
integration cells for the tool result wire, chaos and edge shapes
* test(vertex_ai): teach the fake Gemini servers the list of lists rule
The audit's scripted upstreams now skip lists inside a list like Gemini
does, add list of lists cells on every endpoint, and give the 8 MiB
timing calls their own client timeout
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Mateo Wang
|
d1a7fcde8e
|
refactor(types): replace Any with proven types in 24 files (#46663)
* refactor(types): replace Any with proven types in 84 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep vector store search locals untyped Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep runtime payload validation out of the sweep Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep only annotation-only Any fixes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
452ee4dc08
|
test(caching): baseline breaker metrics from live state after gc (#46652)
* test(caching): baseline breaker metrics from live state after gc Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): collect garbage before the services logger gauge test reads live breakers --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Rick
|
71e508b65d
|
feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos (#43705)
* feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos MinimaxVideoConfig maps litellm's OpenAI Videos API onto MiniMax's V2 task API. Create, status, list, content and delete hit /v2/video_generation and /v2/query/video_generation. Remix is not exposed: MiniMax's /v2/video_regeneration only upscales a finished 768P task to 2K and ignores any prompt, so it raises instead of returning a video that silently drops the edit OpenAI params translate to MiniMax's contract: seconds becomes duration, size reduces via gcd to a valid ratio, and text-to-video requests default to ratio 16:9, resolution 768P and duration 5 when unset. input_reference becomes a first_frame content item, base64 encoded into a data URI for file inputs, and multimodal reference media passes through a content array in extra_body. Caller content is validated strictly as image, video or audio items, and the scanned prompt is always the one text item sent, so text can't reach MiniMax past the guardrails; anything else in content returns a 400. api_base shares MINIMAX_API_BASE with the chat config (a trailing /v1 or /v2 is stripped because the video API carries its own version). The list endpoint pages by number with no cursor or sort, so after and an ascending order are rejected with a 400 rather than ignored, and page_num passes through extra_query. Responses validate into frozen pydantic models at the boundary, video ids are provider-encoded so status and content route back to MiniMax, and create reports duration_seconds and video_resolution in usage for cost tracking End-to-end tests drive the real avideo_generation() entrypoint through an httpx MockTransport and assert on the encoded request body, which is what catches a mapper that drops input_reference before the request is built Both model_prices maps gain minimax/MiniMax-H3 and minimax/MiniMax-H3-Max, priced per output second by resolution tier from MiniMax's pay-as-you-go list (H3 $0.08 at 768P and $0.13 at 2K, H3-Max $0.05 at 480P and $0.08 at 768P). MiniMax also bills input images beyond a per-model free allowance (5 for H3, 2 for H3-Max), so each entry carries input_cost_per_image and provider_specific_entry.minimax_free_input_images, and the create response reports the full charge as provider_reported_cost_usd. Reference video input is rejected with a 400, because MiniMax bills its duration only after the create call that litellm bills Co-authored-by: AaronHowell <237895480@qq.com> * fix(minimax): fail closed on invalid video params and malformed task responses Addresses review on #43705: - A create, status or delete response without a task id (or a task) is a 502 instead of a queued object with an empty id - An unmapped task status is a 502 instead of defaulting to queued - seconds/duration must be a positive whole number and size a positive WIDTHxHEIGHT or W:H, otherwise 400; a 0x0 size no longer divides by zero and an unparseable duration no longer falls back to the billed default - characters and image are rejected with UnsupportedParamsError unless drop_params is set, instead of being dropped silently - Downloading a task that is not ready or failed is a 400 that keeps its message through the handler, instead of a ValueError surfaced as a retried 500 - The video download goes through safe_get / async_safe_get Co-authored-by: AaronHowell <237895480@qq.com> * test(minimax): cover the video download path and invalid-param branches The download now goes through safe_get, so the happy path and the finished-task-without-a-url branch are asserted, alongside the non-object parameters rejection and the remix response stub. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> * fix(minimax): return 400 for video remix and clear new lint gate counts * fix(minimax): send file-tuple images and keep input_reference next to content media * fix(minimax): answer 404 when an H3 video is downloaded before it finishes A download that lands while the task is still queued or running answered 400. That is a client error, which was the intent, but 404 is what the OpenAI videos API documents for content that is not ready, and it is the status the sibling DashScope and Volcengine video paths now answer, so a client polling all three reads one shape A failed task keeps its 400, since that is not a timing mistake the caller can retry away. The download test drives avideo_content and asserts the caller sees NotFoundError rather than an internal server error * fix(minimax): answer 400 for unsupported video params, early downloads and unknown ids A create param MiniMax cannot honor (seed, n, quality and the like) was dropped without a word. It now answers 400 unless drop_params is on, like other providers A download before the video is ready and a status call for an unknown task id now answer 400. The router cools a deployment down on a 404 and retries a 5xx, and neither is a deployment fault. DashScope on main answers 400 for the same case Input images past the free allowance are now billed through the shared video cost path, so a deployment with custom pricing bills them too. Status usage no longer carries MiniMax's token counts, list sets has_more to false since MiniMax pages by number, and the price map stops listing video as an input modality * fix(minimax): keep the polled video id, honor deployment drop settings, and match chat's key order Status re-encodes the id with the polled id's model when MiniMax omits it, drop_params and additional_drop_params never reach MiniMax and listed keys are dropped, MINIMAX_API_KEY wins over litellm.api_key, create sets created_at, and a content variant other than the video answers 400 * fix(minimax): list a task without a model under the routed group MiniMax leaves `model` off some listed tasks. The caller polls the ids a list hands back, and the proxy resolves a poll by the model in the id, so an id encoded with an empty model no longer routes back to the deployment that served the task. Fall back to the group the router was asked for. * fix(minimax): honor deployment drop keys inside parameters and bill images at the map rate when a deployment prices only seconds A field the deployment lists in additional_drop_params no longer reaches MiniMax when it arrives inside parameters, and a deployment that sets only its per-second price now bills reference images past the free allowance at the cost map's per-image rate instead of $0 * fix(minimax): keep deployment-dropped fields out of the body when they arrive through extra_body * fix(minimax): answer 400 for a video resolution with no price unless drop_params is on * fix(minimax): send the default resolution when a deployment drops resolution --------- Co-authored-by: AaronHowell <237895480@qq.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Happy <yesreply@happy.engineering> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Wang Yunzhuo
|
a2f792634e
|
fix: preserve explicit provider for async image, audio, and realtime routes (#36346)
* fix(images): preserve explicit provider for async generation * test(images): prove a bare model name reaches its explicit provider on the async path * fix(images): validate async provider hints instead of reading untyped kwargs Read custom_llm_provider and api_base in aimage_generation through a TypeAdapter over a small TypedDict, so the provider lookup gets typed values and the basedpyright gate stays within its merge-base ceiling * fix: answer bad provider hints with a 400 and keep the explicit provider for async speech and transcription * fix: keep the explicit provider for realtime and cover the audio bad-hint 400 * fix: keep the explicit provider on the realtime HTTP routes and read base_url in async audio * fix: stop async image generation from rejecting a base_url it never reads * fix(audio): skip base_url validation when api_base is set on async speech and transcription * test(integration): cover a declared provider with a bare model on image, audio, and realtime routes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
d5531234ce
|
test(ci): pin xai reported-cost split and coordinator-held Prisma lock timeout (#46645)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c00706218b
|
fix(vertex_ai): return 400 with file line number for malformed batch JSONL row (#40097)
* fix(vertex_ai): return 400 with file line number for malformed batch JSONL row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(vertex_ai): keep yield outside the JSON decode guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): surface BaseLLMException raised while streaming a request body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(http_handler): only follow __cause__ when unwrapping streamed body errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): avoid recursive streamed error unwrapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows Keep main's error path for a directly raised provider error and only unwrap one found behind a transport error. Add the parser column to invalid JSON errors, and cover the real aiohttp chain in the _handle_error test * fix(vertex_ai): report every row conversion error as a 400 naming its line A message with no role, a non-string role, a bad temperature, or a litellm BadRequestError raised while converting a row still returned a 500, and the async upload lost the 400 for litellm's own exceptions. The stream now turns any error from one row's conversion into a 400 naming that line. _handle_error only unwraps an httpx.TransportError, which is what the aiohttp transport raises around a failing request body, so other callers keep their own status. A file that starts with a UTF-8 byte order mark now uploads, and provider messages start with 'Batch input file line N' like the proxy check. * fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line A chat row whose body is missing or not an object now raises a 400 naming its line instead of uploading a blank prompt, and a non-object body on line 1 no longer crashes the GCS object name lookup. Rows too deeply nested or too long to parse return a 400 from both the SDK and the proxy's own batch file check. JSON error columns now count leading whitespace. * fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line * test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos * fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact * test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Raushan kumar
|
f9d45f8a43
|
fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)
* fix(llms): sanitize trailing slashes in provider get_complete_url * test(llms): move trailing-slash url tests into each provider's mapped unit file * fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI * fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders * fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests * fix(gigachat): strip a trailing slash from the pass-through api_base * test(integration): cover trailing-slash api_base across provider URL builders Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kolade Fajimi
|
35970c3eac
|
fix(vertex): report an unreadable credentials file as a read failure (#40173)
* fix(vertex): report an unreadable credentials file as a read failure Tell a path apart from inline JSON by shape, not os.path.exists(). Fixes #40166 * test(vertex): narrow pytest.raises to the expected message * fix(vertex): log the credentials path instead of returning it to the caller * refactor(vertex): log the credential source decision where it is made * fix(vertex): handle paths open() rejects with a bare ValueError * fix(vertex): redact the credentials path before it reaches the log * test(vertex): cover the brace fallback and non-text credentials file * test(vertex): exclude the unreachable exhaustiveness guard from coverage * refactor(vertex): read the failure's fields off the matched object CodeQL's py/uninitialized-local-variable does not follow keyword capture patterns in a match statement and reported `path` and `reason` as possibly unbound. Narrowing on the class and reading the attributes says the same thing without a suppression. * fix(vertex): accept a read-only mapping in extract_aws_params * fix(vertex): keep encoded credentials out of the log and cover the legacy route * fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust * fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file * fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message * test(vertex): integration cells for credentials file and inline value failures * test(vertex): drive the Rust messages cell through Vertex Claude --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ErenAta16
|
bc58a5857b
|
fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema (#34653)
* fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema `translate_anthropic_tools_to_openai` assigned the caller's `input_schema` straight into `function_chunk["parameters"]`, then merged the tool's remaining top-level keys into that same dict. Because the assignment aliased rather than copied, keys such as `display_width_px` on a computer tool were written back into the caller's schema. That matters because callers reuse the tool list across passes: the pass-through guardrail translates the tools, and the request is then translated again for the real call. The first pass pollutes the source and the second forwards a schema carrying non-schema keys. Shallow-copying fixes it. `translate_anthropic_output_format_to_openai` in the same file already copies its schema for this reason, so this brings the two into line. Rebased onto current litellm_internal_staging; the original branch had gone to conflict on surrounding drift while the line it fixes is unchanged. * fix(anthropic/adapter): drop the inline comment and test the strict tool shape that Anthropic rejects * fix(anthropic/adapter): deep copy input_schema so downstream schema edits never reach the caller's tool A shallow copy still shared nested properties with the caller, and the Gemini, vLLM and watsonx bridges strip nested additionalProperties in place, so a fallback to Claude sent a strict tool Anthropic rejects. A deep copy also keeps a null or malformed input_schema as sent, as before this PR * fix(anthropic/adapter): deep copy the incoming tools so downstream edits never reach the caller's tools Copying only input_schema left the extra keys merged into parameters, such as input_examples, and OpenAI-format function tools shared with the caller, so Gemini's strict strip still edited the caller's tool before a fallback to Claude * fix(anthropic/adapter): keep translating tools that deepcopy cannot copy A tool passed as a frozen mapping, or with a schema nested past Python's recursion limit, made copy.deepcopy raise, so the request failed before it reached the provider. Copy each tool on its own and keep the original object when the copy fails, which is how these tools were handled before * fix(anthropic/adapter): copy tools through JSON so the Responses bridge and frozen tools keep the caller's schema * test(anthropic/adapter): parse stub bodies with json and cover the tool copy fallback * test(anthropic/adapter): close the connection after the malformed tools request --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Oliver Jensen
|
84e8204d7f
|
fix(proxy): scope /spend/tags aggregates to the caller's own and permitted-team logs (#46194)
* fix(proxy): scope /spend/tags aggregates to the caller's own and permitted-team logs Non-admin callers previously received the proxy-wide per-tag spend aggregation. The endpoint now resolves the same read scope as /spend/logs/ui and /spend/logs/session/ui, and the query honors the start_date/end_date parameters it previously accepted but ignored. * fix(proxy): use Annotated auth dependency on /spend/tags and sync dashboard API types * fix(proxy): bind /spend/tags dates as UTC datetimes, not session-timezone strings Raw YYYY-MM-DD strings cast to timestamptz are read in the database session timezone, shifting the day window by that offset. Parse them to UTC-aware datetimes first, like every sibling spend query in this file, and reject malformed dates with a 400 before they reach the database. |
||
|
Kolade Fajimi
|
ea777eb21b
|
fix(auth): return a generic 401 for an unknown key (#39787)
* fix(auth): return a generic 401 for an unknown key The wrong-key 401 shipped the internal verification table name and the submitted key's full SHA-256 hash to any unauthenticated caller. Both now go to the proxy's own log and the response carries only "Invalid API key provided." The Admin UI detected a dead session by string-matching that message, so it now reads the structured error type instead. * Update ui/litellm-dashboard/src/lib/http/client.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(auth): log the missing key hash where every resolve path passes Review catch: JWT key-mapping misses resolve through IdentityStore without touching the main key path, so logging the hash there left those callers with nothing but the generic 401. Log it at the store's raise site instead, which every resolve path shares, and drop the now-duplicate log. * test(auth): inject a key-less client instead of patching the DB fetch The store test tripped TQ008 by patching _fetch_key_object_from_db_with_reconnect. Hand IdentityStore a client whose get_data returns no row instead, so the miss is driven through the real lookup rather than around it. * fix(auth): log the unknown key's hash once on the auth failure line The per-lookup warnings in get_key_object and IdentityStore logged whatever value they were handed, which on the failure-logging path is the raw key a caller sent (a provider key included), let a newline in ?key= forge log lines, and fired on lookups that expect a miss. The hash now rides KeyNotFoundError onto the single existing auth failure log line next to the requester IP, with control characters escaped. Also drops the new comments and updates skipped tests that quoted the old message. * test(auth): add the blank lines ruff format expects before _InFlightCountingPrisma * test(integration): expect the generic wrong-key 401 in the management lifecycle tests * fix(auth): keep the operator wording on the unknown-key log line and drop the redundant 401 rewrite * test(auth): cover the generic unknown-key 401 across routes, proxies, and a worker kill * test(auth): cover the realtime upgrade refusing an unknown key without echoing it --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
bf56719f09
|
fix: cap cold MCP schema workers at usable CPUs and deflake wall-clock unit tests (#45791)
* test: inject clocks and wait on conditions in three wall-clock-flaky unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: keep generic API logger tests on 3.10 and cover automatic sending Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: pin router usage minute in shared rate-limit header test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: stabilize three wall-clock-sensitive unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): cap cold schema validation workers at the CPUs the proxy can use Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: inject the cache clock and wait on lock contention instead of 1s deadlines Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): wait on listing events instead of 1s deadlines in the discovery drain test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> |
||
|
Hayden Moulds
|
de0ed1557e
|
fix(proxy): resolve model aliases to their target deployment for listing metadata (#41041)
* fix(proxy): resolve model aliases to their target deployment for listing metadata A model alias (a key's `aliases`, a team's `model_aliases`, or the router's `model_group_alias`) is rewritten at request time and owns no deployment row, so `get_model_listing_info` returns None for it and the alias name is not a cost-map key either. Both metadata candidate sources come up empty and the listing entry carries no `mode`, `max_input_tokens`, or `max_output_tokens`, while the deployment it routes to reports them on the same response. Resolve a listed alias to the deployment it routes to for the metadata lookup, keeping the alias as the response id. Maps are applied in order, each against the result of the last, mirroring how `litellm_pre_call_utils` applies the team map and then the key map, so a chained alias reports the deployment the caller actually reaches rather than an intermediate one. An alias shadowing a same-named deployment follows the alias, because the request rewrite does too. Fixes the model_aliases half of #40553. * fix(proxy): look listed aliases up through the shared alias chain Drop the separate alias resolver and retarget a listed alias in alias_listing_entries and alias_target instead, so /v1/models and /v1/models/{id} report the model /chat/completions rewrites it to, and only when the caller can list that model * fix(proxy): keep deployed listed models' own rows and resolve team alias targets at list and retrieval * fix(proxy): resolve router_settings aliases, keep alias labels, and skip paused targets in model listing * fix(proxy): leave an alias to an undiscoverable model bare in model listing A team alias whose target has discoverable: false showed the hidden model's mode and limits on GET /v1/models and retrieved it by alias. Undiscoverable groups now count as hidden for alias resolution, as on main, while admins still see them. * fix(proxy): leave an alias to a model a listing callback hides bare in model listing * fix(proxy): keep aliases bare for models a listing callback never judged or another team owns --------- Co-authored-by: haydster7 <haydster7@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
3cc9e7ef8e
|
perf(proxy): cache the public provider and agent field endpoints (#46617)
/public/providers/fields re-read and re-parsed the bundled 85K provider_create_fields.json on every request, and /public/agents/fields did the same for that file plus agent_create_fields.json. Both are unauthenticated public endpoints, and both did the work inside an `async def`, so the read and the parse ran on the event loop and serialized against every other in-flight request. Measured against the bundled files, that is roughly 0.40 ms of blocking work per request for the provider endpoint and 0.45 ms for the agent endpoint Both now memoize with `lru_cache`, which is what `/public/endpoints` in the same module already does through its `_cached_endpoints` global, so each file is read once per process rather than once per request The agent endpoint mutated the dicts it had just loaded, appending inherited provider credential fields and popping `inherit_credentials_from_provider`. That is only safe while the data is re-read every time, so caching it meant making the merge pure: `_agent_with_inherited_credentials` builds a new mapping per agent, and the cached value is the merged result rather than the raw file contents. Responses are byte identical before and after, checked on both endpoints against a live proxy The bundled records are typed as Mapping[str, object] and narrowed with isinstance at the point of use, rather than typing.Any, which ruff-strict bans Side effects: 3 of the repo's ASYNC230 violations go away, and basedpyright errors in the file drop from 63 to 52 Rebased onto main after /public/providers/fields started validating the bundled JSON through a pydantic adapter on every request. That validation now runs once and the validated records are cached; the agent endpoint keeps merging the raw provider records, so both responses stay byte identical to main. Co-authored-by: Alok Sharma <alok44170@gmail.com> |
||
|
Kerry Lu
|
4496fe0ebc
|
test: cover response contracts across translation endpoints (#46632)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
22a8cd0cb7
|
test(integration): cache the two-turn tool_search guardrail replays (#46614)
* test(integration): cache the two-turn guardrail replays and derive the thinking pass-through timeout * test(integration): keep the thinking pass-through timeout as on main The derived timeout had no failing run behind it, so this PR now only fixes the guardrail cache cells |
||
|
devin-ai-integration[bot]
|
f08840579e
|
test(cost): add tier 1 gpt cache_read cases and recapture input_text usage shapes (#46623)
Add live-captured cache-read fixtures and align replay usage blocks with current provider responses Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chloe Lu
|
897bfea38d
|
fix(anthropic): keep signed empty thinking blocks on /v1/messages requests (#41362)
The request-side sanitizer on /v1/messages dropped every thinking block whose text is empty, even with a valid signature. Claude Opus 4.8 / Fable 5 / Fable 5.1 return exactly that shape by default (thinking.display omitted): the signature is the preserved reasoning. Replaying the assistant turn as returned therefore lost the model's reasoning on every multi-turn request, silently (the provider answers 200 without it), and left preserved-thinking enforcement (thinking.block_binding) nothing to bind. Anthropic accepts an empty thinking block that carries a signature and binds it to its prefix; only an empty block with no signature (the LIT-6357 case, a turn a non-Anthropic model served through the bridge) is rejected with "each thinking block must contain thinking". Use the signature-aware predicate is_empty_unsigned_thinking_block, which #38809 already applied on the emit side, for the request side too. The unsigned case is still dropped. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
TheMentalCoding
|
ca04c4909e
|
feat(ui): filter active and paused model deployments (#43710)
* feat(ui): status filter + sort and provider-aware search on Models & Endpoints
The models table mixes active and paused deployments with no way to tell
them apart beyond the pause toggle, and the drawer only filters by public
model name and access group.
Backend (/v2/model/info):
- new optional `blocked` query param to filter by routing status
- new `blocked` sort field (asc = active first); the existing `status`
field keeps sorting by config-vs-DB source
- `search` now also matches litellm_params.model, so typing a provider
or upstream model id (e.g. "nvidia_nim", "openai/gpt-4") finds
deployments whose public name doesn't mention them
Frontend (Models & Endpoints):
- visible Status column with Active/Paused badges, sortable
(active first on asc)
- Status filter (All/Active/Paused) in the Filters drawer, wired to the
new server param through URL state
The routing-status filter ignores non-bool sentinels the same way
exclude_auto_routers does, so direct calls bypassing FastAPI keep their
no-filter behavior.
* fix(ui): inverted routing-status filter mapping and compact Status column
Selecting "Active" in the Filters drawer sent blocked=true to the server,
showing the paused deployments instead of the active ones (and vice
versa). The mapping now lives in routingStatusToBlocked() with unit
tests, and the Status column is narrower so the row actions stay in
view.
* fix(ui): pin the Actions column to the right on the models table
With the new Status column the table overflows on narrower screens and
the pause/resume switches (last column) scroll out of view. Pinning the
Actions column keeps them visible; uses the existing DataTable
meta.pinned support (same pattern as MemberTable).
* refactor: readable db search where-condition and ruff-format clean
* fix: apply routing-status filter inside the search flow and regen API types
Greptile flagged (P1): with search + a status filter combined, the
bounded DB fetch ran before status filtering, so matches of the other
status could consume the page budget and leave short pages / stale
totals.
- extract _matches_routing_status() (sentinel-safe) and apply it to the
router-side matches inside _apply_search_filter_to_models
- push the status condition into the DB where clause of
_fetch_db_models_for_search, so rows of the other status never count
against the bounded fetch
- endpoint passes blocked through to the search flow; tests cover the
combined search+status case and the generated where clause
- regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts for the new
query param
- update useModels.test.ts expectations for the new trailing argument
* fix(proxy): keep the flat DB where shape for single-condition searches
The upstream CI test
test_apply_search_filter_honours_exact_model_name_in_db_query pins the
Prisma where shape: exact model_name stays a top-level key, and plain
substring search keeps {model_name: {contains, mode}} at the top level.
Only genuinely multi-condition queries (status filter and/or router-id
exclusion) now wrap in AND.
* fix(proxy): avoid reassigning a Final variable in the db where builder
basedpyright's reportGeneralTypeIssues budget counts the reassignment of
the Final-annotated db_where_condition; express it as a single
conditional expression instead.
* chore(deps): bump pypdf to 6.19.0 to clear osv-scan
pypdf 6.16.2 has 8 High CVEs (GHSA-5jq2-8x83-x246 and aliases) fixed in
6.17.0-6.19.0; bump past all of them
* Revert "chore(deps): bump pypdf to 6.19.0 to clear osv-scan"
This reverts commit 0e2528123db7ab23304ae20f6a4c0961d0e2d073.
* fix(proxy): keep LIT002 budget flat in the model search where-builder
Extract the dynamic Prisma where-condition fragments into module-level
helpers so every dict starts on a short line, split the litellm_params
fallback lookup into a Final local, and annotate suppressions on the
construct start lines the checker attributes
* chore(tests): repoint unbounded-IN baseline at the extracted where-builder helper
The bounded-router-id filter moved from _fetch_db_models_for_search into
_not_in_router_ids, so the old baseline key is stale and the new helper
site is the finding to record
* fix(proxy): avoid searching encrypted deployment model names
* refactor(ui): keep model routing status as a filter only
* docs(api): regenerate model info sorting description
* fix(api): remove remaining paused status sort references
---------
Co-authored-by: thementalcoding <312162298+thementalcoding@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Kerry Lu
|
1a9fa70951
|
test: compare coding-agent response contracts across model tiers (#46597)
* test: compare coding-agent response contracts across model tiers * test: reuse translation cases for coding-agent contracts * test: name response compatibility group cross_model * test: strengthen cross-model structure and stream preservation checks * test: compare complete normalized streaming usage * test: enable Haiku SDK stream checks after upstream fix --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
a9d6123a43
|
test: expect the list-position index on streamed thinking blocks in the integration suite (#46618)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
aa694f316d
|
fix(proxy): keep JSON nulls as null in audit log values (#46577)
LiteLLM_AuditLogs masked before_value and updated_values with
SensitiveDataMasker.mask_dict, which stringifies every non-native leaf, so a
null in the change was stored as the string "None" (or "" under a key-named
field). Walk audit values with mask_credentials_in_payload instead, injecting
the existing {"key"} masker so the masked format and scope stay the same.
|
||
|
Mihidum
|
b025681bef
|
fix(proxy): gzip responses over 16 KiB off the event loop (#46362)
* fix(proxy): gzip responses over 16 KiB off the event loop GZipBufferedResponseMiddleware compressed every body under 1 MiB inline. gzip.compress holds the event loop for its whole run, and its cost follows how well the body compresses: an 824 KB base64 embeddings response (100 inputs, the OpenAI SDK default encoding) takes about 165 ms, so every other request on the worker waits that long, and with several in flight the compressions run back to back. Lower OFF_LOOP_SIZE_BYTES to 16 KiB so only bodies that compress in well under a millisecond stay on the loop. zlib releases the GIL, so the loop keeps serving while a worker thread compresses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): pin the gzip off-loop boundary at exactly OFF_LOOP_SIZE_BYTES A body of exactly OFF_LOOP_SIZE_BYTES compresses in a worker thread, as the `<` in _compress implies. Suggested in review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): pin an embeddings-sized gzip body off the event loop and drop the source comment The off-loop test was relative to OFF_LOOP_SIZE_BYTES, so it still passed with the old 1 MiB line. An 824 KB body, the size of the 100-input base64 embeddings response in the issue, now has to compress in a worker thread. The recorder no longer mutates a list, and the threshold rationale stays in the commit history instead of a source comment. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
3fec5075fa
|
feat(typesafe): support Jev 1.14 images, box questions, and long-context headers (#46596)
* feat(typesafe): support Jev 1.14 images, box questions, and long-context headers * fix(cost): drop the typesafe/jev-1.14.0 entry main already prices * fix(typesafe): send images only with the images header and box questions only to models that answer them TypeSafe takes assets and input_image parts only when the request carries TypeSafe-Danger-Zone-Images: true, so without it they follow drop_params like any other provider that cannot take images. Box support now comes from a supports_box_questions cost-map flag set on jev-1.14.0 and jev-preview, so jev-1.13.0 and jev-latest refuse or drop a box question before the call. A malformed provider_specific_header answers 400 instead of 500. * fix(typesafe): allow null box confidence and validate asset references and base64 * fix(typesafe): gate box questions on the images header, drop unreadable assets under drop_params, forward opt-in headers on the pass-through --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
efd8274315
|
feat(xai): bill grok voice realtime sessions (#46600)
* feat(xai): bill grok voice realtime sessions * chore: revert unrelated formatting churn in test_cost_calculator.py * refactor(xai): tidy realtime usage meter and tests * fix(xai): read push-to-talk mode from client session.update only * fix(xai): clamp decoded audio bytes and lock push-to-talk before content --------- Co-authored-by: kerry <kerry@berri.ai> |
||
|
Kerry Lu
|
0d741a9816
|
fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them (#46611)
* fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them * fix: satisfy the type discipline and basedpyright gates for the thinking block index * chore: sync the OpenAPI snapshot and schema.d.ts with the thinking block index field --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
9444f53300
|
fix(proxy): store list-valued audit log changes as JSON text (#46555)
LiteLLM_AuditLogs parses before_value and updated_values as JSON but only re-serialized dicts, so a list change reached Prisma as a Python list and the row write failed. Every deployment warning ignore, snooze and restore hit this. Mask and re-serialize any JSON value instead |
||
|
Mateo Wang
|
d6e6c6f26c
|
fix(responses): map Anthropic tool search to OpenAI server tool search (#46441)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models * fix(responses): define the tool_search_call output item locally for older openai SDKs * fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500 * fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out A replayed Claude tool search call that the caller answered with a tool message kept its tool_result in the user turn while its tool_use was dropped from the assistant turn, so Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when no tool message answers it. The Responses bridge now strips defer_loading from every tool whenever no tool search tool is sent, whether the caller sent none or the route cannot run it, since a deferred tool with no search tool can never be loaded. Namespace members had started carrying the flag to every bridge provider, which providers that reject unknown tool keys answered 422. The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit. * fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried * fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator * fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic A function tool whose allowed_callers is not a list of strings now answers 400 instead of 500. The guardrail scan reads tools without malformed flags, so the route decides between a 400 and a drop under drop_params. When the caller sends a native Anthropic search tool beside tool_search, only the native one goes out. Databricks Claude no longer counts as an Anthropic request route. A previous_response_id follow-up off Anthropic also drops a search call the client replayed as a function_call with a srvtoolu_ id, and its output. The answered tool id scan in the Anthropic message factory runs once per assistant run instead of once per message. A null created_by on tool_search_call and tool_search_output output items is left out of the serialized response, so a cache hit returns the same keys as the first call. * fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan The scan translated tool_search into Anthropic's regex search tool on every route, so a guardrail returning only function tools removed tool_search from native OpenAI requests and OpenAI answered 400 for the deferred tools * fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai * fix(anthropic): keep answered tool search replays paired with their answers * fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails * fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object * fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent * fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id * fix(responses): answer 400 to a tool_search execution that is not server or client OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed execution is dropped like the other malformed tool flags, which leaves a server search * test(responses): cover the 400 and drop_params paths for a malformed tool_search execution * fix(responses): map Anthropic tool search to OpenAI server tool search Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): sync generated tool search API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): type tool search mapping predicates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): validate tool-search mappings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): use the openai SDK ToolSearchToolParam for the mapped tool search * fix(responses): keep deferred chat tools beside a tool search on the responses bridge Restore the hand-rolled OpenAIToolSearchToolParam, since openai 2.20.0 (the supported floor) has no ToolSearchToolParam, and carry defer_loading from chat function tools onto the Responses request when a tool search tool is present, which OpenAI requires for tool_search * fix(responses): leave the bridge's defer_loading carry to its own PR #46439 owns keeping defer_loading on chat function tools sent through the Responses bridge, so this PR keeps only the tool search mapping and the openai 2.20.0 compatible tool type * fix(openai): keep tool search mapping to the OpenAI API and drop it where the model or history cannot take it * fix(openai): narrow the tool search drop to Anthropic's search tool, drop it when nothing is deferred, and run the wire tests against the OpenAI host * fix(openai): pass malformed tools to OpenAI as sent and keep a caller's own tool_search under drop_params * perf(openai): scan Responses history for Claude search calls only when Anthropic's search tool is sent * test(responses): cover the Anthropic tool search mapping to OpenAI Responses across routes, settings, and chaos * test(integration): define the fallback cell's deployments in the gateway config so both workers load them * test(integration): keep the worker-kill burst held past the replacement wait * fix(responses bridge): count an Anthropic search tool as hosted tool search on the chat bridge main's #46439 refuses defer_loading when no tool_search tool is in the request, and the chat bridge only counted {"type": "tool_search"}, so the Anthropic bm25 or regex tool this PR maps to OpenAI's tool search was refused before the mapping ran Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect the chat bridge to map the search tool now that defer_loading survives --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
berriai-litellm-provider-info-sync[bot]
|
2211fa5f3f
|
chore(prices): add xAI grok-voice-3.0, grok-voice-2.0 and grok-voice-latest realtime rows (#46602)
Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
d0fa05076d
|
fix(xai): name a completed xAI batch's results file and stop looking it up on xAI's files API (#45559)
* fix(xai): report a completed xAI batch's results as its output file details * fix(xai): keep a short results download and per-call timeouts out of the reported file details * fix(xai): describe a batch's results file from the batch alone and let the batch job fill its size Reading an xAI batch id as a file now costs one batch read and never downloads the results: the file reports a 0 byte placeholder, the results filename and the registration time. The managed proxy fills in the size the batch cost job measures when it downloads the results, keeps a known size when a provider reports none, and drops the per-process results download cache. * fix(managed_files): keep a measured batch output size when a refresh races the batch job A refresh that read the row before the batch cost job stored the measured size could save the provider's unsized details over it. The refresh now reads the row back from the writer and saves only while the row still holds what it read, retrying when another write landed in between. The new tests drive the real xAI and OpenAI retrieve paths through a fake HTTP upstream. * fix(managed_files): guard the refresh save with the row the request already read The conditional save first compares against the row the request read, so a refresh adds no database read unless another write landed in between; only then does it read the row back from the writer and try again. * fix(managed_files): save a refreshed file object with one guarded write and no extra database reads A refresh that loses its guarded write to a newer one now leaves that row alone and drops this worker's cached copy, so the next read loads the newer row instead of retrying against the database on the request path * fix(managed_files): retry the batch cost job's measured size after a read refreshes the row first The batch cost job re-reads the row from the writer and saves again when its guarded write loses to a concurrent refresh, so its measured size is never dropped. Reads without a measured size still make one write and no re-read * fix(managed_files): size a concurrently refreshed row instead of restoring the batch job's placeholder copy When the batch cost job loses its guarded write and the row it re-reads holds refreshed details, it applies its measured size to those details, so the results filename and any size the provider reported are kept * fix(managed_files): keep a batch output's earlier created_at when a refresh replaces its placeholder A refresh saved over a stored row now keeps the earlier of the two creation times, so an xAI placeholder keeps the time the proxy registered it instead of the read time, while a provider's own earlier creation time still wins * test(integration): an xAI placeholder row keeps its registration time after the proxy refreshes it * test(integration): drop the cached output row so the placeholder cell reads the seeded database row * test(integration): drop the cached output row without requiring it to still exist * fix(managed_files): limit the batch output size fill, guarded save and kept created_at to xAI rows * fix(xai): keep the batch id as the batch output filename Managed xAI batch output files keep the filename main already returns (the batch id) instead of renaming them to <batch id>_results.jsonl. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
c0663d5202
|
fix(anthropic)!: translate the OpenAI tool_search tool on /v1/chat/completions for Claude models (#46389)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models
* fix(responses): define the tool_search_call output item locally for older openai SDKs
* fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields
Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500
* fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out
A replayed Claude tool search call that the caller answered with a tool message kept its
tool_result in the user turn while its tool_use was dropped from the assistant turn, so
Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when
no tool message answers it.
The Responses bridge now strips defer_loading from every tool whenever no tool search tool is
sent, whether the caller sent none or the route cannot run it, since a deferred tool with no
search tool can never be loaded. Namespace members had started carrying the flag to every
bridge provider, which providers that reject unknown tool keys answered 422.
The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit.
* fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact
A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried
* fix(anthropic): translate the OpenAI tool_search tool on /v1/chat/completions for Claude models
* fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups
With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does
A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result
The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator
* fix(bedrock): answer 400 for the OpenAI tool_search tool on Converse unless drop_params is set
* fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic
A function tool whose allowed_callers is not a list of strings now answers 400
instead of 500. The guardrail scan reads tools without malformed flags, so the
route decides between a 400 and a drop under drop_params. When the caller sends
a native Anthropic search tool beside tool_search, only the native one goes out.
Databricks Claude no longer counts as an Anthropic request route.
A previous_response_id follow-up off Anthropic also drops a search call the
client replayed as a function_call with a srvtoolu_ id, and its output. The
answered tool id scan in the Anthropic message factory runs once per assistant
run instead of once per message. A null created_by on tool_search_call and
tool_search_output output items is left out of the serialized response, so a
cache hit returns the same keys as the first call.
* fix(anthropic): send one search tool and judge tool search on the real model for chat tool_search
Several OpenAI tool_search items now collapse into one tool_search_tool_regex
tool, since Anthropic answers 400 when two tools share a name. A native
Anthropic search tool the caller already sent wins over the OpenAI item.
Bedrock invoke and Vertex AI now decide tool search support on the real model
before tool-based response_format swaps in a stub model name, so a Claude 5.5
request with response_format no longer gets a 400 on a route that runs tool
search.
* fix(anthropic): name the proxy drop_params setting in the tool_search 400
* fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan
The scan translated tool_search into Anthropic's regex search tool on every
route, so a guardrail returning only function tools removed tool_search from
native OpenAI requests and OpenAI answered 400 for the deferred tools
* fix(anthropic): run tool search on azure_ai, gate it on Databricks Claude, and strip deferral without recursion
Anthropic's Foundry docs list no tool search exclusion for either hosting
option, so the claude-tool-search rule now fills azure_ai. Databricks Claude
models route their tools through the same unsupported-tool-search gate, so a
tool_search tool answers 400 or drops under drop_params instead of a 500.
_without_deferral strips two fixed levels instead of walking recursively.
* fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai
* fix(anthropic): keep answered tool search replays paired with their answers
* fix(anthropic): drop Claude tool search calls from history sent to other routes, name the fix in the 400
A second turn that carries Claude's srvtoolu_ search call and its result now has both removed
before completion() sends it to a route that does not build an Anthropic request, including the
responses bridge. The tool search 400 now names model_info supports_tool_search for a model the
cost map does not know, and bedrock/invoke for a Converse model that runs tool search there.
* fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails
* fix(chat): keep answered Claude tool search calls in history sent to other providers
Only a srvtoolu_ search call with no tool result is dropped before a
non-Anthropic route, so an answered pair keeps the caller's shape where
OpenAI and Gemini take it. The history check no longer copies the list
through a TypeAdapter.
* fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object
* fix(chat): drop unanswered Claude tool search calls only on routes that reject them
OpenAI, Azure OpenAI and Bedrock Converse answer 400 for a srvtoolu_ call
with no tool result, while Gemini answers 200. Keep the history as sent
everywhere else so routes that already work see what the caller sent.
* fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent
* test(integration): cover chat tool_search routing, history drop and load on the wire
Audit round 1 of 3 at
|
||
|
Mateo Wang
|
52fc37888b
|
fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444)
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): give the defer_loading casts a cast-ok reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): read replayed tool-search and namespace fields through typed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep tool_search items on a turn with no message or tool call Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep every reasoning item on a search-only turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): leave content unset on a search-only turn so replay adds no empty message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): replay tool search items only with tool_search, count replayed tools in the allowlist, keep namespace off guardrails * fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses Under drop_params, defer_loading is dropped when no tool_search tool is sent, so the 400 it would cause upstream goes away as before. A non-bool value answers 400 without drop_params and is dropped with it. A null tool-level value falls back to the nested function value. A request carrying a tool_search tool now bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort none, since chat completions rejects the tool_search type. * fix(responses bridge): replay tool search items in output order and count replayed tools in tool policy Store the ordered output prefix through the last tool search item, reasoning included, so a second reasoning item no longer makes OpenAI reject the replay. The shared chat-to-Responses converter now replays tool search items and namespace only when the bridge opts in, so Noma, the complexity router and guardrail payloads keep their shape. The tool policy guardrail and the allowlist count replayed tools only when the request can send them. * fix(responses bridge): forward defer_loading only beside a tool_search tool defer_loading only means something with tool_search, and backends without it reject the key (xAI answers 403). A true value with no tool_search tool now answers 400 naming drop_params and is dropped under drop_params; a false value there is a no-op and is dropped. * fix(tool_loop): keep tool search items and namespace between rounds * fix(responses bridge): check replayed tool search loads when a deployment adds tool_search * fix(chat): drop litellm response fields before strict OpenAI-compatible providers A chat reply from the Responses bridge carries provider_specific_fields on the message and on each tool call, plus reasoning_items. Groq, Fireworks, Mistral and Together reject unknown fields, so sending that reply back to them as history failed with a 400 or 422. Strip them in the shared helper and use it in those providers. * fix(caching): keep reasoning_items and tool search items on response cache hits A cached bridge response lost reasoning_items on a non-streaming hit and lost both reasoning_items and provider_specific_fields (tool_search_items) on a streamed hit, so the next turn could not replay the search or the reasoning around it * test(integration): cover tool search item replay on the chat to Responses bridge Proxy, SDK, authorization and chaos cells for returning and replaying tool_search_call and tool_search_output items and the tool call namespace * test(integration): add response cache twins for the tool search guardrail cells Each guardrail cell now has a twin where the answer is already in the Redis response cache: a key opted out of global guardrails caches a replay that loads a policy-blocked tool, the same replay without guardrails is cached before block mode sees it, rewrite mode is not served the unrewritten conversation's cached answer, and an allowed replay is served from the cache. * fix(responses bridge): forward defer_loading beside a client-executed tool_search The bridge now replays a client tool_search_call and the client's tool_search_output through provider_specific_fields.tool_search_items, so a deferred function beside a client tool_search keeps defer_loading like it does beside a hosted one * test(responses bridge): pin the client tool_search turn and its replay, streamed and not * fix(responses bridge): strip nested namespace for strict providers and keep streamed reasoning_items on text replies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep dropping defer_loading beside a client-executed tool_search Forwarding defer_loading beside a client tool_search made GPT stop with a search call the chat client has to answer by hand, so a plain question that used to get tool calls got none. Keep main's behavior of dropping it there. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
1311894845
|
refactor: clean up fresh tech debt in per-user Copilot auth, Ternary FOCUS export and telemetry (#45769)
* refactor: read typed credential fields directly in per-user Copilot auth
- github_copilot_auth_mode reads credential.credential_values directly instead of getattr+isinstance+cast
- _new_async_github_client uses plain attribute access for get_async_httpx_client
- _per_user_oauth_configured and _all_per_user_credential_names iterate typed CredentialItem fields
- list_user_connections builds login/connected_at maps in one shot instead of seeded mutable dicts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: read typed fields directly in team member budget auth and pass-through route registration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "refactor: read typed fields directly in team member budget auth and pass-through route registration"
This reverts commit
|
||
|
Nate Armstrong
|
23025c92ba
|
fix(anthropic): emit one tool_use block per call when a /v1/messages stream chunk carries several (#46492)
Providers like Mistral put every parallel tool call, with full arguments, into one
chat-completions chunk. The /v1/messages adapter opened one tool_use block for the
first call and concatenated every call's arguments into it, so clients got invalid
JSON like {...}{...}. _CombinedChunkSplitter now splits such a chunk into one chunk
per call. The first call streams live and calls opened behind it are held, gathering
their fragments by index, until a non-fragment chunk arrives or the stream ends
Fixes #44029
|
||
|
devin-ai-integration[bot]
|
e136e91273
|
fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084)
Some checks are pending
unit / endpoints-and-responses (push) Blocked by required conditions
unit / enterprise-routing (push) Blocked by required conditions
unit / guardrails-hooks (push) Blocked by required conditions
unit / logging-misc (push) Blocked by required conditions
unit / root (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / assert-shard-coverage (push) Waiting to run
unit / enterprise-managed-files (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / rust-bridge (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Waiting to run
unit / OpenAI 3.0.0 compatibility (push) Waiting to run
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
unit / custom-logging (push) Blocked by required conditions
unit / db-and-spend (push) Blocked by required conditions
unit / jwt-and-keys (push) Blocked by required conditions
unit / key-generation (push) Blocked by required conditions
unit / mcp-oauth (push) Blocked by required conditions
unit / proxy-runtime (push) Blocked by required conditions
unit / proxy-server-core (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / OpenAI 3.25.0 compatibility (push) Waiting to run
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl A prefix marked cache_control ttl 1h lost its pinned deployment after the fixed 300 s, so the next request went to another deployment and paid a cold cache. The pin TTL now follows the longest ephemeral breakpoint in the cacheable prefix, tools included: 3600 s for a 1h breakpoint and 300 s otherwise. * fix(router): give prompt cache pins their own tier and honour any declared ttl Pins now live in a DualCache of their own instead of the router's 200-slot in-memory tier, so an hour-long pin no longer evicts cooldowns and rate windows. The pin lifetime follows any <n>s|m|h cache_control ttl, floored at 300 s, not only the literal "1h". * fix(router): hash the pin ttl off the event loop and flush prompt cache pins with the router cache Compute the cache_control ttl scan through offload_token_count beside the prefix hash, clear the dedicated pin cache in Router.flush_cache, and point the order-fallback and priority-group pin tests at the pin cache they read * fix(router): pin 1h prefixes only on 1h-capable models, cap the pin at 1h, and give pins 10k in-memory slots * fix(router): match the 1h pin gate to what Bedrock forwards Bedrock deployments now decide the 1h pin with is_claude_4_5_on_bedrock, the same check that decides whether the 1h ttl reaches Bedrock, so an application inference profile ARN keeps a 5 minute pin whatever its base_model prices * test(router): give the anthropic_messages pin fixture its provider * fix(router): let each pin write set its own lifetime and require a positive 1h price In a group mixing 1h and 5m deployments the in-memory pin kept the lifetime of its first write, so a 5m deployment could hold an hour-long pin. Each write now clears the in-memory entry first, matching Redis. The Bedrock gate uses supports_bedrock_prompt_cache_ttl after #46058, and a zero 1h price no longer counts as 1h support. * test(router): integration cells for the one-hour prompt cache pin lifetime Cover every surface the pin lifetime touches: chat and /v1/messages through both SDKs, the responses bridge, response-cache hits, each provider family's 1h gate, malformed and duplicated ttl values, rewrites of one prefix, mixed and cross-group routing, concurrency, every Redis attach path, the in-memory slot pressure without Redis reads, a Redis outage and a killed worker * fix(caching): bound the in-memory expiry heap and count prompt cache pins in the memory debug routes * test(integration): cover prompt cache pins in the memory debug routes and the bounded expiration heap * fix(caching): serialize in-memory cache writes so a heap rebuild cannot lose another thread's expiry entry * fix(caching): keep the in-memory cache bounded when entries have no usable expiry * test(caching): cover the response cache bound when cached entries have no usable ttl --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
bbd58931fe
|
test(integration): follow #46101 in the breakpoint cache mark census expectation (#46509)
* test(integration): follow #44788 and #46101 in the openrouter translation and cache mark census expectations Expect prompt_cache_breakpoint for opted-in OpenAI and omit assistant tool-call marks from the capped census Match OpenRouter refusal, logprobs, and system fingerprint nulls Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): follow #46341 blank env and guardrail param handling in keepalive and scope expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): read the first stream chunk through the stream's iterator so MCP streams keep working Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): follow #46493 retry and fallback semantics for streams that fail before content Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop the openrouter expectations #46510 already landed on main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): restore blank lines lost in the main merge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
314f17986a
|
fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models (#45882)
* fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models * fix(responses): define the tool_search_call output item locally for older openai SDKs * fix(responses): gate tool search on the Anthropic request builder and validate deferred tool fields Bedrock Converse builds its own request and has no tool search, so the bridge now drops the tool_search tool there with a warning instead of sending a tool the transport loses, while Bedrock invoke, Anthropic, and Vertex keep it. A replayed tool_search_call with no stored result is dropped from the Anthropic request so a previous_response_id follow-up no longer answers 400. Only srvtoolu_ ids whose name is one of Anthropic's search tools count as search calls, so a caller function named like one stays a function call. A non-boolean defer_loading or a non-list allowed_callers now answers 400 instead of 500 * fix(responses): keep an answered search call on replay and send tools undeferred when no search tool goes out A replayed Claude tool search call that the caller answered with a tool message kept its tool_result in the user turn while its tool_use was dropped from the assistant turn, so Anthropic answered 400 on the OpenAI-SDK tool loop. The search call is now dropped only when no tool message answers it. The Responses bridge now strips defer_loading from every tool whenever no tool search tool is sent, whether the caller sent none or the route cannot run it, since a deferred tool with no search tool can never be loaded. Namespace members had started carrying the flag to every bridge provider, which providers that reject unknown tool keys answered 422. The audit round-1 integration cells for the bridge wire and chaos paths ride in this commit. * fix(responses): gate only the OpenAI tool_search shape and keep native Anthropic search tools intact A native tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119 tool the caller sends through /v1/responses passes through the bridge on every route as before, and the other tools keep defer_loading whenever any Anthropic search tool goes out. The route gate now drops the OpenAI tool_search item before translation instead of filtering the translated regex tool, so bm25 counts as a search tool. answered_tool_call_ids walks the history with islice instead of a slice copy. The tool_search_call output item drops its created_by field, which OpenAI rejects on a replayed input item and the streaming item never carried * fix(responses): drop malformed tool flags under drop_params and keep stored search calls off non-Anthropic follow-ups With drop_params on, a non-boolean defer_loading or a non-list allowed_callers on a Responses API tool (or a namespace member) is dropped before the bridge instead of answering 400, so only a request without drop_params gets the 400. A custom tool now carries defer_loading to the chat tool like a function tool does A previous_response_id follow-up routed to a provider that does not build an Anthropic request no longer replays the stored Claude search call (srvtoolu_ id) as a client tool call. Mistral answered that history 400 "Unexpected tool call id". An assistant turn that held only the search call is dropped there. The Anthropic route keeps the call, since its factory pairs it with the stored result The tool_search_call output model is renamed OutputToolSearchCall so the proxy's OpenAPI schema keeps the OpenAI SDK's ResponseToolSearchCall name. answered_tool_call_ids reads the history by index, so the lookup is linear in the history length. The streaming done event reads the response it is given, which is the same object it used to read from the iterator * fix(responses): validate function allowed_callers, send one search tool, and keep replays clean off Anthropic A function tool whose allowed_callers is not a list of strings now answers 400 instead of 500. The guardrail scan reads tools without malformed flags, so the route decides between a 400 and a drop under drop_params. When the caller sends a native Anthropic search tool beside tool_search, only the native one goes out. Databricks Claude no longer counts as an Anthropic request route. A previous_response_id follow-up off Anthropic also drops a search call the client replayed as a function_call with a srvtoolu_ id, and its output. The answered tool id scan in the Anthropic message factory runs once per assistant run instead of once per message. A null created_by on tool_search_call and tool_search_output output items is left out of the serialized response, so a cache hit returns the same keys as the first call. * fix(guardrails): keep OpenAI tool_search out of the Responses guardrail tool scan The scan translated tool_search into Anthropic's regex search tool on every route, so a guardrail returning only function tools removed tool_search from native OpenAI requests and OpenAI answered 400 for the deferred tools * fix(anthropic): drop answered tool search replays beside client calls, place system before dropped search turns, flag tool search on azure_ai * fix(anthropic): keep answered tool search replays paired with their answers * fix(anthropic): place mid-conversation system messages in one pass and scan tool_search_call in output guardrails * fix(guardrails): send empty incomplete tool_search_call arguments when a rewrite is not a JSON object * fix(responses): keep the caller's defer_loading on Claude routes when no server tool_search is sent * fix(guardrails): deliver streamed rewrites when OpenAI's tool_search_call has no call_id * fix(responses): answer 400 to a tool_search execution that is not server or client OpenAI's own Responses API answers 400 invalid_value to it. With drop_params the malformed execution is dropped like the other malformed tool flags, which leaves a server search * test(responses): cover the 400 and drop_params paths for a malformed tool_search execution * test(responses): cover a Claude search call replayed to Bedrock Converse with cache marks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |