mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-13 08:52:02 +00:00
633 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kerry Lu
|
ce73cf8b9d
|
chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974)
* chore(cost-map): remove the 52 rows whose provider retired them before 2026 * test(utils): pin the proxy function-calling parity case to azure/gpt-4.1 * test(batches): price the Azure batch-logging regression case against gpt-4.1 --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Aditya Batra
|
e9a6a25fe3
|
feat(providers): add Wally as an OpenAI-compatible provider (#45662) | ||
|
devin-ai-integration[bot]
|
d1e4b50d6a
|
fix(router): keep a deployment's token limits off the shared backend cost map key (#49952)
* fix(router): keep a deployment's token limits off the shared backend cost map key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): preserve catalog token limits on shared keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): use catalog for shared token limits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): drop the class attribute patch from the deployment max_tokens test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): avoid provider auth during catalog key resolution Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
65950dba8b
|
fix(main)!: carry the full per-call price entry on the logging object instead of writing litellm.model_cost (#49532)
* fix(main): price per-request custom pricing from the logging object instead of writing litellm.model_cost * fix(main)!: carry the full per-call price entry on the logging object instead of writing litellm.model_cost Direct SDK calls build one ModelInfo from the cost map row plus every custom pricing field on the call and price from it, so audio, tiered and directional per-second rates apply without touching litellm.model_cost. Router calls keep their registered deployment entry. completion_cost(completion_response=...) alone no longer sees per-call prices, pass custom_cost_per_second or custom_cost_per_token * test(cost): assert the exact per-second charge for per-call cost_per_second * fix(main): price Router request overrides and flat rates over inherited tiers in the per-call entry A Router call that sets its own price is priced from it over the registered deployment entry, PTU zeroing applied as at registration. Flat token rates on a call drop an inherited tiered_pricing table, which generic_cost_per_token would otherwise prefer * fix(main): keep tier rates for directions a call does not override and bill per-second-only calls per second A flat rate set on a call is written into every tier for that direction instead of dropping the tier table, so the other direction keeps its tier price. A call that sets only a per-second rate on a model with catalog token rates clears those token rates for the call so the per-second path applies * fix(main): apply a call's flat rate to its own tiered_pricing table when it supplies one * refactor(main): move per-call price entry resolution into litellm_core_utils/per_call_pricing The resolution logic is pure cost-map work with no dependency on dispatch, so it lives next to ptu_pricing and main.py keeps only the call sites in completion() and embedding() Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(cost): rename per_call_model_info parameter to requested_model_info_override Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
norkelbyte
|
3b35ae340d
|
fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks (#49831)
* fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks * fix(rate-limit): count embedding media as media only on models that embed it A data URI sent to a text-only embedding model is read as text, so it keeps the text estimate there. On models flagged supports_embedding_image_input, image and audio use the chat media estimates and PDF and video count at most the model's max_input_tokens. The same estimate now feeds project and key limits, the usage-based, latency and cost routers' deployment tpm, and the deployment itpm check. * fix(proxy): count embedding media only where the backend reads it as media Vertex AI, Gemini, Azure AI and Bedrock count a strict base64 data URI as media per item. Cohere counts it as media only when the whole batch is media, and Voyage, OCI and unknown models keep the text estimate. File blocks carrying a data URI count like the bare URI. The router context window pre-call check and the batch limiter now use the same estimate, resolving each deployment or JSONL row model to its backends. The cost map gains supports_embedding_image_input on voyage/voyage-multimodal-3 and oci/cohere.embed-multilingual-image-v3.0. * fix(router): hold each embedding media item to the context window on its own A PDF or video data URI alone already reserves at most the model's max_input_tokens, but the context window pre-call check then ran the blanked input through the Responses transform and added a few framing tokens, so a PDF that Vertex accepts got a 400. When every input item is media, the check now returns the largest item's media estimate, since an embedding request embeds each item separately * fix(proxy): leave embedding media out of the context check and narrow Bedrock media scope The router context window check now counts only the text of an embedding input, so Titan image lists no longer 400. Bedrock Marengo and Nova with a PDF keep the text estimate since they read those inputs as text or reject them, async_invoke Nova counts images as media, and the unused batch limiter change is dropped. * fix(bedrock): bill Cohere embeddings by Bedrock's input token count header * fix(embeddings): check data URI base64 in place and keep Bedrock Cohere text usage The strict base64 check now scans the payload in place instead of decoding and re-encoding a full copy. Bedrock's billed token header is used only for image requests, so text-only Bedrock Cohere usage matches main. The deployment itpm estimator change is reverted, since the router never passes embedding input to that check. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
883a2d1572
|
revert(bedrock): restore STS identity probe behavior before #49763 (#49939) | ||
|
yuneng-jiang
|
e1c03c5423
|
fix(e2e): restore main qualification across providers and UI (#49763)
* test: check an unsupported budget sort field
* test: wait for dashboard navigation readiness
* test: support clustered provider caches with live fallback
* Revert "test: check an unsupported budget sort field"
This reverts commit
|
||
|
devin-ai-integration[bot]
|
083961731c
|
fix(model_prices): consolidate verified provider metadata and pricing (#49361)
* fix(databricks): mark audio and video input support on the Gemini 3.5 to 3.8 Flash rows * fix(model_prices): add Vertex AI and Together AI retirement dates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model_prices): correct perplexity agent api pricing and add gemini-4-argon, nemotron-3.5-super Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model_prices): consolidate verified pricing and Aplomb metadata Absorb verified pricing from #49547 and #49548 and Aplomb 1 metadata from #49539. Preserve provider tiers and remove an identical duplicate retirement key. Co-authored-by: Liutasil501 <schmid.martin501@gmail.com> Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(model_prices): sync generated EmpirioLabs endpoint metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model_prices): verify Nebius and EmpirioLabs model metadata Verify canonical EmpirioLabs rows from #49695 against published model pages, preserving pricing tiers and excluding unverified variants and specialty billing. Add Nebius models and cached-input prices from the official catalog and correct the Voyage legacy input limit. Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model_prices): absorb Bedrock Opus 5 batch prices, Vertex Omni Flash limits and Gemini 2.5 shutdown dates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model_prices): drop the EmpirioLabs rows and endpoint flags from this sync * fix(model_prices): merge main and dedupe the gemini-2.5 deprecation_date keys * fix(model_prices): set supports_video_input on the two new Nebius video models * fix(model_prices): add Azure AI Claude Mythos 5.1 and match Foundry max effort support Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> |
||
|
zoveki
|
baddc67041
|
fix(anthropic)!: carry prompt_cache_breakpoint to cache_control on chat completions to Claude (#49783)
* fix(anthropic): carry prompt_cache_breakpoint to cache_control on chat completions to Claude
A content part carrying OpenAI's prompt_cache_breakpoint on a /v1/chat/completions request to Claude (Anthropic, Bedrock Invoke and Converse, Vertex AI, Azure AI) now reaches Claude as cache_control {"type": "ephemeral"} on the same block, so prompt caching works for OpenAI-shaped clients instead of the marker being stripped.
Only parts OpenAI accepts the marker on are translated, a client cache_control on the same block wins, and Converse translates only for models that take cache points. A 30m ttl or a malformed marker answers UnsupportedParamsError without drop_params and becomes a default 5m mark or no mark with it.
* fix(anthropic): hold translated prompt_cache_breakpoint marks to the 4-breakpoint cap
Marks past Claude's 4 cache breakpoints now answer 400 without drop_params and
keep the earliest under it, counting the client's own message, tool, and
top-level cache_control marks. The ttl and malformed-marker errors name
drop_params for proxy users too and no longer name a single provider.
* fix(bedrock): count Converse tool cache points in the breakpoint cap and gate unmapped models
A client tool cache_control or a tool_config injection point becomes a Converse tool
cachePoint, so it now takes one of the 4 slots before a translated marker does.
Converse only translates prompt_cache_breakpoint for a model the cost map says takes
prompt caching. An unmapped model such as an application inference profile ARN opts in
through model_info supports_prompt_cache_breakpoint, so a marker sent to every model of a
mixed fleet no longer turns a non-caching profile's 200 into Bedrock's 403.
Router prompt-caching affinity now treats a marked block like a cache_control one, and
requests without a marker skip the translation after one pass over their messages.
* fix(bedrock): count only the tool cache points Converse sends against the breakpoint cap
* fix(anthropic,bedrock): cap only the cache breakpoints each provider sends, and drop marks Nova cannot take on tool results
* perf(anthropic,bedrock): find the breakpoint cut-off by binary search so trimming many marks stays fast
* fix(anthropic): drop the duplicate drop_params left by the main merge
* fix(bedrock): let an inference profile's cache breakpoint flag win over its base_model
* fix(anthropic): keep the first and last prompt_cache_breakpoint marks past the cap
Under drop_params the translated marks that fit the cap are now split between
the start and the end of the request, so the stable prefix and the newest turns
both stay cached. The cap count also includes a native tool search tool that
keeps its cache_control on the wire.
Bedrock Converse now translates the marker only for the Claude and Nova
families, which Bedrock documents as taking cache checkpoints, unless the
deployment sets supports_prompt_cache_breakpoint. A guarded user turn keeps its
cache mark when the request fits the cap.
* refactor(bedrock): decide guarded cache marks with a typed bool helper
* fix(bedrock): validate guarded blocks before reading their cache mark
* fix(anthropic): let injection points replace client marks and keep non-Claude routes untranslated
When a deployment sets cache_control_injection_points, the client's
prompt_cache_breakpoint marks are dropped on targets that do not read them, so
the operator's injection points decide every Claude breakpoint.
OpenCode Go models served over the Anthropic messages endpoint translate the
marker only for Claude, and guarded Converse text keeps a cache mark only on a
model known to take cache points.
* fix(anthropic): keep client marks unless a message injection point replaces them on a translating route
A tool_config-only injection point no longer strips client prompt_cache_breakpoint marks, a flagged Bedrock profile no longer counts as reading the marker natively, and guarded Converse text skips the tool count when no message carries a cache mark.
* fix(anthropic): keep Bedrock and Claude deployments flagged for the marker on cache_control injection
A supports_prompt_cache_breakpoint flag on a Bedrock profile or a Claude deployment made the injection hook write OpenAI markers and prompt_cache_options, which Bedrock answers 400 for. Routes that translate the marker now never take the OpenAI dialect.
* fix(bedrock): reserve only the tool cache points Converse sends when trimming Claude cache marks under drop_params
* fix(anthropic): name the marker's accepted ttl values in the malformed prompt_cache_breakpoint 400
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
|
||
|
tavren27
|
482f3c37ed
|
fix(azure_ai): keep azure_ad_token out of the request body (#49769)
* fix(azure_ai): keep azure_ad_token out of the request body azure_ad_token is not a litellm-owned param, so get_optional_params routes it into extra_body and the azure_ai chat and Claude transformations merge it into the JSON body. Azure AI Foundry then rejects the call with 400 Unknown parameter / Extra inputs are not permitted. The Authorization header already comes from litellm_params, so drop the extra_body copy once at the top of the azure_ai dispatch, as the azure branches do. Type litellm.open_ai_chat_completion_models and open_ai_text_completion_models as Set[str] so the basedpyright gate does not grow from the untyped optional_params read. * fix(azure_ai): keep Entra credentials out of every azure_ai request body * fix(azure_ai): keep the Entra token out of the image edit form * fix(azure_ai): keep Entra credentials out of image edit extra_body too --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
norkelbyte
|
864b98acec
|
fix(bedrock)!: fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 (#49170)
* fix(bedrock): fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 The Bedrock SDK ends the output stream the same way for a dropped connection as for a session that sent sessionEnd, so the handler treated a dropped stream as a clean finish. The client got a 1000 close with no error event and the router logged 200 OK with no retry An output end now counts as clean only after the client left and the handler began closing the Bedrock input. An earlier end raises a 503 BedrockError, so the client gets an error event and a 1011 close, the failure handlers run, partial spend is still logged, and a retry of a committed session is still refused * fix(bedrock): fail a dropped Nova Sonic session right away and log it as failed When Bedrock's output side ends mid-session, by a drop or a modeled stream error, the client side no longer spends the close timeout sending close messages into the dead input stream, so the error reaches the app right after the drop. A session the app ends itself still sends Bedrock its close messages and closes the input The replay refusal now leads with the committed cause, so the error event and the cut 1011 close reason both start with it. It stays a 400 and is tagged as a failure outside the selected deployment, so router retries and fallbacks of a dropped session no longer add extra failure counts A clean realtime session now counts once toward its deployment's cooldown success counter, so cooldown reads the real failure rate instead of 100 percent. Realtime tpm and rpm counting is unchanged The spend row of a session the provider ended now reads failure with its error information, while the events already forwarded are still billed. A session the app ended keeps logging success * fix(bedrock): agree on a dropped realtime session's failure across loggers and the spend row A realtime session's success event now reports status failure, with the same error_information the spend row stores, when the session recorded a provider failure in its metadata. Usage already streamed stays billed and the router still counts one cooldown failure The spend row builder now runs every row's error_information through sanitize_error_information_for_spend_logs (DB size cap plus prompt redaction), so the realtime failure rows get the same treatment as the proxy failure hook, which no longer sanitizes on its own to avoid truncating twice Drop the selection-failure tag on the committed-session refusal, since the shared logging object already keeps a retried refusal out of cooldown, and pin the router's realtime success counting with an OpenAI realtime deployment * fix(bedrock): count realtime router successes only for sessions that logged one, and log first-turn drops The router counts a realtime success only when the session's success handlers ran, so a refused handshake or an upstream close before any frame no longer offsets its failure. The success event's payload status stays success again, so each logger gets one failure record per dropped session, while the spend row still reads failure from the metadata the Bedrock handler records. A committed Bedrock session that fails before any logged event now gets one zero-cost Logs row. After an output failure the input stream is closed in the background, so the SDK body writer finishes without delaying the client's error. Every input send failure reports the same 503 stream-ended cause as an output drop * fix(realtime): count a router success only for a session that ended cleanly A realtime session the upstream closes with 1011 or 1006 after its first frame now marks itself as ended abnormally where that close is seen, in the shared relay and in the OpenAI Live handler, so the router no longer counts it as a cooldown success. The success callbacks still run, so the proxy keeps the budget and parallel-slot settlement it had A Bedrock session that fails before Bedrock sends any usage now logs no success event, as on base. The spend-row failure fields go into a new metadata mapping instead of the request's own dict, validated with a pydantic TypeAdapter instead of an unchecked TypeIs guard. The handler tests drop a wall-clock bound and type their helpers * fix(bedrock): log a dropped realtime session's usage from its own logging state The failure event replaced the shared standard_logging_object while the queued success dispatch was still running its callbacks, so loggers received $0 and 0 tokens for a billed dropped session. * fix(proxy): settle a rate-limit reservation once when a request logs a failure and a success A dropped realtime session fires a failure event and then a success event for the same call. The v3 limiter refunded the TPM reservation on the failure and released it again on the success, so the key's TPM counter went negative. The success path now skips a reservation the failure already released and marks it settled for a later failure. The realtime handler test callbacks are typed * fix(rate-limit): settle each token reservation once when a request logs both a failure and a success A dropped realtime session logs a success for its usage and then a failure. After a success the limiter now clears every reserved amount (TPM, PTU, ITPM, OTPM), so a later failure refunds nothing twice. A new reservation written in the same context after a released one now starts clean, so reserve, fail, reserve, succeed charges the second request's usage once instead of 78 TPM or 28 ITPM for 27 and 20 used * fix(rate-limit): take a token hold before awaiting so concurrent success and failure callbacks settle it once * fix(rate-limit): track the ITPM/OTPM hold's release apart from TPM so a nested pre-call keeps the request's project IO charge * test(rate-limit): pin the split hold-release flags across nested MCP pre-calls * fix(realtime): count a relay that fails on its own as an abnormal end * test(realtime): pin the relay failure exit with a client frame the provider rejects --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
5b546259d5
|
fix(cache_control)!: keep one deployment's prompt-cache opt-in from reaching siblings on the same model (#49685)
* fix(cache_control): read the routed deployment's own prompt-cache flag, not a sibling's shared cost-map entry Two deployments that share one model string share one key in the live litellm.model_cost map, and router registration writes each deployment's supports_prompt_cache_breakpoint into that key. The cache control hook read the live map, so a flagless deployment picked up a sibling's opt-in and sent prompt_cache_options and prompt_cache_breakpoint upstream, which the provider rejects with a 400 When a call carries the routed deployment's model_info, the prompt cache checks now use its declared supports_prompt_cache_breakpoint, and fall back to the loaded catalog instead of the live map. This covers the hook on completion, acompletion, Responses and the /v1/messages route, the azure_ai chat transform, the OpenAI Responses transform under drop_params, and the /v1/messages to Responses adapter * fix(prompt caching): read each deployment's own breakpoint flag in the openai and bedrock readers A deployment's own supports_prompt_cache_breakpoint now picks OpenAI's prompt_cache_breakpoint only where the request goes out OpenAI-shaped on a provider other than openai, such as azure_ai OpenAI models and bedrock_mantle. Bedrock on every route and providers that build Anthropic messages keep cache_control, which Bedrock Converse turns into cachePoint The openai chat reader now passes the routed model_info and checks base_model. The Bedrock cache point reader takes the deployment's flag first and, for a routed call with no flag, reads the catalog as loaded instead of the live map that sibling deployments write into. model_info is threaded from the converse transform, the factory message and tool builders, and the responses bridge * fix(prompt caching): keep gateways on cache_control and read register_model opt-ins on routed calls A declared supports_prompt_cache_breakpoint picks OpenAI's breakpoint only on providers hosting OpenAI's API or for a model the cost map lists under the provider, so OpenRouter, LiteLLM proxy, and vLLM deployments keep cache_control as before. Routed calls read the loaded catalog plus the user's own register_model entries, never router registrations. * fix(prompt caching): honor a deployment's own supports_prompt_caching on Bedrock and its declared litellm_provider on gateways * fix(prompt caching): keep wildcard cache flags off matched models, read deployment flags for auto caching and base_model aliases * test(anthropic): cover auto caching on an unmapped Claude deployment * fix(cache_control): keep the merge base's reads where the per-deployment flag changed them A register_model entry under a provider-prefixed name reaches routed calls again, since the overlay now resolves the bare and prefixed names the way runtime registrations are stored. The hook and the openai chat reader no longer judge base_model, as at the merge base, so a client base_model sent under allow_client_pricing_override cannot pick OpenAI's breakpoint and get a 400. A Bedrock deployment flag that is not a real true reads as false again, as at the merge base. An azure_ai wildcard's supports_prompt_cache_breakpoint speaks for each model it matches again, as at the merge base. Other wildcards still leave both cache flags to the catalog. The s10/s11 Bedrock wire test asserted that a flagless deployment inherits its sibling's flag on a shared ARN, which is the leak this PR removes. It now asserts that each deployment keeps its own flag and that a flagless unmapped ARN keeps its cache points after the delete. * fix(cache_control): let the later register_model entry win between a bare and a provider-prefixed name Routed calls read register_model entries through an overlay that merged the bare name after the provider-prefixed one whatever order they were registered in, so a later opt-out under one name lost to an earlier opt-in under the other. Each registration now moves its key to the end of the kept registrations, and the overlay merges both names oldest first, which matches what SDK calls read from the live model map. Also lists the wildcard cache flag helper in the router coverage script, since the hook's wildcard tests drive it rather than calling it by name. * fix(cache_control): keep register_model entries visible while a cost map reload replays them The previous commit moved a re-registered name to the end of the saved registrations by removing and re-adding it, and the overlay walked that dict to order a bare and a provider-prefixed name. A reload replays every saved entry, so a request on another thread could briefly miss an entry or fail on a dict that changed size mid-walk. Entries are written in place again, a per-name sequence records which registration came last, and both the overlay and the reload replay order by it, so the later registration still wins after a reload. * fix(cache_control): read a string or numeric supports_prompt_caching as the bool it spells and find register_model entries under any route prefix A deployment flag stored as "true", 1, "false" or 0 (POST /model/new and quoted YAML keep strings) now means that bool on the Bedrock cachePoint reader and Anthropic auto caching, and any other value falls to the catalog. register_model records which names wrote to each catalog row, so an opt-out under bedrock/ or bedrock/converse/ reaches routed calls like one under anthropic/. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Emerson Gomes
|
ecb8ef5ca8
|
feat(realtime)!: support latest OpenAI audio models (#35600)
* feat(realtime): support latest OpenAI audio models * fix(transcription): type SDK stream duration metadata explicitly * fix(realtime): preserve transport settings and translation duration * refactor(realtime): reuse SDK client initialization * fix(audio): bill finalized streams and authorize transcription models * refactor(audio): keep transcription pricing selection together * fix(realtime): reconcile model metadata and test contracts * fix(realtime): preserve provider duration and model capabilities * fix(realtime): keep transcription validation compatible with cost map guard * fix(realtime): address transcription and billing review findings * fix(realtime): preserve input usage and provider routing * fix(realtime): bill retained output and route Azure aliases * fix(realtime): reject malformed provider durations * fix(realtime): preserve batch cache rates and Azure v1 routes * fix(transcription): reject nonfinite provider durations * refactor(realtime): align annotations with current lint rules * fix(realtime): preserve translation deployment pricing and input usage * test(transcription): verify synchronous stream finalization * fix(realtime): retain missing translation input usage at close * fix(transcription): reject streams that bypass output guardrails * fix(realtime): reconcile audio protocols billing and current CI * fix(realtime): trust confirmed formats and authorize default models * fix(realtime): preserve absent models and reject unsupported guardrails * fix(realtime): reject translation policy pipelines * chore(ci): rerun transient infrastructure timing failure * fix(realtime): reconcile audio protocols with current upstream * fix(audio): validate provider responses at SDK boundaries * fix(realtime): preserve minimum OpenAI SDK compatibility * fix(realtime): preserve deferred Live session authorization * fix(realtime): preserve upstream authentication and transcription behavior * fix(audio): scope provider duration precedence to configured models * fix(typing): avoid Final declarations inside transcription cost loop * refactor(audio): reuse transcription mapping and duration resolution * fix(realtime): enforce model budgets and preserve translation routing * fix(audio): log streamed transcripts whose done event reports detected languages * fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers Providers without a streaming transcription handler got UnsupportedParamsError for stream=True, languages and keywords, where the merge base forwarded them as provider-specific params. Only openai and azure validate them now. * fix(audio): bill streamed gpt-transcribe from the provider's reported duration The stream path computed cost after logging had replaced the transcription's duration usage, so it fell back to the measured file length while the unstreamed call billed the provider's seconds. * fix(transcription): log streamed transcription usage from the provider response The stream logging path built the spend payload from the response after its duration usage was transformed away, so whisper-1 with stream=true and streamed gpt-transcribe logged an empty usage object. Bill and log from the provider's TranscriptionResponse in both the sync and async paths. * fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes - Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape - Use the input transcription model as the session model only for transcription sessions - Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main - Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set - Return x-litellm headers on streamed transcriptions and mark the request done - Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure - Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing - Bill translation sessions on output audio seconds only, at the single published rate - Document the 201 SDP answer on the translation calls routes and the transcription 200 response * fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the default dated version on the deployments route for every model. The SDK client secret keeps session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the gpt-transcribe format check only reads the request provider's cost map entry. * fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it Transcription reaches every provider as on main again, sends stream only when asked, forwards language and languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from one source and spend rows carry the billed seconds. Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA get beta-shaped session events, and transcription sessions send the deployment model in both shapes. * fix(realtime): address review findings on transcription billing, streaming and realtime auth Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata * fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only * fix(realtime): preserve prior merge resolutions after rebase * fix(realtime): enforce translation policies and normalize Azure transcription URLs * fix(cookbook): stop translation once output idles after the upload ends * fix(realtime): bill translation on input audio and drop pre-read stream cost headers Translation spend now uses the input audio seconds the session sent, priced through input_cost_per_second, instead of the translated output duration. A streamed transcription no longer returns cost headers before the stream is read, since the cost is only known after the last event. Also removes an unused realtime guardrail helper. * fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop * test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription * fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA Bill translation sessions from the deployment's own per-second rates, add per-second prices for Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models * fix(realtime): bill a translation rate the deployment omits from the session model * test(e2e): check translated speech is not English instead of exact French wording * fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending * fix(realtime): keep the proxy's realtime session gate off translation sessions A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects * fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params. * fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree * fix(model-catalog): accept the realtime translation endpoints in the cost map * fix(cookbook): close the translation session and read until session.closed * fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard An empty `stream` form value runs the transcription unstreamed again, as OpenAI does, instead of answering 400. The four translate cost-map entries drop the translation routes from supported_endpoints until the main branch's catalog parser accepts them, and the nested transcription model lookup uses the router registry entry point. * test(proxy): drop the mid-stream error frame pin that main's error shape change replaced * fix(transcription): bill uploaded seconds when a stream fails mid-transcript * fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream * fix(audio): keep transcription's positional arguments and redact streamed error details - Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning - Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do - Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite * fix(audio): stream the guarded transcript to SDK callers that asked for a stream * fix(logging): keep a streamed transcript out of callbacks that turned off message logging A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does * fix(proxy): stream a transcription the deployment streamed when the client sent no stream field --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
f4a2e2b3af
|
fix(anthropic): send an https PDF image_url to Claude as a document (#49708)
Some checks are pending
unit / router (push) Blocked by required conditions
unit / rust-bridge-harness (push) Blocked by required conditions
unit / auth-checks (push) Blocked by required conditions
unit / budgets (push) Blocked by required conditions
unit / custom-logging (push) Blocked by required conditions
unit / db-and-spend (push) Blocked by required conditions
unit / endpoints-and-responses (push) Blocked by required conditions
unit / enterprise-routing (push) Blocked by required conditions
unit / guardrails-hooks (push) Blocked by required conditions
unit / jwt-and-keys (push) Blocked by required conditions
unit / key-generation (push) Blocked by required conditions
unit / logging-misc (push) Blocked by required conditions
unit / mcp-oauth (push) Blocked by required conditions
unit / proxy-runtime (push) Blocked by required conditions
unit / proxy-server-core (push) Blocked by required conditions
unit / root (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* fix(anthropic): send an https PDF image_url to Claude as a document A PDF link sent as image_url to Claude on Anthropic went out as an image block and Anthropic answered 400. A link whose path ends in .pdf is now downloaded like Vertex links are, and the served Content-Type picks the document or image block, so CDN links that serve a PNG from a .pdf path keep working. An explicit format of application/pdf sends a url document with no download. PDF media types match regardless of case or parameters, and count_tokens picks the same block as the request. * fix(anthropic): download https PDF links off the event loop The proxy downloaded an https link ending in .pdf synchronously while building the Anthropic request, so on an async call every other request on that worker waited for the download. Those links are now inlined before the request is built, the same way http links already are, and the /v1/messages guardrail write-back does the same. A non-string format on an image_url is ignored the way it already was inside the image_url mapping, instead of raising an AttributeError. * fix(anthropic): send a PDF link the proxy cannot download as a link and keep guardrail url blocks as sent * test(anthropic): expect the PNG media type the served bytes carry at a .jpg path * fix(anthropic): bound the PDF link download and keep query-string PDF names Give up on a .pdf link after NON_LLM_CONNECTION_TIMEOUT instead of the client's 6000 s default, fall back to main's whole-URL extension check so download?file=report.pdf still counts as a PDF, treat application/x-pdf as a PDF, and set the url document's application/pdf format only for the guardrail write-back so other /v1/messages providers send what main sends * fix(anthropic): probe a .pdf link and send it to Claude as a url document Instead of downloading a .pdf link and inlining it as base64, ask the host for its first kilobyte and send the link as a url document when it serves a PDF. Every other link goes out as the url image main sends, so robots.txt handling, caching and the wire format stay as they are on main. Also restore the guardrail adapter and the content type inference to main, keep the client's url document blocks (title, context, citations, cache_control) through the guardrail write-back, treat a PDF filename in the query string as a PDF link, probe links concurrently on the sync path, cap the async probe's read at the download limit, and catch only transport, URL and SSRF failures in the probe * fix(anthropic): read only the first kilobyte when probing a PDF link, and restore each url document from its own place The probe now streams the response and keeps the first 1 KB, so a host that ignores Range no longer fills memory with up to 20 parallel bodies. The guardrail write-back now matches url documents inside a tool_result and outside one separately, since the write-back moves tool results ahead of other content * fix(anthropic): remember each PDF link probe's verdict and cap its total time, and skip the probe on Bedrock invoke * fix(anthropic): keep a PDF probe's error status for a minute, not ten --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
4573fa1f31
|
fix(embeddings): honor the caller's timeout on Cohere sync and HuggingFace embedding requests (#48593)
* fix(embeddings): honor the caller's timeout on Cohere sync and HuggingFace embedding requests Bedrock Cohere embeddings ran the sync request with the client default instead of the timeout passed to litellm.embedding, and HuggingFace embeddings dropped it on both the sync and async paths. Pass the timeout through to the HTTP request on each path. Fixes #47301 Fixes #47299 * fix(huggingface): pass the caller's timeout to the embedding model lookup too Without input_type the handler first fetches the model's pipeline tag from huggingface.co, and that GET still used the default timeout. Thread the caller's timeout through both lookup helpers and the transform step so every request the embedding makes honors it. Refs #47299 * fix(huggingface): keep the unreachable async lookup path untouched The async model lookup helper and the call_type="async" transform branch are never reached, since the async embedding path transforms through the sync lookup. Leave them as they were so the change only touches requests that actually run. |
||
|
Devin Gupta
|
216041e243
|
feat(decisions): add Parallel AI web-grounded decisions (#49188)
* feat(decisions): add Parallel AI web-grounded decisions * fix(decisions): name the Parallel model decision * fix(parallel_ai): restrict environment credentials to trusted decision endpoints * fix(decisions): keep Parallel evidence fields off other providers and handle missing confidence Only fields a provider names reach the OpenAI response, so existing providers keep their base shape. A malformed api_base answers 400 on both credential paths. The complexity router classifier and the Playground handle a provider that omits confidence. Regenerates the lazy OpenAPI snapshot and the dashboard API types * chore(cost-map): declare the Decisions routes on parallel_ai/decision The provider endpoint files are generated from the cost map, so the row must name both routes for the generated System One flag to hold * fix(decisions): report no classifier confidence when a provider omits it A choice answer with no confidence is a valid classifier answer, so the complexity router picks its tier and leaves classifier_confidence and the confidence signal out instead of reporting a different metric in its place. Also show the Parallel AI logo in the provider picker and add a Parallel translation fixture captured from the live API for both Decisions routes. * fix(decisions): keep a token-priced row on token billing when it also sets a query price A plain input_cost_per_query bills flat only on a row with no token price, and input_cost_per_query_decisions always bills flat, so a custom Decisions row that already bills tokens costs the same as before this PR * fix(decisions): bill a deployment's query price and accept Parallel bases that end in a route A proxy request logs through an object built before the router picks the deployment, so a query price on litellm_params never reached Decisions billing. Parallel Search accepts a base ending in /v1/search and shares PARALLEL_AI_API_BASE with Decisions, so Decisions now accepts that base and one ending in /v1/decisions. * fix(decisions): keep per-query billing and optional confidence on Parallel only A query price on any other Decisions provider moved that provider off token billing, and a choice or score answer with no confidence became a 200 with a null for every provider. Both now apply to Parallel alone, so other providers bill and fail as they do on main. A Parallel deployment with explicit zero token rates and a query price read as free, so every budget check was skipped while it billed per call. It now counts as priced. The dashboard key for Parallel is now PARALLEL_AI so a served bundle that predates the provider still sends parallel_ai as the provider slug, and the README table lists the provider. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
08b951a814
|
fix(vllm): list hosted_vllm wildcard models when the server has no api key (#40066)
* fix(proxy): expand wildcard models for providers with model info but no cost map entry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): stop requiring an api key to list vllm models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve model ids during wildcard expansion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): forward api key for model discovery Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): support bearer auth for model discovery Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): list hosted_vllm models from the server chat uses, with a bounded wait The hosted_vllm listing now resolves HOSTED_VLLM_API_BASE and HOSTED_VLLM_API_KEY like hosted_vllm chat, instead of the /vllm pass-through's VLLM_API_BASE, and waits at most NON_LLM_CONNECTION_TIMEOUT rather than the completion timeout. vllm/* keeps listing nothing, since its chat runs in process. The router pass-through now forwards the deployment's api_key. * fix(vllm): skip the placeholder key, honor deployment TLS and headers, and retry failed listings after a minute * fix(discovery): cache only timed-out model listings, so a fast failure is tried again on the next call * fix(proxy): list wildcard models off the event loop, so a hung provider no longer stalls other requests * fix(utils): keep a concurrent call's model listing when another call times out * fix(proxy): run model discovery on its own thread pool and keep credential lookups off the event loop * fix(proxy): stop the file credential lookup at the first match before listing wildcard models The fallback lookup now walks the allowed model names lazily, in the same order as before, so a hung vLLM server is listed only when no configured name serves the requested provider. The vLLM auth headers no longer mutate the caller's headers dict. * fix(proxy): try every configured wildcard route before listing any wildcard's models in the file credential lookup * fix(proxy): answer model lists from cached listings first and share one listing per server * test(vllm): give the self-signed cert fixture fixed validity dates * fix(proxy): skip wildcard servers a key cannot reach on /v1/model/info /v1/model/info listed every wildcard deployment and filtered by the key afterwards, so a key limited to one model waited on every wildcard server and sent listing calls to servers it cannot use. The listing now skips a wildcard whose patterns match none of the key's models before it asks the server. A key or team granted hosted_vllm/* with no deployment of that name now lists no models, like openrouter/*, since chat answers 400 for every id the environment server would list. * fix(proxy): match a wildcard's own row pattern before asking its server The /v1/model/info pre-filter matched a key's models against both the deployment name and its litellm_params.model, so a key granted hosted_vllm/* still asked a vllm-secure/* -> hosted_vllm/* server whose rows it can never see. It now matches only the pattern expansion names the rows with, picked by one shared helper. * test(vllm): cover the vLLM listings and no-slash wildcard aliases the patch left unrun Adds tests for hosted_vllm get_models with and without a server, vllm/* never asking a configured server, and /model/info pre-filtering of wildcard aliases whose name has no slash. Drops a test comment that only repeated a method signature. --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
28c80048ed
|
fix(tool_choice): accept OpenAI allowed_tools on chat completions and flatten it for the Responses bridge (#49043)
* fix(tool_choice): accept OpenAI allowed_tools on chat completions and flatten it for the Responses bridge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: type the allowed_tools tool_choice fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: raw regex for the allowed_tools provider gate match Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tool_choice): move the allowed_tools provider gate out of completion() so pyright can still analyze it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tool_choice): keep the allowed_tools helpers typed so the basedpyright gate stays level Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tool_choice): type the allowed_tools validation boundary with Any instead of casts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tool_choice): widen tool_choice to object at the allowed_tools helpers with a cast-ok reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(tool_choice): drop allowed_tools test docstrings, type the new tests, and revert unrelated reformatting Also reword the allowed_tools 400 so it no longer blames the flat Responses layout for every malformed choice --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tavren27
|
94b5b6386a
|
fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool (#49703)
* fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool Opus 5.5 and Sonnet 5.5 on Bedrock cannot force the json tool, so a json_schema request without thinking sent an unforced json tool the model ignored. The opt-in native output key now also applies when the model map says the model cannot force a tool, and the Opus 5.5 commercial Bedrock rows carry the opt-in key. The GovCloud Opus 5.5 rows carry it as false, like the Sonnet 5.5 GovCloud rows * fix(bedrock): keep json tool with thinking on Opus 5.5 and Fable 5.1, and GovCloud on the tool Native output without thinking now applies to every Bedrock Claude model that cannot force the json tool unless supports_bedrock_native_structured_output is false, so Opus 5.5 needs no opt-in row and keeps the json tool when thinking is requested, as #46870 chose. The invoke chat path keeps the native output_format it built instead of inlining it, and GovCloud region-path ids read their own false rows. * fix(bedrock): clean caller output_format schemas before sending them natively on invoke * fix(bedrock): keep Mantle Claude on the json tool and send a caller's output_config.format natively on invoke * fix(bedrock): keep the faked Converse stream for native output that replaces the json tool A json_schema stream that goes native only because the model can't force the json tool now keeps the json tool's faked stream, so it still needs only bedrock:InvokeModel. /v1/messages builds a native-output retry only where it sends native output, and the Mantle test drops its mutable capture. * test(bedrock): cover native output and the opt-out in the invoke json-mode refusal tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mika-lt
|
183212e8be
|
feat(anthropic): pick the beta allowlist row per deployment and surface input_transformations (#49689)
* feat(anthropic): pick beta allowlist row per deployment, add mid-2026 beta rows, surface input_transformations
Foundry-hosted anthropic deployments now filter anthropic-beta by the azure_ai row on /v1/messages and chat completions, the bundled config gains the mid-conversation-system-clear-at-2026-08-21 row plus the tool-changes and output-config betas on azure_ai and vertex_ai from a live probe, and Anthropic's input_transformations array reaches chat completions in provider_specific_fields (non-streaming and the first streaming chunk).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): type the provider_specific_fields source dict and beta row test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): input_transformations reaches chat completions and /v1/messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): read the Anthropic response as a Mapping when building provider_specific_fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): fold deployment-row tests into the filtering file and mirror the new beta in the Rust table
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): drop the duplicated beta tuple and mark filter results Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): forward the seven betas Foundry accepts on the azure_ai beta row
A Foundry-hosted anthropic deployment now filters through the azure_ai row, which
dropped compact-2026-01-12, fast mode, fine-grained tool streaming, inline tools,
oauth, output-128k, and token-efficient tools. Foundry answers 200 to each, and
without compact-2026-01-12 a compaction request answers 400, so the row now allows
all seven on both the Python map and the Rust table
* fix(anthropic): keep pass-through and beta-less requests off the Foundry beta row
An openai/ deployment that passes /v1/messages through to a Foundry host lost
every beta, and a request without betas read the beta table from GitHub.
Only an anthropic deployment on a Foundry host now takes the azure_ai row,
and the check reads the provider and host before the table.
* test(anthropic): point the beta-gated wire legs whose row lacks clear-at at a native Databricks deployment
* style(rust): wrap the azure_ai beta header test case the way rustfmt does
* fix(anthropic): forward unlisted betas to Foundry and keep Foundry calls off the Rust messages route
An anthropic deployment on a Foundry host now drops only the betas the azure_ai row marks
unsupported on /v1/messages, and forwards betas the table does not list as sent, as main did.
cognitiveservices.azure.com counts as a Foundry host. The Rust messages route sends an async call
with a Foundry api_base to Python, which filters by row. A sync call stays on Rust, since Python has
no sync /v1/messages handler. The json_mode chat response keeps input_transformations,
and the first /v1/messages call with betas loads the beta table off the event loop.
* fix(anthropic): run the advisor loop for an anthropic deployment on a Foundry host
The advisor interceptor treated every anthropic deployment as native, so a Foundry-hosted one
received the advisor_20260301 tool and Foundry answered 400. Interceptors now get the
deployment's beta row from resolve_beta_provider, so that deployment runs litellm's advisor
loop the same way an azure_ai deployment does
* test(anthropic): drive the Foundry advisor loop test through recorded HTTP
The test now intercepts the Foundry and Anthropic HTTP calls with respx instead of patching litellm's own sub-call helper, so it checks what Foundry receives: no advisor_20260301 tool and no advisor beta
* fix(anthropic): leave the advisor loop to the providers it ran for on main and tidy the PR's tests
Reverts
|
||
|
tavren27
|
640a87f70d
|
fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature (#49697)
* fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): gemini tool-call signature rejected by the backend is resent once with the placeholder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): keep the placeholder resend inside the cache rebuild so either rejection order ends in an accepted body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): resend a cached-prefix signature inline and cover native generateContent When the rejected signature sits inside a cachedContents prefix, the placeholder resend now carries the history inline, since replacing the signature in the request body cannot reach the cache. The native generateContent handlers get the same one-time resend. * fix(gemini): send the history inline when Vertex rejects a signature while creating the cache * fix(gemini): keep the context cache on a signature resend and go inline only when that is rejected too --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
jesus-berri
|
cc142a8191
|
fix(cost): add deepgram/streaming/nova-3-medical and update stale Nova-3 streaming rates (#49733)
The /deepgram/v1/listen WebSocket refuses nova-3-medical with 1008 because the cost map has no streaming row for it. Deepgram's pricing page does not list Nova-3 Medical, so the new row uses the $0.0077/min streaming launch price from Deepgram's 2025-03-03 announcement deepgram/streaming/nova-3 and nova-3-multilingual were $0.0048 and $0.0058 per minute against $0.0077 and $0.0092 on deepgram.com/pricing as of 2026-11-02, so they are updated too Resolves LIT-10393 |
||
|
Prabal Pratap Singh (ZeroX)
|
b3c397e964
|
feat(hosted_vllm)!: keep vLLM's stop_reason and token_ids on chat choices and stream chunks (#48393)
* feat(hosted_vllm): keep vLLM's stop_reason and token_ids on chat choices Non-streamed hosted_vllm chat completions moved vLLM's choice-level stop_reason and token_ids into provider_specific_fields, so clients written against vLLM could not find them where vLLM puts them. They are now also set on the choice itself, and the copy in provider_specific_fields stays for existing readers. Fixes #47945 * feat(hosted_vllm): keep stop_reason and token_ids on streamed chunks and proxy stream output Streamed chat finish chunks and streamed /v1/completions finish chunks now carry vLLM's stop_reason. Each stream chunk keeps the token_ids vLLM sent with it, including held-back empty chunks, and the proxy no longer strips those chunks as empty usage chunks or drops the fields in its fast serializer. * fix(hosted_vllm): keep the upstream choice index on token-id-only stream chunks * fix(hosted_vllm): keep vLLM choice fields for null, cache hits, n>1, and streamed completions Response conversion keeps stop_reason and token_ids on the choice for any OpenAI compatible response, cache replays carry them, finish fields are recorded per choice index, text completion streams keep token_ids, and the proxy passes only these two keys through, nulls included. * fix(hosted_vllm): send empty text on token-id chunks, keep fields on their own choice, and keep null stop_reason on slow-path streams Token-id-only chunks now carry an empty content or text string, a finish chunk only takes stop_reason and token_ids from its own upstream choice, content chunks no longer repeat stop_reason, the slow stream serializer keeps a null stop_reason or token_ids, and fields cached under provider_specific_fields come back on the choice * fix(hosted_vllm): keep stop_reason on the last content chunk, keep both fields in built stream responses, and key the cache on return_token_ids * fix(hosted_vllm): keep another choice's final token_ids and stop_reason on their own stream chunk With n above 1, a finish-only vLLM chunk for a choice other than the one the single finish chunk carries was dropped or taken as choice 0's finish, losing that choice's last token_ids. It now goes out as an id-only chunk on its own choice index. * fix(hosted_vllm): send the first choice's finish chunk before another choice's later token-id chunks * fix(streaming): keep each streamed text completion choice on its own index * fix(streaming): finish each streamed text completion choice on its own index and null token_ids in redacted logs * fix(proxy): drop token_ids from guarded streams, redact choice-level provider fields, and keep one message_delta per Anthropic stream * fix(proxy): skip token-id-only chunks on guarded streams and put stop_reason on the final text chunk * fix(proxy): drop token_ids from the logged and cached response of a guarded stream * fix(proxy): drop token_ids under per-chunk stream checks and redact text-completion stream choices * fix(caching): key return_token_ids and continuation_token sent in extra_body * fix(streaming): drop token ids from streamed chunks while post_call_rules run vLLM holds text back behind a possible stop string but sends its token ids at once, so an SDK stream sent ids before litellm.post_call_rules checked their text. While those rules are set, streamed choices drop token_ids and chunks that carry only ids are skipped, as the proxy already does ahead of a per-chunk check * fix(streaming): keep vLLM fields when a stream's only choice index is not 0 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
simonp9
|
61da6fcb4d
|
fix(mcp)!: answer 400 for a hosted MCP tool a translated route cannot run, drop it under drop_params (#48931)
* fix(bedrock, gemini, deepseek, databricks, gigachat, anthropic): answer 400 for a hosted MCP tool a route cannot run, drop it under drop_params A `type: "mcp"` tool sent to a route that rebuilds tools for its provider (Bedrock Converse and Nova, Bedrock Invoke Claude, Vertex AI Claude, Vertex AI and Google AI Studio Gemini, DeepSeek, Databricks Claude, GigaChat) was dropped silently and the request still answered 200. These routes now answer UnsupportedParamsError (400) naming the model, route and server, and pointing to the LiteLLM MCP gateway. Under drop_params the tool is dropped with a warning, other tools are kept, and no empty tools field is sent. Anthropic, Azure AI Claude and Claude Platform on AWS keep mapping the tool to the MCP connector. Native Responses providers and raw OpenAI-compatible forwarders are unchanged. * fix(mcp): check hosted MCP tools once per request and keep base tool_choice handling Each route now runs one request-level step before its param mapping. When the drop leaves no tools, the request sends no tools, tool_choice or parallel_tool_calls. Converse and GigaChat go back to their base tool_choice and empty-tools handling. * refactor(deepseek): hold the sent tools in a tuple * fix(mcp): keep tool_choice for legacy functions when every hosted MCP tool drops * fix(mcp): answer 400 for MCP servers a route cannot run on Messages, Ollama and the Responses playground /v1/messages requests with mcp_servers on a chat-adapter or Responses-bridge route now answer 400 naming each server, or drop it with a warning under drop_params, instead of being forwarded or silently ignored. Ollama generate no longer pastes a hosted MCP tool (headers included) into the prompt, and ollama_chat filters hosted MCP tools the same way. The Admin UI Responses playground now sends litellm_proxy/mcp references like the other endpoints. * fix(ollama): leave JSON mode off when drop_params drops every hosted MCP tool Ollama generate set format json for any request with tools, even after the hosted MCP check dropped all of them, so a plain answer came back as JSON or as a call to a tool the model never had. The check now runs before the JSON mode switch, which stays on only while a tool is left for the prompt. * fix(messages): honor additional_drop_params for mcp_servers and read the beta map only for hosted MCP tools * fix(mcp_security): accept gateway server names and toolsets the gateway resolves The guardrail compared litellm_proxy/mcp/<name> refs against registry keys, which are server ids, so a saved server referenced by name, alias, or toolset name was blocked although the gateway routes it. It now resolves each ref the way the gateway does and still blocks unknown names. * fix(tool_fields): keep MCP drop warnings on one log line Server labels, model names, and routes come from the caller, so a line break in one could forge extra log lines. The drop warnings now strip CR and LF from each value. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
jesus-berri
|
05b76878b0
|
fix(cohere): map Cohere v2 finish_reason on non-streaming chat responses (#49529)
* fix(cohere): map Cohere v2 finish_reason on non-streaming chat responses * fix(cohere): default a missing Cohere v2 finish_reason to stop instead of raising A 200 body without finish_reason used to return stop and started raising KeyError after the mapping change, so fall back to COMPLETE |
||
|
zoveki
|
6be98d8c75
|
fix(bedrock)!: give Invoke and Mantle Claude the model's own default max_tokens (#49692)
* fix(bedrock): resolve Invoke Claude default max_tokens from the base model id Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): build Invoke Claude request params without mutating Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): type tool-search normalizer return to clear unknown-type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): resolve Invoke and Mantle Claude default max_tokens from the deployment id, ARN base_model, and Mantle row * fix(bedrock): default Invoke Claude max_tokens from the deployment's own cap and honor AnthropicConfig Read the called deployment's model_info.max_output_tokens before the base model's cost row, skip the shared regional key the router fills from every deployment, and leave the default to litellm.AnthropicConfig(max_tokens) when it is set. * fix(bedrock): send the mapped default max_tokens on bedrock_mantle Claude chat * fix(bedrock): bound a global AnthropicConfig max_tokens by the deployment's own cap * test(bedrock): annotate the new Mantle default max_tokens tests with Final * refactor(bedrock): inject the AnthropicConfig max_tokens reader into the Invoke Claude config --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
508a98586e
|
fix(databricks): join plain text blocks for non-Claude json_object requests (#49385)
* fix(databricks): join plain text blocks for non-Claude json_object requests * fix(databricks): join only the last message for json_object and drop cache_control under drop_params Databricks checks for "json" only in the last message, so earlier turns now go out as sent and only the last message, plus any system messages merged into it, is joined into a string. Under drop_params a text block's cache_control is dropped so that message can be joined too, and without it the request still goes out as a list. * fix(databricks): skip dropped assistant turns and blank parts when joining for json_object The json_object join picked the last message before the request transform drops bare assistant turns, so a trailing thinking-only or reasoning-only turn (a Responses input ending in a reasoning item included) left the real last user message as a list and Databricks still returned its 400. The pick now skips the turns that transform drops. The join also ran when only one part had text, which sent whitespace-only parts the existing empty block filter used to drop on requests that already worked. It now joins only two or more non-blank parts, so a request with one non-blank part goes out exactly as before. Without drop_params, a message with a message-level cache_control now goes out as sent instead of being rewritten into one cached block. * fix(databricks): join json_object text parts only on models Databricks hosts Databricks checks for the word "json" only on the models it hosts (databricks-* pay per token names and system.ai.* AI Gateway names). An external model endpoint skips that check and takes a list of text parts today, so its request now goes out as sent instead of joined. The drop_params read now goes through drops_unsupported_params, and a test pins that a text block with a truthy non-string text is kept. * fix(databricks): reject a text part holding a non-string locally with a clear 400 * refactor(databricks): reject a non-string text part at the transform_request boundary * fix(databricks): read pydantic message objects in the non-string text check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
f91774e2f9
|
fix(fireworks): keep developer items in place for models that accept a later system message (#49399)
* fix(fireworks): keep developer items in place for models that accept a later system message * fix(fireworks_ai): keep developer items in place for short Fireworks model names and stop flagging gpt-oss-120b * fix(fireworks): drop the mid-conversation system flag from nemotron-lightning nemotron-lightning-3p5-30b-a3b followed a later instruction less often with the developer item kept in place than folded, so both of its cost map keys go back to folding. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
a2ab27373f
|
fix(proxy): count the context_management edits /v1/messages applies on every bridged provider (#49234)
* fix(proxy): count the context_management edits /v1/messages applies on every bridged provider * fix: apply context_management edits in count_tokens on Responses bridges, compaction blocks and Bedrock Claude The Responses bridge now applies the same edits inference applies for every provider, the chat bridge cuts the history at a client compaction block, and Bedrock Claude forwards the edits AWS accepts with their beta to the mantle and runtime count APIs instead of dropping them. * refactor(count_tokens): return count-view failures as values and raise them once in the caller * fix(bedrock): keep the runtime's 400 when mantle does not serve the model it retries * fix(count_tokens): forward context edits on Vertex and Azure Claude counts and keep Bedrock's mantle answer without edits * fix(proxy): count edited bodies on native routes without a count API, keep counts on editor failures * fix(proxy): keep the Bedrock pass-through count shape and provider 400s on empty bridged bodies The pass-through count picks Converse or InvokeModel from the content again, and the edited InvokeModel count wraps string turns in text blocks. A bridged view with no messages sends the body as before. Azure and Vertex load the beta config off the event loop before a count with edits, and a model with no deployment reads drop_params from the request and router defaults. * fix(bedrock): send an empty-string turn as is when a count forwards context edits Wrapping an empty string into a text block made AWS reject the count with 'text content blocks must be non-empty', while base and /v1/messages accept the empty string. Only non-empty string turns are wrapped now. * test(azure_ai): compare the count's betas with the betas Azure inference sends * test(azure_ai): record the Azure inference request with respx instead of appending to a list --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tavren27
|
a0b1edc0a0
|
fix(providers): trim prompt_cache_breakpoint to what each API takes under drop_params (#49064)
* fix(openai): trim a malformed prompt_cache_breakpoint under drop_params on chat completions * fix(openai): leave a non-object content part for OpenAI to reject under drop_params * fix(openai): trim an unsupported prompt_cache_breakpoint ttl on every path and on azure_ai chat under drop_params * fix: trim prompt_cache_breakpoint to what Azure and Mantle accept under drop_params Azure OpenAI chat now gets the same content part trim as OpenAI chat, and the Azure and Foundry gates also read a deployment's declared base_model. Each Responses config keeps only the ttl its API takes: Azure keeps 30m, and Mantle keeps no ttl. A trimmed ttl is logged at debug. * fix(responses): trim prompt_cache_breakpoint only on APIs known to read it Other Responses providers get the marker whole or not at all, as on main. Azure chat now trims for a deployment with no base_model unless the cost map marks the model as not reading breakpoints, and Azure AI Foundry Responses keeps Azure's 30m-only shape. * fix(azure): trim prompt cache breakpoints only for deployments known to read them Azure and Azure AI Responses keep main's whole-or-nothing rule for unmapped deployment names, an opted-in custom OpenAI host trims the ttl and keeps the mode, and Azure chat strips every marker for a known non-reader under drop_params * fix(providers): keep main's marker handling where the target won't trim it A /v1/responses model with unknown breakpoint support on OpenAI's host gets a marker whole or not at all, as on main. The chat to Responses bridge keeps main's whole-marker check when a non-bool global drop_params leaves the target's trim off. A hyphenated name like gpt-5-6-sol now reads as gpt-5.6. * fix(providers): read a non-version hyphen suffix like -32k as part of the model name * fix(providers): keep main's reading of a dotted minor version like gpt-5.6o * fix(providers): keep main's breakpoint shape for unmapped Mantle models and skip the Azure lookup without drop_params --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mika-lt
|
40e4e512d0
|
fix(anthropic): stream azure_ai natively on an Azure OpenAI host on /v1/messages (#49216)
* fix(anthropic): stream azure_ai natively on an Azure OpenAI host on /v1/messages The /v1/messages bridges got the requested model (azure_ai/gpt-6-luna) with the resolved provider (azure), so the Responses bridge faked the stream and the chat bridge called a deployment named azure_ai/gpt-6-luna (404). Pass the resolved provider prefix when the requested prefix is a different known provider. * test(anthropic): cover the bridged model with respx and a regional bedrock_mantle id * fix(anthropic): carry azure_ai env credentials into bridged /v1/messages calls When an azure_ai deployment's api_base and key come only from AZURE_AI_API_BASE and AZURE_AI_API_KEY, the provider the async wrapper derived from the model alone replaced the model's own resolution, so the bridged call lost both and failed. The model's own resolution now wins when it names the same provider, an explicit provider that disagrees still wins, and the bridge carries the resolved key and base. * fix(anthropic): keep the caller's key, proxy routing and MCP credentials on bridged /v1/messages calls * fix(anthropic): keep an azure deployment's Entra credential over the env key on bridged /v1/messages calls * fix(anthropic): skip the sign-in lookup for chatgpt and github_copilot model prefixes on bridged /v1/messages calls * fix(anthropic): keep the Azure credential of an azure_ai deployment that sets its own api_base on bridged /v1/messages calls * test(anthropic): annotate the parameters of the bridged /v1/messages tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kaustubh Anil Jain
|
6f3251c7e3
|
fix(responses): serialize sequence_number on every bridged streaming event (#44944)
* fix(responses): serialize sequence_number on every bridged streaming event * test(responses): cover sequence_number on bridged streaming events * fix(responses): read undeclared fields through dict accessors and number guardrail block continuations * fix(responses): scope dict reads that see undeclared fields to Responses stream events Moves the extras-aware __getitem__, get, __contains__ and items from the shared response base onto a new base for the 37 stream event classes, so usage, Gemini, Interactions and moderation objects keep reading __dict__ and Gemini responses keep _hidden_params out of .get * fix(router): renumber Responses stream events only after a fallback takes over A stream that never falls back now keeps its provider's own sequence numbers, repeats included, so a starting_after resume still matches them. After a fallback the wrapper continues above the highest number the client saw * fix(guardrails): keep Responses stream numbers unique when LLM Shield flushes held text The restorer's flush copied the last delta and kept its sequence_number, so a native stream repeated a number and a bridged stream sent the flush unnumbered. A flush now takes the done event's slot and later events move up by one, and a stream the provider never closed numbers its flush after the last event. Also pins the created_at in the streaming iterator test fixture instead of reading the clock. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tavren27
|
1046716b6d
|
fix(anthropic)!: sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept (#49409)
* fix(anthropic): sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept * test(e2e): send the azure_ai beta check without the code execution tool * test(anthropic): require a vertex_ai and azure_ai decision for every anthropic beta * fix(anthropic): forward the betas Claude Code sends that anthropic, vertex_ai, and azure_ai accept * test(e2e): send Vertex's header-only betas in the beta acceptance case * fix(anthropic): keep forwarding Vertex computer-use betas and point beta tests at platforms that still lack them A live probe found each Claude 4.5 and 4.6 model on Vertex needs one of the two computer-use betas for the computer tool, so dropping both broke it. The e2e acceptance test leaves those two out of its all-at-once Vertex call, and the integration and unit tests that relied on a beta being dropped now use one the platform still lacks. * test(anthropic): point unsupported-beta tests at betas the platforms still drop The Azure AI handler test used claude-code-20250219 as the beta Azure drops, and the Rust parse test used it as an unknown value. This PR forwards it on Azure AI and maps it, so both now use values that stay dropped or unknown. * fix(anthropic): add computer-use-2025-11-24 for the computer_20251124 tool The computer_20251124 tool fell through to the default computer-use-2024-10-22 beta, which the provider rejects for that tool version. Map it to computer-use-2025-11-24, which the beta map now forwards on anthropic and azure_ai * fix(anthropic): send no computer use beta for an unknown computer tool version An unknown computer tool version fell back to computer-use-2024-10-22, which the anthropic and azure_ai columns now forward. It now gets no beta, as every platform effectively sent before. Every computer tool version also skips defer_loading and allowed_callers, computer_20251124 included * test(azure_ai): read the branch's beta map in the auto-injected beta wire test * fix(anthropic): keep defer_loading and allowed_callers on the computer_20251124 chat tool * test(anthropic): read the branch's beta map in the azure_ai safeguards beta wire test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
572832dea1
|
fix(gemini)!: answer 400 for an unsupported chat audio format, drop it under drop_params (#49423)
* fix(gemini): answer 400 for an unsupported chat audio format, drop it under drop_params * fix(gemini): return the audio format error as a value and keep direct mapper callers refused The format helper returns its UnsupportedParamsError for the caller to raise. The shared voice mapper refuses an unsupported format again for direct SDK callers, the error caps the echoed format at 100 characters, and its advice no longer claims the model only emits pcm16 * fix(gemini): refuse an unsupported audio format nested in speech_config A format inside audio.speech_config skipped the check and was silently dropped with a 200. It now gets the same 400 unless drop_params is set. The 400 test also pins the model name in the chat-path message. * fix(gemini): show a non-string audio format as sent in the 400 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tin-berri
|
36dbb228d9
|
fix(streaming): preserve served models in OpenRouter streams (#49498) | ||
|
tin-berri
|
ae8666b9d1
|
fix(http): diagnose aiohttp connection pool saturation (#49499) | ||
|
moe-berri
|
3bd75f1258
|
fix(proxy): treat empty agent and skill search embedding model as not configured (#49485)
* test(agents): cover empty search embedding model settings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(agents): treat an empty search embedding model as not configured Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(tests): sort search integration imports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(tests): format search embedding tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): annotate search test locals and isolate upstream call checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
alexcastgonz
|
d2321581ec
|
fix(cohere): emit streamed tool calls from v2 tool-call-start and tool-call-delta chunks (#47794)
* fix(cohere): emit streamed tool calls from v2 tool-call-start and tool-call-delta chunks * fix(cohere): type _parse_tool_call_delta without growing basedpyright counts The reassigned Final local and the untyped chunk lookups added basedpyright errors past the base ceiling. Narrow them with casts; per-rule counts for the file are now at or below the merge-base. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(cohere): parse streamed tool calls with a typed model and map Cohere TOOL_CALL to tool_calls Read the streamed tool call through a frozen pydantic model instead of casts, which the strict and type-discipline gates refuse. Map Cohere's TOOL_CALL and STOP_SEQUENCE finish reasons, so a stream that reads Cohere's message-end still ends tool_calls. The regression test now drives litellm.acompletion with two parallel streamed calls and checks the rebuilt calls and the final finish reason * fix(cohere): close a streamed tool call that sent no arguments with an empty JSON object --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
simonp9
|
f7b2cd0000
|
fix(anthropic): answer 400 naming a non-string role or tool call id instead of 500 (#49237)
* fix(anthropic): answer 400 naming a non-string role or tool call id instead of 500 A list role or an object or list tool_call_id crashed the chat conversion with "unhashable type". Depending on the step that hit it, callers got a 500 or a 400 that never named the field. A failed conversion now answers 400 naming the first non-string role or tool call id, and the prompt caching hook reads a non-string answer id as matching no tool call. * fix(anthropic): name the field the conversion failed on, and answer 400 for a non-string Responses call_id The chat field check now runs only when the conversion raised a type error, and scans roles first, then assistant tool call ids, then tool answer ids, so an orphan tool answer that modify_params drops is no longer blamed. A tool output call_id that is not a string on /v1/responses answers 400 naming input[N].call_id, and the clear_tool_uses polyfill leaves a non-string tool_use_id for the provider * fix(anthropic): name a non-string role whatever error the conversion raised An int or bool role fails the conversion with an invalid message error, not a type error, so it lost the named 400. Roles are now named for any conversion failure, while tool call ids stay limited to type errors * fix(responses): name a non-string message item role by its input index * fix(responses): name a malformed input field only where litellm reads it, by the caller's index A tool output call_id is named only when it is an object or a list and the conversion fails, so an orphan numeric id still converts and another field's failure is never blamed on it. On the bridge to Claude a non-string role or function_call call_id is named by its input item, counting items before a compaction item. A numeric assistant tool call id on chat is named too. * fix(anthropic): name a tool id only for the error it causes and count Codex tool items in input indexes A numeric tool answer id that modify_params drops was blamed for any later type or attribute error, so the 400 hid the real cause. Tool ids are now named only for the error their value raises, and a hashable answer id never. The /v1/responses bridge and its input token count now read the input the caller sent, with only additional_tools hoisted into tools, so input[N] counts those items too. * fix(anthropic): name a tool id only for the error its own value causes and keep Codex tool item text out of the request * fix(responses): name a tool output call_id only when the items before it convert * fix(responses): name a non-string input field only on inference so token counting and batch records keep their answer * fix(responses): let a client field named caller_input or names_non_string_fields reach the bridge * test(anthropic): build the non-string id histories without editing them in place * fix(anthropic): never name an empty role for another field's failure * fix(anthropic): name an empty list or object role when the conversion fails on it * fix(anthropic): name a list role under the drop_params cache cap and keep the Bedrock batch line prefix --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
shrey
|
7143aeeb86
|
feat(passthrough): add Deepgram streaming text-to-speech WebSocket route (#49282) | ||
|
moe-berri
|
7e4403ea66
|
fix(a2a): send agent litellm_params.api_key as bearer auth on the A2A hop (#49474)
* test(a2a): cover agent api key authentication Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(a2a): authenticate plain agent hops with api key Co-authored-by: Afnan Mojumder <afnan.mojumder@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Afnan Mojumder <afnan.mojumder@gmail.com> |
||
|
Kerry Lu
|
6f86b117b5
|
fix(moderations): stop the router image check from 500ing on a list input (#49454)
* fix(moderations): stop the router image check from 500ing on a list input * fix(moderations): give the list cast its cast-ok reason * fix(moderations): type the responses input guard and keep SDK compaction items * fix(moderations): import the compaction item through litellm.types and mark the TypeIs import * fix(moderations): sort the compaction item import --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
efc2304b7e
|
fix(streaming)!: fail a stream that ends without a finish reason (#47665)
* fix(streaming): fail on clean EOF without finish reason * fix(streaming): cover shared OpenAI-compatible EOF paths * test(router): complete mocked streaming responses * fix(streaming): keep the EOF error message and cover Anthropic clean EOF The request-less RemoteProtocolError tripped exception mapping into 'The .request property has not been set.' for non-OpenAI providers. Adds Anthropic and generic-chunk unit tests, an integration test across chat, the /v1/messages bridge and the /v1/responses bridge, and moves main's token-estimate test onto a finished stream with a sibling that checks the cut stream still bills the estimate on its failure log. * fix(streaming): count a finish reason any chunk sent, keep LangGraph async streams whole Gemini marks every chunk unfinished and a later usage chunk reset intermittent_finish_reason, so a finished Gemini stream failed at EOF. LangGraph's __aiter__ opened a new aiter_lines on every wrapper call, which hit StreamConsumed after the first chunk and now surfaced as a false EOF error instead of a silent truncation. * fix(streaming): keep the synthetic stop for a stream that sent no chunks Gemini's OpenAI-compatible endpoint answers with only [DONE] when reasoning uses up max_tokens. Failing every openai/azure stream that sent nothing turned that 200 into a 500, while every other provider already kept the stop for an empty stream. Adds integration cells for Gemini's trailing usage chunk, multi-event LangGraph runs, a mocked deployment and the [DONE]-only answer. * test(streaming): keep pytest.raises to one statement, drop the spend read for an empty stream An empty stream writes no spend row on main either, so the [DONE]-only cell checks only what the client sees. * fix(streaming): read the chunk's finish reason for the sticky flag * fix(agentcore): end a stream on messageStop when the runtime sends no metadata or final message * test(streaming): pin the logged-chunks guard, inject the LangGraph client, use current model names * fix(agentcore): say why the end-of-stream flags rebind * test(streaming): cover an AgentCore runtime that ends on messageStop through the proxy * test(streaming): cover SDK clients, request fallbacks and a concurrent burst for a cut stream * test(agentcore): inject the async transport instead of patching the aiohttp flag * test(guardrails): end the decision model chat fixture with a finish reason, as a real stream does * test(streaming): shape the AgentCore and LangGraph fixtures like captures from a real runtime and server * test(streaming): keep each stream's ending and land every spend row once through a spend log outage * test(streaming): end each stream by its provider's finish after a proxy worker is killed mid burst * test(streaming): run the outage test as one burst before and one during the outage instead of a capped wave loop * test(streaming): connect the outage test's proxy to the relay without TLS so its trigger sees the spend-log insert on an ssl=on Postgres * test(streaming): type the new stream EOF unit tests * test(streaming): a merged reasoning stream cut mid reasoning now fails Main's think tag test expected a stream that ends mid reasoning without a finish reason to finish with a closed think block. With this PR that stream fails, so the case moves to its own test that expects MidStreamFallbackError with the open think block as the partial content --------- Co-authored-by: King Star <mcxin.y@gmail.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
ishaan-j-berri
|
54892d6a62
|
fix(oci): preserve all Gemini system instructions (#48813) (#49458) | ||
|
Kerry Lu
|
3cb2e60de0
|
fix(cost): split a provider-reported total between input and output by the cost map rate ratio (#49357)
* fix(cost): split a provider-reported total between input and output by the cost map rate ratio Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): expect the kyma reported-cost headers to follow the cost map rate split Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): keep the kyma test file at its original formatting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): price the response-cost header split from the served model, not the request alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
be4f3d36ce
|
feat(accounting): capture request usage through the durable meter (#49133)
* feat(accounting): capture request usage through the durable meter * fix(accounting): preserve native Google usage compatibility * fix(accounting): preserve prepared usage observation on failure * fix(accounting): exclude sync acknowledgement policy rejections from capture * test(accounting): verify capture races and failure isolation * fix(accounting): preserve request support with sync acknowledgements * fix(accounting): normalize usage metadata before queue admission * fix(accounting): preserve prior attempt links on successful fallback * fix(accounting): preserve attempt evidence through retries and failures * fix(accounting): depend on prior receipts without resubmitting usage * fix(accounting): preserve pricing inputs and resolved capture attribution * fix(accounting): persist batch usage within capture bounds * test(accounting): cover batch dependency failure and backpressure * fix(accounting): isolate batch persistence and preserve trusted attribution * fix(accounting): share byte admission across batch producers and writes * fix(accounting): reserve batch snapshot reads before admission * fix(accounting): compact repeated batch observations without losing requests --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
a713ed6bab
|
fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them (#47692)
* test(e2e): add gemini tool call id length repro for LIT-8912 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them * fix(gemini): pair a tool result with its call when the client re-adds the thought signature to the call id Strands, which the AgentCore harness runs, puts provider_specific_fields.thought_signature back on the assistant tool call id as <id>__thought__<signature> while its tool result keeps the bare id, so the Gemini conversion found no matching call and answered 500. Ids now match with the signature suffix stripped. Also adds thought_signature_in_tool_call_id to the litellm_params inventory test and gives the pass-through tests' mocked logging objects the litellm_params a real Logging object always has. * fix(gemini): keep the thought signature on streamed Messages and Responses tool calls and accept a string setting * fix(gemini): read the thought signature back from Responses input and coerce the per-request setting A Responses function_call input item now keeps its provider_specific_fields, so a client that sends the item back gets its real thought signature replayed to Gemini. A per-request thought_signature_in_tool_call_id of "false" or 0 now keeps ids short, and a value that is not a boolean answers 400, or is dropped under drop_params, instead of a 500. The e2e test now uses the sdk fixture and @meta subjects * fix(caching): key the response cache on thought_signature_in_tool_call_id * fix(responses): send thought signature fields only with short ids and drop malformed routing kwargs whole * fix(main): validate routing kwargs after the locals snapshot so fallbacks and model_list calls keep working The e2e test now uses gemini-3.8-flash with enough max_tokens to finish thinking before the tool call. * fix(gemini): keep non-chat routes tolerant of a malformed thought_signature_in_tool_call_id, match tool results by exact id first, and keep the deployment's value when drop_params drops a malformed override * fix(router): validate strict litellm params through a helper so star imports cannot re-bind a public Final * perf(router): skip strict validation when a request overrides no deployment param * fix(responses): keep a dict tool call's thought signature on the streamed function_call done item * refactor(main): return invalid supplemental params as a value and test the streaming tool call with a real response --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
paveln83
|
febd340120
|
fix(guardrails)!: keep the upstream error when a /v1/messages stream ends in one (#49088)
* fix(guardrails): keep the upstream error when a /v1/messages stream ends in one Raw-SSE guardrail stream hooks assembled the whole stream, and the Anthropic chunk parser raises on an error event, so a stream the upstream ended with event: error failed assembly. tool_permission then blocked it with a 400 that hid the upstream error, bedrock and model_armor emitted a could-not-assemble block, and presidio skipped masking. Assembly now leaves the error events out, so the delivered content is checked as usual and a clean stream passes through with its error frame. A rewritten stream ends with the upstream error instead of a message_delta and message_stop the upstream never sent. tool_permission also passes a stream of only error frames, which carries no tool call to check. * fix(guardrails): do not close a block the failed upstream left open when re-emitting a rewritten stream * fix(guardrails): check only the finished blocks of a stream cut off by an upstream error tool_permission and Presidio now assemble an error-ended stream through the error with an opt-in keyword, leaving out a tool call or thinking block the upstream cut off. Logging, OpenTelemetry, Bedrock, and Model Armor keep the old assembly. A rewrite drops the text block stop only when the upstream left its text block open. * fix(guardrails): keep a cut-off tool call from the client and stop each rewritten block before the next starts A failed stream holding a cut-off tool_use is now re-emitted through the rewrite, which leaves the call out and keeps the upstream error, so a denied call never reaches the client. The rewrite leaves its text block open only when it is the last block, and keeps a message_delta the upstream sent before failing. The new tool_permission tests no longer patch should_run_guardrail. * fix(guardrails): pass a cut-off stream with nothing denied through as sent, minus the cut-off tool call Re-emitting such a stream through the rewrite sent an empty text block and stopped it, and Claude Code does not retry a failed stream that shows a finished block, so it showed a mid-response error instead of retrying. Dropping only the cut-off call's events keeps the stream as the upstream sent it. * fix(guardrails): strip a cut-off tool call only from a stream the upstream failed Assembly leaves unfinished blocks out of the check only when the stream carries an upstream error event, so a stream with an unfinished but allowed tool call and no error frame now reaches the client as sent instead of losing a call the check already read * test(guardrails): type the parameters of the new stream error tests * fix(guardrails): skip SSE data lines that are not JSON objects when reading a stream A data line holding a bare JSON value such as 5 made the stream helpers call .get on an int, so tool_permission and Presidio answered 500 where main answered 400 or passed the stream through * fix(anthropic): skip SSE data lines that are not JSON objects when assembling a stream * test(guardrails): annotate the upstream error stream tests * fix(anthropic): read a stream's non-object lines and named error frames the same way everywhere Run the skip for a data line that is not a JSON object through extract_sse_data and the assembler loop, so the usage fallback keeps a stream's tokens and the chat handler keeps its contract. Presidio now holds a leading non-object frame until the first object frame, so the rest of the stream is still masked or blocked. An event: error frame now counts by its name, so a rewrite keeps it. * fix(guardrails): refuse an unreadable stream in Presidio and cut every kind of tool call A non-object JSON line with no Anthropic frame after it now gets a 400 from Presidio before any data byte, since passing it would send text main refused. A frame named error stays a normal event when its data is another Anthropic event. server_tool_use and mcp_tool_use cut off by an upstream error are dropped like tool_use, and a leading frame with empty data is relayed at once. * fix(guardrails): refuse what Presidio cannot read past a non-object line and keep non-string error types out of the frozenset Presidio now refuses a stream assembly cannot read when it holds a non-object data line, instead of relaying it unmasked. Non-string event types no longer crash a rewrite, and a CRLF stream that opens with a typed error is no longer read as one error, so its tool calls are still checked. * fix(guardrails): refuse a first frame whose type is a list or dict in Presidio Presidio passed PII at 200 where main answered 500 when the first read was a JSON object with a list or dict type, or a JSON array with a character cut at the end of the read. has_unreadable_json_event now reads both as data no reader of these streams can take, so the first-frame check refuses them. * test(guardrails): cover Presidio's refusal when a word or null line comes before a Gemini stream The first read then holds the unreadable line behind one Presidio can pass, so the check has to look at every frame in it. * fix(guardrails): treat a stream as error-only only when every data frame is an error A tool call split over data: lines beside one error frame read as error-only, so tool_permission passed it, and an SDK joins those lines and reads the call. The check now joins a frame's data lines and refuses anything but an error. * fix(guardrails): refuse a stream past an upstream error when a client could read a frame differently Run 8 of the live risk pass showed each stream reader imitating a client and missing a shape: a whitespace-led data line, an error key with no type, a split tool call after a plain message_start. An error frame now has to be plain, and assembly past an error returns None for any frame a client could read differently, so those streams keep main's refusal. * fix(guardrails): on a failed upstream, deliver only what tool_permission read A failed stream now reaches the client through the rewrite that already serves a denial, followed by the upstream error, even when nothing is denied. Raw frames could carry a tool call the check never read in shapes such as a reused block index, mixed line ends, a tool in message_start, interleaved deltas, or a form feed, and each refusal rule only moved the search to the next shape. The error frame goes out as its event name and first data line. An error is found at CR, LF and CRLF line ends, and the rewrite adds no finished block when it read no text, so Claude Code still retries the stream. * fix(guardrails): read an error frame's name as clients do and send the error alone when a rewrite cannot be built A data-less event: error frame with junk after the name is an unknown event to both SDKs, so a healthy stream stays raw. A finished tool call whose input is not JSON cannot be rebuilt, so a failed stream sends its upstream error alone, and a healthy stream still raises as before. * fix(guardrails): read an error frame as a client dispatches it and keep its JSON whole Only the last event line of a frame a blank line ended names an error, so a healthy stream with a later event line or an unterminated error is no failed stream. Any rebuild failure on a failed stream now sends the upstream error alone with a warning, a param rule denies an input nested too deep, and a raw line break inside a JSON error stays escaped so every reader gets it whole. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
27fbcd9895
|
fix(anthropic): map pause_turn and return refusal explanations on chat and responses (#49192)
* fix(anthropic): map pause_turn and return refusal explanations on chat and responses pause_turn now maps to finish_reason length (Python and Rust) and keeps the raw stop_reason in provider_specific_fields. Responses reports a paused turn as incomplete with max_output_tokens and shows the unrun server search as an in_progress web_search_call instead of a client function_call. A refusal's stop_details.explanation now fills message.refusal and the final stream delta.refusal on chat, and becomes a refusal content part on Responses, streamed or not. * fix(anthropic): let a chat client resend a paused turn so Claude continues it * fix(anthropic): replay a paused chat turn without its thinking blocks, and drop its unrun search before a new message Chat cannot say where Claude wrote each thinking block in a paused web search turn, and Anthropic answers 400 when one moves but accepts the turn without them. A new message after a paused turn also 400s on a server_tool_use with no result, so that unrun search is dropped. * fix(responses): send the streamed refusal part's type and text in content_part.added * fix(anthropic): keep pause_turn and Claude refusal explanations out of the Messages adapter's text A paused Claude turn now ends on stop_reason pause_turn instead of max_tokens, streamed or not. A Claude refusal keeps its explanation in stop_details with Claude's category and no longer repeats it as a text block, so a guardrail mask on a streamed refusal is delivered. A paused turn sent back with list content drops its thinking blocks like string content does * fix(anthropic): replay a paused turn with one unrun search and no thinking on chat and Responses * fix(responses): report a paused Claude turn as completed so Codex and the Agents SDK keep going * fix(responses): mark a paused turn's message item completed * fix(responses): mark every item of a paused turn completed * fix(responses): keep a refusal's explanation when a refused turn is sent back A refusal part in a Responses input message now goes to the model as text, so the assistant turn is no longer dropped from the history. The integration test now expects a paused Claude turn on Responses to report completed, and the paused-turn test helper is typed * fix(responses): scan refusal parts in Responses guardrails --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
quietjules
|
cee72ffdd6
|
fix(proxy): count native Responses WebSocket turn tokens toward key and tag TPM limits (#49257)
* fix(proxy): count Responses WebSocket turn tokens toward key and tag TPM limits A native Responses WebSocket turn logs its raw events instead of a usage-carrying response object, so the v3 rate limiter read no usage and never charged the key or tag token window. Read the turn's usage from its events the way the cost calculator does, and charge the TPM-limited tags the socket was opened under. * fix(proxy): refuse socket turns past TPM and count managed, failed, and realtime usage A later response.create turn on an open Responses WebSocket now gets a read-only check of every TPM window it counts toward (key, team, user, org, end user, team member, model-scoped limits and the socket's tags). A turn with no tokens left gets the 429 rate_limit_exceeded frame and the socket stays open. The check never reserves or increments a counter and leaves RPM, budget, PTU and project windows to their own paths. It runs on native and managed sockets. Managed turns carry the socket's call id in their metadata, so they charge the socket's TPM-limited tags. A failed native turn charges the partial usage its failure row already carries. A socket refused at open by a rate limit gets the same refusal frame and closes 1008, logged once without a traceback. /v1/realtime sessions now count toward TPM through the realtime usage combiner, and the v1 limiter reads socket usage through the same helper * fix(proxy): count failed socket turns, keep socket usage off team PTU, check agent windows per turn A native socket turn that ends in response.failed now counts the usage that event reports toward spend and TPM, as the HTTP stream path does. Native socket and realtime usage no longer charge the team PTU counter, matching the merge base. The per-turn TPM check now also reads the agent and agent session windows the socket opened with. * fix(proxy): start missing legacy user, team, and end-user TPM buckets at zero * fix(proxy): stop charging a realtime session's whole usage into its close minute Realtime and live sessions log once at close, so counting them put a long session's tokens into one TPM window and refused calls that stayed under the limit. They count no TPM again, as before this branch. Also covers the legacy v1 count-once fix with an HTTP regression case. * test(proxy): check each per-turn TPM window on its own * fix(proxy): send a fail-closed per-turn socket refusal with its 503 status --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |