825 commits

Author	SHA1	Message	Date
paveln83	7bd342d7ea	fix(proxy): let a config license unlock enterprise settings and spell "environment" right in license errors (#49745) ... * fix(proxy): spell "environment" right in the allowed_ips and AWS KMS V2 license errors * fix(proxy): read general_settings.litellm_license before the allowed_ips and enforced_params gates * fix(proxy): read the config license before callbacks and gates, keeping a valid env license * fix(proxy): keep an invalid config license reported and sync license data with the license in use A config license that fails no longer overwrites the license string when the env license is not valid either, so /health/license still reports has_license. A failed boot-time env license is not re-checked. Both license writes go through one helper that also refreshes premium_user_data, so /health/license and the seat limits follow the license in use. * fix(proxy): keep the env license's boot result when a config license fails, without a second license check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 09:56:08 +00:00
norkelbyte	864b98acec	fix(bedrock)!: fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 (#49170) ... * fix(bedrock): fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 The Bedrock SDK ends the output stream the same way for a dropped connection as for a session that sent sessionEnd, so the handler treated a dropped stream as a clean finish. The client got a 1000 close with no error event and the router logged 200 OK with no retry An output end now counts as clean only after the client left and the handler began closing the Bedrock input. An earlier end raises a 503 BedrockError, so the client gets an error event and a 1011 close, the failure handlers run, partial spend is still logged, and a retry of a committed session is still refused * fix(bedrock): fail a dropped Nova Sonic session right away and log it as failed When Bedrock's output side ends mid-session, by a drop or a modeled stream error, the client side no longer spends the close timeout sending close messages into the dead input stream, so the error reaches the app right after the drop. A session the app ends itself still sends Bedrock its close messages and closes the input The replay refusal now leads with the committed cause, so the error event and the cut 1011 close reason both start with it. It stays a 400 and is tagged as a failure outside the selected deployment, so router retries and fallbacks of a dropped session no longer add extra failure counts A clean realtime session now counts once toward its deployment's cooldown success counter, so cooldown reads the real failure rate instead of 100 percent. Realtime tpm and rpm counting is unchanged The spend row of a session the provider ended now reads failure with its error information, while the events already forwarded are still billed. A session the app ended keeps logging success * fix(bedrock): agree on a dropped realtime session's failure across loggers and the spend row A realtime session's success event now reports status failure, with the same error_information the spend row stores, when the session recorded a provider failure in its metadata. Usage already streamed stays billed and the router still counts one cooldown failure The spend row builder now runs every row's error_information through sanitize_error_information_for_spend_logs (DB size cap plus prompt redaction), so the realtime failure rows get the same treatment as the proxy failure hook, which no longer sanitizes on its own to avoid truncating twice Drop the selection-failure tag on the committed-session refusal, since the shared logging object already keeps a retried refusal out of cooldown, and pin the router's realtime success counting with an OpenAI realtime deployment * fix(bedrock): count realtime router successes only for sessions that logged one, and log first-turn drops The router counts a realtime success only when the session's success handlers ran, so a refused handshake or an upstream close before any frame no longer offsets its failure. The success event's payload status stays success again, so each logger gets one failure record per dropped session, while the spend row still reads failure from the metadata the Bedrock handler records. A committed Bedrock session that fails before any logged event now gets one zero-cost Logs row. After an output failure the input stream is closed in the background, so the SDK body writer finishes without delaying the client's error. Every input send failure reports the same 503 stream-ended cause as an output drop * fix(realtime): count a router success only for a session that ended cleanly A realtime session the upstream closes with 1011 or 1006 after its first frame now marks itself as ended abnormally where that close is seen, in the shared relay and in the OpenAI Live handler, so the router no longer counts it as a cooldown success. The success callbacks still run, so the proxy keeps the budget and parallel-slot settlement it had A Bedrock session that fails before Bedrock sends any usage now logs no success event, as on base. The spend-row failure fields go into a new metadata mapping instead of the request's own dict, validated with a pydantic TypeAdapter instead of an unchecked TypeIs guard. The handler tests drop a wall-clock bound and type their helpers * fix(bedrock): log a dropped realtime session's usage from its own logging state The failure event replaced the shared standard_logging_object while the queued success dispatch was still running its callbacks, so loggers received $0 and 0 tokens for a billed dropped session. * fix(proxy): settle a rate-limit reservation once when a request logs a failure and a success A dropped realtime session fires a failure event and then a success event for the same call. The v3 limiter refunded the TPM reservation on the failure and released it again on the success, so the key's TPM counter went negative. The success path now skips a reservation the failure already released and marks it settled for a later failure. The realtime handler test callbacks are typed * fix(rate-limit): settle each token reservation once when a request logs both a failure and a success A dropped realtime session logs a success for its usage and then a failure. After a success the limiter now clears every reserved amount (TPM, PTU, ITPM, OTPM), so a later failure refunds nothing twice. A new reservation written in the same context after a released one now starts clean, so reserve, fail, reserve, succeed charges the second request's usage once instead of 78 TPM or 28 ITPM for 27 and 20 used * fix(rate-limit): take a token hold before awaiting so concurrent success and failure callbacks settle it once * fix(rate-limit): track the ITPM/OTPM hold's release apart from TPM so a nested pre-call keeps the request's project IO charge * test(rate-limit): pin the split hold-release flags across nested MCP pre-calls * fix(realtime): count a relay that fails on its own as an abnormal end * test(realtime): pin the relay failure exit with a client frame the provider rejects --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:56:23 +00:00
Emerson Gomes	ecb8ef5ca8	feat(realtime)!: support latest OpenAI audio models (#35600) ... * feat(realtime): support latest OpenAI audio models * fix(transcription): type SDK stream duration metadata explicitly * fix(realtime): preserve transport settings and translation duration * refactor(realtime): reuse SDK client initialization * fix(audio): bill finalized streams and authorize transcription models * refactor(audio): keep transcription pricing selection together * fix(realtime): reconcile model metadata and test contracts * fix(realtime): preserve provider duration and model capabilities * fix(realtime): keep transcription validation compatible with cost map guard * fix(realtime): address transcription and billing review findings * fix(realtime): preserve input usage and provider routing * fix(realtime): bill retained output and route Azure aliases * fix(realtime): reject malformed provider durations * fix(realtime): preserve batch cache rates and Azure v1 routes * fix(transcription): reject nonfinite provider durations * refactor(realtime): align annotations with current lint rules * fix(realtime): preserve translation deployment pricing and input usage * test(transcription): verify synchronous stream finalization * fix(realtime): retain missing translation input usage at close * fix(transcription): reject streams that bypass output guardrails * fix(realtime): reconcile audio protocols billing and current CI * fix(realtime): trust confirmed formats and authorize default models * fix(realtime): preserve absent models and reject unsupported guardrails * fix(realtime): reject translation policy pipelines * chore(ci): rerun transient infrastructure timing failure * fix(realtime): reconcile audio protocols with current upstream * fix(audio): validate provider responses at SDK boundaries * fix(realtime): preserve minimum OpenAI SDK compatibility * fix(realtime): preserve deferred Live session authorization * fix(realtime): preserve upstream authentication and transcription behavior * fix(audio): scope provider duration precedence to configured models * fix(typing): avoid Final declarations inside transcription cost loop * refactor(audio): reuse transcription mapping and duration resolution * fix(realtime): enforce model budgets and preserve translation routing * fix(audio): log streamed transcripts whose done event reports detected languages * fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers Providers without a streaming transcription handler got UnsupportedParamsError for stream=True, languages and keywords, where the merge base forwarded them as provider-specific params. Only openai and azure validate them now. * fix(audio): bill streamed gpt-transcribe from the provider's reported duration The stream path computed cost after logging had replaced the transcription's duration usage, so it fell back to the measured file length while the unstreamed call billed the provider's seconds. * fix(transcription): log streamed transcription usage from the provider response The stream logging path built the spend payload from the response after its duration usage was transformed away, so whisper-1 with stream=true and streamed gpt-transcribe logged an empty usage object. Bill and log from the provider's TranscriptionResponse in both the sync and async paths. * fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes - Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape - Use the input transcription model as the session model only for transcription sessions - Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main - Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set - Return x-litellm headers on streamed transcriptions and mark the request done - Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure - Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing - Bill translation sessions on output audio seconds only, at the single published rate - Document the 201 SDP answer on the translation calls routes and the transcription 200 response * fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the default dated version on the deployments route for every model. The SDK client secret keeps session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the gpt-transcribe format check only reads the request provider's cost map entry. * fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it Transcription reaches every provider as on main again, sends stream only when asked, forwards language and languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from one source and spend rows carry the billed seconds. Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA get beta-shaped session events, and transcription sessions send the deployment model in both shapes. * fix(realtime): address review findings on transcription billing, streaming and realtime auth Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata * fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only * fix(realtime): preserve prior merge resolutions after rebase * fix(realtime): enforce translation policies and normalize Azure transcription URLs * fix(cookbook): stop translation once output idles after the upload ends * fix(realtime): bill translation on input audio and drop pre-read stream cost headers Translation spend now uses the input audio seconds the session sent, priced through input_cost_per_second, instead of the translated output duration. A streamed transcription no longer returns cost headers before the stream is read, since the cost is only known after the last event. Also removes an unused realtime guardrail helper. * fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop * test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription * fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA Bill translation sessions from the deployment's own per-second rates, add per-second prices for Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models * fix(realtime): bill a translation rate the deployment omits from the session model * test(e2e): check translated speech is not English instead of exact French wording * fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending * fix(realtime): keep the proxy's realtime session gate off translation sessions A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects * fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params. * fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree * fix(model-catalog): accept the realtime translation endpoints in the cost map * fix(cookbook): close the translation session and read until session.closed * fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard An empty `stream` form value runs the transcription unstreamed again, as OpenAI does, instead of answering 400. The four translate cost-map entries drop the translation routes from supported_endpoints until the main branch's catalog parser accepts them, and the nested transcription model lookup uses the router registry entry point. * test(proxy): drop the mid-stream error frame pin that main's error shape change replaced * fix(transcription): bill uploaded seconds when a stream fails mid-transcript * fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream * fix(audio): keep transcription's positional arguments and redact streamed error details - Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning - Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do - Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite * fix(audio): stream the guarded transcript to SDK callers that asked for a stream * fix(logging): keep a streamed transcript out of callbacks that turned off message logging A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does * fix(proxy): stream a transcription the deployment streamed when the client sent no stream field --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:27:52 +00:00
tessellark	f4a2e2b3af	fix(anthropic): send an https PDF image_url to Claude as a document (#49708) ... * fix(anthropic): send an https PDF image_url to Claude as a document A PDF link sent as image_url to Claude on Anthropic went out as an image block and Anthropic answered 400. A link whose path ends in .pdf is now downloaded like Vertex links are, and the served Content-Type picks the document or image block, so CDN links that serve a PNG from a .pdf path keep working. An explicit format of application/pdf sends a url document with no download. PDF media types match regardless of case or parameters, and count_tokens picks the same block as the request. * fix(anthropic): download https PDF links off the event loop The proxy downloaded an https link ending in .pdf synchronously while building the Anthropic request, so on an async call every other request on that worker waited for the download. Those links are now inlined before the request is built, the same way http links already are, and the /v1/messages guardrail write-back does the same. A non-string format on an image_url is ignored the way it already was inside the image_url mapping, instead of raising an AttributeError. * fix(anthropic): send a PDF link the proxy cannot download as a link and keep guardrail url blocks as sent * test(anthropic): expect the PNG media type the served bytes carry at a .jpg path * fix(anthropic): bound the PDF link download and keep query-string PDF names Give up on a .pdf link after NON_LLM_CONNECTION_TIMEOUT instead of the client's 6000 s default, fall back to main's whole-URL extension check so download?file=report.pdf still counts as a PDF, treat application/x-pdf as a PDF, and set the url document's application/pdf format only for the guardrail write-back so other /v1/messages providers send what main sends * fix(anthropic): probe a .pdf link and send it to Claude as a url document Instead of downloading a .pdf link and inlining it as base64, ask the host for its first kilobyte and send the link as a url document when it serves a PDF. Every other link goes out as the url image main sends, so robots.txt handling, caching and the wire format stay as they are on main. Also restore the guardrail adapter and the content type inference to main, keep the client's url document blocks (title, context, citations, cache_control) through the guardrail write-back, treat a PDF filename in the query string as a PDF link, probe links concurrently on the sync path, cap the async probe's read at the download limit, and catch only transport, URL and SSRF failures in the probe * fix(anthropic): read only the first kilobyte when probing a PDF link, and restore each url document from its own place The probe now streams the response and keeps the first 1 KB, so a host that ignores Range no longer fills memory with up to 20 parallel bodies. The guardrail write-back now matches url documents inside a tool_result and outside one separately, since the write-back moves tool results ahead of other content * fix(anthropic): remember each PDF link probe's verdict and cap its total time, and skip the probe on Bedrock invoke * fix(anthropic): keep a PDF probe's error status for a minute, not ten --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:05:15 +00:00
Devin Gupta	216041e243	feat(decisions): add Parallel AI web-grounded decisions (#49188) ... * feat(decisions): add Parallel AI web-grounded decisions * fix(decisions): name the Parallel model decision * fix(parallel_ai): restrict environment credentials to trusted decision endpoints * fix(decisions): keep Parallel evidence fields off other providers and handle missing confidence Only fields a provider names reach the OpenAI response, so existing providers keep their base shape. A malformed api_base answers 400 on both credential paths. The complexity router classifier and the Playground handle a provider that omits confidence. Regenerates the lazy OpenAPI snapshot and the dashboard API types * chore(cost-map): declare the Decisions routes on parallel_ai/decision The provider endpoint files are generated from the cost map, so the row must name both routes for the generated System One flag to hold * fix(decisions): report no classifier confidence when a provider omits it A choice answer with no confidence is a valid classifier answer, so the complexity router picks its tier and leaves classifier_confidence and the confidence signal out instead of reporting a different metric in its place. Also show the Parallel AI logo in the provider picker and add a Parallel translation fixture captured from the live API for both Decisions routes. * fix(decisions): keep a token-priced row on token billing when it also sets a query price A plain input_cost_per_query bills flat only on a row with no token price, and input_cost_per_query_decisions always bills flat, so a custom Decisions row that already bills tokens costs the same as before this PR * fix(decisions): bill a deployment's query price and accept Parallel bases that end in a route A proxy request logs through an object built before the router picks the deployment, so a query price on litellm_params never reached Decisions billing. Parallel Search accepts a base ending in /v1/search and shares PARALLEL_AI_API_BASE with Decisions, so Decisions now accepts that base and one ending in /v1/decisions. * fix(decisions): keep per-query billing and optional confidence on Parallel only A query price on any other Decisions provider moved that provider off token billing, and a choice or score answer with no confidence became a 200 with a null for every provider. Both now apply to Parallel alone, so other providers bill and fail as they do on main. A Parallel deployment with explicit zero token rates and a query price read as free, so every budget check was skipped while it billed per call. It now counts as priced. The dashboard key for Parallel is now PARALLEL_AI so a served bundle that predates the provider still sends parallel_ai as the provider slug, and the README table lists the provider. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 04:47:02 +00:00
devin-ai-integration[bot]	08b951a814	fix(vllm): list hosted_vllm wildcard models when the server has no api key (#40066) ... * fix(proxy): expand wildcard models for providers with model info but no cost map entry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): stop requiring an api key to list vllm models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve model ids during wildcard expansion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): forward api key for model discovery Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): support bearer auth for model discovery Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vllm): list hosted_vllm models from the server chat uses, with a bounded wait The hosted_vllm listing now resolves HOSTED_VLLM_API_BASE and HOSTED_VLLM_API_KEY like hosted_vllm chat, instead of the /vllm pass-through's VLLM_API_BASE, and waits at most NON_LLM_CONNECTION_TIMEOUT rather than the completion timeout. vllm/* keeps listing nothing, since its chat runs in process. The router pass-through now forwards the deployment's api_key. * fix(vllm): skip the placeholder key, honor deployment TLS and headers, and retry failed listings after a minute * fix(discovery): cache only timed-out model listings, so a fast failure is tried again on the next call * fix(proxy): list wildcard models off the event loop, so a hung provider no longer stalls other requests * fix(utils): keep a concurrent call's model listing when another call times out * fix(proxy): run model discovery on its own thread pool and keep credential lookups off the event loop * fix(proxy): stop the file credential lookup at the first match before listing wildcard models The fallback lookup now walks the allowed model names lazily, in the same order as before, so a hung vLLM server is listed only when no configured name serves the requested provider. The vLLM auth headers no longer mutate the caller's headers dict. * fix(proxy): try every configured wildcard route before listing any wildcard's models in the file credential lookup * fix(proxy): answer model lists from cached listings first and share one listing per server * test(vllm): give the self-signed cert fixture fixed validity dates * fix(proxy): skip wildcard servers a key cannot reach on /v1/model/info /v1/model/info listed every wildcard deployment and filtered by the key afterwards, so a key limited to one model waited on every wildcard server and sent listing calls to servers it cannot use. The listing now skips a wildcard whose patterns match none of the key's models before it asks the server. A key or team granted hosted_vllm/* with no deployment of that name now lists no models, like openrouter/*, since chat answers 400 for every id the environment server would list. * fix(proxy): match a wildcard's own row pattern before asking its server The /v1/model/info pre-filter matched a key's models against both the deployment name and its litellm_params.model, so a key granted hosted_vllm/* still asked a vllm-secure/* -> hosted_vllm/* server whose rows it can never see. It now matches only the pattern expansion names the rows with, picked by one shared helper. * test(vllm): cover the vLLM listings and no-slash wildcard aliases the patch left unrun Adds tests for hosted_vllm get_models with and without a server, vllm/* never asking a configured server, and /model/info pre-filtering of wildcard aliases whose name has no slash. Drops a test comment that only repeated a method signature. --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 04:13:28 +00:00
ryan-crabbe-berri	be50cbe9d8	fix(responses)!: authorize and decode comparison response IDs (#49402) ... * test(responses): reproduce comparison response ID ownership and wire failures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): guard stale retries and malformed comparison envelopes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): authorize and unwrap comparison response ids Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): require comparison IDs to follow response security policy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): deny raw comparison response ids by default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): reject raw comparison response ids by default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): enforce managed ownership with unmanaged ID opt-in Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 03:40:46 +00:00
zoveki	704e73d6fe	fix(cost-map): give the bare gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini aliases their 2027-01-14 shutdown date (#49707) ... * fix(cost-map): give the bare gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini aliases their 2027-01-14 shutdown date * fix(proxy): report a deployment's own provider retirement date before the bare model's An Azure deployment declared as model gpt-4o with custom_llm_provider azure, or as azure/<deployment> with base_model gpt-4o, now resolves azure/gpt-4o's date instead of the bare OpenAI row's. The GPT-4 series alias test now reads both cost maps with typed rows * fix(proxy): read Azure's retirement schedule for azure_ai deployments An azure_ai deployment with no azure_ai row of its own now reports the azure/<model> date, and a cost-map row wins over the provider-less copy the router registers under the deployment's key, which carries the bare model's date. * fix(proxy): let custom and registered base_model rows keep their deprecation date Only the copy the router writes at startup, which has no provider and inherits the bare row's date, gives way to a later candidate * fix(proxy): report a pinned snapshot's own earlier deprecation date over its base_model's * fix(proxy): recognize the router's copied cost-map rows by the loaded catalog * fix(proxy): read Azure's schedule for a provider-prefixed azure_ai base_model --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 02:58:39 +00:00
alberto	e63067a23b	fix(anthropic_endpoints)!: return Anthropic error envelopes on /v1/messages auth failures (#48466) ... * fix: Return Anthropic error envelopes on Messages auth failures Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Answer Anthropic envelopes on Messages auth failures without relabelling unmapped statuses Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(proxy): answer the Anthropic error envelope on the /anthropic pass-through The pass-through relays upstream errors in Anthropic's shape, but the proxy's own refusals there (bad key, model access, budget) still answered the OpenAI body, so an Anthropic SDK pointed at /anthropic could not read them * fix(proxy): answer the Anthropic envelope for 404, 405 and unhandled 500 errors on Messages routes A disabled Claude Code gateway 404, a wrong-method 405 and an unhandled 500 on the Messages and /anthropic/ routes kept FastAPI's detail body or the OpenAI shape. The central HTTP and unhandled-exception handlers now pick the Anthropic envelope by route. The dashboard playground toast titles Anthropic error types when the body has no code. * fix(proxy): keep LiteLLM error types and Anthropic envelopes on Messages routes Error bodies on the Messages routes now carry error.litellm_error_type when the LiteLLM type differs from the Anthropic type, so clients and the Admin UI keep causes like budget_exceeded. Unlisted 4xx statuses map to invalid_request_error per Anthropic's errors page, model-list requests that send anthropic-version get the Anthropic envelope, and the admission, decompression and request-size middleware errors use it on these routes too. * test(proxy): expect litellm_error_type in Messages integration tests and type middleware tests * fix(proxy): keep handler error bodies unchanged and timeouts as api_error Errors the Messages and count_tokens handlers already returned in the Anthropic shape keep their original bodies, so litellm_error_type now appears only on errors the proxy raises before the handler. A 408 stays api_error as before, and the playground toast titles the new Anthropic types and ignores long numbers in front of an embedded error * fix(proxy): keep the base error types inside the Messages handler and scope toast titles to Anthropic bodies The wider status table (402 billing_error, 409 conflict_error, 504 timeout_error, other unlisted 4xx invalid_request_error) now applies only to errors the proxy raises before the handler. Errors inside the Messages handler, streamed error frames and the SDK streaming adapters keep the merge base's table. The dashboard toast reads a status before embedded JSON and titles by Anthropic type only when the body is an Anthropic error envelope, so other pages keep their titles. * test(anthropic): mark the new test variables Final --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 01:53:29 +00:00
jesus-berri	acd084ab38	fix(mcp): answer bare OPTIONS on MCP routes without running LiteLLM auth (#49497) ... Every MCP route (/mcp, /<server>/mcp, /toolset/<name>/mcp, /mcp/proxy) ran LiteLLM auth before looking at the HTTP method, so a bare OPTIONS probe got the RFC 9728 401 OAuth challenge and clients that probe with OPTIONS looped on re-auth. The Streamable HTTP handler now returns 204 with Allow: GET, POST, DELETE, OPTIONS before auth runs. GET, POST and DELETE are unchanged. Resolves LIT-10231	2026-11-02 16:16:29 -08:00
paveln83	b84493d9a7	fix(proxy): log each endpoint's own call type on failed audio, moderation, image, and rerank spend rows (#49684) ... * fix(proxy): record the route's call type on failed speech spend rows A failed /v1/audio/speech request never puts a logging object on the request data, so the failure lift returned nothing and the spend row's call_type came out empty (or whatever the client sent in the body). Without a logging object, lift the call type the route already resolved, so a failure row records the same call type a success row on that route records * fix(proxy): keep the call type of a failure logged under its own route An MCP tool listing that fails inside a Responses, chat, or messages call is logged under /mcp/list_tools, so its spend row keeps list_mcp_tools instead of taking the outer request's call type * fix(proxy): take a failed row's call type from its endpoint, not a route path guess * fix(proxy): log the endpoint call type on failed image generation and rerank rows * fix(proxy): estimate failed-call ledger usage from the route's call type again The failure usage snapshot now gets the same call type the spend row uses for its estimate, so a failed Responses call counts input from the newest compaction item again, as on main. The endpoint call type tests now go through real routers whose provider answers 400, instead of asserting on a mocked failure hook. * test(proxy): type the failed spend row tests and fixture --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 23:57:11 +00:00
Mateo Wang	a5fe912ecf	fix(proxy): preserve DB settings during config reload (#49735) ... * fix(proxy): preserve DB settings during config reload Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): mark reload settings variable as final --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-02 23:55:10 +00:00
paveln83	7fe6b1de34	fix(proxy)!: ignore stored general_settings keys no settings API manages (#48967) ... Apply database general settings only for managed keys, warn about ignored copies, and allow admins to delete stale stored keys Move intended database-only settings into config.yaml before upgrading, including authentication and access restrictions	2026-11-02 14:58:46 -08:00
Etienne Chabert	523b87ec71	fix(proxy): charge budget windows when the cached key or team expired before the request ended (#47608) ... * fix(proxy): charge budget windows when the cached key or team expired before the request ended The post-call spend update reads a key's or team's budget_limits from user_api_key_cache and returned early on a miss, skipping both the window counter increment and the LiteLLM_BudgetWindowSpend row. The cached object expires after user_api_key_cache_ttl (60s by default), so any request that ran longer than that without another request refreshing the entry was counted in the key's lifetime spend but never in its budget windows. With budget reservation disabled nothing else charges the windows, so keys kept running past their window caps On a miss the update now reads budget_limits straight from the key or team row. It does not re-cache the object, since a write from the spend path could race a /key/delete and put a just-deleted key back into the auth cache * test(proxy): assert the budget window fallback never re-caches the key or team * fix(proxy): read a missing key or team row once per cache TTL in the budget window fallback The master key and UI session tokens are logged under names no key row carries, so every one of their requests missed the auth cache and read the database. A missing row is now remembered for the cache TTL in this process only. A failed read is not remembered, and a row that exists is read fresh on every miss so a key update is never served stale. * fix(proxy): skip the budget window row read for a key or team with no windows The fallback remembered only a missing row, so a key or team that exists but holds no budget_limits was read from the database on every request that outlived the key cache. It now remembers both cases under one cache key built in user_api_key_cache.py, for the same TTL as the auth cache * fix(proxy): keep no-window markers in their own cache and clear them when an update adds windows Markers for keys and teams with no budget windows lived in the shared 200-entry auth cache, so a burst of late costs could evict cached teams and users. They now live in a dedicated in-memory cache on UserApiKeyCache, reached by local and broadcast evictions. /key/update and /team/update now evict the marker on every worker, so a late cost after an update charges the windows the update added. * test(proxy): give the spend counter tests the proxy's real key cache and type the window spend helper * fix(proxy): keep a row read that an update outran from restoring the no-window marker * fix(proxy): clear finished no-window reads and evict a marker-shaped user id from both caches * fix(proxy): apply other spend counters before a window row read and let no-window notes follow the key cache settings --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 14:41:59 -08:00
simonp9	250af8f027	fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) ... * fix(responses): log the encrypted response id on streamed /v1/responses spend rows With response id security on, the stream hook encrypts the resp_ id on every frame, but logging had already received the SDK's own copy of the terminal response, so the spend row, the standard logging payload and callbacks got the managed id (or the chatcmpl id on a bridged stream) the client never saw. The hook now records the exact id it advertises on the request's logging object, and the logging layer stamps that id on the logged streamed response * fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response, and a raw provider id still gets a managed id Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or <id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id * fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner A GET or cancel of a stored response now logs a new encrypted id of its own, so callbacks and spend rows never reuse the create's id, while the owner of the id still gets back the exact id they sent. An admin reading someone else's response gets an id bound to the admin, as before. The spend row id suffix for reads and cancels is gone, so with response id security off the rows and session history match the default branch. A streamed GET follows the same rules. * fix(responses): echo the sent id to every reader that passes the response access check A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed the access check, not only when the caller is the user or team the id was issued to. An admin, the master key or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not reach the first turn from it, so a bridged follow-up through that id lost the conversation history Every read and cancel still logs a fresh id bound to the reader, never the create's id * fix(responses): keep a read or cancel row in session history only when it brings new output A GET, streamed GET or cancel sent with a session header joins the create's session, and its stored response repeats the create's output. The replay then sent the same assistant turn once per read. Session history now keeps an aget_responses or acancel_responses row only when its stored Responses output has an item id that no earlier kept row carried. A polled background response keeps the first poll that returns the finished output, since its queued create row has none, and later polls of the same output are dropped. Every other row keeps today's rule * fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart A create row now owns its output items in session history, and a read or cancel row adds only items no create and no earlier read carried, in one linear pass, so a background response polled while it grows, or a read that ends before its streamed create, replays the answer once and after its prompt DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id like a read does Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row id, so a streamed background create and its cost event, which now share the client's id, keep separate files The integration tests that decoded the managed id now match the exact id the client received * fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage file and unbatched GCS object, so batch retrieves and events with no id keep main's names An item a session read replays takes the newest read's copy at the first read's place, so an item a poll caught half written replays finished Tests pin the client's id over the router's continued response id, and the storage test no longer reads the clock after the upload * fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items A streamed background create and its later cost event share the client's id, so OpenMeter dropped the second event and DynamoDB overwrote the create's item. Both now key the cost event as <id>_background_cost, like its spend row and storage file. Session history again replays a read's output items that carry no id, as main did. * fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras * fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id A streamed background create and its later cost event now share the id the client received, so ClickHouse and DeepEval saw both events under one id. Route both through logged_event_key, the way the Azure storage, GCS, OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse request_id matches its spend row's `<id>_background_cost` and its DeepEval span gets its own uuid. ClickHouse response_id stays the raw provider id. * test(deepeval): annotate the background cost span test's locals with Final * test(responses): run the DELETE wire test that pins the echoed client id DELETE /v1/responses/{id} now answers with the id the caller sent, so the skip that marked it as a known bug no longer applies. The test passes on the /v1, /openai/v1 and bare routes. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 14:13:17 -08:00
tessellark	c720b23238	feat(proxy)!: honor model_info.max_input_tokens in the Codex catalog for stock models (#49701) ... * feat(proxy): honor model_info.max_input_tokens in the Codex catalog for stock models A Codex model catalog entry copied from Codex's stock entry (a stock slug, or an alias whose upstream is one) now takes the max_input_tokens every routable deployment of the group sets as its context_window, capped at the stock entry's max_context_window like Codex's own model_context_window. With nothing set, the stock window stays. * feat(proxy): honor the smallest max_input_tokens a group's deployments set in the Codex catalog A deployment without a limit counts as Codex's own window, so a limit on only some deployments can lower the window but never raise it * fix(proxy): skip a zero or negative max_input_tokens when sizing a Codex catalog window * test(proxy): stub the max_input_tokens group field and type the Codex window tests * fix(proxy): skip a max_input_tokens a saved /model/info response copied from the cost map --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 13:27:58 -08:00
Prabal Pratap Singh (ZeroX)	b3c397e964	feat(hosted_vllm)!: keep vLLM's stop_reason and token_ids on chat choices and stream chunks (#48393) ... * feat(hosted_vllm): keep vLLM's stop_reason and token_ids on chat choices Non-streamed hosted_vllm chat completions moved vLLM's choice-level stop_reason and token_ids into provider_specific_fields, so clients written against vLLM could not find them where vLLM puts them. They are now also set on the choice itself, and the copy in provider_specific_fields stays for existing readers. Fixes #47945 * feat(hosted_vllm): keep stop_reason and token_ids on streamed chunks and proxy stream output Streamed chat finish chunks and streamed /v1/completions finish chunks now carry vLLM's stop_reason. Each stream chunk keeps the token_ids vLLM sent with it, including held-back empty chunks, and the proxy no longer strips those chunks as empty usage chunks or drops the fields in its fast serializer. * fix(hosted_vllm): keep the upstream choice index on token-id-only stream chunks * fix(hosted_vllm): keep vLLM choice fields for null, cache hits, n>1, and streamed completions Response conversion keeps stop_reason and token_ids on the choice for any OpenAI compatible response, cache replays carry them, finish fields are recorded per choice index, text completion streams keep token_ids, and the proxy passes only these two keys through, nulls included. * fix(hosted_vllm): send empty text on token-id chunks, keep fields on their own choice, and keep null stop_reason on slow-path streams Token-id-only chunks now carry an empty content or text string, a finish chunk only takes stop_reason and token_ids from its own upstream choice, content chunks no longer repeat stop_reason, the slow stream serializer keeps a null stop_reason or token_ids, and fields cached under provider_specific_fields come back on the choice * fix(hosted_vllm): keep stop_reason on the last content chunk, keep both fields in built stream responses, and key the cache on return_token_ids * fix(hosted_vllm): keep another choice's final token_ids and stop_reason on their own stream chunk With n above 1, a finish-only vLLM chunk for a choice other than the one the single finish chunk carries was dropped or taken as choice 0's finish, losing that choice's last token_ids. It now goes out as an id-only chunk on its own choice index. * fix(hosted_vllm): send the first choice's finish chunk before another choice's later token-id chunks * fix(streaming): keep each streamed text completion choice on its own index * fix(streaming): finish each streamed text completion choice on its own index and null token_ids in redacted logs * fix(proxy): drop token_ids from guarded streams, redact choice-level provider fields, and keep one message_delta per Anthropic stream * fix(proxy): skip token-id-only chunks on guarded streams and put stop_reason on the final text chunk * fix(proxy): drop token_ids from the logged and cached response of a guarded stream * fix(proxy): drop token_ids under per-chunk stream checks and redact text-completion stream choices * fix(caching): key return_token_ids and continuation_token sent in extra_body * fix(streaming): drop token ids from streamed chunks while post_call_rules run vLLM holds text back behind a possible stop string but sends its token ids at once, so an SDK stream sent ids before litellm.post_call_rules checked their text. While those rules are set, streamed choices drop token_ids and chunks that carry only ids are skipped, as the proxy already does ahead of a per-chunk check * fix(streaming): keep vLLM fields when a stream's only choice index is not 0 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 13:26:29 -08:00
simonp9	61da6fcb4d	fix(mcp)!: answer 400 for a hosted MCP tool a translated route cannot run, drop it under drop_params (#48931) ... * fix(bedrock, gemini, deepseek, databricks, gigachat, anthropic): answer 400 for a hosted MCP tool a route cannot run, drop it under drop_params A `type: "mcp"` tool sent to a route that rebuilds tools for its provider (Bedrock Converse and Nova, Bedrock Invoke Claude, Vertex AI Claude, Vertex AI and Google AI Studio Gemini, DeepSeek, Databricks Claude, GigaChat) was dropped silently and the request still answered 200. These routes now answer UnsupportedParamsError (400) naming the model, route and server, and pointing to the LiteLLM MCP gateway. Under drop_params the tool is dropped with a warning, other tools are kept, and no empty tools field is sent. Anthropic, Azure AI Claude and Claude Platform on AWS keep mapping the tool to the MCP connector. Native Responses providers and raw OpenAI-compatible forwarders are unchanged. * fix(mcp): check hosted MCP tools once per request and keep base tool_choice handling Each route now runs one request-level step before its param mapping. When the drop leaves no tools, the request sends no tools, tool_choice or parallel_tool_calls. Converse and GigaChat go back to their base tool_choice and empty-tools handling. * refactor(deepseek): hold the sent tools in a tuple * fix(mcp): keep tool_choice for legacy functions when every hosted MCP tool drops * fix(mcp): answer 400 for MCP servers a route cannot run on Messages, Ollama and the Responses playground /v1/messages requests with mcp_servers on a chat-adapter or Responses-bridge route now answer 400 naming each server, or drop it with a warning under drop_params, instead of being forwarded or silently ignored. Ollama generate no longer pastes a hosted MCP tool (headers included) into the prompt, and ollama_chat filters hosted MCP tools the same way. The Admin UI Responses playground now sends litellm_proxy/mcp references like the other endpoints. * fix(ollama): leave JSON mode off when drop_params drops every hosted MCP tool Ollama generate set format json for any request with tools, even after the hosted MCP check dropped all of them, so a plain answer came back as JSON or as a call to a tool the model never had. The check now runs before the JSON mode switch, which stays on only while a tool is left for the prompt. * fix(messages): honor additional_drop_params for mcp_servers and read the beta map only for hosted MCP tools * fix(mcp_security): accept gateway server names and toolsets the gateway resolves The guardrail compared litellm_proxy/mcp/<name> refs against registry keys, which are server ids, so a saved server referenced by name, alias, or toolset name was blocked although the gateway routes it. It now resolves each ref the way the gateway does and still blocks unknown names. * fix(tool_fields): keep MCP drop warnings on one log line Server labels, model names, and routes come from the caller, so a line break in one could forge extra log lines. The drop warnings now strip CR and LF from each value. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 13:13:03 -08:00
tavren27	a4de720144	fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests the way their hosts bill them (#49074) ... * fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests like their chat templates Render grok-4-1, grok-4.3 and DeepSeek V4.1 tool schemas the way each chat template does, and add a per-deployment token_count_tool_framing next to token_count_framing for the hidden tool preamble a host bills. The Azure AI Foundry grok-4-1, xAI grok-4.3 and Fireworks DeepSeek V4.1 cost map rows carry the measured framing, overridable in model_info * fix(count_tokens): count Azure grok-4-1 Responses requests with their own framing and use the tool framing in place of the base framing * fix(count_tokens): keep operator framing on tool and Responses counts, frame bridged Responses like chat, follow base_model for tool rendering, count grok tool_choice * fix(token_counter): count Azure Responses strict default, skip false strict and built-in-only tools, per-kind operator framing * fix(token_counter): count an explicit null strict like false on Azure Responses * fix(count_tokens): follow the chat completions bridge flag and Azure's strict default on azure/ deployments Keep Azure grok-4-1 chat tool counts at or over the bill by raising its tool framing to 302 * fix(token_counter): keep strict false for non-grok templates, coerce the bridge flag, frame DeepSeek API counts * fix(proxy): frame Responses counts for the api_base router defaults send inference to * fix(proxy): keep the api_base helper's name so main's /v1/messages count caller resolves --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 12:37:35 -08:00
tessellark	a2ab27373f	fix(proxy): count the context_management edits /v1/messages applies on every bridged provider (#49234) ... * fix(proxy): count the context_management edits /v1/messages applies on every bridged provider * fix: apply context_management edits in count_tokens on Responses bridges, compaction blocks and Bedrock Claude The Responses bridge now applies the same edits inference applies for every provider, the chat bridge cuts the history at a client compaction block, and Bedrock Claude forwards the edits AWS accepts with their beta to the mantle and runtime count APIs instead of dropping them. * refactor(count_tokens): return count-view failures as values and raise them once in the caller * fix(bedrock): keep the runtime's 400 when mantle does not serve the model it retries * fix(count_tokens): forward context edits on Vertex and Azure Claude counts and keep Bedrock's mantle answer without edits * fix(proxy): count edited bodies on native routes without a count API, keep counts on editor failures * fix(proxy): keep the Bedrock pass-through count shape and provider 400s on empty bridged bodies The pass-through count picks Converse or InvokeModel from the content again, and the edited InvokeModel count wraps string turns in text blocks. A bridged view with no messages sends the body as before. Azure and Vertex load the beta config off the event loop before a count with edits, and a model with no deployment reads drop_params from the request and router defaults. * fix(bedrock): send an empty-string turn as is when a count forwards context edits Wrapping an empty string into a text block made AWS reject the count with 'text content blocks must be non-empty', while base and /v1/messages accept the empty string. Only non-empty string turns are wrapped now. * test(azure_ai): compare the count's betas with the betas Azure inference sends * test(azure_ai): record the Azure inference request with respx instead of appending to a list --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 10:38:35 -08:00
quietjules	16ee2c71a7	fix(proxy): keep provider fields on streamed chat completion chunks (#49398) ... * fix(proxy): keep provider fields on streamed chat completion chunks The hand-written serializer for plain text chunks dropped every field it did not list, so Azure's per-choice content_filter_results, the chunk's obfuscation, and delta reasoning_details and reasoning_items never reached clients since v1.86.0. Every chunk now serializes with model_dump_json, which keeps them and is faster per chunk. * fix(proxy): send streamed chunk frames as bytes so plain text frames keep their type * fix(proxy): send null for chat chunk values JSON cannot encode instead of failing the stream * fix(proxy): drop chat chunk fields JSON cannot encode instead of failing the stream A callback or provider value pydantic cannot encode, like invalid UTF-8 bytes, a lone surrogate, a circular dict, or an unknown type, now leaves only that field out of the frame with a warning naming it. Responses API events keep raising. * fix(proxy): drop only the nested chunk value JSON cannot encode and keep lone surrogates Main removed the StreamingChoices import, so the field drop helpers now take any BaseModel. A value JSON cannot encode is left out down to the nested key or list item that holds it, and a lone surrogate goes out as an ASCII JSON escape. * fix(proxy): keep streaming when an unencodable value sits under non-string dict keys --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 09:55:38 -08:00
zoveki	33457d3ac9	fix(proxy): check the model in every OpenAI websocket passthrough frame (#48979) ... * test(proxy): cover per-frame model checks and session pricing on the OpenAI websocket passthrough * fix(proxy): check the model in every OpenAI websocket passthrough frame and price the session * fix(proxy): bill each OpenAI websocket turn on its own row and run full auth per frame Each finished turn (completed, incomplete, or failed) now logs its own spend row under the model and access groups that allowed it, so group budgets see spend as turns end instead of at close. Every model a frame names goes through the key's full auth, and a stored prompt reference with no model is refused for a key held to some models. A frame still in flight when the client leaves is recorded before the relay gives up on the send * fix(proxy): keep the stored prompt refusal's first sentence inside the websocket close reason The close reason carries the refusal's first sentence, and the stored prompt sentence ran past the 123 bytes a close frame allows, so it was cut mid-word. The sentence is shorter now and the rest of the explanation stays in the error event. * fix(proxy): bill pipelined websocket turns to their own frame and close the review gaps * fix(proxy): log no extra websocket spend row for frames after the last turn * fix(proxy): price websocket Responses turns by served tier and hosted tools, and bill turns a client walked out of A websocket Responses turn is now priced through the same Responses cost path the HTTP passthrough uses, so a priority-tier answer and a hosted web search cost what they cost over HTTP. When the client leaves mid-answer, the gateway keeps reading the provider's socket until that turn ends, up to 10 minutes, so the turn the provider still bills reaches spend * fix(proxy): bill websocket turns by lane, skip turns OpenAI refuses, and stop frame checks counting the session's own reservation An error event refuses a queued response.create, so the next turn is no longer charged to the refused frame's model. Turns on separate stream_id lanes are charged to the frame that started each one. The frame model gate settles the connect reservation before it re-runs auth, so a key with less budget left than one turn's max cost is no longer refused on its first frame. * fix(proxy): bill each websocket lane cut off at session end on its own row When a client leaves while turns on several stream_id lanes are still running, each unfinished turn now gets its own spend row under the groups and model its own frame asked for, instead of one merged row. Also pins the PR's new single-assigned test variables as Final and swaps datetime.now() in the logging-handler and success-handler fixtures for fixed timestamps. * test(proxy): mark the remaining new websocket test variables Final Builds the logging mocks with model_call_details in the constructor instead of setting it afterwards. * fix(proxy): check every model the realtime URL names before the OpenAI websocket relay connects * fix(proxy): keep reading running answers after a refused websocket frame so they are billed * fix(proxy): bill realtime transcription sessions on the OpenAI websocket passthrough A transcription session names its model only in session.updated under audio.input.transcription, and reports usage on each completed transcription instead of response.done, so its row logged unknown at $0 and a budgeted key never spent. * test(proxy): mark the turn-taking client's delivered list Final * fix(proxy): hold a websocket session's parallel slot until the socket closes Each finished turn logged its spend row under the session's own call id, so the first turn's success callback freed the key's max_parallel_requests slot while the socket was still open, and a key limited to one session could open several at once. The session now claims its slot under its own call id, every turn row logs under a call id of its own, and the slot is released when the session ends, as before Priced turn rows also take their row's call id as their id, so two turns that start in the same instant no longer share a spend log id and drop one row * fix(proxy): bill realtime transcription clips that finish outside an answer A conversation.item.input_audio_transcription.completed frame that arrived after a session's last response.done was never billed, because the session end only owes a row for a turn still running. A clip that completes while no answer is running now logs its own row, and a clip inside a running answer stays in that answer's row Each row also carries the session's latest session.updated frame, since later rows lost the transcription model it names and priced clips against the realtime model. A newer session.updated replaces the older one, so the stale model is never read first It also drops the docstrings this PR had added, since the names already say what each helper does * fix(cost): price a realtime transcription clip at the transcription model the session set A realtime conversation session opens with session.created naming the realtime model and no transcription model, and the client sets the transcription model later in session.update. The cost code took the first session frame and fell back to its session model, so a clip was priced at the realtime model. That model has no per-second rate, so a gpt-transcribe clip cost $0, and a token-billed clip used the realtime model's audio rate instead of the transcription model's. Now the newest session frame that names a transcription model sets the price, and the first session model is only the fallback when no frame names one, as before * fix(cost): price each realtime transcription clip at the model set when it ran A session that changes its transcription model mid-way kept every logged clip on the newest model. Each clip now takes the transcription model from the last session event before it, and falls back to the session's own model * fix(passthrough): keep the session update a buffered clip ran under until its answer is billed A transcription that completes while an answer streams waits in the buffer for that answer's response.done. A session.updated in between used to drop the earlier update, so the clip was priced at the opening configuration. An earlier update now stays while an unbilled clip follows it, and drops at the next update once that clip is billed. * fix(passthrough): keep reading a websocket turn the client left before OpenAI started it A client that sends response.create and hangs up before OpenAI answers with response.created made the walkout checks see no turn in flight, so the gateway closed the provider socket and the turn OpenAI still ran and billed never reached spend. The relay now counts every forwarded create until the provider starts or refuses it, and keeps draining while one is owed * fix(passthrough): settle a websocket turn that ends without its start frame A turn the provider ended without sending response.created first left its create counted as not started, so a client leaving later held the provider socket for the whole walkout drain with nothing left to bill. A turn's end now settles one waiting create when its lane saw no start frame * fix(proxy): refuse unparseable and unroutable websocket frames and hold legacy slots per socket A client frame too deep or too long for the JSON parser closed the provider socket mid-turn, so the turn in flight was never billed. The gate now refuses that frame with an error event and keeps draining the open turn. A stream_id OpenAI rejects comes back as an error that names no lane, which dropped another turn's authorization. The gate now refuses those frames before they reach OpenAI. With the legacy parallel request limiter, each per-turn spend row freed the key's slot while the socket stayed open. Websocket passthrough rows now free nothing, and the socket frees its slot once when it closes. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 09:48:59 -08:00
jesus-berri	3b62b6311c	fix(proxy): make /update/sso_settings a true PATCH and send only changed SSO fields from the UI (#49527) ... * fix(proxy): make /update/sso_settings a true PATCH and send only changed SSO fields from the UI * fix(proxy): mask SSO client secrets in the /update/sso_settings response * fix(proxy): keep SSO values that only live in the environment_variables row on a partial SSO PATCH The SSO PATCH strips every SSO env var from the environment_variables config row. Now that the UI only sends changed fields, an omitted value that lived only in that row was dropped from both tables and lost on restart. Carry those values into the SSO row before stripping them * test(ui): pin CLEARED_SSO_SETTINGS to exactly the backend SSOConfig fields With PATCH semantics an omitted key keeps its stored value, so a field missing from the Clear list would silently survive Clear SSO settings	2026-11-02 09:14:10 -08:00
Kaustubh Anil Jain	6f3251c7e3	fix(responses): serialize sequence_number on every bridged streaming event (#44944) ... * fix(responses): serialize sequence_number on every bridged streaming event * test(responses): cover sequence_number on bridged streaming events * fix(responses): read undeclared fields through dict accessors and number guardrail block continuations * fix(responses): scope dict reads that see undeclared fields to Responses stream events Moves the extras-aware __getitem__, get, __contains__ and items from the shared response base onto a new base for the 37 stream event classes, so usage, Gemini, Interactions and moderation objects keep reading __dict__ and Gemini responses keep _hidden_params out of .get * fix(router): renumber Responses stream events only after a fallback takes over A stream that never falls back now keeps its provider's own sequence numbers, repeats included, so a starting_after resume still matches them. After a fallback the wrapper continues above the highest number the client saw * fix(guardrails): keep Responses stream numbers unique when LLM Shield flushes held text The restorer's flush copied the last delta and kept its sequence_number, so a native stream repeated a number and a bridged stream sent the flush unnumbered. A flush now takes the done event's slot and later events move up by one, and a stream the provider never closed numbers its flush after the last event. Also pins the created_at in the streaming iterator test fixture instead of reading the clock. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 09:01:58 -08:00
devin-ai-integration[bot]	187f936cb3	fix(proxy)!: support reload with multiple workers (#45909) ... * fix(proxy): support reload with multiple workers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): stop the reload worker pool on a broken edit instead of respawning it Workers now load the app through a factory that exits with uvicorn's STARTUP_FAILURE code on any import error, so the pool stops and the reloader waits for the next save, the same as single-worker reload. Multi-worker reload needs uvicorn>=0.50.0 for that, so older versions keep the single-worker reload with a warning. * test(integration): stop the reload test's proxy through the harness shutdown helper Main's TQ010 rule rejects a raw .terminate() under tests/integration, so the multi-worker reload test now calls OwnedProxy.begin_shutdown(), which sends the same SIGTERM * test(proxy): cover the reload worker pool end to end and edit the config the proxy loads * docs(agents): scope the two-worker reload QA rule to uvicorn proxies that use Redis * fix(proxy): keep the original traceback when a reloaded worker fails to import the app * docs(agents): point both router and coordination Redis at the QA Redis and cover single-worker trees * fix(proxy): stop the reload worker pool when the reloader exits and give reloads a longer worker healthcheck Closing the terminal or pressing Ctrl-\ killed the reloader and left the pool parent holding the port while it respawned workers that could not start. The pool now stops once its parent reloader is gone. The reload pool also defaults the worker healthcheck to 30 seconds unless --timeout_worker_healthcheck is set, since uvicorn's 5 second default killed workers that were still booting on a reload. * fix(proxy): watch the reloader through its process sentinel so the pool also stops on Windows * fix(proxy): carry the graceful shutdown timeout into the reload worker pool * fix(proxy): stop the reload worker pool on Ctrl-\ instead of waiting out a dying worker's healthcheck * test(proxy): skip the startup-failure reload tests on uvicorn without a startup failure code * docs(agents): keep the QA proxy rule to two workers, Redis, and reload * fix(proxy): keep Windows on single-worker reload and pass unlisted uvicorn args to the reload pool * fix(proxy): stop reload workers when their pool parent dies * test(proxy): move the reload pool process tests to the integration suite The unit test guide keeps tests/unit in-process, so the subprocess pool cases now live in tests/integration/configuration and use the harness port allocator and process helpers * docs(agents): keep main's proof rules and inline the multi-worker QA command Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): exit reload workers as startup failures when the app factory raises Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): check the reload pool serves the overridden worker app Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-01 00:57:32 +00:00
yucheng-berri	0f4d5c6ad0	feat(ovalix)!: scan files, tool calls and tool results (#49309) ... * feat(ovalix): file and tool scanning, app discovery from the key alias Mirror of BerriAI/litellm#34132 at 5ffc268585, rebuilt on main. * fix(ovalix): keep inspecting requests whose key has no user The tracker allows a checkpoint call with an empty actor without inspecting it, so a key with no user (or a /v1/messages or /v1/responses request that carries its own metadata object) got no Ovalix enforcement. Fall back to "unknown" as the guardrail did before, so every request is inspected. * refactor(ovalix): check texts without mutation and test discovery through tracker requests - _check_texts builds its result from per-text verdicts checked newest first by _check_text - discovery and routing cache unit tests answer the tracker routes instead of replacing private methods - integration test adds streaming cells and a tracker outage during a concurrent burst * fix(ovalix): resolve tool result names in one pass Each tool result rescanned every earlier message for its call id, so a request with thousands of tool results spent seconds in the event loop. Declarations are now indexed once per request and each result takes the nearest earlier one by bisection * test(ovalix): annotate new test bindings and build the burst plan with product * fix(ovalix): drop app discovery from the key alias A config with no application_id no longer boots in discovery mode, where the [Name] prefix of the key alias picked the Ovalix application and so its policy. Anyone who could rename a key could move it to a weaker app, or to an unknown name the tracker created with its default policy. The guardrail now needs application_id at startup again, as on main, and every checkpoint call goes by the configured application and checkpoint ids. Removes enable_routing_cache and fail_if_no_application, the app_name_regex and resolve_application tracker calls, and the per-alias routing cache. Restores main's startup check that a requested pre_call or post_call hook has its checkpoint id. --------- Co-authored-by: Shalom Jamil <shalom@ovalix.io>	2026-10-31 17:01:12 -07:00
ishaan-j-berri	6105aaa63e	Configure full Bedrock guardrail assessments (#49217) (#49470)	2026-10-31 23:37:45 +00:00
ishaan-j-berri	8d26b8bba3	Return configured guardrail native findings (#49217) (#49460)	2026-10-31 23:37:45 +00:00
moe-berri	87e32e7c6d	fix(agents): show non-admin callers the proxy URL in agent cards (#49483) ... * test(agents): cover backend card URL leak to non-admin callers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): assert backend URLs before card parity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): scope unit card regressions to one agent Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): isolate card redaction from access policy loading Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): remove unused proxy fixture binding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): normalize unit test imports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): pass endpoint base URL to redaction helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): cover proxy card deep-copy behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(agents): rewrite non-admin agent card URLs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(agents): simplify proxy card URL helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(agents): drop card mapping adapter import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(agents): use typed card mapping helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): cover alternate card interfaces and real access checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(agents): drop alternate backend interfaces from non-admin cards Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): probe the real health check in card URL test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-31 16:17:11 -07:00
moe-berri	bba63c1934	fix(proxy): serve /a2a/{agent_id}/message/stream routes (#49471) ... * test(a2a): cover message stream route aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): serve /a2a/{agent_id}/message/stream routes Co-authored-by: Rustaman Delvianova <96533738+Rustaman1280@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): document A2A stream route schemas * fix(proxy): use immutable A2A schema mappings * fix(proxy): map v1 a2a stream route to call types and keep stream alias unbilled * test(a2a): use real guardrail registry and typed locals in stream route tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Rustaman Delvianova <96533738+Rustaman1280@users.noreply.github.com>	2026-10-31 16:11:51 -07:00
moe-berri	a7425101bb	fix(a2a): log authenticated pre-dispatch agent rejections (#49492) ... * test(a2a): cover authenticated pre-dispatch denial logging Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(a2a): log authenticated pre-dispatch agent rejections Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(a2a): exercise real permission checks for rejected calls Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-31 15:53:01 -07:00
shrey	ead8aa82ef	fix(passthrough): fit the Deepgram listen unpriced refusal in a WebSocket close frame (#49502)	2026-10-31 15:49:59 -07:00
moe-berri	3bd75f1258	fix(proxy): treat empty agent and skill search embedding model as not configured (#49485) ... * test(agents): cover empty search embedding model settings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(agents): treat an empty search embedding model as not configured Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(tests): sort search integration imports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(tests): format search embedding tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(agents): annotate search test locals and isolate upstream call checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-31 15:32:26 -07:00
moe-berri	580b9266a1	feat(chat): resume replies after browser disconnects (#49461) ... * feat(chat): resume background MCP runs after browser disconnects * feat(chat): background ordinary text conversations * feat(chat): configure background replies from Settings * fix(chat): address background response review findings * test(sdk): finish the HTTP smoke stream explicitly * fix(chat): preserve background response timing metrics * test(router): terminate the scripted completion stream	2026-10-31 14:43:24 -07:00
moe-berri	a5f980600c	fix(a2a): answer proxy refusals on /a2a POST routes as JSON-RPC errors (#49476) ... * test(a2a): expect JSON-RPC errors for proxy refusals on /a2a POST routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(a2a): answer proxy refusals on /a2a POST routes as JSON-RPC errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(a2a): build JSON-RPC refusal data from typed fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(a2a): stamp the OTel server span on JSON-RPC refusals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(a2a): cover JSON-RPC refusal route in unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-31 12:54:10 -07:00
shrey	7143aeeb86	feat(passthrough): add Deepgram streaming text-to-speech WebSocket route (#49282)	2026-10-31 12:46:40 -07:00
ishaan-j-berri	941f52f26f	fix(proxy): preserve discovery after budget exhaustion (#48809) (#49463) ... * fix(proxy): preserve discovery after budget exhaustion (#48809) * refactor(proxy): decide budget exemption once (#48809) * refactor(proxy): centralize request budget policy (#48809) * fix(proxy): keep custom pass-through budget checks (#48809)	2026-10-31 19:00:51 +00:00
Kerry Lu	948445c828	fix(watsonx): read pass-through credentials from use_in_pass_through deployments (#49462) ... * fix(watsonx): read passthrough credentials from use_in_pass_through deployments /watsonx/{endpoint} always called the provider config with api_key=None and api_base=None, so it only ever worked from WATSONX_API_KEY / WATSONX_API_BASE. It now asks PassthroughEndpointRouter for the proxy-wide watsonx deployment flagged use_in_pass_through, like the other passthrough routes, and keeps the env fallback when none is flagged. Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com> * fix(watsonx): keep env key precedence when no deployment is flagged Resolve only use_in_pass_through deployments in the route. get_proxy_wide_credentials also falls back to WATSONX_API_KEY, which put it ahead of WX_API_KEY in generate_iam_token --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>	2026-10-31 11:53:25 -07:00
joshua-berri	6e27da8af9	fix(router): inspect image content without translating routed requests (#49371) ... * fix(router): inspect image content without translating routed requests * fix(router): preserve screenshot image routing * test: make monthly budget reset coverage deterministic --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-31 10:44:24 -07:00
ryan-crabbe-berri	8ff495be96	fix(proxy): price OpenAI pass-through /v1/decisions calls like native Decisions (#49392) ... The OpenAI pass-through dispatcher kept its own list of endpoints it could price, separate from the handler, and /v1/decisions was on neither. Calls reached OpenAI but logged $0 and no tokens. The handler now owns the list the dispatcher reads, and prices a POST to /v1/decisions through the same Decisions response type and cost path native /v1/decisions uses, so the _decisions cost-map rates apply. Reads and sub-routes under /v1/decisions stay unbilled. Fixes #48543	2026-10-31 07:34:29 -07:00
joshua-berri	be4f3d36ce	feat(accounting): capture request usage through the durable meter (#49133) ... * feat(accounting): capture request usage through the durable meter * fix(accounting): preserve native Google usage compatibility * fix(accounting): preserve prepared usage observation on failure * fix(accounting): exclude sync acknowledgement policy rejections from capture * test(accounting): verify capture races and failure isolation * fix(accounting): preserve request support with sync acknowledgements * fix(accounting): normalize usage metadata before queue admission * fix(accounting): preserve prior attempt links on successful fallback * fix(accounting): preserve attempt evidence through retries and failures * fix(accounting): depend on prior receipts without resubmitting usage * fix(accounting): preserve pricing inputs and resolved capture attribution * fix(accounting): persist batch usage within capture bounds * test(accounting): cover batch dependency failure and backpressure * fix(accounting): isolate batch persistence and preserve trusted attribution * fix(accounting): share byte admission across batch producers and writes * fix(accounting): reserve batch snapshot reads before admission * fix(accounting): compact repeated batch observations without losing requests --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-31 07:32:25 -07:00
Mateo Wang	a713ed6bab	fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them (#47692) ... * test(e2e): add gemini tool call id length repro for LIT-8912 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them * fix(gemini): pair a tool result with its call when the client re-adds the thought signature to the call id Strands, which the AgentCore harness runs, puts provider_specific_fields.thought_signature back on the assistant tool call id as <id>__thought__<signature> while its tool result keeps the bare id, so the Gemini conversion found no matching call and answered 500. Ids now match with the signature suffix stripped. Also adds thought_signature_in_tool_call_id to the litellm_params inventory test and gives the pass-through tests' mocked logging objects the litellm_params a real Logging object always has. * fix(gemini): keep the thought signature on streamed Messages and Responses tool calls and accept a string setting * fix(gemini): read the thought signature back from Responses input and coerce the per-request setting A Responses function_call input item now keeps its provider_specific_fields, so a client that sends the item back gets its real thought signature replayed to Gemini. A per-request thought_signature_in_tool_call_id of "false" or 0 now keeps ids short, and a value that is not a boolean answers 400, or is dropped under drop_params, instead of a 500. The e2e test now uses the sdk fixture and @meta subjects * fix(caching): key the response cache on thought_signature_in_tool_call_id * fix(responses): send thought signature fields only with short ids and drop malformed routing kwargs whole * fix(main): validate routing kwargs after the locals snapshot so fallbacks and model_list calls keep working The e2e test now uses gemini-3.8-flash with enough max_tokens to finish thinking before the tool call. * fix(gemini): keep non-chat routes tolerant of a malformed thought_signature_in_tool_call_id, match tool results by exact id first, and keep the deployment's value when drop_params drops a malformed override * fix(router): validate strict litellm params through a helper so star imports cannot re-bind a public Final * perf(router): skip strict validation when a request overrides no deployment param * fix(responses): keep a dict tool call's thought signature on the streamed function_call done item * refactor(main): return invalid supplemental params as a value and test the streaming tool call with a real response --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-31 03:20:42 -07:00
paveln83	febd340120	fix(guardrails)!: keep the upstream error when a /v1/messages stream ends in one (#49088) ... * fix(guardrails): keep the upstream error when a /v1/messages stream ends in one Raw-SSE guardrail stream hooks assembled the whole stream, and the Anthropic chunk parser raises on an error event, so a stream the upstream ended with event: error failed assembly. tool_permission then blocked it with a 400 that hid the upstream error, bedrock and model_armor emitted a could-not-assemble block, and presidio skipped masking. Assembly now leaves the error events out, so the delivered content is checked as usual and a clean stream passes through with its error frame. A rewritten stream ends with the upstream error instead of a message_delta and message_stop the upstream never sent. tool_permission also passes a stream of only error frames, which carries no tool call to check. * fix(guardrails): do not close a block the failed upstream left open when re-emitting a rewritten stream * fix(guardrails): check only the finished blocks of a stream cut off by an upstream error tool_permission and Presidio now assemble an error-ended stream through the error with an opt-in keyword, leaving out a tool call or thinking block the upstream cut off. Logging, OpenTelemetry, Bedrock, and Model Armor keep the old assembly. A rewrite drops the text block stop only when the upstream left its text block open. * fix(guardrails): keep a cut-off tool call from the client and stop each rewritten block before the next starts A failed stream holding a cut-off tool_use is now re-emitted through the rewrite, which leaves the call out and keeps the upstream error, so a denied call never reaches the client. The rewrite leaves its text block open only when it is the last block, and keeps a message_delta the upstream sent before failing. The new tool_permission tests no longer patch should_run_guardrail. * fix(guardrails): pass a cut-off stream with nothing denied through as sent, minus the cut-off tool call Re-emitting such a stream through the rewrite sent an empty text block and stopped it, and Claude Code does not retry a failed stream that shows a finished block, so it showed a mid-response error instead of retrying. Dropping only the cut-off call's events keeps the stream as the upstream sent it. * fix(guardrails): strip a cut-off tool call only from a stream the upstream failed Assembly leaves unfinished blocks out of the check only when the stream carries an upstream error event, so a stream with an unfinished but allowed tool call and no error frame now reaches the client as sent instead of losing a call the check already read * test(guardrails): type the parameters of the new stream error tests * fix(guardrails): skip SSE data lines that are not JSON objects when reading a stream A data line holding a bare JSON value such as 5 made the stream helpers call .get on an int, so tool_permission and Presidio answered 500 where main answered 400 or passed the stream through * fix(anthropic): skip SSE data lines that are not JSON objects when assembling a stream * test(guardrails): annotate the upstream error stream tests * fix(anthropic): read a stream's non-object lines and named error frames the same way everywhere Run the skip for a data line that is not a JSON object through extract_sse_data and the assembler loop, so the usage fallback keeps a stream's tokens and the chat handler keeps its contract. Presidio now holds a leading non-object frame until the first object frame, so the rest of the stream is still masked or blocked. An event: error frame now counts by its name, so a rewrite keeps it. * fix(guardrails): refuse an unreadable stream in Presidio and cut every kind of tool call A non-object JSON line with no Anthropic frame after it now gets a 400 from Presidio before any data byte, since passing it would send text main refused. A frame named error stays a normal event when its data is another Anthropic event. server_tool_use and mcp_tool_use cut off by an upstream error are dropped like tool_use, and a leading frame with empty data is relayed at once. * fix(guardrails): refuse what Presidio cannot read past a non-object line and keep non-string error types out of the frozenset Presidio now refuses a stream assembly cannot read when it holds a non-object data line, instead of relaying it unmasked. Non-string event types no longer crash a rewrite, and a CRLF stream that opens with a typed error is no longer read as one error, so its tool calls are still checked. * fix(guardrails): refuse a first frame whose type is a list or dict in Presidio Presidio passed PII at 200 where main answered 500 when the first read was a JSON object with a list or dict type, or a JSON array with a character cut at the end of the read. has_unreadable_json_event now reads both as data no reader of these streams can take, so the first-frame check refuses them. * test(guardrails): cover Presidio's refusal when a word or null line comes before a Gemini stream The first read then holds the unreadable line behind one Presidio can pass, so the check has to look at every frame in it. * fix(guardrails): treat a stream as error-only only when every data frame is an error A tool call split over data: lines beside one error frame read as error-only, so tool_permission passed it, and an SDK joins those lines and reads the call. The check now joins a frame's data lines and refuses anything but an error. * fix(guardrails): refuse a stream past an upstream error when a client could read a frame differently Run 8 of the live risk pass showed each stream reader imitating a client and missing a shape: a whitespace-led data line, an error key with no type, a split tool call after a plain message_start. An error frame now has to be plain, and assembly past an error returns None for any frame a client could read differently, so those streams keep main's refusal. * fix(guardrails): on a failed upstream, deliver only what tool_permission read A failed stream now reaches the client through the rewrite that already serves a denial, followed by the upstream error, even when nothing is denied. Raw frames could carry a tool call the check never read in shapes such as a reused block index, mixed line ends, a tool in message_start, interleaved deltas, or a form feed, and each refusal rule only moved the search to the next shape. The error frame goes out as its event name and first data line. An error is found at CR, LF and CRLF line ends, and the rewrite adds no finished block when it read no text, so Claude Code still retries the stream. * fix(guardrails): read an error frame's name as clients do and send the error alone when a rewrite cannot be built A data-less event: error frame with junk after the name is an unknown event to both SDKs, so a healthy stream stays raw. A finished tool call whose input is not JSON cannot be rebuilt, so a failed stream sends its upstream error alone, and a healthy stream still raises as before. * fix(guardrails): read an error frame as a client dispatches it and keep its JSON whole Only the last event line of a frame a blank line ended names an error, so a healthy stream with a later event line or an unterminated error is no failed stream. Any rebuild failure on a failed stream now sends the upstream error alone with a warning, a param rule denies an input nested too deep, and a raw line break inside a JSON error stays escaped so every reader gets it whole. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 02:56:20 -07:00
paveln83	27fbcd9895	fix(anthropic): map pause_turn and return refusal explanations on chat and responses (#49192) ... * fix(anthropic): map pause_turn and return refusal explanations on chat and responses pause_turn now maps to finish_reason length (Python and Rust) and keeps the raw stop_reason in provider_specific_fields. Responses reports a paused turn as incomplete with max_output_tokens and shows the unrun server search as an in_progress web_search_call instead of a client function_call. A refusal's stop_details.explanation now fills message.refusal and the final stream delta.refusal on chat, and becomes a refusal content part on Responses, streamed or not. * fix(anthropic): let a chat client resend a paused turn so Claude continues it * fix(anthropic): replay a paused chat turn without its thinking blocks, and drop its unrun search before a new message Chat cannot say where Claude wrote each thinking block in a paused web search turn, and Anthropic answers 400 when one moves but accepts the turn without them. A new message after a paused turn also 400s on a server_tool_use with no result, so that unrun search is dropped. * fix(responses): send the streamed refusal part's type and text in content_part.added * fix(anthropic): keep pause_turn and Claude refusal explanations out of the Messages adapter's text A paused Claude turn now ends on stop_reason pause_turn instead of max_tokens, streamed or not. A Claude refusal keeps its explanation in stop_details with Claude's category and no longer repeats it as a text block, so a guardrail mask on a streamed refusal is delivered. A paused turn sent back with list content drops its thinking blocks like string content does * fix(anthropic): replay a paused turn with one unrun search and no thinking on chat and Responses * fix(responses): report a paused Claude turn as completed so Codex and the Agents SDK keep going * fix(responses): mark a paused turn's message item completed * fix(responses): mark every item of a paused turn completed * fix(responses): keep a refusal's explanation when a refused turn is sent back A refusal part in a Responses input message now goes to the model as text, so the assistant turn is no longer dropped from the history. The integration test now expects a paused Claude turn on Responses to report completed, and the paused-turn test helper is typed * fix(responses): scan refusal parts in Responses guardrails --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 02:53:59 -07:00
quietjules	cee72ffdd6	fix(proxy): count native Responses WebSocket turn tokens toward key and tag TPM limits (#49257) ... * fix(proxy): count Responses WebSocket turn tokens toward key and tag TPM limits A native Responses WebSocket turn logs its raw events instead of a usage-carrying response object, so the v3 rate limiter read no usage and never charged the key or tag token window. Read the turn's usage from its events the way the cost calculator does, and charge the TPM-limited tags the socket was opened under. * fix(proxy): refuse socket turns past TPM and count managed, failed, and realtime usage A later response.create turn on an open Responses WebSocket now gets a read-only check of every TPM window it counts toward (key, team, user, org, end user, team member, model-scoped limits and the socket's tags). A turn with no tokens left gets the 429 rate_limit_exceeded frame and the socket stays open. The check never reserves or increments a counter and leaves RPM, budget, PTU and project windows to their own paths. It runs on native and managed sockets. Managed turns carry the socket's call id in their metadata, so they charge the socket's TPM-limited tags. A failed native turn charges the partial usage its failure row already carries. A socket refused at open by a rate limit gets the same refusal frame and closes 1008, logged once without a traceback. /v1/realtime sessions now count toward TPM through the realtime usage combiner, and the v1 limiter reads socket usage through the same helper * fix(proxy): count failed socket turns, keep socket usage off team PTU, check agent windows per turn A native socket turn that ends in response.failed now counts the usage that event reports toward spend and TPM, as the HTTP stream path does. Native socket and realtime usage no longer charge the team PTU counter, matching the merge base. The per-turn TPM check now also reads the agent and agent session windows the socket opened with. * fix(proxy): start missing legacy user, team, and end-user TPM buckets at zero * fix(proxy): stop charging a realtime session's whole usage into its close minute Realtime and live sessions log once at close, so counting them put a long session's tokens into one TPM window and refused calls that stayed under the limit. They count no TPM again, as before this branch. Also covers the legacy v1 count-once fix with an HTTP regression case. * test(proxy): check each per-turn TPM window on its own * fix(proxy): send a fail-closed per-turn socket refusal with its 503 status --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 02:53:27 -07:00
mika-lt	e5aa3d32cd	fix(proxy): forward a valid metadata.user_id to Anthropic Messages on pass-through routes (#49333) ... * fix(proxy): forward the client's own metadata on Anthropic and custom pass-through routes * fix(proxy): forward only a valid metadata.user_id to Anthropic Messages on pass-through routes The whole-object forward sent keys Anthropic refuses and reached other providers, so a request that used to pass could start failing with a 400. Read the client's user_id after hooks, drop it unless it is a string with no email or phone number, and send only that key when the upstream URL path ends in /v1/messages. Share the validator with the translated route, whose email check now matches any embedded address as Anthropic's does. * fix(proxy): forward metadata.user_id only to Anthropic upstreams no guardrail scans Limit the pass-through user_id to Messages create on api.anthropic.com or the configured Anthropic base, mirror the checks Anthropic answers 400 for, and keep the field off any request a pre-call guardrail inspects. The translated route goes back to its own user_id filter. * fix(proxy): judge user_id forwarding on the whole request and cap it for configured Anthropic bases * refactor(proxy): pass the user_id check a plain mapping instead of an optional body * fix(proxy): drop an unprintable metadata.user_id and warn when a refused id is dropped Anthropic's whitespace is not Python's, so a control character before the at sign hid an email from the filter and turned a 200 into a 400. Fireworks and Moonshot also fail on newline ids. Requiring a printable id covers both, and a refused id now logs a warning that never carries the value. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 00:16:42 -07:00
simonp9	a392a39627	fix(vertex_ai): bill Claude web search per request on deployments priced in model_info (#49160) ... * fix(vertex_ai): bill Claude web search per request on deployments priced in model_info A vertex_ai deployment priced in its model_info is registered under its hashed id, so the web search cost check on the model key never saw "claude" and billed Claude searches with the Gemini grounding calculator: one search per call, or Gemini's $0.035 default with no search price. The check now also reads the model the response names. * fix(vertex_ai): bill token-priced Claude deployments at the served model's search price A Vertex Claude deployment priced in model_info without a search price now bills each web search at the served model's cost map price, read at cost time and never written into the deployment entry. An explicit search price, zero included, still wins. * fix(vertex_ai): price web search on the Vertex Claude Sonnet 4.5 and Haiku 4.5 rows Google bills Claude web search at $10 per 1,000 searches on these models, so token-priced deployments of them no longer bill searches at 0 * fix(cost): bill served-model search price on token-only Anthropic and Azure AI Claude deployments Token-only Claude deployments on anthropic and azure_ai billed web search at 0 because the deployment entry has no search price. They now fall back to the served model's cost map search price, as vertex_ai already does. Vertex routing reads the served model first, so a Gemini deployment whose id contains claude keeps per-prompt grounding * fix(proxy): price streamed usage frames under the deployment's model so usage.cost matches the spend row * test(proxy): give the Vertex streamRawPredict logging doubles the request's model, as a real pass-through Logging has --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 23:14:51 -07:00
shrey	afec145edc	feat(passthrough): let a WebSocket relay bill a session from its client text (#49281)	2026-10-31 05:52:25 +00:00
quietjules	8801a17f0d	fix(proxy)!: bind admin-owned team keys to their team's MCP servers (#49161) ... * test(proxy): pin admin-owned team keys to their team's MCP servers and toolsets A virtual key on a real team owned by a proxy_admin or proxy_admin_viewer, with no MCP list of its own, currently reaches every MCP server in the registry and any toolset, ignoring its team. These tests expect it to resolve only what its team grants, like the same key owned by an internal user, while the admin dashboard session keeps the whole registry. * fix(proxy)!: bind admin-owned team keys to their team's MCP servers A virtual key on a real team now resolves MCP servers and toolsets through its team even when its owner is a proxy_admin or proxy_admin_viewer, the way team model access already binds it. The unscoped admin view stays for the master key, admin keys with no team, the admin dashboard session, and admitted user subjects. The toolset grant checks on /toolset/{name}/mcp and on the Responses litellm_proxy MCP tool use the same rule. BREAKING CHANGE: a team key owned by a proxy admin or admin viewer no longer reaches MCP servers or toolsets its team does not grant. Grant them to the team or use a key with no team. * fix(proxy): answer 403 for a toolset the key cannot use on LLM routes A litellm_proxy MCP tool on /v1/responses, /v1/chat/completions or /v1/messages that named a toolset the key's team was never granted was skipped, which left the server filter empty and handed the model every tool the key could reach. The denied name now stays in the filter, so the request gets the same 403 an unknown server name gets. Admin credentials on a real team stay bound to it whatever carries them, a lite login session included, the way the team already bounds their model access. The tests pin that and drop an unused patch. * fix(proxy): keep a denied toolset out of the scoped key beside a granted one * test(responses): cover toolset name fallbacks and align MCP admin docstrings * test(mcp): cover no-team admins' toolset access on LLM routes and drop a dead admin-check clause * fix(responses): keep a whole-gateway MCP entry serving the key's tools beside a denied toolset * test(responses): check the tools a whole-gateway request lists beside a denied toolset * test(responses): keep the toolset lookup real in the whole-gateway listing test * test(responses): run the toolset gateway tests on a real MCP server manager * fix(responses): keep denied toolset names out of the MCP server filter A denied toolset named like a server the key holds was matched to that server by the loose server-name lookup. Denied names now never reach the filter, and a request naming only denied toolsets answers 403 directly, each name once * fix(responses): let a held MCP access group outrank a same-named ungranted toolset and give denied toolsets the generic 403 * test(responses): pin access group and granted toolset precedence for LLM-route MCP names --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 22:07:32 -07:00
joshua-berri	02a6a96dec	feat(accounting): add bounded transactional usage persistence (#49190) ... * feat(accounting): add bounded transactional usage persistence * fix(accounting): preserve usage flush and shutdown on Python 3.10 * test(accounting): cover batch rejection and partial shutdown recovery * fix(accounting): batch durable usage on a five-second interval * fix(accounting): drain queued full batches without timer delays * fix(accounting): isolate invalid producer submissions within usage batches * fix(accounting): await commit prerequisites within bounded usage admission --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-30 21:08:43 -07:00