mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-03 07:23:33 +00:00
544 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
zoveki
|
e63fc85b7b
|
fix(anthropic): keep Mistral thinking text on streaming /v1/messages (#49059)
* fix(anthropic): keep Mistral thinking text on streaming /v1/messages * test(anthropic): type the signed thinking stream test helpers --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c213234783
|
feat(vertex_ai)!: use HTTP/2 httpx client for search_api vector store (#40631)
* feat(vertex_ai): use HTTP/2 httpx client for search_api vector store Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(httpx): exercise configured request path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(httpx): verify HTTP/2 request negotiation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(httpx): preserve HTTP/2 for sync proxy mounts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * build(deps): add h2 to proxy extra so vertex search HTTP/2 is on by default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): isolate ssl context for http2 clients Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http): add an HTTP/2 opt-out and harden the Vertex AI Search HTTP/2 client litellm.http2 is now unset by default, so http2: false or LITELLM_HTTP2=False turns the provider's HTTP/2 default off. A caller's own SSLContext keeps the search on HTTP/1.1, the HTTP/2 async client honors DISABLE_AIOHTTP_TRUST_ENV, forced IPv4 transports get the CA bundle and client cert, and vector store search retries once when the server drops the connection with a GOAWAY. * fix(http): retry vector store search only on HTTP/2 GOAWAY and keep aiohttp pool and proxy behavior Scope the one retry to a connection Google closes with an HTTP/2 GOAWAY and log it. When HTTP/2 replaces the aiohttp transport, keep aiohttp's idle keepalive and connection cap, and only then skip env proxies under DISABLE_AIOHTTP_TRUST_ENV, so the global LITELLM_HTTP2 toggle keeps HTTPS_PROXY * test(http): cover the vector store search GOAWAY retry in unit tests and drop duplicate SSL cache comments * fix(http): open another HTTP/2 connection once 100 streams are in use httpcore keeps every request to an origin on one HTTP/2 connection and caps it at 100 concurrent streams, so request 101 and later waited with no timeout. The handlers now spread requests over extra connections, opening one only when every open connection has 100 streams in flight. The force_ipv4 wire tests now cover ssl_verify=False and the client certificate. * fix(vertex_ai): reuse one Vertex search OAuth token across searches Every vector store search built a new search config, so the token cache it carries was thrown away and each search fetched a new Google OAuth token with a blocking call on the event loop. Under a burst of searches that blocking fetch held back every HTTP/2 response. The provider config is now shared, so searches with the same credentials reuse one token until it expires. * fix(http): keep the HTTP/2 connection spread within the connection cap * fix(http): keep sync vertex search on HTTP/1.1, since httpcore's sync HTTP/2 is not thread-safe httpcore 1.0.9 hands two threads the same HTTP/2 stream id, so threaded sync searches failed with protocol errors. Only the async client, which the proxy uses, now defaults to HTTP/2, and the sync connection spread is removed. * fix(http): split the connection cap across HTTP/2 lanes Each lane had its own pool with the full connection cap, so an origin answering HTTP/1.1, or several origins on one client, could exceed max_connections once more than 100 requests were in flight. Lanes now open only while lanes times 100 streams stay under the cap, and each lane pool gets at most 100 of it. * test(http2): scope wire-test peer logs to each fixture instance * fix(http2): keep TE, force_ipv4 trust, evicted-client and opt-out behavior on the HTTP/2 path - drop TE values other than trailers on https requests through the HTTP/2 spread, since h2 refuses them - force_ipv4 transports keep httpx's default trust store (SSL_CERT_DIR) unless SSL is configured - the evicted client closer asks the HTTP/2 spread whether a request is in flight - LITELLM_HTTP2 set to 0, no or off, and a quoted "false" http2 setting, opt out of HTTP/2 - the wire test's held-search barrier waits on an event instead of polling * fix(http2): count proxy-mounted transports when checking an evicted client for in-flight requests * test(http2): move the HTTP/2 eviction cases that open a loopback server into the integration wire tests * fix(http2): read an unquoted http2: 0 as off and free the stream slot of a response dropped without closing * fix(http2): free the stream slot of a partly read response dropped without closing * test(http2): check that the evicted client and its stream close after a partly read response is dropped --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Nate Armstrong
|
ead221995e
|
refactor(router): move router callers onto the entry points and invert the proxy imports (#48847)
Code outside the router now reaches it only through the entry points (registry, settings, health, select, apply, observe, register_plugin, retire) or the endpoint methods; the outside-member allowlist shrinks from 139 to 45. litellm/router.py no longer imports litellm.proxy: it asks the installed RouterHost (litellm/router_host.py) for vertex pass-through credentials, member auto-router authorization and auto-router prompt compression, and the proxy installs its host on import. Call sites where a model name can be missing keep the old answer instead of reaching an entry point with None. |
||
|
Rick
|
c98641fd5a
|
feat(dashscope)!: derive protocol bases from one domain (#47812)
* feat(dashscope): derive protocol bases from one domain Accept the DashScope domain root, the OpenAI-compatible base, the Anthropic Messages base, or a native /api/v1 base and derive every surface from it Adds a native Anthropic Messages config for /apps/anthropic so a single deployment serves chat, messages, embeddings, rerank, image generation, and video against the same credential Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com> * fix(dashscope): keep Anthropic credentials off DashScope messages Require a DashScope key or a caller auth header before the Messages path runs, and opt out of workload identity, so the base class cannot fall back to ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN Also format the image generation base selection * style(dashscope): keep the messages config within the format gate * test(dashscope): cover caller Authorization on messages * fix(dashscope): keep the chat bridge for protocol-named bases and the image default for unconfigured calls * fix(dashscope): keep configured image and rerank bases, normalize blank keys, derive brand chat bases - image calls fall back to the provider's own chat base env (DASHSCOPE_API_BASE, QWENCLOUD_API_BASE, QWEN_AI_PLATFORM_API_BASE) before the image default - an explicit /compatible-api/v1/reranks URL wins over DASHSCOPE_API_BASE_RERANK again - a blank DashScope key counts as unset on /v1/messages, so it never falls back to ANTHROPIC_API_KEY - QwenCloud and Qianwen AI Platform chat and embeddings derive their compatible-mode base from a domain root - tests annotate new variables with Final and drop the config-class assertion * fix(dashscope): keep an explicit compatible-api rerank base as given * fix(dashscope): derive chat from compatible-api bases, keep the caller's rerank base, and let explicit bases beat per-endpoint env vars * fix(dashscope): keep a full DashScope image endpoint URL as given and type the brand tests * fix(dashscope): let the image env var beat any shared DashScope base * fix(dashscope): keep a full DASHSCOPE_API_BASE image URL ahead of the image env var * fix(dashscope): keep a native DashScope endpoint URL as given for chat and the api base header --------- Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
06cdee91da
|
fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails (#49035)
* feat(guardrails): scan and rewrite streamed /v1/completions in post_call pipelines Gives OpenAITextCompletionHandler a process_output_streaming_response that assembles the buffered TextCompletionResponse chunks per choice index at end of stream, runs the guardrail over the assembled text, and writes text rewrites back into the first text-carrying chunk of each choice. Tests and live proof still pending. * fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails Assemble each choice's streamed text once the stream ends, run the guardrail on it, write a mask back into the buffered chunks, and end the stream with the guardrail's error on a block. User key metadata now goes through the shared metadata bucket so applied guardrails reach spend logs. * test(guardrails): cover streamed /v1/completions under post_call guardrails * fix(guardrails): keep Gray Swan key metadata, scan a cut-short text stream, and pass non-chunk frames through * fix(guardrails): replace blocked text completion streams like chat, keep Gray Swan's vendor payload, scan text streams in the content filter * fix(guardrails): release an unfinished choice's held text and omit usage the upstream never sent The content filter's streaming hook held back the end of a choice until its finish_reason, which litellm sends only for the first choice of an n>1 stream, so choice 1 lost its last characters on text and chat streams. A closing chunk now releases every unfinished choice when the stream ends. A blocked text stream's closing chunk reported 0/0 usage when the upstream sent none; it now carries usage only when the upstream reported it. * fix(guardrails): send no usage on the content filter's end-of-stream chunk * fix(guardrails): keep the usage chunk last and log the masked text of streamed text completions * fix(guardrails): stream Responses API events through the content filter as they arrive * fix(guardrails): treat a null text completion choice index as choice zero when assembling streams --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
77064d6c70
|
feat(vector_stores): add OpenSearch k-NN search as a vector store provider (#47759)
* feat(vector_stores): add Amazon OpenSearch as a vector store provider * feat(ui): add Amazon OpenSearch to the vector store provider dropdown * feat(vector_stores): add OpenSearch k-NN search as a vector store provider * test(vector_stores): hold the OpenSearch provider tests alone for the first review The provider source comes back in the next commits on this PR * ci(tests): run the OpenSearch vector store unit tests * fix(vector_stores): sign on.aws and FIPS OpenSearch hosts and reject missing AWS credentials before embedding * fix(vector_stores): regenerate provider endpoint support for opensearch * fix(vector_stores): keep opensearch retrieve, update, and delete off the index and refuse half-set AWS keys and per-account endpoints * fix(vector_stores): return only scalar opensearch attributes and keep credential source output out of 401s * test(vector_stores): inject a failing opensearch credential source instead of running a shell * fix(vector_stores): sign opensearch for the endpoint's region, refuse list, add opensearch_index_name, honor drop_params * fix(vector_stores): refuse a caller-chosen opensearch_index_name on search and rag query * fix(vector_stores): refuse opensearch_index_name only on OpenSearch stores, keep the old 400 text Other providers drop the key again as they did before OpenSearch support, and the four embedding keys keep their original error sentence. * fix(vector_stores): refuse a caller index on slash-form OpenSearch stores |
||
|
Mateo Wang
|
be6d082bdc
|
fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166)
* fix(anthropic): send /v1/messages PDF documents to OpenAI-shaped providers as file parts * refactor(anthropic): hold the chat completions document change for a later commit * fix(anthropic): send /v1/messages PDFs as file parts and keep file parts on azure_ai * fix(snowflake): turn file parts into document blocks for Cortex Claude * fix(prompt_security): sanitize chat file parts that carry file_data * fix: answer 400 or drop PDFs for models that cannot read them Models whose model_info or cost map entry sets supports_pdf_input to false, and Ollama, now get the PDF dropped under drop_params and a 400 from litellm without it. GigaChat uploads file parts, SAP keeps sending them as image_url data URLs, azure_ai passes dict content through, and annotations other than url_citation move to provider_specific_fields instead of failing the response. * fix(router): scope supports_pdf_input to its deployment and keep provider annotation keys Mark the azure_ai chat models in the cost map that cannot read PDFs, so drop_params drops the PDF and litellm answers its own 400 without it. Keep url_citation annotations whole, move only typed provider annotations to provider_specific_fields, and split them the same way when a stream is rebuilt. * fix(vertex_ai): send PDF file parts to Vertex partner models as image_url Vertex AI's OpenAI-compatible partner endpoint rejects the chat `file` part, so PDFs from /v1/messages reach those models as the image_url data URL they took before. Mark native DeepSeek chat models as unable to read PDFs, and treat a file part as a PDF by its format or filename when it has no data URL. * fix(cost-map): leave native DeepSeek PDF support unset /v1/messages reaches native DeepSeek through its own Anthropic endpoint, so the marks never applied there and would only change chat completions, which this PR does not otherwise touch. * fix(main): drop a user turn that held only a PDF under drop_params Dropping the PDF from a PDF-only user turn left an empty content list, which Foundry and OpenAI-shaped providers reject with a 400, so drop_params still failed for that request. The emptied user turn is now removed. * fix: read OCI PDFs, gate deployment base models, and stop empty drops OCI generic chat sends file parts as image data URLs again, so OCI Gemini keeps reading PDFs from /v1/messages and chat file parts. The PDF gate now reads model_info.base_model, drops PDFs under drop_params for azure_ai deployments that never declared PDF support (the old flattening behavior), and answers 400 when dropping the PDF leaves no user message. The cost map marks the Vertex partner text models that a live probe showed answering without reading the PDF. * test(integration): cover the PDF input gate, messages PDF file parts and provider annotations Wire cells for the supports_pdf_input check on chat, chat streaming, /v1/messages and the Responses bridge, its model_info, base_model, cost map and Ollama sources, drop_params at deployment and request level, azure_ai file parts, sibling deployment isolation, Anthropic SDK PDFs reaching azure as titled file parts, Gemini and Bedrock document controls, Prompt Security sanitizing a /v1/messages PDF, and OpenRouter file annotations live, cached and streamed * test(integration): cover Snowflake Claude and SAP in the messages PDF wire cells * fix(azure_ai): flatten non-PDF file parts to text as before and cover the GigaChat PDF upload * fix(main): PDF check reads the deployment's top-level base_model * fix(main): keep roles alternating after drop_params removes a PDF-only turn When ensure_alternating_roles is set, re-run role normalization on the messages left after the PDF check drops a turn, so two assistant turns never end up next to each other * test: type the PDF and annotation tests the PR added * test: type the GigaChat, SAP and Snowflake PDF tests |
||
|
Dan Lemon
|
cd58014315
|
feat(soniox): add realtime speech-to-text on /v1/realtime (#44822)
* feat(soniox): add realtime speech-to-text on /v1/realtime Maps OpenAI realtime transcription sessions onto the Soniox websocket API (stt-rt-v5): server VAD uses Soniox endpoint detection, commit sends a manual finalize, deployment litellm_params pass Soniox options through, translation sessions transcribe the translated tokens, idle streams get keepalives, and cost is tracked from processed audio duration. * fix(soniox): bill realtime usage from integer milliseconds * fix(soniox): bound session-start queue and cover realtime backend edge paths * fix(soniox): bill realtime stream time, start sessions on connect and surface client protocol errors Soniox charges for the whole open stream, so session close now bills the connection's open time. The backend announces the session on connect, which flushes audio sent before any session.update, and a single upstream pump makes recv safe to cancel. Invalid client frames now get an OpenAI error event instead of a silent disconnect * fix(soniox): skip deferred setup so client frames reach Soniox in order Drops the connect marker frame, which counted as an upstream frame and hid refusals, and could start a buffered stream at the default rate before the client's session.update. Soniox now opts out of deferred setup, the handler announces a synthetic session.created whenever no setup went upstream, a session.update that would change a started stream gets an error event, and stream time is billed only once a stream started * fix(soniox): judge a follow-up session.update only on the settings it sends A partial follow-up session.update was rebuilt with defaults for the settings it left out, so restating part of a 16 kHz or VAD-off stream was rejected as a change * fix(soniox): treat an omitted rate in a follow-up session.update as unchanged * chore(soniox): drop comments that restate the code * fix(soniox): return speaker labels, log refused sessions as failures, and probe realtime health Deltas carry speaker and language when diarization or language identification is on, and the completed event lists segments per speaker and language. A session Soniox refuses before any audio is processed is logged as a failure with no spend. Billing starts at the start request. Deferred-setup providers (Meta, Vertex Chirp) send the error event and still end the session. Soniox realtime models get a websocket /health probe and cost rows for the stt-rt-v5 aliases. * fix(soniox): bill failed streams, probe health with deployment options, and keep token accumulation linear Only request rejections (400, 401, 402, 403, 429) before any processed audio count as a refused session, so a 408 or 5xx still bills the open stream. The /health probe starts Soniox with the deployment's options, sends 100 ms of silence, and passes only on Soniox's finished frame. A turn's final tokens append in constant time instead of copying the tuple per token. * fix(soniox): log every Soniox error before any processed audio as a failed session * fix(soniox): keep untranslated speech, support input_audio_buffer.clear, ignore an empty language * fix(soniox): reject frames without a type, keep strict language hints, queue finalizes in a deque * fix(soniox): answer a commit with no speech with an empty completed turn * fix(soniox): let a server VAD turn answer a commit that is still in flight * fix(soniox): let a server VAD turn answer only the oldest pending commit --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tavren27
|
b25ea48c4b
|
fix(anthropic)!: drop beta-gated body fields under drop_params (#48793)
* fix(anthropic): strip beta-gated body fields when a leg drops the anthropic-beta When a provider leg drops an anthropic-beta value (the chat handler's allowlist filter, /v1/messages' filtered headers, Bedrock Converse's additionalModelRequestFields), also drop the body fields that beta gates (thinking.block_binding, thinking.display=updates, mid-conversation system clear_at, output_config, tool_addition/tool_removal blocks) so the request degrades on that leg instead of failing with a provider 400. Copy-on-write so a fallback leg still sends the full request, and each strip is logged at WARNING. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove a leg that drops a beta sends neither the beta nor clear_at Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): strip beta-gated fields against the final body anthropic_beta list Converse counts the body anthropic_beta list as active, Invoke strips after its auto-injected betas are known and injects thinking-binding-controls for thinking.block_binding, test helpers are fully typed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated system fields before an older model's system turn becomes a user turn Models without supports_mid_conversation_system send no mid-conversation beta, so tool_addition and tool_removal blocks are dropped (and logged) before the turn is converted instead of riding along as user content. The Invoke output_config regression test reads the real cost map instead of patching _bedrock_model_supports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): pin the integration proxy to the bundled beta allowlist and type the helper locals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): annotate every new test local with Final and type the fixtures and returns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated body fields only under drop_params Without drop_params the request now goes out as sent and the provider answers its own 400, like any other unsupported param. inline-tools-2026-09-15 also unlocks tool_addition and tool_removal blocks, so a leg that carries it keeps them. * fix(anthropic): strip beta-gated fields on Foundry chat and Bedrock chat Invoke under drop_params Foundry chat and Bedrock chat Invoke now drop beta-gated body fields the same way the other legs do when drop_params is on and the request does not send the gating beta. The mid-conversation system normalizer validates only on the conversion path, and the strip WARNING names the beta the caller can send to keep the field. * fix(anthropic): send the thinking betas for block_binding and display updates, strip Databricks chat thinking Every Claude leg now adds the thinking-binding-controls beta when thinking carries block_binding and the display-updates beta for display updates, so those fields go through instead of a 400. Databricks chat rejects both fields with or without the beta, so it drops them under drop_params. A system turn the strip empties stays when it still carries output_config, and the drop warning names both causes. * fix(anthropic): send the thinking betas on every leg and keep fields where the beta is unknown Vertex messages and Claude Platform chat now add the thinking betas when thinking carries block_binding or display updates. Converse sends only the caller's header betas when it has them. A provider with no row in the beta config forwards beta-gated fields instead of dropping them, an emptied trailing system turn that would leave an assistant prefill is sent as is, and the prompt cache baseline body matches the body that goes out. The drop WARNING now names the beta that unlocks each field * fix(anthropic): filter Converse body betas, keep Foundry auto betas, read the bundled beta config for row checks * fix(anthropic): validate messages only when a beta-gated drop runs A non-object messages entry on a Claude model without mid-conversation system support raised a raw pydantic ValidationError before the beta-config and drop_params checks. The strip now passes the messages through untouched unless it drops something, so the provider's 400 reaches SDK callers as BadRequestError again * fix(anthropic): keep system blocks with a non-string type instead of raising --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
1a05aab72a
|
fix(bedrock)!: route region-prefixed embedding models to that region and bill its price (#49072)
* fix(bedrock): send region-prefixed embedding model ids to that region without the prefix * chore(bedrock): hold the embedding fix back until the risk stamp * fix(bedrock): send region-prefixed embedding model ids to that region without the prefix * fix(bedrock): keep a model_id's region and price embeddings at the region the call goes to * fix(bedrock): read a model_id ARN's region for embeddings and keep unlisted regions mappable * fix(bedrock): route async_invoke embeddings whether the region comes before or after it * fix(bedrock): price embeddings by region_name and keep the path region with a plain model_id * fix(bedrock): price a model_id ARN at its region's row * fix(bedrock): keep chat's region rules for embeddings and leave ARN pricing as on main A plain model_id skips the region in the model, as chat does, so a deployment that worked on main keeps calling the same region. An empty aws_region_name fails region validation again instead of falling back. This reverts the ARN regional pricing, since many regional rows lack the bare row's priority, batch, and long-context rates --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
784a52de79
|
test(router_strategy): keep foreign asyncio task noise out of the budget limiter caplog asserts (#43375)
* fix(tests): keep foreign asyncio task noise out of the budget limiter caplog asserts The two gc.collect()-under-caplog tests in test_budget_limiter_hotpath.py asserted on every captured record, so a LoggingWorker task another test had parked on the session fixture loop got destroyed inside their window and asyncio's "Task was destroyed but it is pending!" error failed them. They now assert only the limiter logger's records and park a pending task on a closed loop in the window themselves, so the noise is exercised on every run The edenai and sail spend_capture fixtures no longer start the worker from the session loop: the test's own call and its flush bind the worker to the test loop * test(router_strategy): drop only asyncio records in the limiter caplog asserts and drain the response cache worker on the test loop --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
444423a5ca
|
fix(cost): bill a deployment's per-second rates on native Anthropic /v1/messages (#42417)
* fix(cost): honor per-second custom pricing on chat completions for every provider * test(cost): assert a per-second priced deployment bills instead of staying at $0 The zero-cost diagnostic tests from #42345 used a per-second-only entry as their misconfigured fixture, which this branch now bills. Switch that fixture to a per-query-only entry, which is still selected as the deployment's own pricing and still prices chat usage at $0, and add a per-second test asserting the call duration is billed with no diagnostic Also let a caller's explicit total_time outrank the logging window in completion_cost, so the SDK precedence stays stamped response, caller, logging * test(response_metadata): move the per-second pricing regression into the mapped tests/unit file * fix(cost): bill a deployment's per-second rates on /v1/responses and /v1/messages The proxy builds the request's logging state before the router picks a deployment, and only the chat path re-read the deployment's input_cost_per_second / output_cost_per_second afterwards. The shared update step every other route runs (Logging.update_from_kwargs) now seeds those custom pricing keys from the router-merged kwargs too, so custom_pricing flips on and the cost calculator prices the call against the router-registered deployment id on every surface. * fix(cost): price a /v1/messages call off the response time the proxy reports A /v1/messages result is a plain dict, so the proxy keeps its response time on the logging object. The cost step now reads it there for a non-streamed call, so the cost header and the spend log bill the same duration. * fix(anthropic): bill a deployment's per-second rate only on the native /v1/messages route Move the custom pricing copy from the shared logging refresh into the native Messages handler, reusing the helper the Responses handler already had, so generateContent and other routes keep their current billing * fix(cost): pass only per-second rates to native /v1/messages cost tracking Other deployment overrides keep billing as on main, matching chat's per-second gate * test(anthropic): pin per-second-only pricing on native /v1/messages with a long-context-rate deployment * fix(cost): bill native /v1/messages from logging setup, as main did, to the end the proxy reports --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
m6atlas
|
cd42694b48
|
feat(providers): add Kyma API as an OpenAI-compatible provider (#48756)
Adds Kyma API (kyma/<model>) as a JSON-configured OpenAI-compatible provider. Spend reads the usage.cost Kyma reports, with 83 catalog rows as the fallback price, and image and audio routes refuse before calling Kyma. Carries over #39799 with its author's commits kept |
||
|
tavren27
|
3e3362d461
|
fix(cohere): price native Cohere embed calls on Foundry relays and the Cohere passthrough (#48978)
* fix(azure_ai): price Foundry relays of the native Cohere v2 embed route A proxy relay of POST /azure_ai/<group>/providers/cohere/v2/embed logged spend 0 because the Foundry relay shapes had no entry for it, so the body fell through to the passthrough object with no usage. The Cohere v2 embed body-to-EmbeddingResponse transform is now one shared function that the direct Cohere path and the relay both call, priced from meta.billed_units and falling back to the echoed texts' token count when billed_units is absent. A body that is not a Cohere embed shape raises ValidationError, so the relay fails open as before. * fix(cohere): price native Cohere embeds on the v1 Foundry relay and the v2 passthrough, billing image tokens per caller The Foundry relay now also prices /providers/cohere/v1/embed, and the shared Cohere embed parser accepts the v1 plain list of float vectors next to the by-type object. Direct Cohere bills billed_units.image_tokens (or v1 images) apart from input_tokens, while Foundry already folds the image into input_tokens, so the parser takes an explicit CohereImageTokenBilling and each caller picks its own. The /cohere passthrough now routes and prices /v2/embed through the same parser as the cohere/ SDK path. The relay prices any body that carries billed units whatever texts holds, and the no-billed-units fallback counts special-token text as plain text instead of raising * fix(cohere): keep embed type labels, per-image v3 pricing, and the plain row for unpriceable passthrough embeds Passthrough and Foundry relay log rows keep each vector's embedding type, text-only usage keeps image_tokens null, a Cohere v3 billed image count is priced per image, and an embed call the cost map cannot price falls back to the plain passthrough row instead of raising inside the logger. * fix(cohere): warn when a passthrough embed cannot be priced and pin test timestamps * test(cohere): cover embed-jobs passthrough calls writing the plain passthrough row --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Priyansh Nandwana
|
67ae3fb527
|
fix(websearch)!: keep the provider prefix on the agentic follow-up model (#38885)
* fix(websearch): keep the provider prefix on the agentic follow-up model The agentic follow-up rebuilt its model string with "a slash means it is already qualified". Handlers are handed the provider-stripped model, and for a provider that routes through a sub-path the remainder still holds a slash, so bedrock/mantle/anthropic.claude-sonnet-5 arrived as mantle/anthropic.claude-sonnet-5 and stayed that way. The follow-up acompletion could not resolve a provider from it and raised "LLM Provider NOT provided". Prefix unless the model is already qualified with that same provider, in one shared helper used by both the handler and the websearch patch builder, which carried copies of the same heuristic. This also fixes openrouter, whose stripped model keeps a slash for the same reason, and the old startswith test that read openai as a prefix of openai_like. Fixes #38829 * test(websearch): cover the agentic follow-up call site The helper had unit tests but the call site did not, so the two changed lines in _execute_chat_completion_agentic_plan were uncovered. Drive the real follow-up with mock_response instead of patching litellm, which keeps the test on the behaviour: against the old code the sub-path case raises "LLM Provider NOT provided", and the ordinary-model case passes either way as a guard. * refactor(websearch): trim the comments on the provider-qualify helper * refactor(websearch): cut the helper and test docstrings to one line * fix(websearch): leave a hook-qualified follow-up model on its own provider A hook that patches in its own model owns that choice, so re-qualifying it sent a cross-provider follow-up to the original request's provider instead. Only a bare patched model and the request's provider-stripped model take the prefix now, and the sibling agentic-loop path resolves the model the same way. * fix(websearch): fall back to the request model when a patch carries no model An empty patched model dispatched a provider-only name instead of falling back, and a provider-stripped sub-path copied into the patch lost its prefix. A patched model now keeps its own provider only when its leading segment names one. * fix(websearch): keep the provider prefix on the responses follow-up model too The responses patch builder still used the old "a slash means already qualified" test, so a provider-stripped sub-path model reached the follow-up unqualified on that surface. It now resolves the model the same way the chat completion builder does. * fix(websearch): read custom_llm_provider without a member call on untyped kwargs * fix(websearch): type the chat patch builder kwargs instead of working around them * test(websearch): run the real search path and flatten the qualify helper tests * fix(websearch): route a hook's model on a JSON-configured provider and keep the request provider on an echoed model * fix(websearch): leave a follow-up model alone when its prefix already routes to the request provider * fix(websearch): send the agentic follow-up to the model the caller requested completion() and responses() record the caller's model before routing strips it, and the chat and Responses follow-up builders prefer that model. This keeps bedrock/converse_like/, bedrock/claude_platform/ and openrouter/openrouter/* deployments on their own route for the follow-up. * test(websearch): cover the recorded caller model in Headroom's retrieval follow-up * test(websearch): cover the recorded caller model in Compresr's retrieval follow-up Also merges the repeated provider-prefix check in core_helpers into one helper and drops a test docstring * test(websearch): drive the real search path through a mocked Perplexity HTTP route * refactor(websearch): read the provider from kwargs inside _caller_model * test(http-handler): type the agentic follow-up provider tests * refactor(websearch): record agentic loop params through a Logging method Moving the recording out of completion() keeps it under basedpyright's code-flow complexity limit after the merge with main. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
0cf2bb7296
|
fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params (#47761)
* fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params A GPT-5 request whose reasoning_effort the model map does not enable for that model lost the effort with no log line under drop_params. An Azure deployment name missing from the cost map lost reasoning_effort none this way and ran at the model's default effort. The gate now logs a warning naming the model, the level, the map entry it checked, and the model_info.base_model and model_info.reasoning_effort_levels fixes. * fix(openai): warn when the Responses route or a deployment's declared levels drop a reasoning effort The /v1/responses gate dropped a row-disabled GPT-5 effort, and reasoning for an OpenAI model without supports_reasoning, with no log line, and the router logged a declared-level drop only at INFO. Both Responses drops now log a WARNING and the router logs its drop at WARNING. The warning now leads with model_info.reasoning_effort_levels, since base_model only helps a deployment named differently from its underlying model. * fix: warn only when a reasoning level is actually dropped, and on generic reasoning_effort drops * fix: warn when thinking or a Bedrock Converse reasoning level is dropped, and name the router drop a reasoning level * fix: skip the drop warning when a non-reasoning model is asked for no reasoning, and honor declared levels on Bedrock Responses * fix: stay silent when reasoning is turned off on /v1/responses and warn on the Bedrock Responses fallback drop * fix(reasoning): warn when any reasoning-off request is dropped and give each route only the advice it honors A request to turn reasoning off on a model name the catalog lacks now warns on every route, since it may be a reasoning model. Chat suggests model_info.reasoning_effort_levels and, for a name with no catalog entry, model_info.base_model. OpenAI Responses suggests declared levels only where the config honors them, and Bedrock suggests neither. The Bedrock and Mantle Responses forwarding of declared levels is reverted until it can be proven live. * fix(reasoning): suggest reasoning_effort_levels and base_model only where the route reads them The chat gate suggested model_info.reasoning_effort_levels on routes that never pass declared levels (github_copilot, chatgpt and other OpenAIConfig delegates) and model_info.base_model on OpenAI and Azure AI chat, whose gate looks up the deployment name rather than base_model. /v1/responses/compact suggested levels it never reads either. Declared levels are now None when a caller passes none, so the advice appears only when a caller fed them, and the base_model advice is limited to the Azure chat gate, the one that looks up base_model. The openai row of the chat warning test now expects no base_model advice, the Responses warning test goes through get_optional_params_responses_api the way /v1/responses and compact call it, and the Azure direct-config test passes declared levels the way get_optional_params does. Long test lines are wrapped. * fix: read only the built-in catalog when choosing the reasoning drop warning |
||
|
mika-lt
|
71babd931b
|
fix(llms): move later developer messages for models declared system-first (#48897)
* fix(transformation): merge consecutive system and developer messages for non-OpenAI providers and Responses API
* fix(base_utils): keep later cache_control and billing metadata boundaries when merging system messages
* perf(base_utils): fold consecutive system messages in a single linear pass
The tuple accumulator passed to reduce copied the whole prefix on every step,
making map_developer_role_to_system_role O(n^2) in the number of messages.
Message count is not bounded by default, so a caller could tie up a proxy
worker with thousands of small messages before any provider call. Use a list
and merge into its last element in place instead.
* fix(base_utils): hoist later developer messages into the leading system message for non-OpenAI providers
A developer message that arrives after the first user turn (Codex CLI's
per-turn permissions item, a re-injected instruction after compaction, a
mid-conversation policy update) was translated to a system message in
place, and system-first chat templates such as Fireworks' qwen3.8 reject
any system message that is not at the beginning with 400. The fold that
merges consecutive system messages never saw it because a user turn sat
in between.
Move every developer message that follows the leading system block to
the end of that block before the fold, so it merges into the single
leading system message; client-authored mid-conversation system messages
stay where the client put them.
* fix(openai): hoist later developer messages into one leading system message on OpenAI-compatible backends
Developer role translation moves onto the provider config. OpenAI's own endpoint,
Azure OpenAI, and the o-series keep their current behavior. Every other config
inheriting OpenAIGPTConfig (Fireworks, DeepSeek, Together, vLLM, Groq, openai/
with a custom api_base) hoists developer messages that arrive after the leading
system block into that block and folds the block into a single system message,
which is what system-first chat templates require. Providers with their own
message format are untouched.
Messages carrying cache_control or Anthropic billing metadata fold as their own
text blocks, so breakpoints and provider-side stripping survive the merge. The
Responses bridge goes back to prepending instructions as a separate system
message and lets the hoist fold it.
* refactor(base_utils): fold each run of system messages in one pass
The pairwise fold recopied the accumulated system text or block list on every
merge, so a request carrying many consecutive system or developer messages cost
quadratic time in preprocessing. Group consecutive system messages once and
build the merged message from the whole run: plain strings join with a blank
line, and any cached, billing, or block content keeps one text block per
message. The name of the last named message in the run survives, as before.
* fix(base_utils): keep a closing developer message in place and hoist for OpenAIConfig backends
A developer message that closes the conversation stays where the client put it, translated
to system in place, so the request never ends on the assistant's turn. OpenAIConfig and its
subclasses (Azure AI Foundry, GitHub Copilot, ChatGPT, RAGFlow) now share the OpenAI-compatible
hoist, litellm_proxy passes developer messages through to the downstream proxy, and the Anthropic
billing header literals use the shared constant
* fix(base_utils): keep a closing developer message in place only after an assistant turn
A developer message that closes the conversation after a user or tool message hoists like
any other, so the request shape the ticket reports keeps working on system-first templates;
only a closing run that follows an assistant turn stays in place
* refactor(openai): reuse is_openai_backed_api_base for the hosted endpoint check
* test(base_utils): cover an empty system message inside a block-content run
Codecov flagged the empty-string branch of _text_blocks as the one line the
PR left uncovered. The existing empty-string test only hits the plain-text
join; this one forces the block path by merging an empty system message with
a developer message that carries list content.
* fix(openai): hoist developer messages for litellm_proxy and harden the system fold
- drop the LiteLLMProxyChatConfig passthrough so litellm_proxy/ hoists later developer messages like every other OpenAI-compatible endpoint, whichever backend or LiteLLM version sits downstream
- a system message with null or missing content inside a folded run is dropped instead of raising KeyError('content')
- a message-level cache_control on a list-content member lands on its last block so the breakpoint survives the fold
* fix(litellm_proxy): keep the AllMessageValues import the transform signatures still name
* fix(fireworks_ai): hoist developer items after the first turn on the native Responses path
The native Fireworks Responses path landed on staging mapping developer input items to
system in place, so a developer item after the first input item still hit the same
"System message must be at the beginning" 400 on system-first chat templates. The
ordering rules now live in one shared helper over item roles, and the native path
reorders through it before mapping, keeping a closing developer item after an assistant
turn in place exactly like the chat path does
* fix(transformation): carry the developer-message hoist onto main
|
||
|
quietjules
|
d037d86a94
|
fix(token_counter)!: keep a loaded Hugging Face tokenizer when a slower download fails (#48916)
* fix(utils): keep a loaded Hugging Face tokenizer over a racing failure and retry once per window A download that fails after another count loaded the same tokenizer no longer hides it, the first count after the retry window claims the retry while the others keep using tiktoken, and HUGGINGFACE_TOKENIZER_LOAD_RETRY_SECONDS defaults to 60s to match the proxy's tokenizer reuse window * fix(utils): keep Hugging Face tokenizer outcomes until their retry time instead of a TTL The outcome store no longer expires entries after twice the retry window, so after a long idle stretch the first count still claims the one retry and the other counts keep using tiktoken instead of each downloading again. The store is a small lock-guarded holder of an immutable mapping keyed by tokenizer kind and backend * fix(token_counter): keep the loaded tokenizer in its outcome and run one retry at a time The loaded outcome now holds the tokenizer, so an LRU eviction can no longer send every count back to an unguarded download. A retry blocks further retries until it ends, even when it runs past the window, and a download that fails after another count loaded the tokenizer counts with that tokenizer * fix(token_counter): retry a failed Hugging Face tokenizer download in the background After the retry window, the count that finds the stored failure starts one background download and answers with tiktoken, so no live count waits on a retry. A failure from a download that started before the first failure no longer resets a running retry, and a failed retry never replaces a tokenizer another download loaded. The outcome store takes its clock and retry starter as dependencies, so the tests inject them instead of patching time.monotonic. * fix(token_counter): keep retrying a failed Hugging Face tokenizer download when its retry thread cannot start If starting the background retry thread raises, the count answers with tiktoken and the failure is stored with a fresh retry time, so the next window retries again instead of the tokenizer staying on tiktoken until restart. * fix(token_counter): let a forked process or a crashed retry thread retry a failed Hugging Face tokenizer download again A failed download's in-flight retry mark is now the pid of the process running it, so a process forked during a retry starts its own instead of waiting on a thread it does not have. The retry thread clears its mark in a finally, so a native panic or fork guard error escaping it leaves the next window free to retry. The fork guard error is no longer swallowed in the retry thread and reaches the thread's exception hook, as on the first load * fix(token_counter): let a process that exits during a background tokenizer retry exit cleanly instead of aborting --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
bee9f679dc
|
fix(github_copilot): treat a malformed api-key.json like a missing one (#44723)
* fix(github_copilot): treat a malformed api-key.json like a missing one * fix(github_copilot): read the key and the endpoints of api-key.json as separate strict views * fix(github_copilot): ignore a non-UTF-8 api-key.json like any other malformed file Read the cached key file as bytes so a UTF-16 file, which Windows PowerShell 5.1 writes with >, fails JSON validation instead of raising the decoder's error, and the request gets the refresh or the login hint * fix(github_copilot): decode UTF-16 key files and refresh over a malformed endpoints field * fix(github_copilot): decode a UTF-16 access-token file and refuse an undecodable one like a missing one * fix(github_copilot): use the refreshed key when api-key.json cannot be written * fix(github_copilot): keep a refreshed key that cannot be saved in memory * fix(github_copilot): serve a kept key and its endpoint together after the key file is repaired A key held in memory is now checked before api-key.json, so the token and the endpoint come from the same source, and the map holds one entry per key file with the access token beside it * test(github_copilot): type the authenticator test fixture and the tests this branch adds --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
184ec7b894
|
fix(vertex_ai): build the cachedContents collection URL under a custom api_base (#44783)
* fix(gemini): use collection endpoint for custom context cache * fix(vertex_ai): build the cachedContents collection URL under a custom api_base * fix(vertex_ai): keep a gateway prefix that contains /publishers/ * test(integration): expect the cachedContents collection URL on a custom Gemini api_base * fix(vertex_ai): cut cache collection URLs at the base's location, graft a mount prefix * fix(vertex_ai): cache under a publisher model api_base in that model's location A Vertex AI api_base that names an endpoint or no model keeps its old cache URL, so no cache is created that the chat call cannot use. The cache body takes its project and location from the api_base model, matching the collection URL. * fix(vertex_ai): cache for the api_base's model and skip caching on a Gemini v1 base A Vertex api_base naming a publisher model now gives the cache its model as well as its project and location, so the cache matches the model generateContent calls. A Gemini api_base ending in /v1 skips context caching, since the v1 API has no cachedContents and rejects the cachedContent field. * fix(vertex_ai): check the cache token minimum against the api_base's model The cache is created for the model a publisher-model api_base names, so the minimum prompt size now comes from that model too. A prompt below its minimum is sent uncached instead of failing at cache create. * fix(vertex_ai): use the stricter cache minimum and send a Gemini base's 404 uncached --------- Co-authored-by: paulo037 <paulo.h.carneiro.18@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b93d26bc85
|
fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path (#45330)
* fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path * refactor(responses): reuse the shared prompt_cache_breakpoint_for_wire helper in the chat bridge * fix(responses): read a non-flag deployment drop_params as unset in the marker walk, like the param mapper * fix(responses): drop every prompt_cache_breakpoint under drop_params when the model has no breakpoints Also drop a marker set on the message item itself, sanitize before the cache_control conversion so a client typo no longer erases that breakpoint, and skip items that carry no marker so unmarked history costs no extra walk * fix(responses): keep a valid prompt_cache_breakpoint unless the model is known to reject it A deployment alias the cost map does not know (an Azure deployment name, a fine-tune, a litellm_proxy alias) kept its valid markers before this PR and must keep them under drop_params. base_model is read from litellm_params first, then model_info, as chat completions does. * fix(responses): drop a valid marker on known non-breakpoint models and keep the tool schema gate unchanged * fix(responses): judge breakpoint support by model name so cost-map aliases keep valid markers * fix(responses): keep the caller's request stripped of cache_control as before when a marker is copied * fix(responses): keep valid prompt cache markers in place and let cache_control fill a dropped one Under drop_params the marker step now hands back the caller's own objects when every marker is already wire-safe, so a request with valid markers is no longer copied and walked a second time. A deployment that turns cache_control into markers keeps the caller's own markers too, whatever its base_model says, so one request never gets two answers about breakpoint support. On /v1/messages a malformed raw marker next to cache_control is dropped before the conversion, so the block gets the cache hint cache_control alone would give, as on /v1/responses. * test(responses): keep a null marker and the original key order on a model that reads breakpoints --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
310c96162b
|
fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893)
* fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema * fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema. * fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them. * fix(bedrock): drop tool custom object on the Mantle count_tokens fallback Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way. * fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages * test(vertex): annotate count_tokens tool test locals with Final * fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names * fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
AaronHowell
|
904c092ffe
|
feat(providers): add BigModel support and model pricing (#48563)
* feat: add BigModel provider with native Chat, Responses, and Messages APIs Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> * test(providers): make BigModel unit tests a package Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> * docs(providers): clarify BigModel parameter interface Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> * fix(bigmodel): keep the Coding Plan base, surface Responses auth errors, and scope routes Chat Completions keeps a GLM Coding Plan api_base. Responses raises the status from BigModel's success=false body instead of returning an empty 200, and a JSON body answering a stream request is replayed as a one-shot stream. Messages streams keep ping and [DONE] lines. bigmodel leaves openai_compatible_providers so speech, image, and transcription routes send it no request * fix(responses): replay a JSON body on a stream request only when it has a Content-Length A gateway that labels a chunked SSE stream as application/json kept streaming before the replay, so only a sized body is read in full * fix(bigmodel): scope the JSON stream replay to BigModel, map its business error codes, and echo requested tools * fix(bigmodel): forward Codex's include and parallel_tool_calls, keep the OpenAI org header off BigModel, and pass unchanged Messages stream events through * fix(bigmodel): map disabled thinking to low effort, keep Messages temperature, raise context window and content policy errors, and default parallel_tool_calls to true * fix(bigmodel): keep the caller's reasoning_effort when Messages rewrites disabled thinking * fix(bigmodel): keep context-window and content-policy errors on streamed Responses replays --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
rj-cairn
|
5eb1042bbf
|
fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base (#48728)
* fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): keep tiktoken's encoding for suffixed variants of models it knows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(token_counter): move the o200k provenance note into the assert message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): strip openai/ and azure/ prefixes before picking the encoding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rust_bridge): azure/gpt-4o now resolves to o200k_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): check the provider prefix through _fix_model_name to keep the type gate flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(token_counter): check o200k on the gpt-6 models tiktoken 0.14 still does not know * fix(token_counter): count Azure deployment names and OpenAI alias groups with their OpenAI tokenizer * fix(token_counter): count mixed, alias, and base-model deployments with every served model's tokenizer A group that serves OpenAI and non-OpenAI models now reserves at least each served model's count, model_group_alias names count as their target group, custom Azure deployment names count with their base model, and azure_ai/, chatgpt/ and similar prefixes resolve to the listed OpenAI model. * fix(token_counter): keep mixed groups on OpenAI tokenizers, cache served models, and count context checks, failed rows and stream fallback with the served model * fix(token-counter): keep mixed groups at their old reservation and count new OpenAI release names with o200k A group that serves an OpenAI model next to another vendor now counts its own name the way tiktoken reads it as written, plus its OpenAI models, so it reserves what main did instead of the lower o200k count base_model picks the tokenizer for estimated usage only when it names an OpenAI model, now also in the Cursor stream counter and the output-limit response Unlisted gpt-5+ and o-series names count with o200k_base, and wildcard routes run the context-window check with the matched OpenAI model * fix(token-counter): count an alias to a group no OpenAI model serves by the alias as written Keeps the count main gives such an alias, so its budget reservation and rate limit match main and no other vendor's tokenizer loads for it. Also sorts the import the main merge left unsorted. * test: type the new token-counting tests' parameters * fix(token-counter): count a release-named group no OpenAI model serves as main does, and snapshot the OpenAI model set before iterating it * fix(token-counter): read served models through the router's allowlisted deployments_named, and coordinate the registration race test * perf(token-counter): classify each distinct served model once when walking a model group Budgeted requests walk every deployment in their model group, since the router's cached copy is outside the router surface the proxy may use. Classifying each distinct served model once instead of once per deployment cuts a 200-deployment group from 2.3-4.5 ms to about 1.4 ms with the same result. --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Alon Sokolovski
|
764a7a6536
|
fix(context_management)!: count the system prompt toward the clear_tool_uses input_tokens trigger (#47857)
* fix(context_management): count the system prompt toward the clear_tool_uses input_tokens trigger Anthropic's API compares an input_tokens trigger with the whole prompt, system included; the polyfill counted messages and tools only, so it cleared late by the system prompt's size. compact_20260112 already counts it: share its system_to_text. * fix(context_management): take read-only types in the system prompt count The type-discipline gate (LIT001) rejects new list and dict annotations. * chore: rerun CI (proxy-build-from-pip timed out) * fix(context_management): count the system prompt through compact's shared counter Finishes the main merge's conflict in clear_tool_uses.py. The input_tokens trigger now adds the system prompt through count_system_tokens, the same counter compact uses, while the count reused for cleared_input_tokens stays messages and tools only, so the reported savings are just the cleared tool results * perf(context_management): skip the token count when nothing can be cleared The keep check runs before the trigger count, the system prompt reader accepts any mapping block, and the system prompt test gets short parameter ids * test(context_management): keep the off-loop count test counting after the keep check moved first * test(context_management): cover a client compaction summary counting toward the trigger The trigger also skips the system prompt count when messages and tools alone already pass it * fix(context_management): move a client compaction summary into the system prompt when clearing runs before compact * fix(context_management): clear only the history after a client compaction block so compact still replaces its summary * fix(context_management): keep tool calls beside a client compaction block clearable --------- Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
dbaae521f9
|
fix(responses): drop the null created_by OpenAI rejects when output items are replayed (#48783)
* fix(responses): drop the null created_by OpenAI rejects when output items are replayed * fix(responses): drop a null created_by from replayed input items too A conversation saved from an older proxy still carries created_by: null on its compaction, apply_patch and shell items, so OpenAI keeps rejecting the replay. Drop the null from input items on the HTTP request transform and the WebSocket response.create rewrite, keeping a created_by that is not null. * fix(responses): drop a null created_by from counted input items too * test(responses): cover saved reasoning nulls and re-compaction of a saved conversation * fix(responses): drop a set created_by from input under drop_params * fix(types): type the Responses output serializer's parameters * fix(responses): drop a set created_by on reasoning items and Perplexity presets under drop_params --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
be649f66a6
|
feat(azure_ai): reach Foundry agents through the project Responses API (#47717)
* test(azure_ai): cover Foundry agents on the project Responses API * feat(azure_ai): reach Foundry agents through the project Responses API * fix(azure_ai): keep system prompts and project-endpoint auth on Foundry agent calls and follow-ups * fix(azure_ai): read AZURE_SCOPE before the Foundry audience default on project endpoints * fix(azure_ai): reject or drop instructions on previous_response_id turns and a caller's other agent for Foundry agents * fix(azure_ai): apply the other-agent rule to agent_reference sent at the top level of a Foundry agent request * fix(azure_ai): match the caller's agent by name and version, and hide the agent's own instructions * fix(azure_ai): keep instructions on plain Foundry models that return a null agent_reference * fix(azure_ai): keep a Foundry agent version requested at the top level when extra_body also names the agent * fix(azure_ai): answer 400 for /responses/compact on a Foundry agent instead of calling the project with the agent name |
||
|
kailian zhou
|
3a350cacc9
|
feat!: add StreamLake provider metadata (#41006)
* feat: add StreamLake provider metadata * fix(streamlake): complete provider discovery and test request flows * test(streamlake): assert cost and tool-param invariants instead of pinning vendor prices * fix(streamlake): accept tools and reasoning on endpoint ids and match the current price page Endpoint ids have no cost-map row, so tools and reasoning_effort were rejected before reaching StreamLake. The provider now declares both capabilities, and named non-reasoning rows opt out explicitly. The rows now follow StreamLake's 2026-09-17 price page: drop the KAT-Coder and Qwen3 rows it no longer serves, add the served models that had no price, drop account-level rpm and tpm, and fill missing context windows. StreamLake is also registered for streamlake/* wildcards and its messages and responses support. * fix(streamlake): regenerate the endpoint matrix and type the new tests The backup endpoint matrix advertised chat only while the root map listed Messages and Responses, so /public/endpoints hid both. Regenerating with scripts/generate_provider_endpoints_support.py makes the two files match. Types the StreamLake tests and sets the Add Model test's API key with fireEvent.change, per the dashboard test guide. * fix(streamlake): drop the retired max_tokens field from the StreamLake cost-map rows * fix(streamlake): price the four served models StreamLake publishes rates for and add the Qwen3.8 cache prices * fix(streamlake): send token caps as max_completion_tokens and mark image models StreamLake ignores max_tokens on kimi-k2.6, kimi-k2.7-code, qwen3.6-27b and qwen3.6-35b-a3b, while every served model honors max_completion_tokens, so the JSON provider now maps max_tokens to max_completion_tokens instead of the reverse. Rows whose models read an image in a live probe set supports_vision, and the Qwen3.8 rows drop the explicit cache write price, which this route never uses since cache_control is dropped * fix(streamlake): keep cache_control marks, price explicit cache reads, and use API input limits StreamLake reads cache_control, so the provider keeps the marks through a preserve_cache_control flag. Qwen3.8 explicit cache hits now use their own read price, rows with a cache read price declare prompt caching, three models StreamLake serves without reasoning drop reasoning_effort, and max_input_tokens follows the limits the API reports. * fix(utils): warn when a custom_provider_map name matches a built-in provider --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mika-lt
|
4e5f585c2a
|
fix(guardrails): scan Responses tool results in texts-mode guardrails (#48827)
* fix(guardrails): scan Responses tool results in texts-mode guardrails * fix(guardrails): keep guardrail-added tools and non-text parts on Responses tool scans Tool result texts now come from the same conversion that builds the chat tool message, so items without a call_id, dict outputs, and outputs with no text parts stay aligned for Bedrock and PANW. A changed joined text replaces the first text part and keeps the other parts. scan_only_tool_results appends tools the guardrail returns with new names, like chat, and tool_result and computer_call_output items are scanned through the shared type list. * fix(guardrails): skip empty and screenshot tool outputs and keep text part types on Responses scans * fix(guardrails): keep Bedrock latest-turn masking aligned when tool outputs send no text * fix(guardrails): scan each Responses tool output text part on its own * fix(guardrails): split a Responses tool output's structured row when only textless outputs share its call_id * fix(guardrails): pair each Responses tool output with its own structured row --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
simonp9
|
83ae1bd63e
|
fix(responses): drop params compact rejects and send Azure compaction to the v1 route (#48876)
* fix(responses): drop metadata from compact requests to OpenAI and Azure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): drop user and drop_params-rejected fields on compact and route Azure compaction to v1 * fix(responses): keep stream on compact and keep non-dated Azure api-versions on the v1 compact route * fix(responses): drop compact params only for providers verified to reject them --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
3b9a59348f
|
fix(vertex_ai)!: send the caller's anthropic-version in the Vertex Claude /v1/messages body, not the header (#48814)
* fix(vertex_ai): drop the caller's anthropic-version header on Claude /v1/messages Vertex reads the API version from the body's anthropic_version and answers 400 when a forwarded anthropic-version header names a different one, which anthropic SDK 2.x always does. Strip the header on the Python and LITELLM_RUST Vertex messages routes. * fix(vertex_ai): carry the caller's anthropic-version into the Claude /v1/messages body Vertex reads the API version from the body alone, so the caller's anthropic-version header now becomes the body's anthropic_version instead of a pinned vertex-2023-10-16. The 2023-01-01 and 2023-06-01 first-party versions keep vertex-2023-10-16, and the header itself is still never sent. Same change on the Python proxy and the Rust gateway. * fix(vertex_ai): follow drop_params for an anthropic-version Vertex does not accept Only versions Vertex accepts in the body (vertex-2023-10-16, 2026-10-21) are carried into anthropic_version. 2023-06-01, 2023-01-01, a blank header, or no header keep vertex-2023-10-16. Any other value drops to vertex-2023-10-16 under drop_params and answers a 400 naming drop_params without it, on the Python proxy and the Rust gateway. * refactor(vertex_ai): return the unsupported anthropic-version error as a value and keep Rust helpers and tests in place * fix(rust gateway): resolve os.environ drop_params and accept the same flag words as Python The gateway read drop_params: os.environ/NAME and words like yes as off, so the drop_params opt-out from the new anthropic-version 400 did not work there * docs(rust router): name Router::from_config in the crate README --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
e1580b7b83
|
refactor(images): remove dall-e-2 and dall-e-3 code paths (#48703)
OpenAI shut down dall-e-2 and dall-e-3 on 2026-05-12. Drop their image generation and edit configs, the Azure dall-e-2 create-and-poll flow, and the silent dall-e-2 fallback when no model is passed. image_generation without a model or provider now fails locally with "model is required". image_edit and image_variation leave model out of the provider request instead of sending dall-e-2, so providers that ignore model (Topaz) keep working and OpenAI returns its own 400. |
||
|
rj-cairn
|
d06b889d41
|
fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream The Gemini and Vertex AI streaming path read the SSE body with httpx iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085, U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds one of them was cut in two. The tail is not a data: line and the head never parses, and the accumulate-and-retry fallback then carries the unparsed prefix forward, so every later event is appended to it and the stream ends with no more content and no usage. SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines split decoded text on exactly those, treating a CR at the end of one chunk and an LF at the start of the next as one break. The first request and every long-decode hop now read through them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks. * fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator * fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed Switch the remaining httpx stream readers (Anthropic chat, Google GenAI, Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a raw U+2028 or U+2029 inside a JSON string no longer splits an event. Keep the chunk that closes a think block in the logged and cached response when a stream ends without a finish_reason, close the block before a legacy function_call, and make sync Codestral FIM streaming send stream=true and use the text-completion-codestral stream parser. Test fakes that only mocked iter_lines now stream real httpx bytes and assert the content they feed in. * fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use split_sse_lines from sse_lines.py, which also gains a str.split fast path. Adds tests/unit/llms/codestral to the unit workflow shard. * fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines * test: type the new stream tests, drop a comment, and benchmark the SSE line splitter * test: type the iterator class parameters in the line-separator stream tests * fix(responses): stream tool call arguments opened in a text chunk * fix(responses): hold a tool call that shares a chunk with text until the text ends * test: type the a2a SSE handler helper and the sandbox fake responses --------- Co-authored-by: Tim Vink <vinktim@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> |
||
|
moe-berri
|
9e878cfae2
|
test(gemini): cover every web source for a grounded passage (#48861)
* fix(gemini): preserve every grounding web source * test(gemini): cover all grounding sources per passage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Zabir Raihan <zabirraihan@icloud.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
rj-cairn
|
f89b9c6150
|
refactor(types): replace Any with proven types in 28 files (#48903)
* refactor(types): replace Any with proven types in 52 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): use typing_extensions.TypedDict for validated seams Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): revert strict Anthropic response validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): accept a null finish reason from Bedrock Cohere and TwelveLabs Pegasus * test(bedrock): annotate the Pegasus null finish reason test --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
8e8f6dc66e
|
feat(tencent): add Seedream image generation and editing through TokenHub (#48735)
* feat(tencent): add Seedream image generation and editing through TokenHub * feat(tencent): price Seedream image generation and edits from TokenHub usage * test(tencent): cover Seedream image generation, edits, and pricing through TokenHub * fix(tencent): keep non-Seedream image models on the OpenAI-compatible route Only seedream- models use TokenHub's wand route, so Hy-Image and other tencent image models keep working as on main. The response now carries TokenHub's image size and format, Pro's flat fallback bills its default 2K size, and the new tests are typed. * fix(tencent): bill other Tencent image models as before and forward user and TokenHub params on edits * fix(tencent): keep dropped edit params out and parse form-string watermark and seed * fix(tencent): send extra_body edit fields to TokenHub as given * fix(tencent): keep the routed model on Seedream edits and bill a deployment's per-token price * fix(tencent): accept string drop params on edits, rebuild nested form fields, and honor a per-image deployment price An image edit with a non-list additional_drop_params answered 500 for every provider after this branch validated it as a list, so a non-list value now drops nothing again, as chat treats it. Seedream edits sent through the OpenAI SDK arrive as multipart form fields, so TokenHub-only booleans and integers are parsed and bracketed keys like optimize_prompt_options[mode] are rebuilt into objects. A deployment that sets only output_cost_per_image is billed that price, and the TokenHub URL uses the shared api_base helpers. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
567383c503
|
feat(proxy)!: model family and access group entries in model_max_budget (#47722)
* feat(ui): let the per-model budget field take a model pattern or access group * feat(proxy): let model_max_budget entries name a model family or access group, shared across replicas * fix(proxy): open each shared model budget window once for every replica, and match patterns in linear time Without Redis, the first replica to charge an entry in a new window now opens it in the window spend table and the others join it, so no flush replaces spend another replica already added. Budget patterns match by splitting on `*` instead of a regular expression, a refused request names the entry that refused it, and the key form no longer offers a typed pattern another row already budgets. * fix(proxy): read the shared budget window from the writer, and keep concurrent first charges on one replica Opening a shared window now reads the row from the writer, since a lagging read replica could hand a replica its own start and let a flush replace the others' spend. A charge that waited on the open no longer resets a local window another charge on the same replica opened meanwhile. Window tests use fixed timestamps. * fix(proxy): stop a non proxy admin from lifting a team model cap with a looser group or pattern entry Group and pattern entries now take precedence over other entries, so a team update by anyone but a proxy admin must keep each new or changed entry within every existing cap it can take precedence over. A pattern can match a provider-prefixed or wildcard-routed name, so any two patterns count as overlapping, and a model or group entry counts as outranking every pattern. * fix(proxy): narrow model budget pattern overlap, keep team caps under key patterns, and accept only typed patterns in the UI * fix(proxy): keep model family budgets through database blips, team models and key entries A failed window open no longer restarts the window at this replica's clock, so it cannot wipe shared spend. Spend charged meanwhile stays local and joins the shared window once the database answers A key entry named after a model replaces only the team entry of that name, so team pattern and group caps still apply. Team-scoped models are gated and charged under their public name, and group lookups see team deployments The team update guard compares caps by window length and only ranks patterns that can match the same name, including Bedrock vendor patterns behind a bedrock wildcard route * fix(proxy): refuse team entries that take precedence over a capped model_max_budget entry A non-proxy-admin could add a carve-out (an exact model, a narrower pattern or a group) at or under the family cap, and its own counter then multiplied the team's spend on that family. Any new entry that takes precedence over a capped entry now needs a proxy admin, and group members include the team's own public model names so team deployments cannot hide a group overlap. * test(router): cover team_public_model_names_of * docs(proxy): name pattern and access group keys in model_max_budget docstrings * fix(proxy): name the entry holding the cap in the team model budget raise refusal * fix(proxy): check precedence when a team edit makes an unenforced model_max_budget entry take effect * fix(proxy): let an org admin give a legacy model_max_budget row a budget_duration at the same or a lower cap * fix(proxy): keep a legacy model_max_budget row's stored cap when an org admin gives it a budget_duration |
||
|
Zabir Raihan
|
7d79824391
|
fix(gemini): preserve every grounding web source (#48745) | ||
|
Kerry Lu
|
4a70d80aef
|
fix(aiohttp_transport): keep the owned session open while a response stream is still being read (LIT-10237) (#48920)
* fix(aiohttp_transport): keep the owned session open while a response stream is still being read (LIT-10237) * test(aiohttp_transport): annotate the stream regression test locals with Final and drop its docstring --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
5215c2d3f7
|
fix(pricing): add databricks priority pay per token prices (#48891)
* fix(pricing): add databricks priority pay per token prices * fix(pricing): bill gemini 3.6 flash cache writes at its databricks input rate * fix(pricing): drop image token priority rate the cost map schema does not carry --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
dfd9ad4103
|
fix(responses)!: bill partial usage on mid-stream failures and report bridged /v1/messages failures to the proxy (#47537)
A Responses API stream that fails partway now bills the usage it already consumed, ends with an error instead of a clean stop, and honors stream_timeout between chunks. Bridged /v1/messages calls report those failures to the proxy's failure hooks, so spend, budgets and logs see them |
||
|
Temo
|
99ac0d3e39
|
fix(aiohttp_openai): forward service_tier in transform_response (#47263)
* fix(aiohttp_openai): forward service_tier in transform_response * chore(aiohttp_openai): mark service_tier assignment rebind-ok for the type gate * fix(aiohttp_openai): set service_tier with setattr like other providers ModelResponse has no declared service_tier field, so attribute assignment added a reportAttributeAccessIssue error to the type gate. * test(aiohttp_openai): verify serialized served service tier * fix(aiohttp_openai): copy response when preserving served tier --------- Co-authored-by: tin-berri <tin@berri.ai> |
||
|
devin-ai-integration[bot]
|
83ca14b392
|
fix(pricing): sync databricks model prices (#48884)
* fix(pricing): sync databricks model prices * chore: regenerate provider endpoint support for databricks image rows --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Amaan
|
7c7304c22b
|
feat(providers): add ZeroGPU as an OpenAI-compatible provider (#43874)
* feat(providers): add ZeroGPU as an OpenAI-compatible provider Registers zerogpu through the JSON provider registry so zerogpu/<model> routes to https://api.zerogpu.ai/v1/chat/completions with ZEROGPU_API_KEY and an optional ZEROGPU_API_BASE. Adds LlmProviders.ZEROGPU, the constants.py provider and endpoint lists, cost map entries for the 10 ZeroGPU text generation models, the endpoint matrix, the Add Model form fields, the README row and the dashboard provider entry * feat(zerogpu): add capability flags verified against the live API Live calls to https://api.zerogpu.ai/v1/chat/completions confirmed tools, tool_choice and json_schema response_format on 8 of the 10 catalog text models, cached token reporting on deepseek-v4.1-flash, gpt-4.1-mini, gpt-5.4-nano and gpt-5.6-luna, and reasoning_effort on gpt-5.6-luna. LFM2.5-1.2B-Instruct answers stream requests with plain JSON instead of SSE and LFM2.5-1.2B-Thinking returns 500 on every request, so both stay out of the cost map until the API serves them correctly * feat(zerogpu): price LFM2.5-1.2B-Instruct Adds zerogpu/LFM2.5-1.2B-Instruct to the cost map with the catalog prices and context window. It gets no capability flags because the live API ignores tools, response_format and max_completion_tokens for this model, so LiteLLM drops tool params for it instead of sending them. LFM2.5-1.2B-Thinking stays out of the cost map * fix(zerogpu): reject transcription requests before calling the upstream API ZeroGPU only serves chat completions, so drop it from the OpenAI transcription transport set. A zerogpu transcription request now fails locally instead of hitting an /audio/transcriptions endpoint that does not exist * test(zerogpu): cover the responses and messages bridges Both endpoints reach ZeroGPU through chat completions, so assert the request each one sends upstream and the reply it returns * refactor(zerogpu): move zerogpu out of the provider list tails New upstream providers append to the end of openai_compatible_endpoints, openai_compatible_providers and providers.json, so an entry at the tail conflicts with every one of them. Placing zerogpu next to long-stable entries keeps those merges clean. No behavior change * fix(zerogpu): price cached input for qwen3-30b-a3b-fp8 The ZeroGPU model catalog (https://docs.zerogpu.ai/docs/model-catalog, checked 2026-10-09) lists cached input for qwen3-30b-a3b-fp8 at $0.05 per 1M tokens. Without cache_read_input_token_cost, cached tokens were billed at the full $0.10 input rate * revert(zerogpu): leave the audio transcription provider set unchanged Per review, keep OPENAI_AUDIO_TRANSCRIPTION_PROVIDERS as upstream defines it. This PR only covers chat completions, and ZeroGPU now serves /audio/transcriptions anyway, so the local refusal and its test are removed * fix(zerogpu): add max_tokens so trim_messages can size ZeroGPU history trim_messages reads max_tokens as the fallback for max_input_tokens and evaluates it eagerly, so without it the lookup raised, got swallowed, and the full history came back untrimmed. max_tokens matches each model's maxTokens in ZeroGPU's models API (https://api-dashboard.zerogpu.ai/api/models, checked 2026-10-10) * test(zerogpu): record mocked requests on a recorder object The mock handler appended to a list the caller passed in, which AGENTS.md forbids. A recorder now owns the captured requests as a tuple * fix(zerogpu): cap max_tokens at each model's output limit Each cap is the max_output_tokens ZeroGPU stores for the model in its production models table (read 2026-10-20), the same column its /v1/models serves as max_length. That gives 65536 for deepseek-v4.1-flash, 32768 for gpt-4.1-mini and gpt-oss-120b, 128000 for gpt-5.4-nano and gpt-5.6-luna, 8192 for LFM2.5-1.2B-Instruct and llama-3.1-8b-instruct-fast, and 16384 for qwen3-30b-a3b-fp8. glm-5.3-flash is stored as 32786, which reads as a typo for 32768, so that row uses 32768 trim_messages sizes history from max_input_tokens, so lowering max_tokens leaves it unaffected * fix(zerogpu): flag prompt caching on glm-5.3-flash Repeating the same ~4.3K-token prompt against ZeroGPU on 2026-10-20 returned cached_tokens of 4320 on some glm-5.3-flash calls. gpt-oss-120b and llama-3.1-8b-instruct-fast returned 0 on all nine calls, and qwen3-30b-a3b-fp8 never returned cached_tokens, so they stay unflagged. Their cached prices stay as ZeroGPU publishes them, so any cache hit is still billed at the right rate * fix(zerogpu): set gpt-5.4-nano context window to 272000 ZeroGPU accepts a 271001-token gpt-5.4-nano prompt and rejects 273000 tokens as over the context window (checked 2026-10-21), so max_input_tokens is 272000 rather than the 400000 its catalog lists * fix(zerogpu): list zerogpu models for zerogpu/* wildcards zerogpu had no models_by_provider entry, so get_provider_models("zerogpu") returned None and a zerogpu/* wildcard listed no models. zerogpu_models is now filled from the cost map rows like openinfer_models and registered in models_by_provider * fix(zerogpu): flag gpt-5.4-nano as a reasoning model ZeroGPU honors reasoning_effort on gpt-5.4-nano (checked 2026-10-21). Without it the model spends 0 reasoning tokens, and with low or high it spends 32 or more. LiteLLM only treated it as a reasoning model by falling back to OpenAI's bare gpt-5.4-nano row, so get_model_info and /model/info reported supports_reasoning as null. The row now carries the flag itself * fix(zerogpu): flag deepseek, glm, gpt-oss and qwen3 as reasoning models ZeroGPU's /v1/models marks these four as reasoning models and accepts reasoning_effort on all of them (checked 2026-10-21), but LiteLLM rejected the param with UnsupportedParamsError because their rows had no supports_reasoning flag. glm reports reasoning_tokens under reasoning_effort, and gpt-oss and qwen3 spend hidden reasoning tokens on every reply. The gpt-oss test case now expects reasoning_effort to reach ZeroGPU * fix(zerogpu): set gpt-oss-120b context window to 128000 ZeroGPU routes gpt-oss-120b across several upstream backends, and at least one of them returns 413 past a 128000-token window, so max_input_tokens uses that as the safe floor. The backends reached on 2026-10-21 accepted up to about 131K prompt tokens, so this only gives up a little headroom on the larger ones * feat(zerogpu): call ZeroGPU's Responses API natively ZeroGPU serves /v1/responses for all nine catalog models, so LiteLLM no longer bridges Responses requests through chat completions. Native calls return ZeroGPU's reasoning output items for glm, gpt-oss and qwen3, which the bridge dropped. Checked live on 2026-10-22 through litellm.responses for every model, non-streaming and streaming, plus a gpt-oss tool call. LFM2.5-1.2B-Instruct streams correctly now that ZeroGPU's text_generation lane emits Responses SSE * fix(zerogpu): stop inferring the zerogpu provider from api_base alone Listing https://api.zerogpu.ai/v1 in openai_compatible_endpoints rerouted bare model names on that api_base from openai to zerogpu. A bare gpt-5.6-luna call with verbosity used to return 200 and started raising UnsupportedParamsError. Drop the entry and its api_base-alone test, so zerogpu is only selected through the zerogpu/ prefix * fix(zerogpu): price gpt-6-luna, llama-3.2-3b, LFM2.5-1.2B-Thinking and qwen3.6 A zerogpu/* wildcard served these four chat models with no cost-map row, so their calls logged $0 spend. Add rows at ZeroGPU's catalog prices from https://api-dashboard.zerogpu.ai/api/models, with context windows and output caps from ZeroGPU's models table. LFM2.5-1.2B-Thinking has no cached price, like LFM2.5-1.2B-Instruct. gpt-6-luna is flagged supports_reasoning because LiteLLM already routes gpt-6 names as reasoning models. Checked live on 2026-10-28, all four now report nonzero response_cost * fix(zerogpu): drop the retired max_tokens field from the zerogpu rows Upstream #48471 retired max_tokens from the cost map in favor of max_input_tokens and max_output_tokens, and test_no_row_carries_the_retired_max_tokens_field failed on all 13 zerogpu rows. Every row already carries the same value in max_output_tokens, so nothing is lost. The zerogpu limits test now checks that each row's output limit fits its context window * chore(zerogpu): regenerate provider endpoint support for zerogpu Upstream #48619 generates provider_endpoints_support.json from the cost map and checks it in CI, and the hand-written zerogpu entry no longer matched. Regenerated with scripts/generate_provider_endpoints_support.py. Only the zerogpu entry changed, and it now matches what the generator emits for other JSON-registry providers such as openinfer * fix(zerogpu): flag tools, json_schema and prompt caching on gpt-6-luna ZeroGPU serves function calling, tool_choice, json_schema response_format and prompt caching on gpt-6-luna, so set supports_function_calling, supports_tool_choice, supports_response_schema and supports_prompt_caching on its row in both cost maps. Checked live on 2026-10-28, a weather tool call came back with the right arguments, json_schema output validated, and repeated prompts reported 3260 of 3263 tokens cached * fix(zerogpu): set qwen3.6-35b-a3b context to 262144 and flag its tools and reasoning ZeroGPU accepts up to 262,144 input tokens on qwen3.6-35b-a3b and serves tools, tool_choice, json_schema response_format and reasoning_effort on it. Raise max_input_tokens from 16384 and set supports_function_calling, supports_tool_choice, supports_response_schema and supports_reasoning in both cost maps. Before this, LiteLLM rejected tools and reasoning_effort for the model. Checked live on 2026-10-28, a 209,905-token prompt succeeded, an oversized one was refused with a 262144-token limit, and tool calls, json_schema output and reasoning_effort all worked * fix(zerogpu): flag vision on gpt-6-luna and qwen3.6-35b-a3b ZeroGPU reads image inputs on gpt-6-luna and qwen3.6-35b-a3b, so set supports_vision on both rows in both cost maps. Checked live on 2026-10-28, both models named the animal in an image_url photo, and the image showed up in their prompt token counts |
||
|
tessellark
|
c9a5777739
|
fix(azure): share one Entra token request per worker and audit #47348 (#48769)
* test(azure): audit #47348 Add proxy integration tests for #47348's cached Entra token providers and off-loop token refresh, and fix the token herd the audit found: once the fetch moved to a worker thread, concurrent cold requests each fetched their own token. Every cached bearer token provider now serializes its calls so the first fetch fills the cache and the rest reuse it * fix(azure): build each cached Entra token provider once under concurrent first calls lru_cache does not hold back concurrent misses, so worker threads that miss together each built their own credential and token provider, and each fetched its own token. A lock around each provider cache makes the first caller build it and the rest reuse it * fix(azure): share Entra token fetches only while no valid token is held Concurrent callers now share one result (or one error) from the cold fetch, while callers inside the refresh window keep the still-valid token. The oidc/azure provider build and the client-assertion exchange are single-flighted per audience, and the shared helpers no longer call subscripted generic aliases, which crashed on Python 3.10. * fix(azure): refresh Entra tokens once while other requests keep the valid token MSAL treats a token with less than five minutes left as expired, so every request in that window fetched a new one. Now one request refreshes while the others keep using the still-valid token. Requests that waited on a fetch that failed retry once through a fresh fetch instead of all failing with the first error. Credentials that only implement get_token keep working, and OIDC token exchanges are cached per scope. * fix(azure): bound Entra token waits during an outage Requests waiting on a failed token fetch now retry only when the fetch before it succeeded, so during an Entra outage later bursts fail after one fetch instead of two. Requests during a refresh only get the held token while it has more than 60 seconds left, and a failed refresh falls back to that token instead of failing its request. * fix(azure): share one retry per failed Entra token fetch and raise on a failed refresh Waiters of a failed fetch now retry the flight they joined, so a fast-failing Entra makes one retry instead of one per waiter. A failed refresh raises again instead of returning the held token, which Postgres token auth read as a fresh one. The DefaultAzureCredential fallback reuses one provider per scope. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
43ce3e22aa
|
fix(responses)!: answer 400 before counting a lone Responses built-in tool on Claude (#48798)
* test(responses): audit #47363 Adds integration cells for the input token count paths #47363 changed: the provider id of a managed file and the 403 for another team's file on OpenAI, file_search and additional_tools parity plus cache-mark stripping and untranslatable tools on Claude, and the Bedrock runtime count leaving out the options it refuses. * fix(count-tokens): reject a lone tool with no chat equivalent before the Claude count call A Claude input token count whose only tool was mcp or computer_use_preview skipped the pre-call check and went to the provider, so the caller got the provider error instead of the 400 naming the tool. The check now runs whenever a tool would get a placeholder name. * fix(responses): count a lone Anthropic toolset and name the uncountable tool The lone-tool 400 now fires only for OpenAI Responses built-in tool types, so a nameless Anthropic toolset still counts. The error says LiteLLM cannot count that tool type for the model. Adds a Bedrock unit cell for the lone-tool 400 and pins the runtime wire test's tool shape per request type. * fix(responses): take the uncountable tool types from the OpenAI SDK and keep the region hint off LiteLLM's 400 The Responses built-in tool types now come from openai's ToolParam union, so a type the SDK adds is refused before the count call too. The AWS region hint no longer lands on an error LiteLLM raised before calling the provider. Tests cover two more lone types and the hint. * fix(responses): keep LiteLLM's own list of uncountable tool types and pass a non-string tool type through * fix(responses): count tool_search and namespace tools the way Claude chat inference sends them The count refused a lone tool_search with LiteLLM's 400 and sent a lone namespace raw or squashed it into an empty tool, while Claude chat inference maps tool_search to Anthropic's tool search tool and unpacks a namespace into its function tools. Both now go through the deployment's own map_openai_params, so the count sends what inference sends, and tool_search leaves the list of uncountable types. * fix(bedrock): count a Claude request with its deployment's own chat translation Bedrock invoke and Mantle inference turn a chat tool_search tool into Bedrock's tool search tool, but the Bedrock counter always translated with the Anthropic config, so the count answered 400 for a request inference serves. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
rj-cairn
|
7abff3dd63
|
fix(mistral): replay assistant reasoning as a Mistral thinking chunk (#48778)
* fix(mistral): replay assistant reasoning as a Mistral thinking chunk When a client sends back an earlier Mistral assistant turn, its reasoning (reasoning_content, thinking_blocks, or Mistral's own thinking chunk in content) now reaches Mistral as a ThinkChunk inside content. The top-level LiteLLM fields stay stripped because Mistral rejects them with 422. * fix(mistral): replay reasoning only to Mistral models that accept it Mistral answers 400 to a thinking chunk on models without reasoning support, so a mid-chat model switch, a fallback, or Vertex AI Mistral broke once the replay was added. Gate the replay on the same check reasoning_effort uses and keep dropping the reasoning for the other models, as before. * fix(mistral): drop native thinking chunks for models without reasoning and mark the devstral, code agent, small and leanstral aliases as reasoning * fix(mistral): keep the leanstral rows as on main, since their reasoning support cannot be checked live * fix(mistral): send only the text items of a native thinking chunk, so a reference or image item no longer gets Mistral's 422 * fix(mistral): keep closed false only on the last chunk of a final prefix turn, so an earlier turn no longer gets Mistral's 400 * fix(mistral): drop closed false when merged text follows the thinking chunk, so a prefix turn with text first no longer gets Mistral's 400 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
shrey
|
6e33d7e84b
|
fix(anthropic-messages): merge deployment and client anthropic-beta instead of overwriting (#48594) | ||
|
Onur C. Cakmak
|
d9aae590e3
|
fix(chatgpt): derive the upstream session id from prompt_cache_key (#42014)
* fix(chatgpt): derive Responses session id from prompt_cache_key ChatGPT partitions the Codex prompt cache by the Responses session-id header (the Codex CLI sends a stable per-conversation key derived from prompt_cache_key). The provider only consulted explicit session ids and otherwise stamped a per-request uuid4, so no turn ever reused the prompt cache -- 0% cache hits on repeat turns, an order-of-magnitude quota/cost multiplier on subscription accounts. Resolution order after this change: explicit session config (session_id / litellm_session_id / metadata.session_id) > prompt_cache_key (sanitized with _safe_header_value) > litellm-internal ids > uuid4. Two guards, mirroring get_fireworks_session_id: - proxy-generated session ids (missing_session_id: "generate") are per-request and are skipped, checked in both "metadata" and "litellm_metadata" (LITELLM_METADATA_ROUTES, responses included) - internal per-request ids (litellm_trace_id / litellm_call_id) lose to a stable conversation key Supersedes the body-key forwarding of #37630: a random session header overrides the body key on the provider side, so the header must carry the caller's key. Related: #29978, #29993. Verified against a real ChatGPT Codex account: stable session id -> 97-100% cached on repeat turns; random per-request -> 0%. * style: ruff format * fix(chatgpt): address review - hash unsafe cache keys, move tests, drop comments - hash prompt_cache_keys containing header-unsafe chars instead of mapping them through _safe_header_value, whose replacement char could collapse distinct keys onto one cache shard and serialize unrelated requests (greptile review) - strip the explanatory comment blocks per AGENTS.md comment policy - move the tests into the existing mapped responses transformation test file per AGENTS.md test-placement rule * fix(chatgpt): keep session typing and test diff scoped * fix(chatgpt): hash surrogate cache keys, type session params, flatten tests * test(chatgpt): record upstream requests with a mock instead of appending to a list * fix(chatgpt): keep proxy-mapped session headers ahead of body session_id and ignore non-string cache keys * fix(chatgpt): hash cache keys with leading or trailing whitespace so the session header stays valid * test(chatgpt): type the session id test parameters * fix(chatgpt): keep trace ids copied into litellm_session_id from outranking prompt_cache_key * fix(chatgpt): mark session ids copied from trace_id so explicit ones keep precedence * fix(chatgpt): keep the trace-id session marker out of provider params The marker now uses the _litellm_ internal prefix, so bridges that pass litellm_params back into completion() drop it instead of sending it as a provider param --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
shrey
|
4fa860b64c
|
fix(vertex-ai): preserve image size aspect ratios (#48463) |