148 commits

Author	SHA1	Message	Date
norkelbyte	845f109524	fix(bedrock): forward x-pass- headers on the /bedrock/model pass-through routes (#50728) ... * fix(bedrock): forward x-pass- headers on the /bedrock/model pass-through routes The /bedrock/model/{model}/invoke, invoke-with-response-stream, converse and converse-stream routes built the pass-through call from the request body, so x-pass- headers never reached Bedrock. The processor now hands the caller's x-pass- headers to the pass-through route, keeps them out of the logged body snapshot, and on Converse drops the invoke-only guardrail headers when the deployment sets guardrailConfig and the request metadata header when the proxy owns request metadata * fix(bedrock): bill the served service tier on invoke pass-through and scope x-pass- forwarding to the Bedrock routes * test(bedrock): assert forwarded x-pass- headers on the request Bedrock receives * test(bedrock): gate the router-model x-pass- forwarding on the request Bedrock receives * fix(bedrock): bill a caller's forwarded invoke guardrail and keep x-pass- headers out of guardrail vendor payloads * fix(bedrock): bill a caller guardrail on every invoke model and replace a trace value Bedrock rejects * fix(bedrock): keep guardrail traces out of chat responses, forward the trace header as sent, and drop credential x-pass- headers --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 21:28:49 +00:00
mrinal-berri	9ba551b871	feat(proxy): let teams use their own allowed S3 buckets for batch files (#50571) ... * feat(proxy): let teams send allowed S3 buckets for batch files and read them back A team's metadata can now list allowed_s3_buckets. Keys on that team may send one of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on POST /v1/batches, and the upload and batch-create handlers check the bucket against the team list plus general_settings.allowed_client_side_s3_buckets. The early auth body check still rejects bucket fields on every other route and method Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri through an internal stored_s3_bucket_name param, so content, retrieve, delete and the batch cost job can read files written to a request-chosen bucket. Callers can never send that param, and files in the model bucket resolve exactly as before * test(proxy): assert outcomes in team bucket tests and type the managed content read Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count * fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix * fix(proxy): keep the request's approved S3 bucket when merging deployment credentials Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none * fix(proxy): keep the request bucket only on upload and batch create, not on reads The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for * fix(proxy): only proxy admins can set a team's allowed_s3_buckets Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes	2026-11-10 11:06:17 -08:00
zoveki	901f914ba2	fix(responses): replay Claude tool search on follow-up turns (#50821) ... * fix(responses): replay Claude tool search on follow-up turns Claude tool search on /v1/responses now returns a tool_search_output after each tool_search_call, streamed and not, listing the tools the search loaded as the caller sent them A stateless follow-up sends the search back to Anthropic as the server_tool_use and tool_search_tool_result Claude produced, and the Anthropic request drops a replayed search whose loaded tools are no longer sent, so a changed tool list no longer answers 400 The guardrail write-back for SDK output items reads part text through a typed helper, so the new output item type adds no attribute errors * fix(responses): emit Claude's tool search output in stream order The streamed tool_search_output item was announced at the end of the stream with an output index reserved earlier, so output_item.added events arrived as 0, 2, 1 and the OpenAI SDK's stream helper raised IndexError. The item is now emitted when Claude's search result arrives. * fix(anthropic): keep the caller's tool names in tool search results A tool name Anthropic cannot take gets rewritten on the request, so Claude's tool_search_tool_result named the rewritten tool. The response now maps those tool references back to the caller's names, streamed and not, and a replayed search result maps them forward again on the next turn. * fix(responses): replay Claude's tool search only on routes that run it A follow-up sent to a route that cannot run tool search, a Claude model without it included, now leaves out the replayed search the way it did before, instead of sending Claude search blocks that model cannot take. The stateless gate and the stored-turn filter now use the same check that already decides whether the tool_search tool itself is kept A streamed tool_search_call now gets its output_item.done when Claude's search result arrives, so it finishes before its tool_search_output. Clients that build history from done events, like Codex, kept the output ahead of its call * fix(responses): keep a replayed search when the caller sent Anthropic's own search tool A Claude model the cost map does not flag for tool search still runs it when the request carries Anthropic's own tool_search_tool_regex or bm25 tool, which is how turn 1 already decides to keep it. Both replay gates now follow that rule, so a stored follow-up on such a model keeps Claude's search as it did before, and a stateless one replays it * fix(responses): replay a BM25 tool search under its own name * fix(responses): keep a replayed tool search visible to guardrails and prune only missing tools A replayed Claude tool search now reaches pre-call guardrails and the Purview DLP scan with its search arguments, a stale search drops only the tool references the follow-up no longer sends, and the stored previous_response_id path keeps the base gate for routes the cost map does not flag * fix(responses): replay Claude's search on a stateless follow-up only when it still sends a search tool A store false follow-up that sent no search tool still got the search replayed, and Claude then said it called a tool it no longer had * fix(responses): replay a stored Claude search only when the follow-up still sends a search tool A previous_response_id follow-up that sent no search tool still got the stored search, the same gap a store false follow-up had. The stale search pruning now logs the tools it leaves out at debug level. handler.py reads the reported speed from the usage chunk, and the guardrail write-back skips tool search outputs, so the branch adds no type errors. * fix(responses): leave out a stored Claude search kept in server_tool_calls Main stores Claude's server-side tool search on the assistant message in provider_specific_fields.server_tool_calls and tool_results, not in tool_calls. The previous_response_id path only looked at tool_calls, so a follow-up that sent no search tool still replayed the stored search. The session handler now finds the search in either place and drops it from both server_tool_calls and tool_results when the follow-up cannot run a search. * test(responses): give the replayed search guardrail fixture its own call id The guardrail test reused toolu_01Weather, an id that streaming and bridge tests already put in the process-wide TOOL_CALLS_CACHE. When one of those ran first in the same worker, the guardrail write-back took its rebuild path and the test failed for a reason unrelated to the replayed search. * fix(anthropic): keep Claude's replayed tool search on renamed tools and stored follow-ups A search-only assistant turn now gets its search result renamed to the sanitized tool names too, so the stale-tool trim no longer drops it. Stored follow-ups on Claude routes replay the search as base did, trimmed to the tools still sent, and Bedrock invoke sends the tool search beta whenever the history holds a search * test(responses): pin that guardrails scan a replayed Claude search on a follow-up without a search tool * fix(anthropic): keep Anthropic's tool names in chat tool search results A chat history whose search result used the request's tool names (weather.get) got Anthropic's 400 when replayed on a proxy still on the old code, during a rolling deploy or a rollback. Chat completions now returns Claude's search result as main does, and the Responses bridge maps a rewritten name back to the request's tool when it builds tool_search_output. That also fills tool_search_output on Bedrock invoke streams, which carry no name map --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-10 17:13:40 +00:00
yujonglee	523b3efea2	refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) ... * refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 02:37:35 +00:00
paveln83	0021d5493c	fix(bedrock): answer 400, not 500, for malformed count-tokens requests (#50708) ... * fix(bedrock): answer 400, not 500, for malformed native count-tokens requests The Bedrock count-tokens handler now checks the body before any AWS call and raises a 400 BedrockError for a missing or empty messages list, a message without role or content, or a missing model. The token counter marks that rejection as local, so it skips the Mantle fallback and the framing estimate, and /v1/messages/count_tokens answers invalid_request_error get_bedrock_base_model returns a bare cross-region prefix such as us or global unchanged instead of raising IndexError, so Bedrock judges the model id and its own 400 reaches the caller * fix(bedrock): answer 400 for a malformed message on Mantle token counts * fix(bedrock): limit the region-only model id fix to token counting * fix(bedrock): count a routing-prefixed region-only id and leave other providers' count errors as they were --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-09 23:04:36 +00:00
tavren27	2f430873ae	fix(bedrock): derive the web identity session policy per AWS caller (#50592) ... * fix(bedrock): derive the web identity session policy per AWS caller Each AWS caller (Bedrock embeddings, batches, files, SageMaker, Polly, S3 Vectors, and the S3 and SQS loggers) now sends its own STS session policy on AssumeRoleWithWebIdentity, so a role that allows those actions is no longer capped by the Bedrock-only policy. The default Bedrock policy is unchanged, and the credential cache key includes the policy so callers never share a narrower session. * fix(bedrock): let async invoke write its S3 output under the web identity session policy Bedrock writes an async-invoke job's output with the caller's session, so the embedding session policy now allows s3:PutObject only when Bedrock makes the call (aws:CalledViaLast bedrock.amazonaws.com). * fix(opensearch): give OpenSearch vector store search its own web identity session policy * fix(bedrock): give every remaining AWS caller its own web identity session policy Secrets Manager, knowledge base ingestion, model listing, the S3 config file reader, the SageMaker chat handler, and the Comprehend Medical and Transcribe pass-throughs each send a session policy with only the actions they call Bedrock embeddings now use the Bedrock policy, so chat and embeddings share one STS session. That policy adds async-invoke status, S3 reads and writes only through Bedrock, KMS only through S3, and Mantle token counting bedrock:InvokeModel gets its own statement without aws:SecureTransport, since Bedrock checks a Nova Sonic realtime stream as InvokeModel with that key false KMS grants are limited by kms:ViaService, file downloads get a read-only S3 policy, and batches and speech config export skip underscore attributes * test(batches): expect config export to skip every underscore attribute The session policy class attribute is private, so batches config export no longer returns single-underscore names * fix(bedrock): scope knowledge base ingestion role grants to the roles it creates Knowledge base ingestion only creates and passes litellm-bedrock-kb-* roles, so its session policy no longer lets the session create, edit, or pass any other role, the proxy's own included * fix(bedrock): give embeddings and the agent-runtime pass-through their own web identity session policies * fix(bedrock): grant enforced guardrails and the remaining agent-runtime operations over web identity Every web identity ceiling that grants bedrock:InvokeModel now also grants bedrock:ApplyGuardrail, so an account or organization enforced guardrail no longer denies embeddings, the agent-runtime pass-through, or realtime The agent-runtime ceiling grants every POST operation the pass-through forwards, adding flow executions, document content, and ingested document ACLs, and a test derives that list from botocore Only the realtime caller gets bedrock:InvokeModel without the aws:SecureTransport condition, since chat and embeddings never open a bidirectional stream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-08 19:13:53 +00:00
ryan-crabbe-berri	02d610f1a9	fix(anthropic): disable optional default thinking for reasoning effort none (#50541)	2026-11-07 12:51:37 -08:00
ryan-crabbe-berri	6b745635c8	fix(bedrock): forward Codex agent messages that carry encrypted content intact (#50457) ... * fix(bedrock): forward Codex agent messages that carry encrypted content intact Codex multi-agent v2 sends worker messages as agent_message items whose encrypted_content block only the model server can decrypt. Bedrock Mantle and Bedrock Runtime used to 400 on agent_message, so LiteLLM rewrote every one into assistant text, pasting the ciphertext into the prompt and dropping the sender and recipient. Bedrock now decrypts these items itself, so the rewrite left the recipient model reading an opaque token instead of its task. Agent messages with an encrypted_content block now go to Bedrock untouched on both endpoints. The Bedrock configs declare supports_encrypted_agent_messages, and the shared normalizer keys off that flag so the two cannot drift. Mantle's /v1/responses route for open-weight models still rejects agent_message in any form, so it keeps the old rewrite. Plaintext-only agent messages keep the rewrite on every route. * fix(bedrock): only keep worker messages locked for OpenAI GPT models Gemma and Grok on Mantle's OpenAI route never lock Codex worker messages, so Codex puts plain text in the locked slot and Bedrock rejected it with invalid_encrypted_content once the route alone decided to forward it intact. Decide from the model instead, and restore the full-value rewrite assertion.	2026-11-06 17:47:43 -08:00
Mateo Wang	3f38730d2c	refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) ... * refactor(typing): clear basedpyright Any errors (wave 1) * refactor(typing): clear basedpyright Any errors (wave 2) * refactor(typing): clear basedpyright Any errors (wave 3) * refactor(typing): clear basedpyright Any errors (wave 4) * refactor(typing): clear basedpyright Any errors (wave 5) * refactor(typing): clear basedpyright Any errors (wave 6) * fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves * fix(logging): keep callbacks running when litellm_params is a params model should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request Adds behavior tests for the changed lines codecov listed as not run * fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call. * fix: bill regional uplift multipliers float() accepts, bytes included * fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type	2026-11-06 13:31:15 -08:00
tavren27	2d2290067c	fix(bedrock): run realtime guardrails on Nova Sonic sessions (#49833) ... * fix(bedrock): run realtime guardrails on Nova Sonic sessions * fix(bedrock): quote the spoken violation message and skip it after a blocked Nova Sonic tool output * fix(bedrock): screen every message role, end Nova Sonic sessions on blocked transcripts, drop unparseable frames * fix(realtime): apply guardrail masks, keep block reasons out of spoken notices, fail closed on guardrail crashes * fix(realtime): screen joined text parts, keep the documented notice, honor key opt-outs Text parts of one message are joined before screening again, so a phrase split across parts is still blocked, and a mask collapses them into one part. The spoken notice is realtime_violation_message or the block reason, quoted only on Nova Sonic. Client-written output_text and text parts are screened. Key and team guardrail opt-outs reach regular and translation realtime sessions. A non-string upstream transcript skips screening as before. * fix(realtime): honor guardrail role scope, clear stale response.create drops, screen split keywords, report Nova guardrail outages * fix(realtime): keep every per-part mask when a joined text masks differently * fix(realtime): screen one joined text per item and pass the session's tenant context to guardrails Multi-part items reach each guardrail once as the space-joined text, as on main, and only an item a guardrail rewrites gets its parts screened again to keep per-part masks. Guardrails now receive the session request data with its metadata, so key and team policies apply, and a ModifyResponseException block is reported as a block. Nova scopes every message item as the user turn it sends to Bedrock. * fix(realtime): catch phrases split mid-word across text parts without re-screening single-part items * fix(realtime): screen each joined text in its own guardrail call and keep a Nova opening message for the retry while it is screened * fix(realtime): screen text parts in a loop so large messages keep their masks * fix(realtime): fail closed on a non-string transcript under a guardrail and give each turn its own guardrail metadata * fix(realtime): fail a non-string transcript only when a guardrail would screen the user's turn --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 13:18:48 -08:00
Mateo Wang	36c64ce85d	feat(bedrock): default to native structured output on Opus 5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 (#50221) ... * feat(bedrock): enable native structured output by default for selected Claude models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep govcloud and bare model rows off native output Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected native schemas with the JSON tool on default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep the rejection retry on the unforceable-tool native path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected schemas on the Messages path for default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format Messages native output retry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): allow profile-only native output metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): only disable default-native output on schema-rejection retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): type fixture parameters in native output retry tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): drop strict tools after a native-output retry hits the grammar limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format native output fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): honor the deployment native output opt-out and keep wide batch schemas on the JSON tool A deployment's supports_bedrock_native_structured_output: false now keeps the JSON tool on rows that default to native output. Bedrock batch has no retry, so a record whose schema has more than 64 properties keeps the JSON tool there too. * fix(bedrock): retry a native structured output stream as a buffered stream when Bedrock denies the stream action A role allowed only bedrock:InvokeModel got a 403 on schema streams once these models went native, since native streams call converse-stream while the JSON tool path buffered through converse. The retry now sends that request again as a buffered JSON tool stream, the shape the base sent. The wire test moves its neither-key and opt-in cells to Opus 4.7, its Bedrock-key-only cells to Sonnet 5.5, and its unforceable-tool cells to the bare Opus 5.5 id, since this PR makes Opus 5.5 and Haiku 5.5 always native. The batch wire test now checks a thinking schema record on both sides of the contract the batch unit test already pins, the JSON tool on opt-in Sonnet 5.5 and native output on always-native Haiku 5.5, since its Haiku 5.5 and Opus 4.8 cells assumed those models were opt-in. * fix(bedrock): move the Claude 5 native output default to a key released proxies do not read Released proxies load the cost map from main and have none of the native output retries, so setting supports_native_structured_output on the Opus 5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 profile rows turned working wide or min/max schemas into 400s on v1.108.1. Those rows keep supports_native_structured_output false and gain bedrock_native_structured_output_by_default true, which get_model_info reports as supports_native_structured_output, so this version behaves as before while released versions keep the JSON tool. * fix(bedrock): rename the Claude 5 native output default key to supports_bedrock_native_structured_output_by_default The cost map guard runs main's model catalog on the PR's cost map, and that catalog rejects any undeclared key that does not start with supports_. Declaring the old name on main first needs its own PR, so the key takes the supports_ prefix. Released proxies still never read it * fix(bedrock): honor registered and batch opt-outs, move Opus 5 and 4.8 to the default key A registered supports_bedrock_native_structured_output false now beats the native output default, and batch uploads read the deployment's model_info, so an opted-out deployment keeps the JSON tool on every record. The Opus 5 and Opus 4.8 profile rows move to supports_bedrock_native_structured_output_by_default, so released proxies on main's cost map keep the JSON tool on them. The 64 property batch cutoff now applies only to rows with that key, so models that were already native keep their batch output shape. * fix(proxy): report a deployment's Bedrock native output opt-out on /model/info /model/info merged the deployment's model_info over the backend row, so a deployment that set only supports_bedrock_native_structured_output: false still showed the backend's derived supports_native_structured_output: true. The generic flag is now derived from the merged info unless the deployment set it itself, through the same helper get_model_info uses. * fix(bedrock): keep native output when a denied schema stream is resent buffered A role without bedrock:InvokeModelWithResponseStream had its native output schema stream resent as a buffered JSON tool call. The resend now keeps the native params with fake_stream set, and falls back to the JSON tool only if Bedrock then rejects native output. * fix(bedrock): drop rejected strict tools on the JSON tool retry after a buffered native resend * fix(bedrock): send a buffered schema stream to /invoke and keep the by-default key on its deployment The shared HTTP handler now builds the non-streaming URL for a fake-streamed request, so the invoke route's buffered resend after a denied stream calls /invoke instead of invoke-with-response-stream. supports_bedrock_native_structured_output_by_default is deployment scoped, so setting it on one deployment no longer reaches sibling deployments of the same model * fix(bedrock): answer a buffered invoke stream from the non-streaming reply * fix(bedrock): keep the JSON tool retry buffered after a denied stream's schema is rejected * fix(bedrock): count only a true fake_stream and keep the deployment by-default key out of /model/info A fake_stream sent as "true" or 1 picked the non-streaming URL but kept the streaming reader, so Bedrock invoke streams answered 500 after three billed calls. Only a real True counts now, as on main. /model/info read supports_bedrock_native_structured_output_by_default from the deployment, while routing reads it only from the cost map, so it reported native output for deployments that use the JSON tool. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-06 09:38:00 -08:00
Siddharth	2a4f8b1d19	fix(bedrock): drop another model's signed thinking when replaying to a non-Claude Converse target (#50073) ... * fix(bedrock/converse): omit reasoning signature for non-Anthropic models (#49969) * test(bedrock): cover signed reasoning replay in mapped tests * fix(bedrock): drop another model's signed thinking when replaying to a non-Claude Converse target A signed thinking block from Claude replayed to a Converse model known not to be Claude, such as GPT-6 Luna or Nova 2, now leaves the request instead of going out as reasoningContent. Bedrock rejects that field on those models, so the whole request answered 400 A target that may still be Claude, such as an application inference profile ARN with no base_model, keeps the signed block, since Claude needs it on a tool-use continuation. Unsigned blocks keep the existing text path The thinking-block error matcher added earlier on this branch is removed, because Converse and non-Claude Bedrock requests never reach that retry * fix(bedrock): keep signed thinking for Claude behind a region prefix litellm does not list ca. and in. inference profiles name Claude after a region prefix get_bedrock_cross_region_inference_regions does not have yet, so they counted as non-Claude and lost their signed thinking. The thinking-only turn test now pins the exact merged and continue-message shapes. * fix(bedrock): limit the region-prefix Claude match to signed thinking and drop a dropped block's cache point * fix(bedrock): drop Claude's bridged reasoning items before a Bedrock Responses API replay * fix(bedrock_mantle): drop litellm-minted reasoning items before a Mantle Responses call * test(integration): expect Bedrock and Bedrock Mantle Responses to drop minted reasoning Bedrock and Bedrock Mantle answer 400 on a reasoning item litellm minted, so the wire test now expects both to drop it, like OpenAI and Azure --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 03:15:49 -08:00
Scott Wilson	08f2a98250	fix(bedrock): forward the per-turn-control beta to Bedrock Invoke (#43086) ... * fix(bedrock): forward the per-turn-control beta to Bedrock Invoke Claude Code sends `anthropic-beta: per-turn-control-2026-07-01` together with a mid-conversation `role: system` message carrying a message-level `output_config` (for example an effort change). The beta header config mapped the beta to null for `bedrock`, so LiteLLM dropped it while keeping the message in place on models flagged `supports_mid_conversation_system`, and Bedrock Invoke rejected the turn with "messages.N.output_config: Extra inputs are not permitted". Direct Bedrock InvokeModel probes with a synthetic request show Claude Opus 5.5 accepts that message shape only with this beta, and that Claude Haiku 4.5, Sonnet 4.6, Opus 4.8, Opus 5, and Sonnet 5 accept the beta on a plain request, so map it to itself for `bedrock`. `bedrock_converse` is unchanged. Generated with AI Co-Authored-By: Claude Code * test(anthropic): expect the per-turn-control beta to reach Bedrock Invoke Generated with AI Co-Authored-By: Claude Code * test(bedrock): drop the duplicate per-turn-control filter test and cite the live check * fix(rust): allow the per-turn-control beta on bedrock in the Rust beta table The Rust table must match litellm/anthropic_beta_headers_config.json cell for cell, and the JSON now maps the beta for bedrock. Also type the new Bedrock Invoke test. * fix(bedrock): forward per-turn-control on Bedrock Invoke only where the cost map flags the model Bedrock InvokeModel accepts per-turn-control-2026-07-01 on Claude 4.5 and newer and answers 400 invalid beta flag on Claude 3 Haiku. Released proxies load the beta map from main, so forwarding it in the bedrock row would turn Claude 3 Haiku requests on every running release from 200 into 400. Keep that row null and forward the client's beta from the Invoke transformation when a new claude-per-turn-control cost map rule sets supports_per_turn_control. * fix(bedrock): keep the per-turn-control beta through the messages handler filter on Bedrock Invoke The /v1/messages handler filtered client betas against the beta map before the Bedrock Invoke config could add per-turn-control back for models the cost map flags, so the beta never reached Bedrock. Provider configs now name betas their model accepts beyond the beta map, and the handler filter keeps those * fix(bedrock): keep a deployment's per-turn-control flag on that deployment supports_per_turn_control is now deployment-scoped, so a deployment's opt-in or opt-out no longer changes the beta forward for other deployments of the same Bedrock model. The forward reads the deployment's own model_info first, then the model map. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-05 22:29:17 -08:00
ryan-crabbe-berri	9a473fbbb0	feat(proxy): give Codex accurate model entries for every model in /v1/models (#49825) ... * feat(proxy): give Codex accurate model entries for every model in /v1/models * fix(router): read a deployment's upstream model without a not-required key access * fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider A deployment with no token price no longer votes a Codex service tier off, an undescribed model keeps Codex's default reasoning summary parameter, a Bedrock deployment naming its provider in custom_llm_provider gets Codex's stock entry, and Bedrock provider namespaces follow the AWS model id format. * fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog lite codex imports the catalog before the SDK is initialized, so importing cost tracking, the Bedrock helpers or provider resolution at module level hit a circular import and broke every lite command. * fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped Codex shows /fast for a leftover fast speed tier even when service_tiers no longer offers priority, so an alias, an empty configured tier list or a configured list without priority showed a /fast toggle that sends nothing. One helper now sets service tiers, speed tiers and the default together. * fix(proxy): give an older Codex a translated entry for models past its version Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release	2026-11-05 19:31:32 -08:00
ryan-crabbe-berri	48ff826cbc	feat(proxy)!: only let clients pick S3 buckets the admin allowed (#47496) ... * feat(proxy): let admins allow specific S3 buckets and endpoints in request bodies, block bucket settings otherwise Client requests can no longer pick s3_bucket_name, s3_output_bucket_name or allow_legacy_cloud_file_ids unless the value is on general_settings.allowed_client_side_s3_buckets (buckets) or the admin opted in the usual way. s3_endpoint_url, already blocked, can now be allowed per value through general_settings.allowed_client_side_s3_endpoint_urls * refactor(proxy): drive the client-side S3 allowlist from one typed field-to-setting map * fix(proxy): treat a null client-side S3 allowlist in config as empty instead of failing the request	2026-11-06 02:50:15 +00:00
Mateo Wang	14d50ed40d	feat(bedrock): default to native structured output on Opus 5 and Opus 4.8 (#50222) ... * feat(bedrock): enable native structured output for Opus 5 models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep govcloud and bare model rows off native output Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected native schemas with the JSON tool on default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep the rejection retry on the unforceable-tool native path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected schemas on the Messages path for default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): keep JSON-tool streaming coverage on unchanged route Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format Messages native output retry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): allow profile-only native output metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): only disable default-native output on schema-rejection retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): type fixture parameters in native output retry tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): drop strict tools after a native-output retry hits the grammar limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format native output fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-05 17:22:17 -08:00
quietjules	5a392b9e5f	fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696) ... * fix(anthropic): stop forcing the response_format json tool when Claude will think A forced tool call skips Claude's thinking. The json_schema emulation forced json_tool_call unless thinking.type was enabled, so adaptive thinking and the models that think by default (Opus 5, Sonnet 5, Haiku 5.5, and the always-on families) lost their thinking on every structured-output request. AnthropicModelInfo.response_format_tool_can_be_forced is now the one check the Anthropic, Bedrock Converse (including the native output fallback) and Databricks paths use. It leaves the tool unforced when thinking is requested in any form or the model thinks by default, and keeps forcing it when the request turns thinking off (thinking disabled or reasoning_effort none) on a model that accepts forced tools. The new thinking_on_by_default cost map key marks Opus 5, Sonnet 5 and Haiku 5.5 on the providers that reach these paths, and a miss-only fallback rule covers unmapped Claude 5 and later ids. * fix(bedrock): route native output and thinking token counts by whether Claude will think Bedrock's native output opt-in now uses the same will-think check as the json tool, so Claude 5 models that think by default with no thinking param go native. Converse also asks for /usage/output_tokens_details on those models, so reasoning tokens are reported for them. * fix(anthropic): tell Claude to answer through the unforced response_format tool Moves the integration tests that check the forced JSON tool from Claude 5 models to Opus 4.8, since Claude 5 models now keep their default thinking and take the unforced tool or native output * fix(anthropic): keep open-object schemas on the json tool when Claude will think Native output closes every object, so an open object came back empty. Also declare thinking_on_by_default in the Rust model catalog and expect the Converse thinking-token field in test_main * fix(anthropic): hold streamed prose until the JSON tool decides the answer In json mode with the unforced JSON tool, Claude can write its working before it calls the tool. The stream now holds that text, drops it when the JSON tool answers, and sends it when Claude calls a caller tool or skips the JSON tool, matching the non-streamed reply. * test(anthropic): build the stream test events without nested comprehensions * fix(anthropic): keep always-on thinking models unforced, open objects native when thinking, and the tool instruction Claude-only * test(anthropic): keep main's open-object integration cases and cover the unforced tool on a default-thinking model * fix(anthropic): treat thinking False as thinking off so the JSON tool stays forced Cover thinking False on the Anthropic and Converse forcing tests, and pin that a Converse json_schema stream with the unforced JSON tool never sends Claude's working text as content * fix(anthropic): drop the thinking False forcing case that completion never reaches litellm.completion turns thinking False into no thinking param before any route sees it, so the case was dead on the request path and False keeps meaning the model's default * test(bedrock): collect the Converse json_schema stream chunks in one assignment --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-05 04:07:11 +00:00
shrey	85fdea0fd1	fix(guardrails): apply guardrail masks to Bedrock InvokeModel Claude requests and replies (#49947)	2026-11-04 23:12:50 +00:00
norkelbyte	3b35ae340d	fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks (#49831) ... * fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks * fix(rate-limit): count embedding media as media only on models that embed it A data URI sent to a text-only embedding model is read as text, so it keeps the text estimate there. On models flagged supports_embedding_image_input, image and audio use the chat media estimates and PDF and video count at most the model's max_input_tokens. The same estimate now feeds project and key limits, the usage-based, latency and cost routers' deployment tpm, and the deployment itpm check. * fix(proxy): count embedding media only where the backend reads it as media Vertex AI, Gemini, Azure AI and Bedrock count a strict base64 data URI as media per item. Cohere counts it as media only when the whole batch is media, and Voyage, OCI and unknown models keep the text estimate. File blocks carrying a data URI count like the bare URI. The router context window pre-call check and the batch limiter now use the same estimate, resolving each deployment or JSONL row model to its backends. The cost map gains supports_embedding_image_input on voyage/voyage-multimodal-3 and oci/cohere.embed-multilingual-image-v3.0. * fix(router): hold each embedding media item to the context window on its own A PDF or video data URI alone already reserves at most the model's max_input_tokens, but the context window pre-call check then ran the blanked input through the Responses transform and added a few framing tokens, so a PDF that Vertex accepts got a 400. When every input item is media, the check now returns the largest item's media estimate, since an embedding request embeds each item separately * fix(proxy): leave embedding media out of the context check and narrow Bedrock media scope The router context window check now counts only the text of an embedding input, so Titan image lists no longer 400. Bedrock Marengo and Nova with a PDF keep the text estimate since they read those inputs as text or reject them, async_invoke Nova counts images as media, and the unused batch limiter change is dropped. * fix(bedrock): bill Cohere embeddings by Bedrock's input token count header * fix(embeddings): check data URI base64 in place and keep Bedrock Cohere text usage The strict base64 check now scans the payload in place instead of decoding and re-encoding a full copy. Bedrock's billed token header is used only for image requests, so text-only Bedrock Cohere usage matches main. The deployment itpm estimator change is reverted, since the router never passes embedding input to that check. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 22:06:59 +00:00
yuneng-jiang	883a2d1572	revert(bedrock): restore STS identity probe behavior before #49763 (#49939)	2026-11-03 21:34:59 +00:00
yuneng-jiang	e1c03c5423	fix(e2e): restore main qualification across providers and UI (#49763) ... * test: check an unsupported budget sort field * test: wait for dashboard navigation readiness * test: support clustered provider caches with live fallback * Revert "test: check an unsupported budget sort field" This reverts commit 9694787bb5. * test: check an unsupported budget sort field * test(e2e): refresh retired provider fixtures * test(e2e): read Langfuse observations through the supported API * test(e2e): identify trace roots by kind and request path * fix(e2e): repair UI readiness and regional AWS qualification * fix(e2e): scope LangSmith batch readback to its workspace * fix(e2e): use the configured global Bedrock GPT profile * test(e2e): defer unprovisioned GovCloud coverage * test(e2e): defer Bedrock Runtime Codex cases blocked by IAM * test(e2e): defer intermittent Azure translation coverage	2026-11-03 20:38:47 +00:00
zoveki	baddc67041	fix(anthropic)!: carry prompt_cache_breakpoint to cache_control on chat completions to Claude (#49783) ... * fix(anthropic): carry prompt_cache_breakpoint to cache_control on chat completions to Claude A content part carrying OpenAI's prompt_cache_breakpoint on a /v1/chat/completions request to Claude (Anthropic, Bedrock Invoke and Converse, Vertex AI, Azure AI) now reaches Claude as cache_control {"type": "ephemeral"} on the same block, so prompt caching works for OpenAI-shaped clients instead of the marker being stripped. Only parts OpenAI accepts the marker on are translated, a client cache_control on the same block wins, and Converse translates only for models that take cache points. A 30m ttl or a malformed marker answers UnsupportedParamsError without drop_params and becomes a default 5m mark or no mark with it. * fix(anthropic): hold translated prompt_cache_breakpoint marks to the 4-breakpoint cap Marks past Claude's 4 cache breakpoints now answer 400 without drop_params and keep the earliest under it, counting the client's own message, tool, and top-level cache_control marks. The ttl and malformed-marker errors name drop_params for proxy users too and no longer name a single provider. * fix(bedrock): count Converse tool cache points in the breakpoint cap and gate unmapped models A client tool cache_control or a tool_config injection point becomes a Converse tool cachePoint, so it now takes one of the 4 slots before a translated marker does. Converse only translates prompt_cache_breakpoint for a model the cost map says takes prompt caching. An unmapped model such as an application inference profile ARN opts in through model_info supports_prompt_cache_breakpoint, so a marker sent to every model of a mixed fleet no longer turns a non-caching profile's 200 into Bedrock's 403. Router prompt-caching affinity now treats a marked block like a cache_control one, and requests without a marker skip the translation after one pass over their messages. * fix(bedrock): count only the tool cache points Converse sends against the breakpoint cap * fix(anthropic,bedrock): cap only the cache breakpoints each provider sends, and drop marks Nova cannot take on tool results * perf(anthropic,bedrock): find the breakpoint cut-off by binary search so trimming many marks stays fast * fix(anthropic): drop the duplicate drop_params left by the main merge * fix(bedrock): let an inference profile's cache breakpoint flag win over its base_model * fix(anthropic): keep the first and last prompt_cache_breakpoint marks past the cap Under drop_params the translated marks that fit the cap are now split between the start and the end of the request, so the stable prefix and the newest turns both stay cached. The cap count also includes a native tool search tool that keeps its cache_control on the wire. Bedrock Converse now translates the marker only for the Claude and Nova families, which Bedrock documents as taking cache checkpoints, unless the deployment sets supports_prompt_cache_breakpoint. A guarded user turn keeps its cache mark when the request fits the cap. * refactor(bedrock): decide guarded cache marks with a typed bool helper * fix(bedrock): validate guarded blocks before reading their cache mark * fix(anthropic): let injection points replace client marks and keep non-Claude routes untranslated When a deployment sets cache_control_injection_points, the client's prompt_cache_breakpoint marks are dropped on targets that do not read them, so the operator's injection points decide every Claude breakpoint. OpenCode Go models served over the Anthropic messages endpoint translate the marker only for Claude, and guarded Converse text keeps a cache mark only on a model known to take cache points. * fix(anthropic): keep client marks unless a message injection point replaces them on a translating route A tool_config-only injection point no longer strips client prompt_cache_breakpoint marks, a flagged Bedrock profile no longer counts as reading the marker natively, and guarded Converse text skips the tool count when no message carries a cache mark. * fix(anthropic): keep Bedrock and Claude deployments flagged for the marker on cache_control injection A supports_prompt_cache_breakpoint flag on a Bedrock profile or a Claude deployment made the injection hook write OpenAI markers and prompt_cache_options, which Bedrock answers 400 for. Routes that translate the marker now never take the OpenAI dialect. * fix(bedrock): reserve only the tool cache points Converse sends when trimming Claude cache marks under drop_params * fix(anthropic): name the marker's accepted ttl values in the malformed prompt_cache_breakpoint 400 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>	2026-11-03 12:56:22 +00:00
norkelbyte	864b98acec	fix(bedrock)!: fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 (#49170) ... * fix(bedrock): fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000 The Bedrock SDK ends the output stream the same way for a dropped connection as for a session that sent sessionEnd, so the handler treated a dropped stream as a clean finish. The client got a 1000 close with no error event and the router logged 200 OK with no retry An output end now counts as clean only after the client left and the handler began closing the Bedrock input. An earlier end raises a 503 BedrockError, so the client gets an error event and a 1011 close, the failure handlers run, partial spend is still logged, and a retry of a committed session is still refused * fix(bedrock): fail a dropped Nova Sonic session right away and log it as failed When Bedrock's output side ends mid-session, by a drop or a modeled stream error, the client side no longer spends the close timeout sending close messages into the dead input stream, so the error reaches the app right after the drop. A session the app ends itself still sends Bedrock its close messages and closes the input The replay refusal now leads with the committed cause, so the error event and the cut 1011 close reason both start with it. It stays a 400 and is tagged as a failure outside the selected deployment, so router retries and fallbacks of a dropped session no longer add extra failure counts A clean realtime session now counts once toward its deployment's cooldown success counter, so cooldown reads the real failure rate instead of 100 percent. Realtime tpm and rpm counting is unchanged The spend row of a session the provider ended now reads failure with its error information, while the events already forwarded are still billed. A session the app ended keeps logging success * fix(bedrock): agree on a dropped realtime session's failure across loggers and the spend row A realtime session's success event now reports status failure, with the same error_information the spend row stores, when the session recorded a provider failure in its metadata. Usage already streamed stays billed and the router still counts one cooldown failure The spend row builder now runs every row's error_information through sanitize_error_information_for_spend_logs (DB size cap plus prompt redaction), so the realtime failure rows get the same treatment as the proxy failure hook, which no longer sanitizes on its own to avoid truncating twice Drop the selection-failure tag on the committed-session refusal, since the shared logging object already keeps a retried refusal out of cooldown, and pin the router's realtime success counting with an OpenAI realtime deployment * fix(bedrock): count realtime router successes only for sessions that logged one, and log first-turn drops The router counts a realtime success only when the session's success handlers ran, so a refused handshake or an upstream close before any frame no longer offsets its failure. The success event's payload status stays success again, so each logger gets one failure record per dropped session, while the spend row still reads failure from the metadata the Bedrock handler records. A committed Bedrock session that fails before any logged event now gets one zero-cost Logs row. After an output failure the input stream is closed in the background, so the SDK body writer finishes without delaying the client's error. Every input send failure reports the same 503 stream-ended cause as an output drop * fix(realtime): count a router success only for a session that ended cleanly A realtime session the upstream closes with 1011 or 1006 after its first frame now marks itself as ended abnormally where that close is seen, in the shared relay and in the OpenAI Live handler, so the router no longer counts it as a cooldown success. The success callbacks still run, so the proxy keeps the budget and parallel-slot settlement it had A Bedrock session that fails before Bedrock sends any usage now logs no success event, as on base. The spend-row failure fields go into a new metadata mapping instead of the request's own dict, validated with a pydantic TypeAdapter instead of an unchecked TypeIs guard. The handler tests drop a wall-clock bound and type their helpers * fix(bedrock): log a dropped realtime session's usage from its own logging state The failure event replaced the shared standard_logging_object while the queued success dispatch was still running its callbacks, so loggers received $0 and 0 tokens for a billed dropped session. * fix(proxy): settle a rate-limit reservation once when a request logs a failure and a success A dropped realtime session fires a failure event and then a success event for the same call. The v3 limiter refunded the TPM reservation on the failure and released it again on the success, so the key's TPM counter went negative. The success path now skips a reservation the failure already released and marks it settled for a later failure. The realtime handler test callbacks are typed * fix(rate-limit): settle each token reservation once when a request logs both a failure and a success A dropped realtime session logs a success for its usage and then a failure. After a success the limiter now clears every reserved amount (TPM, PTU, ITPM, OTPM), so a later failure refunds nothing twice. A new reservation written in the same context after a released one now starts clean, so reserve, fail, reserve, succeed charges the second request's usage once instead of 78 TPM or 28 ITPM for 27 and 20 used * fix(rate-limit): take a token hold before awaiting so concurrent success and failure callbacks settle it once * fix(rate-limit): track the ITPM/OTPM hold's release apart from TPM so a nested pre-call keeps the request's project IO charge * test(rate-limit): pin the split hold-release flags across nested MCP pre-calls * fix(realtime): count a relay that fails on its own as an abnormal end * test(realtime): pin the relay failure exit with a client frame the provider rejects --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:56:23 +00:00
tessellark	5b546259d5	fix(cache_control)!: keep one deployment's prompt-cache opt-in from reaching siblings on the same model (#49685) ... * fix(cache_control): read the routed deployment's own prompt-cache flag, not a sibling's shared cost-map entry Two deployments that share one model string share one key in the live litellm.model_cost map, and router registration writes each deployment's supports_prompt_cache_breakpoint into that key. The cache control hook read the live map, so a flagless deployment picked up a sibling's opt-in and sent prompt_cache_options and prompt_cache_breakpoint upstream, which the provider rejects with a 400 When a call carries the routed deployment's model_info, the prompt cache checks now use its declared supports_prompt_cache_breakpoint, and fall back to the loaded catalog instead of the live map. This covers the hook on completion, acompletion, Responses and the /v1/messages route, the azure_ai chat transform, the OpenAI Responses transform under drop_params, and the /v1/messages to Responses adapter * fix(prompt caching): read each deployment's own breakpoint flag in the openai and bedrock readers A deployment's own supports_prompt_cache_breakpoint now picks OpenAI's prompt_cache_breakpoint only where the request goes out OpenAI-shaped on a provider other than openai, such as azure_ai OpenAI models and bedrock_mantle. Bedrock on every route and providers that build Anthropic messages keep cache_control, which Bedrock Converse turns into cachePoint The openai chat reader now passes the routed model_info and checks base_model. The Bedrock cache point reader takes the deployment's flag first and, for a routed call with no flag, reads the catalog as loaded instead of the live map that sibling deployments write into. model_info is threaded from the converse transform, the factory message and tool builders, and the responses bridge * fix(prompt caching): keep gateways on cache_control and read register_model opt-ins on routed calls A declared supports_prompt_cache_breakpoint picks OpenAI's breakpoint only on providers hosting OpenAI's API or for a model the cost map lists under the provider, so OpenRouter, LiteLLM proxy, and vLLM deployments keep cache_control as before. Routed calls read the loaded catalog plus the user's own register_model entries, never router registrations. * fix(prompt caching): honor a deployment's own supports_prompt_caching on Bedrock and its declared litellm_provider on gateways * fix(prompt caching): keep wildcard cache flags off matched models, read deployment flags for auto caching and base_model aliases * test(anthropic): cover auto caching on an unmapped Claude deployment * fix(cache_control): keep the merge base's reads where the per-deployment flag changed them A register_model entry under a provider-prefixed name reaches routed calls again, since the overlay now resolves the bare and prefixed names the way runtime registrations are stored. The hook and the openai chat reader no longer judge base_model, as at the merge base, so a client base_model sent under allow_client_pricing_override cannot pick OpenAI's breakpoint and get a 400. A Bedrock deployment flag that is not a real true reads as false again, as at the merge base. An azure_ai wildcard's supports_prompt_cache_breakpoint speaks for each model it matches again, as at the merge base. Other wildcards still leave both cache flags to the catalog. The s10/s11 Bedrock wire test asserted that a flagless deployment inherits its sibling's flag on a shared ARN, which is the leak this PR removes. It now asserts that each deployment keeps its own flag and that a flagless unmapped ARN keeps its cache points after the delete. * fix(cache_control): let the later register_model entry win between a bare and a provider-prefixed name Routed calls read register_model entries through an overlay that merged the bare name after the provider-prefixed one whatever order they were registered in, so a later opt-out under one name lost to an earlier opt-in under the other. Each registration now moves its key to the end of the kept registrations, and the overlay merges both names oldest first, which matches what SDK calls read from the live model map. Also lists the wildcard cache flag helper in the router coverage script, since the hook's wildcard tests drive it rather than calling it by name. * fix(cache_control): keep register_model entries visible while a cost map reload replays them The previous commit moved a re-registered name to the end of the saved registrations by removing and re-adding it, and the overlay walked that dict to order a bare and a provider-prefixed name. A reload replays every saved entry, so a request on another thread could briefly miss an entry or fail on a dict that changed size mid-walk. Entries are written in place again, a per-name sequence records which registration came last, and both the overlay and the reload replay order by it, so the later registration still wins after a reload. * fix(cache_control): read a string or numeric supports_prompt_caching as the bool it spells and find register_model entries under any route prefix A deployment flag stored as "true", 1, "false" or 0 (POST /model/new and quoted YAML keep strings) now means that bool on the Bedrock cachePoint reader and Anthropic auto caching, and any other value falls to the catalog. register_model records which names wrote to each catalog row, so an opt-out under bedrock/ or bedrock/converse/ reaches routed calls like one under anthropic/. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:55:51 +00:00
tessellark	f4a2e2b3af	fix(anthropic): send an https PDF image_url to Claude as a document (#49708) ... * fix(anthropic): send an https PDF image_url to Claude as a document A PDF link sent as image_url to Claude on Anthropic went out as an image block and Anthropic answered 400. A link whose path ends in .pdf is now downloaded like Vertex links are, and the served Content-Type picks the document or image block, so CDN links that serve a PNG from a .pdf path keep working. An explicit format of application/pdf sends a url document with no download. PDF media types match regardless of case or parameters, and count_tokens picks the same block as the request. * fix(anthropic): download https PDF links off the event loop The proxy downloaded an https link ending in .pdf synchronously while building the Anthropic request, so on an async call every other request on that worker waited for the download. Those links are now inlined before the request is built, the same way http links already are, and the /v1/messages guardrail write-back does the same. A non-string format on an image_url is ignored the way it already was inside the image_url mapping, instead of raising an AttributeError. * fix(anthropic): send a PDF link the proxy cannot download as a link and keep guardrail url blocks as sent * test(anthropic): expect the PNG media type the served bytes carry at a .jpg path * fix(anthropic): bound the PDF link download and keep query-string PDF names Give up on a .pdf link after NON_LLM_CONNECTION_TIMEOUT instead of the client's 6000 s default, fall back to main's whole-URL extension check so download?file=report.pdf still counts as a PDF, treat application/x-pdf as a PDF, and set the url document's application/pdf format only for the guardrail write-back so other /v1/messages providers send what main sends * fix(anthropic): probe a .pdf link and send it to Claude as a url document Instead of downloading a .pdf link and inlining it as base64, ask the host for its first kilobyte and send the link as a url document when it serves a PDF. Every other link goes out as the url image main sends, so robots.txt handling, caching and the wire format stay as they are on main. Also restore the guardrail adapter and the content type inference to main, keep the client's url document blocks (title, context, citations, cache_control) through the guardrail write-back, treat a PDF filename in the query string as a PDF link, probe links concurrently on the sync path, cap the async probe's read at the download limit, and catch only transport, URL and SSRF failures in the probe * fix(anthropic): read only the first kilobyte when probing a PDF link, and restore each url document from its own place The probe now streams the response and keeps the first 1 KB, so a host that ignores Range no longer fills memory with up to 20 parallel bodies. The guardrail write-back now matches url documents inside a tool_result and outside one separately, since the write-back moves tool results ahead of other content * fix(anthropic): remember each PDF link probe's verdict and cap its total time, and skip the probe on Bedrock invoke * fix(anthropic): keep a PDF probe's error status for a minute, not ten --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:05:15 +00:00
tavren27	94b5b6386a	fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool (#49703) ... * fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool Opus 5.5 and Sonnet 5.5 on Bedrock cannot force the json tool, so a json_schema request without thinking sent an unforced json tool the model ignored. The opt-in native output key now also applies when the model map says the model cannot force a tool, and the Opus 5.5 commercial Bedrock rows carry the opt-in key. The GovCloud Opus 5.5 rows carry it as false, like the Sonnet 5.5 GovCloud rows * fix(bedrock): keep json tool with thinking on Opus 5.5 and Fable 5.1, and GovCloud on the tool Native output without thinking now applies to every Bedrock Claude model that cannot force the json tool unless supports_bedrock_native_structured_output is false, so Opus 5.5 needs no opt-in row and keeps the json tool when thinking is requested, as #46870 chose. The invoke chat path keeps the native output_format it built instead of inlining it, and GovCloud region-path ids read their own false rows. * fix(bedrock): clean caller output_format schemas before sending them natively on invoke * fix(bedrock): keep Mantle Claude on the json tool and send a caller's output_config.format natively on invoke * fix(bedrock): keep the faked Converse stream for native output that replaces the json tool A json_schema stream that goes native only because the model can't force the json tool now keeps the json tool's faked stream, so it still needs only bedrock:InvokeModel. /v1/messages builds a native-output retry only where it sends native output, and the Mantle test drops its mutable capture. * test(bedrock): cover native output and the opt-out in the invoke json-mode refusal tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 01:56:52 +00:00
simonp9	61da6fcb4d	fix(mcp)!: answer 400 for a hosted MCP tool a translated route cannot run, drop it under drop_params (#48931) ... * fix(bedrock, gemini, deepseek, databricks, gigachat, anthropic): answer 400 for a hosted MCP tool a route cannot run, drop it under drop_params A `type: "mcp"` tool sent to a route that rebuilds tools for its provider (Bedrock Converse and Nova, Bedrock Invoke Claude, Vertex AI Claude, Vertex AI and Google AI Studio Gemini, DeepSeek, Databricks Claude, GigaChat) was dropped silently and the request still answered 200. These routes now answer UnsupportedParamsError (400) naming the model, route and server, and pointing to the LiteLLM MCP gateway. Under drop_params the tool is dropped with a warning, other tools are kept, and no empty tools field is sent. Anthropic, Azure AI Claude and Claude Platform on AWS keep mapping the tool to the MCP connector. Native Responses providers and raw OpenAI-compatible forwarders are unchanged. * fix(mcp): check hosted MCP tools once per request and keep base tool_choice handling Each route now runs one request-level step before its param mapping. When the drop leaves no tools, the request sends no tools, tool_choice or parallel_tool_calls. Converse and GigaChat go back to their base tool_choice and empty-tools handling. * refactor(deepseek): hold the sent tools in a tuple * fix(mcp): keep tool_choice for legacy functions when every hosted MCP tool drops * fix(mcp): answer 400 for MCP servers a route cannot run on Messages, Ollama and the Responses playground /v1/messages requests with mcp_servers on a chat-adapter or Responses-bridge route now answer 400 naming each server, or drop it with a warning under drop_params, instead of being forwarded or silently ignored. Ollama generate no longer pastes a hosted MCP tool (headers included) into the prompt, and ollama_chat filters hosted MCP tools the same way. The Admin UI Responses playground now sends litellm_proxy/mcp references like the other endpoints. * fix(ollama): leave JSON mode off when drop_params drops every hosted MCP tool Ollama generate set format json for any request with tools, even after the hosted MCP check dropped all of them, so a plain answer came back as JSON or as a call to a tool the model never had. The check now runs before the JSON mode switch, which stays on only while a tool is left for the prompt. * fix(messages): honor additional_drop_params for mcp_servers and read the beta map only for hosted MCP tools * fix(mcp_security): accept gateway server names and toolsets the gateway resolves The guardrail compared litellm_proxy/mcp/<name> refs against registry keys, which are server ids, so a saved server referenced by name, alias, or toolset name was blocked although the gateway routes it. It now resolves each ref the way the gateway does and still blocks unknown names. * fix(tool_fields): keep MCP drop warnings on one log line Server labels, model names, and routes come from the caller, so a line break in one could forge extra log lines. The drop warnings now strip CR and LF from each value. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 13:13:03 -08:00
zoveki	6be98d8c75	fix(bedrock)!: give Invoke and Mantle Claude the model's own default max_tokens (#49692) ... * fix(bedrock): resolve Invoke Claude default max_tokens from the base model id Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): build Invoke Claude request params without mutating Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): type tool-search normalizer return to clear unknown-type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): resolve Invoke and Mantle Claude default max_tokens from the deployment id, ARN base_model, and Mantle row * fix(bedrock): default Invoke Claude max_tokens from the deployment's own cap and honor AnthropicConfig Read the called deployment's model_info.max_output_tokens before the base model's cost row, skip the shared regional key the router fills from every deployment, and leave the default to litellm.AnthropicConfig(max_tokens) when it is set. * fix(bedrock): send the mapped default max_tokens on bedrock_mantle Claude chat * fix(bedrock): bound a global AnthropicConfig max_tokens by the deployment's own cap * test(bedrock): annotate the new Mantle default max_tokens tests with Final * refactor(bedrock): inject the AnthropicConfig max_tokens reader into the Invoke Claude config --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 11:42:45 -08:00
tessellark	a2ab27373f	fix(proxy): count the context_management edits /v1/messages applies on every bridged provider (#49234) ... * fix(proxy): count the context_management edits /v1/messages applies on every bridged provider * fix: apply context_management edits in count_tokens on Responses bridges, compaction blocks and Bedrock Claude The Responses bridge now applies the same edits inference applies for every provider, the chat bridge cuts the history at a client compaction block, and Bedrock Claude forwards the edits AWS accepts with their beta to the mantle and runtime count APIs instead of dropping them. * refactor(count_tokens): return count-view failures as values and raise them once in the caller * fix(bedrock): keep the runtime's 400 when mantle does not serve the model it retries * fix(count_tokens): forward context edits on Vertex and Azure Claude counts and keep Bedrock's mantle answer without edits * fix(proxy): count edited bodies on native routes without a count API, keep counts on editor failures * fix(proxy): keep the Bedrock pass-through count shape and provider 400s on empty bridged bodies The pass-through count picks Converse or InvokeModel from the content again, and the edited InvokeModel count wraps string turns in text blocks. A bridged view with no messages sends the body as before. Azure and Vertex load the beta config off the event loop before a count with edits, and a model with no deployment reads drop_params from the request and router defaults. * fix(bedrock): send an empty-string turn as is when a count forwards context edits Wrapping an empty string into a text block made AWS reject the count with 'text content blocks must be non-empty', while base and /v1/messages accept the empty string. Only non-empty string turns are wrapped now. * test(azure_ai): compare the count's betas with the betas Azure inference sends * test(azure_ai): record the Azure inference request with respx instead of appending to a list --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 10:38:35 -08:00
simonp9	f7b2cd0000	fix(anthropic): answer 400 naming a non-string role or tool call id instead of 500 (#49237) ... * fix(anthropic): answer 400 naming a non-string role or tool call id instead of 500 A list role or an object or list tool_call_id crashed the chat conversion with "unhashable type". Depending on the step that hit it, callers got a 500 or a 400 that never named the field. A failed conversion now answers 400 naming the first non-string role or tool call id, and the prompt caching hook reads a non-string answer id as matching no tool call. * fix(anthropic): name the field the conversion failed on, and answer 400 for a non-string Responses call_id The chat field check now runs only when the conversion raised a type error, and scans roles first, then assistant tool call ids, then tool answer ids, so an orphan tool answer that modify_params drops is no longer blamed. A tool output call_id that is not a string on /v1/responses answers 400 naming input[N].call_id, and the clear_tool_uses polyfill leaves a non-string tool_use_id for the provider * fix(anthropic): name a non-string role whatever error the conversion raised An int or bool role fails the conversion with an invalid message error, not a type error, so it lost the named 400. Roles are now named for any conversion failure, while tool call ids stay limited to type errors * fix(responses): name a non-string message item role by its input index * fix(responses): name a malformed input field only where litellm reads it, by the caller's index A tool output call_id is named only when it is an object or a list and the conversion fails, so an orphan numeric id still converts and another field's failure is never blamed on it. On the bridge to Claude a non-string role or function_call call_id is named by its input item, counting items before a compaction item. A numeric assistant tool call id on chat is named too. * fix(anthropic): name a tool id only for the error it causes and count Codex tool items in input indexes A numeric tool answer id that modify_params drops was blamed for any later type or attribute error, so the 400 hid the real cause. Tool ids are now named only for the error their value raises, and a hashable answer id never. The /v1/responses bridge and its input token count now read the input the caller sent, with only additional_tools hoisted into tools, so input[N] counts those items too. * fix(anthropic): name a tool id only for the error its own value causes and keep Codex tool item text out of the request * fix(responses): name a tool output call_id only when the items before it convert * fix(responses): name a non-string input field only on inference so token counting and batch records keep their answer * fix(responses): let a client field named caller_input or names_non_string_fields reach the bridge * test(anthropic): build the non-string id histories without editing them in place * fix(anthropic): never name an empty role for another field's failure * fix(anthropic): name an empty list or object role when the conversion fails on it * fix(anthropic): name a list role under the drop_params cache cap and keep the Bedrock batch line prefix --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 21:13:24 +00:00
yucheng-berri	efc2304b7e	fix(streaming)!: fail a stream that ends without a finish reason (#47665) ... * fix(streaming): fail on clean EOF without finish reason * fix(streaming): cover shared OpenAI-compatible EOF paths * test(router): complete mocked streaming responses * fix(streaming): keep the EOF error message and cover Anthropic clean EOF The request-less RemoteProtocolError tripped exception mapping into 'The .request property has not been set.' for non-OpenAI providers. Adds Anthropic and generic-chunk unit tests, an integration test across chat, the /v1/messages bridge and the /v1/responses bridge, and moves main's token-estimate test onto a finished stream with a sibling that checks the cut stream still bills the estimate on its failure log. * fix(streaming): count a finish reason any chunk sent, keep LangGraph async streams whole Gemini marks every chunk unfinished and a later usage chunk reset intermittent_finish_reason, so a finished Gemini stream failed at EOF. LangGraph's __aiter__ opened a new aiter_lines on every wrapper call, which hit StreamConsumed after the first chunk and now surfaced as a false EOF error instead of a silent truncation. * fix(streaming): keep the synthetic stop for a stream that sent no chunks Gemini's OpenAI-compatible endpoint answers with only [DONE] when reasoning uses up max_tokens. Failing every openai/azure stream that sent nothing turned that 200 into a 500, while every other provider already kept the stop for an empty stream. Adds integration cells for Gemini's trailing usage chunk, multi-event LangGraph runs, a mocked deployment and the [DONE]-only answer. * test(streaming): keep pytest.raises to one statement, drop the spend read for an empty stream An empty stream writes no spend row on main either, so the [DONE]-only cell checks only what the client sees. * fix(streaming): read the chunk's finish reason for the sticky flag * fix(agentcore): end a stream on messageStop when the runtime sends no metadata or final message * test(streaming): pin the logged-chunks guard, inject the LangGraph client, use current model names * fix(agentcore): say why the end-of-stream flags rebind * test(streaming): cover an AgentCore runtime that ends on messageStop through the proxy * test(streaming): cover SDK clients, request fallbacks and a concurrent burst for a cut stream * test(agentcore): inject the async transport instead of patching the aiohttp flag * test(guardrails): end the decision model chat fixture with a finish reason, as a real stream does * test(streaming): shape the AgentCore and LangGraph fixtures like captures from a real runtime and server * test(streaming): keep each stream's ending and land every spend row once through a spend log outage * test(streaming): end each stream by its provider's finish after a proxy worker is killed mid burst * test(streaming): run the outage test as one burst before and one during the outage instead of a capped wave loop * test(streaming): connect the outage test's proxy to the relay without TLS so its trigger sees the spend-log insert on an ssl=on Postgres * test(streaming): type the new stream EOF unit tests * test(streaming): a merged reasoning stream cut mid reasoning now fails Main's think tag test expected a stream that ends mid reasoning without a finish reason to finish with a closed think block. With this PR that stream fails, so the case moves to its own test that expects MidStreamFallbackError with the open think block as the partial content --------- Co-authored-by: King Star <mcxin.y@gmail.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-31 11:29:26 -07:00
mika-lt	c4b01b2ac8	fix(decisions)!: honor a deployment's own supports_audio_input and supports_video_input flags (#49207) ... * fix(decisions): honor a deployment's own supports_audio_input and supports_video_input flags A Decisions deployment's model_info flags for audio and video now win over the cost map, per media kind, the same way supports_box_questions already does. An unlisted model flagged true gets its clips sent upstream, and a false flag refuses them (or drops them under drop_params). OpenAI-format providers still refuse clips since their request shape has no field for them, and the SystemOne request builder no longer re-reads the cost map. * fix(decisions): ignore media flags on a deployment whose own mode is not decisions A chat-mode deployment's supports_audio_input describes chat input, so it no longer turns on audio or video clips for Decisions. The cost map decides for it, as before. Adds a proxy integration test for flagged, partly flagged, unflagged, and drop_params deployments on both /v1/decisions and /v1/systemone, and for caller-sent model_info. * fix(decisions): follow the cost map's mode for a flagged deployment without its own mode A chat deployment flagged for audio or video usually leaves mode unset, so its flags described chat input yet still let clips through on /v1/decisions. When the deployment sets no mode, the cost map's mode for the model now decides whether its media flags apply, and an unlisted model still takes them. The audio and video lookups also run only when the request carries a clip. * fix(decisions): read the mode a flagged deployment falls back to from the shipped cost map The router copies each deployment's model_info into the model's shared cost map row, so another deployment's mode there could switch off a flagged deployment's media flags. The fallback now reads the cost map as loaded, untouched by router registrations * fix(decisions): route clips in a load-balanced group to a deployment that takes them A Decisions or SystemOne request carrying audio or video now skips group deployments whose own flags or the cost map refuse that media, and keeps the whole group when none takes it, so the 400 still names the media. * fix(decisions): keep deployments that drop or cannot resolve clips in the media filter * fix(router): skip clip-refusing Decisions deployments after cooldown and tag filtering Running the media filter before cooldown turned the 400 a group answered when its only clip-taking deployment was cooling down into a 429. It now runs next to the image filter, just before order filtering, in both the async and sync selection paths. * fix(router): honor router default drop_params when skipping clip-refusing Decisions deployments * test(router): list the router default params helper in the router coverage check * test(bedrock): stop reloading the cost map module in the GovCloud fixture --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 01:09:46 -07:00
Vineeth Sai Varikuntla	a7b66dd8b9	fix(openai): stop stripping cache_control from the caller's own messages and tools (#35602) ... * fix(openai): stop stripping cache_control from the caller's messages Copy before removing the field, so the caller's own message objects are left alone. * fix(fireworks_ai): strip cache_control on a copy so a fallback keeps the caller's hints * fix: keep the caller's request intact on more providers and trim extra Claude cache hints under drop_params * fix: drop only the oldest extra cache hints inside one message when trimming to the Claude limit * fix: keep replayed session turns yielding their hints as whole turns while a request's own message keeps its newest hints * fix(cache_control): count only hints that reach Claude and cap hints on OpenAI-compatible Claude routes under drop_params * fix(cache_control): keep the capped Claude messages a list in OpenAI-compatible request bodies * fix(caching): cap Claude hints for string drop_params, a Claude base_model, and Converse tool messages The 4-hint cap now reads drop_params through drops_unsupported_params on chat, Converse, and Anthropic routes, names Claude by base_model on OpenAI-compatible routes, and counts a Converse tool message's hints as the one cache point it sends. * fix(caching): trim Claude cache hints in one wire-order pass and reserve a top-level cache_control on OpenAI-compatible routes Dropping the oldest hint one at a time re-walked the whole request per hint, which took seconds on long tool histories. The cap now drops each message's oldest hints in the order the provider counts them, so a string assistant turn loses its text hint before its tool call hints. OpenAI-compatible Claude routes also keep room for a request-level or extra_body cache_control under drop_params. * test: drop narrating comments and stacked generator clauses from cache_control tests * fix(vertex_ai): put every system message in the Gemini context cache so generateContent never sends it beside cachedContent --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 00:29:10 -07:00
tessellark	5d5e3685c1	fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths (#48933) ... * fix(logging): warn when drop_params drops a requested reasoning level on the remaining paths * fix(logging): keep guardrail and token-count reads of a request from logging drop warnings Only the real Anthropic-messages adapter request reports a dropped thinking budget. Guardrail, shadow-eval and token-count translations of the same request now drop it silently, so one request logs one warning. The 422 retry also no longer hashes each loc entry, so a list inside loc cannot raise TypeError. * fix(bedrock): keep the DeepSeek R1 Converse reasoning_effort drop at debug level Only allowed_openai_params reaches that branch, and it puts reasoning_effort back after the provider mapping, so a warning there claimed a drop that never happened * fix(logging): warn on Gemini non-integer budget drops and keep the auto-router baseline silent Gemini 2.x now logs the drop of a non-integer thinking budget. The auto-router savings baseline builds a body it never sends, so it no longer warns. Logged values are capped with reprlib, Azure AI marks only its drop_params retry as gated, and Bedrock Chat Completions names the model without its route prefix. * fix(logging): warn on the Gemini 3, Magistral, Ollama, Fireworks and DeepSeek ARN reasoning drops Gemini 3 and later now log a dropped thinking budget, Magistral logs the level it swaps for a reasoning prompt, Ollama logs a level it maps to think off, Fireworks logs chat_template_kwargs effort keys dropped for a model the map does not list as reasoning, and DeepSeek on Bedrock logs a drop when the model is addressed by an ARN that advertises the param. * fix(logging): warn on the remaining Ollama, DeepSeek and Tencent reasoning drops The ollama/ completion route now shares the chat route's think mapping, so a level it turns off logs the same warning. DeepSeek warns when it drops thinking.budget_tokens, and Tencent warns when an adaptive-only model gets a non-integer budget. Magistral no longer warns on low, medium, high or enabled thinking, the values its reasoning prompt stands in for. The Gemini 3 budget warning no longer claims the model uses its default level. * fix(reasoning): warn when Claude drops a non-string reasoning_effort Claude chat and native /v1/messages dropped a reasoning_effort like 5 or {"effort": 5} with no log line. Both now log one warn-only WARNING. A Magistral thinking.type sent as a list no longer raises TypeError. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 08:48:43 -07:00
Siddhant Badola	b71491c6c9	fix(bedrock): route a Bedrock model id with no Invoke provider to Converse (#43290) ... * fix(bedrock): route unmapped converse-only family model ids to converse #42713 stopped Bedrock-native OpenAI ids that are not in the cost map yet from falling to the invoke route. The same gap exists for the other families that have no invoke transformation at all: minimax, nvidia, writer, zai, google and xai. An id from one of them that is newer than the cost map hits get_bedrock_route -> "invoke" -> get_bedrock_invoke_provider() -> None -> BedrockError, when Converse would have served it Add a second family regex for those six and default them to converse, the same way the openai family is handled. _OPENAI_FAMILY_MODEL_RE is left alone because the prompt-caching rule is openai-specific Every shipped id of these families already resolves to a converse entry via its base model (checked against the cost map), so this only changes behaviour for ids that are not in the map yet Tests: six unmapped ids route to converse, and an unmapped meta.* id still routes to invoke since meta has an invoke handler * fix(bedrock): route a model id with no Invoke provider to Converse A bedrock/<model> id missing from the cost map fell to the Invoke route, where an id that names no Invoke provider (a new vendor's model, a bare provisioned-model, imported-model, or custom-model-deployment ARN) fails locally with "Unknown provider=None" before any request is sent. Such ids now go to Converse, which replaces the per-family regex with the general rule. Ids that name an Invoke provider and explicit route prefixes are unchanged. * test(bedrock): expect Converse for ids with no Invoke provider in the nova ARN and not-found tests * fix(bedrock): accept tools on Converse for a model with no cost-map row or Invoke provider, and keep a 404 for an invalid id * fix(bedrock): answer 404 for an invalid model id instead of AWS's 400 AWS answers a mistyped Bedrock model id with a 400 ValidationException. The mapping already raised NotFoundError, but passing AWS's 400 response through made the openai base class report 400, so the proxy answered 400 and the router skipped its cooldown * fix(bedrock): drop lookaround regexes from tool schemas for a model the cost map does not list * fix(bedrock): apply the unknown-model defaults only to bare ids LiteLLM knows nothing about A bare Bedrock id with no cost-map row and no Invoke vendor gets Converse's open params, its reasoning_effort as sent, lookaround-free tool schemas, and a 404 for AWS's invalid-identifier reply, which keeps AWS's response headers. An explicit route like converse/, an ARN, or an id LiteLLM recognizes keeps main's params, regexes and error status, so drop_params still drops tools there * fix(bedrock): judge lookaround support by model id on every route, ignore price-only rows The regex check now strips route and region prefixes from the model and its base_model before asking whether LiteLLM knows the model, so bedrock/converse/<id> gets the same answer in a direct transform call as on the completion path. A cost-map row counts only when it names a provider, so a price-only register_model row leaves an unlisted id unknown, as on the proxy. AmazonInvokeConfig now inherits the Invoke provider lookup instead of keeping an identical copy. * fix(bedrock): treat a region path like the bare id when the cost map does not list it A deployment like bedrock/us-east-1/<unlisted id> now gets the same tools, reasoning_effort and typo 404 as bedrock/<unlisted id>, so Claude Code works on it and a mistyped one still answers 404. * fix(bedrock): keep today's 404 for ids that reach Converse only by default, and keep an empty id local * fix(bedrock): answer 404 for an invalid id only when Converse rejected it, and keep a region path with no id local * fix(bedrock): keep a path-shaped id local, and map /v1/completions errors with the model's provider An id with a slash that is not an ARN, like bedrock/../../converse or an unknown region path, keeps Invoke and its local 404 instead of reaching AWS Converse. atext_completion mapped every error as OpenAI, so a Bedrock typo on /v1/completions answered 400 while chat answered 404. It now resolves the provider from the request's model, as the other async entry points do. * fix(text completion): keep /v1/completions error mapping as it is on main Mapping async /v1/completions errors with each provider's rules is wider than this Bedrock routing change. It moves classes, statuses, cooldown and fallbacks for most providers, and some lose context-window fallbacks. So this restores main's mapping here and leaves that change to its own PR. A Bedrock typo on /v1/completions answers AWS's 400, while chat keeps the 404. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 06:27:59 -07:00
zoveki	68b5a737a1	fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params (#49102) ... * fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params The native /v1/messages path sent tools with `strict` straight through, so when Claude's strict-tool limits rejected them with a 400 the caller got that 400 even with drop_params set. Under drop_params, a 400 that matches Claude's strict-limit phrases on a request with a strict tool is now sent once more with `strict` removed from every tool. Without drop_params nothing changes. The Claude strict-limit matcher moves from bedrock/common_utils.py to anthropic/common_utils.py so Bedrock converse and the base messages config share it, and the Rust bridge routes strict tools under drop_params to Python. * fix(anthropic): retry without strict only when a tool sends strict true A tool with only strict false no longer triggers a resend of an identical body, the warning no longer blames tool schemas when the rejected schema may be the output format, and requests with only additional_drop_params stay on the Rust messages route since the Python retry ignores them * fix(anthropic): keep sync messages calls with strict tools on Rust Python has no sync /v1/messages handler, so routing a sync call with a strict tool to Python raised ValueError where Rust served it before. The strict-tools detector now applies to async calls only. The retry warning also names the strict tool count without a plural. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 17:18:59 -07:00
tavren27	b25ea48c4b	fix(anthropic)!: drop beta-gated body fields under drop_params (#48793) ... * fix(anthropic): strip beta-gated body fields when a leg drops the anthropic-beta When a provider leg drops an anthropic-beta value (the chat handler's allowlist filter, /v1/messages' filtered headers, Bedrock Converse's additionalModelRequestFields), also drop the body fields that beta gates (thinking.block_binding, thinking.display=updates, mid-conversation system clear_at, output_config, tool_addition/tool_removal blocks) so the request degrades on that leg instead of failing with a provider 400. Copy-on-write so a fallback leg still sends the full request, and each strip is logged at WARNING. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove a leg that drops a beta sends neither the beta nor clear_at Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): strip beta-gated fields against the final body anthropic_beta list Converse counts the body anthropic_beta list as active, Invoke strips after its auto-injected betas are known and injects thinking-binding-controls for thinking.block_binding, test helpers are fully typed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated system fields before an older model's system turn becomes a user turn Models without supports_mid_conversation_system send no mid-conversation beta, so tool_addition and tool_removal blocks are dropped (and logged) before the turn is converted instead of riding along as user content. The Invoke output_config regression test reads the real cost map instead of patching _bedrock_model_supports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): pin the integration proxy to the bundled beta allowlist and type the helper locals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): annotate every new test local with Final and type the fixtures and returns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated body fields only under drop_params Without drop_params the request now goes out as sent and the provider answers its own 400, like any other unsupported param. inline-tools-2026-09-15 also unlocks tool_addition and tool_removal blocks, so a leg that carries it keeps them. * fix(anthropic): strip beta-gated fields on Foundry chat and Bedrock chat Invoke under drop_params Foundry chat and Bedrock chat Invoke now drop beta-gated body fields the same way the other legs do when drop_params is on and the request does not send the gating beta. The mid-conversation system normalizer validates only on the conversion path, and the strip WARNING names the beta the caller can send to keep the field. * fix(anthropic): send the thinking betas for block_binding and display updates, strip Databricks chat thinking Every Claude leg now adds the thinking-binding-controls beta when thinking carries block_binding and the display-updates beta for display updates, so those fields go through instead of a 400. Databricks chat rejects both fields with or without the beta, so it drops them under drop_params. A system turn the strip empties stays when it still carries output_config, and the drop warning names both causes. * fix(anthropic): send the thinking betas on every leg and keep fields where the beta is unknown Vertex messages and Claude Platform chat now add the thinking betas when thinking carries block_binding or display updates. Converse sends only the caller's header betas when it has them. A provider with no row in the beta config forwards beta-gated fields instead of dropping them, an emptied trailing system turn that would leave an assistant prefill is sent as is, and the prompt cache baseline body matches the body that goes out. The drop WARNING now names the beta that unlocks each field * fix(anthropic): filter Converse body betas, keep Foundry auto betas, read the bundled beta config for row checks * fix(anthropic): validate messages only when a beta-gated drop runs A non-object messages entry on a Claude model without mid-conversation system support raised a raw pydantic ValidationError before the beta-config and drop_params checks. The strip now passes the messages through untouched unless it drops something, so the provider's 400 reaches SDK callers as BadRequestError again * fix(anthropic): keep system blocks with a non-string type instead of raising --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 06:25:31 -07:00
tessellark	1a05aab72a	fix(bedrock)!: route region-prefixed embedding models to that region and bill its price (#49072) ... * fix(bedrock): send region-prefixed embedding model ids to that region without the prefix * chore(bedrock): hold the embedding fix back until the risk stamp * fix(bedrock): send region-prefixed embedding model ids to that region without the prefix * fix(bedrock): keep a model_id's region and price embeddings at the region the call goes to * fix(bedrock): read a model_id ARN's region for embeddings and keep unlisted regions mappable * fix(bedrock): route async_invoke embeddings whether the region comes before or after it * fix(bedrock): price embeddings by region_name and keep the path region with a plain model_id * fix(bedrock): price a model_id ARN at its region's row * fix(bedrock): keep chat's region rules for embeddings and leave ARN pricing as on main A plain model_id skips the region in the model, as chat does, so a deployment that worked on main keeps calling the same region. An empty aws_region_name fails region validation again instead of falling back. This reverts the ARN regional pricing, since many regional rows lack the bare row's priority, batch, and long-context rates --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 06:24:04 -07:00
Mateo Wang	0cf2bb7296	fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params (#47761) ... * fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params A GPT-5 request whose reasoning_effort the model map does not enable for that model lost the effort with no log line under drop_params. An Azure deployment name missing from the cost map lost reasoning_effort none this way and ran at the model's default effort. The gate now logs a warning naming the model, the level, the map entry it checked, and the model_info.base_model and model_info.reasoning_effort_levels fixes. * fix(openai): warn when the Responses route or a deployment's declared levels drop a reasoning effort The /v1/responses gate dropped a row-disabled GPT-5 effort, and reasoning for an OpenAI model without supports_reasoning, with no log line, and the router logged a declared-level drop only at INFO. Both Responses drops now log a WARNING and the router logs its drop at WARNING. The warning now leads with model_info.reasoning_effort_levels, since base_model only helps a deployment named differently from its underlying model. * fix: warn only when a reasoning level is actually dropped, and on generic reasoning_effort drops * fix: warn when thinking or a Bedrock Converse reasoning level is dropped, and name the router drop a reasoning level * fix: skip the drop warning when a non-reasoning model is asked for no reasoning, and honor declared levels on Bedrock Responses * fix: stay silent when reasoning is turned off on /v1/responses and warn on the Bedrock Responses fallback drop * fix(reasoning): warn when any reasoning-off request is dropped and give each route only the advice it honors A request to turn reasoning off on a model name the catalog lacks now warns on every route, since it may be a reasoning model. Chat suggests model_info.reasoning_effort_levels and, for a name with no catalog entry, model_info.base_model. OpenAI Responses suggests declared levels only where the config honors them, and Bedrock suggests neither. The Bedrock and Mantle Responses forwarding of declared levels is reverted until it can be proven live. * fix(reasoning): suggest reasoning_effort_levels and base_model only where the route reads them The chat gate suggested model_info.reasoning_effort_levels on routes that never pass declared levels (github_copilot, chatgpt and other OpenAIConfig delegates) and model_info.base_model on OpenAI and Azure AI chat, whose gate looks up the deployment name rather than base_model. /v1/responses/compact suggested levels it never reads either. Declared levels are now None when a caller passes none, so the advice appears only when a caller fed them, and the base_model advice is limited to the Azure chat gate, the one that looks up base_model. The openai row of the chat warning test now expects no base_model advice, the Responses warning test goes through get_optional_params_responses_api the way /v1/responses and compact call it, and the Azure direct-config test passes declared levels the way get_optional_params does. Long test lines are wrapped. * fix: read only the built-in catalog when choosing the reasoning drop warning	2026-10-29 04:46:29 -07:00
zoveki	310c96162b	fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893) ... * fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema * fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema. * fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them. * fix(bedrock): drop tool custom object on the Mantle count_tokens fallback Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way. * fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages * test(vertex): annotate count_tokens tool test locals with Final * fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names * fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:36:20 -07:00
rj-cairn	d06b889d41	fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755) ... * fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream The Gemini and Vertex AI streaming path read the SSE body with httpx iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085, U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds one of them was cut in two. The tail is not a data: line and the head never parses, and the accumulate-and-retry fallback then carries the unparsed prefix forward, so every later event is appended to it and the stream ends with no more content and no usage. SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines split decoded text on exactly those, treating a CR at the end of one chunk and an LF at the start of the next as one break. The first request and every long-decode hop now read through them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks. * fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator * fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed Switch the remaining httpx stream readers (Anthropic chat, Google GenAI, Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a raw U+2028 or U+2029 inside a JSON string no longer splits an event. Keep the chunk that closes a think block in the logged and cached response when a stream ends without a finish_reason, close the block before a legacy function_call, and make sync Codestral FIM streaming send stream=true and use the text-completion-codestral stream parser. Test fakes that only mocked iter_lines now stream real httpx bytes and assert the content they feed in. * fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use split_sse_lines from sse_lines.py, which also gains a str.split fast path. Adds tests/unit/llms/codestral to the unit workflow shard. * fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines * test: type the new stream tests, drop a comment, and benchmark the SSE line splitter * test: type the iterator class parameters in the line-separator stream tests * fix(responses): stream tool call arguments opened in a text chunk * fix(responses): hold a tool call that shares a chunk with text until the text ends * test: type the a2a SSE handler helper and the sandbox fake responses --------- Co-authored-by: Tim Vink <vinktim@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>	2026-10-28 17:00:29 -07:00
rj-cairn	f89b9c6150	refactor(types): replace Any with proven types in 28 files (#48903) ... * refactor(types): replace Any with proven types in 52 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): use typing_extensions.TypedDict for validated seams Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): revert strict Anthropic response validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): accept a null finish reason from Bedrock Cohere and TwelveLabs Pegasus * test(bedrock): annotate the Pegasus null finish reason test --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 16:12:10 -07:00
tessellark	43ce3e22aa	fix(responses)!: answer 400 before counting a lone Responses built-in tool on Claude (#48798) ... * test(responses): audit #47363 Adds integration cells for the input token count paths #47363 changed: the provider id of a managed file and the 403 for another team's file on OpenAI, file_search and additional_tools parity plus cache-mark stripping and untranslatable tools on Claude, and the Bedrock runtime count leaving out the options it refuses. * fix(count-tokens): reject a lone tool with no chat equivalent before the Claude count call A Claude input token count whose only tool was mcp or computer_use_preview skipped the pre-call check and went to the provider, so the caller got the provider error instead of the 400 naming the tool. The check now runs whenever a tool would get a placeholder name. * fix(responses): count a lone Anthropic toolset and name the uncountable tool The lone-tool 400 now fires only for OpenAI Responses built-in tool types, so a nameless Anthropic toolset still counts. The error says LiteLLM cannot count that tool type for the model. Adds a Bedrock unit cell for the lone-tool 400 and pins the runtime wire test's tool shape per request type. * fix(responses): take the uncountable tool types from the OpenAI SDK and keep the region hint off LiteLLM's 400 The Responses built-in tool types now come from openai's ToolParam union, so a type the SDK adds is refused before the count call too. The AWS region hint no longer lands on an error LiteLLM raised before calling the provider. Tests cover two more lone types and the hint. * fix(responses): keep LiteLLM's own list of uncountable tool types and pass a non-string tool type through * fix(responses): count tool_search and namespace tools the way Claude chat inference sends them The count refused a lone tool_search with LiteLLM's 400 and sent a lone namespace raw or squashed it into an empty tool, while Claude chat inference maps tool_search to Anthropic's tool search tool and unpacks a namespace into its function tools. Both now go through the deployment's own map_openai_params, so the count sends what inference sends, and tool_search leaves the list of uncountable types. * fix(bedrock): count a Claude request with its deployment's own chat translation Bedrock invoke and Mantle inference turn a chat tool_search tool into Bedrock's tool search tool, but the Bedrock counter always translated with the Anthropic config, so the count answered 400 for a request inference serves. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 12:12:45 -07:00
m6atlas	a0c490ea54	fix(bedrock): retry without strict when Bedrock rejects strict tool schemas under drop_params (#48733) ... * fix(bedrock): retry without strict when Bedrock rejects strict tool schemas under drop_params * test(bedrock): cover the strict tools 400 retry under drop_params * fix(bedrock): anchor the strict tools rejection match and strip toolSpec strict * test(bedrock): return captured requests from the strict tools test helper * fix(bedrock): retry without strict only when a tool sent strict to Bedrock A strict-limit 400 on a request whose tools carry strict: false, or on a model litellm never sends strict to, was sent again with an identical body. The fallback now takes the model and returns nothing unless a tool puts strict on the wire. Also types the fallback test parameter and records the mock transport's requests on the recorder instead of a mutated list * fix(bedrock): decide the strict retry on the profile's base model and retry grammar timeouts An application inference profile with base_model sends strict on the wire, so the retry check now resolves the capability model the way the request transformation does. Bedrock's "Grammar compilation timed out" 400 is a strict-grammar rejection too, so it takes the same single retry under drop_params. * test(bedrock): annotate the strict-tools retry test helper's values as Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 04:04:39 -07:00
Kerry Lu	0b66a27528	feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost (#48278) ... * feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bill native Bedrock guardrail trace usage on chat, responses and pass-through Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): record the Bedrock guardrail trace of a streamed metadata chunk on the logging object Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(bedrock): apply guardrailConfig on /v1/messages and price its trace, record the stream trace on both builder paths, redact the native guardrail_response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): validate the guardrail trace with one pydantic model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): record the native guardrail entry when litellm_params metadata is None Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): import the renamed messages invoke transformation module Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): skip native guardrail billing on cache hits and redact the trace with message logging off Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): walk the guardrail trace iteratively when redacting it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(bedrock): record the native guardrail trace where the Bedrock response is parsed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): type the native guardrail trace fixtures and locals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-27 12:10:45 -07:00
Mateo Wang	14193435b8	refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510) ... * refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages Move the unified /v1/messages code out of a directory called pass_through, since it is not the passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner model experimental_pass_through package becomes messages. The old packages stay as import shims that warn once and alias every submodule in sys.modules, so old deep imports still resolve Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): keep Final variables out of loops in the rename shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): match the warning text in the rename shim test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): resolve old pass_through imports lazily through a meta path finder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): mark the alias loader spec restore for the type discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): format the renamed package shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): keep the rebind-ok marker on the spec restore line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): carry the new package search path on alias specs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): declare the shim test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-27 10:08:25 -07:00
Mateo Wang	7e14638001	refactor(llms): rename model-version transformation files to family names (#48509) ... gpt_5_transformation.py becomes gpt_reasoning_transformation.py (OpenAI and Azure), the Bedrock anthropic_claude3_transformation.py files become anthropic_claude_transformation.py, anthropic_claude2_transformation.py becomes anthropic_legacy_text_completions_transformation.py, and vertex_ai_partner_models/llama3 becomes vertex_ai_partner_models/llama. Class and symbol names are unchanged. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-26 18:53:16 -07:00
Kerry Lu	560372456e	fix(bedrock): price claude_platform calls from their own cost map rows (#48150) ... * fix(bedrock): price claude_platform calls from the anthropic cost map rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: avoid recursive Claude Platform lookups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): price claude_platform calls from their own cost map rows Replace the claude_platform -> anthropic lookup redirect with bedrock/claude_platform/* rows, and apply the inference_geo multiplier in the bedrock cost path so US-only inference is priced at 1.1x Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(bedrock): exempt claude_platform rows from the Converse routing guard The claude_platform/ route is explicit in the row key, so these rows are not accidental non-Converse Bedrock chat rows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-24 13:54:18 -07:00
yucheng-berri	f7e6ddec66	feat(proxy): add LITELLM_OFFLINE_MODE, one switch for the fetches LiteLLM makes on its own (#48096) ... * feat(proxy): add LITELLM_OFFLINE_MODE to stop every request the proxy makes on its own * fix(ui): name LITELLM_OFFLINE_MODE where the cost map panel says local mode is forced * test: isolate the offline-mode tokenizer tests from the local Hugging Face cache * test: cover the offline policy templates and the cached Kimi K3 tokenizer * test(integration): count Kimi K3 tokens in the offline-mode egress test * refactor(token_counter): skip non-bundled tokenizers in offline mode instead of listing downloaded ones --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-23 16:54:19 -07:00
Mateo Wang	66c8f5b1a9	fix(llms): join api_base and route with one shared helper (#47535) ... * fix(llms): sanitize trailing slashes in provider get_complete_url * test(llms): move trailing-slash url tests into each provider's mapped unit file * fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI * fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders * fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests * fix(gigachat): strip a trailing slash from the pass-through api_base * fix(llms): join api_base and route with one shared helper A deployment api_base that ends in / made several get_complete_url builders emit a double slash, which Envoy and other gateways answer with 404. Add join_api_base in url_utils and use it in those builders, with a regression test that drives every get_complete_url under litellm/llms with a trailing-slash api_base * refactor(llms): move per-provider api_base strips onto join_api_base Use the shared join_api_base helper in the search, chat, embedding, image variation, GigaChat and VolcEngine URL builders instead of a per-provider rstrip and suffix check, so one helper owns the single-slash join * ci: run the trailing-slash URL builder test in the llms shard * test(integration): cover trailing-slash api_base across provider URL builders Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell * fix(llms): send every provider route through join_api_base Every place under litellm/llms and litellm/main.py that joined a caller-settable api_base with a route by concatenation or f-string now goes through join_api_base, so an api_base with a trailing slash sends the same upstream path as one without it Routes made only of a caller-supplied id, model, or passthrough path stay on a plain join over an already normalized base, because the helper's suffix dedup would turn a lookup by id into a list or create call The sweep test now also reaches the factory-made JSON provider builders, and a new loopback test drives the public entrypoints with an api_base ending in /gw/ and checks that every request path keeps /gw/ with no double slash * fix(llms): keep the route skip only where a builder already had it join_api_base is a pure join again. The new ensure_api_route skips the route only when the URL path already ends in it, and it is used only where the merge base or the trailing-slash test PR head already skipped. Evals cancel, Predibase, and Azure AI threads append their route again, EdenAI model listing and Gemini RAG ingestion join through join_api_base, and Azure AI Anthropic checks the path so a host named anthropic-* builds /anthropic/v1/messages * fix(azure_ai): read /anthropic from the URL path when building the count_tokens URL * test(llms): run the Gemini RAG trailing-slash case through aingest The sync ingest wrapper needs a current event loop, which an earlier async test in the same worker clears * test(llms): cover the Azure realtime, Bedrock rerank, and oobabooga joins with a trailing-slash api_base * test(integration): cover api_base joins across providers, query strings and named hosts * feat(url_utils): add join_api_base and ensure_api_route helpers * fix(url_utils): keep the scheme slashes when a stripped suffix was the whole host join_api_base and ensure_api_route now keep the // of a base left with only a scheme, so http://v1 with /v1 stripped joins to http://v1/files as before. The Azure v1 client base_url goes through ensure_api_route so a base ending in /openai/v1 no longer doubles it * fix(azure_ai): build the count_tokens URL from the messages URL * fix(azure_ai): match anthropic as a whole path segment in the messages URL * fix(llms): keep main's Azure AI anthropic match and non-string Bedrock endpoints Azure AI chat and messages go back to cutting the base at the first /anthropic, as on main, and count_tokens keeps its path check, so a gateway path like /anthropic-proxy builds the same URLs as main. A base that already ends in /v1/messages still gets count_tokens next to it. A non-string aws_bedrock_runtime_endpoint (an httpx.URL VPC endpoint) is used as its text again in AgentCore, Bedrock responses and the RAGFlow base, instead of falling back to the default host. get_volcengine_base_url and GigaChat's get_api_base strip a trailing slash again for outside callers. The cache-hit wire test is skipped as a known proxy bug, since one cache hit writes two cache_hit spend rows. * fix(llms): drop a slash before the api_base query and tighten the route test checks * test(llms): annotate new test locals as Final and flatten the builder generator * fix(url_utils): keep a scheme-only api_base hostless so it fails before sending --------- Co-authored-by: Raushankumar0720 <raushan.singh.cg@gmail.com>	2026-10-21 06:59:32 +00:00