mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
115 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sameer Kankute
|
816fca939f
|
chore(oss): litellm oss staging 150626 (#30463)
* fix(pricing): add GitHub Copilot MAI Code Flash pricing (#30415) * fix(pricing): add GitHub Copilot MAI Code Flash pricing Add GitHub Copilot pricing entries for MAI-Code-1-Flash and the internal Copilot CLI model name so cost calculation can price input, cached input, and output tokens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(pricing): cover GitHub Copilot MAI Code Flash pricing Add regression coverage for both GitHub Copilot MAI-Code-1-Flash model names, including cached input pricing, chat endpoint metadata, and cost_per_token arithmetic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) (#30213) * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) #28990 added ownership recording for streaming /v1/responses via _wrap_responses_stream_for_container_ownership, which reads `getattr(stream_response, 'completed_response', None)` to extract the ResponsesAPIResponse. The unit test bypassed the Router, so it never exercised the production wrapping path. Through the Router (every proxy deployment), the stream is wrapped by FallbackResponsesStreamWrapper (router.py:2527). Its __init__ set `self.completed_response = None` and __anext__ only forwarded chunks — the inner source iterator's terminal event never bubbled up to the attribute the ownership hook reads, so the hook silently recorded nothing and every follow-up /v1/containers/<id>/files call returned 403 for non-admin keys. This commit: - router.py: pre-resolves the responses-API terminal event tuple (response.completed / .incomplete / .failed) once per _aresponses_streaming_iterator call, and has the wrapper's __anext__ sniff each forwarded chunk's .type. First terminal event hit gets stored on the wrapper's completed_response. Iterator-agnostic — works for source_iterator AND any future wrapper. - common_request_processing.py: when _extract_completed_responses_response returns None we now warn instead of silently skipping. Reporter on #30210 lost a day to this exact silent skip; the warning surfaces future regressions of the same shape directly in operator logs. Fixes #30210 * fix(router): type-ignore wrapper getattr-defaults; broaden ownership-skip warning CI lint (mypy) flagged the three pre-existing getattr(..., None) assignments in FallbackResponsesStreamWrapper.__init__: router.py:2564 self.response = getattr(source_iterator, 'response', None) router.py:2565 self.model = getattr(source_iterator, 'model', None) router.py:2566 self.logging_obj = getattr(..., None) Those lines also exist on litellm_internal_staging and pass mypy there. Adding the typed terminal-event tuple above the class made the function body more narrowable, which surfaced the pre-existing mismatch — base class declares non-Optional types but the bridge path (LiteLLMCompletionStreamingIterator) legitimately omits these. Keep the None fallback and silence with type: ignore[assignment]. Greptile 4/5 note: the ownership-skip warning hard-named code_interpreter which misleads operators when a non-code_interpreter stream aborts. Generalize to 'any tool container (e.g. code_interpreter)'. * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) (#30201) * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) get_model_info synthesizes input_cost_per_token / output_cost_per_token = 0 when they are absent from the raw entry (the price-unknown and free cases share the same representation). register_model then merges that result back into litellm.model_cost, which flips a sparse entry from 'no cost keys' (priced via model name) to 'cost keys = 0' (free). That defeats _is_cost_explicitly_configured (#24949) on re-registration: _is_model_cost_zero returns True, common_checks skips every tag / key / team / user / org budget check for the group, and over-budget traffic keeps returning 200. Spend keeps recording because cost calc still resolves by model name, so the symptom is silent and only triggers on the second register_model pass (router rebuild, /model/update, config sync). Mirror the existing litellm_provider-None guard one block above and pop the cost fields from the synthesized result when they are absent from the raw entry and not in the caller's value. Caller-provided zeros (genuinely free models, BYOK overrides) are preserved. Fixes #30198 * fix(register_model): switch _raw_entry to is-None checks + drop dead test assertion Greptile #30201 review notes: - the `or`-chain in the raw-entry lookup treated an empty dict (a key with no fields) as falsy and fell through to the second arm — replace with explicit `is None` checks so a present-but-empty entry is still taken at face value. - the first assertion in `test_router_double_init_keeps_db_model_entry_sparse` used `in (None, 0)` which passes under the bug condition (cost = 0 matches the tuple); the strong follow-up assertion already covers every shape, so drop the dead branch. * fix(bedrock mantle): use unique function-call id for responses->chat tool calls (#30426) * fix(bedrock mantle): use unique function-call id for responses->chat tool calls ... * fix(bedrock mantle): scope unique tool-call id fallback to degenerate call_id The previous revision preferred the Responses item id for every tool call, which broke providers (and existing tests) where call_id is a unique, canonical correlation key. Restrict the fallback to the degenerate index-based call_id that Bedrock Mantle returns (call_0, call_1, ... resetting per response) and keep call_id otherwise. Revert the change to the OUTPUT_ITEM_DONE streaming handler, whose tool_call_chunk is never emitted (dead code, per review). Extend the regression tests to assert a normal call_id is preserved. * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) (#30241) * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) Router.get_deployment_credentials_with_provider re-validates a deployment's litellm_params through CredentialLiteLLMParams before handing them to file/batch/passthrough callers: return CredentialLiteLLMParams( **deployment.litellm_params.model_dump(exclude_none=True) ).model_dump(exclude_none=True) Any field NOT declared on CredentialLiteLLMParams gets silently dropped on the way through. azure_ad_token was undeclared, so Azure deployments using OAuth/M2M (azure_ad_token instead of a static api_key) silently lost their token at the files endpoint and the proxy returned: Missing credentials. Please pass one of api_key, azure_ad_token, azure_ad_token_provider, ... Declare azure_ad_token on CredentialLiteLLMParams alongside api_key / api_base / api_version so it rides through the round-trip. Static-key deployments stay unaffected (Optional, default None, dropped by exclude_none=True). Provider-callable (azure_ad_token_provider) is a separate concern and out of scope here. Fixes #30235 * fix(ui-types): regenerate schema.d.ts for new azure_ad_token field CI's 'Verify schema.d.ts matches the proxy OpenAPI spec' check auto-detected the new field and emitted the exact diff to apply. Two schemas had `aws_secret_access_key` from CredentialLiteLLMParams, both get the new azure_ad_token marker next to it. * fix(proxy): org_admin with own user_id now sees all org teams on /v2/team/list (#30247) When the UI sends the callers own user_id (as it does for non-Admin global roles), _enforce_list_team_v2_access now nulls it out for org admins so _build_team_list_where_conditions scopes by organization_id only -- matching the legacy /team/list behavior and the documented intent. Fixes #30215 Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * test(vertex_ai): multi-region regression coverage for cachedContents host (#29571) (#29707) litellm_internal_staging already routes the cachedContents URL through get_vertex_base_url, fixing the multi-region 404 reported in #29571 — but carries no test coverage for the actual regression scenario (eu/us must resolve to the REP host aiplatform.{geo}.rep.googleapis.com). Add TestContextCachingMultiRegionUrls: parametrized eu/us REP-host assertions (including absence of the old broken {geo}-aiplatform host), plus regional (us-central1) and global no-regression checks. * fix(proxy): close upstream LLM stream when client disconnects mid-stream (#30245) * fix(proxy): close upstream LLM stream when client disconnects mid-stream When a streaming client disconnects, Starlette abandons the response body iterator without calling aclose(), so the proxy's connection to the upstream backend stays open until garbage collection, which may never come. The backend (e.g. vLLM) keeps generating into a dead pipe: small responses drain invisibly into TCP buffers while large ones block the backend on a full send buffer indefinitely (observed via lsof as an ESTABLISHED proxy->backend connection minutes after the client left) create_response now returns a StreamingResponse subclass that closes both its body iterator and the wrapped upstream-facing generator in a shielded finally. The upstream generator is closed directly rather than through a cascade because aclose() on a never-started generator skips its body, which would make the cascade a no-op when the client disconnects before the first chunk is sent. async_streaming_data_generator also gains the same shielded finally-aclose that async_data_generator in proxy_server.py already had, covering the Anthropic and Google SSE paths With this, killing a streaming client causes the backend to observe the abort within about a second and free its slot, while completed streams are unaffected. No flag is needed, unlike the non-streaming opt-in cancel in #30223: this only releases resources after the client is already gone and does not change any response a client can observe Fixes #30244 * fix(proxy): close upstream even when body iterator aclose raises BaseException Addresses the Greptile finding on #30245: the cleanup loop caught only Exception while the generator-level cleanup catches BaseException, so a CancelledError or GeneratorExit escaping body_iterator.aclose() would skip closing the upstream generator. Both sites now use the same scope and a regression test pins that the upstream is closed even when the body iterator explodes with a BaseException * fix(llms): expose aclose on BaseModelResponseIterator so stream close reaches the provider connection The response-level close added for #30244 only worked for SDK-based providers (e.g. openai), whose streams expose aclose all the way down. Providers served by base_llm_http_handler (hosted_vllm and most modern transformation-based providers) wrap a bare response.aiter_lines() generator in BaseModelResponseIterator, which had no aclose or close at all, and nothing retained the httpx response object; so CustomStreamWrapper.aclose() silently did nothing and the upstream connection stayed open. Verified with a vLLM-style mock: with hosted_vllm/ the backend streamed all 100 chunks to completion after the client disconnected, while openai/ aborted at chunk 6 BaseModelResponseIterator now carries an optional http_response and an aclose() that closes it; make_async_call_stream_helper attaches the response after building the iterator. With this, hosted_vllm aborts the backend within ~1.6s of the client dropping, and completed streams are unaffected --------- Co-authored-by: kursad <kursad.lacin@brado.net> * feat(anthropic): surface compaction usage iterations data (#27065) * feat(anthropic): surface compaction usage iterations data * style: apply black formatting to fix lint checks * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock (#30422) * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock * fix(usage): optimize test imports * feat: add fastCRW search provider (#30434) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider (#30203) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider * libertai: update served endpoints backup + add mode/matrix tests Addresses review feedback: - Add libertai to litellm/provider_endpoints_support_backup.json, the file actually served by GET /public/supported_endpoints (the root provider_endpoints_support.json already had it). - Add tests asserting bge-m3 normalizes to mode='embedding' and that the served matrix lists libertai. embeddings stays false: the JSON-configured provider path only wires chat routing (OpenAILike embedding handler is reached only for literal openai_like/llamafile/lm_studio), matching the llamagate precedent; bge-m3 remains in the cost map for metadata. --------- Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com> * feat(provider): add ModelScope as an OpenAI-compatible provider (#28460) * add ModelScope API support * add modelscope api support * update modelscope model list * add image-genetation support * update test and multimodal * fix: address PR review feedback for modelscope provider * update README * fix(customer_endpoints): restrict /customer/daily/activity to admin-only (#28849) * fix(customer_endpoints): restrict /customer/daily/activity to admin-only * fix(customer_endpoints): check role before prisma_client guard * fix(custom_guardrail): key disable_global_guardrails takes precedence over team guardrail list (#28563) * fix(fallbacks): preserve fallback model in SDK fallback responses (#28260) * fix(fallbacks): preserve fallback model in response when using SDK-level fallbacks * fix(fallbacks): gate x-litellm-* passthrough to trusted callers only The previous patch unconditionally let `x-litellm-*` keys bypass the `llm_provider-` prefix in `process_response_headers`. That function is also called on raw upstream-provider response headers (e.g. from `llm_http_handler.py`), so a malicious provider could return `x-litellm-attempted-fallbacks` and spoof a LiteLLM-internal marker, bypassing the proxy model-override guard. Add a `preserve_litellm_internal_headers` flag (default False). Only `response_metadata.py`, which re-processes the already-built `_hidden_params["additional_headers"]` dict (LiteLLM-owned), passes True. Raw provider header callsites keep the default False, so upstream `x-litellm-*` still gets the `llm_provider-` prefix. Adds a regression test for the spoofing case and renames the existing preserve test to make the trusted-path semantics explicit. * fix(fallbacks): ignore preserve_litellm_internal_headers for raw httpx.Headers inputs * style(core_helpers): apply black formatting * fix(lint): remove banned typing.List/Dict/Any imports and suppress PLR0913 on interface overrides Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): apply black formatting to modelscope chat transformation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): replace noqa with proper fixes — use **kwargs and Awaitable instead of Any/List Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): remove unused AllMessageValues import Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert: restore base_model_iterator.py to original PR state Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): restore full method signatures for MyPy compatibility; bump PLR0913 budget for new provider files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): use @override to suppress PLR0913 on inherited signatures instead of bumping budget The overrides keep their full base-class signatures for MyPy compatibility, but those signatures carry more than five parameters, which tripped PLR0913 on each subclass redeclaration. Since the arity is dictated by the base class and cannot be reduced, decorate the overrides with typing_extensions.override; ruff treats that as the intended signal that the parameter count is not under the author's control and skips PLR0913. This restores the PLR0913 baseline to 1813. * fix(lint): add @override to modelscope image generation overrides Apply the same typing_extensions.override treatment to the image generation config so its inherited-signature overrides do not count against PLR0913. --------- Co-authored-by: Joel Tony <github@jaytau.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: ztko <96878659+koztkozt@users.noreply.github.com> Co-authored-by: Nahrin <nahrin@nahrinoda.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Humphrey <a739376838@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: Dushyant Acharya <dushyantacharya873@gmail.com> Co-authored-by: Yuriy <yuriy.shuyskiy@gmail.com> Co-authored-by: Recep S <22618852+us@users.noreply.github.com> Co-authored-by: Moshe Malawach <moshe.malawach@protonmail.com> Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com> Co-authored-by: Rongkun Yan <2493404415@qq.com> Co-authored-by: Varshith <kvarshithgowda@gmail.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> |
||
|
Sameer Kankute
|
ccc20b121f
|
fix(responses): presidio PII masking for Azure WebSocket and streaming (#30003)
* fix(responses): Presidio PII masking for Azure WebSocket and streaming
Wire Presidio into native Responses WebSocket forwarding and fix streaming output unmasking so masked tokens are restored for HTTP and WS clients.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix unused imports in responses handlers
* fix(responses): address Greptile review - Azure WebSocket model URL and PII logging
- Add model_in_websocket_url() to BaseResponsesAPIConfig (default True) so
providers can opt out of ?model= being appended to WebSocket URLs.
- Override model_in_websocket_url() to return False for Azure, since Azure
sends the model in the response.create body, not the URL query string.
- Use this flag in llm_http_handler to conditionally append ?model=.
- Pass masked message to _store_input() instead of the original PII-containing
message so logging destinations do not receive unmasked PII.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): mask nested response.create input format for Presidio PII
Handle the nested {"type":"response.create","response":{"input":[...]}}
format in _mask_response_create. Previously only the flat top-level input
was masked; the nested shape bypassed Presidio and forwarded raw PII
upstream. Now both shapes are normalized and masked before forwarding.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: apply black formatting to llm_http_handler and streaming_iterator
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: suppress PLR0915 on async_responses_websocket
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): add apply_to_output masking on Responses API WebSocket path
Previously the WebSocket guardrail filter excluded callbacks with
apply_to_output=True, leaving model-generated PII unmasked before
returning to the client.
- Collect apply_to_output callbacks separately in llm_http_handler and
pass them to ResponsesWebSocketStreaming as output_guardrail_callbacks.
- Add _mask_response_completed method that calls check_pii(output_parse_pii=False)
on text blocks in response.completed events, masking model output PII.
- backend_to_client now chains unmask (pii_tokens) → mask (apply_to_output)
before forwarding each event to the client.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): unmask PII tokens in streaming delta events and warn on guardrail init failure
- Rename _unmask_response_completed -> _unmask_response_event and extend
it to also unmask response.output_text.delta (and other delta types)
so real-time streaming clients receive original values, not PII tokens.
- Split the broad except-and-swallow into ImportError (expected in SDK-only
environments) vs Exception (unexpected — now logs a warning so operators
know masking is disabled).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): enforce authorized model on WebSocket frames and remove proxy import
Security: add _enforce_authorized_model to ResponsesWebSocketStreaming that
overwrites both flat and nested model fields in every response.create frame
with the connection-authorized model, preventing deployment-substitution
attacks where an authenticated user sends a different model name in the frame
body after connecting with an allowed model.
Layering: remove the _OPTIONAL_PresidioPIIMasking isinstance check and proxy
import from the SDK handler. Use duck-typed checks (callable check_pii +
get_presidio_settings_from_request_data) so any guardrail implementing the
interface works, not just Presidio.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): add _unmask_pii_text to duck-typed contract and mask delta frames
- Add callable(_unmask_pii_text) check to the guardrail_callbacks filter so
a custom guardrail missing that method cannot cause an AttributeError and
silently kill the WebSocket session.
- Extend _mask_response_completed to also mask response.output_text.delta
(and other delta types) for apply_to_output callbacks, so real-time
streaming clients do not receive unredacted model-generated PII in deltas.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix Responses WebSocket guardrail edge cases
* fix(responses): log masked output and suppress deltas when apply_to_output active
- Move _store_event to after _mask_response_completed so logs receive the
redacted form, not raw model output containing PII.
- Suppress delta event forwarding when output_guardrail_callbacks are
present: per-fragment Presidio cannot catch PII that spans multiple
chunks (e.g. "alice@" + "example.com"). Clients receive only the
fully-masked response.completed, which Presidio scans on complete text.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): mask and suppress response.output_item.done for apply_to_output
response.output_item.done carries completed item text in item.content[*].text
before response.completed arrives, allowing unmasked PII to reach the client.
- _unmask_response_event: unmask input-PII tokens in item.content[*].text
- _mask_response_completed: run check_pii on item.content[*].text for
apply_to_output callbacks (same as response.completed handling)
- backend_to_client suppression: also skip response.output_item.done when
output_guardrail_callbacks are active; client receives only the
fully-masked response.completed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(types): cast response_obj to ResponsesAPIResponse to satisfy mypy
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Revert "fix(types): cast response_obj to ResponsesAPIResponse to satisfy mypy"
This reverts commit d5969557628f9aff58948b9d37cc64d577f95a15.
* Revert "fix(responses): mask and suppress response.output_item.done for apply_to_output"
This reverts commit 219fd54ea3446f4399fde40c07ba0617e2834573.
* fix(types): accept dict responses in guardrail output write-back
Streaming response.completed events pass a dict response object, so widen
_apply_guardrail_responses_to_output to match its existing runtime handling.
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(responses): skip Presidio masking on suppressed WebSocket delta events
Delta events are dropped wholesale when apply_to_output masking is active,
so masking them first issued a wasted check_pii call per fragment. Move the
suppression check ahead of the unmask/mask passes; the event type is
invariant across both, so client-visible behavior is unchanged.
* test(responses): cover Responses WebSocket PII masking hooks
Add regression tests for the native Responses WebSocket guardrail path:
input masking and model enforcement in _mask_response_create, token
unmasking in _unmask_response_event, apply_to_output masking and delta
suppression in _mask_response_completed/backend_to_client, and the
get_websocket_url / model_in_websocket_url defaults for the base and
Azure configs. Raises diff coverage above the codecov patch target.
* fix(responses): suppress text-bearing done events under output PII masking
When apply_to_output masking is active on a native Responses WebSocket,
response.output_text.done, response.content_part.done, and
response.output_item.done carry the full model output before the masked
response.completed arrives, so an authenticated client could read
unmasked PII from those events. Suppress them alongside delta events; the
client receives only the fully-masked response.completed.
* refactor(responses): drop dead delta branch in WebSocket output masking
Delta events are suppressed in backend_to_client before _mask_response_completed
runs when output masking is active, so the method's delta-handling branch was
unreachable. Restrict it to response.completed and cover the Responses API
unmask path with a Pydantic ResponseCompletedEvent regression test.
* fix(presidio): flush buffered chat chunks on mixed unmask stream
_stream_pii_unmasking buffered ModelResponseStream chunks but returned
early once a /v1/responses event was seen, silently dropping the buffered
chat chunks. Flush them in order before switching to passthrough, mirroring
_stream_apply_output_masking, and cover it with a regression test.
* fix(responses): mask instructions and tool-call arguments in WebSocket PII path
Presidio masking on the native Responses WebSocket path left two gaps. On the
request side _mask_response_create only walked the input containers, so PII
placed in the instructions field of a response.create frame was forwarded
upstream and logged unmasked even with output_parse_pii enabled. Now both the
flat and nested instructions strings are masked alongside input.
On the response side _mask_response_completed only masked content text blocks,
so model-produced PII inside function-call arguments could reach the client when
apply_to_output was enabled, both via the standalone
response.function_call_arguments.done event and via the function_call output
items in response.completed. The done event is now suppressed under output
masking and completed function-call arguments are run through check_pii before
forwarding or logging.
* fix(responses): suppress reasoning_summary_text.done under output PII masking
* fix(responses): mask function_call_output.output in WebSocket PII path
response.create input items of type function_call_output carry
user-controlled text in output, not content, so the Presidio masking
pass forwarded that text upstream unmasked. Mask the output field
(string or list of text blocks) alongside content.
* fix(responses): mask reasoning summary PII in WebSocket output path
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Sameer Kankute
|
680f3ff810
|
fix: bedrock mantle fixes (#30083)
* fix: respect aws region * Fix chat completion to responses bridge * Handle response streaming events * Fix bedrock mantle region priority and CI test failures. aws_region_name now overrides BEDROCK_MANTLE_REGION env, and provider tests pass region via litellm_params. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(bedrock_mantle): use warmup id prefix constant and log untyped tool drops * refactor(bedrock_mantle): keep aws_region_name extraction inside chat config * fix(bedrock_mantle): validate aws_region_name before host interpolation The client-supplied aws_region_name flows unvalidated into the Bedrock Mantle host (https://bedrock-mantle.{region}.api.aws), so a value containing a slash could redirect the request, along with the configured bearer API key, to an arbitrary host. Validate the region against the AWS region format in both the chat and responses transformations before it is interpolated. * fix(bedrock_mantle): close AWS_REGION_NAME chat gap and surface dropped tools Chat region resolution now consults AWS_REGION_NAME, matching the responses path precedence. Unsupported Responses tools dropped by map_openai_params are logged at warning level so the loss is visible in production. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Sameer Kankute
|
32c88ca74f
|
Litellm oss staging 080626 (#29932)
* feat(bedrock_mantle): add SigV4/IAM auth to Responses API route (fixes #29665) (#29788) * feat(responses): add default no-op sign_request to BaseResponsesAPIConfig * feat(responses): call sign_request after body is final, send signed bytes when signed * feat(bedrock_mantle): add SigV4 sign_request via composed BaseAWSLLM (bearer path) * test(bedrock_mantle): cover SigV4 access-key, AssumeRole, body bytes, region/auth consistency * feat(bedrock_mantle): defer auth to sign_request; validate_environment no longer requires bearer * docs(bedrock_mantle): document SigV4 + Bearer auth on Responses route * test(responses): cover fake-stream signing order and mantle bearer arg/env precedence * fix(bedrock_mantle): wrap all botocore credential errors with both-paths guidance * fix(bedrock_mantle): catch specific credential errors, not all BotoCoreError, so STS transport failures are not masked * fix(bedrock_mantle): sign the compact Responses route too, not just create * fix(github-copilot): route per-model on /v1/responses based on model info (#29747) * feat(focus): add GCS destination for FOCUS export (#29751) * test: add failing tests for FocusGCSDestination * feat: add FocusGCSDestination reusing GCSBucketBase auth * feat: register FocusGCSDestination in factory; export from __init__ * fix(focus): preserve GCS_PATH_SERVICE_ACCOUNT when service_account_json not in config * style: apply Black formatting to gcs_destination and tests * style: apply Black formatting to factory.py * fix(bedrock): omit empty additionalModelRequestFields and system from Converse API payload (#29565) Amazon Nova Pro (and other strict Bedrock models) return 400 Malformed input request when additionalModelRequestFields: {} or system: [] are present in the payload. Both fields are optional in CommonRequestObject (total=False) and must be omitted rather than sent as empty structures. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible in pass-through cost tracking (#29730) * fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible Azure OpenAI resources created via the newer "Azure AI Foundry" / Cognitive Services pathway live on `*.cognitiveservices.azure.com` subdomains, not the older `openai.azure.com`. Both are valid Azure OpenAI surfaces in production today. The OpenAI pass-through cost-tracking handler hard-codes only the older hostname in five places (four `is_openai_*_route` methods on OpenAIPassthroughLoggingHandler, plus is_openai_route on PassThroughEndpointLogging). As a result, calls from newer Azure deployments are silently classified as "not an OpenAI route", the dispatch into the cost-tracking handler is skipped, and tokens/cost never get extracted into LiteLLM_SpendLogs — the row gets written with prompt_tokens=0, completion_tokens=0, spend=0, model='unknown'. Reproduced 2026-06-04 against a real Azure OpenAI deployment on `*.cognitiveservices.azure.com` proxied through LiteLLM v1.88.0. Fix: factor the hostname check into a single helper `_is_openai_compatible_host` listing all three recognized surfaces (api.openai.com, openai.azure.com, cognitiveservices.azure.com), and have all five call sites delegate to it. Purely additive — never weakens recognition for the originally-supported hostnames. Adds a test `test_is_openai_route_recognizes_cognitiveservices_azure_com` that exercises all four `is_openai_*_route` static methods against `*.cognitiveservices.azure.com` URLs (positive cases per route + a small cross-route negative to confirm route-specific path matching still works on the new hostname). Out of scope for this PR (separate followup): - `openai_passthrough_handler` calls chat/completions `transform_response` on Responses API payloads (`output:` not `choices:`), which throws inside the dispatch and drops the SpendLogs row entirely. Recognized + tracked separately. * ci: trigger fresh run Empty commit to re-run checks. The previous auth-and-jwt failure was a transient HuggingFace Hub 429 rate-limit hitting tokenizer downloads in tests/proxy_unit_tests/test_custom_tokenizer_bug.py — unrelated to this PR's scope (hostname recognition in pass-through cost tracking). No code change. --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(responses): preserve forced-function tool_choice name in Responses to Chat transform (#29812) The Responses API forces a specific function with a top-level name ({"type": "function", "name": "X"}), but _transform_tool_choice only handled the nested Chat Completions shape and fell through to returning "required" for the flat form, silently dropping the function name and degrading a forced function call to force-any-tool. Map the flat Responses shape to the nested Chat shape, keeping the "required" fallback when no name is present. * Preserve x-anthropic-billing-header system blocks for first-party Anthropic (#29584) * Preserve x-anthropic-billing-header system blocks for first-party Anthropic PR #20951 strips system blocks beginning with "x-anthropic-billing-header:" for every Anthropic target. That block is how the first-party Anthropic API recognizes Claude Code subscription (OAuth) traffic, so dropping it makes requests that carry only that block, such as the auto-mode tool-safety classifier, fail with a misleading 429 rate_limit_error; normal turns still work because they also carry the "You are Claude Code" identity block. Gate the strip behind should_strip_billing_metadata(), defaulting to False on the first-party AnthropicConfig and AnthropicMessagesConfig so the block is kept, and overridden to True on the providers that reach these transforms and reject the block (Bedrock platform, Vertex, Azure for the chat path; Minimax, Azure, DeepSeek for the messages path). Behavior for those providers is unchanged. * Strip billing header on Bedrock invoke and Vertex messages pass-through Two more subclasses reach the gated strip but inherited keep-by-default. AmazonAnthropicClaudeConfig (Bedrock invoke) calls AnthropicConfig.transform_request, which calls translate_system_message, and VertexAIPartnerModelsAnthropicMessagesConfig (Vertex messages pass-through) calls super().transform_anthropic_messages_request. Override should_strip_billing_metadata() to True on both. Add a parametrized test asserting the flag for every first-party base (False) and provider subclass (True), covering all overrides, plus a translate_system_message regression test for the Bedrock invoke path. * fix(cache): log hashed cache keys (#29890) * fix(ui): save routing groups as list (#29889) * Revert "fix(ui): save routing groups as list (#29889)" (#29928) This reverts commit |
||
|
Mateo Wang
|
ffd0e9fa7f
|
[internal copy of #27491] fix(realtime): Fix Realtime Audio Token Cost Tracking (#29722)
* Normalize Realtime usage dict keys before ResponseAPIUsage transform * Test usage transform for Realtime versus tokens_details keys * Avoid usage_input dict in-place * Fix audio cost calculation * fix(responses): forward output audio_tokens into completion usage details Pass audio_tokens from output_tokens_details into CompletionTokensDetailsWrapper so cost can use output_cost_per_audio_token. Support dict output details like prompt path. Extend tests for Realtime and mixed completion audio. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix audio token usage formatting * style: Black-format Realtime usage and completion usage merge Resolve combine_usage_objects and responses/utils wrapping for CI black --check. Restore model_fields comments above completion_tokens_details merge loop. Co-authored-by: Cursor <cursoragent@cursor.com> * Add test to cover combined usage objects * Fix merge conflict with test cases Removed unnecessary import statement and cleaned up assertions in test. * fix(cost_calculator): remove dead None guard in completion_tokens_details combiner --------- Co-authored-by: Liam McDonald <lmcdonald@godaddy.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Sameer Kankute
|
b7e978a5c3
|
Litellm oss staging 04 21 2026 2 (#26569)
* fix(bedrock): use model info lookup for output_config support instead of hardcoded check Replace hardcoded _is_claude_4_6_model() string matching with supports_output_config flag in model_prices_and_context_window.json, accessed via _supports_factory(). This follows the project's established pattern for model capability checks (per AGENTS.md rule #8). Bedrock Invoke now conditionally preserves output_config for models that declare supports_output_config=true (currently Claude 4.6 models), while stripping it for older models to avoid request rejection. Ref: https://github.com/BerriAI/litellm/issues/22797 * fix(vertex_ai): single-flight credential refresh to prevent thundering herd (#26024) * fix(vertex_ai): single-flight credential refresh to prevent thundering herd When GCP credentials expire under high concurrency, all requests simultaneously call credentials.refresh() via asyncify, saturating the 40-thread anyio pool and blocking the proxy for 20+ seconds. This adds: - Per-credential asyncio.Lock in get_access_token_async for single-flight refresh (1 coroutine refreshes, others wait on the lock) - Background refresh when token_state is STALE (usable but near expiry), returning the current token immediately with zero added latency - threading.Lock on the sync get_access_token path - Uses google-auth's TokenState enum (FRESH/STALE/INVALID) instead of reimplementing expiry logic Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review comments - Use asyncio.create_task() instead of deprecated get_event_loop().create_task() - Track in-flight background refresh tasks to prevent duplicate refreshes when multiple STALE-path callers pass through the lock before the first background task completes - Add token validation in the STALE branch (consistent with FRESH/INVALID) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: lazy-import TokenState to avoid breaking when google-auth is not installed Also extract helper methods to bring get_access_token_async under the PLR0915 statement limit (50). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: apply Black formatting to test file and update uv.lock Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove user-provided project_id from log messages (CodeQL log injection) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: avoid leaking token value in error message, log type instead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: restore uv.lock to match litellm_oss_branch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove project_id from remaining log message (CodeQL log injection) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove remaining project_id from log and error messages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: reuse cached credentials in VertexAIPartnerModels (#26065) * fix: reuse cached credentials in VertexAIPartnerModels instead of creating new VertexLLM per request VertexAIPartnerModels.completion() was creating a throwaway VertexLLM() instance on every call to get an access token, bypassing the credential cache inherited from VertexBase. This caused a fresh token fetch for every single request, adding significant latency overhead. Fix: call super().__init__() to initialize VertexBase's credential cache, and use self._ensure_access_token() instead of a new VertexLLM instance. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: apply same credential caching fix to VertexAIGemmaModels and VertexAIModelGardenModels Same bug as VertexAIPartnerModels: both classes had `pass` in __init__ instead of `super().__init__()`, and created throwaway VertexLLM() instances per request instead of using self._ensure_access_token(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(fireworks): add glm-5p1 metadata and parallel_tool_calls (#26069) * fix(chatgpt): preserve responses routing and recover empty output (#25403) (#26219) - preserve existing shared backend `mode` when router deployment registration reuses a provider/model key already in `litellm.model_cost` (prevents alias with `mode: chat` from downgrading shared `chatgpt/gpt-5.4` from `responses` to `chat` and triggering 403s on /v1/chat/completions) - teach the ChatGPT Responses parser to recover `response.output_item.done` entries when `response.completed.output` is empty - add defensive /responses -> /chat/completions bridge fallback that reconstructs output items from raw SSE when `raw_response.output` is empty - regression coverage for shared alias routing, empty completed.output parsing, and SSE bridge recovery Closes #25403 Co-authored-by: afoninsky <andrey.afoninsky@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(deps): relax core runtime dependency pins from exact == to ranges When litellm migrated from Poetry to uv (PR #24905, v1.83.1), the core dependency specifications in pyproject.toml changed from Poetry bare-version strings (e.g. openai = "2.30.0") to PEP 621 exact pins (openai==2.24.0). Poetry bare-version strings are actually caret ranges (^X.Y.Z == >=X.Y.Z,<X+1), but PEP 621 == is exact. This means every downstream package that installs litellm as a library dependency is now forced to downgrade aiohttp, pydantic, openai, click, and 8 other common packages to exact old versions. Fix: restore range specifiers for the 12 core runtime dependencies. The optional extras (proxy, proxy-runtime, etc.) are consumed primarily by Docker images where exact pins are appropriate and are left unchanged. The uv.lock file continues to provide exact reproducibility for Docker builds and CI. Fixes: #26154 * Add Rubrik as officially-supported guardrail plugin (#25305) * Add Rubrik as officially-supported guardrail plugin Adds tool blocking and batch logging integration with an external Rubrik webhook service. The plugin validates LLM tool calls against a policy service (fail-open on errors) and batch-logs all requests/responses. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update Rubrik docs: config.yaml as primary, env vars as fallback Restructures the Quick Start to present config.yaml as the recommended approach with tabbed UI, and environment variables as an alternative fallback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add Rubrik env vars to config_settings reference Fixes documentation validation by adding RUBRIK_API_KEY, RUBRIK_BATCH_SIZE, RUBRIK_SAMPLING_RATE, and RUBRIK_WEBHOOK_URL to the environment settings reference table. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add fallback message when blocking service returns empty explanation Prevents whitespace-only violation message when the tool blocking service blocks tools but returns an empty content field. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(ocr): add Reducto parse OCR support (#26068) * feat(ocr): add Reducto parse OCR support * fix(reducto): address OCR review feedback * chore: refresh uv lockfile * Revert "chore: refresh uv lockfile" This reverts commit |
||
|
Sameer Kankute
|
7e61dbb1df
|
fix(responses): preserve cache_control in Responses API -> Chat Completion transformation (#27727)
* fix(responses): preserve cache_control in Responses API -> Chat Completion transformation
cache_control injected by AnthropicCacheControlHook was silently dropped when
_transform_responses_api_content_to_chat_completion_content rebuilt content blocks
with only {type, text}. Now copies cache_control through so Anthropic prompt caching
works correctly when using client.responses.create with cache_control_injection_points.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(responses): preserve cache_control for input_image and input_file blocks
Extends the cache_control fix to image and file content blocks, which were
also silently dropping cache_control during the Responses API -> Chat Completion
transformation. Adds tests for all three content block types.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Babysitter <claude@anthropic.com>
|
||
|
Mateo Wang
|
2e5ebf826f
|
fix(responses): register cooldowns on failure + fail fast on stale encrypted_content (#27820) | ||
|
Sameer Kankute
|
4dbea4e957
|
fix(responses): enforce spec object on completion bridge (#26327)
Ensure Chat Completions -> Responses bridge always emits object="response" so non-native providers return the same top-level schema as native OpenAI Responses. Made-with: Cursor |
||
|
Sameer Kankute
|
f3b80726a7
|
Merge pull request #26301 from BerriAI/litellm_internal_staging
Some checks failed
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (proxy-utils, tests/proxy_unit_tests/test_proxy_utils.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
merge main |
||
|
Ishaan Jaffer
|
e8461b5b97
|
style: run black formatter on files from main merge | ||
|
Sameer Kankute
|
96882e04e7
|
Merge pull request #25942 from BerriAI/litellm_internal_staging
Some checks failed
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
merge litellm_internal_staging |
||
|
Sameer Kankute
|
2506ccb2bc
|
refactor(responses): drop use_responses_api_bridge; fix PLR0915
- Only use_chat_completions_api and openai/chat_completions/ opt into the bridge - Extract MCP gateway and file_search emulation dispatch to cut responses() size - Update docs and tests Made-with: Cursor |
||
|
Sameer Kankute
|
0a8bf4ec9e
|
feat(responses): rename bridge opt-in to use_chat_completions_api
- Add use_chat_completions_api (keep use_responses_api_bridge as deprecated alias) - Support openai/chat_completions/<model> model prefix for the same behavior - Forward use_chat_completions_api in file_search emulation inner calls - Update response_api.md and extend unit tests Made-with: Cursor |
||
|
Sameer Kankute
|
d03ecedba1
|
feat(containers): Azure routing, managed container IDs, delete response parsing (#25287)
* feat(containers): Azure container routing, managed IDs, and delete response wire format - Add AzureContainerConfig and safe URL joining for paths with api-version query - Encode/decode managed container IDs in responses, streaming, and proxy handlers - Accept OpenAI delete response object literal container.file.deleted - Tests for Azure URL regression and DeleteContainerFileResponse parsing Made-with: Cursor * fix(responses): gate response id update on parsed_chunk having response Delta stream events do not include a response body; Mock-based tests (and any truthy synthetic .response on transforms) must not trigger _update_responses_api_response_id_with_model_id. Fixes test_stop_async_iteration_not_logged_as_failure (TypeError: Mock not iterable). Made-with: Cursor * feat(containers): encode container IDs in SDK responses for routing - Add ContainerRequestUtils.encode_container_id_in_response utility - Encode container_id in create/retrieve/delete responses (SDK path) - Fix streaming iterator: gate response ID update on parsed_chunk key - Follows responses API pattern (encode after handler, not in handler) Made-with: Cursor * fix(containers): module-level imports and managed cntr_ ID encoding - Move ResponsesAPIRequestUtils imports to module scope (utils, main, handler_factory). - Serialize absent model_id as empty segment instead of literal None; decode empty and legacy "None" segments as missing for router affinity. - Add unit tests for build/decode round-trip and legacy IDs. Made-with: Cursor * fix(containers): decode managed IDs in endpoint_factory SDK path - Add decode_managed_container_id_for_request in containers/utils and reuse from main. - Strip LiteLLM cntr_ wrappers before generic_container_handler (64-char API limit). - Resolve provider for logging/errors; add unit test for decode helper. - Use resolved_custom_llm_provider after decode for mypy-safe provider typing. Made-with: Cursor * Fix p1 concern * Fix p1 concern |
||
|
Sameer Kankute
|
dc200c34a2
|
fix(responses): map refusal stop_reason to incomplete status in streaming (#25498)
* fix(responses): map refusal stop_reason to incomplete status in streaming Fixes streaming responses API translation where Anthropic's stop_reason="refusal" was incorrectly translated to status="completed" instead of "incomplete". Root cause: build_base_response was unconditionally overwriting finish_reason with None from later chunks, losing the terminal content_filter value. Changes: - streaming_chunk_builder_utils: skip None finish_reason values in build_base_response - streaming_iterator: snapshot chunks before returning pending events (sync path) - streaming_handler: treat usage-only chunks as meaningful content - transformation: map finish_reason=refusal to status=incomplete - tests: add regression tests for refusal handling Made-with: Cursor * Fix test |
||
|
joereyna
|
3ac4333be1
|
fix(responses-ws): use urllib.parse to append model param, fix test mocking | ||
|
joereyna
|
f6dde296fa
|
fix(responses-ws): append ?model= to backend WebSocket URL | ||
|
Sameer Kankute
|
6b7629ec04
|
Fix greptile review | ||
|
Sameer Kankute
|
f6b03a469e
|
feat(responses): add use_responses_api_bridge flag for openai/ models with custom api_base
Allows openai/-prefixed models with a custom api_base pointing to a third-party OpenAI-compatible provider to opt-in to the /responses → /chat/completions bridge, rather than forwarding requests natively to /v1/responses (which may not be supported by the provider). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
ishaan-berri
|
693ad49719
|
Litellm ishaan march23 - MCP Toolsets + GCP Caching fix (#25146) (#25155)
* Litellm ishaan march23 - MCP Toolsets + GCP Caching fix (#25146) * feat(mcp): MCP Toolsets — curated tool subsets from one or more MCP servers (#24335) * feat(mcp): add LiteLLM_MCPToolsetTable and mcp_toolsets to ObjectPermissionTable * feat(mcp): add prisma migration for MCPToolset table * feat(mcp): add MCPToolset Python types * feat(mcp): add toolset_db.py with CRUD helpers for MCPToolset * feat(mcp): add toolset CRUD endpoints to mcp_management_endpoints * fix(mcp): skip allow_all_keys servers when explicit mcp_servers permission is set (toolset scope fix) * feat(mcp): add _apply_toolset_scope and toolset route handling in server.py * fix(mcp): resolve toolset names in responses API before fetching tools * feat(mcp): add mcp_toolsets field to LiteLLM_ObjectPermissionTable type * feat(mcp): register LiteLLM_MCPToolsetTable in prisma client initialization * feat(mcp): validate mcp_toolsets in key-vs-team permission check * feat(mcp): register toolset routes in proxy_server.py * feat(mcp): add MCPToolset and MCPToolsetTool TypeScript types * feat(mcp): add fetchMCPToolsets, createMCPToolset, updateMCPToolset, deleteMCPToolset API functions * feat(mcp): add useMCPToolsets React Query hook * feat(mcp): add toolsets (purple) as third option type in MCPServerSelector * feat(mcp): extract toolsets from combined MCP field in key form * feat(mcp): extract toolsets from combined MCP field in team form * feat(mcp): show toolsets section in MCPServerPermissions read view * feat(mcp): pass mcp_toolsets through object_permissions_view * feat(mcp): add MCPToolsetsTab component for creating and managing toolsets * feat(mcp): add Toolsets tab to mcp_servers.tsx * feat(mcp): pass mcpToolsets to playground chat and responses API calls * feat(mcp): generate correct server_url for toolsets in playground API calls * docs(mcp): add MCP Toolsets documentation * docs(mcp): add mcp_toolsets to sidebar * fix(mcp): replace x-mcp-toolset-id header with ContextVar to prevent client forgery * fix(mcp): use ContextVar + StreamingResponse for toolset MCP routes (fixes SSE streaming) * fix(mcp): cache toolset permission lookups to avoid per-request DB calls * test(mcp): add tests for toolset scope enforcement, ContextVar isolation, and access control * fix(mcp): cache toolset name lookups in MCPServerManager to avoid per-request DB calls * fix(mcp): prevent body_iter deadlock + use cached toolset lookup in responses API - _stream_mcp_asgi_response: add done callback to handler_task that puts the EOF sentinel on body_queue when the task exits, preventing body_iter from hanging forever if the handler raises after headers are sent. - litellm_proxy_mcp_handler: replace raw get_mcp_toolset_by_name() DB call with global_mcp_server_manager.get_toolset_by_name_cached() so toolset resolution uses the 60s TTL cache added for this purpose instead of hitting the DB on every responses-API request. * fix(mcp): toolset access control, asyncio fix, and real unit tests - server.py: _apply_toolset_scope now enforces that non-admin keys must have the requested toolset_id in their mcp_toolsets grant list; admin keys always bypass the check. - mcp_management_endpoints.py: three access-control fixes: * fetch_mcp_toolsets: non-admin keys with mcp_toolsets=None now return [] instead of all toolsets (only admins get 'all' when the field is absent) * fetch_mcp_toolset: non-admin keys that haven't been granted the requested toolset_id now get 403 instead of the full result * add_mcp_toolset: duplicate toolset_name now returns 409 Conflict instead of an opaque 500 - proxy_server.py: use asyncio.get_running_loop() instead of get_event_loop() inside an already-running coroutine (Python 3.10+). - test_mcp_toolset_scope.py: replace four hollow tests that only asserted local variable properties with real tests that call the production fetch_mcp_toolsets() and handle_streamable_http_mcp() functions with mocked dependencies. * fix(mcp): add mcp_toolsets to ObjectPermissionBase, fix multi-toolset overwrite, fix delete 404, allow standalone key toolsets * fix(mcp): add auth check on toolset resolution in responses API; union mcp_servers in _merge_toolset_permissions * fix(mcp): handle RecordNotFoundError in update_mcp_toolset; union direct servers with toolset servers * fix(mcp): use _user_has_admin_view; deny None mcp_toolsets for non-admin; use direct RecordNotFoundError import; fix docstring * fix(mcp): add @default(now()) to MCPToolsetTable.updated_at; fix test for non-admin toolset access * fix: use UniqueViolationError import; guard _ensure_eof for error/cancel only * fix(mcp): preserve mcp_access_groups in toolset scope, use shared Redis cache for toolset perms - Remove mcp_access_groups=[] from _apply_toolset_scope (server.py) and the responses API toolset path (litellm_proxy_mcp_handler.py). A key's access-group grants remain valid even when the request is scoped to a single toolset; clearing them silently revoked legitimate entitlements. - Switch resolve_toolset_tool_permissions and get_toolset_by_name_cached to use user_api_key_cache (Redis-backed DualCache in production) instead of per-instance in-memory dicts. Cache entries are now shared across workers, eliminating the per-worker stale-toolset-permission window flagged as a P1 by Greptile. - Use union merge (set union of tool names per server) when applying toolset permissions in the responses API path so direct-server tool restrictions are not overwritten by toolset permissions. * fix(mcp): return 404 when edit_mcp_toolset target does not exist * fix(mcp): align mcp_toolsets default to None in LiteLLM_ObjectPermissionTable * fix(mcp): admin toolset visibility, in-place tool name mutation, test helper coercion * fix(mcp): treat None/[] team mcp_toolsets as no restriction in key validation * fix(mcp): allow_all_keys backward compat, blocked_tools API write-path, efficient startup query * fix(mcp): use _mcp_active_toolset_id ContextVar to detect toolset scope, avoiding DB-default false-positive * fix(mcp): remove dead toolset cache stubs, log invalidation failures, align schema updated_at defaults * fix(mcp): deserialise MCPToolset from Redis cache hit, replace fastapi import in test * fix(mcp): evict name-cache on toolset mutation, 409 on rename conflict, warning-level list errors * fix(redis): regenerate GCP IAM token per connection for async cluster (#24426) * fix(redis): regenerate GCP IAM token per connection for async cluster clients Async RedisCluster was generating the IAM token once at startup and storing it as a static password. After the 1-hour GCP token TTL, any new connection (including to newly-discovered cluster nodes) would fail to authenticate. Fix: introduce GCPIAMCredentialProvider that implements redis-py's CredentialProvider protocol. It calls _generate_gcp_iam_access_token() on every new connection, matching what the sync redis_connect_func already does. async_redis.RedisCluster accepts a credential_provider kwarg which is invoked per-connection. * refactor(redis): move GCPIAMCredentialProvider to its own file Extract GCPIAMCredentialProvider and _generate_gcp_iam_access_token into litellm/_redis_credential_provider.py. _redis.py imports them from there, keeping the public API unchanged. * fix: address Greptile review issues - GCPIAMCredentialProvider now inherits from redis.credentials.CredentialProvider so redis-py's async path calls get_credentials_async() properly - move _redis_credential_provider import to top of _redis.py (PEP 8) - remove dead else-branch that silently no-oped (gcp_service_account from redis_kwargs.get() was always None since it's popped by _get_redis_client_logic) - remove mid-function 'from litellm import get_secret_str' inline import - remove unused 'call' import from test_redis.py * chore: retrigger CI/review * chore: sync schema.prisma copies from root * chore: sync schema.prisma copies from root * fix(proxy_server): use bounded asyncio.Queue with maxsize to prevent unbounded growth * fix(a2a/pydantic_ai): make api_base Optional to match base class signature * fix(a2a/pydantic_ai): make api_base Optional in handler and guard against None * fix(mcp): remove unused get_all_mcp_servers import * fix(mcp): remove unused MCPToolset import * refactor(mcp): extract toolset permission logic to reduce statement count below PLR0915 limit * fix(tests): update reload_servers_from_database tests to mock prisma directly --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(toolset_db): lazy-import prisma to avoid ImportError when prisma not installed * fix(tests): update UI tests for toolset tab and updated empty state text * fix(tests): add get_mcp_server_by_name to fake_manager stub --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Ishaan Jaffer
|
4b5ca49cd2 | fix: map file_url -> file_id in Responses->Completions translation | ||
|
Lei Nie
|
1716956520 |
test(responses): update expected events and add mock test for content_part.added
- Update test_anthropic_via_responses_api expected_events to include CONTENT_PART_ADDED between OUTPUT_ITEM_ADDED and OUTPUT_TEXT_DELTA - Add TestEnsureOutputItemContentPartAdded with 3 mock tests: message item emits content_part.added, reasoning item does not, and the event is only emitted once |
||
|
Sameer Kankute
|
021540b2e2 |
fix: prevent double prompt management in async path, preserve optional params
- aresponses() now pops prompt_id from kwargs after the async hook runs and passes merged_optional_params via _async_prompt_merged_params. responses() checks for this internal kwarg first and skips the sync hook entirely when present — eliminating double-merge of template messages. - merged_optional_params from async_get_chat_completion_prompt is no longer discarded (_); it flows through to local_vars in responses(). - Async tests now assert get_chat_completion_prompt.assert_not_called() to directly detect any double-execution regression. Made-with: Cursor |
||
|
Sameer Kankute
|
d333dc4077 |
address greptile review feedback (greploop iteration 1)
- Fix async path: call async_get_chat_completion_prompt in aresponses() before executor dispatch, mirroring acompletion() in main.py. Discard merged_optional_params in async path (sync responses() handles them via local_vars), avoiding TypeError from duplicate kwargs in partial(). - Fix provider re-resolution: replace "/" in model heuristic with model != original_model comparison so bare model names are handled. - Add 3 async tests covering hook invocation, optional param propagation, and non-message item filtering in aresponses(). Made-with: Cursor |
||
|
Sameer Kankute
|
b32f5ea379 | Fix greptile comments | ||
|
Sameer Kankute
|
0d70864d09 | Add support for prompt management for responses | ||
|
yuneng-jiang
|
a81a1968ed |
Fix test isolation: clear litellm.callbacks and model_fallbacks between tests
The isolate_litellm_state conftest fixture saved/restored litellm.callbacks but never cleared it before each test, unlike the other callback lists. It also didn't handle litellm.model_fallbacks. Leaked callbacks and fallback config caused mocked tests to route through Router/fallback paths, hitting real APIs with mock keys. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Chesars
|
feed274aa3 |
Reapply "feat: add model_cost aliases expansion support"
This reverts commit
|
||
|
Chesars
|
1be6b31e2f | merge: resolve conflicts between main and litellm_oss_staging_03_11_2026 | ||
|
Varad Khonde
|
9b15f639a4
|
fix(responses): merge parallel function_call items into single assistant message (#23116) | ||
|
Sameer Kankute
|
d143df8591 |
Add tests for resp
onses websocket |
||
|
Shivam Rawat
|
d49abf8577
|
[Fix] Pass MCP auth headers from request into tool fetch for /v1/responses and chat completions (#22291)
* fixed dynamic auth for /responses with mcp * fixed greptile concern |
||
|
Ishaan Jaff
|
6ec16d583d
|
fix(test): add timeout to flush() call to prevent 300s hang in CI (#21819)
GLOBAL_LOGGING_WORKER.flush() calls queue.join() which blocks until all items are task_done(). In CI with pytest-asyncio, each test gets a fresh event loop so the worker reinitializes its queue - items from a previous test never get task_done(), causing an infinite hang. Fix: wrap flush() with asyncio.wait_for(..., timeout=10.0). |
||
|
Ishaan Jaff
|
8483477512
|
fix(test): add asyncio.sleep(0) before flush() to prevent hang in test_async_no_duplicate_spend_logs (#21813) | ||
|
Ishaan Jaff
|
02670582c5
|
fix(tests): replace asyncio.sleep(1) with event-based wait in metadata callback tests (#21805) | ||
|
Sameer Kankute
|
36fd14357c | FIx: replace deprecated claude-3-7-sonnet-20250219 with claude-4-sonnet-20250514 | ||
|
Sameer Kankute
|
5d8f2196d1 | Add test case | ||
|
Sameer Kankute
|
bd0c80406f
|
Merge pull request #21326 from BerriAI/litellm_oss_staging_02_16_2026
Litellm oss staging 02 16 2026 |
||
|
Julio Quinteros Pro
|
c3346962a9 |
fix: replace silent if-hasattr guards with unconditional assertions in MCP streaming tests
The `if hasattr(...)` guards in test_acompletion_with_mcp_adds_metadata_to_streaming and test_acompletion_with_mcp_streaming_metadata_in_correct_chunks could silently skip the provider_specific_fields assertions if chunks lacked choices/delta. Replace with unconditional `assert hasattr(...)` so failures surface immediately. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Julio Quinteros Pro
|
ab6d2eefb9 |
fix: improve test isolation for parallel execution
Fixes test failures that occur during parallel test execution (pytest -n 4) due to module reloading issues with conftest.py reloading litellm. Changes: - Add module reload fixtures to ensure fresh references after conftest reloads - Use patch.object and string-based patches instead of direct attribute assignment - Use class name comparison instead of isinstance for reloaded modules - Handle case where litellm is missing from sys.modules during parallel runs - Move stream consumption inside patch contexts to avoid real API calls - Mock litellm.acompletion instead of low-level HTTP handlers - Add skipif decorator for enterprise-only test classes Affected test files: - test_container_integration.py - test_responses_background_cost.py - test_huggingface_embedding_handler.py - test_vertex_ai_rerank_integration.py - test_volcengine_responses_transformation.py - test_pillar_guardrails.py - test_litellm_pre_call_utils.py - test_proxy_server.py - test_converse_transformation.py - test_chat_completions_handler.py - test_aresponses_api_with_mcp.py - test_anthropic_experimental_pass_through_messages_handler.py Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Chiranjeevisantosh Madugundi
|
dcff3260df
|
fix: preserve metadata for custom callbacks on codex/responses path (… (#21243)
* fix: preserve metadata for custom callbacks on codex/responses path (#21204) - Use metadata or litellm_metadata when calling update_environment_variables in responses/main.py so metadata is not overwritten by None on the bridge path (completion -> responses API). - Add tests for metadata in custom callback for codex models and for litellm_metadata in aresponses(). Co-authored-by: Cursor <cursoragent@cursor.com> * Update tests/test_litellm/responses/test_metadata_codex_callback.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Julio Quinteros Pro
|
8d15996b5a |
test: Fix flaky tests with proper mocking and skip conditions
1. test_acompletion_with_mcp_streaming_metadata_in_correct_chunks:
- Moved stream consumption inside patch context to avoid real API calls
- The previous implementation had assertions outside the `with patch(...)`
block, causing real OpenAI API calls when consuming the stream
2. TestCheckResponsesCost tests:
- Added skip condition when litellm_enterprise module is not available
- These tests import from litellm_enterprise.proxy.common_utils.check_responses_cost
which is only available in the enterprise version
|
||
|
Krish Dholakia
|
fa0b26c611
|
ci/cd fixes (#21218)
* fix: add a2a-sdk to dev deps * fix: fix test * fix: update poetry lock * test: cleanup bad/hanging tests |
||
|
Lei Nie
|
e9c99f41bd |
fix(vertex_ai): forward extra_body to completion transformation handler (#20950)
* fix(vertex_ai): forward extra_body to completion transformation handler The responses() function accepted extra_body as a named parameter but did not pass it to response_api_handler when responses_api_provider_config was None (completion transformation path), silently dropping it. Also adds deep-merge support for extra_body in Vertex AI Gemini transformation, so dict values like generationConfig are merged rather than replaced. * refactor(vertex_ai): extract _merge_extra_body to fix PLR0915 lint Move the extra_body merge loop into a helper function to keep _transform_request_body under the 50-statement limit. |
||
|
Ishaan Jaff
|
3d9b145b04
|
[Feat] Adds support for server-side compaction on the OpenAI Responses API context_management (#21058)
* test_responses_api_context_management_server_side_compaction * Server-side compaction * docs fix * test_responses_api_shell_tool |
||
|
Sameer Kankute
|
5222fd4795
|
Merge pull request #20783 from BerriAI/litellm_oss_staging_02_09_2026
litellm oss staging 09/02/2026 |
||
|
Lei Nie
|
08824f9912 |
[Bug] Fix missing image_tokens in Responses API output_tokens_details (#20404)
When transforming chat completion responses to Responses API format, image_tokens from completion_tokens_details was not being included in output_tokens_details. This affected Vertex AI/Gemini models that return image token counts in candidatesTokensDetails with modality="IMAGE". The fix adds image_tokens handling alongside existing reasoning_tokens and text_tokens transformation. |
||
|
Emerson Gomes
|
28e15f4fd6 |
fix(responses): preserve cached tool call objects in tool result recovery (#20700)
* fix(responses): preserve cached tool call objects in tool result recovery * fix(responses): support attr-based cached tool call recovery * Update litellm/responses/litellm_completion_transformation/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Emerson Gomes
|
cf17a440cd | fix(responses): guard ambiguous tool-call index reuse |