mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-28 01:32:17 +00:00
26 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yuneng Jiang
|
044f88ee91
|
fix(proxy): register transcribe as a known provider for model grants
#41515 added the cost map entry transcribe/StartTranscriptionJob under a new litellm_provider value "transcribe" without registering that provider anywhere else, so litellm.models_by_provider had no "transcribe" key. test_models_by_provider derives its provider set from the cost map itself, so it went red on main. The user-visible half is that get_provider_models returned None for the provider, which get_known_models_from_wildcard turns into an empty list, leaving a transcribe/* key or team grant resolving to no models. Mirror the aws_polly registration: an enum member, a model set, an ingestion branch, and a models_by_provider entry. Amazon Transcribe is reached through the pass-through route rather than the Add Model form, so it joins the frozen unlisted set the Add Model drift test tracks. |
||
|
kerry
|
d2ac51893b |
test: keep the pinning-test removal free of unrelated reformatting
Regenerated every touched file from origin/main applying only the B1 test deletions and the unused import and helper cleanup they leave behind, without running the formatter across untouched code. CI only checks ruff format under litellm/, so the earlier reflows of test files were pure diff noise for reviewers Also drops the tests/local_testing/test_prompt_caching.py entry from the caching-local shard in test-unit.yml since that file is deleted Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
kerry
|
8ecbf3dbc1 |
test: drop tests that pin provider-owned cost map values
The repo rule is that a test must only fail when litellm code changes, never when a vendor updates a price, renames a field, or drops a model. These tests asserted shipped catalog entries directly, comparing lookup results to literals copied from model_prices_and_context_window.json or requiring named entries to exist or be absent, so every cost map sync could break them without any litellm code changing Tests that exercise real litellm behavior with an injected local model_cost, invariants like backup parity, and assertions on non-lookup code paths are untouched Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo-berri
|
a17fcecf70 |
refactor(azure_ai): type the Foundry param mapping override and drop test docstrings
The AzureAIStudioConfig.map_openai_params override now carries dict[str, object] annotations instead of bare dict, and the docstrings added to the new tests go away since the test names already say what they cover. No behavior change |
||
|
mateo-berri
|
15372967c6 |
fix(azure_ai): read the azure_ai card for gpt-5 series reasoning effort gates
Foundry deployments of gpt-6-astra reached through azure_ai used the bare OpenAI card for the reasoning_effort none gates, so temperature and top_p were refused while the azure_ai card says none is supported. AzureAIStudioConfig now dispatches gpt-5 series params through AzureAIGPT5Config, which looks capabilities up under the azure_ai/ prefix the way the azure route does Also carries the search_context_cost_per_query block azure/gpt-6-astra has, adds a flex service tier cost test that fails at the merge base, and keeps the wildcard test from stripping azure_ai/gpt-6-astra out of the provider set |
||
|
mateo-berri
|
635bb3a209 |
feat(cost-map): add azure_ai/gpt-6-astra Foundry pricing
A gpt-6-astra deployment on a Foundry project reached through the azure_ai route had no cost map entry of its own, so it resolved to the OpenAI gpt-6-astra card: missing from the azure_ai/* wildcard listing, flex and priority prices and /v1/batch it does not sell, and no none reasoning effort. Add azure_ai/gpt-6-astra mirroring the azure/gpt-6-astra Standard Global sheet the way azure_ai/gpt-5.5 mirrors azure/gpt-5.5, and extend the cost, reasoning-effort, and wildcard listing tests to the Foundry route. |
||
|
devin-ai-integration[bot]
|
7ae352e5cf
|
fix(model_checks): drop wildcard routes like bedrock/* from /v1/models (#31731)
* fix: remove wildcard routes from /v1/models response Wildcard routes like bedrock/* were leaking into the /v1/models response because _get_wildcard_models only removed them from unique_models in the fallback branches (no router or no deployment), but not when the router had a matching deployment. Now wildcards are always removed from the base list; they are only re-added to the result when return_wildcard_routes=True is explicitly passed. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(model_checks): collapse wildcard expansion branches and tighten regression tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> |
||
|
ryan-crabbe-berri
|
e9d40a8f73 |
test: enforce F811 so a duplicate definition cannot silently replace the first
A name bound twice keeps only the second binding. In `tests/` that is nearly always a repeated import, harmless but misleading, and the same rule is what catches the cases that are not harmless: a local that shadows an import the module still calls, and a second `def test_x` that quietly replaces the first. 311 of the 344 sites were repeated imports and came out with ruff's own fix. The remaining 33 needed a decision. Four modules imported a name they never used because a local definition below already shadowed it. Two comprehensions bound `call` over `unittest.mock.call`, which those modules import and use. One test rebound the two module handles its nested reload closure had captured. One class attribute shadowed an unused `status` import. The load-test fixtures move to a conftest, which is how pytest is meant to share them, so the test module no longer imports three fixture names it never calls. The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that fixture by name before the body runs, so the parameter never shadows anything. |
||
|
ryan-crabbe-berri
|
2d12a3ea41
|
fix(proxy): expand config-defined model access groups when resolving team models for /v2/model/info (#34211)
* fix(proxy): expand config-defined model access groups when resolving team models for /v2/model/info Teams whose only model grant is a config-defined access group (a model_info.access_groups name listed in team.models) got an empty /v2/model/info?include_team_models=true result. _add_team_models_to_all_models passed each team.models entry straight to llm_router.get_model_list(model_name=...), which never matches an access-group name, so the group's member deployments were dropped. Runtime auth and /v1/models were unaffected because they expand team.models through get_team_models first. Resolve team.models through the same get_team_models resolver before iterating, reusing the exact path runtime auth and /v1/models trust so the two can't drift again. The get_model_names and get_model_access_groups accessors are hoisted above the team loop so they run once. * fix(proxy): keep a literal model whose name collides with an access-group name in listings A grant string that names both a deployed model and a config access group grants BOTH at runtime (_check_model_access_helper unions them), but the listing resolver dropped the literal and substituted the group members, hiding a callable model from /v1/models and /v2/model/info. Keep the literal when it is also a deployed model so listings match runtime access exactly. Pure-group names (no collision) are still replaced by their members. Also rewrites _get_models_from_access_groups to build its result without mutating the input list. Addresses the Greptile P1 on this PR. * fix(proxy): type proxy_model_list param as Sequence to satisfy LIT001 budget |
||
|
devin-ai-integration[bot]
|
1a45bf9afe
|
fix(proxy): resolve entity access groups in the model listing endpoints (#36230)
* fix(proxy): resolve entity access groups in the model listing endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): reuse the fetched team object when listing models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover key-level access group resolution in model listing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
7e8d0d3130
|
fix: rebuild models_by_provider in add_known_models so cost map reloads reach wildcard expansion (#36010)
* fix: rebuild models_by_provider in add_known_models so cost map reloads reach wildcard expansion * fix: refresh models_by_provider in place so captured references survive reloads |
||
|
devin-ai-integration[bot]
|
5ece78fb5f
|
revert: undo teamless all-team-models denial from #32022 and #29746 (#32032)
* Revert "fix(auth): deny model access for teamless keys with all-team-models (#32022)"
This reverts commit
|
||
|
Sameer Kankute
|
2b496bc7f7
|
fix(proxy): restore wildcard expansion in /v1/model/info (#31444) | ||
|
Sameer Kankute
|
80c5a84871
|
chore: litellm oss staging (#30968)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens
* test: scope local cost map env var with monkeypatch to avoid test pollution
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold
_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.
mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.
* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers
Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.
Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.
* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview
MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.
* fix(mcp_debug): mask short auth values in debug headers instead of echoing them
Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.
* test(mcp_debug): assert masked short value preserves length
* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)
Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.
Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:
- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
ProviderConfigManager.get_provider_audio_transcription_config() in
litellm/utils.py; update the stale comment in
get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
get_supported_openai_params() in
litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
model_prices_and_context_window.json and
litellm/model_prices_and_context_window_backup.json (both had
mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
imports from tests/llm_translation/test_fireworks_ai_translation.py
No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider (#30876)
* feat: add darkbloom provider
* fix: document darkbloom provider endpoints
* fix: address darkbloom review feedback
* fix: update darkbloom tool metadata
* fix: fail fast for non-Postgres database URLs (#30883)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging
* Validate DIRECT_URL alongside DATABASE_URL startup guards
* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)
* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)
* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)
* style(bedrock): black-format stream-error helper (#24608)
* fix(mcp): re-land native tool preservation with typed annotations (#30645)
* fix(mcp): preserve native tools in semantic filter hook with typed annotations
* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
* fix(sambanova): return embeddings supported params instead of dropping them (#30937)
* fix(router): send fallback metadata when streaming (#30914)
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:
1. The response now correctly populates the fallback headers
(`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
to the client (opt-in) by passing `include_fallback_errors: true` in
the request.
The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
* fix(mistral): drop output-only reasoning fields from input messages (#30884)
LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.
Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)
* fix(perplexity): bill search queries at the per-request price, not 1/1000
The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").
The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.
Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.
* test(perplexity): update integration test search-cost expectations to per-request
The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.
* test(perplexity): drop unused mock imports flagged by ruff
* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)
* fix(fireworks_ai): return None for transcription in get_supported_openai_params
Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting
Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.
Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test
The operator gate added in
|
||
|
Yassin Kortam
|
84266bf924
|
feat(auth): resolve caller identity once into a Principal at the auth seam (#30887)
Introduce a single, typed caller identity that is resolved once at the auth boundary and read by reference downstream, instead of being re-derived from a 50-field key object or rebuilt from request metadata. What this adds (litellm/proxy/auth/resolvers/), organized by responsibility: - Principal: a small, frozen, identity-only value type (user / organization / teams / project / end-user / roles / scopes / network), with its sub-models and the role mapping. No budget or policy state; those stay on the key object. - DbIdentityStore: the auth flow's resolver, owning both halves of resolving a caller. resolve_key does the one combined_view lookup (cache, then DB via the shared lower-level helpers, then write-back) and returns the key object, which still flows for budget / rate-limit / policy unchanged. principal_from_key projects the identity slice of that key object into a Principal, issuing no lookup. user_api_key_auth resolves every key through the store rather than calling get_key_object directly; auth_checks.get_key_object stays as the legacy entrypoint for its other callers until they migrate. - network: the X-Forwarded-For / trusted-proxy CIDR primitives live here in one place. trusted_proxy_utils now imports them rather than keeping a second copy. At the seam, user_api_key_auth projects one per-request Principal off the resolved key object and stamps the request network context onto it once (X-Forwarded-For is trusted only when trusted_proxy_ranges is configured). It is attached to request.state.principal for the downstream consumers later phases add. The projection is additive and defensive: a failure never rejects an already-authenticated request, and a missing principal must be treated as deny by any future reader. The Principal is always identifiable (credential_ref and a stable subject off the token), never anonymous. This is additive and changes no behavior today; it is the identity foundation the spend-attribution and authorization phases build on. |
||
|
Sameer Kankute
|
1ccc1e5b23
|
chore: litellm oss staging160626 (#30527)
* feat(ui): gate "Default Credentials" hint on /ui/login behind env flag (#30234) Adds LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (and an equivalent general_settings.hide_default_credentials_hint) that suppresses the "By default, Username is admin and Password is your set LiteLLM Proxy MASTER_KEY" info card rendered on /ui/login and /fallback/login. Motivation: in production deployments operators set UI_USERNAME / UI_PASSWORD (or SSO), and the hardcoded hint becomes factually incorrect and is flagged by security scanners (Tenable WAS plugin 114625) as information disclosure. There is currently no way to suppress it without forking the dashboard. Behaviour: - Default is unchanged (hint shown), so existing deployments are unaffected. - New field hide_default_credentials_hint on the well-known UI config endpoint, populated from the env var or general_settings. - LoginPage.tsx conditionally renders the Alert based on the flag. Refs: BerriAI/litellm#30232 * fix(router): clean pattern_router state on upsert/delete (#29601) * fix(router): clean pattern_router state on upsert/delete PatternMatchRouter.add_pattern was append-only, and neither Router.upsert_deployment nor Router.delete_deployment removed the existing entry. Rotated-out api_keys stayed in the routing rotation for wildcard deployments (model_name with `*`) until proxy restart, silently defeating key rotation as an admin operation. The same leak applied to provider_default_deployment_ids and per-team pattern routers, and the patterns list grew unboundedly on every edit * test(router): direct unit tests for _remove_deployment_from_wildcard_state router_code_coverage.py greps test files for AST Call nodes and flagged the helper as untested because the existing coverage only exercised it transitively through upsert/delete. Adds two direct tests that pin the helper's contract (cleans across global pattern router, per-team routers with empty-router pop, and provider_default_deployment_ids; noop on falsy model_id) * fix(router): address Greptile review on pattern_router cleanup Widen PatternMatchRouter.remove_deployment annotation to Optional[str]; the implementation already handles None via the falsy guard and the unit test exercises it directly. Move _remove_deployment_from_wildcard_state up one level in upsert_deployment so it runs whenever the prior deployment is on the router, not only when the model_id is present in the fast-mapping index. The scenario is currently unreachable (get_deployment shares the same index), but the cleanup is idempotent so this is defensive against any future divergence between those code paths. * fix(router): widen _remove_deployment_from_wildcard_state to Optional[str] Moving the call out of the inner `deployment_id in deployment_fast_mapping` block in the previous commit lost mypy's narrowing of `deployment_id` from Optional[str] to str, tripping the lint CI. The helper already handles None via its falsy guard, so widening the annotation matches the actual contract. * fix(router): make delete_deployment wildcard cleanup symmetric with upsert After the previous commit moved _remove_deployment_from_wildcard_state out of the inner index-map guard in upsert_deployment, delete_deployment was still calling it only inside `if deployment_idx is not None`. Greptile flagged the asymmetry: under a desynced index_map, delete would silently leave the stale wildcard credential in pattern_router. Moves the cleanup call to the top of the try block, mirroring the upsert path. Cleanup is idempotent so the change is a no-op on the happy path. Adds a regression test that simulates the desync by removing the entry from model_id_to_deployment_index_map and asserts delete still clears pattern_router. * fix(pricing): add 1h cache-write cost for Anthropic Sonnet 4.5/4.6 (#30474) The native anthropic claude-sonnet-4-5/4-6 price-map entries were missing cache_creation_input_token_cost_above_1hr (and the >200K long-context sub-tier for 4.5), so 1-hour-TTL cache writes were costed at the 5-minute rate. Adds 6e-06 regular (and 1.2e-05 long-context) = 2x base input, matching the vertex_ai/azure_ai/bedrock siblings and the older claude-sonnet-4-20250514 entry. Adds a regression test. * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect (#30075) * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect - add _check_request_disconnection to common_request_processing; wrap llm_call as asyncio.Task so it can be cancelled; catch CancelledError and raise HTTPException(499) when client disconnects before LLM responds (non-streaming path) - pass raw httpx.Response into ModelResponseIterator in make_call/make_sync_call so the iterator holds a reference to the underlying connection - implement ModelResponseIterator.aclose() and .close(): close the line iterator then explicitly call response.aclose()/response.close() to release the httpx connection when the client drops mid-stream; errors are debug-logged, not raised - add tests for _check_request_disconnection (cancels task, graceful on exception, does not cancel when client stays connected) and base_process_llm_request 499 behavior; add TestModelResponseIteratorCleanup verifying aclose/close propagation through CustomStreamWrapper * fix(proxy): record 499 on streaming disconnect and cancel orphaned gather tasks Wire streaming generator cleanup to log client_disconnected with error_code 499 in spend logs, cancel pending during_call_hook tasks when the LLM call is cancelled on disconnect, and align the 600s poll limit comment with proxy_server. * fix: extract client disconnect logging helper to satisfy PLR0915 * fix: resolve mypy and code-quality CI failures for client disconnect logging Cast client disconnect error_information for mypy, only await pending gather tasks to avoid masking LLM errors, and add tests for the new logging helper and gather cleanup. * fix(proxy): harden gather cleanup so finally cannot mask LLM errors * fix(proxy): shield streaming disconnect logging and strip spoofable metadata Move streaming disconnect recording into a shielded cancel scope, add gather cleanup regression coverage for guardrail-converted cancels, and strip client_disconnected/error_information from user metadata at the proxy boundary. * fix(proxy): only map CancelledError to 499 for client disconnect Track when the disconnect poller cancels the LLM task and re-raise other CancelledError paths so graceful shutdown is not reported as HTTP 499. * fix(proxy): remove dead _check_request_disconnection helper Non-streaming client disconnect is handled by staging's cancel_on_disconnect path via _await_llm_call_cancelling_on_disconnect. Drop the unused is_disconnected poller and its unit tests; rename the remaining integration tests to TestDisconnectGatherCleanup. * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_wind.. (#29303) * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_window.json Mistral's docs page lists mistral-medium-3-5 as a new model offering. Pricing/specs sourced from Mistral's published model metadata: - input: $1.50 / 1M tokens - output: $7.50 / 1M tokens - context: 262,144 tokens - capabilities: vision, function calling, structured outputs, assistant prefill Adds entry: `mistral/mistral-medium-3-5`, mirroring the pattern used for the rest of the Mistral family. test(mistral): add model_info test for mistral-medium-3-5 + sync backup cost map - Mirror mistral/mistral-medium-3-5 entries into litellm/model_prices_and_context_window_backup.json so the bundled model cost map matches the canonical model_prices_and_context_window.json. - Add tests/test_litellm/test_mistral_medium_3_5_model_metadata.py covering pricing tiers, capability flags, context window, provider routing, and parity between the main and backup cost maps. - Point 'source' at the live Mistral models documentation page. * fix(ui): three small UI fixes — Gemini api_base + credential form reset + Mode badge (#30419) * fix(ui): three small UI fixes — Gemini api_base field + credential form reset + Mode badge Three independent fixes; bundled because they all touch the credential-form / logging-callbacks area. 1. expose api_base field on Google AI Studio credential form The runtime gemini provider supports custom api_base via `vertex_llm_base._check_custom_proxy`; the UI just needs to expose the field. Adds api_base to the Google_AI_Studio credential form ordered before api_key (matching OpenAI/Anthropic conventions). Default value matches the canonical Google AI Studio endpoint that LiteLLM's gemini provider talks to when api_base is unset, so leaving the default in the form behaves identically to leaving it blank. 2. reset credential form state when switching providers Switching the Provider select in AddCredentialModal / EditCredentialModal left the previous provider's field values populated. The form then submitted a mixed payload (e.g. Azure deployment fields under an OpenAI credential), producing confusing failures. Extract `getProviderFieldDefaults` helper and reset the form to it on provider change. Unit-tested via the extracted helper because Antd Select's portal/dropdown behaviour is unreliable in jsdom. 3. logging callbacks table reads backend `type` for Mode badge (#35) The `/get_callbacks` proxy endpoint returns each callback as `{name, type, variables}` where `type` is `"success"` or `"failure"`. The same callback name can appear twice (one per event class) and the two entries fire on disjoint events. `LoggingCallbacksTable` ignored `type` and read `record.mode` (always undefined), so every row fell back to the "Success" badge. A `generic_api` callback registered for both classes showed up as two identical "Success" rows + React duplicate-key warning. Read `record.type` first (fall back to `record.mode` for newly- added not-yet-server-acknowledged rows). Composite rowKey `${name}-${type ?? mode ?? 'success'}`. Removed leftover debug `console.log`. * fix(ui): drop api_base default_value to preserve Gemini v1alpha auto-routing Greptile P2 (PR #30419, threads on lines 1255-1256 of provider_create_fields.json): the api_base field's `default_value` was hard-coded to "https://generativelanguage.googleapis.com/v1beta". This: 1. Bakes v1beta into every credential record saved through the form, even when the user never touched the field. If LiteLLM's internal gemini default URL ever changes, those persisted credentials keep hitting the stale path. 2. Bypasses `_get_gemini_url`'s automatic version routing for Gemini 3+ models. That helper picks v1alpha for Gemini 3+ and v1beta for older models when api_base is unset. With the default pre-filled (and `_check_custom_proxy` then taking over because api_base is non-empty), Gemini 3+ requests get pinned to v1beta and may fail or behave unexpectedly — purely because the user accepted the visible default. Fix: set `default_value` to `null` and move the canonical URL guidance into the `placeholder` (visible to the user, never persisted) and an expanded tooltip. UX is unchanged — the URL is still shown in the greyed-out input — but the auto-version-routing path stays default. Updated test_google_ai_studio_provider_fields_expose_api_base to assert the new contract (`default_value is None`, `placeholder` carries the canonical URL), with a comment pointing at the Greptile threads as the rationale so future contributors don't accidentally re-introduce the default. 26/26 tests in the file pass. JSON validates (`json.load` clean). * feat(azure_ai): add gpt-5.5 to model cost map (#30428) * feat(azure_ai): add gpt-5.5 to model cost map Adds azure_ai/gpt-5.5 and its dated snapshot azure_ai/gpt-5.5-2026-04-23 to both the canonical and bundled cost maps. gpt-5.5 is generally available on Azure AI Foundry; pricing mirrors the openai gpt-5.5 entry, matching the established azure_ai convention (verified identical for gpt-5.4), in the azure tier structure (base / above-272k / priority). supports_minimal_ reasoning_effort is false, the capability that changed from gpt-5.4. Fixes #30306 * Update tests/test_litellm/test_gpt_5_5_model_metadata.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: guard check_and_fix_namespace against None key (#30435) * fix: guard check_and_fix_namespace against None key When user_id is None, the cache key can be None, causing AttributeError: 'NoneType' object has no attribute 'startswith' in check_and_fix_namespace. Add an early return for None key to prevent the error and the ERROR-level log noise it produces on every unauthenticated request. Fixes #30424 * fix: update type annotations for check_and_fix_namespace - key: str -> Optional[str] (now handles None input) - return: str -> Optional[str] (returns None when input is None) Addresses Greptile review concern about type signature mismatch. * fix: revert check_and_fix_namespace type signature to str to fix MyPy downstream errors * fix: update type annotations for check_and_fix_namespace - Change signature from str -> str to Optional[str] -> Optional[str] - Remove type: ignore comment on None return - Add None guard in async_set_cache_sadd before passing to helper Addresses review feedback from Sameerlite on type mismatch. * Revert "fix: update type annotations for check_and_fix_namespace" This reverts commit |
||
|
yuneng-jiang
|
37ef8d9059
|
fix(proxy): hydrate wildcard discovery credentials (#28284) (#28419)
* fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> |
||
|
Ishaan Jaffer
|
e8461b5b97
|
style: run black formatter on files from main merge | ||
|
Chesars
|
feed274aa3 |
Reapply "feat: add model_cost aliases expansion support"
This reverts commit
|
||
|
Chesars
|
1be6b31e2f | merge: resolve conflicts between main and litellm_oss_staging_03_11_2026 | ||
|
yuneng-jiang
|
1755a281bd |
Fix mutation bug: copy lists in get_key_models to prevent corrupting cached UserAPIKeyAuth
`all_models = user_api_key_dict.models` was creating an alias, so `_get_models_from_access_groups` (which uses `.pop()`/`.extend()`) would mutate the cached object in-place. Now both `.models` and `.team_models` assignments create copies via `list()`. Added test to verify the input is not mutated. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
1cf191d9ad |
[Fix] Deduplicate model lists and remove dead assignment
Adds dedup to get_key_models and get_team_models to prevent duplicate entries when access group member models overlap with proxy_model_list. Removes dead assignment of all_models in get_team_models. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
c829733200 |
[Fix] Include model access groups when expanding All Proxy Models
When a team has "all-proxy-models", the model list expansion now includes model access group names so they appear in the UI key creation form. Also fixes get_key_models not forwarding include_model_access_groups to _get_models_from_access_groups, and removes unused _unfurl_all_proxy_models. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Shivam Rawat
|
c0e87f7ffb
|
fixed byok models for teams issue (#21408) | ||
|
Alexander Yastrebov
|
825923e7be
|
litellm/proxy: preserve model order of /v1/models and /model_group/info (#13178)
Closes #12644 Signed-off-by: Alexander Yastrebov <alexander.yastrebov@zalando.de> |
||
|
Krish Dholakia
|
17d2711cf4
|
UI - fix invitation link + ensure team models returned when team has 'all-proxy-models' + team only models (#11524)
* fix(onboarding_link.tsx): fix adding ui/invitation id * fix(onboarding_link.tsx): update invitation link function to handle w/ and w/out custom server path cases * fix(model_checks.py): ensure team only models returned when all proxy models set for team |