mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-10 22:41:41 +00:00
31 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yuneng-jiang
|
6a0d03914c
|
test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
|
||
|
tin-berri
|
4307b34aca
|
fix: omit thinking.type=disabled for always-on thinking Claude models (#37510) | ||
|
devin-ai-integration[bot]
|
33bafd0402
|
fix(router): make prompt caching affinity aware of auto-injected cache_control (#37689)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tin-berri
|
e35ee4e5fa
|
feat(router): independent, default-on deployment affinity for the auto-router (#36146) | ||
|
Tin Chi Lo
|
fc5848174e |
fix(router): take the lowest minimum across a model group, not the highest
The read gate cannot cause a wrong pin. A deployment is only pinned when the cache already holds an entry for the prefix, and async_log_success_event writes entries against the deployment's real model rather than the group alias, so a model that will not cache a prefix never records one and there is nothing to pin it to That makes this gate purely a cheap short-circuit deciding whether the cache lookup is worth doing, so the threshold must be the lowest minimum in the group. Taking the highest skipped the lookup for a prefix a lower-minimum member had genuinely cached, losing a hit it earned, and protected against nothing. It also broke the Fable 5 direction this ticket is meant to fix: its real minimum is 512, so a group gate stuck at a higher value would skip the lookup for a prefix Fable 5 had actually cached |
||
|
Tin Chi Lo
|
25b2f83f97 |
test(router): clear the lru_cache when forcing the local cost map
get_model_info is lru_cached, so swapping litellm.model_cost is not enough on its own. An earlier test that resolved these models against the remote map, which does not carry prompt_cache_min_tokens yet, leaves cached entries without it, and the stale hit resolves to the default. The assertions would then pass for the wrong reason or fail depending on execution order Clear on teardown as well, so entries these tests warm against the local map do not leak into later tests, matching the fixture already used in test_utils.py Also pin that a wildcard route resolves the underlying model's minimum. That works only because pattern_match_deployments substitutes the real model name into litellm_params before the deployment reaches the check; without the assertion that claim is unpinned and the threshold would silently fall back to the default |
||
|
Tin Chi Lo
|
ba70189e32 |
fix(router): resolve prompt cache minimum per model instead of a flat 1024
MINIMUM_PROMPT_CACHE_TOKEN_COUNT was a flat 1024 described as "minimum number of tokens to cache a prompt by Anthropic". Anthropic's minimum cacheable prefix is per-model and ranges from 512 to 4096, and it can differ per platform for the same model, so one constant is wrong in both directions is_prompt_caching_valid_prompt gates PromptCachingDeploymentCheck, which is what optional_pre_call_checks: ["prompt_caching"] turns on. When it believes a prompt is cacheable, async_filter_deployments pins routing to whichever deployment previously served that prefix. For a prompt between 1024 and 4096 tokens on Opus 4.6, Opus 4.5 or Haiku 4.5, litellm judged it cacheable and constrained routing while the provider never cached it, so the pin cost load balancing for nothing. In the other direction Fable 5 caches from 512 tokens, so a 512 to 1024 token prefix was refused a pin it had earned The minimum now resolves from prompt_cache_min_tokens in the model cost map, which keeps it current with new models and lets the Bedrock override for Fable 5 fall out of the existing per-entry keys with no special casing. MINIMUM_PROMPT_CACHE_TOKEN_COUNT stays as a global escape hatch when explicitly set, and as the fallback for models the cost map has no entry for async_filter_deployments only ever receives the model group alias, never a model name, so it resolves the threshold from healthy_deployments instead. A group may mix models with different minimums, so it takes the max: a prompt is only treated as cacheable when it clears every member's minimum, because an unnecessary pin is the defect being fixed while a missed pin only forfeits an optimization Gemini context caching shares this gate and has the same defect; its entries are left unset so they keep today's behavior, tracked separately in LIT-4525 |
||
|
Sameer Kankute
|
3b40ac987f
|
Litellm oss 090626 (#30021)
* fix(mcp): report scoped server name during initialize (#29865) * fix mcp scoped server name * Update litellm/proxy/_experimental/mcp_server/mcp_context.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(mcp): cover scoped server name in the SSE initialize handler --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(ui): show all session logs in the drawer, not just the first 50 (#29795) * fix(ui): show newest session logs first * test(ui): keep session log pagination coverage * fix(ui): show all session logs in the drawer, not just the first page The session detail drawer fetched session logs via sessionSpendLogsCall without page/page_size, so it only ever received the backend default of one page (50 rows). Sessions with more than 50 calls had the rest unreachable in the UI (#29153). sessionSpendLogsCall now takes page/page_size, and the drawer fetches the first page, reads total_pages, then fetches the remaining pages and accumulates them before the existing client-side sort. This keeps the single continuous list (and the selected-log lookup and keyboard navigation, which all assume the full session) correct. Fetching is bounded by a page cap, and the sidebar shows a "showing most recent N" note if a session exceeds it. The rows are lightweight metadata (the endpoint excludes messages/response), so the full set is small; request/response bodies are still loaded per log on demand. * fix(ui): default session drawer to most recent log, newest first Open a session with its most recent log selected, and order the sidebar newest-first to match the all-sessions logs overview. MCP calls stay grouped last. The latest log by time is computed explicitly, since the MCP grouping means it is not always the first row. * Apply fetching pages in batches suggestion from @greptile-apps[bot] Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(ui): derive session total from accumulated rows when backend omits it Compute the session total after all pages are fetched, falling back to the accumulated row count rather than the first page's. Guards the truncation note against a backend response that omits total but spans multiple pages. --------- Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): handle Mistral multipart passthrough (#29927) * fix(proxy): handle Mistral multipart passthrough * chore: satisfy passthrough ci formatting * test(proxy): cover Mistral passthrough in CI shard * fix(vertex_ai): use REP host for context caching on eu/us multi-region endpoints (#29573) Context caching built the cachedContents URL as https://{location}-aiplatform.googleapis.com, which is an invalid host for the eu/us multi-region endpoints and returns 404. The inference path already resolves these to the REP host (https://aiplatform.{geo}.rep.googleapis.com) via get_vertex_base_url(); reuse that helper in _get_token_and_url_context_caching so caching uses the same host as inference. Adds tests covering the eu/us multi-region cachedContents URLs (v1 and v1beta1). Fixes #29571 * Support per-model encrypted content affinity config (#29760) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix: propagate upstream status code in proxy API exception handler (#29402) * fix: propagate upstream status code in proxy API exception handler When Google GenAI / Vertex returns a 404 for deprecated or missing models via streamGenerateContent, the exception was falling through to a generic handler that defaulted to 500. Now provider exceptions carrying a valid HTTP status_code correctly propagate it through to the ProxyException. * fix: apply black formatting to common_request_processing.py * fix: tighten status code range to 400-599 and deduplicate ProxyException raise * fix(tests): use valid vertex_location in context caching tests Replace "test_location" (contains underscore) with "us-central1" so tests pass the regex validation added in get_vertex_base_url(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(sdk): add xAI OAuth provider (#29866) * Add xAI OAuth provider * Update oauth.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Fix xAI OAuth CI failures * Add xAI OAuth coverage tests * Move xAI OAuth coverage tests to core utils * Address xAI OAuth review comments * Prevent xAI OAuth api_base token exfiltration * Treat blank xAI OAuth api keys as absent * Wrap invalid xAI OAuth JSON responses * Use xAI OAuth behind explicit flag --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy) #27734 allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update (#27751) * fix(proxy): allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update Fixes #27734 Sending null for budget_duration, team_member_budget, team_member_budget_duration, team_member_rpm_limit, or team_member_tpm_limit via /key/update or /team/update returned 200 OK but silently ignored the null value. The fields remained unchanged in the database. Root causes: - /key/update: prepare_key_update_data() popped budget_duration from the update dict but never re-added it (or budget_reset_at) when the value was None. - /team/update: _set_budget_reset_at() only acted when budget_duration was non-None, leaving a stale budget_reset_at in the DB. - /team/update: team_member_* null values bypassed the budget table update entirely because should_create_budget() requires at least one non-None field. * test(proxy): cover no-budget-row path in clear_team_member_budget_fields * fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes (#30028) * fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes When output_parse_pii=true on the Anthropic native path (anthropic/claude-*), response chunks arrive as raw bytes in SSE format. _stream_pii_unmasking was yielding those bytes unchanged, so <PERSON_1> tokens were never replaced with the original values before reaching the caller. Add _unmask_sse_bytes_chunk to parse each data: line, find content_block_delta / text_delta events, and apply _unmask_pii_text before re-encoding. Wire it into _stream_pii_unmasking so bytes chunks are unmasked when pii_tokens exist. * fix(presidio): handle CRLF line endings and non-ASCII PII in SSE unmask Strip trailing \r before the [DONE] guard so CRLF-terminated SSE chunks don't bypass it and silently swallow a JSONDecodeError. Add ensure_ascii=False to json.dumps so non-ASCII replacement values like accented names are preserved as UTF-8 on the wire rather than being \uXXXX-escaped. Add regression tests for both cases. * feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) (#29925) * feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) Bedrock Mantle serves the Responses API on two upstream paths: - gpt frontier models (gpt-5.5 / gpt-5.4) on /openai/v1/responses - every other Responses-capable model (e.g. gpt-oss) on the standard /v1/responses BedrockMantleResponsesAPIConfig gains a `use_openai_path` flag; the provider gate in utils.py picks the path per model: openai.gpt-* (non gpt-oss) -> /openai/v1/responses; any model declared mode=responses (price-map entry or user model_info) -> /v1/responses; everything else returns None and keeps the existing chat-completions emulation. Adds gpt-5.5 / gpt-5.4 price-map entries, registry wiring, and the routing-matrix tests. * feat(bedrock_mantle): data-driven frontier routing via use_openai_responses_path Addresses the Greptile review point that frontier detection should be a price-map field rather than a hardcoded name match. The gate now routes a model to /openai/v1/responses when its price-map entry declares use_openai_responses_path, so a frontier model whose name does not follow the openai.gpt- convention can be onboarded by JSON alone. The name-convention check is kept as a fallback that needs no price-map entry, which preserves zero-change routing for a future gpt-6 before its entry loads. gpt-5.5 / gpt-5.4 get the flag in both price maps. Adds tests for the data-driven flag path and for the flag presence on the gpt-5.x entries; both branches are mutation-tested. * test(model_prices): allow use_openai_responses_path in price-map schema The model_prices_and_context_window.json schema validator (test_aaamodel_prices_and_context_window_json_is_valid) enforces additionalProperties: false, so the new use_openai_responses_path flag on the gpt-5.5 / gpt-5.4 entries failed validation. Add it to the schema as a boolean, alongside the other supports_* / capability flags. * Add Tensormesh serverless models to the model cost map (#30037) * Add Tensormesh serverless models to the model cost map * Flag reasoning support on the Tensormesh models that expose thinking mode * fix(proxy): invalidate stale key spend counter after budget reset or manual spend update (#30001) * fix(proxy): reconcile stale key spend counter after budget reset * fix(proxy): invalidate stale key spend counter after budget reset or manual spend update * fix(proxy): remove read-time stale counter reconciliation to prevent budget bypass * revert: undo unrelated formatting changes in enterprise directory * test(proxy): add unit test for key spend update invalidating counter * test(proxy): fix mocked update_data and hash token expectations in unit test * fix(proxy): use Responses-API transformer in pass-through cost tracking (#29728) The `elif is_responses:` branch of `openai_passthrough_handler` was calling the chat-completions `transform_response` on a Responses API payload. The chat-completions transformer expects `choices: [...]` in the raw response; the Responses API uses `output: [...]` and `usage.input_tokens` / `usage.output_tokens` (not `prompt_tokens` / `completion_tokens`). The result was a KeyError 'choices' deep inside `convert_to_model_response_object`, swallowed by the surrounding `except Exception` in the handler, and the SpendLogs row was written by the fallback path with zeroed-out tokens, spend, and model. This bug silently undercounts cost for every successful pass-through call to either OpenAI's `/v1/responses` or Azure's `/openai/v1/responses` (deployments configured for the Responses API). Reproduced 2026-06-04 against a real Azure OpenAI Responses API deployment proxied through LiteLLM v1.88.0. Fix: use the dedicated `OpenAIResponsesAPIConfig.transform_response_api_response` for the Responses branch. This transformer already exists in LiteLLM (`litellm/llms/openai/responses/transformation.py`) and knows the Responses-API on-the-wire shape. `litellm.completion_cost` already handles `ResponsesAPIResponse` natively with `call_type="responses"`, so no downstream changes are needed. Tests: test_responses_api_uses_responses_transformer_not_chat_completions NEW. Real regression test — exercises the openai_passthrough_handler with a real-shaped Responses payload (no `choices`, has `output` and Responses-API `usage` keys) and NO mocked `get_provider_config`. Pre-fix: raises KeyError 'choices' inside the chat-completions transformer (the bug). Post-fix: returns a ResponsesAPIResponse, completion_cost is called with call_type="responses" and a ResponsesAPIResponse instance (asserted). Verified to fail on un-fixed handler + pass on fixed handler before commit. test_responses_api_cost_tracking UPDATED. Old test mocked `get_provider_config` (no longer called in the responses branch post-fix). Now mocks the Responses transformer directly (`OpenAIResponsesAPIConfig.transform_response_api_response`) to test the downstream cost-calc contract. Out of scope for this PR (separate followup): - Recognizing *.cognitiveservices.azure.com (the newer Azure OpenAI hostname) in the is_openai_*_route checks. Separate PR. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(skills): execute DB skills by matching the litellm_skill_ tool name prefix (#30116) Skill IDs are generated as litellm_skill_<uuid> and the model-facing tool name is the sanitized skill ID, but the post-call execution gates in SkillsInjectionHook only ran tools whose name starts with "skill_", so DB skills were silently returned to the client as raw tool calls. Fixes #28122. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): synthesize content_block_start when Responses stream omits output_item.added (#30115) * fix(team): reserve team budget raises for proxy admins on /team/update (#30030) The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a team's spend ceiling has nothing to do with the admin's own key budget. That comparison was an unintended side effect of reusing _check_user_team_limits() (which exists for the /team/new path) and broke the UI, which re-sends the unchanged budget on every save. New behavior on /team/update for standalone teams: - A team admin (already authorized via _verify_team_access) may freely KEEP or LOWER the team budget, and change models/tpm/rpm, without being gated by their personal limits. - GROWING a team's spend ceiling is a budget-authority action reserved for proxy admins -> 403 for team admins. "Growing" covers both raising max_budget above the team's current finite value and removing the cap entirely (max_budget=null, detected via model_fields_set so an explicit null is distinguished from an omitted field). For a team that currently has no cap, setting a finite value is a restriction and is allowed. - Org-scoped teams remain governed by _check_org_team_limits() (capped by the org budget). Also reverts the #29525 existing_team_max_budget workaround in _check_user_team_limits() back to the create-only form; /team/new still enforces the creator's personal caps. docs(access_control): resolve the contradiction in the team-admin section — team admins can keep/lower the budget and manage rate limits/models, but cannot raise the team budget (proxy-admin only). tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed, keep/lower/resend allowed, and unchanged create-path guards. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974) * test(ui): add a data-driven App Router migration E2E smoke Add a growing Playwright smoke for migrated pages: for each segment it deep-links to the path route, asserts the URL and that the dashboard shell rendered, then clicks off to a legacy page and asserts navigation still works. Driven by e2e_tests/fixtures/migratedPages.ts, so adding a page is one line. Runs in two situations against the same proxy: the default mount (npm run e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root). globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage state is valid under a prefix. Seeded with api-reference; append the rest as their migrations merge. * test(ui): support headed slow-motion + watch pauses in the migration smoke Honor SLOWMO in the server-root-path config (the default config already did), and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state. Both are no-ops by default, so CI behavior is unchanged. * test(ui): make the migration smoke a sidebar-click user journey Rework the smoke from deep-linking to a real navigation journey: start at the landing page, click the migrated page in the sidebar (expanding submenus for nested items), assert the path route rendered, reload it (the check a wrong server_root_path breaks), bounce to a legacy page and back, and — once two pages are migrated — navigate directly between two migrated pages. Verifies via URL + shell render, driven by the same fixture list. * test(ui): address review on the migration smoke Escape ROOT and segment before interpolating them into RegExp URL matchers so a future segment containing regex metacharacters can't silently widen the match. Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead of silently re-running the default mount and passing without exercising the prefix. * test(ui): drop unused watch helper and fix stale smoke README * test(ui): run the migration smoke under a server root path in CI * test(ui): harden + instrument the server-root-path proxy reboot in CI * test(ui): run the server-root-path migration smoke as its own CI job Replace the in-place proxy reboot in e2e_ui_testing with a dedicated e2e_ui_testing_server_root_path job that boots the proxy once with SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the config gets its own job rather than killing and relaunching the live proxy. The reboot was failing deterministically: after pkill -9 and relaunch the prefixed proxy never came back up on :4000 (connection refused), so the smoke never ran. The readiness step that was supposed to surface the cause could never reach its boot-log tail because CircleCI runs steps under bash -eo pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's exit 7. Booting the proxy as the job's own background step lets any boot crash land in that step's log instead of being swallowed. The default e2e_ui_testing job is unchanged aside from dropping the reboot, prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at the root mount there via the default Playwright config. * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232) * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through * test: mock post_call_response_headers_hook in audio speech route tests * chore(ui): remove dead App Router route stubs under (dashboard) (#30045) models-and-endpoints, organizations, and virtual-keys each had a page.tsx route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and deep links never resolve to it and the route is unreachable. Each was a thin wrapper that handed the shared view empty or no-op props (empty modelData with a no-op setModelData, hardcoded empty organizations, no-op setUserRole/setUserEmail), so reaching one would render a degraded page in any case. The real wrapper belongs in the PR that flips each page into MIGRATED_PAGES, written with eyes on it and a test This continues the dead-scaffolding cleanup from #28891. The shared components these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay, since the legacy ?page= switch in app/page.tsx and src/components still import them * fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000) * fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session * fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss * fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041) * fix(mcp): honor team access-group grants in OAuth authorize/token access check * test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation * docs(security): require a reproduction video for vulnerability reports (#30048) (#30063) With AI models capable of automated vulnerability discovery now publicly available, we expect a large increase in report volume, much of it unverified. Requiring a video of the exploit running against a live instance raises the bar for submissions and keeps triage focused on reproducible issues. Reports without a video will be closed and reopened if one is added later. Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> * feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796) * feat(ui): add admin flag to disable in-product UI nudges for everyone Admins can now suppress the survey and Claude Code feedback popups for all users via a single disable_ui_nudges UI setting, instead of relying on each user dismissing them individually. * fix(ui): suppress nudges while ui settings are loading Gate nudgesDisabled on the ui-settings loading state so an admin with disable_ui_nudges on doesn't see the survey prompt flash, and the getInProductNudgesCall fetch doesn't fire, on a cold page load before the flag resolves. Falls back to showing nudges if the fetch errors. * test(ui): wrap CreateKeyPage test in QueryClientProvider page.tsx now calls useUISettings (react-query), which needs a QueryClient that layout.tsx supplies in production but the test did not. Add the provider and mock getUiSettings so the query resolves. * chore(ui): remove dead dashboard files and unused dependencies (#30047) * chore(ui): remove dead dashboard files and unused dependencies knip flagged seven orphaned source/config files with no importers and five declared dependencies that nothing in the tree uses. Removing them shrinks the dashboard bundle's source surface and keeps the manifest honest; vite stays installed transitively via vitest, so test tooling is unaffected. * fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec (tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml workflow step still depend on it, so the redirect e2e job failed to load a config that no longer existed. * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009) * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593) Restores the reverse-lookup for the JSONL body.model fallback path so that legacy/pre-target_model_names managed files still map stripped provider IDs back to proxy aliases before auth. Also cleans up redundant `or None`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)" This reverts commit |
||
|
Mateo Wang
|
2e5ebf826f
|
fix(responses): register cooldowns on failure + fail fast on stale encrypted_content (#27820) | ||
|
Cursor Agent
|
40db114a23
|
fix(router): accept Pydantic LiteLLM_Params in encryption-boundary key lookup
Greptile flagged that the strict isinstance(dict) guard in
_encryption_boundary_key would silently return None for any non-dict input,
including a LiteLLM_Params Pydantic instance, which exposes a custom .get()
method and is intended to be used dict-style in some router paths. If such
an instance ever flowed into healthy_deployments, the guard would drop every
candidate from boundary matching and fall through to the full deployment
pool, i.e. trigger the exact invalid_encrypted_content failure this check
exists to prevent.
Loosen the guard to accept any object exposing a callable .get(): plain
dicts (the common case) and LiteLLM_Params-style Pydantic instances. The
function still returns None for non-dict-like values (None, lists, strings,
ints, bare objects).
Adds regression tests covering:
- LiteLLM_Params Pydantic instance resolves to the same boundary tuple as
an equivalent plain dict
- non-dict-like values and dicts missing required fields still return None
|
||
|
mateo-berri
|
f3b8aad883
|
fix(router): pin Responses API affinity to Azure resource on model-group switch
When a Responses API follow-up switches model_name (e.g. gpt-5.3-codex -> gpt-5.4, or to a LiteLLM-side alias of the same Azure deployment), the router has already filtered healthy_deployments to the new group, so the originating model_id is no longer present. The encrypted_content_affinity check would log "decoded deployment not found" and fall back to the full deployment pool, where simple-shuffle could land on a different Azure resource and trip a 400 invalid_encrypted_content. Fall back to pinning by the originating deployment's encryption boundary (api_base + api_key) when the model_id miss is across model groups. The encrypted_content travels with the Azure resource, not the model_name, so any deployment on the same resource accepts it. LIT-2531 |
||
|
Ishaan Jaffer
|
e8461b5b97
|
style: run black formatter on files from main merge | ||
|
Sameer Kankute
|
3a4ed48f54
|
fix(router): don't create litellm_metadata for non-Responses API calls in encrypted_content_affinity_check (#25347)
Using setdefault('litellm_metadata', {}) unconditionally created an empty
litellm_metadata key for chat completions and embeddings. This caused
_get_metadata_variable_name_from_kwargs to return 'litellm_metadata' instead
of 'metadata', so tag-based routing looked for tags in the wrong dict and
ignored all tag filters.
Fix: only set the encrypted_content_affinity_enabled flag when litellm_metadata
already exists (Responses API path). Chat completions and embeddings never have
this key, so nothing is created and tag routing works correctly.
|
||
|
Sameer Kankute
|
528daa8cf4 |
feat(router): add per-model-group deployment affinity configuration
Enable deployment_affinity, responses_api_deployment_check, and session_affinity to be configured per model group via router_settings.model_group_affinity_config, falling back to global settings for unconfigured groups. - Add model_group_affinity_config parameter to Router and DeploymentAffinityCheck - Add _get_effective_flags helper to resolve flags per model group - Update async_filter_deployments and async_pre_call_deployment_hook to use per-group config - Add 4 comprehensive tests covering per-group config, fallback, and override scenarios This allows fine-grained control of affinity behavior across model groups, e.g., enabling stickiness only for cross-provider deployments while leaving other groups free to load-balance. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
1092c17468 |
Fix flaky encrypted_content_affinity tests: mock at handler level
The 4 integration tests were flaky in CI because the AsyncHTTPHandler.post mock was bypassed when aiohttp transport is used. Mock at the higher BaseLLMHTTPHandler.async_response_api_handler level instead, which bypasses the HTTP layer entirely while still exercising router deployment selection, pre-call checks, and response post-processing (item ID rewriting). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
67e905f0d0 |
Fix flaky encrypted_content_affinity tests: clear HTTP client cache, disable retries
Tests failed intermittently in CI (-n 8 workers) because cached AsyncHTTPHandler instances from other tests bypassed the class-level mock on AsyncHTTPHandler.post, causing real requests to OpenAI with mock API keys. Router retries (default 2) masked the root cause. - Add autouse fixture to flush litellm.in_memory_llm_clients_cache before/after each test so mocks always apply to fresh clients - Set num_retries=0 on all Router instances to surface mock failures immediately instead of silently retrying Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Sameer Kankute
|
521f804350 | Fix encrypted content streaming affinity issue | ||
|
Sameer Kankute
|
2bc4da76ce | Update the tests | ||
|
Sameer Kankute
|
18bf3f2df6 | Fix mock github test | ||
|
Sameer Kankute
|
a88a17796b | Fix logging and encrypted content extraction | ||
|
Sameer Kankute
|
9f627c67d8 | Add tests for encrypted_content_affinity | ||
|
Sameer Kankute
|
394c49d303 | Add tests for encrypted_content_affinity | ||
|
Harshit Jain
|
456d8f5524 | feat: add session_id to have better routing | ||
|
Emerson Gomes
|
3dd55a7b61 | Merge main into affinity_callback and address deployment affinity review feedback | ||
|
Emerson Gomes
|
f7726c8950 | Fix wrong keys being used for model sticky entry | ||
|
Emerson Gomes
|
059e75ad88 |
fix(router): scope deployment affinity by model_map_key
- Key affinity by (user_api_key_hash, model_map_key) -> model_id - Ignore OpenAI 'user' param for affinity - Avoid double hashing user_api_key_hash - Add unit tests + docs clarifications |
||
|
Emerson Gomes
|
e23bd21a28 | Add deployment affinity routing | ||
|
Sameer Kankute
|
f804ab6de5
|
Add LLM provider response headers to Responses API (#16091)
* Add llm headers to responses api * fix mock test |
||
|
Ishaan Jaff
|
76f1064229
|
[Bug Fix] litellm incompatible with newest release of openAI v1.100.0 (#13728)
* fix imports OpenAI SDK * ResponseText fixes * fixes ResponseText * fix imports * catch AttributeError * fix import * use openai==1.100.1 * fix build from PIP * fix lint test * Print OpenAI version * fix Install dependencies |
||
|
Ishaan Jaff
|
39955129f5
|
fix mapped tests (#12320)
* fix - use flush llm client cache
* faster mapped tests
* test_async_multiple_response_ids_routing
* fix tests
* test_ateam_member_update_admin_requires_premium
* regular mapped tests
* Revert "Fix: Initialize JSON logging for all loggers when JSON_LOGS=True (#12206)"
This reverts commit
|
||
|
Krish Dholakia
|
ef42461c1e
|
Litellm fix GitHub action testing (#11163)
* test: add __init__.py files * refactor: rename test folder to avoid naming conflict * test: update workflows * test: update tests * test: update imports * test: update tests * test: remove unused import * ci(test-litellm.yml): add pytest retry to github workflow * test: fix test |