mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
11554 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
ryan-crabbe-berri
|
7745fe887f | Merge branch 'litellm_window_spend_writer' into litellm_window_spend_reader | ||
|
ryan-crabbe-berri
|
041cae8280 |
fix(proxy): bound the window spend seed exclusion to the batch's own start time
request_id can be chosen by the client through x-litellm-call-id, so an unbounded NOT (request_id = ANY(batch)) let a replayed old id drop that id's historical LiteLLM_SpendLogs row from the one-time seed while its increment still landed. The increment now carries the request start, the batch keeps the earliest one, and the seed only excludes ids whose startTime is at or after it. |
||
|
ryan-crabbe-berri
|
124d08d592 |
perf(proxy): read budget-window spend from the maintained window table
Per-window budget enforcement aggregated LiteLLM_SpendLogs on every cold counter and on every 5s authoritative floor check. SpendLogs has no index on api_key or team_id, so each check range-scanned the highest-volume table. Reads now hit the LiteLLM_BudgetWindowSpend row by primary key and only fall back to the aggregate when no row exists for the window being enforced. A row is current when its window_start is at or past the caller's expected start, so a pod holding a stale reset_at trusts a window another pod already rolled instead of summing the previous window back in. window_duration is threaded from the budget_limits entry through to the read rather than parsed back out of the counter key. The reader never writes rows. |
||
|
ryan-crabbe-berri
|
187e0fab60 |
feat(proxy): maintain per-window budget spend rows in the spend writer
Multi-window budgets (budget_limits on keys and teams) enforce off Redis counters with a 60s TTL. Every cold counter, and every authoritative floor check, aggregates LiteLLM_SpendLogs with a range scan over startTime; that table has no index on api_key or team_id, so the scan lands on the highest volume table in the schema and saturates the connection pool (#35766). Every other budget feature reads a maintained running spend value instead. This gives windows the same shape by keeping LiteLLM_BudgetWindowSpend up to date: one row per configured window, whose window_start rolls forward in place. The cost callback already iterates a key's and a team's windows with the window start computed and the actual cost in hand, so it enqueues there, onto a new WindowSpendUpdateQueue. Increments are enqueued even when the cache increment is skipped for a reserved counter, since the reservation only pre-charged an estimate and the row still owes the actual cost. Windows with no reset_at slide with wall clock and cannot be represented by a single row, so they are left to the read path's existing aggregate. The queue flushes alongside the daily spend queues, through the Redis buffer when one is configured (only the pod-lock winner commits) and directly otherwise. A flush selects the primary keys that already exist, seeds the ones that do not from LiteLLM_SpendLogs so a new row cannot undercount spend that predates it, and applies one batch of upserts ordered by primary key. An increment at or behind the stored window_start adds into the row, matching how in-flight requests carry into a window after a reset; a newer one rolls the window and starts from that increment. The conflict arm adds only the increment, never the seeded base, so two pods seeding the same new window cannot double count it. The seed excludes the requests its own batch is about to apply. Spend logs are drained by a separate monitor that fires on a ~2s poll whenever anything is queued, while window increments flush on the much slower batch tick, so by seed time the batch's log rows are normally already in the table; counting them in the aggregate and again in the increments made a fresh row land at exactly twice the true spend. Each increment therefore carries the LiteLLM_SpendLogs request_id it was recorded under, which update_database now returns rather than having the callback re-derive it (a cache hit appends time.time() to that id, so a second derivation would not match). The reset job rolls each expired window's row alongside the counter it zeroes, conditional on the stored window_start still being behind the new one so a pod that already rolled it is not clobbered. |
||
|
Mateo Wang
|
1798d9d2a9
|
Merge pull request #35834 from BerriAI/litellm_fix_cursor_variant_budget_bypass
fix(proxy): enforce per-model budgets against resolved cursor model variants |
||
|
ryan-crabbe-berri
|
f4538679c0
|
fix(proxy): apply key_alias/key_hash filters to all /key/list visibility branches (#35840)
* fix(proxy): apply key_alias/key_hash filters to all /key/list visibility branches The filters previously lived only in the own-keys OR branch, so a team admin's admin-team branch matched every team key and the Key Alias filter in the Virtual Keys UI appeared broken. Both filters are now global AND conditions alongside team_id/project_id/access_group_id/agent_id, narrowing every visibility branch while leaving unfiltered visibility unchanged. * chore: drop new explanatory comments flagged by review * chore: restore schema.d.ts to base enum order |
||
|
yucheng-berri
|
58ead7f653
|
fix(azure_storage): honor AZURE_STORAGE_ENDPOINT_SUFFIX for sovereign clouds (#35806)
The azure_storage logging callback and the azure blob files backend built every storage URL against the hardcoded commercial host, so an Azure Government account was unreachable with no way to override it. Read AZURE_STORAGE_ENDPOINT_SUFFIX (default core.windows.net) once in AzureBlobStorageLogger and derive the Data Lake and Blob hosts from it, so all seven previously hardcoded sites follow the configured cloud. Parse stored blob URLs with urlparse instead of matching the commercial host, so URLs persisted before the suffix was configured still resolve, and pin the resulting host-validation boundary with tests. |
||
|
Deepanshu Lulla
|
e2950a8995
|
fix(router): eagerly fetch Vertex AI deferred stream to surface HTTP errors in _acompletion fallback path (#34627)
* fix(router): eagerly fetch deferred stream to surface HTTP errors in fallback path
Providers like Vertex AI and Bedrock defer their HTTP call until the first
__anext__ on the returned CustomStreamWrapper (completion_stream=None,
make_call set). Errors raised inside __anext__ (e.g. 429, 503) escape the
_acompletion try/except block, so fail_calls is never incremented, deployment
cooldown does not fire, and the standard fallback chain is bypassed.
Call fetch_stream() on the wrapper before delegating to
_acompletion_streaming_iterator when completion_stream is None and make_call
is set. Any HTTP error now propagates through _acompletion's except block,
increments fail_calls, and enters the normal retry/fallback chain.
Strip Content-Length, Transfer-Encoding, Content-Encoding, and Content-Type
from exception headers at the same point to prevent HTTP framing mismatches
when LiteLLM builds its own error response body.
Add a re-raise guard in _acompletion_streaming_iterator (async and sync paths)
so MidStreamFallbackError with already-generated content re-raises to the
caller instead of silently injecting a continuation prompt into a fresh request
to a fallback model.
Apply logging cleanup in async_function_with_fallbacks_common_utils: use
%s-style formatting and exc_info=True instead of f-strings with
traceback.format_exc().
* fix(router): undo success_calls on deferred-stream fetch failure; broaden header strip
* fix(router): extract header-strip helper to keep _acompletion under strict C901 threshold
* test(router): add unit tests for _strip_http_framing_headers to satisfy router coverage gate
* test(router): add sync _completion_streaming_iterator re-raise test for mid-chunk MidStreamFallbackError
* fix(router): restore Fallbacks context in no-fallback log; document update_team mcp_rpm_limit
The log and debug message when no fallback model group is found was missing
the Fallbacks list, making it hard to understand why routing failed.
Also adds the missing mcp_rpm_limit documentation to update_team to fix
the documentation_test_api_docs CI check.
* fix(router): preserve original traceback in deferred stream fetch error re-raise
Using bare `raise` instead of `raise fetch_err` keeps the full inner
traceback from fetch_stream() intact so the error origin is visible in
logs and debuggers without being anchored to this line.
* style(test): restore black-style formatting in test_router.py
An earlier commit on this branch collapsed the file's pre-existing
multi-line formatting into single lines while adding the deferred-stream
tests, producing a diff full of unrelated reformatting noise. Restores
the untouched code to its original formatting; the actual new/changed
test content is unaffected (verified via AST comparison).
* fix(router): re-raise mid-stream fallback on any generated content, not just text
The re-raise guard added for MidStreamFallbackError only checked
generated_content, which tracks text deltas alone. A stream that emitted a
tool-call or reasoning-only chunk before failing had generated_content=""
despite already streaming to the client, so the router silently retried
and the client saw duplicated/inconsistent output. The guard now also
inspects the wrapper's raw chunks for tool_calls/reasoning_content.
Also moves the deferred-stream HTTP-framing-header stripping out of
Router._acompletion into the proxy's _handle_llm_api_exception: Router is
used directly as an SDK as well as by the proxy, and stripping headers
there dropped legitimate provider metadata (content-type,
proxy-authenticate) for direct SDK callers who never see the proxy's own
response construction.
schema.d.ts regenerated via make pre-commit; unrelated to this change.
* test(router): add direct coverage for _stream_chunks_have_generated_content
CI's router_code_coverage check flags any router.py function never referenced
by name in a test file; the new helper was only exercised indirectly through
the mid-stream re-raise guard tests.
* revert(ui): drop incidental schema.d.ts regeneration
Committing router.py/common_request_processing.py touched
pre_commit_lint.sh's litellm/proxy trigger for the API-type-sync check,
which force-regenerated schema.d.ts even though neither file changes any
route or model. The regenerated ordering of two unrelated Union/enum
fields (stream_timeout, user_role) isn't stable across process
invocations even against completely unmodified backend code (confirmed
by regenerating twice against the pre-existing committed code and getting
the same diff both times), so this reverts to the original committed
file rather than chase non-deterministic output.
* fix(proxy): strip framing headers on the pre-existing ProxyException branch too
_handle_llm_api_exception filtered framing headers into a local `headers`
dict, but for an exception that's already a ProxyException, it merged
{**e.headers, **headers}: the original e.headers came first, so a framing
header present there but absent from the filtered `headers` (because it
was just stripped) was never overwritten and survived into the response
unfiltered. Filters the merged result instead of relying on the merge
order to do it implicitly.
* chore: retrigger CI (no GitHub Actions check-suite was created for the previous two pushes)
* fix(router): detect thinking_blocks as generated content in mid-stream guard
Greptile flagged that a thinking-only delta (Anthropic extended thinking,
Delta.thinking_blocks) wasn't recognized as already-streamed content, so
a stream that emitted only thinking blocks before failing could still
restart via fallback and append an unrelated response after content the
client already received.
* fix(proxy): strip browser-facing security headers from provider exceptions too
veria-ai flagged that the framing-header denylist still let a malicious or
misconfigured provider set browser-facing headers (Access-Control-Allow-Origin,
Content-Security-Policy, Clear-Site-Data, etc.) on the proxy's own error
response. Adds a dedicated _BROWSER_SECURITY_HEADERS set alongside the
existing framing one and strips both wherever provider exception headers
reach the client response.
* refactor(router): address maintainer review mechanicals
- List[ModelResponseStream] -> list[ModelResponseStream] in
_stream_chunks_have_generated_content (ruff UP006 strict-budget gate)
- drop _strip_http_framing_headers and its 3 tests: the proxy inlines the
filter directly now, so the helper has had no production caller since
the header-stripping was moved out of Router
- move HTTP_FRAMING_HEADERS/BROWSER_SECURITY_HEADERS/
UNSAFE_PROXY_RESPONSE_HEADERS from router.py into litellm/constants.py,
removing the router.py <-> proxy import path the two CodeQL
cyclic-import alerts were pointing at
- move the eager fetch_stream() call before success_calls/logging/
_track_deployment_metrics instead of incrementing then compensating
with a manual decrement on failure
- fix a dead assert message: `mock_fallback.assert_not_called(), "..."`
built a tuple, not an assert-with-message; assert_not_called() already
raises on its own so this just drops the inert string
* revert(router): pull mid-stream continuation-removal out of this PR
Removing the continuation-prompt fallback (retrying with the partial
response as a prefixed assistant message) so a stream failing after
partial content always re-raises instead was a scope decision beyond
what this PR's title/issue (#31874) describe, and it directly conflicts
with #30242/#30743, which are already fixing the same code path for
Anthropic's removal of assistant-message prefill on Sonnet 4.6+/Opus
4.6+. Landing this PR's version first would delete the branch those PRs
are patching; landing theirs first would have this PR undo their fix on
rebase.
Restores the original prefill-based continuation-resume behavior
(including the is_pre_first_chunk guard already in litellm_internal_staging)
in both _acompletion_streaming_iterator and _completion_streaming_iterator,
and removes _stream_chunks_have_generated_content along with the tests
that only existed to cover the guard. This PR now only touches the
deferred-stream eager-fetch fix and the header-stripping fixes; the
non-text-content re-raise idea becomes a follow-up PR built on top of
whichever of #30242/#30743 lands.
* fix(proxy): re-filter unsafe headers after the response-headers hook merge
_handle_llm_api_exception filtered provider/framing headers once, then
merged in post_call_response_headers_hook's return value afterward
without re-filtering. The ProxyException branch happened to re-filter
after its own header merge, but the HTTPException/httpx.HTTPStatusError/
generic-exception branches passed the post-hook headers straight through
unfiltered, so a callback hook (any custom guardrail/logging plugin)
returning an unsafe header would bypass the strip entirely for those
paths. Filters once, right after the hook merge, so every branch gets
the same guarantee.
* Revert "revert(router): pull mid-stream continuation-removal out of this PR"
This reverts commit c5ca101f61746a9b12a480c4bc48d95fc0c69f8d.
* fix(router): detect reasoning_items as generated content in mid-stream guard
Greptile flagged that a structured reasoning-only delta (Delta.reasoning_items,
the OpenAI Responses-API-style reasoning item) wasn't recognized as
already-streamed content by _stream_chunks_have_generated_content, alongside
the existing thinking_blocks/tool_calls checks, so a stream that emitted only
reasoning_items before failing could still restart via fallback.
* fix(router): annotate _stream_chunks_have_generated_content with Sequence, not list
The type_discipline_gate LIT001 check flags mutable-collection parameter
annotations. chunks is only iterated, never mutated, so Sequence is the
correct read-only annotation and clears the ratcheted budget ceiling.
* fix(router): surface original provider exception, not the internal wrapper, when mid-stream fallback gives up
When content has already streamed and MidStreamFallbackError carries
original_exception (e.g. RateLimitError), both the async and sync
streaming iterators bare-re-raised the wrapper itself, so the client
lost the specific error type/code/provider_specific_fields instead of
seeing the real provider error. The fallback-failure path a few lines
below already unwraps to original_exception for the same reason; apply
the same pattern here.
Also extend _stream_chunks_have_generated_content to recognize audio,
images, and annotations deltas as generated content, matching
is_chunk_non_empty's existing annotations check and Delta's treatment
of audio/images as first-class content fields — a stream carrying only
one of these before failing was not recognized as already-streamed,
so the router could still restart it via fallback after the client had
received real content.
* chore: retrigger CI (frontend-lint cancelled, schema.d.ts flake)
frontend-lint's check-run shows conclusion=cancelled on
|
||
|
ryan-crabbe-berri
|
9ea5cfce0e
|
fix(proxy): persist periodic reload schedule state so status survives restarts and fires without store_model_in_db (#35165)
* fix(proxy): persist periodic reload schedule state so status survives restarts and fires without store_model_in_db The model cost map and Anthropic beta headers reload schedules kept their last-run time in a per-pod module global, so GET /schedule/*/status reported last_run null after any restart and the Admin UI showed the reload as never having run. The reload check also only ran from the add_deployment job, which is registered only when store_model_in_db is true, so config-file deployments stored a schedule that never fired. Persist last_run_at and reload_requested_at as dedicated columns on LiteLLM_Config, owned by the reload job and manual reload endpoints, while the schedule endpoints own the param_value JSON (interval_hours); no writer can clobber another's fields. Serve status entirely from the row. Register the check as its own periodic_reload_job outside the store_model_in_db gate. Replace the force_reload boolean with a reload_requested_at timestamp each pod compares against its own in-memory last reload, so a manual reload reaches every pod exactly once instead of being cleared by the first poller. Run the blocking fetches via asyncio.to_thread, and stamp last_run_at with update_many so a schedule cancelled mid-poll is not resurrected. * fix(proxy): compare reload requests against pod data age seeded at boot A pod that had never reloaded kept its in-memory clock at None, and with no interval configured nothing ever set it, so every manual reload request was ignored by every pod except the one serving the click (Greptile P1 on the previous commit). Seed the per-pod timestamp at boot as the time its data was loaded and reload whenever a request or the interval is older than that, which also removes both None special cases from the due predicate. A schedule whose row has no last_run_at fires on the next tick so the first run does not wait a full interval. * fix(proxy): scope reload persistence to the model cost map and seed the pod clock from the actual load time Revert the Anthropic beta headers reload path to its previous JSON-flag implementation so this PR only changes the price data reload; the beta headers path keeps working exactly as before and can migrate to the shared module in a follow-up. The unused columns on its config row are inert. Seed model_cost_map_loaded_at from the timestamp get_model_cost_map records at the actual import-time fetch instead of ProxyConfig construction time, closing the startup window where a manual reload request stamped between the fetch and the constructor compared as older than the pod's data and was skipped (Greptile P1 on the previous commit). * refactor(proxy): drop the legacy force_reload backfill from the reload tracking migration The backfill only carried over a manual reload clicked in the seconds before an upgrade, and every upgrade restarts the pods, which re-fetch the cost map at import and so already deliver what that request asked for. Removing it makes the migration schema-only, so prisma db push and prisma migrate deploy leave the database in the same state instead of diverging on a data statement that only one of them runs. * fix(proxy): stamp reload timestamps at the precision they are stored at Postgres stores these columns as TIMESTAMP(3) while Python stamps microseconds, so a pod comparing its in-memory clock against the persisted copy of the same instant read as newer and skipped the reload request it had just recorded. Truncate every stamp to milliseconds at the source, and floor the boot seed the same way, so the in-memory value and its persisted copy compare exactly. * fix(proxy): identify manual reloads by revision instead of comparing timestamps Comparing a request timestamp against each pod's data age made correctness depend on clock resolution: Postgres stores TIMESTAMP(3) while Python stamps microseconds, and two events inside the same millisecond are indistinguishable no matter how the comparison is written. Replace reload_requested_at with a reload_revision counter the manual reload endpoint increments atomically in the database. Each pod records the revision it last applied and reloads whenever the row's differs, so a request reaches every pod exactly once regardless of clock skew or precision, and concurrent requests publish distinct revisions instead of overwriting one another. A pod adopts the current revision on its first poll, since data it loaded at boot already satisfies any earlier request. Interval reloads still key off the pod's own data age, where hour scale comparisons make precision irrelevant. * fix(proxy): seed the applied reload revision at startup A pod adopted whatever revision it found on its first poll, so a manual reload published while the pod was starting was marked applied without ever being served and the pod kept the prices it fetched at import. Read the row once at startup instead, right after that fetch, and treat a missing row as revision 0 * style(tests): revert incidental reformatting of test_proxy_server.py An earlier ruff format run reflowed the whole file from its 88-column formatting, adding ~1150 lines of churn unrelated to this PR. Replay only the real test changes onto the original formatting * fix(proxy): serve an outstanding reload request on a booting pod Seeding the applied revision at startup left a window: a manual reload published after the import-time cost map fetch but before startup read the row was marked applied without ever being fetched, stranding that pod on stale prices when no interval was configured. A pod now starts unapplied and serves any outstanding request on its first poll, which costs one redundant fetch per boot and removes the window along with the seeding step * fix(proxy): accept a reload interval still encoded as JSON text param_value is written with safe_dumps, and a raw row read can return it decoded or as a string depending on the driver. Strict validation rejected the string, so the schedule read as disabled and an admin's configured reloads silently stopped. Mirrors the guard ConfigRepository.get_param already carries for the same column * fix(proxy): cancel a reload schedule without resetting the revision * fix(proxy): null the interval in JSON so cancelling keeps the revision prisma rejects a null literal for a Json? column, so update_many writes an interval-less object instead. The fake config table now rejects the same input the database does, which is what the live run caught and the mock did not. Also records the run before adopting the revision, so a failed status write leaves the request unserved for the next poll rather than reporting a run that never landed. * fix(ui): match the CI-generated user_role union order in schema.d.ts |
||
|
yuneng-jiang
|
e64536c425
|
test(e2e): retry provider-transient statuses at the transport with bounded backoff (#35824)
* test(e2e): retry provider-transient statuses at the transport with bounded backoff The Anthropic passthrough cost test failed a full-suite run on a real 529 overloaded_error. Passthrough routes forward provider responses verbatim and bypass the router's num_retries, so provider blips reach the harness only on those paths. Following standard practice, the retry is scoped to the dependency boundary instead of rerunning tests: only the enumerated transient statuses (500/502/503/504/529, the set production SDKs retry by default) are retried, with bounded exponential backoff and a printed line per retry so flakiness stays visible in run logs. 429 is deliberately excluded: the quota suites assert the proxy's own rate-limit and budget 429s, and a transport that absorbed them would break those tests. Network errors and timeouts are not retried either, so a hang surfaces as a hang. request_with_retry takes injected callables, and the new harness tests pin the contract with protocol fakes, no monkeypatching * test(e2e): narrow the transport retry to 529, the one status the proxy cannot emit Greptile's review is right that status-only classification could absorb an intermittently failing proxy: at the transport a 500/502/503/504 from the proxy is indistinguishable from one it relayed, and the proxy is the system under test. 529 is the only status litellm provably never originates (Anthropic's overload signal, forwarded verbatim on passthrough) and the only transient observed across the full-suite runs, so the set shrinks to exactly that. The canary tests now also pin 500/502/503/504 as never retried |
||
|
mateo-berri
|
1cd481d4f2 | fix(proxy): enforce per-model budgets against resolved cursor model variants | ||
|
Mateo Wang
|
27885076e7
|
Merge pull request #35738 from BerriAI/litellm_bedrock_tool_choice_parallel_conflict
fix(bedrock): drop conflicting tool_choice.type when toolConfig.toolChoice is set |
||
|
Mateo Wang
|
5159cba6b0
|
Merge pull request #35807 from BerriAI/litellm_enforce_final_variables
feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) |
||
|
yuneng-jiang
|
e86f2209a4
|
test(e2e): move load/perf testing out of the main suite and drop the vllm passthrough test (#35820)
The Locust throughput SLO test is a different testing category from functional e2e (variance-driven, historically flaky, currently skip-annotated against LIT-5119) and erodes trust in the suite as a release gate; it comes out of the default collection along with its exclusive plumbing (locustfile, load-mock registration fixtures, run_chat_load). Re-implementation as its own pipeline is tracked in LIT-5163. The weekly session-anomaly test never ran in the suite (opt-in via E2E_WEEKLY_ANOMALY, driven by its own workflow) and stays, as do the markerless aggregation unit tests. The vllm passthrough test read-times-out (60s) against the shared vllm-cpu backend in every run on the per-SHA e2e stack; it is removed until LIT-5164 establishes whether that is backend capacity or a passthrough defect. Its registry cells return to the gap list, which is the honest state |
||
|
mateo-berri
|
72983e28c0 |
fix(lint): exempt the runtime-settable config surface in litellm/__init__.py from LIT010
Module-level names in litellm/__init__.py are the SDK's documented config
surface: users assign litellm.api_key and friends directly, and the proxy
rebinds them via setattr from litellm_settings. The package ships py.typed,
so the Final sweep made every such documented assignment a mypy error
("Cannot assign to final name") in downstream codebases. Strip Final from
the module scope of that file, keep it on function locals, and teach LIT010
that the config surface's module scope is exempt so the gate stays green
without suppression comments
|
||
|
mubashir1osmani
|
ad79b314c5
|
test(e2e): cover legacy text /completions endpoint (#34431)
* test(e2e): cover legacy text /completions endpoint The /completions (and /v1/completions) text-completion route had zero e2e coverage despite being the second-busiest endpoint in production; everything 'completions' in the suite was chat. Add a text-completion endpoint test that registers an OpenAI instruct deployment, drives /v1/completions through the gateway, and asserts real generated text. Adds text_completions() + the completion request/result models to EndpointsClient, the 'completions' endpoint to the coverage registry vocab, and the registry cell. * test(e2e): assert /v1/completions choices shape, not just joined text Assert the response carries a choices array and the first choice has real text, so a malformed response (no choices) and a clean-but-empty completion are distinct failures. Drop the unused text property / id / model fields (model only what the test reads). |
||
|
mubashir1osmani
|
dcb4e5033c
|
test(e2e): vendor API strategy coverage across endpoints (#34649)
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure * test(e2e): expand vendor API strategy coverage across endpoints Adds validation cases on existing endpoint suites, plus vector stores, search, bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches contract, and chat stream SSE. Registers coverage cells for LIT-4778 * test(e2e): finish vendor strategy open items Audio transcription negatives, vector-store file attach/poll/search, OpenAI moderation category matrix across chat/messages/responses, and smoke model matrix for chat (LIT-4778) * test(e2e): harden vendor strategy suite against live env edges Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing, vector store list/search models, responses validation, and provider-denied Bedrock paths so the suite is stable against a live proxy * test(e2e): rename suites, drop vendor_contract, fix greptile gaps Move shared status helpers into e2e_http, rename chat auth headers and chat security suites, remove vendor_contract and dev_config files_settings, and tighten transcription validation plus vector-store search assertions * test(e2e): route bedrock stream disconnects through e2e_http Catch mid-stream RequestException in the shared harness so bedrock native tests do not import requests directly * fix(e2e): address greptile and veria review on vendor strategy suite Store search tool keys as os.environ refs and resolve them in SearchAPIRouter. Tighten validation helpers and assertions so 5xx/empty/unrelated failures no longer pass coverage cells * fix(e2e): drop search_api_router os.environ expansion from vendor suite Keep the PR test-only. Search tools register without an api_key so the proxy falls back to its own PERPLEXITY/TAVILY env, same pattern as a2a. * test(e2e): drop search e2e suite from vendor strategy PR Remove the /v1/search coverage file and its registry rows so this PR no longer carries search endpoint testing. |
||
|
mateo-berri
|
2708620d6a | feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) | ||
|
devin-ai-integration[bot]
|
355ae9989b
|
fix(proxy): propagate user_email and bind api_key on JWT auth attribution paths (#34331)
* fix(proxy): propagate user_email and bind api_key on JWT auth paths Standard JWT auth built UserAPIKeyAuth with user_id but never user_email, and the first auto-registered request early-returned a key with token set but api_key unset, so spend-log attribution logged user_api_key_user_email and user_api_key_hash as null. Bind api_key to the token hash on the auto-registered key, copy user_email from the resolved user object on both the standard and auto-register JWT paths, and warn when enable_jwt_auth/litellm_jwtauth are placed at the config top level where they are silently ignored. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover misplaced top-level JWT config warning Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan <ryan@berri.ai> |
||
|
mateo-berri
|
9eeff06263 | Merge origin/litellm_internal_staging into litellm_lit4395_cursor_agent | ||
|
yuneng-jiang
|
5ac1edcd59
|
fix(e2e): make spend-counter redis connection env-driven for non-cluster deployments (#35732) | ||
|
yuneng-jiang
|
6b3d4f2380
|
feat(ui): add admin-configurable user banner (#35729)
* feat(ui): add admin-configurable user banner Proxy admins can publish a markdown announcement that renders as a dismissible banner on every dashboard page for all authenticated users, editable from Admin Settings > UI Settings without a redeploy. Backed by new /get/user_banner and /update/user_banner endpoints persisting to the existing LiteLLM_UISettings table * fix(ui): re-surface dismissed banner on identical republish Stamp a server-side revision on every banner update and fold it into the client dismissal signature, so unpublishing and republishing the same message reaches users who dismissed the earlier run * fix(ui): stamp banner revision as an opaque uuid instead of a counter Two overlapping admin updates could read the same prior revision and both persist the same incremented value, letting an identical republish collide with a previously dismissed signature. A server-generated uuid per update makes every publication identity unique by construction with no read-modify-write * refactor(ui): drop the server-side banner cache Reads go straight to the single-row table; the dashboard already throttles fetches client-side, so the cache only added staleness windows under concurrent updates and multiple workers * refactor(ui): move banner storage behind a domain repository and drop the store_model_in_db gate UserBannerRepository owns the row shape instead of the endpoint reaching through the generic .table bridge, and publishing no longer depends on the unrelated STORE_MODEL_IN_DB flag; a connected database remains the only requirement |
||
|
Ahmed N
|
368dd0be5b
|
fix(groq): translate web_search_options to the browser_search tool (#34971) | ||
|
devin-ai-integration[bot]
|
a625d1e1ca
|
feat(otel): stamp service tier attributes on inference spans (#35679)
* feat(otel): stamp service tier attributes on inference spans Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): bound requested service tier to known values The requested tier is caller-controlled and reaches the span verbatim, so an arbitrary string lands on every litellm_request span on success and on failure. A 100k character value was stamped uncapped; safe_set_attribute does not truncate and no span limits are configured. Apply KNOWN_REQUEST_SERVICE_TIERS in get_requested_service_tier so both the span attribute and the Prometheus label bound the value the same way. The served tier stays unrestricted since it comes from the provider, so a tier a provider adds later is still reported. Prometheus label behavior is unchanged. * fix: derive known service tiers from the ServiceTier enum The allowlist omitted "fast", which litellm models as a real tier and prices through the priority cost key, so a request naming it resolved to no tier on the span and no Prometheus label. Deriving the set from ServiceTier keeps the two in sync, so a tier added there for cost calculation cannot go missing here. Behavior change: a request with service_tier "fast" now carries the tier on the span and on the Prometheus service_tier label, where it previously resolved to none. Every other value resolves as before. * refactor: build the known service tiers without a mutable intermediate The set comprehension and set literal tripped LIT002, which bounds mutable collections. Concatenating tuples keeps the derivation from ServiceTier while every intermediate stays immutable; the resulting frozenset is unchanged. --------- Co-authored-by: milan <milan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yucheng Zhu <yucheng@berri.ai> |
||
|
ryan-crabbe-berri
|
abe3289398
|
fix(proxy): retry model cost map fetch with Retry-After-aware backoff and keep current map on reload failure (#35739)
* fix(proxy): retry model cost map fetch with Retry-After-aware backoff and stop downgrading to the packaged backup on reload failure A 429 or transient network error during a manual or scheduled model cost map reload used to silently replace litellm.model_cost with the stale backup JSON bundled in the installed wheel, stamp the reload as successful, and clear the force_reload flag, so a fleet could serve months-old pricing until the next interval. Runtime reloads now go through refetch_model_cost_map, which retries 429/5xx/transport errors up to 3 times honoring Retry-After (capped at 30s, exponential backoff with jitter otherwise) and returns a failure value instead of the backup when the fetch or integrity validation fails. On failure the pod keeps its currently loaded map, the periodic job leaves last_run and force_reload untouched so it retries on the next config poll, and the manual endpoint returns 502 with the reason instead of reporting a fake success. Startup behavior is unchanged: boot still falls back to the packaged backup since there is no previously loaded map to keep. * fix(proxy): use shared async httpx client for cost map reload and make retry tests CI-env-proof The reload fetch now goes through get_async_httpx_client with a dedicated httpxSpecialProvider.ModelCostMap pool instead of constructing a raw httpx.AsyncClient, so it inherits deployment-level TLS and transport settings and passes the ensure_async_clients gate. Tests inject a MockTransport-backed client through the same seam. An autouse fixture clears LITELLM_LOCAL_MODEL_COST_MAP, which CI exports and which short-circuited the retry tests; the two TestPriceDataReloadAPI tests and the config sync pubsub reload test that still patched get_model_cost_map now patch refetch_model_cost_map instead. |
||
|
Classic298
|
c9887a1f94
|
perf: build log messages lazily so filtered-out log records cost nothing (#35703) | ||
|
tin
|
d39c557743 |
fix(bedrock): drop conflicting tool_choice.type when toolConfig.toolChoice is set
Converse rejects a request that carries both toolConfig.toolChoice and an
additionalModelRequestFields.tool_choice.type, so any request that pairs
parallel_tool_calls with an explicit tool_choice 400s with "The additional field
tool_choice/type conflicts with the existing field toolConfig.toolChoice.auto".
That pairing is what agentic clients send by default; Codex CLI sends
tool_choice "auto" and parallel_tool_calls false on every turn, so tool calling
was broken outright on Bedrock models that advertise
supports_parallel_tool_use_config.
Drop the type from the Anthropic passthrough once toolChoice carries it, and keep
disable_parallel_tool_use, which has no toolConfig equivalent and is accepted
alongside toolChoice. Measured against Bedrock directly: toolChoice plus
{disable_parallel_tool_use} succeeds for auto, any and tool, while an empty
tool_choice with no toolChoice is rejected for a missing type, so the type still
has to be emitted when the caller sends no tool_choice.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Classic298
|
042ef4cc48
|
perf: install hiredis so redis-py parses replies with its C parser (#35709) | ||
|
tin-berri
|
22f68c0c6b
|
fix(spend): read what a request cost from the record instead of pricing it again (#35736)
The auto-router savings driver recomputes what the served request cost, but that request is not a counterfactual: it ran, and the cost calculator already billed it and wrote the number down. Recomputing means restating every pricing dimension the biller applied, and the two this missed were enough to halve it. A request billed at a priority tier is recomputed at standard rates, and a regional host's uplift is dropped entirely, so the driver writes a savings figure into the same rollup row as the `spend` it disagrees with. On `gpt-5.4-mini` at priority the row is billed 0.024 and the driver prices the same usage at 0.012. Neither omission cancels between the two arms, because both are per-model. The uplift is a multiplier read off each model's own entry, so 1.1*A - 1.1*B is 1.1*(A-B) and a model without one does not move at all. Tier coverage is sparser and asymmetric: `gpt-5.6` has priority rates and `gpt-5.4-nano` has none. `cost_breakdown` already carries the answer and already reaches the call site. The cost calculator records it, it rides the standard logging payload into the spend log's metadata, and OTEL, the log drawer and the response headers all read it rather than re-deriving; this driver was the only downstream consumer in the tree still pricing a completed request from its tokens. `input_cost` and `output_cost` sum to exactly what the pricer returns, so the served arm reads them. Tool spend, discount and margin stay out, since the counterfactual cannot be priced with them and charging them to one arm alone would read as the router losing money on every tool call. The baseline never ran, so it is still priced through the cost engine, now on the basis the biller used. `CostBreakdown` carries that basis because it cannot be recovered afterwards: the tier the biller used comes from `optional_params`, which no log record keeps, and the served tier that does survive on the usage object is a different fact with the opposite precedence. Rows written before this shipped carry no basis and price at standard rates, exactly as they do today; there is no backfill. Two smaller things in the same path. The router is passed as a provider rather than a router, so a spend write that was never auto-routed no longer fetches and discards one, and the complexity router resolves its messages once per hook instead of once per consumer. |
||
|
tin-berri
|
9e3a8df6c0
|
feat(spend): add net auto-router savings to the cost-optimization dashboard (#35521)
* feat(spend): add net auto-router savings to the cost-optimization dashboard The dashboard credited compression and prompt caching but said nothing about the optimization that picks the model, so the driver with the largest lever on a bill was the one an operator could not see. Savings are the counterfactual: without a router a deployment runs one model, and it has to be one that can carry the hardest request, so the baseline is the priciest model in the router's hardest configured tier. A cheap tier is a choice the router made, not a ceiling it was bounded by. `auto_router_savings_baseline_model` overrides it for operators who would genuinely have run something else. Both are provider-qualified before pricing, because a bare name can resolve to a different vendor's rates or to nothing at all, and a deployment is priced by its `base_model` where it has one, which is how Azure deployments are priced everywhere else. Both arms price the request's real usage through `generic_cost_per_token` rather than re-deriving per-token arithmetic, so tiered rates, ephemeral cache-write tiers and regional uplifts stay consistent with what was actually billed. `prompt_tokens` already includes the cache buckets, so charging them again at the input rate would price the same tokens twice. Cache state is what makes this hard. The baseline serves every turn, so whether it had the prompt cached is whether the conversation was already underway. On a continuing conversation it wrote the prompt earlier and would only read it now, so this request's write is what switching cost and counts against the saving. On a first turn nothing was cached for any model, the baseline would have written the same prompt, and both arms carry the write at their own rates. Charging the write to both cases understates a first turn to a few percent of its value, and because the write premium is fixed by prompt size while the saving grows with completion length, it can render a profitable route as a loss. That shape is read off the conversation rather than remembered: a second human ask means an earlier turn was served. No cache, no session id, and no dependence on a caller sending a session header. It cannot see a switch on a turn the router did not classify, and it reads a few-shot prompt's synthetic turns as prior conversation; both err toward charging the write, which under-claims. The baseline and the shape ride on the existing `routing_decision` record, which is already carried from the router to the spend log, already classified for redaction, and already written-or-cleared per attempt. A fallback that re-enters the hook therefore cannot leave either fact behind to be attributed to a deployment that never routed, and no new metadata key crosses the trust boundary. The result is signed. Whether a switch pays off is a race between the rate gap and the cache-write cost, and a narrow gap loses; flooring at zero would hide exactly the routing behaviour an operator needs to see. The donut plots only drivers that saved, while the card and range total keep the sign. Savings accrue into a new `autorouter_savings_spend` column on the six daily rollup tables, declared `NotRequired` because rows queued by a pod on the previous release carry no such key. It is summed by the rollup merge the cross-pod Redis drain also runs, and carried through the aggregation query, the per-row accumulation and the response model, so the dashboard reads a value the API actually sends. Tests enumerate the drivers from the response model itself and assert each is summed, accumulated, carried and totalled, so one added later cannot be half-wired. * fix(spend): let the baseline pay for a continuing turn's own growth `_baseline_usage` moved every cache-creation token into the baseline's read bucket whenever the conversation was underway. That is right for a switch, where the baseline never left the model it was on and really would only read, but wrong for a turn that stayed put: the prompt grew, and the tokens written are that growth. They are new to every model, so the baseline would have paid to write them too. Forgiving it that write made the counterfactual cheaper than it was and shrank the reported saving on ordinary steady-state traffic, by about 2% per turn. The selected arm was never involved; it has always been priced on the real usage. The error sat entirely on the baseline. The condition is that the request read more than it wrote, not that it read anything. A switch onto a model already holding a small prefix of this prompt still writes most of it, and that write is the switch's own cost; keying off a nonzero read would have handed such a request the full rate gap, turning +$0.0056 into +$0.1177. Comparing the two buckets separates a warm continuation, which reads far more than it writes, from a cold arrival, which does the reverse, and it leaves the existing invariant intact: a request reading 0 and one reading 1 both still land in the same place. * fix(spend): price each arm under the key litellm billed it, and see agent turns Two ways the savings number read the wrong thing, both from identifying a model by its name when the name is not what it costs. The counterfactual was ranked and priced on the public rate for the model a deployment names. A deployment may not be charged that rate: the router registers its configured prices under the deployment's own id and deliberately keeps them off the shared model-name key so deployments sharing a backend model do not pollute each other. So a hardest-tier deployment configured above its public rate lost the ranking to a cheaper candidate, and once chosen was priced at a rate nobody pays. Which key prices a deployment is now `_select_model_name_for_cost_calc`'s decision, the resolver the real request is billed through, rather than a second rule here that would have to re-learn that per-second and tiered overrides count, that a partial override still counts, and that a deployment configured at zero is priced at zero rather than treated as unpriced. The arm being subtracted had the same fault and a sharper edge. It priced the spend log's `model`, which on Azure is the deployment name, absent from the cost map, so the whole driver silently read zero for that traffic. It no longer re-derives anything: `model_map_information.model_map_key` is what litellm actually billed the request under, recorded at request time by that same resolver with `base_model` and custom pricing already applied. Separately, the conversation-shape discriminator counted human asks, and an agent loop can run twenty turns on one of them. Its tool traffic rides `tool_result` blocks on user turns that flatten to empty text, and `tool` roles that are never read, so a long agentic conversation looked like its own first turn and was handed the arithmetic that leaves the cache write on both arms. That is the one direction this must never fail in, because it inflates. An assistant turn is the direct evidence that something answered earlier, and it is blind to how the tool plumbing is spelled on either surface. * fix(spend): give the cost-key resolver both inputs the selected arm needs The served model was resolved through one input at a time, and each choice broke the half the other fixed. `model_map_key` is the served model already resolved through `base_model`, which is the only way an Azure deployment name reaches the cost map at all; without it the selected arm priced a name absent from the map, returned nothing, and the whole driver silently read zero for that traffic. But it is built without `router_model_id`, so it never carries a deployment's own price overrides, and a custom-priced deployment was compared at its public rate while the baseline used the real override. On a deployment configured well above its public rate that inverted the answer outright: a route that lost $21.88 reported saving $0.10. `_select_model_name_for_cost_calc` takes both, so it gets both. Which key prices a deployment stays its decision rather than a rule restated here. * fix(spend): same model is only the same cost when it is the same deployment The short-circuit compared resolved model identity, so two deployments of one model collapsed to "no switch" and reported zero. They are not the same cost: a deployment can carry a negotiated rate, and routing from the dear one to the list-price one is a real saving the dashboard reported as $0.00 against a true $21.93. Both arms now carry the key litellm prices them under, so the comparison is between deployments rather than between names. * refactor(spend): price from resolved rates, not from a name we keep re-resolving Four review rounds landed on one mechanism: which identifier prices a deployment. base_model, then the deployment id, then cache-only overrides. Each round added a clause to a resolution rule that should not exist, and a wrong primitive fails once per input shape, so each shape arrived as its own finding. `Router.get_deployment_model_info` already owns this. It merges a deployment's configured prices over the built-in map, folds in `base_model` defaults for deployments whose name is not a model, and falls back to the model name when nothing is overridden. Every shape hand-rolled here (cache-only, partial, per-second, Azure) was that function re-implemented badly. `generic_cost_per_token` now accepts already-resolved rates instead of demanding a name it looks up itself, which is what forced the name-bending in the first place. Both arms resolve through the owner and pass what they got: the counterfactual by the deployment the router would have used, the served request by the deployment that served it. The invented cost-key resolver is gone, and `Baseline` carries a deployment id rather than a key we chose on litellm's behalf. Net 64 insertions against 79 deletions. * test(spend): follow _most_expensive onto the router that prices its candidates Ranking moved through `Router.get_deployment_model_info`, since what a deployment costs is the router's answer to give; these four cases were still calling the old free-function signature. * fix(spend): rank baseline candidates by what a request costs, not by two rates "Most expensive" was decided by comparing output rate then input rate. That is a property of a rate, not of a request: a deployment dearer per output token can be cheaper per cached token, so the comparison ordered cache-heavy traffic backwards and recorded the wrong counterfactual. Candidates are now costed on one reference request through the same engine the savings themselves use, which leaves cache read and write rates, tiered tables and every other billing dimension to that engine rather than to another rule restated here. The reference request is cache-heavy because auto-routed traffic is. * fix(spend): pick the baseline against the request that ran, not a stand-in for one Ranking happened in the pre-routing hook, where the request has not executed yet, so candidates were costed against a hard-coded reference workload: 20k prompt, 19k of it cached, 1k out. Which candidate is dearest depends on that mix, so a pooled hardest tier holding a deployment with non-proportional configured rates could be ranked for a request nothing like the one served. The mix is known on the spend path, so the ranking belongs there. The routing decision now carries the tier's candidates rather than a winner already chosen, and the baseline is resolved against the usage that actually happened. The reference workload is gone; nothing here assumes a traffic shape any more. The router is passed in rather than imported from `proxy_server` inside the computation, so the savings stay a pure function of their arguments and the caller owns where the router comes from. That also makes the spend path testable without a running proxy, which the previous shape was not. * refactor(spend): measure savings against one configured model, not a derived one The counterfactual was derived per request: enumerate the hardest tier's deployments, resolve each one's effective pricing, price them all, take the dearest. That machinery produced a review finding per input shape it had not anticipated, and every answer it gave was one an operator could have stated in a line of config. So they state it. `litellm_settings.autorouter_savings_baseline_model` names the model the traffic would have run on without a router, for every auto-router on the proxy, and unset means the driver is off rather than a model nobody named being guessed at. `savings_baseline.py` and its tests are deleted outright, along with the tier enumeration, the candidate list on the routing decision, and the per-deployment override that shadowed it. Cache-state handling is untouched: the baseline is still priced on this request's own read and write split, so a switch still pays for re-warming the cache and a first turn still charges the write to both arms. 45 insertions against 482 deletions. * refactor(router): compute the conversation shape once and pass it down `_classify_and_route` re-derived it from the messages the hook had already resolved, so an ordinary routed request walked the turn list twice for one boolean. The hook computes it and hands it over, which is also where the affinity-hit path already got it from. Also moves `_get_llm_router` below the imports it sat among. * fix(router): drop the dead conversation_continuing parameter off the hook It was added to `async_pre_routing_hook` by mistake and immediately overwritten by the value the hook computes, so it never did anything. It also widened a signature every pre-routing strategy shares with the protocol in `types/router.py`, leaving this one router diverged from `AutoRouter` and the interface for no reason. Also records why an unreadable request counts as continuing: no messages is no evidence a turn was served, so it pays the cache write and under-claims rather than being handed a first turn's larger saving on nothing. * fix(spend): charge a baseline its input rate for cache buckets it cannot price A model with no cache_creation_input_token_cost, which is every OpenAI, Azure and Gemini entry, resolved that rate to 0.0 and carried the whole written prompt for free, so a first turn routed onto a cheaper model reported a loss. Same hole on cache reads. Those tokens are plain input on such a model, so they move into the text bucket. * refactor(spend): build the daily upsert payloads in one shot `common_data` and `update_data` were constructed and then appended to: `request_id` conditionally for tag rows, `endpoint` unconditionally a few lines later. A dict that grows after its literal cannot be reasoned about by reading the literal, which is the whole point of building it at once. The conditional key resolves to a spreadable value before either payload, so both are single expressions and the tag branch appears once instead of twice. Not wrapped in MappingProxyType, though it was suggested: these go straight to prisma, whose query builder branches on `isinstance(value, dict)` to tell a nested node from a scalar. A mappingproxy is a Mapping but not a dict, so it falls through to the serializer and raises `TypeError: Type <class 'mappingproxy'> not serializable` inside the batch upsert, where the surrounding except would log it and leave the rollups silently unwritten. * fix(spend): keep the one-shot upsert payloads under the type-discipline budget Building both payloads as single literals traded a mutation for two dict literals, and LIT002 counts construction rather than mutation, so the change the review asked for is the one the gate charges for. The empty branch is the avoidable half: it is the same value every time, so it moves to a module constant built once instead of a literal per transaction, and it is a read-only mapping so none of the call sites that spread it can fill it in later. |
||
|
Mateo Wang
|
f60e99c583
|
Merge pull request #34531 from BerriAI/litellm_forward_client_headers_responses_api
fix(responses): forward client headers to the provider on /v1/responses |
||
|
yucheng-berri
|
2d1f650e9a
|
fix(guardrails/rubrik): attribute blocked requests to the caller that made them (#35734)
The block event Rubrik receives sourced caller identity from model_call_details[metadata], where the enriched litellm metadata never lives; it sits under litellm_params. Every block therefore reported user_api_key_hash as an empty string, so a security block could not be traced to a key, user, or team. Read identity off the authenticated UserAPIKeyAuth the failure hook is already handed, via the same mapper the success path and the proxy spend logger use, so a block log and a success log describe their caller with an identical key set. |
||
|
tin-berri
|
cb8c734dbe
|
fix(ui): reject an auto-router keyword rule left empty instead of dropping it (#35705)
"Add keyword rule" seeds a row with no keywords, and the only check that a rule carried one lived inside getSemanticConfigError, which returns early when semantic keyword matching is off. Off is the default, so an unfilled row fell through to serializeKeywordTierRules and was discarded on the way to the payload; the create reported success and the rule was gone. The row now reports the gap itself and the submit is withheld while one is outstanding, on the create form and the edit modal alike, both reading emptyKeywordTierRuleIndexes so the row named and the row marked cannot differ. Enter commits a typed keyword: the dropdown is kept closed, which left antd nothing for Enter to select, and submitting was what used to supply the blur that saved the word. The backend already refused such a rule, but only when the router built the deployment, so a caller that sent one anyway got the row written, dropped on reload, and a 500. The management write paths now parse the incoming complexity_router_config with the router's own ComplexityRouterConfig, judged on the config alone so a patch that writes one without naming a model is covered too, and reject it with a 400 having persisted nothing. |
||
|
tin-berri
|
ec9acf7362
|
fix(bedrock): stop forwarding no-op toolSpec.strict to Converse (#35688)
* fix(bedrock): stop forwarding no-op toolSpec.strict to Converse `strict: false` is the Chat Completions default, so sending it to Bedrock Converse communicates nothing the provider does not already assume, while Bedrock rejects the key by presence rather than by value: any Claude model routed through its Anthropic-compatible validator 400s with `tools.0.custom.strict: Extra inputs are not permitted`. The existing `bedrock_converse_supports_strict_tools` gate only protects models whose `model_prices_and_context_window.json` entry carries the flag, which makes every newly released Claude model broken by default until someone adds it. That is a losing race for a field that carries no information when false, and it is unrecoverable from the client side on `/v1/responses`, where the Responses to Chat Completions bridge stamps `strict: false` onto every function tool even when the caller never sent one. `drop_params` cannot help there because the caller never supplied the param. Drop the key when falsy instead. `strict: true` still honors the per-model gate, so models that accept strict schemas keep the behavior they have today and the flag keeps doing its job for the values that actually mean something. * fix(bedrock): flag Claude Sonnet 5 as rejecting toolSpec.strict Bedrock routes Sonnet 5 through the Anthropic-compatible validator that rejects `toolSpec.strict`, but its six pricing-map entries never got `bedrock_converse_supports_strict_tools: false`, so the gate fell back to forwarding for Anthropic models and every tool call carrying `strict: true` 400'd. Verified live in us-east-1: before this, `strict: true` against `us.anthropic.claude-sonnet-5` returns `tools.0.custom.strict: Extra inputs are not permitted`; after, it returns a real tool call. Measured the rest of the family the same way rather than trusting the map: Sonnet 4.5, Sonnet 4.6 and Haiku 4.5 all accept `strict: true`, and Opus 4.8 already carries the flag. Sonnet 5 was the only entry where the map disagreed with the provider, so it is the only one changed here. Same shape as the Opus 4.7/4.8 and Sonnet 4 fixes before it. |
||
|
yuneng-jiang
|
965968e052
|
ci(circleci): install a pinned Rust toolchain on the Linux jobs (#35519)
* ci(circleci): install a pinned Rust toolchain on the Linux jobs The cimg/python images have no Rust toolchain, so every Linux job that runs `uv sync` or `uv build` builds litellm-rust through maturin with no cargo on PATH. maturin's puccinialin helper then fetches rustup-init from the unversioned /rustup/dist/ path with no checksum and provisions a floating `stable` toolchain, so the compiler a job builds with drifts with whatever upstream published that day. uv hides build-backend output on a successful sync, so none of this shows up in the job log. Add an install_rust command that mirrors the Windows job: download a pinned rustup 1.28.2, verify its SHA-256 against rust-lang's published sidecar, install toolchain 1.97.1 with the minimal profile, and export ~/.cargo/bin through BASH_ENV. Run it after install_uv in every job that builds the workspace; upload-coverage only runs `uv tool run coverage` and is left alone. Net download cost is unchanged, since puccinialin was already pulling a rustup and a toolchain in each of these jobs. * test(ci): guard that no CircleCI job builds the workspace without a pinned Rust A green CI run does not notice the gap this closes: uv suppresses build-backend output on a successful sync, so a job that syncs with no cargo on PATH silently gets maturin's own unpinned rustup and a floating toolchain, and the log looks identical either way. Pin the invariant statically instead. Every job and reusable command is walked in step order, and reaching a `uv sync` / `uv build` without a Rust toolchain provisioned first is a failure. install_rust and the Windows job's inline pinned install both satisfy it, so a new job that forgets one is named in the assertion message at PR time. Separate cases cover install_rust's own pins: a versioned /rustup/archive/ URL, a SHA-256 verified before the installer is executed, and an exact toolchain version rather than a channel name. * ci(circleci): provision Rust for base_sdk_install base_sdk_install landed on staging while this branch was open. It runs `uv build --wheel` on cimg/python:3.12 behind install_uv alone, so it built the bridge with maturin's own unpinned rustup. The guardrail added here caught it on the merge result, which is the case it exists for. |
||
|
yuneng-jiang
|
cd87fee9c5
|
feat(team): custom metadata validation hook for team create and update (#33353)
* feat(team): custom metadata validation hook for team create and update
Operators can point general_settings.custom_team_metadata_validate at an
async Python function that validates team metadata before /team/new,
POST /team/update, and PATCH /team/{team_id} commit their writes. The
hook receives the metadata that will actually be written (the merged
result on PATCH) plus the stored metadata and requester context, and
fails closed: a rejected value returns the function's own message as a
400 while any exception or timeout blocks the write with a configurable
generic message as a 503. Premium-gated like enforced_params.
* fix(team): validate metadata before model alias writes and strip system keys from validator input
Review follow-ups on the team metadata validation hook: run the validator
before the model_aliases table insert so a rejected create leaves no
orphaned model rows, strip system-managed keys from existing_metadata so
the validator sees symmetric input on both fields, and accept class
instances exposing an async __call__ as validators. Adds a three-way
validator implementation matrix (allowlist function, HTTP-service-backed
function, immutability-enforcing class instance) driven through the real
create, update, and patch endpoints, including an HTTP stub service and
outage coverage.
* test(team): run the metadata validation matrix against the DB-backed proxy in CI
Adds the validator matrix to the proxy_store_model_in_db_tests CircleCI
job so every scenario runs full e2e against a Postgres-backed proxy. The
proxy config registers a dispatching validator that routes each request
to one of the three implementations via a metadata key and accepts
anything that does not opt in, keeping the rest of the suite unaffected.
CI starts a stand-in cost center service on the host for the HTTP-backed
implementation, reached from the container via host.docker.internal, and
the outage path targets a closed port to prove the fail-closed 503
without stopping services.
* feat(ui): edit team metadata as key-value pairs in team create and edit forms
The team create and edit forms asked for metadata as a raw JSON blob in a
textarea buried under Additional Settings. Both forms now render a key-value
pair editor directly under the TPM/RPM limit fields, backed by a shared
MetadataKeyValueFields component. Values round-trip losslessly: non-string
values display as JSON and parse back to their typed form on save, and
JSON-ambiguous strings are quoted so their type survives the trip. The edit
form hides UI-managed keys (logging, guardrails, model rate limits, etc.)
that dedicated controls already own and re-add on save.
* fix(ui): explain typed JSON parsing in the team metadata help text
* feat(team): schema-driven metadata fields from team_metadata_schema config
* refactor(team): render schema metadata fields as locked key-value rows, drop allowed_values
* refactor(team): schema fields reduce to key and label, tag-rendered keys, clean rejection toasts
* refactor(ui): prepopulate declared metadata keys as ordinary key-value rows
* fix(team): let non-admin dashboard users read the team metadata schema
* test(proxy): pin timeout wiring, boundary, and error-message contracts for team metadata validation
* fix(proxy): use pooled async httpx client in the e2e team metadata validator example
* refactor(team): satisfy staging lint ratchets inherited by the merge
|
||
|
devin-ai-integration[bot]
|
ba1bde70e4
|
feat(guardrails/rubrik): prompt moderation, response-text blocking, streaming buffer, failure logging (#35722)
* feat(guardrails/rubrik): prompt moderation, response-text blocking, streaming buffer, failure logging (#34019) * feat(guardrails/rubrik): add prompt moderation, response-text blocking, streaming buffer, failure logging - Add `pre_call` prompt moderation via `/v1/before_prompt/openai/v1` webhook: structured messages are flattened and sent before the LLM is called; blocked prompts surface a `ModifyResponseException` with the refusal text. - Extend `post_call` response moderation to cover assistant text in addition to tool calls; text blocks (wholesale replacement) are distinguished from tool-block explanations (appended) via `startswith` diffing. - Add `streaming_end_of_stream_only = True` and `streaming_buffer_until_moderated = True` so streamed responses are withheld until end-of-stream moderation passes (requires litellm >= BerriAI/litellm#31389; older versions fall back to detect-only). - Add `_MalformedToolBlockingResponseError` for structurally invalid service responses; `_guarded` logs at CRITICAL so operators notice misconfiguration. - Add `max_queue_size = 10_000`, `_enforce_max_queue_size`, and drop-oldest backpressure so a webhook outage cannot grow the retry queue unboundedly. - Add `flush_queue` override that snapshots once for both send and drain, preventing duplicate delivery on concurrent flush calls. - Make `_log_batch_to_rubrik` re-raise on error so `flush_queue` preserves undelivered events for the next retry. - Add `async_post_call_failure_hook` to log blocked requests (`ModifyResponseException`) with a best-effort fallback payload for prompt blocks (where no `standard_logging_object` exists yet). - Add `_correlation_id` / `_apply_correlation_id` / `_prepend_system_prompt` helpers; `_prepare_log_payload` now applies them for all providers (not just Anthropic) so every log correlates by `litellm_call_id`. - Add `get_supported_event_hooks` classmethod advertising `[pre_call, post_call]`. - Use dedicated `httpx.AsyncClient` (`moderation_client`) for webhook calls with explicit pool limits, separate from the shared logging client. - Drop module-level `rubrik_handler` singleton (inappropriate for a library). - Update `initialize_guardrail` docstring to explain `pre_call` vs `post_call` mode. - Update tests: rename `tool_blocking_client` → `moderation_client`, `tool_blocking_endpoint` → `response_moderation_endpoint`, `_flush_task` → `_periodic_flush_task`; migrate `TestExtractBlockedTools` to `TestExtractResponseBlock` for the new combined text+tool block API; add tests for prompt moderation, text blocking, streaming flags, and failure payload construction. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * test(guardrails/rubrik): add tests to reach 100% coverage 50 new tests across 18 classes covering previously-untested paths: - Prompt moderation: passthrough, block, no-messages skip, message flattening (content-list → string), payload construction with tools/user/correlation_key/litellm_call_id fallback, refusal extraction - async_post_call_failure_hook: non-matching exception no-op, missing stash warning, valid stash → enqueue, AttributeError in payload build, flush exception handling - Block payload building: standard_logging_object present vs fallback path, missing start_time - async_log_success_event: _rubrik_blocked=True skip path - aclose: task cancel + moderation_client.aclose() - Edge cases: sampling rate clamp warning, unknown input_type passthrough, empty-inputs early return, model_call_details warning, _stash_block_context, duck-typed tool-call normalization, request_data["tools"] preference over optional_params, system-prompt exception handler, flush-at-batch-size, enqueue exception swallowing, queue empty/lock-None guards, non-dict JSON response TypeError Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): use get_async_httpx_client, ruff format - Replace bare httpx.AsyncClient with get_async_httpx_client (required by ensure_async_clients_test; avoids per-request client creation) - aclose() calls close() (AsyncHTTPHandler interface, not aclose()) - ruff format on rubrik.py and guardrail_hooks/rubrik/__init__.py - Update 3 tests for AsyncHTTPHandler type (isinstance check, close()) osv-scan and documentation CI failures are pre-existing on the base branch and unrelated to this PR. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): fix UP006 strict ruff violation get_supported_event_hooks return type used List[...] (UP006) instead of list[...]. Replace with the built-in generic and remove the now-unused List import from typing. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): fix 3 reportArgumentType basedpyright violations Use `# pyright: ignore[reportArgumentType]` (not `# type: ignore`) to suppress the three errors basedpyright reports in --outputjson mode: - convert_content_list_to_str call (dict vs AllMessageValues) - _apply_correlation_id call (StandardLoggingPayload vs dict[str, Any]) - _prepend_system_prompt call (same) Also tighten _apply_correlation_id and _prepend_system_prompt signatures from bare `dict` to `dict[str, Any]`. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): don't close shared HTTP client in aclose() moderation_client and async_httpx_client both come from LiteLLM's global HTTP-client cache (get_async_httpx_client keys on llm_provider + params). Two RubrikLogger instances with the same parameters share the same underlying AsyncHTTPHandler object. Calling close() in aclose() closed the shared connection pool for all instances, breaking any subsequent moderation request on other loggers. aclose() now only cancels the periodic flush task and lets LiteLLM manage the shared client lifecycle. Tests updated to assert close() is NOT called. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): use Counter for duplicate tool-call ID detection Set-based comparison lost ID multiplicity: two original tool calls with the same ID both appeared "allowed" even when the service returned only one (e.g. one allowed + one prohibited sharing an ID). Replace with Counter so returned_id_counts[id] >= required_id_counts[id] must hold for every ID. Matches the approach in the original _extract_blocked_tools. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): respect default_on=true when omitted from config LitellmParams.__init__ converts an omitted default_on to False before initialize_guardrail receives it, so litellm_params.default_on is always bool and never None. The is-None guard in RubrikLogger.__init__ therefore never fired on the proxy path, leaving prompt/response moderation inactive for any config that omitted default_on. Fix: read the raw guardrail dict (before LitellmParams coercion) to distinguish an explicit `default_on: false` from the absent-means-True default. When the key is absent from the raw config, default_on=True is used; when it is explicitly set (either True or False), that value wins. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * style: ruff format rubrik.py after Counter import addition Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): detect ID-less tool call removal; fix UP045 ID-less tool calls (tc.id is falsy) were excluded from required_id_counts, so the Counter comparison never caught their removal. Add a cardinality check (len(returned) < len(original)) that fires on any removal regardless of ID presence, combined with the Counter check for duplicate-ID attacks. Also fix 5 UP045 violations (Optional[X] → X | None) introduced by our new code against the daily-branch baseline. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): filter optional_params through ModelParamHelper in fallback payload _build_fallback_payload forwarded the raw optional_params dict as model_parameters. optional_params can contain extra_headers, api_key, and other upstream provider credentials that must not reach the Rubrik webhook. The normal standard_logging_object path already filters through ModelParamHelper.get_standard_logging_model_parameters(), which allowlists only safe LLM API parameters. Apply the same filter here. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): scope failure hook by guardrail_name; moderate text-completions Guard async_post_call_failure_hook by guardrail_name so multiple Rubrik instances don't cross-log: the failure hook is called for every registered callback; without the check the first instance pops the stash and the originating instance finds None and silently skips logging. Now each instance only handles blocks raised by itself. Also moderate /v1/completions prompts: _moderate_prompt returned early when structured_messages was absent. For text-completion requests litellm supplies inputs["texts"] with no structured_messages. Added a fallback that synthesises a user-message from texts so the before_prompt webhook can evaluate text-completion prompts. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(lint): add reason comments to pyright: ignore suppressions type-discipline budget requires each # pyright: ignore[...] to carry an explanatory comment. Add reasons to the three bare suppressions on lines 483, 651, 652. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): include tool-call arguments in prompt moderation _flatten_messages_for_moderation only sent the content field, silently dropping tool_calls[].function.arguments and function_call.arguments. An attacker could embed prohibited text in tool-call arguments inside assistant history turns and bypass prompt moderation entirely. Now collects all attacker-controlled text per message: text content via convert_content_list_to_str, plus all tool_calls[].function.arguments and the deprecated function_call.arguments, joined with newlines before being sent to the before_prompt webhook. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): tighten append detection to prevent prefix bypass startswith(sent_content) allowed any replacement whose text shares the original as a prefix (e.g. "Hello" → "Hello, blocked.") to be classified as a tool-block append rather than a text block, bypassing detection. Use startswith(f"{sent_content}\n\n") to require the exact two-newline separator the webhook uses between original text and appended tool-block explanations. Also add `returned_content != sent_content` to text_blocked so an unchanged passthrough is never classified as a block. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(guardrails/rubrik): default_on=False when omitted (follow existing pattern) Remove the custom raw-dict lookup that was defaulting default_on to True when omitted from the guardrail config. Follow the standard litellm convention: omitted resolves to False (users must explicitly opt in with default_on: true). - initialize_guardrail: pass litellm_params.default_on directly - RubrikLogger.__init__: is-None guard defaults to False not True - Test updated to assert the correct False default Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * chore(rubrik): keep the ported guardrail within staging lint budgets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: credit the original author of the rubrik guardrail work Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: keep this mirror PR's diff limited to the rubrik files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
c98d595359
|
fix(proxy): redact credential headers from request logging copies (#35678)
* fix(proxy): redact credential headers from request logging copies clean_headers preserves an Anthropic subscription OAuth token, and other client-supplied provider credentials, so they can be forwarded upstream. The same dict was also stored as proxy_server_request["headers"] and metadata["headers"], so those credentials reached every logging callback and the SpendLogs proxy_server_request column that the Admin UI logs page renders. Build the observability facing copies through redact_credential_headers, and drop the transport-only keys (provider_specific_header, headers, api_key) from the request body snapshot since they have to keep the real values. * fix(proxy): use the redacted header copy in the request debug log The stdout secret filter matches Bearer and sk- shaped values, so an MCP auth token printed by the request-header debug line survived it in cleartext. * fix(proxy): resolve the configured MCP auth header name through the secret manager get_secret_str also consults a configured secret manager, so a deployment that stores the header name there now gets that header masked too. Drops the added comments in favour of a named constant. * perf(proxy): resolve the MCP auth header name once per process get_secret_str issues a blocking secret-manager SDK call when one is configured, and configured_credential_header_names runs on every proxied request. * fix(proxy): read the MCP auth header name live, cache only the secret manager The config reloader rewrites os.environ on an interval and after /config/update, and MCPRequestHandler resolves the same setting per request, so caching the env lookup left a renamed header logged in the clear until the process restarted. Only the blocking secret-manager call stays cached. * refactor(proxy): narrow header redaction to the reported credential set Drops the MCP header-name resolution, its per-request config and secret-manager lookups, and the x-mcp- prefix rule. Those cover a separate credential family than the one this ticket reports and carried their own config-reload staleness surface; they belong in their own change. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
7dab1ff75f
|
fix(datadog): read team callback dd_* params from kwargs instead of blocked dynamic params (#35115) (#35687)
Team-scoped DD credentials (dd_api_key, dd_site) set via POST /team/{id}/callback were silently dropped because _request_blocked_callback_params blocks them from standard_callback_dynamic_params. The security block is correct for request-level injection, but team callback_vars are admin-configured and trusted.
Store the raw init kwargs on the Logging instance and read dd_* params from there in _process_dynamic_callback_list instead of from standard_callback_dynamic_params.
Adds an integration test that exercises the full Logging.__init__ flow with team callback_vars to prevent regression.
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
|
||
|
tin-berri
|
8ad5d144a1
|
feat(complexity_router): default session affinity off and expose it in the UI (#35714)
* feat(ui): expose an Auto-Router session affinity toggle session_affinity on ComplexityRouterConfig defaults to True, and neither the create form nor the edit modal ever emitted the key, so every auto-router built in the UI silently pinned each session to its first turn's model for an hour with no way to see or change that. Adds an "Advanced: Session Affinity" switch to both surfaces, defaulted on to match the backend field. Both paths now write the key explicitly instead of falling through to the backend default, so a stored config states what the router actually does. A stored config with the key absent hydrates as on, since those routers are running with affinity enabled today; showing them as off would report the opposite of reality and persist it on the next save. * feat(complexity_router): default session affinity off and expose it in the UI session_affinity defaulted to True and the Auto-Router UI never emitted the key, so every router built there silently pinned each session to whatever model its first turn classified into for an hour, refreshed on every hit. There was no way to see that from the UI and no way to change it without hand-editing config.yaml. The default flips to False, so every turn is classified on its own merits and lands on the cheapest adequate tier. Pinning is now opt-in. The toggle added in the previous commit follows the field: it renders off, and both the create tab and the edit modal keep writing the key explicitly, so a stored config states what the router does instead of inheriting a default that can move under it. Behavior change for existing routers: those created before this have no session_affinity key stored, so they pick up the new default and start reclassifying every turn. That gives up the provider prompt cache the pin was preserving, and a multi-turn session can now change model between turns. Set session_affinity: true to keep the old behavior. |
||
|
Yassin Kortam
|
8cf2e2eb43
|
fix(proxy): apply key/team router_settings.model_group_alias (#35486)
Key and team `router_settings.model_group_alias` was accepted, persisted and echoed back by `/key/info`, but never applied at request time, so the request ran on the group the caller asked for. `route_request` forwards only the settings the Router accepts as per-request kwargs, and `model_group_alias` is not one of them: the Router resolves aliases from its own instance attribute, which holds the global config map and is shared across requests. Resolve the alias in the proxy instead, alongside the existing model-alias rewrites and ahead of the pre-call hooks, so per-model limits and guardrails key off the group that actually serves the request. Authorize the alias target before the rewrite; model access was checked against the requested group, so a key whose alias points at a group it cannot call gets the usual 403 rather than being quietly served it. Resolves LIT-4879 |
||
|
yuneng-jiang
|
9d5984b358
|
refactor(ui): rename the create MCP server component to PascalCase (#35686)
Pure rename, no behavior change. create_mcp_server.tsx and its test move to CreateMCPServer, the two importers and one stale e2e comment follow, and the local/filename-pascal-case suppression drops now that the file passes the rule on its own. The rename is scoped to this one component rather than the whole directory because three PRs are currently open against its snake_case siblings; the rest can follow once those land. |
||
|
Yassin Kortam
|
66bc70365f
|
fix(caching): close evicted LLM clients so their connections are reclaimed (#35492)
An evicted client was left for the garbage collector, but every OpenAI/Azure SDK client is a reference cycle, so nothing freed the client or its pooled TCP connections until a generational sweep ran. Driving 2000 azure calls through the official image with no forced collection, live clients and open sockets climbed from 202 to 1361 while the cache stayed at its 200-entry bound, and RSS grew 279 MB to 456 MB against a TLS upstream. Closing on eviction is what caused the earlier 'Cannot send a request, as the client has been closed' regression, so an evicted client litellm created is now closed only once a grace window has passed, by which point any request that was already holding it has finished. A client the caller supplied is never closed, since litellm does not own its lifecycle. Resolves LIT-4883 |
||
|
ryan-crabbe-berri
|
46b6eae799
|
feat(teams): apply default organization to new teams from default team settings (#35540)
* feat(teams): apply default organization to new teams from default team settings Adds organization_id to DefaultTeamSSOParams so proxy admins can pick a default organization in Default Team Settings. new_team applies it before org validation whenever a team is created without an explicit organization_id, so API, Admin UI, SCIM, SSO, and team upsert creations all inherit it and go through the same existence and org-limit checks. Explicit organization selections win and existing teams are untouched. The default is validated at save time (PATCH /update/default_team_settings returns 400 for an unknown org) and at create time, where a missing org now surfaces as a clean 400 instead of a 500 by routing OrganizationNotFoundError into the previously dead org_table None guard. The Admin UI Default Team Settings tab gets a Default Organization row backed by the shared OrganizationDropdown. * fix(teams): validate org limits against final team state including defaults Applies default_team_params and the legacy max_budget fallback before the organization validation block, so _check_org_team_limits sees the values the team will actually be persisted with. Also loads the org's budget table in the lookup; without include_budget_table every budget comparison in _check_org_team_limits was skipped because litellm_budget_table was None. * test(proxy_behavior): pin org team limits as enforced on /team/new The dead-code pins existed to turn red when include_budget_table went live; that happened, so the scenarios now assert the 400 rejections plus within-cap acceptance, and the unknown-org pin asserts the handler's 400 instead of the surfaced 500. |
||
|
devin-ai-integration[bot]
|
5b6194f427
|
fix(proxy): backfill null user_email on existing users during JWT auth (#34588)
* fix(proxy): backfill null user_email on existing users during JWT auth Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): guard mapped-key email backfill and make null update atomic Resolve Greptile review on the JWT user_email backfill: - only backfill when the mapped virtual-key owner is the JWT principal, so a mismatched admin-created mapping cannot write one user's email onto another - make the best-effort mapped-key enrichment non-fatal so a database outage on a cached-key request no longer fails otherwise-valid authentication - persist the backfill with an atomic null-guarded update_many so concurrent writers cannot overwrite an already-populated email Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep cache coherent when a concurrent backfill wins the null-email update * fix(proxy): cache DB-persisted email after JWT backfill, not the proposed value Resolve the Greptile finding that a successful null-guarded backfill could cache this request's proposed email even if a concurrent ordinary user update wrote a different email first. The helper now always re-reads the row after the atomic update and refreshes the cache from the value the database holds, so cache-hit auth and attribution stay consistent with the persisted record. Annotate the Prisma and model_copy dict literals to keep the LIT002 budget within its ceiling. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> |
||
|
yucheng-berri
|
7c8364c991
|
fix(team-callbacks): actually stop logging when disable_logging is called (#35520)
disable_team_logging cleared only metadata["callback_settings"], but callbacks
registered through POST /team/{team_id}/callback and the Admin UI live in
metadata["logging"], and request-time resolution stops at that slot without
ever reading callback_settings. The endpoint reported success while the team
kept sending request and response data to its third-party destination.
Empty the logging slot alongside the existing callback_settings reset, and
refresh the cached team object so the change applies to keys that are already
in flight rather than at the next cache expiry. The same refresh is added to
add_team_callbacks, which has the symmetric problem of a newly registered
callback staying dormant until the entry expires.
Resolves LIT-5101
|
||
|
yuneng-jiang
|
491eda319c
|
Merge pull request #35572 from BerriAI/litellm_e2e_budget_spend_poll_to_deadline
Some checks are pending
Unit Tests: LLM Provider Transformations / All Other Providers (push) Waiting to run
Unit Tests: MCP, Secrets, Containers & Misc / misc (push) Waiting to run
Unit Tests: Proxy Auth & Key Management / proxy-auth (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy API Endpoints / proxy-endpoints (push) Waiting to run
Unit Tests: Proxy API Endpoints / proxy-server (push) Waiting to run
Unit Tests: Proxy Infrastructure / proxy-infra (push) Waiting to run
Unit Tests: Proxy Legacy Tests / auth-and-jwt (push) Waiting to run
Unit Tests: Proxy Legacy Tests / key-generation (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-config (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-response-and-misc (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-server (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-server-extras (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-token-counter (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-user-auth-and-spend (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-utils (push) Waiting to run
Unit Tests: Responses, Caching & Types / responses-caching-types (push) Waiting to run
test(e2e): poll key spend to a deadline in budget reset advances tests |
||
|
ryan-crabbe-berri
|
bf873dffe4 |
test(e2e): skip the strict-priority and throughput SLO tests pending LIT-5118 / LIT-5119
The strict-priority e2e (added with the zero-increment limiter fix) can never pass on stage: the proxy there does not run the dynamic_rate_limiter_v3 callbacks + priority_reservation settings the module requires, confirmed by zero limiter log lines across every gateway and backend pod during the 2026-08-02 run. Config lives in the infra repo; LIT-5118 tracks adding it. The throughput SLO test failed the same run with 65.9% of requests dying at the ELB as 502/503 before reaching a pod. The per-replica SLO rework fixed the RPS-floor assertion but cannot help when stage idles at one warm gateway replica; LIT-5119 tracks pre-scaling the fleet for the load phase. Both skips name their ticket, and the coverage registry returns the two cells to the gap list while they are in place. |
||
|
ryan-crabbe-berri
|
ebdd854b98 |
test(e2e): poll key spend to a deadline in budget reset advances tests
A single read of key_info.spend races the batched spend writer: deltas earned before a reset flush to the DB up to ~60s later (proxy_batch_write_at) and land on the row after the reset zeroed it. The stage runs on Jul 30 and Aug 2 failed test_key_budget_reset_at_advances_after_window exactly this way, with spend back at the driven total while budget_reset_at had advanced and calls flowed again. Replace the single reads in rung 3 (spend zeroed after reset) and rung 4 (roomy window keeps spend) with _poll_key_spend, which re-reads to a 90s deadline covering one full flush-plus-reset cycle. A reset that never zeroes the row keeps spend pinned and still times out, so the regression guard keeps its teeth. |
||
|
mateo-berri
|
f25f1d2921 |
feat(proxy): resolve Cursor thinking/fast model-name suffixes on /cursor/chat/completions
Cursor appends -thinking-<level> and -fast to custom model names when the user picks a thinking level or fast mode, so a model configured as claude-opus-5 arrives as claude-opus-5-thinking-xhigh-fast and fails routing with no healthy deployments. When the raw name is not servable by the router but the suffix-stripped base name is, rewrite the body to the base model and carry the thinking level into reasoning_effort (chat bodies) or reasoning.effort (Responses bodies), never clobbering an effort the client already sent. Explicitly configured aliases keep winning because the raw-name servability check runs first. |