mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
904 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo
|
9d12c2c4e2 |
test: give the stream usage persistence test its own ProxyConfig
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo
|
ce1f43ec1a |
test: keep unrelated formatting out of the config list test change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo
|
0bc5b2ecf6 |
refactor: declare Admin UI general settings on ConfigGeneralSettings fields
The /config/list type map in proxy_server.py repeated every UI-editable general setting by name, so a setting had to be added in two files. The UI type now sits on the field itself and the map is derived from it. always_include_stream_usage was in the map but not on the model, so the Admin UI never listed it and /config/field/update rejected it. It is now a declared field. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yujonglee
|
574fcea691
|
fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473)
* fix(redis): share one native circuit breaker between RedisCache and the Rust stores RedisCache and the native Redis client each kept their own breaker, so once the cache learned the coordination Redis was wedged, a spend-counter write through the native store still paid a full socket timeout. Every Redis target now has one Rust breaker, looked up by NativeRedisBreaker.shared and injected into both the RedisCache guard and the native client. The Python FSM is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(rust): cargo fmt the shared breaker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(redis): close the breaker when the recovery probe gets a Redis reply error A command error such as WRONGTYPE means Redis answered, but the probe was left unsettled and its drop reopened the breaker for another recovery timeout. Both the RedisCache guard and the native client now settle such a probe as a success. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): pass a NativeRedisBreaker and isolate the shared breaker per test The sensitive routing test still passed RedisBreakerSettings as the store breaker, and the session affinity fixture reused one Redis URL, so a breaker an earlier test opened leaked into the next. Also make the RUF043 match pattern raw. * fix(redis): key the shared breaker by server and close the three audit gaps The breaker registry now keys on the Redis server (host and port, unix socket, cluster node set or sentinel service) instead of the full connection target, so caches that differ only in db, credentials or url spelling share one breaker. A READONLY reply from a replica mid-failover no longer counts as Redis answering, so it can't reset a run of failures. A new RedisCache refreshes the breaker state gauge, which the deleted Python breaker did at construction. A shared native client is only reused by a store that passes the same breaker. * refactor(redis): split redis-types into modules and drop redundant pyo3 attribute annotations lib.rs now only declares modules and re-exports, with resolution, target, tls, auth and endpoint each in their own file. Derived FromPyObject already reads a struct field as the attribute of the same name, so #[pyo3(attribute)] on the breaker and store settings did nothing. * refactor(redis): drop redundant pyo3 attribute annotations in the stores and let PyO3 build the cooldown snapshot Derived FromPyObject already reads each field as the attribute of the same name. The cooldown snapshot returns an IndexMap, which PyO3 turns into the same dict in the same order. * refactor(redis): return a HashMap from the three memory-tier snapshots The snapshots read cache-memory's HashMap, so there is no order to keep. PyO3 builds the same dict from a std HashMap, which replaces the hand-written PyDict loops in the spend counter and router usage stores and the IndexMap in the cooldown store. The rate-limit slot registry keeps IndexMap because it hands back a Python dict in the order Python wrote it. * refactor(redis): share Redis breakers and clients through one RedisHosts per host redis-connect's RedisHosts now owns the per-server breaker and the shared clients that python-bridge kept in two process-wide statics. A client takes breaker settings and admits through its server's breaker, so no caller can pair a client with a breaker of its own, and clients are keyed by Resolution alone. This drops RedisClient::guarded_by and Breaker::same. python-bridge keeps one RedisHosts per process and calls it with the GIL released. The gateway builds one per connect, so its coordination and response cache on the same server now share a breaker. RedisStoreSettings.breaker is back to breaker settings, as on main. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
quietjules
|
210e2a075a
|
fix(vertex_ai): authenticate Vertex AI pass-through requests once (#48942)
* test(vertex_ai): run auth once per Vertex pass-through request Drive /vertex_ai, /vertex-ai and /vertex_ai/discovery requests through the real proxy app with a counting custom_auth, and assert each request authenticates once and is served as the caller that auth returned. The generate-content routes fail today because the route authenticates a second time. * fix(vertex_ai): authenticate Vertex pass-through requests once vertex_proxy_route already resolves the caller through Depends(user_api_key_auth), but _base_vertex_proxy_route ignored it and authenticated the request again. A custom_auth function ran twice per request, and a key with max_budget reserved its estimated cost twice, so the unreleased first reservation exhausted the budget after one or two calls. The base route now uses the caller it is given and authenticates once itself only when none is passed, which is the discovery route's case. * fix(vertex): read the caller's key only from x-litellm-api-key or Authorization on Vertex routes The single auth check on /vertex_ai and /vertex-ai now reads the same two headers main's effective check did, so a key sent only in another key header gets 401 again instead of authenticating. * fix(vertex): authenticate the discovery route through the same Vertex key check Both Vertex pass-through routes now resolve the caller once through _authenticate_vertex_caller and hand it to _base_vertex_proxy_route, which no longer re-runs auth. x-litellm-api-key accepts the same scheme prefixes on /vertex_ai/discovery as on /vertex_ai and every other proxy route. * fix(proxy): size a Gemini request's budget hold from its prompt and maxOutputTokens A Gemini generateContent body's budget hold now counts its contents and systemInstruction and reads generationConfig.maxOutputTokens. Before, it held the model's whole context window, about $0.85 on gemini-3.8-flash, so a key with a small budget got 422s on back-to-back calls it could afford Also removes get_litellm_virtual_key, which nothing calls since the Vertex routes stopped using it * fix(proxy): count every Gemini system part and candidateCount in the budget hold A Gemini generateContent body with several systemInstruction text parts lost every part after the first when LiteLLM translated it to a chat request. That dropped the later system text for non-Gemini models behind /v1beta/models/<model>:generateContent, and made the budget hold count only the first part. The adapter now keeps every text part, as a plain string when there is one part and as text blocks when there are several. The output hold now also multiplies by generationConfig.candidateCount, the same way chat bodies use n, so a request asking for several candidates holds budget for all of them. * fix(proxy): keep the context-window hold for Gemini bodies the prompt count cannot see A Gemini generateContent body with media, cachedContent, other roles, or non-function tools keeps the full context-window hold, since the chat translation drops or undercounts those parts. The hold also reads the snake_case generation_config output fields, and several systemInstruction text parts reach non-Gemini models joined into one system string * fix(proxy): read only camelCase Gemini output fields in the budget hold /v1beta forwards only generationConfig.maxOutputTokens to non-Gemini models, so a snake_case cap the model never receives must not shrink the hold. Snake_case bodies now get the default output hold. * fix(proxy): count snake_case candidate_count in the Gemini budget hold * fix(vertex_ai): skip malformed system parts and read Gemini output fields only on Gemini bodies A systemInstruction part whose text is not a string is skipped, as main did, instead of failing the request. The budget hold reads generationConfig output and candidate fields, and counts contents, only when the body has no messages, prompt, input, query or documents field. An output cap of 0 or below counts as no cap. Test recorders keep tuples instead of mutating lists. * fix(proxy): take the budget hold's request shape from the route On generateContent routes the hold counts only contents and reads its output cap and multiplier from generationConfig, so stray fields of other APIs can no longer hide the prompt. A max_tokens of 0 holds no output again, while negative values and a Gemini maxOutputTokens of 0 or below hold the default output * fix(proxy): hold one candidate for a count too large to price and read system parts like content parts * fix(proxy): keep the spend already counted when a budget hold is far over the budget * fix(proxy): count candidateCount in the budget hold only for models that generate the candidates A non-Gemini model reached through /v1beta generateContent never receives candidateCount, so a huge count held the whole budget for the length of the call while the model produced one answer * fix(proxy): hold a candidate count only when every deployment generates it, and drop n when every deployment drops it * fix(proxy): count candidateCount when any deployment generates it and always count n The stricter rules from the previous commit let a key spend past max_budget through a Gemini name no deployment serves, an allowed_openai_params n, and a router fallback. Restore the earlier rules and add regression tests for those three shapes. * fix(proxy): cap the candidateCount hold at main's default output hold A candidateCount like 1e300 sent to a mixed Gemini and Claude group, or through a key alias from a Gemini name to Claude, held a whole key while Claude served the call and answered a strict key 422. On generateContent routes the counted candidates now raise the output hold at most to the default output hold main used for the model, so no count makes a call hold more than on main. Chat n is unchanged. * fix(proxy): hold every Gemini candidate up to the merge base's whole hold The previous cap stopped the candidateCount output hold at the default 16384 tokens. Because the hold now counts the Gemini prompt instead of the whole input window, a strict key admitted a call like 4 candidates of 8000 tokens and then spent past max_budget, where the merge base refused it. Every candidate is now held at its output cap, like chat n, and the whole hold is clamped at what the merge base held for the same call (the model's input window plus the default output), never below one candidate's hold. A huge count still holds no more than the merge base did. Chat n is unchanged * fix(proxy): let a Gemini output cap that /v1beta drops only raise the hold The Vertex and Gemini pass-throughs forward every spelling of the output cap, but /v1beta passes only generationConfig.maxOutputTokens to other models. The hold read only that spelling, so a snake_case cap above the default output held less than the pass-through could spend. A cap in any other spelling now raises the hold to it and never lowers it * fix(proxy): hold Gemini candidates of a model with no context window up to the default output A Gemini model the cost map has no context window for, priced in its deployment, held one candidate for any candidateCount, so parallel calls could pass their budget holds. Its candidates are held again up to its counted prompt plus the default output hold * fix(proxy): keep main's Gemini hold unless the request caps its output An uncapped generateContent call, or one capped only by a snake_case spelling, now holds what main holds (the context window plus the default output). A call with generationConfig.maxOutputTokens holds its counted prompt plus cap times candidateCount, never clamped to the deployment's max_output_tokens and never more than main's hold. A model with no context window keeps main's no-hold, and candidateCount always counts on generateContent routes. Also keeps get_litellm_virtual_key in pass_through_endpoints/common_utils.py, so an out-of-tree importer does not break * fix(proxy): keep main's budget hold on generateContent routes The PR now leaves the generateContent budget estimate as main has it. A live risk check showed the counted Gemini hold under-reserves (the prompt counter undercounts Gemini bodies, tiered and image output rates are not read, and request fields the handler merges later are not counted), so that estimate moves to a follow-up. The single auth pass on the Vertex pass-through and the systemInstruction adapter fix stay * fix(vertex_ai): keep main's Gemini adapter and pass a missing Authorization as no key The Gemini adapter change for later systemInstruction parts goes back to main, since it is not the double-auth fix this PR is for. The Vertex auth dependency now hands a missing Authorization header to the auth check as no key, like main's first auth pass, so custom auth and the 401 text see what they see on main --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
HootRock
|
8f19607259
|
fix(proxy)!: preserve Responses SSE event framing (#47980)
* fix(proxy): preserve Responses SSE event framing
* style(proxy): format Responses stream terminator condition
* fix(proxy): validate Responses guardrail event names
* fix(proxy): keep Responses first-event errors and keepalive failures in Responses framing
* test(proxy): use controlled clocks for Responses keepalive failures
* fix(proxy): end late Responses failures with an error event the SDK raises on
Errors that arrive after keepalive pings, and guardrail error strings yielded
mid-stream, now end /v1/responses with a named `error` event carrying the
same error object chat sends, so the OpenAI SDK still raises APIError with a
clean message and provider_specific_fields. Background polling passes the
same framing and records an `error` event as a failed response. The first
data line of a frame is found by splitting on newlines only, so a U+2028 in
an error message no longer hides a first-event error.
* fix(proxy): match OpenAI's Responses error event and stream late passthrough blocks as block responses
* fix(proxy): keep the reason when a background Responses request fails
A background poll of a request a post-call guardrail blocked saved a
null code and message, because the guardrail's error event nests them.
A request whose first chunk was an error polled as completed with no
error. Both now poll failed with the same code and message a late
failure gets.
* test(proxy): type the first-chunk poll test parameters
* fix(proxy): keep first-chunk error detection to lone error frames and handle string poll errors
* fix(proxy): keep the HTTP status on late Responses errors and end Responses streams on error events
A late error body without a code now takes the response's status, an upstream error event counts as the stream's terminal event, and a Responses stream that fails before its first chunk ends with an error event instead of data: [DONE].
* test(proxy): type the first-chunk failure test's parameters
* fix(proxy): keep cursor chat framing and JSON-encode Responses hook errors
* fix(proxy): encode non-dict Mapping Responses chunks as JSON objects
* fix(proxy): encode mappings and models nested in Responses stream chunks as JSON objects
* test(proxy): freeze the hook-output fixture
* fix(proxy): keep masked mappings and odd keys safe in Responses stream frames
Only read-only dict views and pydantic models nested in a hook chunk or error event expand into JSON objects. Other mappings, such as httpx.Headers, keep their own string form so masked values stay masked. Keys JSON cannot take become strings, and a chunk JSON still cannot encode goes out as the plain data frame it was before.
* fix(proxy): never fail a Responses stream frame on a value JSON cannot encode
Only dicts and read-only dict views expand into JSON, at the top level too, so a header mapping yielded as a whole chunk keeps its masked string form. Any failure while encoding a chunk falls back to the plain data frame the base sent, and an error event that cannot encode still goes out with its code and message, so a late block never drops the connection.
* fix(proxy): keep what a nested value's repr hides out of Responses stream frames
* fix(proxy): print values in Responses stream frames the way each base path did
* fix(proxy): send string and number subclasses whose repr shows their value as that value in Responses stream frames
* fix(proxy): print container subclasses and SDK models in Responses stream frames the way base did
* fix(proxy): read hook container storage the way repr does and send error-event models as str()
* fix(proxy): frame streamed Responses reads and send unreadable hook values as base did
Track pre-framed and raw SSE frames so a late failure never follows a terminal event, accept CRLF and CR line breaks in a first named error, frame GET /v1/responses/{id}?stream=true as Responses events, print a whole hook proxy chunk once and fall back to its text, ignore a non-integer status_code in a hook error frame, read a late-block dict subclass through items() when that view differs from its storage, and keep the order and inline form base's repr printed for nested mappings
* fix(proxy): match base on Responses stream edge cases
Read a decimal-string or integral-float status_code as its integer again, so
"429" and 429.0 still map to rate_limit_exceeded. Print each hook value's repr
once and reuse it, so a repr that changes between calls cannot reach the
client. Track only pre-framed sequence numbers from 0 to 2**63-1, so a huge
one cannot break the failure event. Build nested JSON with one frame per level
again, which restores the nesting depth sent as JSON. Count a pre-framed frame
as terminal from its data type, and a named error frame only when its data
carries an error, the way the OpenAI SDKs read them.
* fix(proxy): print hook proxies once and keep base's chunk tracking
Read-only mappings in hook chunks now emit the text from their first print, hook
chunks are read without their error field, only a non-empty error object makes a
framed error event terminal, and framed sequence numbers stay below 2**63-1 while
hook chunks keep base's unbounded tracking
* fix(proxy): print each read-only mapping item once and poll framed error events as failed
A MappingProxyType in a hook chunk or error detail now expands from the
mapping it wraps, so each item is printed once, the way base printed it.
A value whose repr changes between calls stays masked, a value whose repr
works once is kept, and deep nesting keeps one stack frame per level
The background reader takes each frame's type from the same terminal rule
the stream uses, so an event: error frame with an error object polls
failed even when its data names another type or none
* fix(proxy): print self-holding hook values once and poll how the stream ended
A container that holds itself now goes out as the marker its repr prints, like {...}, [...] or mappingproxy({...}), and the dict behind a read-only mapping joins the cycle check, so no item is printed a second time. A dict whose item repr adds a key keeps going after the items already shown, the way dict's repr does, instead of printing the whole dict again
A whole read-only mapping chunk that cannot go out as JSON is printed once by the formatter, so a print that fails fails the stream as it did on the base
The background poll reads the event type from the block it parses, as on the base, and keeps the first error frame the stream treats as terminal anywhere in a chunk. That frame fails the poll only when no response.completed, response.failed or response.incomplete follows it
* fix(proxy): name hook chunk events from the sent JSON and keep polls on frames they cannot read
A mapping chunk now takes its event: line from the JSON it sends, so a masked type value never reaches that line, and a chunk the event model cannot read goes out as a plain data frame instead of failing the stream
A dict whose item repr deletes or swaps keys keeps printing every key not yet shown, and an error code past the integer digit limit reads as no status
The background poll skips a frame whose shape it cannot read, as base skipped frames it could not parse, stops reading a chunk at [DONE], and lets only the last terminal frame in a chunk decide its status
A Responses stream's first chunk answers as a JSON error only for codes 400 and up
* fix(proxy): read every frame of a polled chunk and name model chunks from the sent JSON
A background poll parsed only the first data frame of each chunk, so a terminal event behind an error or delta frame in the same chunk was lost, and the poll stored a status its own stream never ended with. It now reads every frame up to [DONE]
A terminal event whose response is a string or null kept no status and polled completed. Its status now comes from the event type, and final output items that are not objects are skipped, so the rest of the final output is still stored
Model chunks now take their event line from the JSON they send, as mapping chunks already do, so a serializer that drops type no longer sends event: response.completed over {}
* fix(proxy): read polled chunks as SSE events and keep draining past a mid-chunk [DONE]
A background poll split a chunk into data lines, so one event with two data lines parsed as two payloads, and a frame with an integer past the digit limit raised a ValueError the parse guard did not catch, which failed the poll and dropped its spend row. The poll now joins each event's data lines the way an SSE reader does and skips any frame it cannot parse
A [DONE] behind another frame in one chunk ended the poll and left the stream unbilled. Only a chunk that opens with [DONE] ends the poll now, as on main, and the poll stops reading a chunk at its first terminal event, so two terminal events in one chunk poll the first one as main did
* fix(proxy): JSON-encode only plain hook chunks on Responses streams
A hook chunk whose whole tree is builtin JSON types streams as its named
event. Any other chunk keeps the base data frame, so nothing a custom
repr hides reaches the wire. Error event details encode the way the chat
error frame does.
* fix(proxy): skip containers already visited when checking hook chunks for JSON
* fix(responses): keep streaming past an event with no type
The Responses stream iterator read event.type directly when it noted a
yielded event, so an event whose JSON had no type, like a custom hook's
dict that the proxy now sends as JSON, raised AttributeError and ended
the stream. Read it with getattr, as the iterator's other checks do.
* fix(proxy): keep hook dicts with no type on the base Responses frame
A hook dict with no type is not a Responses event, so it gets no event
name either way. Sending it as JSON only made litellm SDKs that read
type on every event stop the stream. Keep such dicts on the frame the
base sends, so every SDK version reads them as before.
* fix(responses): keep a background poll's first terminal status and take a first-chunk error code from its HTTP status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Krrish Dholakia
|
b3c4a1c3e4
|
feat(spend-logs): turn off prompt storage per key or team (#50135)
* Add per-key and team prompt storage restrictions for spend logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Apply spend-log prompt restrictions to passthrough requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Apply team prompt storage restrictions to queued requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend-logs): honor tenant prompt storage in the collector and on "false" strings The spend-event collector decided whether to ship request and response bodies from the global setting alone, so an opted-out key or team still sent prompts through the collector transport. It now uses the same key/team check as the spend log writer. A key or team setting of the string "false" was ignored and prompts kept being stored. It now counts as false, the same way the global setting parses strings. Splits the 42-case tenant storage test into one test for the policy and one for the pass-through logging contexts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(spend-logs): carry key/team prompt storage into failure, batch and side-call rows Failure rows, executed batch rows, semantic index embeddings, websearch interception calls and errors rejected before request setup built their logging metadata without the key and team metadata, so a restricted key or team still had prompts stored on those rows. They now stamp it through one helper, LiteLLMProxyRequestSetup.get_tenant_logging_metadata. The dashboard toggle now shows a saved string "false" as off, matching the server, and its label and help text come from the translation file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Mateo Wang
|
98acc01e0d
|
feat(videos): add first-class OpenRouter video request fields (#50319)
* feat(videos): add first-class OpenRouter video request fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): remove recursive image parameter fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): preserve EdenAI native parameter passthrough Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): deduplicate frame roles in linear time Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): preserve image selection and dropped parameter semantics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): preserve provider scalar parameters and raw extra body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(videos): retain accepted scalar inputs for Vertex and DashScope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yujonglee
|
2b4485dee5
|
fix(guardrails): check a model-level post_call response once across the proxy and deployment layers (#50300)
#40154 made the deployment post_call hook really run apply_guardrail guardrails, so the proxy checked the same response a second time (LIT-10481). The deployment hook now records guardrail + response hash on the request and on the response, and the proxy skips a guardrail only on a matching record. The hash covers every field except ids, usage, model and other metadata set between layers, so a response rewritten between the layers (OCR included) is still checked. Guardrails with their own post_call hook keep their pre-#40154 behavior Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
devin-ai-integration[bot]
|
5d66f2d8ff
|
fix(usage): keep spend when metadata lookups hit database failures (#50253)
* fix(usage): tolerate entity metadata database failures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(usage): keep spend when metadata lookups hit database failures Treat transport errors and the query engine crash on a closed connection as database failures in the key, user and guardrail lookups of the aggregated, keys, users, model_top_keys and cache_leakage_keys routes, and keep prisma out of module-level imports so a proxy install without prisma still imports the usage endpoints. * fix(usage): fall back only on database outages, on every daily activity route Data-layer Prisma errors such as a rejected query or a missing table return 500 again, a key whose details failed to load reports key_exists null instead of false, and the non-aggregated daily activity routes keep spend when the key or entity metadata lookup hits an outage * fix(proxy): skip the key metadata lookup when the daily rows carry no keys * fix(usage): keep the guardrail breakdown fallback for any Prisma error A missing guardrail usage table or column made the Usage call return 500 after the outage-only narrowing. The guardrail lookup now falls back on outages and on any Prisma error, as it did before this PR * fix(usage): keep spend when the team, organization, customer, or agent name lookup drops --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
ab3b12b951
|
feat(proxy): serve Codex its own release's model catalog, fetched from GitHub (#50318)
* feat(proxy): give Codex accurate model entries for every model in /v1/models * fix(router): read a deployment's upstream model without a not-required key access * fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider A deployment with no token price no longer votes a Codex service tier off, an undescribed model keeps Codex's default reasoning summary parameter, a Bedrock deployment naming its provider in custom_llm_provider gets Codex's stock entry, and Bedrock provider namespaces follow the AWS model id format. * fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog lite codex imports the catalog before the SDK is initialized, so importing cost tracking, the Bedrock helpers or provider resolution at module level hit a circular import and broke every lite command. * fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped Codex shows /fast for a leftover fast speed tier even when service_tiers no longer offers priority, so an alias, an empty configured tier list or a configured list without priority showed a /fast toggle that sends nothing. One helper now sets service tiers, speed tiers and the default together. * fix(proxy): give an older Codex a translated entry for models past its version Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release * feat(proxy): serve Codex its own release's model catalog, fetched from GitHub When Codex asks /v1/models for its catalog, the proxy now reads models.json and prompt.md at that release's tag on GitHub and serves those entries, falling back to the bundled 0.166.0 copy when the fetch fails, times out after 2 seconds, does not validate, the version has no tag, or LITELLM_OFFLINE_MODE is on. A failed fetch is remembered per version for an hour and only the eight most recent versions are kept. The integration tests ask as the bundled release so their expectations hold whether or not the proxy can reach GitHub * fix(proxy): bound the GitHub fetches a caller can trigger through Codex catalog versions Failed versions now live in their own cache so they never push out a fetched catalog, requests for a version already being fetched share that fetch, each worker starts at most six fetches a minute and serves the bundled catalog past that, and the version check only accepts ASCII digits * fix(proxy): keep a failed Codex version's cooldown past the fetched catalog cap Failures are tiny, so they get their own cap of 256 instead of sharing the eight-catalog cap, which let nine failing versions retry each other well before their hour was up. The TypeVar takes Final like the rest of the codebase |
||
|
Mateo Wang
|
3f38730d2c
|
refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986)
* refactor(typing): clear basedpyright Any errors (wave 1) * refactor(typing): clear basedpyright Any errors (wave 2) * refactor(typing): clear basedpyright Any errors (wave 3) * refactor(typing): clear basedpyright Any errors (wave 4) * refactor(typing): clear basedpyright Any errors (wave 5) * refactor(typing): clear basedpyright Any errors (wave 6) * fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves * fix(logging): keep callbacks running when litellm_params is a params model should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request Adds behavior tests for the changed lines codecov listed as not run * fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call. * fix: bill regional uplift multipliers float() accepts, bytes included * fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type |
||
|
mrinal-berri
|
aeede53112
|
feat(guardrails): add mcp_block_behavior to raise 400 on MCP tool blocks (#50284)
* Allow guardrails to propagate MCP tool blocks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Refresh generated API schemas for MCP block behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Fix MCP guardrail deployment selection and streaming errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Enable content filter blocks during MCP calls Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Preserve content filter MCP hook restrictions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
de0b389205
|
feat(router): add the Switchyard stage router as an auto router strategy (#50407)
* feat(router): add Switchyard Stage Router strategy * feat(router): decision-model classifier and safer fallbacks for the Switchyard stage router Moves to nemo-switchyard 0.4.0a1, which adds decision-model classifiers and makes ModelCall.respond and fail async. The classifier is now a tagged union of an LLM judge and a decision model, and decision calls go through the router's adecisions so any decisions provider works. Algorithm misbehavior and library errors route to the capable model instead of failing the request, and the session id plus client session headers are always passed so capable holds apply without a classifier Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): keep the Switchyard router inside the type-check gate The gate's environment did not install the optional switchyard extra, so every libsy symbol counted as Unknown. nemo-switchyard joins proxy-dev like the other optional runtime packages, and imports go through switchyard_rust.libsy, which ships py.typed. Step handling moves into helpers so no Final is rebound in the loop, and the dashboard API types are regenerated for the new routing fields Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): route Switchyard on every endpoint and reject unsafe router saves The router now passes each request's call type to routing strategies, and Switchyard converts the request to chat messages with the existing converter for that call type before translating it. /v1/messages turns with tool or thinking blocks used to fall back to the capable model, and now route on their signals. The Anthropic converter can keep tool_result.is_error on an analysis copy, so Switchyard sees tool failures there, while requests sent to models never carry it Saves of a strategy router that lists itself now fail with a 400 instead of a 500 with a dead row, the capable and efficient models must differ, judge and decision calls are not retried before failing open, the Switchyard routing fields are classified for redaction, and the edit dialog returns the saved router instead of the stale one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): catch self-routing on patch and rename, and trim Switchyard tests The self-routing check now reads the stored params through the decrypting helper and runs when a patch carries no litellm_params, so a PATCH that points a router at itself or renames it onto one of its targets returns 400 instead of a 500 with a dead row. The Switchyard tests share one transcript builder and router fixture, and reformatting noise in unrelated test files is reverted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(ui): make Switchyard a strategy in the existing auto router form Switchyard is now a card in the existing strategy picker. Picking it swaps the tier and classifier fields for the Switchyard fields, and the shared name, team and save button create it through one submit branch, the same flow as the other strategies. The separate add form, the extracted picker and the hidden-form swap are gone, and the edit dialog keeps its own Switchyard form The judge's session_affinity and message_hash_fallback settings are removed, since Switchyard rejects both for the stage router Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): accept partial tool semantics and show server errors on model edit The Switchyard tool semantics field rejected mappings that list only some categories, although the server accepts them. Model PATCH errors were reduced to a generic message, so it now goes through the shared API client like model create and surfaces the server text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): show the routing strategy picker up front in the auto router form Every strategy was hidden behind an Advanced configuration link, so the quick setup read as the only option and never said it creates a heuristic router. The picker now always shows with Heuristics as the labelled default, the quick setup sits under it, and picking another strategy opens that strategy prefilled with the chosen models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): show the server's reason when a model settings or API key update fails Both screens replaced the rejection with a fixed message, so a 400 such as a router routing to itself read only as a generic failure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(router): route Switchyard on the Rust router core main replaced the Python router core with the Rust one, which left Switchyard routers registered nowhere. Register them with the other pre-routing strategies, hand each strategy the endpoint the request came through, and evict Switchyard routers on reload and delete like the rest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): scope Switchyard sessions per API key and fix review gaps in the Switchyard form Switchyard session state is keyed by a hash of the caller's key hash and session id, and client session headers no longer reach Switchyard unscoped. The add form carries the quick setup models into Switchyard, sends the team and access groups, clears a stale tool semantics error, and the schema reads API-saved configs with nulls and structured decision instructions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(router): make the Switchyard test folder a package * fix(router): read the Switchyard session key hash only from the proxy's own metadata * fix(proxy): keep nested nulls as null when masking deployment params * fix(router): send Switchyard judge calls with the caller's own metadata bucket so tag routing applies * fix(router): put Switchyard judge metadata in the bucket the judge endpoint reads --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
simonp9
|
1f15088fc9
|
feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes (#49408)
* feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes
The native Codex catalog now lists the key's other chat and responses models after
ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list
refreshes on Codex's timer. A request for a model configured by exact name on another
provider goes through /v1/responses without the ChatGPT bearer or account header,
while ChatGPT names and wildcard-only matches keep the subscription passthrough.
Reasoning items another provider produced are dropped before forwarding to ChatGPT.
* fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models
Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while
`codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native
/backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no
default, so the provider applies its own default exactly as `codex -m` does
* fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes
Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses
request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may
call /v1/responses, drop unpersisted reasoning ids on the passthrough when store
is not true, keep any slug the caller's ChatGPT account lists on the subscription
via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog
added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard
precedence over a catch-all deployment.
* fix(proxy): keep Codex's native route on the subscription when the account list is unknown
A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the
account's list is known and leaves it out. A failed catalog read is remembered for 60 s and
the Codex version comes only from a User-Agent whose product matches the originator header.
The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows,
and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so
Codex stops refetching or dropping its catalog cache.
LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure
blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge
decodes wrapped thinking blocks with their signature.
The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the
native route checks the chatgpt/ form of a name the key holds only through that group. A
none reasoning effort is dropped only under drop_params. The guide covers app tools, every
routing case, and the ETag, and the catalog logs how many models the byte limit left out.
* fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API
* fix(proxy): keep Codex native turns working when the key's model listing fails
A listing callback that raises no longer turns chatgpt.com's catalog or
responses answer into a 500. The catalog keeps upstream's body and ETag, and
responses keep upstream's X-Models-Etag.
Under drop_params, a `none` reasoning effort is now dropped only when a
candidate deployment does not explicitly support it, so an alias to a model
like openai/gpt-5.4 keeps it.
The account slug list and its unavailable marker are keyed on the account id
plus the caller's ChatGPT Authorization header, so a failed fetch with one
bearer cannot turn off LiteLLM routing for another caller on the same account.
The auth step checks the Codex route before importing the Codex module.
* fix(proxy): keep Codex model switches working across LiteLLM providers
On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400
Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts
A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows
* fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers
Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the
native route now hoists them without flattening, like OpenAI and Azure and
the chat completions bridge. Other native Responses providers still get
flat names, now built with the bridge's namespace_tool_name_map, and each
function_call they return gets its tool name and namespace back in the
non-streaming body and in streamed output_item.added, output_item.done,
and completed/incomplete/failed events. Every other event passes through
unchanged
* fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools
Codex sends its freeform exec tool as a custom tool on every turn, and xAI's
Responses API answers 422 for custom, namespace and additional_tools shapes.
On the Codex native route, a model whose deployments include a native
Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now
asks for the chat completions bridge with use_chat_completions_api, which
already turns custom and namespace tools into functions, hoists
additional_tools, and puts the namespace back on each returned call
OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock
Mantle still get it on their own Responses API with additional_tools hoisted.
The route's namespace flattening and the response-side namespace restore are
removed, since no path sends flattened names anymore
* fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve
A reasoning item another deployment produced, replayed with store false and no blob the
target can decrypt, now reaches the target as its readable summary without id or status,
and is dropped when nothing readable is left. OpenAI answered 400 on the status field and
404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment
replays and stored conversations keep the item whole
* refactor(codex): reach encrypted reasoning stripping through the router registry
The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from
litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry
gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with
encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise
every target must decrypt it
* fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog
* fix(proxy): drop Codex's none effort for an alias whose native entry names no default
* fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop
* test(proxy): map the pinned reasoning efforts field in the router registry stub
* fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases
Codex native routes now send a key to LiteLLM only when admin_only_routes allows it,
a native alias with configured service tiers starts with no tier picked, and the
account slug cache declares its Redis key family.
* fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items
* fix(proxy): check vector store grants on file_search tools inside additional_tools items
A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input.
* fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes
* fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply
* fix(guardrails): check additional_tools items next to top-level function tools in tool_policy
ToolPolicyGuardrail read tools from the request body only when no top-level
function tool existed, so a blocked tool defined in an additional_tools item
reached Fireworks once the Responses transform moved the item into tools after
the pre-call hooks. Check every list on each request, including namespace
members inside items.
* fix(router): serve decryptable_reasoning from the Rust core's optional checks
The merge with main removed the Python router core, so the registry's host has no
optional_callbacks and no routing context, and native Codex requests with encrypted
reasoning raised RustCoreUnsupported. The filter now lives with the optional checks,
which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core
Also give the static litellm.Router a declared constructor. Since the Python core was
deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...)
is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change
* fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns
Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures.
* fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias
* fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases
Auth no longer asks ChatGPT for the account's model list when the bearer is
missing or is the LiteLLM key, so a key sent as the bearer never reaches it.
tool_policy now reads the members of a top-level namespace, which is where
the native route's hoist puts them.
A global alias now reads encrypted content affinity from its target group.
A non-list top-level tools value and a non-string function name now reach the
provider, which refuses them, instead of raising a 500.
The catalog ETag no longer depends on the order a process iterates priced
service tiers in, so workers and pods agree on it.
* fix(proxy): list a team's models in the order the team set them
get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker
* fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models
A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models
and /backend-api/codex/responses only when the JWT auth flow would admit a
direct /v1/responses call for that model, so team_allowed_routes,
admin_allowed_routes and RBAC route limits hold on the handoff too.
The Fireworks Responses transform also leaves additional_tools input items
in place when the tools param is not a list, instead of raising while hoisting
* fix(fireworks): hoist additional_tools when tools is a tuple
litellm.responses passes a tuple tools argument through to the Fireworks
transform unchanged, so the non-list guard skipped hoisting and left the
additional_tools item in the input. Treat tuples like lists and cover both
shapes in the drop_params merge test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
ryan-crabbe-berri
|
ec46a981fc
|
feat(model-hub): show each deployment's full prices in the model hub (#50145)
* feat(model-hub): show each deployment's full prices in the model hub The model hub only showed one input and one output price per model, the highest across its deployments. Each model group now lists every deployment's prices, read from the same entry billing charges, including cached tokens, long-context tiers, flex, priority, batch and tiered ranges. Both the logged-in and public hub detail views render them in one table per deployment, labeled by provider only * refactor(ui): move model hub pricing breakdown text onto i18n Route every string the deployment pricing breakdown writes through t() under a new deploymentPricing group in en.json, plus the three hub headings that host it. The English on screen is unchanged * fix(model-hub): show Vertex regional prices and explicit cached reads as billed A Vertex deployment outside the global endpoint now shows its token prices with the regional uplift billing applies at the deployment's location. The explicit cached read price shows per 1M tokens on its own row instead of a raw per-token value. The breakdown also says that adjustments made per request, like off-peak discounts and regional or data residency surcharges, aren't included * fix(model-hub): show prices written as text the way billing reads them A price written in config.yaml as 3e-7 arrives as text. Billing turns it into a number, but the hub dropped it, so a deployment showed no price or the cost map's price instead of its own. The hub now reads numeric text for regular prices and size-based price ranges, and skips text that isn't a number * style(ui): format public model hub test * test(model-hub): name the deployment pricing fixture to stay within the inline object lint budget |
||
|
Kerry Lu
|
d4f338ef1e
|
feat(tags): reset a tag's spend through /tag/update (#50323)
* feat(tags): reset a tag's spend through /tag/update Co-authored-by: Arnold Galovics <arnold_galovics@docktape.com> * test(tags): assert spend update response contract * fix(tags): seed spend counter after evicting tag cache * refactor(spend): expose shared counter reset helper * fix(spend): sanitize counter key in spend counter warning log --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Arnold Galovics <arnold_galovics@docktape.com> |
||
|
Mateo Wang
|
36c64ce85d
|
feat(bedrock): default to native structured output on Opus 5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 (#50221)
* feat(bedrock): enable native structured output by default for selected Claude models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep govcloud and bare model rows off native output Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected native schemas with the JSON tool on default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep the rejection retry on the unforceable-tool native path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry rejected schemas on the Messages path for default-native models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format Messages native output retry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): allow profile-only native output metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): only disable default-native output on schema-rejection retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): type fixture parameters in native output retry tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): drop strict tools after a native-output retry hits the grammar limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format native output fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): honor the deployment native output opt-out and keep wide batch schemas on the JSON tool A deployment's supports_bedrock_native_structured_output: false now keeps the JSON tool on rows that default to native output. Bedrock batch has no retry, so a record whose schema has more than 64 properties keeps the JSON tool there too. * fix(bedrock): retry a native structured output stream as a buffered stream when Bedrock denies the stream action A role allowed only bedrock:InvokeModel got a 403 on schema streams once these models went native, since native streams call converse-stream while the JSON tool path buffered through converse. The retry now sends that request again as a buffered JSON tool stream, the shape the base sent. The wire test moves its neither-key and opt-in cells to Opus 4.7, its Bedrock-key-only cells to Sonnet 5.5, and its unforceable-tool cells to the bare Opus 5.5 id, since this PR makes Opus 5.5 and Haiku 5.5 always native. The batch wire test now checks a thinking schema record on both sides of the contract the batch unit test already pins, the JSON tool on opt-in Sonnet 5.5 and native output on always-native Haiku 5.5, since its Haiku 5.5 and Opus 4.8 cells assumed those models were opt-in. * fix(bedrock): move the Claude 5 native output default to a key released proxies do not read Released proxies load the cost map from main and have none of the native output retries, so setting supports_native_structured_output on the Opus 5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 profile rows turned working wide or min/max schemas into 400s on v1.108.1. Those rows keep supports_native_structured_output false and gain bedrock_native_structured_output_by_default true, which get_model_info reports as supports_native_structured_output, so this version behaves as before while released versions keep the JSON tool. * fix(bedrock): rename the Claude 5 native output default key to supports_bedrock_native_structured_output_by_default The cost map guard runs main's model catalog on the PR's cost map, and that catalog rejects any undeclared key that does not start with supports_. Declaring the old name on main first needs its own PR, so the key takes the supports_ prefix. Released proxies still never read it * fix(bedrock): honor registered and batch opt-outs, move Opus 5 and 4.8 to the default key A registered supports_bedrock_native_structured_output false now beats the native output default, and batch uploads read the deployment's model_info, so an opted-out deployment keeps the JSON tool on every record. The Opus 5 and Opus 4.8 profile rows move to supports_bedrock_native_structured_output_by_default, so released proxies on main's cost map keep the JSON tool on them. The 64 property batch cutoff now applies only to rows with that key, so models that were already native keep their batch output shape. * fix(proxy): report a deployment's Bedrock native output opt-out on /model/info /model/info merged the deployment's model_info over the backend row, so a deployment that set only supports_bedrock_native_structured_output: false still showed the backend's derived supports_native_structured_output: true. The generic flag is now derived from the merged info unless the deployment set it itself, through the same helper get_model_info uses. * fix(bedrock): keep native output when a denied schema stream is resent buffered A role without bedrock:InvokeModelWithResponseStream had its native output schema stream resent as a buffered JSON tool call. The resend now keeps the native params with fake_stream set, and falls back to the JSON tool only if Bedrock then rejects native output. * fix(bedrock): drop rejected strict tools on the JSON tool retry after a buffered native resend * fix(bedrock): send a buffered schema stream to /invoke and keep the by-default key on its deployment The shared HTTP handler now builds the non-streaming URL for a fake-streamed request, so the invoke route's buffered resend after a denied stream calls /invoke instead of invoke-with-response-stream. supports_bedrock_native_structured_output_by_default is deployment scoped, so setting it on one deployment no longer reaches sibling deployments of the same model * fix(bedrock): answer a buffered invoke stream from the non-streaming reply * fix(bedrock): keep the JSON tool retry buffered after a denied stream's schema is rejected * fix(bedrock): count only a true fake_stream and keep the deployment by-default key out of /model/info A fake_stream sent as "true" or 1 picked the non-streaming URL but kept the streaming reader, so Bedrock invoke streams answered 500 after three billed calls. Only a real True counts now, as on main. /model/info read supports_bedrock_native_structured_output_by_default from the deployment, while routing reads it only from the cost map, so it reported native output for deployments that use the JSON tool. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
ddd162439f
|
fix(proxy): stop /tag/update 500 when the body carries the tag's budget_id (#50324)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
JeongCheolLee
|
febbe38326
|
fix(anthropic)!: do not surface server_tool_use as client tool_calls (#48385)
* fix(anthropic): do not surface server_tool_use as client tool_calls Anthropic server-executed tools (web search, tool search, code execution) were returned in chat completions tool_calls with srvtoolu_ ids, so OpenAI-protocol agent clients treated them as pending work and re-invoked the model until their recursion limit. Move them to provider_specific_fields.server_tool_calls on both the non-streaming and streaming paths. History replay, the /v1/responses bridge and the /v1/messages streaming guardrail read that field, so multi-turn reconstruction, web_search_call output items and the guardrail's fail-closed check on server tool blocks keep working. * fix(responses): return Claude web_fetch calls as web_search_call open_page items * fix(responses): keep Claude server web calls without a result as hosted items A non-streamed Claude turn that calls web_search or web_fetch next to a client tool can return the server call before its result. Build an in_progress web_search_call for it, so the Responses bridge no longer returns it as a client function_call * fix(anthropic): keep pending server calls native and hide server calls from client tool paths Replaying a server call that has no result yet now sends it back as server_tool_use, so Anthropic accepts the next turn. The tool permission guardrail checks server calls again and rewrite mode removes a denied one. The /v1/messages adapter keeps Claude's server blocks, and the Responses bridge never returns a server call as a client function_call. * fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls, so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own server_tool_use instead of running again * fix(anthropic): let guardrails and loggers read Claude's server tool calls again Guardrails, tool policies, tracing loggers and the Responses guardrail translation read only client tool calls after server calls moved to provider_specific_fields. They now read server calls first, then client calls, through one shared helper, and the Responses guardrail treats Claude's web_search_call items as web_search or web_fetch calls. Non-streaming splits server calls by block type, as streaming does. * fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy, streamed or not. Redaction with message logging off hides server call arguments and hosted web call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls. * fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use. * fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy. * test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions * test(guardrails): type the fake Pillar client like its Protocol * fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or web_search still needs that call and its thinking replayed, or Claude loses the fetch result * fix(guardrails): scan replayed Claude server calls and hosted web calls Request guardrails now see and rewrite server_tool_calls replayed in chat messages. Cato and Purview scan Claude web_search and web_fetch calls on Responses output, and a function tool call with an extra custom key is no longer dropped from Cisco, Purview, Repello and PANW scans. * fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks * fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows * test: annotate the new server tool call test locals with Final and type their parameters * test: type the new guardrail test helpers with litellm's tool call types instead of Any * test: narrow the remaining new test helper parameters from bare dict and object * fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls * fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera * fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
moe-berri
|
b02bb99b08
|
feat: add OpenRouter-style -latest model aliases for first-party providers (#50315)
* Add first-party floating model aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Align Moonshot alias parameter test with its target Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Share floating alias proxy budgets and rate limits with targets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Prove target-only team budgets count and block floating aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Share floating alias quotas with project and organization targets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Coalesce colliding floating alias rate limit counters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Fix latest alias Rust fallback and failure accounting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: adapt latest alias checks to typed rate-limit store Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ishaan-j-berri
|
1bcd11706b
|
fix(proxy): discover models for LiteLLM Gateway wildcard providers (#50374)
* fix(proxy): discover upstream LiteLLM Gateway wildcard models * fix(proxy): use the typed shared client for gateway discovery * fix(ui): show the selected LiteLLM Gateway logo |
||
|
yucheng-berri
|
4086d5ccaa
|
feat(proxy): hash UI passwords with PBKDF2 in FIPS mode (#49874)
* feat(proxy): hash UI passwords with PBKDF2 in FIPS mode A FIPS OpenSSL refuses scrypt, so every password write fails there and nobody can sign in with a password. When OpenSSL enforces FIPS, passwords now use PBKDF2-HMAC-SHA256 at 600k iterations with a salt from OpenSSL's DRBG, stored as "pbkdf2:" plus base64 of salt and key. Any other row is refused with one error log and left as it is, and the boot migration does nothing, because FIPS mode is for new deployments. PBKDF2 at 600k takes about five times as long as scrypt, so in FIPS mode the five request paths run it on one thread instead of the event loop. A single thread keeps a login burst from using up a CPU quota, which is what stalls the loop with asyncio.to_thread. A login that queued for that thread asks the throttle again before it answers, so a burst of guesses can't get past a block that the failures ahead of it just set. The breached password check is off in FIPS mode, because it runs the password through SHA-1. A standard OpenSSL reads False at every new branch, so standard mode keeps scrypt, inline, on the same rows. Resolves LIT-8425 * fix(proxy): take FIPS logins one at a time and recheck the throttle first A login that waited for the KDF thread asked the throttle again only after deriving, so a burst for one user kept that thread busy long after the user was blocked. Logins now wait for their turn on a lock and ask the throttle when it comes, so a blocked login costs no PBKDF2 and only one login at a time holds a place in the KDF queue. Password changes and invite claims no longer wait behind a whole burst of logins. * fix(proxy): keep PBKDF2 rows on a standard boot and warn when FIPS skips the breach check A standard boot treated a pbkdf2: row as plaintext and hashed its text, so the real password stopped working and the old row text signed in. The migration now leaves a row of that exact shape alone. Any other pbkdf2: text still migrates. FIPS mode now logs a warning when it skips a breach check that is turned on. The one-thread test no longer depends on timing, and the new test variables are annotated Final. * fix(proxy): keep the breached password check on in FIPS mode * fix(proxy): choose the password KDF from fips_mode instead of the host OpenSSL * test(proxy): fake the scrypt probe in the FIPS fixture now that the gate checks scrypt |
||
|
yujonglee
|
b1b236826a
|
feat(redis): serve the proxy's common Redis call with Rust stores (#50170)
* feat(redis): route the router, rate limiters and auth through the native store families * style(redis): satisfy the python lint gates Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): give the registry's default usage store a memory-only handle RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): build stores from native handles and wait for router tasks Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
jesus-berri
|
4f06ca217f
|
feat(team): let default_team_params attach access_group_ids to every new team (#49908)
* feat(team): let default_team_params attach access_group_ids to every new team * fix(types): keep default access_group_ids as Sequence for the type discipline gate * fix(settings): chunk the default access group existence lookup * test(team): prove default access groups reach the saved team row and type the settings fixtures * fix(team): check default access groups on the writer so a fresh group is not rejected behind a read replica |
||
|
jesus-berri
|
18b02380e5
|
feat(spend-logs): opt-in latest_turn mode that stores only the current turn of stateless requests (#49544)
* feat(spend-logs): opt-in latest_turn mode that stores only the current turn of stateless requests * fix(spend-logs): keep bridged Responses rows and classifier audit in latest_turn mode, reject unknown values at boot * fix(spend-logs): drop the recursive call in the latest_turn trim and type the stateless test cases * test(spend-logs): format the stateless latest_turn cases * test(spend-logs): wrap the classifier audit fixture * fix(spend-logs): type the latest_turn trim inputs * fix(spend-logs): leave the raw request body unvalidated before the latest_turn trim * fix(spend-logs): trim classifier rows and ignore continuation keys off /v1/responses in latest_turn mode * fix(spend-logs): take the classifier's originating request from the logging payload and redact a copy of vector store results * fix(spend-logs): redact the body selected for storage, after the latest_turn trim |
||
|
ryan-crabbe-berri
|
9a473fbbb0
|
feat(proxy): give Codex accurate model entries for every model in /v1/models (#49825)
* feat(proxy): give Codex accurate model entries for every model in /v1/models * fix(router): read a deployment's upstream model without a not-required key access * fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider A deployment with no token price no longer votes a Codex service tier off, an undescribed model keeps Codex's default reasoning summary parameter, a Bedrock deployment naming its provider in custom_llm_provider gets Codex's stock entry, and Bedrock provider namespaces follow the AWS model id format. * fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog lite codex imports the catalog before the SDK is initialized, so importing cost tracking, the Bedrock helpers or provider resolution at module level hit a circular import and broke every lite command. * fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped Codex shows /fast for a leftover fast speed tier even when service_tiers no longer offers priority, so an alias, an empty configured tier list or a configured list without priority showed a /fast toggle that sends nothing. One helper now sets service tiers, speed tiers and the default together. * fix(proxy): give an older Codex a translated entry for models past its version Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release |
||
|
ryan-crabbe-berri
|
48ff826cbc
|
feat(proxy)!: only let clients pick S3 buckets the admin allowed (#47496)
* feat(proxy): let admins allow specific S3 buckets and endpoints in request bodies, block bucket settings otherwise Client requests can no longer pick s3_bucket_name, s3_output_bucket_name or allow_legacy_cloud_file_ids unless the value is on general_settings.allowed_client_side_s3_buckets (buckets) or the admin opted in the usual way. s3_endpoint_url, already blocked, can now be allowed per value through general_settings.allowed_client_side_s3_endpoint_urls * refactor(proxy): drive the client-side S3 allowlist from one typed field-to-setting map * fix(proxy): treat a null client-side S3 allowlist in config as empty instead of failing the request |
||
|
yucheng-berri
|
9f489430a6
|
fix(proxy): probe scrypt instead of MD5 in the fips_mode OpenSSL check (#50305)
* fix(proxy): probe scrypt instead of MD5 in the fips_mode OpenSSL check * test(proxy): fake a working scrypt so the fips_mode check tests pass on FIPS hosts |
||
|
mrinal-berri
|
fb2a1450dd
|
fix(users)!: validate custom IDs and reject concurrent duplicate creation (#49748)
* fix(users): validate custom IDs and reject concurrent duplicate creation * fix(users): return precise conflicts without retrying them * fix(users): preserve retries for existing-user upserts * fix(scim): preserve opaque IDs during user provisioning * fix(sso): preserve identity provider user IDs |
||
|
tin-berri
|
a7f9f892be
|
feat(skills): add sortable Skills Hub catalog and publication audit (#50195)
* feat(skills): add sortable Skills Hub catalog and publication audit * refactor(skills): remove redundant publisher table column * style(skills): apply dashboard formatting * fix(skills): preserve approval audit and complete hub sorting |
||
|
Kerry Lu
|
d295f76c4a
|
fix(types): define the responses compact call types for during_call guardrails and deployment hooks (#50274)
* fix(types): define the responses compact call types so during_call guardrails and deployment hooks see them * test: regenerate dashboard api types and skip compact in the azure sdk client sweep --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
3b20b48473
|
feat(interactions): forward continuation_token and type requires_continuation so callers can resume a long decode (#50151)
* feat(interactions): continue requires_continuation long decodes with continuation_token hops A foreground Interactions request stops at Google's per-hop output cap with status requires_continuation and a continuation_token. litellm now re-posts the request with previous_interaction_id and continuation_token until the status moves on, merging the hops into one interaction (last id, every hop's steps, summed usage with initial_input_tokens from hop 1), for non-streaming and streaming calls. continuation_token is accepted as a request param, and requires_continuation is a known status for billing. * refactor(gemini interactions): validate streaming chunks with model_validate * fix(interactions): keep the held completion across stream hops, stop on a repeated continuation_token, follow the spec redirect * fix(interactions): add grounding_tool_count per tool type, stop on any seen continuation_token, keep streaming past a [DONE] frame when a hop is pending * ci(interactions): point the spec canary at aistudio.google.com, where Google moved the Interactions OpenAPI spec * test(interactions): scripted Google ends each long decode hop with a [DONE] frame * refactor(interactions): pass requires_continuation and continuation_token through instead of looping hops in the proxy The caller drives the continuation, the same as against Google directly. /interactions accepts continuation_token in place of input so a resume request needs no input. * test(proxy): assert the accepted /interactions bodies pass the required-param check * fix(proxy): keep the router default fallback for input on /interactions, continuation_token stands in for it --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
ff2709f5c3
|
fix(zscaler_ai_guard)!: scan only the last user turn on requests (#50139)
* fix(zscaler_ai_guard): scan only the last user turn on requests * fix(zscaler_ai_guard): keep scanning A2A requests and refuse scan_only_tool_results * fix(zscaler_ai_guard): join the last user turn's text parts with spaces like before * fix(zscaler_ai_guard): keep scanning tool results when scan_only_tool_results is set * fix(zscaler_ai_guard): scan the latest user turn when assistant or tool messages follow it * fix(zscaler_ai_guard): skip the scan when the latest user turn is only whitespace A blank latest user turn was sent to Zscaler on its own, which answers Content is empty, so the proxy returned 500 where main returned 200 --------- Co-authored-by: Arnav Maiti <amaiti@zscaler.com> |
||
|
Nate Armstrong
|
067f93340f
|
refactor(router): delete the python router core, the rust core is the only core (#50178) | ||
|
Mateo Wang
|
88ea40e69c
|
fix(github-copilot): send the GitHub token directly instead of exchanging it (#50126)
* fix(github-copilot): use GitHub token directly Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(github-copilot): expire API base cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(github-copilot): restore authenticator coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(github-copilot): update direct token auth tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(github-copilot): cover direct-token error paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(github-copilot): satisfy type-discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
a12a6cd154
|
feat(accounting)!: make the SpendLogs ledger the default accounting path (#49872)
* fix(accounting): keep one undecodable delivery from blocking the ledger A request whose metadata carried a 401-digit tokens_saved stored that number in its delivery payload. The query engine could not decode it when the worker claimed the batch, so every later delivery on the proxy stayed pending The worker now claims the payload as text and decodes each row in Python. A row that still cannot be decoded is parked alone while the rest of the batch applies. The payload builder also clamps integers to the JSON-safe range, so caller-supplied counts can no longer reach the engine at that size Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): take spend locks in one order so the ledger cannot deadlock A ledger delivery updates many spend tables in one transaction, so it can deadlock with any writer that locks the same rows in another order. It took the team advisory lock after the user and team rows, while team roster writers take it first The delivery now takes every team advisory lock before any row, then writes tables in one canonical order with ids sorted. It also runs under the rollup lock_timeout, applies 500 deliveries per batch by default, and spreads retries with jitter The writers that could still collide with it now follow the same order. The budget reset cascade, model allowlist rename, global spend reset, bulk user update, org delete and the delete-by-list paths lock their rows in key order first, and access group endpoints update teams and keys in one sorted pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): prove a ledger delivery waits instead of deadlocking Real Postgres tests hold the first lock of a team roster change, an access group update and a budget reset cascade, start a delivery, and let the other writer finish. The delivery must wait and then charge once. Before the lock order fix each case ends in a Postgres deadlock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(accounting)!: make the SpendLogs ledger the default accounting path use_spend_logs_ledger now defaults to true, so every request is charged through LiteLLM_SpendLogDelivery. Setting it to false rolls back to the in-memory and Redis spend queues, which stay in place. Tests that cover those legacy writers pin the setting off, and the rest check the queued charge the ledger stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): let the mocked client open a transaction for key and team deletes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): take team locks first when a model rename rewrites allowlists A rename locked team rows before user rows while a roster change holds the team advisory lock and a user row, so the two could deadlock. The rename now takes the advisory lock for every team that lists the model, sorted, before any row lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): drain a backlog in one worker run, one batch per transaction One run applied a single fresh batch, so at 500 per batch charges fell behind any load above that rate. The worker now keeps claiming fresh batches while they come back full, for up to five seconds, and each batch still commits on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): queue only the charge when a request's spend log is suppressed With logging off for a request, the ledger still queued the full row, so a failed flush or a shutdown could leave its prompt and response in Redis. The queue now holds only the fields the charge needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(proxy): format the rename team listing query Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): remove expired delivery receipts as fast as the worker applies them The worker now applies several batches per run but still removed one cleanup batch of applied receipts, so the table could outgrow its retention window. Cleanup now repeats while each pass removes a full batch, for up to five seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): retry a model rename when a team starts listing the model mid-lock The team advisory locks come from a snapshot, so a team that gained the old name before its row lock was locked without its advisory lock. That attempt now commits without writing and the next one locks the fresh set of teams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
tin-berri
|
aa7917a3d2
|
feat(skills): show download counts and downloader history (#50165) | ||
|
tin-berri
|
79e63d3e06
|
feat(skills): add optional manual security scans (#50194) | ||
|
ishaan-j-berri
|
639625d679
|
fix(usage): restore per-user spend breakdown on the user aggregated route (#49464)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tin-berri
|
7e4ad6eb33
|
feat(skills)!: add portable submissions and admin review (#50160) | ||
|
tin-berri
|
0ad5d750d4
|
feat(skills)!: version published Skills Hub updates (#50152) | ||
|
ishaan-j-berri
|
da2ecbc5bb
|
feat: email affected owners before model deprecations (#50172)
* feat: opt-in model deprecation emails for affected owners * fix: annotate deprecation sender initialization * fix: clarify deprecation emails and use consistent settings toggles * fix: exclude former team admins from deprecation emails * feat: clarify provider deprecations and model migration steps |
||
|
tin-berri
|
3f4dc93b40
|
feat(skills)!: add portable Agent Skills Hub API (#50159)
Some checks failed
unit / auth-checks (push) Blocked by required conditions
unit / budgets (push) Blocked by required conditions
unit / custom-logging (push) Blocked by required conditions
unit / db-and-spend (push) Blocked by required conditions
unit / endpoints-and-responses (push) Blocked by required conditions
unit / enterprise-routing (push) Blocked by required conditions
unit / guardrails-hooks (push) Blocked by required conditions
unit / jwt-and-keys (push) Blocked by required conditions
unit / key-generation (push) Blocked by required conditions
unit / logging-misc (push) Blocked by required conditions
unit / mcp-oauth (push) Blocked by required conditions
unit / proxy-runtime (push) Blocked by required conditions
unit / proxy-server-core (push) Blocked by required conditions
unit / root (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
|
||
|
Nate Armstrong
|
9752869d84
|
feat(router): add the Rust router core with the host as its only boundary (#49513)
Adds the litellm-router-core crate, its step-driven PyO3 bridge (RouterCoreHandle / RouterCall) and RustCore (litellm/router_rust/), served behind LITELLM_RUST through the ROUTER route rule. The host call-out is the only boundary: every attempt goes through Host::invoke, there is no native executor path, and RustCore never builds, wraps or forwards to PythonCore, so router_python_core.py stays deletable. Attempt preparation lives in litellm/router_attempt.py so it outlives router_python_core.py. RustCore matches PythonCore's behavior, including its known bugs (fixed in follow-up PRs). Routing decisions (strategy picks, routing groups, PTU, pre-routing, pre-call checks, fallback walk, health routing, access groups, weights, cooldowns, allowed fails, usage) live in the crate; work that needs live Python objects (request shaping, endpoints, streaming, model info facts, callbacks, alerting, prompt management) lives in the host-side litellm/router_rust/ package (steps/, invoke/, surface/) driven by Rust steps. The crate owns the registry. It runs the deployment build loop (secret references, ids, organization fan-out, install order) and asks the host for what needs Python, one deployment at a time so errors surface where Python raised them: secrets, validate, provider and added steps. It holds the model list, the index maps and the pattern routers with the Python core's history rules (host_index.rs), and answers the registry lookups (lookup.rs). The Python views of that state in litellm/router_rust/surface/ (model_list, index(), names(), patterns()) read the handle, and every write to them becomes a changeset. Model info, deployment model info, configured limits and model group info are merged in the crate (model_info.rs, group_info.rs) from catalog rows and facts the host answers on a model_info step asked at query time, so price reloads and register_model stay visible. The usage cache key format is defined once, in the crate. The crate records each attempt's outcome itself rather than through litellm callbacks. A streamed attempt is deferred: it counts its request when the stream opens, and the host's stream wrappers report the success with the final usage, or a failure after the first chunk, when the stream ends, as the Python core's callbacks do. tests/unit/test_router/core characterizes PythonCore's public behavior and runs every test on both cores, with nothing pending; only the names of the Python core's own outcome callbacks are asserted on Python alone. litellm.Router is now final: subclassing it raises TypeError at the class statement, and the machinery that kept subclass overrides working is gone. The Rust core never honoured subclass overrides, so they would have silently stopped running there. Custom routing goes through register_plugin(), settings through apply() and outcomes through observe(). This is a breaking change for anyone subclassing Router. Fuzz parity fixes on top of the rebase: the Rust core keeps num_retries, max_fallbacks, allowed_fails, retry_after, timeout, the tag settings, routing_strategy_args and the fallback lists as given (the crate reads them as numbers, update_settings casts as Python does); a deprecated `model_list = [...]` write reaches the Rust core; a failed set_model_list is all or nothing on both cores (the deployments served before the call are put back and the error raised); config_deployments() is a generator and `del` of a plain attribute works on the Rust core. Least-busy pick reservations from main now work on the Rust core: its selectors share one reservation store, routed calls reserve and free their unstarted picks, and select() and batch cancel do not reserve. On the Python core a dropped upsert takes its name back off deployment_names, and get_deployment_order reads every numeric spelling as one int so mixed orders no longer raise TypeError. Rulings on the remaining differences, the Python core changed where it was the odd one out: sync picks and picks by deployment id apply the team filter as async picks do, so another team's deployment is refused; the fallback lists and default_litellm_params in RouterSettings are copies, so an in-place edit is ignored and settings change only through apply(); one health record with an unreadable timestamp drops out alone; an empty fallback entry is skipped; a bool cooldown_time is refused at construction, a bool or string cooldown_time on a deployment or a call falls back to the router default, and a failure status that no HTTP status can be is recorded as 500; an infinite or NaN cost and a whole number wider than 64 bits (ptu_shares) are refused on both cores with the same error (litellm/router_utils/deployment_numbers.py); and a /v1/messages stream whose provider iterator raises cools the deployment and retries like a non-stream call, with the Rust core waiting for that cooldown before its retry picks. Differences the proxy parity fuzz and its regression tests found on the Rust core: a file upload or a batch on an uploaded file falls back only within its own model group, as on the Python core, since the file lives with one provider (the host sends same_group_fallbacks_only and the walk skips targets in other groups); Decisions calls drop deployments whose provider has no Decisions config, and deployments that refuse the audio or videos a call carries, unless that would leave none, as the Python core does, while the retry count still reads the whole group; a file upload that fails on one order level is created on the next level's deployments rather than the first level again; and a response that cannot be converted (a text_completion body without a message) reports only its own traceback, so a rate limit handled by the attempt before it no longer turns the error into a 429, and both cores return 500. |
||
|
ishaan-j-berri
|
71fde48016
|
feat(alerts): show recommended model provenance and provider notices (#50162)
* feat(alerts): show recommended model provenance and provider notices * fix(ui): avoid nested recommendation source conditionals |
||
|
joshua-berri
|
49f56e8b23
|
refactor(proxy)!: drop the unused accounting event tables (#49923)
LiteLLM_AccountingEvent and LiteLLM_AccountingDelivery have had no reader since the SpendLogs projection replaced them, so the migration drops them with their enums. AccountingDeliveryState stays because LiteLLM_SpendLogDelivery uses it The schema now spells the delivery_id default the way Postgres stores it, which ends the drift Prisma reported against the ledger migration. The schema check compares base types only, so an attribute-only change is no longer reported as a type change Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
ishaan-j-berri
|
d2b2e76374
|
feat(proxy): add model deprecation alerts with daily requests by key (#50022)
* feat(ui): show recent model deprecations in notification bell * feat(ui): simplify deprecation request trends and add model identity |
||
|
Krrish Dholakia
|
c5da69cd13
|
feat(ui): show model deprecation banner on the models page (#50080)
* feat(ui): show model deprecation banner on the models page Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ui): format model deprecation banner Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): refresh model deprecations when models change Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): address review feedback on model deprecation banner Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): cover deprecation role gate and agent builder refresh Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): separate deprecation banner toggle from alerts link Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): dedupe and allow dismissing the model deprecation banner Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): refresh table once and key banner entries on successor Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: quiet successor logs, count banner models, use local dates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): pin banner test date and bucket upcoming entries locally Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover mixed-successor log level through the deprecations endpoint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): move deprecation banner cases into integration tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): keep deprecation banner when a refetch fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
906fa7ed28
|
feat(accounting): add opt-in SpendLogs ledger (#49856)
* feat(accounting): add opt-in SpendLogs ledger delivery * fix(accounting): preserve missing organization daily spend parity * fix(accounting): preserve ledger receipts when replaying migration * fix(accounting): preserve callback windows and tune ledger batches * fix(accounting): preserve window charges and bound ledger statements * test(accounting): verify budget callback registration order * test(accounting): verify captured charges after repeated startup * feat(accounting): store durable payloads independently of spend logs * fix(accounting): preserve first delivery and trace ledger writes * fix(accounting): keep SDK imports independent of proxy tracing * fix(accounting): honor log retention during batch takeover * test(accounting): cover retained batch alerts and invalid settings * test(accounting): verify downgrade command against pending work * fix(accounting): drain retries without starving fresh deliveries * test(accounting): cover delivery projection, metrics and window key reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): read delivery metrics without the shared registry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): charge a leftover $0 batch row when its claim fails A failed batch claim buffers the retrieve's row and reports it charged. At flush the row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back and the charge never reached the spend tables. The flush now takes that row over in the same transaction and keeps the delivery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): trace the batch takeover at flush as a SpendLogs update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client The export client is never created, so main always writes SpendLogs to the proxy's own database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was set, which dropped those request logs. It now treats a flush as an export only when the client exists, and batch claims write local rows as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep every request's charge when its log row is shared, evicted or rejected - Key non-batch deliveries on a per-request delivery_id so a reused client call id is still charged once per request, while buffer replays stay deduplicated - Persist the redacted api_key instead of an opaque bearer token - Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it - Strip NUL from payloads and skip non-text tags so projection never parks a paid request Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging a request whose tag is too long for the daily tag index A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store only guardrail billing fields in the accounting payload The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): move spend payload stamping out of update_database Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): restore the modules the import-layering guard evicts The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): keep update_database's payload stamping on its own local Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): type guardrail entries and use a public key redaction helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging requests whose end-user id is too long for an index An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): type the spend log queue snapshot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): leave out any projected row Postgres cannot index or store Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep the guardrail fields compression savings are counted from Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): cap projected counters so daily totals cannot overflow Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): let a buffered batch charge take over its zero-cost receipt A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read a rescued batch row's spend from its accounting payload Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read budget windows through the delivery transaction Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): retry a transient failure while keeping a dropped row's charge The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): give delivery projection payloads their charged entities project() now reads entities only from accounting_entities, so the hand-built delivery payloads carry them like real stored payloads do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): own ledger rows by their payload and key batches on the batch id Drop the accounting_path stamp and its SpendLogs column. A buffered row that carries an accounting payload is ledger owned, one without it is legacy, and spend counters get a plain windows_from_ledger flag. Window seeding tells applied ledger rows apart by their delivery row. A ledger batch is plain transaction A keyed by the batch id, so a repeated completed retrieve conflicts and charges nothing. A $0 retrieve records no delivery, a batch a legacy retrieve already charged is not charged again, and the legacy batch path is unchanged from main apart from refusing a batch the ledger already owns. Batch takeover and its delta bookkeeping are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): drop a projection test duplicated while rebasing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop previous_payload now that deliveries are never replaced Batch takeover was the only writer of previous_payload, so the worker applies each delivery's payload as stored and the column leaves the migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop the ledger downgrade checklist script The use_spend_logs_ledger flag is the rollback path, so a separate downgrade gate is no longer needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): name the window seed reads so their Prisma span resolves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): remove the unused accounting event types and protocols types.py, protocols.py and measurements.py were left from the old event scaffolding. Nothing outside the package imports them, so they go with their tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): park only deliveries whose own data fails A delivery that keeps failing on a database outage, lock timeout or deadlock stays pending and retries with capped backoff, so a recovered database applies it. Only a data error or an invalid payload parks the isolated row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a spend log row the insert statement rejects The raw SpendLogs insert validates columns and JSON before Postgres sees the row, so a row that fails there is now bisected out and dropped alone while its charge is kept, instead of failing and replaying the whole flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): carry a job's model window on every completed poll in ledger mode The ledger keeps the poll whose delivery claims the job, which need not be the poll that won the limiter's in-memory charge. Every completed poll now carries the job's window, so the delivery that lands always has it, while the live model budget is still charged once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): leave model-budget windows on the existing window writer Model-budget windows are written by the model budget limiter rather than derived from a request's spend, so they stay on main's window queue like live budget counters do. This drops window capture from the accounting payload, the deferred limiter registration and the job-window capture, and the ledger keeps owning key and team budget windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep a delivery pending when the database is unreachable Prisma reports a can't reach database server outage as a DataError, so the worker now checks for an unavailable database before treating a data error as the delivery's own fault. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate only rows the raw insert cannot encode Catching every ValueError also swallowed unexpected insert errors that main re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column or non-JSON value, and only that is isolated as a rejected row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store zero savings when caller data makes them uncomputable A huge caller-supplied tokens_saved overflowed the savings math inside build_accounting_payload, which dropped the request's charge in ledger mode. Savings that cannot be computed are now recorded as zero and the charge lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |