* fix(count_tokens): normalize Google system parts and null tools
* fix(count_tokens): preserve typed validator return
* fix(count_tokens): validate single system part without typing.cast
* fix(count_tokens): accept every Google spelling of a repeated field
Google countTokens now reads a repeated field sent as one object, null, or a
list with null items the same way as its list form, for system parts, tools,
and function declarations, matching what Vertex accepts.
* fix(count_tokens): read every Google spelling of contents and skip null system text
contents and each content's parts now read one object, null, and null items like their list form, the same as system parts and tools. A one-object parts used to translate to no messages, so non-Google models counted it as empty. A system part whose text is null is skipped, as Vertex AI does.
* fix(google_genai): read null-text media system parts as media and one-object content parts on generateContent
* fix(google_proxy): count null-text system parts that carry only part metadata, like Vertex AI
A system part with null or missing text and only metadata such as thought or thoughtSignature, or an empty part, now counts as empty instead of answering 400. Parts that carry an image or file with null text still answer 400.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(context_management): count the system prompt toward the clear_tool_uses input_tokens trigger
Anthropic's API compares an input_tokens trigger with the whole prompt, system
included; the polyfill counted messages and tools only, so it cleared late by the
system prompt's size. compact_20260112 already counts it: share its system_to_text.
* fix(context_management): take read-only types in the system prompt count
The type-discipline gate (LIT001) rejects new list and dict annotations.
* chore: rerun CI (proxy-build-from-pip timed out)
* fix(context_management): count the system prompt through compact's shared counter
Finishes the main merge's conflict in clear_tool_uses.py. The input_tokens trigger now adds the system prompt through count_system_tokens, the same counter compact uses, while the count reused for cleared_input_tokens stays messages and tools only, so the reported savings are just the cleared tool results
* perf(context_management): skip the token count when nothing can be cleared
The keep check runs before the trigger count, the system prompt reader accepts any mapping block, and the system prompt test gets short parameter ids
* test(context_management): keep the off-loop count test counting after the keep check moved first
* test(context_management): cover a client compaction summary counting toward the trigger
The trigger also skips the system prompt count when messages and tools alone already pass it
* fix(context_management): move a client compaction summary into the system prompt when clearing runs before compact
* fix(context_management): clear only the history after a client compaction block so compact still replaces its summary
* fix(context_management): keep tool calls beside a client compaction block clearable
---------
Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): enforce an end user's per-model budget from their linked budget
The shared auth checks now refuse a request once the end user's spend on
that model reaches the model_max_budget of their budget (a linked budget or
the default applied to them), for virtual keys and the master key alike.
The cap is read from the end-user row, since the master key's cached auth
object keeps whichever end user's limits it was cached with. The per-model
end-user check inside virtual-key validation is removed: it read a token
field that is never set there, so it never ran.
* fix(auth): enforce the end user's per-model budget on JWT auth and skip it on budget-exempt routes
JWT tokens now carry the end user's per-model caps so spend is charged against them and over-cap requests are refused. Routes that skip budget checks, like /spend/calculate, skip this check too. The proxy default end-user budget is covered by a test.
* fix(auth): write the end user's per-model cap onto every token and check it after the member fallback
The shared auth checks now resolve the end user's per-model cap, falling back to the key's or the proxy's default end-user budget when the end user has no row yet, and write it onto the token so the spend hook charges the cap the check enforces on every auth path. The check runs after the team-member budget fallback, on the model it picked, and the JWT builder's duplicate check is removed
* fix(auth): keep only a per-model end-user cap that custom auth itself set
A key-auth token reached through custom auth's auto hand-back now takes the current end user's cap, so a cached master-key or virtual-key object no longer carries an earlier end user's cap into another end user's request.
* test(auth): build the custom-auth token with model_copy instead of setting the flag in place
* test(auth): hold an end user to their own or their key's budget cap when the proxy has no default budget
* fix(proxy): give each request its own copy of a custom-auth token so end-user caps and ids stay per request
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cli): give lite codex aliases of a Codex-known model that model's catalog entry
lite codex built its Codex model catalog from /v1/models ids only, so a proxy
alias of a model Codex knows (api-gpt-6-luna for openai/gpt-6-luna) got the
fallback entry. That entry has no supports_search_tool or tool_mode, so Codex
sent every MCP and app tool inline and each turn cost far more input tokens.
lite now reads each listed model's upstream from the proxy's /v1/model/info
(team_public_model_name when set, else model_name, first deployment wins) and
passes it to the shared catalog builder, the same way the proxy's own Codex
route does. If that request fails, the catalog is built as before.
* fix(cli): give a lite codex alias an upstream only when all its deployments route to one OpenAI model
* fix(cli): cap a lite codex alias's context window at its listed max_input_tokens and count only OpenAI-hosted deployments
* fix(cli): keep Codex's tier, upgrade and reasoning defaults off lite codex aliases
* fix(cli): keep Codex's own effort, verbosity and promo off lite codex aliases and skip credential-hosted deployments
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): drop the null created_by OpenAI rejects when output items are replayed
* fix(responses): drop a null created_by from replayed input items too
A conversation saved from an older proxy still carries created_by: null on its compaction, apply_patch and shell items, so OpenAI keeps rejecting the replay. Drop the null from input items on the HTTP request transform and the WebSocket response.create rewrite, keeping a created_by that is not null.
* fix(responses): drop a null created_by from counted input items too
* test(responses): cover saved reasoning nulls and re-compaction of a saved conversation
* fix(responses): drop a set created_by from input under drop_params
* fix(types): type the Responses output serializer's parameters
* fix(responses): drop a set created_by on reasoning items and Perplexity presets under drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(azure_ai): cover Foundry agents on the project Responses API
* feat(azure_ai): reach Foundry agents through the project Responses API
* fix(azure_ai): keep system prompts and project-endpoint auth on Foundry agent calls and follow-ups
* fix(azure_ai): read AZURE_SCOPE before the Foundry audience default on project endpoints
* fix(azure_ai): reject or drop instructions on previous_response_id turns and a caller's other agent for Foundry agents
* fix(azure_ai): apply the other-agent rule to agent_reference sent at the top level of a Foundry agent request
* fix(azure_ai): match the caller's agent by name and version, and hide the agent's own instructions
* fix(azure_ai): keep instructions on plain Foundry models that return a null agent_reference
* fix(azure_ai): keep a Foundry agent version requested at the top level when extra_body also names the agent
* fix(azure_ai): answer 400 for /responses/compact on a Foundry agent instead of calling the project with the agent name
* test(proxy): keep the spend sidecar socket path under the macOS AF_UNIX limit
* test(unit): share the short socket fixtures across unit tests
Move socket_dir and socket_path to tests/unit/conftest.py so the datadog
redis test's absent socket fails for the right reason on a Mac, rename
the pgbouncer helper's directory argument, and record the rule in
tests/unit/AGENTS.md
* test(unit): fall back to /tmp when TMPDIR leaves no room for a socket path
socket_dir now uses /tmp when TMPDIR plus the longest socket suffix
would pass macOS's sun_path limit, and its teardown ignores errors so a
failed test reports only its own failure
* test(unit): keep socket fixtures inside tmp_path behind a short link
socket_dir now links a short temp directory to tmp_path, so artifacts keep
pytest's retention and a killed run leaks only an empty link. The fallback
to /tmp counts the temp directory in bytes, and socket_tempdir points
tempfile at it for the pooler tests whose socket directory production picks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(router): reset_time picks when deployment rpd and tpd daily limits start over
reset_time is "HH:MM" in UTC next to rpd/tpd in litellm_params or model_info. The daily cache keys are dated by now minus that offset and the daily 429 retry-after counts to the next reset, so a free tier whose provider quota resets at 08:00 UTC comes back at 08:00 instead of midnight. Without it the window stays the UTC calendar day.
* chore(ui): regenerate schema.d.ts with reset_time
* refactor(router): rpd, tpd and reset_time are read from litellm_params only
* feat(router): rename reset_time to daily_reset_time, drop the request-sent value, accept only HH:MM
* chore(ui): regenerate schema.d.ts for daily_reset_time
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): list TypeSafe Jev models in the admin model picker
* fix(proxy): let typesafe/* allowlists cover bare /typesafe Jev ids
Adds a typesafe-jev-ids routing rule to the cost map so a bare jev-* id resolves to TypeSafe, which lets typesafe/* on a key, team or project allowlist match /typesafe calls. The picker gate now only checks that a TypeSafe key is set, so an unresolvable OIDC reference cannot fail the listing, and the /v1/models route tests mock deployments with litellm_params like real ones.
* fix(typesafe): match only published Jev ids and keep picker loads off the secret manager
Narrows the typesafe-jev-ids rule to jev-latest, jev-preview and versioned ids so a model group such as jev-internal-router never resolves to TypeSafe, checks TYPESAFE_API_KEY in the environment instead of through the secret manager on each picker load, and moves the picker tests into the mapped test file.
* test(typesafe): type the picker test fixture and tests
* fix(typesafe): match only published Jev versions and cover non-version suffixes in tests
* fix(typesafe): match only priced Jev ids and offer TypeSafe in the no-team key picker
* fix(proxy): offer TypeSafe allowlist names only to the Admin UI allowlist pickers
The dashboard sends include_model_access_groups on every /models call, so
gate the TypeSafe names on wildcard routes too and skip access-group-only
listings. Also say the typesafe-jev-ids rule matches case-insensitively.
* fix(ui): offer only the TypeSafe names the cost-map rule routes, and keep /v1/models listing as on main
A typesafe/* router deployment registers typesafe/* in the cost map, so the
allowlist pickers offered a bare *, which allows every model. The pickers now
list only priced names the rule sends to TypeSafe. The non-expand list also
stops de-duplicating every entry and only skips TypeSafe names already listed.
* feat: add StreamLake provider metadata
* fix(streamlake): complete provider discovery and test request flows
* test(streamlake): assert cost and tool-param invariants instead of pinning vendor prices
* fix(streamlake): accept tools and reasoning on endpoint ids and match the current price page
Endpoint ids have no cost-map row, so tools and reasoning_effort were rejected
before reaching StreamLake. The provider now declares both capabilities, and
named non-reasoning rows opt out explicitly.
The rows now follow StreamLake's 2026-09-17 price page: drop the KAT-Coder and
Qwen3 rows it no longer serves, add the served models that had no price, drop
account-level rpm and tpm, and fill missing context windows. StreamLake is also
registered for streamlake/* wildcards and its messages and responses support.
* fix(streamlake): regenerate the endpoint matrix and type the new tests
The backup endpoint matrix advertised chat only while the root map listed
Messages and Responses, so /public/endpoints hid both. Regenerating with
scripts/generate_provider_endpoints_support.py makes the two files match.
Types the StreamLake tests and sets the Add Model test's API key with
fireEvent.change, per the dashboard test guide.
* fix(streamlake): drop the retired max_tokens field from the StreamLake cost-map rows
* fix(streamlake): price the four served models StreamLake publishes rates for and add the Qwen3.8 cache prices
* fix(streamlake): send token caps as max_completion_tokens and mark image models
StreamLake ignores max_tokens on kimi-k2.6, kimi-k2.7-code, qwen3.6-27b and
qwen3.6-35b-a3b, while every served model honors max_completion_tokens, so
the JSON provider now maps max_tokens to max_completion_tokens instead of the
reverse. Rows whose models read an image in a live probe set supports_vision,
and the Qwen3.8 rows drop the explicit cache write price, which this route
never uses since cache_control is dropped
* fix(streamlake): keep cache_control marks, price explicit cache reads, and use API input limits
StreamLake reads cache_control, so the provider keeps the marks through a preserve_cache_control flag. Qwen3.8 explicit cache hits now use their own read price, rows with a cache read price declare prompt caching, three models StreamLake serves without reasoning drop reasoning_effort, and max_input_tokens follows the limits the API reports.
* fix(utils): warn when a custom_provider_map name matches a built-in provider
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): skip embeddings input in google_text_moderation and openai_moderations
Since v1.103.0 the proxy dispatches these legacy moderation callbacks on every route, and they read the embeddings input field as text, so benign embedding inputs were blocked. Return early for embedding call types, matching the Aim and Cato guardrails.
* test(guardrails): type the embeddings moderation test parameters
* fix(guardrails): keep openai_moderations moderating embeddings, skip only google_text_moderation
Google's moderation scores topic categories like Finance and Health, so it blocks
documents being indexed by subject. OpenAI's moderation scores harm categories only,
so it keeps moderating embedding input as it has since v1.103.0
* fix(guardrails): skip Gigachat passthrough embeddings in google_text_moderation
Treat an allm_passthrough_route call whose endpoint ends in embeddings as
non-conversational, log the skip, return None from the early exit, and type
the hook's data so the basedpyright gate does not grow
* fix(guardrails): match only plain embeddings passthrough paths in google_text_moderation
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): scan Responses tool results in texts-mode guardrails
* fix(guardrails): keep guardrail-added tools and non-text parts on Responses tool scans
Tool result texts now come from the same conversion that builds the chat tool message, so items without a call_id, dict outputs, and outputs with no text parts stay aligned for Bedrock and PANW. A changed joined text replaces the first text part and keeps the other parts. scan_only_tool_results appends tools the guardrail returns with new names, like chat, and tool_result and computer_call_output items are scanned through the shared type list.
* fix(guardrails): skip empty and screenshot tool outputs and keep text part types on Responses scans
* fix(guardrails): keep Bedrock latest-turn masking aligned when tool outputs send no text
* fix(guardrails): scan each Responses tool output text part on its own
* fix(guardrails): split a Responses tool output's structured row when only textless outputs share its call_id
* fix(guardrails): pair each Responses tool output with its own structured row
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): drop metadata from compact requests to OpenAI and Azure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop user and drop_params-rejected fields on compact and route Azure compaction to v1
* fix(responses): keep stream on compact and keep non-dated Azure api-versions on the v1 compact route
* fix(responses): drop compact params only for providers verified to reject them
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): copy the stream's hidden response_cost into the client's streamed usage.cost
The stream is priced once into hidden response_cost (stream builder for chat and text completions, the Responses iterator for the terminal event) and the proxy egress copies that bill into the streamed usage.cost on a copy of the event. Inside the pipeline usage.cost stays what the upstream reported. Spend rows do not change; the billing fix for upstream-reported usage.cost follows in the next PR cut from #48287.
* fix(proxy): type the bridge bill carry and the builder's reported-cost guard for the lint gates
* style: format the chat-over-Responses bridge helper signature
* refactor(bridge): build the billed Responses event with model_copy instead of a TypeAdapter chain
* fix(stream): keep writing the computed cost into usage.cost as on main, the hidden copy is only for the proxy egress
* fix(stream): write litellm's bill into the usage frame's usage.cost instead of a hidden field the proxy copies
* test(stream): keep main's formatting in test_streaming_handler.py, only the new test changes
* refactor(stream): drop the docstring on _stamp_usage_cost, the body says it
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): stop dumping the request body at every layer under --detailed_debug
The proxy dumped the full request body at every layer it passed through
(proxy entry, pre-call, router, SDK call echo, param mapping, logging
object), so one Codex "hi" wrote over 500KB of debug log. Keep the inbound
request line and the outgoing provider request, summarize large values in
the SDK call echo, and stop per-request lines that report nothing
(callback disable checks with no disable header, cost map name probes,
response id decode misses, vector store path misses).
* fix(logging): summarize tools in the param mapping debug lines
Non-Default params and Final returned optional params still printed full
tool definitions, so a Codex-style request repeated them twice more. Use
the same size summary as the SDK call echo and cover tools in the test.
* perf(logging): build the param mapping debug summary only when debug is on
* fix(bedrock): omit system prompt from audio transcription
* fix(bedrock): report Converse token usage on audio transcription
Bedrock transcriptions returned usage null and were billed at $0. Map
Converse inputTokens, outputTokens and totalTokens into the transcription
usage so spend uses the model's per-token price. input_token_details is
optional because Converse does not split audio and text tokens.
Also switch the updated request-shape test to rstest.
* fix(bedrock): default transcription temperature to 0 and price ap-southeast-2 Voxtral regionally
Voxtral wrapped some transcripts in quotes at its default sampling
temperature. In direct Converse calls in ap-southeast-2, 0 of 48 calls at
temperature 0 were framed against 7 of 72 at the default, and instruction
wording made no clear difference, so the request now sends temperature 0
when the caller sets none, matching OpenAI's transcription default.
Calls in ap-southeast-2 billed at the us-east-1 price. The cost map gains
bedrock/ap-southeast-2 Voxtral rows with all four service tiers, the Bedrock
transcription response carries the deployment region in its hidden params,
and the transcription token cost uses the region-resolved model key.
The bridge now builds the response with model_copy instead of assigning
usage after construction, and the native wheel test checks the usage the
compiled route returns.
* fix(bedrock): forward transcription language, reject unsupported formats, and bill env and Tokyo/London regions
get_optional_params_transcription had no Bedrock branch, so language,
response_format and timestamp_granularities were dropped before the Rust
transform ever saw them. Bedrock now forwards language, prompt and
temperature, accepts response_format json or text (the only shapes it can
return), and answers 400 for srt, vtt, verbose_json or
timestamp_granularities unless drop_params is set.
The billing region now falls back to AWS_REGION_NAME and then AWS_REGION
when aws_region_name is unset, the same order the Rust resolver uses to
pick the endpoint, so an env-configured region gets its regional price.
Adds ap-northeast-1 and eu-west-2 Voxtral rows (standard, batch, flex and
priority) from the AWS Bedrock offer files, the other two regions AWS
serves Voxtral audio in.
* fix(bedrock): send transcription temperature as a number and name the rejected param
The proxy keeps multipart form values as strings, so -F temperature=0.2
reached Converse as "0.2" and Bedrock answered 400. The Bedrock mapper now
converts temperature to a float and answers 400 (or drops it under
drop_params) when it is not a finite number.
get_optional_params_transcription copied prompt and temperature back from
the raw call after the Bedrock branch ran, which undid that conversion and
any drop. The Bedrock branch now owns every OpenAI transcription param.
An unsupported param such as timestamp_granularities now gets a 400 that
names it instead of the generic "user/encoding format" message.
* refactor(bedrock): return unsupported transcription params as a value and raise at the param step
* fix(bedrock): drop the retired max_tokens field from the regional voxtral price rows
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(health): ask OpenAI and Azure audio chat models for audio output in the health check probe
* fix(health): check the probed model when health_check_model is set, and mark Azure audio models as audio output
A deployment with an audio base_model and a text health_check_model got the audio fields sent to its text probe model. The Azure gpt-audio and gpt-4o audio preview cost-map entries now set supports_audio_output, so a dated Azure name like azure/gpt-audio-1.5-2026-02-23 gets the audio probe too
* fix(health): read litellm_params.base_model, and drop the stored base_model when Test Connect swaps the model
* fix(health): judge the audio probe on a swapped-in health_check_params model, and keep Test Connect's stored base_model
* fix(health): look up a provider-prefixed base_model under its own provider and keep text-only operator modalities
* fix(vertex_ai): drop the caller's anthropic-version header on Claude /v1/messages
Vertex reads the API version from the body's anthropic_version and answers 400 when a forwarded anthropic-version header names a different one, which anthropic SDK 2.x always does. Strip the header on the Python and LITELLM_RUST Vertex messages routes.
* fix(vertex_ai): carry the caller's anthropic-version into the Claude /v1/messages body
Vertex reads the API version from the body alone, so the caller's
anthropic-version header now becomes the body's anthropic_version instead
of a pinned vertex-2023-10-16. The 2023-01-01 and 2023-06-01 first-party
versions keep vertex-2023-10-16, and the header itself is still never sent.
Same change on the Python proxy and the Rust gateway.
* fix(vertex_ai): follow drop_params for an anthropic-version Vertex does not accept
Only versions Vertex accepts in the body (vertex-2023-10-16, 2026-10-21) are
carried into anthropic_version. 2023-06-01, 2023-01-01, a blank header, or no
header keep vertex-2023-10-16. Any other value drops to vertex-2023-10-16 under
drop_params and answers a 400 naming drop_params without it, on the Python proxy
and the Rust gateway.
* refactor(vertex_ai): return the unsupported anthropic-version error as a value and keep Rust helpers and tests in place
* fix(rust gateway): resolve os.environ drop_params and accept the same flag words as Python
The gateway read drop_params: os.environ/NAME and words like yes as off, so the drop_params opt-out from the new anthropic-version 400 did not work there
* docs(rust router): name Router::from_config in the crate README
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
OpenAI shut down dall-e-2 and dall-e-3 on 2026-05-12. Drop their image
generation and edit configs, the Azure dall-e-2 create-and-poll flow, and
the silent dall-e-2 fallback when no model is passed.
image_generation without a model or provider now fails locally with
"model is required". image_edit and image_variation leave model out of
the provider request instead of sending dall-e-2, so providers that
ignore model (Topaz) keep working and OpenAI returns its own 400.
Add rustdoc overviews for a new litellm-router-core crate and for
litellm-model-catalog, a module docstring on litellm/router.py, and a
shrink-only allowlist of the Router members and router_utils modules used
outside the router, checked in CI by router_surface_check.py.
Move the router-free helpers out of litellm/router_utils: batch_utils to
litellm_core_utils, and the auto-router naming contract, tuning baseline
and baseline request next to the auto-router plugin. No behaviour change.
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream
The Gemini and Vertex AI streaming path read the SSE body with httpx
iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks
on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085,
U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds
one of them was cut in two. The tail is not a data: line and the head never
parses, and the accumulate-and-retry fallback then carries the unparsed
prefix forward, so every later event is appended to it and the stream ends
with no more content and no usage.
SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines
split decoded text on exactly those, treating a CR at the end of one chunk and
an LF at the start of the next as one break. The first request and every
long-decode hop now read through them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF
Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks.
* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator
* fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed
Switch the remaining httpx stream readers (Anthropic chat, Google GenAI,
Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent
Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and
others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a
raw U+2028 or U+2029 inside a JSON string no longer splits an event.
Keep the chunk that closes a think block in the logged and cached response
when a stream ends without a finish_reason, close the block before a legacy
function_call, and make sync Codestral FIM streaming send stream=true and use
the text-completion-codestral stream parser.
Test fakes that only mocked iter_lines now stream real httpx bytes and
assert the content they feed in.
* fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests
Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on
U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use
split_sse_lines from sse_lines.py, which also gains a str.split fast path.
Adds tests/unit/llms/codestral to the unit workflow shard.
* fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines
* test: type the new stream tests, drop a comment, and benchmark the SSE line splitter
* test: type the iterator class parameters in the line-separator stream tests
* fix(responses): stream tool call arguments opened in a text chunk
* fix(responses): hold a tool call that shares a chunk with text until the text ends
* test: type the a2a SSE handler helper and the sandbox fake responses
---------
Co-authored-by: Tim Vink <vinktim@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
* fix(gemini): preserve every grounding web source
* test(gemini): cover all grounding sources per passage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Zabir Raihan <zabirraihan@icloud.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): replace Any with proven types in 52 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): use typing_extensions.TypedDict for validated seams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): revert strict Anthropic response validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): accept a null finish reason from Bedrock Cohere and TwelveLabs Pegasus
* test(bedrock): annotate the Pegasus null finish reason test
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch): strip Claude Code instructional prefix in short-circuit search
Signed-off-by: agustin18 <agustinsaiz02@gmail.com>
* fix(websearch): preserve quotes for unwrapped search queries
Signed-off-by: agustin18 <agustinsaiz02@gmail.com>
* test(websearch): replace mock with pure unit tests and cover empty queries
Signed-off-by: agustin18 <agustinsaiz02@gmail.com>
* style(websearch): fix ruff formatting blank line in handler
Signed-off-by: agustin18 <agustinsaiz02@gmail.com>
* refactor(websearch): enforce Final annotations, prevent rebinding, and retain e2e coverage
Signed-off-by: agustin18 <agustinsaiz02@gmail.com>
* test(websearch): assert server_tool_use block unconditionally
Signed-off-by: agustin18 <agustinsaiz02@gmail.com>
* fix(websearch): search only the query inside Claude Code's web search prompt
Match Claude Code's exact "Perform a web search for the query:" prefix instead of a broad regex family, keep any quotes the user typed so exact-phrase searches still work, and test the search request at the HTTP boundary.
---------
Signed-off-by: agustin18 <agustinsaiz02@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): accept request priority from the x-litellm-priority header
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only apply x-litellm-priority on scheduler-queued routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply x-litellm-priority only where the router schedules the request
The header now maps onto priority only for chat and text completions that a
router serves, so the Cursor route's responses call and proxies started without
a config never forward it to the provider. /queue/chat/completions now honors
the header too, with the header winning over a body priority.
* test(proxy): use a real ProxyLogging in the priority header routing test
* fix(proxy): ignore a malformed priority header under drop_params instead of dropping the body priority
* fix(proxy): treat an empty priority header as unset and pass a valid one on as an integer
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(tencent): add Seedream image generation and editing through TokenHub
* feat(tencent): price Seedream image generation and edits from TokenHub usage
* test(tencent): cover Seedream image generation, edits, and pricing through TokenHub
* fix(tencent): keep non-Seedream image models on the OpenAI-compatible route
Only seedream- models use TokenHub's wand route, so Hy-Image and other
tencent image models keep working as on main. The response now carries
TokenHub's image size and format, Pro's flat fallback bills its default
2K size, and the new tests are typed.
* fix(tencent): bill other Tencent image models as before and forward user and TokenHub params on edits
* fix(tencent): keep dropped edit params out and parse form-string watermark and seed
* fix(tencent): send extra_body edit fields to TokenHub as given
* fix(tencent): keep the routed model on Seedream edits and bill a deployment's per-token price
* fix(tencent): accept string drop params on edits, rebuild nested form fields, and honor a per-image deployment price
An image edit with a non-list additional_drop_params answered 500 for every provider after this branch validated it as a list, so a non-list value now drops nothing again, as chat treats it.
Seedream edits sent through the OpenAI SDK arrive as multipart form fields, so TokenHub-only booleans and integers are parsed and bracketed keys like optimize_prompt_options[mode] are rebuilt into objects. A deployment that sets only output_cost_per_image is billed that price, and the TokenHub URL uses the shared api_base helpers.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(router): add deployment rpd and tpd daily limits to enforce_model_rate_limits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): route rate limit cache increments through typed helpers
* chore(ui): regenerate schema.d.ts for deployment rpd and tpd fields
* test(types): add rpd and tpd to the owned litellm_params inventory
* fix(router): run daily rpd/tpd checks after the minute checks and increment rpm and rpd in one cache trip
* fix(router): flatten the async rpm reject branch for the ruff strict gate
* fix(router): hand rpm and rpd back on any rate limit reject
A request the rpm, rpd or tpd check rejects no longer keeps its request counts, so retries against a deployment whose rpd is spent cannot fill its rpm and block the in-group fallover with the minute 429.
* test(router): a minute-rejected request no longer keeps its rpm count
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(skills): add a litellm-setup agent skill
* fix(skills): run the quickstart for Docker with Postgres in litellm-setup
Use scripts/quickstart.sh for the Docker plus Postgres path instead of a
hand-rolled Postgres container, keep uv with the user's own Postgres, load
.env as data instead of sourcing it, pin the uv Python range, chmod an
existing .env, list bedrock_converse and vertex_ai rows in the cost map
lookup, ask before editing the shell profile, and stop the health wait
when the proxy exits
* fix(skills): guide empty Postgres, docker sudo, and key fixes in litellm-setup
Ask for a new empty database on the uv path and DIRECT_URL behind a pooler,
add the baseline error to troubleshooting, and drop inherited database
settings in the loader line. Check python3 and lsof, explain Docker's
permission denied, give quickstart commands by the folder's path, and fix
provider keys by recreating the container or deleting the quickstart model.
Drop the structure-only test.
* fix(skills): keep shell profiles untouched when litellm-setup installs uv
The uv installer edits shell profiles unless UV_NO_MODIFY_PATH is set, so the skill now sets it and puts ~/.local/bin on PATH for later commands. It also says litellm --setup runs in a separate terminal and how to retire the shared master-key UI login
* fix(skills): offer litellm --setup only on the uv path
The Docker paths install no host litellm CLI, and the old wording also passed the skill's own step number on to the user
* fix(skills): ask before using an env provider key, print model names config.yaml accepts, probe the port with curl
* fix(skills): state the env key default in the question, chmod the wizard config, keep uv's bin dir on PATH
* fix(skills): keep litellm-setup on its own .env and install when the shell has others
* fix(skills): read the setup wizard's config without printing keys and hand off the exact restart command
* fix(skills): hide the database-access dev skill from end-user skill installs
* fix(skills): give the litellm --setup path a restart command that keeps its config
* docs(skills): point litellm --setup users to the master key in litellm_config.yaml
* fix(skills): hide every non-end-user skill, use Azure deployment names, keep the restart block out of the user's shell
* test(skills): find non-end-user skills in-process instead of through git
* docs(skills): price Azure deployments with base_model and put uv on PATH in the start block
* feat(ui): let the per-model budget field take a model pattern or access group
* feat(proxy): let model_max_budget entries name a model family or access group, shared across replicas
* fix(proxy): open each shared model budget window once for every replica, and match patterns in linear time
Without Redis, the first replica to charge an entry in a new window now opens it in the window spend table and
the others join it, so no flush replaces spend another replica already added. Budget patterns match by splitting
on `*` instead of a regular expression, a refused request names the entry that refused it, and the key form no
longer offers a typed pattern another row already budgets.
* fix(proxy): read the shared budget window from the writer, and keep concurrent first charges on one replica
Opening a shared window now reads the row from the writer, since a lagging read replica could hand a replica its
own start and let a flush replace the others' spend. A charge that waited on the open no longer resets a local
window another charge on the same replica opened meanwhile. Window tests use fixed timestamps.
* fix(proxy): stop a non proxy admin from lifting a team model cap with a looser group or pattern entry
Group and pattern entries now take precedence over other entries, so a team update by anyone but a proxy admin
must keep each new or changed entry within every existing cap it can take precedence over. A pattern can match a
provider-prefixed or wildcard-routed name, so any two patterns count as overlapping, and a model or group entry
counts as outranking every pattern.
* fix(proxy): narrow model budget pattern overlap, keep team caps under key patterns, and accept only typed patterns in the UI
* fix(proxy): keep model family budgets through database blips, team models and key entries
A failed window open no longer restarts the window at this replica's clock, so it cannot wipe shared spend. Spend charged meanwhile stays local and joins the shared window once the database answers
A key entry named after a model replaces only the team entry of that name, so team pattern and group caps still apply. Team-scoped models are gated and charged under their public name, and group lookups see team deployments
The team update guard compares caps by window length and only ranks patterns that can match the same name, including Bedrock vendor patterns behind a bedrock wildcard route
* fix(proxy): refuse team entries that take precedence over a capped model_max_budget entry
A non-proxy-admin could add a carve-out (an exact model, a narrower pattern or
a group) at or under the family cap, and its own counter then multiplied the
team's spend on that family. Any new entry that takes precedence over a capped
entry now needs a proxy admin, and group members include the team's own public
model names so team deployments cannot hide a group overlap.
* test(router): cover team_public_model_names_of
* docs(proxy): name pattern and access group keys in model_max_budget docstrings
* fix(proxy): name the entry holding the cap in the team model budget raise refusal
* fix(proxy): check precedence when a team edit makes an unenforced model_max_budget entry take effect
* fix(proxy): let an org admin give a legacy model_max_budget row a budget_duration at the same or a lower cap
* fix(proxy): keep a legacy model_max_budget row's stored cap when an org admin gives it a budget_duration
* feat(daily-activity): read guardrail spend per key from the daily guardrail usage table
Adds DailyActivityRepository.guardrail_spend, which sums LiteLLM_DailyGuardrailUsageUnits.cost by date, guardrail and key. A guardrail row counts only when its key has a spend row on that day under the same scope filters as the rest of the daily activity response, so user, key and date filters apply to guardrail cost too
* feat(daily-activity): add a per-guardrail spend breakdown to /user/daily/activity/aggregated
Each day now has breakdown.guardrails[<guardrail_id>] with metrics.spend and an api_key_breakdown carrying each key's spend and metadata. Guardrail entries only report spend, since that is the one metric stored per key. The breakdown is on the user endpoint only, is empty when a model filter is set because guardrail cost is not stored per model, and lists only keys within api_key_limit while the guardrail total keeps every key. spend and budgets are unchanged
Resolves LIT-9785
* feat(daily-activity): add the guardrail breakdown to team and organization activity, with guardrail names
/team/daily/activity/aggregated and /organization/daily/activity/aggregated now return breakdown.guardrails the same way the user endpoint does, since each key belongs to one team and one organization. Each guardrail entry also carries metadata.guardrail_name, read from LiteLLM_GuardrailsTable for guardrails created through the UI or API and falling back to the breakdown key for config.yaml guardrails, whose key is already their name
* fix(daily-activity): keep daily activity up when the guardrail breakdown query fails, and treat an empty model filter as none
A database error from the guardrail spend query now logs a warning and returns the response without guardrails, where it used to fail the whole aggregated response with a 500. An empty model= no longer drops the guardrail breakdown, matching how the rest of the endpoint already ignores it. The daily activity golden files and fixtures gain the empty guardrails key every breakdown now carries
* refactor(daily-activity): fold guardrail breakdown in a worker thread and drop docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(daily-activity): report untracked guardrail units next to spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(migration_data_checks): size btree entries by the bytes Postgres stores (LIT-10244)
* test(migration_data_checks): flag exactly the compressed keys Postgres rejects at the entry limit (LIT-10244)
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(aiohttp_transport): keep the owned session open while a response stream is still being read (LIT-10237)
* test(aiohttp_transport): annotate the stream regression test locals with Final and drop its docstring
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pricing): add databricks priority pay per token prices
* fix(pricing): bill gemini 3.6 flash cache writes at its databricks input rate
* fix(pricing): drop image token priority rate the cost map schema does not carry
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): make hide-secrets honor skip_tool_message_in_guardrail
With the flag on, per guardrail or in litellm_settings, hide-secrets leaves
role: tool messages and Anthropic tool_result blocks as sent, as the
unified guardrails do. Tool call arguments are still masked.
* test(guardrails): type the skip-tool walker test and assert the whole request
* fix(ci): spend logs guard requires the approval to cover the head's spend-log lines
An approving review survives later pushes (dismiss_stale_reviews_on_push is off on main), so a PR approved once could add more LiteLLM_SpendLogs schema or migration changes and the guard kept passing. The guard now reads the review's commit_id and passes only when the approval is on the head, or the head adds no spend-log line the approved commit did not already carry.
* fix(ci): keep approved_by untouched, add latest_approval beside it for the stale-approval check
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(files): cover owner-scoped file listing under require_managed_files
* fix(files): list only the caller's managed files on every list route under require_managed_files
* fix(files): narrow owner-scoped managed file lists to the named provider or model and page with after_id and before_id
* fix(files): narrow scoped lists the way uploads route, and refuse lists that cannot be scoped
A model name now matches every deployment that serves it, wildcards and
aliases included, and a provider matches deployments by the provider
uploads resolve, so anthropic/* and unprefixed Claude models count. The
OpenAI and Azure lists also return the caller's pass-through uploads.
An unknown provider and different after and after_id cursors answer 400,
and a list with no matching deployment skips the database read.
* fix(files): match stored files by model name and keep flag-off lists as on main
Under require_managed_files, a narrowed list now matches a file's model
mappings by deployment id or model name, so files LiteLLM stores in its own
database show on their model and provider lists. The Anthropic list defaults
to 20 rows when no limit is sent, after_id, before_id and the cursor 400s
apply only under the flag, and an admin keeps the provider list when the
managed files hook is not loaded. List errors are returned as values and
mapped to HTTP errors in one place.
* fix(files): keep admin provider lists and index the owner-scoped walk
Under require_managed_files, an admin keeps the provider list as on main, and
only a non-admin list reads the caller's own and team managed files. A stored
file now also shows on the provider its model name routes to through a
wildcard, with exact model names first, and a list routed to a model takes the
default page size of the provider serving it. A new index on created_by and
created_at keeps each chunk of the walk an index read, and the model check
runs before the file object is parsed.
* fix(files): filter provider and model lists in the query
A non-admin list scoped to a provider or model now asks Postgres only for the
caller's rows that map to one of that scope's deployments or models, or that
live in file storage, instead of reading every owned row and filtering them in
Python. A caller with many files on other providers no longer waits for the
whole set to be read. Team members' files are matched in the same query.
* test(files): type the fake raw query in the files list tests
* refactor(proxy): list managed files through one typed database handle
* fix(files): list alias and routing-group storage uploads on their provider
A Responses API stream that fails partway now bills the usage it already consumed, ends with an error instead of a clean stop, and honors stream_timeout between chunks. Bridged /v1/messages calls report those failures to the proxy's failure hooks, so spend, budgets and logs see them
* fix(streaming): surface upstream abort finish reasons as errors
* fix(streaming): keep SDK chunks with null choices streaming past the abort check
* fix(streaming): treat null choices as empty in the raw SSE abort check
* refactor(streaming): bind raw SSE choices as Final instead of rebinding
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(rust-bridge): require the native extension
_native ships in every wheel and the sdist will not install without it, yet the bridge loaded it optionally and carried None paths that could fail open for fork safety, exception mapping and OTLP errors. Delete loader.py and import _native directly. Tests inject a fake module through NativeBinding(source=...) or with_source() instead of patching the loader
The install smoke checks now verify the compiled module, and tests fail instead of skipping when _native is missing. The interactions spec canary installs the project so it has the extension
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ci): build Rust bridge in Windows smoke tests
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(aiohttp_openai): forward service_tier in transform_response
* chore(aiohttp_openai): mark service_tier assignment rebind-ok for the type gate
* fix(aiohttp_openai): set service_tier with setattr like other providers
ModelResponse has no declared service_tier field, so attribute assignment
added a reportAttributeAccessIssue error to the type gate.
* test(aiohttp_openai): verify serialized served service tier
* fix(aiohttp_openai): copy response when preserving served tier
---------
Co-authored-by: tin-berri <tin@berri.ai>
* test(cli): print-token and whoami let the proxy judge a classic token past the client window
* fix(cli): let the proxy decide when a classic lite login token has expired
* fix(cli): treat 403 and 404 as a rejected token and judge lite up logins the same way
* fix(cli): keep lite up on the client window and bound the proxy check to 3 seconds
* test(cli): prove the proxy check gives up on a silent proxy and pin stale fixtures
Point the timeout test at a local socket that never answers, use fixed
timestamps for stale CLI token fixtures, and correct the print-token docs
for tokens younger than the client window.
* fix(cli): expire a CLI token with no recorded proxy and stop treating 404 as rejected
A token file without base_url cannot be checked, so it keeps the client
window's verdict instead of printing forever. A 404 also comes from a proxy
that does not serve /v1/models, so only 401 and 403 count as a rejected key,
matching the CLI's existing key check. The timeout test now runs in process.
* test(cli): judge the whoami proxy check on a fixed timestamp and name the client window for lite up
