* fix(router): cap the default OTPM reservation for requests without max_tokens
* fix(router): reconcile ITPM and OTPM reservations after every successful call
* fix(router): read the default OTPM reservation from the served model and make its fraction configurable
The default reservation now looks up max output from litellm_params.model, as main does, so a pricing base_model never changes it. NO_MAX_TOKENS_TPM_RESERVATION_FRACTION sets the fraction for both the router check and the key and team TPM floor. A max_tokens that is not a number or string counts as no cap instead of raising inside the check.
* fix(router): refund a failed attempt's io token reservation in make_call
The proxy passes one logging object to every retry, and it runs failure
callbacks only for the first attempt, so the last attempt's ITPM and
OTPM reservation stayed held until the minute ended. Each attempt that
raises now gives its reservation back before the retry or fallback
* fix(router): count Anthropic cache writes once in ITPM reconcile
Native /v1/messages usage reports input_tokens without cache reads and writes, so the reconcile now maps it through AnthropicConfig.calculate_usage before subtracting cached tokens.
* fix(router): use a deployment max_tokens, charge late usage to its finish minute, and read the reservation fraction as a fraction
* fix(router): reserve with the deployment cap only through the param the request's API sends
* fix(router): never let a deployment's output cap raise the reservation above the default
* test(router): move the reservation fraction env var test to the integration suite
The unit test started a child interpreter, which the unit guide forbids. The integration test boots a real proxy with the env var set and checks admission against the held reservation.
* test(router): retry the reservation fraction scenario when the minute rolls over
The OTPM counter is keyed by minute, so a rollover between the held stream and the capped requests gave the capped request a fresh counter. The control request now leaves room for usage the first attempt left in the new minute.
* test(router): give the rollover retry its own deployment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(zai): fall back to BIGMODEL/ZHIPUAI api keys and add GLM-5.3 aliases and glm-5v-turbo
Resolve the zai api key from ZAI_API_KEY, BIGMODEL_API_KEY or ZHIPUAI_API_KEY
so deployments predating the zai rename keep working. Accept response_format
and max_completion_tokens as supported openai params. Register zai/GLM-5.3
and zai/GLM-5.3-Flash aliases plus the glm-5v-turbo vision model in both the
main and backup cost maps
* fix(zai): read only ZAI_API_KEY, dropping the BigModel key fallbacks
* fix(zai): map max_completion_tokens to max_tokens, correct glm-5v-turbo pricing, drop redundant aliases
* fix(zai): reject json_schema response_format, add glm-5v-turbo cache read price
* fix(zai): forward every response_format type so allowed_openai_params keeps working
* fix(zai): follow json_schema natively on glm-5.3, prompt other models, drop over-limit max_tokens under drop_params
* fix(zai): keep BigModel's json_schema as sent and never drop the caller's max_tokens under drop_params
* fix(zai): write the prompted JSON schema as JSON so models answer with valid JSON
* test(zai): type the zai provider tests and mark their locals Final
---------
Co-authored-by: DewellNaa <210961864+DewellNaa@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): bill a provider reported usage.cost on every path unless the deployment is priced
* test(integration): a reported cost is attributed to output in the gateway parity cases
* fix(cost): skip a chat usage.cost that already holds litellm's stamped bill and Responses API objects
* fix(cost): record litellm's bill on the Responses terminal event so a later pricing pass does not read it as the upstream's charge
* fix(cost): mark usage.cost litellm wrote as its own bill and keep a breakdown for a reported cost
* fix(cost): keep the token breakdown when a header reported cost repeats an already priced bill
* fix(cost): read the stored breakdown total through the logging object's typed field
* refactor(cost): name the providers billed from their own usage fields for what they are
* fix(cost): keep the upstream's usage.cost beside litellm's bill and stamp a zero operator price on the stream frame
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Neither chart bundles Lens anymore. The Lens subchart, its lock entry,
the vendored chart archive, the templates/lens resources and the
bundled-Lens install smoke are gone. A connection-only lens block wires
the gateway and backend to a Lens deployed from its own chart, and an
enabled lensWorker value fails the render instead of being ignored.
Supersedes #47869, which made the same change for helm/litellm only.
* fix(cost): bill the requested model's rates when the response names an unpriced model
Fireworks answers chat and /v1/messages calls with a display name such as
"Qwen 3.8 Max" in the response's model field. The cost path priced that
name, the Fireworks calculator fell back to its free default bucket without
raising, and the call billed $0 even though the requested model has a row.
When the response model resolves to no cost-map row or rule that declares
rates and the requested model does, price from the requested model. A priced
response model still wins, so dated snapshots keep their own rates, and an
unpriced requested model such as a router alias leaves the result unchanged.
* perf(cost): check the exact cost-map row before generalization rules when picking the priced model
* fix(cost): apply the priced fallback to the provider response model streaming records
* perf(cost): skip the served model's rule lookup when the requested model has no rates
* fix(cost): price the requested model without a model arg and keep Fireworks size buckets behind the proxy
When completion_cost runs with no model, the requested model now comes from the
call's hidden litellm_model_name, so a display-name response still bills at the
requested model's rates. A Fireworks model whose name carries a parameter size
counts as priced through its size bucket.
The Fireworks per-token lookup now treats a cost-map row that prices nothing, like
the empty row the Router registers for an unlisted deployment, as missing, so an
unlisted Fireworks model bills at its size bucket instead of $0 even when Fireworks
echoes its slug. The "row prices anything" helpers move to llm_cost_calc/utils so
the Fireworks lookup can reuse them without a circular import.
* fix(fireworks): keep prices from a matching fallback rule for an unlisted model
An unlisted Fireworks model that only a fallback generalization rule prices
has no raw cost map row, so the empty-row check sent it to the size bucket
and dropped the rule's rates. The Fireworks lookup now keeps the model info
when the row or a matching rule declares a rate.
* fix(cost): price a bare response model under the requested provider and drop the unlisted Fireworks bucket change
The logging payload picks the pricing model without a provider, so a bare echo
like mistral-large-latest failed its price check and the requested alias took
over model_map_key. The served name is now priced under the requested model's
provider, and an echo that is the requested name minus its provider prefix
skips the checks.
Treating the Router's empty price row as missing let unlisted Fireworks
deployments bill from a name guess, which misprices dedicated deployment ids
and routes some short names to another provider's rates. That change and its
tests go back to the merge base, so unlisted Fireworks models keep billing $0
behind the proxy as on main.
* perf(cost): read the response model's price row without building full model info
An unpriced display name like Qwen 3.8 Max made the price check build full
model info, and Fireworks' supported params lookup scans the whole cost map
for an unknown name. The row check now resolves the cost map key through the
cached model info helper, so that pick takes about 0.45 ms instead of several.
A regional test covers a Bedrock call whose response names an unpriced display
name with no model passed, which bills the requested model's regional row.
The duplicate get_model_info wrapper goes, since its twin already does the
same thing.
* fix(cost): keep the served model's filled-in and size-bucket rates, and cache misses
A served row priced only by a fill_missing_for_providers rule bills those
rates on main, but the price check read only the raw row and billed the
requested alias instead. It now merges the rule's fill-in keys through the
helper model info lookup uses.
A served Fireworks or Together AI id that names its parameter count keeps its
size-bucket price, as on main, instead of switching to the requested row.
The served model is checked first, cost map misses are cached and cleared with
the model info cache, and the rules are read only for a name that resolved
through one. A priced snapshot behind an Azure deployment name takes about
19 us of CPU time against 7 us on the merge base, down from about 300 us.
* fix(cost): bill the requested model's listed rates over a served size-bucket guess
A served Fireworks or Together AI name that is not in the cost map but names
its parameter count, like a display name with a size in it, now bills at the
requested model's listed rates instead of the provider's size-bucket price.
The bucket price still applies when the requested model has no rates, as on
main
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): batch Claude models in Anthropic's native row shape
Claude deployments on Vertex were filed under publishers/google/models/<claude>
when the batch input was uploaded, so create_batch asked Vertex for a Google
model that doesn't exist and got a bare 404 on batchPredictionJobs. The rows
were also written as Gemini GenerateContentRequests, which Vertex rejects for
Claude even with the right model path
Claude uploads now go under publishers/anthropic/models/<model> and each line
becomes {"custom_id", "request": <Anthropic Messages body with
anthropic_version vertex-2023-10-16>} using the existing Vertex Claude
transform. Output rows in the Anthropic message shape are converted back to
OpenAI batch rows (custom_id from the row, errors as failed rows), native rows
are costed from Anthropic usage, and create_batch fails fast with a 400 when a
Claude job targets the global location, which Vertex doesn't serve for
partner model batches
Fixes#45671
* test(vertex_ai): cover malformed Claude output rows and rows without custom_id
* fix(vertex_ai): build Claude batch rows with chat's param mapping and surface Vertex batch create errors
Claude batch rows now go through get_optional_params like Vertex Claude chat, so unsupported params, top_k, response_format and extra_body follow drop_params. Tool names Claude would rename get a 400 at upload. A refused batch create returns Vertex's own status and message.
* fix(vertex_ai): price Claude batch output rows without a request echo, keep @ version ids, refuse a json_tool_call user tool, keep Vertex error headers
* fix(vertex_ai): keep httpx.HTTPStatusError on Vertex batch create errors and apply deployment chat params to Claude batch rows
* fix(vertex_ai): keep Vertex's status code on batch create errors through the proxy
* fix(vertex_ai): keep Vertex's error body on batch create errors and require custom_id on Claude lines
* fix(vertex_ai): keep the old batch create error text and Google object names
A failed Vertex batch create keeps httpx's error line first and adds Vertex's body after it, and logs the status and body again. Only Claude object names keep the @ version, so Google models are stored under the same names as before
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(providers): add CLF AI Gateway (clf_ai_gateway), OpenAI-compatible, 9 models
OpenAI-compatible gateway at https://api.clfaigateway.dev/v1 serving open-weight models (GLM, Kimi, DeepSeek, Qwen) on Cloudflare Workers AI upstream. Follows the DeepInfra pattern: ClfAiGatewayConfig(OpenAIGPTConfig), CLF_AI_GATEWAY_API_KEY / _API_BASE, provider registered in enum, provider lists, endpoint inference, lazy-import registry, ProviderConfigManager, validate_environment. Supported params deliberately omit functions/function_call/logit_bias (gateway returns 400 for them). 9 cost-map entries (root + backup) with measured context/output limits and currently billed prices; endpoint-support entry; README row. Mocked tests only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(clf_ai_gateway): declare each model's real reasoning_effort levels
The per-level supports_*_reasoning_effort flags cannot express the sets
this gateway actually takes, so all nine models advertised a wrong one:
every model offered "minimal" and eight offered "none", neither of which
the gateway accepts, five dropped "xhigh", which it does accept, and
qwen3.8-27b offered "high", which it rejects. That set surfaces through
/model_group/info and the model picker, so a user picking a level off it
gets a 400 from the gateway.
Declare reasoning_effort_levels instead, taken from the gateway's own
GET /v1/public/models, which resolve_supported_reasoning_efforts reads
first and uses whole. Declaring it also stops clf_ai_gateway/deepseek-v4-pro
and clf_ai_gateway/deepseek-v4-flash inheriting effort metadata from the
unprefixed deepseek entries of the same name.
* test(clf_ai_gateway): name the gateway-rejected params instead of commenting them
* fix(clf_ai_gateway): list the provider in the proxy Add Model form
main now requires every LlmProviders value to have an entry in
provider_create_fields.json (or be frozen as unlisted), so the Add Model
dropdown cannot silently miss a backend provider. Add the clf_ai_gateway
entry in the same shape as the SCX.ai registration: required API key,
optional API base with https://api.clfaigateway.dev/v1 as placeholder, and
a cost-map model as the default model placeholder. A provider test pins
the entry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(clf_ai_gateway): sync prices and reasoning_effort levels with the live catalog
CLF AI Gateway raised its launch discount from 40% to 60% off list on
2026-09-26. It also re-measured every model's reasoning_effort levels
against the current Workers AI behaviour.
- input/output/cached per-token prices for all 9 models now match
GET https://api.clfaigateway.dev/v1/public/models to the nano; qwen3.8-27b
cached input also follows the upstream cached rate published on 2026-09-25
- reasoning_effort_levels: add "max" to deepseek-v4-flash, deepseek-v4-pro,
glm-5.2 and glm-5.3-flash; add "none" to kimi-k2.6 (it really disables
thinking there); drop "none" from glm-5.3 (upstream runs it as "max")
Context windows, output limits and vision flags are unchanged and were
checked against the same endpoint.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(clf_ai_gateway): drop a rebind-ok that no longer suppresses anything
After rebasing onto main, the type-discipline gate's new LIT013 rule flags
the `# rebind-ok` on the endpoint-inference `dynamic_api_key` assignment.
The rebind rule no longer fires on that line, so the suppression is inert.
The sibling endpoint branches carry no comment there either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(clf_ai_gateway): expect the re-measured reasoning_effort levels
Follows the catalog sync: max on deepseek-v4-flash, deepseek-v4-pro,
glm-5.2 and glm-5.3-flash, none on kimi-k2.6, and no none on glm-5.3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(clf_ai_gateway): move the provider tests to tests/unit/llms
main moved every per-provider test from tests/test_litellm/llms/<provider>/
to tests/unit/llms/<provider>/ (DeepInfra lives at
tests/unit/llms/deepinfra/). This branch still added the old path, so no CI
shard claimed it. That failed assert-ci-coverage and assert-shard-coverage,
and the file never ran (hence the low codecov/patch).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(clf_ai_gateway): assert routing, billing, and dropped params instead of pinned gateway facts
* fix(clf_ai_gateway): keep the type-discipline gate at its merge-base count
* chore(clf_ai_gateway): regenerate provider endpoint support
* ci(clf_ai_gateway): run the provider tests in the llms unit shard
* fix(clf_ai_gateway): drop the retired max_tokens field from the cost map rows
* fix(clf_ai_gateway): forward params the gateway accepts, drop undeclared reasoning levels, add deepseek-v4.1-flash
* fix(clf_ai_gateway): drop the unreachable non-chat supported-params branch
* fix(clf_ai_gateway): send a dict reasoning_effort as its level
* fix(clf_ai_gateway): refuse a reasoning_effort dict without a level unless drop_params drops it
---------
Co-authored-by: bap1106 <bap1106@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* refactor(types): replace Any with proven types in 35 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): keep anthropic chat transformation json path untyped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): accept null gemini vector store fields and load encryption TypedDicts on python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): validate Python client user responses and type delete_user as the count it returns
* fix(types): keep loose payloads working under the sweep's validators
An off-schema Anthropic stream event is now skipped instead of ending the
guardrail's text and stop_reason scan, and MAI image usage accepts the loose
numbers it did before. New dict and list adapters hide their input from
validation errors, RerankResponse indexing keeps its field types, and the
delete_user test mocks the int /user/delete returns.
* fix(types): make the changed DynamoAI fields ReadOnly and skip non-object A2A stream items
* refactor(types): tighten round 2 parse sites and keep payloads out of errors
Parse Anthropic SSE events with validate_json and ignore extra fields, validate admin team and key rows per row with hidden inputs, type rerank get and __getitem__ per key, and move the Phoenix prompt version response to a frozen BaseModel
* fix(types): keep MAI image calls alive on unreadable usage counts
An unreadable MAI usage count now counts as absent instead of failing a
billed image call, and the provider's usage fields pass through as sent.
The Anthropic stream scan warnings no longer log raw model output, and the
off-schema skip has tests that fail on the old scan.
* fix(types): count non-object MAI usage as absent and parse Anthropic SSE with json.loads
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(batches): read OpenAI batch usage into the batch token counts
* fix(batches): serve OpenAI's batch usage shape from retrieve and the cost poller
* fix(batches): keep the chat usage keys next to OpenAI's batch usage fields
Batch usage now carries every chat key it had plus OpenAI's five batch fields with the same counts, so poller-billed batches keep prompt_tokens and cache-write tokens, and older proxies still read the new rows. Usage with missing or null detail counts reads its real input and output counts.
* test(batches): check the batch usage counts in the e2e terminal-state test
* fix(types): apply nested include and exclude to both batch usage shapes
* fix(batches): log no cost for a batch retrieve that is not final yet
A retrieve now reads OpenAI's batch usage into the chat counts, so a retrieve
that saw the batch completed before its output file arrived priced those counts
for the logging callbacks, and the final retrieve priced the whole batch again.
A batch that is not final now logs no cost, as it did before.
Batch usage dumps also pass exclude_unset and exclude_defaults through to the
usage keys.
* fix(batches): pass through every usage detail field the provider sends
Batch dumps rebuilt input_tokens_details and output_tokens_details from the chat keys, which dropped any detail field past cached_tokens and reasoning_tokens that the merge base passed through. The dump now starts from the provider's stored details and only sets the two counts derived from the chat keys
* fix(batches): skip sync loggers on the cost poller's own retrieve and honor exclude_none on provider details
* fix(batches): keep sync logger events at $0 on poller-owned retrieves, pass unparsed provider counts through, log batch usage in chat shape
* fix(batches): store only chat usage extras on batch spend rows
Spend rows for batch retrieves copied OpenAI's batch fields (input_tokens,
output_tokens and their details) into metadata.additional_usage_values next
to the chat counts they duplicate. Build the row from the batch's chat usage,
shared with the logging payload through batch_chat_usage.
* fix(streaming): rebuild each choice of an n>1 chat stream on its own
stream_chunk_builder folded every choice of an n>1 chat stream into choice 0, so post-call guardrails such as tool_permission re-emitted one choice holding every choice's tool calls, and spend logs and logging callbacks kept only the first choice's text. Each choice is now assembled from its own chunks, while usage, cost, and hidden params still read the whole stream. Cisco AI Defense compares each choice's streamed text on its own so safe n>1 streams still pass through unchanged.
* fix(streaming): keep bridged reasoning summaries on choice 0 and infer an unreported choice finish reason
* fix(streaming): match MCP follow-ups and bridged summary text to the rebuilt choices
MCP auto-execute now answers only the first choice's tool calls, since its
follow-up turn continues that choice alone. The Responses bridge streams a
space before each reasoning summary part after the first, matching the
non-streamed reply. build_base_response now takes the per-choice streams
instead of having its choices replaced afterwards
* fix(streaming): space every later reasoning summary part, deny per judged call, and rebuild in one text pass
* fix(guardrails): key tool_permission denials for custom tool calls too, and type the stream rebuild's messages
* fix(mcp): continue the first choice that called a tool when an n>1 reply auto-executes MCP tools
* fix(noma): scan each choice of a multi-choice reply on its own
* fix(noma): keep masked choices when a later choice's scan fails open on a stream
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ci): measure PR gates from the target branch's current tip
The lint gate base, the ui lint and ui unit test scopes, and the spend logs
guard compared the event's pull_request.base.sha with the head to find the
merge base. GitHub can leave base.sha weeks behind the base branch, and when
that old commit is an ancestor of a head that merged newer main, compare
returns the old commit itself, so the strict gates blamed main's own code on
the PR.
Compare base.ref with the head instead, passed through env, so GitHub
resolves the branch tip when the job runs. This is the same base make check
uses locally. The ui unit test push guard now keys on BASE_REF, which is
empty on push and workflow_dispatch.
The regression test runs every workflow step that reads merge_base_commit
against a fake gh backed by a temp repo with a stale base.sha, for main and a
litellm_/ base with a slash.
* fix(ci): check out the base branch's current guard code in the PR guards
* fix(ci): read the PR's current base branch so re-runs survive a deleted base
Every merge-base step now asks the pulls API for the PR's base branch instead of
reading the event's base.ref, which a re-run keeps after a stacked base is merged
and deleted. The guard checkouts go back to base.sha, which stays reachable.
* test(ci): mark the step script path Final
* fix(ci): keep the spend logs guard on main and run both guard scripts from the base branch tip
* fix(ci): measure from the event's base branch unless it was deleted, and test what each step hands on
A re-run after a retarget off main kept following the PR's current base, so lint, ui-unit and the cost map guard could pass without measuring against main. They now read the current base only when the event's base branch no longer exists. The tests run each merge base step whole and check the value its consumer gets
* fix(ci): follow a renamed base branch to its new name before the compare
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(token_counter): price PDF pages for OpenAI and Azure models the way they bill
* fix(token_counter): price PDF pages for OpenAI and Azure models the way they bill
* fix(token_counter): count azure, data-zone and dated GPT names with their family's encoding
* fix(token_counter): price only drawn PDF images and keep other gateways' GPT pricing
Detect raster images from the page content stream, including inline
images on pages with empty resources, and walk each form once. Strip
only openai/, azure/ and Azure data-zone prefixes, bound the model
family scan, read Azure-only chat models from the cost map, and keep
cl100k_base for listed non-chat OpenAI models.
* fix(token_counter): price GPT names behind any prefix like the bare name and walk shared forms per resources
Prices a PDF by the model name's last path segment, the same name the proxy counts with,
keeps listed non-chat models on Claude pricing, walks each form once per inherited resources
dictionary so a shared child form drawing an image is not missed, and resolves PDF pricing
only when a message carries a PDF
* fix(token_counter): keep the rendered-page count for image pages, parse each form once, and match chat models only
A PDF page that draws a raster image keeps the rendered-page estimate, since newer GPT models bill such a page near
it. The image walk parses each form once and stops at a fixed number of drawn XObjects, so a form shared under many
resources dictionaries no longer slows the count. The family prefix match skips non-chat entries, and ft: fine-tunes
count like their base model
* fix(token_counter): count Azure-only chat models with o200k like other OpenAI chat models
* fix(token_counter): count reservation and project ITPM with the serving deployment's model
Budget reservation and project ITPM counted with the requested group name, so a
GPT-named group served by Claude reserved a PDF at OpenAI's price. They now count
with each deployment's model minus its provider prefix, the name count_tokens uses,
and take the largest. A custom OpenAI tokenizer with no model counts with cl100k
again instead of raising, and listed Azure gpt- chat entries price as OpenAI.
* perf(token_counter): find drawn PDF images with a byte scan instead of parsing every operator
pypdf parsed every operator of a page's content stream to find the images it draws, so a
compressed page of a million operators took about 30 seconds to price for an OpenAI model.
A linear scan of the decoded stream for name Do and inline images finds the same draws.
* fix(token-counter): count each team deployment by its PDF pricing and honor base_model
Budget reservation and the TPM limiter now look up the deployments that serve the
caller's team, by base_model when set. A deployment that reads PDF pages and prices
them unlike the group name counts with its own model, any other with the group name.
A comment between an XObject name and Do now reads as whitespace, and the timing
tests moved to CodSpeed benchmarks.
* fix(benchmarks): install pypdf in the benchmark environment
The PDF token-count benchmarks parse pages with pypdf, which only the proxy-runtime extra installed, so CodSpeed took the no-parser path.
* fix(proxy): count PDF pages for deployments served under multi-segment model names
* fix(proxy): count a team's wildcard route the way routing serves it and reserve OpenAI chat's per-PDF charge
* fix(proxy): count routing-group and alias members with the caller's team, as routing serves them
* fix(proxy): stop answering with the literal string "None" on the proxy-server, auth, and health routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* fix(proxy): stop answering with the literal string "None" on the spend and management routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* test(proxy): pin the mid-stream failure frame and the DB outage body to a JSON null param
* fix(proxy): type upstream timeouts as timeout_error and update integration asserts for the null param
A 408 with no carried type now answers timeout_error instead of invalid_request_error, so a client branching on type keeps retrying it. Three integration suites still compared auth rejections against param="None".
* fix(proxy): keep a 408 on main's invalid_request_error type and pin the transcription file cap's 400
* fix(proxy): type a route failure by the status it answers with after the tag-routing and no-deployment remap
* fix(proxy): keep auth_error on proxy-raised 401/403 route failures and ban the string None as an error type or param
* fix(proxy): type a hook's 401 or 403 on completions, moderations and transcription by its status, as chat does
Only a status the proxy remapped into 401 or 403, such as a tag-routing denial, keeps auth_error.
* test(proxy): build the transcription file-cap test on a real router so it holds after the router refactor
* feat(lite): offer each model group's reasoning efforts in pi, OpenCode and Codex
lite pi, lite opencode and lite codex now read supported_reasoning_efforts from /model_group/info and write them into each client's model config. pi gets reasoning and a thinkingLevelMap, OpenCode gets reasoning_options plus one variant per effort, and Codex fallback entries get supported_reasoning_levels with a default level. The proxy-served Codex catalog on /v1/models?client_version= offers the same levels
Codex before 0.138.0 fails the whole catalog on an effort it cannot name, so those releases, and an unreadable version, get only the six efforts every release parses. A group with unknown efforts keeps today's output, and a group with none is written as non-reasoning
The strength order and nearest_declared_reasoning_effort move to an SDK-free module so the lite CLI, which loads without initializing the SDK, can import the Codex catalog
* fix(lite): describe team models by their public name and keep Codex's none default
/model_group/info now looks up a team's own deployments when the caller's
team serves the requested name, so a team model reports its reasoning levels.
Codex fallback entries default to none when the group lists it, and the proxy
builds the Codex catalog in a worker thread
* fix(lite): round Codex's default level down and describe a team's shared-name model from its own deployments
Move the reasoning strength helper to litellm_core_utils so lite and the Codex catalog import it without the SDK or the router, and let make lint-format-check-changed diff the working tree so a staged rename passes.
* fix(lite): keep pi's system role on reasoning model groups
pi switches its system prompt to the developer role for any model marked reasoning, so an operator's
cache_control_injection_points entry for role system stopped matching on Claude groups. lite pi now sets
compat.supportsDeveloperRole false on those entries, so pi sends system as it did before.
* fix(codex): treat a client_version too long to parse as an old Codex and keep undescribed stock levels
A client_version past Python's integer digit limit made the Codex catalog answer 500. It now gets the
levels every Codex parses. A stock Codex level with no description validates again, as on base, and
lite codex treats undecodable codex --version output as an unknown version
* fix(lite): keep REASONING_EFFORT_STRENGTH_ORDER importable from router_utils and skip the Codex sync on undecodable debug models output
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore(caching)!: remove the disk cache backend
Drop the disk cache type and its diskcache dependency, along with the
caching install extra that existed only to pull it in. diskcache has not
had a release since August 2023
Cache now raises a ValueError listing the supported types when it gets a
type it does not know. Before, an unknown type built a Cache with no
backend, which only failed on first use
BREAKING CHANGE: cache_params type "disk" and the litellm[caching] extra
are gone. A proxy configured with type disk now refuses to start. Use
local or redis instead
* test(proxy): expect the unsupported cache type error from _init_cache
The test pinned the old fallthrough, where an unknown type left the cache
without a backend and the proxy hit an AttributeError reading it
* chore(deps): drop the diskcache scan ignore main extended
diskcache is no longer in the lock, so the ignore matches nothing
* fix(caching): ignore a leftover disk_cache_dir instead of crashing redis and s3
Dropping the disk_cache_dir parameter sent the key into **kwargs, which
the redis and s3 branches hand to their clients, so a config that moved
from the disk cache to redis or s3 and kept the key failed with a
TypeError. Accept the key again, ignore it, and log a warning
The unsupported type test now checks the whole supported list, and two
proxy docstrings no longer name disk as a cache backend
* fix(caching): warn about a leftover disk_cache_dir only once the cache is built
The warning ran before the type check, so a config with type disk logged
that the key was ignored and then failed on the type anyway. It now logs
only after a supported backend is built. The test also checks that a
cache without the key logs nothing
* feat(pass-through): add eu/jev decision API pass-through with spend tracking
Adds a /eu_jev/{endpoint:path} pass-through route for eu/jev, an EU-hosted
decision API that speaks the same POST /v1/systemone contract as TypeSafe.
It reads EU_JEV_API_KEY and EU_JEV_API_BASE (defaults to
https://jev.bevel.software), books spend through the shared decision logging
handler, and prices eu_jev/eu-jeff-1.0.0 and eu_jev/eu-jeff-latest from the
cost map
* fix(pass-through): default eu_jev to its canonical eu-jev.com origin
* fix(pass-through): reject custom_body on the eu_jev route
A key limited to other models could send an eu/jev request through
custom_body, because auth checks the outer body while the shared
pass-through forwards custom_body upstream. The route now answers 400
before anything is forwarded, the same way validate_oss_request guards
the laya and bespoke routes
* fix(pass-through): give the eu_jev route request and 200 response schemas
The OpenAPI docs gate flagged POST /eu_jev/{endpoint} for OA001 and
OA003. The route now documents the eu/jev System One request body and
its 200 answer, usage and meta shape, taken from the eu/jev OpenAPI
document. The lazy OpenAPI snapshot and the dashboard API types are
regenerated with python -m litellm.proxy._lazy_openapi_snapshot and
npm run gen:api
* fix(pass-through): reject streaming on the eu_jev route
A body with stream=true switched the route to streaming, which skips the
decision spend tracking. Reject it with a 400, as validate_oss_request does
for laya and bespoke.
* fix(pass-through): type the eu_jev request body like the other decision routes
* fix(pass-through): check eu_jev decisions against the model eu/jev runs
Serve only POST /eu_jev/v1/systemone, eu/jev's one endpoint, resolve an omitted
model and the legacy names to eu-jeff-latest for key model checks and pricing,
check scopes under eu_jev/<model> like Laya and Bespoke, and answer 503 when
EU_JEV_API_KEY is unset
* fix(pass-through): apply eu_jev key limits, register eu_jev, price unlisted releases
Pre-call hooks and guardrails now see the body model as eu_jev/<release>, so
model_rpm_limit and model_tpm_limit set on eu_jev/eu-jeff-latest apply, and
upstream still gets the client's exact body. eu_jev is a registered decisions
provider, so access groups, config deployments, /v1/systemone and /v1/decisions
accept it, and the Admin UI can add it. A release without its own cost map row
is priced as eu-jeff-latest instead of $0
* fix(eu_jev): use the pass-through deployment's api_base and accept provider-prefixed models
* fix(eu_jev): declare /v1/systemone on the eu/jev cost map entries so the generated endpoint table keeps it
* fix(eu_jev): take pass-through credentials only from proxy-wide deployments
The eu/jev route now ignores team-scoped deployments when it picks the
upstream key and base, so one team's deployment can no longer redirect
another team's calls, and a base ending in /v1 is not doubled
* fix(eu_jev): let a configured pass-through at /eu_jev/v1/systemone keep serving it
Scope the mapped pass-through and gateway allowlist entries to the decision
route instead of the whole /eu_jev prefix, so custom routes under /eu_jev keep
their auth, metering, and gateway behavior from before this PR
* fix(eu_jev): treat /eu_jev/v1/systemone as a provider route only while the built-in route serves it
Drop the decision path from the mapped pass-through list and classify it with one
check that asks the pass-through registry, so an operator's own pass-through at that
path keeps its auth, metering, LLM-route status and litellm_user_api_key handling
from before this PR. The built-in route now yields on a per-request root path and
on any configured method, and the gateway keeps only the exact decision path
* fix(eu_jev): find a configured route at the decision path with or without a trailing slash, and stop a deleted one forwarding
* test(eu_jev): annotate new test variables with Final
* fix(proxy): keep eu/jev pass-through routing stable during updates and method mismatches
Swap an updated pass-through's routes with no await between removal and
re-registration, so the built-in eu/jev route never sees the path as
unconfigured mid-update. The built-in eu/jev handler no longer takes a
configured route's target, and the built-in route also yields on a
method-only match, so GET and PUT on the no-slash form of a slash route
redirect again.
* test(pass-through): annotate the route-swap test's locals as Final
* fix(pass-through): keep an updated auth route's LLM API route checks on the updating worker
* fix(eu_jev): leave a null model out of the upstream decision request
A JSON model of null is limited and counted as eu-jeff-latest, so send the
request upstream without a model field, the way an omitted model goes, since
eu/jev answers a null model with 400 Unknown model
* fix(pass-through): keep route checks on a path another auth endpoint still serves after an update
Forward a non-string eu/jev model unchanged so eu/jev rejects it, and drop only a null or missing one
* test(eu_jev): keep the non-string model test in the eu/jev route test file so a CI shard runs it
* refactor(eu_jev): build the scoped pass-through body as a new dict instead of updating it in place
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(dashscope): add image edits and text-to-speech
* fix(dashscope): bill reported speech characters and input images, follow the chat region for edits, keep the call timeout on the audio download
* fix(dashscope): bill deployment image prices as main does and route edits to the host speech uses
A deployment that sets its own input_cost_per_image, output_cost_per_image or input_cost_per_pixel bills through the default image calculator again, so an override is never read as a per-input-image price and pixel pricing on qwen-image-3.0-pro no longer raises.
Edits now resolve the caller's api_base or DASHSCOPE_API_BASE through the same API-root helper speech uses, on any host, with DASHSCOPE_API_BASE_IMAGE still winning. An image response whose usage is null or not an object keeps its images instead of raising.
* fix(images): bill a deployment's own prices on image edits through the proxy
The proxy builds the logging object before the router picks a deployment, so image_edit never handed the deployment's price fields to its cost calculation and billed the cost-map price instead. Pass the custom pricing params with the pre-call update, as the Responses API does.
* fix(images): bill per-pixel image prices at a DashScope W*H size
DashScope's mapped size reads 1328*1328, which the cost size reader rejected, so a deployment's input_cost_per_pixel billed every DashScope generation or edit at the 1024x1024 default.
* fix(dashscope): bill image prices by their meaning, report the rendered size, prefer the deployment edit base, fetch audio over https
* fix(dashscope): read the rendered image size from output_width and output_height
* test(images): bill a fal edit at its deployment's own image price
* fix(dashscope): keep the qwen-image-3.0 input-image fee out of the shared cost map row
Released proxies read the cost map from main and bill input_cost_per_image as the
flat price of each DashScope image, so the row's $0.003 input fee billed every
qwen-image-3.0 image there at a tenth of its $0.03 price. A deployment can still
set input_cost_per_image to bill DashScope's input images
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(context_management): run clear_tool_uses in the gateway on the Responses API path
/v1/messages to openai/ models goes through the Responses API, which translated
only compact_20260112 (to its native compaction) and dropped every other edit
without a warning, so clear_tool_uses_20250919 never cleared anything. Edits
without a Responses API form now run in the same polyfill as the chat completions
path, before the call, and their applied_edits come back on the response and the
final message_delta. compact stays native.
* fix(context_management): check the sync polyfill result instead of casting it
The strict ruff gate (TID251) bans typing.cast.
* test(context_management): cover the Responses path's polyfill error handling
A polyfill crash is a 500 before the call, a rejected edit under drop_params is
skipped, and the sync stream reports its edits. normalize_spec never raises for
a dict spec, so its guard is gone.
* fix(context_management): count unreadable blocks and check tool_result text before clearing
clear_tool_uses now maps blocks the token counter cannot read (server tool, compaction) to JSON text, so an input_tokens trigger no longer 500s on that history. The Responses API path checks malformed tool_result text on the original messages before clearing, as the chat completions path does.
Also removes comments that repeated the code and makes the test fixtures immutable.
* fix(context_management): count only blocks the model receives and keep list specs unsent
clear_tool_uses now leaves blocks token_counter cannot read, like server tool
results and compaction, out of its count instead of counting them as JSON text.
No adapter sends them to the model, so an input_tokens trigger no longer fires
on tokens the model never sees.
On the Responses API path, an OpenAI-style list context_management stays unsent,
as it was before, instead of reaching OpenAI's native compaction, where a
threshold under 1000 is a 400.
On the chat completions path, a clear edit listed before compact_20260112 now
runs on the turns after the latest compaction block, as Anthropic does, instead
of counting and clearing turns the compaction block drops.
The crash test that replaced apply_context_management with a mock is removed.
* fix(context_management): apply clear edits in OpenAI count_tokens and skip nested off-spec blocks when counting
* fix(context_management): run edits on the history drop_params sends, so count_tokens and /v1/messages clear alike
* fix(context_management): count clear_at_least freed tokens over sent blocks only
* fix(context_management): count tools with JSON Schema shorthands and reject a non-object trigger or keep
A tool whose schema has an array without items, a boolean subschema, or a search_result block without a title made the token counter raise, so a clear edit answered 500. A string trigger or keep also answered 500, even under drop_params. Both now count or get the API's 400, and drop_params drops the malformed field.
* fix(token_counter): count a malformed tool schema keyword as absent and move shorthand framed-estimate cases
* fix(count_tokens): pass the system prompt into clearing so count_tokens matches inference
* fix(context_management): keep a malformed system prompt from failing clear_tool_uses on the Responses API
A number, an object, or a list holding non-object items as system made the
polyfill answer 500. The polyfill now keeps a list's object items and treats
any other shape as no system, on both /v1/messages and count_tokens, so the
two paths still count the same text
---------
Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): keep the proxy's error body on litellm.responses errors from litellm_proxy models
litellm.responses() with a litellm_proxy/ model raised BadRequestError with
body None when the proxy answered a 4xx, while litellm.completion() on the
same proxy carries the decoded error object. The litellm_proxy Responses
config now decodes the error JSON into the exception body the way the
OpenAI SDK does, so body, code, type and the message match chat completions
on sync, async and streaming calls.
* fix(proxy): relay a downstream proxy's provider_specific_fields in error responses
A LiteLLM proxy routing to another LiteLLM proxy now copies the downstream's
provider_specific_fields (which guardrail fired) from the exception body into
its own error answer, on chat and Responses alike. Adds tests for the two
_openai_sdk_error_body branches the existing tests left open.
* fix(proxy): redact internal details from relayed provider_specific_fields
* fix(proxy): redact relayed provider_specific_fields per key so credential-named values stay hidden
* fix(proxy): redact numeric credentials in relayed provider_specific_fields
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params
The native /v1/messages path sent tools with `strict` straight through, so
when Claude's strict-tool limits rejected them with a 400 the caller got that
400 even with drop_params set. Under drop_params, a 400 that matches Claude's
strict-limit phrases on a request with a strict tool is now sent once more
with `strict` removed from every tool. Without drop_params nothing changes.
The Claude strict-limit matcher moves from bedrock/common_utils.py to
anthropic/common_utils.py so Bedrock converse and the base messages config
share it, and the Rust bridge routes strict tools under drop_params to Python.
* fix(anthropic): retry without strict only when a tool sends strict true
A tool with only strict false no longer triggers a resend of an identical body, the warning no longer
blames tool schemas when the rejected schema may be the output format, and requests with only
additional_drop_params stay on the Rust messages route since the Python retry ignores them
* fix(anthropic): keep sync messages calls with strict tools on Rust
Python has no sync /v1/messages handler, so routing a sync call with a
strict tool to Python raised ValueError where Rust served it before.
The strict-tools detector now applies to async calls only. The retry
warning also names the strict tool count without a plural.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): bill streamed /v1/completions without stream_options like chat
* test(proxy): mark the text completion stream usage test variable Final
* fix(completions): drop Azure's empty prompt-filter frame when the proxy asked for stream usage
* test(spend): drop the module docstring from the text completion stream usage test
* fix(proxy): keep the internal stream usage strip flag out of the logged request body
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openapi): build native extension for base counts
* test(openapi): exercise native import in base worktree
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GenericGuardrailAPI.get_supported_event_hooks only listed pre_call,
post_call and during_call, so event hook validation rejected the MCP
modes and the guardrail was skipped at proxy startup instead of being
attached to MCP tool calls. Add pre_mcp_call, during_mcp_call and
post_mcp_call. during_mcp_call runs alongside the tool call, so it can
fail the call but cannot rewrite its arguments.
#48718 added TestLangfuseDestinationFields while #48723 made the unconfigured
stdout placeholder opt-in (console_default=True, the otel callback's setting).
Both merged within a minute, so main's integrations job fails six tests that
read the placeholder's exporter. Build those providers with console_default=True
so the operator-exporter checks keep their target.
* fix(caching): key cached async clients by a per-loop token instead of id(loop)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): drop deferred closes queued on a collected event loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): pin that a close queued on a live loop waits for that loop
* test(caching): run loop-reuse tests until a loop address repeats
The collected-loop tests ran a fixed 50 loops and passed with the fix reverted whenever no loop reused an earlier address. They now stop at the first reuse and skip if none happens, through a shared helper.
* test(caching): cover the cache key fallback for loops without weak references
Also corrects the evicted client closer docstring: a dropped bucket's clients are left to the garbage collector.
* fix(http): free short-lived event loops whose handler finalizer close finished as the loop stopped
* fix(caching): share sync clients across event loops and catch a failed finalizer close
* fix(agentops): stop closing the shared sync http client after the auth call
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Without drop_params, every copy of a tool from additional_tools input items
is forwarded as sent, so a provider that rejects duplicate tools returns its
own 400. Codex keeps the merge because the proxy turns drop_params on for it.
* test: add a manually dispatched integration trial on larger runners
Runs the providers, database and management integration groups on 16-core GitHub-hosted runners with Postgres and Redis as service containers, to measure wall time and how far each group scales with more workers
run_integration.sh now keeps a preset INTEGRATION_WORKERS, takes the service container names from the environment, and seeds test order from the GitHub run when there is no CircleCI one
* test: run the integration trial on push to this branch
* test: give integration trial workers a local redis-server
* test: run the providers trial on a 64-core runner with 32 workers
The 16-core run reached 53% of providers in 45 minutes with 16 workers, which points at CPU. Postgres gets max_connections=1000 and a 1 GB shm because one server now backs 32 worker proxies, where each CircleCI machine backs 4
* test: run every integration group in the trial on sized runners
* test: add a scan that refuses to upload integration results holding the license
* test: drop the member-name pass the archive byte scan already covers
* test: use a secret whose base64 and base64url encodings differ
* ci: run integration groups on run-ci pull requests and scheduled main runs
* ci: scope the integration license to a protected environment
* fix(otel): stop OTel v2 vendor presets from printing every span to stdout
With LITELLM_OTEL_V2 on and a vendor callback such as langfuse_otel or arize
holding operator credentials, the proxy printed every span as JSON to stdout,
prompts included, on the request path. Nobody set OTEL_EXPORTER, but the
console placeholder that OpenTelemetryV2Config folds in for an empty exporter
list stayed next to the vendor's own exporter.
The provider now skips that placeholder when the config has any other
exporter. A console exporter the operator names with OTEL_EXPORTER=console is
kept, and so is the placeholder when it is the only exporter.
* test(otel): type the console-placeholder tests and drop inherited OTEL_* env from the stdout check
* fix(otel): print the console default only for the otel callback
The proxy's background V2 logger (published when no callback built one, which is
what key-only and team-only Langfuse destinations run under) also kept the
unconfigured console exporter, so it printed every span with prompts to stdout.
The console default now prints only for a listed otel callback or an exporter the
operator named.
* fix(otel): keep the otel callback's console output on service-name routes
A key or team otel_service_name moves the call onto a provider the tenant
tracer cache builds, which never got the console choice, so the plain otel
callback stopped printing those calls. The no-operator-keys integration rig
also drops inherited Langfuse keys.
* feat(guardrails): scan sampled stream windows in the background (LIT-9338)
streaming_scan_in_background runs each sampled post_call scan as a task so
the stream keeps flowing; a block ends the stream before the next chunk.
* fix(guardrails): hold the stream end for the background verdict and cancel the scan on upstream errors (LIT-9338)
* fix(guardrails): wait for the cancelled background scan on upstream errors (LIT-9338)
* test(guardrails): cover /v1/responses, a failing scan peer and a mixed burst for background scans (LIT-9338)
* test(guardrails): build the burst calls without a nested comprehension (LIT-9338)
* fix(otel)!: keep a credential-less langfuse_otel on OTel v2 so key destinations get the Langfuse mapper
With LITELLM_OTEL_V2 on, langfuse_otel in callbacks and no operator
LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY, the preset raised at startup and
the callback fell back to the legacy logger. Key and team destinations were
then served by a generic v2 logger without the Langfuse mapper, so their
traces lost tags, user and cost, and the legacy logger added an orphaned
raw_gen_ai_request span.
The preset now always degrades to the credential-gated exporter when the
operator has no keys, the same way arize_preset does, so the startup logger
stays on v2, exports nowhere for traffic without a destination, and carries
the Langfuse mapper to every key and team destination.
* test(otel): type the langfuse tenant-only test helpers
* fix(otel): give key/team Langfuse destinations the Langfuse fields when langfuse_otel is not a callback
With langfuse_otel only in a key's or team's logging, the startup logger is the
generic one and its spans reached the key's Langfuse project without user,
session or tags. The emitter now adds the Langfuse mapper for a request with a
Langfuse destination, and every non-Langfuse exporter strips the Langfuse
request metadata, since any logger can now write it.
* fix(otel): stop a key's Langfuse logger from re-sending spans to the operator's collector
With langfuse_otel only in key or team logging, the proxy publishes the generic
V2 logger at startup and it already exports to the operator's OTLP collector.
The Langfuse logger a key's request builds kept the same base exporter, so the
collector got every model span twice. A logger built after the publish now keeps
only its preset's own exporters, as it already did beside a configured V2 logger,
and a later call that would rebuild one exporting nowhere reuses it.
* test(otel): assert the collector gets one copy instead of reading the key logger's exporters
* test(otel): count the collector's spans with one comprehension clause
* fix(otel): keep the legacy langfuse_otel logger and stand it down for requests a Langfuse destination carries
Reverts the preset degrade and the published-provider exporter trim, so every
request without a key or team Langfuse destination keeps main's legacy logger.
For a request with one, the V2 fan-out already delivers the whole trace with
the Langfuse fields, so the legacy logger emits nothing for it.
* test(otel): pin the legacy langfuse_otel paths a tenant's project still gets without a destination
* fix(otel): shed a destination's Langfuse request metadata before its other Langfuse fields
At its attribute limit a span sheds the Langfuse fields only a destination asked
for first, in the mapper's order, so the generation marker went before the
request metadata blob. Put the request metadata first, as a Langfuse-configured
logger already does.
* fix(otel): keep the legacy langfuse_otel span for a destination's call rejected before any provider
The OTel v2 fan-out records no call for a request the proxy rejects before a
provider (rate limit, pre-call hook), so standing the legacy logger down there
lost the user, session and tags main sent the tenant. Stand it down only for a
call the fan-out records, using the same check the v2 logger uses.
The tenant and collector integration tests now hold their exact counts for a
few seconds, so a late duplicate copy fails them.
* fix(otel): give a key's Langfuse destination the langfuse_environment the legacy logger wrote
The destination's resource carries langfuse.environment from the key's or team's langfuse_environment, else LANGFUSE_TRACING_ENVIRONMENT, so a key's project files its traces under the environment it chose instead of default. Only that destination's export carries it.
* fix(otel): keep the Langfuse fields written for a key's Langfuse destination off every other exporter
* fix(otel): give a key's Langfuse destination the prompt and response the legacy logger sent, only where it ran
* fix(otel): keep a key's Langfuse destination on the global capture setting when it sets none, with or without global Langfuse keys
* fix(guardrails)!: make lakera_v2 on_flagged block reject PII-only detections, add mask
on_flagged block now returns 400 when Lakera flags only PII, instead of
masking it and forwarding the request. on_flagged mask keeps the old
behavior: mask PII-only detections and block anything else.
* test(guardrails): cover lakera_v2 on_flagged on every endpoint, hook and a Lakera outage
* test(guardrails): type the lakera_v2 on_flagged unit test helpers
* fix(proxy): honor litellm_settings.drop_params for Claude Code and Codex requests
The agentic CLI drop_params default checked the operator's config with
isinstance(config, dict), but ProxyConfig.config is a MappingProxyType, so the
check never matched and litellm_settings.drop_params could not turn the
default off. Read it as a Mapping so an operator who sets drop_params, true or
false, keeps the decision on /v1/messages, chat, and count_tokens
* fix(proxy): keep the Claude Code and Codex drop_params default unless the admin's value reads as false
A drop_params: true saved through /config/update never reaches the global setting, so treating the key's presence as an opt-out made Claude Code and Codex requests answer 400 there. Only a value that reads as false now turns the default off; true, empty, or missing keeps it.
* test(proxy): cover string drop_params values in the Claude Code and Codex default rows
Adds rows for "false", " False ", "true" and an unreadable "maybe" under litellm_settings, so swapping the value check for a plain is-not-False comparison fails the suite
* docs(proxy): say only a false operator drop_params turns off the agentic CLI default
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(guardrails): opt-in response_masked_data on PANW Prisma AIRS block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): integration cells for PANW include_masked_response_on_block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): reject non-boolean include_masked_response_on_block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(guardrails): update PANW tool-call comment for include_masked_response_on_block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): build PANW kwargs without mutation and clean up DB-created test guardrails
Restore main's single-expression kwargs in the PANW initializer. The two DB-backed integration tests now create their guardrail inside the scenario and delete it on exit.
* fix(guardrails): type raw PANW boolean setting before validation
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep Mistral thinking text on streaming /v1/messages
* test(anthropic): type the signed thinking stream test helpers
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(vertex_ai): use HTTP/2 httpx client for search_api vector store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(httpx): exercise configured request path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(httpx): verify HTTP/2 request negotiation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(httpx): preserve HTTP/2 for sync proxy mounts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): add h2 to proxy extra so vertex search HTTP/2 is on by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): isolate ssl context for http2 clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http): add an HTTP/2 opt-out and harden the Vertex AI Search HTTP/2 client
litellm.http2 is now unset by default, so http2: false or LITELLM_HTTP2=False
turns the provider's HTTP/2 default off. A caller's own SSLContext keeps the
search on HTTP/1.1, the HTTP/2 async client honors DISABLE_AIOHTTP_TRUST_ENV,
forced IPv4 transports get the CA bundle and client cert, and vector store
search retries once when the server drops the connection with a GOAWAY.
* fix(http): retry vector store search only on HTTP/2 GOAWAY and keep aiohttp pool and proxy behavior
Scope the one retry to a connection Google closes with an HTTP/2 GOAWAY and
log it. When HTTP/2 replaces the aiohttp transport, keep aiohttp's idle
keepalive and connection cap, and only then skip env proxies under
DISABLE_AIOHTTP_TRUST_ENV, so the global LITELLM_HTTP2 toggle keeps HTTPS_PROXY
* test(http): cover the vector store search GOAWAY retry in unit tests and drop duplicate SSL cache comments
* fix(http): open another HTTP/2 connection once 100 streams are in use
httpcore keeps every request to an origin on one HTTP/2 connection and caps it at 100 concurrent streams, so request 101 and later waited with no timeout. The handlers now spread requests over extra connections, opening one only when every open connection has 100 streams in flight. The force_ipv4 wire tests now cover ssl_verify=False and the client certificate.
* fix(vertex_ai): reuse one Vertex search OAuth token across searches
Every vector store search built a new search config, so the token cache it carries was thrown away and each search fetched a new Google OAuth token with a blocking call on the event loop. Under a burst of searches that blocking fetch held back every HTTP/2 response. The provider config is now shared, so searches with the same credentials reuse one token until it expires.
* fix(http): keep the HTTP/2 connection spread within the connection cap
* fix(http): keep sync vertex search on HTTP/1.1, since httpcore's sync HTTP/2 is not thread-safe
httpcore 1.0.9 hands two threads the same HTTP/2 stream id, so threaded sync
searches failed with protocol errors. Only the async client, which the proxy
uses, now defaults to HTTP/2, and the sync connection spread is removed.
* fix(http): split the connection cap across HTTP/2 lanes
Each lane had its own pool with the full connection cap, so an origin answering
HTTP/1.1, or several origins on one client, could exceed max_connections once
more than 100 requests were in flight. Lanes now open only while lanes times
100 streams stay under the cap, and each lane pool gets at most 100 of it.
* test(http2): scope wire-test peer logs to each fixture instance
* fix(http2): keep TE, force_ipv4 trust, evicted-client and opt-out behavior on the HTTP/2 path
- drop TE values other than trailers on https requests through the HTTP/2 spread, since h2 refuses them
- force_ipv4 transports keep httpx's default trust store (SSL_CERT_DIR) unless SSL is configured
- the evicted client closer asks the HTTP/2 spread whether a request is in flight
- LITELLM_HTTP2 set to 0, no or off, and a quoted "false" http2 setting, opt out of HTTP/2
- the wire test's held-search barrier waits on an event instead of polling
* fix(http2): count proxy-mounted transports when checking an evicted client for in-flight requests
* test(http2): move the HTTP/2 eviction cases that open a loopback server into the integration wire tests
* fix(http2): read an unquoted http2: 0 as off and free the stream slot of a response dropped without closing
* fix(http2): free the stream slot of a partly read response dropped without closing
* test(http2): check that the evicted client and its stream close after a partly read response is dropped
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Code outside the router now reaches it only through the entry points
(registry, settings, health, select, apply, observe, register_plugin, retire)
or the endpoint methods; the outside-member allowlist shrinks from 139 to 45.
litellm/router.py no longer imports litellm.proxy: it asks the installed
RouterHost (litellm/router_host.py) for vertex pass-through credentials,
member auto-router authorization and auto-router prompt compression, and the
proxy installs its host on import. Call sites where a model name can be
missing keep the old answer instead of reaching an entry point with None.
* fix(proxy): default the uvicorn worker healthcheck timeout to 60 s so multi-worker boots survive a slow import
* fix(proxy): warn on uvicorn below 0.37 when multi-worker boots cannot get the longer healthcheck
uvicorn 0.33 to 0.36 has no timeout_worker_healthcheck, so the 60 s default never reaches it and a slow worker start still crash-loops with no hint. Print an upgrade warning when the proxy runs more than one uvicorn worker on such a version, keep granian out of the uvicorn-only path, and type the new tests.
* fix(proxy): name --run_granian in the healthcheck help text and add per-worker grace guidance to the helm chart
* fix(proxy): keep the litellm command light so spawned workers answer uvicorn's first health ping
Each worker uvicorn spawns re-runs the parent's main script before it starts
answering health pings. The litellm console script imported the whole SDK at
the top, so on a busy or CPU-limited host a new worker missed uvicorn's 5 s
ping and was killed in a loop. The script now runs litellm_cli.main:proxy_server,
which imports the proxy only when called, and gateway.launch imports LiteLLM
inside main the same way.
This replaces the earlier 60 s healthcheck default, so uvicorn's 5 s
healthcheck and SIGTERM timing stay as they are.
* fix(proxy): load the worker app before its log config and without GC pauses
* fix(gateway): ship litellm_cli in the gateway image
gateway.launch imports without_gc_pauses from litellm_cli, which the
gateway image never copied, so python -m gateway.launch failed at import.
Also build the gateway image when the launcher or litellm_cli changes.
* fix(proxy): apply the worker log config before the proxy app import, as earlier releases did
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): preserve reasoning fields on streaming cache hits
* fix(proxy): validate cached streaming delta fields
Signed-off-by: jesco-absolut <team@srswti.com>
* fix(caching): replay cached audio, images, reasoning and citations in stream order
A cached answer replayed as a stream now ends with its own empty finish chunk, so a chunk that
carries only audio or images keeps its payload and role. Reasoning fields lead on a chunk with no
text, and annotations arrive once on the last text slice, matching a live stream's order.
* fix(caching): replay a cached tool call after the text, like a live stream
Only the role, reasoning fields and provider fields lead a replayed answer. Tool calls now arrive on
their own chunk after the last text slice, and logprobs ride the first text chunk, so a cached
answer with reasoning and a tool call keeps a live stream's reasoning, text, tool call order.
* fix(caching): replay the reasoning before a cached tool call that has no text
A cached answer that thinks and then only calls a tool now replays its reasoning on its own chunk before the
tool call, the order a live stream sends, instead of one chunk carrying both.
* fix(caching): keep blank-only text when replaying a cached answer with reasoning
A cached answer that thinks, sends only whitespace as text, and then calls a tool now replays that whitespace
between the reasoning and the tool call, instead of dropping it.
---------
Signed-off-by: jesco-absolut <team@srswti.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Router gains eight entry points (registry, settings, health, select, apply/aapply,
observe, register_plugin, retire) in litellm/router_api.py. The 123 public members
outside code used become DeprecatedRouterMembers (litellm/router_deprecated.py):
same signatures and results, a DeprecationWarning naming the replacement and a use
counter for callers outside the router. Router internals move to _core_ names;
subclass overrides of old members are still called by the router.
* feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations
* feat(proxy): roll back each migration of the dry run on its own and baseline-check databases without a ledger
* test(proxy): cover the dry run CLI and PrismaManager wrapper without a database
* fix(proxy): refuse the no-ledger dry run for a custom migration folder like boot does, and probe the rolled-back indexes directly
* ci(codecov): stop carrying forward the roi-postgres and redis-compat flags
Their carried-forward sessions were measured against older proxy_cli.py revisions (633 and 689 line maps against 710 today) and mark docstring lines of the merged file as missed, which failed codecov/patch on this PR at 70% while every unit shard reports the lines hit
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: drain shutdown callbacks and persist pending tag spend
* fix: catch shutdown timeouts on Python 3.10
* fix: share shutdown callback budget with HTTP draining
* fix: type shutdown cancellation tasks and document its clock
* test: retain callbacks after bounded worker cancellation
* test(proxy): inspect the lifespan body after the SIGTERM wrapper split
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): keep the success callback enqueue on its own task
The synchronous enqueue spawned the logging worker loop inside the awaited
call and ran callbacks that per-call loops used to drop, which hung the SDK
callback tests on the autouse worker flush. The shutdown worker flush covers
the queued callback on its own, so the one-tick task is restored
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging_worker): keep the queue's unfinished count exact when a full queue rejects a callback
The aggressive clear ran the rejected callback and marked it done against
the queue it was never put in, so flush() could return while a real
callback was still running and the shutdown drain cancelled it early. The
rejected callback is now re-enqueued once the clear made room, and only
runs uncounted when the queue is still full
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(logging_worker): build the aggressive-clear run set immutably
Final tuple instead of an appended list, with the re-enqueue in a
_requeue helper, and Final annotations on the overflow regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): track overflow callbacks through shutdown
* chore(logging): explain shared callback ownership
* test(logging): isolate deferred callback completion
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dashscope): derive protocol bases from one domain
Accept the DashScope domain root, the OpenAI-compatible base, the Anthropic
Messages base, or a native /api/v1 base and derive every surface from it
Adds a native Anthropic Messages config for /apps/anthropic so a single
deployment serves chat, messages, embeddings, rerank, image generation, and
video against the same credential
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* fix(dashscope): keep Anthropic credentials off DashScope messages
Require a DashScope key or a caller auth header before the Messages path
runs, and opt out of workload identity, so the base class cannot fall back
to ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN
Also format the image generation base selection
* style(dashscope): keep the messages config within the format gate
* test(dashscope): cover caller Authorization on messages
* fix(dashscope): keep the chat bridge for protocol-named bases and the image default for unconfigured calls
* fix(dashscope): keep configured image and rerank bases, normalize blank keys, derive brand chat bases
- image calls fall back to the provider's own chat base env (DASHSCOPE_API_BASE, QWENCLOUD_API_BASE, QWEN_AI_PLATFORM_API_BASE) before the image default
- an explicit /compatible-api/v1/reranks URL wins over DASHSCOPE_API_BASE_RERANK again
- a blank DashScope key counts as unset on /v1/messages, so it never falls back to ANTHROPIC_API_KEY
- QwenCloud and Qianwen AI Platform chat and embeddings derive their compatible-mode base from a domain root
- tests annotate new variables with Final and drop the config-class assertion
* fix(dashscope): keep an explicit compatible-api rerank base as given
* fix(dashscope): derive chat from compatible-api bases, keep the caller's rerank base, and let explicit bases beat per-endpoint env vars
* fix(dashscope): keep a full DashScope image endpoint URL as given and type the brand tests
* fix(dashscope): let the image env var beat any shared DashScope base
* fix(dashscope): keep a full DASHSCOPE_API_BASE image URL ahead of the image env var
* fix(dashscope): keep a native DashScope endpoint URL as given for chat and the api base header
---------
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(guardrails): scan and rewrite streamed /v1/completions in post_call pipelines
Gives OpenAITextCompletionHandler a process_output_streaming_response that
assembles the buffered TextCompletionResponse chunks per choice index at end
of stream, runs the guardrail over the assembled text, and writes text
rewrites back into the first text-carrying chunk of each choice. Tests and
live proof still pending.
* fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails
Assemble each choice's streamed text once the stream ends, run the guardrail on it, write a mask back into the buffered chunks, and end the stream with the guardrail's error on a block. User key metadata now goes through the shared metadata bucket so applied guardrails reach spend logs.
* test(guardrails): cover streamed /v1/completions under post_call guardrails
* fix(guardrails): keep Gray Swan key metadata, scan a cut-short text stream, and pass non-chunk frames through
* fix(guardrails): replace blocked text completion streams like chat, keep Gray Swan's vendor payload, scan text streams in the content filter
* fix(guardrails): release an unfinished choice's held text and omit usage the upstream never sent
The content filter's streaming hook held back the end of a choice until its
finish_reason, which litellm sends only for the first choice of an n>1
stream, so choice 1 lost its last characters on text and chat streams. A
closing chunk now releases every unfinished choice when the stream ends.
A blocked text stream's closing chunk reported 0/0 usage when the upstream
sent none; it now carries usage only when the upstream reported it.
* fix(guardrails): send no usage on the content filter's end-of-stream chunk
* fix(guardrails): keep the usage chunk last and log the masked text of streamed text completions
* fix(guardrails): stream Responses API events through the content filter as they arrive
* fix(guardrails): treat a null text completion choice index as choice zero when assembling streams
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
