mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-16 04:29:31 +00:00
888 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
devin-ai-integration[bot]
|
da2f5a498f
|
feat(agents): support Databricks Agent Platform agents in the agent gateway (#45681)
* feat(agents): support Databricks Agent Platform agents in the agent gateway Add a databricks_agent provider that speaks the MLflow ResponsesAgent wire (Responses-shaped input, output_text items, output_text.delta SSE) to Model Serving endpoints (/invocations or the unified /serving-endpoints/responses) and Databricks Apps (/responses), with a PAT or OAuth M2M client credentials Register it through the agents config block or the Admin UI agent catalog and call it through /a2a/<agent> (message/send, message/stream) or model: a2a/<agent> chat completions. The chat route now runs every completion-bridge agent through its own provider, and attaches the minted Databricks OAuth token for both bridge and URL agents. Consumed OAuth settings never reach the provider call * fix(agents): harden databricks_agent input, output, and OAuth handling Non-text content parts answer 400 unless drop_params is set, a reply without an output list answers 502, flat OAuth fields resolve an os.environ api_base, the bridge debug log no longer echoes the request model, routing tests stub only the HTTP boundary, and the catalog description fits the dropdown * fix(agents): merge agent and client extra_headers on the a2a chat route, and reject an empty databricks_oauth block * fix(agents): bridge only databricks_agent agents on the a2a chat route, keep the card url over a client api_base, and require workspace_url for a Databricks App * refactor(agents): read the a2a chat bridge providers from a constants set instead of branching on a provider name * fix(databricks_agent): let the configured credential win, reject malformed client extra_headers, and keep a missing endpoint name empty * test(agents): drive the flat OAuth a2a send through the real bridge * fix(agents): keep an empty databricks_oauth block ignored on url agents, as before this change * fix(agents): accept bare App hosts, env service principals, and os.environ refs on Databricks agents * fix(databricks_agent): address review caveats on chat bridging, params, and auth Map max_tokens and max_completion_tokens to max_output_tokens and pass temperature, top_p, and user, so /v1/messages works without drop_params. Send A2A message metadata only with string values, answering 400 or dropping the rest under drop_params. Join message items with a blank line in both stream and non-stream replies. Cache the env service principal token until shortly before it expires. Keep databricks_agent replies out of the response cache. On the chat bridge, let request fields win over the agent's stored params while pinning the agent's api_base and api_key, and apply the agent's static_headers between client and agent extra_headers, as /a2a does. Resolve os.environ/ only for databricks_agent api_base and api_key instead of every bridge provider. Ignore an unusable databricks_oauth block on URL agents on chat, as at the merge base, instead of answering 400. * test(internal_context): list the databricks agent token cache as process-local * fix(databricks_agent): keep the agent's endpoint, input, and target out of client control extra_body model and input answer 400, or are dropped under drop_params. A client api_base or base_url never replaces the agent's card URL on the bridge. Chat to a URL agent keeps its merge-base call, and an incomplete databricks_oauth block on a URL agent keeps its merge-base ValueError. * test(databricks): expect the merge-base 400 for a client api_base on agent chat * fix(databricks): keep agent-owned inputs and identity headers out of a chat caller's reach * fix(databricks): bound the agent-over-caller merge to two levels so it needs no recursion --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
d9b5d9e19f
|
Revert "feat(proxy): add LITELLM_FIPS_MODE startup gate with provider assertion and loud password migration failure (#42700)" (#46740)
This reverts commit
|
||
|
Mateo Wang
|
1938f537bf
|
fix(decisions): read a deployment's own supports_box_questions flag before the cost map (#46696)
* fix(decisions): read a deployment's own supports_box_questions flag before the cost map The box-question check only read the shared cost-map row and required a decisions mode there, so an OpenAI-format deployment whose model_info sets supports_box_questions was refused with a 400, which kept the wire test from #46633 red on main * fix(decisions): keep the box flag on its deployment and out of request bodies A model_info that is not a plain dict no longer fails the call, the proxy decisions endpoint drops a caller-sent model_info, and the router stops copying supports_box_questions onto the shared cost-map key. * fix(decisions): keep the box flag shared with same-model deployments like main Scoping supports_box_questions to one deployment turned box questions on an unflagged sibling of the same backend model from reaching the provider into a 400. Siblings inherit the flag through the shared cost-map row again, as they do on main. Also formats two new tests. * test(decisions): pin box flag inheritance by an unflagged same-model deployment * test(decisions): audit box-question deployment flag across providers, routes, and chaos cases |
||
|
Kerry Lu
|
fac0135d82
|
fix(search): bill an unpriced fallback search tool at its own price, not the failed primary's (#46734)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
11e9fcc033
|
fix(cost): price bridged /v1/messages reasoning tokens from the chat completion, not the served Anthropic body (#46726)
* fix(cost): price bridged /v1/messages reasoning tokens from the chat completion, not the served Anthropic body * fix(cost): keep the type gate flat for the bridged /v1/messages billing path --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
67d75522ed
|
fix(test): wait for the wedged Redis to be reached instead of a fixed 0.1s sleep in team roster sync tests (#46727)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
1fc456917f
|
fix(mcp): preserve progress and cancel upstream work promptly (#46134)
* fix(mcp): preserve progress and cancel upstream work promptly * fix(mcp): apply output guardrails to progress messages --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
f470e418ea
|
fix(anthropic): keep signed empty thinking blocks and honor thinking.block_binding (#46342)
* fix(anthropic): treat adaptive thinking as enabled when mapping response_format
is_thinking_enabled only recognised thinking.type=enabled, so a request that
paired thinking.type=adaptive with response_format took the synthetic JSON
tool path with a forced tool_choice, which Anthropic rejects alongside
thinking. The same check also gates the Bedrock Converse and Databricks
forced tool_choice downgrades, so they now see adaptive thinking too.
* fix(anthropic): keep signed thinking blocks with empty text on chat completions
Models that default to display: omitted return signed thinking blocks whose
text is empty, and Anthropic rejects the next turn when they are not replayed
unchanged. is_unsignable_thinking_block dropped them along with unsigned
blocks, and the inline content path required non-empty text, so a tool-use
turn on Opus 4.7+ or the 5 families lost its thinking blocks and 400ed on the
tool-result turn. Only a missing signature now drops a block, and the
mid-conversation system placement mirrors that rule. The /v1/messages strip
helper is left to #41362.
* fix(anthropic): skip the thinking-strip retry when block_binding asks for an error
The /v1/messages retry loop strips thinking blocks and resends on any
invalid-thinking 400, including the binding mismatch a caller opted into
with thinking.block_binding.prefix_mismatch_behavior: error. The retry
decision now sees the request body and surfaces that 400 instead of
overriding the caller's choice. #42888 covers the litellm_params side.
* fix(bedrock): carry thinking.block_binding through the Invoke thinking rebuilds
Bedrock Invoke rebuilt the thinking param from scratch when display: updates
was requested, on both the /v1/messages and chat completions paths, and the
legacy enabled-to-adaptive translation plus the clear_thinking injection did
the same, so thinking.block_binding never reached Bedrock. Every rebuild now
spreads the caller's thinking and only overrides the keys it owns, and
AnthropicThinkingParam types block_binding. The thinking-binding-controls
beta was already on the Bedrock allowlist.
* fix(anthropic): keep forcing the response_format tool under adaptive thinking
This reverts
|
||
|
devin-ai-integration[bot]
|
e344dbd8db
|
feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043)
* feat(proxy): split Azure PTU deployments across teams by share
A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to
ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on,
routing hides the deployment from teams holding no share, the v3 limiter enforces each
share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily
flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team
Usage page report PTU-hours next to tokens.
* fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name
The PTU flag is read from the process environment on every check instead of
through the secret-manager path, so a proxy with a hosted secret manager does
not pay a round trip per request. A team's ceiling now carries the model's
cached-input ratio and settlement charges cache reads at that ratio, matching
how Azure sizes a PTU. A deployment registered for one team is found by its
team_public_model_name for PTU-hours and the ceiling, since its model_name is
rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and
parsed_ptu_shares refuses booleans, strings, and non-string team ids, a
fractional ptu_count reserves nothing, and models_processed counts distinct
deployments rather than holdings.
* fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations
* fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id
* fix(ptu): refuse ptu_shares naming a team that does not exist
* refactor(ptu): keep Azure detection under llms/azure and check share teams with one query
* fix(proxy): drop the reverted TeamIdSearchFilter import after merging main
Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally
* fix(proxy): chunk the ptu_shares team lookup under the IN-list bound
* fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling
* fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling
* fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour
* fix(ui): seed total_ptu_hours in the empty daily activity metadata
* fix(ptu): size usage rows by the name the request used
* fix(ptu): size a team's usage rows by the deployment it is served from
* fix(ptu): never size a team's usage row by another team's reservation
* fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length
* fix(ptu): read a share map only through the terms registration requires
A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and
ptu_effective_from, or one that does not add up to ptu_count, is refused at
registration but was enforced once loaded: every other team got the 400 and
the share holder a ceiling while no flat cost accrued. _deployment_shares now
reads the shares through ptu_terms, so such a row reserves nothing, the same
as before the map was read.
* fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off
|
||
|
moyai-devin-berriai[bot]
|
3151c47960
|
fix(guardrails): resolve decision models in caller team scope (#46085)
* fix(guardrails): resolve decision and judge models in caller team scope * fix(guardrails): scope guardrail model calls to the caller's team only where it owns the deployment Read the team only from the auth object the proxy attached to the request, never from caller JSON, and use it only when the router's route to the guardrail model for that team reaches a deployment a teamless caller cannot. Shared names, aliases, legacy team tags and wildcards keep resolving and billing as before. Decision calls pass the team in litellm_metadata only when it is used, so the logged model group stays intact. Adds integration cells for chat, responses and messages, forged team ids, a team deployment that shares a name with a shared model or wildcard, and billing, plus unit cases for legacy team tags and aliases. * fix(guardrails): route team guardrail calls to the team deployment without charging the team Guardrail calls on a team's own model now go to the deployment the router picks for that team, by its id, with no team on the call. Like every other guardrail call, they write no spend row and add nothing to the team's or the key's spend * fix(guardrails): pin the team deployment only for the judge guardrail, not shadow eval judge calls * fix(guardrails): keep the teamless route when only a team wildcard covers the guardrail model * refactor(guardrails): scope the team model fix to the decision model guardrail only * fix(guardrails): send decision calls on a team's own model with the team, and keep team wildcards out The decision call on a team's own model now carries the caller's team instead of being pinned to a deployment id, so it is charged to that team and goes through the team's normal routing: an access window that reserves the deployment for the team lets the call through, and pre-call checks see the input and skip a deployment too small for it. A team deployment listed under a wildcard public name, such as team_public_model_name openai/*, no longer captures a guardrail model the team never named, whatever model it points at. Adds unit and integration cells for the reserved deployment, mixed context windows under enable_pre_call_checks, a team wildcard name over a fixed model, and the owner team's spend. * fix(guardrails): log the owner team's name on decision calls charged to it The decision call on a team's own model now carries the team alias beside the team id, so the spend row and the Logs page name the team the call was charged to. * fix(decision_model): use the caller's team only for names nothing else serves A team model named like a shared deployment, a shared wildcard or an SDK public name took over the guardrail's decision calls for that team, so a team admin could point a proxy-wide guardrail at their own endpoint. The team is now used only when a caller without a team reaches no model under the name. * refactor(decision_model): drop the team route subset check the served check covers Once nothing serves the name without a team, every deployment a caller without a team can reach is a wildcard, so the subset check only ever rejected an empty team route. Pin team wildcards over a name nothing else serves. * fix(decision_model): use the caller's team only when judge_target finds nothing without a team The served check rebuilt the router's rules and missed a shared wildcard whose provider comes from custom_llm_provider, so a team model under the guardrail's name took over that team's decision calls. Ask judge_target, the same lookup the teamless dispatch makes, instead --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
30ada9d26a
|
feat(batches): store each batch line's request and response in S3 and GCS (#41691)
* feat(batches): emit per-line JSONL batch records to callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): only emit line items for final batches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): bump soupsieve to 2.9.2 for GHSA-gjv8-xp57-g29c and GHSA-j934-xhv5-fg8f
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(batches): satisfy ruff format for line item logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for store_batch_line_items_in_callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): cover line-item edge shapes and general settings propagation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep deployment credentials out of callback-visible params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(batches): declare credential transport attribute on Logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): integration contract for per-line batch callback events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve store_batch_line_items_in_callbacks through the settings store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): assert per-line events beside the aggregate and create events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop resurrected use_legacy_interactions_schema shim from merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): assert line item events carry no credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reset batch line-item flag when the setting is absent
Removing store_batch_line_items_in_callbacks from the DB general settings left the module flag at its previous value because the effect early-returned when the key was neither in the DB nor owned by YAML. The proxy owns the global, so an absent resolved value now writes False.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): map anthropic batch lines through the anthropic message transformer
Anthropic result lines carry a message-shaped body, so ModelResponse(**body) produced empty choices and a usage key that blocked the batch-rate usage attach. Extract the existing AnthropicResponse -> ModelResponse conversion used for /v1/messages logging into anthropic_message_to_model_response and reuse it for anthropic line items.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: tag the anthropic result cast with a cast-ok reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: shorten cast-ok reason so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip line-item callbacks for providers without a line reconstruction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): reconstruct bedrock and mistral batch lines through their provider transformations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: wrap widened provider literals so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): pair bedrock lines by recordId and type them from the reconstructed response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep line item events out of proxy limiters and strip credentials from child params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep cast-ok tags on the flagged cast lines for the discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip failed line item events in parallel request limiter failure hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): tolerate lifecycle spend rows when asserting no line item rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): place mutable-ok tags on the lines the discipline gate flags
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip router budget accounting for batch line item events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep anthropic line errors and skip native vertex batch rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): restore test-quality tags on line item tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move batch line-item unit test to tests/unit after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): emit batch line-item callbacks once per batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): release line-item claim when fan-out emitted nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): make the line-item claim release ownership-safe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep bodyless failed batch lines reaching callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): seed batch line hidden params in get_hidden_params defaults
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(batches): reuse fetched batch result files for line-item logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): release line-item claim when nothing was emitted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): type batch line-item callback sites to satisfy basedpyright budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: wrap batch hidden-params cast for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): reason suppressions for batch line-item lint deltas
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop stale mutable-ok suppressions after LIT002 removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters (#44486)
* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters
Batch line-item callbacks (call_type=acompletion + litellm_params.batch_parent_id)
bypassed the call-type-only batch guards:
- ClickHouseSpendLogger wrote one spend row per line on top of the aggregate
aretrieve_batch row, billing every batch twice with the flag on.
- Router deployment callbacks and the routing-strategy usage handlers charged the
historical batch's tokens/rpm to the live TPM/RPM counters, so after
Router.aretrieve_batch the next chat raised RouterRateLimitError.
Skip these events wherever the aggregate batch retrieve is already skipped.
* fix(logging): keep internal litellm_params authoritative in model_call_details
A request body key named litellm_params lands in optional_params, and
update_environment_variables merged **optional_params after the internal
entry, so a caller could overwrite the logging kwargs' litellm_params and
forge markers like batch_parent_id. Live-reproduced: a Gemini chat whose
body carried {"litellm_params": {"batch_parent_id": ...}} returned 200
with no ClickHouse spend row. Re-assert the internal litellm_params after
the merge so every sink (ClickHouse, Postgres, router quota counters)
reads the same trusted value, and assert the aggregate aretrieve_batch
row still writes in the ClickHouse line-item test (review finding).
* fix(clickhouse): skip failed batch line items in the spend sink too
A failed JSONL line is part of the aggregate aretrieve_batch row's
request counts; writing it its own spend_logs row inflated request
counts with the line-item flag on. Extend the batch line-item guard to
the failure handler (review finding) and pin both directions: a failed
line item writes no row, a genuine live failure still does.
* fix(batches): declare batch_line_items service family on claim calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(integrations): keep batch line items out of the built-in metering sinks
With store_batch_line_items_in_callbacks on, per-line batch events carry a
real response_cost and reached six metering sinks that lacked the
is_batch_line_item_event guard, so a completed batch metered aggregate +
per-line (roughly 2x true spend): prometheus (litellm_spend_metric and
request counters), openmeter, lago, datadog cost management (FOCUS
BilledCost), newrelic metrics, and the OTel v1/v2 gen_ai.usage.cost
metrics. Skip line-item events in each sink's success/failure metering
entry points, matching the guards already on the Postgres and ClickHouse
spend sinks. OTel spans for line items are still emitted; only their
cost/token metrics are skipped. Tests drive real sink instances on the
real dispatch lists through a completed 2-line batch and assert each
sink meters exactly the aggregate.
* style: format metering-sinks test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): skip synthetic line-item durations in v2 metrics; isolate prometheus test
Follow-ups from review on the sink guards: record_failure() now skips
batch line items too (a failed line's interval measures retrieval and
emission, not its model call), and the prometheus guard test snapshots
and restores the process-wide registry collectors instead of leaving
them unregistered. The OTel v2 test is now a flag on/off A/B over the
whole metric snapshot: cost and usage values must match exactly and no
metric may gain datapoints with the flag on.
* chore: finish main merge conflict resolution for batch line items
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: sort imports in litellm_logging after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): use the public hook class names main now exports
* fix(batches): only the batch writer can mark a callback event as a line item
amoderation, transcription and speech build litellm_params from every request kwarg,
so a body field named batch_parent_id reached Logging.litellm_params and every
spend, budget and rate-limit hook skipped the request. update_environment_variables
now drops an incoming batch_parent_id; the line-item writer sets it through
Logging.mark_batch_line_item. Batch line hidden params are emitted only when set.
* chore(batches): keep the lint gates within main's ceilings after the merge
Typed the helpers main now types (Mapping litellm_params, Any Span), replaced the
banned cast with a literal match, dropped isinstance checks main made redundant,
and moved test-quality reasons onto the lines the gate reads.
* fix(anthropic): read the message id off the validated message
The anthropic_messages logging path can hand over an already parsed message object,
which has no .get; main read its id through getattr, so take it from the validated
AnthropicResponse instead.
* fix(batches): an unreadable error file no longer drops the output line events
Costing counts an error file it could not fetch as no failed lines; line logging
refetched it and the raise aborted the whole fan-out, so the successful lines were
never emitted. The error file now reads as empty with a warning.
* test(batches): pin billing for batch_parent_id on every endpoint and value shape
Chat completions, embeddings, responses and messages join moderations, speech and
transcriptions, and speech is driven with an empty, int, list, object and 5 KB value.
* test(batches): bill every request of a concurrent burst mixing forged and plain bodies
* revert: keep the line-item fan-out all-or-nothing on an unreadable error file
|
||
|
ryan-crabbe-berri
|
1f041a3265
|
fix(openai): strip prompt_cache_breakpoint under drop_params for OpenAI models that cannot read it (#46453)
A chat completions request to gpt-4.1 (or any OpenAI model older than 5.6) with drop_params on still sent the marker inside messages and tools. It is now removed there, while drop_params off keeps sending it through and models that read it keep it either way Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yujonglee
|
c0d7264e8b
|
feat(rust): add native OpenRouter Responses (#46000)
The Responses route now resolves openrouter alongside openai. The OpenRouter config reuses the OpenAI request and response transforms and owns only its bearer credential (OPENROUTER_API_KEY, then OR_API_KEY) and base URL (OPENROUTER_API_BASE). Wire model ids follow Python's get_llm_provider, including the openrouter/auto native id with an explicit provider. The Python route host no longer hands litellm.openai_key to OpenRouter. The OpenRouter live test lives in tests/live/openrouter.rs and returns early when the provider key is not exported, so cargo test stays offline by default instead of hiding it behind --ignored. Co-authored-by: nate-berri <nate@berri.ai> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
devin-ai-integration[bot]
|
fbc2e358bc
|
fix(responses): bill a background response once, on the first read or poll that sees it finished (#45703)
* fix(responses): bill a background response once, from the poll job
Reading a completed background response with GET /v1/responses/{id} wrote a
full-cost spend row on every read, on top of the row the background cost poll
writes. Reads of a Responses object (GET, DELETE, cancel) now never bill, and
the poll job is the single biller.
The poll job now bills with the creating request's attribution (key hash, team,
org, user, aliases, tags) read off the managed object row, through a helper it
shares with the batch cost poll. The background create persists the creating
key and request tags so that attribution survives until the poll runs.
* test(proxy): a background response read reports zero cost
* fix(responses): bill a background response once, on the first read or poll that sees it finished
A conditional update moves the stored row out of queued or in_progress so only one worker or pod bills it, through one re-read stamped with the creator's attribution. A failed billing read puts the row back for the next poll
* fix(responses): keep the caller's response id out of the billing failure warning
* fix(responses): keep the creating client's User-Agent tags on the one background charge
* fix(responses): keep managed background reads free and bill each response once
The managed files hook marks reads and cancels of proxy-issued response ids, so only those stay free. SDK and database-less reads of a background response still bill, extra_body background creates are tracked, and the poll's status read no longer cools down a deployment on a 404.
* fix(responses): bill finished background responses from the fetched body and store rows Postgres accepts
* fix(responses): give the background billing event its own spend log row and bill responses the create left untracked
* fix(responses): narrow the managed files hook with a protocol instead of an enterprise import
* fix(responses): leave a read the caller retargets with client-side credentials to the cost poll
* fix(responses): bill background responses on the creator's end user and report the charge on the settling read
The read that settles the bill returns the charge in x-litellm-response-cost, the billed event
carries the creator's end user and the key, team, and user budget fields, and its spend row is
named after the id the client got. Override reads, cancels, and deletes settle from the
configured deployment, a streamed create marks its row billed, stored rows keep no model
output, and model insights counts the billed row's spend without a request.
* test(responses): cover background response billing across proxies, reads and chaos
* fix(responses): settle background billing only for callers allowed to address the response
A delete now settles the bill only after the response id access check passes, a streamed
create is marked billed when it sets background inside extra_body or its completed event
arrives after the last chunk, the billing event keeps the deployment's litellm_params
base_model, a key with no user and no team bills no user instead of its owner scope, and the
enterprise cost check import is optional
* fix(responses): record the bound on the pending status IN filters
* fix(responses): send the billed cost split on the settling read and keep the batch cost event as it was
The read that settles a background response now sends the whole cost header
family from the billed event, so the breakdown headers add up to the charge.
Batch cost events go back to main's attribution (creator id as the user, no
budget fields); only background responses get the end user and budgets.
Integration tests expect the charge on the settling read (or none when the
poll billed first), one charge for a streamed create, and SDK flows that poll
with the id each read returns.
* fix(responses): bill each background response once across upgrades, streams, and failed settles
Rows stored before this billing model are found by the issued id and rekeyed by the
provider id, and the cost poll retires one a read already billed. A streamed background
create is free on its own and bills the finished response through the same managed row
before the client sees response.completed, so a racing read cannot bill it again. A
settle that fails retries in the background, a retargeted read with no row tracks and
bills from the configured deployment, and a delete keeps its billed cost header.
Reverts the unrelated json.dumps narrowing in the completion transformation.
* fix(responses): bill cache-polled background streams and match poll rows by the issued id
* test(responses): cover deletes, failed settles, budgets, base models, and cache-polled background billing
Adds integration cells for the background response billing paths the audit
inventory lists: a DELETE that settles the bill and refuses a stranger, a
settle the database refuses that retries once it recovers, the Prometheus
remaining budget, base_model pricing from model_info and litellm_params, the
daily model usage rollup, the end user, the settling read's cost split, stored
rows without model output, a forged managed marker on the create, streamed
creates with background in extra_body, and the cache-polled create
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Yusuf Efe
|
c9ce9a733d
|
fix(proxy): allow model instances in LiteLLM_EndUserTable validation (#41677)
* fix(proxy): allow model instances in LiteLLM_EndUserTable validation * fix(proxy): preserve property-backed attributes when coercing spend * refactor(proxy): default null end-user spend with a field validator * fix(proxy): drop model-wide from_attributes from the end-user table * test(proxy): unskip the /customer/block integration contracts --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
622daf3cdf
|
fix(scim): write a group's roster in one locked transaction instead of one member at a time (#45197)
* fix(scim): write a group's roster in one locked transaction instead of one member at a time POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the membership rows and the users' team tags in one transaction under the team's advisory lock, so a 500-member push runs a fixed number of statements instead of thousands. PUT now answers with the roster it just wrote * test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers * fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock * fix(scim): evict the cached team on a group rename and read a no-op roster from the writer A rename-only PUT or PATCH wrote the new alias and left the team cached under its id and old alias on every worker, so keys kept logging spend against the old team alias until the TTL expired. The group write now evicts the cached team (and broadcasts the eviction) right after the alias or metadata update. sync_team_roster reads the team from the writer when the plan names nobody, so a replica cannot hand back the pre-rename row; a plan already matching the roster returns inside the lock without writing, evicting, or emitting the members metric; the response is built from the written row, so lastModified reflects the roster write; and detaching a removed member bumps the user's updated_at like every other user write. * fix(scim): bump detached members' updated_at in UTC The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts into Prisma's naive timestamp column in the session timezone, so on a database whose session is not UTC the removed member's row landed hours away from the UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and have the bulk roster cell assert the bumped row sits between two reads of the database's own UTC clock taken around the replace. * fix(scim): chunk the group write's member lookups under the bind cap Every Prisma "in" filter the group write issues bound one parameter per member with nothing bounding the list, which check_unbounded_in_lists reports as 11 new findings. The single-field lookups now go through litellm.repositories.chunked_in (caller_transaction inside the roster transaction, per_chunk_ok for the role recompute), and the one cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two slices, and a member named only in the last slice still resolves. * fix(proxy): send bulk cache evictions in slices so the broadcaster drops none Veria pointed out that the invalidation publisher caps its pending tasks at 1,024 and drops the excess with a warning, and that a group push now detaches every removed member's keys in one call instead of one member at a time, so a removal of more than roughly 340 members (three cache keys each) silently left stale key objects on the other replicas. The merge base never hit the cap because each member was a separate request. Bulk evictions (team, user, organization, and session key deletes, plus the roster settle) now publish in slices of 256 and wait, bounded at two seconds, for the backlog already handed to the event loop before each slice. A Redis that never answers costs a bulk caller at most that wait per slice instead of wedging the request, and single-key callers are untouched. Regression tests push 1,500 keys and a 500-member removal through a slow publisher and assert every eviction reaches Redis; both fail against the previous tip. * fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts Greptile pointed out that pacing the broadcasts in slices also paced the local eviction: a bulk delete waited for the publish backlog before dropping the next slice from this worker's own memory, so on a Redis that accepts connections but never answers a key deleted from the database kept authenticating on the handling worker for up to two seconds per slice. The merge base cleared every key locally at once. Bulk evictions now drop every affected entry from this worker's memory first, through the cache's own partition lookup since key objects live apart from the other auth objects, and only then publish in paced slices. The roster settle does the same for the deleted tokens, the JWT mapping keys and every changed member's user and membership entries before any step that can wait. Regression tests run both paths against a publisher that never answers and check that nothing is still cached on this worker before the first bound elapses; both fail with the local eviction removed. * fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts * docs(proxy): drop the module docstring that repeated the PR description * fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps Bulk key evictions now drop entries from memory, delete them from Redis in one chunked DEL, and broadcast every key in one pipelined publish, so a large roster removal no longer overflows the publisher's pending cap or costs a Redis round trip per member. A push waits for in-flight membership loads before evicting. User rows are locked in byte order to match the spend writer, a bulk user delete takes every team lock before its first user write, and a POST whose roster write fails deletes the empty team so the IdP retry creates it again. * test(team): observe delete broadcasts at the Redis layer * fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently * fix(team): chunk the email lookup behind the member row lock * fix(scim): create a member deleted mid-push again instead of failing the group write * fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch A member deleted mid-push is created again only when scim_upsert_user allows it, under the verb's own created_via. New group members are checked against the license seat limit as one batch before any is created. Auth cache publish drops log the key count instead of the key. * fix(scim): lock member rows while recomputing admin roles after a group push * fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member * fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member * fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time. The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team. A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup. * fix(user-delete): keep the team alias label on the team members gauge after removals * test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them * fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404 * fix(scim): answer 404 and roll back when the team is deleted during a roster write * test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast |
||
|
Thijmen Stavenuiter
|
a76e723b26
|
feat(proxy): report when each per-model budget window resets (#45797)
* feat(proxy): report when each per-model budget window resets model_max_budget_usage on /key/info, /user/info and /team/info now carries budget_reset_at per model, the open window's start plus its period, or null when no window is open. The window start is read in the same batched lookup as the spend counter * fix(proxy): count a monthly budget window from its own start when reporting its reset duration_in_seconds measures Nmo from now, so a window opened on January 15 and read on February 10 reported February 12, then null while the counter still refused requests until February 15. duration_in_seconds_from measures the month from the window start, and duration_in_seconds now shares that month arithmetic * fix(proxy): measure months from timezone-aware datetimes and assert whole usage values datetime.fromtimestamp without a tz tripped DTZ006. The month arithmetic now runs on an aware local datetime, which keeps the same wall-clock result. The duration tests move to parametrized functions, since subTest payloads carrying datetimes cannot cross the xdist worker boundary, and the new usage tests compare the whole entry * fix(budgets): end a monthly budget window a calendar month after it opens Each charge recounted the month from the charge date, so a monthly window opened in a month of another length expired days before or after the reset /key/info reports. The window is now measured from its start, by the same helper the usage report uses. A period ending after year 9999 reports no reset instead of failing the read. * fix(budgets): count monthly budget windows in UTC for every router budget Months were counted on the host's local calendar, so replicas with different TZ settings disagreed on when a monthly window ends. Provider, deployment, tag and per-model budgets now all measure a window from its start in UTC, and a test covers the router budgets. A reset past year 9999 is documented as null. * test(budgets): cover monthly tag budgets in the router window test * test(budgets): add integration cells for budget_reset_at and monthly budget windows * test(budgets): find a monthly window in another span on every calendar day --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yujonglee
|
dd0c023338
|
refactor(rust-bridge): remove the LoggerRule concept from the catalog (#46711)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
L
|
bdd80781f2
|
fix(proxy): list team access group models in /v1/model/info and keep key model lists narrow (#32599)
* fix(proxy): list models from team access groups * 修复 PR #32599 的类型检查并补齐模型列表回退测试 * fix(proxy): keep a key's own model list narrow when adding team access group models A key with its own model list was shown every model from its team's access groups on /v1/models, /v1/model/info and /model_group/info, though auth denies those calls. Team access group models now widen a key's list only when the key inherits the team's models. /v1/model/info reuses the shared listing helper without a catch-all, and route-level tests replace the mock-only ones * fix(proxy): skip the list build in the unrestricted model info check and test every listing route The unrestricted check for /v1/model/info only needs to know whether the key and team lists are empty, so it no longer builds and expands the model list a second time. The route test now covers /v1/models and /model_group/info next to both model info routes * fix(proxy): scope model info for personal and agent keys with no model list * fix(proxy): keep config keys' team access groups and own-team team_id scope in model listing --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3740d48844
|
test(cost): add tier 1 claude messages stream twins and recapture drifted usage shapes (#46690)
* test(cost): add tier 1 claude messages stream twins and recapture drifted usage shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): freeze the login throttle clock so the retry-after assertion is deterministic Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
694582aeb3
|
fix(jwt): honor end_user_id_jwt_field on the JWT-to-virtual-key mapped path (#46637)
* fix(jwt): honor end_user_id_jwt_field on the JWT-to-virtual-key mapped path A JWT mapped to a virtual key dropped the end-user claim, so spend rows had no end user and customer budgets, limits and region never applied. The mapped token now carries the claim, and it takes precedence over the body user and the x-litellm-end-user-id header, which remain the fallback when the claim is missing or empty. * test(jwt): budget-check the body or header end user when a mapped JWT has no end-user claim * fix(jwt): replace the annotation offsets with a targeted pyright suppression to keep LIT001 flat |
||
|
Mateo Wang
|
2c4ec9c6fa
|
fix(responses): open faked stream items empty so tool arguments and text arrive once (#46533)
* fix(responses): open faked stream items empty so tool arguments and text arrive once * fix(responses): open faked stream items in progress with a fresh empty list, and open mid-stream guardrail blocks empty * fix(responses): open faked refusals, reasoning summaries, custom tool calls and MCP calls empty so each value arrives once * test(integration): cover the empty-open Responses stream items on every fake, replay and block path Integration cells for the faked /v1/responses stream, the chat and messages bridges, cache replay, guardrail block events and the MCP gateway call. Each opened item starts empty and each value arrives once, checked with raw httpx, the OpenAI SDK and the Anthropic SDK, plus upstream errors, 5 KB values, a client drop, a dropped upstream connection and a worker SIGKILL |
||
|
yujonglee
|
7d79ccb7cc
|
test(rust-bridge): select native routes per test without LITELLM_RUST (#46620)
* feat(rust-bridge): scope catalog rules with a contextvar Co-Authored-By: Yujong Lee <yujong@berri.ai> * test(rust-bridge): select native integration routes with scoped rules Co-Authored-By: Yujong Lee <yujong@berri.ai> * test(rust-bridge): run the Rust extension suite without LITELLM_RUST Co-Authored-By: Yujong Lee <yujong@berri.ai> * test(rust-bridge): use scoped catalog rules in dispatch tests Co-Authored-By: Yujong Lee <yujong@berri.ai> * refactor(rust-bridge): keep catalog rules a static global and override them once per test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Taylor Hawkes
|
fd84e97f49
|
feat(providers): add Inference APIs as OpenAI-compatible provider (#41669)
* feat(providers): add Inference APIs as OpenAI-compatible provider Registers https://api.inferenceapis.com/v1 as a JSON-configured provider (chat completions and Responses API). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(providers): list Inference APIs in Add Model and the provider docs index, add request-level tests - provider_create_fields.json entry so the Add Model dropdown test passes - provider_endpoints_support.json (+ backup) entry so the provider docs check passes - tests now capture the outgoing HTTP request (URL, auth header, body) instead of only inspecting registrations - drop a redundant comment Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(providers): inject a recording HTTP session instead of patching httpx.Client Request-level tests now set litellm.client_session to an httpx.Client with a MockTransport, so nothing is monkeypatched on a class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4 * chore(providers): drop redundant comment on the inferenceapis entry Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4 * feat(inferenceapis): add cost map rows for the 12 chat models and test the responses route Prices and context windows come from https://inferenceapis.com/pricing and GET /v1/models. Moves the provider test to tests/unit and adds the interactions flag the endpoint matrix now carries. * feat(inferenceapis): price the vendor's chat model aliases and mark /v1/messages supported * fix(inferenceapis): serve /v1/messages natively and keep audio routes off the provider * fix(json providers): send provider-specific params in extra_body and report context overflow as ContextWindowExceededError * fix(exception mapping): match the Inference APIs overflow phrase instead of any context_length_exceeded substring * fix(exception mapping): map context overflow to ContextWindowExceededError for Inference APIs only Providers without their own mapper keep reporting an overflow as BadRequestError, since the router raises a ContextWindowExceededError instead of trying generic fallbacks when context_window_fallbacks has no entry for the group. Adds the integration cells for the provider, its fallbacks, and the extra_body and overflow families. * test(inferenceapis): hold upstream replies across the worker kill in the chaos cell A reply the killed worker sent before SIGKILL can lose its queued spend row, which the product never promised to keep. Holding the upstream until after the kill means only the surviving worker answers, so every 200 must log exactly once. * test(inferenceapis): make the chaos cell kill the quieter worker and always release held replies The survivor now must hold at least one in-flight request, so the burst check cannot pass as 0 == 0, and the held vendor is released on exit so a failed assertion cannot stall the wire server teardown --------- Co-authored-by: TaylorHawkes <857858+TaylorHawkes@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
구동하
|
a9e45a24d3
|
fix(streaming): keep provider response headers on SDK streaming errors (#41528)
* fix(streaming): keep provider headers on MidStreamFallbackError A 429 raised while opening a stream is wrapped in MidStreamFallbackError, which dropped the litellm_response_headers exception_type() attaches to the wrapped error. retry-after and x-request-id reached callers on the non-streaming path but not on the streaming one, so retry backoff, provider request id logging and proxy error headers lost them. * refactor(streaming): drop explanatory comment from MidStreamFallbackError header copy * test(streaming): cover sync and async header carry on MidStreamFallbackError Parametrizes the provider-header test over the sync and async stream wrappers, and drops the hasattr test, which pinned how the attribute is stored rather than what any caller observes * test(streaming): add integration cells for stream-open provider headers Cover the SDK stream openers (gemini, vertex_ai, codestral, predibase and the eager providers), the Router stream retry and fallback paths, and the proxy on both the router and the config-less serving paths, including a mixed burst, a worker SIGKILL and a SIGTERM restart. Add a helper that boots a config-less proxy owned by the test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
17f621e116
|
fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836)
* fix(vertex_ai): route Model Garden models through the generateContent adapter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vertex_ai): annotate generateContent route test parameters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): type litellm_params in get_vertex_ai_model_route * fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions Streamed Model Garden calls through the generateContent adapter now request stream usage where the deployment supports it, keep cached-token details, and look up the provider-relative cost key, so they bill like the non-stream call. The adapter forwards router metadata, passes upstream errors through with their status, countTokens falls back to local counting for non-Gemini routes, and the native Gemini config is limited to the Gemini routes. * fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing Streamed generateContent through the chat adapter now ends with the upstream usage, honors always_include_stream_usage false and the caller's stream_options, counts one failure per failed call, and routes Vertex deployments by their base_model like chat does * fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage * test(google_genai): expect the trailing usage chunk on bridged OpenAI streams The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone. * fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk * fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent * test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire * test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
08a24d391a
|
fix(count_tokens): count system and tools on Vertex Gemini, fall back to Mantle on Bedrock 404 (#46430)
* fix: align token counting with inference context Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve Vertex Gemini count message shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: forward Google countTokens tools in shapes every provider counter accepts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: add package marker for Vertex count-token unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): ignore deployment system_instruction when counting tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): forward system and tools on Google AI Studio, keep strict on Anthropic tools Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(count_tokens): ruff format gemini count handler Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): return Mapping from count request body builders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(count_tokens): truncate long Anthropic tool names on Vertex count, update stale strict test expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(count_tokens): simplify Gemini count context and tests Put the Vertex Gemini count branch back in the same shape as main and the partner model branch, keeping only the system and tools context, the provider error mapping and the gemini tokenizer default Share one error response helper between the Vertex and AI Studio callers, drop the unread kind fields and the extra validation of map_function output, and say Gemini instead of Vertex AI in the error that AI Studio users also see Tighten the tests by removing a stray parametrize on a helper, sharing the SDK count args, asserting the exact forwarded system and tools, and folding the Anthropic tool shape tests into one table * fix(count_tokens): reuse inference tool translation for Vertex Gemini count Vertex and AI Studio count now convert tools with the same functions inference uses, translate_anthropic_tools_to_openai and the Responses tool converter, before VertexGeminiConfig.map_function. Unnamed and Anthropic hosted tools now count the same declarations inference sends instead of being dropped Tools that cannot be converted, such as a non list tools value, a non object tool or a schema that is not a JSON object, now return a 400 before any request instead of being counted without them --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
18831b818c
|
fix: keep model capability flags out of provider requests (#46647)
* fix: keep model capability flags out of provider requests A supports_* flag or another model map field set on a deployment's litellm_params was forwarded to the provider as a request field, so Bedrock answered 400 Extra inputs are not permitted on chat traffic and on the background health check alike. The shared owned-kwarg predicate now treats every ProviderSpecificModelInfo field as litellm-owned. Builds on the health-check-only approach in #40952 by leilei3167. * fix: treat every supports_ model map flag as a capability field and warn when one sits in litellm_params * fix(router): skip declared litellm params in the capability flag warning and warn for runtime-added deployments * fix(router): strip model capability flags from the router's deployments instead of every request * fix(router): keep capability flags out of health probes and read them from the loaded catalog * perf(router): read the catalog's supports_ field set once per cost map load Cache the loaded catalog's field names when the cost map snapshot is taken so the capability-flag check stops walking every catalog entry per deployment, and add integration cells that drive the flagged and control deployments through chat, streaming, messages, responses, embeddings, health and model management against a vendor that rejects the flags |
||
|
Mateo Wang
|
56dad9077d
|
fix(decisions): answer 400 on a malformed provider_specific_header instead of dropping it (#46619)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
606ecf56de
|
fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals (#46633)
* fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove an OpenAI-format box answer reaches the caller as a box Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): give the box wire test a deployment that advertises supports_box_questions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Sushit Lal Pakrashy
|
958cba7016
|
fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results (#38284)
* fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results
* fix(vertex_ai): use object instead of Any in schema-ref traversal
* fix(vertex_ai): use immutable tuple traversal to satisfy type-discipline gate
* test(vertex_ai): add direct contract test for _contains_json_schema_ref
* fix(vertex_ai): type the $ref walk over JsonValue and drop the private-helper test
* fix(vertex_ai): wrap a tool result only when a $ref holds a non-empty string
Gemini rejects {"$ref": "<name>"} in function_response.response but accepts null, empty,
numeric, boolean, object, and list values, so those tool results stay structured
* perf(vertex_ai): skip the $ref walk when the tool result text cannot spell the key
* test(vertex_ai): cover a $ref on the tool result's root object
* docs(vertex_ai): cite where the Gemini $ref rule was observed
* fix(vertex_ai): keep a $ref under a list of lists structured
Gemini accepts a $ref reached through a list nested directly in a list,
so those tool results keep their structured form. Adds the audit's
integration cells for the tool result wire, chaos and edge shapes
* test(vertex_ai): teach the fake Gemini servers the list of lists rule
The audit's scripted upstreams now skip lists inside a list like Gemini
does, add list of lists cells on every endpoint, and give the 8 MiB
timing calls their own client timeout
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
452ee4dc08
|
test(caching): baseline breaker metrics from live state after gc (#46652)
* test(caching): baseline breaker metrics from live state after gc Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): collect garbage before the services logger gauge test reads live breakers --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Rick
|
71e508b65d
|
feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos (#43705)
* feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos MinimaxVideoConfig maps litellm's OpenAI Videos API onto MiniMax's V2 task API. Create, status, list, content and delete hit /v2/video_generation and /v2/query/video_generation. Remix is not exposed: MiniMax's /v2/video_regeneration only upscales a finished 768P task to 2K and ignores any prompt, so it raises instead of returning a video that silently drops the edit OpenAI params translate to MiniMax's contract: seconds becomes duration, size reduces via gcd to a valid ratio, and text-to-video requests default to ratio 16:9, resolution 768P and duration 5 when unset. input_reference becomes a first_frame content item, base64 encoded into a data URI for file inputs, and multimodal reference media passes through a content array in extra_body. Caller content is validated strictly as image, video or audio items, and the scanned prompt is always the one text item sent, so text can't reach MiniMax past the guardrails; anything else in content returns a 400. api_base shares MINIMAX_API_BASE with the chat config (a trailing /v1 or /v2 is stripped because the video API carries its own version). The list endpoint pages by number with no cursor or sort, so after and an ascending order are rejected with a 400 rather than ignored, and page_num passes through extra_query. Responses validate into frozen pydantic models at the boundary, video ids are provider-encoded so status and content route back to MiniMax, and create reports duration_seconds and video_resolution in usage for cost tracking End-to-end tests drive the real avideo_generation() entrypoint through an httpx MockTransport and assert on the encoded request body, which is what catches a mapper that drops input_reference before the request is built Both model_prices maps gain minimax/MiniMax-H3 and minimax/MiniMax-H3-Max, priced per output second by resolution tier from MiniMax's pay-as-you-go list (H3 $0.08 at 768P and $0.13 at 2K, H3-Max $0.05 at 480P and $0.08 at 768P). MiniMax also bills input images beyond a per-model free allowance (5 for H3, 2 for H3-Max), so each entry carries input_cost_per_image and provider_specific_entry.minimax_free_input_images, and the create response reports the full charge as provider_reported_cost_usd. Reference video input is rejected with a 400, because MiniMax bills its duration only after the create call that litellm bills Co-authored-by: AaronHowell <237895480@qq.com> * fix(minimax): fail closed on invalid video params and malformed task responses Addresses review on #43705: - A create, status or delete response without a task id (or a task) is a 502 instead of a queued object with an empty id - An unmapped task status is a 502 instead of defaulting to queued - seconds/duration must be a positive whole number and size a positive WIDTHxHEIGHT or W:H, otherwise 400; a 0x0 size no longer divides by zero and an unparseable duration no longer falls back to the billed default - characters and image are rejected with UnsupportedParamsError unless drop_params is set, instead of being dropped silently - Downloading a task that is not ready or failed is a 400 that keeps its message through the handler, instead of a ValueError surfaced as a retried 500 - The video download goes through safe_get / async_safe_get Co-authored-by: AaronHowell <237895480@qq.com> * test(minimax): cover the video download path and invalid-param branches The download now goes through safe_get, so the happy path and the finished-task-without-a-url branch are asserted, alongside the non-object parameters rejection and the remix response stub. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> * fix(minimax): return 400 for video remix and clear new lint gate counts * fix(minimax): send file-tuple images and keep input_reference next to content media * fix(minimax): answer 404 when an H3 video is downloaded before it finishes A download that lands while the task is still queued or running answered 400. That is a client error, which was the intent, but 404 is what the OpenAI videos API documents for content that is not ready, and it is the status the sibling DashScope and Volcengine video paths now answer, so a client polling all three reads one shape A failed task keeps its 400, since that is not a timing mistake the caller can retry away. The download test drives avideo_content and asserts the caller sees NotFoundError rather than an internal server error * fix(minimax): answer 400 for unsupported video params, early downloads and unknown ids A create param MiniMax cannot honor (seed, n, quality and the like) was dropped without a word. It now answers 400 unless drop_params is on, like other providers A download before the video is ready and a status call for an unknown task id now answer 400. The router cools a deployment down on a 404 and retries a 5xx, and neither is a deployment fault. DashScope on main answers 400 for the same case Input images past the free allowance are now billed through the shared video cost path, so a deployment with custom pricing bills them too. Status usage no longer carries MiniMax's token counts, list sets has_more to false since MiniMax pages by number, and the price map stops listing video as an input modality * fix(minimax): keep the polled video id, honor deployment drop settings, and match chat's key order Status re-encodes the id with the polled id's model when MiniMax omits it, drop_params and additional_drop_params never reach MiniMax and listed keys are dropped, MINIMAX_API_KEY wins over litellm.api_key, create sets created_at, and a content variant other than the video answers 400 * fix(minimax): list a task without a model under the routed group MiniMax leaves `model` off some listed tasks. The caller polls the ids a list hands back, and the proxy resolves a poll by the model in the id, so an id encoded with an empty model no longer routes back to the deployment that served the task. Fall back to the group the router was asked for. * fix(minimax): honor deployment drop keys inside parameters and bill images at the map rate when a deployment prices only seconds A field the deployment lists in additional_drop_params no longer reaches MiniMax when it arrives inside parameters, and a deployment that sets only its per-second price now bills reference images past the free allowance at the cost map's per-image rate instead of $0 * fix(minimax): keep deployment-dropped fields out of the body when they arrive through extra_body * fix(minimax): answer 400 for a video resolution with no price unless drop_params is on * fix(minimax): send the default resolution when a deployment drops resolution --------- Co-authored-by: AaronHowell <237895480@qq.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Happy <yesreply@happy.engineering> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Wang Yunzhuo
|
a2f792634e
|
fix: preserve explicit provider for async image, audio, and realtime routes (#36346)
* fix(images): preserve explicit provider for async generation * test(images): prove a bare model name reaches its explicit provider on the async path * fix(images): validate async provider hints instead of reading untyped kwargs Read custom_llm_provider and api_base in aimage_generation through a TypeAdapter over a small TypedDict, so the provider lookup gets typed values and the basedpyright gate stays within its merge-base ceiling * fix: answer bad provider hints with a 400 and keep the explicit provider for async speech and transcription * fix: keep the explicit provider for realtime and cover the audio bad-hint 400 * fix: keep the explicit provider on the realtime HTTP routes and read base_url in async audio * fix: stop async image generation from rejecting a base_url it never reads * fix(audio): skip base_url validation when api_base is set on async speech and transcription * test(integration): cover a declared provider with a bare model on image, audio, and realtime routes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c00706218b
|
fix(vertex_ai): return 400 with file line number for malformed batch JSONL row (#40097)
* fix(vertex_ai): return 400 with file line number for malformed batch JSONL row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(vertex_ai): keep yield outside the JSON decode guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): surface BaseLLMException raised while streaming a request body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(http_handler): only follow __cause__ when unwrapping streamed body errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): avoid recursive streamed error unwrapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows Keep main's error path for a directly raised provider error and only unwrap one found behind a transport error. Add the parser column to invalid JSON errors, and cover the real aiohttp chain in the _handle_error test * fix(vertex_ai): report every row conversion error as a 400 naming its line A message with no role, a non-string role, a bad temperature, or a litellm BadRequestError raised while converting a row still returned a 500, and the async upload lost the 400 for litellm's own exceptions. The stream now turns any error from one row's conversion into a 400 naming that line. _handle_error only unwraps an httpx.TransportError, which is what the aiohttp transport raises around a failing request body, so other callers keep their own status. A file that starts with a UTF-8 byte order mark now uploads, and provider messages start with 'Batch input file line N' like the proxy check. * fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line A chat row whose body is missing or not an object now raises a 400 naming its line instead of uploading a blank prompt, and a non-object body on line 1 no longer crashes the GCS object name lookup. Rows too deeply nested or too long to parse return a 400 from both the SDK and the proxy's own batch file check. JSON error columns now count leading whitespace. * fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line * test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos * fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact * test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Raushan kumar
|
f9d45f8a43
|
fix(llms): sanitize trailing slashes in provider get_complete_url (#42887)
* fix(llms): sanitize trailing slashes in provider get_complete_url * test(llms): move trailing-slash url tests into each provider's mapped unit file * fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI * fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders * fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests * fix(gigachat): strip a trailing slash from the pass-through api_base * test(integration): cover trailing-slash api_base across provider URL builders Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kolade Fajimi
|
35970c3eac
|
fix(vertex): report an unreadable credentials file as a read failure (#40173)
* fix(vertex): report an unreadable credentials file as a read failure Tell a path apart from inline JSON by shape, not os.path.exists(). Fixes #40166 * test(vertex): narrow pytest.raises to the expected message * fix(vertex): log the credentials path instead of returning it to the caller * refactor(vertex): log the credential source decision where it is made * fix(vertex): handle paths open() rejects with a bare ValueError * fix(vertex): redact the credentials path before it reaches the log * test(vertex): cover the brace fallback and non-text credentials file * test(vertex): exclude the unreachable exhaustiveness guard from coverage * refactor(vertex): read the failure's fields off the matched object CodeQL's py/uninitialized-local-variable does not follow keyword capture patterns in a match statement and reported `path` and `reason` as possibly unbound. Narrowing on the class and reading the attributes says the same thing without a suppression. * fix(vertex): accept a read-only mapping in extract_aws_params * fix(vertex): keep encoded credentials out of the log and cover the legacy route * fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust * fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file * fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message * test(vertex): integration cells for credentials file and inline value failures * test(vertex): drive the Rust messages cell through Vertex Claude --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ErenAta16
|
bc58a5857b
|
fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema (#34653)
* fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema `translate_anthropic_tools_to_openai` assigned the caller's `input_schema` straight into `function_chunk["parameters"]`, then merged the tool's remaining top-level keys into that same dict. Because the assignment aliased rather than copied, keys such as `display_width_px` on a computer tool were written back into the caller's schema. That matters because callers reuse the tool list across passes: the pass-through guardrail translates the tools, and the request is then translated again for the real call. The first pass pollutes the source and the second forwards a schema carrying non-schema keys. Shallow-copying fixes it. `translate_anthropic_output_format_to_openai` in the same file already copies its schema for this reason, so this brings the two into line. Rebased onto current litellm_internal_staging; the original branch had gone to conflict on surrounding drift while the line it fixes is unchanged. * fix(anthropic/adapter): drop the inline comment and test the strict tool shape that Anthropic rejects * fix(anthropic/adapter): deep copy input_schema so downstream schema edits never reach the caller's tool A shallow copy still shared nested properties with the caller, and the Gemini, vLLM and watsonx bridges strip nested additionalProperties in place, so a fallback to Claude sent a strict tool Anthropic rejects. A deep copy also keeps a null or malformed input_schema as sent, as before this PR * fix(anthropic/adapter): deep copy the incoming tools so downstream edits never reach the caller's tools Copying only input_schema left the extra keys merged into parameters, such as input_examples, and OpenAI-format function tools shared with the caller, so Gemini's strict strip still edited the caller's tool before a fallback to Claude * fix(anthropic/adapter): keep translating tools that deepcopy cannot copy A tool passed as a frozen mapping, or with a schema nested past Python's recursion limit, made copy.deepcopy raise, so the request failed before it reached the provider. Copy each tool on its own and keep the original object when the copy fails, which is how these tools were handled before * fix(anthropic/adapter): copy tools through JSON so the Responses bridge and frozen tools keep the caller's schema * test(anthropic/adapter): parse stub bodies with json and cover the tool copy fallback * test(anthropic/adapter): close the connection after the malformed tools request --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Oliver Jensen
|
84e8204d7f
|
fix(proxy): scope /spend/tags aggregates to the caller's own and permitted-team logs (#46194)
* fix(proxy): scope /spend/tags aggregates to the caller's own and permitted-team logs Non-admin callers previously received the proxy-wide per-tag spend aggregation. The endpoint now resolves the same read scope as /spend/logs/ui and /spend/logs/session/ui, and the query honors the start_date/end_date parameters it previously accepted but ignored. * fix(proxy): use Annotated auth dependency on /spend/tags and sync dashboard API types * fix(proxy): bind /spend/tags dates as UTC datetimes, not session-timezone strings Raw YYYY-MM-DD strings cast to timestamptz are read in the database session timezone, shifting the day window by that offset. Parse them to UTC-aware datetimes first, like every sibling spend query in this file, and reject malformed dates with a 400 before they reach the database. |
||
|
Kolade Fajimi
|
ea777eb21b
|
fix(auth): return a generic 401 for an unknown key (#39787)
* fix(auth): return a generic 401 for an unknown key The wrong-key 401 shipped the internal verification table name and the submitted key's full SHA-256 hash to any unauthenticated caller. Both now go to the proxy's own log and the response carries only "Invalid API key provided." The Admin UI detected a dead session by string-matching that message, so it now reads the structured error type instead. * Update ui/litellm-dashboard/src/lib/http/client.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(auth): log the missing key hash where every resolve path passes Review catch: JWT key-mapping misses resolve through IdentityStore without touching the main key path, so logging the hash there left those callers with nothing but the generic 401. Log it at the store's raise site instead, which every resolve path shares, and drop the now-duplicate log. * test(auth): inject a key-less client instead of patching the DB fetch The store test tripped TQ008 by patching _fetch_key_object_from_db_with_reconnect. Hand IdentityStore a client whose get_data returns no row instead, so the miss is driven through the real lookup rather than around it. * fix(auth): log the unknown key's hash once on the auth failure line The per-lookup warnings in get_key_object and IdentityStore logged whatever value they were handed, which on the failure-logging path is the raw key a caller sent (a provider key included), let a newline in ?key= forge log lines, and fired on lookups that expect a miss. The hash now rides KeyNotFoundError onto the single existing auth failure log line next to the requester IP, with control characters escaped. Also drops the new comments and updates skipped tests that quoted the old message. * test(auth): add the blank lines ruff format expects before _InFlightCountingPrisma * test(integration): expect the generic wrong-key 401 in the management lifecycle tests * fix(auth): keep the operator wording on the unknown-key log line and drop the redundant 401 rewrite * test(auth): cover the generic unknown-key 401 across routes, proxies, and a worker kill * test(auth): cover the realtime upgrade refusing an unknown key without echoing it --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
bf56719f09
|
fix: cap cold MCP schema workers at usable CPUs and deflake wall-clock unit tests (#45791)
* test: inject clocks and wait on conditions in three wall-clock-flaky unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: keep generic API logger tests on 3.10 and cover automatic sending Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: pin router usage minute in shared rate-limit header test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: stabilize three wall-clock-sensitive unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): cap cold schema validation workers at the CPUs the proxy can use Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: inject the cache clock and wait on lock contention instead of 1s deadlines Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): wait on listing events instead of 1s deadlines in the discovery drain test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> |
||
|
Hayden Moulds
|
de0ed1557e
|
fix(proxy): resolve model aliases to their target deployment for listing metadata (#41041)
* fix(proxy): resolve model aliases to their target deployment for listing metadata A model alias (a key's `aliases`, a team's `model_aliases`, or the router's `model_group_alias`) is rewritten at request time and owns no deployment row, so `get_model_listing_info` returns None for it and the alias name is not a cost-map key either. Both metadata candidate sources come up empty and the listing entry carries no `mode`, `max_input_tokens`, or `max_output_tokens`, while the deployment it routes to reports them on the same response. Resolve a listed alias to the deployment it routes to for the metadata lookup, keeping the alias as the response id. Maps are applied in order, each against the result of the last, mirroring how `litellm_pre_call_utils` applies the team map and then the key map, so a chained alias reports the deployment the caller actually reaches rather than an intermediate one. An alias shadowing a same-named deployment follows the alias, because the request rewrite does too. Fixes the model_aliases half of #40553. * fix(proxy): look listed aliases up through the shared alias chain Drop the separate alias resolver and retarget a listed alias in alias_listing_entries and alias_target instead, so /v1/models and /v1/models/{id} report the model /chat/completions rewrites it to, and only when the caller can list that model * fix(proxy): keep deployed listed models' own rows and resolve team alias targets at list and retrieval * fix(proxy): resolve router_settings aliases, keep alias labels, and skip paused targets in model listing * fix(proxy): leave an alias to an undiscoverable model bare in model listing A team alias whose target has discoverable: false showed the hidden model's mode and limits on GET /v1/models and retrieved it by alias. Undiscoverable groups now count as hidden for alias resolution, as on main, while admins still see them. * fix(proxy): leave an alias to a model a listing callback hides bare in model listing * fix(proxy): keep aliases bare for models a listing callback never judged or another team owns --------- Co-authored-by: haydster7 <haydster7@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
3cc9e7ef8e
|
perf(proxy): cache the public provider and agent field endpoints (#46617)
/public/providers/fields re-read and re-parsed the bundled 85K provider_create_fields.json on every request, and /public/agents/fields did the same for that file plus agent_create_fields.json. Both are unauthenticated public endpoints, and both did the work inside an `async def`, so the read and the parse ran on the event loop and serialized against every other in-flight request. Measured against the bundled files, that is roughly 0.40 ms of blocking work per request for the provider endpoint and 0.45 ms for the agent endpoint Both now memoize with `lru_cache`, which is what `/public/endpoints` in the same module already does through its `_cached_endpoints` global, so each file is read once per process rather than once per request The agent endpoint mutated the dicts it had just loaded, appending inherited provider credential fields and popping `inherit_credentials_from_provider`. That is only safe while the data is re-read every time, so caching it meant making the merge pure: `_agent_with_inherited_credentials` builds a new mapping per agent, and the cached value is the merged result rather than the raw file contents. Responses are byte identical before and after, checked on both endpoints against a live proxy The bundled records are typed as Mapping[str, object] and narrowed with isinstance at the point of use, rather than typing.Any, which ruff-strict bans Side effects: 3 of the repo's ASYNC230 violations go away, and basedpyright errors in the file drop from 63 to 52 Rebased onto main after /public/providers/fields started validating the bundled JSON through a pydantic adapter on every request. That validation now runs once and the validated records are cached; the agent endpoint keeps merging the raw provider records, so both responses stay byte identical to main. Co-authored-by: Alok Sharma <alok44170@gmail.com> |
||
|
Chloe Lu
|
897bfea38d
|
fix(anthropic): keep signed empty thinking blocks on /v1/messages requests (#41362)
The request-side sanitizer on /v1/messages dropped every thinking block whose text is empty, even with a valid signature. Claude Opus 4.8 / Fable 5 / Fable 5.1 return exactly that shape by default (thinking.display omitted): the signature is the preserved reasoning. Replaying the assistant turn as returned therefore lost the model's reasoning on every multi-turn request, silently (the provider answers 200 without it), and left preserved-thinking enforcement (thinking.block_binding) nothing to bind. Anthropic accepts an empty thinking block that carries a signature and binds it to its prefix; only an empty block with no signature (the LIT-6357 case, a turn a non-Anthropic model served through the bridge) is rejected with "each thinking block must contain thinking". Use the signature-aware predicate is_empty_unsigned_thinking_block, which #38809 already applied on the emit side, for the request side too. The unsigned case is still dropped. Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
TheMentalCoding
|
ca04c4909e
|
feat(ui): filter active and paused model deployments (#43710)
* feat(ui): status filter + sort and provider-aware search on Models & Endpoints
The models table mixes active and paused deployments with no way to tell
them apart beyond the pause toggle, and the drawer only filters by public
model name and access group.
Backend (/v2/model/info):
- new optional `blocked` query param to filter by routing status
- new `blocked` sort field (asc = active first); the existing `status`
field keeps sorting by config-vs-DB source
- `search` now also matches litellm_params.model, so typing a provider
or upstream model id (e.g. "nvidia_nim", "openai/gpt-4") finds
deployments whose public name doesn't mention them
Frontend (Models & Endpoints):
- visible Status column with Active/Paused badges, sortable
(active first on asc)
- Status filter (All/Active/Paused) in the Filters drawer, wired to the
new server param through URL state
The routing-status filter ignores non-bool sentinels the same way
exclude_auto_routers does, so direct calls bypassing FastAPI keep their
no-filter behavior.
* fix(ui): inverted routing-status filter mapping and compact Status column
Selecting "Active" in the Filters drawer sent blocked=true to the server,
showing the paused deployments instead of the active ones (and vice
versa). The mapping now lives in routingStatusToBlocked() with unit
tests, and the Status column is narrower so the row actions stay in
view.
* fix(ui): pin the Actions column to the right on the models table
With the new Status column the table overflows on narrower screens and
the pause/resume switches (last column) scroll out of view. Pinning the
Actions column keeps them visible; uses the existing DataTable
meta.pinned support (same pattern as MemberTable).
* refactor: readable db search where-condition and ruff-format clean
* fix: apply routing-status filter inside the search flow and regen API types
Greptile flagged (P1): with search + a status filter combined, the
bounded DB fetch ran before status filtering, so matches of the other
status could consume the page budget and leave short pages / stale
totals.
- extract _matches_routing_status() (sentinel-safe) and apply it to the
router-side matches inside _apply_search_filter_to_models
- push the status condition into the DB where clause of
_fetch_db_models_for_search, so rows of the other status never count
against the bounded fetch
- endpoint passes blocked through to the search flow; tests cover the
combined search+status case and the generated where clause
- regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts for the new
query param
- update useModels.test.ts expectations for the new trailing argument
* fix(proxy): keep the flat DB where shape for single-condition searches
The upstream CI test
test_apply_search_filter_honours_exact_model_name_in_db_query pins the
Prisma where shape: exact model_name stays a top-level key, and plain
substring search keeps {model_name: {contains, mode}} at the top level.
Only genuinely multi-condition queries (status filter and/or router-id
exclusion) now wrap in AND.
* fix(proxy): avoid reassigning a Final variable in the db where builder
basedpyright's reportGeneralTypeIssues budget counts the reassignment of
the Final-annotated db_where_condition; express it as a single
conditional expression instead.
* chore(deps): bump pypdf to 6.19.0 to clear osv-scan
pypdf 6.16.2 has 8 High CVEs (GHSA-5jq2-8x83-x246 and aliases) fixed in
6.17.0-6.19.0; bump past all of them
* Revert "chore(deps): bump pypdf to 6.19.0 to clear osv-scan"
This reverts commit 0e2528123db7ab23304ae20f6a4c0961d0e2d073.
* fix(proxy): keep LIT002 budget flat in the model search where-builder
Extract the dynamic Prisma where-condition fragments into module-level
helpers so every dict starts on a short line, split the litellm_params
fallback lookup into a Final local, and annotate suppressions on the
construct start lines the checker attributes
* chore(tests): repoint unbounded-IN baseline at the extracted where-builder helper
The bounded-router-id filter moved from _fetch_db_models_for_search into
_not_in_router_ids, so the old baseline key is stale and the new helper
site is the finding to record
* fix(proxy): avoid searching encrypted deployment model names
* refactor(ui): keep model routing status as a filter only
* docs(api): regenerate model info sorting description
* fix(api): remove remaining paused status sort references
---------
Co-authored-by: thementalcoding <312162298+thementalcoding@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
yuneng-jiang
|
aa694f316d
|
fix(proxy): keep JSON nulls as null in audit log values (#46577)
LiteLLM_AuditLogs masked before_value and updated_values with
SensitiveDataMasker.mask_dict, which stringifies every non-native leaf, so a
null in the change was stored as the string "None" (or "" under a key-named
field). Walk audit values with mask_credentials_in_payload instead, injecting
the existing {"key"} masker so the masked format and scope stay the same.
|
||
|
Mihidum
|
b025681bef
|
fix(proxy): gzip responses over 16 KiB off the event loop (#46362)
* fix(proxy): gzip responses over 16 KiB off the event loop GZipBufferedResponseMiddleware compressed every body under 1 MiB inline. gzip.compress holds the event loop for its whole run, and its cost follows how well the body compresses: an 824 KB base64 embeddings response (100 inputs, the OpenAI SDK default encoding) takes about 165 ms, so every other request on the worker waits that long, and with several in flight the compressions run back to back. Lower OFF_LOOP_SIZE_BYTES to 16 KiB so only bodies that compress in well under a millisecond stay on the loop. zlib releases the GIL, so the loop keeps serving while a worker thread compresses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): pin the gzip off-loop boundary at exactly OFF_LOOP_SIZE_BYTES A body of exactly OFF_LOOP_SIZE_BYTES compresses in a worker thread, as the `<` in _compress implies. Suggested in review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): pin an embeddings-sized gzip body off the event loop and drop the source comment The off-loop test was relative to OFF_LOOP_SIZE_BYTES, so it still passed with the old 1 MiB line. An 824 KB body, the size of the 100-input base64 embeddings response in the issue, now has to compress in a worker thread. The recorder no longer mutates a list, and the threshold rationale stays in the commit history instead of a source comment. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
3fec5075fa
|
feat(typesafe): support Jev 1.14 images, box questions, and long-context headers (#46596)
* feat(typesafe): support Jev 1.14 images, box questions, and long-context headers * fix(cost): drop the typesafe/jev-1.14.0 entry main already prices * fix(typesafe): send images only with the images header and box questions only to models that answer them TypeSafe takes assets and input_image parts only when the request carries TypeSafe-Danger-Zone-Images: true, so without it they follow drop_params like any other provider that cannot take images. Box support now comes from a supports_box_questions cost-map flag set on jev-1.14.0 and jev-preview, so jev-1.13.0 and jev-latest refuse or drop a box question before the call. A malformed provider_specific_header answers 400 instead of 500. * fix(typesafe): allow null box confidence and validate asset references and base64 * fix(typesafe): gate box questions on the images header, drop unreadable assets under drop_params, forward opt-in headers on the pass-through --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
efd8274315
|
feat(xai): bill grok voice realtime sessions (#46600)
* feat(xai): bill grok voice realtime sessions * chore: revert unrelated formatting churn in test_cost_calculator.py * refactor(xai): tidy realtime usage meter and tests * fix(xai): read push-to-talk mode from client session.update only * fix(xai): clamp decoded audio bytes and lock push-to-talk before content --------- Co-authored-by: kerry <kerry@berri.ai> |
||
|
Kerry Lu
|
0d741a9816
|
fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them (#46611)
* fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them * fix: satisfy the type discipline and basedpyright gates for the thinking block index * chore: sync the OpenAPI snapshot and schema.d.ts with the thinking block index field --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |