* feat(nvidia_riva): add riva_offline mode for offline-only ASR models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(nvidia_riva): assert offline vs streaming via fake service call tracking
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts after user soft_budget docstring removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(nvidia_riva): parse riva_offline strings, map oversize offline audio to 413
A multipart riva_offline=false no longer turns offline mode on, a malformed value drops under drop_params and answers 400 without it, and a Riva RESOURCE_EXHAUSTED for a message over the server's max now returns a non-retryable 413 instead of a retried 429
* fix(nvidia_riva): apply the request timeout to offline recognition and keep riva_offline on an api_base override
Offline Recognize now runs as a gRPC future waited on with the request
timeout, so a stalled Riva server returns a 408 and the call is cancelled.
riva_offline is a mode flag, not a credential, so an api_base override no
longer clears it. The 413 hint for audio over the gRPC message limit is
reworded, and the new tests carry type annotations.
* fix(nvidia_riva): wait the request timeout in seconds whatever its form and cap the riva client to versions that import
A timeout from os.environ, a multipart form field, or an httpx.Timeout reached the gRPC future as a string or object,
so result() raised TypeError and every offline call came back as a false 408. The handler now turns the timeout into
float seconds with CompletionTimeout.as_limit, taking the read limit of an httpx.Timeout, for both the offline future
and the streaming deadline
The stt-nvidia-riva extra caps nvidia-riva-client below 2.17, since every later release either pins an older grpcio
or ships stubs that need a newer one than the grpcio==1.78.0 pin, so a fresh install failed at import
* fix(nvidia_riva): parse the timeout only where the call uses it and drop the riva client cap
A malformed timeout no longer fails streaming calls on clients that take no
timeout. On offline calls it is a 400, or the default under drop_params, and
an httpx.Timeout with no read limit waits the default instead of forever.
The nvidia-riva-client cap made installs next to a newer client unresolvable,
so the extra goes back to main's specifier.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mcp): upgrade to MCP SDK 2.3.0 and raise the per-event SSE size cap
MCP SDK 2.x reads Streamable HTTP responses through httpx2's SSE parser,
which rejects any single event over 1 MiB, so a large tools/call result
failed with 'SSE stream ended without a response'. mcp 2.3.0 exposes
max_sse_event_size, and MCPClient now passes a 64 MiB default,
configurable with LITELLM_MCP_MAX_SSE_EVENT_SIZE_BYTES
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): fall back for invalid SSE event limits
* test(mcp): verify configured SSE limits and recovery
* chore(mcp): remove redundant dependency comment
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(extraction): add Clairic extraction types and Cloudflare provider config
* feat(extraction): add the /v1/extraction route, router wiring, and Clairic billing
* fix(extraction): refuse or drop stream, pass unknown keys, run guardrails, keep the requested model name
Clairic does not stream, so stream=true answers 400 and is dropped and billed under drop_params. Keys LiteLLM does not know pass through to Cloudflare. Guardrails scan state, field instructions and text answers. Frozen decision and extraction responses now carry the requested model name. List fields must name their item type. /health/test_connection and the Add Model page accept the extraction mode.
* fix(extraction): keep upstream model names on decisions, scan images, keep typed values, refuse OpenAI params
Scope the frozen-response model stamp to extraction so /v1/decisions and /v1/systemone return the upstream name as on main. Pass request images to guardrails as data URLs and drop answer spans when a response guardrail rewrites state_text. Field spec values keep their JSON type, so a wrong type answers 400 instead of being coerced. OpenAI generation params answer 400 unless drop_params. The Model Hub shows a /v1/extraction snippet for extraction models.
* fix(extraction): send only Clairic's request fields, count extraction tokens, keep typed answers typed, drop spans a request guardrail made stale, and honor --max_tokens
* fix(extraction): scan number answers in response guardrails, rewrite guardrail fields in linear time, and leave --max_tokens as on main
* fix(extraction): scan float answers as whole numbers so fraction digits never trip a response guardrail
* fix(extraction): round only mean and median for response guardrails so fractional values keep their decimal text
* fix(extraction): count state and fields for budget reservations and scan estimate quantiles in response guardrails
* fix(extraction): send each repeated text to guardrails once and keep the context-window budget reservation
* test(extraction): type request_data in the guardrail test helpers
* fix(logging): report pay-as-you-go rates for spilled over PTU requests without base_model
* fix(logging): keep the provider response model so spilled over PTU payloads price the served model
* fix(logging): keep a provider_response_model the response already carries
* fix(logging): read the carried provider_response_model through a typed adapter
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bill every earlier timed-out retry attempt at its own deployment's model rate
* fix(router): read the retry record's base_model through the typed metadata helper
* fix(router): keep the four-record retry breadcrumb cap
* fix(router): carry a deployment's litellm_params base_model on its retry record
* chore(router): place the cast-ok reason on the cast line and list the base_model helper as covered through log_retry
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): price cost-based routing through model_info.base_model
* fix(router): keep cost-based routing pricing helpers fully typed
* fix(router): drop typing.cast from cost-based routing pricing helpers
* fix(router): read the cost map through a typed accessor in cost-based routing
* fix(router): avoid a Final rebind inside the cost-based routing loop
* refactor(router): share deployment_pricing_models across spend, listing and cost-based routing
* refactor(router): keep the /v1/models listing key inline, share pricing models for spend and routing
* refactor(router): inline the deployment pricing-model wrapper in cost-based routing
* refactor(router): read deployment pricing models through one shared helper
deployment_pricing_models(deployment) now owns the model_info.base_model / litellm_params.base_model /
litellm_params.model unpacking, used by cost-based routing and the /v1/models listing. pricing_models(base_model, model)
stays for the failure-row spend estimate, which already holds the two names
* refactor(router): share the str-keyed mapping validator with cost-based routing
* ci: re-run after flaky test_aiohttp_transport read timeout in llms-providers
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): fire one CustomLogger hook per async call outside the async flag list
aspeech, atranscription, aimage_generation, aimage_edit, aocr and asearch are
missing from is_sync_litellm_request's flag list, so the sync success and
failure handlers called a CustomLogger's sync hook on top of the async hook the
async handler had already called for the same request.
The sync handlers now skip a CustomLogger's log_success_event and
log_failure_event when that run is paired with an async handler run
(for_async_request) and the async handler gives the logger its own async hook.
The async client wrapper, dispatch_failure_handlers, the proxy-only failure
dispatch and the native bridge's sync failure step for an async call mark their
sync failure run as paired.
Resolves LIT-9486
* fix(logging): keep the sync CustomLogger hook when no async success runs
The async client wrapper skips async success logging for calls carrying a
fallbacks kwarg, and the native bridge skips it for internal and fallbacks
calls, yet the paired sync success run still dropped a both-hooks logger's
sync hook, so that logger got no success event at all.
The sync success run now suppresses the sync hook only when the async success
handler was actually scheduled for the call: the client wrapper passes
async_handler_runs=not is_completion_with_fallbacks, and the native host
records per call whether it enqueued or deferred the async success.
Tests that pinned the doubled failure hook on native aocr, messages and
responses calls and on proxy-rejected requests now expect the async hook
alone. The integration test adds /v1/messages and /v1/responses cells and
runs the key-auth rejection on every route.
* fix(logging): type dispatch_success_handlers kwargs
The new async_handler_runs keyword on success_handler made the untyped
**kwargs that dispatch_success_handlers forwards an unknown argument for it,
which grew the reportUnknownArgumentType count past its merge-base ceiling.
Typing the forwarded kwargs as Any, as _success_handler_body already does,
removes those diagnostics without changing what is forwarded.
* test(integration): cover CustomLogger hook counts across modes, chaos and router strategies
Moves the hook count rig into tests/integration/_support/hook_counts.py and
extends the cells it drives:
- streamed chat, Messages and Responses requests, success and failure
- loggers the async handler never reaches (sync-only overrides, sync event
lists only) keep one sync hook per request on every surface outside the
async flag list, key-auth rejections included
- a concurrent burst while the logger's destination stalls and then fails,
and a burst while a worker is killed, never give a request two hooks
- least-busy keeps counting a request in flight while async speech requests
finish around it, and usage-based routing v2 counts an async image
request's tokens once
* test(integration): Datadog and LangSmith deliver one record per async request
Points the built-in Datadog, LangSmith and OpenTelemetry loggers at a local
sink and checks every async surface outside the flag list: one Datadog log
entry and one LangSmith run per request (two on the merge base), and never two
OpenTelemetry spans, which its own emit-once guard already ensured.
* fix(logging): give a both-hooks logger its sync hook when the async run never reaches it
The sync hook was skipped whenever the logger sat in the async callback list, so a
logger the async run never reached got no hook at all: a sibling stalled past the
logging worker timeout, a batch poll that skips default logging, or an async
failure run cut off. The sync run now holds the hook and the async run hands it
back unless it reached the logger
* fix(logging): run the held sync success hooks when the proxy drops a blocked response's success
A post-call guardrail block drops the parked async success, which left the sync
success hook held for it unrun. Base gave a both-hooks logger that hook on these
call types, so the drop now ends the pairing and the held hook runs as before
* fix(logging): run held sync hooks only after every sync run of the request has closed
A held hook released by the async run could run on another thread while the sync
run was still walking its other callbacks. The pairing now waits until the async
run has ended and every sync run has closed, and the last sync run to close runs
the held hooks itself
* fix(logging): end the async run when its coroutine is dropped before it starts
A logging queue that drops the async success coroutine, or a deferred success
that fails to schedule, left the sync hooks held for it unrun. The enqueued async
run now ends when its coroutine is discarded, and the proxy ends it when firing
the deferred success raises
* fix(logging): leave a still-queued async run to the exit flush
The discard finalizer ran at interpreter exit before the logging worker's exit
flush, so a queued request could get its held sync hook and then its async hook
* test: keep the router and file-stream tests in step with main
Main gives each owned proxy a private Redis by default, so the router cells
share the worker's Redis to read its counters, and the file-stream stub takes
the hook pairing the stream now passes
* fix(logging): run held sync hooks when a during_call guardrail raises or the exit flush gives up
A during_call guardrail that raised after the LLM call parked its deferred
success skipped the proxy's flush, and async runs the exit flush never started
were never ended, so a both-hooks logger lost its sync hook in both cases
* fix(logging): skip the sync failure hook when the async failure run logs to the logger
An async call outside is_sync_litellm_request's flag list ran both failure
handlers, so a CustomLogger with both failure hooks got both. The sync run
of an async request now skips log_failure_event for a logger that overrides
async_log_failure_event and is registered for async failures. The four sync
runs of async failures (dispatch_failure_handlers, the @client async wrapper,
the proxy-only failure thread and the Rust host) pass for_async_request=True.
Drops the HookPairing fallback. Successes are left to #47192, which already
gives such a logger only async_log_success_event.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): price a 499 spend row only for the proxy's own disconnect, and never after the call already completed
* fix(proxy): gate the zero-cost 499 row on the success log being enqueued, not merely claimed
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): detect Python-only features in the catalog instead of Rust declines
* refactor(rust-bridge): split and document route-specific detectors
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): decide native routing in the catalog, drop the bypass predicates
* refactor(rust-bridge): simplify catalog compatibility checks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): scope the search free-tier counter to the request's provider
* fix(proxy): read the provider with get so partial spend payloads still get the free tier
* test(proxy): give the grounded spend payload fixture its provider
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): apply the deployment guardrailConfig on the /bedrock pass-through routes
The router-model pass-through dropped a Bedrock deployment's guardrailConfig, so
converse, converse-stream, invoke and invoke-with-response-stream reached Bedrock
unguarded while /v1/chat/completions on the same deployment was blocked. The
signer now sends it the way Bedrock reads it, in the body for Converse and as the
X-Amzn-Bedrock-Guardrail* headers for Invoke, replacing any guardrail the caller sent
* test(bedrock): reject version-less guardrails in the scripted Bedrock and cover the 400
The mantle and fallback fixtures named a guardrail with no guardrailVersion, which real Bedrock refuses, yet the scripted Bedrock answered them as guarded successes. The fake now answers a version-less guardrail with a 400 ValidationException, the fixtures carry versions, and a new case checks a version-less deployment guardrail reaches Bedrock and the caller gets its rejection on all four actions
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): strip caller base_model from spend pricing and reject lone surrogates in multipart metadata
* fix(proxy): type the parsed multipart metadata so basedpyright does not grow reportArgumentType
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep tool names over 64 chars when a guardrail rewrites a /v1/messages request
The guardrail translation shortens long tool names for the OpenAI-shaped guardrail input
and wrote the shortened names back into tools and tool_use blocks, while tool_reference
blocks and tool_choice kept the original names, so Bedrock rejected the request
* fix(anthropic): restore declared tool names only from the request's own tool list
A history tool_use name whose shortened form matched a declared short tool renamed that
declared tool, so a forced tool_choice no longer matched and a tool allowlist saw a
different name than the one forwarded
* fix(anthropic): keep a tool_use name the caller already used as written
A history tool_use named exactly like the shortened form of another long name was
rewritten to that long name; such an ambiguous alias is now left alone
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(cost): price deployments from their own entry and add model_info.pricing_provider
A deployment whose model_info.id equals another provider's cost-map key
billed $0: register_model merged its prices onto that catalog row, and the
cost calculator then resolved the id by name through get_model_info, whose
provider check rejected the row. completion_cost now hands the deployment's
entry straight to generic_cost_per_token on the openai and generic branches
instead of re-resolving it by name.
model_info.pricing_provider names the provider whose cost-map rows price a
deployment when it differs from custom_llm_provider (an openai-compatible
deployment pointed at a third-party host). It scopes the catalog row the
deployment's entry is merged onto at registration and the provider the
request is priced under at bill time. Unset keeps custom_llm_provider.
* fix(logging): type the pricing_provider metadata read and sync dashboard API types
* fix(logging): read pricing_provider without loop-scoped Final
* refactor(router): inline the deployment pricing provider pick
* fix(cost): keep only model_info.pricing_provider, drop the deployment-entry cost path
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): redact Responses instructions and callback output
* fix(logging): avoid copying Responses output twice
* fix(logging): return redacted Responses output without mutation
* fix(logging): type Responses redaction payloads
* fix(logging): redact streamed Responses instructions
* fix(logging): redact request-side Responses instructions in logged params and spend log bodies
* fix(redaction): keep per-callback opt-out off shared instructions and skip redacted replays
A callback that sets message_logging=False no longer strips instructions
from sibling callbacks, the spend log, or the client's response. Dict
logging surfaces keep the reasoning config, and previous_response_id no
longer sends the redacted placeholder upstream as the system prompt.
* fix(redaction): redact instructions in an opted-out callback's own copy
A callback with message_logging=False now gets instructions redacted in its kwargs copy and response object, while sibling callbacks and the spend log keep them
* fix(redaction): give a message_logging false callback its redacted copy on sync calls
The sync log_success_event and log_stream_event branch now hands an opted-out callback the same copy as the async path.
* fix(redaction): redact a message_logging false callback's sync copy and keep failure estimates whole
Give an opted-out callback a copy with messages, input, prompt, the sync stream's
complete_response and its response object redacted, and stop the proxy's failed
request estimate from counting the redaction marker in place of real instructions
* test(redaction): wait on callback events instead of sleep polls
The new redaction tests now wait for each callback to log through an event it sets, so they no longer poll on a timer
* test(redaction): expect redacted follow-ups to replay no instructions
This PR stops bridged previous_response_id follow-ups from reusing
instructions the spend log stored redacted, so the integration test from
#45923 that asserted the replay now asserts that neither the original
instructions nor the redaction sentinel reach the provider
* test(proxy): type the redacted-instructions failure estimate test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(harness): set PWD to the working directory of local sandbox commands
LocalSandbox started CLIs in the sandbox workdir but passed the parent's PWD through. OpenCode reads PWD as its project directory, so a litellm.agent() turn worked in the Python process cwd instead of the sandbox.
* fix(harness): re-home resumed OpenCode sessions into the sandbox folder
OpenCode files each session under the folder it ran in and only streams a resumed session's events from that folder, so a session saved before PWD pointed at the sandbox, or resumed in another sandbox, hung. Resume now runs opencode export and import in the sandbox first. Also types the CLI handler's config so its member accesses are known.
* fix(harness): fail an OpenCode resume that cannot move its session, with a timeout
The export and import step now has a 30 s bound, and a failure or timeout raises HarnessError instead of leaving the resumed turn waiting forever. A restart of a session the handler already ran skips the step.
* fix(harness): resume an OpenCode session elsewhere as a copy and fail the turn when it cannot
Importing the exported session moved it, so the original stopped working
where it was saved. Resume now imports a copy under fresh ids, filed under
the sandbox folder, and continues the copy. A failed or timed-out export or
import ends the next turn with runtime_error, as base does for a missing
session, instead of raising from agent_resume.
* fix(harness): retry a failed OpenCode resume on the next turn and keep resumed copies stable
Resume now only records the saved id, and the first turn runs the OpenCode export and import under the caller's timeout. A failed export fails that turn and the next turn tries again. The local sandbox kills a command when its run is cancelled, and the copy names the sandbox folder so OpenCode 1.15.5 and older keep it there.
* test(harness): keep subprocess tests out of tests/unit
The PWD regression test moves to tests/integration/sdk with an isolated child interpreter, and the cancel test checks collect_output with an in-process process instead of a real one. collect_output takes the sandbox Process protocol.
* fix(harness): rewrite only id fields when copying an OpenCode session for resume
* chore: update Next.js build artifacts (2026-10-20 18:15 UTC, node v24.19.0)
* test(security): cover the generated dashboard bundle in the example master key check
* test(security): keep a clean-repo success case for the example master key check
* fix(bedrock): send batch jobs to a configured control-plane endpoint
Bedrock batch create, retrieve and cancel always built the regional
bedrock.{region} host, so a VPC endpoint or compatible gateway could not
be used. Add aws_bedrock_control_plane_endpoint as a deployment setting
read by the signed HTTP path and the boto3 path, carry it through the
proxy credential filter, and refuse it in request bodies like the other
endpoint overrides. api_base stays out of it because on a Bedrock
deployment it points at the runtime host.
Fixes#45056
* fix(bedrock): read the control-plane endpoint from typed kwargs in job status
The named parameter added one unknown-argument error in litellm/batches/main.py,
which unpacks untyped kwargs into this function
* docs(bedrock): explain batch control-plane endpoint configuration
* fix(bedrock): keep the batch control-plane endpoint out of chat request bodies
A deployment that sets aws_bedrock_control_plane_endpoint also serves chat,
and the unregistered key was swept into the Bedrock request body, so every
chat call on it failed with "Extra inputs are not permitted". Register it
with the other Bedrock batch connection fields so it stays a LiteLLM-owned
kwarg on every route while batches still read it from litellm_params.
* test(bedrock): verify endpoint signatures without internal mocks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): bill partial Responses API streams when the client disconnects
Streamed /v1/responses requests and the default-route OpenAI and Azure /v1/messages requests behind them logged a 499 with zero spend when the client went away mid-stream, while /v1/chat/completions already billed the chunks it had sent. The Responses API iterator now keeps the response.created payload and can hand the disconnect cleanup a response.incomplete event with counted usage and the served service_tier, and the messages-route byte stream and router wrappers delegate to it.
* fix(proxy): satisfy lint gates for partial Responses API disconnect billing
Make the disconnect helper public so the proxy, router and messages adapter can import it without reportPrivateUsage, type the SSE byte stream as an AsyncGenerator so aclose is known, read litellm_call_id off the logging object, and keep the new tests on Final locals with the recorder injected through dynamic_async_success_callbacks instead of litellm.callbacks.
* fix(router): bill the primary stream's tokens after a mid-stream hop and chat-bridge Responses streams on disconnect
The router's fallback wrapper now folds the primary Responses stream's relayed tokens into the fallback's disconnect event the way _combine_responses_fallback_usage does for a natural end of stream, and LiteLLMCompletionStreamingIterator (chat-only providers served through /v1/responses) builds its disconnect event from the chat completion chunks it collected. stamp_responses_usage_cost is now public so the bridge module can price that event.
* Potential fix for pull request finding 'CodeQL / Log Injection'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Every /v1/responses spend log asked its worker to flush the queue right away so a
chained previous_response_id could find the row. Only the chat-completions bridge
reads that row. Native Responses providers hold the conversation upstream, so their
rows now wait for the batch write like chat rows do
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): replace Any with proven types in 9 shared files
* refactor(types): replace Any with proven types in the remaining sweep files
* refactor(types): keep unproven integrations on the merge base and pin malformed-body handling
* refactor(types): leave five request paths on the merge base so non-object bodies keep their 400
* refactor(types): drop a docstring that repeats the helper name
* refactor(types): leave six files with no live malformed-payload proof on the merge base
* refactor(types): leave Anthropic pass-through SSE parsing on the merge base
* refactor(types): leave Responses output walking, the retry ceiling and concatenated tool JSON on the merge base
* fix(types): keep main's parsing in the budget window reset, utils loaders and Langfuse attributes
The new strict checks rejected inputs main skips or keeps, such as a JSONB
string in budget_limits, so these three files go back to main's version
* refactor(types): return the request tag parse failure as a value
* fix(bedrock): return a 400 for malformed batch JSONL lines
A malformed line in a Bedrock batch input file escaped as a JSONDecodeError
from the upload and file content conversions, which the proxy reported as a
500. Parse those files line by line and raise a 400 that names the bad line.
Object name inference keeps its fallback for files that are not JSONL.
Fixes#45060
* test(bedrock): cover blank lines in malformed batch JSONL
* fix(bedrock): reject a malformed batch line whatever the upload's MIME type
The upload only converts, and so re-checks, batch files whose MIME type is on the
allowlist. Falling back to the generic object name let a malformed file sent as
application/x-jsonl skip both checks and land in S3 unchanged, so the object-name
step now runs the same strict per-line parse and raises the 400.
* fix(bedrock): reject every malformed batch input line with a 400 naming the line
Non-object lines, a missing or non-object body, a non-string body.model,
non-UTF-8 bytes, nesting too deep to parse, and rows the Bedrock mapping
cannot convert escaped as AttributeError, TypeError, UnicodeDecodeError or
RecursionError and surfaced as 500s. They now raise a 400 BedrockError in
the same "Batch input file line N ..." wording the proxy pre-check and the
Vertex adapter use. Object naming reads only the first line and never
raises, and a UTF-8 byte order mark is accepted.
* docs(bedrock): point the batch file name comment at the helper that builds it
* test(bedrock): mark new batch test locals Final and give the line builders explicit params
* test(bedrock): use single quotes inside the batch-line f-strings so the file parses on Python 3.10 and 3.11
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* test: bill each completed request once and price unpriced returned decision models at the requested model
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(logging,decisions): bill each request once and price returned decision models only when they carry rates
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(decisions): leave the double-billing fix to #47510 and keep the decisions pricing and missing-answers fixes
* fix(decisions): name malformed answer entries and pick the Arize cache-hit span by attribute
* test(decisions): cover a returned-model row with metadata but no rates
* fix(decisions): keep upstream reply text out of the parse error so it stays a 500
* fix(decisions): answer every malformed upstream reply with a 500, whatever text it carries
A reply that fails the decisions response schema now raises
DecisionsResponseValidationError, which the decisions call maps straight to
InternalServerError instead of passing it through exception_type. Its text
matching read an answer key like rate_limit, or a value like "Request timed
out", as a 429 or a 408, so the router cooled down a deployment for a reply
that fails the same way on every retry.
The message also drops the empty location for a reply that is not an
object, the Arize family test loses the cache_hit entries main already
added, and the Bedrock Mantle count-tokens wire test calls
_metadata_deployment, since main renamed the _deployment helper under it.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
The Gemini TTS billing and text completion guardrail tests read spend rows from the owned proxy's own database, which #47213 made private. The cursor wire test derives its effort levels from the gpt-5 cost map row that #43935 now enforces, and the Azure effort check accepts the unquoted level. The pin TTL test asserts the one hour pin #46411 writes for a tool-only mark, the video callback test decodes ids with the product decoder after #44614, and the two Gemini 3.8 flash stream cost cases bill audio at the text rate their rows imply
* feat(proxy): size pass drops the oldest spend log partitions first
On a range-partitioned LiteLLM_SpendLogs the size pass drops fully
elapsed partitions oldest first, never the current or DEFAULT one,
remeasures after each drop and falls back to oldest-first row deletes
only while still above the target
* fix(proxy): a partition drop rebases the size state instead of reading as a statistics reset
Bugbot on #47449: dropping a partition removes that child's n_tup_ins and
n_tup_del from the summed probe, which advance_size_state took for a reset
and reseeded from n_dead_tup. The pass now saves the counters of the last
post-drop probe as the state's base and scales the reclaimable count to
the surviving files
* style(proxy): keep the mutable-ok reason on the line it suppresses
* fix(proxy): size pass stops when a post-drop re-measure fails and keeps day partitions ensured
A failed storage probe after a partition drop no longer reads as under target. The pass logs a warning, skips the row step and saves no count, so neither live rows get deleted on the pre-drop numbers nor the next check reads the drop as a statistics reset. The partitioned size path also calls ensure_partitions before dropping, so a size-only setup keeps getting day partitions instead of writing into DEFAULT
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stabilize auth and team delete checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): make loop-blocking and tampered-signature checks deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): isolate model-cost state and synchronize loop progress
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): fail loop-progress handshakes when the blocking call never starts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): opt-in free-tier allowance for Google Search grounding spend
* fix(proxy): declare the search free tier counter's service target
* test(cost_map): allow search_free_requests in the cost map schema
* fix(proxy): skip cache hits and never free more than the billed cost in the search free tier
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds general_settings.maximum_spend_logs_size,
maximum_spend_logs_cleanup_target_size and
maximum_spend_logs_size_check_interval, the spend_log_size_check job,
maximum_spend_logs_cleanup_cron precedence, oldest-first bounded deletes
down to the target, storage fields in the run summary and the
litellm_spend_log_storage_bytes gauge, with a Postgres integration test
covering the post-VACUUM no-refire case
Resolves LIT-9831
Fixes#44589
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): send integer enums to Gemini instead of dropping them
Gemini rejects enum values that are not strings, so _fix_enum_types removed
every non-string enum. Integer enums are now sent as type INTEGER with
format "enum" and string values, which Gemini accepts and enforces.
* fix(vertex_ai): keep converted integer enums when a tool is reused
The schema is edited in place, so a tool definition passed again already
carries the converted enum. Keep an integer enum that is already in the
format "enum" form instead of dropping it, and type the new locals.
* fix(vertex_ai): use Sequence at runtime in the enum cast
* fix(vertex_ai): treat integer-string enum values as an integer enum
A reused schema already holds its integer enum as strings. Instead of
reading `format` to recognise that, accept integer strings as well as
integers on an integer-typed enum and send both in Gemini's form. This
drops the untyped `schema.get("format")` read.
* fix(vertex_ai): map Gemini tools from a copy so a fallback keeps the caller's schema
Gemini tool mapping rewrote the caller's tools in place, so a router fallback to another provider received the Gemini-converted integer enum (string values with format enum) and returned the integer as a string
* fix(vertex_ai): send integer enums only to Gemini 3 and newer, and only ones Gemini accepts
gemini-2.5-flash stalls for minutes on a larger integer enum, so Gemini 2.x
keeps main's behavior and drops it. An enum outside int64 or longer than 1000
values is dropped too, since Gemini answers 400 for those. A nullable integer
enum written as a type list now keeps its values like the anyOf form does
* fix(vertex_ai): drop integer enum strings longer than int64 instead of failing the request
* fix(vertex_ai): drop integer enums when a request's enums pass Gemini's size limit, and send int enum members as numbers
Gemini limits the total size of all enums in one request, so the per-enum 1000-value cap still let several large integer enums through to a 400. Integer enums are now sent only while every enum value in the request adds up to 2000 characters or fewer, and an (int, Enum) member goes out as its number instead of its name
* fix(vertex_ai): send only integer enums Gemini returns exactly, and count them in Gemini token counts
* fix(vertex_ai): send Gemini integer enums without format enum, which made gemini-3.8-flash think much longer
Also skip the response cache in the repeated-request integration test, since copying tools makes the second identical request a cache hit
* fix(vertex_ai): keep dropping integer enums in Vertex batch rows, which Vertex batch cannot parse
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
_failure_spend_rows yields the final row plus one re-keyed row per lifted billed_failed_attempts entry (<request_id>-attempt-<position>, the attempt's usage, cost, retry count and deployment id), each written through DBSpendUpdateWriter.update_database so it also deducts from key, team and user spend
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): set every perf ceiling to 1.2x the lane median
* test(perf): derive the upgrade verdict test's samples from the case ceilings
* test(perf): keep memory_growth_megabytes at 100 and 600, the metric swings 60 MB between reruns
* test(perf): keep small litellm_overhead_seconds p99 at 1.2x the lane max, it is the 3rd slowest of 300 requests
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
