- Reject reducto:// file IDs sent through the proxy /v1/ocr JSON API.
The IDs are not bound to a LiteLLM key, so an authenticated user
could submit another user's file ID and receive OCR text via the
proxy's shared Reducto credentials. Force fresh uploads (multipart
form or inline base64 data URI) so every OCR call is server-mediated
and implicitly bound to the originating request.
- Add ocr_cost_per_credit=0.015 to reducto/parse-v3 and
reducto/parse-legacy in both pricing JSONs so successful Reducto OCR
calls debit key/team spend instead of recording zero.
- vertex_llm_base: guard background refresh's cache write with an
identity check so a stale write cannot overwrite a credentials
reference replaced by a concurrent reauthentication path.
- router: make shared backend mode preservation directional - only
preserve when an existing 'responses' mode would be downgraded to
'chat', or when the deployment mode is None (which would otherwise
clear the existing mode). Legitimate upgrades now apply.
- rubrik: remove unused preserve_events_added_during_flush attribute;
RubrikLogger overrides flush_queue, so the base-class flag never
applied. Drop the test that exercised the parent path on a Rubrik
instance since it does not reflect real flush behavior.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Previously the inner guard 'if _deployment_mode is not None' prevented
_shared_model_info['mode'] from being set back to the existing shared
mode when the deployment mode was None, which then overwrote the shared
backend's mode with None via register_model.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Both ChatGPTResponsesAPIConfig and LiteLLMResponsesTransformationHandler
duplicated the same OUTPUT_ITEM_DONE / OUTPUT_TEXT_DONE recovery
algorithm. Move that logic into litellm.responses.sse_output_recovery
and have both call sites use the shared helpers, so future fixes apply
in one place.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
refresh_auth is invoked from three call sites that can run on different
threads (sync get_access_token, async slow path via asyncify, and the
background proactive refresh task). Only the sync path was protected
by _sync_refresh_lock, so a concurrent sync + async/background call
could invoke google-auth's Credentials.refresh() on the same object
from two threads simultaneously, mutating internal credential state.
Move the lock acquisition into refresh_auth itself; the lock is an
RLock so reentrant acquisition from the sync path remains safe.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Replace the asyncio.Lock._waiters inspection in
_maybe_prune_async_refresh_lock with an explicit refcount so the entry
is pruned exactly when no coroutine is holding or waiting on the lock,
without depending on any private asyncio internals.
- Build a memoized index of fireworks_ai/* entries from litellm.model_cost,
invalidated by (id, len) of the model_cost dict. Avoids re-scanning the
full ~30k-entry model_cost dictionary on every get_provider_info call.
- Replace plain substring containment with hyphen-aligned boundary matching
so a known short model name (e.g. 'some-model') cannot falsely match an
unrelated longer query (e.g. 'awesome-model').
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Raise a clear ValueError when Reducto /upload returns 200 without a
file_id key (or with a non-JSON body), instead of letting downstream
callers see a confusing KeyError.
Address the two new findings from greptile's 19:45 review of the
vertex+router surfaces.
- vertex_llm_base: when the slow path sees TokenState.INVALID, await any
in-flight background refresh task before invoking refresh_auth
ourselves. google-auth's Credentials.refresh() is not safe to call
concurrently on the same credentials object, and the background task
runs outside the per-key lock. After the wait, re-check the cached
token so we can short-circuit if the background refresh already
restored it. Extracted the helper into
_await_in_flight_background_refresh so get_access_token_async stays
under ruff's PLR0915 statement budget.
- router.py: when alias registration would overwrite the deployment's
declared `mode` to keep the shared backend mode stable, emit a
verbose_router_logger.warning so the override is visible to operators
instead of silently winning. The existing fix (preventing alias
registration from downgrading a shared `mode: responses` to chat) is
preserved; the warning just surfaces it.
- Fireworks: _get_model_cost_capability fallback now picks the longest
substring match in model_cost so more specific entries win over less
specific ones (instead of returning the first match by insertion order).
- Reducto OCR: drop per-request _api_key/_api_base instance attributes on
_BaseReductoOCRConfig and instead thread api_key/api_base through
transform_ocr_request/async_transform_ocr_request kwargs from the
shared OCR HTTP handler. Makes the config safe to share/cache across
concurrent requests with different credentials.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Swap the # 5a / # 5b step labels so they appear in numerical order
within the file. The new output_config-strip block was added with
label # 5b above the pre-existing # 5a 'remove custom field from
tools' block; rename the new block to # 5a and the pre-existing
block to # 5b so the labels match the order of the steps in the
file.
No behavior change.
Co-authored-by: Greptile Reviewer <greptile-apps@users.noreply.github.com>
Address greptile's P2 thread (#3144753707) about ocr_cost silently
under-reporting billing when response.usage_info.pages_processed is
missing. The credit-priced and unpriced fallback still has to return
0.0 (we don't know how to bill without usage), but emit a warning so
the missing-data case is visible in logs instead of disappearing.
The per-page-priced branch still raises, preserving the original
ValueError signal callers may catch.
Bedrock Invoke transformation files (chat and messages) called
_supports_factory(custom_llm_provider=None, ...) which relies on
auto-detection. For short Bedrock model names (e.g. 'anthropic.claude-opus-4-6'
without the version suffix) auto-detection fails and the lookup falls back
through the exception path. Passing the known 'bedrock' provider explicitly
makes the lookup deterministic for all Bedrock model variants, including
cross-region inference profile IDs.
Co-authored-by: Claude <noreply@anthropic.com>
Address greptile's open thread on _async_refresh_locks growing
unboundedly in high-cardinality deployments.
- Add _maybe_prune_async_refresh_lock: drops the per-key Lock from
the registry once no coroutine holds it and no coroutine is queued
in lock._waiters. The check-then-pop sequence is safe under
asyncio's cooperative scheduler — a waiter that arrives after the
pop simply creates a fresh lock under the same key, which is fine
because the previous batch is already done.
- Wrap the slow-path async with lock in a try/finally so the prune
runs on every exit (return, exception, reauth retry).
- Extract the existing background-refresh task scheduling into
_schedule_background_refresh so get_access_token_async stays under
ruff's PLR0915 ("Too many statements") limit. No behaviour change.
- Regression tests cover both pruning after release (the dict
shrinks back to zero after each call) and the safeguard that
keeps the lock alive while a waiter is still queued.
The `allowed_tools` field on `BlockedToolsResult` was computed in
`_extract_blocked_tools` but never read by the only caller — when any
tool was blocked the integration unconditionally raised
`ModifyResponseException` to reject the full response, never doing
partial filtering. Drop the dataclass and return the blocking
explanation directly as `Optional[str]` so there's no misleading shape
hinting at unused partial-filter capability.
Co-authored-by: Greptile <greptile-apps[bot]@users.noreply.github.com>
- Fireworks _get_model_cost_capability: after exact-key lookups, fall back
to substring matching against fireworks_ai/* entries in model_cost so
model name variants (e.g. fine-tuned suffixes) continue to inherit
capability flags like supports_reasoning.
- Vertex vertex_llm_base: replace non-reentrant threading.Lock with RLock
on the sync refresh path so the reauthentication retry, which recurses
into get_access_token while still holding the lock, does not deadlock
when reloaded credentials are also expired.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Address bugbot review concerns:
- Sanitize proxy_server_request before forwarding to the Rubrik webhook.
The previous code passed the entire inbound HTTP context (Authorization,
Cookie, x-api-key, and the raw request body) through to a third-party
endpoint, which exfiltrates proxy credentials and upstream secrets. The
new _sanitize_proxy_server_request allowlists only url and method.
(Cursor Bugbot HIGH severity #3192354895)
- Treat a null choices[0].message.tool_calls as 'all blocked' rather than
letting iteration raise and silently fall through the outer except in
apply_guardrail (which would fail open). Iterate over a defensive
fallback list instead of relying on the dict default.
(Cursor Bugbot MEDIUM severity #3192349538)
Co-authored-by: Cursor Bugbot <bugbot@cursor.com>
Commit dfb2524 intentionally reverted _async_refresh_locks from a
WeakValueDictionary back to a regular Dict so the per-key asyncio.Lock
identity is stable across concurrent callers — preserving
single-flight semantics. The test asserting that the dict shrinks
back to 0 after refreshes was added when the WeakValueDictionary
backing was still in place; it now contradicts the deliberate design
and is failing CI.
- cost_calculator.ocr_cost: log a warning when pages_processed is reported
but no ocr_cost_per_page is configured, instead of silently billing zero
via an implicit '(... or 0.0) * pages_processed' fallback. Behavior is
preserved (zero cost) so free-tier / unpriced models still work, but
configuration gaps are now visible in logs.
- ChatGPTResponsesAPIConfig._extract_completed_response_from_sse: also
collect response.output_text.done events into a text-only items map and
merge them into the recovered output (OUTPUT_ITEM_DONE wins on duplicate
output_index), mirroring the LiteLLMResponses handler. This recovers
text content when a provider only emits OUTPUT_TEXT_DONE and the final
response.completed event has an empty output list.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- Guard background refresh done_callback with an identity check so a
stale callback cannot remove a newer task that already replaced it in
the tracking dict (done_callbacks are scheduled via call_soon, so a
fresh task can be stored for the same credential key before the old
callback fires).
- Replace WeakValueDictionary with a regular dict for
_async_refresh_locks so the per-key asyncio.Lock identity is stable
across concurrent callers; otherwise a lock can be GC'd between two
coroutines arriving for the same key, breaking single-flight.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- vertex: back `_async_refresh_locks` with a WeakValueDictionary so a per-key
Lock is auto-evicted once no coroutine holds it, preventing unbounded growth
in deployments with many credential combinations while keeping single-flight
semantics intact.
- rubrik: defer the periodic flush task to the first log event when the logger
is constructed without a running event loop, so low-traffic batches still
get drained instead of being silently stranded by a swallowed RuntimeError.
Pull in the base-branch fix from #28320 which (1) adds the
mistral/ministral-8b-2512 cost map entry on main and (2) backfills the
in-tree local backup into litellm.model_cost in tests/local_testing/conftest.py
so cost-calculator lookups succeed even before main's hosted cost map
catches up. This unblocks the failing test_completion_mistral_api and
test_completion_mistral_api_modified_input cases in local_testing_part1.
Co-authored-by: Claude <noreply@anthropic.com>
* feat(gemini): add gemini-3.1-flash-lite model cost map entries
Co-authored-by: Cursor <cursoragent@cursor.com>
* Update model_prices_and_context_window.json
* Update source URL for model pricing information
* Sync source URL for gemini-3.1-flash-lite in backup JSON
* fix(model_cost_map): add mistral/ministral-8b-2512 entry
Mistral rotated the 'mistral/mistral-tiny' alias to return
'ministral-8b-2512' as the response model, which is not in the cost map.
This caused test_completion_mistral_api and
test_completion_mistral_api_modified_input to fail in
completion_cost lookup. Add the entry mirroring the existing
openrouter/mistralai/ministral-8b-2512 pricing.
* test(cost_calculator): assert output_cost_per_reasoning_token for gemini-3.1-flash-lite
* fix(tests): backfill local backup entries into runtime model_cost
litellm.model_cost is loaded from LITELLM_MODEL_COST_MAP_URL (pinned to
main) at import time, so any pricing entries added to the in-tree backup
on this branch aren't visible at test runtime until they also land on
main. The Mistral cassette currently returns model=ministral-8b-2512
and the cost-calculator lookup in test_completion_mistral_api /
test_completion_mistral_api_modified_input fails despite the entry
existing in the local backup. Backfill missing backup entries into
litellm.model_cost in the local_testing conftest so these lookups
succeed against the cassette state the branch is being tested with.
* fix(tests): guard conftest backfill against empty local cost map
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Mistral's API now returns model='ministral-8b-2512' when 'mistral-tiny' is requested. Adding the entry so completion_cost can resolve the cost for that response.
- fireworks: resolve supports_reasoning lookup for short model names by also
trying the full accounts/fireworks/models/ path in model_cost
- ocr_cost: drop reducto-specific guard in shared utility; treat missing
pages_processed as zero cost when no per-page pricing is configured
- docs: remove reducto/rubrik markdown stubs from this repo (canonical docs
live in litellm-docs)
Use model-map lookup with _model_supports_effort_param fallback so Bedrock
Invoke keeps output_config for Claude 4.6/4.7 when pricing flags are missing.
Revert custom_llm_provider=bedrock for supports_output_config checks, fix
allowlist test model, and apply black to xai/vertex files failing lint CI.
Co-authored-by: Cursor <cursoragent@cursor.com>
Previously RubrikLogger relied on CustomBatchLogger.flush_queue, which
captured len(self.log_queue) separately from the snapshot taken inside
async_send_batch. Although both happen without an intervening await today
(so they agree in practice), they are semantically disconnected: a future
refactor that adds an await between the two captures, or that changes the
async_send_batch contract, could cause the parent to delete a different
number of items than were actually sent and trigger duplicate deliveries
to Rubrik.
Override flush_queue on RubrikLogger so a single snapshot drives both the
HTTP POST and the queue truncation. async_send_batch is preserved for
direct callers/tests but no longer participates in the canonical flush
path. Existing tests (including the one that explicitly invokes the base
CustomBatchLogger.flush_queue path) still pass.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
_parse_sse_json_chunk in ChatGPTResponsesAPIConfig passed the raw chunk
directly to _strip_sse_data_from_chunk, which only matches the 'data:'
prefix at position 0. Chunks with leading whitespace (e.g. ' data: {...}')
were returned unchanged and silently failed JSON parsing, dropping the
contained event.
Mirror the existing fix in LiteLLMResponsesTransformationHandler._parse_raw_sse_chunk
by calling chunk.strip() before stripping the SSE prefix.
Adds a regression test using whitespace-padded data: lines and verifies
that the response.output_item.done payload is recovered into the final
ResponsesAPIResponse output.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Previously, _log_batch_to_rubrik swallowed all HTTP errors and exceptions,
and the parent flush_queue unconditionally drained the queue afterwards.
On Rubrik 5xx responses, network errors, or timeouts the in-flight events
were silently dropped without ever being delivered.
- Re-raise from _log_batch_to_rubrik so failures surface to the caller.
- In CustomBatchLogger.flush_queue, catch exceptions from async_send_batch
and leave the queue intact for retry on the next flush. Existing loggers
that override flush_queue (e.g. Datadog) or that swallow their own errors
inside async_send_batch (e.g. Langsmith, GCS, Argilla) are unaffected.
- Tests now assert events are preserved on HTTP errors, network errors,
and that mid-flush appended events are also preserved on failure.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
When recovering output from raw SSE, OUTPUT_ITEM_DONE and OUTPUT_TEXT_DONE
events were treated as mutually exclusive fallbacks. If a stream emitted
OUTPUT_ITEM_DONE for some output indices and only OUTPUT_TEXT_DONE for
others, the text-only items at the missing indices were silently dropped.
Merge both dicts before returning, with OUTPUT_ITEM_DONE entries taking
precedence at any shared index (preserving the existing behavior covered
by test_transform_response_preserves_output_item_when_text_done_arrives_later).
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
asyncio.create_task() raises RuntimeError when called outside a running
event loop. Wrap the call in a try/except RuntimeError so that RubrikLogger
can be instantiated in synchronous contexts (e.g. during startup, testing)
without crashing. The periodic_flush background task simply won't start in
those cases; it starts normally when the constructor is called inside an
event loop.
Add a test that verifies instantiation outside an event loop does not raise
(does not patch asyncio.create_task).
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
Completed asyncio.Task objects were never removed from
_background_refresh_tasks. In long-running proxies with many distinct
credential keys the dict grows indefinitely, retaining references to
finished tasks and their results.
Fix:
- Pop the existing (done) entry before creating a replacement task.
- Attach a done_callback to each new task that removes its entry from
the dict once the task finishes (success or failure).
Tests:
- test_background_refresh_task_removed_after_completion: verifies the
done-callback cleans up a single entry after the task completes.
- test_background_refresh_tasks_no_accumulation_across_many_keys:
drives 20 distinct credential keys and confirms the dict is empty
after all background refreshes finish.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
xAI can return total_tokens inconsistent with prompt_tokens +
completion_tokens when caching is enabled. Align with OpenAI-style
usage so shared LLM tests and downstream consumers see coherent totals.
Apply to non-streaming responses and streaming usage chunks.
Made-with: Cursor
