* fix(security): restrict and validate file uploads at /v1/files and /upload/logo
Extends fast-fail upload validation to every purpose at POST /v1/files,
not just purpose=batch: a configurable max_file_size_mb size cap and a
blocked_file_extensions denylist, plus rejection of filenames carrying a
directory-traversal component before anything is read, stored, or
forwarded to a provider.
Also fixes two concrete gaps found while auditing every upload surface:
the Azure Blob Storage backend derived a blob path's extension with
filename.split(".")[-1], which does not parse path structure and let a
crafted filename embed a directory traversal sequence into the stored
blob path; and POST /upload/logo (the admin UI logo upload) had no
role check at all, so any authenticated API key, not just a proxy
admin, could write a file to the server's disk.
* fix(lint): drop cast()/mutation from settings coercion, sync blocked_file_extensions on reload
Replaces the TypeAdapter+cast() reads of max_file_size_mb and
blocked_file_extensions with small isinstance-based validators, since the
codebase's cast() budget (LIT006) had no headroom left. Also adds the
blocked_file_extensions reload block that was missing from
_update_general_settings: it was registered as an editable setting but
never re-synced into runtime state, so a value set through the DB-backed
settings editor would silently never take effect (Greptile finding).
* fix(security): declare max_file_size_mb and blocked_file_extensions on ConfigGeneralSettings
The DB-backed general-settings update endpoints validate every field
through ConfigGeneralSettings.model_fields before persisting it, so
without these declarations an operator could never actually set either
setting through that path even though both were registered for the
Admin UI's settings editor and reloaded on config refresh (Greptile
finding). blocked_file_extensions is typed as a tuple, not a list, to
stay out of the immutable-collections lint budget; the stored JSON
value is unaffected since the raw request payload, not the validated
model, is what gets persisted.
* chore: regenerate schema.d.ts for the new ConfigGeneralSettings fields
* fix(security): normalize configured blocked_file_extensions casing
check_blocked_extension lowercased the uploaded filename's extension
before comparing but compared it against blocked_extensions verbatim,
so an admin-configured blocked_file_extensions: ['.EXE'] would never
match an uploaded payload.exe (Greptile finding). Normalizes the
configured values the same way at comparison time, and adds the
missing case (mismatched-case config, lowercase upload) as a
regression test, mutation-checked against the unfixed comparison.
* fix(security): restore caller-owned stream position after size inspection
_file_size_bytes unconditionally seeked back to 0 after measuring a
BinaryIO's length, discarding wherever the caller had actually
positioned it (Greptile finding). Saves and restores the original
position instead. Rewrites the existing test that had encoded the
old "always resets to 0" behavior as its expectation, and adds a
sibling case for the under-cap path; both are mutation-checked
against the unfixed always-reset-to-0 behavior.
* test: add regression coverage for twelve closed issues
Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:
- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges
Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.
* fix: stop the new regression tests polluting and tripping over shared global state
Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.
test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.
The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
The field is declared optional on OpenAIFileObject and its own docstring says it
is absent on every upload guardrails did not touch, but the /v1/files routes have
no response_model, so FastAPI falls through to jsonable_encoder with exclude_none
off and serialises the unset default as an explicit null. Every create and
retrieve response on a proxy with no guardrails configured at all picked up a
litellm_batch_guardrail: null it never had before, and so did every row of a file
list, since those rows are the same object.
A wrap serializer drops the key only when nothing set it, so the populated report
still reaches the wire intact, including a record whose guardrail is null. The
managed-files list route spreads a stored file_object blob rather than the model,
so rows persisted before this lands keep their null until it is dropped there too.
Eleven DummyManagedFiles stubs still declared afile_list(self, purpose,
litellm_parent_otel_span). The real hook grew user_api_key_dict, limit and
after, so the doubles no longer stand in for the interface they replace.
Their tests pass today only because every one of them takes a provider
branch that never reaches the hook, which means a stub going stale is
invisible until some later test does reach it and reads a TypeError as a
behavior change.
Signatures only; no test changes behavior.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
The managed hook returned the plain dict build_list_page builds, while
every other GET /v1/files path returns an SDK page object. A post-call
success hook or a logging callback that reads response.data off the
listing raised AttributeError as soon as a request took the managed path
FileListPage is a pydantic model over the same five fields, so hooks read
.data again and the response body does not move: jsonable_encoder gives
the same keys in the same order for the model and for the dict. It sits
in litellm.types.llms.openai because base_llm/files/transformation.py
already imports from there and cannot import proxy modules. It is
deliberately not subscriptable, since the provider-backed path returns a
page object that is not either, and dict access would be a third contract
to keep alive
Also reject a purpose the Files API never accepts. An unknown purpose
matches no row, so the listing answered an empty page for what is really
a bad request, while the upload route in this same file already refuses
those values against get_args(OpenAIFilesPurpose). The check runs before
the first query, and only in the managed hook, so providers that define
their own purposes keep them
Also put back the route's original except tail. Sending every error
through handle_exception_on_proxy changed error.type on a bad
target_model_names from "None" to the exception class name, which a
caller matching on the body would read as a break. create_file in this
file already pairs base's tail with a ProxyException passthrough, so
list_files does the same and the handle_exception_on_proxy import is gone
The managed file listing cut the page to `limit` first and applied the
purpose filter in Python afterwards, so a page whose rows all failed the
filter came back as `data: []` with `has_more: true`. openai-python stops
paging the moment `data` is empty, so `files.list(purpose="batch", limit=1)`
returned nothing at all instead of every batch file.
Read successive keyset chunks until the page holds `limit + 1` matches or
the caller's rows run out, then return at most `limit` of them. `data` is
now non-empty whenever matching files remain, its last id is always a
usable cursor, and `has_more: false` only ever means the caller has seen
everything. Rows whose stored blob will not parse drop out in the same
loop, so they cannot empty a page either.
That also makes the `next_cursor_id` escape hatch on `build_list_page`
dead, so it goes back to what it was for the batch and vector-store
listings that share it.
Also move `validate_file_list_limit` up into the list_files route, so the
target_model_names and provider branches reject an out-of-range limit the
same way the managed file store already did.
The unscoped GET /v1/files limit check accepted 0, which OpenAI's minimum
of 1 does not allow, and the route's except block rebuilt every error with
getattr(e, "status_code", 500). ProxyException has no status_code, so the
400 it raises went out as a 500 and the OpenAI SDK retried it three times.
Errors now go through handle_exception_on_proxy, the helper the sibling
batches route already uses, and the unknown-cursor error is a ProxyException
so it carries type invalid_request_error and param after instead of the
literal "None". The cursor still 400s whether the file belongs to someone
else or does not exist at all
The owner-scoped listing read every row the caller owns in one query, so an
admin key that owns every file on the proxy pulled the whole table into one
response. Page it with a keyset cursor on unified_file_id instead, and accept
limit and after on GET /v1/files so a client can walk the pages. limit follows
what OpenAI documents for that route: 1 to 10000, default 10000.
An after cursor is resolved inside the caller's own scope, so an id they do not
own gets a 400 rather than a page, and has_more now reflects whether another
row exists instead of always being false.
Refs #37714
* fix(proxy): scan batch records with the content hooks that are not guardrails
Guardrails were made to run on batch uploads by scanning each record through the pre-call hook
with the walk limited to guardrails. That limit exists because the same branch carries the rate
limiters and budget accounting, which must count an upload once rather than once per line. It
also excluded every enforcement hook written as a plain CustomLogger, so prompt-injection
detection, Azure content safety, banned keywords and the blocked-user check never saw a batch
record at all. Content that is a hard 400 online reached the provider verbatim through batch.
A CustomLogger now declares whether its pre-call hook judges the payload or merely counts the
request. The four that judge it opt in, the walk admits them, and both short-circuits learn
about them, including the one that decides whether the file is streamed off disk in the first
place: a proxy configured only with one of these hooks was skipping the scan entirely. Nothing
that counts a request is marked, so an upload still costs one slot and one budget check.
* refactor(proxy): drop the per-hook comment the attribute contract already states
* test(proxy): make the classification a ledger, and pin the wiring with a real hook
The classification test listed the two non-enterprise hooks by hand, so unmarking either
enterprise one changed nothing and the mutation matrix passed with both surviving. It now walks
the hook registries and fails on any pre-call CustomLogger that is on neither side, which also
gives the flag the forcing function it lacked: an enforcement hook added later would otherwise
default to off and silently skip batch records, which is the bug being fixed here.
Nothing exercised the path the bug actually lived on either, since every test raised its own
exception rather than a real hook's. One test now drives the shipped prompt-injection hook
through the scan, which pins the part no synthetic exception reaches: a chained exception reads
as a failure to judge, so refactoring any of these hooks to `raise ... from` would turn every
per-record drop into an aborted upload.
Also records why a hook that rewrites the payload for routing stays unmarked, and that only the
leaf class is consulted.
* test(proxy): set the callback list through monkeypatch rather than writing the global
* fix(proxy): read batch records the same way the upload validation does
The upload validation parses each JSONL line as bytes, where the json module sniffs the
encoding itself and accepts a leading byte order mark or a lone surrogate. The guardrail scan
that runs immediately after decoded each line to text first, which is stricter, so a file the
validation had just accepted could fail the scan. A `.jsonl` written by any of the editors that
emit a BOM, which includes PowerShell's Out-File and classic Notepad, uploaded fine until a
pre_call guardrail was configured and then returned 500 with a decode error and no indication
of which line or why. The scan now parses the same bytes the validation did, and an untouched
record is copied through as the bytes it arrived as rather than re-encoded.
A numeric custom_id was reported as null. The spec asks for a string, but callers do send
numbers, and null leaves the one field a caller reconciles on empty for exactly the records
that need it.
* fix(proxy): read the load-balancing record the same way, so a byte order mark keeps its routing
The first record is parsed to pick a deployment when batch load balancing is on, and it was
decoded to text before parsing, which rejects a leading byte order mark. The lookup returns None
on any parse failure, so such a file silently lost its routing and went to the default provider
rather than the configured one. That was already reachable for an upload no guardrail changed,
since the original bytes are passed straight through, and preserving the mark through a rewrite
widens it. Parsed as bytes now, like the validation and the scan.
* fix(proxy): find the routing record past a blank first line
The upload validation and the guardrail scan both skip blank lines, but deployment selection
read only the first physical line, so a file starting with a blank line lost its routing model
and went to the default provider rather than the configured one. It now skips blanks the way
the other two readers do, reading lazily so a large file is not read past its first record.
* fix(proxy): do not crash deployment selection on a record whose body is not an object
The upload validation checks that a record has a `body`, not that it is an object, so a record
can carry a string or a list there. Deployment selection called `.get` on it unconditionally and
raised, returning 500. That was already reachable for a plain file, and reading past a byte order
mark or a blank first line widened it to files that previously fell through to the default
provider instead. A record whose body names no readable model now resolves to no model, which is
the same answer the default-provider branch already handled.
* fix(proxy): keep a custom_id that cannot be encoded from failing the whole upload
A record identifier is echoed back in the create response. JSON parses a lone surrogate happily
but it cannot be encoded again, so a file the upload validation accepts returned 500 from the
response renderer rather than a report. Unencodable characters are replaced, which leaves every
ordinary identifier untouched and keeps a pathological one reconcilable.
This predates the reader change; reading past a byte order mark only altered which error the
same file produced first.
* fix(proxy): treat a url the parser rejects as one we do not recognize
Resolving a record's call type from its url runs the url through urlsplit, which raises on a few
malformed authorities such as an unclosed bracket. That happens before the try that wraps the
guardrail call, so it escaped the scan and returned 500 on a file the upload validation had just
accepted. An unreadable url is simply one we cannot recognize, which the body-shape fallback
already handles, so the record is still scanned rather than lost.
Reachable on staging today for a proxy running any guardrail. Enabling the scan for a proxy that
runs only a content-enforcing CustomLogger widens it to that configuration too, which is why it
is fixed here rather than left.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* feat(proxy): redact or drop individual batch records instead of rejecting the file
A single record tripping a guardrail rejected the whole upload, which is unusable for a file
holding thousands of rows. A record a guardrail rewrites is now submitted in its rewritten
form, a record it blocks is left out, and the create response reports every changed record by
both custom_id and line so a caller can reconcile against the file it sent. The same outcome
is written to the proxy log and to request metadata, so it is not visible only to the caller.
A rewritten record goes straight to a spool and only its offset is carried, so a masking
guardrail touching most rows of a large upload does not build a second copy of the file on the
heap, and the rewrite runs off the event loop the way the sibling full-file validation does.
Both proxy-injected metadata keys are captured from the record and restored exactly, including
an explicit null, so a masked row keeps the tags that decide how it is attributed.
A record is dropped only when a guardrail judged its content. `GuardrailRaisedException` now
carries `blocked_content` for that, because half its raise sites in the repo signal an
unreachable or unparseable backend under a fail-closed policy, and treating those as blocks
would turn "refuse this request" into "drop this record and submit the rest". The default is
off, so a raise that does not say what it means aborts the upload instead of silently
shrinking the file.
* fix(proxy): only drop a batch record on a verdict the guardrail actually reached
A guardrail that reports a technical failure as an HTTPException carrying a block status was
read as a content block, so an unreachable backend under a fail-closed policy quietly shrank the
file instead of failing the upload. Two in-tree integrations do exactly that, and one of them
defaults to fail-closed, so the broken configuration was the default one. Such an exception is
raised `from` the underlying error, which is a deliberate statement that something else caused
it, and no content verdict in the repo is raised that way, so the chain now settles it. Implicit
context is left alone, since a block raised inside an unrelated `except` would read as a failure.
Two annotation errors in the same family: the one GuardrailRaisedException subclass in tree never
opted into blocked_content, so a real block took the whole upload down with it, and straiker's
block helper is reached both from its verdict and from its fail-closed handler, so it claimed a
verdict for an outage. The helper now takes the flag from its caller.
A record could also opt itself out of the chain. Guardrail selection reads a body-level
`guardrails` key ahead of the proxy-injected list, and online that key can only add to the key
and team selection, never replace it, so a batch record naming an empty list skipped every
guardrail that was not default_on and was still reported as scanned. Every injected key is now
stripped before dispatch and restored afterwards.
A guardrail that reroutes a record to another model is honoured on the online path by rewriting
the model, which the scan read as a rewrite and submitted in the same file, sending content to
the provider the reroute existed to avoid. Every record of a batch file goes to one provider, so
the upload is refused instead, naming the line.
The scan spool is closed on the paths that never read it back.
* fix(proxy): give the scan the metadata bag guardrails actually read, and close its spools
The narrowed request metadata was installed under `litellm_metadata` only, but a record is
scanned as the chat request it describes, and the guardrails that pick a policy from a request
header read `metadata` instead. Noma choosing an application and Aim choosing a user both look
there, so the header allowlist added for them did not reach either one and a batch record was
still evaluated under the fallback policy. The scan metadata now goes into both bags, which are
both stripped and restored, so neither survives into the record that ships.
The scan spool was closed on the paths that abort, which are exactly the paths where it is
empty, and left open on the one path where it holds the rewritten records. Nothing closed the
rewrite output either, where before this feature the uploaded handle belonged to Starlette. The
upload now owns both and closes them however it exits.
* fix(proxy): register the scan spool before the rewrite can fail
The scan spool was added to the request's cleanup list only after the rewrite returned, so a
rewrite that raised, which for a spilled file can be as ordinary as the disk filling up, jumped
to the handler with the list still empty and left the scan's own handle open. The rewrite also
left its half-written output behind on that path, since nothing owns that handle until it is
returned. Both now close.
* fix(proxy): run pre-call guardrails on batch input file uploads
POST /v1/files with purpose=batch was the only route in files_endpoints that
never reached pre_call_hook, so guardrails did not see batch content at all and
records reached the provider unscanned.
Stream the uploaded JSONL a record at a time and run each record's body through
the existing pre_call_hook dispatch under the call type its url maps to, so
guardrail resolution, key and team config, and the per-endpoint translations are
reused rather than reimplemented. The hook gains a guardrails_only mode for this,
since the same callback loop also drives rate limiters, budget hooks, prompt
templates and hanging-request alerting, none of which should fire once per record.
A guardrail that blocks raises its own exception, which propagates untouched so
its status code survives. A record a guardrail would rewrite, a record that
cannot be parsed, and a record whose url cannot be scanned all reject the upload,
since silently skipping any of them is the bypass this is meant to close.
Per-record redaction lands separately.
The scan only runs when a guardrail that actually runs pre_call, or a guardrail
pipeline, is configured, so deployments without one are byte for byte unchanged.
* fix(proxy): compare the dict a batch guardrail returns, not the one it was given
async_pre_call_hook may return a replacement dict instead of mutating its input, and
process_pre_call_hook_response then makes that replacement the request. The scan only
inspected the dict it passed in, so a guardrail that redacts by returning a copy was
treated as a no-op and its record uploaded unchanged.
* fix(proxy): treat a missing batch body key as different from a null one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(proxy): document the guardrails_only mode on pre_call_hook
* fix(proxy): resolve a batch record's scan type from its body when the url is unfamiliar
The scanner only accepted five exact urls, but callers write that field by hand and the
provider transformers are far more permissive: bedrock treats any non-empty url as chat
and vertex strips query strings and trailing slashes. Uploads that work today would have
started failing the moment a pre-call guardrail was configured.
Normalize the url before lookup and fall back to the body shape when it is unfamiliar, so
a record we can still read is a record we still scan. Only a body with no messages, prompt
or input is now refused, and the error says so instead of listing urls that were never the
whole set.
Also pins the default side of the guardrails_only gate: the hanging-request alert and
prompt templating are asserted to still fire when the flag is absent.
* refactor(proxy): drop batch guardrail checks the upload validation already makes
check_batch_file_upload now runs first and rejects a line that does not parse, a line that
is not an object, and a line missing custom_id, method, url or body, so the guardrail scan
can rely on all four. Its own parse handling was unreachable through the endpoint and is
gone, along with the tests for it. What is left is the case that validation does not cover,
a body whose value is not an object, since it only checks that the key is present.
* fix(proxy): resolve a batch record's call type from the url path, not the whole url
A record naming its route in full, which is how callers actually write batch files, matched
no known route, so it fell through to the body shape. A Responses record carries `input`,
and that reads as an embedding, so the record was scanned as the wrong call type and any
guardrail scoped to chat or Responses skipped it while the upload was accepted. Chat records
survived only because their body shape happens to map back to the same call type. The url is
now reduced to its path before matching.
Guardrails that pick their policy from a request header, such as noma choosing an application
id, saw no headers at all during the scan and fell back to a default, so a batch record could
be evaluated under a different policy than the same content sent online. The sanitized headers
the proxy already stores in request metadata now travel with the scan.
Also drops the bare `dict` annotation, the unreachable non-dict branch on the guardrail chain's
own return, and the type alias that was missing its `TypeAlias`, which together were failing
the lint gate.
* fix(proxy): give each batch record its own copy of the scan metadata
The narrowed metadata was handed to every record as a shallow copy, so `headers` and `tags`
stayed shared with the upload request and with the other records in the same window. A guardrail
that writes into one of those in place, which several do to record their own bookkeeping, would
have its write show up in every record scanned after it and in the request itself. The narrowing
already removed the values that cannot be copied, so each record now gets a deep copy.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Probe the column before the scheduler registers CheckBatchCost, closing the window where a retrieve that decided the poller was inactive billed a batch the first poll cycle then billed again. Also drop narration docstrings and section banners from the new tests.
The ownership question was asked twice for one retrieve: once before the provider
call to decide whether to suppress inline accounting, and again afterwards to
decide whether to mark the batch accounted. Between those two points the poller
can complete its first successful filtered query and become usable, so the two
answers disagree. The retrieve then accounts for the batch inline, having decided
the poller was unusable, while the later check sees a usable poller and leaves the
marker unset, so the poller accounts for the same batch again and its spend is
counted twice.
The retrieve now decides once and passes that decision to
update_batch_in_database, which prefers it over re-deriving one. Callers that
record no cost of their own leave it unset and keep deriving it as before, so the
cancel path is unchanged.
The handoff asked whether the poller was running, when what matters is whether it
will actually account for the batch. Those differ on a schema without the
batch_processed column: the poller cannot filter on it, so it falls back to a
query that excludes complete and completed rows, and it cannot set it either. A
caller retrieving a provider-completed batch before the poller saw it therefore
suppressed inline accounting, then marked the row complete, and the fallback query
could never find it again. Nobody accounted for that batch, so its cost escaped
the caller's budget entirely.
The poller now publishes batch_processed_support_confirmed, set only once a
filtered query has actually succeeded, and the handoff requires it. Defaulting to
unconfirmed keeps accounting on the retrieve path in exactly the cases the poller
would drop the batch, including the window before the poller's first cycle. All
four combinations account exactly once: unconfirmed leaves the retrieve
accounting and setting the marker, whether or not the column exists, and
confirmed is only reachable when the column is present, where the poller accounts
and sets it.
A scheduler that hands back something other than a bound method leaves no poller
to interrogate, which reads as unconfirmed rather than as working.
Two components computed a managed batch's cost and each assumed it was the only
one. Retrieving a batch computed it through the @client decorator's success
callback, and CheckBatchCost computed it on its own schedule. Whichever observed
completion first decided the outcome, so cost was either counted once per
retrieve or not at all.
The lockout is the worse half. Retrieving a batch that had reached completion set
batch_processed=True, which is what takes a batch out of CheckBatchCost's queue,
since it selects batch_processed=False. That write claimed the cost had been
accounted for on behalf of a callback that had not run yet and was not awaited.
When the callback then failed the cost was gone permanently, with the poller
already retired and no retry left. Observed on a live proxy: two completed
batches whose callbacks raised inside the logging worker, one on a provider
output path that did not resolve and one on a batch whose output file id was
still None, both left marked processed with no spend row and no way to recover
them. Nothing logged at error level for the batches themselves.
The over-count is the other half. Nothing suppressed recomputation, so each
retrieve of an already-completed batch recorded that batch's full cost again. A
caller polling its own batch to see whether it had finished inflated spend by
however many times it looked.
The flag now means what its name says, and only the component that actually
recorded the cost sets it. When the poller is running it owns accounting, so
retrieving a managed batch records no cost and leaves the flag alone; the poller
computes once and sets it. When the poller cannot be relied on, either because
polling is disabled by config or because the enterprise job never registered,
the retrieve path is the only accountant and behaves exactly as before. Batches
with no managed object row are untouched either way, since neither the flag nor
the poller queue applies to them.
The helper that carries the credential snapshot into litellm_params lived private
in files/main.py, and the batch retrieve needed it too. It now sits beside
get_litellm_params, which is what it augments, so neither caller reaches into the
other's private surface. Typed as Mapping/MutableMapping of object rather than
Any, which the strict import rules ban.
The file-content route builds the snapshot through the same helper as the batch
route instead of assembling a conditional mapping inline, which drops two mutable
constructions and leaves one way to attach it. Its name loses the batch suffix now
that both routes use it.
A third path reads a completed batch's output file, and it could not resolve the
bucket either. When cost is accounted from the retrieve itself rather than from
the poller, the batch success handler calls _handle_completed_batch, which fetches
the output file through _extract_file_access_credentials. That helper forwarded a
whitelist covering Azure and Vertex, gcs_bucket_name included, but nothing for
Bedrock, and retrieve_batch built its litellm_params through get_litellm_params,
whose fixed signature drops the trusted credential snapshot. So the snapshot never
reached the file read and it failed with "S3 bucket_name is required" for a bucket
the deployment had configured, leaving the batch's cost unrecorded.
Adding s3_bucket_name to that whitelist would not have worked. The Bedrock file
config deliberately resolves the bucket only from the immutable server-side
snapshot or the environment, never from a request param, because the bucket is
what managed file ids are validated against. The snapshot is therefore what has to
flow, exactly as it already does for the model-routed and cost-poller paths.
retrieve_batch now re-adds the snapshot after get_litellm_params, the same way the
file operations already do, the whitelist forwards it, and the proxy attaches it
for router-routed managed batches from the deployment behind the unified id.
Verified against a live proxy reading a real completed Bedrock batch: the cost row
appears within seconds of the retrieve carrying the batch's real spend and usage,
where before the read raised and no row was written.
Resolving those credentials is best effort. A batch whose deployment no longer
resolves, which happens when a model group is removed while batches are in
flight, still serves its status instead of failing the request on the lookup.
This matters for the OSS and polling-disabled configurations, where the retrieve
path is the only thing that accounts for a batch at all.
The mock merged every call into one shared dict, so a second routed retrieval would
overwrite the first and the assertions would still pass. Keep one frozen snapshot per
call and assert exactly one call, which also makes an unintended second retrieval a
failure rather than something the merge hides
get_configured_s3_bucket_name accepts the output bucket only from the immutable
_litellm_internal_model_credentials snapshot or AWS_S3_BUCKET_NAME. That refusal to read
litellm_params is deliberate: the bucket is what validate_managed_cloud_file_id checks a
file id against, so trusting a request-supplied value would let a caller redirect reads
to a bucket of their choosing
Two live entry points reach the Bedrock file-content transformation without ever building
that snapshot. The managed-files pre-call hook sets data["model"] for any id carrying
llm_output_file_id, which is every batch output, so get_file_content always takes the
model-routed branch; that branch called llm_router.afile_content directly, and
managed_files_obj.afile_content, the only caller that built the snapshot, is therefore
unreachable for batch output. CheckBatchCost spread the deployment credentials as plain
kwargs, and get_litellm_params does not carry s3_bucket_name across (gcs_bucket_name is
listed for exactly this reason, its S3 counterpart is not), so the poller lost the bucket
the same way
The result was that every completed Bedrock managed batch failed files.content with
"S3 bucket_name is required" and never had its cost tracked, leaving the row to be
re-polled every cycle. Both paths now resolve the deployment credentials and pass the
same MappingProxyType snapshot the managed-files hook already builds
* fix(proxy/batches): stop forwarding custom_llm_provider twice in list and cancel
The model-routing branches of list_batches and cancel_batch passed
custom_llm_provider as an explicit kwarg while also leaving it inside the dict
they splat, so every such call raised "got multiple values for keyword argument
'custom_llm_provider'" and returned a 500.
list_batches SCENARIO 2 called data.update(credentials) but never removed
custom_llm_provider before litellm.alist_batches(custom_llm_provider=..., **data);
it now uses prepare_data_with_credentials, the same helper the create and
retrieve branches already use, which pops it out.
cancel_batch SCENARIO 3 resolved the provider with
`provider or data.pop("custom_llm_provider", None) or ...`, so when the path
param provider was set the pop short-circuited and a body custom_llm_provider
stayed in data and collided with the explicit kwarg. The body value is now
popped unconditionally before the fallback chain, so the path param wins cleanly
and data no longer carries a duplicate.
Both paths already had strict-xfail regression tests documented "remove when
fixed"; those markers are dropped so the tests now guard the fix.
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
* fix(proxy/files): avoid duplicate custom_llm_provider in list
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
---------
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
require_managed_files was only checked on upload, so raw provider ids still
reached the batch, fine-tuning and vector store file routes. Ownership rows
exist only for managed ids, so those requests were forwarded under shared
credentials with no tenant check: knowing another tenant's id was enough to
read, run against, cancel or delete their object.
Generalise the file-id guard to validate_managed_id_requirement(resource_id,
resource_kind) and call it on batch create/retrieve/cancel, fine-tuning
create/retrieve/cancel (training_file and validation_file both) and the shared
vector store file id resolver. Behaviour is unchanged when the setting is off.
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (#30708)
OpenAI GPT-5 models require max_completion_tokens >= 16.
Health checks were using 5 (proxy/health_check.py) and 10
(health_check_helpers.py), causing failures on GPT-5 models.
Fixes#23836
* fix: increase health check max_tokens from 5 to 16 (#23836) (#26610)
GPT-5 models enforce a minimum of 16 for max_output_tokens. The current
default of 5 still causes health checks to fail for these models. Bump
the non-wildcard default to 16 — the smallest value that satisfies all
known provider minimums while keeping health checks lightweight.
Also tightens the wildcard test assertion from a weak disjunctive check
to strict key-absence.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (#30696)
* fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema.
* fix: remove async keyword from test.
* fix: make Bedrock Mantle Responses routing data-driven per model (#30700)
* Make Bedrock Mantle Responses routing data-driven per model
Route Bedrock Mantle models to the native Responses API based on each
model's price-map capability signal instead of a hardcoded model-name
heuristic, and derive the OpenAI-compatible base path segment per model.
Responses dispatch now selects the native config when the model advertises
responses support (/v1/responses in supported_endpoints, or mode=responses),
both overridable via register_model and proxy model_info. This enables
native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping
chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing
chat-completions emulation. Capability is per-model, so gpt-oss-120b routes
natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss
substring.
The wire path is a separate concern, driven by the existing
use_openai_responses_path flag rather than a model-name match: gpt-5.x and
gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat
config now derives its base from the same flag, fixing gemma-4
chat-completions requests that previously went to /v1 instead of /openai/v1.
Cost maps: add supported_endpoints to the gpt-oss entries (responses for the
non-safeguard variants, chat-only for safeguard) and supported_endpoints +
use_openai_responses_path to all three gemma-4 entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: move capability helper into bedrock_mantle package
Move the Responses capability check out of utils.py into
litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses,
alongside its companion wire-path helper mantle_base_segment. Both are now
pure functions of (model, model_cost): the price-map mode/supported_endpoints
read replaces the get_model_info call, so the rules are unit-testable without
patching global state and the Bedrock Mantle package is self-contained.
Use str | None instead of Optional[str] on the new signatures to satisfy the
ruff UP045 strict-rule gate. Add direct unit tests for both helpers.
Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b
now legitimately supports Responses, so it can no longer be the
"None after restore" vehicle; use the chat-only safeguard variant, which
isolates the register/restore effect from the model's own capability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)
The tiered cost calculator resolved a tier's per-token cost with
`tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or`
short-circuits on any falsy value, a tier that legitimately prices a
component at 0.0 (e.g. a free-cache-read tier with
cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated
as missing and silently billed at the full fallback rate
(input_cost_per_token / output_cost_per_token).
The flat-pricing path in the same module already handles this correctly
with an `is None` guard. Resolve tier costs through a small helper that
mirrors it, so 0.0 is honored at both the in-range and overflow sites.
No shipped model currently has a 0.0 tier cost, so this is a latent
defect; the fix makes the tiered path consistent with the flat path and
prevents over-charging the first time such a tier appears. Adds unit
tests covering the in-range and overflow paths, and drops an unused
import flagged by ruff in the touched test file.
* feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)
* fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (#30618)
In the messages->chat/completions bridge, translate_anthropic_tools_to_openai
merged every non-mapped tool key into the function parameters dict. The
Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object'
-> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type).
Exclude 'type' from the passthrough. Fixes#30557.
* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)
An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:
1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
engine, spawn new one.
2. The engine-death watcher sees that kill, assumes a crash, and calls
`attempt_db_reconnect(force=True)` (a different lock,
`_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
3. In-flight queries failing during the swap are classified as transport
errors and trigger their own `attempt_db_reconnect` -> recreate again.
Fix coordinates planned restarts across the wrapper and the watcher:
- PrismaWrapper records the old engine PID in `_expected_engine_deaths`
before killing it; all four watcher death-detectors (waitpid thread,
pidfd, already-dead probe, os.kill poll) consume that PID and skip the
reconnect instead of treating it as a crash.
- `recreate_prisma_client` now serializes through `_reconnection_lock` and
bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
as an optimistic-lock token, so racing/cascading recreates collapse into a
single restart (losers no-op). This closes the two-lock gap.
- The direct reconnect path probes the writer with SELECT 1 before
recreating; a healthy connection (e.g. engine already replaced by a
refresh) skips the recreate entirely.
- `_safe_refresh_token` coalesces: it skips when the current token still has
more than the refresh buffer of runway, so stacked triggers (proactive
loop + __getattr__ fallback) don't each restart the engine. An
`on_engine_replaced` hook re-arms the watcher on the new PID.
RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.
* feat(bedrock): support file content retrieval for batch output files (#30595)
Implements transform_file_content_request and transform_file_content_response
in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch
files. The request transform resolves the file id (direct s3:// URI or base64
unified id) to its S3 object, validates bucket and key prefix against the
server-configured bucket, and SigV4-signs an S3 GetObject using the same
credential and region resolution as the existing upload path. The credential
and region params are validated into a typed model at the boundary, so the only
untyped values left are the botocore signing primitives.
Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries
s3_bucket_name (previously dropped when building deployment credentials) and
the managed-files hook passes the deployment credential snapshot when routing
afile_content, so unified-id content retrieval works with per-model bucket
config instead of only the AWS_S3_BUCKET_NAME env var.
Preserves managed-file access control: the proxy file-content endpoint now
rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the
owner/team check that only runs for unified ids and let a caller read another
tenant's batch output by its object key. Managed outputs are reachable only
through their unified file id. The afile_content "not found" error now reports
the caller's unified id rather than the resolved internal S3 URI.
Fixes#16186, #15563
* fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)
* fix(oci): map Cohere tool array/object params to lowercase builtins
OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare
"List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema
arrays. MLflow {{trace}} judges trip this: their tools (get_root_span,
get_span) take an attributes_to_fetch array. The lowercase builtins list/dict
are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but
both are lowercased for consistency).
Verified live against us-chicago-1 (cohere.command-a-03-2025 and
command-latest). Adds a unit regression on the transformed parameterDefinitions
plus a gated integration test exercising an array-param tool end to end.
* fix(oci): make Cohere agentic tool-calling continuation work
Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges
drive once a tool has been executed and its result is fed back.
Request side: litellm pulled the last user message into the top-level `message`
and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that
("cannot specify message if the last entry in chat history contains tool
results"), and an empty message alone is rejected too ("message must be at least
1 token long or tool results must be specified"). OCI carries the current turn's
results in a dedicated top-level `toolResults` field. The Cohere transform now
sends an empty message, keeps the user turn in chatHistory, and puts the results
in `toolResults`, matching the langchain-oracle reference. Tool results are no
longer represented as chatHistory entries.
Response side: tool-grounded answers come back with citations carrying
`documentIds` (camelCase) and no `document_ids`, which made the required
`CohereCitation.document_ids` field fail validation and sink the whole response
parse. Those citations are never surfaced, so the field (and CohereSearchQuery's
generation_id) is now optional.
Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest),
single and multi-round tool loops. Adds unit regressions on the transformed
request shape and on citation parsing, plus gated integration tests for the
continuation.
* feat: integrate Repelloai Argus guardrail (#30673)
* feat(guardrails): add RepelloAI Argus guardrail integration (#1)
* feat(guardrails): add RepelloAI Argus guardrail integration
Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.
* fix(guardrails): harden RepelloAI Argus guardrail
- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
block on unknown/malformed verdicts so an API change can't silently
disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
expose unreachable_fallback in the config schema, and stop leaking the
raw provider response / exception strings to the client
* fix(guardrails): address RepelloAI Argus review feedback
- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description
* docs(guardrails): add RepelloAI Argus docs page and dashboard listing
- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution
* fix(guardrails): keep RepelloAI asset_id optional in config model
A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.
* Add comment for last user turn scanning
* feat(guardrails): harden repelloai scanning
* feat(guardrails): expand repelloai scanning to include tool definitions
Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.
* refactor(guardrails): fix and harden repelloai schema text extraction
- Fix duplicate text in _iter_schema_text: previously all dict values were
re-queued onto the stack even after scalar/list keys were already extracted
explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param
* fix(guardrails): resolve pyright type errors in repelloai guardrail
- Narrow async_handler.post return from Response|None to Response with
explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
type in pyright
* fix(guardrails/repello): include Responses API instructions field in prompt scan
The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.
* feat: add api_key to config model and read prompt from data dict
* fix(guardrails/repello): plug input_text and tool-call response bypass gaps
Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.
Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.
Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.
* fix(guardrails/repello): scan Responses API function_call output arguments
Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.
* refactor(guardrails/repello): clean up typing and remove lint-any workarounds
- Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout
- Use dict[str, object] instead of bare dict in all signatures
- Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly
- Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel
- Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks
- Use TypeAdapter.validate_json() instead of response.json() + manual dict construction
- Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any
- Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check
- Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType
- Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel]
* fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning
- Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate
- Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks
* refactor: modifications for lint check
* feat: add Pinstripes as an OpenAI-compatible provider (#30567)
* feat: add Pinstripes as an OpenAI-compatible provider
Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference
provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.)
with per-token pricing and no subscriptions.
Changes:
- `litellm/llms/openai_like/providers.json`: register pinstripes with
base_url, api_key_env, and max_completion_tokens→max_tokens mapping
- `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders
- `litellm/constants.py`: add to openai_compatible_providers and
openai_compatible_endpoints lists
- `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect
provider when api_base is "https://pinstripes.io/v1"
- `provider_endpoints_support.json`: document supported endpoints
- `tests/`: 7 unit tests covering provider registration, resolution,
URL auto-detection, api_base override, and Router config
Usage:
import litellm
response = litellm.completion(
model="pinstripes/ps/glm-4.5-air",
messages=[{"role": "user", "content": "Hello"}],
api_key=os.environ["PINSTRIPES_API_KEY"],
)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): resolve Greptile P1 review comments
- Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works
- Set responses: false in provider_endpoints_support.json — not actually wired up
- Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): add api_base_env and correct responses capability
- Add api_base_env: PINSTRIPES_API_BASE to providers.json
- Set responses: false in provider_endpoints_support.json
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): wire up Responses API — add supported_endpoints
Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so
JSONProviderRegistry.supports_responses_api returns true correctly,
matching what provider_endpoints_support.json advertises.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(pinstripes): enable embeddings endpoint
Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings.
Add /v1/embeddings to supported_endpoints and set embeddings: true.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json
Matches the file's existing convention. Flagged by Greptile review.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): set a2a: false — A2A protocol not implemented
All comparable JSON-configured providers (tensormesh, parasail, empiriolabs,
libertai, neosantara) have a2a: false. Pinstripes does not implement the
Google A2A protocol, so this should be false to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: inference_provider <max@redactedlab.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(rag): attach existing OpenAI file ids (#30628)
* fix(rag): attach existing OpenAI file ids
* chore: use modern typing in rag ingest fix
* chore: retrigger ci
* fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)
cache_control_injection_points was only consumed by the chat/completions
prompt-management hook; on the native Anthropic /v1/messages path it was
forwarded unused, so deployment-level cache injection was silently dropped
(cache_creation_input_tokens stayed 0 for Anthropic-native clients).
Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject
cache_control at block level for system / tools / message locations (the only
forms /v1/messages accepts), wire it into the native anthropic_messages
handler, and pop the param so it does not leak upstream as an unknown field.
A {location: message, role: system} config is redirected to the top-level
system prompt so the same YAML works on both endpoints.
Injection respects Anthropic's 4-block cache_control limit shared across
system, tools, and messages: client-supplied markers count toward the cap and
are never overwritten, a slot is reserved per Bedrock tool_config point, and
injection stops once the budget is exhausted. Locations this path cannot
represent (tool_config) are forwarded downstream instead of being silently
consumed, mirroring get_chat_completion_prompt's remaining_points pass-through.
Built on litellm_internal_staging. Refs BerriAI/litellm#30293
* fix(proxy): release budget reservation when a request is cancelled mid-flight (#30522)
* fix(proxy): release budget reservation on cancel when no chunk was delivered
The pre-call budget reservation increments the cross-pod spend counter by a
request's worst-case cost, then reconciles it on success (cost callback) or
error (failure hook). A client disconnect or timeout cancels the request and
surfaces as CancelledError / GeneratorExit, which neither path catches, so the
reservation leaks. Under a retry storm the leaked holds accumulate, pin the
counter above real spend, and return spurious 429 "Budget has been exceeded" to
keys whose spend is far below budget; the counter only recovers when its TTL
lapses, so the failure is intermittent and self-healing.
Release the reservation in async_streaming_data_generator (which the Anthropic
and Google SSE generators delegate to) on the (CancelledError, GeneratorExit)
path, alongside the existing max_parallel_requests release. release_budget_
reservation_on_cancel runs under asyncio.shield so it completes despite the
in-progress cancellation, is guarded by the reservation's finalized flag, and
swallows a failing release so it cannot replace the in-flight cancellation.
The refund is gated on whether a chunk reached the client. The flag is set
immediately before the yield, after the slow-path hook await: an async generator
suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk
sees it True (keep the hold), while a cancellation during the slow-path await
leaves it False (refund, nothing sent). A non-streaming cancellation delivers
nothing and a completed non-streaming response is reconciled by the success
callback, so neither needs a release here.
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(proxy): reconcile a cancelled reservation to input cost, not zero
A streaming request cancelled before the first chunk previously reconciled its
reservation to zero and finalized it. But by the time the generator is
consuming the response the provider call was already dispatched, so the input
tokens were billed even though no chunk reached the client, and the
success/failure cost callbacks are skipped on cancellation. Refunding to zero
let a caller send an expensive request and abort pre-token to dodge the input
charge.
Compute the request's input-token cost at reservation time and reconcile the
cancelled reservation to it instead of zero. The worst-case output portion of
the reservation is still released (so a legitimate mid-flight cancellation no
longer pins the counter and 429s the key), while the input the provider already
processed is charged.
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(caching): encode object name in GCS cache GET path (#30378)
GCS cache reads always missed when gcs_path was set. The GET methods
interpolated the object name directly into the URL path, while the GCS
JSON API requires it to be URL-encoded (a "/" must be sent as %2F).
With gcs_path configured the object name is "<prefix>/<sha256>", so the
raw slash produced a malformed object path and GCS returned 404. httpx
does not raise on 4xx, so the status_code == 200 check fell through and
get/async_get returned None, silently missing on every read. Without
gcs_path the key has no slash, which is why this went unnoticed.
Wrap the object name with urllib.parse.quote(..., safe="") in get_cache
and async_get_cache. Apply the same encoding to the name= query
parameter in set_cache and async_set_cache so the key written matches
the key read back.
Adds regression tests asserting the GET path and SET query are encoded
(%2F) when gcs_path is set, for both sync and async paths; these fail on
the unpatched code.
Fixes#30377
* chore: add soniox stt-async-v5 model (#30672)
* fix(proxy): include model group aliases in v1 model info (#30626)
* Include model group aliases in v1 model info
* Fix model info alias implementation
* removed extra blank line
* chore: rerun CI
* fix(lint): remove redundant noqa directive in proxy_cli.py
* fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme
* Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme"
This reverts commit 52c7a07777.
* Revert "fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)"
This reverts commit c9e8a177bd.
* Revert "fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)"
This reverts commit 85828da695.
* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)
An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:
1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
engine, spawn new one.
2. The engine-death watcher sees that kill, assumes a crash, and calls
`attempt_db_reconnect(force=True)` (a different lock,
`_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
3. In-flight queries failing during the swap are classified as transport
errors and trigger their own `attempt_db_reconnect` -> recreate again.
Fix coordinates planned restarts across the wrapper and the watcher:
- PrismaWrapper records the old engine PID in `_expected_engine_deaths`
before killing it; all four watcher death-detectors (waitpid thread,
pidfd, already-dead probe, os.kill poll) consume that PID and skip the
reconnect instead of treating it as a crash.
- `recreate_prisma_client` now serializes through `_reconnection_lock` and
bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
as an optimistic-lock token, so racing/cascading recreates collapse into a
single restart (losers no-op). This closes the two-lock gap.
- The direct reconnect path probes the writer with SELECT 1 before
recreating; a healthy connection (e.g. engine already replaced by a
refresh) skips the recreate entirely.
- `_safe_refresh_token` coalesces: it skips when the current token still has
more than the refresh buffer of runway, so stacked triggers (proactive
loop + __getattr__ fallback) don't each restart the engine. An
`on_engine_replaced` hook re-arms the watcher on the new PID.
RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.
* fix(lint): modernize type annotations in IAM-refresh prisma client files (UP006/UP045)
* Revert "feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)"
This reverts commit f530b2237c.
* Revert "fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)"
This reverts commit 4f58bd0df5.
* Revert "fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)"
This reverts commit 50f34e0b15.
* Revert "fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)"
This reverts commit 0544eed6ea.
* fix(bedrock_mantle): restore BedrockMantleAuthMixin and constants removed by routing rewrite
* fix(key management): restore exact /key/list user_id & key_alias matching by default (#30593)
Before substring search was added (commit 33bd570d5e), /key/list matched user_id
and key_alias exactly. That change made admin-authenticated calls substring-match
by default, breaking the prior contract: a caller passing an exact user_id as an
access filter (e.g. an integration scoping to one user with an admin key) then
received other users' keys -- user_id="alice" also returned "alice2",
"alice-test", etc. This is a cross-user key disclosure.
Make substring matching opt-in via a new admin-only substring_matching=true query
param; default to exact, restoring the prior behavior. The dashboard search box
(keyListCall) passes the flag so partial search still works. Non-admins remain
exact and scoped to their own keys.
Updates the proxy-behavior key_alias test to opt in and adds an exact-by-default
guard; adds list_keys unit coverage for the opt-in gate.
---------
Co-authored-by: perseus <51974392+tcconnally@users.noreply.github.com>
Co-authored-by: Hannah Smith <64043506+hannahmadison@users.noreply.github.com>
Co-authored-by: Charlie Patterson <Pattersoncharlesl@gmail.com>
Co-authored-by: Matthew Lapointe <mlapointe@alpha-sense.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com>
Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com>
Co-authored-by: Nitish Agarwal <1592163+nitishagar@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com>
Co-authored-by: AD Mohanraj <admohanraj@gmail.com>
Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com>
Co-authored-by: Lavish Bansal <lavish.bansal619@gmail.com>
Co-authored-by: max-amos <gruffulom@gmail.com>
Co-authored-by: inference_provider <max@redactedlab.com>
Co-authored-by: NK <93352237+Nithish-Yenaganti@users.noreply.github.com>
Co-authored-by: 安妮的心动录 <74543653+anneheartrecord@users.noreply.github.com>
Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com>
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Burak Ömür <burak.omur.1998@gmail.com>
Co-authored-by: Dan Lemon <daniel.lemon@amazee.io>
Co-authored-by: Vanika Dangi <166420943+vanika02@users.noreply.github.com>
Co-authored-by: Jay Gowdy <130084966+jgowdy-godaddy@users.noreply.github.com>
* fix(proxy): resolve list files credentials from team BYOK deployments
GET /v1/files without target_model_names now prefers the team's own
deployment (model_info.team_id) over shared global provider keys, so JWT
team auth lists files against the correct upstream account.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): scope list files credential lookup to team allowlist
Remove the unrestricted deployment scan that could leak global provider
keys to teams without access, normalize all-proxy-models to the team-scoped
model list, and fix TID251 violations by using dict instead of Dict/Any.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
