OpenAIFilesPurpose was missing evals, which OpenAI documents. The upload
route validates against that set, so POST /v1/files with purpose=evals was
already being rejected, and the new listing validator extended the same
rejection to GET /v1/files?purpose=evals, turning a purpose OpenAI accepts
into a hard 400. Nothing branches exhaustively on the type, so widening it
changes no routing.
The managed-file listing test fake only understood a created_by filter. The
OR filter a key carrying both a user_id and a team_id produces, the team_id
filter a service-account key produces, and the empty filter a proxy admin
produces all fell through it and returned every row, so the shapes most real
keys send went uncovered. The fake now applies the filter it is handed, and
the listing is tested against all three, including paging an OR filter
across a cursor.
Two docstrings claimed the continuation chunk bounds what a filtered page
costs. It bounds queries per row scanned; the walk is still linear in the
rows the caller owns.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
The managed hook returned the plain dict build_list_page builds, while
every other GET /v1/files path returns an SDK page object. A post-call
success hook or a logging callback that reads response.data off the
listing raised AttributeError as soon as a request took the managed path
FileListPage is a pydantic model over the same five fields, so hooks read
.data again and the response body does not move: jsonable_encoder gives
the same keys in the same order for the model and for the dict. It sits
in litellm.types.llms.openai because base_llm/files/transformation.py
already imports from there and cannot import proxy modules. It is
deliberately not subscriptable, since the provider-backed path returns a
page object that is not either, and dict access would be a third contract
to keep alive
Also reject a purpose the Files API never accepts. An unknown purpose
matches no row, so the listing answered an empty page for what is really
a bad request, while the upload route in this same file already refuses
those values against get_args(OpenAIFilesPurpose). The check runs before
the first query, and only in the managed hook, so providers that define
their own purposes keep them
Also put back the route's original except tail. Sending every error
through handle_exception_on_proxy changed error.type on a bad
target_model_names from "None" to the exception class name, which a
caller matching on the body would read as a break. create_file in this
file already pairs base's tail with a ProxyException passthrough, so
list_files does the same and the handle_exception_on_proxy import is gone
* test: use monkeypatch.setenv for env writes in tests/test_litellm
`os.environ["X"] = v` inside a test leaks the value into every test that runs
after it in the same worker, so ordering decides the result. 262 of those
writes across 40 files now go through pytest's `monkeypatch` fixture, which
restores the previous value at teardown.
The rewrite skips any test that a mock.patch-family decorator wraps, any test
with defaulted positional parameters, any test whose own name is called
directly elsewhere, and rebinds nothing inside nested defs, because in each of
those cases appending a fixture parameter changes what pytest or mock binds.
Ratchets the TQ004 ceiling from 768 to 506.
* fix(test): delete the key through monkeypatch instead of popping it first
Five tests popped a key straight out of `os.environ`, ran, then restored it with
`monkeypatch.setenv`. By the time monkeypatch saw the name it was already gone,
so it recorded "absent" as the value to go back to and deleted the key at
teardown. On a worker that inherited a real `RESEND_API_KEY`, `SENDGRID_API_KEY`,
`UI_PASSWORD`, `LITELLM_SALT_KEY` or `OPENAI_API_KEY`, every test after the first
one ran without it.
`monkeypatch.delenv(..., raising=False)` removes the key and restores whatever
was there, so the try/finally the manual restore needed goes with it.
* chore(test): leave the two cost-calc files to the PR that rewrites them fully
Both files are also in #37815, which converts the module-global writes as well
as the env writes and folds them into one fixture. Two PRs rewriting the same
lines differently is a conflict nobody benefits from resolving, so this one
drops back to staging on those two and keeps the other 39.
TQ004 clears 200 here instead of 275; the rest moves with #37815.
The chunk loop read `limit + 1` rows at a time, so a small limit whose
matches sit far behind the newest rows advanced a couple of rows per
query. A `purpose` that matches only the last of 10000 owned rows at
`limit=1` cost 5001 sequential find_many calls for one HTTP request,
which any authenticated caller could ask for on purpose.
Once a scan has to continue past its first chunk, widen the chunk to
FILE_LIST_CONTINUATION_CHUNK_SIZE. That same case now costs 21 queries.
The first chunk keeps its `limit + 1` size, so a page the newest rows
already fill still costs exactly one query and reads nothing extra.
Rows whose blob will not parse drop out of a page the way a filter does,
so they get the bound too, not just the purpose filter.
The floor only changes how many round trips a page costs, never what it
returns: chunk boundaries do not affect a keyset scan, so the page is
still `matches[:page_size]`, `has_more` is still `len(matches) >
page_size`, and empty data still implies `has_more` false.
The managed file listing cut the page to `limit` first and applied the
purpose filter in Python afterwards, so a page whose rows all failed the
filter came back as `data: []` with `has_more: true`. openai-python stops
paging the moment `data` is empty, so `files.list(purpose="batch", limit=1)`
returned nothing at all instead of every batch file.
Read successive keyset chunks until the page holds `limit + 1` matches or
the caller's rows run out, then return at most `limit` of them. `data` is
now non-empty whenever matching files remain, its last id is always a
usable cursor, and `has_more: false` only ever means the caller has seen
everything. Rows whose stored blob will not parse drop out in the same
loop, so they cannot empty a page either.
That also makes the `next_cursor_id` escape hatch on `build_list_page`
dead, so it goes back to what it was for the batch and vector-store
listings that share it.
Also move `validate_file_list_limit` up into the list_files route, so the
target_model_names and provider branches reject an out-of-range limit the
same way the managed file store already did.
The unscoped GET /v1/files limit check accepted 0, which OpenAI's minimum
of 1 does not allow, and the route's except block rebuilt every error with
getattr(e, "status_code", 500). ProxyException has no status_code, so the
400 it raises went out as a 500 and the OpenAI SDK retried it three times.
Errors now go through handle_exception_on_proxy, the helper the sibling
batches route already uses, and the unknown-cursor error is a ProxyException
so it carries type invalid_request_error and param after instead of the
literal "None". The cursor still 400s whether the file belongs to someone
else or does not exist at all
A page whose rows are all dropped by the purpose filter, or by a row
that does not parse, used to come back with an empty data list, has_more
true and last_id null, so the caller had no cursor to advance with and
stopped one page short of files it owns. last_id now falls back to the
last row the page read.
Also drops the OpenAIFilesPurpose import that the widened purpose
annotation left unused.
The owner-scoped listing read every row the caller owns in one query, so an
admin key that owns every file on the proxy pulled the whole table into one
response. Page it with a keyset cursor on unified_file_id instead, and accept
limit and after on GET /v1/files so a client can walk the pages. limit follows
what OpenAI documents for that route: 1 to 10000, default 10000.
An after cursor is resolved inside the caller's own scope, so an id they do not
own gets a 400 rather than a page, and has_more now reflects whether another
row exists instead of always being false.
Refs #37714
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
Every pod and uvicorn worker schedules its own CheckBatchCost poller against the
shared managed-object table, so two of them can select the same completed batch in
one polling window and both write an aretrieve_batch spend log for it, counting
that batch's cost twice.
Claim the row with a compare-and-swap on batch_processed, and skip the batch when
another pod already holds it. The claim sits immediately before the spend log is
written rather than before the results fetch, because batch_processed is also what
blocks deletion of the files the fetch reads and what keeps an unbilled row
selectable by later poll cycles, so claiming up front would strand the spend of any
worker that died mid-fetch. A failed spend log write hands the row back.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The retrieve path now defers a managed batch's accounting to CheckBatchCost, which
bills the key, team, and tags stored on the managed object row. The /v1/batches
create hook never persisted api_key or request_tags there (only the passthrough
creates did), so the poller attributed the cost to the user alone and the creating
key's spend stayed at zero.
The Anthropic batch create never persisted the creating key's hashed token or its
request tags on the managed object, so when CheckBatchCost billed the batch hours
later there was nothing to attribute it to. Key spend, key budgets and tag spend
never moved for batch usage.
Persist both from the create, the way the Vertex passthrough already does, and
register the batch only from the collection route. An id-scoped route cannot
rebuild the unified object id, because it embeds the model and the model comes
from the create's request body, so it could only claim a row it did not create or
fail the model_object_id unique constraint.
The shared metadata helpers, the route predicate and the registration-result
logging now live in batch_attribution instead of being copied per provider. The
Anthropic write previously logged success unconditionally, before the
fire-and-forget task had run.
Resolves LIT-5288
A poll of a Vertex passthrough batch wrote nothing to the managed-object row,
so status and file_object stayed frozen at the create-time snapshot and
GET /v1/batches served a stale status and an empty output file id for the life
of the batch. Only the create may claim a batch, but every observation of one
may refresh its state.
store_unified_object_id takes create_if_missing, which the poll clears: it
refreshes status and file_object through update_many, and leaves a row that is
absent absent rather than creating one owned by the observer, since created_by
and team_id are written by whoever reaches the create branch. The update payload
is now shared with the upsert so it cannot drift into writing api_key,
request_tags, created_by or team_id.
The passthrough identity re-assertion that was previously part of this PR ships
separately in #36121, so this PR keeps only the batch attribution work.
The creating key owns user_api_key_alias only when it actually has one. Guarding
the overwrite on the presence of a key rather than on a resolved alias nulled the
field out for every key generated without key_alias, and for any key rotated or
deleted before its batch finished, losing the creating user's alias that the spend
row previously carried. The guard now matches the team-alias line below it.
update_batch_in_database now fetches the batch row by unified_object_id
when the caller omits db_batch_object, so the cancel endpoint attributes
newly registered output and error files to the batch owner and returns
unified managed ids instead of raw provider ids. Idempotent cancels that
do not change the stored status also skip the redundant DB write now.
Repair two pre-existing mock tests in test_openai_batches_endpoint.py
that asserted values inside lazy percent-format log strings, and give
the cancel test's prisma mock an awaitable find_first.
The skip warning interpolated the full pydantic ValidationError, whose
string embeds input_value with the rejected row's contents. Managed-file
rows carry a caller-supplied filename, so a malformed row copied that
into operational logs.
Log the error locations, types, and messages via errors() with input,
url, and context excluded, keeping the field-level diagnostics without
the values. Non-validation failures fall back to the exception type.
get_user_created_file_ids validated every row's file_object without a
guard, so a single row failing OpenAIFileObject validation raised
ValidationError and turned the whole GET /v1/files response into a 500.
#35365 covered the null case only, leaving malformed or partial rows
able to take the entire listing down.
Rows now parse through a helper that returns None on failure and logs a
warning, matching how list_user_batches already tolerates rows it cannot
parse, so one bad row costs its own entry instead of the caller's whole
listing. Null rows stay silent since the batch cost poller registers
those legitimately.
Refs #35361
Terminal batch retrieve could return the raw provider output_file_id, which
skips managed-file ownership checks on /v1/files/{id}/content and lets any key
on the proxy download another user's batch output.
Retrieve now registers the missing managed-file row before responding, and
attributes ownership to the durable batch owner rather than the retrieving
caller, so output and error ids always come back as unified managed ids.
Fixes#33989
* refactor(ui): point invitation links at the dedicated /onboarding route
Invitation and reset-password links were built as /ui?invitation_id=..., which lands on the dashboard index and renders the onboarding form inline. They now point at the standalone /ui/onboarding route, so the index no longer has to special-case invitations. Old links keep working unchanged; the index still renders onboarding inline for ?invitation_id until the migration closeout removes that branch.
Updates the three generators (the enterprise email builder, bulk user create, and the invitation/reset-password modal) and extracts the modal's URL building into a pure, unit-tested buildOnboardingUrl
Refs LIT-3687
* refactor(ui): guard buildOnboardingUrl against a missing invitation id
Return "" instead of emitting an invitation_id=undefined link when the id is not yet available, matching the existing empty-baseUrl guard. Placed after the SSO branch so the SSO link, which does not use the id, is unaffected
Refs LIT-3687
* fix: prevent duplicate budget alert emails on concurrent threshold crossings
Budget alert emails were sent more than once for a single threshold crossing. The email dedup guard read the "already sent" marker, awaited the send, then wrote the marker, so concurrent requests crossing the same threshold within the send window all saw no marker and each sent. This affected the multi-threshold path (default_key_max_budget_alert_emails), the legacy single-threshold path (EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE), and the soft budget path, all in EmailBaseCallback.budget_alerts
All three branches now claim the send slot atomically before sending via async_increment_cache, which is atomic per event loop for the in-memory cache and across workers via Redis INCR; only the caller that observes a count of 1 sends. On send failure the marker is released with async_delete_cache so a transient failure does not suppress the alert for the full 24h TTL
* fix: harden budget alert claim release and skip-path event allocation
Addresses review feedback on the claim-before-send change. The claim release in each send-failure handler now logs the send error first and releases the claim best-effort through a shared helper, so a transient cache error during async_delete_cache cannot propagate out of the fire-and-forget budget_alerts task, drop the send-failure log, and leave the claim stuck for the full 24h TTL. In the multi-threshold branch the increment claim now runs before the WebhookEvent is built, so skipped concurrent crossings no longer construct and discard the event, matching the single-threshold and soft budget branches
* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (#30708)
OpenAI GPT-5 models require max_completion_tokens >= 16.
Health checks were using 5 (proxy/health_check.py) and 10
(health_check_helpers.py), causing failures on GPT-5 models.
Fixes#23836
* fix: increase health check max_tokens from 5 to 16 (#23836) (#26610)
GPT-5 models enforce a minimum of 16 for max_output_tokens. The current
default of 5 still causes health checks to fail for these models. Bump
the non-wildcard default to 16 — the smallest value that satisfies all
known provider minimums while keeping health checks lightweight.
Also tightens the wildcard test assertion from a weak disjunctive check
to strict key-absence.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (#30696)
* fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema.
* fix: remove async keyword from test.
* fix: make Bedrock Mantle Responses routing data-driven per model (#30700)
* Make Bedrock Mantle Responses routing data-driven per model
Route Bedrock Mantle models to the native Responses API based on each
model's price-map capability signal instead of a hardcoded model-name
heuristic, and derive the OpenAI-compatible base path segment per model.
Responses dispatch now selects the native config when the model advertises
responses support (/v1/responses in supported_endpoints, or mode=responses),
both overridable via register_model and proxy model_info. This enables
native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping
chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing
chat-completions emulation. Capability is per-model, so gpt-oss-120b routes
natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss
substring.
The wire path is a separate concern, driven by the existing
use_openai_responses_path flag rather than a model-name match: gpt-5.x and
gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat
config now derives its base from the same flag, fixing gemma-4
chat-completions requests that previously went to /v1 instead of /openai/v1.
Cost maps: add supported_endpoints to the gpt-oss entries (responses for the
non-safeguard variants, chat-only for safeguard) and supported_endpoints +
use_openai_responses_path to all three gemma-4 entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: move capability helper into bedrock_mantle package
Move the Responses capability check out of utils.py into
litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses,
alongside its companion wire-path helper mantle_base_segment. Both are now
pure functions of (model, model_cost): the price-map mode/supported_endpoints
read replaces the get_model_info call, so the rules are unit-testable without
patching global state and the Bedrock Mantle package is self-contained.
Use str | None instead of Optional[str] on the new signatures to satisfy the
ruff UP045 strict-rule gate. Add direct unit tests for both helpers.
Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b
now legitimately supports Responses, so it can no longer be the
"None after restore" vehicle; use the chat-only safeguard variant, which
isolates the register/restore effect from the model's own capability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)
The tiered cost calculator resolved a tier's per-token cost with
`tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or`
short-circuits on any falsy value, a tier that legitimately prices a
component at 0.0 (e.g. a free-cache-read tier with
cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated
as missing and silently billed at the full fallback rate
(input_cost_per_token / output_cost_per_token).
The flat-pricing path in the same module already handles this correctly
with an `is None` guard. Resolve tier costs through a small helper that
mirrors it, so 0.0 is honored at both the in-range and overflow sites.
No shipped model currently has a 0.0 tier cost, so this is a latent
defect; the fix makes the tiered path consistent with the flat path and
prevents over-charging the first time such a tier appears. Adds unit
tests covering the in-range and overflow paths, and drops an unused
import flagged by ruff in the touched test file.
* feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)
* fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (#30618)
In the messages->chat/completions bridge, translate_anthropic_tools_to_openai
merged every non-mapped tool key into the function parameters dict. The
Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object'
-> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type).
Exclude 'type' from the passthrough. Fixes#30557.
* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)
An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:
1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
engine, spawn new one.
2. The engine-death watcher sees that kill, assumes a crash, and calls
`attempt_db_reconnect(force=True)` (a different lock,
`_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
3. In-flight queries failing during the swap are classified as transport
errors and trigger their own `attempt_db_reconnect` -> recreate again.
Fix coordinates planned restarts across the wrapper and the watcher:
- PrismaWrapper records the old engine PID in `_expected_engine_deaths`
before killing it; all four watcher death-detectors (waitpid thread,
pidfd, already-dead probe, os.kill poll) consume that PID and skip the
reconnect instead of treating it as a crash.
- `recreate_prisma_client` now serializes through `_reconnection_lock` and
bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
as an optimistic-lock token, so racing/cascading recreates collapse into a
single restart (losers no-op). This closes the two-lock gap.
- The direct reconnect path probes the writer with SELECT 1 before
recreating; a healthy connection (e.g. engine already replaced by a
refresh) skips the recreate entirely.
- `_safe_refresh_token` coalesces: it skips when the current token still has
more than the refresh buffer of runway, so stacked triggers (proactive
loop + __getattr__ fallback) don't each restart the engine. An
`on_engine_replaced` hook re-arms the watcher on the new PID.
RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.
* feat(bedrock): support file content retrieval for batch output files (#30595)
Implements transform_file_content_request and transform_file_content_response
in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch
files. The request transform resolves the file id (direct s3:// URI or base64
unified id) to its S3 object, validates bucket and key prefix against the
server-configured bucket, and SigV4-signs an S3 GetObject using the same
credential and region resolution as the existing upload path. The credential
and region params are validated into a typed model at the boundary, so the only
untyped values left are the botocore signing primitives.
Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries
s3_bucket_name (previously dropped when building deployment credentials) and
the managed-files hook passes the deployment credential snapshot when routing
afile_content, so unified-id content retrieval works with per-model bucket
config instead of only the AWS_S3_BUCKET_NAME env var.
Preserves managed-file access control: the proxy file-content endpoint now
rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the
owner/team check that only runs for unified ids and let a caller read another
tenant's batch output by its object key. Managed outputs are reachable only
through their unified file id. The afile_content "not found" error now reports
the caller's unified id rather than the resolved internal S3 URI.
Fixes#16186, #15563
* fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)
* fix(oci): map Cohere tool array/object params to lowercase builtins
OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare
"List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema
arrays. MLflow {{trace}} judges trip this: their tools (get_root_span,
get_span) take an attributes_to_fetch array. The lowercase builtins list/dict
are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but
both are lowercased for consistency).
Verified live against us-chicago-1 (cohere.command-a-03-2025 and
command-latest). Adds a unit regression on the transformed parameterDefinitions
plus a gated integration test exercising an array-param tool end to end.
* fix(oci): make Cohere agentic tool-calling continuation work
Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges
drive once a tool has been executed and its result is fed back.
Request side: litellm pulled the last user message into the top-level `message`
and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that
("cannot specify message if the last entry in chat history contains tool
results"), and an empty message alone is rejected too ("message must be at least
1 token long or tool results must be specified"). OCI carries the current turn's
results in a dedicated top-level `toolResults` field. The Cohere transform now
sends an empty message, keeps the user turn in chatHistory, and puts the results
in `toolResults`, matching the langchain-oracle reference. Tool results are no
longer represented as chatHistory entries.
Response side: tool-grounded answers come back with citations carrying
`documentIds` (camelCase) and no `document_ids`, which made the required
`CohereCitation.document_ids` field fail validation and sink the whole response
parse. Those citations are never surfaced, so the field (and CohereSearchQuery's
generation_id) is now optional.
Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest),
single and multi-round tool loops. Adds unit regressions on the transformed
request shape and on citation parsing, plus gated integration tests for the
continuation.
* feat: integrate Repelloai Argus guardrail (#30673)
* feat(guardrails): add RepelloAI Argus guardrail integration (#1)
* feat(guardrails): add RepelloAI Argus guardrail integration
Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.
* fix(guardrails): harden RepelloAI Argus guardrail
- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
block on unknown/malformed verdicts so an API change can't silently
disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
expose unreachable_fallback in the config schema, and stop leaking the
raw provider response / exception strings to the client
* fix(guardrails): address RepelloAI Argus review feedback
- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description
* docs(guardrails): add RepelloAI Argus docs page and dashboard listing
- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution
* fix(guardrails): keep RepelloAI asset_id optional in config model
A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.
* Add comment for last user turn scanning
* feat(guardrails): harden repelloai scanning
* feat(guardrails): expand repelloai scanning to include tool definitions
Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.
* refactor(guardrails): fix and harden repelloai schema text extraction
- Fix duplicate text in _iter_schema_text: previously all dict values were
re-queued onto the stack even after scalar/list keys were already extracted
explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param
* fix(guardrails): resolve pyright type errors in repelloai guardrail
- Narrow async_handler.post return from Response|None to Response with
explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
type in pyright
* fix(guardrails/repello): include Responses API instructions field in prompt scan
The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.
* feat: add api_key to config model and read prompt from data dict
* fix(guardrails/repello): plug input_text and tool-call response bypass gaps
Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.
Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.
Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.
* fix(guardrails/repello): scan Responses API function_call output arguments
Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.
* refactor(guardrails/repello): clean up typing and remove lint-any workarounds
- Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout
- Use dict[str, object] instead of bare dict in all signatures
- Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly
- Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel
- Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks
- Use TypeAdapter.validate_json() instead of response.json() + manual dict construction
- Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any
- Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check
- Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType
- Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel]
* fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning
- Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate
- Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks
* refactor: modifications for lint check
* feat: add Pinstripes as an OpenAI-compatible provider (#30567)
* feat: add Pinstripes as an OpenAI-compatible provider
Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference
provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.)
with per-token pricing and no subscriptions.
Changes:
- `litellm/llms/openai_like/providers.json`: register pinstripes with
base_url, api_key_env, and max_completion_tokens→max_tokens mapping
- `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders
- `litellm/constants.py`: add to openai_compatible_providers and
openai_compatible_endpoints lists
- `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect
provider when api_base is "https://pinstripes.io/v1"
- `provider_endpoints_support.json`: document supported endpoints
- `tests/`: 7 unit tests covering provider registration, resolution,
URL auto-detection, api_base override, and Router config
Usage:
import litellm
response = litellm.completion(
model="pinstripes/ps/glm-4.5-air",
messages=[{"role": "user", "content": "Hello"}],
api_key=os.environ["PINSTRIPES_API_KEY"],
)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): resolve Greptile P1 review comments
- Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works
- Set responses: false in provider_endpoints_support.json — not actually wired up
- Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): add api_base_env and correct responses capability
- Add api_base_env: PINSTRIPES_API_BASE to providers.json
- Set responses: false in provider_endpoints_support.json
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): wire up Responses API — add supported_endpoints
Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so
JSONProviderRegistry.supports_responses_api returns true correctly,
matching what provider_endpoints_support.json advertises.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(pinstripes): enable embeddings endpoint
Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings.
Add /v1/embeddings to supported_endpoints and set embeddings: true.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json
Matches the file's existing convention. Flagged by Greptile review.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): set a2a: false — A2A protocol not implemented
All comparable JSON-configured providers (tensormesh, parasail, empiriolabs,
libertai, neosantara) have a2a: false. Pinstripes does not implement the
Google A2A protocol, so this should be false to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: inference_provider <max@redactedlab.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(rag): attach existing OpenAI file ids (#30628)
* fix(rag): attach existing OpenAI file ids
* chore: use modern typing in rag ingest fix
* chore: retrigger ci
* fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)
cache_control_injection_points was only consumed by the chat/completions
prompt-management hook; on the native Anthropic /v1/messages path it was
forwarded unused, so deployment-level cache injection was silently dropped
(cache_creation_input_tokens stayed 0 for Anthropic-native clients).
Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject
cache_control at block level for system / tools / message locations (the only
forms /v1/messages accepts), wire it into the native anthropic_messages
handler, and pop the param so it does not leak upstream as an unknown field.
A {location: message, role: system} config is redirected to the top-level
system prompt so the same YAML works on both endpoints.
Injection respects Anthropic's 4-block cache_control limit shared across
system, tools, and messages: client-supplied markers count toward the cap and
are never overwritten, a slot is reserved per Bedrock tool_config point, and
injection stops once the budget is exhausted. Locations this path cannot
represent (tool_config) are forwarded downstream instead of being silently
consumed, mirroring get_chat_completion_prompt's remaining_points pass-through.
Built on litellm_internal_staging. Refs BerriAI/litellm#30293
* fix(proxy): release budget reservation when a request is cancelled mid-flight (#30522)
* fix(proxy): release budget reservation on cancel when no chunk was delivered
The pre-call budget reservation increments the cross-pod spend counter by a
request's worst-case cost, then reconciles it on success (cost callback) or
error (failure hook). A client disconnect or timeout cancels the request and
surfaces as CancelledError / GeneratorExit, which neither path catches, so the
reservation leaks. Under a retry storm the leaked holds accumulate, pin the
counter above real spend, and return spurious 429 "Budget has been exceeded" to
keys whose spend is far below budget; the counter only recovers when its TTL
lapses, so the failure is intermittent and self-healing.
Release the reservation in async_streaming_data_generator (which the Anthropic
and Google SSE generators delegate to) on the (CancelledError, GeneratorExit)
path, alongside the existing max_parallel_requests release. release_budget_
reservation_on_cancel runs under asyncio.shield so it completes despite the
in-progress cancellation, is guarded by the reservation's finalized flag, and
swallows a failing release so it cannot replace the in-flight cancellation.
The refund is gated on whether a chunk reached the client. The flag is set
immediately before the yield, after the slow-path hook await: an async generator
suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk
sees it True (keep the hold), while a cancellation during the slow-path await
leaves it False (refund, nothing sent). A non-streaming cancellation delivers
nothing and a completed non-streaming response is reconciled by the success
callback, so neither needs a release here.
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(proxy): reconcile a cancelled reservation to input cost, not zero
A streaming request cancelled before the first chunk previously reconciled its
reservation to zero and finalized it. But by the time the generator is
consuming the response the provider call was already dispatched, so the input
tokens were billed even though no chunk reached the client, and the
success/failure cost callbacks are skipped on cancellation. Refunding to zero
let a caller send an expensive request and abort pre-token to dodge the input
charge.
Compute the request's input-token cost at reservation time and reconcile the
cancelled reservation to it instead of zero. The worst-case output portion of
the reservation is still released (so a legitimate mid-flight cancellation no
longer pins the counter and 429s the key), while the input the provider already
processed is charged.
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(caching): encode object name in GCS cache GET path (#30378)
GCS cache reads always missed when gcs_path was set. The GET methods
interpolated the object name directly into the URL path, while the GCS
JSON API requires it to be URL-encoded (a "/" must be sent as %2F).
With gcs_path configured the object name is "<prefix>/<sha256>", so the
raw slash produced a malformed object path and GCS returned 404. httpx
does not raise on 4xx, so the status_code == 200 check fell through and
get/async_get returned None, silently missing on every read. Without
gcs_path the key has no slash, which is why this went unnoticed.
Wrap the object name with urllib.parse.quote(..., safe="") in get_cache
and async_get_cache. Apply the same encoding to the name= query
parameter in set_cache and async_set_cache so the key written matches
the key read back.
Adds regression tests asserting the GET path and SET query are encoded
(%2F) when gcs_path is set, for both sync and async paths; these fail on
the unpatched code.
Fixes#30377
* chore: add soniox stt-async-v5 model (#30672)
* fix(proxy): include model group aliases in v1 model info (#30626)
* Include model group aliases in v1 model info
* Fix model info alias implementation
* removed extra blank line
* chore: rerun CI
* fix(lint): remove redundant noqa directive in proxy_cli.py
* fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme
* Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme"
This reverts commit 52c7a07777.
* Revert "fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)"
This reverts commit c9e8a177bd.
* Revert "fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)"
This reverts commit 85828da695.
* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)
An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:
1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
engine, spawn new one.
2. The engine-death watcher sees that kill, assumes a crash, and calls
`attempt_db_reconnect(force=True)` (a different lock,
`_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
3. In-flight queries failing during the swap are classified as transport
errors and trigger their own `attempt_db_reconnect` -> recreate again.
Fix coordinates planned restarts across the wrapper and the watcher:
- PrismaWrapper records the old engine PID in `_expected_engine_deaths`
before killing it; all four watcher death-detectors (waitpid thread,
pidfd, already-dead probe, os.kill poll) consume that PID and skip the
reconnect instead of treating it as a crash.
- `recreate_prisma_client` now serializes through `_reconnection_lock` and
bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
as an optimistic-lock token, so racing/cascading recreates collapse into a
single restart (losers no-op). This closes the two-lock gap.
- The direct reconnect path probes the writer with SELECT 1 before
recreating; a healthy connection (e.g. engine already replaced by a
refresh) skips the recreate entirely.
- `_safe_refresh_token` coalesces: it skips when the current token still has
more than the refresh buffer of runway, so stacked triggers (proactive
loop + __getattr__ fallback) don't each restart the engine. An
`on_engine_replaced` hook re-arms the watcher on the new PID.
RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.
* fix(lint): modernize type annotations in IAM-refresh prisma client files (UP006/UP045)
* Revert "feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)"
This reverts commit f530b2237c.
* Revert "fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)"
This reverts commit 4f58bd0df5.
* Revert "fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)"
This reverts commit 50f34e0b15.
* Revert "fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)"
This reverts commit 0544eed6ea.
* fix(bedrock_mantle): restore BedrockMantleAuthMixin and constants removed by routing rewrite
* fix(key management): restore exact /key/list user_id & key_alias matching by default (#30593)
Before substring search was added (commit 33bd570d5e), /key/list matched user_id
and key_alias exactly. That change made admin-authenticated calls substring-match
by default, breaking the prior contract: a caller passing an exact user_id as an
access filter (e.g. an integration scoping to one user with an admin key) then
received other users' keys -- user_id="alice" also returned "alice2",
"alice-test", etc. This is a cross-user key disclosure.
Make substring matching opt-in via a new admin-only substring_matching=true query
param; default to exact, restoring the prior behavior. The dashboard search box
(keyListCall) passes the flag so partial search still works. Non-admins remain
exact and scoped to their own keys.
Updates the proxy-behavior key_alias test to opt in and adds an exact-by-default
guard; adds list_keys unit coverage for the opt-in gate.
---------
Co-authored-by: perseus <51974392+tcconnally@users.noreply.github.com>
Co-authored-by: Hannah Smith <64043506+hannahmadison@users.noreply.github.com>
Co-authored-by: Charlie Patterson <Pattersoncharlesl@gmail.com>
Co-authored-by: Matthew Lapointe <mlapointe@alpha-sense.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com>
Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com>
Co-authored-by: Nitish Agarwal <1592163+nitishagar@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com>
Co-authored-by: AD Mohanraj <admohanraj@gmail.com>
Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com>
Co-authored-by: Lavish Bansal <lavish.bansal619@gmail.com>
Co-authored-by: max-amos <gruffulom@gmail.com>
Co-authored-by: inference_provider <max@redactedlab.com>
Co-authored-by: NK <93352237+Nithish-Yenaganti@users.noreply.github.com>
Co-authored-by: 安妮的心动录 <74543653+anneheartrecord@users.noreply.github.com>
Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com>
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Burak Ömür <burak.omur.1998@gmail.com>
Co-authored-by: Dan Lemon <daniel.lemon@amazee.io>
Co-authored-by: Vanika Dangi <166420943+vanika02@users.noreply.github.com>
Co-authored-by: Jay Gowdy <130084966+jgowdy-godaddy@users.noreply.github.com>
* fix(proxy): skip double-wrapping unified batch output file ids on retrieve
After ensure_batch_response_managed_file_ids normalizes output_file_id, the managed files post-call hook was re-encoding the unified id and storing the nested id as the provider mapping. Use the decoded llm_output_file_id for retrieve and model_mappings instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): guard managed file id parsing for non-output unified formats
Only treat decoded unified ids as already-wrapped output files when they contain llm_output_file_id. Skip other litellm_proxy id shapes instead of IndexError on split.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): rename loop variable to satisfy mypy unified file id typing
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): normalize batch file IDs before ManagedObjectTable write
Run post_call_success_hook before update_batch_in_database on retrieve/cancel,
and ensure_batch_response_managed_file_ids so file_object never stores raw
provider output_file_id or error_file_id.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): address Greptile review on batch file ID normalization
Remove redundant resolve_* calls after update_batch_in_database and rename
loop variable to avoid shadowing hidden_params unified_file_id.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): add mistral/ministral-8b-2512 to cost map and backfill in conftest
Mistral rotated the 'mistral/mistral-tiny' alias to return
'ministral-8b-2512' as the response model, which was missing from the
cost map. This caused test_completion_mistral_api and
test_completion_mistral_api_modified_input to fail in
litellm.completion_cost lookup.
- Add mistral/ministral-8b-2512 entry to both the in-tree
model_prices_and_context_window.json and the bundled
litellm/model_prices_and_context_window_backup.json (mirrors the
existing openrouter/mistralai/ministral-8b-2512 pricing).
- litellm.model_cost is loaded at import time from the URL pinned to
main, so the new backup entry isn't visible at test runtime until
it also lands on main. Backfill any entries missing from the
remote-fetched map into litellm.model_cost in the local_testing
conftest so cost-calculator lookups succeed on this branch.
* fix(tests): drop unnecessary del of conftest backfill loop vars
* fix: resolve batch response file IDs even when status unchanged
The status-unchanged early return in update_batch_in_database was
skipping ensure_batch_response_managed_file_ids, leaving raw provider
input_file_id (and other raw IDs) in the user-facing response when
polling an in-progress batch. Move the in-place file ID normalization
above the early return so the response always carries unified managed
IDs while still skipping the DB write when nothing changed.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(batches): cover ensure_batch_response_managed_file_ids branches
Add tests for the previously-uncovered paths in
ensure_batch_response_managed_file_ids: error_file_id normalization,
swallowed conversion errors, UserAPIKeyAuth fallback from
db_batch_object, model_name resolution from unified_file_id, and early
returns when managed_files_obj, model_id, or auth context are missing.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(managed_batches): convert raw output_file_id to managed ID in CheckBatchCost poller
CheckBatchCost bypasses async_post_call_success_hook, causing raw provider
output_file_ids to be persisted in LiteLLM_ManagedObjectTable. This fix converts
output_file_id and error_file_id to managed base64 IDs before the DB write.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(check_batch_cost): persist managed file before mutating response and propagate team_id
- Move setattr after store_unified_file_id so the response only receives the
managed ID once the DB record is successfully written. Avoids serializing
an orphaned managed ID into file_object when the store call fails.
- Populate team_id on the minimal UserAPIKeyAuth from job.team_id so the
managed file record is created with the correct team ownership, allowing
other team members to access the batch output file via /files/{id}/content.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(managed_batches): extend test to cover error_file_id conversion
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix managed file test
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The enterprise package is installed as `litellm_enterprise` (per
enterprise/pyproject.toml), but several tests imported it as
`enterprise.litellm_enterprise.*` — a path that only resolves
because the repo root happens to sit on sys.path, letting Python's
implicit namespace package machinery discover `enterprise/` as a
directory.
This breaks any test runner that relocates source (e.g. the
mutation-testing workflow, which copies tests under `mutants/`) and
also caused two `patch()` strings to target a module path that does
not match what production code imports — meaning those mocks were
never actually patching the production module's attribute.
Replace `from enterprise.litellm_enterprise.` with the canonical
`from litellm_enterprise.` across 6 test files, and fix two
`patch()` target strings (and one `sys.modules` patch key in the
SSO test) to match.
Service-account API keys are issued without a `user_id`, and managed
file/batch/vector-store ownership checks compared
`resource.created_by == user_api_key_dict.user_id`. Because Python
evaluates `None == None` as True, any service-account key passed
ownership checks for any resource also created without a user id, and
listing endpoints skipped the `created_by` filter entirely when the
caller had no user id — returning every tenant's records.
Replace the bare equality with an identity-aware helper:
- Admins (PROXY_ADMIN, PROXY_ADMIN_VIEW_ONLY) keep their unscoped view.
- Callers with a `user_id` are scoped to records they created.
- Callers without a `user_id` but with a `team_id` are scoped to records
created within their team via a new `created_by_team_id` column.
- Callers with no admin role and no identifying ids are denied — the
listing path returns an empty page without issuing a query.
Schema migration adds `created_by_team_id` to LiteLLM_ManagedFileTable,
LiteLLM_ManagedObjectTable, and LiteLLM_ManagedVectorStoreTable, plus
indexes for the new filter. Writes in BaseManagedResource and the
enterprise managed_files hook now stamp the column from
`user_api_key_dict.team_id`. Reads in `can_user_access_unified_resource_id`,
`can_user_call_unified_file_id`, `can_user_call_unified_object_id`,
`list_user_resources`, `list_user_batches`, and `get_user_created_file_ids`
all delegate to the new helper.
Tests cover the helper in isolation, the base-class listing/access paths,
and the enterprise file-access hook (including a regression test for the
original `None == None` bypass).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Users can set metadata.max_budget_alert_emails as a JSON map of threshold
percentages to email recipients on virtual keys. When configured, the email
handler loops over each threshold, checks per-threshold dedup cache, and
sends to the configured recipients (auto-including the key owner's email).
When no map is set, the existing single 80% threshold behavior is preserved
unchanged. Teams support is out of scope for this v0.
- Remove unused `completed_jobs` list (dead code after per-job update refactor)
- Wrap DB update in try/except to prevent one failed update from aborting remaining jobs
- Add test assertions verifying batch_processed, status, and file_object are written to DB
CheckBatchCost poller updated the status column but not the file_object
JSON column. The list_batches endpoint reads status from file_object,
so batches appeared stuck in "validating" even after Azure reported
them as completed. Now update file_object alongside status in the
per-job DB write.
