The skip warning interpolated the full pydantic ValidationError, whose
string embeds input_value with the rejected row's contents. Managed-file
rows carry a caller-supplied filename, so a malformed row copied that
into operational logs.
Log the error locations, types, and messages via errors() with input,
url, and context excluded, keeping the field-level diagnostics without
the values. Non-validation failures fall back to the exception type.
get_user_created_file_ids validated every row's file_object without a
guard, so a single row failing OpenAIFileObject validation raised
ValidationError and turned the whole GET /v1/files response into a 500.
#35365 covered the null case only, leaving malformed or partial rows
able to take the entire listing down.
Rows now parse through a helper that returns None on failure and logs a
warning, matching how list_user_batches already tolerates rows it cannot
parse, so one bad row costs its own entry instead of the caller's whole
listing. Null rows stay silent since the batch cost poller registers
those legitimately.
Refs #35361
* fix(proxy): give proxy_admin_viewer read parity with proxy_admin
Route-level checks already default-allow management GETs for the viewer
role, but ~15 handlers compared user_role to PROXY_ADMIN only, dropping
viewers into regular-user scoping (/key/list, /user/info, /model/info,
guardrails, prompts, agents, memory, workflows, MCP catalog, coordination
redis settings, credential migration check, enterprise projects). Swap
those read paths to user_api_key_has_admin_view; write gates unchanged.
The dashboard now presents the viewer session as Admin for all gating
(effectiveSessionRole) so every page fetches with admin visibility, with
userRoleLabel/isViewOnly preserving the account-menu label and the
playground cost guard. The server remains the write authority.
* refactor(agents): remove side-effectful health_check param from GET /v1/agents
Addresses a security review finding on the admin viewer read parity change:
listing agents with health_check=true made the proxy issue a server-side GET
to every agent URL, so a read-scoped caller could trigger request fan-out
beyond their object permissions. The list endpoint is now a pure read for
every role.
Removes the query param, the URL probing helper and its timeouts, the
AgentHealthCheck httpx provider tag, and the dashboard's Health Check
toggle. Requests still passing health_check=true get the full list back
with the param ignored.
* fix(proxy): keep credential encryption check proxy_admin only
The residual scan behind GET /credentials/migrate-encryption/check loads
every model, credential, MCP, team, and verification-token row and runs a
decryption attempt on each stored value. Extending it to proxy_admin_viewer
let a read-only account repeatedly trigger deployment-wide scans, so the
route keeps its original full-admin gate.
* fix(agents): restore health_check, keep list fast path proxy_admin only
Restores the agent health_check feature exactly as before this PR: the
query param, the URL probing helper, the httpx provider tag, and the
dashboard toggle all return, so existing callers keep the filtering
contract. The viewer expansion is instead reverted at its source: the
GET /v1/agents admin fast path stays PROXY_ADMIN only, so a
proxy_admin_viewer goes through the object-permission scoped branch as
before and cannot fan out health checks beyond their allowlist. The
viewer read of a single agent stays viewer-inclusive since it has no
side effects.
LiteLLM_ManagedObjectTable only stores created_by (user_id) and team_id,
never the raw API key hash. A batch created with the master key or a
team-less key has both null, so CheckBatchCost's synthetic logging_obj
for the completed batch carried no attributable key/user/team/end-user.
_should_track_cost_callback silently skipped the DB write in that case
(by design, to avoid tracking truly anonymous requests), with no error
or warning: batch_processed still became true, but no LiteLLM_SpendLogs
row was ever written despite real, already-incurred provider cost.
Extend the same allowance already made for unauthenticated pass-through
requests to aretrieve_batch's cost event, and pass job.team_id through
so a batch's team gets real attribution when one exists.
CheckBatchCost built unified output file ids with the provider model name, so key model-access checks resolved the file to e.g. gpt-5.5 and every GET /v1/files/{output_file_id}/content failed. Resolve the model group from the batch's managed input file, falling back to the deployment's model_name.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Swap `Model(**payload)` for `Model.model_validate(payload)` at the seams where
the payload comes back untyped, so basedpyright stops widening every target
field to Any. None of the models involved override `__init__`, so validation
goes through the same core validator either way.
Also route UserRepository through its own typed helpers (find_many, update,
find_by_id) instead of the raw Prisma table, drop the redundant `_to_model`
override signature, and call generate_key_helper_fn with explicit arguments in
the SSO callback rather than splatting an untyped dict.
Whole-tree basedpyright: reportAny 21481 -> 20834, reportExplicitAny 7258 ->
7252, with every other rule unchanged or lower.
Replace Any-typed seams with real types in the files carrying the highest
reportAny/reportExplicitAny density: typed Prisma read helpers in the MCP
db layer and verification token repository, TypedDicts for OAuth credential
payloads and aggregated spend rows, a DailySpendRecord protocol for the
daily activity endpoints, and concrete request/response types in the
volcengine, openai evals, azure batches, azure_ai count_tokens, and ocr
transformation modules. Modernize touched annotations to PEP 604/585 forms.
No casts, no type: ignore, no noqa, no new Any annotations, no behavior
changes. Whole-tree basedpyright: reportAny 27,005 -> 24,427,
reportExplicitAny 7,439 -> 7,280, no rule increased anywhere. Budgets
ratcheted: basedpyright -2,869, ruff-strict -1,505, type-discipline -167.
Validating the cursor whenever `after` was non-None turned `?after=` into a
400, which the listing has always read as "no cursor". Only a cursor the
client actually sent is looked up now, matching the sibling managed-resource
listing.
An `after` that does not resolve to a batch the caller can list now returns
400 instead of an empty page. An empty page is indistinguishable from the end
of the list, so a stale or malformed cursor silently truncated a client's batch
list. The lookup is scoped to the caller's own rows, so a Prisma cursor can no
longer be anchored to another user's batch.
`has_more` now comes from whether an extra row exists rather than from whether
the page came back full. Reporting fullness made every client fetch one extra
empty page when the batch count was an exact multiple of `limit`, and made a
page shortened by an unparseable row look like the end of the list, hiding the
older batches behind it.
Also drops the unreachable `target_model_names` oversampling branch; that
argument raises a few lines above it.
GET /batches served from the managed-objects table paged with a
where id > after filter, but the after cursor clients send back is a
batch's unified_object_id (the value returned as .id and last_id), and
id is the table's random-uuid primary key. Comparing the two unrelated
fields, while ordering by created_at desc but filtering with gt, made
pages repeat the same last_id (pagination loops) and silently drop
batches. Switch to Prisma cursor pagination on the unique
unified_object_id column so listing walks every batch exactly once in
reverse-chronological order, matching OpenAI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(logging): add user and team level spend and budget to StandardLoggingPayload metadata
* fix(logging): include user and team budget fields in dummy standard logging payload
* feat(batches): track cost for unmanaged Bedrock batches, generalize the flag
CheckBatchCost skipped Bedrock batches whose unified_object_id is a raw
model-invocation-job ARN, the same root cause previously fixed for
unmanaged Vertex batches. Bedrock batches embed the model name in their
s3:// input file name instead (litellm-bedrock-files-{model}-{uuid}.jsonl),
so the same routing mechanism now derives the model from that layout and
matches it to a configured bedrock deployment.
track_unmanaged_vertex_batch_cost is renamed to track_unmanaged_batch_cost
since two providers now share this mechanism.
* fix(batches): parse Bedrock batch output and price with deployment model name
Bedrock model-invocation-job results use modelOutput/error rows and short
internal model ids that are not in the cost map, so unmanaged batch cost
tracking logged tokens but $0 spend. Use deployment model name for pricing
and add regression tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(ui): point invitation links at the dedicated /onboarding route
Invitation and reset-password links were built as /ui?invitation_id=..., which lands on the dashboard index and renders the onboarding form inline. They now point at the standalone /ui/onboarding route, so the index no longer has to special-case invitations. Old links keep working unchanged; the index still renders onboarding inline for ?invitation_id until the migration closeout removes that branch.
Updates the three generators (the enterprise email builder, bulk user create, and the invitation/reset-password modal) and extracts the modal's URL building into a pure, unit-tested buildOnboardingUrl
Refs LIT-3687
* refactor(ui): guard buildOnboardingUrl against a missing invitation id
Return "" instead of emitting an invitation_id=undefined link when the id is not yet available, matching the existing empty-baseUrl guard. Placed after the SSO branch so the SSO link, which does not use the id, is unaffected
Refs LIT-3687
* fix(batches): price anthropic passthrough message batches correctly in batch cost job
Anthropic message batches created via the /anthropic passthrough were never
cost tracked. The CheckBatchCost job fetched batch results from the Files API
(POST /v1/files/msgbatch_.../content), which Anthropic rejects with "File id
must have file_ prefix"; the error response was silently wrapped as file
content, parsed as zero successful rows, logged as a $0 aretrieve_batch spend
row, and the job was marked batch_processed=true so the $0 was permanent.
Route msgbatch_ file ids to GET /v1/messages/batches/{id}/results in the
anthropic files transformation, raise on HTTP error status in
retrieve_file_content instead of returning the error body as content, parse
Anthropic's results JSONL shape (result.type == "succeeded",
result.message.usage with cache creation/read tokens) in batch_utils, price
cache creation tokens at cache_creation_input_token_cost in the batch cost
fallback (50% batch discount preserved for base input, cache reads, cache
writes, and output), and leave the managed object row unprocessed when cost
tracking fails so a later poll retries instead of permanently recording $0.
* fix(batches): carry cache token details into aggregated anthropic batch usage
* fix: prevent duplicate budget alert emails on concurrent threshold crossings
Budget alert emails were sent more than once for a single threshold crossing. The email dedup guard read the "already sent" marker, awaited the send, then wrote the marker, so concurrent requests crossing the same threshold within the send window all saw no marker and each sent. This affected the multi-threshold path (default_key_max_budget_alert_emails), the legacy single-threshold path (EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE), and the soft budget path, all in EmailBaseCallback.budget_alerts
All three branches now claim the send slot atomically before sending via async_increment_cache, which is atomic per event loop for the in-memory cache and across workers via Redis INCR; only the caller that observes a count of 1 sends. On send failure the marker is released with async_delete_cache so a transient failure does not suppress the alert for the full 24h TTL
* fix: harden budget alert claim release and skip-path event allocation
Addresses review feedback on the claim-before-send change. The claim release in each send-failure handler now logs the send error first and releases the claim best-effort through a shared helper, so a transient cache error during async_delete_cache cannot propagate out of the fire-and-forget budget_alerts task, drop the send-failure log, and leave the claim stuck for the full 24h TTL. In the multi-threshold branch the increment claim now runs before the WebhookEvent is built, so skipped concurrent crossings no longer construct and discard the event, matching the single-threshold and soft budget branches
* feat(proxy): track cost for unmanaged Vertex AI batch jobs
CheckBatchCost previously skipped Vertex batches created via the raw GCS
input_file_id path, since their unified_object_id is a raw provider job id
that fails the base64 managed-id check. Behind the opt-in general_settings
flag track_unmanaged_vertex_batch_cost, the poller now derives the model
from the gs:// input_file_id, maps it to a configured vertex_ai deployment,
polls the batch, computes cost, and marks batch_processed=True.
* Update tracking for failed", "expired", "cancelled"
* fix(proxy): apply ruff format to proxy_server.py
* address greptile review feedback (greploop iteration 1)
Filter unmanaged Vertex batch deployments by vertex_ai provider so a
shared model group name can't route to a wrong-provider deployment.
Move gs:// URI parsing into VertexAIBatchTransformation. Add test
coverage for the failed/expired/cancelled terminal-status DB update.
* fix: route unmanaged vertex batches to matching deployment
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* fix: p0 issues, added types and shared functions for each test suite
* style: carry clearer status_code comparison into renamed e2e dir
* refactor: migrate to gateway client
* fix: add new tests, split gateway
* test(e2e): add live batches suite across providers and routing scenarios
* test(batches): cover real cost tracking on completed batch retrieve
* test(e2e): assert managed vs raw file and batch id shapes per routing scenario
* test(e2e): assert full response shape of each batches and files endpoint
* test(e2e): only accept transitional statuses for a freshly created batch
* test(prompt-factory): make test_convert_url deterministic with a data URL
picsum.photos is down (HTTP 522), so test_convert_url failed on every
run. Swap the live external image for an inline data: URL and assert the
round-trip through convert_url_to_base64 genuinely.
A data URL is already inline base64 image data, so convert_url_to_base64
now short-circuits it instead of attempting an impossible HTTP fetch;
add a regression for that branch in the mapped image_handling test
* fix: pass through async image data urls
* fix(image-handling): short-circuit data URLs in async path too
Bugbot flagged that convert_url_to_base64 returns data: base64 URLs
unchanged but async_convert_url_to_base64 still tried to fetch them,
so async OCR flows (Bedrock, Azure) would reject inline images the sync
path accepts. Add the same guard to the async function and a regression
test that asserts the async path returns the data URL without touching
the HTTP client
* Fix: openai batches lifecycle
* Fix: add e2e azure openai tests
* Fix e2e for vertex ai
* Add all models for testing
* test(managed-files): assert idempotent upsert in store_unified_file_id
store_unified_file_id switched from create to upsert to avoid
UniqueViolationError when re-storing the same unified_file_id (e.g.
batch output files stored before metadata is available). Update the
unit test to assert the upsert call and its create payload instead of
the removed create call.
* test(batches): reconcile vertex_ai native batch-id comment with fallback guard
* fix(test-config): keep rust-ocr models in model_list by moving files_settings after it
* fix(test-config): move batch models after OCR block to keep merge with internal_staging clean
* fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: ruff format transformation.py and endpoints.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition
* test(vertex-ai/batches): align completion_window assertion to 24h
* fix: update managed file metadata on upsert
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy): count only active users toward license seat limit
SCIM-deactivated users (metadata.scim_active == false) are kept in LiteLLM_UserTable for audit and reactivation, but they were still counted toward the per-user license limit, so deactivating a user never freed a seat. Okta never sends a SCIM DELETE and Entra only hard-deletes well after deactivation, so deactivation has to be what frees the seat
Add UserRepository.count_billable_users(), which counts every row except those where metadata.scim_active is false (absent, null, and true all count), and route the user-create license gate, the free-SSO 5-user cap, and the enterprise /user/available_users display through it. A separate litellm_active_users Prometheus gauge reports the billable count while litellm_total_users keeps its original meaning so existing dashboards are unaffected
* fix(proxy): floor billable user count at zero
count_billable_users() runs two separate count queries (total, then deactivated). Under a burst of deactivations between them, the deactivated count can momentarily exceed the earlier total and produce a negative result, which would flow into is_over_limit as a negative and show a negative seat count in the display and gauge. Clamp the result to zero so a transient race can never yield a nonsensical negative; the value self-corrects on the next call
Addresses Greptile P1 on the PR
* refactor(proxy): count teams via TeamRepository in available_users
* style: ruff format changed files at line-length 120
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (#30708)
OpenAI GPT-5 models require max_completion_tokens >= 16.
Health checks were using 5 (proxy/health_check.py) and 10
(health_check_helpers.py), causing failures on GPT-5 models.
Fixes#23836
* fix: increase health check max_tokens from 5 to 16 (#23836) (#26610)
GPT-5 models enforce a minimum of 16 for max_output_tokens. The current
default of 5 still causes health checks to fail for these models. Bump
the non-wildcard default to 16 — the smallest value that satisfies all
known provider minimums while keeping health checks lightweight.
Also tightens the wildcard test assertion from a weak disjunctive check
to strict key-absence.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (#30696)
* fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema.
* fix: remove async keyword from test.
* fix: make Bedrock Mantle Responses routing data-driven per model (#30700)
* Make Bedrock Mantle Responses routing data-driven per model
Route Bedrock Mantle models to the native Responses API based on each
model's price-map capability signal instead of a hardcoded model-name
heuristic, and derive the OpenAI-compatible base path segment per model.
Responses dispatch now selects the native config when the model advertises
responses support (/v1/responses in supported_endpoints, or mode=responses),
both overridable via register_model and proxy model_info. This enables
native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping
chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing
chat-completions emulation. Capability is per-model, so gpt-oss-120b routes
natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss
substring.
The wire path is a separate concern, driven by the existing
use_openai_responses_path flag rather than a model-name match: gpt-5.x and
gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat
config now derives its base from the same flag, fixing gemma-4
chat-completions requests that previously went to /v1 instead of /openai/v1.
Cost maps: add supported_endpoints to the gpt-oss entries (responses for the
non-safeguard variants, chat-only for safeguard) and supported_endpoints +
use_openai_responses_path to all three gemma-4 entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: move capability helper into bedrock_mantle package
Move the Responses capability check out of utils.py into
litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses,
alongside its companion wire-path helper mantle_base_segment. Both are now
pure functions of (model, model_cost): the price-map mode/supported_endpoints
read replaces the get_model_info call, so the rules are unit-testable without
patching global state and the Bedrock Mantle package is self-contained.
Use str | None instead of Optional[str] on the new signatures to satisfy the
ruff UP045 strict-rule gate. Add direct unit tests for both helpers.
Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b
now legitimately supports Responses, so it can no longer be the
"None after restore" vehicle; use the chat-only safeguard variant, which
isolates the register/restore effect from the model's own capability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)
The tiered cost calculator resolved a tier's per-token cost with
`tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or`
short-circuits on any falsy value, a tier that legitimately prices a
component at 0.0 (e.g. a free-cache-read tier with
cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated
as missing and silently billed at the full fallback rate
(input_cost_per_token / output_cost_per_token).
The flat-pricing path in the same module already handles this correctly
with an `is None` guard. Resolve tier costs through a small helper that
mirrors it, so 0.0 is honored at both the in-range and overflow sites.
No shipped model currently has a 0.0 tier cost, so this is a latent
defect; the fix makes the tiered path consistent with the flat path and
prevents over-charging the first time such a tier appears. Adds unit
tests covering the in-range and overflow paths, and drops an unused
import flagged by ruff in the touched test file.
* feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)
* fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (#30618)
In the messages->chat/completions bridge, translate_anthropic_tools_to_openai
merged every non-mapped tool key into the function parameters dict. The
Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object'
-> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type).
Exclude 'type' from the passthrough. Fixes#30557.
* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)
An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:
1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
engine, spawn new one.
2. The engine-death watcher sees that kill, assumes a crash, and calls
`attempt_db_reconnect(force=True)` (a different lock,
`_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
3. In-flight queries failing during the swap are classified as transport
errors and trigger their own `attempt_db_reconnect` -> recreate again.
Fix coordinates planned restarts across the wrapper and the watcher:
- PrismaWrapper records the old engine PID in `_expected_engine_deaths`
before killing it; all four watcher death-detectors (waitpid thread,
pidfd, already-dead probe, os.kill poll) consume that PID and skip the
reconnect instead of treating it as a crash.
- `recreate_prisma_client` now serializes through `_reconnection_lock` and
bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
as an optimistic-lock token, so racing/cascading recreates collapse into a
single restart (losers no-op). This closes the two-lock gap.
- The direct reconnect path probes the writer with SELECT 1 before
recreating; a healthy connection (e.g. engine already replaced by a
refresh) skips the recreate entirely.
- `_safe_refresh_token` coalesces: it skips when the current token still has
more than the refresh buffer of runway, so stacked triggers (proactive
loop + __getattr__ fallback) don't each restart the engine. An
`on_engine_replaced` hook re-arms the watcher on the new PID.
RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.
* feat(bedrock): support file content retrieval for batch output files (#30595)
Implements transform_file_content_request and transform_file_content_response
in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch
files. The request transform resolves the file id (direct s3:// URI or base64
unified id) to its S3 object, validates bucket and key prefix against the
server-configured bucket, and SigV4-signs an S3 GetObject using the same
credential and region resolution as the existing upload path. The credential
and region params are validated into a typed model at the boundary, so the only
untyped values left are the botocore signing primitives.
Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries
s3_bucket_name (previously dropped when building deployment credentials) and
the managed-files hook passes the deployment credential snapshot when routing
afile_content, so unified-id content retrieval works with per-model bucket
config instead of only the AWS_S3_BUCKET_NAME env var.
Preserves managed-file access control: the proxy file-content endpoint now
rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the
owner/team check that only runs for unified ids and let a caller read another
tenant's batch output by its object key. Managed outputs are reachable only
through their unified file id. The afile_content "not found" error now reports
the caller's unified id rather than the resolved internal S3 URI.
Fixes#16186, #15563
* fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)
* fix(oci): map Cohere tool array/object params to lowercase builtins
OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare
"List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema
arrays. MLflow {{trace}} judges trip this: their tools (get_root_span,
get_span) take an attributes_to_fetch array. The lowercase builtins list/dict
are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but
both are lowercased for consistency).
Verified live against us-chicago-1 (cohere.command-a-03-2025 and
command-latest). Adds a unit regression on the transformed parameterDefinitions
plus a gated integration test exercising an array-param tool end to end.
* fix(oci): make Cohere agentic tool-calling continuation work
Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges
drive once a tool has been executed and its result is fed back.
Request side: litellm pulled the last user message into the top-level `message`
and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that
("cannot specify message if the last entry in chat history contains tool
results"), and an empty message alone is rejected too ("message must be at least
1 token long or tool results must be specified"). OCI carries the current turn's
results in a dedicated top-level `toolResults` field. The Cohere transform now
sends an empty message, keeps the user turn in chatHistory, and puts the results
in `toolResults`, matching the langchain-oracle reference. Tool results are no
longer represented as chatHistory entries.
Response side: tool-grounded answers come back with citations carrying
`documentIds` (camelCase) and no `document_ids`, which made the required
`CohereCitation.document_ids` field fail validation and sink the whole response
parse. Those citations are never surfaced, so the field (and CohereSearchQuery's
generation_id) is now optional.
Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest),
single and multi-round tool loops. Adds unit regressions on the transformed
request shape and on citation parsing, plus gated integration tests for the
continuation.
* feat: integrate Repelloai Argus guardrail (#30673)
* feat(guardrails): add RepelloAI Argus guardrail integration (#1)
* feat(guardrails): add RepelloAI Argus guardrail integration
Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.
* fix(guardrails): harden RepelloAI Argus guardrail
- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
block on unknown/malformed verdicts so an API change can't silently
disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
expose unreachable_fallback in the config schema, and stop leaking the
raw provider response / exception strings to the client
* fix(guardrails): address RepelloAI Argus review feedback
- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description
* docs(guardrails): add RepelloAI Argus docs page and dashboard listing
- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution
* fix(guardrails): keep RepelloAI asset_id optional in config model
A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.
* Add comment for last user turn scanning
* feat(guardrails): harden repelloai scanning
* feat(guardrails): expand repelloai scanning to include tool definitions
Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.
* refactor(guardrails): fix and harden repelloai schema text extraction
- Fix duplicate text in _iter_schema_text: previously all dict values were
re-queued onto the stack even after scalar/list keys were already extracted
explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param
* fix(guardrails): resolve pyright type errors in repelloai guardrail
- Narrow async_handler.post return from Response|None to Response with
explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
type in pyright
* fix(guardrails/repello): include Responses API instructions field in prompt scan
The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.
* feat: add api_key to config model and read prompt from data dict
* fix(guardrails/repello): plug input_text and tool-call response bypass gaps
Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.
Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.
Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.
* fix(guardrails/repello): scan Responses API function_call output arguments
Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.
* refactor(guardrails/repello): clean up typing and remove lint-any workarounds
- Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout
- Use dict[str, object] instead of bare dict in all signatures
- Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly
- Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel
- Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks
- Use TypeAdapter.validate_json() instead of response.json() + manual dict construction
- Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any
- Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check
- Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType
- Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel]
* fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning
- Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate
- Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks
* refactor: modifications for lint check
* feat: add Pinstripes as an OpenAI-compatible provider (#30567)
* feat: add Pinstripes as an OpenAI-compatible provider
Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference
provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.)
with per-token pricing and no subscriptions.
Changes:
- `litellm/llms/openai_like/providers.json`: register pinstripes with
base_url, api_key_env, and max_completion_tokens→max_tokens mapping
- `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders
- `litellm/constants.py`: add to openai_compatible_providers and
openai_compatible_endpoints lists
- `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect
provider when api_base is "https://pinstripes.io/v1"
- `provider_endpoints_support.json`: document supported endpoints
- `tests/`: 7 unit tests covering provider registration, resolution,
URL auto-detection, api_base override, and Router config
Usage:
import litellm
response = litellm.completion(
model="pinstripes/ps/glm-4.5-air",
messages=[{"role": "user", "content": "Hello"}],
api_key=os.environ["PINSTRIPES_API_KEY"],
)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): resolve Greptile P1 review comments
- Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works
- Set responses: false in provider_endpoints_support.json — not actually wired up
- Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): add api_base_env and correct responses capability
- Add api_base_env: PINSTRIPES_API_BASE to providers.json
- Set responses: false in provider_endpoints_support.json
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): wire up Responses API — add supported_endpoints
Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so
JSONProviderRegistry.supports_responses_api returns true correctly,
matching what provider_endpoints_support.json advertises.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(pinstripes): enable embeddings endpoint
Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings.
Add /v1/embeddings to supported_endpoints and set embeddings: true.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json
Matches the file's existing convention. Flagged by Greptile review.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): set a2a: false — A2A protocol not implemented
All comparable JSON-configured providers (tensormesh, parasail, empiriolabs,
libertai, neosantara) have a2a: false. Pinstripes does not implement the
Google A2A protocol, so this should be false to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: inference_provider <max@redactedlab.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(rag): attach existing OpenAI file ids (#30628)
* fix(rag): attach existing OpenAI file ids
* chore: use modern typing in rag ingest fix
* chore: retrigger ci
* fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)
cache_control_injection_points was only consumed by the chat/completions
prompt-management hook; on the native Anthropic /v1/messages path it was
forwarded unused, so deployment-level cache injection was silently dropped
(cache_creation_input_tokens stayed 0 for Anthropic-native clients).
Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject
cache_control at block level for system / tools / message locations (the only
forms /v1/messages accepts), wire it into the native anthropic_messages
handler, and pop the param so it does not leak upstream as an unknown field.
A {location: message, role: system} config is redirected to the top-level
system prompt so the same YAML works on both endpoints.
Injection respects Anthropic's 4-block cache_control limit shared across
system, tools, and messages: client-supplied markers count toward the cap and
are never overwritten, a slot is reserved per Bedrock tool_config point, and
injection stops once the budget is exhausted. Locations this path cannot
represent (tool_config) are forwarded downstream instead of being silently
consumed, mirroring get_chat_completion_prompt's remaining_points pass-through.
Built on litellm_internal_staging. Refs BerriAI/litellm#30293
* fix(proxy): release budget reservation when a request is cancelled mid-flight (#30522)
* fix(proxy): release budget reservation on cancel when no chunk was delivered
The pre-call budget reservation increments the cross-pod spend counter by a
request's worst-case cost, then reconciles it on success (cost callback) or
error (failure hook). A client disconnect or timeout cancels the request and
surfaces as CancelledError / GeneratorExit, which neither path catches, so the
reservation leaks. Under a retry storm the leaked holds accumulate, pin the
counter above real spend, and return spurious 429 "Budget has been exceeded" to
keys whose spend is far below budget; the counter only recovers when its TTL
lapses, so the failure is intermittent and self-healing.
Release the reservation in async_streaming_data_generator (which the Anthropic
and Google SSE generators delegate to) on the (CancelledError, GeneratorExit)
path, alongside the existing max_parallel_requests release. release_budget_
reservation_on_cancel runs under asyncio.shield so it completes despite the
in-progress cancellation, is guarded by the reservation's finalized flag, and
swallows a failing release so it cannot replace the in-flight cancellation.
The refund is gated on whether a chunk reached the client. The flag is set
immediately before the yield, after the slow-path hook await: an async generator
suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk
sees it True (keep the hold), while a cancellation during the slow-path await
leaves it False (refund, nothing sent). A non-streaming cancellation delivers
nothing and a completed non-streaming response is reconciled by the success
callback, so neither needs a release here.
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(proxy): reconcile a cancelled reservation to input cost, not zero
A streaming request cancelled before the first chunk previously reconciled its
reservation to zero and finalized it. But by the time the generator is
consuming the response the provider call was already dispatched, so the input
tokens were billed even though no chunk reached the client, and the
success/failure cost callbacks are skipped on cancellation. Refunding to zero
let a caller send an expensive request and abort pre-token to dodge the input
charge.
Compute the request's input-token cost at reservation time and reconcile the
cancelled reservation to it instead of zero. The worst-case output portion of
the reservation is still released (so a legitimate mid-flight cancellation no
longer pins the counter and 429s the key), while the input the provider already
processed is charged.
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(caching): encode object name in GCS cache GET path (#30378)
GCS cache reads always missed when gcs_path was set. The GET methods
interpolated the object name directly into the URL path, while the GCS
JSON API requires it to be URL-encoded (a "/" must be sent as %2F).
With gcs_path configured the object name is "<prefix>/<sha256>", so the
raw slash produced a malformed object path and GCS returned 404. httpx
does not raise on 4xx, so the status_code == 200 check fell through and
get/async_get returned None, silently missing on every read. Without
gcs_path the key has no slash, which is why this went unnoticed.
Wrap the object name with urllib.parse.quote(..., safe="") in get_cache
and async_get_cache. Apply the same encoding to the name= query
parameter in set_cache and async_set_cache so the key written matches
the key read back.
Adds regression tests asserting the GET path and SET query are encoded
(%2F) when gcs_path is set, for both sync and async paths; these fail on
the unpatched code.
Fixes#30377
* chore: add soniox stt-async-v5 model (#30672)
* fix(proxy): include model group aliases in v1 model info (#30626)
* Include model group aliases in v1 model info
* Fix model info alias implementation
* removed extra blank line
* chore: rerun CI
* fix(lint): remove redundant noqa directive in proxy_cli.py
* fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme
* Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme"
This reverts commit 52c7a07777.
* Revert "fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)"
This reverts commit c9e8a177bd.
* Revert "fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)"
This reverts commit 85828da695.
* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)
An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:
1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
engine, spawn new one.
2. The engine-death watcher sees that kill, assumes a crash, and calls
`attempt_db_reconnect(force=True)` (a different lock,
`_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
3. In-flight queries failing during the swap are classified as transport
errors and trigger their own `attempt_db_reconnect` -> recreate again.
Fix coordinates planned restarts across the wrapper and the watcher:
- PrismaWrapper records the old engine PID in `_expected_engine_deaths`
before killing it; all four watcher death-detectors (waitpid thread,
pidfd, already-dead probe, os.kill poll) consume that PID and skip the
reconnect instead of treating it as a crash.
- `recreate_prisma_client` now serializes through `_reconnection_lock` and
bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
as an optimistic-lock token, so racing/cascading recreates collapse into a
single restart (losers no-op). This closes the two-lock gap.
- The direct reconnect path probes the writer with SELECT 1 before
recreating; a healthy connection (e.g. engine already replaced by a
refresh) skips the recreate entirely.
- `_safe_refresh_token` coalesces: it skips when the current token still has
more than the refresh buffer of runway, so stacked triggers (proactive
loop + __getattr__ fallback) don't each restart the engine. An
`on_engine_replaced` hook re-arms the watcher on the new PID.
RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.
* fix(lint): modernize type annotations in IAM-refresh prisma client files (UP006/UP045)
* Revert "feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)"
This reverts commit f530b2237c.
* Revert "fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)"
This reverts commit 4f58bd0df5.
* Revert "fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)"
This reverts commit 50f34e0b15.
* Revert "fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)"
This reverts commit 0544eed6ea.
* fix(bedrock_mantle): restore BedrockMantleAuthMixin and constants removed by routing rewrite
* fix(key management): restore exact /key/list user_id & key_alias matching by default (#30593)
Before substring search was added (commit 33bd570d5e), /key/list matched user_id
and key_alias exactly. That change made admin-authenticated calls substring-match
by default, breaking the prior contract: a caller passing an exact user_id as an
access filter (e.g. an integration scoping to one user with an admin key) then
received other users' keys -- user_id="alice" also returned "alice2",
"alice-test", etc. This is a cross-user key disclosure.
Make substring matching opt-in via a new admin-only substring_matching=true query
param; default to exact, restoring the prior behavior. The dashboard search box
(keyListCall) passes the flag so partial search still works. Non-admins remain
exact and scoped to their own keys.
Updates the proxy-behavior key_alias test to opt in and adds an exact-by-default
guard; adds list_keys unit coverage for the opt-in gate.
---------
Co-authored-by: perseus <51974392+tcconnally@users.noreply.github.com>
Co-authored-by: Hannah Smith <64043506+hannahmadison@users.noreply.github.com>
Co-authored-by: Charlie Patterson <Pattersoncharlesl@gmail.com>
Co-authored-by: Matthew Lapointe <mlapointe@alpha-sense.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com>
Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com>
Co-authored-by: Nitish Agarwal <1592163+nitishagar@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com>
Co-authored-by: AD Mohanraj <admohanraj@gmail.com>
Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com>
Co-authored-by: Lavish Bansal <lavish.bansal619@gmail.com>
Co-authored-by: max-amos <gruffulom@gmail.com>
Co-authored-by: inference_provider <max@redactedlab.com>
Co-authored-by: NK <93352237+Nithish-Yenaganti@users.noreply.github.com>
Co-authored-by: 安妮的心动录 <74543653+anneheartrecord@users.noreply.github.com>
Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com>
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Burak Ömür <burak.omur.1998@gmail.com>
Co-authored-by: Dan Lemon <daniel.lemon@amazee.io>
Co-authored-by: Vanika Dangi <166420943+vanika02@users.noreply.github.com>
Co-authored-by: Jay Gowdy <130084966+jgowdy-godaddy@users.noreply.github.com>
Drops PLR0915 from ruff's extend-select along with its per-file-ignores,
and strips the now-unused `# noqa: PLR0915` directives across the codebase
(RUF100 would otherwise flag them as unused). The C901 suppression that
shared a directive with PLR0915 in streaming_handler.py is preserved.
* fix(proxy): skip double-wrapping unified batch output file ids on retrieve
After ensure_batch_response_managed_file_ids normalizes output_file_id, the managed files post-call hook was re-encoding the unified id and storing the nested id as the provider mapping. Use the decoded llm_output_file_id for retrieve and model_mappings instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): guard managed file id parsing for non-output unified formats
Only treat decoded unified ids as already-wrapped output files when they contain llm_output_file_id. Skip other litellm_proxy id shapes instead of IndexError on split.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): rename loop variable to satisfy mypy unified file id typing
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix overiding of fastapi_response headers
* fix(bedrock): support tool search results and surface citations as annotations
Add an optional tool-message search_results path that maps directly to Bedrock toolResult.searchResult blocks, and convert Converse citationsContent into chat completion annotations for user-facing citation metadata.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(format): align bedrock prompt factory with black
Reformat the updated bedrock prompt template conversion file so CI black --check passes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): harden citations, search_results mapping, and token counting
Resolve mypy issues in citation parsing, only attach url_citation annotations when citation text is stitched into content, fall back to tool content when search_results is empty, and count search_results text in token/TPM preflight paths.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): extract tool result helpers to satisfy PLR0915
Refactor _convert_to_bedrock_tool_call_result into smaller helpers so lint passes without changing Bedrock tool result behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): count all forwarded search_results fields in token estimates
Include source, title, content text, and citations when estimating tokens so large metadata cannot bypass TPM preflight checks. Reformat factory.py with black.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(managed-files): skip content blocks without a type key in get_file_ids_from_messages
* fix(bedrock): stitch citations for any punctuation-only text block
* fix(bedrock): map null citation source/title to empty annotation strings
* fix(bedrock): advance citation offset for text-only citationsContent blocks
* fix(bedrock): complete citation TypedDicts for grounding annotations
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
