mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-20 00:11:50 +00:00
39817 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
ryan-crabbe-berri
|
bb77cc0841 |
fix(mcp): stop logging tool-call input in MCP client (#31393)
Backport of #31393 to stable/1.90.x.
Cherry-picked from
|
||
|
Shivam Rawat
|
bb90565092 |
fix(bedrock): honor ttl for tool_config cache injection points (#31929)
Backport of #31929 to stable/1.90.x.
Cherry-picked from
|
||
|
Mateo Wang
|
6a5353196d |
fix(bedrock/converse): drop toolSpec.strict for Opus 4.7/4.8 (#31582) (#31923)
Backport of #31923 to stable/1.90.x.
Cherry-picked from
|
||
|
yuneng-jiang
|
1e60f26596
|
Merge pull request #31782 from BerriAI/litellm_backport_1_90_x_bp_31519_31733
chore(release): backport #31519, #31733 to stable/1.90.x and cut 1.90.2 |
||
|
Yuneng Jiang
|
8f96737c78
|
chore: refresh uv.lock for 1.90.2 | ||
|
Yuneng Jiang
|
9c88929209
|
bump: version 1.90.1 → 1.90.2 | ||
|
mubashir1osmani
|
9426f7b9dd
|
fix(logging): route realtime success logging through the bounded worker (#31733)
RealTimeStreaming.log_messages dispatched the success handler with a bare
asyncio.create_task, bypassing GLOBAL_LOGGING_WORKER (which gives a per-coroutine
timeout and a concurrency cap). On a long-lived realtime websocket a slow logging
callback left one suspended task per logged turn, each pinning that turn's
assembled response, accumulating without bound (~12-15k in-flight under load in a
repro) until OOM. Route realtime success logging through the bounded worker so
in-flight logging is capped and a hung callback is cancelled at the worker
timeout.
The chat and responses streaming success-logging paths are intentionally left
unchanged: their success callbacks must complete within the call's event-loop run
(the non-streaming path pairs the worker with a synchronous callback; the
streaming path has no such companion), so deferring them through the worker would
drop logs for one-shot SDK calls and breaks test_async_custom_handler_stream.
Bounding those paths needs a load-shedding approach and is left to a follow-up.
(cherry picked from commit
|
||
|
mubashir1osmani
|
13285671f2
|
fix(realtime): stop second Gemini Live setup, retry hung handshake, close guardrail bypass (#31519)
* fix(realtime): stop sending a second Gemini Live setup on follow-up session.update
Gemini Live (BidiGenerateContent) accepts setup as the first-and-only client
message; a second setup closes the socket with 1007 Request contains an invalid
argument. The AI Studio Gemini path forwarded every client session.update after
the first as a follow-up setup, and GA clients (pipecat) send several while
configuring the session, so the second one tore the session down before the
first turn. Callers saw silence after the first response, exponential per-turn
latency from reconnect/retry churn, and intermittent 1011 errors.
Drop subsequent session.updates instead of resending setup, matching what the
Vertex subclass already does. Tools and instructions must ride on the first
session.update before any conversation content.
Adds regression tests covering the plain follow-up, a follow-up that adds tools
(the case the previous identical-only dedup still forwarded), and the guardrail
create_response=False warning path.
* fix(realtime): retry the backend open handshake instead of failing with 1011
The upstream Live API open handshake (e.g. Gemini Live) intermittently hangs;
waiting longer never recovers a hung attempt, but a fresh attempt almost always
connects in ~1s. The proxy opened the backend websocket once with the default
open_timeout and no retry, so a single slow handshake surfaced to the caller as
a fatal 1011 internal error and dropped the call.
Bound each open attempt with a short open_timeout and retry; a bounded attempt
that already timed out spaces out the next try, so no backoff is needed.
Deterministic handshake-status rejections (auth/4xx) are not retried, and the
retry only ever wraps the open, never a live session.
Adds tests for retry-then-succeed, raise-after-max-attempts, and
no-retry-on-auth-failure.
* fix(realtime): close guardrail bypass + surface handshake status; drop obsolete tests
Three review fixes on the Gemini Live realtime path.
Transcription-guardrail bypass: Gemini Live rejects a second setup (1007), so once
the initial setup is sent the guardrail's automaticActivityDetection.disabled=true
can no longer be delivered as a follow-up session.update. With that follow-up now
dropped, the model's auto-response stayed enabled and a realtime_input_transcription
guardrail was bypassed (the model answered before the proxy could gate the turn).
Fold the disable into the one-and-only setup instead: the handler injects it into
the auto-sent setup (gemini_live_defer_setup false) and _send_to_backend injects it
into the deferred first setup. OpenAI sessions accept follow-up updates and are left
untouched.
Backend handshake status: the open-retry treated only InvalidStatusCode as
deterministic; websockets>=15 raises InvalidStatus for a rejected client handshake,
so a 401/403 fell into the broad WebSocketException branch and was retried before
the caller closed the client with 1011 instead of the upstream status. Treat both as
non-retryable.
Obsolete tests: the four tests asserting a follow-up session.update is merged and
re-sent as a second setup asserted behavior that crashes Gemini Live with 1007
(verified directly against the API). Removed; the drop is covered by new regression
tests.
* style(realtime): reformat changed files to ruff line-length 120
Post-merge with litellm_internal_staging, which unified ruff format width to 120
(#31518). The realtime change set was formatted at 88, so the changed lines
tripped the whole-file ruff format check. Reformat with ruff 0.15.3 at the repo's
120 width; no logic changes.
(cherry picked from commit
|
||
|
yuneng-jiang
|
ca6149ab99
|
Merge pull request #31667 from BerriAI/litellm_backport_1_90_x_31036_31342_31653
chore(release): backport #31036, #31342, #31653 to stable/1.90.x and cut 1.90.1 (litellm-enterprise 0.1.43.post1) |
||
|
Yuneng Jiang
|
92da671c68
|
chore: refresh uv.lock for 1.90.1 and litellm-enterprise 0.1.43.post1 | ||
|
Yuneng Jiang
|
22421112ea
|
bump: version 1.90.0 → 1.90.1 | ||
|
Yuneng Jiang
|
3d06452b36
|
chore(release): bump litellm-enterprise 0.1.43 -> 0.1.43.post1 for stable/1.90.x | ||
|
mubashir1osmani
|
ccd7bb6754
|
fix(vertex_ai/files): single media upload for batch files to fix 499s on large uploads (#31653)
* fix(vertex_ai/files): upload batch files in a single media request to fix 499s on large uploads
PR #31036 switched the vertex batch file upload from a single GCS media
upload to a chunked resumable session. The resumable path sends the body as
many sequential PUTs, each waiting a full round-trip to GCS before the next,
so a multi-GB upload accumulates hundreds of round-trips and overruns the
client/load-balancer request timeout, surfacing as 499s (client closed
connection) on files as small as 500MB. This was a regression from the
last-known-good commit, where the upload completed as one continuous request.
Revert the batch upload to a single uploadType=media request, but stage the
transformed payload to a temp file first so peak memory stays bounded (the
goal of the resumable rewrite) without the per-chunk round-trips. The temp
file is closed deterministically (TemporaryFile unlinks on close), not left
to the GC. The now-unused resumable chunked-upload plumbing is removed.
Also swap the per-row transform's stdlib json for orjson (parse + serialize),
which is ~4x faster on this hot path; the streaming body now emits compact
orjson bytes.
The request stays synchronous, so the returned file object is real and
POST /v1/batches keeps working immediately against the uploaded object.
Tests: single media request carries the whole payload with a real
Content-Length (no chunked transfer-encoding); failed upload raises; the
staged temp file is closed deterministically; byte-for-byte transform parity.
* test(vertex_ai/files): mock single media upload POST instead of removed resumable method
test_avertex_batch_prediction patched BaseLLMHTTPHandler._aresumable_chunked_upload, which was removed when the batch jsonl upload moved from a chunked resumable GCS session to a single uploadType=media request. Patch the raw httpx.AsyncClient.post that _astage_and_upload_media issues so the real staging, upload and response transform run while the GCS object response is mocked, and assert the media URL and Content-Type.
* fix(vertex_ai/files): forward request timeout to media upload, drop orjson, sort imports
Forward the per-request timeout through _stage_and_upload_media /
_astage_and_upload_media to the GCS POST. Every other upload branch forwards
it; the new media path was dropping it, so a caller-provided timeout was
silently ignored (the files path passes 600s by default, but a custom
request_timeout would not have reached this upload). Regression test asserts
the resolved timeout reaches the request (mutation-verified).
Revert the orjson swap in the batch transform: importing orjson at module load
in this core-path file broke `import litellm` on environments without orjson
(the Windows import test). Back to stdlib json; the upload leg dominates large
uploads anyway, so the transform-side win was marginal.
Fix import ordering in llm_http_handler.py (I001) introduced by the new imports.
* fix(vertex_ai/files): stream batch upload to GCS instead of staging to a temp file
Addresses a disk-exhaustion concern: staging the full transformed batch body to
a local temp file before the GCS request meant an authenticated user could fill
the proxy's temp volume with large concurrent uploads (on top of Starlette's
input spool).
GCS's simple/media upload accepts chunked transfer-encoding, so stream the
transform straight to the single media request instead. Each block is produced
on a worker thread (the transform never runs on the event loop) and sent
chunked, so the body is neither buffered in memory nor written to disk, and the
upload is still one continuous request (no per-chunk round-trips, no 499). Drops
the temp-file staging, the tempfile/IO imports, and Content-Length computation.
Regression test asserts the upload streams (chunked transfer-encoding, no
Content-Length) and creates no temp file; mutation-verified that reintroducing
staging fails it.
(cherry picked from commit
|
||
|
yucheng-berri
|
6a1d2b0c66
|
fix(proxy/client): redact api key from key/info client error messages (#31342)
* fix(proxy/client): redact api key from key/info client error messages
The keys management client builds GET /key/info?key=<key> and lets the
requests HTTPError propagate. str(HTTPError) renders the failing request URL
verbatim ("... for url: .../key/info?key=sk-..."), so any caller that logs the
exception leaks the full key; the 401 branch leaked the same way through
UnauthorizedError(str(orig_exception))
Redact both branches with the existing redact_secrets helper so the
secret-bearing query param is scrubbed to ?REDACTED while the status code,
reason, and response object are preserved. Server-side responses already mask
the key, so this closes the remaining client-side surface
* fix: preserve key info unauthorized response
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
(cherry picked from commit
|
||
|
mubashir1osmani
|
45b6ece18c
|
fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads (#31036)
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
(cherry picked from commit
|
||
|
tin-berri
|
6e8282d406
|
fix(mcp): challenge delegate-auth OAuth servers with upstream resource_metadata (#31255)
An oauth2 MCP server with delegate_auth_to_upstream=true never prompted the user to sign in. On an unauthenticated initialize the gateway answered locally (200, no tools) and emitted no WWW-Authenticate, so clients like Claude Desktop either connected empty or hit "OAuth probe timeout after 10000ms". #30124 added a bare `continue` in _raise_preemptive_401_for_unauthenticated_servers to stop sending LiteLLM's gateway authorization_uri challenge for delegate-auth servers, expecting the upstream to emit its own challenge. On initialize the gateway never probes upstream, so no challenge ever reached the client. Replace the `continue` with a preemptive 401 carrying the proxied resource_metadata (RFC 9728) challenge, the same form passthrough servers and MCPUpstreamAuthError already use. This keeps #29770 fixed (still no authorization_uri) while restoring the upstream PKCE sign-in prompt. |
||
|
Yuneng Jiang
|
0b9dce00e3
|
chore: update Next.js build artifacts (2026-06-27 01:58 UTC, node v20.20.2) | ||
|
ryan-crabbe-berri
|
95a0c30a05
|
fix(ui): render logos under a custom server_root_path (#31156)
The App Router migration moved pages to deeper path segments and the proxy can be mounted under a sub-path (e.g. /litellm behind a reverse proxy). Local logo asset paths were emitted without the server root prefix, so they resolved off the origin root and 404'd. Route every local logo src through a single resolver that prefixes the live server root path and leaves external URLs untouched, fixing provider, guardrail, vector store, callback, MCP and audit-log logos at any route depth and root path. |
||
|
yuneng-jiang
|
dcf1b445e6
|
Merge pull request #30907 from BerriAI/litellm_internal_staging
Some checks failed
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
CodSpeed Benchmarks / benchmarks (push) Has been cancelled
Helm unit test / unit-test (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
GitHub Actions Security Analysis / zizmor (push) Has been cancelled
chore(ci): promote internal staging to main |
||
|
yuneng-jiang
|
df5619f9b2
|
chore: update Next.js build artifacts (2026-06-21 00:50 UTC, node v20.20.2) (#30906) | ||
|
ryan-crabbe-berri
|
8125ddd2f8
|
fix(ui): source api-keys identity from useAuthorized to stop "User ID is not set" (#30903)
The migrated /ui/api-keys route gates rendering on useAuthorized() but read userID from the AuthContext (useAuth), which hydrates asynchronously. On a hard refresh or deep link the route could render UserDashboard before AuthContext had populated userID, so UserDashboard hit its `userID == null` guard and showed "User ID is not set". The legacy index page avoided this by gating on AuthContext's own authLoading; the migration switched the gate to useAuthorized without aligning the identity source. Read identity from useAuthorized (a synchronous cookie decode) so userID is populated whenever the route is authorized. useAuth is kept only for the backfill setters UserDashboard still expects, until the planned AuthContext consolidation removes them. Refs LIT-3687 |
||
|
Krrish Dholakia
|
53593f697d
|
feat(sandbox): e2b code execution primitive (#30898)
* feat(sandbox): add e2b code execution primitive Add a provider-agnostic code execution primitive that runs model-generated code in an isolated sandbox and returns the output, with e2b as the first backend over raw httpx (no SDK dependency). Public API: litellm.acode_interpreter_tool (ephemeral create -> run -> delete) plus the low-level lifecycle litellm.acreate_sandbox / arun_code / adelete_sandbox. Each is @client-decorated so operations are logged like litellm.asearch. Backends implement BaseSandboxConfig; resolved via ProviderConfigManager.get_provider_sandbox_config. * fix(sandbox): address review feedback and CI gates - document e2b provider in provider_endpoints_support.json and add a sandbox endpoint definition - regenerate dashboard CallTypes after the sandbox call-type additions - guard explicit timeout=0 instead of coercing it to the default - require a ContainerHandle access token before running code; reject bare-id runs - return False on a 404 delete now that the shared http handler raises for status - skip non-JSON NDJSON lines and cap streamed output to bound memory - move the real-network integration tests out of tests/test_litellm into tests/integration/sandbox * fix(sandbox): satisfy strict ruff gate and scope star-exports - modernize annotations in the new sandbox modules to PEP 585/604 (list/dict, X | None) and drop the now-unnecessary quoted forward refs so the strict-rule budget delta for UP006/UP037/UP045 returns to zero - add __all__ to litellm/sandbox/main.py so 'import *' only re-exports the four public entrypoints instead of leaking module-level imports * fix(sandbox): drop quotes on sandbox config return annotation utils.py uses 'from __future__ import annotations', so the quoted forward ref tripped UP037; the unquoted union is lazily evaluated and keeps the strict-rule delta at zero * chore(sandbox): re-trigger automated review after addressing feedback |
||
|
Mateo Wang
|
b16cfd7de9
|
test: point router/completion/triton tests at the local fake OpenAI endpoint (#30900)
* test: point router/completion/triton tests at the local fake OpenAI endpoint The shared Railway-hosted mock (exampleopenaiendpoint-production.up.railway.app) takes down unrelated CI jobs whenever it is unreachable. #30695 moved the mounted proxy configs onto a job-local fake server but left these in-Python api_base literals pointing at the dead host, so litellm_router_testing, local_testing_part1, local_testing_part2 and llm_translation_testing still fail with a 404 "Application not found" when Railway is down Resolve the api_base from FAKE_OPENAI_API_BASE (default http://127.0.0.1:8190) through a shared helper, auto-start the canned server from the local_testing and llm_translation conftests when nothing is already serving, and extend the server with a Triton embeddings route and a slow-endpoint delay so the triton and latency-timeout tests run fully offline. The deliberately broken fallback URL is left as-is so fallback handling still has a failing upstream * fix: ignore non-loopback FAKE_OPENAI_API_BASE so the local mock is used in CI * fix: drop 0.0.0.0 from loopback hosts, an unreliable client connect target * fix(tests): keep fake OpenAI mock alive across xdist workers ensure_fake_openai_endpoint registered atexit on the worker that spawned the subprocess, so under -n 4 the first worker to drain its queue would terminate the shared mock while siblings were still hitting it. Detach the child via start_new_session and drop the per-worker teardown; reuse on /health handles re-runs and CI containers clean up themselves |
||
|
Shivam Rawat
|
c7efa77de3
|
fix(watsonx): wrap string embedding input in array for WatsonX API (#30897)
* fix(watsonx): wrap string embedding input in array for WatsonX API WatsonX text/embeddings expects inputs as []string; OpenAI clients often send a single string. Co-authored-by: Cursor <cursoragent@cursor.com> * style(watsonx): format watsonx embed transformation for black Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watsonx): avoid UP006 in embed transformation strict lint gate Use list[str] and branch-based input normalization instead of List and cast so the watsonx embedding change does not add strict ruff UP006 violations. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
yuneng-jiang
|
46d2861001
|
chore(ci): bump deps (#30899)
* bump: version 0.1.42 → 0.1.43 * bump: version 1.89.0 → 1.90.0 * adding uv lock |
||
|
yuneng-jiang
|
140ca3012a
|
chore: update Next.js build artifacts (2026-06-20 21:24 UTC, node v20.20.2) (#30894) | ||
|
Mateo Wang
|
a7b0b0ba09
|
feat: add lint-gate target and truncation-proof summary to the strict ruff gate (#30877)
* feat: add CI-parity mode and truncation-proof summary to strict ruff gate * refactor: tolerant worktree cleanup and concrete GateInputs types * fix: clean up temp dir when git worktree add fails * fix: align lint-gate with CI by dropping unused --ci-parity path The lint-gate Makefile target invoked ruff_strict_gate.py with --ci-parity, which counted violations on a throwaway merge of base into HEAD against base counts at the base tip. CI in test-linting.yml runs the same script without --ci-parity on a PR-head checkout, taking the gather_fast path that counts on the live tree against base counts at the merge-base. A local pass could therefore disagree with CI. Drop --ci-parity from the Makefile and remove the now-unused gather_ci_parity branch and flag so there is one code path that both local and CI exercise. The docstring claim that CI runs against the synthetic merge ref was also wrong; the workflow checks out github.event.pull_request.head.sha. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Mateo Wang
|
15aa40b36e
|
test(ui): isolate OldTeams delete-warning tests from leaked mock (#30871)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
The deprecated OldTeams component takes only accessToken, userID, userRole and premiumUser; it ignores the teams prop these tests passed and instead populates its table from the mocked teamListCall. The delete-warning block never set teamListCall, and vi.clearAllMocks clears call history but not implementations, so the table rendered the "Legacy Team" (keys.length 2) left behind by the previous block's last test. Both delete tests therefore ran against that leaked team: the keys-present case passed only because the leaked count happened to be 2, and the no-keys case rendered the same warning it asserted should be absent, so it failed. Seed the team through the channel the component actually reads (teamListCall) and drop the props it never consumes, so each test renders exactly the team it declares. The keys-present case now uses a distinctive count so it can no longer pass on a coincidental leak |
||
|
Yassin Kortam
|
9c3ad1b094
|
feat(caching): add valkey-semantic cache backend and fix semantic cache scope keys (#30675)
Adds a "valkey-semantic" cache type so semantic prompt caching can run against Valkey clusters (for example AWS ElastiCache for Valkey) using the valkey-search module. The existing "redis-semantic" backend cannot drive valkey-search. RedisVL gates the connection on a RediSearch module version that valkey-search does not report, and its SemanticCache index declares the prompt as a TEXT field, which valkey-search does not implement. ValkeySemanticCache therefore talks to valkey-search directly over redis-py: it builds a vector index from the field types valkey-search supports (TAG for caller scope, VECTOR for the prompt embedding) and runs KNN queries for retrieval. Prompt extraction, embedding generation, and cached-response parsing are reused from RedisSemanticCache since those are backend agnostic. The redis dependency is imported lazily in the cache dispatch so importing litellm without redis installed still works. It also fixes semantic-cache scope keys so similarity matching works across reworded prompts. get_cache_key() hashed messages / prompt / input into the litellm_cache_key that every semantic backend filters its KNN search on, so a paraphrase landed in a different bucket and never matched, even far above the similarity threshold. For semantic cache types the prompt-bearing params are now excluded from the scope key and the server-set tenant identity (user_api_key, team, org) is appended instead, restoring embedding matching within a tenant while keeping cache entries scoped to the authenticated key / team / org. The three semantic backends share this key, so the same change fixes redis-semantic and qdrant-semantic. Connections resolve from VALKEY_HOST / VALKEY_PORT / VALKEY_PASSWORD, falling back to REDIS_* for drop-in compatibility, and passwordless clusters (IAM or no-auth) are supported. Resolves #29121 Fixes #29086 |
||
|
yuneng-jiang
|
ea17236a1e
|
fix(ui): warn that team models are deleted in the delete-team modal (#29990)
The delete-team confirmation modal warned that a team's keys would be deleted but said nothing about models. #29977 made team deletion also delete the team's BYOK models, so the modal copy was understating what gets removed. The warning banner now mentions models alongside keys, and the always-shown confirmation message does too so a team that has models but no keys (the banner only renders when keys exist) still gets warned. |
||
|
yuneng-jiang
|
60dc8420ed
|
fix(ui): repoint dead usage guide link to cost tracking docs (#30859)
The "View Usage Guide" button on the legacy Usage page (shown when DISABLE_EXPENSIVE_DB_QUERIES is set, i.e. SpendLogs has 1M+ rows) linked to docs/proxy/spending_monitoring, which was removed from the docs and now returns 404. Point it at docs/proxy/cost_tracking, which is live. Fixes LIT-2724 |
||
|
Yassin Kortam
|
4847fa5dd5
|
fix(proxy): record partial spend on the failure row for interrupted streams (#30788)
A streaming request that breaks mid-flight, for example on a mid-stream read timeout, still bills the provider for the chunks already delivered, yet the proxy recorded that interrupted request as a zero-spend failure. An earlier revision logged the recovered partial usage through the success path, which mislabeled a failed request as a success and produced a misleading spend row This recovers the partial usage where the failure is actually logged. The streaming handler assembles the usage from the chunks seen so far and stashes it, with its cost, on the logging object before firing the failure handlers. The proxy failure hook lifts that usage and cost onto request_data before the non-serialisable logging object is popped, and the spend-log writer records the real partial spend on the failure row instead of a hardcoded zero; get_logging_payload honors the recovered usage for the token columns and _failure_handler_helper_fn preserves the recovered cost so the non-DB failure loggers stay consistent A request that recovers via a successful fallback is unaffected: the failure hook only fires when the whole request fails, so the fallback's combined-usage success row stays the single source of truth and there is no double counting Resolves LIT-3825 Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
bd74c62ff1
|
fix(passthrough): recover output tokens for interrupted anthropic streams (#30787) | ||
|
yucheng-berri
|
1f9323792c
|
fix(otel): one v2 logger owns the global provider; scope tenant OTLP creds per exporter (#30590)
* fix(otel): one v2 logger owns the global provider; scope tenant creds per exporter The proxy published the OTel global TracerProvider before callbacks were initialized, so no preset logger existed yet and a second generic logger was built that won the global provider. Server spans then exported through a different provider than the preset's gen-ai spans, orphaning the LLM span on the preset backend. Publish after callback init and reuse the already-built logger instead. Separately, per-request tenant OTLP credentials were stamped onto every OTLP exporter, leaking one backend's key onto a co-configured backend. Tag each exporter with the preset that contributed it and apply dynamic credentials only to the matching owner. * fix(otel): satisfy Any-discipline on changed lines Type the logger-selection parameter as Sequence[object] (isinstance narrows it), cast the list[Any] global at the single call site, and pass model_copy a typed dict[str, str] update so no changed line carries an Any value. * fix(otel): annotate the untyped-global boundary with any-ok select_global_otel_v2_logger consumes litellm._in_memory_loggers, a shared List[Any] global this change does not own. A cast doesn't satisfy the Any-discipline checker (it inspects the inner expression), and re-annotating the global is out of scope, so mark the single boundary line any-ok. * test(otel): cover the startup global-provider publish via injectable helper The publish step lived inline in proxy_startup_event (a FastAPI lifespan unit tests do not execute), so its lines were uncovered though the selection logic was tested. Extract publish_global_otel_v2_provider, which selects the single v2 logger and publishes its provider through an injected setter, and unit-test that the published provider is the selected logger's. proxy_server delegates to it. * refactor(otel): select global provider from the registered owner, not a list scan The startup publish picked the global TracerProvider by scanning _in_memory_loggers for the first OpenTelemetryV2, re-deriving an answer the factory already settled: the first logger built registers itself as proxy_server.open_telemetry_logger, and every other v2 path (guardrail, identity seeding, phase spans) routes through that owner via _registered_v2_logger. Pass that owner into select_global_otel_v2_logger so the global provider reuses the same logger instead of an independent, order-dependent guess; the list scan remains the SDK-path fallback. The owner is injected at the proxy call site to keep the helper free of hidden global reads. * refactor(otel): type ExporterSpec.owner as an ExporterOwner enum The owner field carried free-form strings that had to match preset callback names. Introduce a str-based ExporterOwner enum (values equal to the callback names, so per-request credential routing's owner==callback_name comparison still holds) and have each preset tag its exporter with the enum member. * refactor(otel): rename ExporterOwner.ARIZE to ARIZE_AX Distinguish the hosted Arize AX backend from Arize Phoenix at the member level while keeping the value 'arize' (the public callback name routing compares against). Add a comment noting AX and Phoenix are separate backends. |
||
|
Mateo Wang
|
1bd603d1ac
|
chore(typing): add boto3/botocore stubs so basedpyright resolves the AWS SDK (#30815)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
|
||
|
Mateo Wang
|
31eca17007
|
chore: make pr template linear portion clearer (#30766) | ||
|
Mateo Wang
|
f9b8b9700c
|
fix(proxy): use e.request_data for logging_obj in ModifyResponseException streaming passthrough (#30800)
* fix(proxy): use e.request_data for logging_obj in ModifyResponseException streaming passthrough
When a guardrail blocks a streaming request pre-call by raising
ModifyResponseException (or RejectedRequestError), chat_completion streams the
violation message back as a 200 by building a CustomStreamWrapper. It read the
logging object from the outer request body (`data.get("litellm_logging_obj")`),
but that dict never carries litellm_logging_obj -- it diverges from the
processor's data at function_setup, and only the processor copy (exposed as
e.request_data, already bound to `_data` here) gets the logging object
attached. CustomStreamWrapper.__init__ then dereferences
`logging_obj.model_call_details` on None and 500s the request with
"AttributeError: 'NoneType' object has no attribute 'model_call_details'".
Read logging_obj from `_data` (= e.request_data) in both streaming
passthrough handlers so the refusal streams correctly. The non-streaming and
the anthropic/responses passthrough paths were unaffected.
Adds a regression test asserting the wrapper receives the logging object from
e.request_data rather than None.
* test(proxy): cover RejectedRequestError streaming passthrough
The streaming logging_obj fix was applied to both the ModifyResponseException
and RejectedRequestError handlers, but only the former had a regression test.
Extract a shared helper and add a parallel test for the RejectedRequestError
streaming path so both handlers stay guarded against the None-logging_obj crash.
---------
Co-authored-by: Joseph Barker <joseph.barker@rubrik.com>
|
||
|
yuneng-jiang
|
7e5699c7ab
|
ci(zizmor): gate PRs on medium+ findings and clear existing ones (#30797)
Switch the zizmor check to fail on any finding at medium severity or above (advanced-security off, min-severity medium, annotations on) so it can be promoted to a required check, and pin the engine to zizmor 1.24.1 through zizmor-action v0.5.6 for deterministic runs. Clear the findings that were outstanding so the check passes: correct mismatched action pin version comments, scope the proxy endpoint workflow's id-token and pull-requests permissions to the jobs that use them, and mark the server-root-path docker build as non-publishing while dropping its shared gha build cache. |
||
|
yucheng-berri
|
5637b3212e
|
feat(proxy): configurable response headers and login-page hint (#30792)
* feat(proxy): add configurable response headers middleware Adds a small ASGI middleware that sets standard response headers (X-Frame-Options, Content-Security-Policy frame-ancestors, X-Content-Type-Options) on proxy and UI responses. Strict-Transport-Security is optional and gated behind LITELLM_ENABLE_HSTS for HTTPS deployments. Values use setdefault so a route that sets its own header is preserved. * feat(proxy/ui): make login page credentials hint configurable build_ui_login_form accepts a hide_default_credentials_hint parameter and google_login reads LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (or general_settings) so the legacy login page behaves consistently with the new UI. Also collapses a duplicated branch and removes an unused variable and module-level constant. * fix(proxy/ui): apply credentials hint flag on /fallback/login The /fallback/login handler still rendered the default-credentials hint regardless of LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT. Collapse its duplicate branch and forward the flag, matching google_login, so all login surfaces behave consistently. Adds regression tests for /fallback/login and makes the ui_sso test helper restore os.environ so env vars do not leak across tests. |
||
|
ryan-crabbe-berri
|
32bdd004bd
|
feat(ui): migrate api-keys landing to App Router path route (#30699)
Cut the default "Virtual Keys" landing (?page=api-keys) over to a path route at (dashboard)/api-keys. The dashboard is extracted into a shared ApiKeysDashboard component used by both the new route and the index's inline render, so there's no duplication. Adding the MIGRATED_PAGES entry repoints the sidebar item and redirects ?page=api-keys to /ui/api-keys. The index is the post-login landing and still hosts the legacy switch for the not-yet-migrated pages (models, pass-through, usage) plus the invitation flow, so it stays. The auto-redirect now fires only for an explicit ?page= param, leaving the bare /ui/ landing to render inline; this keeps the return-URL handling and the invitation_id flow (both of which run at the bare landing) intact, where a blanket redirect would have dropped them. The new route uses useAuthorized for the login gate, matching every other migrated route. |
||
|
ryan-crabbe-berri
|
e4a53f50de
|
chore: remove in-product survey and Claude Code feedback nudges (#30773)
Delete the in-product survey and Claude Code feedback prompts end to end. Frontend: remove the src/components/survey/ module, the index page's nudge state/effects/handlers, the getInProductNudgesCall helper, and the orphaned "Disable UI nudges" toggle in the admin UI Settings page; prune the stale eslint-suppressions entries. Backend: remove the now-dead /in_product_nudges route, the InProductNudgeResponse type, and the disable_ui_nudges UI setting (Field + allowlist). Nothing read it for logic and the UISettings model is extra="allow", so existing stored configs are unaffected (the value is just no longer surfaced). schema.d.ts is regenerated and the two tests covering the removed route/setting are dropped. |
||
|
ryan-crabbe-berri
|
ba0233c4ce
|
fix(test): drop references to removed Agent Shin workflows (#30791)
PR #30784 deleted .github/workflows/review_gate.yml and triage_pr_with_llm.yml, but test_github_triage_workflows.py still listed both in its parametrize tables, so _load_workflow raised FileNotFoundError for every case naming them. Remove the two stale entries from DESTRUCTIVE_GATE_ENV and LLM_CLIENT_INSTALLER_WORKFLOWS; the remaining four workflows that still exist keep their guardrail coverage. |
||
|
yuneng-jiang
|
e7532b72df
|
ci(zizmor): also run on litellm_internal_staging (#30789)
* chore(ci): remove Agent Shin pull_request_target workflows Drop the two Agent Shin workflows that ran on the pull_request_target trigger: the PR triage workflow and the review gate. Both were dry-run and gated behind AGENT_SHIN_ENABLED, so no live automation changes. The shared scripts under .github/scripts stay in place; four other Agent Shin workflows still depend on them and run on schedule, dispatch, and issue events rather than pull_request_target * ci(zizmor): also run on litellm_internal_staging |
||
|
Sameer Kankute
|
4c25b7a13d
|
chore: litellm oss staging (#30745)
* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (#30708) OpenAI GPT-5 models require max_completion_tokens >= 16. Health checks were using 5 (proxy/health_check.py) and 10 (health_check_helpers.py), causing failures on GPT-5 models. Fixes #23836 * fix: increase health check max_tokens from 5 to 16 (#23836) (#26610) GPT-5 models enforce a minimum of 16 for max_output_tokens. The current default of 5 still causes health checks to fail for these models. Bump the non-wildcard default to 16 — the smallest value that satisfies all known provider minimums while keeping health checks lightweight. Also tightens the wildcard test assertion from a weak disjunctive check to strict key-absence. Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (#30696) * fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema. * fix: remove async keyword from test. * fix: make Bedrock Mantle Responses routing data-driven per model (#30700) * Make Bedrock Mantle Responses routing data-driven per model Route Bedrock Mantle models to the native Responses API based on each model's price-map capability signal instead of a hardcoded model-name heuristic, and derive the OpenAI-compatible base path segment per model. Responses dispatch now selects the native config when the model advertises responses support (/v1/responses in supported_endpoints, or mode=responses), both overridable via register_model and proxy model_info. This enables native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing chat-completions emulation. Capability is per-model, so gpt-oss-120b routes natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss substring. The wire path is a separate concern, driven by the existing use_openai_responses_path flag rather than a model-name match: gpt-5.x and gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat config now derives its base from the same flag, fixing gemma-4 chat-completions requests that previously went to /v1 instead of /openai/v1. Cost maps: add supported_endpoints to the gpt-oss entries (responses for the non-safeguard variants, chat-only for safeguard) and supported_endpoints + use_openai_responses_path to all three gemma-4 entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: move capability helper into bedrock_mantle package Move the Responses capability check out of utils.py into litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses, alongside its companion wire-path helper mantle_base_segment. Both are now pure functions of (model, model_cost): the price-map mode/supported_endpoints read replaces the get_model_info call, so the rules are unit-testable without patching global state and the Bedrock Mantle package is self-contained. Use str | None instead of Optional[str] on the new signatures to satisfy the ruff UP045 strict-rule gate. Add direct unit tests for both helpers. Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b now legitimately supports Responses, so it can no longer be the "None after restore" vehicle; use the chat-only safeguard variant, which isolates the register/restore effect from the model's own capability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653) The tiered cost calculator resolved a tier's per-token cost with `tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or` short-circuits on any falsy value, a tier that legitimately prices a component at 0.0 (e.g. a free-cache-read tier with cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated as missing and silently billed at the full fallback rate (input_cost_per_token / output_cost_per_token). The flat-pricing path in the same module already handles this correctly with an `is None` guard. Resolve tier costs through a small helper that mirrors it, so 0.0 is honored at both the in-range and overflow sites. No shipped model currently has a 0.0 tier cost, so this is a latent defect; the fix makes the tiered path consistent with the flat path and prevents over-charging the first time such a tier appears. Adds unit tests covering the in-range and overflow paths, and drops an unused import flagged by ruff in the touched test file. * feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507) * fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (#30618) In the messages->chat/completions bridge, translate_anthropic_tools_to_openai merged every non-mapped tool key into the function parameters dict. The Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object' -> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type). Exclude 'type' from the passthrough. Fixes #30557. * fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183) An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the running query-engine and spawns a new one. That planned kill was indistinguishable from a crash, and three reconnect paths used two uncoordinated locks, so a single refresh triggered a cascade of engine kill/respawn cycles: 1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old engine, spawn new one. 2. The engine-death watcher sees that kill, assumes a crash, and calls `attempt_db_reconnect(force=True)` (a different lock, `_db_reconnect_lock`) -> recreate again -> kills the fresh engine. 3. In-flight queries failing during the swap are classified as transport errors and trigger their own `attempt_db_reconnect` -> recreate again. Fix coordinates planned restarts across the wrapper and the watcher: - PrismaWrapper records the old engine PID in `_expected_engine_deaths` before killing it; all four watcher death-detectors (waitpid thread, pidfd, already-dead probe, os.kill poll) consume that PID and skip the reconnect instead of treating it as a crash. - `recreate_prisma_client` now serializes through `_reconnection_lock` and bumps a monotonic `_engine_generation`. Callers pass `expected_generation` as an optimistic-lock token, so racing/cascading recreates collapse into a single restart (losers no-op). This closes the two-lock gap. - The direct reconnect path probes the writer with SELECT 1 before recreating; a healthy connection (e.g. engine already replaced by a refresh) skips the recreate entirely. - `_safe_refresh_token` coalesces: it skips when the current token still has more than the refresh buffer of runway, so stacked triggers (proactive loop + __getattr__ fallback) don't each restart the engine. An `on_engine_replaced` hook re-arms the watcher on the new PID. RoutingPrismaWrapper forwards `expected_generation` and skips recreating the reader when the writer recreate was skipped. * feat(bedrock): support file content retrieval for batch output files (#30595) Implements transform_file_content_request and transform_file_content_response in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch files. The request transform resolves the file id (direct s3:// URI or base64 unified id) to its S3 object, validates bucket and key prefix against the server-configured bucket, and SigV4-signs an S3 GetObject using the same credential and region resolution as the existing upload path. The credential and region params are validated into a typed model at the boundary, so the only untyped values left are the botocore signing primitives. Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries s3_bucket_name (previously dropped when building deployment credentials) and the managed-files hook passes the deployment credential snapshot when routing afile_content, so unified-id content retrieval works with per-model bucket config instead of only the AWS_S3_BUCKET_NAME env var. Preserves managed-file access control: the proxy file-content endpoint now rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the owner/team check that only runs for unified ids and let a caller read another tenant's batch output by its object key. Managed outputs are reachable only through their unified file id. The afile_content "not found" error now reports the caller's unified id rather than the resolved internal S3 URI. Fixes #16186, #15563 * fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646) * fix(oci): map Cohere tool array/object params to lowercase builtins OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare "List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema arrays. MLflow {{trace}} judges trip this: their tools (get_root_span, get_span) take an attributes_to_fetch array. The lowercase builtins list/dict are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but both are lowercased for consistency). Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest). Adds a unit regression on the transformed parameterDefinitions plus a gated integration test exercising an array-param tool end to end. * fix(oci): make Cohere agentic tool-calling continuation work Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges drive once a tool has been executed and its result is fed back. Request side: litellm pulled the last user message into the top-level `message` and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that ("cannot specify message if the last entry in chat history contains tool results"), and an empty message alone is rejected too ("message must be at least 1 token long or tool results must be specified"). OCI carries the current turn's results in a dedicated top-level `toolResults` field. The Cohere transform now sends an empty message, keeps the user turn in chatHistory, and puts the results in `toolResults`, matching the langchain-oracle reference. Tool results are no longer represented as chatHistory entries. Response side: tool-grounded answers come back with citations carrying `documentIds` (camelCase) and no `document_ids`, which made the required `CohereCitation.document_ids` field fail validation and sink the whole response parse. Those citations are never surfaced, so the field (and CohereSearchQuery's generation_id) is now optional. Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest), single and multi-round tool loops. Adds unit regressions on the transformed request shape and on citation parsing, plus gated integration tests for the continuation. * feat: integrate Repelloai Argus guardrail (#30673) * feat(guardrails): add RepelloAI Argus guardrail integration (#1) * feat(guardrails): add RepelloAI Argus guardrail integration Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed asset policies enforced via an asset_id and X-API-Key auth. * fix(guardrails): harden RepelloAI Argus guardrail - scan streaming responses on output (was bypassing the guardrail) - log blocked verdicts as guardrail_intervened instead of success - treat auth/config errors (401/403/404/422) as misconfiguration that always blocks, not a fail-open-able unreachable error - default unreachable_fallback to fail_closed and read it directly; block on unknown/malformed verdicts so an API change can't silently disable enforcement - type unreachable_fallback as a Literal, drop the duplicate config model, expose unreachable_fallback in the config schema, and stop leaking the raw provider response / exception strings to the client * fix(guardrails): address RepelloAI Argus review feedback - support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback) - make asset_id required in the config model - normalize unreachable_fallback so only fail_open opens; block on 400 misconfig - correct the shared unreachable_fallback field description * docs(guardrails): add RepelloAI Argus docs page and dashboard listing - add docs page covering config, env vars, modes, verdicts, failure semantics - list RepelloAI Argus in the Guardrail Garden with provider/logo mappings - add a regression test for the provider logo and display-name resolution * fix(guardrails): keep RepelloAI asset_id optional in config model A required asset_id leaked onto the shared LitellmParams (which inherits RepelloAIGuardrailConfigModel), breaking validation for every other guardrail. Keep it optional like sibling models; the guardrail __init__ still raises when asset_id is missing, which is the real enforcement. * Add comment for last user turn scanning * feat(guardrails): harden repelloai scanning * feat(guardrails): expand repelloai scanning to include tool definitions Add extraction of tool definitions and tool call arguments to the RepelloAI guardrail scanning. Improves detection coverage by including function schemas and parameters in the prompt sent to the guardrail service. Also captures detailed error responses in logs and adds guardrail header to streaming responses. * refactor(guardrails): fix and harden repelloai schema text extraction - Fix duplicate text in _iter_schema_text: previously all dict values were re-queued onto the stack even after scalar/list keys were already extracted explicitly, causing names/descriptions to appear twice in the scanned prompt - Extract schema key frozensets to module-level constants so they are not reconstructed on every call - Change _iter_schema_text from @classmethod to @staticmethod (cls unused) - Narrow _call_analyze stage param from str to Literal["prompt", "response"] - Add HttpxResponse type annotation to _raise_for_config_error - Add LLMResponseTypes annotation to async_post_call_success_hook response param * fix(guardrails): resolve pyright type errors in repelloai guardrail - Narrow async_handler.post return from Response|None to Response with explicit None guard before calling raise_for_status/json - Fix list comprehension returning str|None by switching to explicit loop with isinstance guard so pyright tracks the narrowing - Cast model_dump() result to Dict since hasattr does not narrow object type in pyright * fix(guardrails/repello): include Responses API instructions field in prompt scan The /v1/responses top-level `instructions` field was not included in _extract_prompt_text, allowing a caller to bypass guardrail policy checks by putting blocked content in `instructions` while keeping `input` benign. * feat: add api_key to config model and read prompt from data dict * fix(guardrails/repello): plug input_text and tool-call response bypass gaps Responses API input content parts with type 'input_text' were silently dropped by build_inspection_messages (which only handles type='text'), allowing callers to send blocked content via that path without triggering the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail and call it when walking the Responses API input messages. Post-call scanning skipped responses whose choices contained only tool_calls or function_call (message.content=None), letting models put blocked output in function arguments undetected. Fix: _extract_chat_completion_text now calls _extract_tool_call_args_from_message on each choice message. Also replace typing.Dict/List with builtin dict/list to clear TID251 strict ruff violations introduced by this file. * fix(guardrails/repello): scan Responses API function_call output arguments Output items with type 'function_call' in a /v1/responses response were skipped by _extract_responses_api_text; only 'message' items were walked. A model could return blocked content in function_call.arguments undetected. Now extract arguments from function_call output items before scanning. * refactor(guardrails/repello): clean up typing and remove lint-any workarounds - Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout - Use dict[str, object] instead of bare dict in all signatures - Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly - Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel - Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks - Use TypeAdapter.validate_json() instead of response.json() + manual dict construction - Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any - Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check - Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType - Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel] * fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning - Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate - Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks * refactor: modifications for lint check * feat: add Pinstripes as an OpenAI-compatible provider (#30567) * feat: add Pinstripes as an OpenAI-compatible provider Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.) with per-token pricing and no subscriptions. Changes: - `litellm/llms/openai_like/providers.json`: register pinstripes with base_url, api_key_env, and max_completion_tokens→max_tokens mapping - `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders - `litellm/constants.py`: add to openai_compatible_providers and openai_compatible_endpoints lists - `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect provider when api_base is "https://pinstripes.io/v1" - `provider_endpoints_support.json`: document supported endpoints - `tests/`: 7 unit tests covering provider registration, resolution, URL auto-detection, api_base override, and Router config Usage: import litellm response = litellm.completion( model="pinstripes/ps/glm-4.5-air", messages=[{"role": "user", "content": "Hello"}], api_key=os.environ["PINSTRIPES_API_KEY"], ) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): resolve Greptile P1 review comments - Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works - Set responses: false in provider_endpoints_support.json — not actually wired up - Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): add api_base_env and correct responses capability - Add api_base_env: PINSTRIPES_API_BASE to providers.json - Set responses: false in provider_endpoints_support.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): wire up Responses API — add supported_endpoints Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so JSONProviderRegistry.supports_responses_api returns true correctly, matching what provider_endpoints_support.json advertises. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(pinstripes): enable embeddings endpoint Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings. Add /v1/embeddings to supported_endpoints and set embeddings: true. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json Matches the file's existing convention. Flagged by Greptile review. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): set a2a: false — A2A protocol not implemented All comparable JSON-configured providers (tensormesh, parasail, empiriolabs, libertai, neosantara) have a2a: false. Pinstripes does not implement the Google A2A protocol, so this should be false to match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: inference_provider <max@redactedlab.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(rag): attach existing OpenAI file ids (#30628) * fix(rag): attach existing OpenAI file ids * chore: use modern typing in rag ingest fix * chore: retrigger ci * fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341) cache_control_injection_points was only consumed by the chat/completions prompt-management hook; on the native Anthropic /v1/messages path it was forwarded unused, so deployment-level cache injection was silently dropped (cache_creation_input_tokens stayed 0 for Anthropic-native clients). Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject cache_control at block level for system / tools / message locations (the only forms /v1/messages accepts), wire it into the native anthropic_messages handler, and pop the param so it does not leak upstream as an unknown field. A {location: message, role: system} config is redirected to the top-level system prompt so the same YAML works on both endpoints. Injection respects Anthropic's 4-block cache_control limit shared across system, tools, and messages: client-supplied markers count toward the cap and are never overwritten, a slot is reserved per Bedrock tool_config point, and injection stops once the budget is exhausted. Locations this path cannot represent (tool_config) are forwarded downstream instead of being silently consumed, mirroring get_chat_completion_prompt's remaining_points pass-through. Built on litellm_internal_staging. Refs BerriAI/litellm#30293 * fix(proxy): release budget reservation when a request is cancelled mid-flight (#30522) * fix(proxy): release budget reservation on cancel when no chunk was delivered The pre-call budget reservation increments the cross-pod spend counter by a request's worst-case cost, then reconciles it on success (cost callback) or error (failure hook). A client disconnect or timeout cancels the request and surfaces as CancelledError / GeneratorExit, which neither path catches, so the reservation leaks. Under a retry storm the leaked holds accumulate, pin the counter above real spend, and return spurious 429 "Budget has been exceeded" to keys whose spend is far below budget; the counter only recovers when its TTL lapses, so the failure is intermittent and self-healing. Release the reservation in async_streaming_data_generator (which the Anthropic and Google SSE generators delegate to) on the (CancelledError, GeneratorExit) path, alongside the existing max_parallel_requests release. release_budget_ reservation_on_cancel runs under asyncio.shield so it completes despite the in-progress cancellation, is guarded by the reservation's finalized flag, and swallows a failing release so it cannot replace the in-flight cancellation. The refund is gated on whether a chunk reached the client. The flag is set immediately before the yield, after the slow-path hook await: an async generator suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk sees it True (keep the hold), while a cancellation during the slow-path await leaves it False (refund, nothing sent). A non-streaming cancellation delivers nothing and a completed non-streaming response is reconciled by the success callback, so neither needs a release here. Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): reconcile a cancelled reservation to input cost, not zero A streaming request cancelled before the first chunk previously reconciled its reservation to zero and finalized it. But by the time the generator is consuming the response the provider call was already dispatched, so the input tokens were billed even though no chunk reached the client, and the success/failure cost callbacks are skipped on cancellation. Refunding to zero let a caller send an expensive request and abort pre-token to dodge the input charge. Compute the request's input-token cost at reservation time and reconcile the cancelled reservation to it instead of zero. The worst-case output portion of the reservation is still released (so a legitimate mid-flight cancellation no longer pins the counter and 429s the key), while the input the provider already processed is charged. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(caching): encode object name in GCS cache GET path (#30378) GCS cache reads always missed when gcs_path was set. The GET methods interpolated the object name directly into the URL path, while the GCS JSON API requires it to be URL-encoded (a "/" must be sent as %2F). With gcs_path configured the object name is "<prefix>/<sha256>", so the raw slash produced a malformed object path and GCS returned 404. httpx does not raise on 4xx, so the status_code == 200 check fell through and get/async_get returned None, silently missing on every read. Without gcs_path the key has no slash, which is why this went unnoticed. Wrap the object name with urllib.parse.quote(..., safe="") in get_cache and async_get_cache. Apply the same encoding to the name= query parameter in set_cache and async_set_cache so the key written matches the key read back. Adds regression tests asserting the GET path and SET query are encoded (%2F) when gcs_path is set, for both sync and async paths; these fail on the unpatched code. Fixes #30377 * chore: add soniox stt-async-v5 model (#30672) * fix(proxy): include model group aliases in v1 model info (#30626) * Include model group aliases in v1 model info * Fix model info alias implementation * removed extra blank line * chore: rerun CI * fix(lint): remove redundant noqa directive in proxy_cli.py * fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme * Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme" This reverts commit |
||
|
yuneng-jiang
|
c0352c5aa8
|
chore(ci): remove Agent Shin pull_request_target workflows (#30784)
Drop the two Agent Shin workflows that ran on the pull_request_target trigger: the PR triage workflow and the review gate. Both were dry-run and gated behind AGENT_SHIN_ENABLED, so no live automation changes. The shared scripts under .github/scripts stay in place; four other Agent Shin workflows still depend on them and run on schedule, dispatch, and issue events rather than pull_request_target |
||
|
Yassin Kortam
|
a8b94b9a87
|
fix(proxy): enforce budgets against authoritative DB spend when the cross-pod counter is unreliable (#30684)
Some checks failed
Agent Shin — rollout heads-up (one-shot) / heads-up (push) Has been cancelled
Budget enforcement reads spend from the cross-pod Redis counter via get_current_spend, which trusted the counter whenever Redis returned a value. A Redis instance that restarts and reloads an older RDB snapshot (the customer's logs repeat "Redis is loading the dataset in memory") comes back with a stale-low counter; that read is a hit, not a clean miss, so the existing DB reseed never ran and a key kept getting admitted even though its recorded spend was already over max_budget. The symptom was recorded spend sitting above the limit while requests kept succeeding. Read-time enforcement: get_current_spend takes an optional max_budget and, when the counter would admit the request but reads below this caller's last-known recorded spend, re-reads the authoritative spend and enforces against the higher value. The authoritative source depends on the counter: key/team/user/org/team-member read the DB row, per-window budgets aggregate spend logs, and end-user/tag have no DB row so the caller's freshly-loaded recorded spend is used. Healthy primary counters and freshly reset keys stay off the DB path, and the value is cached in-process for a few seconds, so a persistently stale counter drives at most one read per counter per window. When the DB value is higher, the counter is repaired with a monotonic, atomic set-max (RedisCache.async_set_max) so every worker reads the corrected total and a concurrent increment is never clobbered. Reconcile no longer fails open: when the post-call reservation reconcile found the counter missing or an adjustment that would drive it negative, it deleted the counter and continued (the deletion is what left counters nil/unenforced after a Redis reload). It now reseeds from the DB's lagging authoritative floor instead of deleting; the monotonic set-max can only raise a stale-low counter, and the read-time floor converges to the true total as the spend buffer flushes. The pre-call admission resize path keeps its original fail-closed behavior. Opt-in strict enforcement: general_settings.fail_closed_budget_enforcement (default False) makes the authoritative re-check run for every budgeted entity (closing the gap where a stale-low counter and a stale-low cached fallback would otherwise both pass the cheap guard), and rejects a request with 503 when the spend backing an admit decision can be verified against neither Redis nor the database. Default behavior is unchanged; the re-check stays bounded by the in-process cache. Resolves LIT-3772 |
||
|
ryan-crabbe-berri
|
382d78ec16
|
feat(ui): migrate old usage report to App Router path route (#30694)
Cut the legacy "Old Usage" report (?page=usage) over from the switch in
(dashboard)/page.tsx to a path route at (dashboard)/old-usage. The segment is
old-usage rather than usage because the modern usage dashboard (new_usage)
already owns /usage. Adding the MIGRATED_PAGES entry repoints the sidebar item
and redirects existing ?page=usage links to /ui/old-usage.
The report was the switch's catch-all else, so removing it means choosing a new
fallback: collapse the now-redundant explicit api-keys arm into the else so the
main dashboard (UserDashboard) is the default. Unknown ?page= values now land on
the dashboard instead of the Old Usage report, which is the sensible default.
The new route sources identity from useAuthorized() and passes keys={null}: the
key-filter dropdown read the parent's keys state, which was already empty on
direct navigation to ?page=usage, so this preserves that rather than wiring a
paginated key fetch into a deprecated report.
|
||
|
Simantak Dabhade
|
fb34c184b4
|
feat(search): add TinyFish as search provider (#30634)
* feat(search): add TinyFish as search provider Adds TinyFish web search (GET https://api.search.tinyfish.ai) as the 16th search provider in LiteLLM. Follows the BaseSearchConfig pattern used by other GET-based providers like Brave. Includes unit tests in tests/test_litellm/ for full patch coverage. * fix(search/tinyfish): use concrete types to pass any-discipline and ruff UP006/UP045 Replace typing.Dict/List/Optional/Union with modern syntax (dict, list, X | None) and use concrete type parameters (dict[str, str] for headers, dict[str, object] for params) to eliminate LIT009 Any-discipline violations. Move _append_domain_filters to module level to avoid leaking Any through self. * fix(search/tinyfish): eliminate Any-typed values for any-discipline gate Use Pydantic BaseModel and TypeAdapter at httpx/base-class boundaries to validate untyped inputs (json(), params.get(), bare set). Three genuine external boundaries annotated with any-ok. * style: fix black formatting for long line * fix(search/tinyfish): move any-ok comment to violation line for any-discipline gate The any-discipline checker matches `# any-ok` comments by line number. The comment was on the closing-paren line (127) but the violation was on the call-expression line (126), so the suppression did not apply. * fix(search/tinyfish): align with approved PR #30158 Drop explicit AND from domain filter query to match the approved implementation. Set pricing to zero. Rename test to match behavior. |
||
|
Mateo Wang
|
a9e651d994
|
fix(bedrock_mantle): add SigV4 fallback to chat completions auth (#30714) | ||
|
Sameer Kankute
|
e33e2917c6
|
chore: litellm oss 170626 (#30637)
* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes (#30089) * fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes Add the realtime WebRTC HTTP sub-routes (/realtime/client_secrets, /realtime/calls and their /v1 + /openai/v1 variants) to LiteLLMRoutes.openai_routes so is_llm_api_route() classifies them as LLM API routes. Without this, non-admin virtual keys received 401 'Only proxy admin can be used to generate, delete, update info for new keys/users/teams' when calling these endpoints. Fixes #29923 * fix(proxy): validate session.model for realtime routes in model-access check The GA Realtime WebRTC HTTP routes resolve the effective model from the nested session.model (falling back to the top-level model), but the auth layer's get_model_from_request() only extracted the top-level model. A model-restricted virtual key could therefore place a disallowed model in session.model, leave the top-level model unset, and skip can_key_call_model() entirely - obtaining an ephemeral token for a model it is not allowed to use. Extract session.model for the realtime client_secrets/calls routes so the model-access check runs against the model the request will actually use. Legitimate callers are unaffected; their permitted model still validates. Relates to https://github.com/BerriAI/litellm/issues/29923 * fix(proxy): classify realtime transcription_sessions routes as LLM API routes Add the GA Realtime WebRTC transcription_sessions HTTP routes to openai_routes so is_llm_api_route() returns True for them, matching the client_secrets and calls routes already fixed. These endpoints are registered with user_api_key_auth in realtime_endpoints/endpoints.py, so without this a non-admin virtual key calling POST /v1/realtime/transcription_sessions would hit the admin-only 401 branch. Extends the regression test parametrization accordingly. --------- Co-authored-by: habonlaci <4699494+habonlaci@users.noreply.github.com> * feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models (#30272) * feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models * fix(proxy): degrade /v1/models gracefully when model-group lookup fails --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: sort tiered token-cost thresholds numerically (#30375) * fix: sort tiered token-cost thresholds numerically _get_token_base_cost iterated input_cost_per_token_above_<N>_tokens keys with a lexicographic sort, so for tiers whose thresholds have different digit lengths (e.g. 90k vs 128k) a request crossing both was billed at the lower tier that sorted first. Sort by the parsed numeric threshold instead, so the highest tier the request actually crosses is applied. * refactor: reuse _parse_above_token_threshold for inline threshold parse --------- Co-authored-by: Eric (GabiDevFamily) <271972409+santino18727-debug@users.noreply.github.com> * fix(openai): preserve cache_control for openai-compatible custom endpoints (#30387) * fix(openai): preserve cache_control for openai-compatible custom endpoints * fix(openai): use parsed hostname to detect real OpenAI for cache_control preservation * fix(proxy): drain all daily-spend batches per flush cycle (#30281) (#30505) * fix(types): prevent internal parallel_request_limiter fields from leaking to upstream providers (#30545) * fix(types): add internal parallel_request_limiter fields to all_litellm_params to prevent forwarding to upstream providers * test(types): add regression test for internal rate-limit fields in all_litellm_params * fix(init): add bool type annotation to suppress_debug_info (#30531) Module-level `suppress_debug_info = False` had no annotation, so strict type checkers (e.g. ty) infer it as `Literal[False]`. Reassigning it to `True` (as done in proxy_server.py and router.py) then fails with an invalid-assignment error. Annotate it as `bool` to match every other flag in this module. * fix: coalesce null aggregates in update_metrics for no-spend keys (#29945) * feat(team_endpoints): add query parameter `key_limit` to `/team/info` endpoint (#30006) * feat(team_endpoints): Add query parameter key_limit to /team/info * feat(team_endpoints): update schema.d.ts to include the new query parameter * feat(team_endpoints): add tests for limitting key count in /team/info response * feat(team_endpoints): Apply suggestions from greptile * Set greater-than constraint on key-limit * Fix type * fix(router): release aiohttp connection when stream iteration ends abnormally (#30271) * fix(router): release aiohttp connection when stream iteration ends abnormally A streaming response that terminates with a mid-stream read timeout, a task cancellation (client disconnect), or GeneratorExit never closed the underlying aiohttp ClientResponse. aiohttp only auto-releases the connector slot at body EOF, so each abnormally terminated stream permanently leaked one slot from the shared TCPConnector pool. During a backend traffic spike the pool drains; once exhausted every subsequent request to that host waits for a slot, times out and surfaces as a 408, indefinitely, even after the backend recovers. Only a proxy restart cleared the in-memory sessions, which matched the reported symptom of a router stuck returning 408 for a healthy vLLM backend. Close the response in a finally clause when iteration ends. On a fully read response the connection was already released at EOF and close() is a no-op, so keep-alive reuse for normal requests is unchanged. Fixes #30192 * test(aiohttp): cover GeneratorExit path with a mock instead of a live socket The previous slot-release test started a real aiohttp TCP server, which can flake in offline CI and does not exercise this fix's code path directly. Replace it with a dependency-injected mock that closes the stream generator (GeneratorExit) and asserts the response is closed, covering the third abnormal-exit path the finally block handles * feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (#30273) * feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery * refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils * fix(proxy): make model_list request param optional for direct callers * feat(dashscope): add Responses API support (#30286) * feat(dashscope): add Responses API support DashScope's OpenAI-compatible endpoint serves /responses, so register a DashScopeResponsesAPIConfig that routes dashscope/* responses calls to {api_base}/responses without rewriting the upstream model id, instead of falling back to the chat-completions -> responses emulation pipeline. Closes #29780 * feat(dashscope): mark responses API as not supporting native websocket Matches the hosted_vllm/perplexity/openrouter responses configs, which all override supports_native_websocket() to False since the OpenAI-compatible endpoint has no native wss:// responses transport. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(spend-logs): preserve error_message on ProxyException failures (#30381) * fix(spend-logs): preserve error_message on ProxyException failures `StandardLoggingPayloadSetup.get_error_information` used `str(original_exception)` to populate the human-readable error message stored in `spend_logs.metadata.error_information.error_message`. `ProxyException` (litellm/proxy/_types.py:3453) sets `self.message` in its constructor but does NOT call `super().__init__(message)` and does NOT define `__str__`. As a result, `str(ProxyException(...))` returns the empty string, and every auth/budget/quota rejection was landing in spend_logs with `error_message=""` despite a fully populated traceback. Operator impact: dashboard "LLM Failure" rows became untriageable — the only way to tell a 401 from a 429 was to manually unpack the traceback JSON via psql. Burst failure patterns (e.g. a UI session polling with a stale token) produced 20-30 indistinguishable `error_code=401` rows per second. Fix: prefer the `.message` attribute (set by ProxyException and every litellm.exceptions.* class) over `str(exc)`. The `str(exc)` fallback is retained for non-litellm exception types, preserving prior behavior. Test plan: - 2 new unit tests in tests/test_litellm/litellm_core_utils/ test_litellm_logging.py: * test_get_error_information_prefers_message_attribute_over_str * test_get_error_information_falls_back_to_str_when_no_message_attr - Existing test_get_error_information_error_code_priority still passes - End-to-end verified: bad-key 401 now stores full "Authentication Error, Invalid proxy server token passed..." message in spend_logs.metadata.error_information.error_message * fix(spend-logs): preserve explicit empty .message + drop dead reference Greptile P2 on #30381. The truthiness check `if message_attr:` silently skipped an explicit empty-string `.message` and fell through to `str(original_exception)`. For ProxyException-shaped objects both produce empty, so the bug was latent; for other exception types it would inject a different string into error_information.error_message and corrupt the signal. Use `is not None` so an empty string survives verbatim. Also drop the stale `See e2e/cases/11.` comment reference — that path does not exist anywhere in the repo and confuses future readers. Regression test added: an exception with `.message=""` and a non-empty `super().__init__()` arg must yield error_message == "". * ci: retrigger workflows after base branch change to litellm_internal_staging * fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response (#30382) * fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response The non-streaming /v1/messages response carries a LiteLLM-injected usage.total_tokens = input_tokens + output_tokens that is not part of the Anthropic API spec. This caused three problems: 1. Shape divergence with streaming on the same endpoint. message_delta.usage in the SSE path never carries total_tokens. Clients parsing both paths get two different schemas from one endpoint. 2. Shape divergence with upstream. Direct calls to https://api.anthropic.com/v1/messages return no total_tokens field, so clients using the official Anthropic SDK couldn't rely on it, and clients that did rely on the LiteLLM-injected one broke when bypassing the proxy. 3. Numerical misuse. total = input + output undercounts when cache_read_input_tokens and cache_creation_input_tokens are non-zero, because cache tokens are reported in their own fields. A 100k-token cached prompt with 1 non-cache input token + 200 output tokens reports total_tokens = 201, off by ~99.8% from any reasonable definition of "total." Fix: add _strip_total_tokens_from_anthropic_response in litellm/proxy/anthropic_endpoints/endpoints.py and invoke it in the success path of anthropic_response right before returning. Only mutates dict-shaped responses; streaming (which already lacks the field) is left untouched. spend_logs / Prometheus continue to compute total_tokens internally for billing — this fix only strips the field from the wire response. Scope: only the Anthropic passthrough endpoint /v1/messages. The OpenAI-shape /v1/chat/completions is unaffected. * fix(anthropic): gate total_tokens strip behind flag + handle Pydantic .usage Two P1 greptile threads on #30382: P1 — **Backwards-incompatible removal without a feature flag** Stripping `usage.total_tokens` unconditionally breaks any client currently reading the LiteLLM-shaped non-streaming /v1/messages response. Per the codebase's policy (mirrors #30418), gate behind a new flag. - `litellm.strip_anthropic_total_tokens: bool = False` (default — backward-compat: clients keep seeing total_tokens). - Env override: `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS=true`. - Docstring: planned to flip to True in a future major release; opt in early. P1 — **Silent no-op if `result` is a Pydantic model** `base_process_llm_request` may return a Pydantic-style object whose `.usage` is a plain dict (the most common shape — e.g. objects wrapping raw upstream JSON). The original `isinstance(response, dict)` guard skipped strip on those, so `total_tokens` would still hit the wire. Helper now also reads `getattr(response, "usage", None)` and strips when that's a dict. Strongly-typed Pydantic `Usage` sub-models with required `total_tokens` fields are still skipped — those impose type constraints the helper doesn't try to subvert. Tests: - `test_strips_total_tokens_on_pydantic_model_with_dict_usage` - `test_flag_defaults_off` 8/8 pass locally. * fix(anthropic): drop env var for strip flag (docs CI) Mirrors #30418's pattern (`expose_router_debug_in_errors: bool = True`, no `os.getenv`). The `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS` env var introduced in the prior commit was flagged by `tests/documentation_tests/test_env_keys.py` because the documentation file `docs/my-website/docs/proxy/config_settings.md` lives in `BerriAI/litellm-docs` (separate repo) and registering a new env key requires a parallel docs PR — a friction we avoid here by exposing the flag only as a Python attribute + `litellm_settings` config key, both of which load through the existing proxy config plumbing without needing the env-var registry to be updated. No semantic change: default still False, behavior identical when set via `litellm.strip_anthropic_total_tokens = True` or `litellm_settings.strip_anthropic_total_tokens: true` in config.yaml. Verified locally: env scan no longer surfaces the key; 8/8 tests pass. * ci: retrigger workflows after base branch change to litellm_internal_staging * fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 (#30413) * fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 * test: resolve model prices JSON relative to test file for pip installs * fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError (#30417) * fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError Some Gemini-compatible gateways (e.g. new-api) wrap a 429 rate-limit signal from upstream inside an HTTP 500/503 envelope, with the real code only surfaced in the JSON body: {"error":{"message":"...high demand...","type":"upstream_error", "param":"","code":429}} Previously LiteLLM only looked at the HTTP status and mapped this to InternalServerError, which Router treats as non-retryable for many configs — so users got hard 500s instead of fallback/retry. Now the Gemini/Vertex exception mapper parses error.code from the body and routes code 429 to RateLimitError before falling through to the HTTP-status branches. Other body codes fall through unchanged. Tests cover: - new-api gateway's `code:429` payload now maps to RateLimitError - Genuine 500-body responses stay InternalServerError - Non-JSON body strings fall through to status-code mapping unchanged * fix(exception-mapping): scope body-code 429 promotion to 5xx envelopes Addresses greptile P1/P2 + @Sameerlite's review on #30417. The new elif branch was firing for any HTTP status, so a gateway response of HTTP 400 with body {"error":{"code":429,...}} would be incorrectly promoted to RateLimitError (retryable) instead of falling through to BadRequestError. Same trap for 401 -> AuthenticationError. Scoped the body-code 429 check to `500 <= status_code < 600` — covers 500/502/503/504 (gateways wrapping upstream 429 in any 5xx envelope) without inviting the 4xx misclassification. Tests: parametrized table now covers 5xx (500/502/503), 4xx (400/401), and the existing fall-through cases, asserting each maps to the exception type that matches the HTTP status code. 50/50 pass locally. * ci: retrigger workflows after base branch change to litellm_internal_staging * feat(router): add expose_router_debug_in_errors flag (default True) to redact internal model_group/fallback names (#30418) * feat(router)!: redact internal model_group/fallback names from exception messages The Router was unconditionally appending internal config names onto exception.message: - "Received Model Group=..." - "Available Model Group Fallbacks=..." - "No fallback model group found... Fallbacks={...}" - "context_window_fallbacks={...}" - Deployment-timeout messages including model_group - Fallback failure detail listing fallback chain ProxyException forwards .message verbatim to clients, so gateways were leaking their model_name / fallback wiring in every failed call. Fix: gate all five mutation sites on a new `litellm.expose_router_debug_in_errors` flag (default False). Set to True to restore upstream debug behavior for local debugging. Why: matches the redaction posture this codebase already has for upstream model identifiers (cf. _litellm_returned_model_name) and removes the last common error-path leak of internal model_group names. Breaking change marker (!): if anything parses "Received Model Group=" out of client error messages, flip the flag on or migrate to the x-litellm-* response headers instead. Tests: 7 cases covering each of the 5 redaction sites + the flag-on inverse path, plus a "default off" sanity check. * test(router): cover sites 1 + 3 of expose_router_debug_in_errors gate Addresses Greptile / codecov feedback on #30418: patch coverage was 55.6% with 4 lines uncovered in litellm/router.py. The existing tests exercised sites 2 (ContextWindowExceededError), 4 (no-fallback-found), and 5 (Received Model Group) — both default and flag-on. Sites 1 and 3 were declared in the PR description as covered by "site 5 also fires" but the gate body lines for each (the `e.message +=` inside the `if litellm.expose_router_debug_in_errors:` branch) only execute when the flag is on AND the specific exception path is taken, which neither existing test triggered. Added 4 new tests (default + flag-on × 2 sites): - test_default_does_not_leak_deployment_timeout_debug - test_flag_on_leaks_deployment_timeout_debug - test_default_does_not_leak_content_policy_fallback_hint - test_flag_on_leaks_content_policy_fallback_hint Trigger details: - Site 1 (litellm.Timeout in _acompletion) is reached via the Router-supported `mock_timeout=True` + `timeout=0.001` kwargs on `acompletion(...)`. Cannot embed a Timeout instance in model_list because Router.__init__ deep-copies it and Timeout.__reduce__ does not preserve the required positional args. - Site 3 (ContentPolicyViolationError without content_policy_fallbacks set, in async_function_with_fallbacks_common_utils) is reached by passing a `mock_response=litellm.ContentPolicyViolationError(...)` instance via the call-site kwarg — same deepcopy-avoidance reason. 11/11 tests pass locally. Patch coverage on litellm/router.py for this PR's diff should now be 100%. * chore(router): flip expose_router_debug_in_errors default to True Addresses @Sameerlite's review on #30418 — maintain backward compat on the wire. Redact becomes opt-in via setting the flag to False; the historical behavior (leak internal model_group / fallback wiring through exception messages) is preserved as the default. - litellm/__init__.py: default flipped to True, docstring rewritten with deprecation note pointing at a future flip to False (redact by default) in a major release. - tests/test_litellm/test_router_exception_redaction.py: fixture resets to True (was False); the "off" tests now explicitly set False; the "default_leaks_*" tests rely on the fixture default. test_flag_defaults_off -> test_flag_defaults_on. - No router.py change needed; the gate keys off the same flag, only the default changes. - PR title no longer needs the breaking-change `!` marker — no client sees a behavior change at default settings. 11/11 pass locally. * ci: retrigger workflows after base branch change to litellm_internal_staging * feat(guardrails): integrate Repelloai Argus guardrail (#30465) * feat(guardrails): add RepelloAI Argus guardrail integration (#1) * feat(guardrails): add RepelloAI Argus guardrail integration Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed asset policies enforced via an asset_id and X-API-Key auth. * fix(guardrails): harden RepelloAI Argus guardrail - scan streaming responses on output (was bypassing the guardrail) - log blocked verdicts as guardrail_intervened instead of success - treat auth/config errors (401/403/404/422) as misconfiguration that always blocks, not a fail-open-able unreachable error - default unreachable_fallback to fail_closed and read it directly; block on unknown/malformed verdicts so an API change can't silently disable enforcement - type unreachable_fallback as a Literal, drop the duplicate config model, expose unreachable_fallback in the config schema, and stop leaking the raw provider response / exception strings to the client * fix(guardrails): address RepelloAI Argus review feedback - support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback) - make asset_id required in the config model - normalize unreachable_fallback so only fail_open opens; block on 400 misconfig - correct the shared unreachable_fallback field description * docs(guardrails): add RepelloAI Argus docs page and dashboard listing - add docs page covering config, env vars, modes, verdicts, failure semantics - list RepelloAI Argus in the Guardrail Garden with provider/logo mappings - add a regression test for the provider logo and display-name resolution * fix(guardrails): keep RepelloAI asset_id optional in config model A required asset_id leaked onto the shared LitellmParams (which inherits RepelloAIGuardrailConfigModel), breaking validation for every other guardrail. Keep it optional like sibling models; the guardrail __init__ still raises when asset_id is missing, which is the real enforcement. * Add comment for last user turn scanning * feat(guardrails): harden repelloai scanning * feat(guardrails): expand repelloai scanning to include tool definitions Add extraction of tool definitions and tool call arguments to the RepelloAI guardrail scanning. Improves detection coverage by including function schemas and parameters in the prompt sent to the guardrail service. Also captures detailed error responses in logs and adds guardrail header to streaming responses. * refactor(guardrails): fix and harden repelloai schema text extraction - Fix duplicate text in _iter_schema_text: previously all dict values were re-queued onto the stack even after scalar/list keys were already extracted explicitly, causing names/descriptions to appear twice in the scanned prompt - Extract schema key frozensets to module-level constants so they are not reconstructed on every call - Change _iter_schema_text from @classmethod to @staticmethod (cls unused) - Narrow _call_analyze stage param from str to Literal["prompt", "response"] - Add HttpxResponse type annotation to _raise_for_config_error - Add LLMResponseTypes annotation to async_post_call_success_hook response param * fix(guardrails): resolve pyright type errors in repelloai guardrail - Narrow async_handler.post return from Response|None to Response with explicit None guard before calling raise_for_status/json - Fix list comprehension returning str|None by switching to explicit loop with isinstance guard so pyright tracks the narrowing - Cast model_dump() result to Dict since hasattr does not narrow object type in pyright * fix(guardrails/repello): include Responses API instructions field in prompt scan The /v1/responses top-level `instructions` field was not included in _extract_prompt_text, allowing a caller to bypass guardrail policy checks by putting blocked content in `instructions` while keeping `input` benign. * feat: add api_key to config model and read prompt from data dict * fix(guardrails/repello): plug input_text and tool-call response bypass gaps Responses API input content parts with type 'input_text' were silently dropped by build_inspection_messages (which only handles type='text'), allowing callers to send blocked content via that path without triggering the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail and call it when walking the Responses API input messages. Post-call scanning skipped responses whose choices contained only tool_calls or function_call (message.content=None), letting models put blocked output in function arguments undetected. Fix: _extract_chat_completion_text now calls _extract_tool_call_args_from_message on each choice message. Also replace typing.Dict/List with builtin dict/list to clear TID251 strict ruff violations introduced by this file. * fix(guardrails/repello): scan Responses API function_call output arguments Output items with type 'function_call' in a /v1/responses response were skipped by _extract_responses_api_text; only 'message' items were walked. A model could return blocked content in function_call.arguments undetected. Now extract arguments from function_call output items before scanning. * fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (#30486) * fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients When an Anthropic server-side tool (web_search, id `srvtoolu_...`) is used, its result is carried in `provider_specific_fields.web_search_results` — PRs #17746 / #17798 restore it for callers that round-trip provider_specific_fields. A generic OpenAI client that does NOT preserve provider_specific_fields (e.g. Open WebUI talking to a Vertex/Anthropic model over /chat/completions) drops it on replay and instead sends back an assistant `tool_call` + a `tool` message both keyed to the `srvtoolu_` id. The transform then produced a bare `server_tool_use` (with no following *_tool_result) plus a user `tool_result` for the same id — both invalid, so the next turn 400s: messages.N.content.0: unexpected `tool_use_id` found in `tool_result` blocks: srvtoolu_... Each `tool_result` block must have a corresponding `tool_use` block in the previous message. This is the commonly-reported vertex_ai symptom where Gemini works but Claude 400s on the 2nd turn of a web-search chat. Fix (litellm/litellm_core_utils/prompt_templates/factory.py): - convert_to_anthropic_tool_invoke: only emit a server_tool_use when its matching *_tool_result is available to pair with it; otherwise skip it (a bare server_tool_use is itself rejected). - anthropic_messages_pt: drop a replayed `tool`/`function` message whose tool_call_id starts with `srvtoolu_` (a server-executed tool produces no client result; a user tool_result for it is invalid). The existing reconstruction path (provider_specific_fields present, e.g. the litellm SDK) is unchanged, as is regular client tool_use/tool_result. Tests (tests/llm_translation/test_prompt_factory.py): - update test_convert_to_anthropic_tool_invoke_server_tool -> test_convert_to_anthropic_tool_invoke_server_tool_without_result_is_dropped - add test_anthropic_messages_pt_generic_client_drops_orphan_server_tool Follow-up to #17746 / #17798; addresses the generic-client (no provider_specific_fields) case of #17737. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(anthropic): cover the srvtoolu_ round-trip fix in the test_litellm unit suite The regression tests added in tests/llm_translation/test_prompt_factory.py aren't run by the coverage CI job (it runs tests/test_litellm), so the new factory.py branches showed as uncovered (codecov patch coverage). Add equivalent focused tests in the unit suite so both new branches are exercised there: - convert_to_anthropic_tool_invoke drops a srvtoolu_ server_tool_use when no matching *_tool_result is available. - anthropic_messages_pt drops the orphaned srvtoolu_ tool message a generic OpenAI client replays. Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(anthropic): cover the server_tool_use + result valid-pair path in unit suite Covers the remaining patch-coverage lines codecov flagged: convert_to_anthropic_tool_invoke emitting server_tool_use followed by its web_search_tool_result when the matching result is present (the litellm-SDK round-trip path). Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(anthropic): flatten srvtoolu_ tool-message guard to a negated if Addresses the Greptile style nit: replace the if-pass/else with a single negated `if not (...)` guard around the tool_result append. Behavior unchanged. Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(proxy): require premium only when enabling premium metadata fields (#30285) (#30506) Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(perplexity): stop double-billing reasoning tokens in manual cost fallback (#30488) * fix(perplexity): stop double-billing reasoning tokens in manual cost fallback When perplexity_cost_per_token cannot use the API-provided usage.cost.total_cost short-circuit and falls back to manual calculation, it multiplies the full usage.completion_tokens by output_cost_per_token and then adds reasoning_tokens * output_cost_per_reasoning_token on top. Per the OpenAI/Perplexity usage convention codified for the central path in PR #18607, completion_tokens already INCLUDES reasoning_tokens, so the manual fallback double-bills reasoning at both the output and reasoning rate. Concrete impact on perplexity/sonar-deep-research (input 2e-6, output 8e-6, reasoning 3e-6): for the exact usage shape exercised by the live response fixture in tests/llm_translation/test_perplexity_reasoning.py (prompt_tokens=9, completion_tokens=20, reasoning_tokens=15) the current code charges 0.000223 vs the convention-correct 0.000103, a 2.165x overcharge. The bug is reachable whenever Perplexity omits the cost object (streaming chunks, fixture-driven paths, older API versions). Subtracts reasoning_tokens (clamped at zero) from completion_tokens before applying the output rate, mirroring how dashscope/cost_calculator.py and the central generic_cost_per_token already handle it. Preserves the existing fallback behaviour when output_cost_per_reasoning_token is unset (all completion_tokens stay at the output rate). Existing tests in tests/test_litellm/llms/perplexity/test_perplexity_cost_calculator.py asserted the buggy math and are updated to the convention-correct math. Adds a focused regression test using the exact usage shape from the live response fixture so this class of bug cannot be silently reintroduced. * style(perplexity): drop redundant type annotation on else branch to satisfy mypy mypy [no-redef] flagged 'completion_cost' as declared in both if and else arms; keeping the annotation only on the first declaration matches existing patterns in this file. * fix(perplexity): update integration test expected costs for non-double-billed math Three tests in test_perplexity_integration.py asserted the old buggy expectation that reasoning_tokens are billed in addition to the full completion_tokens count. After the fix in cost_per_token, reasoning_tokens are billed at the reasoning rate and the remaining (completion_tokens - reasoning_tokens) at the standard output rate, matching OpenAI/Perplexity convention (PR #18607). Updates: test_end_to_end_cost_calculation_with_transformation, test_main_cost_calculator_integration, test_high_volume_cost_calculation. The high-volume sanity threshold drops to 0.25 to reflect the corrected total. * fix(ui): use dynamic proxy base URL in MCP usage examples (#30487) Replace hardcoded http://localhost:4000 with getProxyBaseUrl() in the MCP server usage example and copy-to-clipboard snippet so the generated configuration works for non-local deployments. Fixes #30466 * feat: add missing UK PII entity types to Presidio guardrail (#30537) * feat: add missing UK PII entity types to Presidio guardrail Add UK_PASSPORT, UK_POSTCODE, and UK_VEHICLE_REGISTRATION to PiiEntityType enum and PII_ENTITY_CATEGORIES_MAP. These entity types are supported by Microsoft Presidio but were missing from litellm's type definitions, preventing users from configuring UK-specific PII detection. * test: remove fragile hardcoded entity count test Remove test_uk_category_entity_count which hardcodes len() == 5. The test_uk_entities_match_presidio_recognizers test already verifies exact set equality, making the count test redundant and fragile to future Presidio additions. * style: apply Black formatting to match CI requirements * fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (#30357) Volcengine (Doubao) models define `tiered_pricing` but no flat per-token cost, so cost_per_token fell through to generic_cost_per_token (which only reads flat costs) and tracked them at $0 Route custom_llm_provider == "volcengine" to the shared tiered-pricing handler in litellm/llms/dashscope/cost_calculator.py, which already computes graduated tier costs. Make that handler provider-agnostic by adding a custom_llm_provider argument (default "dashscope" preserves existing behavior) so get_model_info resolves the correct model map entry Fixes #30346 * feat(mcp): make MCP gateway name and description configurable via env vars (#30473) * feat(mcp): make MCP gateway name and description configurable via env vars * Rename function _restore_env to _apply_env * docs(mcp): document import-time capture of env-backed identity constants Address Greptile review feedback: clarify that LITELLM_MCP_SERVER_NAME and LITELLM_MCP_SERVER_DESCRIPTION are read once at import and require a module reload to observe env changes after import. Generated with AI assistance Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Yevhen Luhovtsov <yevhen.luhovtsov@intapp.com> Co-authored-by: Claude <noreply@anthropic.com> * fix(mcp): preserve native tools in semantic filter hook (#26650) * fix(mcp): preserve native tools in semantic filter hook The SemanticToolFilterHook.async_pre_call_hook passed ALL tools (MCP + native) to filter_tools(), which only knows MCP-registered tool names. Native tools silently failed the name match in _get_tools_by_names() and were dropped from the request. Fix: partition tools into native and MCP-registered before filtering. Run the semantic filter only on MCP tools, then merge native tools back unconditionally. Changes: - Robust _is_mcp_tool() using shape-based detection for OpenAI-format dicts, safe regardless of future _extract_tool_info changes - Single-pass partition loop (no double _is_mcp_tool calls) - Preserve native tools in MCP expansion path (mixed requests) - Track MCP expansion to prevent expanded tools bypassing filtering - filter_stats reports MCP-only counts for accurate metrics - Extracted _emit_filter_metadata() helper - Skip spurious filter headers for all-native tool requests Closes #26212 * remove stale docstring note referencing tools_expanded_from_mcp * fix: handle Responses API name collision and preserve tool ordering - Classify Responses API tools ({type: 'function', name: '...'}) as native to prevent name collisions with MCP canonical names - Preserve original request tool ordering using id()-based merge instead of naive native+mcp concatenation - Add 2 regression tests: name collision and ordering preservation * style: apply black formatting * fix(mcp): harden semantic filter — preserve all native tool formats, safe metadata access, graceful expansion failure, name-based merge * lint: suppress PLR0915 on async_pre_call_hook (matches codebase convention) * ci: retrigger checks after rebase onto litellm_internal_staging * feat(fireworks): sync Fireworks AI model registry with current platform catalog (#30616) Adds 12 new Fireworks serverless models and updates 3 existing entries in model_prices_and_context_window.json and its bundled backup to match the current Fireworks platform model list. New direct models: glm-5p2, qwen3p7-plus, minimax-m3, minimax-m2p7, kimi-k2p7-code, kimi-k2p6, deepseek-v4-pro, deepseek-v4-flash. New router endpoints: glm-5p1-fast, kimi-k2p6-fast, kimi-k2p7-code-fast. Updated: glm-5p1, gpt-oss-120b, and gpt-oss-20b now carry correct output token caps, cache-read pricing, and explicit capability flags max_tokens is set equal to max_output_tokens (not the full context window) for models whose generation cap is below their context window. This avoids the shared input+output budget path in get_modified_max_tokens, which would otherwise let callers request output sizes the model cannot produce. The same fix corrects the pre-existing glm-5p1, gpt-oss-120b, and gpt-oss-20b entries that had max_tokens equal to the full context window Short-form aliases (fireworks_ai/<model>) are added for every direct accounts/fireworks/models/ entry so cost attribution works for callers using bare model names. Router endpoints get short-form aliases too, and transform_request now routes bare names ending in -fast to the accounts/fireworks/routers/ path instead of defaulting every bare name to models/. This keeps the kimi-k2p6-fast router from being misrouted to the nonexistent models/kimi-k2p6-fast endpoint kimi-k2p6-turbo is intentionally excluded; kimi-k2p6-fast is its replacement. Context windows for deepseek-v4 and kimi models use the power-of-two values (1048576 and 262144) published on the Fireworks model pages, matching the convention already used by existing entries Two regression tests in test_utils.py assert the exact per-token costs, token limits, capability flags, and short-form-to-long-form equality for all 15 models against both the main and backup cost maps. Two routing tests in test_fireworks_ai_chat_transformation.py verify bare -fast names route to routers/ and bare direct-model names route to models/ * fix(bedrock): handle role:"system" inside the messages array on /v1/messages (#29698) (#30443) * feat(anthropic): hoist leading in-array system to top-level (helper) * test(anthropic): cover _system_content_to_blocks edge cases; deepcopy cache_control * test(anthropic): mid-conversation system normalization cases * feat: add supports_mid_conversation_system flag to Claude Opus 4.8 Add supports_mid_conversation_system: true to all 9 claude-opus-4-8 cost-map entries (Anthropic-native, Bedrock, Vertex, Azure AI) in both the root cost map and the bundled package backup, since the runtime helper and tests read the backup in local/offline mode. Pin the mid-system passthrough regression test to the local cost map via the existing local_model_cost_map fixture so it reads the branch-local flag rather than the network-fetched main copy. * fix(bedrock): normalize in-array system in /v1/messages handler (#29698) Wire normalize_system_messages_for_anthropic into anthropic_messages_handler so all Bedrock /v1/messages paths (Invoke / Mantle / ClaudePlatform / Converse-bridge) hoist leading in-array system entries (and demote mid-conversation ones on models lacking supports_mid_conversation_system) into the top-level system field. The normalized messages/system are written back into the local_vars snapshot the base_llm branch reads from, otherwise the Invoke/Mantle fix would silently no-op. Also fix the helper to resolve supports_mid_conversation_system through the prefix-aware AnthropicModelInfo._supports_model_capability resolver. The raw _supports_factory could not see the flag once get_llm_provider left the invoke/ prefix on the model id, which would have wrongly demoted mid-conversation system on a Bedrock invoke opus-4-8 path. * fix(bedrock): resolve mid-conversation-system flag through mantle/invoke/converse route prefixes; drop unused param * fix(types): widen system param to Union[str, List] for hoisted system blocks * refactor(bedrock): drop dead local_vars messages writeback * fix(bedrock/converse): translate in-array system in anthropic->openai adapter (#29698) * fix(bedrock/converse): preserve cache_control on in-array system; test drop-empty * fix(bedrock/converse): rename colliding local to satisfy mypy; test handler system-merge branches * fix(types): register supports_mid_conversation_system in model-info schema The cost-map JSON-schema validation test (test_aaamodel_prices_and_context_window_json_is_valid) rejects unknown properties, so adding supports_mid_conversation_system to the opus-4-8 cost-map entries failed CI with 'Additional properties are not allowed'. Register the flag in the INTENDED_SCHEMA allow-list and in the ProviderSpecificModelInfo TypedDict so it is a typed, first-class capability flag alongside its peers (supports_output_config, etc.). --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload (#28885) * fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload By default the agentcore provider flattens the last message to a text-only {"prompt": "..."} payload via convert_content_list_to_str, silently dropping OpenAI multimodal blocks (image_url, file, input_audio, ...). This adds an opt-in `forward_multimodal_content` litellm param. When truthy and the last message's content is a list containing a non-text block, the original OpenAI content list is forwarded verbatim under a new "content" field so an attachment-aware AgentCore agent can read it. Default off keeps the payload byte-identical to the legacy {"prompt": "..."} shape — existing agents are unaffected. The flag is read from optional_params (where other AgentCore params land) with a litellm_params fallback, and accepts a bool or a config/env string ('true', '1', ...). AgentCore Runtime is schemaless on the agent side — the agent's @app.entrypoint parses arbitrary JSON up to 100 MB (per https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-invoke-agent.html), so this is a purely upstream change; no AgentCore-side schema is asserted. * fix(bedrock/agentcore): shallow-copy forwarded multimodal content list Address review feedback (Sameerlite): payload["content"] = last_content aliased the caller's mutable messages[-1]["content"] list. Harmless today because the payload is JSON-serialized immediately, but a latent footgun if a future caller mutates the returned payload before serialization. Forward list(last_content) so the payload owns its own list. Block dicts stay shared on purpose — a deep copy would clone potentially large base64 media on the request hot path, and the flagged risk was the shared list, not the blocks. Update the passthrough tests to assert equality + distinct identity, and add a regression test that mutating the payload list can't leak back into the original message content. * Revert "fix(mcp): preserve native tools in semantic filter hook (#26650)" This reverts commit |