A daily-spend batch upsert that outlives prisma-client-py's 30s HTTP read
timeout keeps running server side after the client gives up, holding its
row locks for as long as the database takes. Every later flush cycle
queues behind those locks, which is how one slow batch cascaded into
exhausted database sessions.
The query engine's own transaction timeout cannot end that wait: it
cannot interrupt a statement that is already executing. Measured against
real Postgres, a batch wrapped in db.tx(timeout=60s) still held its locks
for the full 90s the statement ran. Only a Postgres-side statement_timeout
bounded it.
database_statement_timeout and database_lock_timeout (seconds) are now
first-class general_settings keys, emitted as libpq
options=-c statement_timeout=<ms> on DATABASE_URL. They are opt-in, so an
unset config keeps today's behavior, and they are never applied to
DIRECT_URL, which serves migrations that legitimately run long.
Resolves LIT-4718
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
* fix(embeddings): accept encoding_format='float' for vertex_ai/gemini embeddings (#33617)
OpenAI SDKs (and litellm's own client since ~1.84) send
encoding_format='float' by default, but the vertex embedding config only
supports ['dimensions'], so get_optional_params_embeddings raised
UnsupportedParamsError at the provider default value. Any
OpenAI-compatible client talking to a litellm proxy with vertex
embedding models got a 400 unless the operator set proxy-wide
drop_params: true.
Float lists are exactly what the vertex API returns, so the param is a
no-op: pop it before validation. Other values (e.g. 'base64') keep the
existing unsupported-param behavior (dropped with drop_params, raise
otherwise).
Fixes#33173
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add Singulr guardrail integration for LiteLLM gateway (#31302)
* singulr guardrail support for litellm gateway
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix comments
* improvement
* fix: resolve review comments and implement requested improvements
* fix:Guardrail bypass through uninspected messages
* fix:tool text scanning
* fix: Legacy function definitions bypass scanning by adding indirect message scaning
* chore: remove unintended basedpyright budget file
* fix:Response schema bypasses guardrail scanning (response_format.json_schema)
* chore: restore basedpyright-code-budget.json and update lint baselines
Restores the file deleted in c698b88686 to match upstream litellm_internal_staging.
Regenerates basedpyright and ruff-strict budget baselines via make lint-budget-update.
* fix: scan system messages as indirect prompt injection in Singulr guardrail
* chore: restore lint budget files to upstream baseline
* fix: resolve ruff UP006 and I001 violations in singulr guardrail
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* resolve review comments on Singulr guardrail
* fix: scan tool call results as indirect prompt injection in Singulr guardrail
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* minor
* formating fix
* refactor: shift extraction logic to singulr side
* refactor:keep precall hook only
* fix:formatting
* fix:linting
* improve config description
* Trigger CI
* fix
* fix:field description
* fix:errors due to change in field names
* style: apply ruff line-wrap formatting to singulr guardrail
* fix:exception
* fix:formatting
* fix playground
* improved
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix
* fix ci issues
* remove uv.lock from pr
* fix
* fix:resolved comments
* chore: trigger CI
* remove uv.lock
* fix
* fix linting
* fix linting
* fix linting
* remove doc strings
* remove test fixes
* chore: retrigger CI
* change in singulr api contract
* remove some ut
* send litellm call_id to singulr
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* Fix non-conformant UUIDv7 generation in native Opik integration (#31294)
create_uuid7() encoded the timestamp in units of 16 seconds instead of
milliseconds, so the top 48 bits came out ~4096x the real unix-ms. Opik's
backend validates the embedded UUIDv7 timestamp on ingestion (OPIK-7067);
the bad encoding decoded to ~year 2201 and every trace/span batch was
rejected with HTTP 400.
Rewrite create_uuid7() to be RFC 9562 conformant (top 48 bits = unix-ms),
using the standard library only so no new dependency is added. Add unit
tests covering UUIDv7 validity and millisecond timestamp encoding.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(proxy): expose uvicorn concurrency limit (#33077)
Expose uvicorn's limit_concurrency as a --limit_concurrency CLI flag and
LIMIT_CONCURRENCY environment variable. Uvicorn counts both active tasks and
accepted connections and returns HTTP 503 once the configured limit is reached.
Reject non-positive limits at CLI parse time and only add the setting to the
uvicorn startup arguments. Because idle connections also consume capacity,
deployments should use upstream connection/header timeouts and per-client
connection limits.
* test: reorder test_utils tail to keep the daily merge conflict-free (#33788)
The daily OSS branch and litellm_internal_staging each appended an
independent test block at the very end of tests/test_litellm/test_utils.py,
so merging the two collides on that shared end-of-file position even though
the additions are unrelated (this branch adds the vertex embedding
encoding-format tests; staging adds the per-model prompt-cache-minimum
tests). Moving this branch's new TestVertexEmbeddingEncodingFormat class
above test_gemini_image_models_do_not_support_reasoning, which both branches
share, gives the two additions different anchors, so git applies both
without a conflict and without pulling staging into this branch. Pure
reorder; no test bodies change
---------
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: madan-singulr <150280287+madan-singulr@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
Co-authored-by: Aliaksandr Kuzmik <98702584+alexkuzmik@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Salva Madrid <50212436+salvamadrid@users.noreply.github.com>
* fix(proxy_cli): reap orphaned prisma query-engine processes when a worker dies
When the proxy runs multi-worker (uvicorn multiprocess supervisor or the
gunicorn arbiter), a worker that crashes or is force-killed never runs its
in-process atexit cleanup, so its prisma query-engine subprocess reparents
to PID 1 and keeps its database connection pool established forever while
the replacement worker opens a fresh pool. Active DB connections then grow
past database_connection_pool_limit with every worker death.
Run a reaper thread in the supervisor process that marks itself a child
subreaper on Linux, scans for adopted query-engine children whose worker
is gone, and terminates them with SIGTERM escalating to SIGKILL after a
bounded grace period. Engines owned by live workers are children of those
workers, never of the supervisor, so they are structurally out of reach.
Resolves LIT-4449
Fixes https://github.com/BerriAI/litellm/issues/33023
* fix(proxy_cli): address review findings on the query-engine reaper
Make start_query_engine_reaper idempotent, reap simultaneous orphans
under one shared grace period instead of serially, and log when a PID
survives SIGKILL. Also regenerate schema.d.ts for the update_team
docstring line that documents the existing mcp_rpm_limit param (fixes
the walk-order-dependent documentation CI failure) and avoid a cast in
the prctl wrapper
* test(proxy): fix reaper idempotency-test isolation and widen coverage
The daemon-thread startup test now stubs threading.enumerate so a
reaper thread left running by an earlier test in the same xdist worker
cannot satisfy the idempotency guard and skip the code under test. Add
coverage for stat-file truncation, non-numeric ppid, non-child reap,
signal-to-dead-pid, subreaper capability, and reaper-loop resilience
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens
* test: scope local cost map env var with monkeypatch to avoid test pollution
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold
_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.
mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.
* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers
Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.
Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.
* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview
MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.
* fix(mcp_debug): mask short auth values in debug headers instead of echoing them
Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.
* test(mcp_debug): assert masked short value preserves length
* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)
Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.
Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:
- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
ProviderConfigManager.get_provider_audio_transcription_config() in
litellm/utils.py; update the stale comment in
get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
get_supported_openai_params() in
litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
model_prices_and_context_window.json and
litellm/model_prices_and_context_window_backup.json (both had
mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
imports from tests/llm_translation/test_fireworks_ai_translation.py
No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider (#30876)
* feat: add darkbloom provider
* fix: document darkbloom provider endpoints
* fix: address darkbloom review feedback
* fix: update darkbloom tool metadata
* fix: fail fast for non-Postgres database URLs (#30883)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging
* Validate DIRECT_URL alongside DATABASE_URL startup guards
* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)
* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)
* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)
* style(bedrock): black-format stream-error helper (#24608)
* fix(mcp): re-land native tool preservation with typed annotations (#30645)
* fix(mcp): preserve native tools in semantic filter hook with typed annotations
* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
* fix(sambanova): return embeddings supported params instead of dropping them (#30937)
* fix(router): send fallback metadata when streaming (#30914)
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:
1. The response now correctly populates the fallback headers
(`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
to the client (opt-in) by passing `include_fallback_errors: true` in
the request.
The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
* fix(mistral): drop output-only reasoning fields from input messages (#30884)
LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.
Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes#30835
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)
* fix(perplexity): bill search queries at the per-request price, not 1/1000
The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").
The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.
Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.
* test(perplexity): update integration test search-cost expectations to per-request
The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.
* test(perplexity): drop unused mock imports flagged by ruff
* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)
* fix(fireworks_ai): return None for transcription in get_supported_openai_params
Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting
Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.
Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test
The operator gate added in e7ff3e1 means include_fallback_errors is only
honoured when general_settings.expose_fallback_errors_to_caller is True.
Set that flag via monkeypatch in the test that exercises the emit path.
* test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos
test_convert_url called convert_url_to_base64 against a live picsum.photos
URL and asserted nothing, so it added no real signal and broke CI whenever
the host was unreachable (it was returning 522 and blocking this branch).
Replace the live call with a mocked HTTP client and assert the produced
base64 data URL, so the conversion path is exercised deterministically with
no network dependency. This suite runs under VCR, which is why a transport
level mock (respx) does not reliably intercept; mocking the client object
itself is robust regardless.
* fix(interactions): drop role from Interaction response to match Google spec
Google removed the output-only role field from the Interaction schema (it
now lives only on Turn), so the live OpenAPI compliance canary started
failing with 'role' not in spec. Reconcile our generated types by removing
role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams
and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse,
stop stamping role=model in the responses-to-interactions transformation, and
update the compliance and integration tests accordingly. Turn.role is kept
since the spec still defines it.
* fix: align all-team-models sentinel access
* fix(router): forward include_fallback_errors through multi-hop fallbacks
run_async_fallback received include_fallback_errors as an explicit named
parameter, so it was bound out of **kwargs and never reached the nested
async_function_with_fallbacks call. Multi-hop fallback chains (a fallback
group that itself fails over) therefore stopped collecting fallback errors
beyond the first hop when a caller opted in. Re-inject the flag into kwargs
before the nested call so inner hops keep accumulating errors, which
add_fallback_headers_to_response already merges across levels.
---------
Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu>
Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com>
Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com>
Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com>
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com>
Co-authored-by: dav nguyxn <hoangson091104@gmail.com>
Co-authored-by: Tal Marian <tal.marian@island.io>
Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com>
Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Setting --max_requests_before_restart alone recycles every worker at almost the
same time once they have served a similar number of requests, which under
sustained load can drop a whole pod's capacity at once roughly every 7-10 days.
This exposes a jitter knob that adds a random amount in [0, jitter] to the
restart threshold per worker so restarts are staggered. It maps to uvicorn's
limit_max_requests_jitter and gunicorn's max_requests_jitter. uvicorn only
gained limit_max_requests_jitter in 0.41.0 while litellm still allows
uvicorn>=0.33.0, so the uvicorn path feature-detects the parameter via the
Config signature and warns instead of crashing on older versions. The flag has
no effect without --max_requests_before_restart, so the kwarg is not forwarded
in that case and a warning is printed on both the uvicorn and gunicorn paths.
Resolves LIT-3774
* feat(proxy): hot-reload .env in dev when running with --reload
The --reload watcher already restarts the worker on *.py and --config YAML
edits, but .env was unwatched, so changing a key there did nothing until a
manual restart. Add .env to the uvicorn reload_includes (and to the
StatReload monkeypatch, which ignores reload_includes) so an edit triggers a
worker restart.
A reloaded worker is a fresh process that inherits the reloader's
environment, so load_dotenv(override=False) would keep serving the stale
inherited value for any key already in the environment. The CLI now exports
LITELLM_DEV_ENV_HOT_RELOAD when --reload is set, and litellm/__init__.py
reads it to load .env with override=True only on that dev path, leaving
normal startup precedence untouched.
* feat(proxy): warn that --reload makes .env override shell env vars
When --reload is active, worker processes re-read .env with override=True, so
.env values win over shell-exported environment variables. Surface this dotenv
precedence change with a startup warning so a developer who relies on a
shell-exported override is not silently surprised.
* fix(proxy): type reload helper paths as Optional[str] to satisfy mypy
* fix(proxy): watch the cwd .env in both reload backends for parity
WatchFiles only watches cwd (and the --config dir) for .env, while the
StatReload fallback used find_dotenv(usecwd=True), which walks up to a
parent-dir .env that WatchFiles never sees. Point StatReload at the same
cwd .env so the two reload backends react to the same file.
* Add granian as a ASGI compliant web server. Provides better stability, 10-20 RPS improvement under standard LT conditions.
TODO: Verify poetry lock details and add locust numbers to PR
* Update granian version in license_cache.json and pyproject.toml to 2.5.7
* Enhance proxy CLI tests by adding SSL initialization checks for Granian server. Remove Python version skip conditions and implement tests to ensure SSL certificate and key are required for server initialization.
* update uv lock to fix granian import error
* fix(proxy): expose Prisma idle/connect timeout + extra DB URL params
Operators have reported large numbers of idle Prisma connections that
never get closed. The proxy already forwards `connection_limit` and
`pool_timeout` to the DATABASE_URL, but had no knob for capping idle
or slow connections. Add three new `general_settings` keys that thread
through to the DATABASE_URL / DIRECT_URL query string:
- `database_connect_timeout` -> Prisma `connect_timeout`
- `database_socket_timeout` -> Prisma `socket_timeout` (the main
knob for closing idle connections from the LiteLLM side)
- `database_extra_connection_params` -> untyped passthrough dict for
any other Prisma URL param (`pgbouncer`, `statement_cache_size`,
`sslmode`, ...); keys here override LiteLLM defaults.
Refactors the duplicated DATABASE_URL/DIRECT_URL param dicts into a
single `_build_db_connection_url_params` helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update litellm/proxy/proxy_cli.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
test_keepalive_timeout_flag and test_timeout_worker_healthcheck_flag
were the only run_server tests in test_proxy_cli.py that neither
stripped DATABASE_URL/DIRECT_URL nor mocked the prisma DB path. When a
DATABASE_URL is present (CI/env leak), run_server --local enters the DB
block and blocks in the un-timeout'd subprocess.run(["prisma"]) at
proxy_cli.py:987 plus the ProxyExtrasDBManager migrate-deploy retry
loops, ~370s per test on the CI runner. --dist=loadscope pins both to
one xdist worker, so the proxy-infra job appears stuck at 99% and hits
the 20-min timeout.
Apply the same isolation every other run_server test in this file
already uses: mock PrismaManager.setup_database +
should_update_prisma_schema and strip DATABASE_URL/DIRECT_URL. Full
module drops from 31.7s to 2.9s locally; both tests fall off the slow
list.
* proxy: hot-reload config YAML when --reload is set
Uvicorn's --reload only watches *.py by default, so editing the
--config YAML did not restart the proxy. _get_reload_options() now
extends reload_dirs/reload_includes with the config file's directory
and basename when --config is provided.
* proxy: qualify reload_includes with absolute config path
Address Greptile review on PR #27274. When the --config file lives
outside cwd, reload_includes previously stored only the basename, which
meant uvicorn/watchfiles would also reload on edits to any same-named
file inside cwd. Use the absolute config path as the include pattern in
that case so only the actual proxy config triggers a restart.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy): use basename for reload_includes config pattern
Uvicorn's resolve_reload_patterns() calls pathlib.Path.glob(), which
raises NotImplementedError on absolute patterns (uvicorn discussion
2156). Passing config_abs (an absolute path) when the config file lived
outside cwd crashed startup under --reload. The config_dir is already
added to reload_dirs, so using just the basename as the include pattern
is sufficient to match the specific config file.
* fix: make it reload app when yaml changes
* style: remove unneeded comments
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds a CLI flag (`--timeout_worker_healthcheck`, env `TIMEOUT_WORKER_HEALTHCHECK`)
that forwards to uvicorn's `timeout_worker_healthcheck` Config kwarg (added in
uvicorn 0.37.0). Lets operators raise the supervisor's worker-ping timeout above
the default 5s when triaging workers being killed and respawned under load.
The helper introspects `uvicorn.Config.__init__` and only sets the kwarg if
supported, otherwise prints a warning - so the existing uvicorn>=0.32.1,<1.0.0
floor pin is unaffected. Gunicorn and Hypercorn paths are unchanged (the uvicorn
supervisor isn't running there); the value is also not passed to the helper at
all on those paths so the "uvicorn too old" warning never fires spuriously.
Existing tests pinned exact kwargs on `PrismaManager.setup_database`,
but the opt-in v2 resolver added `use_v2_resolver=False` to every call.
Update the three assertions to reflect the new signature.
Fixes:
- TestHealthAppFactory::test_use_prisma_db_push_flag_behavior
- TestHealthAppFactory::test_startup_fails_when_db_setup_fails
* build: migrate packaging metadata to uv
* ci: move automation and local tooling to uv
* docker: migrate image builds and runtime setup to uv
* docs: update install and deployment guidance for uv
* chore: align auxiliary scripts and tests with uv
* test: harden test_litellm isolation
* fix: keep release and health check images self-contained
* build: pin uv tooling and health check deps
* test: isolate bedrock image request formatting from suite state
* test: cover sandbox executor requirements flow
* ci: fix circleci no-op command steps
* ci: fix circleci publish workflow parsing
* fix: stabilize remaining uv migration CI checks
* ci: increase matrix test timeout headroom
* fix: restore published docker and license coverage
* fix: restore proxy runtime build parity
* fix: restore proxy extras parity and venv migrations
* ci: persist uv path across circleci steps
* fix: keep psycopg binary in default test env
* docker: preserve prisma cache across stages
* test: run local proxy checks through uv python
* build: restore runtime deps moved into ci
* build: refresh uv lock after upstream merge
* fix: restore module import in test_check_migration after merge
The conflict resolution imported only the function but the test body
references check_migration as a module throughout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: revert dependency promotions, remove nodejs-wheel-binaries, fix Docker layer caching
- Move google-generativeai, Pillow, tenacity back to ci group (they are
lazily imported and bloat the base SDK install needlessly)
- Remove nodejs-wheel-binaries from extra_proxy and proxy-dev (redundant
in Docker where system Node.js is already installed via apk)
- Remove all nodejs-wheel node replacement and venv npm patching blocks
from Dockerfiles since the wheel is no longer installed
- Add --no-default-groups to CodSpeed benchmark workflow so the benchmark
environment matches the old minimal pip install footprint
- Apply standard uv two-phase Docker pattern: copy metadata first, install
deps (cached layer), then copy source and install project
- Replace CircleCI enterprise no-op with proper uv sync command
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: regenerate uv.lock after removing nodejs-wheel-binaries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): use cache/restore instead of cache to prevent cache poisoning
The old workflow used actions/cache/restore (read-only). The uv migration
changed it to actions/cache (read-write), which zizmor flags as a cache
poisoning risk. Restore the safer read-only variant.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv built-in cache to silence cache-poisoning alert
The setup-uv action enables caching by default, which zizmor flags as a
cache poisoning risk. Disable it since we already use a read-only
cache/restore step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): disable setup-uv cache in publish workflow
Silences zizmor cache-poisoning alert. Publishing workflow runs
infrequently on protected branches so caching adds no real benefit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(test): remove duplicate verbose_logger mock in test_check_migration
The logger was patched twice — first via mocker.patch() then via
mocker.patch.object(autospec=True). The second call fails because
autospec cannot inspect an already-mocked attribute. Remove the
redundant first patch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ci): free disk space before Docker build in test-server-root-path
The Dockerfile.non_root build ran out of disk on the CI runner. Remove
Android SDK, .NET, Boost, and GHC toolchains (~12GB) to free space.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Aligns proxy default with litellm.AZURE_DEFAULT_API_VERSION (2025-02-01-preview)
so Azure response_format + json_schema works without tools fallback.
Made-with: Cursor
* fix(test): add missing mocks for test_streamable_http_mcp_handler_mock
The test was missing mocks for extract_mcp_auth_context and set_auth_context,
causing the handler to fail silently in the except block instead of reaching
session_manager.handle_request. This mirrors the fix already applied to the
sibling test_sse_mcp_handler_mock.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): route OpenAI models through chat completions in pass-through tests
The test_anthropic_messages_openai_model_streaming_cost_injection test fails
because the OpenAI Responses API returns 400 for requests routed through the
Anthropic Messages endpoint. Setting LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES=true
routes OpenAI models through the stable chat completions path instead.
Cost injection still works since it happens at the proxy level.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): fix assemblyai custom auth and router wildcard test flakiness
1. custom_auth_basic.py: Add user_role='proxy_admin' so the custom auth
user can access management endpoints like /key/generate. The test
test_assemblyai_transcribe_with_non_admin_key was hidden behind an
earlier -x failure and was never reached before.
2. test_router_utils.py: Add flaky(retries=3) and increase sleep from 1s
to 2s for test_router_get_model_group_usage_wildcard_routes. The async
callback needs time to write usage to cache, and 1s is insufficient on
slower CI hardware.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* ci: retrigger CI pipeline
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mypy): use LitellmUserRoles enum instead of raw string in custom_auth_basic
Fixes mypy error: Argument 'user_role' has incompatible type 'str'; expected 'LitellmUserRoles | None'
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: don't close HTTP/SDK clients on LLMClientCache eviction (#22926)
* fix: don't close HTTP/SDK clients on LLMClientCache eviction
Removing the _remove_key override that eagerly called aclose()/close()
on evicted clients. Evicted clients may still be held by in-flight
streaming requests; closing them causes:
RuntimeError: Cannot send a request, as the client has been closed.
This is a regression from commit fb72979432. Clients that are no longer
referenced will be garbage-collected naturally. Explicit shutdown cleanup
happens via close_litellm_async_clients().
Fixes production crashes after the 1-hour cache TTL expires.
* test: update LLMClientCache unit tests for no-close-on-eviction behavior
Flip the assertions: evicted clients must NOT be closed. Replace
test_remove_key_closes_async_client → test_remove_key_does_not_close_async_client
and equivalents for sync/eviction paths.
Add test_remove_key_removes_plain_values for non-client cache entries.
Remove test_background_tasks_cleaned_up_after_completion (no more _background_tasks).
Remove test_remove_key_no_event_loop variant that depended on old behavior.
* test: add e2e tests for OpenAI SDK client surviving cache eviction
Add two new e2e tests using real AsyncOpenAI clients:
- test_evicted_openai_sdk_client_stays_usable: verifies size-based eviction
doesn't close the client
- test_ttl_expired_openai_sdk_client_stays_usable: verifies TTL expiry
eviction doesn't close the client
Both tests sleep after eviction so any create_task()-based close would
have time to run, making the regression detectable.
Also expand the module docstring to explain why the sleep is required.
* docs(AGENTS.md): add rule — never close HTTP/SDK clients on cache eviction
* docs(CLAUDE.md): add HTTP client cache safety guideline
* [Fix] Install bsdmainutils for column command in security scans
The security_scans.sh script uses `column` to format vulnerability
output, but the package wasn't installed in the CI environment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle string callback values in prometheus multiproc setup
When callbacks are configured as a plain string (e.g., `callbacks: "my_callback"`)
instead of a list, the proxy crashes on startup with:
TypeError: can only concatenate str (not "list") to str
Normalize each callback setting to a list before concatenating.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* bump: version 1.82.2 → 1.82.3
* fix(test): update test_startup_fails_when_db_setup_fails for opt-in enforcement
The --enforce_prisma_migration_check flag is now required to trigger
sys.exit(1) on DB migration failure, after #23675 flipped the default
behavior to warn-and-continue.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(cost_calculator): use model name for per-request custom pricing when router_model_id has no pricing
When custom pricing is passed as per-request kwargs (input_cost_per_token/output_cost_per_token),
completion() registers pricing under the model name, but _select_model_name_for_cost_calc was
selecting the router deployment hash (which has no pricing data), causing response_cost to be 0.0.
Now checks whether the router_model_id entry actually has pricing before preferring it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add LITELLM_WORKER_STARTUP_HOOKS for per-worker initialization (gflags support)
Add support for running user-defined startup hooks in each worker process
during proxy_startup_event. This enables re-initialization of in-process
state (like gflags.FLAGS) that doesn't survive uvicorn worker spawning.
Usage:
export LITELLM_WORKER_STARTUP_HOOKS=mymodule:init_fn,other:setup_fn
Hooks run early in proxy_startup_event (before config/DB loading).
Supports both sync and async callables. Errors propagate to prevent
broken workers from serving traffic. No-op when env var is unset.
Includes 5 tests covering sync/async hooks, multiple hooks, error
propagation, and no-hooks-set scenarios.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: add Worker Startup Hooks page with gflags usage example
- New docs page: docs/proxy/worker_startup_hooks.md
- Explains the problem (per-process state lost in multi-worker deployments)
- Full gflags example with wrapper module and startup script
- Covers multiple hooks, async hooks, error behavior
- Architecture diagram showing master→worker flow
- Added LITELLM_WORKER_STARTUP_HOOKS to config_settings.md env var table
- Added to sidebar under Setup & Deployment
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Update litellm/proxy/proxy_server.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(lint): suppress PLR0915 for 3 complex methods that exceed 50-statement limit
- streaming_iterator.py: _process_event (84 statements)
- transformation.py: translate_messages_to_responses_input (51 statements)
- transformation.py: transform_realtime_response (54 statements)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mypy): resolve type errors in public_endpoints, user_api_key_auth, common_utils, transformation
- public_endpoints.py: fix _cached_endpoints type annotation
- user_api_key_auth.py: accept Optional[str] for end_user_id parameter
- common_utils.py: add NewProjectRequest/UpdateProjectRequest to Union type
- transformation.py: add ChatCompletionRedactedThinkingBlock and list[Any] to content type
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(proxy-extras): bump version to 0.4.50 and sync schema
- Bump litellm-proxy-extras from 0.4.49 to 0.4.50
- Sync schema.prisma with main proxy schema
- Includes new LiteLLM_ClaudeCodePluginTable model
- Includes new @@index([startTime, request_id]) on SpendLogs
- Update version references in requirements.txt and pyproject.toml
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(router): use string id in test_add_deployment and add defensive str() in register_model
- Change test to use string '100' instead of int 100 for model_info.id
- Add str() conversion in register_model to prevent AttributeError on non-string keys
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(security): update minimatch to 10.2.4 to fix CVE-2026-27903 and CVE-2026-27904
- Run npm audit fix in docs/my-website
- Updates minimatch from 10.2.1 to 10.2.4 (fixes HIGH severity ReDoS vulnerabilities)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(test): update realtime guardrail test assertions to match actual guardrail behavior
- test_text_message_blocked_by_guardrail_no_ai_response: allow guardrail's own block
message text in response.done (previously expected empty content)
- test_voice_transcript_blocked_by_guardrail: allow guardrail to send response.cancel
+ block message + response.create flow (previously expected no response.create)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: revert proxy-extras version in requirements.txt and pyproject.toml
The litellm-proxy-extras 0.4.50 is not published to PyPI yet, so consumer
references must stay at 0.4.49. Only the source package pyproject.toml
should be bumped to 0.4.50 for the publish_proxy_extras CI job.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: make transcript delta check optional in voice guardrail test
The guardrail sends an error event (guardrail_violation) when blocking
voice transcripts; it does not always produce transcript deltas. Remove
the assertion requiring response.audio_transcript.delta since the error
event is the primary signal that blocked content was handled.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Add missing env keys to documentation: LITELLM_MAX_STREAMING_DURATION_SECONDS and LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES
These two environment variables were used in code but not documented in the
environment variables reference section of config_settings.md, causing the
test_env_keys.py CI test to fail.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Fix 13 mypy type errors across 6 files
- in_flight_requests_middleware.py: Fix type: ignore error codes from
[union-attr] to [attr-defined], add [arg-type] for Gauge **kwargs
- transformation.py: Add [assignment] ignore for output_format reassignment,
add fallback empty string for tool use id to fix arg-type
- responses/main.py: Remove redundant type annotation on second
secret_fields assignment to fix no-redef
- streaming_iterator.py: Add [assignment] ignores for intermediate
cache token assignments
- handler.py: Add [typeddict-item] ignore for AnthropicMessagesRequest
construction from dict
- public_endpoints.py: Add [arg-type] ignore for _load_endpoints()
return type mismatch with SupportedEndpoint model
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add auth overrides to spend tracking tests, fix realtime guardrail assertion, update UI minimatch
- Add app.dependency_overrides for user_api_key_auth in 4 spend tracking tests
that were returning 401 Unauthorized (error_code, error_message,
error_code_and_key_alias, key_hash)
- Fix realtime guardrail test to check ANY error event for guardrail_violation
instead of just the first (OpenAI may send its own errors first)
- Update ui/litellm-dashboard/package-lock.json to fix minimatch vulnerability
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Fix failing MCP e2e and create_mcp_server UI tests
Test 1 (test_independent_clients_no_shared_session):
- Add allow_all_keys: true to MCP servers in test config. With master_key
and no DB, get_allowed_mcp_servers returned empty, causing 0 tools and
403 on tool calls. allow_all_keys bypasses per-key restrictions.
- Add asyncio.sleep(0.5) between client connections to allow MCP SDK
TaskGroup cleanup and avoid ExceptionGroup on connection close (MCP #915).
Test 2 (create_mcp_server 'auth value is provided'):
- Use userEvent.setup({ delay: null }) for instant keystrokes to avoid
timeout from default typing delay on CI.
- Increase per-test timeout to 15000ms for CI environments.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: stabilize proxy unit tests for parallel execution
- test_response_polling_handler: add xdist_group to prevent heavy import OOM
- test_db_schema_migration: use temp dir for worker isolation, sync schema.prisma index
- test_custom_tokenizer_bug: use lighter tokenizer to prevent OOM in parallel
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add auth overrides to more spend tracking and model info tests
- Fix test_ui_view_spend_logs_pagination missing auth override (401)
- Fix test_view_spend_tags missing auth override (401)
- Fix test_view_spend_tags_no_database missing auth override (401)
- Fix test_empty_model_list.py to use app.dependency_overrides instead of patch()
for FastAPI dependency injection auth
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(test): use patch.object for aiohttp transport test to work in parallel execution
The @patch decorator was not intercepting the static method call in parallel
xdist workers. Using patch.object on the directly-imported class is more reliable.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(security): update minimatch from 10.2.1 to 10.2.4 in Dockerfile
The Docker image was explicitly pinning minimatch@10.2.1 which has HIGH
severity ReDoS vulnerabilities (GHSA-7r86-cg39-jmmj, GHSA-23c5-xmqv-rm74).
Update to 10.2.4 which includes fixes for both CVEs.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ui): prevent MCP and TeamInfo test timeouts on CI
- Add userEvent.setup({ delay: null }) to all tests using userEvent in both files
- Add timeout: 15000 to tests with significant user interaction (typing, multiple clicks)
- Fixes: create_mcp_server Bearer Token test, TeamInfo cancel button test
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: stabilize parallel test execution and aiohttp transport test
- test_aiohttp_handler: rewrite transport test to not rely on static method mock
(consistently fails in parallel xdist workers)
- test_proxy_cli: add xdist_group to prevent timeout during heavy imports
- test_swagger_chat_completions: add xdist_group to prevent timeout
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(security): add serialize-javascript override to fix GHSA-5c6j-r48x-rmvq
Add npm override for serialize-javascript>=7.0.3 in docs/my-website
to fix HIGH severity RCE vulnerability via RegExp.flags.
Also bump minimatch override to >=10.2.4.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Fix flaky tests: remove broken Vertex model, add retries for Anthropic
- Remove vertex_ai/meta/llama-4-scout-17b-16e-instruct-maas from
test_partner_models_httpx_streaming - consistently returns 400 BadRequest
- Add @pytest.mark.flaky(retries=6, delay=10) to test_function_call_parsing
for transient Anthropic API overload errors
- Add @pytest.mark.flaky(retries=6, delay=10) to test_openai_stream_options_call
for transient Anthropic InternalServerError
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): add xdist_group(proxy_heavy) to prevent OOM in parallel proxy tests
- Add pytestmark = pytest.mark.xdist_group('proxy_heavy') to test_proxy_utils.py
- Change test_db_schema_migration.py from schema_migration to proxy_heavy group
- Add @pytest.mark.xdist_group('proxy_heavy') to test_proxy_server.py::test_health
Groups heavy proxy tests to run on same worker, avoiding worker OOM crashes.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Fix vertex AI qwen global endpoint test to mock vertexai module import
The test_vertex_ai_qwen_global_endpoint_url test was failing because the
VertexAIPartnerModels.completion() method tries to 'import vertexai' before
any of the mocked code runs. In environments without google-cloud-aiplatform
installed, this import fails with a VertexAIError(status_code=400).
Fix by:
- Adding patch.dict('sys.modules', {'vertexai': MagicMock()}) to mock the
vertexai module import
- Adding vertex_ai_location parameter to the acompletion call for completeness
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): add xdist_group to health endpoint and watsonx tests for parallel stability
- test_health_liveliness_endpoint: add xdist_group('proxy_health') to prevent timeout
- test_watsonx_gpt_oss tests: add xdist_group('watsonx_heavy') to prevent mock interference
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(test): pre-populate WatsonX IAM token cache to prevent parallel test interference
The watsonx prompt transformation test was failing in parallel execution because
litellm.module_level_client.post mock was being interfered with by other tests.
Pre-populating the IAM token cache avoids the HTTP call entirely.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(test): add spend data polling with retries for e2e pass-through tests
- test_vertex_with_spend.test.js: Replace 15s fixed wait with polling loop
(up to 6 attempts, 10s apart) for spend data to appear in DB
- Increase test timeout from 25s to 90s to accommodate polling
- base_anthropic_messages_tool_search_test.py: Add flaky(retries=3) for
streaming test that depends on live Anthropic API
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): reduce parallel workers from 8 to 4 for proxy tests to prevent OOM
- litellm_proxy_unit_testing_part2: -n 8 -> -n 4
- litellm_mapped_tests_proxy_part2: -n 8 -> -n 4, timeout 60 -> 120
- Worker crashes consistently caused by too many parallel proxy tests
each loading the full FastAPI app and heavy dependency tree
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(db): add migration for SpendLogs composite index (startTime, request_id)
The @@index([startTime, request_id]) was added to schema.prisma but had no
corresponding migration. This caused test_aaaasschema_migration_check to fail
because prisma migrate diff detected the missing index.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(db): add migration for MCP available_on_public_internet default change to true
The schema.prisma changed the default for available_on_public_internet from
false to true, but no migration was created. This caused the schema migration
test to detect drift.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(test): increase server wait time and add retry to flaky external API tests
- test_basic_python_version.py: increase server startup wait from 60s to 90s
for slower CI environments (fixes installing_litellm_on_python_3_13)
- test_a2a_agent.py: add flaky(retries=3, delay=5) for non-streaming test
that depends on live A2A agent endpoint
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(test): add flaky retries to all intermittent external API tests for 0-fail CI
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(test): add auth overrides to file endpoint tests that return 500
The test_target_storage tests were getting 500 because the FastAPI auth
dependency wasn't overridden. Added app.dependency_overrides for proper
auth bypass in test environment.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Replace Click CliRunner with standalone_mode=False to avoid
"I/O operation on closed file" errors caused by Click's stream
isolation in CI environments.
- TestSpendLogsPayload: save/restore litellm.cache in setup_method/teardown_method
so tests that run after a cache-setting test don't see a non-None cache and get
a hash instead of "Cache OFF" in the cache_key field
- test_use_prisma_db_push_flag_behavior: apply clean_env pattern (strip DATABASE_URL/DIRECT_URL,
then set DATABASE_URL to test value) inside the with block instead of using @patch.dict
decorator, matching the pattern from test_skip_server_startup to avoid Click 8.3.x
StreamMixer stream lifecycle issues in CI
* fix(test): remove deprecated Click mix_stderr param in test_use_prisma_db_push_flag_behavior
Click 8.2+ removed the mix_stderr parameter from CliRunner. Use CliRunner() without it.
* fix(test): use app.dependency_overrides for auth mock in test_role_mappings_stored_and_retrieved
monkeypatch.setattr doesn't affect FastAPI's Depends() resolution in parallel
test execution. Use app.dependency_overrides which is the proper FastAPI pattern.
* fix: reset db_health_cache in source module to prevent stale cache hits
The test was reassigning db_health_cache via `global` in the test module,
which doesn't affect the _health_endpoints module's variable. When a prior
test set the cache to "connected" within 2 minutes, _db_health_readiness_check
returned early without calling health_check(), causing assert_called_once to fail.
Also use PrismaError with a connection message so it's properly recognized
as a connection error by PrismaDBExceptionHandler.is_database_connection_error.
* fix: replace asyncio.sleep with polling loop in spend logs tests
The GLOBAL_LOGGING_WORKER processes callbacks via an async queue, so
asyncio.sleep(1) is a race condition - under CI load the worker may not
have processed the queued task within 1 second. Replace with a polling
helper that waits up to 10 seconds for the mock to be called.
Also add metadata.attempted_retries and metadata.max_retries to
ignored_keys since these are new fields.
* fix: isolate test_skip_server_startup from CI environment
Remove mix_stderr=False (unsupported in some Click versions). Strip
DATABASE_URL/DIRECT_URL from environment during the test to prevent
real prisma operations when these are set in CI.
- Add new SpendLogsMetadata keys to ignored_keys in spend logs tests
(regression from ccecc10c82 which intentionally includes all keys)
- Mock PrismaManager.setup_database and should_update_prisma_schema in
proxy CLI tests to prevent real DB migrations from running in CI
- Use CliRunner(mix_stderr=False) to fix Click stream lifecycle issues
- Use unique UUID suffix for Redis TTL test keys to avoid stale state
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add DATABRICKS_API_KEY to environment settings reference
* fix: streaming test usage check on Pydantic model
* fix: mock litellm.proxy.proxy_server in test_skip_server_startup
* [Fix] Containers API - Allow routing to regional endpoints (#19118)
* fix get_complete_url
* fix url resolution containers API
* TestContainerRegionalApiBase
* feat(proxy): add keepalive_timeout support for Gunicorn server
Add configurable keepalive timeout parameter for Gunicorn workers to
match existing Uvicorn functionality. This allows users to tune the
keep-alive connection timeout based on their deployment requirements.
Changes:
- Add keepalive_timeout parameter to _run_gunicorn_server method
- Configure Gunicorn's keepalive setting (defaults to 90s if not specified)
- Update --keepalive_timeout CLI help text to document both Uvicorn and Gunicorn behavior
- Pass keepalive_timeout from run_server to _run_gunicorn_server
Tests:
- Add test to verify keepalive_timeout flag is properly passed to Gunicorn
- Add test to verify default 90s timeout when flag is not specified
Co-Authored-By: lizhen921 <294474470@qq.com>
Signed-off-by: Kris Xia <xiajiayi0506@gmail.com>
---------
Signed-off-by: Kris Xia <xiajiayi0506@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: lizhen921 <294474470@qq.com>
* fix(litellm_proxy/chat/transformation.py): support 'user' and all other openai chat completion params
Fixes issue where 'user' was not being sent in request to litellm proxy via sdk
* fix(prisma_migration.py): remove 'use_prisma_migrate' flag, is now default
* docs: cleanup docs
* fix(proxy_cli.py): remove --use_prisma_migrate flag
* refactor: remove references to use_prisma_migrate env var
This is now the default flow for db migrations
* Enhance proxy CLI with Rich formatting and improved user experience
- Integrated Rich library for better console output in `proxy_cli.py`, including version display, health check results, and test completion responses.
- Updated health check and test completion methods to provide progress indicators and formatted tables.
- Refactored feedback display in `proxy_server.py` to use Rich for a more visually appealing user interface.
- Adjusted tests in `test_proxy_cli.py` to mock console output instead of using print statements, ensuring compatibility with Rich formatting.
* fix linting error
* refactor(proxy_cli.py): simplify DB setup logging
- Removed progress indicators for IAM token generation and environment variable decryption to simplify the code.
- Consolidated the logic for generating the database URL and setting environment variables.
- Enhanced error handling for configuration loading and database setup, ensuring clearer feedback
* Update test-linting workflow to include proxy-dev dependencies in Poetry installation
* Enhance proxy server initialization with Rich console for improved model display. Added support for loading model parameters from environment variables and refined provider identification logic. Fallback to original print formatting if Rich is not available.
* Refactor feedback handling: Moved feedback message generation and custom warning display to utils.py. Enhanced feedback box with rich formatting and fallback to ASCII for environments without rich. Cleaned up proxy_server.py by removing obsolete code.
* fix linting error
* Refactor model initialization display: Moved model initialization logic to a new utility function `display_model_initialization` for improved readability and maintainability. Enhanced model provider extraction with a dedicated function. Fallback to basic logging if Rich console is unavailable.
* Refactor model provider extraction: Replace the `_extract_provider_from_model` function with a more robust approach using `get_llm_provider`. Implement fallback logic for provider identification and improve error handling. Ensure compatibility with Rich console for model initialization display.
