The cost map's fal_ai/fal-ai/trellis-2 entry prices its output by resolution
with output_cost_per_image_512, output_cost_per_image_1024, and
output_cost_per_image_1536, which litellm/types/utils.py types and the fal_ai
cost calculator reads, but INTENDED_SCHEMA in
test_aaamodel_prices_and_context_window_json_is_valid never allowed them, so
the test fails on main with "Additional properties are not allowed". Add the
three keys next to output_cost_per_image in the schema and in the cost-under-1
field list so a per-size image price is validated like the per-size video ones
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The test priced gemini/gemini-3.1-flash-image-preview, which #42435 removed
from the cost map as deprecated, so the calculator had no per-token rates to
keep and the hardcoded expected value no longer matched. Price the live
gemini/gemini-3.1-flash-image row instead and derive the expected cost from
that row in litellm.model_cost, so a rate change on it cannot break the test
while a calculator that drops the map's token rates still fails it.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): search users by id or email when assigning a key owner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): label users without an email by user id in key owner picker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): freeze merged user-filter where, format create key test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): suppress module-global patch findings in ui_view_users search test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep merged user-filter where as a plain dict for prisma serialization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): forward search param from userFilterUICall to /user/filter/ui
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): mention user ID in the Create Key user picker helper text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add immutable model catalog crate
* feat(rust): add typed model info mirror, error module, and schema feature
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): split catalog module, rstest tests, and repo data parity tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
#42521 retired the cohere.command-r ids from the cost map, and a bare Bedrock id resolves its provider through that map, so test_model_group_info and the cohere cases in test_bedrock_dynamic_auth_params_unit_tests failed with LLM Provider NOT provided. The completion tests keep the cohere invoke request and URL assertions through the bedrock/ prefix, the bare parametrize entry moves to amazon.nova-2-lite-v1:0 with the mock response shape picked by the real Bedrock route, and the router test builds its group info from bedrock/amazon.nova-2-lite-v1:0.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): skip non-dict content items in beta-header and file-id helpers so malformed content lists return 400 instead of 500
Fixes#42094
Supersedes #42101
Co-authored-by: Pawan-Shahane <shahanepawan511@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): spawn the DB-less regression proxy with -P so the cwd cannot shadow the pinned checkout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): launch the DB-less proxy via -I -c with an explicit sys.path so python 3.10 works, drop DIRECT_URL, remove restating docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): gate the self-booted DB-less proxy behind the owned_gateway opt-in the Buildkite container cannot satisfy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): move the bare string content item repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(tests): wrap the anthropic bare string wire test to the 120 column limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(tests): wrap anthropic common_utils test literals to the 120 column limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: fix ruff findings in touched test files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Pawan-Shahane <shahanepawan511@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): return 400 from /spend/calculate for a model with no pricing row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): assert error type and param for unpriced /spend/calculate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): move the repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: alias ModelNotMappedError re-export to satisfy F401
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): raise ModelNotMappedError only when the pricing row is missing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): record the GenAI exception event without the removed Events API
OpenTelemetry removed opentelemetry._events in 1.44.0, so the three imports of
it broke 7 modules under litellm.integrations.otel, including the entry point.
Two things then failed quietly: with LITELLM_OTEL_V2 set the otel callback
resolved to None and nothing was exported, and with it unset the newrelic
callback was dropped as well, because that branch imports the v2 logger
ungated
Build and emit the event through the Logs API, which both lines carry. The
event name keeps riding the event.name attribute: the event_name log record
field that replaces it only exists from 1.44.0, and this package pins 1.28.0,
so the attribute is the only form both can write. It is also what the Events
API wrote, so exported events keep their shape
Emitting a plain record drops the default the Events SDK applied, so the
timestamp now falls back to time_ns() here
* style(otel): trim the event name key and regression test prose
Keep only the constraint a reader cannot infer from the code, that the
event_name record field does not exist on the pinned OpenTelemetry line
* fix(otel): export the GenAI exception event on both OpenTelemetry 1.28 and 1.44 lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): drop the record selection comment
Co-authored-by: Pawan-Shahane <shahanepawan511@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(otel): collapse the record selection conditional for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(otel): build the record fields with a dict literal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): set the native event_name on the 1.44 log record
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(otel): spell out the record kwargs so the type gate sees each call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(otel): suppress the version-gated kwargs for the type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): drop the version-window prose and correct the event_name suppression reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(otel): wrap the compat test docstring to the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Pawan-Shahane <shahanepawan511@gmail.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): keep embedding cache hits aligned with request inputs
Partial hits now send only the uncached inputs to the provider and merge
fresh vectors back into their original positions. Responses whose item
count differs from the input count (one input scoring many documents)
are no longer written to the per-input cache, since a later hit would
return a single item.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): drop mutable collection builds flagged by the type discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): bypass embedding cache entries written before the per input cardinality check
Embedding cache entries now carry format_version and readers treat entries without it as
misses, so entries that only hold the first row of a multi row response are refetched instead
of served until their TTL expires. The provider call also receives a copy of the request kwargs
with the uncached inputs rather than mutating the caller's mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): assert a partial embedding cache hit becomes a full hit on repeat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): await pending embedding cache writes before asserting on cache hits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): validate cached embeddings without mutating responses or request kwargs
Validate cache rows through a frozen pydantic model so import does not depend on
TypeAdapter support for ReadOnly TypedDicts, accept string embeddings, build the
merged partial hit response instead of mutating the cached one, and hand the
provider request mapping to post call hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): keep cache_hit and response_ms on merged partial embedding hits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): run the memory cell alone on the shared stack
* test(e2e): hold the stack lock for every collected test, marker or not
* test(e2e): prove the stack lock's reader sharing and writer preference across processes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: count a zombie grandchild as gone in the migrate deploy timeout test
A SIGKILLed grandchild whose parent died in the same killpg reparents to
PID 1 or the nearest subreaper and stays a zombie until reaped, and signal 0
still succeeds on a zombie, so the timeout test read it as alive wherever
PID 1 is slow to reap or never does. The sibling test in
tests/test_litellm/proxy/db already handled that; both now share one
process_is_gone helper that reads the /proc state and reaps its own
children, with unit tests for the live, reaped, unreaped, and foreign
zombie shapes.
* test: move the pre-commit interrupt test onto the shared zombie-aware liveness helper
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): make two proxy-infra tests independent of sibling-test state
Both tests read process-global state that another module in the same xdist
worker can change, so they passed or failed on shard scheduling rather than on
the behaviour they assert.
test_gateway_plus_backend_covers_full_app checked allowlist coverage against
the live route table. gateway/main.py trims routes once, inside the lifespan,
so it only ever sees what is registered at startup; a lazy feature appends its
router on demand afterwards and is never filtered. Those routes cannot be
dropped on the floor, but they do enter the assertion the moment a sibling test
warms the feature, and 110 of them sit in neither allowlist. Subtract exactly
the lazy features this process has loaded, which leaves the assertion at full
strength for every eagerly registered route.
test_real_proxy_child_auth_privacy_and_body_policy pins prisma_client to a bare
object(). litellm.max_budget is a module global that nothing restores between
tests; once a sibling leaves it above zero, user_api_key_auth takes the
global-spend branch, dereferences prisma_client.db, and the AttributeError
surfaces as HTTP 401. Pin max_budget next to the other globals the test already
controls.
* test(proxy): measure allowlist coverage in a pristine interpreter
The previous revision subtracted paths matching a loaded lazy feature's
prefixes. Those prefixes are broad enough to swallow eagerly registered routes:
31 of them, including /openai/deployments/*, /access_group/*, /cursor/* and
/mcp, which would have made a real allowlist regression invisible.
Run the coverage check in a fresh interpreter instead. No lazy feature is
loaded there, so the route table is exactly the one gateway/main.py's lifespan
trim sees, and nothing has to be subtracted for the result to be deterministic.
The probe also reports the lazy modules it loaded and its route count, so an
empty uncovered set cannot pass vacuously.
Drop _component_paths and the four allowlist constants it used; the probe
reproduces the predicate in the child process.
* feat(proxy): configurable key_alias_pattern for key generate, update, and regenerate
Adds litellm_settings.key_alias_pattern, a regex every key_alias sent to
/key/generate, /key/service-account/generate, /key/update, and
/key/{key}/regenerate has to fully match. A non-matching alias gets a 400
that names the setting and the pattern. When set, it replaces the built-in
rule enable_key_alias_format_validation turns on, and the baseline
unsafe-name check still runs first. An invalid regex fails config load.
* fix(proxy): cap key_alias length under key_alias_pattern and type the test fixtures
* style(proxy): declare key_alias_pattern with a PEP 604 union
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): hold every worker under an idle RSS budget before any traffic
The harness reads /debug/memory/summary on every replica once at collection
time, right after the readiness gate and before this pytest process sends any
traffic, and the memory suite's first test fails when any worker idles past
E2E_MEMORY_IDLE_RSS_BUDGET_MB (768 MB by default) or gives no reading at all.
A v1.100.x worker with a database idled at 836-886 MB where v1.101.0rc1 idled
at 544 MB on the same database: prisma-client-py's default recursive type
depth generated 91k TypedDict classes that v1.101.0's recursive_type_depth = -1
cut to 19k. The budget starts at the rc1 reading plus headroom.
* test(e2e): read idle RSS only when the idle budget test is selected
Gate the collection-time /debug/memory/summary read on a selected test using
the idle_rss fixture and skip it under --collect-only, so sessions that never
run the idle budget test pay no round trip. Drop the markerless unit test file
the e2e guide bans and assert live that every configured replica was measured
* test(e2e): take the idle RSS read after collection settles
Read every replica's RSS from a tryfirst pytest_collection_finish hook so -k
and -m deselection has already run, and only when a selected test still asks
for the idle_rss fixture and the run is not --collect-only
* test(e2e): record the heaviest idle RSS reading as junit properties
* test(e2e): attach the idle RSS properties from the harness's setup hook
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune
* test(vcr): derive the leak guard's patch points from vcrpy's own reset list
* test(vcr): share CapturingTransport and switch the encoding_format embedding test to it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore(cost-map): remove models past their deprecation date
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop the empty parametrize left behind by the gemini web search removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): drop merge base block left by conflict resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop gemini image cost tests pinned on removed model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): attribute provider and model info on pre-call rejected requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep pre-call rejections out of deployment cooldown and prometheus deployment state
Stamp model_info only into the logging metadata so the router's failure callbacks do not
count a key-level 429 or guardrail 403 against the deployment, treat a resolved plus an
unresolved deployment as ambiguous provider attribution, and stop the prometheus deployment
counters and deployment_state from treating a proxy-side reject as a selected deployment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): skip deployment attribution when the rejected body's model is not a string
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): bucket non-string request models as other instead of raising in failure hook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve team deployments and treat guardrail rejects as proxy-side in failure attribution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): flag pre-routing rejects instead of matching exception names
Post-call GuardrailRaisedException failures kept their deployment labels on main but lost them
on this branch because every GuardrailRaisedException was treated as a pre-routing reject.
The proxy failure path now flags litellm_params with proxy_rejected_before_routing only when
it adds deployment attribution itself, and the Prometheus logger keys deployment selection
off that flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): key pre-routing reject flag off provider handoff, not caller metadata
Caller-supplied metadata.model_info (kept for keys allowed to override pricing) no longer
suppresses proxy_rejected_before_routing. The hook now checks the logging object's
first_api_call_start_time, which only the provider handoff sets, so Prometheus never
records a deployment failure for a request that was rejected before routing.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): poll for both served and rejected spend rows before asserting attribution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(proxy): honor DATABASE_DISABLE_PREPARED_STATEMENTS in the litellm CLI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): pooled pgbouncer url keeps a single pgbouncer=true when the upstream already carries it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject a malformed DATABASE_DISABLE_PREPARED_STATEMENTS even when the config already enables it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gateway): expose /api/event_logging/batch on the gateway allowlist
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gateway): route /api/event_logging to gateway pods in helm and terraform
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): stop max_budget leaking between proxy_server and native_compaction tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): registry audit 2026-09-22, absorb open pricing PRs
Rolls the open registry-only PRs into one PR after re-verifying every value against the official provider source: OpenAI, Azure, Vertex AI and Gemini batch cache-read prices, Baseten model metadata from the authenticated inference API, Bedrock eu-west-2 Nemotron Super 3 pricing from the AWS offer file, and OpenRouter prices refreshed from the live OpenRouter models API
Co-authored-by: sinksilk <785976238@qq.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): add groq/llama-guard-3-8b from the Groq model page
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): refresh openrouter deepseek aliases from live api and drop stale off-peak windows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): resolve baseten merge conflicts against main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: sinksilk <785976238@qq.com>
* fix(ssrf): point the blocked-address remediation at litellm_settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ssrf): pin the block message's named section to litellm_settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(compaction): pin litellm.max_budget so leaked proxy budget cannot 401 the child auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): honor SSL_CERT_FILE and ssl_verify in OTLP HTTP exporters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): assert OTLP HTTP TLS behavior against a real TLS sink
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): assert rejected exports by outcome, not by exception type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): honor SSL_CERT_FILE and ssl_verify in the v2 OTLP HTTP exporters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): hoist otlp_tls imports and type the TLS sink fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): gate the OTLP TLS export test behind an otel_tls opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(e2e): drop CONTRIBUTING.md edit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(mcp): apply async_post_mcp_tool_call_hook content changes to the tool result
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): drop structuredContent when a post-call hook rewrites tool content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): satisfy type discipline and result contract
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): document internal logging patch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): avoid Final assignments inside callback loops
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): run every post-call hook and chain the rewritten content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(mcp): credit the original fix from #33403
Co-authored-by: eric <mitrecx@163.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): cover post-call logging fallback paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): cover proxy hook logging context
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): preserve native structured guardrail replacements
* fix(mcp): invalidate stale structure after direct content edits
* fix(mcp): reconcile direct edits after callback exceptions
* fix(mcp): preserve successful in-place callback rewrites
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: eric <mitrecx@163.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* test(e2e): one request lands the same spend on every surface
One priced chat request must show the same response_cost on the spend log row, /key/info, /team/info, the usage export's /user/daily/activity/aggregated row, and the litellm_spend_metric Prometheus sample; each is a separate writer, so the test fails naming the surface that drifted
* fix(e2e): scrape every replica's /metrics/ and enable prometheus in the replay lane
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): share model rate-limit buckets between a model_group_alias and its target
A request sent under a model_group_alias counted in its own per-key, per-team,
per-org, and per-project model bucket, so a key could double a deployment's
default_api_key_rpm_limit / tpm_limit by alternating the alias and the model
group name, and a metadata model_rpm_limit / model_tpm_limit keyed by the
model group never applied to alias requests. The limiter now resolves the
requested name to its model group before keying any model bucket, looks the
limit up by the requested name first and the model group second, and charges
post-call tokens to the same bucket.
* fix(proxy): charge the model group resolved at admission when reconciling reserved tokens
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: enforce disable_custom_api_keys from general_settings
The gate in _check_custom_key_allowed read the persisted UI settings row
through get_ui_settings_cached, which had two consequences.
A config-file general_settings.disable_custom_api_keys was never enforced,
because the gate only ever looked at the stored ui_settings row. POST
/key/generate with a custom key value returned 200 even with the flag set
to true in config.yaml.
The read went through a DualCache with a 600s TTL, which is per worker
without Redis, and only the worker that served PATCH /update/ui_settings
refreshed it. A gate flipped through the UI was then a coin flip across
workers for up to ten minutes.
Both go away by routing the flag the way the other runtime UI flags are
already routed. Adding it to _RUNTIME_GENERAL_SETTINGS_FLAGS and to the
settings rules' _UI_SETTINGS_FIELDS makes SettingsStore resolve it from the
ui_settings row with the config file winning, and every pod re-reads it on
its own settings sync rather than holding a private cached copy. The two
lists have to stay in step: a flag in one and not the other resolves
against the wrong stored row and silently never reaches a reader, so there
is a test for that invariant.
Writes to the ui_settings table did not publish on the config-sync channel,
so other pods only discovered a change on their next periodic reload. Adding
litellm_uisettings to _CONFIG_SYNCED_TABLE_NAMES puts it on the same pubsub
path model and SSO config writes already use, which cuts cross-pod
propagation from tens of seconds to a few.
The value reaching the gate is run through coerce_bool first. Resolution
hands back the raw YAML value, so a quoted "true" in config.yaml is a str
and the old `is True` check let custom keys straight through.
* test: assert both directions of the coerced config value
* test: assert the runtime flags read back instead of inspecting the registry
* fix(fal_ai): honour global api_base for image generation and reject non-string reasoning_effort with 400
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): add return annotations to reasoning_effort regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): move the LIT-8340 repro from tests/e2e to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): align the cache crates with Python and activate every backend
The cache port had drifted: lifecycle and Redis-only operations sat on
`BaseCache`, counters were pinned to `f64`, each semantic backend defined its
own embedder and prompt handling, and only the in-memory backend could be
selected natively.
- Split `disconnect` and `test_connection` out of `BaseCache` into optional
capabilities, implemented only where the Python class defines them, and give
every Redis-only operation its own capability trait.
- Decouple counters from the stored value type, so one backend can serve both
responses and counters as Python's `RedisCache` does.
- Share one `Embedder` and prompt contract in `litellm_cache::semantic`, and
make the Redis and Valkey semantic backends generic over their codec.
- Port the Python operations that were missing: `async_refresh_ttl`,
`async_rpush_and_trim`, `async_set_cache_pipeline_with_ttls`, the DualCache
pipeline, sadd, bulk delete and TTL reads, and the semantic-similarity
write-back.
- Take the HTTP client from the host pool in the GCS, S3 and Azure backends.
- Activate all nine backends through the Rust catalog, whose rules all stay
`PYTHON_ONLY`, and route the `Cache` facade's storage calls to the native
runtime when one is selected.
- Give every crate the same layout, move all tests to `tests/` on rstest, and
add the shared `litellm-cache-testing` contract suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: freeze native cache request kwargs and batch entries for type discipline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: declare semantic lookup methods in the native stub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): align the cache crates with Python and activate every backend
The cache port had drifted: lifecycle and Redis-only operations sat on
`BaseCache`, counters were pinned to `f64`, each semantic backend defined its
own embedder and prompt handling, and only the in-memory backend could be
selected natively.
- Split `disconnect` and `test_connection` out of `BaseCache` into optional
capabilities, implemented only where the Python class defines them, and give
every Redis-only operation its own capability trait.
- Decouple counters from the stored value type, so one backend can serve both
responses and counters as Python's `RedisCache` does.
- Share one `Embedder` and prompt contract in `litellm_cache::semantic`, and
make the Redis and Valkey semantic backends generic over their codec.
- Port the Python operations that were missing: `async_refresh_ttl`,
`async_rpush_and_trim`, `async_set_cache_pipeline_with_ttls`, the DualCache
pipeline, sadd, bulk delete and TTL reads, and the semantic-similarity
write-back.
- Take the HTTP client from the host pool in the GCS, S3 and Azure backends.
- Activate all nine backends through the Rust catalog, whose rules all stay
`PYTHON_ONLY`, and route the `Cache` facade's storage calls to the native
runtime when one is selected.
- Give every crate the same layout, move all tests to `tests/` on rstest, and
add the shared `litellm-cache-testing` contract suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: freeze native cache request kwargs and batch entries for type discipline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: declare semantic lookup methods in the native stub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust): opt the native Messages and tokenizer suites into Rust explicitly
#42517 made the Messages, token counter and tokenizer routes Python-only, so
tests/test_litellm_rust silently exercised the Python path or failed outright.
Each suite now prepends a RUST_OPT_IN rule for its route, keeping native
coverage without changing the shipped default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust): pop one at a time in the Redis 6 lpop pipeline and drop explanatory comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): reuse the status client and headers on the video result probe
The Fal result GET issued after a COMPLETED status poll built its own default
client and only carried Authorization and Content-Type, so an injected client,
a request-level ssl_verify and extra_headers were honored on the status GET but
not on the result GET, and a transport failure on that probe escaped as a 500.
The handler now hands the selected sync or async client to the provider status
transform, Fal reuses it with the full validated header set, and a probe
transport error stays non-terminal like the existing 429 and 5xx handling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): mark the result probe header dict as mutable-ok for the type discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): move the result probe repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): restore the shared e2e helpers to the merge base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): align /fal_ai queue gate with the pricer and normalise resolution type
The /fal_ai gate admitted catalog keys the Fal pricer cannot price, so
those jobs were forwarded and logged at 0.0 spend. The gate now reuses
the pricer as its eligibility predicate. Resolution is normalised to a
string before the keyed price lookup so int and str spellings bill the
same.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): assert passthrough pricing invariants on synthetic catalog entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): price the submitted body in the /fal_ai queue gate so keyed-only entries are admitted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): move the queue gate repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
gemini-2.5-flash-lite retires on 2026-10-20 per Google's Vertex model
lifecycle table, which would turn the google_generate_content_endpoint_testing
CircleCI job red on every PR from that day. gemini-3.5-flash-lite is the newest
flash-lite on both AI Studio and Vertex, with a retirement of 2027-07-21 or
later.
Vertex serves gemini-3.5-flash-lite only on the global, us, and eu endpoints,
never on a single region, so the direct SDK Vertex tests now pass
vertex_location global like the suite's proxy config already does.
The 429 backoff test derives its mocked generateContent URL from litellm's own
Gemini URL builder, since litellm routes Gemini 3 and newer models to v1alpha
instead of v1beta.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(e2e): route credential, cost map, and UI login calls to the control plane
* test(e2e): stop pinning the upstream model name in the credential-backed messages test
* test(e2e): drop the harness unit test, the split stack run proves the routing live
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(logs): add span type filter to request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logs): look up span type sql conditions from a mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): opt-in include_guardrail_response returns guardrail_information in the response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): read include_guardrail_response from the request metadata bucket the router did not reseed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format common request processing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): redact matched content in guardrail_information and stop mutating cached responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): traverse guardrail diagnostics iteratively
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): annotate guardrail traversal cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): reuse core redaction helper for guardrail_information
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): justify response rebind when attaching guardrail information
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): assert a cooldown reaches a sibling replica within the 1s Redis read interval
* test(e2e): skip the sibling replica cooldown cell when one gateway URL is named
* test(e2e): collapse repeated gateway addresses so the sibling cooldown cell skips instead of erroring
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
