* fix: enforce disable_custom_api_keys from general_settings
The gate in _check_custom_key_allowed read the persisted UI settings row
through get_ui_settings_cached, which had two consequences.
A config-file general_settings.disable_custom_api_keys was never enforced,
because the gate only ever looked at the stored ui_settings row. POST
/key/generate with a custom key value returned 200 even with the flag set
to true in config.yaml.
The read went through a DualCache with a 600s TTL, which is per worker
without Redis, and only the worker that served PATCH /update/ui_settings
refreshed it. A gate flipped through the UI was then a coin flip across
workers for up to ten minutes.
Both go away by routing the flag the way the other runtime UI flags are
already routed. Adding it to _RUNTIME_GENERAL_SETTINGS_FLAGS and to the
settings rules' _UI_SETTINGS_FIELDS makes SettingsStore resolve it from the
ui_settings row with the config file winning, and every pod re-reads it on
its own settings sync rather than holding a private cached copy. The two
lists have to stay in step: a flag in one and not the other resolves
against the wrong stored row and silently never reaches a reader, so there
is a test for that invariant.
Writes to the ui_settings table did not publish on the config-sync channel,
so other pods only discovered a change on their next periodic reload. Adding
litellm_uisettings to _CONFIG_SYNCED_TABLE_NAMES puts it on the same pubsub
path model and SSO config writes already use, which cuts cross-pod
propagation from tens of seconds to a few.
The value reaching the gate is run through coerce_bool first. Resolution
hands back the raw YAML value, so a quoted "true" in config.yaml is a str
and the old `is True` check let custom keys straight through.
* test: assert both directions of the coerced config value
* test: assert the runtime flags read back instead of inspecting the registry
* fix(fal_ai): honour global api_base for image generation and reject non-string reasoning_effort with 400
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): add return annotations to reasoning_effort regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): move the LIT-8340 repro from tests/e2e to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): align the cache crates with Python and activate every backend
The cache port had drifted: lifecycle and Redis-only operations sat on
`BaseCache`, counters were pinned to `f64`, each semantic backend defined its
own embedder and prompt handling, and only the in-memory backend could be
selected natively.
- Split `disconnect` and `test_connection` out of `BaseCache` into optional
capabilities, implemented only where the Python class defines them, and give
every Redis-only operation its own capability trait.
- Decouple counters from the stored value type, so one backend can serve both
responses and counters as Python's `RedisCache` does.
- Share one `Embedder` and prompt contract in `litellm_cache::semantic`, and
make the Redis and Valkey semantic backends generic over their codec.
- Port the Python operations that were missing: `async_refresh_ttl`,
`async_rpush_and_trim`, `async_set_cache_pipeline_with_ttls`, the DualCache
pipeline, sadd, bulk delete and TTL reads, and the semantic-similarity
write-back.
- Take the HTTP client from the host pool in the GCS, S3 and Azure backends.
- Activate all nine backends through the Rust catalog, whose rules all stay
`PYTHON_ONLY`, and route the `Cache` facade's storage calls to the native
runtime when one is selected.
- Give every crate the same layout, move all tests to `tests/` on rstest, and
add the shared `litellm-cache-testing` contract suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: freeze native cache request kwargs and batch entries for type discipline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: declare semantic lookup methods in the native stub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): align the cache crates with Python and activate every backend
The cache port had drifted: lifecycle and Redis-only operations sat on
`BaseCache`, counters were pinned to `f64`, each semantic backend defined its
own embedder and prompt handling, and only the in-memory backend could be
selected natively.
- Split `disconnect` and `test_connection` out of `BaseCache` into optional
capabilities, implemented only where the Python class defines them, and give
every Redis-only operation its own capability trait.
- Decouple counters from the stored value type, so one backend can serve both
responses and counters as Python's `RedisCache` does.
- Share one `Embedder` and prompt contract in `litellm_cache::semantic`, and
make the Redis and Valkey semantic backends generic over their codec.
- Port the Python operations that were missing: `async_refresh_ttl`,
`async_rpush_and_trim`, `async_set_cache_pipeline_with_ttls`, the DualCache
pipeline, sadd, bulk delete and TTL reads, and the semantic-similarity
write-back.
- Take the HTTP client from the host pool in the GCS, S3 and Azure backends.
- Activate all nine backends through the Rust catalog, whose rules all stay
`PYTHON_ONLY`, and route the `Cache` facade's storage calls to the native
runtime when one is selected.
- Give every crate the same layout, move all tests to `tests/` on rstest, and
add the shared `litellm-cache-testing` contract suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: freeze native cache request kwargs and batch entries for type discipline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: declare semantic lookup methods in the native stub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust): opt the native Messages and tokenizer suites into Rust explicitly
#42517 made the Messages, token counter and tokenizer routes Python-only, so
tests/test_litellm_rust silently exercised the Python path or failed outright.
Each suite now prepends a RUST_OPT_IN rule for its route, keeping native
coverage without changing the shipped default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust): pop one at a time in the Redis 6 lpop pipeline and drop explanatory comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): reuse the status client and headers on the video result probe
The Fal result GET issued after a COMPLETED status poll built its own default
client and only carried Authorization and Content-Type, so an injected client,
a request-level ssl_verify and extra_headers were honored on the status GET but
not on the result GET, and a transport failure on that probe escaped as a 500.
The handler now hands the selected sync or async client to the provider status
transform, Fal reuses it with the full validated header set, and a probe
transport error stays non-terminal like the existing 429 and 5xx handling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): mark the result probe header dict as mutable-ok for the type discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): move the result probe repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): restore the shared e2e helpers to the merge base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): align /fal_ai queue gate with the pricer and normalise resolution type
The /fal_ai gate admitted catalog keys the Fal pricer cannot price, so
those jobs were forwarded and logged at 0.0 spend. The gate now reuses
the pricer as its eligibility predicate. Resolution is normalised to a
string before the keyed price lookup so int and str spellings bill the
same.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): assert passthrough pricing invariants on synthetic catalog entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): price the submitted body in the /fal_ai queue gate so keyed-only entries are admitted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): move the queue gate repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
gemini-2.5-flash-lite retires on 2026-10-20 per Google's Vertex model
lifecycle table, which would turn the google_generate_content_endpoint_testing
CircleCI job red on every PR from that day. gemini-3.5-flash-lite is the newest
flash-lite on both AI Studio and Vertex, with a retirement of 2027-07-21 or
later.
Vertex serves gemini-3.5-flash-lite only on the global, us, and eu endpoints,
never on a single region, so the direct SDK Vertex tests now pass
vertex_location global like the suite's proxy config already does.
The 429 backoff test derives its mocked generateContent URL from litellm's own
Gemini URL builder, since litellm routes Gemini 3 and newer models to v1alpha
instead of v1beta.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(e2e): route credential, cost map, and UI login calls to the control plane
* test(e2e): stop pinning the upstream model name in the credential-backed messages test
* test(e2e): drop the harness unit test, the split stack run proves the routing live
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(logs): add span type filter to request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logs): look up span type sql conditions from a mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): opt-in include_guardrail_response returns guardrail_information in the response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): read include_guardrail_response from the request metadata bucket the router did not reseed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format common request processing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): redact matched content in guardrail_information and stop mutating cached responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): traverse guardrail diagnostics iteratively
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): annotate guardrail traversal cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): reuse core redaction helper for guardrail_information
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): justify response rebind when attaching guardrail information
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): assert a cooldown reaches a sibling replica within the 1s Redis read interval
* test(e2e): skip the sibling replica cooldown cell when one gateway URL is named
* test(e2e): collapse repeated gateway addresses so the sibling cooldown cell skips instead of erroring
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch_interception): propagate parent session/trace ids into intercepted searches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(websearch_interception): let parent correlation win over configured search params and type test params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): bill an intercepted web search under the parent request session
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop unrelated reformatting from the websearch session harness change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): keep the websearch interception session suite out of the stage-mirror gate
The stage-mirror stack runs no websearch_interception callback or search tool, so the suite is deselected there and the changed-tests gate fails on a file that executed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): run the websearch interception session suite on the stage-mirror stack
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(e2e): leave CONTRIBUTING.md untouched
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Python 3.13+ strips the common indentation of docstrings at compile time and 3.12
keeps it, and the 429 error description in ERROR_RESPONSES came straight from
RateLimitError.__doc__, so regenerating litellm/proxy/_lazy_openapi_snapshot.json
on a 3.13+ venv produced a one-line diff that the check-ui-api-types job (Python
3.12) rejected. Run the docstring through inspect.cleandoc before it lands in the
spec, regenerate the snapshot once, and pin the behavior with a test
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(langsmith): json.dumps with default=str so non-serializable metadata does not crash batch flush
Serialize the runs/batch payload with json.dumps(default=str, allow_nan=False) and send it as content= with an explicit Content-Type, so datetime, Decimal and similar metadata values no longer raise TypeError and drop the batch. Forward content= on the AsyncHTTPHandler retry path so a retried batch re-sends the identical body
Replaces #39133, which was cut from the retired staging branch and conflicts with main
Co-authored-by: Damien Smrt <dsmrt@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): drop test docstrings and replace monkeypatch with a client-injecting handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): add live e2e for non-native metadata reaching LangSmith
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): scope the e2e docstring to the values the test injects
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(http_handler): close injected retry clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): deselect the LangSmith live e2e on the stage-mirror stack
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Damien Smrt <dsmrt@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add python-compat crate for Python data formats
Add litellm-python-compat, a PyO3-free crate that reproduces the Python
data formats LiteLLM persists, so Rust readers and writers can interoperate
with state written by the Python proxy:
- literal::literal_eval: a linear recursive-descent port of
ast.literal_eval (prefixes, escapes, implicit concatenation, numeric
underscores and radixes, single unary sign, real +/- complex with 3.14
mixed-mode rules, set(), Python-equality key dedup)
- repr::{repr, to_str}: byte-exact repr()/str(), with a printable table
generated from CPython's str.isprintable (Unicode 16.0.0)
- json::{dumps, from_json, to_json}: json.dumps defaults and the
json.loads mapping
- pickle::{loads, dumps}: plain-data pickles via serde-pickle's serde
interface, which keeps dict insertion order
- truthy::truthy: bool() for plain data
Tests replay fixtures generated by CPython 3.14 (values across every
format and pickle protocol 0-5, plus 154 literal_eval source texts).
Accepted divergences are pinned in a KNOWN table that fails once one
starts matching. A criterion bench covers each format and literal_eval
cost by nesting depth, guarding the linear parse: the py_literal grammar
doubled per nested bracket (105 ms at 16 nested dicts; 19 us at 128 now).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(rust): split python-compat modules and harden the pickle verifier
- Disable class resolution in scripts/verify_rust_pickles.py, and truncate
the export file once instead of removing and appending to it, so the
verifier cannot be pointed at a pre-created file whose rows execute code
through pickle.loads
- Move Error to error.rs and Value to value.rs, leaving lib.rs as the crate
overview, module list and MAX_DEPTH
- Move the generator and verifier to scripts/, beside the Unicode table
generator, leaving tests/ to the Rust tests
- Group the bench by measured surface, give every case a Throughput so
criterion reports bytes per second, and document baseline comparison
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat(proxy): add admin-only /debug/report sharing the bug report environment fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add verbose=true to /debug/report listing every config key with typed values
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): inject auth into /debug/report through Annotated to keep the B008 budget flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound the verbose config walk, drop nested-list recursion from the safe renderer, regenerate schema.d.ts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count pass-through and mcp header maps plus operator-named budget maps in verbose /debug/report, single-exit scalar renderers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts after dropping the verbose query from /debug/report
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust_bridge): keep the Messages route on Python until the Rust path is ready
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust_bridge): keep token counter and tokenizer routes on Python
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): write key deleted audit logs for cascade and alias key deletions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert persisted key deleted audit rows for cascade paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): route /audit and /v2/login to the control plane in split transport
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): e2e for minimax h3 auto duration and oversized size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): keep auto duration literal and make h3 tier lookup total
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): omit duration for h3 when seconds is auto
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): move the minimax h3 auto duration and oversized size repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): let a later usage event zero out stale cache counts (#40736)
The usage merger only replaced cache_creation_input_tokens and
cache_read_input_tokens with a positive value, so when Anthropic's
message_delta restated the cache block with a 0 write, the 58k write from
message_start survived next to the 58k read. prompt_tokens minus both cache
counts then went negative and the write was billed twice
A usage event that reports any prompt-side count is now authoritative for both
cache fields, zeros included. An event with no prompt-side counts, such as an
output-only message_delta, still leaves the earlier values alone
* test(streaming): cover an input-only message_delta and the derived uncached input in the cache count merger test
---------
Co-authored-by: muhammadwaqar12 <m_waqar@live.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: shrey kharbanda <shreshth@berri.ai>
* fix image cost: honor deployment pricing
* fix types: coerce fal deployment price, drop private import
* fix: forward every custom pricing field through get_litellm_params
* test: assert optional keys are absent, not merely None, in get_litellm_params
* test: type the deployment image pricing test parameters
* fix: bill deployment per-image and per-pixel prices on unlisted image models
* test: type the remaining image cost test parameters
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): explain fallback outcome in plain words in the raised error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep the fallback outcome trailer on the outermost hop only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): explain failed context-window and content-policy fallbacks too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): name both Router and proxy fallback config in the no-fallback hint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add gpt-6-sol and gpt-6-luna to the model cost map with pricing from the OpenAI pricing page and reasoning effort levels none through max. Extend the long-context priority pricing and reasoning effort capability tests to cover both models
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(realtime): surface an upstream handshake refusal as an error event and policy close
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(realtime): tidy the handshake refusal e2e
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(realtime): keep upstream exception text out of the Azure client error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(realtime): map handshake refusal close codes with a lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): time-windowed team reservation of deployments via model_info.access_windows
Deployments can declare model_info.access_windows, a list of local wall-clock windows (IANA timezone, cross-midnight allowed) that reserve the deployment for the listed team_ids. While a window is active the router drops the deployment for every other request, including no-team and admin requests, on every candidate path (model name, model id, specific_deployment, early-resolve, wildcard, litellm_params.model lookup, fallbacks). If every candidate is reserved the request fails with a 400 naming the window end instead of falling back. Outside a window routing is unchanged and reserved deployments stay visible in /model/info and /v1/models. Malformed windows (bad time, unknown timezone, empty team_ids, start equal to end, offset-aware times) fail proxy startup with a clear error since the proxy router runs with ignore_invalid_deployments=True
Resolves LIT-8308
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover _filter_reserved_deployments directly for coverage gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): keep reservation filtering immutable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): drop strategy markers before reservation filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): give every ui settings endpoint test a fresh settings store
Under xdist the module-scoped reload in tests/test_litellm/conftest.py is skipped, so a health endpoint test's lifespan startup leaves proxy_server.general_settings bound to the ProxyConfig singleton's SettingsStore for the rest of the worker. Every /get or /update ui_settings call in test_proxy_setting_endpoints.py then writes into that shared store, and since get_ui_settings merges proxy_config.settings over the stored row, a leaked team_admin_editable_team_fields: [] from the PTU round-trip test beat the mocked DB row in test_get_reports_the_stored_list_and_advertises_supported_fields. An autouse fixture now binds both proxy_config.settings and general_settings to a fresh store for each test in the file, and a regression pair checks a runtime-flag write lands in the store the endpoint reads and does not survive into the next test.
* test(proxy): assert ui settings isolation through the endpoints
Replace the fixture-structure assertions with a parametrized GET/PATCH round trip on /get/ui_settings and /update/ui_settings that fails whenever one case's PATCH leaks into the next case's GET.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): surface the owner's user budget on keys without their own budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): apply the owner's budget hint to team keys when apply_user_budget_to_team_keys is on
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): read only the apply_user_budget_to_team_keys flag from general_settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the general_settings cast as cast-ok
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): load the owner's budget for keys opened outside the current page
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: jesus <jesus@berri.ai>
* fix(jwt): say x-litellm-team-id matched no team id or alias in the 403
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): tell the caller when x-litellm-team-id names an alias shared by several teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): deny a shared x-litellm-team-id alias exactly like an unknown value
A distinct 403 for an alias several teams share was raised before the
allowed-teams check, so any JWT could probe which aliases exist. The
alias lookup now treats the duplicate as a miss, and both denials say
the value does not resolve to a team id or a unique team alias, which
is true for unknown, unauthorized and duplicate values alike
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
add_provider_specific_params_to_optional_params built the extra_body dropped-key
set as frozenset(additional_drop_params), so one non-string entry raised
TypeError: unhashable type: 'list' and every openai-compatible call carrying one
failed with a 500 before it reached the transport. The set now takes only the
string entries, the element type every other signature in this chain already
declares as list[str].
A list-form entry still drops nothing: is_nested_path() tests a string, so
delete_nested_value() has never applied one on any provider. This removes the
crash only, so a working string path such as "tools[*].function.x" sitting
beside a malformed list entry is applied instead of taking the request down.
* fix(otel v2): map rerank and search output and the OCR, image edit and search input onto the Langfuse generation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): summarize OCR data URIs by media type and size and log an empty document URL as empty
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): keep URL-less search results, name OCR file streams and skip non-str query parts when logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(otel v2): drop the unused typing imports and the decorative section divider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): bill batch prompts above 272K at OpenAI's long-context batch tier
* fix(cost): mirror batch long-context keys on custom pricing params
Register the two *_above_272k_tokens_batches keys on CustomPricingLiteLLMParams so a per-deployment override stays out of the shared backend key, add them to the inline model-info schema and alias-count tests, and build LiteLLM_Params and GenericLiteLLMParams through model_validate at the two dict-splat call sites so basedpyright's reportArgumentType budget ratchets down instead of blocking the new fields.
* fix(cost): add the gpt-5.5-pro batch long-context tier and ignore malformed batch tier keys
* fix(cost): bill cached batch tokens at OpenAI's cached batch rate
Adds cache_read_input_token_cost_batches and
cache_read_input_token_cost_above_272k_tokens_batches for the tiered
OpenAI entries at half the standard cached rate, bills cached batch
tokens at that rate per output line, and parses string-valued batch
rates in deployment-level model_info.
* fix(cost): bill batch cache writes at the batch cache-write rate and carry published batch rates for one-sided deployments
OpenAI's Batch table prices cache writes for gpt-6-astra, gpt-5.6, gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna at half the standard cache-write rate, so the cost map gains cache_creation_input_token_cost_batches and its above_272k tier for those entries and batch cost pulls written tokens out of the input bucket at that rate; models without the key keep billing writes at the batch input rate.
A deployment declaring only one side of its batch pricing now carries every published batch rate of the other side (tier, cached, cache write), its own keys win, and a lone tier, cached or cache-write batch key counts as declared pricing instead of being ignored.
* fix(cost): select the batch long-context tier from any batch tier key
A deployment that declares its own flat standard input rate keeps every
published batch rate of the output direction, including the 272K output
tier, but the tier was only ever selected when an input tier key was also
present. Detect the crossed tier from any of the four batch tier keys so
the carried output, cache-read, and cache-write tiers bill at their tier
rate above 272K tokens.
* chore(proxy): keep the OpenAPI snapshot as CI generates it
* fix(cost): pick each batch price component's tier from its own keys
The batch rate picker crossed one threshold for every component, so a
deployment declaring only an output tier also moved its input, cached, and
cache-write rates to that cutoff. Each component now crosses its own
*_above_<N>k_tokens_batches keys and falls back to its flat key.
The JSON schema is regenerated with the generator as it is on main:
cost-map-guard renders the PR's cost map with the base branch's generator,
so the descriptions for the new batch cache keys move to a follow-up.
* chore(proxy): restore the lazy OpenAPI snapshot to what CI's Python 3.12 generates
The merge commit carried a snapshot regenerated on a Python 3.14 venv, which dedents
docstrings at compile time, so one description line differed from the file CI regenerates
on 3.12 and the schema.d.ts sync check went red. The snapshot is byte-identical to main again
* fix(bedrock/claude_platform): strip body params the AWS endpoint rejects
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock/claude_platform): assert exact bodies through a strict fake gateway for every workspace alias and auth mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): block Claude Platform workspace id aliases in request bodies without admin opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Venkata Donavalli <vdonavalli@live.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shrey kharbanda <shreshth@berri.ai>
Adds the anthropic cost map entry for claude-opus-5-5 at $4/$20 per MTok
with $5 per MTok 5m cache writes, $8 per MTok 1h cache writes, $0.20 per
MTok cache reads (0.05x base), and fast mode at 2x. The entry sets
thinking_always_on (Opus 5.5 cannot turn thinking off) and
supports_forced_tool_use false (tool_choice required/named 400s, same as
Fable 5.1), mirrors that model by omitting thinking cache preservation,
and registers the model in the setup wizard provider list
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): make bedrock collector and secret scan timing tests deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): count interpreter calls instead of wall clock in the secret scan scaling test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): profile the secret scan with cProfile, restore the outer profiler and tighten the scaling bound
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(arize): per-team success and error sampling rates for the Arize AX callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(arize): fail open on invalid sampling rates and type the new sampling code
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(arize): type the sampling test helpers and parametrized fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
