Budget reservation tokenized every request twice, once for the max-cost
estimate and once for the input-cost estimate, and again per pricing
candidate. Tokenizing is O(prompt) and ran inline, so admitting one large
request stalled every other request the worker was serving.
Count the input tokens once per request and reuse the counts for both
estimates. Prompts above 30K characters of input text are counted in a
worker thread so the event loop stays free. The size heuristic renders the
body rather than walking its values, so tool-schema property names count
toward the threshold, and it sizes every field the counter tokenizes,
tool_choice included.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The shared logging ThreadPoolExecutor uses an unbounded work queue, so
sync callbacks that fall behind request arrival pin every queued payload
in memory until the task restarts. Cap queued-plus-running work with a
semaphore, shed submissions past the cap, and warn at most once every 30
seconds naming the knob that raises it. No caller of the shared executor
reads the returned future, so shedding is safe.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(helm): compose DATABASE_URL_READ_REPLICA from a reader host secret key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(helm): cover reader host composition and readReplicaUrlKey precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): suppress unused reader host env when readReplicaUrlKey is set
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): emit reader host only when readReplicaUrl composition is active
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Agent registry CRUD (/v1/agents*) sat in agent_routes, which feeds
llm_api_routes, so DISABLE_LLM_API_ENDPOINTS returned "LLM API routes are
disabled for this instance." for every Admin UI Agents tab call. Split the
group the same way MCP is split: agent_inference_routes stays on the data
plane, agent_management_routes joins management_routes, and agent_routes
remains their union for keys configured with allowed_routes=["agent_routes"].
Non-admin callers reached agent CRUD through llm_api_routes before, so the
management paths also join self_managed_routes and the llm_api_routes virtual
key carve-out; the handlers already scope reads by role and 403 non-admin
writes.
Both new groups are tuples, so check_route_access now takes a Sequence and
matches wildcards through a generator instead of materializing an
intermediate list on every call.
A dropped connection anywhere in the budget reset tick used to abort the whole
phase, so every due key, user, team and budget tier stayed unreset until the
next tick ten minutes later. Route the job's DB calls through
call_with_db_reconnect_retry so a transport blip costs one reconnect instead.
Reads replay on any transport error, since re-running a SELECT has nothing to
double-apply. Writes are non-idempotent, a reset assigns spend = 0
unconditionally, so they narrow to DB_RETRY_SAFE_ERROR_TYPES: only a
ConnectError proves the statements never reached the database. A post-send
error like ReadError or ReadTimeout leaves the commit outcome unknown, and
replaying one that already landed would erase whatever was spent since, so
those keep the pre-existing behaviour of failing the tick.
Resolves LIT-5372
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(scim): fail group sync when a member add or user creation fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(scim): apply ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The hook swallowed errors into the console, so an admin flipping the switch without
STORE_MODEL_IN_DB saw nothing happen and got no reason why. Adds the missing hook tests.
The AUTH exchange it runs is the blocking client API, so on an async
connection send_command and read_response hand back coroutines nobody
awaits and the connect fails outright.
REDIS_URL-based async clients and every async connection pool dropped the
managed-identity credential the caller configured, so they connected
unauthenticated against an auth-enforcing Redis. The conversion from
redis_connect_func to a CredentialProvider now happens once, before any
branch, and covers the url, sentinel, cluster, and pool paths alike.
Also adds credential_provider to the cluster kwargs allowlist, which
silently filtered it out.
A streaming chat completion that ends early (client disconnect, or the proxy
cutting the stream at LITELLM_MAX_STREAMING_DURATION_SECONDS) wrote a spend log
row with spend 0.0, prompt_tokens 0 on the proxy-cut path, and no cache fields
in usage_object. The proxy restamps chunk.model in place to the client-facing
alias, so the partial response rebuilt from those chunks priced the unmapped
alias and came out at 0. The failure path also rebuilt usage without the
request messages, so prompt tokens counted to 0, and a cut stream never sees
the final usage event that normally zero-fills the cache fields.
Restamp the rebuilt partial response with the wrapper's real model before cost
calculation on both the disconnect and the failure paths, pass the request
messages when rebuilding usage on the failure path, and zero-fill missing
cache usage fields the way completed streams already do.
Peer workers previously kept their startup value for block_requests_for_models_without_pricing
until a restart, so a toggle from the UI only took effect on the worker that served the request.
Route fal.ai's openai/gpt-image-2 endpoints through a dedicated transformation that maps OpenAI image params (n, size, quality, output_format) into fal's schema, and register the model in the cost map.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): surface the paginated fallback on Cost Optimization
The page streamed its fallback silently: useDailyActivityRange dropped
the hook's progress and cancel fields and CacheLeakageCard only showed
a loading state while empty. Extract the Usage page's fetch banner into
a shared PaginationStatusAlerts component, render it above the tabs,
and note on the cache leakage tables when pages are still arriving.
* fix(ui): gate the cache leakage streaming note on isFetchingMore only
loading also covers a fresh aggregated request over the previous
range's rows, where pagination copy mislabels stale data. Drop the
redundant component comment flagged against the repo comment policy.
* fix(ptu): refuse an incomplete config.yaml reservation the way the endpoints do
POST /model/new answers 400 when PTU fields are set without a team_id, a
ptu_effective_from, or the count and rate together. config.yaml ran none of
those checks, so the same deployment loaded and served, billing per token
while accruing no flat cost, with nothing logged.
The rule moves into litellm_core_utils.ptu_pricing so both paths state it
once. Registration refuses such a deployment and names it, and the proxy's
ignore_invalid_deployments keeps that to the one entry. Only enforced while
PTU cost attribution is enabled, so a proxy that never opted in is unchanged.
* refactor(ptu): build the refusal message in the module that owns the rule
router.py raised a message it composed itself, which put proxy-facing
wording on the shared SDK surface. ptu_config_error now takes the
deployment name and returns the whole sentence; the endpoints still ask
without a name and their 400 bodies are unchanged.
JSON-RPC 2.0 types `id` as string, integer or null, but
LiteLLMSendMessageResponse annotated it as a bare required `str`. Pydantic v2
dropped v1's int-to-str coercion, so an upstream agent echoing an integer id was
rejected outright, and a null id, which section 5 requires for an error that
cannot be correlated to a request, was rejected too. Both surfaced as -32603 with
a pydantic ValidationError in the message: five distinct 500s on
/a2a/{agent_id}, across message/send and tasks/get.
Everything around the model already handled the full union: the endpoint reads
the id off the body as Any, its helpers are typed `str | int | None`, the error
builder takes `object`, and the streaming path passes the id through untouched.
The response model was the only narrowing left.
Backfilling an id the agent omitted keeps the caller's type too, since JSON-RPC
requires the response id to equal the request id and a caller that sent 7 cannot
correlate a response carrying "7".
`bool` is excluded from the integer half even though it subclasses `int`, so a
boolean id is stringified rather than relayed as 1 or 0, where it would collide
with a real integer id another in-flight request may be using.
* feat(ci): freeze the conftest save/restore inventory so it can only shrink
* fix(ci): resolve the named constant a conftest save loop iterates
* fix(ci): match the snapshot shape instead of a list of blessed dict names
* feat(ci): fail a branch that clears TQ violations without lowering the ceiling
A limit that only ever falls is not the same as one that falls when it can.
Clearing violations and leaving the ceiling above the new count let the same
violations return later under a limit nobody moved, so the gate now fails on
that and names `make lint-budget-update` as the fix. It needs both head below
base and head below limit, so headroom already in the base is never blamed on
the branch that happens to run next.
Drops the seeded-rule exemption from the ratchet along with it. Its stated
reason was that the base tree predates a rule introduced on this branch, but
base counts are measured with the current checker, so such a rule is counted at
the base too and its grandfathered total was never at risk of reading as fixed.
Removing the exemption is what lets a newly seeded rule ratchet like the six
that came before it.
The base scan is skipped when the branch touches neither the test tree nor the
checker, since neither count can have moved.
* feat(ci): assert .github/workflows holds only workflows, correctly named
* style(tests): annotate the hygiene test module's names with Final
* fix(ci): report a .yaml workflow as a naming finding, not a stray
GitHub reads .yml and .yaml alike, so WF001 telling you to move a valid
.yaml workflow to .github/scripts/ was wrong advice. WF001 now covers only
files that are not workflows at all, and the .yml spelling this directory
keeps moves to WF004, which says to rename rather than relocate.
WF001 also never looked into subdirectories, since GitHub does not read
them either; the message now says so. The directory is injected rather
than read off a module constant, so the cases are testable without
monkeypatching.
* test: merge three stranded twins into the files that shadow them
The second mirror's last four files each share a filename with a live test, so
the previous commit could not move them. Three of the four turn out to be plain
additions: their classes collide with nothing in the live file, so the tests are
extra coverage that has sat unrun rather than a competing version of anything.
Appending them takes the three files from 156 collected tests to 196, and all
196 pass. The 40 recovered are 13 OCI cases covering key normalization,
credential validation, complete-URL building and image-url transformation, 15
management-endpoint cases covering empty-value handling and the premium check,
and 12 DeepSeek thinking-parameter cases.
One assertion had to change. test_map_reasoning_effort_none_does_not_enable_thinking
asserted that reasoning_effort='none' leaves no thinking key, while the handler
maps it to {'type': 'disabled'} on purpose, documented in map_openai_params as
the OpenAI-style way to ask for thinking off. The test's stated intent holds,
since disabled does not enable anything, so it now asserts the disabled mapping
instead of the key's absence. Two imports moved to module scope for the
appended code, and no live test was touched.
test_discoverable_endpoints.py is the one left. Its twin grew from 1268 lines
to 9434, 25 of its assertions fail against today's code, and only 5 of its 19
tests have no counterpart, so deciding what survives that rewrite is a
judgement about the endpoints rather than a merge. The allowlist now holds
exactly that file and that reasoning.
* test(oci): stop the OCI suite reading credentials from the environment
validate_environment falls back to os.environ for every OCI credential and only
defaults the region when OCI_REGION is unset, so on a machine with OCI
configured the missing-credential test finds credentials it never passed and the
default-region test builds a URL for the ambient region. The suite then passes
or fails depending on who runs it.
A fixture drops the seven OCI variables for the four classes this branch added
and for TestOCIChatConfig, which had the same dependency before any of this and
fails the same way: with OCI_USER and friends exported, two of its cases fail on
origin/litellm_internal_staging today.
clean env: 83 passed
ambient OCI env: 83 passed
Same numbers either way, where the pre-existing file gave 68 passed / 2 failed
under the second.
Coverage is the single biggest time lever on the unit shards: the legacy
no-coverage workflow ran the same directory in about 5 minutes against 11 to 13
with coverage on. coverage.py's sys.monitoring backend (PEP 669) is the cheapest
core it ships, and it is not in use here today.
It has to be asked for explicitly. coverage 7.14 only defaults to sysmon from
Python 3.14 (`SYSMON_DEFAULT = CPYTHON and PYVERSION >= (3, 14)`) and these
shards pin 3.12, so without `COVERAGE_CORE` they get the slow tracer.
The audit left open whether sysmon survives turning on branch coverage. It does
not, at this Python. coverage gates branch measurement under sysmon on
`branch_right_left`, which needs newer than 3.14.0a5; on 3.12 it refuses and
falls back to the default core with a `no-sysmon` warning. Verified directly
against Python 3.12.13 with coverage 7.14.0:
$ COVERAGE_CORE=sysmon python -m coverage run --branch --source=. run.py
CoverageWarning: Can't use core=sysmon: sys.monitoring can't measure
branches in this version, using default core (no-sysmon)
So this speedup and `branch = true` are mutually exclusive until the runners
move to 3.14. Nothing here turns branch coverage on, so the two never collide
in this change, but whoever does turn it on is choosing to give this back.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
