* fix(auto_router): bill the routing embedding to the caller's key and team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(auto_router): validate the forwarded caller metadata with a pydantic model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The cancel test accumulated chunk_count and reassigned response_id on every
iteration, and the lock watcher rebound its query result on every poll. Both are
the mutable-local pattern the repo avoids.
The stream now drains through a generator that stops at the first chunk carrying
a response id, so the caller binds streamed_ids once and reads the id off the
tail. Empty stream, no-id stream and first-chunk-id all behave exactly as the
loop did. The watcher inlines its poll result.
The block accepts output_cost_per_reasoning_token and cache_creation_input_token_cost. The generic
cost path and the DashScope calculator swap them in while a window is open, and unset keys keep the
standard rate. One shared TokenRates value replaces the DashScope-local copy, and
apply_off_peak_pricing takes and returns it.
Without the auto_router feature in the signed enterprise license a proxy may hold
one complexity router with classifier_type heuristic_v2 across config.yaml and the
DB; with it the limit is lifted. The ceiling is derived once from LicenseCheck and
handed to the Router, which refuses the extra router at registration. config.yaml
over the limit refuses to start, and /model/new, /model/update and
PATCH /model/{id}/update refuse the write with a 403 before touching the DB.
Expiry follows the existing max_users/max_teams pattern: judged when the
license is verified, not on every call, and a verify that rejects the license
(expired or unreadable) leaves no signed payload behind. The rollback after a
failed upsert re-admits state that was already serving, so it is exempt from the
ceiling: an edit that fails, including one refused by a ceiling that has since
tightened, leaves the router serving its previous configuration.
A write that leaves a row on heuristic_v2 under a limited license runs in one
transaction that takes a Postgres advisory lock before counting the DB rows plus
this proxy's config.yaml routers, so concurrent writes on any pod cannot both
claim the sole slot and no surplus row is ever persisted.
Only the row insert runs under that lock: the team model bookkeeping, which
needs a second pool connection, runs after the transaction has committed.
PATCH /model/{id}/update follows the same order as create: the row is written
through the slot first and the team's model list is updated only afterwards, so
a refused write leaves the team as it was.
The slot transaction bypasses the repository's publish-on-write, so it
publishes the config change once after commit, as delete_team_models does.
425e3069b9 added Router.get_configured_mode but only exercised it
through create_model_info_response, which the router coverage gate does
not count. The code-quality workflow has been failing on staging and on
every open PR since.
* fix(vector_stores): only list vector stores the caller was granted
/vector_store/list returned every managed vector store with no team_id to any
key, and let a dashboard session see stores created from the dashboard because
every session shares the litellm-dashboard team id. Non-admin listings now show
a store only when the key or one of the caller's real teams is allowlisted for
it via object_permission.vector_stores, or the team owns it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vector_stores): keep a dashboard session key's own grants when the user has no teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CI unit shards load the price map from main at import, so a model that only
exists on the branch never reaches open_ai_chat_completion_models and
cost_per_token cannot infer its provider. Refresh the sets after swapping in
the local map so the tier pricing cases resolve gpt-6-astra before merge
OpenAI documents low, medium, high, xhigh, and max for gpt-6-astra, with no none level, so the entry stops advertising none and starts advertising max.
Adds the OpenAI gpt-6-astra entry to both price files with standard, flex, priority (fast mode), batch, and above-272K long-context rates, and regression tests covering each tier and the batch rates.
A five-token response can complete before the cancel lands, which put the
test back on the "Cannot cancel a completed response" path it used to
swallow. Ask for a long generation so the cancel always beats completion,
and assert the cancelled status unconditionally
test_cancel_streaming_response drained the whole background stream before
cancelling, so an OpenAI keepalive stall held the e2e_openai_endpoints job for
301s and failed it on a generic APIError, and on a healthy day it cancelled an
already completed response and swallowed the 400 without verifying a cancel.
Cancel at the first event carrying a response id, bound admission to 90s, skip
naming the stall when only keepalives arrived, and assert status == cancelled
test_cancel_streaming_response drained the whole stream and only then called
cancel, by which point the response had finished and the cancel was expected to
fail. The test passed only because the surrounding except matched the literal
string "Cannot cancel a completed response", which is upstream OpenAI's wording,
not ours: it appears nowhere in this repo. Any change to that text, a different
status code, or the background job still running flipped the result. The test
also never exercised cancellation, and its only positive assertion was
hasattr(cancel_response, "id").
Break out of the stream at the first chunk carrying a response id and cancel
there, with a prompt long enough that the response cannot have completed in the
meantime. Both proxy cancel paths, the polling handler and the provider
passthrough, settle on status "cancelled", so assert that and the returned id
rather than a provider error string.
The three race tests claimed to pin the interleaving deterministically, but
lock_acquired.set() ran as the first statement of the task coroutine, before the
awaited endpoint call, so it only signalled that the task had started. The real
synchronisation was `await asyncio.sleep(0.2)` followed by
`assert not task.done()`, which is a timing assumption on a box running four
xdist workers against one Postgres.
Take the blocking connection's advisory lock key straight out of pg_locks, then
poll from the unblocked watcher connection until a non-granted lock on that same
key appears. That is the condition the sleep was standing in for, and it holds
however slow the machine is. If the endpoint returns without ever queueing, the
helper now fails with the endpoint's own exception chained on instead of a bare
assert.
The two tests reported failing in CI used sleep(0.2); the third used sleep(0.3)
and was not reported, which is consistent with the margin being the cause.
* fix: stop a cleared Organization field from failing key creation
Clearing the Organization combobox in the Create Key modal left organization_id set to an empty string, so /key/generate looked up an organization named "" and failed with "Organization doesn't exist in db. Organization=".
OrganizationDropdown now emits null on clear, and GenerateKeyRequest normalizes an empty organization_id or project_id to None the same way it already does for team_id.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop customer-specific docstring from key request normalization test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
