* refactor(router): move litellm/router.py to litellm/router_python_core.py
A pure rename with no content change, so git and review tools track the router body as a move.
The next commit turns it into PythonCore and adds the Router facade back at litellm/router.py
* refactor(router): extract PythonCore behind RouterCore and pass the router its config
The routing logic moves unchanged from litellm/router.py into PythonCore in
litellm/router_python_core.py. Router becomes a facade that builds a RouterConfig, picks a
RouterCore with make_core and delegates every member to it. The core no longer reads or writes
litellm.* directly: process settings come from RouterDynamicConfig, and inference calls, sleeps,
callback lists and the global cache go through PythonHost. A CI check bans litellm.* in the router
modules, and the proxy's construction sites build the config outside the router
Settings are static: the constructor arguments and changeset values are copied when the router is
built or the change applied, and a write to a setting attribute becomes apply(ChangeSet(assign=...)).
A Router subclass's core class also subclasses the facade, so an override's super(Sub, self) works,
and an instance-level patch of a core method is kept on the facade so nested patches unwind.
tests/unit/test_router/core tests the core in isolation: ScriptedHost serves every host operation
(scripted inference answers, recorded sleeps, settings, callbacks, cache), and make_core runs each
test once per core in CORE_BUILDERS
Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality
sub-routers now take a RoutingContext instead of the Router, and read the router only through
registry(), settings(), health() and the context's forwarding methods. Router-free helpers move
from router_utils to litellm_core_utils, outside callers of router state switch to entry points,
and the surface allowlist shrinks to the endpoint methods plus the entry points
Code outside the router now reaches it only through the entry points
(registry, settings, health, select, apply, observe, register_plugin, retire)
or the endpoint methods; the outside-member allowlist shrinks from 139 to 45.
litellm/router.py no longer imports litellm.proxy: it asks the installed
RouterHost (litellm/router_host.py) for vertex pass-through credentials,
member auto-router authorization and auto-router prompt compression, and the
proxy installs its host on import. Call sites where a model name can be
missing keep the old answer instead of reaching an entry point with None.
* refactor(types): replace Any with proven types in 57 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): hide input values in validation errors and drop secret-bearing seams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): revert the Phoenix prompt seam that rejects string content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): revert files that break mocked tests and Python 3.10 imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): revert pegasus response validation to main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): revert azure search, milvus and onyx response validation to main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep accepting numeric keys in /cache/delete
* fix(caching): answer 400 for a non-list /cache/delete keys and accept null Stability lists
Return pydantic errors without the caller's input on a bad /cache/delete body, let the Bedrock Stability
image edit schema take null finish_reasons and images like the old reads did, declare user_id and user_role
on the plugin session claim, wrap the retyped Vertex fields in ReadOnly, and drop unused Guardrails AI types
* refactor(types): leave provider reply parsers for a follow-up with live provider calls
Restores the Bedrock Stability, Bedrock KB, AgentCore, Replicate, Recraft, Aleph Alpha and Guardrails AI parsers to main, since no live provider call could prove the stricter schemas. Also types the cache delete route tests.
* refactor(types): keep /cache/delete on main's behavior
Restores the cache delete route and its test to main, so this types PR changes no response. Rejecting a non-list keys is left for its own change, since an object keys deletes its keys on main today.
* fix(types): keep text completion, batch ids and guardrail callback on main's code
* fix(types): keep policy suggestion and auto router parsing on main's code
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(tests): keep foreign asyncio task noise out of the budget limiter caplog asserts
The two gc.collect()-under-caplog tests in test_budget_limiter_hotpath.py asserted
on every captured record, so a LoggingWorker task another test had parked on the
session fixture loop got destroyed inside their window and asyncio's "Task was
destroyed but it is pending!" error failed them. They now assert only the limiter
logger's records and park a pending task on a closed loop in the window themselves,
so the noise is exercised on every run
The edenai and sail spend_capture fixtures no longer start the worker from the
session loop: the test's own call and its flush bind the worker to the test loop
* test(router_strategy): drop only asyncio records in the limiter caplog asserts and drain the response cache worker on the test loop
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Add rustdoc overviews for a new litellm-router-core crate and for
litellm-model-catalog, a module docstring on litellm/router.py, and a
shrink-only allowlist of the Router members and router_utils modules used
outside the router, checked in CI by router_surface_check.py.
Move the router-free helpers out of litellm/router_utils: batch_utils to
litellm_core_utils, and the auto-router naming contract, tuning baseline
and baseline request next to the auto-router plugin. No behaviour change.
* test(complexity_router): Codex compaction requests hold the user_turn decision
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity_router): treat Codex compaction traffic as user_turn continuation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity_router): validate Codex turn metadata with frozen models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(complexity_router): flatten body system text through _message_text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(complexity_router): drop unrelated reformatting from the mapped test file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity_router): skip pin escalation on Codex compaction turns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): stop concurrent deployment deletes from crashing deployment lookups
delete_deployment and upsert_deployment shrink model_list before they repair
the index maps, so a lookup on another thread in that window read a stale
index: IndexError, or the neighbouring deployment and its credentials. An add
racing a delete could corrupt the index for good.
One RLock now covers model_list and the three index maps. Delete, upsert, add
and the set_model_list reset change them under it, and every lookup that turns
an index into a deployment reads under it.
Fixes#45242
* test(router): release the held delete when the read reaches the index lock
The race tests released the held write after a fixed 0.2 s, so a reader the runner
stalled past it read the repaired list and passed on unfixed code. The read now runs
on its own thread and the write stays held until the read returns or reaches the
router's index lock; the 10 s bound only fails a broken synchronization.
The integration test gains the audit cells: chat, streaming, Responses and
Anthropic messages calls (sync and async, by model name and team public name)
during a held delete, the same for upsert, add and set_model_list, a request for
the deployment being deleted, a provider error, a double delete, and thread and
async bursts during upsert churn.
* test(router): cover the early return of a delete and the team filter of get_model_ids
delete_deployment now returns None from inside the index lock, and get_model_ids by
name reads its deployments through _deployments_named; pin both: deleting an unknown
id returns None and leaves the list alone, and exclude_team_models drops only the
team deployment.
* fix(router): read deployments by name under the index lock outside the Router
The response polling provider check, the complexity router's adaptive pool and the
savings baseline turned the name index into list positions themselves, so a concurrent
delete could still hand them a stale slot. They now read through Router.deployments_named,
which takes the deployment index lock.
* fix(router): type the locked deployment lookups as DeploymentTypedDict
deployments_named and _team_deployments_named returned tuple[Any, ...]. They now return
DeploymentTypedDict rows, which is what model_list holds. The adaptive pool setup in the
Router and the complexity router drop their branches for pydantic Deployment rows, which
these lookups never return, and should_include_deployment accepts the typed row.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(router): keep latency routing alive when latency samples are unusable
latency-based-routing ranks deployments by the average of the cached samples
and keeps everything within lowest_latency_buffer times the lowest. A negative
or NaN sample breaks that invariant, because the minimum no longer satisfies
its own bound. The candidate list then comes back empty and random.choice
raises IndexError, which turns every request to the group into a 500 until the
cache entry expires.
Samples come from completion_start_time - start_time, and that difference is
negative whenever the two timestamps invert. Production recorded samples
around -600s from successful calls.
Drop non-finite and negative samples before ranking, and fall back to every
candidate if the window is empty anyway, for example with a negative buffer.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* test(router): seed latency_normalized in the unusable TTFT fallback test
Main reads latency_normalized since #47727, so seeding latency left both deployments ranked at zero and the test picked at random.
Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): assert ranked latency after dropping unusable TTFT samples
Greptile on #48216: the new tests passed with the sample filter removed because the empty-window fallback alone saved them. They now assert _latency_per_deployment so unfiltered negative or NaN samples fail deterministically. Also adds Final to ttft_samples and drops a comment that restated the code
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): drop Final on a loop-local in latency ranking
basedpyright rejects Final on a variable assigned inside a loop, and the type_check_gate counts it as a new reportGeneralTypeIssues error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): type the request kwargs in the unusable TTFT tests
Greptile on #48216: bare dict annotations. The tests now hold the metadata dict in a typed local and assert on it directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): unregister strategy selectors from global callbacks on discard
Router.discard only removed its own bound methods from litellm's callback
lists, so the strategy selectors it registered (least-busy, usage-based,
latency-based, cost-based and routing group selectors) stayed behind and
kept receiving events after the router was gone. Reuse the identity based
unregister that update_settings already uses so a discarded router takes
only its own selectors with it
* fix(router): purge discarded selectors from every event callback list
function_setup copies litellm.callbacks into the success and failure
lists on every request, so a router that served traffic still leaked its
selectors there after discard. Route the cleanup through
remove_callback_from_all_lists, which now also covers input_callback, and
fold the selector gathering into discard so the router coverage check
sees no untested helper
* fix(router): keep live routers, in-flight hooks and user_config limits intact on discard
discard() now rebuilds every global callback list without the router's selectors, optional pre-call checks and own usage callbacks instead of removing them in place, so a callback loop already running never skips a hook. It then re-registers what the remaining live routers need, so a same-strategy survivor keeps its deduped handler. The proxy's per-request user_config router is discarded only after its call returns.
* fix(router): bind user_config handlers to the request and tighten discard
When a user_config request carries a logging object, its router is
discarded before the request runs and its handlers ride on that
logging object, so v1 rpm and v2 tpm limits count every request and
concurrent requests count once. Without one, the router is discarded
after the request as before. The router's pre-call loops also read
the request-scoped handlers.
discard() now refills only the handler keys it vacated, guards the
callback lists and the live-router set with a lock, covers routers
built without a model_list, and drops its Slack alerting, adaptive
router hook and budget limiter sync task
* fix(proxy): reuse one user_config router per config and bind its handlers to each request
Every user_config request now goes through a cached router built once per
config. Its strategy handlers leave the global callback lists and are bound
to the request's logging object instead, so limits count each request once
and nothing is left behind after the request.
* fix(router): keep the proxy budget limiter on user_config requests
A user_config router now swaps in only its strategy selectors, so the proxy's
RouterBudgetLimiting stays on the request and keeps counting its spend. Usage
logged after a strategy is retired, such as a request still in flight when its
user_config router is evicted, now schedules one more push to Redis.
* fix(proxy): init the retire fields on the key model budget limiter
The key model budget limiter builds its own state instead of calling the base
init, so with the proxy's Redis attached every spend increment raised on the
missing retire flag and the key's model budget never counted spend.
* fix(router): bind the deployment callbacks to user_config requests
A cached user_config router serves requests through each request's logging
object but bound only its selectors and policies there, so a failing
deployment was never cooled down and its usage counters missed requests.
* fix(router): retry an evicted router's budget writes and keep the adaptive header on user_config requests
A retired budget limiter now keeps retrying its queued Redis increments every
sync interval until Redis accepts them, instead of giving up after one failed
write.
The x-litellm-adaptive-router-model header now comes from one stateless hook in
litellm.callbacks that holds no router, so user_config adaptive routers, whose
per-router hooks are request-scoped, still answer with the header.
* fix(router): lock request callback setup against router cleanup and drain in-flight budget writes on retire
function_setup now edits the global callback lists under the same lock that
router cleanup uses, so a router dropping its callbacks on another thread can
no longer shift the indexes function_setup pops or put a dropped handler back.
A budget limiter retired while a Redis write is still in flight now keeps
retrying that batch if the write fails, instead of leaving the spend queued
with no task left to send it.
* fix(utils): set up loggers after releasing the callback lists lock
function_setup held the callback lists lock while set_callbacks ran its one-time
logger setup, which can block on a Slack package install, so other threads and
Router.discard waited on it. The lock now covers only the list reads and writes.
* test(router): annotate the retire test's locals as Final
* fix(logging): skip replaced global callbacks on the input path too
replace_global_callback already kept a same-key global logger out of
success and failure dispatch. pre_call and post_call still sent it the
input events, so a logger with an input hook could count a request start
it never saw finish. Both now share one filter with the success path
* test(logging): type the input-path regression test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): price cost-based routing through model_info.base_model
* fix(router): keep cost-based routing pricing helpers fully typed
* fix(router): drop typing.cast from cost-based routing pricing helpers
* fix(router): read the cost map through a typed accessor in cost-based routing
* fix(router): avoid a Final rebind inside the cost-based routing loop
* refactor(router): share deployment_pricing_models across spend, listing and cost-based routing
* refactor(router): keep the /v1/models listing key inline, share pricing models for spend and routing
* refactor(router): inline the deployment pricing-model wrapper in cost-based routing
* refactor(router): read deployment pricing models through one shared helper
deployment_pricing_models(deployment) now owns the model_info.base_model / litellm_params.base_model /
litellm_params.model unpacking, used by cost-based routing and the /v1/models listing. pricing_models(base_model, model)
stays for the failure-row spend estimate, which already holds the two names
* refactor(router): share the str-keyed mapping validator with cost-based routing
* ci: re-run after flaky test_aiohttp_transport read timeout in llms-providers
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): map baggage values to budget tags
* fix(proxy): preserve mapped tags in the parsed request cache
* fix(proxy): preserve mapper accounting and tag policy
* fix(proxy): preserve baggage tags across native transports
* fix(proxy): keep mapper rejection and passthrough logging safe
* fix(proxy): scope baggage tag mappings to billed inference and keep them out of tag routing
Mapped tags now apply only to billed LLM API routes, skipping token counting,
model listing, MCP and A2A. They classify spend and budgets but never select
deployments unless a key, team or client set the same tag. WebSocket inference
returns 403 only while a required rule exists. A tag created with /tag/new resets
its spend counter so its budget counts from creation. The General settings editor
keeps a new mapping unsaved until every required field is filled in.
* fix(proxy): keep client-tag rejection identical with baggage, map A2A calls, skip request previews
* fix(proxy): keep mapped tags out of requester metadata, answer 503 on tag lookup errors, map custom pass-through routes, check the key before the WebSocket 403
* fix(proxy): keep caller-sent tags routable, keep mapped tags out of policy and guardrail matching, honor allow_requests_on_db_unavailable, reject control characters in baggage values
* test(proxy): cover baggage tag mappings end to end
Adds integration cells for request_tag_mappings across chat, messages,
responses, A2A and custom pass-through spend, required and budgeted
rules, the WebSocket refusal, unbilled routes, reject_clientside_metadata_tags,
tag routing and policy isolation, a dropped tag read with and without
allow_requests_on_db_unavailable, and the /tag/new spend counter reset
* fix(proxy): keep base tag merges for requests that map nothing, accept JSON-string metadata tags, count litellm_metadata caller tags
Requests with no mapped baggage now reuse the cached body exactly as before, so header and key tag merges still normalize metadata for routing. The client body copy is made only when a tag maps, and the base merges are mirrored onto it. A request that maps nothing no longer gets a 400 for non-object metadata. caller_tags now counts tags sent in both metadata and litellm_metadata. Optional mappings no longer reject JSON-string metadata tags under reject_clientside_metadata_tags.
* fix(router): keep baggage-mapped tags from selecting tag-scoped pre-routing strategies
Auto, complexity, adaptive and quality router selection, the consumed-tags stamp and compression policy lookup now read request tags without baggage-mapped tags, unless the caller sent the tag or the key, team or project carries it, matching deployment tag filtering.
* feat(proxy): parse baggage headers into request tag mappings
* fix(proxy): match plus-sign baggage keys and keep deployment tags in guardrail modes
* fix(proxy): keep baggage-mapped tags behind key, team and deployment tags and serve non-object or invalid DB mapping settings like unset
Mapped tags are added after the key, team, deployment and caller tags, so a tag-mode guardrail picks the same mode with or without baggage. A non-object metadata field maps like empty metadata, and an invalid request_tag_mappings value loaded from the DB is ignored with an ERROR log instead of failing every request
* fix(router): merge deployment tags in linear time and keep malformed client tags as sent under mapped baggage
Merge serving deployment tags into the request tags in one pass, so a large client tag list no longer blocks the event loop. Leave a non-list metadata.tags untouched when baggage maps a tag, so the request gets the same answer as without baggage. Add the C17 integration cell for the team tag's guardrail mode under a mapped deployment tag.
* test(spend): cover string metadata.tags under mapped baggage
* fix(proxy): check mapped tag budgets when metadata.tags is a non-list or holds non-strings
Mapped baggage tags are applied again after the header and key tag merges, so
a header or key tag that replaces a string metadata.tags no longer hides the
mapped tag from the tag budget check. An empty or false metadata.tags is read
like a missing one, and a tags list with non-string items is kept as sent with
the mapped tags added after it, instead of failing auth with a 401
* fix(router): scope tag routing to prefixed tags once tag_routing_prefix is set
With router_settings.tag_routing_prefix configured, an unprefixed caller tag
(for example a spend tag such as user_id:234) no longer drives tag routing, so
it can no longer refuse a model group whose deployments are untagged.
Key/team/project tags (metadata.inherited_tags) keep applying with or without
the prefix. Rows of batches the proxy runs itself now carry those tags as
inherited_tags too, so they keep routing on them once the prefix is set.
Takes over #39903.
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
* test(integration): pin that proxy-run batch rows keep key/team tag protection
* refactor(router): filter routing tags from the raw request tags; deterministic tests and key-tag cells
* test(integration): audit cells for tag_routing_prefix: project tags, forged inherited_tags, odd tag values, outage, fallback, user-agent routing, concurrent burst
* test(integration): proxy-run batch rows keep key and project tags too
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
* fix(secret_managers): resolve oidc/file and oidc/env refs in admin-configured api_key per request
An api_key set to oidc/file/<path> or oidc/env/<VAR> in a credential_list entry or a deployment's
litellm_params was sent upstream as literal text, so every call answered 401. The SDK client wrapper
now reads the referenced file or env var on each call and never writes the token back, so a rotated
token is used on the next call. The existing server-owned federation gates treat an oidc/ api_key as
server-owned by value, so a request body, a non-admin credential or model write, user_config, fallback
targets and routing tier params can never carry or name one. Other oidc/ kinds answer 400
* fix(secret_managers): strip only the leading oidc/ from a secret reference
A token file under a directory named oidc resolved to the wrong path, because every oidc/ in the reference was removed
* fix(auth): refuse oidc/ api_key refs in advisor tools and scope the route gates to them
The advisor tool's own api_key now answers a 400 when it holds an oidc/
reference, since its sub-call runs through the client wrapper that resolves
one. Stored fallback targets, auto-router tiers and user_config refuse only an
oidc/ api_key, inline or through the credential they name, so their workload
identity federation handling stays as it was. Provider model discovery for a
wildcard deployment resolves the reference before calling the provider.
* fix(auth): allow admin oidc api_key on deployment writes, resolve refs on discovery, batch probe and realtime health, refuse client api_base on oidc deployments
* fix(utils): resolve oidc api_key refs only from admin-configured sources
Mark an oidc/ api_key read from a proxy-level deployment or a stored
credential, and resolve only that marked value. Team deployments, prompt
templates, nested configs and a direct SDK api_key send the text as
written. An unsupported reference kind raises a 401 so the router fails
over to the group's next deployment.
* fix(utils): send a credential's oidc token only to its own api_base host
A stored credential's oidc/ api_key now resolves only for the scheme, host and port of the
credential's own api_base or base_url, so a team model or vector store that names the
credential with another api_base gets a 401 and that host gets nothing. Saving a team model
with an inline oidc/ api_key answers 400, since team rows never resolve one.
* fix(proxy): send an admin oidc api_key only to the hosts configured beside it
Every admin oidc/ reference now carries the origins of the host params set beside it, checked for api_base, base_url and the Bedrock, S3, SageMaker and deployment_url params. Token counting reads the reference only when it calls the provider, answering 401 when it cannot, and pass-through 401s redact internal paths.
* fix(proxy): redact the token path from token count auth errors
* test(security): cover token count, Bedrock endpoint, api_base update and pass-through path redaction for oidc api_key refs
* fix(proxy): redact the token path from secret reference errors where they are raised
* fix(proxy): keep the secret reference's path out of the error a caller sees
The 401 for an unreadable oidc api_key or pass-through credential now names only
the reference kind and a fixed reason, so a token path under /run/secrets or a
custom credential dir never reaches the caller, while the server log keeps the
full reference for the operator
* fix(secret_managers): strip line breaks from the logged secret reference
A reference holding a newline can no longer forge a server log line
* fix(secret_managers): label an unknown oidc reference kind instead of echoing it
An api_key such as oidc/env_MY_VAR (a typo for oidc/env/MY_VAR) came back in
full in the caller's 401 and in the spend log, since the kind was whatever
text followed oidc/. A kind get_secret does not know now reads
oidc/<unknown kind>. The relative-path hint uses oidc/file//<absolute path>,
which the client-message redaction on chat routes leaves whole.
* test(health): run the real model auth check in the oidc test connection test
* test(secret_managers): read an oidc/file token stored under a directory named oidc
* fix(secret_managers): count a URL in the model name as an upstream host for oidc api_key references
A huggingface/* wildcard call could name its own endpoint URL as the model and
receive the resolved token. The origin check now treats a URL in the model as
a host the reference must allow. The integration contract also reads the
proxy's own database instead of the pytest worker's.
* fix(secret_managers): treat a default port as explicit when matching an oidc api_key origin
A credential on https://api.mistral.ai refused a call to https://api.mistral.ai:443,
though both reach the same server. The origin check now compares scheme, host
and effective port, as assert_same_origin does, and an unparseable api_base is
refused instead of raising. The advisor test sets the real proxy flag instead of
replacing our own helpers.
* test: type the oidc request-body helpers and drive the tier test through mock_response
* fix(secret_managers): refuse an oidc api_key reference whose call drops the configured host
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns.
* fix(responses): bill the usage a failed native WebSocket turn reports
* fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect
The v3 limiter stashed whichever litellm_call_id the request body carried as
the slot owner, so a client-supplied id on a pass-through body left the
proxy's own success and failure events unable to release the slot. The owner
is now the proxy's logging call id. A WebSocket connection whose turns log
under their own ids never settles the TPM, ITPM, or OTPM estimate the
connection reserved, so the disconnect release now refunds any reservation
no callback settled, once.
* fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect
A failed turn's reported usage now counts toward the key's TPM and the
project ITPM/OTPM scopes the same way a success does, and closing the
socket hands back only the estimate still unused in the window that
holds it, so a window that already rolled is left alone.
An id-less error event is booked against the newest pending turn, the
one the backend rejected at parse time, and a turn that nests its input
under response logs its user message. A connection's one budget
reservation goes to the first turn that settles it, every later turn
charging its cost directly.
* test(proxy): expect the settling call id on a pass-through budget reservation
* fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM
A native Responses WebSocket turn reaches the rate limiter's success
callback with its raw event list as the response and no
combined_usage_object, so the turn's tokens never reached the key's TPM
window or the project's input and output token limits. Read the usage
from the turn's events when the call type is a Responses WebSocket turn.
* fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas
The limiter's failure hook looked up the parallel slot by the request
body's litellm_call_id, while the slot belongs to the proxy's logging
call id, so a pass-through request a pre-call hook rejected held its
slot for an hour when its body carried an id. Both now resolve the id
the same way.
Each response.create frame already charges its estimate to project
ITPM/OTPM, so the turn's own usage no longer adds a second count.
The cost callback's failure hook read the body's litellm_call_id
through a strict model and raised on a non-string value, which dropped
the failure row. It now reads a string id or none.
* fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right
Free a rejected request's parallel slot through the request it runs in, and skip
native Responses WebSocket connections, whose disconnect frees the slot. Count
WebSocket turns toward the connection's tag TPM. In collector mode, claim the
shared budget reservation before the spend event is built, so only the first
turn settles it. Store each turn's own frame as its request body, and send an
id-less error to the oldest waiting turn unless an earlier turn is streaming.
* fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag
* fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls
* fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes
* fix(router): count native Responses WebSocket turns toward deployment limits and budgets
* fix(router): re-check deployment budget and RPM on each Responses WebSocket turn
* test(router): list the WebSocket turn admission method as covered through its partial
* fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn
* fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure
* fix(proxy): log the routed model group on a WebSocket turn's failure row
* fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn
* test(integration): cover per-turn admission on Responses WebSockets
Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the
project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP
controls for TPM counting and the failure row model_group
* fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails
A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success
callback such as the response cache no longer runs each CustomLogger's sync success hook on top of
the async one. That second run counted every turn's tokens twice toward deployment TPM.
A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket
closes with 1011, and the error becomes the socket's failure, on both the native and the managed
relay, instead of the socket dropping with no frame and no failure row.
A refused turn's failure row now carries the request's tags from the router's metadata, the same
way it already carries the model group.
* test(integration): cover a crashed WebSocket turn check and failure-row tags
A turn whose admission check raises gets a 500 server_error frame, the
socket closes with 1011 and its one failure row carries the error, on
both relays. The refused turn's failure row keeps the caller's tags, and
HTTP failure rows on chat, responses and messages keep them too.
* fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows
A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket
now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which
a client that sends its next response.create early would lose.
A refused turn's failure row now takes the router's tags only when every tag is a string, so the
fallback never copies a caller's unvalidated tags onto the row.
* test(integration): cover a turn still streaming when the next WebSocket turn's check crashes
A native turn still streaming when the next turn's admission check raises now has its own cell. It
must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for
one more turn, so a relay that stopped checking turns after a crash goes red.
* fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets
A turn after the first now runs the project quota check, then the proxy's
pre_call guardrails, then the deployment's turn admission, on both the
native and the managed relay, the order HTTP calls use. A turn a guardrail
blocks gets the guardrail's error even past the budget and no longer counts
toward the deployment's rpm or runs its pre-call checks.
* fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it
A turn the per-turn guardrail passes and the deployment then refuses now writes
its own failed spend row with that turn's guardrail information, on both relays,
the way a guardrail-blocked frame and an HTTP refusal already do
* fix(responses): bill a refused WebSocket turn on one row only
A refused turn whose guardrails recorded a cost is already billed on its own row,
so it no longer also becomes the connection's close-time failure row
* fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row
The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them
* fix(responses): bill a WebSocket's refused turns when the session limit closes it
* fix(responses): bill refused WebSocket turns when the session limit cancels the relay
* fix(responses): free the WebSocket slot at the session limit without waiting on the failure row
* fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot
* fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown
* fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after
* fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it
* fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close
* fix(proxy): always commit spend again after the shutdown wait for pending logging tasks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version
* fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider
* fix(utils): refresh provider model sets after register_model
* fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row
* test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version
* fix(utils): keep router deployment ids out of provider model sets
register_model now refreshes the provider sets, so the per-request and
router registrations keyed by a deployment id put those opaque ids into
wildcard model listings. Those callers now pass
add_to_provider_model_sets=False.
* fix(azure): carry responses_api_version through the chat to Responses bridge
completion() dropped responses_api_version from the litellm_params it hands
the Responses bridge, so azure/responses/<model> on a dated chat api_version
still built the 404ing /openai/responses URL. Also cover the auto-router
marker never forwarding the new param.
* fix(azure): stop listing a wrong set of reasoning efforts when none is rejected
The message said only low, medium and high are supported, which is wrong for
models that also take xhigh such as gpt-6.1-sol.
* test(azure): use the latest grok and gpt-5.4 rows in the new azure tests
* fix(proxy): list each wildcard-expanded model once
A provider set can hold a model both bare and provider-prefixed, which now
happens after any priced request registers its openai/<model> key, so the
openai/* listing showed that model twice.
* fix(azure): route Responses past dated GA api_versions by default
A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before
* chore(azure_ai): mark the grok chunk_parser dict as an override contract
* fix(azure): rewrite only the route's /openai segment when moving a call to v1
The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway
api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses.
Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too,
so the rewrite now only touches the last /openai path segment.
The Responses version resolver no longer duplicates the builder's default for an unset api_version.
Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite
now has one test for the dated GA default and one where only responses_api_version can make the call work
* test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none
Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params
* fix(utils): list ai21 chat and vertex code models under their provider wildcard
register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too.
* fix(main): keep caller-chosen model names out of provider wildcard listings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(router): stop queueing budget increments for Redis when no Redis is configured
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert Redis-less model budget limiter tracks spend locally without queueing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): report when each per-model budget window resets
model_max_budget_usage on /key/info, /user/info and /team/info now carries budget_reset_at per model, the open window's start plus its period, or null when no window is open. The window start is read in the same batched lookup as the spend counter
* fix(proxy): count a monthly budget window from its own start when reporting its reset
duration_in_seconds measures Nmo from now, so a window opened on January 15 and read on February 10 reported February 12, then null while the counter still refused requests until February 15. duration_in_seconds_from measures the month from the window start, and duration_in_seconds now shares that month arithmetic
* fix(proxy): measure months from timezone-aware datetimes and assert whole usage values
datetime.fromtimestamp without a tz tripped DTZ006. The month arithmetic now runs on an aware local datetime, which keeps the same wall-clock result. The duration tests move to parametrized functions, since subTest payloads carrying datetimes cannot cross the xdist worker boundary, and the new usage tests compare the whole entry
* fix(budgets): end a monthly budget window a calendar month after it opens
Each charge recounted the month from the charge date, so a monthly window
opened in a month of another length expired days before or after the reset
/key/info reports. The window is now measured from its start, by the same
helper the usage report uses. A period ending after year 9999 reports no
reset instead of failing the read.
* fix(budgets): count monthly budget windows in UTC for every router budget
Months were counted on the host's local calendar, so replicas with different TZ
settings disagreed on when a monthly window ends. Provider, deployment, tag and
per-model budgets now all measure a window from its start in UTC, and a test
covers the router budgets. A reset past year 9999 is documented as null.
* test(budgets): cover monthly tag budgets in the router window test
* test(budgets): add integration cells for budget_reset_at and monthly budget windows
* test(budgets): find a monthly window in another span on every calendar day
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl
A prefix marked cache_control ttl 1h lost its pinned deployment after the fixed 300 s, so the next request went to another deployment and paid a cold cache. The pin TTL now follows the longest ephemeral breakpoint in the cacheable prefix, tools included: 3600 s for a 1h breakpoint and 300 s otherwise.
* fix(router): give prompt cache pins their own tier and honour any declared ttl
Pins now live in a DualCache of their own instead of the router's 200-slot
in-memory tier, so an hour-long pin no longer evicts cooldowns and rate windows.
The pin lifetime follows any <n>s|m|h cache_control ttl, floored at 300 s, not
only the literal "1h".
* fix(router): hash the pin ttl off the event loop and flush prompt cache pins with the router cache
Compute the cache_control ttl scan through offload_token_count beside the
prefix hash, clear the dedicated pin cache in Router.flush_cache, and point
the order-fallback and priority-group pin tests at the pin cache they read
* fix(router): pin 1h prefixes only on 1h-capable models, cap the pin at 1h, and give pins 10k in-memory slots
* fix(router): match the 1h pin gate to what Bedrock forwards
Bedrock deployments now decide the 1h pin with is_claude_4_5_on_bedrock,
the same check that decides whether the 1h ttl reaches Bedrock, so an
application inference profile ARN keeps a 5 minute pin whatever its
base_model prices
* test(router): give the anthropic_messages pin fixture its provider
* fix(router): let each pin write set its own lifetime and require a positive 1h price
In a group mixing 1h and 5m deployments the in-memory pin kept the lifetime of its first write, so a 5m deployment could hold an hour-long pin. Each write now clears the in-memory entry first, matching Redis. The Bedrock gate uses supports_bedrock_prompt_cache_ttl after #46058, and a zero 1h price no longer counts as 1h support.
* test(router): integration cells for the one-hour prompt cache pin lifetime
Cover every surface the pin lifetime touches: chat and /v1/messages through
both SDKs, the responses bridge, response-cache hits, each provider family's
1h gate, malformed and duplicated ttl values, rewrites of one prefix, mixed
and cross-group routing, concurrency, every Redis attach path, the in-memory
slot pressure without Redis reads, a Redis outage and a killed worker
* fix(caching): bound the in-memory expiry heap and count prompt cache pins in the memory debug routes
* test(integration): cover prompt cache pins in the memory debug routes and the bounded expiration heap
* fix(caching): serialize in-memory cache writes so a heap rebuild cannot lose another thread's expiry entry
* fix(caching): keep the in-memory cache bounded when entries have no usable expiry
* test(caching): cover the response cache bound when cached entries have no usable ttl
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(ui): add decisions and classifier providers to the Add Model form
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): use literal upstream decisions paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: drop duplicate TYPESAFE provider fields entry superseded by main's TypeSafe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): drop stray blank line in provider_info_helpers test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the OSS decisions provider fields test
* fix(decisions): keep Laya and Bespoke env keys on their configured server
Route laya and bespoke Decisions deployments through the same oss_connection
and validate_oss_model rules the pass-through and the classifier router use,
so LAYA_API_KEY and BESPOKE_API_KEY only go to LAYA_API_BASE and BESPOKE_API_BASE,
an api_base with credentials, a query, or a non-HTTP scheme answers 400, and
an unknown checkpoint name answers 400 before any upstream call
* fix(decisions): send the Laya and Bespoke env key to a form-added model on the env server
A deployment added through Add Model always carries an API base, so the
env key only reached deployments with no base. The env key now goes to a
deployment whose base matches LAYA_API_BASE or BESPOKE_API_BASE, an empty
deployment key counts as no key, and the Bespoke placeholder is the name
Ollama serves.
* test(decisions): cover the env-base classifier case in the complexity router test
* fix(decisions): match the env server with or without /v1 and point Ollama users at bespoke/nimble
* fix(decisions): call /v1/systemone once from classifiers and pass-through on a /v1 base
* fix(decisions): strip /v1 from the API base path only, so a host named v1 still works
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e
* test: fix disk cache collection, ungate saturation check, drop RSS memory tests
* test: assert one delivered log per completion in Datadog and LangSmith checks
* test: drop SDK-direct e2e copies and align cost-map path filter cases
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* fix(databricks): call the ai_decide AI Function route for decisions and the auto-router
Databricks serves ai_decide at POST <host>/api/2.0/ai-functions/ai-decide, not
on a serving endpoint's /invocations route. The decisions provider and the
auto-router classifier now accept only the ai_decide model, post to that route
without a model field, unwrap the response envelope, and map a noul answer's
probability to the System One noul field.
* test(databricks): bound the megabyte model name test by scaling, not wall time
* test(databricks): assert the whole OpenAI-format decision and floor the liveliness bound
* test(decisions): move the Databricks OpenAI-format wire row to the ai_decide route
* fix(databricks): keep serving endpoint decisions beside the ai_decide route
* test(databricks): assert the whole classifier request body in the test routing case
* test(databricks): correct System One spend call type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): move complexity router request models next to their base config
Part of LIT-9423
* test(types): cover the old auto-router import path for the moved request models
Part of LIT-9423
* test: move routing, caching and callback tests out of local_testing
Moves the 37 routing, caching and callback files in tests/local_testing into mirrored tests/unit paths and tests/integration sdk and security groups per the migration manifest, and deletes the 8 rows it marks for deletion
* test: cover least-busy routing and tighten migrated router, cache and pass-through tests
Add router-level least-busy tests for the three legacy rows that had no
destination, move completion fallback and model alias tests to their mirror
files, move router batch tests next to the router, and strengthen tests
whose mutants survived (specific deployment, batch pair grouping, custom
cache key, router cache ttl). Add a unit test for configured pass-through
Authorization forwarding.
* test: tighten migrated callback, proxy config and cache ttl tests
Assert the full StandardLoggingPayload key set and the wire user on callback
tests, add an async embedding failure callback test, drive the prometheus
redis failure through a real RedisCache, use the real salt key in proxy
config tests, pin the redis ttl value and rename fallback test helpers
* test: fix semantic cache assertions and order-dependent router and callback tests
Count semantic cache embeddings on the route, drain async cache writes before
the second lookup, echo the requested model from the callback provider mock so
response_cost does not depend on model_cost entries other tests register, and
move the two-key authorization assertion to the auth failure test it belongs to
* test: cover model_list fan-out and the usage-based routing tpm limit lost in the move
* test: wait for the streamed cache entry in redis and drop deleted files from the hosted mock guard
* test: restore sync callback, audio redaction and langfuse export coverage, keep redis-compat filter on the moved file
* ci: run the remaining router-keyword legacy tests on one node
* test: refuse the redis write in-process instead of resolving an invalid host
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider
* fix(decisions): reject Databricks endpoint names carrying URL separators and keep the provider wiring under llms/
* fix(decisions): reject dot-segment Databricks endpoint names and add Databricks to the dashboard's OSS classifier editor
* fix(ui): name every OSS classifier provider in the classifier radio copy
* feat(decisions): register the Databricks OpenJev decider as an evaluation-mode cost-map entry
* fix(cost-map): declare zero cache rates on the databricks-openjev-qwen35-4b entry
* fix(proxy): reconcile every DB deployment when a registry miss lands on an auto router
A /model/new for an auto router lands on one replica. A request routed to
that auto router on a sibling replica missed it, the model read-through
loaded only the auto router's own row, and the pre-routing hook then picked
a tier the sibling had not loaded, so the caller got a 400 "no healthy
deployments for <tier>" until the next periodic reload. An auto router miss
now runs the full add_deployment reconcile, tiers included. Found by the
audit's two-worker rig on this PR's routing cells; CircleCI's one-worker
proxy never opens the window.
* test(integration): audit cells for Databricks decisions and the Databricks auto-router classifier
Wire cells for /v1/decisions with databricks/<endpoint> deployments, routing
cells for the complexity router's databricks classifier on all three
endpoints with the Postgres rows read back, management cells for the
classifier's spend writes and for the evaluation-mode health check of a
Databricks serving endpoint.
* test: move offline router, latency-routing, retry and exception tests from local_testing into tests/unit
Moves 29 offline nodes into the tests/unit files that mirror the code they exercise.
Hosted-provider calls are mocked at the HTTP boundary with respx, and sleeps that only
built a start/end gap are replaced by explicit timestamps. Legacy files left empty are deleted.
* ci: record the live prompt caching cases left in tests/local_testing as unrun
Moving the router prompt caching test to tests/unit leaves only live Anthropic and Vertex cases in tests/local_testing/test_anthropic_prompt_caching.py, which every CircleCI -k already deselects
* test: freeze the clock in the moved lowest-latency routing tests
The latency logger buckets usage by the current minute, so a run that crosses a minute boundary could grow the cache and fail the memory check. The moved tests now use fixed timestamps and a frozen clock in the logger module
* ci: describe the mixed prompt caching file accurately in the coverage allowlist
* test: move 76 live top-level tests to offline unit and integration coverage
* test: keep the mock-param opt-in guard valid once the top-level senders are gone
* test: address review on offline replacements
* docs(test): drop the deleted harness test file from the harness check command
* test: make the key rebind, team member delete and routes integration tests exercise the legacy paths
* test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing
* test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks
* test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds
* test: scope the admin health check to the test's own deployment
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: move the unit half of 126 mixed legacy files into tests/unit
* test: restore litellm globals that moved tests set
* test: finalize migration test cleanup
* test: restore original bodies of moved legacy tests
The move into tests/unit had rewritten 612 test bodies, and some of the rewrites dropped assertions. Each moved test now carries its original body from the legacy file, with only the imports, helpers, fake provider credentials and monkeypatched env it needs to run under tests/unit
test_timeout_streaming goes back to tests/local_testing because it needs the fake OpenAI endpoint server. The image payload fixture moves with its only user, and two tests that leaked global state (a registered model cost entry and queued logging tasks) are now isolated
* test: drop module imports shadowed by restored local imports
* test: assert on LiteLLM output in no-assertion moved tests and isolate leaks
Twenty no-assertion candidates get one assertion on the value LiteLLM returns, with the original lines unchanged. Four tests go back to their legacy files because they only check types or imports, write into the working directory, or cannot assert without a body change
Two moved tests leaked globals into later tests in the same worker, so monkeypatch fixtures now restore the retry-after header parser and the end user cost tracking flags
* test: drain queued logging tasks before the Phoenix span test
The moved Phoenix test counted spans from logging tasks that earlier tests had queued, so the drain fixture moves to tests/unit/conftest.py and both it and the Datadog batch test use it. test_factory_function goes back to its legacy file because its returned wrapper calls the real Assistants API and cannot be asserted on without a body change
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: delete unconditionally skipped legacy tests
* test: move whole-unit legacy test files into tests/unit
* test: keep moved legacy tests free of import-time global state
* test: keep the module-level invocation scan pointed at tests/local_testing
* ci: drop the agent_testing CircleCI job emptied by the move
* test: fix moved-test isolation and router coverage
* test: add Tinyfish search package marker
* test: isolate moved tests from logger state leaks
* test: isolate Helicone logging fixture state
* test: isolate Vertex pass-through credentials between moved tests
* test: cancel S3 periodic flush tasks started by moved tests
* ci: restore CircleCI assistant test selection after move
* test: prevent Datadog datetime import shadowing
* ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move
* test: deduplicate imports in rebased unit tests
* test: remove duplicate passthrough router patch import
* test: remove moved legacy source files after rebase
* test: align moved tests with rebased main
* test: carry main's legacy-file edits into moved destinations
* test: make the moved cost map fallback tests assert the fetch and the backup
The four fallback cases only checked the result was non-empty, so they still
passed with integrity validation disabled. They now inject a mock client, assert
one fetch happened, and assert the result is exactly the local backup with the
fallback reason recorded.
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* refactor(types): prove runtime types at harness, search, rag and client boundaries
Replace Any with adapter-validated types in the litellm.agent() harness, the
search provider transformations, RAG ingestion and query, the vector store
pre-call hook and registry, the galileo and opik logging integrations and the
proxy client CLI. Each boundary gets unit tests for well-formed and malformed
payloads.
* chore(typing): prove types at more provider boundaries and restore search transformations
Second pass over a2a, embedding, rerank, image, audio and small provider
modules. Search transformations go back to their previous form because
validating their response bodies would change the proxy status for
malformed upstream bodies from 400 to 500.
* refactor(types): prove types at logging, files, rerank, image, audio and management boundaries
Replace Any with validated or annotation-only types in 37 more files: logging
integrations, token counters, provider files/rerank/image generation/audio
transcription transformations, pass-through logging handlers and management
endpoints. No proxy HTTP status or error type changes.
* refactor(types): prove types at repository, spend, files and router boundaries
Replace Any with repository table accessors, validated mappings and
annotation-only types in 33 more files: Prisma repositories, the enterprise
batch and responses cost checkers, budget reservation, files endpoints,
management endpoints, the policy registry, the adaptive and complexity routers
and the secret managers. No proxy HTTP status or error type changes.
* test(types): run the aiohttp transformation test in-process and cover repository row conversion
The aiohttp chat transformation test no longer starts a server. It feeds the
transformation a response whose json() returns the body under test.
The proxy unit shards now exercise stored model rows whose params are JSON
strings and the object permission create and update paths.
