The template rendered affinity and tolerations but never nodeSelector, so a
values file that pinned the chart to a node pool got the gateway and every
subchart placed correctly while the migration Job silently fell through to
whatever the cluster's default pool was.
That is worse than an outright failure. On EKS Auto Mode the default pool hands
out 3 GiB nodes and the migration container needs roughly 3.6 GB, so the Job
was OOM-killed on a pool it was never meant to run on, while the values file
that would have placed it on a large enough node looked correct.
The new test fails against the old template with "unknown path
spec.template.spec.nodeSelector".
* fix(guardrails): scan and re-emit raw Anthropic SSE streams in the bedrock post-call hook
* fix(guardrails): keep upstream id and model on a blocked Anthropic stream
* fix(guardrails): deliver a blocked Anthropic stream as an error frame
* fix(guardrails): deliver an unscannable Anthropic stream as an error frame
* fix(guardrails): emit the guardrail block detail as JSON in the stream error frame
* fix(guardrails): deliver an Anthropic block through the shared block-SSE builder
* fix(guardrails): keep the shared SSE assembler behavior-identical for existing callers
* fix(guardrails): keep the stream error message a string and drop an unreachable branch
* chore(guardrails): drop a comment that repeated its own docstring
* fix(guardrails): let bedrock service failures keep their status instead of framing them as blocks
* fix(guardrails): key the streamed block decision on status, not detail shape
InvokeGuardrailChecks details a Mapping on its 500 for an unparseable response,
so a detail-shape test read that outage as a policy block and framed it as a 200
guardrail_error. Both block sites raise 400, so gate on the status too.
* refactor(guardrails): narrow the SSE error-frame helper to the input it actually takes
Both callers pass a string, so the Mapping overload and its json.dumps branch
were unreachable. Folds the block branch's narrative comment into the rebind
suppressions that already carry a reason.
* test(ui): pin behaviour of guardrails-monitor, projects and logs components before migration
Adds role- and text-based characterisation tests for EvaluationSettingsModal,
GuardrailDetail and AuditLogDrawer, which had none, and moves the remaining
antd-specific assertions (.ant-spin, the icon role of an antd Spin indicator)
onto library-neutral ARIA queries. Also covers the enterprise banner on the
deleted keys and deleted teams pages, which no test reached.
All of these pass against the current antd and Tremor components.
* refactor(ui): migrate guardrails-monitor, projects and logs to shadcn
Replaces antd and Tremor with installed shadcn primitives across the files
these three routes exclusively own. Markup only, except where noted below.
Deletes AntDLoadingSpinner, an antd-only primitive living in the shadcn ui/
folder, and moves its single call site onto ui/ui-loading-spinner.
Two behaviour notes. The logs tab handler previously mapped every tab past
the first to "audit logs", so the audit panel kept polling while Deleted
Keys or Deleted Teams was on screen; each tab now reports its own value and
panels stay mounted via keepMounted. The evaluation settings dialog is
bounded to the viewport and scrolls internally, which the antd Modal got
from being top-anchored on a scrolling page.
The tests added in the previous commit pass unedited against these
components.
* test(interactions): follow Google spec drift replacing Turn with typed steps
* test(interactions): send step and content-list input to the live Gemini API
The migration turned each collapse row into a real button, but the existing
tests only click, so a regression in Enter or Space activation would still
pass. Add one test per component that tabs to the row, expands with Enter and
collapses with Space, asserting visibility rather than markup.
Both fail against the antd version and pass against the migrated one.
TruncatedValue swaps antd Tooltip and Typography for the shadcn Tooltip
and the shared CopyButton, so the full value now lives in the page and is
clipped with CSS rather than shortened in the text node.
OutputCard drops antd Typography for token-styled markup and folds its two
duplicated branches into one. Its border and the drawer's mono font stay on
the module's existing constants so the card still matches InputCard, which
is out of scope here.
Both files retire their no-restricted-imports suppression.
Replace antd Typography and inline hex styling in the log drawer's message
and tool-call blocks with plain elements and design tokens. Both components'
existing tests pass unedited before and after.
Retires their no-restricted-imports entries from the eslint suppressions
baseline.
The assertion hard-coded en-US separators while toLocaleString follows the
host locale, so it failed under de_DE. Building the expected string the same
way keeps it deterministic everywhere and still catches a dropped
toLocaleString wherever the locale groups at all.
Drops the antd Typography import from both. TokenFlow renders inside an antd
Descriptions.Item that already sets the colour, font size, line height and
wrapping the Text wrapper restated, so a bare span is pixel-identical there.
JsonViewer's placeholder moves onto the muted-foreground token.
The characterisation tests from the previous commit are unchanged and stay
green, which is what shows the markup swap did not move behaviour.
Both log-drawer collapse rows hand-rolled the same pattern: a click
handler on a plain div, hover tracked in React state, and a max-height
tween on an always-mounted panel. Move both onto the ui/collapsible
primitive with lucide chevrons, so the row is a real button that keyboard
users can reach and the open state lives in the primitive.
CollapsibleContent keeps keepMounted, which preserves the existing
contract that panel content stays in the DOM while collapsed.
Neither test file is touched: both were already role and text based, and
they pass unedited against the new markup.
Both components are shared by the logs, guardrails-monitor and tool-policies
routes and had no test. These assert on rendered text and roles only, so they
hold against antd Typography and against its replacement.
* fix(langfuse): emit otel trace version and release on the keys langfuse v4 reads
The langfuse_otel exporter wrote version to langfuse.generation.version and
langfuse.trace.version, and release to langfuse.trace.release. Langfuse v4
recognizes neither, so both landed in the generic span attribute bag and every
trace reported version and release as null. v4 has a single langfuse.version
key, lifted to the trace when it sits on the root span, plus langfuse.release.
Also routes the otel v2 preset's per-request headers through the shared builder
so key-scoped and team-scoped exports carry x-langfuse-ingestion-version like
the other three exporter paths already do.
* fix(langfuse): give trace_version precedence over version on the shared v4 key
Matches the documented contract in docs/observability/langfuse_integration.md
and the legacy langfuse SDK callback, which both treat trace_version as the
authoritative trace version with version as its fallback.
cost_per_token reads litellm.model_cost, which CI loads from the
model_prices_and_context_window.json on main, so the new entries were
missing until merge. Use the same local_model_cost_map fixture the
sonnet 5 metadata test uses to force the branch's own backup map.
Pre-routing now reads the request's tags on every request with a registered
strategy, including the single-strategy case that used to short-circuit before
looking at tags. Metadata is request-controlled, so a caller that sends
`litellm_metadata` (or `tags`) as a string or any other non-dict shape crashed
tag lookup with an AttributeError instead of routing untagged.
The virtual-key edit view never rendered the router settings stored on a
key, so fallbacks configured at creation could not be verified or changed
afterwards. The key info panel now summarises them and the edit view
embeds the router settings accordion.
The accordion is a fixed-field editor, so its value is merged over the
stored object instead of replacing it. Routing fields it cannot render,
such as tag_routing_prefix or model_group_retry_policy, survive an
unrelated edit, while a field it does own that the admin emptied still
goes out as null so the clear reaches the server. Emptying every field
sends {}, which the proxy reads as no key-level override, so the key
falls back to its team and global routing rather than being pinned to a
blob of nulls.
Adds vpc_id/public_subnet_ids/private_subnet_ids to deploy into existing
networking, plus create_database/database_url and create_redis/redis_url to
use existing data stores or none at all. Defaults keep today's module-owned
behavior.
A container reads a secret by ARN, which gives Terraform no edge to the
_version that writes the value. The managed-Aurora path hid that behind the
cluster create; the bring-your-own path has nothing slow in between, so the
migration, the bootstrap, and both services now depend on the versions
explicitly.
Supplied private subnets must cover two AZs whenever Aurora or ElastiCache is
module-created, since both subnet groups require it, and a check block warns
when a Redis-less stack can run more than one gateway process, because per-key
rate limits are counted per process without Redis.
* feat(model_prices): add NVIDIA Nemotron 3.5 Lightning on OpenRouter and DeepInfra
Nemotron 3.5 Lightning shipped 2026-08-11 with public per-token pricing on
OpenRouter and DeepInfra at $0.05/M in and $0.20/M out. Without cost map
entries both ids raise "This model isn't mapped yet" and log at zero spend.
* fix(model_prices): stop asserting an output cap for Nemotron 3.5 Lightning
262144 is the native context window, not the output budget, and neither
OpenRouter nor DeepInfra publishes an output cap. Keeps max_input_tokens at the
native 256K window: 1M needs VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 on a self-hosted
deployment, so it is not what these hosted endpoints serve.
* chore(tests): drop the Nemotron 3.5 Lightning metadata test
Requested on the review thread: the cost map entries stand on their own.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: never price a strategy-router alias
A strategy-router alias (auto_router/complexity_router/<name>) is never the
deployment that gets called or billed, but custom pricing configured on it was
being treated as real pricing in two places:
- registered in litellm.model_cost under the alias deployment id, so an
explicit zero made _is_cost_explicitly_configured() report the group as a
genuinely free model and every budget check was skipped, while the request
routed to a paid deployment and accrued real spend
- copied onto request_kwargs by the alias-params merge, so the routed
deployment got re-registered at the alias price and the request billed 0.0
Both are fixed at the writer, so config, /model/new and price-map reload all
take the same path
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: annotate filtered cost-map copy for the mutable-collection gate
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(proxy): serialize model reconciles so concurrent writes stop evicting each other
A model write is a read-modify-write of the shared `llm_router` global: read the
db into a snapshot, then make the router match that snapshot. Nothing serialized
it, so two of them interleaving was not a lost update but an eviction --
_delete_deployment removes every live deployment absent from the snapshot it was
handed, so the request holding the older snapshot reconciles the newer request's
model straight back out of the router. The row survives in the db, which is what
makes it easy to miss: the pod simply stops serving a model it was told to serve
until some later reload happens to put it back.
clear_cache compounds it. It deletes every db model from the router before
reloading them, so for the width of that reload the pod serves none of them --
and any concurrent write sampling the router in that window sees the hole.
Fix is one lock (MODEL_RECONCILE_LOCK) held across both, so each reconcile reads
the db and applies it atomically and no stale snapshot can evict a newer model.
clear_cache holds it across wipe+reload and calls the already-locked
_add_deployment_locked, since asyncio.Lock is not reentrant and routing back
through the public add_deployment would deadlock the pod's whole model-write
path.
The verdict needed the same treatment. raise_if_reload_degraded_serving compared
a desired-set read during the reload against a router snapshot taken after it,
so a neighbouring reconcile's in-flight wipe was reported to the caller as
collateral damage from its own reload -- a 500 on a create that had in fact
succeeded. Reconciles now return a ReconcileOutcome carrying both the desired set
and the post-reconcile serving state, captured before the lock is released, and
the verdict judges against that. Omitting live_after keeps the old live re-read,
which stays correct for the no-reconcile-ran case.
Found by running the e2e suite with pytest-xdist at 8 workers: three unrelated
tests failed together on "Previously served model id(s) [...] are also no longer
being served by this pod", which is this. Serial runs concurrent enough to hit it
are rare, which is why 78 minutes of sequential e2e never surfaced it -- but any
customer provisioning models in parallel (terraform, CI) is in exactly this race.
test_reconciles_serialize_so_no_stale_snapshot_can_evict fails with 5 == 1
without the lock.
* fix(tests): return a ReconcileOutcome from the PTU test's add_deployment mock
test_ptu_model_settings.py stubs proxy_config.add_deployment with
AsyncMock(return_value=None). Now that add_deployment returns a
ReconcileOutcome, add_new_model reads .still_desired off that None and
the two PTU gate tests fail with "'NoneType' object has no attribute
'still_desired'".
Return ReconcileOutcome(still_desired=None, live_after=None), matching
the other reconcile mocks. Both fields None means no reconcile state was
captured, so the serving verdict falls back to reading the router live,
which is what the test's mock_router already drives — the PTU assertions
are unchanged.
Two sibling test files were updated for this in the parent commit; this
one was missed because the local env cannot collect four modules under
tests/test_litellm/proxy (prisma generate artifacts), so the full shard
only ran in CI.
Also applies ruff format to proxy_server.py: the new add_deployment
wrapper's single call fits on one line under the project's line length.
* fix(proxy): lock the delete evictions and stop clear_cache wiping deployments
Two follow-ups to MODEL_RECONCILE_LOCK, both found by review.
1. delete_model and delete_team_models evict from llm_router directly,
outside the lock. The db row is gone by then, but a reconcile that
snapshotted the db BEFORE the delete still lists that id as desired and
upserts the deployment straight back, so the pod keeps serving a model
the database no longer has until some later reconcile notices. Taking
the lock orders the eviction after any in-flight reconcile's re-add.
Both new tests fail without the lock ("did not wait for
MODEL_RECONCILE_LOCK") and pass with it.
2. clear_cache no longer wipes deployments. It used to delete_deployment()
every db model before the reload restored them, which left the router
serving ZERO db models for the entire width of the reload -- every
inference request landing in that window fell into a real hole, and
serializing reconciles made the aggregate outage additive rather than
overlapping. The wipe was also redundant: _delete_deployment evicts
exactly the ids the db no longer lists, and upsert_deployment
pops-and-re-adds a deployment whose params changed while no-opping one
that did not, so the reconcile converges to the same state on its own.
Every mutation is visible to that comparison (blocked, and updated_at
for premium, are written into model_info).
The auto-router pops are NOT redundant and stay: they are keyed by
model_name, which no deployment-id reconcile touches.
The new tests patch their own lock rather than contending the module-level
one: asyncio.Lock binds to the event loop of its first contended acquire
and raises on every other loop after that, which would poison the next
asyncio test in the process. The proxy has a single event loop for its
lifetime so this is test-only, but it is a trap worth naming for whoever
writes the next concurrency test here.
* fix(proxy): scope the clear_cache wipe to auto-router deployments
Review caught a regression in the previous commit. Dropping the wipe
entirely stranded every db-backed auto-router on the pod.
The strategy registries (auto_routers, complexity_routers,
adaptive_routers, quality_routers) are keyed by model_name, which no
deployment-id reconcile touches, so clear_cache pops them and relies on
the reload to rebuild them. But the rebuild only happens on the ADD path:
Router.upsert_deployment returns early when a deployment is unchanged and
never reaches add_deployment -> _add_deployment ->
init_auto_router_deployment, which is what repopulates them. With the wipe
gone the deployment was always unchanged, so the pop was permanent: ANY
unrelated model write -- a team admin patching one team-owned model --
left every db-backed auto, complexity, adaptive and quality router
unroutable across tenants until a restart.
Restore the wipe for exactly the auto_router/* db deployments, whose
strategy entries are the ones being popped. Deleting them forces upsert
down the add path so both the deployment and its strategy entry come back.
Ordinary db models stay un-wiped, which is the point of the previous
commit: wiping them un-served every db model for the width of the reload,
and the reconcile converges without it.
test_clear_cache_wipes_auto_routers_but_leaves_ordinary_db_models pins
both halves against each other, since fixing either one naively breaks the
other. Both clear_cache tests fail with the pop-without-delete version.
* refactor(clear_cache): fold auto-router wipe into the classification pass
The auto-router scoping added in 5deddfd introduced two new mutable-collection
constructions, pushing LIT002 five over its budget ceiling.
Rather than suppress, do the work in the single pass that already walks
current_models: detect and delete the auto_router/* db deployments while
classifying, accumulating names into a set that replaces the old
db_router_deployments comprehension. Net-zero LIT002, same behaviour.
Comment updated to describe where the wipe actually happens now.
