An expression at `jobs.<id>.strategy` is legal on GitHub, but the model
required a mapping there, so a workflow using one made the whole file
unreadable and turned code-quality red. That job's names are now a blind
spot like any other name the sweep cannot work out offline.
A matrix whose `name:` holds no matrix value publishes that one name once
per combination, which leaves a required context just as ambiguous as two
jobs sharing a name, so it now reports instead of deduping.
A file that does not parse as one YAML document is reported the way the
module already promised, rather than escaping as a traceback.
Three ways the sweep could fail a workflow GitHub would publish fine.
`github.workflow` and `github.job` were counted as fixed for the whole run, so
two jobs naming themselves after the workflow they sit in were reported as a
collision. `runner` and `vars` were wrong the same way. Drop the exception
entirely: a name still holding an expression is one GitHub resolves per job, so
it is nothing to compare, which is what the rest of the module already does.
`format()` was resolved with Python's semantics, so an attribute lookup crashed
the script and a width specifier padded a name GitHub never pads. Fill `{0}`
holes and escaped braces, and treat anything richer as unresolved.
A matrix `include` or `exclude` row holding a value that is not a scalar lost
that key and became an empty row, which excludes every combination. Report the
row instead of quietly reshaping the matrix around it.
A job name holding an expression the sweep could not resolve was compared as
if it were the published name. Two jobs whose names differ per matrix value or
per caller input were reported as a collision, and a matrix that was itself an
expression collapsed onto the bare job id and did the same.
Model what a job publishes as known names beside the reasons the rest stay
unknown. Anything the sweep cannot work out contributes no name and is
reported as a note instead of guessed at. An expression over contexts that are
fixed for the whole run still compares, so two jobs sharing one of those are
still caught.
The collision sweep read a job's name as its `name:` or bare job id, which is
wrong for a matrix job that sets no name: GitHub publishes `build (3.12)`, one
per combination. That missed real duplicates and invented ones that don't exist.
It also crossed every matrix value while ignoring `exclude`, so it checked
combinations no job ever runs.
Four smaller gaps went with it. Boolean matrix values reached a name as `True`
rather than `true`. A `format()` whose arguments cannot fill its placeholders
raised straight out of the script instead of leaving the name unresolved. A job
calling a reusable workflow only ever chained one level, and a call outside the
repo fell back to the caller's own name, which GitHub never posts. A job whose
`name:` was not a string failed validation and silently dropped every job in
that file, so the sweep now renders any scalar and reports a file it cannot read
instead of skipping it.
The checker raised a custom exception and caught it two lines down in the
same module, which is the throw-then-catch the repo's coding guide rules
out. `main` now prints the same message and returns the exit code, so the
collision list stays a value the whole way out
The guard read each matrix key's values independently and crossed them, so
a job name reading two keys off one include row published pairs no job ever
runs, which could fail a valid workflow on a required check
It now builds the combinations GitHub builds: the listed keys crossed, each
include row folded into the combinations it overwrites nothing in, and a row
that fits nowhere standing on its own
The guard only substituted a bare `${{ matrix.key }}`, so any name built from a
larger expression stayed in the string as its own template. `_test-unit-base.yml`
names its job with a ternary over `format()`, which meant every shard published
an opaque name and 23 of the 33 required contexts, all of them `<shard> / Run
tests`, were invisible to the very check meant to protect them.
Job names are now evaluated per matrix combination over the pieces a name can
hold: string literals, `matrix.<key>`, `format()`, `==` and `!=`, and the
`<cond> && <a> || <b>` idiom. All 33 required contexts now resolve, and nothing
in the repo leaves an expression unresolved. An expression the evaluator does not
understand still falls back to its verbatim template, so two jobs sharing one
stays a collision.
The test also drops its `sys.path.insert`, which the test-quality budget counts
under TQ003; pytest already puts the file's own directory on the path.
A ruleset's required status check names a check run and GitHub matches it by
that name alone, so two jobs publishing the same name leave the gate unable to
say which job proved it. The new code-quality check reads every workflow,
expands matrix values and local reusable-workflow calls the way Actions does,
and fails when one name has more than one job behind it.
The selector picked up two suites that can never pass in this stack, so
editing either one turned the check permanently red: the presidio masking
suite calls pytest.fail without an analyzer and anonymizer that up.sh
never starts, and the pipecat audio suite skips itself at import time
unless the NLTK punkt_tab data is present, which nothing installs.
tests/e2e/coverage_registry/test_collector.py had the same problem for a
different reason. Its nested pytest.main autoloads pytest-retry from the
ci group the workflow installs and dies with "INTERNALERROR: no option
named 'filtered_exceptions'", so the collect-only pass now disables that
plugin. The plugin's entry point is pytest-retry, not retry, so the same
one-word fix lands on mutmut's pytest_add_cli_args, where "-p no:retry"
was disabling nothing.
Two smaller holes in the harness: a canary argument the shell never
expanded used to select nothing and let the gate pass green, and a secret
that cannot be represented in both bash and dotenv was rejected without
naming the key.
The first canary run failed pass 1 because the stage-mirror config had no
openai-text-embedding-3-small while test_llm_api_routes_group_grants_every_llm_endpoint
calls /embeddings with it; the public log named the test, which is the
behavior the previous commit added
A harness-only change (proxy_client.py, conftest.py, pytest.ini, the gateway
config, .github/e2e-stack, or the workflow) selected nothing, so the stack was
never exercised by the change that touched it. select_tests.py keeps the
changed-file rule and adds the access_control suite whenever a harness file
changes. The run step now reports the pytest exit code before the evidence
check, prints pytest's summary line per pass so the rerun count is visible,
and assert_tests_ran.py names each failed or errored test as classname::name
The retrieve tool was injected whenever any hash=<24hex> string appeared in the
restored conversation, including protected rows and caller-authored text, so a
git SHA in a tool result registered a bogus hash and billed a useless retrieval
round trip on every later turn. The compression service reports the hashes it
actually stored in ccr_hashes; that field is now the only source, validated to
the service's own 12 to 24 hex grammar before it reaches the retrieve URL.
Assistant rows are no longer flattened to strings before compression: the
service protects assistant text blocks but has no gate for assistant strings,
so the model's own earlier tables came back as a schema line plus CSV.
Adds ccr_retrieval (default true) so operators on a marker-free sidecar can
turn the retrieval loop off entirely.
The chat and Responses bridges serialize tool_use blocks with model_dump(), so every
bridged /v1/messages response carried LiteLLM's internal provider_specific_fields key
(null, or a Gemini thought signature). Clients replay the block verbatim, and the next
turn that lands on a native Anthropic deployment (auto-router tier change, model swap)
is rejected with "tool_use.provider_specific_fields: Extra inputs are not permitted"
Strip the key from replayed content blocks at the single native Anthropic dispatch so
already-poisoned transcripts self-heal on every native provider, and stop emitting the
null on new responses. The bridges keep reading the signature for the Gemini round trip
Closes#19739
* fix(guardrails): don't inspect embeddings in the AIM and Cato hooks
`pre_call_hook` fires for /embeddings as well as chat. An embeddings body
carries `input` — documents being indexed, not a prompt — which
`build_inspection_messages` lifts into synthetic chat messages, so both hooks
inspect it as a conversation and a policy verdict on that text breaks a request
that was never one:
- AIM, anonymize + batched `input`: `has_non_string_content` is true for any
list, so `_anonymize_request` raises 400 "...multimodal input...".
- AIM, anonymize + single-string `input`: no error — the input is rewritten to
redacted text and the caller embeds text it never sent.
- AIM and Cato, block: the embeddings request is blocked outright.
Gate both hooks on a new `NON_CONVERSATIONAL_CALL_TYPES` deny-list. This is
deliberately not `TEXT_CONTENT_CALL_TYPES`: that allow-list omits
`anthropic_messages`, `responses` and `call_mcp_tool`, so gating on it would
stop these guardrails inspecting real chat traffic. An unrecognised or newly
added call type is still inspected.
* feat(guardrails): add inspect_embeddings toggle for AIM and Cato
* fix(guardrails): redact batched embedding input on anonymize
A list of plain strings is the /embeddings batch shape. AIM rejected it as
multimodal and Cato forwarded the original strings, so anonymize never
reached the provider for batched input. Redactions are now written back
element-wise, one redacted message per non-empty element, so a fully
redacted element cannot shift the following documents into the wrong slot.
* fix(guardrails): reject partial embedding redactions
* fix(guardrails): avoid unnecessary batch type check
* style(tests): drop trailing blank line in cato guardrail tests
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject aim redactions with no text content
The anonymize path read role and content off every entry of the vendor's
redacted_chat before the shared write-back helper could refuse the payload,
so a message missing content, or a bare string in place of a message, raised
out of the hook as a 500. Validate the vendor list first and return the 400
the guardrail already uses for an unusable redaction.
* fix(guardrails): validate all aim redaction paths
Validate AIM redaction containers before request or output rewrites, reject
cardinality mismatches and empty output, and cover malformed vendor payloads
with regression tests.
* fix(guardrails): preserve aim output redaction alignment
AIM returns the inspected request messages followed by the assistant output.
Validate that full response and select the final redacted message instead of
requiring a single entry.
* test(guardrails): cover aim output anonymize alignment and malformed redactions
---------
Co-authored-by: Guy Levi <guy.levi@catonetworks.com>
* fix(azure_sentinel): split batches under the 1MB ingestion cap and keep undelivered records queued
Azure Monitor rejects any Logs Ingestion body over 1MB with a 413. The Sentinel logger
posted the whole queue as one body and cleared it in a finally block, so an oversize
batch, a transient 5xx, or a failed token call dropped every queued record, and records
logged while a send was in flight were cleared with it. Both the standard and the audit
queue share the sender.
Move Datadog's proactive size split and 413 halving into a shared helper,
litellm/integrations/batch_utils.send_batch_with_413_split, and route Sentinel through it
with a 1MB size check. A lone record that still 413s is dropped, everything a transient
failure leaves undelivered goes back to the front of its queue, and the retry queue is
capped at max_queue_size so an unreachable workspace cannot grow memory without bound
* fix(azure_sentinel): retry undelivered records on the flush timer only
Requeued records made every later event cross the batch_size threshold, so a
down ingestion endpoint got one full-queue resend per request. Threshold sends
now go through flush_queue, so they take the flush lock instead of racing the
timer, and they stand down while records are awaiting retry.
A record that cannot be serialized raised out of the size probe and killed the
periodic flush task. The probe now runs inside the failure handling, so the
batch is split and only the record that cannot be serialized is dropped.
* fix(azure_sentinel): decide threshold sends under the flush lock
Concurrent callbacks all read logs_awaiting_retry before the first send
finished, so each one resent the whole queue once that send failed. The
flag and the batch_size threshold are now rechecked while holding the
flush lock, and each queue sends only itself instead of going through
flush_queue, which was retrying the other queue too.
* test(azure_sentinel): cover successful threshold waiters
* fix(azure_sentinel): preserve cancelled batches for retry
* fix(azure_sentinel): requeue only the undelivered part of a cancelled split
A batch over the ingestion cap goes out in pieces, so a cancellation partway
through requeued pieces the destination had already accepted and sent them a
second time on the next flush
The split helper now raises a cancellation carrying the records it never
delivered, and Azure Sentinel requeues those instead of the whole batch
* fix(azure_sentinel): drop batches a permanent rejection will never accept
A non-413 4xx from the ingestion endpoint or from the OAuth token call means the request
will fail the same way on every retry, so requeueing it held the batch, and every record
logged behind it, until the queue cap dropped them. Retryable statuses (5xx, 408, 429)
still keep the whole batch, and a shared classifier gives Datadog the same rule
The serialization probe now catches any exception, not just TypeError and ValueError,
because safe_dumps hands pydantic models to model_dump and can raise anything. It also
splits on record count, so a recovery flush sends batch_size records per request instead
of serializing the whole requeued queue to measure it
Both integrations re-raise a cancelled send as exactly asyncio.CancelledError. Python
3.12's asyncio.wait_for only translates the exact class into TimeoutError, so the
BatchSendCancelled subclass escaped the logging worker as an unhandled error
The awaiting-retry flag now follows the queue that survived the max_queue_size trim, so
a deployment with the cap at zero is not left waiting for a timer flush with nothing
queued to retry
* chore(logging): document mutable queue ownership
Annotate the queue detach and requeue constructions required by the logger's appendable queue contract so the type-discipline budget stays clean
* fix(datadog): preserve non-413 retry behavior
Keep Datadog's existing contract of requeuing every non-413 HTTP failure while Azure Sentinel applies its permanent-client-error policy through the shared splitter
* fix(batch_utils): requeue by default and let Sentinel opt into dropping
The shared splitter's default non-success handler is now requeue_after_http_error, the behavior Datadog had before the extraction, so a caller that omits the argument keeps its records. Azure Sentinel passes undelivered_after_http_error explicitly to drop permanent 4xx rejections
Also drops an explicit return None the strict ruff gate flags in the test helper
Format the model-picked id with %r so control characters in it cannot
break the log line. The regression test for the unlisted id keeps to
generic scoping wording
The sidebar and header were still keyed on legacy ?page= ids and mapped
back and forth through MIGRATED_PAGES, legacyPageHref and
legacyKeyForPathname. Leaves are now plain Next links to their path
route, the active item and breadcrumb come from usePathname, and the
setPage/defaultSelectedKey prop chain is gone.
The id-to-route table moves next to the dashboard root page as its only
consumer. That redirect now forwards the remaining query params instead
of dropping them, so deep links such as the proxy's MCP env-var setup
link (?page=mcp-servers&fill_env_vars=) no longer rely on the target page
reading the pre-redirect URL during its first render. The proxy builds
that link as /ui/mcp-servers?fill_env_vars= directly, and the Playground
warnings link to the real routes instead of relative ?page= URLs.
migratedHref is renamed uiHref, the /ui base-path helper it always was.
A one-character value in the provider secret bundle was masked too, which
turned every 1 in the run log into ***, including the pass numbers and the
gateway addresses, so the only public diagnostics were unreadable
Emulated file_search now warns when the model returns a vector_store_id that is
not one of the request's stores, naming the dropped id and the stores that were
searched instead. H16 asserts the warning is emitted exactly once.
_average_latency skipped integer samples in the sum while counting them in the denominator, which contradicted its own
Sequence[float | int] signature; it now averages every sample. Both success loggers in cost-based routing computed
response_ms / completion_tokens and threw the result away, so a chat response with zero completion tokens raised
ZeroDivisionError inside the handler. The proxy swallows and logs it, but the handler then skips that request's tpm and
rpm update, so cost-based routing undercounts the deployment's usage. The QA run for the latency fix hit it on real
gpt-5.5 traffic through /v1/chat/completions and /v1/messages
The regression test's recording logger overrode async_post_call_failure_hook
with untyped parameters. It now mirrors the base signature, and the
UserAPIKeyAuth import moves to module level so the annotation resolves.
The emulated file_search handler searched whatever vector_store_id the
model returned, so a model steered to an id outside the request's
file_search tool reached a store the per-key vector store permission
check never saw. An id outside the request's stores now falls back to
those stores; an id that is one of them still narrows the search to it.
The changed-tests workflow overrode the suite's `--reruns 1` with `--reruns 0`, so a
transport blip failed a pass that pytest.ini already scopes to network errors and
5xx responses. Pass 2 of run 33692484803 also went red 15s after a model write with
"no healthy deployments": the barrier only polled /v1/models through nginx, which
proves one gateway converged, and the next request rolled the other. The stack now
exports LITELLM_PROXY_REPLICA_URLS, the barrier polls every replica with the full
budget before settling, and up.sh refuses to boot without DD_API_KEY, since the
gateway config enables the datadog callback on every run
Latency-based routing averaged a deployment's cached samples with total / len(samples) and raised ZeroDivisionError
once an entry held none, which the proxy answered as a 500 for every later request on that model group. Cost-based
routing writes the same {model_group}_map entry with minute counters only, so a group used by both strategies hit this
on every latency-routed request. A deployment with no samples now counts as 0 latency, the same as one the router has
never seen
Resolves LIT-7053
A failed pass-through call logged the httpx traceback, whose message
quotes the upstream URL with the provider API key in its query string,
into the spend log's error information and into every failure callback.
The error information built for logging now redacts its traceback and
error message, and the traceback is redacted once before the failure
callbacks receive it.
The test-quality gate counts every patch of a litellm internal against a ceiling, and the three patches this test needs pushed it over. The callback imports increment_spend_counters, update_cache and proxy_logging_obj from proxy_server inside its own body, so there is no seam to inject fakes through; every other test in this file uses the same three patches for the same reason
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The previous cleanup only ran inside a caller's own except handler, so a
build that failed after its only caller had already been cancelled left
the failed task cached with nothing left to clear it. Move the cleanup
onto the task itself as a done-callback, which fires whether or not
anyone is still awaiting it, so the next request always gets a fresh
attempt instead of replaying the stale failure.
Adds a regression test for exactly that ordering (cancel the only
caller, let the build fail unobserved, then confirm the next request
builds successfully); it fails against the previous except-based
cleanup, which left the task cached.
test_load_state_from_db_handles_unknown_request_type compared the
WRITING cell after load against a cold-start value captured for
GENERAL. They happened to be equal for this fixture (the fast model's
empty strengths list makes every request type's prior identical), which
hid that the assertion was comparing the wrong baseline. Capture each
request type's own cold-start value instead.
The proxy cost callback zeroed response_cost whenever cache_hit was true. That rule dates from Jan 2024 when it was the only place cache hits were priced. The logging layer has priced the LLM share at 0 on a cache hit since Aug 2024, and since guardrail cost joined the standard logging payload the proxy-side zeroing has thrown away a real provider charge: a pre_call guardrail runs before the cache is consulted, so a cached response still cost whatever the guardrail billed. Drop the redundant zeroing so the payload's response_cost, which is already LLM 0 + guardrail cost, reaches spend logs, daily tables and budgets untouched
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
_ensure_routelayer previously awaited asyncio.to_thread(...) directly
inside the lock. Under cancel_on_disconnect, cancelling that await
released the lock while the worker thread kept running, so a second
concurrent request would see no lock held and start a duplicate billed
build. Store the build as a task on self and have every caller await it
through asyncio.shield: cancelling one caller's wait no longer cancels
the build or lets another caller start a second one. A real build
failure (not merely a cancelled caller) clears the slot so the next
call retries fresh instead of replaying the same failure forever.
Also replaces the two tests that monkeypatched _build_routelayer (an
anti-pattern per this repo's conventions) with ones that instrument the
already-injected embedding router dependency instead, and adds a third
proving the cancellation race is actually closed.
- Shrink the fallback comments to one line each; the fuller rationale
was redundant per repo comment policy.
- Add test_pick_model_favors_the_cheaper_model_info_priced_deployment
and its hybrid-router counterpart, which exercise pick_model's actual
Thompson-sampling/scoring output instead of only asserting the
model_to_cost dict. Both are ordered so the expensive model wins
pick_best's insertion-order tie-break on the pre-fix code (proven by
reverting the production diff and rerunning), so they fail before the
fix and pass after it.
Both places that build an adaptive router's model_to_cost (the plain
auto_router/adaptive_router path in router.py, and the hybrid
adaptive-inside-complexity_router path in complexity_router.py) read
input_cost_per_token from litellm_params only. Custom pricing is
conventionally declared under model_info everywhere else in LiteLLM
(cost_calculator.py, add_deployment's litellm.model_cost registration),
so a deployment priced that way silently costs 0.0 in adaptive-router
scoring: every candidate ties on cost, the cost term contributes
nothing, and routing runs on quality alone with no warning.
Fall back to model_info at both call sites when litellm_params does not
declare a cost, matching how quality_router.py already sources cost.
litellm_params still wins when both are set.
Fixes#31481.
