Add tests/e2e/jwt_issuer.py, a test-only RS256 issuer that serves a JWKS
document and signs arbitrary claims over an open loopback-only /token
endpoint, so e2e tests can mint tokens without ever holding a signing key.
One key per process keeps the proxy's cached JWKS valid for the whole run.
Add the first five live JWT auth tests in tests/e2e/other: a valid token for
an existing team is accepted and its spend row carries the claimed team and
user, a tampered signature and an expired token are refused with 401, a
token naming a team that does not exist is refused with 403, and a plain
sk- virtual key keeps working with enable_jwt_auth on. Harness unit tests
cover the issuer itself. Team and user create/delete land on the shared
ProxyClient (warn-only teardown, /user/delete typed as the int it returns)
instead of a fourth per-suite copy.
Document the issuer command, the E2E_JWT_ISSUER_PORT convention (default
4190), JWT_PUBLIC_KEY_URL and the litellm_jwtauth config block in
tests/e2e/CONTRIBUTING.md, and register the new cells in other.yaml.
* fix(guardrails): don't inspect embeddings in the AIM and Cato hooks
`pre_call_hook` fires for /embeddings as well as chat. An embeddings body
carries `input` — documents being indexed, not a prompt — which
`build_inspection_messages` lifts into synthetic chat messages, so both hooks
inspect it as a conversation and a policy verdict on that text breaks a request
that was never one:
- AIM, anonymize + batched `input`: `has_non_string_content` is true for any
list, so `_anonymize_request` raises 400 "...multimodal input...".
- AIM, anonymize + single-string `input`: no error — the input is rewritten to
redacted text and the caller embeds text it never sent.
- AIM and Cato, block: the embeddings request is blocked outright.
Gate both hooks on a new `NON_CONVERSATIONAL_CALL_TYPES` deny-list. This is
deliberately not `TEXT_CONTENT_CALL_TYPES`: that allow-list omits
`anthropic_messages`, `responses` and `call_mcp_tool`, so gating on it would
stop these guardrails inspecting real chat traffic. An unrecognised or newly
added call type is still inspected.
* feat(guardrails): add inspect_embeddings toggle for AIM and Cato
* fix(guardrails): redact batched embedding input on anonymize
A list of plain strings is the /embeddings batch shape. AIM rejected it as
multimodal and Cato forwarded the original strings, so anonymize never
reached the provider for batched input. Redactions are now written back
element-wise, one redacted message per non-empty element, so a fully
redacted element cannot shift the following documents into the wrong slot.
* fix(guardrails): reject partial embedding redactions
* fix(guardrails): avoid unnecessary batch type check
* style(tests): drop trailing blank line in cato guardrail tests
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject aim redactions with no text content
The anonymize path read role and content off every entry of the vendor's
redacted_chat before the shared write-back helper could refuse the payload,
so a message missing content, or a bare string in place of a message, raised
out of the hook as a 500. Validate the vendor list first and return the 400
the guardrail already uses for an unusable redaction.
* fix(guardrails): validate all aim redaction paths
Validate AIM redaction containers before request or output rewrites, reject
cardinality mismatches and empty output, and cover malformed vendor payloads
with regression tests.
* fix(guardrails): preserve aim output redaction alignment
AIM returns the inspected request messages followed by the assistant output.
Validate that full response and select the final redacted message instead of
requiring a single entry.
* test(guardrails): cover aim output anonymize alignment and malformed redactions
---------
Co-authored-by: Guy Levi <guy.levi@catonetworks.com>
* fix(azure_sentinel): split batches under the 1MB ingestion cap and keep undelivered records queued
Azure Monitor rejects any Logs Ingestion body over 1MB with a 413. The Sentinel logger
posted the whole queue as one body and cleared it in a finally block, so an oversize
batch, a transient 5xx, or a failed token call dropped every queued record, and records
logged while a send was in flight were cleared with it. Both the standard and the audit
queue share the sender.
Move Datadog's proactive size split and 413 halving into a shared helper,
litellm/integrations/batch_utils.send_batch_with_413_split, and route Sentinel through it
with a 1MB size check. A lone record that still 413s is dropped, everything a transient
failure leaves undelivered goes back to the front of its queue, and the retry queue is
capped at max_queue_size so an unreachable workspace cannot grow memory without bound
* fix(azure_sentinel): retry undelivered records on the flush timer only
Requeued records made every later event cross the batch_size threshold, so a
down ingestion endpoint got one full-queue resend per request. Threshold sends
now go through flush_queue, so they take the flush lock instead of racing the
timer, and they stand down while records are awaiting retry.
A record that cannot be serialized raised out of the size probe and killed the
periodic flush task. The probe now runs inside the failure handling, so the
batch is split and only the record that cannot be serialized is dropped.
* fix(azure_sentinel): decide threshold sends under the flush lock
Concurrent callbacks all read logs_awaiting_retry before the first send
finished, so each one resent the whole queue once that send failed. The
flag and the batch_size threshold are now rechecked while holding the
flush lock, and each queue sends only itself instead of going through
flush_queue, which was retrying the other queue too.
* test(azure_sentinel): cover successful threshold waiters
* fix(azure_sentinel): preserve cancelled batches for retry
* fix(azure_sentinel): requeue only the undelivered part of a cancelled split
A batch over the ingestion cap goes out in pieces, so a cancellation partway
through requeued pieces the destination had already accepted and sent them a
second time on the next flush
The split helper now raises a cancellation carrying the records it never
delivered, and Azure Sentinel requeues those instead of the whole batch
* fix(azure_sentinel): drop batches a permanent rejection will never accept
A non-413 4xx from the ingestion endpoint or from the OAuth token call means the request
will fail the same way on every retry, so requeueing it held the batch, and every record
logged behind it, until the queue cap dropped them. Retryable statuses (5xx, 408, 429)
still keep the whole batch, and a shared classifier gives Datadog the same rule
The serialization probe now catches any exception, not just TypeError and ValueError,
because safe_dumps hands pydantic models to model_dump and can raise anything. It also
splits on record count, so a recovery flush sends batch_size records per request instead
of serializing the whole requeued queue to measure it
Both integrations re-raise a cancelled send as exactly asyncio.CancelledError. Python
3.12's asyncio.wait_for only translates the exact class into TimeoutError, so the
BatchSendCancelled subclass escaped the logging worker as an unhandled error
The awaiting-retry flag now follows the queue that survived the max_queue_size trim, so
a deployment with the cap at zero is not left waiting for a timer flush with nothing
queued to retry
* chore(logging): document mutable queue ownership
Annotate the queue detach and requeue constructions required by the logger's appendable queue contract so the type-discipline budget stays clean
* fix(datadog): preserve non-413 retry behavior
Keep Datadog's existing contract of requeuing every non-413 HTTP failure while Azure Sentinel applies its permanent-client-error policy through the shared splitter
* fix(batch_utils): requeue by default and let Sentinel opt into dropping
The shared splitter's default non-success handler is now requeue_after_http_error, the behavior Datadog had before the extraction, so a caller that omits the argument keeps its records. Azure Sentinel passes undelivered_after_http_error explicitly to drop permanent 4xx rejections
Also drops an explicit return None the strict ruff gate flags in the test helper
Format the model-picked id with %r so control characters in it cannot
break the log line. The regression test for the unlisted id keeps to
generic scoping wording
The sidebar and header were still keyed on legacy ?page= ids and mapped
back and forth through MIGRATED_PAGES, legacyPageHref and
legacyKeyForPathname. Leaves are now plain Next links to their path
route, the active item and breadcrumb come from usePathname, and the
setPage/defaultSelectedKey prop chain is gone.
The id-to-route table moves next to the dashboard root page as its only
consumer. That redirect now forwards the remaining query params instead
of dropping them, so deep links such as the proxy's MCP env-var setup
link (?page=mcp-servers&fill_env_vars=) no longer rely on the target page
reading the pre-redirect URL during its first render. The proxy builds
that link as /ui/mcp-servers?fill_env_vars= directly, and the Playground
warnings link to the real routes instead of relative ?page= URLs.
migratedHref is renamed uiHref, the /ui base-path helper it always was.
Emulated file_search now warns when the model returns a vector_store_id that is
not one of the request's stores, naming the dropped id and the stores that were
searched instead. H16 asserts the warning is emitted exactly once.
The regression test's recording logger overrode async_post_call_failure_hook
with untyped parameters. It now mirrors the base signature, and the
UserAPIKeyAuth import moves to module level so the annotation resolves.
The emulated file_search handler searched whatever vector_store_id the
model returned, so a model steered to an id outside the request's
file_search tool reached a store the per-key vector store permission
check never saw. An id outside the request's stores now falls back to
those stores; an id that is one of them still narrows the search to it.
A failed pass-through call logged the httpx traceback, whose message
quotes the upstream URL with the provider API key in its query string,
into the spend log's error information and into every failure callback.
The error information built for logging now redacts its traceback and
error message, and the traceback is redacted once before the failure
callbacks receive it.
The test-quality gate counts every patch of a litellm internal against a ceiling, and the three patches this test needs pushed it over. The callback imports increment_spend_counters, update_cache and proxy_logging_obj from proxy_server inside its own body, so there is no seam to inject fakes through; every other test in this file uses the same three patches for the same reason
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The proxy cost callback zeroed response_cost whenever cache_hit was true. That rule dates from Jan 2024 when it was the only place cache hits were priced. The logging layer has priced the LLM share at 0 on a cache hit since Aug 2024, and since guardrail cost joined the standard logging payload the proxy-side zeroing has thrown away a real provider charge: a pre_call guardrail runs before the cache is consulted, so a cached response still cost whatever the guardrail billed. Drop the redundant zeroing so the payload's response_cost, which is already LLM 0 + guardrail cost, reaches spend logs, daily tables and budgets untouched
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The Anthropic and Together AI /v1/messages streaming tests required at
least two content_block_delta events. How many deltas a reply is split into
is the provider's choice, and Haiku answers a short count in one or two, so
the assertion failed on provider variance with no change in the proxy: four
of the day's full runs on the PR e2e gate went red on it on 2026-09-05.
The harness now stamps when each SSE event reached the client
(StreamingResponse.stream_event_arrivals, index-aligned with stream_events,
with the clock injectable so the reader has a unit test). Both tests ask for
a reply long enough to take seconds to generate and require the first
content delta to land at least STREAM_MIN_LEAD_SECONDS before message_stop.
A relayed stream shows a lead of about two seconds. A proxy that buffered
the response delivers every event in one burst and fails every time, which
a whole-response buffering relay in front of a live proxy confirmed. The
event-grammar assertions are unchanged.
Replay hands the proxy its recorded chunks back to back, so timing says
nothing there. The assertion is gated on provider_paces_stream() and replay
proves the grammar only, which tests/e2e/CLAUDE.md now says.
* fix(mcp): scan and mask MCP tool call arguments in unified guardrails
A guardrail configured with mode pre_mcp_call was handed only a synthetic
tool definition (name plus an empty parameters schema), so it never saw the
argument values it was configured to inspect, and any rewrite it returned was
discarded. Detection could not fire and masking could not take effect, while
the applied-guardrails metadata still reported the guardrail as having run.
Pass every string leaf of the tool call arguments as texts, and fold the
guardrail's rewritten leaves back into modified_arguments, which is the channel
the MCP call path reads to decide what to send upstream. The leaf walk reuses
the json_string_leaves / with_json_string_leaves helpers the tool result path
already uses, so both directions share one bounded traversal.
Two guardrails running concurrently under run_in_parallel scan the same payload
snapshot, so each returns a full replacement derived from the original leaf.
Rewrites of the same leaf to different values are rejected rather than silently
losing one redaction; a leaf that already holds this guardrail's own replacement
is convergent and still masks, which is what the bundled content filter does
when it rewrites the arguments itself as well as through texts.
* fix(mcp): annotate guardrail argument rewrites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): isolate MCP guardrail callback state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: ratchet LIT010 budget after merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): remove duplicate Bedrock hook parameter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): fail closed when guardrail rewrites cannot be mapped to MCP arguments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): patch the guardrail translation mappings cache where staging now keeps it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): serve Prometheus /metrics from a separate process via --prometheus_metrics_port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format prometheus_metrics_server
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): fail fast when the separate metrics server cannot start and force the multiproc dir whenever it is enabled
- wait for the child's /health before starting uvicorn; raise a ClickException if it exits first (port in use)
- create PROMETHEUS_MULTIPROC_DIR whenever --prometheus_metrics_port is set, so DB-configured prometheus callbacks work
- honour lowercase prometheus_multiproc_dir; validate the port before spawning
- cover main() entry point, readiness, bind failure and wildcard-host probing in tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): pin metrics-server readiness to the child pid so another service on the port cannot pass the health check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): probe metrics-server readiness through the shared HTTPHandler instead of bare httpx.get
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): serve only /metrics on the prometheus metrics port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): validate metrics server CLI args with pydantic instead of typing.cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): satisfy metrics server lint gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Anthropic now returns the 64-token 'count to 20' reply in one to three content_block_delta events, measured directly against api.anthropic.com and through proxies at 7672399 and 49a1145 alike, so the incrementality assertion (at least two deltas) failed in litellm-e2e builds 125, 130 and the 278 rerun with no proxy change behind it. A 'count to 100' reply at max_tokens 400 arrived in five to fifty deltas across every measured run
Custom code guardrails could only allow(), block(reason) or modify(). This adds flag(reason, metadata={}) which lets the request or response through unchanged and records a guardrail_flagged entry carrying the guardrail name, configured mode, evaluated input_type (request or response), reason and structured metadata. The new status is threaded through the request-level guardrail_status aggregation, the Guardrails Monitor rollup (flagged_count), Request Logs (action=flagged, most severe phase wins when a guardrail runs pre and post call) and the Request Logs detail view in the dashboard, which now renders FLAGGED with warning styling instead of falling into FAILED.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): cover presidio post_call, tool_permission, and weave logging cells
Five registry cells in Logging & Guardrails had no covering test. Each one now
has a live scenario read back from the real destination:
- guardrail.presidio.post_call.masks: an output-scoped Presidio guardrail
anonymizes the PII the model repeats back. The prompt also asks for the
address's local part, which Presidio does not mask, so one response proves the
model saw the raw address (no pre-call masking) while the address itself comes
back as <EMAIL_ADDRESS>
- guardrail.tool_permission.pre_call.blocks / .allows: an allow-list of one tool.
A request declaring an unlisted tool is rejected 400 naming it; a request
declaring the permitted tool is served and carries
x-litellm-applied-guardrails, so the allow half cannot pass by the guardrail
never running
- logging.niche_integrations.success.logs_spend / .failure.logs_spend: a
key-scoped weave_otel callback delivers to the real Weave project, read back
through Weave's query API. Success asserts exactly one call whose
llm.response.cost equals the x-litellm-response-cost header; failure asserts
one ERROR-status call naming the provider exception and carrying no cost
Logging & Guardrails coverage goes 24/59 to 29/59. No registry rows are added.
* test(e2e): make the tool-permission allow case deterministic and scope the Weave read-back
Review follow-ups on the coverage PR.
- the allow scenario forced the outcome to depend on whether the model felt like
calling an optional tool, and checked for the tool name as a substring of the
whole body, which a prose mention would satisfy. It now sends
tool_choice="required" and asserts the parsed response carries exactly one tool
call, for the permitted tool
- the Weave read-back queried the newest 200 calls of a shared project and
filtered client-side, so busy traffic could push the target out of the window
and read as a delivery failure. The query now scopes server-side to the
litellm_request op and to calls started after the request, and pages through
the window with offset
- the reader builds its results as tuples instead of accumulating into lists
Also unblocks the lint gate: `basedpyright tests/e2e` runs only on PRs that touch
tests/e2e, and it has been failing on staging for three FakeItem arguments in
test_junit_properties.py. The stand-in now goes through one typed adapter that
says why, so the gate is green without touching junit_properties.py itself.
* test(e2e): scope the presidio post_call guardrail to email and phone
Running the suite three times in a row caught a real flake: Presidio's broader
recognizers sometimes claim the email's local part as an NRP entity, so the
answer came back as `<NRP>\n<EMAIL_ADDRESS>\n<PHONE_NUMBER>` and the assertion
that the raw local part survives failed. That token is what tells output masking
apart from input masking, so it has to survive.
The post_call guardrail now registers pii_entities_config for EMAIL_ADDRESS and
PHONE_NUMBER only, which is also the narrower thing the scenario means. Verified
against the exact marker that failed, plus two others.
* test(e2e): mark weave logging cells stage red
* test(e2e): use per-test stage red skips for the weave logging cells
* feat(mcp): warn when an oauth2_id_jag server outruns the SSO provider's assertion capture
Only the generic OIDC login path captures the IdP id_token that an oauth2_id_jag MCP
server spends as its RFC 8693 subject token. Under Google, Microsoft, SAML or no SSO at
all, registration succeeds and then every ID-JAG credential resolution fails for every
user, with nothing in the logs, the config or the API response to say why.
Report the mismatch from the two places it is knowable: when an oauth2_id_jag server is
created or updated through the management endpoint, and at SSO callback time when a login
hands the arm nothing while such a server is registered. Provider selection mirrors the
callback's precedence, so a generic client id sitting behind GOOGLE_CLIENT_ID does not
clear the warning.
* test(sso): update merged CLI diagnostic patch target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): warn about the ID-JAG capture gap for config-declared servers and on the SSO debug page (#39350)
* feat(sso): surface the ID-JAG capture gap on the SSO debug page
/sso/debug/callback is where an operator lands when they are already trying to work out
why ID-JAG is failing, so the reason belongs on it. The annotation appears only when the
active SSO provider captures no identity assertion AND an oauth2_id_jag server is
registered for that gap to break; a deployment without both renders the page it rendered
before, byte for byte. Only the provider name and the remedy are rendered, never a
configured value, and an unreachable MCP table costs the page its annotation rather than
the page itself.
The payload carries the one mutable-ok in this work. Conditionally including a member of a
JSON document has to construct a mapping, and the rejected alternatives are recorded on the
helper so the next reader does not rediscover them.
Held out of the diagnosability PR deliberately: that PR is already reviewed and green, and
this surface ships with the remaining config-load warning as one follow-up.
* feat(mcp): warn at config load when an oauth2_id_jag server outruns the SSO provider's assertion capture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(sso): trim comments on the ID-JAG debug page diagnostic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): clean up merged imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): satisfy type discipline for diagnostic payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(sso): keep the optional ID-JAG payload member on one line for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): use Python 3.10-compatible assert_never
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): keep the ID-JAG capture-gap diagnostic out of the unauthenticated debug page
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(sso): inject the retention check and log via caplog so the ID-JAG tests pass the test-quality gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(sso): keep the debug-page outage test on the capture-gap path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sso): annotate the retention check type alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): renew the stored SSO identity assertion behind ID-JAG
The oauth2_id_jag arm asserts the id_token captured at the user's last
interactive SSO login, and nothing ever renewed it, so an agent holding a
brokered LiteLLM key could act for that user only until that token's exp.
The assertion already carried the IdP refresh token beside it; this
redeems it.
RefreshingSSOAssertionStore wraps the database reader and satisfies the
same protocol, so the egress arm is unchanged. Renewal is lazy and
single-flighted per user through the same RefreshCoordinator the
authorization_code arm uses, since an IdP that rotates refresh tokens
treats two concurrent redemptions as replay. A refusal leaves the expired
assertion in place so the reader still challenges the user; an
unreachable IdP surfaces as a store outage instead.
* fix(mcp): let a cross-replica loser settle the SSO assertion renewal itself
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): satisfy type discipline lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ci): rerun checks after docs main added the missing router setting row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): answer a cross-replica loser retryable instead of re-electing it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): bypass stale assertion cache during renewal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: ratchet type-discipline budget after merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cloudzero): infer daily batch schema from every row
pl.DataFrame defaults to inferring column types from the first 100 rows,
so a day whose batch starts with more than 100 rows missing team_alias,
api_key_alias or user_email typed that column as Null and then raised a
ComputeError on the first row that had a value, failing the whole export
with a 500 and sending nothing.
Pass infer_schema_length=None when rebuilding each day's DataFrame, the
same guard the usage query already uses.
* test(cloudzero): cover late tag schema inference
Exercise the CloudZero resource tag field after a long run of missing values so a finite inference window fails the regression test.
* fix(cloudzero): preserve late resource tags
* style(cloudzero): remove redundant test comment
* fix(cloudzero): infer daily batch schema from every row
pl.DataFrame defaults to inferring column types from the first 100 rows,
so a day whose batch starts with more than 100 rows missing team_alias,
api_key_alias or user_email typed that column as Null and then raised a
ComputeError on the first row that had a value, failing the whole export
with a 500 and sending nothing.
Pass infer_schema_length=None when rebuilding each day's DataFrame, the
same guard the usage query already uses.
* test(cloudzero): cover late tag schema inference
Exercise the CloudZero resource tag field after a long run of missing values so a finite inference window fails the regression test.
* fix(router): coordinate async and sync failure handlers at remaining router call sites
Five router failure paths still scheduled logging_obj.async_failure_handler
as a task while starting logging_obj.failure_handler on a raw thread, so
both handlers mutated the same logging object concurrently. Route them
through dispatch_failure_handlers like the streaming paths already do.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): wait on the real logging executor and justify the callbacks global patch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): submit sync failure handler even when the dispatch task is cancelled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): justify the executor submit patch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_no_linear_scans_in_router: #39674 renamed heuristic_v2_router_limit_violation
to auto_router_capability_violation, so the allowlist entry stopped matching and the
same admin-only scan tripped the static check. Rename the entry to follow it.
tableScrolling.spec.ts: 9ba6cab889 (LIT-4738) gave the Tags and Model Hub tables
client-side pagination at 25 rows, so the 40 seeded rows no longer render on one
page. Select 50 rows per page before counting, as the Logs case already does.
