* feat(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing
AWS's ApplyGuardrail API rejects requests whose content exceeds the
account's per-request "maximum input size in text units" quota with a
400 ValidationException. That cap is account/region/policy-dependent
and cannot be predicted from config, so it can only be reacted to.
_make_apply_guardrail_request now tries the whole-content call first
(no behavior change for requests that already fit). On a too-large
ValidationException it bisects the flat content list and retries each
half sequentially, recursing until every piece fits or cannot be split
further, then merges the per-chunk responses (action, assessments,
outputs, usage) into one so callers cannot tell chunking happened. A
real guardrail block on any (sub-)chunk still raises immediately.
Contextual-grounding requests are never chunked: grounding scores the
response holistically against the whole reference source, so
fragmenting it would produce misleading scores.
Each chunk call also gets a small exponential backoff retry on AWS
ThrottlingException (429), since chunking increases the number of
per-second API calls and can trade a 400 for a 429.
All new state is local to a single request's call stack (no shared
cache, no cross-process coordination), so this is safe for
single-pod, multi-pod, and cache-less LiteLLM proxy deployments alike.
* fix(guardrails): address Bedrock ApplyGuardrail chunking review feedback
Fixes three issues flagged in review of the chunking fallback: a single
oversized content item couldn't be split (only list-length bisection was
supported), a chunked request that got recovered still logged a stray
failure telemetry entry alongside the real outcome, and flattening chunk
outputs without positional bookkeeping could misalign masked text onto
the wrong original message once a chunk had nothing to mask.
* test(guardrails): add regression test for multi-level Bedrock guardrail chunking
Confirms the too-large bisection recursion isn't capped at a single split:
a payload that is still oversized after the first halving keeps splitting
until every piece fits, converging on however many chunks it takes rather
than only ever producing two.
* fix(guardrails): hybrid bin-pack+bisection chunking, whitespace-safe splits
Rework Bedrock ApplyGuardrail chunking from pure reactive bisection to a
hybrid strategy: bin-pack content into fixed-budget batches up front as
the fast path, falling back to the existing recursive bisection only for
a batch AWS still rejects as too large. Avoids paying O(log n) round
trips on every oversized request when a single pass would do.
Also switch single-item text splitting from a raw character midpoint to
the nearest whitespace boundary, so a fragment never starts or ends
mid-word. Closes the accidental-severing case from review; the residual
gap (a multi-word denied phrase deliberately straddling the boundary) is
documented as an accepted limitation, since fixing it would require an
overlap window reconciled against masked output with no documented
length-preservation guarantee from AWS.
* chore(ui): regenerate dashboard API types
* fix(guardrails): don't retry an oversized Bedrock guardrail call as a throttle
AWS reports an ApplyGuardrail request that exceeds the per-request
text-unit cap as a ThrottlingException (429), not only as the documented
ValidationException (400). Verified against a live guardrail with an
active content-filter policy: a 3273-text-unit request comes back as
"Input text size (3273 text units) exceeds the maximum allowed (1000 text
units) for the content filter policy (Classic tier)".
The throttle retry keyed off status 429 alone, so every oversized chunk
burned the full backoff-retry budget - each attempt a billed AWS call
preceded by a sleep - before the bisection fallback got a chance, at every
level of the recursion. A size error is not transient; re-posting the same
content can never succeed. It now short-circuits straight to bisection.
Also rename _is_input_too_large_validation_error to
_is_input_too_large_error (it never keyed off the status code, and the
error is not always a ValidationException), correct the docstrings that
asserted a 400, and log at warning level when a split happens so the
recovery is visible without --detailed_debug.
* Revert "chore(ui): regenerate dashboard API types"
This reverts commit ebf8ba2fd57f13bccf7aa6c5dfcac41c74db1ed9.
* fix(guardrails): group all fragments of one item and stop double-logging
Two defects found in review, both invisible to the existing tests.
Fragment grouping assumed a split content item always produces exactly two
adjacent fragments. That holds for one bisection level but not two: an item
split twice yields four fragments, which were regrouped in fixed pairs into
two output entries for a single message. Since masking walks the merged
outputs by a running index across the original, unchunked message list, that
message was written back truncated to its first half and every later message
shifted. Fragments now carry the size of the group they belong to, so any
number of them collapse back into exactly one output entry.
Telemetry was also double-counted. AsyncHTTPHandler.post calls
raise_for_status(), so every non-200 from Bedrock reaches _sign_and_post's
error path, which logged guardrail_failed_to_respond before re-raising as an
HTTPException that the consolidating caller then logged again. A request
recovered by chunking reported one failure per rejected attempt plus a
success. The ApplyGuardrail path now opts out of that per-attempt logging,
since it owns consolidated per-request logging; the connection-level branch
still logs, as nothing else records it.
The existing tests missed both because their mocks return a non-200 response
object, while the real client raises. Added a helper that raises a genuine
httpx.HTTPStatusError so these paths are covered the way production hits
them, plus a case asserting an unrecoverable failure still logs exactly once
rather than zero times.
* refactor(guardrails): move Bedrock chunking rationale into docstrings
The chunking work explained itself with inline comment blocks, which this
repo's conventions do not want. Folded that reasoning into the docstrings of
the functions it describes and dropped the comments, including the
module-level constant blocks and the test-file banner.
No behavior change. The banner also claimed AWS rejects an oversized request
with a 400 ValidationException, which live testing disproved, so removing it
drops a stale claim as well as an internal ticket reference from a public repo.
* feat(guardrails): match AWS default chunk budget and make it configurable
ApplyGuardrail's default quota is 25 text units, roughly 25,000 characters,
per second. Chunking has to respect that throughput limit rather than just the
per-request size, otherwise splitting an oversized request trades a size error
for a throttle. The budget now defaults to 25,000 to match that default for
every user, up from an arbitrary 20,000.
Accounts with raised quotas can spend fewer calls by setting
chunk_budget_chars on the guardrail. A value AWS still rejects as too large is
bisected automatically, so an over-large setting costs an extra round trip
rather than failing the request.
* fix(guardrails): never split a Bedrock text into an empty fragment
_nearest_whitespace_split_index could return len(text) when the only space at
or after the midpoint was the final character, so the first fragment came back
identical to the text AWS had just rejected as too large and the second came
back empty. AWS rejects the unchanged fragment again, and each retry re-splits
it into the same fragment, so an oversized single item shaped like a long
unbroken token with one trailing space exhausted the stack with a
RecursionError instead of scanning or surfacing Bedrock's error.
Candidate boundaries that would leave either side empty are now discarded, and
the raw midpoint is used when none remain. The midpoint is always safe because
_split_bedrock_content only calls this for text of at least two characters.
* style(guardrails): move chunking rationale out of comments and into docstrings
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Restores the source changes intended for a08e4cf309, which landed with only the
test. Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* style(guardrails): sort the constants import into the first-party block
* refactor(guardrails): bring the Bedrock chunking path under the LIT lint budgets
Annotates never-rebound locals with Final, replaces the retry counter and the two
branch-assigned locals with single bindings, and moves the internal chunking chain
to Sequence parameters and tuple returns. Collections that reach the logged payload
stay lists on purpose: redact_nested_match_and_regex_keys only traverses dict and
list, so a tuple would carry PII past redaction. The remaining constructions are
contract-bound and carry inline reasons
* fix(guardrails): keep the pre-chunking contract for failures reported inside a 200
Reverts the 500 this branch introduced for an AWS 200 whose body carries an
Output.__type exception marker: the request proceeds as it did before chunking
existed. The logged status is now derived from the merged response instead of
being hardcoded to success, so that shape is still reported as
guardrail_failed_to_respond. The consolidated failure logger also goes back to
logging a dict rather than a bare string, matching both the pre-chunking code and
the InvokeGuardrailChecks path in this file
* docs(guardrails): correct the docstring for failures reported inside a 200 body
The raise was reverted, so the docstring no longer describes the code. Records that
the request proceeds by design and points at LIT-5338 for closing the fail-open path
behind the existing unreachable_fallback setting
---------
Co-authored-by: spencer-burridge <265588760+spencer-burridge@users.noreply.github.com>
* feat(proxy): return per-group model provenance on /team/info
/team/info now carries access_group_details, one entry per resolved access
group with its id, name, and model list, so the UI can attribute each
inherited model to the group granting it. The batch resolver returns the
access group rows keyed by id instead of a stringly dict of lists, and the
team member budget helper returns a copy instead of mutating its parameter.
Type discipline and basedpyright budgets ratchet down accordingly.
* feat(ui): allow group-only teams and show model provenance on hover
Team create and edit no longer require a model selection: an empty
selection is saved as the no-default-models sentinel, never as a bare
empty list, since an empty team model list means unrestricted access.
The team info Models card now renders every badge with a hover tooltip
naming how the team got that model: directly, via named access groups,
or both, and group-granted badges stay visible when the direct list is
empty or a sentinel.
* refactor(proxy): dedupe access group ids and return copies instead of mutating
Duplicate access_group_ids no longer amplify the /team/info response: ids
collapse order-preserving before provenance is built, pinned by a regression
test. The resolver returns a model_copy rather than mutating its parameter,
and the team create call sends a new object instead of reassigning
formValues.models. Budgets ratchet down further with the mutation removal.
* fix(proxy): resolve entity access groups in the model listing endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): reuse the fetched team object when listing models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover key-level access group resolution in model listing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(auto-router): track turns per complexity tier (LIT-5302)
Stamps complexity tier at decision time (rollup never re-derives from routed
model, since tier->model mapping is mutable config). Records per-tier turn
counts in LiteLLM_AutoRouterSession.tier_turns (jsonb), rolls up per router
in benchmarks SQL via jsonb_object_agg, returns on AutoRouterBenchmarkGroup
for dashboard turns/share metrics.
Addresses Greptile/Bugbot findings:
- Missing _SessionAggRow.tier_turns field: added with field_validator to
parse jsonb text cast and handle NULL. Would 500 every benchmarks read.
- Missing ::text cast on tier parameter: Postgres fails type inference on
parameterized CASE/IS NULL without explicit cast. Added to all usages.
- Docstring false claim (only complexity routers produce tiers): quality
router stamps numeric tier '1'/'2'/'3'. Per-type grouping in SQL prevents
cross-contamination. Rewrote docstring to clarify isolation.
- Comment convention violations: stripped per CLAUDE.md rule.
- Test gaps: 8 unit tests for extraction/validation/aggregation, 7 behavior
tests for SQL semantics against real Postgres. 12 mutations killed.
Fixed fragile complexity_router test that broke on nested function calls.
No API change; extends existing GET /auto_router/benchmarks response only.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-router): address review findings on tier turns tracking
- Guard router_type update so a mid-session reconfigure can't pool
foreign tier names into tier_turns
- Keep pinned turns attributed to the tier that actually serves them
- Drop stray -- AlterTable comment from hand-written migration
- Drop the now-unnecessary ::text/json.loads round-trip; prisma
already returns tier_turns as a parsed dict
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-router): satisfy type-discipline lint gate
- tier_turns fields: dict[str, int] -> Mapping[str, int] (LIT001,
mutable collection in annotation); these are read-only after
construction
- _summed_agg_row: {} -> MappingProxyType({}) (LIT002, mutable dict
literal)
- default-fallback branch: replace the reassigned-without-Final
fallback_tier with a Final default_model_first flag and a single
ternary assignment (LIT010)
Verified locally: type_discipline_gate.py, ruff_strict_gate.py, and
type_check_gate.py all pass against the litellm_internal_staging
merge-base; full test_complexity_router.py (374), auto_router
management-endpoint tests (26), db-layer rollup tests (31), and the
live-Postgres proxy_behavior rollup suite (17) all pass.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
PR #36020 changed AgentRegistry.get_agent_list to return tuple[AgentResponse, ...], and PR #35163 added a test asserting the result equals []. Both were green on their own branches and only collided once they were both on litellm_internal_staging, so proxy-server has been failing on every PR since with 'assert () == []'.
Nothing user-facing was wrong: get_agents only iterates the result and rebuilds it with comprehensions, and FastAPI serializes a tuple to the same JSON array. The test expectation was simply stale, so it now compares against ().
The get_agents local was still annotated list[AgentResponse] while two branches assign the registry tuple straight into it, so it widens to Sequence[AgentResponse]. That covers both the tuple and the list branches without pretending the value is mutable.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): deny when agent grants resolve to nothing
`get_allowed_agents` returned a plain list where the empty value meant both
"this caller was never restricted" and "this caller's grants resolved to
nothing". Downstream read either as allow-all, so a key restricted to one
agent inside a team restricted to another reached every agent on the proxy,
and an access group that resolved to no agents did the same.
Replace it with `resolve_agent_access`, returning a tagged
UnrestrictedAgentAccess | RestrictedAgentAccess. Only a caller with no grant
anywhere is unrestricted; an empty restricted set denies. Access group lookup
failures now propagate to the key/team resolvers so a DB error still fails
open exactly as before, while a group that genuinely resolves to nothing
denies.
* style(proxy): drop redundant comments from the agent access match
* fix(proxy): derive config agent ids from agent_name so grants survive secret rotation
Config-defined A2A agents were identified by a sha256 of the whole resolved
config entry, secrets included, so rotating an os.environ secret re-minted the
agent_id on restart and orphaned every object_permission.agents grant while
grant-less keys kept access (LIT-5144). The id now hashes only agent_name, and
the old full-entry hash is kept as a legacy alias: permission checks,
GET /v1/agents filtering, spend and key attachment, and public_agent_groups all
normalize legacy ids so pre-upgrade grants keep working
* fix(proxy): persist stable agent ids into stored grants at startup
The runtime alias only translates a legacy grant while the current config
still hashes to it, so a secret rotation after upgrading would orphan the
grant, and an orphaned grant intersecting a stable team grant collapses to
an empty list that downstream reads as allow-all. Rewriting the stored ids
once at boot removes both. This cannot be a SQL migration because only the
running proxy can recompute the legacy hash from resolved config secrets
* fix(proxy): make the grant id migration a compare-and-swap
A grant edited between the migration's read and write kept the stale
snapshot. The update now predicates on the agents array read at scan time
via update_many, so a concurrently modified row is skipped and the runtime
alias covers it until the next boot retries
* fix(proxy): retry the grant id migration and stay within the LIT002 ceiling
The one-shot startup task now retries up to three times with a short delay
so a transient DB error at boot cannot leave a legacy grant unmigrated
until an operator's next restart is the rotation itself. The new list
constructions in the migration and the alias-expanded agent id lookups are
tuples now, keeping the branch under the mutable-collection budget
* fix(proxy): count compare-and-swap misses in the grant id migration
migrate_legacy_grant_ids now returns rewritten and missed counts from the
update_many results instead of reporting scanned rows as migrated, and the
startup task retries while any rows remain unmigrated, not just on errors
* fix(lint): clear basedpyright budget breaches in agent id aliasing
* feat(proxy): add apply_user_budget_to_team_keys opt-in
PR #32005 made a user's personal max_budget apply to their team-scoped keys
too, and PR #35271 reverted the whole thing (behavior plus the
skip_user_budget_on_team_key opt-out) because that flipped the default for
everyone. This brings the behavior back the other way round: default is
unchanged, and general_settings.apply_user_budget_to_team_keys opts a
deployment into charging the key owner's personal budget on team keys.
The flag reaches all three personal-budget gates so an opted-in deployment
enforces consistently: the read-time check in common_checks, the optimistic
reservation counter in _get_budget_counters, and the _PROXY_MaxBudgetLimiter
pre-call hook. It is also in the /config/list allowed args and, unlike the
reverted flag, in the _update_general_settings propagation allowlist, so the
Admin UI General Settings toggle actually takes effect at runtime; an explicit
YAML value still wins over the DB value on reload.
get_config_list's allowed_args moves to a module-level frozen mapping of
field name to type string, dropping 18 LIT002 violations and rebuilding one
less dict per request.
* style(proxy): drop explanatory comments from the budget flag paths
* fix(proxy): invalidate cached project object on /project/update and /project/delete
The auth path reads projects cache-first via get_project_object with a 60s
TTL and no freshness check, but no project write endpoint ever evicted the
project_id:{id} cache entry. A project cached before /project/update added a
model allowlist kept an empty models list in cache, so _run_project_checks
skipped can_project_access_model and project-bound keys could call team
models outside the project allowlist until the TTL expired. The same
staleness applied to blocked status and budget fields, and /project/delete
left the deleted project enforceable from cache.
Evict the cache entry after the DB write in update_project and
delete_project via a shared delete_cached_project_object helper, with the
cache key derivation shared with get_project_object.
* fix(proxy): broadcast project cache invalidation to all workers and make eviction best-effort
Single-worker eviction leaves every other worker serving its in-memory copy
of the mutated project until the 60s TTL expires, so a project allowlist
change was still bypassable on multi-worker deployments. Add a coordination
Redis pub/sub channel (litellm_proxy.auth_cache_invalidation): project
eviction publishes the cache key and a per-worker subscriber deletes the
local in-memory entry, with the next auth read refetching from the DB.
Subscriber starts on any deployment with a coordination Redis and falls back
to the TTL when none is configured.
Also wrap the eviction in a best-effort catch: the DB write has already
committed when eviction runs, so a cache backend error must not turn a
successful update into a 500 or abort the remaining ids in /project/delete.
* fix(lint): sort auth cache invalidation import and suppress best-effort shutdown catch
The strict-budget gate flagged the new import block as un-sorted (I001) and
the broad except in stop_auth_cache_invalidation_subscriber (BLE001); the
catch is intentional since a failing stop must not break proxy shutdown, so
it carries a named suppression instead of counting against the budget.
* fix(proxy): return the real status code when a credential update is rejected
update_credential ended its except clause with 'return handle_exception_on_proxy(e)'. Returning the exception makes it the response body, so FastAPI answers 200 and every rejection on this route reads as a successful write to any caller that checks the status; the admin dashboard's API client is one. Patching a name that does not exist answered 200 with the real 404 buried in the body.
The sibling handlers in this file already raise. The route had no test coverage, which is why it survived.
* Update tests/test_litellm/proxy/credential_endpoints/test_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
create_a2a_client took the raw client off a process-wide cached handler and
called headers.update() on it, then leaned on folding the header set into the
cache key (through the unrelated disable_aiohttp_transport field) to keep one
caller's credentials away from the next.
Per-caller headers now ride with each request through the a2a SDK's call
context, and the agent card fetch gets them through resolver_http_kwargs, so
the shared client is never written to and its cache key no longer varies by
header set. Since the proxy puts a fresh trace id in every request's headers,
that key previously changed on every call, giving each request its own httpx
client and flushing the 200-entry client cache that every other provider
shares. All A2A callers on one timeout now reuse a single pooled client.
Sharing that client also means sharing its httpx cookie jar, which httpx fills
from every Set-Cookie and replays on any later request to a matching domain, so
one agent's session cookie would arrive at another agent on the same host. The
pooled client now carries a cookie policy that stores and sends nothing, which
neither litellm nor the a2a SDK relies on: the SDK's auth interceptor skips
cookie-borne API keys outright.
* fix(proxy): promote caller metadata trace fields into litellm_metadata
Routes in LITELLM_METADATA_ROUTES keep the caller's metadata as a provider
passthrough field and track proxy state in litellm_metadata, which is the dict
the logging integrations read. The caller's trace_id, session_id, trace_user_id
and trace_metadata therefore never reached any callback on /v1/responses,
/v1/messages, /v1/batches or /v1/files, and mask_input / mask_output were
dropped with them so a caller asking for redaction had their prompt logged in
full.
Promote an explicit allow-list of those fields from the requester_metadata
snapshot into litellm_metadata, never overwriting a value already set so
header-derived ids keep precedence. Trace-mutation controls (existing_trace_id,
update_trace_keys) and trace_public are deliberately excluded: langfuse applies
them to an arbitrary caller-chosen trace with no ownership check. tags is
excluded because per-tag budget enforcement runs earlier, at auth time.
This covers providers with a native Responses API config. Providers reaching
/v1/responses through the chat-completions bridge need the companion change to
get_litellm_params.
* ci: retrigger workflows
* warn at startup when a proxy-wide budget is set but no DB is connected
litellm.max_budget is only enforced via DB-loaded global spend, so a DB-less proxy silently ignores it. Log a one-time startup warning.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): inject max_budget into DB-less budget warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover DB-less budget warning startup call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): pin DB-less budget warning call site
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): stabilize budget warning call-site pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: tin <tin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Prompt Security drops tool and function rows unless check_tool_results
is on, so it now reports scan-only support from that setting and the
registry refuses the pairing at boot. Pairing scan_only_tool_results
with skip_tool_message_in_guardrail excludes every message, so guardrail
initialization now rejects that combination too.
Replaces the hand-rolled recording double with AsyncMock so the awaited team ids
come from await_args_list instead of a mutated list, and annotates the response
factory now that SearchToolInfoResponse is imported at module level.
The aggregated route was missing from LiteLLMRoutes.self_managed_routes
while its paginated sibling /user/daily/activity was listed, so auth
rejected every internal user with a 401 before the handler ran. That
route backs the default "Your Usage" view in the dashboard, which left
the main Usage page broken for non-admin users.
The handler already self-scopes: it checks admin view first, then falls
back to require_caller_user_id_for_non_admin, defaults a missing user_id
to the caller's own, and returns 403 when a non-admin asks for someone
else's data. Listing the route restores reachability without widening
what a caller can read.
check_route_access matches exactly (plus explicit wildcards), so the
parent entry never covered the /aggregated sub-path.
Every Admin UI session key is stamped with the reserved team id
`litellm-dashboard`, which never has a row in LiteLLM_TeamTable, so the
team lookup in _filter_visible_search_tools raised 404 and the endpoint
returned 500 for every non-admin dashboard session.
Skip the lookup for that sentinel and scope the caller by its key-level
allowlist alone, matching how MCP and agent permission checks already
treat it. A real team id is still resolved, and a genuine lookup failure
now surfaces with its own status instead of being masked as a 500.
* fix(proxy): include today's UTC bucket when a daily activity range ends at the caller's current day
* fix(proxy): gate the current-UTC-day extension behind an opt-in param sent by the cost optimization dashboard
* fix(ui): label cost optimization savings dates as UTC days
An adversarial review of the previous commit found annotations that
described what the code wished were true rather than what flows through.
A false annotation is worse than the Any it replaced, since it launders a
wrong assumption past the type checker.
- purview: `_resolve_user_id` claimed every request-body value was a
Mapping, contradicting `_resolve_trusted_user_id` one method over, which
types the same argument `Mapping[str, object]`. `_should_block` claimed
every Graph response value was a sequence of str->str mappings and was
not assignable from its own producer's return type.
- cato: `_CatoAnalyzeResponse.required_action` was required and
non-nullable while the API returns null, as seven fixtures in the
guardrail's own suite assert. `analysis_result` had the same problem.
The streaming hook narrowed an override parameter below what
`ProxyLogging` actually passes it.
- marketplace: `_PluginRecord.manifest_json` was `str` against a nullable
column. Making it honest surfaced a latent crash, covered below.
- ownership: two functions took an attribute Protocol while their own
bodies branch on `isinstance(response, dict)`, which no Protocol can
satisfy.
- openapi generator: `paths` claimed every path-item value was an
operation, though path items also carry `parameters`, `summary` and
`$ref`.
- custom openapi spec: a TypedDict asserted a shape that the function
returns raw Pydantic sub-schemas out of. Reverted to Any, which is
imprecise but not false.
`get_marketplace` did an unguarded `json.loads` on the nullable
`manifest_json` inside an `except json.JSONDecodeError`, which cannot
catch the TypeError a NULL raises, so one NULL row 500s the endpoint. It
now skips the plugin like the file's other two read sites already do, with
a regression test that fails without the guard.
Where honesty cost precision, precision lost. `_should_block` went back to
its original signature entirely: the narrowing needed to type it turned a
fail-closed DLP control fail-open, because the TypeError it used to raise
on a malformed response reached `except Exception` and became a 400.
When the batch cost poller found a batch in a terminal failed, expired, or
cancelled state it wrote the provider response straight to the managed object
table, so the stored blob kept raw provider file ids and a raw batch id. Since
the row is final after batch_processed=True and the read paths only resolve
existing managed ids, every later GET /batches/{id} and GET /batches leaked
raw provider output and error file ids that clients cannot fetch through the
proxy. The terminal branch now normalizes the response with
ensure_batch_response_managed_file_ids before persisting, minting managed ids
under the batch owner's identity
POST /batches/{id}/cancel had the same gap: it called update_batch_in_database
without the caller's auth context, so a cancel response that already carried
provider file ids could never mint managed ids. The endpoint now forwards
user_api_key_dict
Terminal batch retrieve could return the raw provider output_file_id, which
skips managed-file ownership checks on /v1/files/{id}/content and lets any key
on the proxy download another user's batch output.
Retrieve now registers the missing managed-file row before responding, and
attributes ownership to the durable batch owner rather than the retrieving
caller, so output and error ids always come back as unified managed ids.
Fixes#33989
* fix: rebuild models_by_provider in add_known_models so cost map reloads reach wildcard expansion
* fix: refresh models_by_provider in place so captured references survive reloads
A connection test that redirects the destination already leaves the configured
credentials behind. It kept litellm_credential_name, which names the same stored
secrets and is resolved further down the call, so the reference is now dropped
with them. A request that sets no connection fields of its own is unaffected,
which is how the Admin UI tests a configured model.
The proxy-wide opt-in that already governs callers supplying their own
connection parameters now also governs whether a connection test may pair a
request-supplied endpoint with the configured deployment's credentials. Off by
default, which keeps configured credentials scoped to the endpoint the
configuration names; on, the previous merge behaviour is available unchanged.
A request that supplies its own connection fields describes a connection of its
own, so the configured deployment's credentials are no longer merged underneath
it. Anything the request leaves unset still comes from the configuration, so
naming a configured model and testing it as configured is unchanged, and adding
a second deployment for an already-configured name works as before.
Replaces the earlier outright rejection, which also refused requests that
supplied a complete connection of their own.
When a connection test names a model that resolves to a configured deployment,
that deployment's routing and credential parameters are authoritative. A request
supplying a complete connection of its own is unaffected.
BREAKING CHANGE: /health/test_connection no longer lets a request replace the
routing or credential parameters of a configured model it names. Supply the full
connection parameters instead of naming a configured model.
