* fix(proxy): fail parked DB lookups at a deadline and flip readiness while they stall
Under a load burst with a slow authentication database every request parked inside
the pod with no deadline while /health/readiness kept answering 200 (its own ping
gets a fresh connection), so the load balancer kept sending traffic until the pod hit
its memory limit, and the parked requests completed against the provider minutes
after every client had hung up
Every pre-request read (key, team, user, end user, budget, membership, organization,
object permission, jwt mapping, project, proxy budget, spend counter reseed) now runs
under one deadline, PROXY_DB_LOOKUP_DEADLINE_SECONDS (default 10 s). A lookup that
hits it fails the request with the existing 503 "authentication database is
temporarily unreachable" answer, honours allow_requests_on_db_unavailable, and never
triggers the transport reconnect (the transport is fine, the query is slow), which is
what turned the repro's stall into "too many clients". Writes stay unbounded
A deadline hit marks the pod stalled for PROXY_DB_LOOKUP_STALL_WINDOW_SECONDS
(default 30 s, 0 disables), during which /health/readiness answers 503 with
"db": "stalled" behind the same fail-open gate, so the pod leaves rotation before it
fills its memory. The existing litellm_in_flight_requests gauge already exposes the
parked set on /metrics
The deadline is enforced on the wall clock: bounded_db_lookup waits on the lookup
task with asyncio.wait and raises DBLookupDeadlineExceeded when the deadline passes
even if the lookup absorbs its cancellation, where asyncio.wait_for on 3.12+ would
sit on the cancelled task for as long as it takes
The failure spend-log row no longer re-runs the key and team lookups when the
failure itself is a database connection or deadline error, so a request that hit
the deadline is answered after one deadline instead of two
* fix(proxy): bound the spend counter gate wait and narrow the stalled lookup shortcut
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the global spend lookup on the prisma client handle
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): list key and team model aliases in GET /v1/models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep alias listing helpers within the type discipline budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover alias rows on GET /v1/models and /v1/models/{id}
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply team then key aliases like chat completions and keep the alias as the retrieved id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply key aliases twice like chat completions and skip only malformed alias entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply the global model_alias_map between the key alias passes like chat completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): list only the caller's own aliases and never rewrite a listed model id on retrieval
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format model_info alias lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): hide undiscoverable names from model retrieval so an alias named like one resolves to its target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep undiscoverable models retrievable by id while excluding them from the alias guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): pass an immutable name sequence into the model_info alias guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the model list alias test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): annotate the new alias listing test fixtures and helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): bound concurrent S3 uploads per flush and add opt-in JSONL batch files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): keep failed uploads queued, parse env-backed flags, add integration coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(s3_v2): type test helpers and honor constructor bound when config value is null
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(s3): annotate required casts for the type-discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): keep tenant prefixes, stable retries and cold storage safety in batch file mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): cover root-level batch file keys for codecov patch target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): audit matrix across chat, messages and responses surfaces with sink faults
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): read sink objects under the lock in the audit cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): rebind the retry queue instead of slicing in place
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): ignore stray non-POST requests in the surface upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): keep fake upload state on the fake client instead of nonlocal counters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* test(integration): reproduce passthrough upstream error body missing from logs and spend row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(passthrough): log upstream 4xx/5xx error bodies and carry them into the failure hook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(error_normalization): let the passthrough prefix win over upstream body text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(passthrough): honor message redaction for upstream error bodies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(passthrough): bound the upstream error body read and sanitize it before logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(passthrough): use the Sequence import directly in the allowed-routes cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(passthrough): rechunk the upstream error stream so the preview read stays bounded
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit matrix for passthrough upstream error visibility
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(passthrough): drop the restating docstring on the upstream failure logger
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the retired covers markers from the passthrough error tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(passthrough): keep the upstream status when the error body peek fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(passthrough): cover the relay aclose in the mid-read failure test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(passthrough): relay decoded partial body on mid-read failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): do not requeue a daily spend batch whose commit already left for postgres
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): settle an interrupted daily spend commit from the shutdown flush instead of blocking the cancelled tick
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): burst two workers and SIGTERM during daily spend COMMIT, expect exactly once
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep deployment labels on cache-hit post_call rejections
A post-call failure on a response served from the litellm cache set no first_api_call_start_time, so the failure hook flagged it as rejected before routing and dropped the model_id and provider labels
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): read the cache hit from caching_details in the failure hook
model_call_details[cache_hit] is stamped inside the enqueued success handler, so a post-call failure can observe it too early; logging_obj.caching_details is set synchronously before the cached response returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover cache-hit guardrail reject deployment labels across endpoints, modes and chaos
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bound the worker-kill reject count by in-flight losses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert provider and model labels on the cache-hit regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): bound the exceeded budget regex so a crafted error message cannot stall the proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit normalized_error clustering on long messages with a real two worker proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): poll for the budget denial and correlate upstream 503 bursts by request marker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): replace the bounded exceeded budget regex with a linear scan that keeps the original semantics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): compare upstream error wording against the decoded message
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): tolerate a reaped worker while listing proxy children
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): stream non-Anthropic raw SSE through the post_call hook unbuffered
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(presidio): keep the pytest.raises block to a single await
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(presidio): move raw SSE format check into a helper to keep hook complexity flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): fold the raw SSE format check into the existing bytes branch to stay within the complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): decide raw SSE stream shape on a complete first frame, not a transport fragment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover presidio post_call streaming for native gemini passthrough and anthropic messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): cap first SSE frame coalescing at 64 KiB so an unterminated first event cannot buffer unbounded
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): name raw SSE passthrough in the skipped output masking warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): registry audit 2026-09-23, in-region Bedrock Claude and OpenRouter price fixes
Absorbs #42698
Co-authored-by: coldStoneSoul <shekharpachlore@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): keep registry formatting unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: use eu.amazon.nova-pro for regional pricing probe after in-region parity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: lock in-region parity for bare Bedrock Claude ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: compare every pricing field for bare Bedrock Claude parity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model-catalog): add above_32k cost fields to ModelInfo round-trip
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "fix(model-catalog): add above_32k cost fields to ModelInfo round-trip"
This reverts commit c71d5a3de3.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: coldStoneSoul <shekharpachlore@gmail.com>
* fix(proxy): document responses API request and response schemas in openapi
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): namespace colliding openapi defs instead of overwriting existing components
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): regenerate lazy openapi snapshot and dashboard schema types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): require model and input in responses schema, document event stream, fix def collision refs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark responses request fields readonly required
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reuse existing OpenAPI components when a $defs entry has the same shape
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ollama): read the JSON thinking field on non-streaming completions
Ollama's /api/generate returns reasoning in a top-level `thinking` field, but
the completion transport only looked for inline <think> tags. reasoning_content
was therefore always null, and a model that spent its whole turn reasoning
returned an empty assistant message with tokens billed.
Port the precedence the ollama_chat transport already uses: the field wins and
inline tags stay the fallback. Applied to both non-streaming paths, including
the JSON-mode text fallback. The two fields are read through a small validated
model rather than off the untyped JSON, so absent and explicitly null
`response` stay distinct exactly as before.
* fix(ollama): keep the thinking field on JSON-mode completions
The first pass read `thinking` for plain replies and for JSON-mode text that
failed to parse, but the three JSON-mode branches that succeed still dropped
it: an empty `response`, a valid JSON object, and a function-call shaped one.
A model that spent its whole turn reasoning under `format: json` therefore
still came back blank with the tokens billed.
Carry the field on all three, type the new test helper's parameters, and cover
the null and malformed `response` fallbacks.
---------
Co-authored-by: Pawan-Shahane <shahanepawan511@gmail.com>
* fix(models): sync openrouter prices from the models API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(models): allow above_32k_tokens cost fields in price map schema test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): gate disable_global_guardrails on keys and teams to proxy admins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover metadata smuggle with explicit false and UI toggle gating
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): satisfy PT017 in resend-stored guardrail flag test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): keep regenerate_key_fn under the C901 ceiling via a guardrail opt-out helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for guardrail opt-out docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): gate disable_global_guardrails on caller-sent metadata, not server defaults
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit cells for disable_global_guardrails admin gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): restore contracts.json formatting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): share guardrail opt-out helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): hide the team disable_global_guardrails switch from non proxy admins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop covers markers and bound the slow sink check to the sink delay
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): honor model_info.discoverable on the model listing endpoints
A model_list entry marked model_info: {discoverable: false} is left out of GET /v1/models (OpenAI and Anthropic shapes, scope=expand and wildcard routes included), the list path of GET /v1/model/info and GET /model_group/info for every caller without the admin view, while direct requests naming the model keep routing to it. The field defaults to None so an absent flag reads as discoverable and nothing is persisted or echoed for configs that never set it.
* fix(proxy): hide flagged team models under their public name and cover the scope=expand filter
The discoverability lookup now resolves a listed name with the caller's team context, so a team-scoped deployment marked discoverable: false drops out for that team's keys under its public name instead of failing open. The scope=expand branch is now exercised by a team admin caller, and the OCI secrets test builds a real UserAPIKeyAuth instead of a spec mock that has no pydantic fields.
* perf(proxy): resolve only candidate names in the discoverable filter
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mcp): reject duplicate MCP server names and aliases
MCP server_name and alias were unchecked at write time, so two servers
could share one tool prefix and tool routing resolved to an arbitrary
winner. Writes now run inside an advisory-locked transaction that
rejects a collision on either column case-insensitively with a 400
naming the colliding identifier, covering create, edit, connector
import and restricted-admin submission. Server reload logs one warning
per identifier already shared in the database.
Co-Authored-By: bot_apk <apk@cognition.ai>
* fix(ui): block duplicate MCP server names and aliases before submit
The create and edit forms now check the normalized name/alias against
the loaded server list (case-insensitive, spaces to underscores, own
row excluded on edit) and show a field error instead of submitting.
Structured proxy error bodies are unwrapped so a 400 no longer renders
as 'Error: [object Object]'.
Co-Authored-By: bot_apk <apk@cognition.ai>
* fix(mcp): check identifier conflicts when an alias is cleared
Clearing an alias drops the tool prefix to the stored server_name, so
that name must go through the conflict check too; an explicit alias:null
is now treated as an identifier write. Also narrows the new db tests to
behavioral assertions instead of pinning prisma where shapes.
Co-Authored-By: bot_apk <apk@cognition.ai>
* fix(mcp): treat an empty alias as a clear in conflict checks
An empty-string alias was written unchecked even though the prefix falls
back to server_name; the update path now treats any falsy alias like a
clear. The edit form likewise compares a cleared alias as empty instead
of re-checking the alias being removed.
Co-Authored-By: bot_apk <apk@cognition.ai>
* test(mcp): cover clearing an alias to an empty string
Co-Authored-By: bot_apk <apk@cognition.ai>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: bot_apk <apk@cognition.ai>
* fix(otel): root post-response service spans in their own trace linked to the request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): trim service span context docstring
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(redis): authenticate sync clusters with IAM credential providers
Signed-off-by: Silu Panda <31051721+SiluPanda@users.noreply.github.com>
* test(redis): exercise IAM cluster authentication over TCP
Run Azure and GCP regressions against a real local cluster with only
cloud token issuance stubbed. Build a checksum-verified Redis server
in the compatibility workflow and report its coverage.
Signed-off-by: Silu Panda <31051721+SiluPanda@users.noreply.github.com>
* test(redis): separate unit and cluster integration coverage
Keep the mapped test tree mock-only. Run the live cluster cases
from the existing local caching integration file, selected by
explicit node IDs in the Redis compatibility workflow.
Signed-off-by: Silu Panda <31051721+SiluPanda@users.noreply.github.com>
* test(ci): isolate workflow coverage audit fixtures
Replace the stale unrun caching-file assumption with isolated workflow
fixtures for file and node-ID selectors. Keep the unnamed-file negative
check and clarify which live caching cases remain outside CI.
Signed-off-by: Silu Panda <31051721+SiluPanda@users.noreply.github.com>
---------
Signed-off-by: Silu Panda <31051721+SiluPanda@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): serve the OpenAI models on bedrock-runtime's native Responses API
AWS serves the OpenAI models on bedrock-runtime through an OpenAI-compatible
surface at /openai/v1/responses, alongside Converse. LiteLLM had no Responses
config for the bedrock provider, so /v1/responses fell back to the Chat
Completions bridge and was translated into Converse. A realistic Codex session
does not survive that translation: its function_call / function_call_output
history becomes Converse toolUse / toolResult blocks with no toolConfig, and
Converse rejects the request outright.
Add a Responses config for that surface, opted into per model from the price-map
supported_endpoints so models without the signal keep the bridge exactly as
before. Auth is Bearer when a Bedrock API key is present, SigV4 otherwise.
Both Bedrock endpoints reject the Codex history item types agent_message,
context_compaction and local_shell_call, so the normalization bedrock_mantle
carried privately moves into a shared module and both providers use it. They are
history items, so they only bite from the second turn onward -- a first-turn
smoke test passes and hides the problem. Verified against bedrock-runtime with
global.openai.gpt-5.6-sol: additional_tools is accepted there (unlike on
bedrock-mantle) while those three types are rejected, so the two endpoints do
not share one validator and each provider opts in explicitly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(bedrock): build the Responses endpoint from the region's partition suffix
get_complete_url hardcoded amazonaws.com in an f-string, so every non-commercial
partition got the wrong host: cn-north-1 resolved to amazonaws.com instead of
amazonaws.com.cn, and GovCloud/ISO regions were wrong the same way. Defer to
BaseAWSLLM._select_default_endpoint_url, which this config already inherits and
which resolves the suffix per partition.
test_no_fstring_hardcodes_the_commercial_dns_suffix scans the whole tree, so it
caught this even though it is not one of this PR's test files. Register the
config in ENDPOINT_BUILDERS so the cn/GovCloud endpoint sweep covers this
surface from now on rather than only the f-string guard.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(bedrock): opt the gpt-6 family into the native Responses API
* fix(bedrock): drop the Responses tool types bedrock-runtime rejects
Codex sends a web_search tool on every turn. api.openai.com runs that tool
itself, and the Converse bridge dropped it silently, but bedrock-runtime's
native Responses endpoint rejects the whole request with 400 "web search is
not supported for this request". Filter the request's tools down to the
types bedrock-runtime's own validation error names, logging what was dropped,
through a helper shared with the Mantle route, which already did the same.
* fix(bedrock): emulate file_search and collapse custom Responses paths
* fix(bedrock): keep background and remote image inputs working on the native Responses route
* fix(bedrock): inline remote images inside tool outputs on the native Responses route
* fix(bedrock): inline remote computer screenshots on the native Responses route
---------
Co-authored-by: Leonardo Freitas dos Santos <leonardo.freitas.s@outlook.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(rust-bridge): declare native chat completions and responses bindings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(dispatch): cover chat completions and messages dispatch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): keep secret manager stub formatting unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust-bridge): match stub parameter names and exports to the native surface
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust-bridge): name declining entrypoint parameters and export embeddings in the stub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust-bridge): cover embeddings bindings in the route matrix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust-bridge): keep secret manager stub formatting unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): return 401 challenge for REST token-exchange tool calls without a subject token
Co-Authored-By: bot_apk <apk@cognition.ai>
* fix(mcp): keep tool_server_mismatch when server_id disagrees with the tool prefix
Co-Authored-By: bot_apk <apk@cognition.ai>
* test(mcp): type the token-exchange challenge test helpers
Co-Authored-By: bot_apk <apk@cognition.ai>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: bot_apk <apk@cognition.ai>
The tool_result user message built by the /v1/messages MCP loop used tuple
content, which the Messages to Chat Completions adapter silently dropped, so
non-Anthropic models re-requested the tool until the iteration cap or the
provider rejected the follow-up. Emit list content so the existing tool_result
branch translates it into a role tool message keyed by tool_call_id.
Resolves LIT-8474
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: bot_apk <apk@cognition.ai>
* feat(guardrails): speak the Straiker v3 platform API (/api/v3/detect)
The Straiker guardrail posted a webhook envelope to /api/v1/detect/webhook.
The v3 platform exposes /api/v3/detect instead, and its integration keys
(sk_agt_…) are rejected by the v1 route with an empty 401, so a tenant on
the v3 platform could not run this guardrail at all. Measured on a
customer gateway on 2026-09-17 after they rotated to a v3 key.
v3 parses the gateway's own traffic server-side, the same contract as
Straiker's unified Kong plugin. So on v3 the guardrail relays: the
request phase posts the provider body LiteLLM received (Anthropic
Messages or OpenAI chat), the response phase posts
{straiker_phase, sse, model, request}, the answer beside the request it
answers, and Straiker derives prompt, answer, agent and archetype. Both
phases also carry the flat prompt / app_response pair: a gateway-mode
integration key scores only the flat pair and an api-mode key only the
relayed body, each ignoring the other, so one payload serves whichever
key the console issued and it is one turn either way (measured on tenant
123, both key modes, 2026-09-18).
- api_version: "v1" | "v3", unset follows the key prefix, so a v3 key
needs no extra configuration. Explicit override still wins.
- The relayed body is an allowlist of provider fields. The hook sees the
client body merged with proxy state: `deployment` carries the resolved
provider credential and `proxy_server_request` the client's own
Authorization header. Neither travels. Identity survives as the
metadata subset Straiker's LiteLLM adapter reads.
- Identity never sends a proxy placeholder. `default_user_id` and the
master-key alias were being forwarded as a user and became the
session's identity on the platform.
- Headers: x-tool: litellm (ingress), x-straiker-phase, x-straiker-user,
and x-claude-code-session-id forwarded when the client sent it.
- Verdict: hookSpecificOutput.permissionDecision on the gateway envelope,
`action` on the flat one; block on block/deny, and on a non-empty
blocked_by as a backstop. A detect-mode control reads NONE.
- An error status from Straiker is now a webhook failure. LiteLLM's HTTP
client raises on any non-2xx and the retry loop caught only connection
errors, so a 401 or 503 from Straiker escaped the guardrail as an
exception and was relayed raw to the client, bypassing fail_open /
fail_closed. Retryable statuses retry; the rest are final.
- v1 is unchanged: same envelope, same X-Straiker-Webhook-Format header.
Tests: 15 new, fixtures from the request dict a hook sees on 1.98.0 and
the verdict envelopes the v3 platform returned on 2026-09-18. Each fix
was mutation-checked (handling removed, the test fails). Live: the same
eight-case battery (chat, /v1/messages, streaming, tool call; benign,
injection, PII) passes on a gateway-mode and an api-mode key, blocks at
pre_call with the tenant's block message, and lands under the declared
agent with the end user attributed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(guardrails): name the agent per application on v3 (x-s6r-agent)
One integration key can front several applications. Straiker enumerates them
as separate agents when the turn names one, which is what the unified Kong
plugin sends as x-s6r-agent. Without it every application on a gateway
collapses onto a single agent.
- Forwards a client-supplied x-s6r-agent.
- New `agent_ref` config names one agent for a route when the client sends
nothing. The client wins, matching Kong's precedence.
- Neither set: no header, and the platform derives the agent from the traffic.
Verified live on tenant 123 against an integration whose connector is
`gateway`: three distinct values minted three observed agents, and a turn
with no hint derived one from the traffic shape. An integration whose
connector is `custom-agent` declares its agent, so every turn attributes to
that one agent and the hint is ignored (agent_ref_source: attested).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(guardrails): which v3 shape is scored depends on the connector, not the key mode
The earlier comment said a gateway-mode key scores only the flat pair. Re-measured
on tenant 123 across all three integration types with one injection prompt:
custom-agent connector (Add Agent) raw body ignored flat prompt scored
gateway connector raw body scored flat prompt scored
api mode raw body scored flat prompt ignored
Behaviour unchanged: the payload already carries both shapes, which is why it works
on every type. Comment only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(guardrails): send exactly what the unified Kong plugin sends on v3
The v3 platform parses the gateway's traffic itself and derives agent,
archetype and identity from it. The earlier commits added to the relayed
body (a flat prompt / app_response pair, source, user_name) and to the
headers (x-tool, x-straiker-phase, x-straiker-user). None of that is in
the Kong v0.12 contract, and traffic through this guardrail was not
classifying by shape the way the same traffic through Kong does. Match
Kong byte for byte and leave classification to the platform.
Request phase: the provider body, plus session_id and
original.processed.Meta.user. Response phase: {straiker_phase, sse,
model, request} plus the same two. No flat fields, no phase or user
headers, no x-tool.
Session id follows Kong's precedence: the client's x-claude-code-session-id,
then the session LiteLLM resolved, then an md5 of system prompt + first
message so a conversation that states no session still groups across its
replays.
Routing hints complete the Kong set: x-s6r-agent (client header, else
`agent_ref`), and new `client` (x-s6r-client) and `format_hint`
(x-s6r-format) config, both optional.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(guardrails): sort imports in the v3 session test
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): send a streamed Messages answer back in the Messages shape on v3
On a streamed /v1/messages call the proxy rebuilds the answer as a chat
completion before the post-call hook runs, and that is what the plugin put in
the response envelope's sse field. Straiker's coding-agent reader parses a
Messages answer, so a Claude Code turn relayed this way came back
coding_agent/claude with no session and zero events scored: the model's tool
calls were never screened on the response phase. Captured live on 2026-09-18
against tenant 123, a real Claude Code Bash tool call through the proxy.
The proxy's own Anthropic adapter turns the rebuilt answer back into a Messages
response when the call arrived on the anthropic_messages route, which is what a
transport relay forwards. Chat completions calls keep the chat completion shape
and a buffered Messages answer is relayed untouched.
The regression test's fixture is the chat completion the proxy actually built
for that captured turn. After the fix the same turn scores on the response
phase (session resolved, one event, the Bash tool_use block present).
* style(straiker): ruff format the v3 guardrail and its tests
* refactor(straiker): one attempt per call in the webhook retry loop
The HTTPStatusError branch added for v3 duplicated the non-200 branch and put
_post_webhook over the strict complexity ceiling. One attempt is now its own
method that returns the verdict or a failure marked retryable, and the loop only
decides whether to try again. Behaviour is unchanged: retryable statuses and
transport errors retry, everything else is final.
* fix(straiker): name Claude Code's client and agent on v3 so its session lands under one coding agent
Straiker types a gateway turn as a coding agent from the "You are Claude Code"
preamble, which only the main agent turns carry. Claude Code's title and
topic-detection sidecars have their own system prompts, so they resolved by
shape as autonomous, and because they share the session id with the main turns
the whole session was filed under Autonomous rather than under a coding agent.
Kong does not hit this because its plugin config names the client and agent on
every call.
The User-Agent (claude-cli/...) is on every call including the sidecars, so the
plugin now reads it and sends x-s6r-client: claude plus, when the route names no
agent, x-s6r-agent: "Claude (LiteLLM)". A client-supplied x-s6r-agent or the
agent_ref config still wins. Verified live on tenant 123: a real Claude Code
session now lands as one coding_agent labelled "Claude (LiteLLM)" with its turns
scored, where before it split across Autonomous.
Identity: the key's own user (email then id) now outranks the end user the
request named. LiteLLM resolves Claude Code's hashed metadata.user_id as the end
user when nothing better is set, so a per-user key was being shadowed by a
session token. The key is the authenticated principal, the way a Kong consumer
is, so it wins; the request end user is the fallback.
* refactor(straiker): build the v3 request, envelope and headers as frozen mappings
The v3 builders seeded dicts and grew them, which the type-discipline gate
counts as mutable accumulators. Each is now one expression over a tuple of
pairs, frozen with MappingProxyType, and the JSON encoder unwraps a frozen
mapping through a default. The session seed and the verdict parser no longer
rebind locals. The wire is unchanged: 36 live calls through the proxy on this
commit carry the same fields, shapes, headers and identities as before, with
no mappingproxy text in any body.
* fix(straiker): satisfy basedpyright on the v3 builders
The frozen-mapping refactor left a shadowed headers local, a Mapping handed to
an HTTP client that takes a dict, an unguarded optional response, a turn id
typed object, and a redundant isinstance on already-typed texts. No behaviour
change: 4 live calls (chat, Messages, Bedrock, injection) return 200 with the
expected verdicts on this commit.
* fix(straiker): type the v3 config fields at the initializer and keep the verbose log as JSON
The four v3 routing fields (api_version, agent_ref, client, format_hint)
travelled through the untyped kwargs passthrough, which basedpyright counts
against the budget. They are now validated through a small Pydantic model at
the initializer and passed by name.
The verbose log serialized the frozen payload with default=str, which printed
a Python repr instead of JSON once the builders returned MappingProxyType.
Every serializer now unwraps a frozen mapping first. A test asserts the logged
payload parses as JSON and carries the identity; mutating the log site back to
default=str fails it.
* fix(straiker): address review findings on the v3 relay
Text completions relay their prompt: `prompt`, `suffix`, `echo` and `best_of`
join the provider allowlist, so /v1/completions traffic is screened.
The route's `agent_ref` now outranks the caller's `x-s6r-agent` header. The
header is caller-supplied, and letting it beat a pinned route would let any key
file its traffic under another application's agent and controls. On a route
that names nothing the header still names the application, which is how
several applications enumerate behind one key.
Credentials inside `tools` and `mcp_servers` (an OpenAI `mcp` tool's `headers`,
Anthropic's `authorization_token`) are replaced with `[redacted]` before the
body leaves the proxy, on both phases and in the verbose log. Detection reads
tool names, descriptions and schemas, never these.
A 200 whose body is valid JSON but not an object now reports an invalid
schema and follows the failure policy instead of raising out of the hook.
Comments that restated a constant are gone. Tests cover each change and the
failure paths (unreadable error body, client exceptions, missing response,
unmodellable request, session seeds from Anthropic block shapes); every fix
fails its test when reverted.
* fix(straiker): scrub tool credentials one level deep, without recursion
* fix(straiker): scrub only the fields that carry a credential, never a schema
The credential set is now the three fields that actually hold one on a tools
or mcp_servers entry (headers, authorization, authorization_token), read one
level deep. A function tool whose parameter schema defines a token, headers or
api_key property is relayed exactly as sent; a test pins that, and fails
against the recursive version.
* test(straiker): use example.com identities; drop a comment that restated its branch
* fix(straiker): present a legacy completion as the chat exchange it is
Straiker scores chat on both phases of a gateway turn but has no reader for a
text_completion answer: the request phase of a /v1/completions call was
scored and the response phase was refused with 501, whether or not the call
named an agent. A completion is one user turn and one assistant turn, so both
phases now present that exchange: the prompt becomes the single user message
and the TextCompletionResponse becomes a chat completion. Measured through the
proxy on this commit, both phases return 200 and score, and the derived
session is shared between them.
The derived session seed accepts the tuple the conversion produces; the test
pins the session on both phases and fails against the list-only check. The
unreachable "parsed is None" branch is folded into the failure branch, and a
malformed tools value is shown to relay as sent.
* fix(straiker): screen a completions prompt as the text the model receives
LiteLLM's /v1/completions accepts a string, a list of strings, a list of
token ids or a list of token-id lists, and decodes token ids with the
text-davinci-003 tokenizer before calling the model. The relay now renders
the prompt the same way, one user message per prompt, so a pre-tokenized
prompt is screened as the text it stands for rather than as digit strings.
A prompt in a shape this cannot render (empty, mixed, or with no tokenizer
available) is relayed untouched instead of being replaced with something
else. Tests cover all four accepted shapes and six unrenderable ones.
* fix(straiker): seed the derived session on the preamble and the first user turn
An OpenAI chat body carries its system prompt as messages[0], and the derived
session seeded on the Anthropic `system` field plus messages[0] with no role
check. For that shape the seed was the system prompt twice and the first user
turn never counted, so every unnamed conversation behind one system prompt
collapsed into one Straiker session. The seed now takes the preamble from
wherever the API puts it (`system`, `instructions`, or a leading system or
developer message) and the first message with role `user`, else a Responses
`input` string, else `prompt`. Two conversations sharing a system prompt are
two sessions again; a replayed conversation stays one.
* fix(straiker): seed the derived session on every text block of the first turn
A user turn that opens with an image or a document block and carries its
text later seeded the session on an empty string, so two different
conversations under the same preamble shared one Straiker session. Read
every text block of the turn instead of only the first block. A plain
string or a single text block seeds exactly as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(straiker): cover the tokenizer fallback, a textless first turn and Responses instructions
Three branches of the v3 relay had no test: a token-id prompt relayed as
sent when the tokenizer cannot be fetched, a first user turn with no text
seeding the session on the preamble alone, and a Responses API body
seeding on its instructions and first input turn. Each test fails when
its branch is mutated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): seed the derived session on the principal as well as the conversation
Straiker skips turns it has already scored for a session. The derived
session hashed the system prompt and the first user turn alone, so two
users who opened a conversation with the same words shared one session,
and the second user's copy of an attack came back as a replay: unscored
and allowed. Measured live on 2026-09-20: the first user's SSN turn was
blocked (`social_security_number`, scored=2), the second user's identical
turn was allowed (`controls: []`, replayed=2).
The principal now joins the seed. Explicit session ids, the Claude Code
header and LiteLLM's own session are unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): derive the session id with sha256 and drop comments that restated constants
The derived session now hashes the principal, and CodeQL flags MD5 over an
identity as a weak hash on sensitive data. SHA-256 truncated to the same
32 hex characters keeps the id shape. Comments that only labelled the
allowlist groups or restated a constant are removed; the two that explain
a non-obvious choice stay.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): keep a blocked conversation blocked when it is replayed
Straiker de-duplicates turns it has already scored per session and
answers a replay `allow`, whatever the first verdict was. A client that
resends a blocked request, or grows the conversation past the blocked
turn, was let through: measured on 2026-09-20, `block` then `allow,
events_replayed=2` for the same session and body, and Claude Code's
automatic retry after the 400 turned a blocked poisoned-file read into
a pass.
The guardrail now remembers, per session, a fingerprint of every
conversation it blocked (a bounded, day-long in-memory cache) and blocks
a request that repeats or extends one without asking again. A different
session with the same words is a new conversation and is scored afresh.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): scope the block memory by session or principal, never by content alone
A request with no derivable session keyed the replay memory on the
conversation fingerprint alone, so one caller's block could answer
another caller's identical request. The memory is now scoped by the
session, else by the principal, and a request with neither is not
remembered at all.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(straiker): remember only a block that names a control, never one that comes from state
The replay memory kept every block, including one the platform returns
because a kill switch is engaged (`action: block` with `blocked_by: []`).
An administrator lifting the kill switch then left the conversation
refused by the remembered copy: measured on 2026-09-21, traffic stayed
blocked after `POST /inventory/agents/{id}/restore` returned `engaged:
false`.
The same words are the same attack tomorrow, so a control-named block is
still worth remembering; state is not ours to cache. The parsed verdict
now carries `blocked_by` so the two can be told apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Phimmasone Phonpaseuth <PhimmStraiker@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(prices): add baseten/zai-org/GLM-5.3-Fast pricing with cost tracking e2e
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): assert message instead of comment on breakdown row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(baseten): drop the live e2e cost tracking test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* ci: add dashboard and core smoke checks across supported Python versions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: tighten merge smoke harness and keep mapped test diffs additive
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: terminate proxy on readiness timeout and use contextlib.suppress in teardown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(params): carry stream_chunk_size through litellm_params instead of provider params
* test(integration): fence stream_chunk_size out of every provider request body
* test(bedrock): type parametrized stream chunk test params
* test(integration): drop the contracts manifest resurrected by the main merge
* test(bedrock): type the stream_chunk_size test helpers
* test(params): finish AGENTS.md typing pass on stream_chunk_size tests
* test(integration): drop the covers marker from the stream_chunk_size wire test
---------
Co-authored-by: shrey kharbanda <shreshth@berri.ai>
* test(autoroute): wait for a valid fuzzy selection index and cancel the prompt on driver failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(autoroute): read the fuzzy selection through the public InquirerPy property
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject the HIBP client into change_password so the breached-password test never touches the network
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): only use the 200ms read timeout in the silent mode of the transport completion test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): record HIBP requests so the ordering test asserts no lookup happened
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(codeql): filter the weak-sensitive-data-hashing false positive on the HIBP k-anonymity lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): add Azure Foundry pricing for gpt-6-sol and gpt-6-luna
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): give azure/eu gpt-6-sol and gpt-6-luna full model metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the contracts.json manifest and the covers requirement
Groups live as a GROUPS literal in run.py, the browser expectations move next to the
browser tests, and the runner fails only on pytest failure, collection errors or a
selected file that collects zero tests. The covers marker stays registered for the
existing tests but is no longer checked. The mcp directory gets its own group
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): run mcp as its own shard with xdist and a peer proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): INTEGRATION_COVERAGE=1 runs the proxy under coverage for the MCP modules
The mcp shard sets it. The proxy and its peer start under coverage run in parallel mode,
get SIGTERM after the tests so coverage flushes, and the combined text and HTML reports
land in the suite results that CircleCI already stores as artifacts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): let the test proxy flush coverage when uvicorn re-raises SIGTERM
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add SSE, stdio, scripted, OpenAPI and OAuth 2.1 MCP peer doubles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add MCP transport and access-control matrices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add MCP credential and OAuth flow coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add MCP LLM endpoint, accounting, guardrail, resilience and lifecycle coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stop the same-URL grant test from counting a late initialize as a leaked call and satisfy the test-tree lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the REST denied-server listing is refused or empty
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin the REST denied-server listing to 403 access_denied
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
PATCH /model/{id}/update rejected read-modify-write edits that resent an unchanged but dangling litellm_credential_name. Existence validation now runs only when the requested name differs from the stored one; empty string and non-admin detach rejections are unchanged
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): revoke UI session tokens on logout and password change
Adds POST /session/logout to revoke the presented UI session key server
side (previously logout was client-side only and the key stayed valid
until expiry). Password changes now revoke the user's other UI sessions:
self-change keeps the caller's session, admin reset and onboarding claim
revoke all. The BYOK OAuth cookie auth now re-resolves the embedded key
against the DB so revoked sessions get a 401.
* fix(proxy): satisfy B008 budget and backend allowlist for /session/logout
* refactor(proxy): satisfy type-discipline budget in session_endpoints
* feat(otel): emit gen_ai.conversation.id from the caller's session id on v2 LLM spans
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep the caller's header session under missing_session_id: generate and read replayed payload session ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): drop only the proxy-minted session id so a caller id on the other metadata key survives
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep a replayed session id hidden when it only echoes the payload trace id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep a replayed session id even when the payload trace id fell back to it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): stop reading the replayed payload's session id, the generated marker does not survive replay
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit gen_ai.conversation.id on otel v2 spans through a real proxy, sink and postgres
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): keep otel conversation rigs alive for the whole session so shuffled shards do not reboot the proxy per test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): stop the audit rig proxies from probing sibling test peers for model info
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): record accepted OTLP batches in the sink instead of mutating the collector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): guard the accepted batch deque so snapshots cannot race sink appends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(proxy): bound auth cache invalidation publish so a wedged coordination Redis cannot stall user updates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve publish callable at call time in evict_and_broadcast
The keyword-only default bound publish_auth_cache_invalidation at
function-definition time, so tests patching the module attribute observed
zero calls. Default to None, resolve the real publisher inside the body,
and keep the keyword-shaped cache_key call the existing contract asserts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): publish auth cache invalidations in the background so a wedged coordination Redis costs handlers nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): cap in-flight auth cache invalidation publishes so a wedge cannot drain the redis pool
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(policy_engine): keep inherited parent guardrails when a child policy condition misses
Attachment applicability now walks the policy inheritance chain, so an attached child whose own condition does not match still contributes the guardrails of its unconditional ancestors, and a non-default attachment that applies through an ancestor still suppresses default attachments. The resolver continues to skip only the chain members whose own condition fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(policy_engine): skip a policy's pipeline when its own condition misses
resolve_pipelines_for_context returned the pipeline of a matched policy without evaluating its own condition, so a condition-missing child admitted by the chain-aware matcher still ran its pipeline. It now mirrors resolve_policy_guardrails and drops the pipeline when the policy's own condition does not match.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(policy_engine): property test that chain matching only widens to applicable ancestors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(policy_engine): log policies admitted only through an inherited ancestor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(policy_engine): log ancestor admissions once per attachment scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vector_stores): keep config-defined vector stores listed and read-only
Vector stores declared in config.yaml were purged from the in-memory registry by /vector_store/list because the database was treated as the only source of truth. Config-defined stores now carry is_config=True, stay in the list beside database rows, are never overwritten or evicted by database state, and reject /vector_store/new, /vector_store/update and /vector_store/delete with 400. The Admin UI renders them read-only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show vector store source and read-only state for config-defined stores
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit config-owned vector stores across list, writes, search, authz, peers and redis outage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show a visible read-only hint in the config vector store actions menu
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pricing): correct cached-token fields on realtime cost-map entries
azure/gpt-realtime-2 was the only member of the gpt-realtime-2 family priced
on one side of its cached-audio meter. Azure publishes that meter as
"gpt-realtime-2 Audio cd inp Gl 1M Tokens" at 0.4 per 1M and charges the
same rate for the write that populates the cache and the read that hits it,
so cache_creation_input_audio_token_cost lands at 4e-07, matching
azure/gpt-realtime-2.1, azure/gpt-realtime-2.1-mini and the openai
gpt-realtime-2 entry. No cost path reads that field yet, so this corrects
what get_model_info reports rather than what anything bills.
The gemini Live entries go the other way. Google's Vertex context-caching
page publishes separate supported-model lists for implicit and explicit
caching, and no Live or native-audio model is in either one. Its pricing
page prints N/A in both cached-input columns for every Gemini 2.5 Flash
Live API row, where plain 2.5 Flash and 2.5 Flash-Lite both carry real
cached prices, and the Vertex model card for the family marks context
caching not supported outright. Vertex never reports cachedContentTokenCount
on a Live session either, including for a byte-identical 7,021-token prefix
replayed across sessions minutes apart, which is well past the 2,048-token
minimum the same page sets for the Gemini 2 family.
So the 7.5e-08 on the two preview siblings priced something the provider does
not sell, and supports_prompt_caching on all three claimed a capability the
model does not have. The rate comes out. The flag is set to false rather than
removed, because get_model_info maps an absent key to None, and None is how
this map spells "nobody checked" across the 2,788 entries that omit it, where
false records the vendor's documented no. Both readers of the flag gate on
`is True`, so nothing bills or behaves differently either way.
Only the cached fields change on the two 09-2025 preview entries. Their
source field points at the Gemini API pricing page rather than the Vertex
one, so they describe a different surface with its own published limits, and
their context windows are left alone rather than assumed to match the Vertex
model card that drives the GA entry.
Tests cover all three halves: the family invariant that a cached audio read
implies an equal cached audio write, a cached count on a Live entry leaving
the bill at the fresh-input total instead of adding the old 7.5e-08, and
supports_prompt_caching answering false for all three entries while still
answering true for 2.5 Flash, so the false cannot be a swallowed lookup
error.
* fix(cost): correct gemini-live-2.5-flash-native-audio limits and capabilities
Google's model card for model ID gemini-live-2.5-flash-native-audio gives a
128K context window and 64K maximum output tokens, and marks structured
output, context caching and URL context as not supported. Its modality list
is text in and out, image in, audio in and out, and video in, with no
document input of any kind.
The entry advertised a 1M context window, an off-by-one 65535 output cap, and
three capability flags the vendor marks unsupported. Context caching is the
fourth and is handled in the cached-fields change alongside its two preview
siblings.
Both the bare id and vertex_ai/gemini-live-2.5-flash-native-audio resolve to
this single entry, so the test drives the corrected values through both.
* test(integration): cover live preview cached tokens billed at the fresh rate
Co-authored-by: Marty Sullivan <marty@martysullivan.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): cite dated sources for Live entry pins and drop restating docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Marty Sullivan <marty@martysullivan.com>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
