* fix(mcp): admit dcr_bridge envelope alongside an explicit litellm credential and mint under jwt principals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(mcp): suppress LIT002 on concrete dict header payloads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): mint bridge envelope for jwt mapped to a key without a user_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): mint and admit bridge envelopes under the master key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): bind mapped JWT envelopes to stored key tokens
* fix(mcp): preserve master envelope scope enforcement
* fix(mcp): reject bridge minting that loses JWT restrictions
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): keep the in-flight daily spend batch when shutdown cancels the flush
A daily spend batch drained from the in-memory queue was dropped for good when
the scheduler tick was cancelled by shutdown, because asyncio.CancelledError
bypasses the except Exception requeue. The flush now requeues the drained rows
on cancellation and re-raises, and each daily batch upsert runs in an
interactive transaction so a statement that already reached Postgres is rolled
back with the cancel instead of committing behind the requeue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): requeue the cancelled daily spend batch before its rollback returns
Behind a lock the rollback of the cancelled interactive transaction only
returns once the blocked statement does, which is after the shutdown flush
has already run. The commit now runs as a shielded task so the cancelled
tick requeues the batch at once and lets the rollback finish in the
background. The final flush then finds the rows and writes them exactly once
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): give the recording db a transaction seam for the bulk upsert tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): route the mocked daily tag spend upsert through the transaction seam
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): restore the drained Redis tag batch when shutdown cancels its commit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): search users by id or email when assigning a key owner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): label users without an email by user id in key owner picker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): freeze merged user-filter where, format create key test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): suppress module-global patch findings in ui_view_users search test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep merged user-filter where as a plain dict for prisma serialization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): forward search param from userFilterUICall to /user/filter/ui
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): mention user ID in the Create Key user picker helper text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): return 400 from /spend/calculate for a model with no pricing row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): assert error type and param for unpriced /spend/calculate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): move the repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: alias ModelNotMappedError re-export to satisfy F401
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): raise ModelNotMappedError only when the pricing row is missing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: count a zombie grandchild as gone in the migrate deploy timeout test
A SIGKILLed grandchild whose parent died in the same killpg reparents to
PID 1 or the nearest subreaper and stays a zombie until reaped, and signal 0
still succeeds on a zombie, so the timeout test read it as alive wherever
PID 1 is slow to reap or never does. The sibling test in
tests/test_litellm/proxy/db already handled that; both now share one
process_is_gone helper that reads the /proc state and reaps its own
children, with unit tests for the live, reaped, unreaped, and foreign
zombie shapes.
* test: move the pre-commit interrupt test onto the shared zombie-aware liveness helper
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): make two proxy-infra tests independent of sibling-test state
Both tests read process-global state that another module in the same xdist
worker can change, so they passed or failed on shard scheduling rather than on
the behaviour they assert.
test_gateway_plus_backend_covers_full_app checked allowlist coverage against
the live route table. gateway/main.py trims routes once, inside the lifespan,
so it only ever sees what is registered at startup; a lazy feature appends its
router on demand afterwards and is never filtered. Those routes cannot be
dropped on the floor, but they do enter the assertion the moment a sibling test
warms the feature, and 110 of them sit in neither allowlist. Subtract exactly
the lazy features this process has loaded, which leaves the assertion at full
strength for every eagerly registered route.
test_real_proxy_child_auth_privacy_and_body_policy pins prisma_client to a bare
object(). litellm.max_budget is a module global that nothing restores between
tests; once a sibling leaves it above zero, user_api_key_auth takes the
global-spend branch, dereferences prisma_client.db, and the AttributeError
surfaces as HTTP 401. Pin max_budget next to the other globals the test already
controls.
* test(proxy): measure allowlist coverage in a pristine interpreter
The previous revision subtracted paths matching a loaded lazy feature's
prefixes. Those prefixes are broad enough to swallow eagerly registered routes:
31 of them, including /openai/deployments/*, /access_group/*, /cursor/* and
/mcp, which would have made a real allowlist regression invisible.
Run the coverage check in a fresh interpreter instead. No lazy feature is
loaded there, so the route table is exactly the one gateway/main.py's lifespan
trim sees, and nothing has to be subtracted for the result to be deterministic.
The probe also reports the lazy modules it loaded and its route count, so an
empty uncovered set cannot pass vacuously.
Drop _component_paths and the four allowlist constants it used; the probe
reproduces the predicate in the child process.
* feat(proxy): configurable key_alias_pattern for key generate, update, and regenerate
Adds litellm_settings.key_alias_pattern, a regex every key_alias sent to
/key/generate, /key/service-account/generate, /key/update, and
/key/{key}/regenerate has to fully match. A non-matching alias gets a 400
that names the setting and the pattern. When set, it replaces the built-in
rule enable_key_alias_format_validation turns on, and the baseline
unsafe-name check still runs first. An invalid regex fails config load.
* fix(proxy): cap key_alias length under key_alias_pattern and type the test fixtures
* style(proxy): declare key_alias_pattern with a PEP 604 union
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore(cost-map): remove models past their deprecation date
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop the empty parametrize left behind by the gemini web search removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): drop merge base block left by conflict resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop gemini image cost tests pinned on removed model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): attribute provider and model info on pre-call rejected requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep pre-call rejections out of deployment cooldown and prometheus deployment state
Stamp model_info only into the logging metadata so the router's failure callbacks do not
count a key-level 429 or guardrail 403 against the deployment, treat a resolved plus an
unresolved deployment as ambiguous provider attribution, and stop the prometheus deployment
counters and deployment_state from treating a proxy-side reject as a selected deployment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): skip deployment attribution when the rejected body's model is not a string
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): bucket non-string request models as other instead of raising in failure hook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve team deployments and treat guardrail rejects as proxy-side in failure attribution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): flag pre-routing rejects instead of matching exception names
Post-call GuardrailRaisedException failures kept their deployment labels on main but lost them
on this branch because every GuardrailRaisedException was treated as a pre-routing reject.
The proxy failure path now flags litellm_params with proxy_rejected_before_routing only when
it adds deployment attribution itself, and the Prometheus logger keys deployment selection
off that flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): key pre-routing reject flag off provider handoff, not caller metadata
Caller-supplied metadata.model_info (kept for keys allowed to override pricing) no longer
suppresses proxy_rejected_before_routing. The hook now checks the logging object's
first_api_call_start_time, which only the provider handoff sets, so Prometheus never
records a deployment failure for a request that was rejected before routing.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): poll for both served and rejected spend rows before asserting attribution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(proxy): honor DATABASE_DISABLE_PREPARED_STATEMENTS in the litellm CLI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): pooled pgbouncer url keeps a single pgbouncer=true when the upstream already carries it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject a malformed DATABASE_DISABLE_PREPARED_STATEMENTS even when the config already enables it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gateway): expose /api/event_logging/batch on the gateway allowlist
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gateway): route /api/event_logging to gateway pods in helm and terraform
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): stop max_budget leaking between proxy_server and native_compaction tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ssrf): point the blocked-address remediation at litellm_settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ssrf): pin the block message's named section to litellm_settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(compaction): pin litellm.max_budget so leaked proxy budget cannot 401 the child auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): apply async_post_mcp_tool_call_hook content changes to the tool result
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): drop structuredContent when a post-call hook rewrites tool content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): satisfy type discipline and result contract
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): document internal logging patch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): avoid Final assignments inside callback loops
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): run every post-call hook and chain the rewritten content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(mcp): credit the original fix from #33403
Co-authored-by: eric <mitrecx@163.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): cover post-call logging fallback paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): cover proxy hook logging context
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): preserve native structured guardrail replacements
* fix(mcp): invalidate stale structure after direct content edits
* fix(mcp): reconcile direct edits after callback exceptions
* fix(mcp): preserve successful in-place callback rewrites
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: eric <mitrecx@163.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): share model rate-limit buckets between a model_group_alias and its target
A request sent under a model_group_alias counted in its own per-key, per-team,
per-org, and per-project model bucket, so a key could double a deployment's
default_api_key_rpm_limit / tpm_limit by alternating the alias and the model
group name, and a metadata model_rpm_limit / model_tpm_limit keyed by the
model group never applied to alias requests. The limiter now resolves the
requested name to its model group before keying any model bucket, looks the
limit up by the requested name first and the model group second, and charges
post-call tokens to the same bucket.
* fix(proxy): charge the model group resolved at admission when reconciling reserved tokens
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: enforce disable_custom_api_keys from general_settings
The gate in _check_custom_key_allowed read the persisted UI settings row
through get_ui_settings_cached, which had two consequences.
A config-file general_settings.disable_custom_api_keys was never enforced,
because the gate only ever looked at the stored ui_settings row. POST
/key/generate with a custom key value returned 200 even with the flag set
to true in config.yaml.
The read went through a DualCache with a 600s TTL, which is per worker
without Redis, and only the worker that served PATCH /update/ui_settings
refreshed it. A gate flipped through the UI was then a coin flip across
workers for up to ten minutes.
Both go away by routing the flag the way the other runtime UI flags are
already routed. Adding it to _RUNTIME_GENERAL_SETTINGS_FLAGS and to the
settings rules' _UI_SETTINGS_FIELDS makes SettingsStore resolve it from the
ui_settings row with the config file winning, and every pod re-reads it on
its own settings sync rather than holding a private cached copy. The two
lists have to stay in step: a flag in one and not the other resolves
against the wrong stored row and silently never reaches a reader, so there
is a test for that invariant.
Writes to the ui_settings table did not publish on the config-sync channel,
so other pods only discovered a change on their next periodic reload. Adding
litellm_uisettings to _CONFIG_SYNCED_TABLE_NAMES puts it on the same pubsub
path model and SSO config writes already use, which cuts cross-pod
propagation from tens of seconds to a few.
The value reaching the gate is run through coerce_bool first. Resolution
hands back the raw YAML value, so a quoted "true" in config.yaml is a str
and the old `is True` check let custom keys straight through.
* test: assert both directions of the coerced config value
* test: assert the runtime flags read back instead of inspecting the registry
* fix(fal_ai): align /fal_ai queue gate with the pricer and normalise resolution type
The /fal_ai gate admitted catalog keys the Fal pricer cannot price, so
those jobs were forwarded and logged at 0.0 spend. The gate now reuses
the pricer as its eligibility predicate. Resolution is normalised to a
string before the keyed price lookup so int and str spellings bill the
same.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): assert passthrough pricing invariants on synthetic catalog entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): price the submitted body in the /fal_ai queue gate so keyed-only entries are admitted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): move the queue gate repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(logs): add span type filter to request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logs): look up span type sql conditions from a mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): opt-in include_guardrail_response returns guardrail_information in the response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): read include_guardrail_response from the request metadata bucket the router did not reseed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format common request processing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): redact matched content in guardrail_information and stop mutating cached responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): traverse guardrail diagnostics iteratively
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): annotate guardrail traversal cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): reuse core redaction helper for guardrail_information
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): justify response rebind when attaching guardrail information
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Python 3.13+ strips the common indentation of docstrings at compile time and 3.12
keeps it, and the 429 error description in ERROR_RESPONSES came straight from
RateLimitError.__doc__, so regenerating litellm/proxy/_lazy_openapi_snapshot.json
on a 3.13+ venv produced a one-line diff that the check-ui-api-types job (Python
3.12) rejected. Run the docstring through inspect.cleandoc before it lands in the
spec, regenerate the snapshot once, and pin the behavior with a test
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): add admin-only /debug/report sharing the bug report environment fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add verbose=true to /debug/report listing every config key with typed values
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): inject auth into /debug/report through Annotated to keep the B008 budget flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound the verbose config walk, drop nested-list recursion from the safe renderer, regenerate schema.d.ts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count pass-through and mcp header maps plus operator-named budget maps in verbose /debug/report, single-exit scalar renderers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts after dropping the verbose query from /debug/report
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): write key deleted audit logs for cascade and alias key deletions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert persisted key deleted audit rows for cascade paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): route /audit and /v2/login to the control plane in split transport
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): time-windowed team reservation of deployments via model_info.access_windows
Deployments can declare model_info.access_windows, a list of local wall-clock windows (IANA timezone, cross-midnight allowed) that reserve the deployment for the listed team_ids. While a window is active the router drops the deployment for every other request, including no-team and admin requests, on every candidate path (model name, model id, specific_deployment, early-resolve, wildcard, litellm_params.model lookup, fallbacks). If every candidate is reserved the request fails with a 400 naming the window end instead of falling back. Outside a window routing is unchanged and reserved deployments stay visible in /model/info and /v1/models. Malformed windows (bad time, unknown timezone, empty team_ids, start equal to end, offset-aware times) fail proxy startup with a clear error since the proxy router runs with ignore_invalid_deployments=True
Resolves LIT-8308
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover _filter_reserved_deployments directly for coverage gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): keep reservation filtering immutable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): drop strategy markers before reservation filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): give every ui settings endpoint test a fresh settings store
Under xdist the module-scoped reload in tests/test_litellm/conftest.py is skipped, so a health endpoint test's lifespan startup leaves proxy_server.general_settings bound to the ProxyConfig singleton's SettingsStore for the rest of the worker. Every /get or /update ui_settings call in test_proxy_setting_endpoints.py then writes into that shared store, and since get_ui_settings merges proxy_config.settings over the stored row, a leaked team_admin_editable_team_fields: [] from the PTU round-trip test beat the mocked DB row in test_get_reports_the_stored_list_and_advertises_supported_fields. An autouse fixture now binds both proxy_config.settings and general_settings to a fresh store for each test in the file, and a regression pair checks a runtime-flag write lands in the store the endpoint reads and does not survive into the next test.
* test(proxy): assert ui settings isolation through the endpoints
Replace the fixture-structure assertions with a parametrized GET/PATCH round trip on /get/ui_settings and /update/ui_settings that fails whenever one case's PATCH leaks into the next case's GET.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): surface the owner's user budget on keys without their own budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): apply the owner's budget hint to team keys when apply_user_budget_to_team_keys is on
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): read only the apply_user_budget_to_team_keys flag from general_settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the general_settings cast as cast-ok
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): load the owner's budget for keys opened outside the current page
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: jesus <jesus@berri.ai>
* fix(jwt): say x-litellm-team-id matched no team id or alias in the 403
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): tell the caller when x-litellm-team-id names an alias shared by several teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): deny a shared x-litellm-team-id alias exactly like an unknown value
A distinct 403 for an alias several teams share was raised before the
allowed-teams check, so any JWT could probe which aliases exist. The
alias lookup now treats the duplicate as a miss, and both denials say
the value does not resolve to a team id or a unique team alias, which
is true for unknown, unauthorized and duplicate values alike
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock/claude_platform): strip body params the AWS endpoint rejects
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock/claude_platform): assert exact bodies through a strict fake gateway for every workspace alias and auth mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): block Claude Platform workspace id aliases in request bodies without admin opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Venkata Donavalli <vdonavalli@live.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shrey kharbanda <shreshth@berri.ai>
* feat(arize): per-team success and error sampling rates for the Arize AX callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(arize): fail open on invalid sampling rates and type the new sampling code
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(arize): type the sampling test helpers and parametrized fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): accept a team alias in x-litellm-team-id
The header only matched canonical team ids, so a JWT caller selecting one of their teams by its alias got a 403 even though they belonged to it. The header value is now resolved through the existing alias lookup before the JWT allowed-team check and the DB membership fallback, while a value that is already a team id never costs an alias lookup and denials keep naming the value the caller sent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): only alias a header team id the database provably lacks
Under fallback_to_db_teams a header value whose team row read fails for any reason other than TeamNotFoundError now keeps the membership denial instead of falling through to the alias lookup, so a degraded read cannot select a different team that carries the value as an alias. Drops the HeaderTeam docstring that only restated its fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): store the masked output in spend logs when a post_call guardrail rewrites the response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): record served output without re-narrowing the logging object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): overlay the served output before message redaction so turn_off_message_logging still wins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): type the monkeypatch fixture in the redaction ordering regression
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): poll through raw card output until the guardrail reaches the serving worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep blanked, multi-choice and disconnected served output out of raw spend logs
Served text keeps empty strings and tracks unavailable choices as None so a guardrail that blanks
the output still overrides the raw provider text. Stream choices are sized from the highest choice
index, served chunks are recorded before a client disconnect or stream failure propagates, and
message-logging redaction drops the served text from callback kwargs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Proxy bug reports now carry the request's stream flag and a config block
built from dotted paths like router_settings.routing_strategy. A line is
emitted only when its key is defined by a LiteLLM schema and its value is
a bool or a LiteLLM-defined value (providers, callbacks, routing
strategies, cache types, guardrail integrations and modes, key management
systems). Secrets, URLs, numbers and custom values leave no line
* fix(proxy): answer 503 no_db_connection on management routes when the caller's user read hits a database outage
Under allow_requests_on_db_unavailable, once the caller's key row lapses the
request runs as the restricted fallback identity, and its own user read fails
on the outage. /v2/team/list, /user/list, and /user/filter/ui answered a bare
500 for that; every route now answers the same 503 body auth gives, through
one shared builder consulted by the generic exception handler and by
ui_view_users' own catch-all
* fix(proxy): log the database outage before answering 503 on user search
* chore(proxy): drop the docstrings on the db outage 503 helper and its tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): add conversational matrix across chat, messages and responses
Parameterizes one behavioral contract (reply, stream, cost log, tool call,
tool round trip) across /v1/chat/completions, /v1/messages and /v1/responses,
OpenAI and Anthropic models, and env-ref vs stored-credential auth, with
record/replay fixtures.
Adds general_settings.disable_model_info_refresh so the proxy fronting a
replay fixture does not poll every OpenAI-compatible deployment's /v1/models
in the background, which otherwise leaves unconsumed interactions in the
recorded bundle.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): force the weather tool on the first turn and rename Provider to Deployment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(errors): prefilled GitHub issue link on unmapped internal errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(errors): scope bug report link to unmapped exceptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(errors): cover bug report link on unmapped SDK exceptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(errors): exercise unmapped branch without changing openai mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: format bug report changes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): guard bug report generation for invalid exceptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep bug report notice out of client error bodies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(errors): strip only the notice separator from client messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(errors): only link bug reports for exceptions without a provider status
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(errors): allowlist-only bug report link, drop exception message and model from URL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): read request.url.path for the unhandled-exception bug report route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bug_report): drop non-string provider values before the allowlist lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bug_report): drop the OS line from the prefilled issue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): mask PII in streaming /v1/messages output
Raw Anthropic SSE frames were passed through the post_call output masking
callback untouched, and ProxyLogging rerouted the callback to the unified
apply_guardrail path on /v1/messages because mask_response_content was
false. Buffer the raw frames, assemble them with the shared Anthropic SSE
helpers, mask through Presidio, and re-emit the masked frames.
Resolves LIT-8288
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): replay raw SSE frames when masking fails mid-stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): propagate upstream stream errors instead of returning an empty stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(presidio): extract buffered stream masking to satisfy complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): let BLOCK on generated PII refuse the streaming /v1/messages response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(presidio): fold the BLOCK re-raise into the existing except to stay within the complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(presidio): move the blocked stream consumption into a helper so pytest.raises holds one statement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(presidio): fail closed when output masking of a raw SSE stream errors
A Presidio outage on streaming /v1/messages replayed the unscanned frames
to the caller. Propagate the error instead, matching the non streaming
path and the merge base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(presidio): cover structured chat stream output masking and trailing bytes passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: benchmark and gate an installed release wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: simplify installed-wheel benchmark check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add native tokenizer codec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): route Python tokenization through the Rust extension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(lint): format tokenizer call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(packaging): restore runtime dependencies and native images
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve Python SDK behavior with Rust tokenizers
* fix(tokenizer): restore compatibility paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): count custom tokenizers directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve caller-supplied Python tokenizer counts
* fix(tokenizer): reuse packaged vocabularies in the native wheel
* refactor(rust_bridge): route token counting through the catalog as RUST_OPT_IN
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): compare tokenizer groups by value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): re-resolve filelock under the <4.0 pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): align transformation override signatures with base configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): use fat LTO to keep the native wheel under the 35 MB limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): preserve Python defaults with opt-in Rust dispatch
* test(proxy): tolerate missing litellm.utils.Tokenizer when patching it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): patch the tokenizer dispatch function instead of the removed alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): give the Rust wrappers the tiktoken and tokenizers surface
Callers of litellm.encoding and litellm.create_tokenizer must see the same
read-only API whichever backend the catalog selects.
- OpenAIEncoding mirrors tiktoken.Encoding: n_vocab, max_token_value,
token_byte_values, encode_single_token, encode_with_unstable,
encode_to_numpy, decode_with_offsets, is_special_token, repr; the Rust
tiktoken crate keeps a Vocabulary beside each CoreBPE and reports the
requested encoding name (gpt2 stays gpt2).
- HuggingFaceTokenizer mirrors the read-only tokenizers.Tokenizer surface
(token_to_id, id_to_token, get_vocab, get_vocab_size,
get_added_tokens_decoder, num_special_tokens_to_add, padding, truncation,
encode_special_tokens, from_buffer); HuggingFaceEncoding gains the
char/word/token lookups, pad, truncate, set_sequence_id and merge.
Mutators stay on the Python tokenizer.
- from_json/from_pretrained claim the fork gate only when the huggingface
feature is compiled in; the surrogate fallback matches on the Codec.
- Tokenizer caching is keyed on the same catalog Context the dispatch runs
on; rust_tokenizer reads the encoding name without loading an encoding;
LITELLM_RUST parsing is cached.
- Drop the unused tiktoken_encoding_for_model export and Error::Download.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tokenizer): close the exhaustive matches with assert_never
CodeQL reads a `match` over a Literal with no default arm as an implicit
`None` return. `assert_never` makes the exhaustiveness explicit for both the
HuggingFace tokenizer loader and the Rust token-counter factory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tokenizer): derive the fast counter from the shared tokenizer
The count-only counter (`fast` feature) and the codec each parsed the same
artifact: TokenCounter took the Anthropic JSON and the tiktoken rank files
from Python while Tokenizer loaded them again. One parse now serves both.
- FastTokenizer builds from a model another loader holds: `from_shared`
takes the Arc<tokenizers::Tokenizer> the HF codec keeps, and
`from_*_pairs` take the ranks the tiktoken vocabulary already parsed.
- `FastCounter::fast_counter` in the core crate derives it from either codec;
encodings the fast scanner does not reproduce are refused.
- Native `Tokenizer.count(text, fast=False)` opts into that counter, built
once per tokenizer on first use; `TokenCounter.from_tokenizer(tokenizer,
fast=False)` replaces the JSON and rank-file constructors.
- The Python route counts over the native tokenizers the codec path shares
(`native_encoding`, `native_anthropic`) and no longer reads rank files;
the packaged Anthropic tokenizer has one loader, `tokenizer_dispatch.anthropic`.
- Public wrappers gain `count(text, fast=False)`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* feat(proxy): add TinyFish Agent API passthrough with per-step billing
* chore(ui): regenerate dashboard API types for /tinyfish passthrough
* fix(proxy): satisfy strict lint budget for tinyfish passthrough
* style: ruff format tinyfish passthrough handler
* test(proxy): exercise tinyfish route through the app with a faked upstream
* refactor(proxy): make cross-module tinyfish billing hooks public
* fix(proxy): tolerate transient tinyfish poll failures instead of dropping the charge
* fix(proxy): defer billing for disconnected tinyfish SSE runs to the background poller
* Revert "fix(proxy): defer billing for disconnected tinyfish SSE runs to the background poller"
This reverts commit ef0bcfb4a0.
* fix(proxy): bill tinyfish SSE runs via detached poller and only COMPLETED runs
Disconnected run-sse clients previously left completed runs unbilled: the stream-end
handler saw a still-RUNNING run and logged $0. The poller now spawns from the streaming
path on the first run_id frame, outlives the disconnect, and writes the one spend row when
the run turns terminal; the stream-end path only logs the $0 fallback for run_id-less
streams. Costs now apply only to COMPLETED runs ($0 for FAILED/CANCELLED, matching the
upstream invoice), spend rows carry the request's litellm_call_id (previously NULL
request_ids collided and were silently dropped), and the GET /v1/runs listing is blocked
so callers behind the shared key cannot discover each other's runs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* fix(proxy): drop GET /v1/runs from the tinyfish allowlist error message
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* chore(proxy): sync openapi artifacts for tinyfish docstring, suppress LIT011 on flag write
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): ruff-format the sse poller flag write
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): keep the rebind-ok suppression on the flag write's own line
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* fix(proxy): harden tinyfish billing paths from review findings
Skip failure dispatch when the SSE poller owns billing (a failure row collided with the
poller's billed row on request_id and dropped the charge), late-spawn the poller for
run_ids that arrive in unterminated frames instead of mispricing RUNNING runs at $0,
thread litellm_params into poller-billed standard logging objects so SLO consumers see
attribution, untype the run error field so upstream error-shape drift cannot void a
billable run, normalize a schemeless TINYFISH_AGENT_API_BASE, extend the poll budget to
cover queue wait (3600s) with ~60s outage tolerance, and log poller cancellation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* chore(proxy): satisfy ratcheted BLE001/LIT002 budgets from main in tinyfish handler
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): drop stray blank line from merge resolution
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* fix(proxy): reject passthrough envelope controls on tinyfish route, raise blocking-run timeout
The generic passthrough unwraps a caller-supplied custom_body as the forwarded request and
honors a caller stream flag, so custom_body.use_vault bypassed the credentialed-run 403 and
stream: true flipped a blocking run into the streaming pipeline. The route now 400s the
envelope fields (custom_body, stream, query_params); streaming comes from the endpoint.
Blocking runs also get a 1500s default timeout covering the upstream 1200s run cap, unless
the operator configured pass_through_request_timeout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): resolve operator timeout without a dict-literal default
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* test(passthrough): list the TinyFish route among protocol-constrained pass-through routes
* chore(proxy): regenerate the lazy OpenAPI snapshot after merging main
* chore(proxy): keep the lazy OpenAPI snapshot as CI's Python 3.12 renders it
* fix(proxy): reject TinyFish POST bodies that are not a JSON object
A form-encoded or text body carried stream and use_vault past both field
gates, because the gates only saw fields the body parsed to as JSON. The
route now checks the content type before reading the body and answers 400
for anything that is not a JSON object.
* fix(tinyfish): reject submit paths with extra slashes so run-async always bills
The allowlist dropped empty path segments, so POST /v1/automation/run-async/
was forwarded upstream while the billing dispatch only recognises the exact
path and would have logged the submit at $0 without starting the poller.
Any path with a trailing or doubled slash now returns 403 before forwarding.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
get_user_object wrapped every failed read, a refused connection included, in
ValueError("User doesn't exist in db ..."), so JWT callers got a 401 naming a
missing user while Postgres was down and virtual-key callers got 503
no_db_connection for the same outage. A connection or transport error now
propagates as-is and the auth exception mapper answers 503 no_db_connection;
a genuinely missing row and query-level errors still answer 401.
The MCP auth and token-exchange docstrings and the exception-chain helper's
docstring described the old wrap and are updated to the new contract.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): release unclaimed budget reservations at request end
* fix(proxy): release unclaimed budget reservations of websocket sessions too
* test(proxy): drop the structural middleware inheritance check
* fix(proxy): claim the budget reservation on streaming pass-through before its cost callback
The SSE chunk processor hands its success handler to the logging worker
after the response, so the request-end release freed the reservation
first and left the key unguarded until the worker drained. Claim it at
both end-of-stream hand-offs, the immediate enqueue and the coroutine
parked for deferred dispatch.
Give the xai realtime test double the litellm_params attribute every
real Logging object carries, since the wrapper now reads it.
* test(pass-through): give the vertex streaming test doubles a litellm_params dict
The spec'd Logging mocks in test_vertex_ai_anthropic_streaming_cost_injection.py
lacked the instance attribute the chunk processor now reads to claim the budget
reservation. Also restores main's _lazy_openapi_snapshot.json: the branch's copy
had been regenerated under Python 3.14, which dedents one docstring description
that the CI regeneration on Python 3.12 keeps indented, and the PR adds no lazily
loaded route, so main's file is the correct one.
* fix(pass-through): claim the budget reservation only after its cost callback is enqueued
Every pass-through success hand-off stamped callback_bound before handing the
coroutine to the logging worker. When that enqueue raised, the reservation stayed
claimed with no callback left to reconcile it, so the request-end release skipped it
and the reserved cost stayed pinned on the key's counter. Enqueue first, then claim,
so a failed hand-off leaves the reservation for the request-end release.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
