The unified guardrail's streaming iterator hook processes every Nth
chunk (sampling_rate, default 5). On each sampled chunk it calls
process_output_streaming_response, which combines all accumulated text
into the first chunk and clears all subsequent chunks to "".
The hook then yielded `processed_items[-1]` — the last item, whose
content had been cleared to "". This permanently lost every Nth
chunk's content, causing random missing words/tokens in the client
output (observed in Roo Code, Open WebUI, etc.).
Fix: deep-copy the current chunk before guardrail processing runs,
then yield the original (unmodified) chunk. The guardrail validation
still executes and can block if it detects a problem, but the stream
content is preserved.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: enforce team MCP permissions when using JWT authentication
Root cause: When JWT auth was used with teams in groups (via team_ids_jwt_field),
the team's MCP permissions were not being enforced because:
1. The default team_allowed_routes did not include mcp_routes
2. allowed_routes_check() failed for MCP endpoints like /mcp/tools/list
3. find_team_with_model_access() skipped the team due to failed route check
4. team_id was None in UserAPIKeyAuth
5. MCPRequestHandler._get_allowed_mcp_servers_for_team() returned empty list
Fix: Add 'mcp_routes' to the default team_allowed_routes in LiteLLM_JWTAuth.
This ensures that teams can access MCP endpoints by default, allowing the
team's MCP server permissions to be properly enforced.
Added tests:
- test_reproduce_jwt_mcp_enforcement_issue: Reproduces the exact bug scenario
- test_verify_mcp_routes_in_default_team_allowed_routes: Verifies fix
- test_mcp_route_check_passes_for_team: Verifies route check works
Co-authored-by: ishaan <ishaan@berri.ai>
* test: add comprehensive E2E tests for JWT + team MCP permission enforcement
Added tests:
- test_e2e_jwt_team_mcp_permissions_enforced: Full E2E test verifying JWT auth
with teams in groups properly sets team_id and MCPRequestHandler returns
the team's MCP servers
- test_e2e_jwt_without_team_no_mcp_servers: Verifies no MCP servers returned
when JWT has no teams
- test_e2e_jwt_team_mcp_key_intersection: Verifies intersection logic when
both key and team have MCP permissions (result = intersection)
These tests verify the complete flow:
1. JWT token with team in groups field
2. JWT auth properly sets team_id on UserAPIKeyAuth
3. MCPRequestHandler.get_allowed_mcp_servers() returns team's MCP servers
4. Key/team permission intersection works correctly
Co-authored-by: ishaan <ishaan@berri.ai>
* test: add simple tests for JWT + MCP permission enforcement
Simple, focused tests that validate:
1. test_simple_jwt_mcp_permissions_enforced: JWT user with team gets team's MCP servers
2. test_simple_jwt_no_team_no_mcp_servers: JWT user without team gets no MCP servers
3. test_simple_jwt_team_id_required_for_mcp_permissions: Verifies team_id is required
4. test_jwt_auth_sets_team_id_for_mcp_route: JWT auth sets team_id for MCP routes
These tests directly verify the core MCP permission enforcement logic works
when using JWT authentication with teams.
Co-authored-by: ishaan <ishaan@berri.ai>
* Add test: MCP route without model still returns team_id
Co-authored-by: ishaan <ishaan@berri.ai>
* Add 2 debug logs for JWT+MCP troubleshooting
- handle_jwt.py: Log team route check result (team_id, route, is_allowed)
- user_api_key_auth_mcp.py: Log team_id when looking up MCP permissions
Co-authored-by: ishaan <ishaan@berri.ai>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
_cached_get_model_group_info uses @lru_cache but had no invalidation,
causing stale model group info (TPM/RPM limits) after dynamic deployment
changes. Add cache_clear() at all 5 model_list mutation sites.
AsyncHTTPHandler.__del__ was closing httpx clients still in use by
AsyncOpenAI/AsyncAzureOpenAI due to independent cache lifecycles.
Restores standalone httpx client creation for OpenAI/Azure providers.
The no-args hot path (called on every proxy request) was rebuilding
a defaultdict by iterating the full model list each time. Cache the
result and invalidate at all 5 model_list mutation sites following
the _invalidate_model_cost_lowercase_map() pattern.
Line profile: 31.8µs/call → 1.1µs/call (29x improvement).
* fix fail-open for grayswan; pass metadata to cygnal api endpoint; update docs
* pass litellm_metadata to cygnal in payload
* switch error msg to const, and clean exception handling.
* update pyproject.toml as requested
* Revert "update pyproject.toml as requested"
This reverts commit 4eece154d0.
* fix: check for model_response_choices before guardrail input
* test: add tests for responses api translation
* fix: protect other guardrail translations
* refactor: remove type ignores
* anthropic request body got mutated fix
* add warning when extra_body is provided but user is non premium
* fix: resolve mypy union-attr errors in anthropic guardrail handler
Cast choices[0] to Choices type before accessing .message attribute
to satisfy mypy's union type checking for Choices | StreamingChoices.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* add logger when model response has no choices for streaming /response and /messages
* update pyproject.toml as requested
* Revert "update pyproject.toml as requested"
This reverts commit 541a2b075a.
* update pyproject.toml as requested
* Revert "update pyproject.toml as requested"
This reverts commit 716ea0caa1.
---------
Co-authored-by: Xiaohan Fu <xiaohan@grayswan.ai>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Add the following SCIM v2 discovery endpoints per RFC 7643/7644:
- GET /scim/v2 - Base resource discovery (ListResponse of ResourceTypes)
- GET /scim/v2/ResourceTypes - List all supported resource types
- GET /scim/v2/ResourceTypes/{id} - Get a specific resource type (User/Group)
- GET /scim/v2/Schemas - List all supported schemas
- GET /scim/v2/Schemas/{uri} - Get a specific schema by URI
These endpoints are required by identity providers (Okta, Azure AD, etc.)
for SCIM resource discovery. Previously, GET /scim/v2 returned 404.
Also adds SCIMResourceType, SCIMSchema, and SCIMSchemaAttribute Pydantic
models to the SCIM types module.
Fixes#20295
* fix: strip stale mcp-session-id header to prevent 'Session not found' error loop
When VSCode reconnects to LiteLLM's MCP endpoint after a reload, it sends
a stale mcp-session-id header. The session was already cleaned up, causing
a 404 'Session not found' error. VSCode retries with the same stale ID,
creating an infinite error loop.
Before forwarding requests to the StreamableHTTP session manager, check if
the mcp-session-id header references a valid session. If the session doesn't
exist, strip the header so a new session is created automatically.
Fixes#20292
* refactor: extract stale session handling into _strip_stale_mcp_session_header helper
* perf: cache _get_relevant_args_to_use_for_logging() as module-level frozenset
The set of valid LLM API parameter names for logging was being rebuilt
on every request from 8 OpenAI SDK type annotations + set operations.
Since these are static TypedDict annotations that never change at
runtime, compute once at import time and store as a class-level
frozenset.
Line profiler: get_standard_logging_model_parameters() dropped from
774ms to 77ms across 12K calls (90% reduction, ~25µs/req saved).
* test: add tests for cached ModelParamHelper logging args
Verify cached frozenset matches dynamic computation and that
prompt content keys (messages, prompt, input) are excluded from
logged model parameters.
Pass-through endpoints (like vLLM classify) were not setting
standard_logging_object because _get_assembled_streaming_response
returns None for non-ModelResponse results.
This caused model_max_budget_limiter.async_log_success_event to raise
ValueError('standard_logging_payload is required').
The fix adds an elif branch in async_success_handler that mirrors the
non-pass-through code path.
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Add x-api-key header to CountTokens handler to match chat completion
authentication. Azure AI Anthropic requires this header per Microsoft's
native API format.
Previously, when Model Armor guardrail blocked a request/response,
the `applied_guardrails` field was not populated in the logs because
`add_guardrail_to_applied_guardrails_header()` was called after the
HTTPException was raised.
This fix moves the `add_guardrail_to_applied_guardrails_header()` call
to before the blocking check in all hooks:
- async_pre_call_hook (pre_call mode)
- async_moderation_hook (during_call mode)
- async_post_call_success_hook (post_call mode)
- async_post_call_streaming_iterator_hook (streaming)
This ensures that even when a guardrail blocks content, the guardrail
name is properly recorded in the logs for observability.
Added regression tests to verify applied_guardrails is populated when
content is blocked.
Co-authored-by: Cursor <cursoragent@cursor.com>
