mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-05 08:07:05 +00:00
350 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yuneng-jiang
|
6ff668c7aa
|
[Infra] Promote internal staging to main (#27245)
Some checks failed
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
CodSpeed Benchmarks / benchmarks (push) Has been cancelled
Helm unit test / unit-test (push) Has been cancelled
Read Version from pyproject.toml / read-version (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
Unit Tests: Caching (Redis) / caching-redis (push) Has been cancelled
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
GitHub Actions Security Analysis / zizmor (push) Has been cancelled
Unit Tests: Proxy DB Operations / auth-checks (push) Has been cancelled
Unit Tests: Proxy DB Operations / budgets (push) Has been cancelled
Unit Tests: Proxy DB Operations / custom-logging (push) Has been cancelled
Unit Tests: Proxy DB Operations / db-and-spend (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-runtime (push) Has been cancelled
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Has been cancelled
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Has been cancelled
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Has been cancelled
Unit Tests: Proxy DB Operations / key-generation (push) Has been cancelled
Unit Tests: Proxy DB Operations / logging-misc (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-server-core (push) Has been cancelled
Unit Tests: Proxy DB Operations / schema-migration (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-utils (push) Has been cancelled
* default requested_model to empty string on litellm-side rejects * Update litellm/router.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: scope key access_group_ids override by team's assigned groups A team member could set any access_group_ids on their key (e.g. a group assigned only to a different team) and override the team's model restriction. Intersect the key's access_group_ids with team_object.access_group_ids in _key_access_group_grants_model so foreign groups are dropped before model expansion. Adds a regression test that asserts expansion is never called for foreign groups. * [Fix] Proxy: Skip Personal Budget Hook When Reservation Covers Counter The reservation path (PR #26845) atomically pre-fills `spend:user:{user_id}` and admits at the strict-`<` boundary. The legacy `_PROXY_MaxBudgetLimiter` pre-call hook re-reads the same counter with `>=`, so a reservation that fills the counter to exactly `max_budget` (e.g. a request without a `max_tokens` cap that falls back to reserving the smallest remaining headroom) is rejected by the hook even though the reservation already admitted it. Skip the hook when the request's active `budget_reservation` covers `spend:user:{user_id}`. The reservation is the source of truth for that counter cross-pod; the legacy `>=` path remains in place for requests without a reservation (e.g. paths that bypass the reservation entirely). Reproduces as `tests/otel_tests/test_prometheus.py::test_user_budget_metrics` on a fresh user with `max_budget=10` calling `fake-openai-endpoint` without `max_tokens`. Adds focused unit coverage in `tests/test_litellm/proxy/hooks/test_max_budget_limiter.py`. * harden bedrock file bucket validation * Fix syntax errors from botched merge in router.py * Fix Vertex batch output edge cases * [Fix] RBAC: Drop management_routes Write Fallback for Admin Viewer Greptile P1: the unsafe-method branch of `_check_proxy_admin_viewer_access` ended with a blanket `if route in management_routes: return`. That set is a mix of reads (info/list — handled via the safe-method GET branch above) and writes. The fallback let Admin Viewer POST to write endpoints not enumerated in `_ADMIN_VIEWER_BLOCKED_WRITE_ROUTES`, including: - /team/block, /team/unblock, /team/permissions_update - /jwt/key/mapping/{new,update,delete} - /key/bulk_update - /key/{key_id}/reset_spend Remove the fallback. The two remaining allow sets (admin_viewer_routes and global_spend_tracking_routes) are both read-only, so removal does not affect the legitimate POST-as-read cases (e.g. /spend/calculate, which is in spend_tracking_routes ⊂ admin_viewer_routes). Tests: - 8 new parametrized cases pinning each previously-leaking management write endpoint to 403 on POST for PROXY_ADMIN_VIEW_ONLY. * fix(tests): anchor VCR redis cassette key to repo root `os.path.relpath` with no `start` arg uses the current working directory, so running pytest from a subdirectory produced a different Redis key than running from the repo root. CI-recorded cassettes and locally-replayed runs would silently miss each other's cache. Anchor the path to the repo root (derived from `__file__`) so the key is stable regardless of CWD. https://claude.ai/code/session_018uCx7pcrkdUJZrCVMaTdPx * fix: gate key access_group override on group's own assignment Replaces the previous intersect-with-team.access_group_ids check, which made the override unreachable in practice (the team-gate fallback already covered every case the intersection allowed). The override now resolves each of the key's access_group_ids via get_access_object and accepts the group only if its assigned_team_ids includes the key's team_id, or its assigned_key_ids includes the key's token. This fulfills the original ask (a key can extend a team's allow-list via a group the admin granted to that team or that specific key) while still rejecting foreign groups referenced by team members of other teams. * [Fix] Proxy/Key Management: Honor team_member_permissions /key/list In /key/list Endpoint When a team grants /key/list via team_member_permissions, non-admin members should see all keys for that team — same as a team admin. Previously the classification in list_keys() only checked admin status, so permitted members fell into the service-account-only path and could not see other members' personal keys. Routes those members into the full-visibility set. * Fix access-group bypass via litellm-model fallback path When _get_all_deployments returns 0 candidates and the litellm-model fallback branch (_get_deployment_by_litellm_model) finds deployments that the access-group filter then empties, _access_group_filter_emptied_candidates remained False (it was captured before that branch ran). The router would then proceed to default fallbacks; the fallback model could have no access_groups and short-circuit the filter, silently serving a caller blocked by access-group restrictions. Update the flag inside the litellm-model branch when filtering empties a non-empty candidate set so the default-fallback guard still triggers. * fix(proxy): redact MCP server URL and headers for non-admin viewers (VERIA-8) Many MCP integrations (Zapier, etc.) embed an upstream API key directly in the server URL, e.g. ``https://actions.zapier.com/mcp/<api-key>/sse``. The list and single-server endpoints were returning the full URL to any authenticated user — `_redact_mcp_credentials` only stripped the explicit ``credentials`` field, and `_sanitize_mcp_server_for_virtual_key` only ran for restricted virtual keys. Non-admin internal users could read the dashboard, click the unmask toggle, and exfiltrate the raw token. Add `_sanitize_mcp_server_for_non_admin` that runs on top of the existing credential redaction and clears the credential-bearing fields: - ``url`` (the primary leak vector) - ``spec_path`` (OpenAPI spec URLs that may carry tokens) - ``static_headers`` / ``extra_headers`` (Authorization) - ``env`` (arbitrary secrets) - ``authorization_url`` / ``token_url`` / ``registration_url`` Identity fields (``server_id``, ``alias``, ``mcp_info``, etc.) are preserved so the UI can still list servers a non-admin's team has access to. Apply the new sanitizer in `fetch_all_mcp_servers` and the per-server fetch path right after the existing virtual-key branch. Update the existing `test_list_mcp_servers_non_admin_user_filtered` assertions that previously checked URL visibility. Frontend defense-in-depth: hide the URL unmask toggle on `mcp_server_view.tsx` unless the viewer is a proxy admin. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix runtime policy attachment initialization Mark runtime-created policies and attachments initialized so global policy attachments created from the policy builder apply immediately without requiring a restart. Co-authored-by: Cursor <cursoragent@cursor.com> * test(router): cover _try_early_resolve_deployments_for_model_not_in_names The router_code_coverage CI check requires every function in router.py to be referenced by at least one test under tests/{local_testing, router_unit_tests,test_litellm} in a file with "router" in its name. The recently-extracted helper had no direct test, so the check failed with "0.45% of functions in router.py are not tested". Add a focused test that exercises the four return paths: model already in self.model_names, no fallback applies, pattern-router match, and default_deployment substitution (also asserting the stored default isn't mutated). https://claude.ai/code/session_019AVp1XL7RT9RxRe4qRLkay * Fix policy registry teardown in tests Reset the policy ID index during policy engine test cleanup so stale policy versions cannot leak between tests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(batches): count non-chat tokens, validate batch-file model access (VERIA-39) (#27015) * fix(batches): count non-chat tokens and validate every model in batch file Two security control bypasses on POST /v1/batches: 1. `_get_batch_job_input_file_usage` only summed tokens for `body.messages` (chat completions). Embedding (`input`) and text completion (`prompt`) batches reported zero, letting massive non-chat workloads slip past TPM rate limits. Extend the counter to handle string and list shapes for both fields. 2. The batch input file was forwarded to the upstream provider without inspecting the models named inside the JSONL — only the outer `model` query parameter was checked against the caller's allowlist. A caller restricted to gpt-3.5 could submit a batch targeting gpt-4o and the upstream would execute it under the proxy's shared API key. Add `_get_models_from_batch_input_file_content` (returns the distinct `body.model` values) and call it from `_enforce_batch_file_model_access` in the pre-call hook, which runs each model through `can_key_call_model` so the same allowlist semantics (wildcards, access groups, all-proxy-models, team aliases) the proxy enforces on `/chat/completions` apply here too. Any unauthorized model raises a 403 before the file is forwarded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(batches): count pre-tokenized prompt/input shapes, classify 403 logs Two follow-ups from the Greptile review on the batch validation PR: 1. P1 TPM bypass via integer token arrays. The OpenAI batch schema accepts ``prompt`` and ``input`` as ``list[int]`` (a single pre-tokenized prompt) or ``list[list[int]]`` (multiple) in addition to the string and ``list[str]`` shapes. Pre-fix only the string shapes were counted, so a caller could submit a batch with hundreds of millions of pre-tokenized tokens and the rate limiter would record zero. Extract the per-field logic into ``_count_prompt_or_input_tokens`` and count each int as one token. 2. P2 access-denial logs were indistinguishable from I/O failures. ``count_input_file_usage`` caught every exception under a generic "Error counting input file usage" message, so an intentional 403 from ``_enforce_batch_file_model_access`` looked the same in the logs as a missing file or a Prisma timeout. Catch ``HTTPException`` separately and log 403s at WARNING level with a security-relevant message before re-raising. Tests cover the new shapes: single ``list[int]``, ``list[list[int]]`` (the worst-case bypass vector), and embeddings ``input`` with pre-tokenized arrays. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(proxy): re-validate user_id after /user/info re-parses query (#27009) * fix(proxy): re-validate user_id ownership after /user/info re-parses query The route-level access check in `RouteChecks.non_proxy_admin_allowed_routes_check` reads `request.query_params.get("user_id")`, which decodes literal `+` to spaces. The endpoint then re-parses the raw query string with `urllib.unquote` in `get_user_id_from_request` to preserve `+` characters (so plus-addressed emails work as user_ids). Those two paths produce different ids: a caller who registered a user_id containing a literal space could pass the route check and then read another user's row by sending the encoded `+` form. Add `_enforce_user_info_access` and call it after `_normalize_user_info_user_id` returns the final id. Proxy admin / view-only admin still bypass; everyone else must match the resolved user_id (or have no user_id, which falls back to the caller's own id later in the handler). Tests cover the admin bypass, owner-match path, and the cross-user lookup that this change blocks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(proxy): apply user_info ownership check to PROXY_ADMIN_VIEW_ONLY `_enforce_user_info_access` was bypassing both PROXY_ADMIN and PROXY_ADMIN_VIEW_ONLY, but the upstream route check in `RouteChecks.non_proxy_admin_allowed_routes_check` only treats PROXY_ADMIN as a true admin for the `/user/info` route — view-only admins go through the `user_id == valid_token.user_id` enforcement along with regular users. Mirroring that asymmetry left the same encoded-`+` bypass open for view-only admins whose user_id contains a literal space. Drop the PROXY_ADMIN_VIEW_ONLY exemption so the post-decode re-check matches the upstream rule. Update tests: a view-only admin must now be blocked from cross-user lookups but still allowed to read their own row. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(spend-logs): opt-in suppression of stack traces in spend-tracking error logs Adds LITELLM_SUPPRESS_SPEND_LOG_TRACEBACKS env var. When set to true and the proxy log level is INFO or above, spend-tracking error paths emit a single ERROR line without the full traceback. Stack traces are preserved at DEBUG and the Sentry / proxy_logging_obj.failure_handler path is unchanged. The new spend_log_error helper is wired through the spend write hot path: - DBSpendUpdateWriter (update_database, _update_*_db, batch upsert, redis-commit fallbacks) - _ProxyDBLogger._PROXY_track_cost_callback - get_logging_payload exception path - update_spend / update_daily_tag_spend / spend logs queue monitor Resolves LIT-2704. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(spend-logs): preserve no-traceback behavior for update_daily_tag_spend This call site previously logged a single-line error via verbose_proxy_logger.error() with no traceback. Switching it to spend_log_error(..., exc=e) caused a full stack trace to render by default (when LITELLM_SUPPRESS_SPEND_LOG_TRACEBACKS is unset), which contradicts the PR goal of leaving default behavior unchanged. Revert this specific site to the original error log call. * fix(spend-logs): preserve no-traceback behavior for update_daily_tag_spend Bugbot caught a regression: the previous error log here was a single-line verbose_proxy_logger.error(...) with no traceback. spend_log_error attaches the active exception's traceback by default (when the suppression env var is unset), so swapping it in changed default behavior. Revert this one site to its original .error() call to keep the PR strictly opt-in. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(spend-logs): suppress traceback in SpendLogs error_information row Extend LITELLM_SUPPRESS_SPEND_LOG_TRACEBACKS to the failure callback so the per-row Metadata pane in the UI no longer shows the stack trace when the opt-in env var is set, matching the existing console-side suppression. https://claude.ai/code/session_014dztoRbRnRvq54HL9EyHx6 * [Fix] Proxy: Repair Merge Fallout In Router-Override Fallback Auth Conflict resolution for #26968 dropped the `Iterator` typing import (NameError at module load), left a dead `fallback_models = cast(...)` block, and the new tests called `_enforce_key_and_fallback_model_access` without the now-required `request` kwarg. * isolate dual OTEL handlers * harden cloud file compatibility path * harden cloud file compatibility path * [Fix] Proxy/Key Management: Align Key-Org Membership Checks On Generate And Regenerate Mirrors the membership rule on /key/update so that /key/generate and /key/{key}/regenerate apply the same `_validate_caller_can_assign_key_org` gate when the caller specifies an `organization_id`. Proxy admins bypass. The check no-ops when `organization_id` is not being set. * thread trusted params through vertex file content * trust only server legacy file flag * chore(proxy): keep public AI hub unauthenticated * fix(proxy): preserve low-detail readiness status * [Test] Anthropic: Replace Legacy Claude-4-Sonnet Alias With Haiku 4.5 Three live-API tests pinned to claude-4-sonnet-20250514, which is a non-canonical alias of claude-sonnet-4-20250514. Anthropic's main API no longer resolves the legacy form under freshly issued keys, so the tests fail with not_found_error. The token counter test pinned to claude-sonnet-4-20250514 itself (deprecation_date 2026-05-14, two weeks out) was on borrowed time too. Bump all four to claude-haiku-4-5-20251001 — capability superset for what these tests exercise (streaming, parallel tool calling, extended thinking, token counting), no upcoming deprecation, cheaper per-token. * chore(proxy): move URL-valued model/file_id guard from SDK to proxy The previous per-provider guards in HuggingFace, Oobabooga, and Gemini files lived in the SDK layer, breaking SDK callers who legitimately pass URL-valued model identifiers. Move the check to the proxy boundary in add_litellm_data_to_request so SDK users keep working while proxy users default-deny URL-valued model and file_id, with admin opt-in via litellm.provider_url_destination_allowed_hosts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [Chore] Proxy/UI: Drop stray _experimental/out/chat/index.html This file is a regenerable UI build artifact that should not be tracked in source. Removing so the merge into litellm_internal_staging stays clean. * [Test] Anthropic Passthrough: Bump Streaming Cost-Injection Test To Haiku 4.5 test_anthropic_messages_streaming_cost_injection hits the proxy's /v1/messages route, which routes via the anthropic/* wildcard to api.anthropic.com. The 404 surfaced in the test was Anthropic's own not_found_error propagated back through the proxy (visible from the x-litellm-model-id hash on the response — the proxy did route). Same root cause as the prior commit: the legacy claude-4-sonnet-20250514 alias is no longer recognized by Anthropic's main API under the new key. Swap to claude-haiku-4-5-20251001 — same routing path, canonical model. * fix(proxy): handle ownership-recording failures after upstream create If record_container_owner raises after the upstream container is created, the user previously got a 500 with no usable container — they were billed for an unreachable resource. Move ownership recording into the create path's exception handling and split the two failure modes: - HTTPException from the recorder (auth conflicts) propagates verbatim so the client sees the real status code, not a generic LLM error. - Unexpected exceptions are logged and swallowed; the response is returned to the caller so they aren't billed for a container they can't address. The DB row stays untracked until an operator reconciles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(guardrails): close post-call coverage gaps * fix(types): add /team/permissions_bulk_update to management_routes The blocklist check in _check_proxy_admin_viewer_access only fires for routes that match LiteLLMRoutes.management_routes — the bulk-update endpoint was missing from that list, so the test for view-only admins on /team/permissions_bulk_update fell through to "allow." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [Test] Anthropic Passthrough: Bump Thinking Tests Off Legacy Sonnet 4 Alias base_anthropic_messages_test.test_anthropic_messages_with_thinking and test_anthropic_streaming_with_thinking still pinned to claude-4-sonnet-20250514 — the same legacy alias Anthropic no longer recognizes under freshly issued keys. The other four tests in this base class already use claude-sonnet-4-5-20250929; these two were missed. Bump to claude-haiku-4-5-20251001 (supports_reasoning=true, no upcoming deprecation). Subclasses including TestAnthropicPassthroughBasic inherit these methods. * fix(guardrails): cover multi-choice output variants * fix(proxy): preserve public ai hub ui setting * fix(scim): cascade FK cleanup on user delete and surface block status in UI SCIM DELETE /Users/{id} previously called litellm_usertable.delete without clearing rows that FK back to the user, so Postgres rejected the delete with LiteLLM_InvitationLink_user_id_fkey and the SCIM caller saw a 500. Add a helper to drop invitation_link, organization_membership, and team_membership rows before the user delete (mirrors /user/delete in internal_user_endpoints). Also add a Status column to the Virtual Keys and Internal Users tables so admins can see at a glance which keys are blocked and which users SCIM has deactivated. SCIM-blocked keys carry a tooltip explaining the origin. Pin the dashboard's Node version to 20 via .nvmrc to match CI. * chore: update Next.js build artifacts (2026-05-02 03:21 UTC, node v20.20.2) * perf(proxy): cache container/skill ownership reads on the hot path Container ownership and skill rows are looked up on every retrieve / delete / list / file-content / chat-completion-with-skill call. The new stores wrapped raw Prisma queries with no cache, putting one DB round-trip on each request. Add an in-process TTL'd cache mirroring the _byok_cred_cache pattern in mcp_server/server.py: per-key (value, monotonic_timestamp), 60s TTL, 10000-entry cap with full-clear on overflow, invalidated by every write. Negative results (`None`) are cached too so untracked-resource checks also skip the DB. Tests cover: cache-after-first-hit, negative caching, write invalidation, no-caching-on-DB-error, TTL expiry, capacity eviction. 56 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update Next.js build artifacts (2026-05-02 03:39 UTC, node v20.20.2) * fix: remove traceback key instead of it being "" * fix: linting error * fix(scim): preserve scim_active on PUT when client omits the field A SCIM PUT may legally omit `active` (full-replace with the field absent). Pydantic fills the SCIMUser.active default of True, so the PUT handler was overwriting metadata.scim_active with True even when the client never sent it — silently reactivating a previously SCIM-blocked user and unblocking their keys. Use model_fields_set to detect whether the client actually sent `active`. If omitted, preserve the prior scim_active value and skip the cascade to virtual keys. Also drop comments added in this PR that just narrate what the code does; keep only the docstrings and the SQL-NULL pitfall note that explain non-obvious behaviour. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy): use set lookup for permitted agent filters * fix(mcp): redact command fields for non-admin server views * fix(proxy): forward decoded container ids after ownership checks * fix(caching): handle stale isolated Redis semantic index * fix(cloudflare): support response_text in streaming chunk parser Newer Cloudflare Workers AI models (e.g. Nemotron) emit 'response_text' instead of 'response' on streamed chunks. The non-streaming path was already updated to fall back to 'response_text' (#26385), but the streaming chunk parser still only read 'response', which caused streaming requests against those models to silently produce empty content. Mirror the non-streaming fallback in CloudflareChatResponseIterator.chunk_parser and add a streaming test for the response_text shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Fix code qa * Address bugbot: drop dead encode/decode helpers; preserve empty custom_id - Remove unused _encode_gcp_label_value / _decode_gcp_label_value singular helpers; only the _chunks variants are actually called. - Use 'is not None' check for custom_id so empty-string custom_ids are still labeled and round-trip through batch outputs. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Forward Vertex file content logging context * test vertex file content logging forwarding Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * Fix Vertex batch output logging mutation * fix: don't mutate caller's logging_obj in _try_transform_vertex_batch_output_to_openai The method was overwriting logging_obj.optional_params, logging_obj.model, and logging_obj.start_time on the caller's Logging instance. When invoked from llm_http_handler.py's generic framework path, the framework's own logging_obj (which already went through pre_call) had its properties clobbered, causing model and start_time to reflect the last batch line's values rather than the original call context. Fix: create a fresh local Logging instance for the per-line transformation instead of mutating the incoming logging_obj. The caller's object is now left entirely untouched regardless of whether a logging_obj was passed in or not. Regression tests added to verify model, start_time, and optional_params are not mutated on the caller's logging_obj. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * feat: add opt-out flag for Vertex batch output transformation Adds litellm.disable_vertex_batch_output_transformation (default False). When True, afile_content returns raw Vertex predictions.jsonl untouched so users that parse candidates/modelVersion directly are not broken. * fix(anthropic,bedrock): omit thinking/output_config when reasoning_effort="none" Setting reasoning_effort="none" on Anthropic chat models (direct, Bedrock Invoke, Bedrock Converse, Vertex AI Anthropic, Azure AI Anthropic) crashed LiteLLM with: litellm.APIConnectionError: 'NoneType' object has no attribute 'get' Both the Anthropic chat transformation and Bedrock Converse called ``AnthropicConfig._map_reasoning_effort`` and assigned the ``None`` it returns for ``"none"`` directly to ``optional_params["thinking"]``. Downstream ``is_thinking_enabled`` then did ``optional_params["thinking"].get("type")`` and crashed. Pop ``thinking`` (and on Claude 4.6/4.7, ``output_config``) instead of assigning ``None``, restoring the documented contract that ``reasoning_effort="none"`` means "do not enable thinking". This also prevents downstream Anthropic 400s ("thinking: Input should be an object", "output_config.effort: Input should be ...") if the bug were ever masked. Verified end-to-end against the live Anthropic API and Bedrock Converse on claude-opus-4-{5,6,7} and claude-sonnet-4-6, plus Bedrock Invoke for Claude 4.5/4.6. Vertex AI Anthropic and Azure AI Anthropic inherit the fixed ``map_openai_params`` from ``AnthropicConfig`` and need no further changes. * fix(vertex-ai): set response=null on batch error entries per OpenAI spec The Vertex batch output transformer was emitting both a populated 'response' and 'error' for failed batch entries. The OpenAI Batch output spec defines them as mutually exclusive: on error 'response' MUST be null. This broke any consumer using 'result["response"] is None' to detect failures. * test(vertex-ai): cover transformation_error path emits response=null * fix(security): sandbox jinja2 in gitlab/arize/bitbucket prompt managers DotpromptManager was hardened to render through ImmutableSandboxedEnvironment. The three sibling managers (gitlab, arize, bitbucket) were missed and still instantiate plain jinja2.Environment(), leaving the same attribute-traversal SSTI primitive open: a template fetched from a GitLab/BitBucket repo or Arize Phoenix workspace can reach __class__.__init__.__globals__ and execute arbitrary Python on the proxy host. Match the dotprompt pattern by switching all three to ImmutableSandboxedEnvironment. The sandbox blocks the dunder-traversal chain while leaving normal {{ var }} substitution intact, so the template surface is unchanged for legitimate use. Adds tests/test_litellm/integrations/test_prompt_manager_ssti.py (18 cases) verifying each manager's jinja_env is a sandbox, that classic SSTI payloads raise SecurityError, and that ordinary variable rendering still works. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(proxy): drop client-supplied pricing fields from request bodies The proxy currently forwards request-body pricing parameters (the fields on `CustomPricingLiteLLMParams`, plus `metadata.model_info`) into the core call path. Those fields belong to deployment configuration, not to per-request input — sending them from a client mutates the request's recorded cost and, via `litellm.completion` → `register_model`, the process-wide `litellm.model_cost` map for every later caller in the worker. Strip them at the boundary. The strip set is built from `CustomPricingLiteLLMParams.model_fields` so pricing fields added later are covered automatically. Operators who do want clients to supply per-request pricing can opt back in per key or team via `metadata.allow_client_pricing_override = true`, mirroring the existing `allow_client_mock_response` and `allow_client_message_redaction_opt_out` flags. Tests cover the strip set's coverage, root and metadata strips, the opt-in skip on both key and team metadata, and a regression check that the global `litellm.model_cost` map is unmutated after a stripped request. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(proxy): log stripped pricing fields at debug for operator visibility Operators upgrading would otherwise see client-supplied pricing overrides silently stop applying with no diagnostic. Emit a debug-level line listing the dropped fields and pointing at the opt-in flag when any are stripped; stay silent on the no-op path so the log isn't filled with noise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(proxy): move pricing strip below the litellm_metadata JSON-string parse The strip ran before the proxy parses ``litellm_metadata`` from a JSON string into a dict (a path used by multipart/form-data and ``extra_body`` callers), so ``isinstance(metadata, dict)`` was False and ``model_info`` survived the strip. Move the call to the same post-parse position the ``user_api_key_*`` strip already uses for the same reason. Adds a regression test exercising the JSON-string ``litellm_metadata`` path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(responses): replace legacy claude-4-sonnet alias in multiturn tool-call test Anthropic's main API no longer resolves the non-canonical 'claude-4-sonnet-20250514' alias for freshly issued keys, returning 404 not_found_error. PR #27031 already swept three other live tests pinned to this alias to claude-haiku-4-5-20251001 but missed test_multiturn_tool_calls in the responses API suite, which is now failing reliably on PR CI runs (e.g. PR #27074, job 1603363). Bump the two model references in test_multiturn_tool_calls to the same claude-haiku-4-5-20251001 snapshot used by PR #27031 -- it covers everything this test exercises (tool calling, multi-turn) and isn't on a deprecation schedule. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(proxy): close callback-config and observability-credential side channels Two related gaps in the proxy's request bouncer: 1. ``is_request_body_safe`` (auth_utils.py) walked the request-body root and the ``litellm_embedding_config`` nested dict, but not ``metadata`` or ``litellm_metadata``. The same fields it bans at root — Langfuse / Langsmith / Arize / PostHog / Braintrust / Phoenix / W&B Weave / GCS / Humanloop / Lunary credentials and routing — were silently accepted when the caller put them inside metadata, retargeting observability callbacks to a caller-controlled host with caller-supplied creds. Walk both metadata containers (and parse the JSON-string form sent via multipart / ``extra_body``) through the same banned-params helper, so the existing ``allow_client_side_credentials`` opt-in covers both paths consistently. 2. The banned-params list was hand-maintained and lagged the canonical ``_supported_callback_params`` allow-list in ``initialize_dynamic_callback_params``. Derive the observability bans from that allow-list (minus a small ``_SAFE_CLIENT_CALLBACK_PARAMS`` set for informational fields like ``langfuse_prompt_version`` and ``langsmith_sampling_rate``) so future integrations are covered automatically; ``_EXTRA_BANNED_OBSERVABILITY_PARAMS`` carries the handful of fields integrations read but the allow-list hasn't caught up to. A guard test fails CI if a new entry is added to ``_supported_callback_params`` without an explicit safe-list decision. Separately in ``litellm_pre_call_utils.py``: add ``callbacks``, ``service_callback``, ``logger_fn``, and ``litellm_disabled_callbacks`` to ``_UNTRUSTED_ROOT_CONTROL_FIELDS``. The first three are appended to worker-wide ``litellm.{input,success,failure,_async_*,service}_callback`` lists / ``litellm.user_logger_fn`` from inside ``function_setup`` — one request poisons every subsequent caller in that worker. The last is the inverse primitive: the legitimate path reads it from key/team metadata, the request-body version silently disables admin-configured audit / observability for the call. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(auth): per-param allow must continue, not return early A pre-existing logic bug in ``_check_banned_params``: when the deployment-level ``configurable_clientside_auth_params`` permitted one banned field, the loop ``return``-ed on the first match instead of ``continue``-ing, so any other banned param later in the same body or metadata dict was never checked. This PR's metadata walk multiplies the surface where that bypass matters — a body pairing an allowed ``api_base`` with an observability credential like ``langfuse_host`` would silently pass. Proxy-wide ``allow_client_side_credentials`` keeps ``return`` (it's a global opt-in for every banned param). The per-param branch becomes ``continue`` so only the one explicitly-permitted field is skipped. Adds a regression test that exercises the api_base + langfuse_host pair. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(vector_store): resolve embedding config at request time, never persist creds The vector store create/update path previously called ``_resolve_embedding_config`` against the admin-configured router/DB model and persisted the resolved ``litellm_embedding_config`` dict (``api_key`` / ``api_base`` / ``api_version``) into the ``litellm_managedvectorstorestable.litellm_params`` column. Because the resolver expanded ``os.environ/...`` references via ``get_secret``, the DB row carried cleartext provider credentials, and the ``/vector_store/{new,info,update,list}`` responses returned them to any authenticated caller who could supply a known admin model name. Move the auto-resolve out of ``create_vector_store_in_db`` and out of the update path. Persist only the user-supplied ``litellm_embedding_model`` reference. Resolve at request-handling time inside ``_update_request_data_with_litellm_managed_vector_store_registry`` so the resolved config lives in the per-request ``data`` dict and is garbage-collected after the response. Legacy rows that were created by an earlier proxy version and already carry a resolved ``litellm_embedding_config`` skip the re-resolution and pass through unchanged so embedding calls keep working. The ``new_vector_store`` response now also runs the existing ``_redact_sensitive_litellm_params`` masker (already used by ``info``, ``update``, and ``list``), defending against caller-supplied cleartext on the create path and against legacy rows whose persisted credentials are still in the database. Existing tests that asserted the old write-time-resolve behaviour are updated to assert the new persistence shape (no embedding config stored, just the model reference). Two new tests cover the use-time path: one asserting fresh resolution happens when a row carries only the model reference, the other asserting legacy rows with persisted config skip re-resolution and continue to work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(vector_store): tighten registry-mutation comment and dedupe test helpers * fix(vector_store): cache use-time embedding-config resolution Hold the resolved config in a process-memory TTL cache so the request-handling path doesn't run litellm_proxymodeltable.find_first on every vector-store call. * fix(anthropic,bedrock,vertex): forward output_config.effort + 400 on garbage reasoning_effort Follow-up bugs surfaced by the QA sweep on PR #27039 (https://github.com/BerriAI/litellm/pull/27039#issuecomment-4363363610). 1. Stop stripping output_config.effort on Bedrock + Vertex adaptive routes. - Vertex AI Claude 4.6/4.7 accepts output_config.effort on rawPredict (verified end-to-end against us-east5 / global). The strip helper now no-ops for effort. - Bedrock Converse routes output_config into additionalModelRequestFields for anthropic base models so the requested adaptive tier (low/medium/ high/xhigh/max) actually reaches the wire instead of all collapsing to identical thinking. - Bedrock Invoke chat transformation (AmazonAnthropicClaudeConfig) stops popping output_config from the post-AnthropicConfig request body. - Bedrock Invoke /v1/messages allowlist (BedrockInvokeAnthropicMessagesRequest) now lists output_config so the runtime allowlist filter forwards it. 2. Validate effort across Bedrock Converse so 'disabled' / 'invalid' / '' / unsupported tiers (xhigh/max on Sonnet 4.6 or budget-mode 4.5 models) surface as a clean 400 BadRequestError instead of 500. 3. ValueError -> BadRequestError throughout (AnthropicConfig.map_openai_params, _apply_output_config, AmazonConverseConfig._handle_reasoning_effort_parameter). Empty-string effort is now rejected (was silently passing the 'if effort and ...' short-circuit). 4. Floor reasoning_effort='minimal' at the Anthropic provider minimum (1024 budget_tokens) via new ANTHROPIC_MIN_THINKING_BUDGET_TOKENS so it's a usable tier on direct Anthropic / Azure AI Anthropic / Vertex AI Anthropic / Bedrock Invoke (all of which 400 below 1024). 5. model_prices: dedupe duplicate supports_max_reasoning_effort key on claude-opus-4-7 / claude-opus-4-7-20260416. Adds regression tests across all five affected paths; existing tests asserting the silent-strip behavior were updated to reflect the new pass-through and clean 400 surfaces. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(constants): make ANTHROPIC_MIN_THINKING_BUDGET_TOKENS a plain constant The documentation CI test (tests/documentation_tests/test_env_keys.py) asserts every os.getenv() key in the source has a matching entry in the litellm-docs config_settings.md table. ANTHROPIC_MIN_THINKING_BUDGET_TOKENS tracks Anthropic's published wire-protocol minimum (1024) — it's not a user-tunable, so making it env-overridable was wrong anyway. Drop the os.getenv() wrapper; the value is now a plain literal. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(anthropic,bedrock): correct effort error message and dedupe effort_map - Remove 'none' from the Bedrock _validate_anthropic_adaptive_effort error message; it was listed as a valid value but rejected by the membership check, leaving users in a feedback loop if they tried 'none'. - Hoist the duplicated reasoning_effort -> output_config.effort mapping out of AnthropicConfig.map_openai_params and AmazonConverseConfig._handle_reasoning_effort_parameter into a single AnthropicConfig.REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT class constant so the two routes cannot drift. * fix(anthropic): translate reasoning_effort on /v1/messages route Closes the remaining QA-sweep gap on PR #27074: Bedrock Invoke /v1/messages was silently ignoring ``reasoning_effort`` because the shared param filter only kept native Anthropic keys, so every effort tier collapsed to the same behavior on the wire (27/231 cells failing across opus-4-5 / opus-4-6 / sonnet-4-6). Map ``reasoning_effort`` to native Anthropic ``thinking`` / ``output_config.effort`` at the ``AnthropicMessagesConfig`` layer so all four /v1/messages routes (direct Anthropic, Azure AI, Vertex AI, Bedrock Invoke) inherit the same translation: - Add ``reasoning_effort`` to ``AnthropicMessagesRequestOptionalParams`` so the param filter in ``AnthropicMessagesRequestUtils.get_requested_anthropic_messages_optional_param`` no longer drops it before the transformation runs. - Add ``_translate_reasoning_effort_to_anthropic`` and call it from ``transform_anthropic_messages_request``. Mirrors ``AnthropicConfig.map_openai_params`` on the chat completion path (re-uses ``_map_reasoning_effort`` and ``REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT``) so the two routes cannot drift. Pops ``reasoning_effort`` so it never reaches the wire. - Caller-supplied native ``thinking`` / ``output_config.effort`` always win — same precedence as ``_translate_legacy_thinking_for_adaptive_model``. - Garbage values (``""``, ``"disabled"``, ``"invalid"``) raise ``AnthropicError(status_code=400)`` instead of falling through and surfacing as 500s from the provider. - ``"none"`` clears thinking + output_config so callers can opt out per request. Also restores the non-adaptive-model test coverage on Bedrock Invoke /v1/messages that the previous commit lost when ``test_bedrock_messages_strips_output_config`` was renamed to the ``forwards`` variant on Opus 4.7. Adds a new test file ``test_reasoning_effort_translation.py`` covering the translation at the shared config level (adaptive + non-adaptive models, none, garbage, caller precedence) so all four /v1/messages routes are exercised by a single suite. Adds parametrized + behavioral tests on the Bedrock Invoke /v1/messages suite covering: minimal/low/medium/high/xhigh/max mapping for adaptive models, thinking-budget mapping for non-adaptive Opus 4.5, ``none`` clears both, garbage raises 400, explicit ``output_config`` wins. Refs: https://github.com/BerriAI/litellm/pull/27074 * fix(anthropic,bedrock): reject unmapped reasoning_effort at mapping site Both the chat completion path (AnthropicConfig.map_openai_params) and the Bedrock Converse path (_handle_reasoning_effort_parameter) used REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT.get(value, value) which falls back to the raw input on unmapped keys. Combined with _map_reasoning_effort returning type='adaptive' for any string on Claude 4.6/4.7, garbage values (e.g. 'disabled') could leak into optional_params['output_config']['effort'] unvalidated if map_openai_params ran without the downstream transform_request or _validate_anthropic_adaptive_effort check. Mirror the /v1/messages pattern: use .get(value) (no fallback) and raise BadRequestError immediately when the value is unmapped, co-locating validation with the mapping for defense in depth. * style: black formatting Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(anthropic): stop class-attr leak; gate xhigh/max on every route The reasoning-effort mapping dict was a public class attribute on AnthropicConfig, so BaseConfig.get_config returned it as a request parameter and every Anthropic-backed call (Anthropic / Azure / Vertex / Bedrock Invoke) hit a 400 'REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT: Extra inputs are not permitted' from the provider. Move the mapping to a module-level constant. _supports_effort_level only looked the model up under custom_llm_provider='anthropic', so bedrock-prefixed model ids (e.g. bedrock/invoke/us.anthropic.claude-opus-4-7) returned False for both 'max' and 'xhigh' even when the underlying model entry has the flag set. Strip known provider prefixes and retry the lookup against litellm.model_cost directly so per-model gating works on every route. Mirror the per-model xhigh/max gate from AnthropicConfig._apply_output_config in AnthropicMessagesConfig._translate_reasoning_effort_to_anthropic so the /v1/messages route also raises a clean 400 instead of forwarding the unsupported tier. * feat(anthropic,bedrock): strip output_config under drop_params for non-effort models When a proxy fronts Claude Code (which always sends `output_config.effort`) at a pre-4.5 Anthropic model — haiku-3, sonnet-3.5, opus-3, sonnet-4 — the forwarded knob causes a forced 400 the client can't fix. Gating a strip behind the existing `drop_params` flag lets operators opt into silent fixup once and stop worrying about per-model param hygiene. Default (`drop_params=False`) still forwards and surfaces the provider's error, preserving the strict, debuggable contract from #27074. Per https://platform.claude.com/docs/en/build-with-claude/effort the supporting set is Opus 4.5+, Sonnet 4.6+, and Mythos Preview; everything else is dropped (with a verbose_logger warning so the strip is visible). Recognition uses model-name patterns plus a fallback to any `supports_*_reasoning_effort` flag in the model map for forward compatibility with new entries. https://claude.ai/code/session_01WjHq31rvXT6xYNdVmSJvRp (cherry picked from commit |
||
|
user
|
7497674661 | fix(proxy): sanitize redaction controls at ingress | ||
|
user
|
842eea0131 | chore(proxy): harden request control fields | ||
|
Sameer Kankute
|
b516120036
|
Merge pull request #26737 from BerriAI/litellm_internal_staging
merge internal staging |
||
|
Sameer Kankute
|
cf74f55b79
|
Fix extra body error | ||
|
milan-berri
|
10aed9e981
|
feat(logging): add retry settings for generic API logger (#26645)
* Add retry settings for generic API logger Made-with: Cursor * Refine generic API retry behavior Made-with: Cursor |
||
|
ishaan-berri
|
8a9faa81b2
|
feat(guardrails): LLM-as-a-Judge guardrail (#26360)
* feat(guardrails): add LLM_AS_A_JUDGE to SupportedGuardrailIntegrations * feat(types): add EvalVerdict, StandardLoggingEvalInformation; wire eval_information into SpendLogsMetadata * feat(guardrails): add self-contained llm_as_a_judge guardrail hook * fix(a2a): filter agent-only litellm_params from acompletion kwargs; pass agent_id into body * feat(ui): add LLMJudgeFields criteria builder component * feat(ui): wire LLM-as-a-Judge into add guardrail form * feat(ui): update EvalViewer — title 'LLM Judge Results', weighted score column, summary row * fix(ui): wire EvalViewer into LogDetailContent to show LLM judge results on logs page * fix(guardrails-ui): route llm_as_a_judge to criteria builder step; rename to LiteLLM LLM as a Judge; add litellm logo * fix(guardrail-viewer): stack lifecycle + eval details vertically to avoid badge overflow in narrow drawer * fix(guardrail-create): surface config validation errors on create instead of silently orphaning guardrail in DB * fix(guardrail-registry): hardcode llm_as_a_judge in initializer registry so it loads regardless of package install path * fix(llm-as-a-judge): fix P1 code quality issues - validate weights/on_failure, guard pre_call, handle multimodal, move imports to module level, fix spurious finally logging * fix(guardrail_endpoints): use correct PK field in rollback delete and log rollback failure * fix(llm_as_a_judge): support Pydantic object in _get_litellm_param fallback chain * fix(LLMJudgeFields): replace @tremor/react Button with antd Button * fix(llm_as_a_judge): remove dead registry dicts, fix KeyError in prompt builder, set correct status on judge failure * test(llm_as_a_judge): add unit tests for guardrail hook * fix(llm_as_a_judge): remove @log_guardrail_information decorator to fix duplicate guardrail_information entries The decorator and the manual finally block both called add_standard_logging_guardrail_information_to_request_data, producing two entries per request. The decorator also misclassified HTTPException(422) blocks as guardrail_failed_to_respond (it checks for 400). The finally block correctly tracks status throughout, so removing the decorator is sufficient. * fix(test_gcs_pub_sub): ignore metadata.eval_information in comparison * fix(test_spend_management): ignore metadata.eval_information in payload comparison * fix(types/guardrails): add input_type and messages to ApplyGuardrailRequest * fix(guardrail_endpoints): pass input_type and messages through apply_guardrail endpoint * fix(guardrail_endpoints): auto-detect post_call guardrails and use input_type=response * fix(a2a_endpoints): merge agent litellm_params guardrails into data before post_call hooks * fix(llm_as_a_judge): use float sum with tolerance for weight validation * fix(guardrail_registry): split long import line for black formatting * fix(llm_as_a_judge): guard guardrail_name Optional for mypy * fix(llm_as_a_judge): set guardrail_status=guardrail_intervened when score fails, regardless of on_failure mode * fix(a2a_endpoints): use try/finally so deferred spend log fires even when guardrail blocks with 422 * fix(litellm_logging): declare _defer_async_logging and _enqueue_deferred_logging on Logging class for mypy * fix(logging_worker): restore queue.join() in flush() to wait for in-flight callbacks |
||
|
ishaan-berri
|
8a4a775b1b
|
fix(logging): add litellm_call_id to StandardLoggingPayload and OTel span (#26133)
* add litellm_call_id field to StandardLoggingPayload * populate litellm_call_id in get_standard_logging_object_payload * emit litellm.call_id span attribute in OTel integration * test: litellm_call_id is present in StandardLoggingPayload * test: litellm.call_id emitted as OTel span attribute * test: allow litellm. prefix attributes in redacted span validator |
||
|
Yuneng Jiang
|
11c3270cdc
|
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_yj_apr17
# Conflicts: # litellm/__init__.py |
||
|
Yuneng Jiang
|
ee2cf0e6e8
|
fix: address three CI failures from recent security PR merges
- url_utils.py: narrow sockaddr[0] from str|int to str via a helper with a fail-closed isinstance check. Fixes the two mypy errors introduced by the SSRF hardening without masking unexpected stdlib behavior. - key_management_endpoints.py: restore the documented team member_permissions path for /key/update. The cross-key admin check added to close the cross-org rewrite attack was over-broad: it rejected non-admin team members even when can_team_member_execute_key_management_endpoint had already validated their team membership and /key/update grant. Now skip the admin check when the key has a team_id and the change is non-budget (membership + permission already enforced above). Budget/spend changes still require team/org admin. The cross-org attack remains blocked: an outside org admin fails the earlier team membership check. - test_logging_redaction_e2e_test.py: rename and rewrite two parametrized tests to assert that request-body turn_off_message_logging has no effect. Reflects the intentional removal of turn_off_message_logging from _supported_callback_params so the caller cannot override admin logging policy via the request body. - test_key_management_endpoints.py: add two tests covering the restored team member permission path — one positive (non-budget update succeeds for a team member with /key/update grant), one negative (max_budget change still rejected without admin role). |
||
|
Ishaan Jaffer
|
e8461b5b97
|
style: run black formatter on files from main merge | ||
|
Ishaan Jaffer
|
98c2d90f5c
|
fix(logging): update test_get_additional_headers to reflect provider header passthrough | ||
|
David Chen
|
b7ccc5b691
|
[Test Fix] fix gov pricing tests (#25022)
* fix pricing tests * fix mypy * fix cost expectation since us based model is used now. * fix test get model info |
||
|
David Chen
|
d1df4e838b
|
Litellm fix update bedrock models (#24947)
* update bedrock models in tests * updated more tests and model_prices_and_context_window * fix model id and pricing * replace more sonnet models * update tests * git push * update pricing * flaky total cost * monkey patch * relax the cost change * fix and revert some changes * revert the pricing * chore: move cost/pricing changes to bedrock-cost-fixes branch * chore: split Bedrock file-api beta stripping to separate branch Removes strip_unsupported_file_api_betas_for_bedrock_invoke from this branch; see litellm_bedrock_invoke_strip_file_api_betas for that fix. Made-with: Cursor |
||
|
ishaan-berri
|
e4442a4d98
|
test fix us.anthropic.claude-haiku-4-5-20251001-v1:0 (#24931)
* test fix us.anthropic.claude-haiku-4-5-20251001-v1:0 * ignore mypy cache files --------- Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> Co-authored-by: David Chen <clfhhc@gmail.com> |
||
|
Ishaan Jaffer
|
0298c1f58d | test_basic_s3_v2_logging | ||
|
Ishaan Jaffer
|
443566d4f5 | test fixes | ||
|
Ishaan Jaffer
|
28afbc152f | test_async_gcs_pub_sub_v1 | ||
|
Ishaan Jaffer
|
431782c3fe | test azure blob storage | ||
|
Krrish Dholakia
|
25f2baad71 | test: cleanup dead tests | ||
|
Krrish Dholakia
|
0fef88d67c | test: remove dead tests | ||
|
Krrish Dholakia
|
bc829d51f2 | test: test | ||
|
Ishaan Jaff
|
81dadb698a
|
Ishaan - March 18th changes (#24056)
* add DD Tracing (#24033) * feat(models): add Azure GPT-5.4 mini and nano variants (#24045) Add `azure/gpt-5.4-mini` and `azure/gpt-5.4-nano` to the model database with official pricing from Azure OpenAI: - GPT-5.4 mini: $0.75/M input, $0.075/M cached, $4.5/M output - GPT-5.4 nano: $0.20/M input, $0.02/M cached, $1.25/M output Both models support: - 1.05M input / 128K output context window - Chat, batch, and responses endpoints - Function calling, tools, vision, reasoning - Prompt caching with automatic tiered pricing Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * Add new model pricing details for volcengine Doubao-Seed-2.0 series (#23871) Add entries for volcengine Doubao-Seed-2.0 series * fix(mcp): support refresh_token grant type in OAuth token endpoint (#23701) * fix(mcp): support refresh_token grant type in OAuth token endpoint (#23700) The .well-known/oauth-authorization-server metadata advertises refresh_token as a supported grant type, but the token endpoint rejected it with HTTP 400. This adds refresh_token grant support so MCP clients can refresh expired tokens without re-authenticating. * test(mcp): add tests for refresh_token grant type in OAuth token endpoint * fix(mcp): move code_verifier guard into authorization_code branch code_verifier is only relevant for authorization_code grants (PKCE). Move it inside the else branch so it doesn't apply to refresh_token. * fix(mcp): guard None client_secret and forward scope in token exchange - Conditionally include client_secret in form data to prevent httpx from sending the literal string "None" (applies to both authorization_code and refresh_token branches) - Forward optional scope parameter per RFC 6749 §6, allowing clients to request a subset of originally-granted scopes on refresh * fix(mcp): validate code param in authorization_code grant Guard against None code being form-encoded as literal string "None" by httpx, symmetric with the existing refresh_token guard. * docs: add incident report for guardrail logging secret exposure (#24059) Add blog post documenting the guardrail logging path exposing internal request data (e.g. Authorization headers) in spend logs and OTEL traces. Fix available in LiteLLM 1.82.3+. Made-with: Cursor * [Fix] Datadog LLM Observability tags format (env, service, version missing) (#23673) * tag fix * greptile comment * fix(ci): stabilize 6 failing CI jobs 1. mypy: remove duplicate type annotation for token_data in discoverable_endpoints.py 2. integrations tests: add parameterized to CI test deps 3. doc quality: document OTEL_IGNORE_CONTEXT_PROPAGATION env key 4. security: allowlist CVE-2026-2673, CVE-2026-3644, CVE-2026-4224 (no fix available) 5. proxy_store_model_in_db: fix missing x-litellm-call-id header on error responses 6. google tests: add --retries 3 for transient Vertex AI rate limits Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(streaming): handle RuntimeError during model_copy in streaming handler The race condition occurs when model_copy(deep=True) tries to deepcopy _hidden_params dict while it's being concurrently modified by logging callbacks. Fall back to shallow copy if the deep copy fails. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(cost): handle non-string traffic_type in cost calculator + add retries 1. Fix AttributeError in _map_traffic_type_to_service_tier when traffic_type is an integer (cast to str before calling .upper()). This was causing pass-through vertex spend logging to fail silently. 2. Add --retries to llm_translation_testing for flaky external API calls. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> --------- Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: ExMatics HydrogenC <33123710+HydrogenC@users.noreply.github.com> Co-authored-by: Jack Venberg <jack.venberg@rover.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> |
||
|
Ishaan Jaff
|
8e61b32b8e
|
[Staging] - Ishaan March 17th (#23903)
* feat(xai): add grok-4.20 beta 2 models with pricing (#23900)
Add three grok-4.20 beta 2 model variants from xAI:
- grok-4.20-multi-agent-beta-0309 (reasoning + multi-agent)
- grok-4.20-beta-0309-reasoning (reasoning)
- grok-4.20-beta-0309-non-reasoning
Pricing (from https://docs.x.ai/docs/models):
- Input: $2.00/1M tokens ($0.20/1M cached)
- Output: $6.00/1M tokens
- Context: 2M tokens
All variants support vision, function calling, tool choice, and web search.
Closes LIT-2171
* docs: add Quick Install section for litellm --setup wizard (#23905)
* docs: add Quick Install section for litellm --setup wizard
* docs: clarify setup wizard is for local/beginner use
* feat(setup): interactive setup wizard + install.sh (#23644)
* feat(setup): add interactive setup wizard + install.sh
Adds `litellm --setup` — a Claude Code-style TUI onboarding wizard that
guides users through provider selection, API key entry, and proxy config
generation, then optionally starts the proxy immediately.
- litellm/setup_wizard.py: wizard with ASCII art, numbered provider menu
(OpenAI, Anthropic, Azure, Gemini, Bedrock, Ollama), API key prompts,
port/master-key config, and litellm_config.yaml generation
- litellm/proxy/proxy_cli.py: adds --setup flag that invokes the wizard
- scripts/install.sh: curl-installable script (detect OS/Python, pip
install litellm[proxy], launch wizard)
Usage:
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh
litellm --setup
* fix(install.sh): remove orange color, add LITELLM_BRANCH env var for branch installs
* fix(install.sh): install from git branch so --setup is available for QA
* fix(install.sh): remove stale LITELLM_BRANCH reference that caused unbound variable error
* fix(install.sh): force-reinstall from git to bypass cached PyPI version
* fix(install.sh): show pip progress bar during install
* fix(install.sh): always launch wizard via $PYTHON_BIN -m litellm, not PATH binary
* fix(install.sh): use litellm.proxy.proxy_cli module (no __main__.py exists)
* fix(install.sh): suppress RuntimeWarning from module invocation
* fix(install.sh): use Python bin-dir litellm binary to avoid CWD sys.path shadowing
* fix(install.sh): use sysconfig.get_path('scripts') to find pip-installed litellm binary
* fix(install.sh): redirect stdin from /dev/tty on exec so wizard gets terminal, not exhausted pipe
* fix(install.sh): warn about git clone duration, drop --no-cache-dir so re-runs are faster
* feat(setup_wizard): arrow-key selector, updated model names
* fix(setup_wizard): use sysconfig binary to start proxy, not python -m litellm
* feat(setup_wizard): credential validation after key entry + clear next-steps after proxy start
* style(install.sh): show git clone warning in blue
* refactor(setup_wizard): class with static methods, use check_valid_key from litellm.utils
* address greptile review: fix yaml escaping, port validation, display name collisions, tests
- setup_wizard.py: add _yaml_escape() for safe YAML embedding of API keys
- setup_wizard.py: add _styled_input() with readline ANSI ignore markers
- setup_wizard.py: change DIVIDER to _divider() fn to avoid import-time color capture
- setup_wizard.py: validate port range 1-65535, initialize before loop
- setup_wizard.py: qualify azure display names (azure-gpt-4o) to avoid collision with openai
- setup_wizard.py: work on env_copy in _build_config to avoid mutating caller's dict
- setup_wizard.py: skip model_list entries for providers with no credentials
- setup_wizard.py: prompt for azure deployment name
- setup_wizard.py: wrap os.execlp in try/except with friendly fallback
- setup_wizard.py: wrap config write in try/except OSError
- setup_wizard.py: fix _validate_and_report to use two print lines (no \r overwrite)
- setup_wizard.py: add .gitignore tip next to key storage notice
- setup_wizard.py: fix run_setup_wizard() return type annotation to None
- scripts/install.sh: drop pipefail (not supported by dash on Ubuntu when invoked as sh)
- scripts/install.sh: use litellm[proxy] from PyPI (not hardcoded dev branch)
- scripts/install.sh: guard /dev/tty read with -r check for Docker/CI compat
- scripts/install.sh: remove --force-reinstall to avoid downgrading dependencies
- tests/test_litellm/test_setup_wizard.py: 13 unit tests for _build_config and _yaml_escape
* style: black format setup_wizard.py
* fix: address remaining greptile issues - Windows compat, YAML quoting, credential flow
- guard termios/tty imports with try/except ImportError for Windows compat
- quote master_key as YAML double-quoted scalar (same as env vars)
- remove unused port param from _build_config signature
- _validate_and_report now returns the final key so re-entered creds are stored
- add test for master_key YAML quoting
* fix: add --port to suggested command, guard /dev/tty exec in install.sh
* fix: quote api_base in YAML, skip azure if no deployment, only redraw on state change
* fix: address greptile review comments
- _yaml_escape: add control character escaping (\n, \r, \t)
- test: fix tautological assertion in test_build_config_azure_no_deployment_skipped
- test: add tests for control character escaping in _yaml_escape
* feat(ui): remove Chat UI page link and banner from sidebar and playground (#23908)
* feat(guardrails): MCPJWTSigner - built-in guardrail for zero trust MCP auth (#23897)
* Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers
* Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present.
* Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings.
* Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers
* Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present.
* Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings.
* feat(guardrails): add MCPJWTSigner built-in guardrail for zero trust MCP auth
Signs outbound MCP tool calls with a LiteLLM-issued RS256 JWT so MCP servers
can trust a single signing authority instead of every upstream IdP.
Enable in config.yaml:
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
JWT carries sub (user_id), act.sub (team_id, RFC 8693), tool-level scope, iss,
aud, iat/exp/nbf. RSA-2048 keypair auto-generated at startup unless
MCP_JWT_SIGNING_KEY env var is set.
Adds /.well-known/jwks.json endpoint and jwks_uri to /.well-known/openid-configuration
so MCP servers can verify LiteLLM-issued tokens via OIDC discovery.
* Update MCPServerManager to raise HTTPException with status code 400 for extra headers in OpenAPI-backed servers. Adjust tests to verify the correct status code and exception message.
* fix: address P1 issues in MCPJWTSigner
- OpenAPI servers: warn + skip header injection instead of 500
- JWKS Cache-Control: 5min for auto-generated keys, 1h for persistent
- sub claim: fallback to apikey:{token_hash} for anonymous callers
- ttl_seconds: validate > 0 at init time
* docs: add MCP zero trust auth guide with architecture diagram
* docs: add FastMCP JWT verification guide to zero trust doc
* fix: address remaining Greptile review issues (round 2)
- mcp_server_manager: warn when hook Authorization overwrites existing header
- __init__: remove _mcp_jwt_signer_instance from __all__ (private internal)
- discoverable_endpoints: copy dict instead of mutating in-place on OIDC augmentation
- test docstring: reflect warn-and-continue behavior for OpenAPI servers
- test: update scope assertions for least-privilege (no mcp:tools/list on tool-call JWTs)
* fix: address Greptile round 3 feedback
- initialize_guardrail: validate mode='pre_mcp_call' at init time — misconfigured
mode silently bypasses JWT injection, which is a zero-trust bypass
- _build_claims: remove duplicate inline 'import re' (module-level import already present)
- _types.py: add TODO comment explaining jwt_claims is forward-compat plumbing
for a follow-up PR that will forward upstream IdP claims into outbound MCP JWTs
* feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes
Addresses all missing pieces from the scoping doc review:
FR-5 (Verify + re-sign): MCPJWTSigner now accepts access_token_discovery_uri
and token_introspection_endpoint. When set, the incoming Bearer token is
extracted from raw_headers (threaded through pre_call_tool_check), verified
against the IdP's JWKS (JWT) or introspected (opaque), and only re-signed if
valid. Falls back to user_api_key_dict.jwt_claims for LiteLLM JWT-auth mode.
FR-12 (Configurable end-user identity mapping): end_user_claim_sources
ordered list drives sub resolution — sources: token:<claim>, litellm:user_id,
litellm:email, litellm:end_user_id, litellm:team_id.
FR-13 (Claim operations): add_claims (insert-if-absent), set_claims (always
override), remove_claims (delete) applied in that order.
FR-14 (Two-token model): channel_token_audience + channel_token_ttl issue a
second JWT injected as x-mcp-channel-token: Bearer <token>.
FR-15 (Incoming claim validation): required_claims raises HTTP 403 when any
listed claim is absent; optional_claims passes listed claims from verified
token into the outbound JWT.
FR-9 (Debug headers): debug_headers: true emits x-litellm-mcp-debug with kid,
sub, iss, exp, scope.
FR-10 (Configurable scopes): allowed_scopes replaces auto-generation. Also
fixed: tool-call JWTs no longer grant mcp:tools/list (overpermission).
P1 fixes:
- proxy/utils.py: _convert_mcp_hook_response_to_kwargs merges rather than
replaces extra_headers, preserving headers from prior guardrails.
- mcp_server_manager.py: warns when hook injects Authorization alongside a
server-configured authentication_token (previously silent).
- mcp_server_manager.py: pre_call_tool_check now accepts raw_headers and
extracts incoming_bearer_token so FR-5 verification has the raw token.
- proxy/utils.py: remove stray inline import inspect inside loop (pre-existing
lint error, now cleaned up).
Tests: 43 passing (28 new tests covering all FR flags + P1 fixes).
* feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes (core)
Remaining files from the FR implementation:
mcp_jwt_signer.py — full rewrite with all new params:
FR-5: access_token_discovery_uri, token_introspection_endpoint,
verify_issuer, verify_audience + _verify_incoming_jwt(),
_introspect_opaque_token()
FR-12: end_user_claim_sources ordered resolution chain
FR-13: add_claims, set_claims, remove_claims
FR-14: channel_token_audience, channel_token_ttl → x-mcp-channel-token
FR-15: required_claims (raises 403), optional_claims (passthrough)
FR-9: debug_headers → x-litellm-mcp-debug
FR-10: allowed_scopes; tool-call JWTs no longer over-grant tools/list
mcp_server_manager.py:
- pre_call_tool_check gains raw_headers param to extract incoming_bearer_token
- Silent Authorization override warning fixed: now fires when server has
authentication_token AND hook injects Authorization
tests/test_mcp_jwt_signer.py:
28 new tests covering all FR flags + P1 fixes (43 total, all passing)
* fix(mcp_jwt_signer): address pre-landing review issues
- Remove stale TODO comment on UserAPIKeyAuth.jwt_claims — the field is
already populated and consumed by MCPJWTSigner in the same PR
- Fix _get_oidc_discovery to only cache the OIDC discovery doc when
jwks_uri is present; a malformed/empty doc now retries on the next
request instead of being permanently cached until proxy restart
- Add FR-5 test coverage for _fetch_jwks (cache hit/miss),
_get_oidc_discovery (cache/no-cache on bad doc), _verify_incoming_jwt
(valid token, expired token), _introspect_opaque_token (active,
inactive, no endpoint), and the end-to-end 401 hook path — 53 tests
total, all passing
* docs(mcp_zero_trust): rewrite as use-case guide covering all new JWT signer features
Add scenario-driven sections for each new config area:
- Verify+re-sign with Okta/Azure AD (access_token_discovery_uri,
end_user_claim_sources, token_introspection_endpoint)
- Enforcing caller attributes with required_claims / optional_claims
- Adding metadata via add_claims / set_claims / remove_claims
- Two-token model for AWS Bedrock AgentCore Gateway
(channel_token_audience / channel_token_ttl)
- Controlling scopes with allowed_scopes
- Debugging JWT rejections with debug_headers
Update JWT claims table to reflect configurable sub (end_user_claim_sources)
* fix(mcp_jwt_signer): wire all config.yaml params through initialize_guardrail
The factory was only passing issuer/audience/ttl_seconds to MCPJWTSigner.
All FR-5/9/10/12/13/14/15 params (access_token_discovery_uri,
end_user_claim_sources, add/set/remove_claims, channel_token_audience,
required/optional_claims, debug_headers, allowed_scopes, etc.) were
silently dropped, making every advertised advanced feature non-functional
when loaded from config.yaml.
Add regression test that asserts every param is wired through correctly.
* docs(mcp_zero_trust): add hero image
* docs(mcp_zero_trust): apply Linear-style edits
- Lead with the problem (unsigned direct calls bypass access controls)
- Shorter statement section headers instead of question-form headers
- Move diagram/OIDC discovery block after the reader is bought in
- Add 'read further only if you need to' callout after basic setup
- Two-token section now opens from the user problem not product jargon
- Add concrete 403 error response example in required_claims section
- Debug section opens from the symptom (MCP server returning 401)
- Lowercase claims reference header for consistency
* fix(mcp_jwt_signer): fix algorithm confusion attack + add OIDC discovery 24h TTL
- Remove alg from unverified JWT header; use signing_jwk.algorithm_name from JWKS key instead.
Reading alg from attacker-controlled headers enables alg:none / HS256 confusion attacks.
- Add _oidc_discovery_fetched_at timestamp and _OIDC_DISCOVERY_TTL = 86400 (24h).
Without a TTL the cached discovery doc never refreshes, so IdP key rotation is invisible.
---------
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
* fix(ci): stabilize CI - formatting, type errors, test polling, security CVEs, router bug, batch resolution
Fix 1: Run Black formatter on 35 files
Fix 2: Fix MyPy type errors:
- setup_wizard.py: add type annotation for 'selected' set variable
- user_api_key_auth.py: remove redundant type annotation on jwt_claims reassignment
Fix 3: Fix spend accuracy test burst 2 polling to wait for expected total
spend instead of just 'any increase' from burst 2
Fix 4: Bump Next.js 16.1.6 -> 16.1.7 to fix CVE-2026-27978, CVE-2026-27979,
CVE-2026-27980, CVE-2026-29057
Fix 5: Fix router _pre_call_checks model variable being overwritten inside
loop, causing wrong model lookups on subsequent deployments. Use local
_deployment_model variable instead.
Fix 6: Add missing resolve_output_file_ids_to_unified call in batch retrieve
non-terminal-to-terminal path (matching the terminal path behavior)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* chore: regenerate poetry.lock to sync with pyproject.toml
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: format merged files from main and regenerate poetry.lock
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mypy): annotate jwt_claims as Optional[dict] to fix type incompatibility
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): update router region test to use gpt-4.1-mini (fix flaky model lookup)
Replace deprecated gpt-3.5-turbo-1106 with gpt-4.1-mini + mock_response in
test_router_region_pre_call_check, following the same pattern used in commit
|
||
|
yuneng-jiang
|
8f56ddb9c6 |
Merge remote main into litellm_ci_optimize
Resolved conflict in test_claude_agent_sdk.py by keeping main's additions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
cc027a2b90 |
Fix flaky test_langsmith_queue_logging: poll instead of fixed sleep
The test waited a fixed 3s for async callbacks to populate log_queue. Under xdist -n 4, CPU contention can delay the GLOBAL_LOGGING_WORKER background task beyond 3s. Replace fixed sleeps with polling loops (up to 10s) that break as soon as the expected condition is met. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
9b77524354 |
Fix logging_testing: capture true defaults at conftest import time
Module-level mutations (litellm.num_retries=3 in test_langfuse_e2e_test.py and test_amazing_s3_logs.py, litellm.success_callback=['langfuse']) run at import time, BEFORE any function fixture. The save/restore pattern captured these polluted values as 'originals' and kept restoring them. Fix: capture litellm defaults when conftest.py is first imported (before test modules), then reset to those true defaults before each test instead of saving/restoring the current (potentially polluted) state. |
||
|
yuneng-jiang
|
13a46598e7 |
Fix logging_testing: clear _in_memory_loggers and add missing globals
- Clear _in_memory_loggers before/after each test to prevent cached logger instances (LangsmithLogger, SlackAlerting, etc.) from leaking stale state - Add pre_call_rules, post_call_rules to list attrs save/restore - Add vector_store_registry to scalar attrs save/restore Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
92ad90de2a |
Fix logging_testing: expand save/restore to cover redaction and other globals
The logging tests mutate many more litellm globals than guardrails tests (turn_off_message_logging, s3_callback_params, datadog_params, service_callback, etc.). The initial save/restore list only covered callbacks and a few basics, causing state leaks like redaction settings bleeding across tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
19e8a16cce |
Optimize logging_testing CI: suppress DEBUG logs, fix xdist isolation
- Add LITELLM_LOG=WARNING to suppress verbose DEBUG log output - Remove -s flag to stop capturing all stdout - Bump xdist workers from -n 2 to -n 4 - Add --timeout=120 for safety - Rewrite conftest.py to use save/restore pattern (matching guardrails_tests) instead of per-function importlib.reload + event loop creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Harshit28j
|
d7c9ec6276 | add tests for fix | ||
|
yuneng-jiang
|
89d8401d72
|
Merge pull request #23483 from BerriAI/litellm_update_deprecated_test_models
[Fix] Update Deprecated Model Names in CI Tests |
||
|
yuneng-jiang
|
cc81e3c226 |
Replace deprecated model names in tests that were removed from remote model cost map
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Cesar Garcia
|
e01d722803
|
Merge branch 'main' into litellm_oss_staging_03_11_2026 | ||
|
Chesars
|
feed274aa3 |
Reapply "feat: add model_cost aliases expansion support"
This reverts commit
|
||
|
Chesars
|
1be6b31e2f | merge: resolve conflicts between main and litellm_oss_staging_03_11_2026 | ||
|
Sameer Kankute
|
36ec80d90c | Fix azure model router | ||
|
Cesar Garcia
|
3d2df7e8b5
|
Revert "feat: add model_cost aliases expansion support" | ||
|
Sameer Kankute
|
b08445837b |
fix(logging): preserve ModelResponse choices format in redacted standard_logging_object + add Charity Engine provider endpoint
- Fix perform_redaction to handle dict representation of ModelResponse (from model_dump()) - Preserve full choices structure when redacting, redact content/audio in place - Add _redact_standard_logging_object helper for standard_logging_object field - Update test_logging_redaction_e2e_test assertions to expect choices format - Add charity_engine to provider_endpoints_support.json Fixes: test_standard_logging_payload, test_standard_logging_payload_audio Made-with: Cursor |
||
|
Ishaan Jaff
|
28c33f53a3
|
CircleCI test stability (#23055)
* fix: resolve ruff lint errors and mypy type error
- Remove unused import get_user_credential (F401)
- Add noqa: PLR0915 for 3 large functions exceeding 50 statements
- Cast result_data['q'] to str for _append_domain_filters (mypy arg-type)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add /vertex_ai/live to supported endpoints and azure gpt-5.1 reasoning flags
- Add /vertex_ai/live to JSON schema validation enum in test_utils.py
- Add supports_none_reasoning_effort=true to 10 azure/gpt-5.1 model entries
(matching the OpenAI gpt-5.1 behavior)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: handle non-string team_alias/key_alias in PolicyMatchContext
Prevent Pydantic validation errors when team_alias or key_alias are not
proper strings (e.g. MagicMock in tests). Only pass values that are
actually strings; default to None otherwise.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: initialize jwt_handler.litellm_jwtauth in JWT test
The test_jwt_non_admin_team_route_access test was failing because
user_api_key_auth now accesses jwt_handler.litellm_jwtauth.virtual_key_claim_field
before reaching the mocked JWTAuthManager.auth_builder. Initialize the
jwt_handler with a default LiteLLM_JWTAuth object.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add missing mock attributes to MCP server test
The test_add_update_server_fallback_to_server_id test was failing because
MagicMock auto-creates attributes when accessed. build_mcp_server_from_table
accesses many fields via getattr(), which on a MagicMock returns another
MagicMock instead of None, causing Pydantic validation errors in MCPServer.
Explicitly set all required mock attributes.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: update UI tests for leftnav, navbar, and KeyLifecycleSettings
- leftnav: Add mock for useTeams hook, add isUserTeamAdminForAnyTeam to
roles mock, update topLevelLabels to match current component menu items
- navbar: Add mocks for useDisableBouncingIcon, BlogDropdown, UserDropdown,
and serverRootPath. Update test to work with the new component structure.
- KeyLifecycleSettings: Fix placeholder and tooltip assertions to match
actual component behavior
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: update health check test assertion from 'connected' to 'healthy'
The /health/readiness endpoint now returns {"status": "healthy"} with the
DB status in a separate field, instead of the previous {"status": "connected"}.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: clear litellm.api_key in OpenRouter validate_environment test
The test_validate_environment_raises_without_key test was failing because
litellm.api_key may be set globally in the test environment. Clear it
along with OPENROUTER_API_KEY and OR_API_KEY env vars using monkeypatch.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: patch HTTPHandler class-level in VLLM embedding test
The test_encoding_format_not_sent_in_actual_request test was patching
client.post on an instance, but the handler uses the class method.
Patch HTTPHandler.post at class level, add caching=False to prevent
cache hits, and remove broad try/except that hid errors.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: make test_redaction_responses_api_stream resilient to async callback timing
Replace fixed 1s sleep with polling wait for async_log_success_event.
Streaming success handler runs via asyncio.create_task; 1s was insufficient
in CI. Add 0.5s initial sleep for event loop to schedule the task, then
poll up to 10s for the callback to fire.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: update dompurify and svgo to fix security CVEs
- CVE-2026-0540: dompurify XSS vulnerability - fix by upgrading to 3.3.2+
- CVE-2026-29074: svgo DoS via entity expansion - fix by upgrading to 3.3.3+
Added npm overrides in docs/my-website/package.json and regenerated
package-lock.json.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: remove unused json import in config_override_endpoints.py
Ruff F401: json is imported but unused (safe_json_loads/safe_dumps
are used instead)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add missing MCP mock attributes and provider documentation entries
- Add missing mock attributes to test_add_update_server_with_alias and
test_add_update_server_without_alias (same fix as fallback test)
- Add bedrock_mantle and searchapi to provider_endpoints_support.json
- Remove unused json import from config_override_endpoints.py
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: override _supports_reasoning_effort_level for Azure gpt5_series prefix
The Azure GPT-5 config uses 'gpt5_series/' as a routing prefix, but
_supports_factory(model='gpt5_series/gpt-5.1') fails to resolve because
'gpt5_series' is not a recognized provider. Override the method to strip
the prefix and prepend 'azure/' for correct model info lookup.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: accept both 'healthy' and 'connected' in health check test
The test_health_and_chat_completion test runs against both source builds
(which return 'healthy') and pip-installed versions (which may return
'connected'). Accept both values.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: mock extract_mcp_auth_context in streamable HTTP MCP handler test
The handle_streamable_http_mcp function now calls extract_mcp_auth_context
before session_manager.handle_request, but the test didn't mock it. The
auth extraction fails with the minimal mock scope, preventing
handle_request from being called. Also relax assertion to not check
exact args since the send wrapper may be modified by debug injection.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add test for _combine_fallback_usage to satisfy router code coverage
The router_code_coverage.py check requires all functions in router.py
to be called in test files. Add a basic test for _combine_fallback_usage.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add @log_guardrail_information decorator to CrowdStrike AIDR guardrail
The check_guardrail_apply_decorator.py CI check requires all guardrail
apply_guardrail methods to have the @log_guardrail_information decorator.
The CrowdStrike AIDR handler was missing it.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: document PRISMA_RECONNECT_ESCALATION_THRESHOLD and REDIS_CLUSTER_NODES env keys
Add missing environment variable documentation to config_settings.md
to satisfy the test_env_keys.py CI check.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: document enforced_file_expires_after and enforced_batch_output_expires_after in new_team docstring
The test_api_docs.py CI check validates that all Pydantic model fields
are documented in the function docstring. Add missing parameter docs
for enforced_file_expires_after and enforced_batch_output_expires_after.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: regenerate poetry.lock to match pyproject.toml
The poetry.lock file was out of sync with pyproject.toml, causing
proxy_e2e_azure_batches_tests to fail during dependency installation.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: set master_key=None in test_create_file_with_deep_nested_litellm_metadata
The test was missing the master_key monkeypatch that other tests in the
same file set. In CI with parallel execution (-n 4), another test may
set master_key to a non-None value, causing auth failures (500) when
the test sends 'Bearer test-key'.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: document enforced_*_expires_after in update_team docstring too
Same missing params as new_team - also needed in update_team docstring
for the test_api_docs.py CI check to pass.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: use get_async_httpx_client in a2a_protocol and add master_key monkeypatch to files tests
- Replace httpx.AsyncClient() with get_async_httpx_client() in a2a_protocol/main.py
to satisfy the ensure_async_clients_test CI check
- Add httpxSpecialProvider.A2AProvider enum value
- Add master_key=None monkeypatch to test_managed_files_with_loadbalancing
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: remove unused httpx import from a2a_protocol/main.py
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: use cache-key-only param for A2A extra_headers to avoid AsyncHTTPHandler init error
The 'extra_headers' key in params was being passed to AsyncHTTPHandler.__init__()
which doesn't accept it. Use 'disable_aiohttp_transport' as the cache-key-only
param since it's explicitly filtered out before reaching the constructor.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: add additionalProperties:false and resolve $defs/$ref in Anthropic output_format schemas
Anthropic API now requires additionalProperties=false for all object-type
schemas in output_format. Also resolve $defs/$ref references by inlining
them using unpack_defs before sending to Anthropic, since Anthropic
doesn't support external schema references.
Fixes: llm_translation_testing Anthropic JSON schema failures
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: allowlist CVE-2026-2297 and GHSA-qffp-2rhf-9h96 in security scans
- CVE-2026-2297: Python 3.13 SourcelessFileLoader audit hook bypass,
no fix available in base image
- GHSA-qffp-2rhf-9h96: tar hardlink path traversal, from nodejs_wheel
bundled npm, not used in application runtime code
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: isolate files endpoint tests from shared proxy state in CI parallel execution
Override user_api_key_auth dependency to return a fixed UserAPIKeyAuth
with PROXY_ADMIN role, avoiding auth lookups via prisma_client,
user_api_key_cache, or master_key. Set prisma_client=None to prevent
DB state contamination. Use try/finally to clean up dependency overrides.
Fixes persistent test_create_file_with_deep_nested_litellm_metadata and
test_managed_files_with_loadbalancing 500 errors in CI with -n 4.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: apply same auth override to test_managed_files_with_loadbalancing
Same CI parallel execution fix as test_create_file_with_deep_nested -
override user_api_key_auth dependency and set prisma_client=None.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
|
||
|
Sameer Kankute
|
16d6c279da
|
Merge pull request #22180 from BerriAI/litellm_fix_vllm_test
Add JSON exact match test for vLLM embeddings |
||
|
Sameer Kankute
|
d9cd3ea185
|
Merge pull request #22181 from mubashir1osmani/fix/arize-phoenix-nested-traces-test-update
fix(test): update Phoenix OTEL test |
||
|
mubashir1osmani
|
3c595f6fd2 |
fix(test): update Phoenix OTEL test for dedicated TracerProvider architecture
The old test assumed ArizePhoenixLogger reused the global TracerProvider. With the nested traces fix, Phoenix now creates its own dedicated provider and produces litellm_proxy_request + litellm_request + raw_gen_ai_request spans independently. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
Sameer Kankute
|
1a3d4c80db | Add JSON exact match test for vLLM embeddings | ||
|
Sameer Kankute
|
828ce40eac | Fix test_async_gcs_pub_sub_v1 | ||
|
Ryan Crabbe
|
75bc8329e2 |
Merge origin/main into litellm_perf_skip_usage_roundtrip
Resolve conflict in litellm_logging.py: take main's version and re-apply get_usage_as_dict optimization on top. |
||
|
Ishaan Jaff
|
a5e886de79
|
fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var instead of hardcoding model (#21781)
* fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var in bedrock KB tests * fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var in test_router * fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var in test_router_retries * fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var in test_router_timeout |
||
|
Ishaan Jaff
|
0726bdb67c
|
fix(tests): update gcs pubsub v1 fixture with new SpendLogsMetadata fields (#21779)
SpendLogsMetadata added new fields (user_api_key, status, error_information, etc.) that weren't in the expected spend_logs_payload.json fixture, causing test_async_gcs_pub_sub_v1 to fail. |
||
|
Emerson Gomes
|
cba3bcf1a9 | fix(logging): avoid shared callback list references (#20984) | ||
|
Sameer Kankute
|
59d6ab8a00
|
Merge branch 'main' into litellm_oss_staging_02_11_2026 |