diff --git a/docs/my-website/blog/april_townhall_announcement/index.md b/docs/my-website/blog/april_townhall_announcement/index.md index 466d9e845f0..1f842536f89 100644 --- a/docs/my-website/blog/april_townhall_announcement/index.md +++ b/docs/my-website/blog/april_townhall_announcement/index.md @@ -4,6 +4,7 @@ title: "April Townhall: Security + Product Roadmap" date: 2026-04-02T07:30:00 authors: - krrish + - ishaan-alt description: "Join the LiteLLM April townhall on Friday, 10 April at 7:30 AM to learn about LiteLLM's security and product roadmap." tags: [announcement, townhall] hide_table_of_contents: true diff --git a/docs/my-website/blog/april_townhall_updates/index.md b/docs/my-website/blog/april_townhall_updates/index.md new file mode 100644 index 00000000000..c726d1b7f8e --- /dev/null +++ b/docs/my-website/blog/april_townhall_updates/index.md @@ -0,0 +1,162 @@ +--- +slug: april-townhall-updates +title: "April Townhall Updates: CI/CD v2, Stability, and Product Roadmap" +date: 2026-04-10T12:00:00 +authors: + - krrish + - ishaan-alt +description: "A recap of the April LiteLLM town hall covering CI/CD v2, product stability work, and the near-term roadmap." +tags: [townhall, security, reliability, product] +hide_table_of_contents: false +--- + +import Image from '@theme/IdealImage'; + +Thank you to everyone who joined our April town hall. + +We used the session to share our CI/CD v2 improvements, product stability work, and what we are prioritizing next across reliability and product roadmap. + +{/* truncate */} + +## CI/CD v2 improvements + +Our CI/CD v2 work is centered around four goals: + +1. **Limit** what each package can access +2. **Reduce** the number of sensitive environment variables +3. **Avoid** compromised packages +4. **Reduce the risk of** release tampering + +#### New architecture: isolated environments + +We have begun moving to isolated environments for distinct CI/CD stages to reduce the chance that a single compromised step can inherit broad access across the entire pipeline. + + + +#### Current rollout status + +These changes are deployed in our current release workflow. [See here](https://github.com/BerriAI/litellm/tags) + +#### Independently verify releases + +A key part of CI/CD v2 is supporting independent verification of release artifacts using our published verification process, while reducing reliance on any single credential or release path. + +[**Learn more about how to verify releases**](https://docs.litellm.ai/docs/proxy/docker_image_security) + + + +## Stability improvements + +### SDLC improvements + +This month, we're focusing on process stability improvements around: +- Improving main-branch stability +- Mapping UI QA to built Docker images for 1:1 environment parity +- Consistent release tags across PyPI and Docker +- Fixing release notes publication + +#### Improving main-branch stability + +We're introducing a staging-gated flow: + + + +- Only an internal staging branch can push to `main`. +- PRs to that staging branch must pass CircleCI LLM API testing. +- Collision handling happens on staging, which is designed to reduce unstable changes reaching `main`. + +#### UI QA in Docker environment + +Moving forward, all UI QA will be performed in the built Docker image that users run. + +Previously, some UI QA paths were run in local environments that did not fully replicate Docker runtime conditions. + +That contributed to release-specific issues, including MCP registration problems in `v1.82.3`. + +#### Consistent release tags + +Today we publish releases for multiple scenarios: +- Dev (Built of a PR for a customer-specific scenario) +- Nightly (Passes all CI/CD checks) +- Release Candidate (Passes all CI/CD checks + manual UI QA) +- Stable (intended to pass all CI/CD checks + manual UI QA + 7 days of production testing) + +We are targeting a consistent naming convention across PyPI and Docker by the end of April. + +#### Release notes + +CI/CD v2 changes moved release notes to a manual path. This is a temporary solution while we investigate a better automated workflow. We are targeting a more consistent process by the end of April. + +### Product stability improvements + +#### Stable Prisma migrations + +Today, we have observed several migration failure classes: +- Migration not applied +- Migration marked applied but incomplete +- Migration not applied due to non-root image issues + +We're prioritizing this work this month and have assigned an engineering owner to the effort. Our target is to resolve these error classes by the end of April. + +#### UI type safety + +Another area of focus is improving the stability of the UI. Today, one cause of errors is that the UI maintains its own assumptions about backend API types. This can lead to issues when backend responses differ from UI assumptions. + +We aim to move to having the UI and Backend be in sync with each other, and are exploring OpenAPI-driven mapping to achieve this. + +## Product roadmap + +### Our Assumptions + +Over the next few years, we expect: +- Companies will give employees more AI tools. +- More AI agents will move into production workflows across HR, finance, support, and operations. + +### Our Inferences +#### Near-term + +- AI spend will increase. +- Uptime and latency will become even more important. +- More AI resources (skills, CLIs, and related assets) will require governance. +- Agent and MCP usage patterns will require deeper controls. +- Broader developer adoption will increase the need for simpler, more discoverable tooling. + +#### Long-term + +- We expect many organizations to treat agent auditability (how decisions were made across LLM + MCP + sub-agent inputs/outputs) as a compliance expectation. +- Permission management will get more complex as user-agent interaction chains deepen. + +Roadmap timelines in this post are targets and may evolve based on validation and user feedback. + +## April investments + +### Reliability + +- Increase uptime for 10k+ RPS scenarios. +- Investigate latency overhead for long-running Claude Code requests. + +### Feature reliability + +- Polish MCP authentication. +- Better understand how teams are using agents through LiteLLM. + +### Governance + +- Launch Skills as a first-class citizen in LiteLLM. + +## Q&A + +Thank you again for all the questions and direct feedback. We will keep sharing concrete progress updates as these efforts ship. + +## Hiring + +We are actively hiring across several roles, please apply [here](https://jobs.ashbyhq.com/litellm) if you're interested! \ No newline at end of file diff --git a/docs/my-website/blog/authors.yml b/docs/my-website/blog/authors.yml index 1b1ef4d34c4..c8a1bab7ed3 100644 --- a/docs/my-website/blog/authors.yml +++ b/docs/my-website/blog/authors.yml @@ -24,7 +24,7 @@ ishaan: # Alias for typo in name ishaan-alt: - name: Ishaan Jaff + name: Ishaan Jaffer title: CTO, LiteLLM url: https://www.linkedin.com/in/reffajnaahsi/ image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg diff --git a/docs/my-website/img/april_townhall_isolated_environments.png b/docs/my-website/img/april_townhall_isolated_environments.png new file mode 100644 index 00000000000..120e5cec9b7 Binary files /dev/null and b/docs/my-website/img/april_townhall_isolated_environments.png differ diff --git a/docs/my-website/img/stable_main.png b/docs/my-website/img/stable_main.png new file mode 100644 index 00000000000..f050b54f6e0 Binary files /dev/null and b/docs/my-website/img/stable_main.png differ diff --git a/docs/my-website/img/verify_releases.png b/docs/my-website/img/verify_releases.png new file mode 100644 index 00000000000..270a999d8dc Binary files /dev/null and b/docs/my-website/img/verify_releases.png differ diff --git a/litellm/integrations/dotprompt/prompt_manager.py b/litellm/integrations/dotprompt/prompt_manager.py index 997a40d545e..6407a18d0b3 100644 --- a/litellm/integrations/dotprompt/prompt_manager.py +++ b/litellm/integrations/dotprompt/prompt_manager.py @@ -7,7 +7,8 @@ from pathlib import Path from typing import Any, Dict, List, Optional, Tuple, Union import yaml -from jinja2 import DictLoader, Environment, select_autoescape +from jinja2 import DictLoader, select_autoescape +from jinja2.sandbox import ImmutableSandboxedEnvironment class PromptTemplate: @@ -59,7 +60,10 @@ class PromptManager: self.prompt_directory = Path(prompt_directory) if prompt_directory else None self.prompts: Dict[str, PromptTemplate] = {} self.prompt_file = prompt_file - self.jinja_env = Environment( + # Sandboxed env: templates can come from user input via /prompts/test, + # so we must block access to unsafe Python attributes and mutation of + # caller-supplied mutables. + self.jinja_env = ImmutableSandboxedEnvironment( loader=DictLoader({}), autoescape=select_autoescape(["html", "xml"]), # Use Handlebars-style delimiters to match Dotprompt spec diff --git a/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py b/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py index c1eccaebd04..3ac579dae1b 100644 --- a/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py +++ b/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py @@ -538,9 +538,9 @@ class AmazonAnthropicClaudeMessagesConfig( merges usage from message_start and message_delta but ignores message_stop. This method buffers message_delta and, when message_stop arrives with cache usage, merges those fields into the - message_delta usage and also updates the input_tokens on - message_delta to include the full count (uncached + cache_creation + - cache_read). + message_delta usage. input_tokens is kept as the uncached-only + count; downstream calculate_usage adds cache tokens to + prompt_tokens. """ _CACHE_FIELDS = ("cache_creation_input_tokens", "cache_read_input_tokens") pending_delta = None @@ -569,12 +569,7 @@ class AmazonAnthropicClaudeMessagesConfig( raw_input = stop_usage.get("input_tokens") if raw_input is not None: - uncached = raw_input if isinstance(raw_input, int) else 0 - raw_cc = delta_usage.get("cache_creation_input_tokens", 0) - cache_creation = raw_cc if isinstance(raw_cc, int) else 0 - raw_cr = delta_usage.get("cache_read_input_tokens", 0) - cache_read = raw_cr if isinstance(raw_cr, int) else 0 - delta_usage["input_tokens"] = uncached + cache_creation + cache_read + delta_usage["input_tokens"] = raw_input if isinstance(raw_input, int) else 0 if delta_usage: pending_delta["usage"] = delta_usage # type: ignore[arg-type] diff --git a/litellm/llms/litellm_proxy/skills/prompt_injection.py b/litellm/llms/litellm_proxy/skills/prompt_injection.py index 2b86f74122b..86b6e223512 100644 --- a/litellm/llms/litellm_proxy/skills/prompt_injection.py +++ b/litellm/llms/litellm_proxy/skills/prompt_injection.py @@ -5,6 +5,7 @@ Handles extraction of skill content (SKILL.md) from stored ZIP files and injection into the system prompt for non-Anthropic models. """ +import posixpath import zipfile from io import BytesIO from typing import Any, Dict, List, Optional @@ -103,8 +104,18 @@ class SkillPromptInjectionHandler: else: clean_path = name - if clean_path: - files[clean_path] = zf.read(name) + if not clean_path: + continue + + # Ensure the path stays within the intended directory + normalized = posixpath.normpath(clean_path) + if normalized.startswith("..") or posixpath.isabs(normalized): + verbose_logger.warning( + f"SkillPromptInjectionHandler: Skipping entry with invalid path in skill {skill.skill_id}: {name}" + ) + continue + + files[normalized] = zf.read(name) except Exception as e: verbose_logger.warning( f"SkillPromptInjectionHandler: Error extracting files from skill {skill.skill_id}: {e}" diff --git a/litellm/llms/litellm_proxy/skills/sandbox_executor.py b/litellm/llms/litellm_proxy/skills/sandbox_executor.py index a5c0a539c96..af13d3c091d 100644 --- a/litellm/llms/litellm_proxy/skills/sandbox_executor.py +++ b/litellm/llms/litellm_proxy/skills/sandbox_executor.py @@ -94,9 +94,15 @@ class SkillsSandboxExecutor: # Create a temp directory to stage files with tempfile.TemporaryDirectory() as tmpdir: + tmpdir_abs = os.path.abspath(tmpdir) for path, content in skill_files.items(): # Create the file in temp directory - local_path = os.path.join(tmpdir, path) + local_path = os.path.abspath(os.path.join(tmpdir, path)) + if not local_path.startswith(tmpdir_abs + os.sep): + verbose_logger.warning( + f"SkillsSandboxExecutor: Skipping file with invalid path: {path}" + ) + continue os.makedirs(os.path.dirname(local_path), exist_ok=True) with open(local_path, "wb") as f: f.write(content) diff --git a/litellm/proxy/agent_endpoints/endpoints.py b/litellm/proxy/agent_endpoints/endpoints.py index 6e5d4562b55..64c20d5ed5e 100644 --- a/litellm/proxy/agent_endpoints/endpoints.py +++ b/litellm/proxy/agent_endpoints/endpoints.py @@ -28,6 +28,7 @@ from litellm.types.agents import ( MakeAgentsPublicRequest, PatchAgentRequest, ) +from litellm.litellm_core_utils.litellm_logging import _get_masked_values from litellm.types.llms.custom_http import httpxSpecialProvider from litellm.types.proxy.management_endpoints.common_daily_activity import ( SpendAnalyticsPaginatedResponse, @@ -36,6 +37,28 @@ from litellm.types.proxy.management_endpoints.common_daily_activity import ( router = APIRouter() +def _redact_sensitive_agent_fields( + agents: List[AgentResponse], +) -> List[AgentResponse]: + """ + Return copies of the given agents with sensitive configuration fields + redacted. The original objects are not modified. + """ + redacted: List[AgentResponse] = [] + for agent in agents: + copy = agent.model_copy(deep=True) + copy.static_headers = None + copy.extra_headers = None + if copy.litellm_params: + copy.litellm_params = _get_masked_values( + copy.litellm_params, + unmasked_length=4, + number_of_asterisks=4, + ) + redacted.append(copy) + return redacted + + def _check_agent_management_permission(user_api_key_dict: UserAPIKeyAuth) -> None: """ Raises HTTP 403 if the caller does not have permission to create, update, @@ -183,6 +206,14 @@ async def get_agents( agent.agent_id in litellm.public_agent_groups ) + # Redact sensitive fields for non-admin users + is_admin = ( + user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN + or user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN.value + ) + if not is_admin: + returned_agents = _redact_sensitive_agent_fields(returned_agents) + if health_check: agents_with_url = [ agent @@ -399,6 +430,14 @@ async def get_agent_by_id( status_code=404, detail=f"Agent with ID {agent_id} not found" ) + # Redact sensitive fields for non-admin users + is_admin = ( + user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN + or user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN.value + ) + if not is_admin: + agent = _redact_sensitive_agent_fields([agent])[0] + return agent except HTTPException: raise diff --git a/litellm/proxy/guardrails/guardrail_endpoints.py b/litellm/proxy/guardrails/guardrail_endpoints.py index 422bdc13780..6814729258f 100644 --- a/litellm/proxy/guardrails/guardrail_endpoints.py +++ b/litellm/proxy/guardrails/guardrail_endpoints.py @@ -60,12 +60,20 @@ def _get_guardrails_list_response( """ Helper function to get the guardrails list response """ + from litellm.litellm_core_utils.litellm_logging import _get_masked_values + guardrail_configs: List[GuardrailInfoResponse] = [] for guardrail in guardrails_config: + litellm_params = guardrail.get("litellm_params") or {} + masked_params = _get_masked_values( + litellm_params, + unmasked_length=4, + number_of_asterisks=4, + ) guardrail_configs.append( GuardrailInfoResponse( guardrail_name=guardrail.get("guardrail_name"), - litellm_params=guardrail.get("litellm_params"), + litellm_params=masked_params, guardrail_info=guardrail.get("guardrail_info"), ) ) diff --git a/litellm/proxy/management_endpoints/key_management_endpoints.py b/litellm/proxy/management_endpoints/key_management_endpoints.py index 323ff7fd531..6e8a691ce93 100644 --- a/litellm/proxy/management_endpoints/key_management_endpoints.py +++ b/litellm/proxy/management_endpoints/key_management_endpoints.py @@ -456,6 +456,34 @@ def handle_key_type(data: GenerateKeyRequest, data_json: dict) -> dict: return data_json +def _check_allowed_routes_caller_permission( + allowed_routes: Optional[list], + user_api_key_dict: UserAPIKeyAuth, +) -> None: + """ + Only proxy admins may set `allowed_routes` on a key. + + `allowed_routes` bypasses the standard role-based route gate in + RouteChecks.non_proxy_admin_allowed_routes_check, so if a non-admin is + allowed to set it they can grant themselves access to any endpoint. + Non-admins should use `key_type` to pick a preset route bucket instead. + """ + # Empty list is the default on GenerateKeyRequest — treat as "not set". + if not allowed_routes: + return + if user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN.value: + return + raise HTTPException( + status_code=403, + detail={ + "error": ( + "Only proxy admins can set `allowed_routes` on a key. " + "Use `key_type` to pick a preset route bucket instead." + ) + }, + ) + + async def validate_team_id_used_in_service_account_request( team_id: Optional[str], prisma_client: Optional[PrismaClient], @@ -740,9 +768,9 @@ async def _common_key_generation_helper( # noqa: PLR0915 request_type="key", **data_json, table_name="key" ) - response[ - "soft_budget" - ] = data.soft_budget # include the user-input soft budget in the response + response["soft_budget"] = ( + data.soft_budget + ) # include the user-input soft budget in the response response = GenerateKeyResponse(**response) @@ -1254,6 +1282,12 @@ async def generate_key_fn( raise HTTPException( status_code=status.HTTP_403_FORBIDDEN, detail=message ) + + _check_allowed_routes_caller_permission( + allowed_routes=data.allowed_routes, + user_api_key_dict=user_api_key_dict, + ) + # For non-admin internal users: auto-assign caller's user_id if not provided # This prevents creating unbound keys with no user association (LIT-1884) _is_proxy_admin = ( @@ -1888,6 +1922,11 @@ async def _validate_update_key_data( """Validate permissions and constraints for key update.""" _is_proxy_admin = user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN.value + _check_allowed_routes_caller_permission( + allowed_routes=data.allowed_routes, + user_api_key_dict=user_api_key_dict, + ) + # Prevent non-admin from removing user_id (setting to empty string) (LIT-1884) if data.user_id is not None and data.user_id == "" and not _is_proxy_admin: raise HTTPException( @@ -3233,10 +3272,10 @@ async def delete_verification_tokens( try: if prisma_client: tokens = [_hash_token_if_needed(token=key) for key in tokens] - _keys_being_deleted: List[ - LiteLLM_VerificationToken - ] = await prisma_client.db.litellm_verificationtoken.find_many( - where={"token": {"in": tokens}} + _keys_being_deleted: List[LiteLLM_VerificationToken] = ( + await prisma_client.db.litellm_verificationtoken.find_many( + where={"token": {"in": tokens}} + ) ) if len(_keys_being_deleted) == 0: @@ -3436,9 +3475,9 @@ async def _rotate_master_key( # noqa: PLR0915 from litellm.proxy.proxy_server import proxy_config try: - models: Optional[ - List - ] = await prisma_client.db.litellm_proxymodeltable.find_many() + models: Optional[List] = ( + await prisma_client.db.litellm_proxymodeltable.find_many() + ) except Exception: models = None # 2. process model table @@ -4078,11 +4117,11 @@ async def validate_key_list_check( param="user_id", code=status.HTTP_403_FORBIDDEN, ) - complete_user_info_db_obj: Optional[ - BaseModel - ] = await prisma_client.db.litellm_usertable.find_unique( - where={"user_id": user_api_key_dict.user_id}, - include={"organization_memberships": True}, + complete_user_info_db_obj: Optional[BaseModel] = ( + await prisma_client.db.litellm_usertable.find_unique( + where={"user_id": user_api_key_dict.user_id}, + include={"organization_memberships": True}, + ) ) if complete_user_info_db_obj is None: @@ -4165,10 +4204,10 @@ async def _fetch_user_team_objects( if complete_user_info is None or not complete_user_info.teams: return [] - teams: Optional[ - List[BaseModel] - ] = await prisma_client.db.litellm_teamtable.find_many( - where={"team_id": {"in": complete_user_info.teams}} + teams: Optional[List[BaseModel]] = ( + await prisma_client.db.litellm_teamtable.find_many( + where={"team_id": {"in": complete_user_info.teams}} + ) ) if teams is None: return [] diff --git a/litellm/proxy/ui_crud_endpoints/proxy_setting_endpoints.py b/litellm/proxy/ui_crud_endpoints/proxy_setting_endpoints.py index 60bf41709ef..0349f289b4e 100644 --- a/litellm/proxy/ui_crud_endpoints/proxy_setting_endpoints.py +++ b/litellm/proxy/ui_crud_endpoints/proxy_setting_endpoints.py @@ -1,6 +1,7 @@ #### CRUD ENDPOINTS for UI Settings ##### import json from typing import Any, Dict, List, Optional, Union +from urllib.parse import urlparse from fastapi import APIRouter, Depends, File, HTTPException, UploadFile @@ -817,6 +818,29 @@ async def get_ui_theme_settings(): ) +def _validate_public_image_url(value: Optional[str], field_name: str) -> None: + """ + Reject anything that isn't a plain http(s) URL with a host. This value is + later served via the unauthenticated /get_image endpoint, so local paths + like "/etc/passwd" or "file://..." must not be accepted. + """ + if value is None: + return + if not isinstance(value, str) or not value.strip(): + return + parsed = urlparse(value.strip()) + if parsed.scheme not in ("http", "https") or not parsed.netloc: + raise HTTPException( + status_code=400, + detail={ + "error": ( + f"Invalid {field_name}: must be an http(s) URL with a host. " + "Local filesystem paths and non-http schemes are not allowed." + ) + }, + ) + + @router.patch( "/update/ui_theme_settings", tags=["UI Theme Settings"], @@ -831,6 +855,9 @@ async def update_ui_theme_settings(theme_config: UIThemeConfig): from litellm.proxy.proxy_server import proxy_config, store_model_in_db + _validate_public_image_url(theme_config.logo_url, "logo_url") + _validate_public_image_url(theme_config.favicon_url, "favicon_url") + if store_model_in_db is not True: raise HTTPException( status_code=500, diff --git a/litellm/proxy/utils.py b/litellm/proxy/utils.py index 635204f3362..88a2e1e95cd 100644 --- a/litellm/proxy/utils.py +++ b/litellm/proxy/utils.py @@ -2645,7 +2645,7 @@ class PrismaClient: raise e async def _query_first_with_cached_plan_fallback( - self, sql_query: str + self, sql_query: str, *args ) -> Optional[dict]: """ Execute a query with automatic fallback for PostgreSQL cached plan errors. @@ -2664,7 +2664,7 @@ class PrismaClient: Original exception if not a cached plan error """ try: - return await self.db.query_first(query=sql_query) + return await self.db.query_first(sql_query, *args) except Exception as e: error_str = str(e) if "cached plan must not change result type" in error_str: @@ -2679,7 +2679,7 @@ class PrismaClient: "retrying with fresh plan. This may occur during rolling deployments " "when schema changes are applied." ) - return await self.db.query_first(query=sql_query_retry) + return await self.db.query_first(sql_query_retry, *args) else: raise @@ -2978,7 +2978,7 @@ class PrismaClient: detail={"error": f"No token passed in. Token={token}"}, ) - sql_query = f""" + sql_query = """ SELECT v.*, t.spend AS team_spend, @@ -3016,11 +3016,11 @@ class PrismaClient: LEFT JOIN "LiteLLM_ProjectTable" AS p ON v.project_id = p.project_id LEFT JOIN "LiteLLM_OrganizationTable" AS o ON v.organization_id = o.organization_id LEFT JOIN "LiteLLM_BudgetTable" AS b2 ON o.budget_id = b2.budget_id - WHERE v.token = '{token}' + WHERE v.token = $1 """ response = await self._query_first_with_cached_plan_fallback( - sql_query + sql_query, hashed_token ) # If not found in main table, check deprecated keys (grace period) diff --git a/litellm/responses/main.py b/litellm/responses/main.py index c82574278ba..bcc3f7f05e8 100644 --- a/litellm/responses/main.py +++ b/litellm/responses/main.py @@ -1967,11 +1967,18 @@ async def _aresponses_websocket( ) # Extract params that we're passing explicitly to avoid duplicates in **kwargs - remaining_kwargs = { - k: v - for k, v in kwargs.items() - if k not in {"user_api_key_dict", "litellm_metadata"} + _explicit_keys = { + "user_api_key_dict", + "litellm_metadata", + "custom_llm_provider", + "model", + "websocket", + "litellm_logging_obj", + "api_base", + "api_key", + "timeout", } + remaining_kwargs = {k: v for k, v in kwargs.items() if k not in _explicit_keys} await base_llm_http_handler.async_responses_websocket( model=model, diff --git a/pyproject.toml b/pyproject.toml index f4442ee23c4..5fcfe900f0e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "litellm" -version = "1.83.5" +version = "1.83.6" description = "Library to easily interface with LLM API providers" authors = ["BerriAI"] license = "MIT" @@ -181,7 +181,7 @@ requires = ["poetry-core", "wheel"] build-backend = "poetry.core.masonry.api" [tool.commitizen] -version = "1.83.5" +version = "1.83.6" version_files = [ "pyproject.toml:^version" ] diff --git a/tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.py b/tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.py index ea208007cd6..570e11e1bb7 100644 --- a/tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.py +++ b/tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.py @@ -3,6 +3,7 @@ import json import os import sys from datetime import datetime +from unittest.mock import Mock import pytest @@ -125,7 +126,8 @@ async def test_bedrock_sse_wrapper_keeps_usage_in_message_start_and_message_delt assert "usage" in delta_json assert delta_json["usage"]["cache_creation_input_tokens"] == 1562 assert delta_json["usage"]["cache_read_input_tokens"] == 32392 - assert delta_json["usage"]["input_tokens"] == 3 + 1562 + 32392 + assert delta_json["usage"]["input_tokens"] == 3 + assert delta_json["usage"]["output_tokens"] == 8 def test_chunk_parser_usage_transformation(): @@ -402,3 +404,111 @@ def test_bedrock_messages_strips_output_config_with_output_format(): assert "output_config" not in result assert "output_format" not in result + + +@pytest.mark.asyncio +async def test_promote_message_stop_usage_preserves_message_delta_output_tokens(): + """ + Bedrock unified /messages streaming can send full usage on message_delta and a + conflicting smaller usage on message_stop (e.g. output_tokens 9 vs 12). + _promote_message_stop_usage must not replace message_delta output_tokens. + """ + cfg = AmazonAnthropicClaudeMessagesConfig() + + async def _stream(): # type: ignore[return-type] + yield { + "type": "message_delta", + "delta": {"stop_reason": "end_turn", "stop_sequence": None}, + "usage": { + "input_tokens": 3, + "cache_creation_input_tokens": 10553, + "cache_read_input_tokens": 25490, + "output_tokens": 12, + }, + } + yield { + "type": "message_stop", + "usage": {"input_tokens": 3, "output_tokens": 9}, + } + + merged: list[dict] = [] + async for chunk in cfg._promote_message_stop_usage(_stream()): + if isinstance(chunk, dict): + merged.append(chunk) + + assert len(merged) >= 1 + delta_out = merged[0] + assert delta_out["type"] == "message_delta" + assert delta_out["usage"]["output_tokens"] == 12 + assert delta_out["usage"]["cache_creation_input_tokens"] == 10553 + assert delta_out["usage"]["cache_read_input_tokens"] == 25490 + assert delta_out["usage"]["input_tokens"] == 3 + + +@pytest.mark.asyncio +async def test_unified_bedrock_messages_sse_usage_and_cost_claude_sonnet_46(): + """ + End-to-end for Bedrock Invoke Anthropic Messages (unified) streaming path: + dict chunks -> _promote_message_stop_usage -> bedrock_sse_wrapper SSE bytes -> + same logging reconstruction as Anthropic /messages. Ensures token counts and + completion_cost match model_prices for us.anthropic.claude-sonnet-4-6. + """ + from litellm import completion_cost + from litellm.proxy.pass_through_endpoints.llm_provider_handlers.anthropic_passthrough_logging_handler import ( + AnthropicPassthroughLoggingHandler, + ) + + cfg = AmazonAnthropicClaudeMessagesConfig() + + async def _stream(): # type: ignore[return-type] + yield { + "type": "message_delta", + "delta": {"stop_reason": "end_turn", "stop_sequence": None}, + "usage": { + "input_tokens": 3, + "cache_creation_input_tokens": 10553, + "cache_read_input_tokens": 25490, + "output_tokens": 12, + }, + } + yield { + "type": "message_stop", + "usage": {"input_tokens": 3, "output_tokens": 9}, + } + + logging_obj = LiteLLMLoggingObj( + model="bedrock/us.anthropic.claude-sonnet-4-6", + messages=[{"role": "user", "content": "Hello"}], + stream=True, + call_type="chat", + start_time=datetime.now(), + litellm_call_id="test_unified_bedrock_messages_sse_cost", + function_id="test_unified_bedrock_messages_sse_cost", + ) + + collected: list[bytes] = [] + async for sse in cfg.bedrock_sse_wrapper( + completion_stream=_stream(), + litellm_logging_obj=logging_obj, + request_body={"model": "us.anthropic.claude-sonnet-4-6"}, + ): + collected.append(sse) + + built = AnthropicPassthroughLoggingHandler._build_complete_streaming_response( + all_chunks=collected, + model="us.anthropic.claude-sonnet-4-6", + litellm_logging_obj=Mock(), + ) + assert built.usage is not None + assert built.usage.completion_tokens == 12 + assert built.usage.prompt_tokens == 36046 + assert built.usage.total_tokens == 36058 + assert built.usage.cache_creation_input_tokens == 10553 + assert built.usage.cache_read_input_tokens == 25490 + + cost = completion_cost( + completion_response=built, + model="bedrock/us.anthropic.claude-sonnet-4-6", + custom_llm_provider="bedrock", + ) + assert cost == pytest.approx(0.052150725, rel=0, abs=1e-9) diff --git a/tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py b/tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py index 72092a97f5b..096e0b2bc41 100644 --- a/tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py +++ b/tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py @@ -8579,3 +8579,170 @@ def test_enforce_upperbound_no_config_is_noop(): assert data.tpm_limit == 999999 finally: litellm.upperbound_key_generate_params = original + + +class TestAllowedRoutesCallerPermission: + """ + Non-admins must not be able to set `allowed_routes` on a key. The field + bypasses the role-based route gate in + RouteChecks.non_proxy_admin_allowed_routes_check, so allowing a non-admin + to populate it grants them arbitrary endpoint access. + """ + + @pytest.mark.asyncio + async def test_non_admin_generate_key_with_allowed_routes_rejected(self): + data = GenerateKeyRequest( + key_alias="escalate", + allowed_routes=["/*"], + ) + user_api_key_dict = UserAPIKeyAuth( + user_id="internal-user-123", + user_role=LitellmUserRoles.INTERNAL_USER, + ) + mock_prisma_client = AsyncMock() + + with patch("litellm.proxy.proxy_server.prisma_client", mock_prisma_client), patch( + "litellm.proxy.proxy_server.user_api_key_cache", MagicMock() + ), patch("litellm.proxy.proxy_server.user_custom_key_generate", None), patch( + "litellm.proxy.management_endpoints.key_management_endpoints._common_key_generation_helper", + new_callable=AsyncMock, + return_value=MagicMock(), + ): + with pytest.raises(ProxyException) as exc_info: + await generate_key_fn( + data=data, + user_api_key_dict=user_api_key_dict, + litellm_changed_by=None, + ) + assert str(exc_info.value.code) == "403" + assert "allowed_routes" in str(exc_info.value.message) + + @pytest.mark.asyncio + async def test_admin_generate_key_with_allowed_routes_allowed(self): + data = GenerateKeyRequest( + key_alias="admin-key", + allowed_routes=["/chat/completions"], + user_id="admin-user", + ) + user_api_key_dict = UserAPIKeyAuth( + user_id="admin-user", + user_role=LitellmUserRoles.PROXY_ADMIN, + ) + mock_prisma_client = AsyncMock() + stub_response = MagicMock() + + with patch("litellm.proxy.proxy_server.prisma_client", mock_prisma_client), patch( + "litellm.proxy.proxy_server.user_api_key_cache", MagicMock() + ), patch("litellm.proxy.proxy_server.user_custom_key_generate", None), patch( + "litellm.proxy.management_endpoints.key_management_endpoints._common_key_generation_helper", + new_callable=AsyncMock, + return_value=stub_response, + ): + result = await generate_key_fn( + data=data, + user_api_key_dict=user_api_key_dict, + litellm_changed_by=None, + ) + assert result is stub_response + + @pytest.mark.asyncio + async def test_non_admin_generate_key_default_empty_allowed_routes_ok(self): + """ + Regression guard: GenerateKeyRequest.allowed_routes defaults to [], so + the helper must treat empty-list as "not set" or every non-admin key + creation breaks. + """ + data = GenerateKeyRequest(key_alias="plain-key") + user_api_key_dict = UserAPIKeyAuth( + user_id="internal-user-123", + user_role=LitellmUserRoles.INTERNAL_USER, + ) + mock_prisma_client = AsyncMock() + stub_response = MagicMock() + + with patch("litellm.proxy.proxy_server.prisma_client", mock_prisma_client), patch( + "litellm.proxy.proxy_server.user_api_key_cache", MagicMock() + ), patch("litellm.proxy.proxy_server.user_custom_key_generate", None), patch( + "litellm.proxy.management_endpoints.key_management_endpoints._common_key_generation_helper", + new_callable=AsyncMock, + return_value=stub_response, + ): + result = await generate_key_fn( + data=data, + user_api_key_dict=user_api_key_dict, + litellm_changed_by=None, + ) + assert result is stub_response + + @pytest.mark.asyncio + async def test_non_admin_update_key_with_allowed_routes_rejected(self): + from litellm.proxy.management_endpoints.key_management_endpoints import ( + update_key_fn, + ) + + data = UpdateKeyRequest(key="sk-test", allowed_routes=["/*"]) + user_api_key_dict = UserAPIKeyAuth( + user_id="internal-user-123", + user_role=LitellmUserRoles.INTERNAL_USER, + ) + mock_prisma_client = AsyncMock() + + with patch("litellm.proxy.proxy_server.prisma_client", mock_prisma_client), patch( + "litellm.proxy.proxy_server.user_api_key_cache", MagicMock() + ), patch("litellm.proxy.proxy_server.user_custom_key_update", None), patch( + "litellm.proxy.proxy_server.llm_router", None + ), patch("litellm.proxy.proxy_server.premium_user", True), patch( + "litellm.proxy.proxy_server.proxy_logging_obj", MagicMock() + ), patch( + "litellm.proxy.management_endpoints.key_management_endpoints._get_and_validate_existing_key", + new_callable=AsyncMock, + return_value=MagicMock(), + ): + with pytest.raises(ProxyException) as exc_info: + await update_key_fn( + request=MagicMock(), + data=data, + user_api_key_dict=user_api_key_dict, + litellm_changed_by=None, + ) + assert str(exc_info.value.code) == "403" + assert "allowed_routes" in str(exc_info.value.message) + + +def test_jinja_prompt_manager_is_sandboxed(): + """ + PromptManager renders user-supplied templates via /prompts/test, so its + jinja env must reject access to unsafe Python attributes like + ``__class__`` and ``__mro__``. + """ + from jinja2.exceptions import SecurityError + + from litellm.integrations.dotprompt.prompt_manager import PromptManager + + pm = PromptManager() + template = pm.jinja_env.from_string("{{ ''.__class__.__mro__ }}") + with pytest.raises(SecurityError): + template.render() + + +def test_validate_public_image_url_rejects_local_paths(): + from litellm.proxy.ui_crud_endpoints.proxy_setting_endpoints import ( + _validate_public_image_url, + ) + + for bad in ("/etc/passwd", "file:///etc/passwd", "../../etc/passwd"): + with pytest.raises(HTTPException) as exc_info: + _validate_public_image_url(bad, "logo_url") + assert exc_info.value.status_code == 400 + + +def test_validate_public_image_url_accepts_http_and_noop_empty(): + from litellm.proxy.ui_crud_endpoints.proxy_setting_endpoints import ( + _validate_public_image_url, + ) + + _validate_public_image_url("https://example.com/logo.png", "logo_url") + _validate_public_image_url("http://cdn.internal/logo.svg", "logo_url") + _validate_public_image_url(None, "logo_url") + _validate_public_image_url("", "logo_url") + _validate_public_image_url(" ", "logo_url") diff --git a/ui/litellm-dashboard/src/components/user_edit_view.test.tsx b/ui/litellm-dashboard/src/components/user_edit_view.test.tsx index 7f78ef45127..7aeaae94fab 100644 --- a/ui/litellm-dashboard/src/components/user_edit_view.test.tsx +++ b/ui/litellm-dashboard/src/components/user_edit_view.test.tsx @@ -1,6 +1,6 @@ -import { screen, waitFor } from "@testing-library/react"; +import { cleanup, screen, waitFor } from "@testing-library/react"; import userEvent from "@testing-library/user-event"; -import { beforeEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { renderWithProviders } from "../../tests/test-utils"; import { UserEditView } from "./user_edit_view"; @@ -140,6 +140,15 @@ describe("UserEditView", () => { vi.clearAllMocks(); }); + afterEach(() => { + // Tremor's internal Tooltip sets a setTimeout that fires after teardown, + // causing "window is not defined". Flush pending timers before cleanup. + vi.useFakeTimers(); + vi.runAllTimers(); + vi.useRealTimers(); + cleanup(); + }); + it("should render", async () => { renderWithProviders();