mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-07 07:58:09 +00:00
feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes (#49408)
* feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes
The native Codex catalog now lists the key's other chat and responses models after
ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list
refreshes on Codex's timer. A request for a model configured by exact name on another
provider goes through /v1/responses without the ChatGPT bearer or account header,
while ChatGPT names and wildcard-only matches keep the subscription passthrough.
Reasoning items another provider produced are dropped before forwarding to ChatGPT.
* fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models
Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while
`codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native
/backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no
default, so the provider applies its own default exactly as `codex -m` does
* fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes
Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses
request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may
call /v1/responses, drop unpersisted reasoning ids on the passthrough when store
is not true, keep any slug the caller's ChatGPT account lists on the subscription
via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog
added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard
precedence over a catch-all deployment.
* fix(proxy): keep Codex's native route on the subscription when the account list is unknown
A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the
account's list is known and leaves it out. A failed catalog read is remembered for 60 s and
the Codex version comes only from a User-Agent whose product matches the originator header.
The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows,
and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so
Codex stops refetching or dropping its catalog cache.
LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure
blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge
decodes wrapped thinking blocks with their signature.
The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the
native route checks the chatgpt/ form of a name the key holds only through that group. A
none reasoning effort is dropped only under drop_params. The guide covers app tools, every
routing case, and the ETag, and the catalog logs how many models the byte limit left out.
* fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API
* fix(proxy): keep Codex native turns working when the key's model listing fails
A listing callback that raises no longer turns chatgpt.com's catalog or
responses answer into a 500. The catalog keeps upstream's body and ETag, and
responses keep upstream's X-Models-Etag.
Under drop_params, a `none` reasoning effort is now dropped only when a
candidate deployment does not explicitly support it, so an alias to a model
like openai/gpt-5.4 keeps it.
The account slug list and its unavailable marker are keyed on the account id
plus the caller's ChatGPT Authorization header, so a failed fetch with one
bearer cannot turn off LiteLLM routing for another caller on the same account.
The auth step checks the Codex route before importing the Codex module.
* fix(proxy): keep Codex model switches working across LiteLLM providers
On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400
Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts
A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows
* fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers
Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the
native route now hoists them without flattening, like OpenAI and Azure and
the chat completions bridge. Other native Responses providers still get
flat names, now built with the bridge's namespace_tool_name_map, and each
function_call they return gets its tool name and namespace back in the
non-streaming body and in streamed output_item.added, output_item.done,
and completed/incomplete/failed events. Every other event passes through
unchanged
* fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools
Codex sends its freeform exec tool as a custom tool on every turn, and xAI's
Responses API answers 422 for custom, namespace and additional_tools shapes.
On the Codex native route, a model whose deployments include a native
Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now
asks for the chat completions bridge with use_chat_completions_api, which
already turns custom and namespace tools into functions, hoists
additional_tools, and puts the namespace back on each returned call
OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock
Mantle still get it on their own Responses API with additional_tools hoisted.
The route's namespace flattening and the response-side namespace restore are
removed, since no path sends flattened names anymore
* fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve
A reasoning item another deployment produced, replayed with store false and no blob the
target can decrypt, now reaches the target as its readable summary without id or status,
and is dropped when nothing readable is left. OpenAI answered 400 on the status field and
404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment
replays and stored conversations keep the item whole
* refactor(codex): reach encrypted reasoning stripping through the router registry
The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from
litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry
gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with
encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise
every target must decrypt it
* fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog
* fix(proxy): drop Codex's none effort for an alias whose native entry names no default
* fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop
* test(proxy): map the pinned reasoning efforts field in the router registry stub
* fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases
Codex native routes now send a key to LiteLLM only when admin_only_routes allows it,
a native alias with configured service tiers starts with no tier picked, and the
account slug cache declares its Redis key family.
* fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items
* fix(proxy): check vector store grants on file_search tools inside additional_tools items
A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input.
* fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes
* fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply
* fix(guardrails): check additional_tools items next to top-level function tools in tool_policy
ToolPolicyGuardrail read tools from the request body only when no top-level
function tool existed, so a blocked tool defined in an additional_tools item
reached Fireworks once the Responses transform moved the item into tools after
the pre-call hooks. Check every list on each request, including namespace
members inside items.
* fix(router): serve decryptable_reasoning from the Rust core's optional checks
The merge with main removed the Python router core, so the registry's host has no
optional_callbacks and no routing context, and native Codex requests with encrypted
reasoning raised RustCoreUnsupported. The filter now lives with the optional checks,
which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core
Also give the static litellm.Router a declared constructor. Since the Python core was
deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...)
is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change
* fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns
Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures.
* fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias
* fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases
Auth no longer asks ChatGPT for the account's model list when the bearer is
missing or is the LiteLLM key, so a key sent as the bearer never reaches it.
tool_policy now reads the members of a top-level namespace, which is where
the native route's hoist puts them.
A global alias now reads encrypted content affinity from its target group.
A non-list top-level tools value and a non-string function name now reach the
provider, which refuses them, instead of raising a 500.
The catalog ETag no longer depends on the order a process iterates priced
service tiers in, so workers and pods agree on it.
* fix(proxy): list a team's models in the order the team set them
get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker
* fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models
A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models
and /backend-api/codex/responses only when the JWT auth flow would admit a
direct /v1/responses call for that model, so team_allowed_routes,
admin_allowed_routes and RBAC route limits hold on the handoff too.
The Fireworks Responses transform also leaves additional_tools input items
in place when the tools param is not a list, instead of raising while hoisting
* fix(fireworks): hoist additional_tools when tools is a tuple
litellm.responses passes a tuple tools argument through to the Fireworks
transform unchanged, so the non-list guard skipped hoisting and left the
additional_tools item in the input. Treat tuples like lists and cover both
shapes in the drop_params merge test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
This commit is contained in:
simonp9
GitHub
parent
a94118c6cd
commit
1f15088fc9
39 changed files with 3834 additions and 243 deletions
|
|
@ -46,7 +46,43 @@ For a managed workspace rollout, use OpenAI's [gateway configuration guide](http
|
|||
|
||||
## Native compatibility and Pi
|
||||
|
||||
For a native Codex model catalog and request format, change Codex's `base_url` to `https://litellm.example.com/backend-api/codex`. This route uses the same credentials but does not need a model deployment or resolve model aliases. Restricted keys must allow the upstream model name.
|
||||
For a native Codex model catalog and request format, change Codex's `base_url` to `https://litellm.example.com/backend-api/codex`. This route uses the same credentials. A ChatGPT model keeps its upstream name and needs no deployment of its own.
|
||||
|
||||
### Mix ChatGPT and LiteLLM models
|
||||
|
||||
On the native route, Codex's model picker shows OpenAI's catalog first and then the other chat and responses models the virtual key can use. Pick one of those and Codex's requests go through LiteLLM's `/v1/responses` with the key's usual routing, budgets, guardrails, and logging. The proxy removes the ChatGPT bearer and account header from that request, so the other provider and your callbacks never receive them. A name is routed this way when another provider serves it as an exact `model_name`, through a router `model_group_alias`, or as a team's public model name. A ChatGPT model, a name that only a wildcard deployment matches, or a name OpenAI's catalog lists for the caller's ChatGPT account keeps the passthrough, so a model OpenAI adds later still uses the subscription even when a deployment of another provider has the same name. The proxy learns the account's list from the catalog Codex fetches, or asks OpenAI with the caller's login. When it cannot get the list, for example because OpenAI is down, the request times out, or the client is not Codex, every request keeps the passthrough for the next minute, as it did before LiteLLM models were added to the picker. When the key can use many models, the picker lists the ones that fit in Codex's catalog size limit, and the proxy logs a warning with how many it left out.
|
||||
|
||||
Codex's picker sends a `none` reasoning effort for a LiteLLM model whose catalog entry lists no effort levels. With `litellm_settings: drop_params: true`, the proxy drops it for a model that does not support `none`, and the provider applies its own default. A model whose cost map entry or `model_info` sets `supports_none_reasoning_effort: true` keeps it. Without `drop_params`, the effort reaches the provider unchanged.
|
||||
|
||||
Codex also sends its ChatGPT app tools with every turn, which on a LiteLLM model adds a few hundred thousand prompt tokens to each turn. Set `apps = false` under `[features]` in Codex's `config.toml` to turn them off.
|
||||
|
||||
Only a key that may call `/v1/responses` gets these models. A key whose `allowed_routes` covers only the native routes sees OpenAI's catalog unchanged, and every request it sends uses the subscription.
|
||||
|
||||
Codex sends earlier reasoning items back with each turn. The proxy tags the reasoning that a LiteLLM model returns on this route, OpenAI and Azure deployments included. When a session switches to a ChatGPT model, the proxy drops those items before forwarding the request to OpenAI, which would reject them. Messages, tool calls, and ChatGPT's own reasoning are kept, and a LiteLLM model still gets its own reasoning back.
|
||||
|
||||
When the catalog adds LiteLLM models for a key, its `ETag` is a digest of OpenAI's ETag and those models, and that key's responses carry the same value in `X-Models-Etag`. Codex keeps its cached catalog while the two match and fetches it again when OpenAI's catalog or the key's models change. A key whose catalog adds no LiteLLM models keeps OpenAI's ETags.
|
||||
|
||||
### Restrict native models with an access group
|
||||
|
||||
A key with a model allowlist needs a rule for ChatGPT names that OpenAI adds over time. Put the subscription wildcards in an access group under `chatgpt/` names:
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: chatgpt/gpt-*
|
||||
litellm_params:
|
||||
model: chatgpt/gpt-*
|
||||
model_info:
|
||||
access_groups: ["chatgpt-native"]
|
||||
- model_name: chatgpt/codex-*
|
||||
litellm_params:
|
||||
model: chatgpt/codex-*
|
||||
model_info:
|
||||
access_groups: ["chatgpt-native"]
|
||||
```
|
||||
|
||||
Then add `chatgpt-native` to the key's models, next to any other model names it should reach. On the native route, a key that holds only the `chatgpt/` form of a name is checked as that form, so a request for `gpt-5.6-sol` is allowed when the key may call `chatgpt/gpt-5.6-sol`. The key can call every ChatGPT model that matches the wildcards, including ones the proxy configuration does not name.
|
||||
|
||||
On `/v1` these wildcards match only a request that asks for a `chatgpt/` name, so a request for a plain `gpt-` or `codex-` name still goes where it went before, such as a `*` catch-all deployment. A `/v1` request for a `chatgpt/` name without a ChatGPT login falls back to the proxy's own saved login if it has one, so keep that login unset as described above.
|
||||
|
||||
### Configure Pi
|
||||
|
||||
|
|
@ -71,7 +107,7 @@ Keep Pi's subscription login as the provider credential. The header value names
|
|||
|
||||
Send a prompt and check the completed request in LiteLLM's Logs page. Its gateway user/key and token counts should match the caller. Give each person their own LiteLLM virtual key for separate attribution. The gateway credential and ChatGPT login are independently authenticated; LiteLLM does not establish that both identities belong to the same person.
|
||||
|
||||
The native routes are `GET /backend-api/codex/models` and `POST /backend-api/codex/responses`. Model requests use the upstream model name, so a restricted LiteLLM key must allow that name. OpenAI's model catalog is returned in its native format; seeing a model there does not override gateway permissions.
|
||||
The native routes are `GET /backend-api/codex/models` and `POST /backend-api/codex/responses`. Model requests use the upstream model name, so a restricted LiteLLM key needs the `chatgpt-native` access group above or that exact name. OpenAI's model catalog is returned in its native format; seeing a model there does not override gateway permissions.
|
||||
|
||||
The proxy forwards the ChatGPT bearer and account header to the fixed OpenAI Codex endpoint. It removes gateway credentials and cookies from the upstream request and redacts credential headers in logging callbacks. The client remains responsible for refresh and reauthentication. Native passthrough returns authentication and quota errors without switching subscriptions. Standard model routes retain the caller login across ChatGPT retries; explicitly configured cross-provider fallbacks use their own credentials. An explicit workspace header is preserved for OpenAI to validate.
|
||||
|
||||
|
|
|
|||
|
|
@ -54,6 +54,7 @@ from litellm.llms.base_llm.bridges.completion_transformation import (
|
|||
CompletionTransformationBridge,
|
||||
)
|
||||
from litellm.llms.custom_httpx.sse_lines import split_sse_lines
|
||||
from litellm.responses.additional_tools import additional_tools_in_input
|
||||
from litellm.responses.litellm_completion_transformation.tool_search import is_anthropic_tool_search_chat_tool
|
||||
from litellm.responses.sse_output_recovery import (
|
||||
parse_sse_json_chunk,
|
||||
|
|
@ -426,6 +427,17 @@ def replayed_tool_search_tool_names(request: object) -> tuple[str, ...]:
|
|||
return tuple(chain.from_iterable(_tool_definition_name(tool) for tool in replayed_tool_search_tools(request)))
|
||||
|
||||
|
||||
def _top_level_namespaces(request: object) -> Iterator[object]:
|
||||
tools: Final = _json_list(_json_field(request, "tools")) or ()
|
||||
return (tool for tool in tools if _json_field(tool, "type") == "namespace")
|
||||
|
||||
|
||||
def nested_tool_member_names(request: object) -> tuple[str, ...]:
|
||||
tools: Final = (*_top_level_namespaces(request), *additional_tools_in_input(_json_field(request, "input")))
|
||||
members: Final = chain.from_iterable(_namespace_members(tool) for tool in tools)
|
||||
return tuple(chain.from_iterable(_tool_definition_name(member) for member in members))
|
||||
|
||||
|
||||
def _loads_any_tool(message: object, names: frozenset[str]) -> bool:
|
||||
loaded_names: Final = chain.from_iterable(_tool_definition_name(tool) for tool in _message_loaded_tools(message))
|
||||
return any(name in names for name in loaded_names)
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
from collections.abc import Mapping
|
||||
from collections.abc import Mapping, Sequence
|
||||
from types import MappingProxyType
|
||||
from typing import TYPE_CHECKING, Final
|
||||
from typing import TYPE_CHECKING, Final, cast # noqa: TID251 # the base signature leaves the params bag untyped
|
||||
from urllib.parse import unquote
|
||||
|
||||
import httpx
|
||||
|
||||
from litellm.litellm_core_utils.core_helpers import drops_unsupported_params
|
||||
from litellm.litellm_core_utils.url_utils import join_api_base
|
||||
from litellm.llms.base_llm.responses.transformation import (
|
||||
STANDARD_MODE_WITHOUT_CONTEXT_REASONING_FIELD_VALUES,
|
||||
|
|
@ -20,8 +21,9 @@ from litellm.llms.fireworks_ai.common_utils import (
|
|||
)
|
||||
from litellm.llms.openai.responses.instruction_items import developer_items_as_system, instruction_items_folded
|
||||
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
|
||||
from litellm.responses.additional_tools import HoistedAdditionalTools, hoist_additional_tools
|
||||
from litellm.secret_managers.main import get_secret_str
|
||||
from litellm.types.llms.openai import ResponseInputParam
|
||||
from litellm.types.llms.openai import ALL_RESPONSES_API_TOOL_PARAMS, ResponseInputParam
|
||||
from litellm.types.responses.main import DeleteResponseResult
|
||||
from litellm.types.router import GenericLiteLLMParams
|
||||
from litellm.types.utils import LlmProviders
|
||||
|
|
@ -98,6 +100,23 @@ class FireworksAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
|
|||
headers: dict[str, str], # mutable-ok: overrides the base class signature
|
||||
) -> dict[str, object]: # mutable-ok: overrides the base class signature
|
||||
validated_input: Final = self._validate_input_param(input)
|
||||
tools_param: Final = response_api_optional_request_params.get("tools")
|
||||
hoisted: Final = (
|
||||
hoist_additional_tools(
|
||||
validated_input,
|
||||
cast( # cast-ok: the base signature leaves the params bag untyped
|
||||
"Sequence[ALL_RESPONSES_API_TOOL_PARAMS] | None", tools_param
|
||||
),
|
||||
drop_params=drops_unsupported_params(litellm_params.drop_params),
|
||||
)
|
||||
if tools_param is None or isinstance(tools_param, (list, tuple))
|
||||
else HoistedAdditionalTools(input=validated_input, tools=(), hoisted=())
|
||||
)
|
||||
hoisted_params: Final = (
|
||||
{**response_api_optional_request_params, "tools": list(hoisted.tools)}
|
||||
if hoisted.hoisted
|
||||
else response_api_optional_request_params
|
||||
)
|
||||
model_info: Final = deployment_model_info(litellm_params)
|
||||
declared: Final = declared_mid_conversation_system_support(model, self.custom_llm_provider.value, model_info)
|
||||
keeps_developer_items_in_place: Final = declared is True and (
|
||||
|
|
@ -107,14 +126,9 @@ class FireworksAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
|
|||
folded: Final = (
|
||||
None
|
||||
if keeps_developer_items_in_place
|
||||
else instruction_items_folded(
|
||||
validated_input, response_api_optional_request_params, "user" if declared is False else "system"
|
||||
)
|
||||
)
|
||||
placed_input, placed_params = folded or (
|
||||
developer_items_as_system(validated_input),
|
||||
response_api_optional_request_params,
|
||||
else instruction_items_folded(hoisted.input, hoisted_params, "user" if declared is False else "system")
|
||||
)
|
||||
placed_input, placed_params = folded or (developer_items_as_system(hoisted.input), hoisted_params)
|
||||
request: Final = super().transform_responses_api_request(
|
||||
model=resolve_fireworks_resource_name(model),
|
||||
input=placed_input,
|
||||
|
|
|
|||
|
|
@ -78,6 +78,7 @@ from litellm.llms.openai.responses.guardrail_translation.tool_merge import (
|
|||
append_returned_tools,
|
||||
merge_guardrailed_tools,
|
||||
)
|
||||
from litellm.responses.additional_tools import additional_tools_in_input
|
||||
from litellm.responses.litellm_completion_transformation.compaction_items import (
|
||||
compaction_summary,
|
||||
decode_compaction_item,
|
||||
|
|
@ -989,9 +990,14 @@ class OpenAIResponsesHandler(BaseTranslation):
|
|||
|
||||
def extract_request_tool_names(self, data: Mapping[str, object]) -> list[str]:
|
||||
"""Extract tool names from Responses API request (tools[].name for function
|
||||
and custom, tools[].server_label for mcp)."""
|
||||
and custom, tools[].server_label for mcp), including tools carried by
|
||||
``additional_tools`` input items."""
|
||||
names: Final[list[str]] = []
|
||||
for tool in list_field(data, "tools"):
|
||||
request_tools: Final[tuple[object, ...]] = (
|
||||
*list_field(data, "tools"),
|
||||
*additional_tools_in_input(data.get("input")),
|
||||
)
|
||||
for tool in request_tools:
|
||||
if not isinstance(tool, dict):
|
||||
continue
|
||||
if tool.get("type") in ("function", "custom") and tool.get("name"):
|
||||
|
|
|
|||
|
|
@ -9633,7 +9633,7 @@
|
|||
"paths": {
|
||||
"/backend-api/codex/{endpoint}": {
|
||||
"get": {
|
||||
"description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication.",
|
||||
"description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the\nkey's other chat models through LiteLLM.",
|
||||
"operationId": "codex_passthrough_route_backend_api_codex__endpoint__get",
|
||||
"parameters": [
|
||||
{
|
||||
|
|
@ -9681,7 +9681,7 @@
|
|||
]
|
||||
},
|
||||
"post": {
|
||||
"description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication.",
|
||||
"description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the\nkey's other chat models through LiteLLM.",
|
||||
"operationId": "codex_passthrough_route_backend_api_codex__endpoint__post",
|
||||
"parameters": [
|
||||
{
|
||||
|
|
@ -9724,7 +9724,7 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
"description": "Unmodified Codex response or server-sent events"
|
||||
"description": "Codex response, the model catalog with the key's LiteLLM models, or server-sent events"
|
||||
},
|
||||
"422": {
|
||||
"content": {
|
||||
|
|
|
|||
|
|
@ -180,6 +180,7 @@ from litellm.repositories.table_repositories import (
|
|||
)
|
||||
from litellm.repositories.team_repository import TeamRepository
|
||||
from litellm.repositories.user_repository import UserRepository
|
||||
from litellm.responses.additional_tools import additional_tools_in_input
|
||||
from litellm.router import Router
|
||||
from litellm.router_api import RegistryReader
|
||||
from litellm.types.proxy.auth.auth_checks import UserNotFoundError
|
||||
|
|
@ -1037,9 +1038,6 @@ async def check_tools_allowlist(
|
|||
call_types: Final = get_call_types_for_route(route)
|
||||
if not call_types or not any(ct.value in TOOL_CAPABLE_CALL_TYPES for ct in call_types):
|
||||
return
|
||||
tool_names: Final = extract_request_tool_names(route, request_body)
|
||||
if not tool_names:
|
||||
return
|
||||
key_meta: Final = (valid_token.metadata or {}) if isinstance(valid_token.metadata, dict) else {}
|
||||
team_meta: Final = (valid_token.team_metadata or {}) if isinstance(valid_token.team_metadata, dict) else {}
|
||||
key_allowed: Final = key_meta.get("allowed_tools")
|
||||
|
|
@ -1047,6 +1045,9 @@ async def check_tools_allowlist(
|
|||
effective: Final = key_allowed if (isinstance(key_allowed, list) and len(key_allowed) > 0) else team_allowed
|
||||
if not isinstance(effective, list) or len(effective) == 0:
|
||||
return
|
||||
tool_names: Final = extract_request_tool_names(route, request_body)
|
||||
if not tool_names:
|
||||
return
|
||||
allowed_set: Final = {str(t) for t in effective}
|
||||
disallowed: Final = [n for n in tool_names if n not in allowed_set]
|
||||
if disallowed:
|
||||
|
|
@ -7988,6 +7989,10 @@ def _tools(tools: object) -> tuple[object, ...]:
|
|||
return ()
|
||||
|
||||
|
||||
def _request_tools(request_body: Mapping[str, object]) -> tuple[object, ...]:
|
||||
return (*_tools(request_body.get("tools")), *additional_tools_in_input(request_body.get("input")))
|
||||
|
||||
|
||||
def _tool_vector_store_ids(tool: object) -> tuple[str, ...]:
|
||||
try:
|
||||
tool_fields: Final = _TOOL_ADAPTER.validate_python(tool, strict=True)
|
||||
|
|
@ -7998,14 +8003,14 @@ def _tool_vector_store_ids(tool: object) -> tuple[str, ...]:
|
|||
|
||||
def _strict_requested_vector_store_ids(request_body: Mapping[str, object]) -> tuple[str, ...]:
|
||||
"""
|
||||
Same fields VectorStoreRegistry.get_vector_store_ids_to_run reads, but a vector_store_ids that is
|
||||
not a list of strings is a 400 instead of being skipped or iterated, and tools that are not
|
||||
objects name no store.
|
||||
Same fields VectorStoreRegistry.get_vector_store_ids_to_run reads, plus the tools of `additional_tools`
|
||||
input items, but a vector_store_ids that is not a list of strings is a 400 instead of being skipped or
|
||||
iterated, and tools that are not objects name no store.
|
||||
"""
|
||||
return tuple(
|
||||
chain(
|
||||
_validated_vector_store_ids(request_body.get("vector_store_ids")),
|
||||
chain.from_iterable(_tool_vector_store_ids(tool) for tool in _tools(request_body.get("tools"))),
|
||||
chain.from_iterable(_tool_vector_store_ids(tool) for tool in _request_tools(request_body)),
|
||||
)
|
||||
)
|
||||
|
||||
|
|
@ -8044,7 +8049,7 @@ def _legacy_requested_vector_store_ids(
|
|||
return tuple(
|
||||
chain(
|
||||
top_level,
|
||||
chain.from_iterable(vector_store_ids_in_tool(tool) for tool in _tools(request_body.get("tools"))),
|
||||
chain.from_iterable(vector_store_ids_in_tool(tool) for tool in _request_tools(request_body)),
|
||||
)
|
||||
)
|
||||
|
||||
|
|
@ -8113,8 +8118,8 @@ async def vector_store_access_check(
|
|||
"""
|
||||
Checks whether the caller may use every vector store the request names.
|
||||
|
||||
Requested stores come from `vector_store_ids`, `tools[].vector_store_ids` and the RAG
|
||||
`retrieval_config.vector_store_id`. Grants come from each identity's
|
||||
Requested stores come from `vector_store_ids`, `tools[].vector_store_ids`, the tools of
|
||||
`additional_tools` input items and the RAG `retrieval_config.vector_store_id`. Grants come from each identity's
|
||||
`object_permission.vector_stores`.
|
||||
|
||||
Both modes read the same request fields, with or without a vector store registry, and answer 400
|
||||
|
|
|
|||
|
|
@ -863,7 +863,7 @@ async def pre_db_read_auth_checks(
|
|||
Raises:
|
||||
- HTTPException if request fails initial auth checks
|
||||
"""
|
||||
from litellm.proxy.proxy_server import general_settings, llm_router, premium_user
|
||||
from litellm.proxy.proxy_server import general_settings, llm_router
|
||||
|
||||
# Check 1. request size
|
||||
await check_if_request_size_is_safe(request=request)
|
||||
|
|
@ -891,19 +891,12 @@ async def pre_db_read_auth_checks(
|
|||
)
|
||||
|
||||
# Check 4. Check if request route is an allowed route on the proxy
|
||||
if "allowed_routes" in general_settings:
|
||||
_allowed_routes: Final = general_settings["allowed_routes"]
|
||||
if premium_user is not True:
|
||||
verbose_proxy_logger.error(
|
||||
"Trying to set allowed_routes. This is an Enterprise feature. %s",
|
||||
CommonProxyErrors.not_premium_user.value,
|
||||
)
|
||||
if route not in _allowed_routes:
|
||||
verbose_proxy_logger.error("Route %s not in allowed_routes=%s", route, _allowed_routes)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail=f"Access forbidden: Route {route} not allowed",
|
||||
)
|
||||
if not RouteChecks.is_allowed_on_proxy(route):
|
||||
verbose_proxy_logger.error("Route %s not in allowed_routes=%s", route, general_settings["allowed_routes"])
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail=f"Access forbidden: Route {route} not allowed",
|
||||
)
|
||||
|
||||
|
||||
def route_in_additonal_public_routes(current_route: str):
|
||||
|
|
|
|||
|
|
@ -192,34 +192,25 @@ def get_team_models(
|
|||
- Empty list if no models set
|
||||
- If model_access_groups is provided, only return models that are in the access groups
|
||||
"""
|
||||
all_models_set: Final[set[str]] = set()
|
||||
if len(team_models) > 0:
|
||||
all_models_set.update(team_models)
|
||||
if SpecialModelNames.all_team_models.value in all_models_set:
|
||||
all_models_set.update(team_models)
|
||||
# GH#30619: expand all-team-models sentinel
|
||||
# to the actual proxy model list
|
||||
all_models_set.discard(SpecialModelNames.all_team_models.value)
|
||||
all_models_set.update(proxy_model_list)
|
||||
if include_model_access_groups:
|
||||
all_models_set.update(model_access_groups.keys())
|
||||
if SpecialModelNames.all_proxy_models.value in all_models_set:
|
||||
all_models_set.update(proxy_model_list)
|
||||
if include_model_access_groups:
|
||||
all_models_set.update(model_access_groups.keys())
|
||||
|
||||
all_models = _get_models_from_access_groups(
|
||||
expands_to_proxy_models: Final = (
|
||||
SpecialModelNames.all_team_models.value in team_models
|
||||
or SpecialModelNames.all_proxy_models.value in team_models
|
||||
)
|
||||
expansion: Final = (
|
||||
(*proxy_model_list, *(model_access_groups if include_model_access_groups else ()))
|
||||
if expands_to_proxy_models
|
||||
else ()
|
||||
)
|
||||
named_models: Final = (model for model in team_models if model != SpecialModelNames.all_team_models.value)
|
||||
all_models: Final = _get_models_from_access_groups(
|
||||
model_access_groups=model_access_groups,
|
||||
all_models=list(all_models_set),
|
||||
all_models=list(dict.fromkeys((*named_models, *expansion))),
|
||||
include_model_access_groups=include_model_access_groups,
|
||||
proxy_model_list=proxy_model_list,
|
||||
)
|
||||
|
||||
# deduplicate while preserving order
|
||||
all_models = list(dict.fromkeys(all_models))
|
||||
|
||||
verbose_proxy_logger.debug("ALL TEAM MODELS - %s", len(all_models))
|
||||
return all_models
|
||||
return list(dict.fromkeys(all_models))
|
||||
|
||||
|
||||
def get_complete_model_list(
|
||||
|
|
|
|||
|
|
@ -439,6 +439,19 @@ class RouteChecks:
|
|||
detail=f"user not allowed to access this route. Route={route} is an admin only route",
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def is_allowed_on_proxy(route: str) -> bool:
|
||||
from litellm.proxy.proxy_server import general_settings, premium_user
|
||||
|
||||
if "allowed_routes" not in general_settings:
|
||||
return True
|
||||
if premium_user is not True:
|
||||
verbose_proxy_logger.error(
|
||||
"Trying to set allowed_routes. This is an Enterprise feature. %s",
|
||||
CommonProxyErrors.not_premium_user.value,
|
||||
)
|
||||
return route in general_settings["allowed_routes"]
|
||||
|
||||
@staticmethod
|
||||
def is_unbilled_route(route: str) -> bool:
|
||||
resource, _, action = route.rsplit("/", 1)[-1].partition(":")
|
||||
|
|
|
|||
|
|
@ -309,6 +309,17 @@ async def _normalize_claude_model(
|
|||
rewrite_request_model(request_data, request, source)
|
||||
|
||||
|
||||
_CODEX_NATIVE_RESPONSES_ROUTE: Final = "/backend-api/codex/responses"
|
||||
|
||||
|
||||
def _is_codex_native_litellm_turn(route: str, model: object, llm_router: litellm.Router, team_id: str | None) -> bool:
|
||||
if route != _CODEX_NATIVE_RESPONSES_ROUTE:
|
||||
return False
|
||||
from litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints import routes_to_litellm
|
||||
|
||||
return routes_to_litellm(model, llm_router, team_id)
|
||||
|
||||
|
||||
async def _resolve_router_settings_model_group_alias(
|
||||
request_data: dict[str, object], # mutable-ok: the request body is rewritten in place for every downstream reader
|
||||
valid_token: UserAPIKeyAuth,
|
||||
|
|
@ -325,9 +336,12 @@ async def _resolve_router_settings_model_group_alias(
|
|||
if request.scope.get(MODEL_GROUP_ALIAS_RESOLVED_SCOPE_KEY) is True:
|
||||
return
|
||||
request.scope[MODEL_GROUP_ALIAS_RESOLVED_SCOPE_KEY] = True
|
||||
if request_dispatched_to_pass_through_endpoint(request) or request_dispatched_to_provider_pass_through(request):
|
||||
return
|
||||
requested: Final = request_data.get("model")
|
||||
if request_dispatched_to_pass_through_endpoint(request) or (
|
||||
request_dispatched_to_provider_pass_through(request)
|
||||
and not _is_codex_native_litellm_turn(route, requested, llm_router, valid_token.team_id)
|
||||
):
|
||||
return
|
||||
if not isinstance(requested, str) or await read_raw_json_body(request=request) is None:
|
||||
return
|
||||
settings: Final = await proxy_config.get_hierarchical_router_settings(
|
||||
|
|
@ -338,6 +352,11 @@ async def _resolve_router_settings_model_group_alias(
|
|||
target: Final = resolve_model_group_alias(settings.get("model_group_alias"), requested)
|
||||
if target is None or target == requested:
|
||||
return
|
||||
if request_dispatched_to_provider_pass_through(request):
|
||||
from litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints import native_turn_serves_through_litellm
|
||||
|
||||
if not await native_turn_serves_through_litellm(requested, llm_router, request, valid_token):
|
||||
return
|
||||
verbose_proxy_logger.debug(
|
||||
"router_settings.model_group_alias resolved %s -> %s before auth",
|
||||
requested.replace("\r", "").replace("\n", ""),
|
||||
|
|
@ -347,6 +366,27 @@ async def _resolve_router_settings_model_group_alias(
|
|||
rewrite_request_model(request_data, request, target)
|
||||
|
||||
|
||||
def _resolve_codex_native_chatgpt_model(
|
||||
request_data: dict[str, object], # mutable-ok: the request body is rewritten in place for every downstream reader
|
||||
valid_token: UserAPIKeyAuth,
|
||||
request: Request | None,
|
||||
route: str,
|
||||
) -> None:
|
||||
requested: Final = request_data.get("model")
|
||||
if route != _CODEX_NATIVE_RESPONSES_ROUTE or request is None or not isinstance(requested, str):
|
||||
return
|
||||
from litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints import native_chatgpt_model
|
||||
from litellm.proxy.proxy_server import llm_router
|
||||
|
||||
if llm_router is None:
|
||||
return
|
||||
target: Final = native_chatgpt_model(requested, llm_router, valid_token)
|
||||
if target is None:
|
||||
return
|
||||
request.scope.setdefault(CLIENT_REQUESTED_MODEL_SCOPE_KEY, requested)
|
||||
rewrite_request_model(request_data, request, target)
|
||||
|
||||
|
||||
def _get_model_names_for_budget_checks(
|
||||
model: str | list[str] | None,
|
||||
) -> list[str]:
|
||||
|
|
@ -4172,6 +4212,7 @@ async def _authorize_authenticated_request(
|
|||
RouteChecks.should_call_route(route=route, valid_token=user_api_key_auth_obj, request=request)
|
||||
await _normalize_claude_model(request_data, user_api_key_auth_obj, request, route)
|
||||
await _resolve_router_settings_model_group_alias(request_data, user_api_key_auth_obj, request, route)
|
||||
_resolve_codex_native_chatgpt_model(request_data, user_api_key_auth_obj, request, route)
|
||||
|
||||
# Single authorization point. Builder paths MUST NOT call common_checks.
|
||||
# Route through the same exception handler the builder uses so
|
||||
|
|
@ -4650,6 +4691,7 @@ async def enforce_key_and_fallback_model_access(
|
|||
"""
|
||||
await _normalize_claude_model(request_data, valid_token, request, route)
|
||||
await _resolve_router_settings_model_group_alias(request_data, valid_token, request, route)
|
||||
_resolve_codex_native_chatgpt_model(request_data, valid_token, request, route)
|
||||
config: Final = valid_token.config
|
||||
|
||||
if config != {}:
|
||||
|
|
|
|||
|
|
@ -24,8 +24,10 @@ a newer tag.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import asdict, dataclass
|
||||
from functools import cache
|
||||
from itertools import accumulate, pairwise
|
||||
from pathlib import Path
|
||||
|
|
@ -33,7 +35,7 @@ from types import MappingProxyType
|
|||
from typing import TYPE_CHECKING, Final, Literal, TypeAlias
|
||||
|
||||
from packaging.version import Version
|
||||
from pydantic import ConfigDict, TypeAdapter, ValidationError
|
||||
from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
|
||||
|
||||
from litellm._logging import verbose_proxy_logger
|
||||
from litellm.proxy.common_utils.codex_model_translation import (
|
||||
|
|
@ -61,6 +63,8 @@ CODEX_CHAT_MODES: Final = frozenset({"chat", "responses"})
|
|||
|
||||
_BODY_PREFIX: Final = '{"models":['
|
||||
_BODY_SUFFIX: Final = "]}"
|
||||
_JSON_OBJECT: Final = TypeAdapter(dict[str, object])
|
||||
_JSON_OBJECTS: Final = TypeAdapter(tuple[dict[str, object], ...])
|
||||
_NO_UPDATE: Final[Mapping[str, object]] = MappingProxyType({})
|
||||
|
||||
|
||||
|
|
@ -119,6 +123,7 @@ class CodexStockCatalog(LiteLLMBaseModel):
|
|||
|
||||
CodexCatalogEntry = CodexFallbackModel | CodexStockModel
|
||||
CodexAliasDefaults: TypeAlias = Literal["codex", "deployment"]
|
||||
_NATIVE_ALIAS_DEFAULTS: Final[CodexAliasDefaults] = "deployment"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
|
|
@ -408,6 +413,7 @@ def _entry(
|
|||
stock_model.upgrade if stock_model.upgrade is not None and stock_model.upgrade.model in served else None
|
||||
)
|
||||
is_alias: Final = row.id != stock_model.slug
|
||||
deployment_decides: Final = is_alias and alias_defaults == "deployment"
|
||||
stock_name: Final = row.id if is_alias else stock_model.display_name
|
||||
listing: Final = {
|
||||
"slug": row.id,
|
||||
|
|
@ -417,12 +423,18 @@ def _entry(
|
|||
"supported_in_api": True,
|
||||
"upgrade": upgrade,
|
||||
**({} if search_tool is None else {"supports_search_tool": search_tool}),
|
||||
**(_deployment_defaults(row, stock_model) if is_alias and alias_defaults == "deployment" else {}),
|
||||
**(_deployment_defaults(row, stock_model) if deployment_decides else {}),
|
||||
**_configured_window(row, stock_model),
|
||||
}
|
||||
if configured is None:
|
||||
return stock_model.model_copy(update={**_priced_tiers(row, stock_model), **listing})
|
||||
return stock_model.model_copy(update={**listing, **_tier_fields(stock_model, configured)})
|
||||
return stock_model.model_copy(
|
||||
update={
|
||||
**listing,
|
||||
**_tier_fields(stock_model, configured),
|
||||
**({"default_service_tier": None} if deployment_decides else {}),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _serialized(entry: CodexCatalogEntry) -> str:
|
||||
|
|
@ -435,13 +447,12 @@ def _bytes_left_after(left: int, size: int) -> int:
|
|||
return left - size if size <= left else left
|
||||
|
||||
|
||||
def _fits(serialized: Sequence[str], byte_limit: int | None) -> tuple[bool, ...]:
|
||||
"""Whether each entry, taken in order, fits in what is left of a body of at most `byte_limit`
|
||||
bytes: an entry larger than the bytes left is passed over and a smaller one after it is still
|
||||
taken, each entry counted with its separating comma and the last one's comma given back."""
|
||||
if byte_limit is None:
|
||||
def _fits(serialized: Sequence[str], budget: int | None) -> tuple[bool, ...]:
|
||||
"""Whether each entry, taken in order, fits in what is left of `budget` bytes: an entry larger
|
||||
than the bytes left is passed over and a smaller one after it is still taken, each entry
|
||||
counted with its separating comma."""
|
||||
if budget is None:
|
||||
return (True,) * len(serialized)
|
||||
budget: Final = byte_limit - len(_BODY_PREFIX) - len(_BODY_SUFFIX) + 1
|
||||
left: Final = tuple(
|
||||
accumulate((len(entry.encode()) + 1 for entry in serialized), _bytes_left_after, initial=budget)
|
||||
)
|
||||
|
|
@ -449,17 +460,34 @@ def _fits(serialized: Sequence[str], byte_limit: int | None) -> tuple[bool, ...]
|
|||
|
||||
|
||||
def _kept_positions(
|
||||
entries: Sequence[CodexCatalogEntry], serialized: Sequence[str], byte_limit: int | None
|
||||
entries: Sequence[CodexCatalogEntry], serialized: Sequence[str], budget: int | None
|
||||
) -> frozenset[int]:
|
||||
"""The listing positions that survive the byte limit: an entry offering a service tier is taken
|
||||
ahead of one offering none, each group in listing order, since a model without a tier is driven
|
||||
by name just as well off Codex's bundled catalog; an entry too large for the bytes left never
|
||||
costs the smaller ones after it their place."""
|
||||
survival: Final = sorted(range(len(entries)), key=lambda position: (not entries[position].service_tiers, position))
|
||||
fits: Final = _fits(tuple(serialized[position] for position in survival), byte_limit)
|
||||
fits: Final = _fits(tuple(serialized[position] for position in survival), budget)
|
||||
return frozenset(position for position, fit in zip(survival, fits, strict=True) if fit)
|
||||
|
||||
|
||||
def _entries(
|
||||
drivable: Sequence[CodexCatalogRow],
|
||||
stock: Mapping[str, CodexStockModel] | None,
|
||||
instructions: str | None,
|
||||
first_priority: int,
|
||||
alias_defaults: CodexAliasDefaults,
|
||||
client_version: str | None,
|
||||
) -> tuple[CodexCatalogEntry, ...]:
|
||||
known: Final = bundled_codex_models() if stock is None else stock
|
||||
prompt: Final = codex_base_instructions() if instructions is None else instructions
|
||||
served: Final = frozenset(row.id for row in drivable)
|
||||
return tuple(
|
||||
_entry(first_priority + index, row, known, served, prompt, alias_defaults, client_version)
|
||||
for index, row in enumerate(drivable)
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CodexCatalogBody:
|
||||
"""A `ModelsResponse` body, the ids it lists, and the chat-capable ids the byte limit left out."""
|
||||
|
|
@ -495,15 +523,12 @@ def codex_models_response_json(
|
|||
requesting Codex's version, which decides the reasoning levels a fallback entry may offer; None counts
|
||||
as a version too old for custom levels.
|
||||
"""
|
||||
known: Final = bundled_codex_models() if stock is None else stock
|
||||
prompt: Final = codex_base_instructions() if instructions is None else instructions
|
||||
drivable: Final = tuple(row for row in rows if _codex_can_drive(row.id, row.mode))
|
||||
served: Final = frozenset(row.id for row in drivable)
|
||||
entries: Final = tuple(
|
||||
_entry(index, row, known, served, prompt, alias_defaults, client_version) for index, row in enumerate(drivable)
|
||||
)
|
||||
entries: Final = _entries(drivable, stock, instructions, 0, alias_defaults, client_version)
|
||||
serialized: Final = tuple(_serialized(entry) for entry in entries)
|
||||
kept: Final = _kept_positions(entries, serialized, byte_limit)
|
||||
kept: Final = _kept_positions(
|
||||
entries, serialized, None if byte_limit is None else byte_limit - len(_BODY_PREFIX) - len(_BODY_SUFFIX) + 1
|
||||
)
|
||||
return CodexCatalogBody(
|
||||
json=f"{_BODY_PREFIX}{','.join(serialized[position] for position in sorted(kept))}{_BODY_SUFFIX}",
|
||||
listed=tuple(row.id for position, row in enumerate(drivable) if position in kept),
|
||||
|
|
@ -511,6 +536,74 @@ def codex_models_response_json(
|
|||
)
|
||||
|
||||
|
||||
def _priority(model: Mapping[str, object]) -> int:
|
||||
priority: Final = model.get("priority")
|
||||
return priority if isinstance(priority, int) else -1
|
||||
|
||||
|
||||
def _compact_json(value: object) -> str:
|
||||
return json.dumps(value, separators=(",", ":"))
|
||||
|
||||
|
||||
def _entry_json(entry: CodexCatalogEntry) -> Mapping[str, object]:
|
||||
return _JSON_OBJECT.validate_python(entry.model_dump(mode="json", exclude_unset=isinstance(entry, CodexStockModel)))
|
||||
|
||||
|
||||
def merged_codex_catalog(
|
||||
upstream_body: bytes,
|
||||
rows: Sequence[CodexCatalogRow],
|
||||
*,
|
||||
stock: Mapping[str, CodexStockModel] | None = None,
|
||||
instructions: str | None = None,
|
||||
byte_limit: int = CODEX_CATALOG_BYTE_LIMIT,
|
||||
client_version: str | None = None,
|
||||
) -> str | None:
|
||||
try:
|
||||
top: Final = _JSON_OBJECT.validate_json(upstream_body)
|
||||
upstream: Final = _JSON_OBJECTS.validate_python(top.get("models"))
|
||||
except ValidationError:
|
||||
return None
|
||||
slugs: Final = frozenset(model.get("slug") for model in upstream)
|
||||
added: Final = tuple(row for row in rows if _codex_can_drive(row.id, row.mode) and row.id not in slugs)
|
||||
first_priority: Final = max((_priority(model) for model in upstream), default=-1) + 1
|
||||
entries: Final = _entries(added, stock, instructions, first_priority, _NATIVE_ALIAS_DEFAULTS, client_version)
|
||||
added_models: Final = tuple(_entry_json(entry) for entry in entries)
|
||||
serialized: Final = tuple(_compact_json(model) for model in added_models)
|
||||
budget: Final = byte_limit - len(_compact_json({**top, "models": upstream}))
|
||||
kept: Final = tuple(sorted(_kept_positions(entries, serialized, budget)))
|
||||
if len(kept) < len(entries):
|
||||
verbose_proxy_logger.warning(
|
||||
"Codex native model catalog left out %d of %d LiteLLM models to stay within Codex's %d byte limit",
|
||||
len(entries) - len(kept),
|
||||
len(entries),
|
||||
byte_limit,
|
||||
)
|
||||
if not kept:
|
||||
return None
|
||||
return _compact_json({**top, "models": [*upstream, *(added_models[position] for position in kept)]})
|
||||
|
||||
|
||||
def _canonical_value(value: object) -> object:
|
||||
return value.model_dump(mode="json") if isinstance(value, BaseModel) else repr(value)
|
||||
|
||||
|
||||
def codex_catalog_etag(upstream_etag: str, rows: Sequence[CodexCatalogRow]) -> str:
|
||||
canonical: Final = json.dumps(
|
||||
[asdict(row) for row in rows], sort_keys=True, separators=(",", ":"), default=_canonical_value
|
||||
)
|
||||
digest: Final = hashlib.sha256(json.dumps([upstream_etag, canonical]).encode()).hexdigest()
|
||||
return f'"{digest}"'
|
||||
|
||||
|
||||
def upstream_codex_slugs(upstream_body: bytes) -> frozenset[str] | None:
|
||||
try:
|
||||
upstream: Final = _JSON_OBJECTS.validate_python(_JSON_OBJECT.validate_json(upstream_body).get("models"))
|
||||
except ValidationError:
|
||||
return None
|
||||
slugs: Final = (model.get("slug") for model in upstream)
|
||||
return frozenset(slug for slug in slugs if isinstance(slug, str))
|
||||
|
||||
|
||||
def _group_reasoning_efforts(
|
||||
row: ModelInfoResponse, lookup_id: str, registry: RouterRegistry | None, team_id: str | None
|
||||
) -> tuple[str, ...] | None:
|
||||
|
|
@ -520,6 +613,11 @@ def _group_reasoning_efforts(
|
|||
return group.supported_reasoning_efforts if group is not None else None
|
||||
|
||||
|
||||
def _shared_pinned_effort(pinned: tuple[object, ...]) -> str | None:
|
||||
first: Final = next(iter(pinned), None)
|
||||
return first if isinstance(first, str) and all(effort == first for effort in pinned) else None
|
||||
|
||||
|
||||
def deployment_capabilities(model_info: Mapping[str, object] | None, model_id: str) -> CodexDeploymentCapabilities:
|
||||
"""One deployment's model info, merged with the cost map, as the capabilities a Codex entry describes.
|
||||
Model info that does not validate declares nothing, logged, so one bad value never fails the listing."""
|
||||
|
|
@ -562,6 +660,11 @@ def _catalog_row(
|
|||
registry.resolve(lookup_id, kind="upstream_provider", team_id=team_id) if registry is not None else None
|
||||
),
|
||||
display_name=registry.group(label_id, field="display_name") if registry is not None else None,
|
||||
reasoning_effort=(
|
||||
_shared_pinned_effort(registry.group(lookup_id, field="pinned_reasoning_efforts", team_id=team_id))
|
||||
if registry is not None
|
||||
else None
|
||||
),
|
||||
service_tiers=registry.group(lookup_id, field="service_tiers", team_id=team_id) if registry is not None else (),
|
||||
supported_reasoning_efforts=_group_reasoning_efforts(row, lookup_id, registry, team_id),
|
||||
search_tool_support=(
|
||||
|
|
@ -619,3 +722,21 @@ def codex_model_list_body(
|
|||
", ".join(body.left_out),
|
||||
)
|
||||
return body.json
|
||||
|
||||
|
||||
def codex_entry_names_default_reasoning_level(
|
||||
model_id: str, llm_router: Router, team_id: str | None, client_version: str | None
|
||||
) -> bool:
|
||||
"""Whether the native catalog entry for `model_id` names a `default_reasoning_level`. Codex's model picker
|
||||
sends reasoning effort `none` for a model whose entry names none, while showing that choice as
|
||||
"default"."""
|
||||
listed: Final[ModelInfoResponse] = {"id": model_id, "object": "model", "created": 0, "owned_by": ""}
|
||||
(entry,) = _entries(
|
||||
(_catalog_row(listed, model_id, model_id, llm_router, team_id),),
|
||||
None,
|
||||
"",
|
||||
0,
|
||||
_NATIVE_ALIAS_DEFAULTS,
|
||||
client_version,
|
||||
)
|
||||
return _entry_json(entry).get("default_reasoning_level") is not None
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ from types import NoneType
|
|||
from typing import Annotated, Final, Literal, TypeAlias
|
||||
|
||||
from packaging.version import Version
|
||||
from pydantic import ConfigDict, Field, StringConstraints
|
||||
from pydantic import ConfigDict, Field, StringConstraints, field_serializer
|
||||
|
||||
from litellm.litellm_core_utils.reasoning_effort_strength import nearest_declared_reasoning_effort
|
||||
from litellm.types.llms.base_model import LiteLLMBaseModel
|
||||
|
|
@ -117,6 +117,10 @@ class CodexDeploymentCapabilities(LiteLLMBaseModel):
|
|||
prices_tokens: bool = False
|
||||
priced_service_tier_keys: frozenset[str] = frozenset()
|
||||
|
||||
@field_serializer("priced_service_tier_keys")
|
||||
def _sorted_priced_service_tier_keys(self, keys: frozenset[str]) -> tuple[str, ...]:
|
||||
return tuple(sorted(keys))
|
||||
|
||||
|
||||
@cache
|
||||
def tier_price_keys() -> tuple[str, ...]:
|
||||
|
|
|
|||
|
|
@ -27,7 +27,10 @@ from pydantic import TypeAdapter
|
|||
from typing_extensions import TypedDict, Unpack
|
||||
|
||||
from litellm._logging import verbose_proxy_logger
|
||||
from litellm.completion_extras.litellm_responses_transformation.transformation import replayed_tool_search_tool_names
|
||||
from litellm.completion_extras.litellm_responses_transformation.transformation import (
|
||||
nested_tool_member_names,
|
||||
replayed_tool_search_tool_names,
|
||||
)
|
||||
from litellm.integrations.custom_guardrail import (
|
||||
CustomGuardrail,
|
||||
log_guardrail_information,
|
||||
|
|
@ -47,6 +50,14 @@ class _CustomGuardrailOptions(TypedDict, total=False, extra_items=object):
|
|||
"""Base-class constructor options this guardrail forwards untouched to CustomGuardrail."""
|
||||
|
||||
|
||||
def _function_tool_name(tool: object) -> str | None:
|
||||
match tool:
|
||||
case {"function": {"name": str(name)}}:
|
||||
return name
|
||||
case _:
|
||||
return None
|
||||
|
||||
|
||||
def _attribute(source: object, name: str) -> object:
|
||||
return getattr(source, name, None)
|
||||
|
||||
|
|
@ -140,17 +151,18 @@ class ToolPolicyGuardrail(CustomGuardrail):
|
|||
"""
|
||||
if input_type == "request":
|
||||
tools: Final = inputs.get("tools") or []
|
||||
function_tool_names: Final = [
|
||||
t["function"]["name"]
|
||||
for t in tools
|
||||
if isinstance(t, dict) and isinstance(t.get("function"), dict) and t["function"].get("name")
|
||||
]
|
||||
function_tool_names: Final = [name for name in map(_function_tool_name, tools) if name]
|
||||
route: Final = _get_request_route_from_data(request_data)
|
||||
replayed_tool_names: Final = replayed_tool_search_tool_names(_REQUEST_DATA.validate_python(request_data))
|
||||
tool_names = (
|
||||
extract_request_tool_names(route, request_data)
|
||||
if route and not function_tool_names
|
||||
else [*function_tool_names, *replayed_tool_names]
|
||||
request: Final = _REQUEST_DATA.validate_python(request_data)
|
||||
tool_names = list(
|
||||
dict.fromkeys(
|
||||
(
|
||||
*function_tool_names,
|
||||
*(extract_request_tool_names(route, request_data) if route else ()),
|
||||
*replayed_tool_search_tool_names(request),
|
||||
*nested_tool_member_names(request),
|
||||
)
|
||||
)
|
||||
)
|
||||
else:
|
||||
tool_calls: Final = inputs.get("tool_calls") or []
|
||||
|
|
|
|||
|
|
@ -1,15 +1,94 @@
|
|||
import asyncio
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import re
|
||||
from collections.abc import Iterator, Mapping
|
||||
from dataclasses import dataclass
|
||||
from types import MappingProxyType
|
||||
from typing import Annotated, Final, Literal
|
||||
from typing import TYPE_CHECKING, Annotated, Final, Literal, Protocol
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import TypeAdapter, ValidationError
|
||||
from starlette.types import Message
|
||||
|
||||
import litellm
|
||||
from litellm._internal_context import with_service_target
|
||||
from litellm._logging import verbose_proxy_logger
|
||||
from litellm.exceptions import BadRequestError
|
||||
from litellm.litellm_core_utils.core_helpers import drops_unsupported_params, normalize_drop_params
|
||||
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
|
||||
from litellm.llms.chatgpt.common_utils import CHATGPT_API_BASE
|
||||
from litellm.proxy._types import UserAPIKeyAuth
|
||||
from litellm.llms.custom_httpx.http_handler import (
|
||||
get_async_httpx_client, # pyright: ignore[reportUnknownVariableType] # shared client factory has untyped params
|
||||
)
|
||||
from litellm.proxy._types import LitellmUserRoles, UserAPIKeyAuth
|
||||
from litellm.proxy.auth.auth_checks import check_model_access_helper, check_tools_allowlist
|
||||
from litellm.proxy.auth.handle_jwt import JWTAuthManager
|
||||
from litellm.proxy.auth.route_checks import RouteChecks
|
||||
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
|
||||
from litellm.proxy.common_utils.codex_model_catalog import (
|
||||
CodexCatalogRow,
|
||||
bundled_codex_models,
|
||||
codex_catalog_etag,
|
||||
codex_entry_names_default_reasoning_level,
|
||||
merged_codex_catalog,
|
||||
upstream_codex_slugs,
|
||||
)
|
||||
from litellm.proxy.common_utils.http_parsing_utils import (
|
||||
get_client_requested_model,
|
||||
read_request_body, # pyright: ignore[reportUnknownVariableType] # returns a bare dict, validated at the call
|
||||
)
|
||||
from litellm.proxy.litellm_pre_call_utils import should_auto_drop_params_for_agentic_cli
|
||||
from litellm.proxy.pass_through_endpoints.pass_through_endpoints import pass_through_request
|
||||
from litellm.proxy.response_api_endpoints.endpoints import responses_api
|
||||
from litellm.responses.additional_tools import with_additional_tools_hoisted
|
||||
from litellm.responses.litellm_completion_transformation.reasoning_items import is_other_provider_reasoning_item
|
||||
from litellm.responses.utils import ResponsesAPIRequestUtils
|
||||
from litellm.types.llms.custom_http import httpxSpecialProvider
|
||||
from litellm.types.router import Deployment
|
||||
from litellm.types.utils import LlmProviders
|
||||
from litellm.utils import ProviderConfigManager, supports_none_reasoning_effort
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from litellm.router import Router
|
||||
|
||||
router: Final = APIRouter()
|
||||
_CHATGPT_PROVIDER: Final = "chatgpt"
|
||||
_CHATGPT_CREDENTIAL_HEADERS: Final = frozenset({b"authorization", b"chatgpt-account-id"})
|
||||
_MERGED_CATALOG_DROPPED_HEADERS: Final = frozenset({"etag", "content-length", "content-type"})
|
||||
_ETAG_HEADER: Final = "etag"
|
||||
_MODELS_ETAG_HEADER: Final = "x-models-etag"
|
||||
_ENCRYPTED_CONTENT_AFFINITY_FLAG: Final = "encrypted_content_affinity_enabled"
|
||||
_LITELLM_RESPONSES_PATH: Final = "/v1/responses"
|
||||
_CODEX_NO_DEFAULT_EFFORT: Final = "none"
|
||||
_NONE_EFFORT_SUPPORT_FLAG: Final = "supports_none_reasoning_effort"
|
||||
_JSON_OBJECT: Final = TypeAdapter(dict[str, object])
|
||||
_INPUT_ITEMS: Final = TypeAdapter(tuple[object, ...])
|
||||
_CACHED_SLUGS: Final = TypeAdapter(frozenset[str])
|
||||
_CACHED_MODELS_ETAG: Final = TypeAdapter(dict[Literal["etag"], str])
|
||||
_MODEL_NAMES: Final = TypeAdapter(list[str])
|
||||
_HEADER_DERIVED_STATE: Final = frozenset({"_cached_headers"})
|
||||
_ACCOUNT_CATALOG_HEADERS: Final = frozenset({"authorization", "chatgpt-account-id", "originator", "user-agent"})
|
||||
_CODEX_USER_AGENT_PRODUCT: Final = re.compile(r"([^/]+)/(\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.+-]+)?)")
|
||||
_ACCOUNT_CATALOG_TIMEOUT_SECONDS: Final = 5.0
|
||||
_SHARED_CACHE_TTL_SECONDS: Final = 3600
|
||||
_ACCOUNT_SLUGS_UNAVAILABLE: Final = "unavailable"
|
||||
_ACCOUNT_SLUGS_UNAVAILABLE_TTL_SECONDS: Final = 60
|
||||
_ACCOUNT_SLUGS_CACHE_PREFIX: Final = "codex_account_slugs:"
|
||||
_ACCOUNT_SLUGS_TARGET: Final = "codex_account_slugs"
|
||||
_ACCOUNT_SLUGS_FETCHES_CACHE_PREFIX: Final = "codex_account_slugs_fetches:"
|
||||
_ACCOUNT_SLUGS_FETCHES_PER_CALLER: Final = 5
|
||||
_ACCOUNT_SLUGS_FETCHES_WINDOW_SECONDS: Final = 60
|
||||
_MODELS_ETAG_CACHE_PREFIX: Final = "codex_models_etag:"
|
||||
_MODELS_ETAG_TTL_SECONDS: Final = 300
|
||||
_OPENAI_API_PROVIDERS: Final = frozenset({LlmProviders.OPENAI.value, LlmProviders.AZURE.value})
|
||||
_CODEX_TOOL_PROVIDERS: Final = frozenset(
|
||||
{*_OPENAI_API_PROVIDERS, LlmProviders.FIREWORKS_AI.value, LlmProviders.BEDROCK_MANTLE.value}
|
||||
)
|
||||
_CHAT_COMPLETIONS_BRIDGE: Final = "use_chat_completions_api"
|
||||
_HEADERS: Final = frozenset(
|
||||
{
|
||||
"authorization",
|
||||
|
|
@ -29,10 +108,637 @@ _HEADERS: Final = frozenset(
|
|||
)
|
||||
|
||||
|
||||
@router.get("/backend-api/codex/{endpoint}", tags=["ChatGPT Pass-through", "pass-through"])
|
||||
def _is_chatgpt(deployment: Deployment) -> bool:
|
||||
return (
|
||||
deployment.litellm_params.custom_llm_provider == _CHATGPT_PROVIDER
|
||||
or deployment.litellm_params.model.partition("/")[0] == _CHATGPT_PROVIDER
|
||||
)
|
||||
|
||||
|
||||
def _serves_by_name(deployment: Deployment, names: frozenset[str]) -> bool:
|
||||
return not _is_chatgpt(deployment) and (
|
||||
deployment.model_name in names or deployment.model_info.team_public_model_name in names
|
||||
)
|
||||
|
||||
|
||||
def _candidate_deployments(model: str, llm_router: "Router", team_id: str | None) -> tuple[Deployment, ...]:
|
||||
registry: Final = llm_router.registry()
|
||||
candidates: Final = tuple(
|
||||
registry.deployment(model_id) for model_id in registry.names(model, kind="route_candidates", team_id=team_id)
|
||||
)
|
||||
return tuple(deployment for deployment in candidates if deployment is not None)
|
||||
|
||||
|
||||
def routes_to_litellm(model: object, llm_router: "Router | None", team_id: str | None) -> bool:
|
||||
if (
|
||||
not isinstance(model, str)
|
||||
or llm_router is None
|
||||
or model in bundled_codex_models()
|
||||
or f"{_CHATGPT_PROVIDER}/{model}" in litellm.model_cost # pyright: ignore[reportUnknownMemberType] # registry is dynamically typed
|
||||
):
|
||||
return False
|
||||
deployments: Final = _candidate_deployments(model, llm_router, team_id)
|
||||
names: Final = frozenset({model, llm_router.registry().resolve(model, kind="alias") or model})
|
||||
return bool(deployments) and all(_serves_by_name(deployment, names) for deployment in deployments)
|
||||
|
||||
|
||||
def _key_and_team_allow(model: str, llm_router: "Router", user_api_key_dict: UserAPIKeyAuth) -> bool:
|
||||
allowed_lists: Final = (
|
||||
_MODEL_NAMES.validate_python(user_api_key_dict.models),
|
||||
_MODEL_NAMES.validate_python(user_api_key_dict.team_models),
|
||||
)
|
||||
return all(
|
||||
check_model_access_helper(model, llm_router, allowed, team_id=user_api_key_dict.team_id)
|
||||
for allowed in allowed_lists
|
||||
)
|
||||
|
||||
|
||||
def native_chatgpt_model(model: str, llm_router: "Router", user_api_key_dict: UserAPIKeyAuth) -> str | None:
|
||||
if model.startswith(f"{_CHATGPT_PROVIDER}/") or _key_and_team_allow(model, llm_router, user_api_key_dict):
|
||||
return None
|
||||
chatgpt_model: Final = f"{_CHATGPT_PROVIDER}/{model}"
|
||||
deployments: Final = _candidate_deployments(chatgpt_model, llm_router, user_api_key_dict.team_id)
|
||||
if not deployments or not all(_is_chatgpt(deployment) for deployment in deployments):
|
||||
return None
|
||||
return chatgpt_model if _key_and_team_allow(chatgpt_model, llm_router, user_api_key_dict) else None
|
||||
|
||||
|
||||
class _SharedCache(Protocol):
|
||||
async def async_get_cache(self, key: str) -> object: ...
|
||||
|
||||
async def async_set_cache(self, key: str, value: object, *, ttl: int) -> None: ...
|
||||
|
||||
async def async_increment_cache(self, key: str, value: float, *, ttl: int) -> float | None: ...
|
||||
|
||||
|
||||
def codex_shared_cache() -> _SharedCache:
|
||||
from litellm.proxy.proxy_server import proxy_logging_obj
|
||||
|
||||
return proxy_logging_obj.internal_usage_cache.dual_cache
|
||||
|
||||
|
||||
def _key_may_call_litellm_responses(user_api_key_dict: UserAPIKeyAuth) -> bool:
|
||||
if not RouteChecks.is_allowed_on_proxy(_LITELLM_RESPONSES_PATH):
|
||||
return False
|
||||
try:
|
||||
_ = RouteChecks.should_call_route(route=_LITELLM_RESPONSES_PATH, valid_token=user_api_key_dict)
|
||||
if user_api_key_dict.user_role != LitellmUserRoles.PROXY_ADMIN:
|
||||
RouteChecks.custom_admin_only_route_check(route=_LITELLM_RESPONSES_PATH)
|
||||
except HTTPException:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
async def _jwt_policy_admits_litellm_responses(
|
||||
gateway_token: str, model: str, request_headers: Mapping[str, str]
|
||||
) -> bool:
|
||||
from litellm.proxy.proxy_server import (
|
||||
general_settings_view,
|
||||
jwt_handler,
|
||||
prisma_client,
|
||||
proxy_logging_obj,
|
||||
user_api_key_cache,
|
||||
)
|
||||
|
||||
try:
|
||||
_ = await JWTAuthManager.auth_builder(
|
||||
api_key=gateway_token,
|
||||
jwt_handler=jwt_handler,
|
||||
request_data={"model": model},
|
||||
general_settings=dict(general_settings_view()),
|
||||
route=_LITELLM_RESPONSES_PATH,
|
||||
prisma_client=prisma_client,
|
||||
user_api_key_cache=user_api_key_cache,
|
||||
parent_otel_span=None,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
request_headers=request_headers,
|
||||
request_method="POST",
|
||||
)
|
||||
except Exception: # noqa: BLE001 # any JWT policy refusal keeps the caller on ChatGPT's models
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _LitellmResponsesAccess:
|
||||
key_admits: bool
|
||||
jwt_token: str | None
|
||||
request_headers: Mapping[str, str]
|
||||
|
||||
async def admits(self, model: str) -> bool:
|
||||
if not self.key_admits:
|
||||
return False
|
||||
if self.jwt_token is None:
|
||||
return True
|
||||
return await _jwt_policy_admits_litellm_responses(self.jwt_token, model, self.request_headers)
|
||||
|
||||
|
||||
def _litellm_responses_access(user_api_key_dict: UserAPIKeyAuth, request: Request) -> _LitellmResponsesAccess:
|
||||
standard_jwt: Final = bool(user_api_key_dict.jwt_claims) and not user_api_key_dict.token
|
||||
return _LitellmResponsesAccess(
|
||||
key_admits=_key_may_call_litellm_responses(user_api_key_dict),
|
||||
jwt_token=_gateway_token(request) if standard_jwt else None,
|
||||
request_headers=request.headers,
|
||||
)
|
||||
|
||||
|
||||
def _account_slugs_cache_key(headers: Mapping[str, str]) -> str | None:
|
||||
account_id: Final = headers.get("chatgpt-account-id")
|
||||
if not account_id:
|
||||
return None
|
||||
caller: Final = json.dumps([account_id, headers.get("authorization", "")])
|
||||
return f"{_ACCOUNT_SLUGS_CACHE_PREFIX}{hashlib.sha256(caller.encode()).hexdigest()}"
|
||||
|
||||
|
||||
def _cached_slugs(cached: object) -> frozenset[str] | None:
|
||||
if cached is None:
|
||||
return None
|
||||
try:
|
||||
return _CACHED_SLUGS.validate_python(cached)
|
||||
except ValidationError:
|
||||
return None
|
||||
|
||||
|
||||
async def _remember_account_slugs(cache_key: str, slugs: frozenset[str] | None, cache: _SharedCache) -> None:
|
||||
if slugs is None:
|
||||
await cache.async_set_cache(cache_key, _ACCOUNT_SLUGS_UNAVAILABLE, ttl=_ACCOUNT_SLUGS_UNAVAILABLE_TTL_SECONDS)
|
||||
return
|
||||
await cache.async_set_cache(cache_key, tuple(sorted(slugs)), ttl=_SHARED_CACHE_TTL_SECONDS)
|
||||
|
||||
|
||||
def _codex_client_version(headers: Mapping[str, str]) -> str | None:
|
||||
product: Final = _CODEX_USER_AGENT_PRODUCT.fullmatch(headers.get("user-agent", "").partition(" ")[0])
|
||||
if product is None or product.group(1) != headers.get("originator"):
|
||||
return None
|
||||
return product.group(2)
|
||||
|
||||
|
||||
async def _fetched_account_slugs(headers: Mapping[str, str], client_version: str) -> frozenset[str] | None:
|
||||
client: Final = get_async_httpx_client(
|
||||
llm_provider=httpxSpecialProvider.PassThroughEndpoint,
|
||||
params={"timeout": _ACCOUNT_CATALOG_TIMEOUT_SECONDS},
|
||||
)
|
||||
try:
|
||||
response: Final = await client.client.get(
|
||||
f"{CHATGPT_API_BASE}/models",
|
||||
params={"client_version": client_version},
|
||||
headers={name: value for name, value in headers.items() if name in _ACCOUNT_CATALOG_HEADERS},
|
||||
)
|
||||
except httpx.HTTPError:
|
||||
return None
|
||||
return upstream_codex_slugs(response.content) if response.status_code == 200 else None
|
||||
|
||||
|
||||
async def _admits_account_slugs_fetch(caller: str, cache: _SharedCache) -> bool:
|
||||
fetches: Final = await cache.async_increment_cache(
|
||||
f"{_ACCOUNT_SLUGS_FETCHES_CACHE_PREFIX}{caller}", 1, ttl=_ACCOUNT_SLUGS_FETCHES_WINDOW_SECONDS
|
||||
)
|
||||
return fetches is not None and fetches <= _ACCOUNT_SLUGS_FETCHES_PER_CALLER
|
||||
|
||||
|
||||
@with_service_target(_ACCOUNT_SLUGS_TARGET)
|
||||
async def _account_slugs(headers: Mapping[str, str], caller: str, cache: _SharedCache) -> frozenset[str] | None:
|
||||
cache_key: Final = _account_slugs_cache_key(headers)
|
||||
if cache_key is None:
|
||||
return None
|
||||
cached_value: Final = await cache.async_get_cache(cache_key)
|
||||
if cached_value == _ACCOUNT_SLUGS_UNAVAILABLE:
|
||||
return None
|
||||
cached: Final = _cached_slugs(cached_value)
|
||||
if cached is not None:
|
||||
return cached
|
||||
client_version: Final = _codex_client_version(headers)
|
||||
if client_version is None or not await _admits_account_slugs_fetch(caller, cache):
|
||||
return None
|
||||
fetched: Final = await _fetched_account_slugs(headers, client_version)
|
||||
await _remember_account_slugs(cache_key, fetched, cache)
|
||||
return fetched
|
||||
|
||||
|
||||
async def _serves_through_litellm(
|
||||
model: object,
|
||||
llm_router: "Router | None",
|
||||
user_api_key_dict: UserAPIKeyAuth,
|
||||
access: _LitellmResponsesAccess,
|
||||
caller: str,
|
||||
headers: Mapping[str, str],
|
||||
cache: _SharedCache,
|
||||
) -> bool:
|
||||
if not isinstance(model, str) or not routes_to_litellm(model, llm_router, user_api_key_dict.team_id):
|
||||
return False
|
||||
if not await access.admits(model):
|
||||
return False
|
||||
account_slugs: Final = await _account_slugs(headers, caller, cache)
|
||||
return account_slugs is not None and model not in account_slugs
|
||||
|
||||
|
||||
def _gateway_token(request: Request) -> str:
|
||||
gateway_parts: Final = request.headers.get("x-litellm-api-key", "").split()
|
||||
return (
|
||||
gateway_parts[-1]
|
||||
if len(gateway_parts) == 1 or (len(gateway_parts) == 2 and gateway_parts[0].lower() == "bearer")
|
||||
else ""
|
||||
)
|
||||
|
||||
|
||||
def _chatgpt_bearer(request: Request) -> str:
|
||||
scheme, _, token = request.headers.get("authorization", "").partition(" ")
|
||||
return token.strip() if scheme.lower() == "bearer" else ""
|
||||
|
||||
|
||||
def _forwarded_headers(request: Request) -> Mapping[str, str]:
|
||||
return MappingProxyType({name: value for name, value in request.headers.items() if name in _HEADERS})
|
||||
|
||||
|
||||
async def native_turn_serves_through_litellm(
|
||||
model: str, llm_router: "Router", request: Request, user_api_key_dict: UserAPIKeyAuth
|
||||
) -> bool:
|
||||
chatgpt_bearer: Final = _chatgpt_bearer(request)
|
||||
gateway_token: Final = _gateway_token(request)
|
||||
if not chatgpt_bearer or not gateway_token or hmac.compare_digest(chatgpt_bearer.encode(), gateway_token.encode()):
|
||||
return False
|
||||
return await _serves_through_litellm(
|
||||
model,
|
||||
llm_router,
|
||||
user_api_key_dict,
|
||||
_litellm_responses_access(user_api_key_dict, request),
|
||||
_models_etag_caller(user_api_key_dict, gateway_token),
|
||||
_forwarded_headers(request),
|
||||
codex_shared_cache(),
|
||||
)
|
||||
|
||||
|
||||
def _json_object(value: object) -> Mapping[str, object]:
|
||||
try:
|
||||
return _JSON_OBJECT.validate_python(value)
|
||||
except ValidationError:
|
||||
return MappingProxyType({})
|
||||
|
||||
|
||||
def _scope_state(request: Request) -> Mapping[str, object]:
|
||||
return _json_object(request.scope.get("state"))
|
||||
|
||||
|
||||
def _as_litellm_responses_request(request: Request, body: bytes, parsed_body: Mapping[str, object]) -> Request:
|
||||
replayed: Final[Iterator[Message]] = iter(({"type": "http.request", "body": body, "more_body": False},))
|
||||
|
||||
async def receive() -> Message:
|
||||
return next(replayed, None) or await request.receive()
|
||||
|
||||
return Request(
|
||||
{
|
||||
**request.scope,
|
||||
"path": _LITELLM_RESPONSES_PATH,
|
||||
"raw_path": _LITELLM_RESPONSES_PATH.encode(),
|
||||
"headers": [
|
||||
(name, value) for name, value in request.headers.raw if name not in _CHATGPT_CREDENTIAL_HEADERS
|
||||
],
|
||||
"parsed_body": (tuple(parsed_body), parsed_body),
|
||||
"state": {
|
||||
name: value for name, value in _scope_state(request).items() if name not in _HEADER_DERIVED_STATE
|
||||
},
|
||||
},
|
||||
receive,
|
||||
)
|
||||
|
||||
|
||||
def _drops_params(body: Mapping[str, object], model: str, llm_router: "Router", team_id: str | None) -> bool:
|
||||
deployments: Final = _candidate_deployments(model, llm_router, team_id)
|
||||
return drops_unsupported_params(body.get("drop_params")) or (
|
||||
bool(deployments)
|
||||
and all(normalize_drop_params(deployment.litellm_params.drop_params) is True for deployment in deployments)
|
||||
)
|
||||
|
||||
|
||||
def _deployment_supports_no_effort(deployment: Deployment) -> bool:
|
||||
configured: Final = _json_object(deployment.model_info.model_extra).get(_NONE_EFFORT_SUPPORT_FLAG)
|
||||
if isinstance(configured, bool):
|
||||
return configured
|
||||
return supports_none_reasoning_effort(
|
||||
model=deployment.litellm_params.model, custom_llm_provider=deployment.litellm_params.custom_llm_provider
|
||||
)
|
||||
|
||||
|
||||
def _supports_no_effort(model: str, llm_router: "Router", team_id: str | None) -> bool:
|
||||
deployments: Final = _candidate_deployments(model, llm_router, team_id)
|
||||
return bool(deployments) and all(_deployment_supports_no_effort(deployment) for deployment in deployments)
|
||||
|
||||
|
||||
def _with_provider_default_reasoning(
|
||||
body: Mapping[str, object], llm_router: "Router | None", team_id: str | None, client_version: str | None
|
||||
) -> Mapping[str, object]:
|
||||
model: Final = body.get("model")
|
||||
try:
|
||||
reasoning: Final = _JSON_OBJECT.validate_python(body.get("reasoning"))
|
||||
except ValidationError:
|
||||
return body
|
||||
if (
|
||||
reasoning.get("effort") != _CODEX_NO_DEFAULT_EFFORT
|
||||
or not isinstance(model, str)
|
||||
or llm_router is None
|
||||
or not _drops_params(body, model, llm_router, team_id)
|
||||
or codex_entry_names_default_reasoning_level(model, llm_router, team_id, client_version)
|
||||
or _supports_no_effort(model, llm_router, team_id)
|
||||
):
|
||||
return body
|
||||
return {**body, "reasoning": {key: value for key, value in reasoning.items() if key != "effort"}}
|
||||
|
||||
|
||||
def _with_wrapped_reasoning(body: Mapping[str, object]) -> Mapping[str, object]:
|
||||
return {
|
||||
**body,
|
||||
"litellm_metadata": {**_json_object(body.get("litellm_metadata")), _ENCRYPTED_CONTENT_AFFINITY_FLAG: True},
|
||||
}
|
||||
|
||||
|
||||
def _with_reasoning_the_targets_can_decrypt(
|
||||
body: Mapping[str, object], model: str, llm_router: "Router", deployments: tuple[Deployment, ...]
|
||||
) -> Mapping[str, object]:
|
||||
try:
|
||||
items: Final = _INPUT_ITEMS.validate_python(body.get("input"))
|
||||
except ValidationError:
|
||||
return body
|
||||
model_group: Final = llm_router.registry().resolve(model, kind="alias") or model
|
||||
kept: Final = llm_router.registry().decryptable_reasoning(model_group, items, deployments)
|
||||
if body.get("store") is True:
|
||||
return {**body, "input": list(kept)}
|
||||
resolvable: Final = (_without_unpersisted_reasoning_reference(item) for item in kept)
|
||||
return {**body, "input": [item for item in resolvable if item is not None]}
|
||||
|
||||
|
||||
def _without_unpersisted_reasoning_reference(item: object) -> object | None:
|
||||
if not _is_unpersisted_reasoning_reference(item):
|
||||
return item
|
||||
return ResponsesAPIRequestUtils.readable_reasoning(_JSON_OBJECT.validate_python(item))
|
||||
|
||||
|
||||
def _model_and_provider(deployment: Deployment) -> tuple[str, str] | None:
|
||||
try:
|
||||
model, provider, _, _ = get_llm_provider(
|
||||
model=deployment.litellm_params.model, custom_llm_provider=deployment.litellm_params.custom_llm_provider
|
||||
)
|
||||
except BadRequestError:
|
||||
return None
|
||||
return model, provider
|
||||
|
||||
|
||||
def _is_openai_api(deployment: Deployment) -> bool:
|
||||
resolved: Final = _model_and_provider(deployment)
|
||||
return resolved is not None and resolved[1] in _OPENAI_API_PROVIDERS
|
||||
|
||||
|
||||
def _takes_codex_tools(deployment: Deployment) -> bool:
|
||||
resolved: Final = _model_and_provider(deployment)
|
||||
if resolved is None:
|
||||
return False
|
||||
model, provider = resolved
|
||||
return (
|
||||
provider in _CODEX_TOOL_PROVIDERS
|
||||
or ProviderConfigManager.get_provider_responses_api_config(
|
||||
provider=provider, model=model, api_base=deployment.litellm_params.api_base
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
|
||||
def _with_tools_the_targets_accept(
|
||||
body: Mapping[str, object], deployments: tuple[Deployment, ...], drop_params: bool
|
||||
) -> Mapping[str, object]:
|
||||
if all(_is_openai_api(deployment) for deployment in deployments):
|
||||
return body
|
||||
if all(_takes_codex_tools(deployment) for deployment in deployments):
|
||||
return with_additional_tools_hoisted(body, drop_params=drop_params)
|
||||
return {**body, _CHAT_COMPLETIONS_BRIDGE: True}
|
||||
|
||||
|
||||
def _drops_params_for_client(
|
||||
body: Mapping[str, object], model: str, llm_router: "Router", team_id: str | None, user_agent: str
|
||||
) -> bool:
|
||||
from litellm.proxy.proxy_server import proxy_config
|
||||
|
||||
return _drops_params(body, model, llm_router, team_id) or should_auto_drop_params_for_agentic_cli(
|
||||
user_agent, "drop_params" in body, proxy_config
|
||||
)
|
||||
|
||||
|
||||
def _litellm_body(
|
||||
body: Mapping[str, object], llm_router: "Router | None", team_id: str | None, headers: Mapping[str, str]
|
||||
) -> Mapping[str, object]:
|
||||
wrapped: Final = _with_wrapped_reasoning(
|
||||
_with_provider_default_reasoning(body, llm_router, team_id, _codex_client_version(headers))
|
||||
)
|
||||
model: Final = body.get("model")
|
||||
if not isinstance(model, str) or llm_router is None:
|
||||
return wrapped
|
||||
deployments: Final = _candidate_deployments(model, llm_router, team_id)
|
||||
return _with_tools_the_targets_accept(
|
||||
_with_reasoning_the_targets_can_decrypt(wrapped, model, llm_router, deployments),
|
||||
deployments,
|
||||
_drops_params_for_client(body, model, llm_router, team_id, headers.get("user-agent", "")),
|
||||
)
|
||||
|
||||
|
||||
def _is_unpersisted_reasoning_reference(item: object) -> bool:
|
||||
try:
|
||||
fields: Final = _JSON_OBJECT.validate_python(item)
|
||||
except ValidationError:
|
||||
return False
|
||||
item_id: Final = fields.get("id")
|
||||
encrypted_content: Final = fields.get("encrypted_content")
|
||||
return (
|
||||
fields.get("type") == "reasoning"
|
||||
and isinstance(item_id, str)
|
||||
and bool(item_id)
|
||||
and not (isinstance(encrypted_content, str) and bool(encrypted_content))
|
||||
)
|
||||
|
||||
|
||||
def _rejected_by_chatgpt(item: object, stored: bool) -> bool:
|
||||
return is_other_provider_reasoning_item(item) or (not stored and _is_unpersisted_reasoning_reference(item))
|
||||
|
||||
|
||||
def _input_chatgpt_accepts(body: Mapping[str, object]) -> tuple[object, ...] | None:
|
||||
try:
|
||||
items: Final = _INPUT_ITEMS.validate_python(body.get("input"))
|
||||
except ValidationError:
|
||||
return None
|
||||
stored: Final = body.get("store") is True
|
||||
kept: Final = tuple(item for item in items if not _rejected_by_chatgpt(item, stored))
|
||||
return kept if len(kept) < len(items) else None
|
||||
|
||||
|
||||
def _chatgpt_body(body: Mapping[str, object], client_model: str | None) -> Mapping[str, object] | None:
|
||||
kept_input: Final = _input_chatgpt_accepts(body)
|
||||
restored_model: Final = client_model is not None and body.get("model") == f"{_CHATGPT_PROVIDER}/{client_model}"
|
||||
if kept_input is None and not restored_model:
|
||||
return None
|
||||
return {
|
||||
**body,
|
||||
**({"input": list(kept_input)} if kept_input is not None else {}),
|
||||
**({"model": client_model} if restored_model else {}),
|
||||
}
|
||||
|
||||
|
||||
async def _litellm_catalog_rows(
|
||||
user_api_key_dict: UserAPIKeyAuth, access: _LitellmResponsesAccess, llm_router: "Router | None"
|
||||
) -> tuple[CodexCatalogRow, ...]:
|
||||
from litellm.proxy.proxy_server import codex_catalog_rows_for_key
|
||||
|
||||
if llm_router is None or not access.key_admits:
|
||||
return ()
|
||||
try:
|
||||
rows: Final = await codex_catalog_rows_for_key(user_api_key_dict)
|
||||
except Exception as error: # noqa: BLE001 # a failed listing must never fail the ChatGPT answer it extends
|
||||
verbose_proxy_logger.warning(
|
||||
"Codex catalog keeps only ChatGPT's models: listing the key's LiteLLM models raised %s",
|
||||
type(error).__name__,
|
||||
)
|
||||
return ()
|
||||
routed: Final = tuple(row for row in rows if routes_to_litellm(row.id, llm_router, user_api_key_dict.team_id))
|
||||
admitted: Final = await asyncio.gather(*(access.admits(row.id) for row in routed))
|
||||
return tuple(row for row, row_admitted in zip(routed, admitted, strict=True) if row_admitted)
|
||||
|
||||
|
||||
def _models_etag_caller(user_api_key_dict: UserAPIKeyAuth, gateway_token: str) -> str:
|
||||
return user_api_key_dict.token or hashlib.sha256(gateway_token.encode()).hexdigest()
|
||||
|
||||
|
||||
def _models_etag_cache_key(caller: str, upstream_etag: str) -> str:
|
||||
return f"{_MODELS_ETAG_CACHE_PREFIX}{hashlib.sha256(json.dumps([caller, upstream_etag]).encode()).hexdigest()}"
|
||||
|
||||
|
||||
async def _remembered_models_etag(
|
||||
caller: str, upstream_etag: str, rows: tuple[CodexCatalogRow, ...], cache: _SharedCache
|
||||
) -> str:
|
||||
etag: Final = codex_catalog_etag(upstream_etag, rows) if rows else upstream_etag
|
||||
await cache.async_set_cache(
|
||||
_models_etag_cache_key(caller, upstream_etag), {"etag": etag}, ttl=_MODELS_ETAG_TTL_SECONDS
|
||||
)
|
||||
return etag
|
||||
|
||||
|
||||
async def _cached_models_etag(caller: str, upstream_etag: str, cache: _SharedCache) -> str | None:
|
||||
try:
|
||||
return _CACHED_MODELS_ETAG.validate_python(
|
||||
await cache.async_get_cache(_models_etag_cache_key(caller, upstream_etag))
|
||||
)["etag"]
|
||||
except ValidationError:
|
||||
return None
|
||||
|
||||
|
||||
async def _key_models_etag(
|
||||
upstream_etag: str,
|
||||
user_api_key_dict: UserAPIKeyAuth,
|
||||
access: _LitellmResponsesAccess,
|
||||
caller: str,
|
||||
llm_router: "Router | None",
|
||||
cache: _SharedCache,
|
||||
) -> str:
|
||||
cached: Final = await _cached_models_etag(caller, upstream_etag, cache)
|
||||
if cached is not None:
|
||||
return cached
|
||||
rows: Final = await _litellm_catalog_rows(user_api_key_dict, access, llm_router)
|
||||
return await _remembered_models_etag(caller, upstream_etag, rows, cache)
|
||||
|
||||
|
||||
def _with_etag(response: Response, content: bytes | str, etag: str | None) -> Response:
|
||||
kept_headers: Final = {
|
||||
name: value for name, value in response.headers.items() if name not in _MERGED_CATALOG_DROPPED_HEADERS
|
||||
}
|
||||
return Response(
|
||||
content=content,
|
||||
status_code=response.status_code,
|
||||
headers={**kept_headers, **({_ETAG_HEADER: etag} if etag is not None else {})},
|
||||
media_type="application/json",
|
||||
)
|
||||
|
||||
|
||||
async def _catalog(
|
||||
request: Request,
|
||||
user_api_key_dict: UserAPIKeyAuth,
|
||||
access: _LitellmResponsesAccess,
|
||||
caller: str,
|
||||
headers: Mapping[str, str],
|
||||
cache: _SharedCache,
|
||||
) -> Response:
|
||||
from litellm.proxy.proxy_server import llm_router
|
||||
|
||||
upstream: Final = await pass_through_request(
|
||||
request=request,
|
||||
target=f"{CHATGPT_API_BASE}/models",
|
||||
custom_headers=dict(headers),
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
custom_llm_provider=_CHATGPT_PROVIDER,
|
||||
upstream_headers_filter=lambda _: headers,
|
||||
)
|
||||
if upstream.status_code != 200 or isinstance(upstream, StreamingResponse):
|
||||
return upstream
|
||||
upstream_body: Final = bytes(upstream.body)
|
||||
slugs: Final = upstream_codex_slugs(upstream_body)
|
||||
if slugs is None:
|
||||
return upstream
|
||||
account_slugs_key: Final = _account_slugs_cache_key(headers)
|
||||
if account_slugs_key is not None:
|
||||
await _remember_account_slugs(account_slugs_key, slugs, cache)
|
||||
rows: Final = await _litellm_catalog_rows(user_api_key_dict, access, llm_router)
|
||||
upstream_etag: Final = upstream.headers.get(_ETAG_HEADER)
|
||||
etag: Final = None if upstream_etag is None else await _remembered_models_etag(caller, upstream_etag, rows, cache)
|
||||
if not rows:
|
||||
return upstream
|
||||
merged: Final = await asyncio.to_thread(
|
||||
merged_codex_catalog, upstream_body, rows, client_version=request.query_params.get("client_version")
|
||||
)
|
||||
return _with_etag(upstream, merged if merged is not None else upstream_body, etag)
|
||||
|
||||
|
||||
async def _responses(
|
||||
request: Request,
|
||||
fastapi_response: Response,
|
||||
user_api_key_dict: UserAPIKeyAuth,
|
||||
access: _LitellmResponsesAccess,
|
||||
caller: str,
|
||||
headers: Mapping[str, str],
|
||||
cache: _SharedCache,
|
||||
) -> object:
|
||||
from litellm.proxy.proxy_server import llm_router
|
||||
|
||||
body: Final = _JSON_OBJECT.validate_python(await read_request_body(request))
|
||||
if await _serves_through_litellm(body.get("model"), llm_router, user_api_key_dict, access, caller, headers, cache):
|
||||
litellm_body: Final = _litellm_body(body, llm_router, user_api_key_dict.team_id, headers)
|
||||
await check_tools_allowlist(
|
||||
request_body=dict(litellm_body),
|
||||
valid_token=user_api_key_dict,
|
||||
team_object=None,
|
||||
route=_LITELLM_RESPONSES_PATH,
|
||||
)
|
||||
return await responses_api(
|
||||
_as_litellm_responses_request(request, json.dumps(litellm_body).encode(), litellm_body),
|
||||
fastapi_response,
|
||||
user_api_key_dict.model_copy(update={"request_route": _LITELLM_RESPONSES_PATH}),
|
||||
)
|
||||
chatgpt_body: Final = _chatgpt_body(body, get_client_requested_model(request))
|
||||
upstream: Final = await pass_through_request(
|
||||
request=request,
|
||||
target=f"{CHATGPT_API_BASE}/responses",
|
||||
custom_headers=dict(headers),
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
custom_body=dict(chatgpt_body) if chatgpt_body is not None else None,
|
||||
custom_llm_provider=_CHATGPT_PROVIDER,
|
||||
upstream_headers_filter=lambda _: headers,
|
||||
)
|
||||
upstream_models_etag: Final = upstream.headers.get(_MODELS_ETAG_HEADER)
|
||||
if upstream_models_etag is not None:
|
||||
upstream.headers[_MODELS_ETAG_HEADER] = await _key_models_etag(
|
||||
upstream_models_etag, user_api_key_dict, access, caller, llm_router, cache
|
||||
)
|
||||
return upstream
|
||||
|
||||
|
||||
@router.get("/backend-api/codex/{endpoint}", tags=["ChatGPT Pass-through", "pass-through"], response_model=None)
|
||||
@router.post(
|
||||
"/backend-api/codex/{endpoint}",
|
||||
tags=["ChatGPT Pass-through", "pass-through"],
|
||||
response_model=None,
|
||||
openapi_extra={
|
||||
"requestBody": {
|
||||
"required": True,
|
||||
|
|
@ -41,7 +747,7 @@ _HEADERS: Final = frozenset(
|
|||
},
|
||||
responses={
|
||||
200: {
|
||||
"description": "Unmodified Codex response or server-sent events",
|
||||
"description": "Codex response, the model catalog with the key's LiteLLM models, or server-sent events",
|
||||
"content": {
|
||||
"application/json": {"schema": {"type": "object", "additionalProperties": True}},
|
||||
"text/event-stream": {"schema": {"type": "string"}},
|
||||
|
|
@ -52,36 +758,29 @@ _HEADERS: Final = frozenset(
|
|||
async def codex_passthrough_route(
|
||||
endpoint: Literal["models", "responses"],
|
||||
request: Request,
|
||||
fastapi_response: Response,
|
||||
user_api_key_dict: Annotated[UserAPIKeyAuth, Depends(user_api_key_auth)],
|
||||
) -> Response:
|
||||
"""Forward a client's ChatGPT subscription with separate LiteLLM authentication."""
|
||||
cache: Annotated[_SharedCache, Depends(codex_shared_cache)],
|
||||
) -> object:
|
||||
"""Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the
|
||||
key's other chat models through LiteLLM."""
|
||||
expected_method: Final = "GET" if endpoint == "models" else "POST"
|
||||
if request.method != expected_method:
|
||||
raise HTTPException(status_code=405, detail="Method not allowed", headers={"Allow": expected_method})
|
||||
|
||||
authorization: Final = request.headers.get("authorization", "")
|
||||
scheme, _, token = authorization.partition(" ")
|
||||
gateway_key: Final = request.headers.get("x-litellm-api-key", "")
|
||||
gateway_parts: Final = gateway_key.split()
|
||||
gateway_token: Final = (
|
||||
gateway_parts[-1]
|
||||
if len(gateway_parts) == 1 or (len(gateway_parts) == 2 and gateway_parts[0].lower() == "bearer")
|
||||
else ""
|
||||
)
|
||||
if scheme.lower() != "bearer" or not token.strip() or not gateway_token:
|
||||
chatgpt_bearer: Final = _chatgpt_bearer(request)
|
||||
gateway_token: Final = _gateway_token(request)
|
||||
if not chatgpt_bearer or not gateway_token:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Send the ChatGPT bearer in Authorization and the LiteLLM key in x-litellm-api-key",
|
||||
)
|
||||
if hmac.compare_digest(token.strip().encode(), gateway_token.encode()):
|
||||
if hmac.compare_digest(chatgpt_bearer.encode(), gateway_token.encode()):
|
||||
raise HTTPException(status_code=401, detail="ChatGPT and LiteLLM credentials must be separate")
|
||||
|
||||
headers: Final = MappingProxyType({name: value for name, value in request.headers.items() if name in _HEADERS})
|
||||
return await pass_through_request(
|
||||
request=request,
|
||||
target=f"{CHATGPT_API_BASE}/{endpoint}",
|
||||
custom_headers=dict(headers),
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
custom_llm_provider="chatgpt",
|
||||
upstream_headers_filter=lambda _: headers,
|
||||
)
|
||||
headers: Final = _forwarded_headers(request)
|
||||
caller: Final = _models_etag_caller(user_api_key_dict, gateway_token)
|
||||
access: Final = _litellm_responses_access(user_api_key_dict, request)
|
||||
if endpoint == "models":
|
||||
return await _catalog(request, user_api_key_dict, access, caller, headers, cache)
|
||||
return await _responses(request, fastapi_response, user_api_key_dict, access, caller, headers, cache)
|
||||
|
|
|
|||
|
|
@ -236,6 +236,7 @@ if TYPE_CHECKING:
|
|||
|
||||
from litellm.integrations.opentelemetry import OpenTelemetry
|
||||
from litellm.proxy.health_check_utils.shared_health_check_manager import SharedHealthCheckManager
|
||||
from litellm.types.proxy.model_listing import ModelInfoResponse
|
||||
|
||||
Span = _Span | Any
|
||||
else:
|
||||
|
|
@ -520,7 +521,7 @@ from litellm.proxy.common_utils.auth_cache_invalidation_pubsub import (
|
|||
publish_auth_cache_invalidation,
|
||||
)
|
||||
from litellm.proxy.common_utils.callback_utils import initialize_callbacks_on_proxy
|
||||
from litellm.proxy.common_utils.codex_model_catalog import codex_model_list_body
|
||||
from litellm.proxy.common_utils.codex_model_catalog import CodexCatalogRow, codex_catalog_rows, codex_model_list_body
|
||||
from litellm.proxy.common_utils.config_includes import resolve_include_file_path, resolve_includes
|
||||
from litellm.proxy.common_utils.config_sync_pubsub import ConfigSyncSubscriber
|
||||
from litellm.proxy.common_utils.credential_hydration import decrypted_or_stored
|
||||
|
|
@ -559,6 +560,7 @@ from litellm.proxy.common_utils.http_parsing_utils import (
|
|||
from litellm.proxy.common_utils.load_config_utils import get_config_from_bucket
|
||||
from litellm.proxy.common_utils.model_deprecation import collect_model_deprecations
|
||||
from litellm.proxy.common_utils.model_listing_utils import (
|
||||
CallerAliases,
|
||||
ClaudeCodeRoutingNames,
|
||||
DeployedModelGroups,
|
||||
TeamModelNameTranslator,
|
||||
|
|
@ -12616,6 +12618,116 @@ async def _deployment_hidden_by_listing_callbacks(deployment: Deployment, user_a
|
|||
return listed_name in await _names_hidden_by_listing_callbacks(user_api_key_dict, (listed_name,))
|
||||
|
||||
|
||||
async def _caller_aliases(user_api_key_dict: UserAPIKeyAuth, team_id: str | None) -> CallerAliases:
|
||||
caller_router_settings: Final = await proxy_config.get_hierarchical_router_settings(
|
||||
user_api_key_dict, prisma_client, proxy_logging_obj
|
||||
)
|
||||
return caller_alias_maps(
|
||||
user_api_key_dict.aliases,
|
||||
user_api_key_dict.team_model_aliases,
|
||||
user_api_key_dict.team_id,
|
||||
team_id,
|
||||
caller_router_settings.get("model_group_alias") if isinstance(caller_router_settings, Mapping) else None,
|
||||
)
|
||||
|
||||
|
||||
async def _listing_hidden_names(healthy_only: bool | None) -> frozenset[str]:
|
||||
blocked_names: Final = llm_router.health().fully_blocked_model_names() if llm_router is not None else set[str]()
|
||||
unhealthy_names: Final = await get_hidden_unhealthy_model_names(
|
||||
healthy_only=healthy_only,
|
||||
general_settings=general_settings_view(),
|
||||
llm_router=llm_router,
|
||||
)
|
||||
return frozenset(blocked_names | unhealthy_names)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _KeyModelListing:
|
||||
rows: "tuple[ModelInfoResponse, ...]"
|
||||
entries: tuple[tuple[str, str], ...]
|
||||
labels: tuple[tuple[str, str], ...]
|
||||
|
||||
|
||||
def _listed_model_info(
|
||||
response_id: str, lookup_id: str, include_metadata: bool, fallback_type: str | None
|
||||
) -> "ModelInfoResponse":
|
||||
from litellm.proxy.utils import create_model_info_response
|
||||
|
||||
model_info: Final = create_model_info_response(
|
||||
model_id=lookup_id,
|
||||
provider="openai",
|
||||
include_metadata=include_metadata,
|
||||
fallback_type=fallback_type,
|
||||
llm_router=llm_router,
|
||||
)
|
||||
model_info["id"] = response_id
|
||||
return model_info
|
||||
|
||||
|
||||
async def _key_model_listing(
|
||||
user_api_key_dict: UserAPIKeyAuth,
|
||||
caller_aliases: CallerAliases,
|
||||
hidden_names: frozenset[str],
|
||||
*,
|
||||
team_id: str | None = None,
|
||||
include_model_access_groups: bool = False,
|
||||
only_model_access_groups: bool = False,
|
||||
include_wildcard_routes: bool = False,
|
||||
include_metadata: bool = False,
|
||||
fallback_type: str | None = None,
|
||||
offered_model_names: Sequence[str] = (),
|
||||
) -> _KeyModelListing:
|
||||
from litellm.proxy.utils import get_available_models_for_user
|
||||
|
||||
available: Final = await get_available_models_for_user(
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
llm_router=llm_router,
|
||||
general_settings=general_settings,
|
||||
user_model=user_model,
|
||||
prisma_client=prisma_client,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
team_id=team_id,
|
||||
include_model_access_groups=include_model_access_groups,
|
||||
only_model_access_groups=only_model_access_groups,
|
||||
return_wildcard_routes=include_wildcard_routes,
|
||||
user_api_key_cache=user_api_key_cache,
|
||||
)
|
||||
names: Final = (*available, *(name for name in offered_model_names if name not in available))
|
||||
excluded: Final = hidden_names | undiscoverable_model_names(
|
||||
names, llm_router, user_api_key_dict, team_id or user_api_key_dict.team_id
|
||||
)
|
||||
listed_entries: Final = await _entries_kept_by_listing_callbacks(
|
||||
TeamModelNameTranslator.listing_entries(
|
||||
[name for name in names if name not in excluded], llm_router, general_settings_view()
|
||||
),
|
||||
user_api_key_dict,
|
||||
)
|
||||
entries: Final = alias_listing_entries(
|
||||
listed_entries,
|
||||
caller_aliases,
|
||||
_describable_model_groups(listed_entries, hidden_names, user_api_key_dict, team_id),
|
||||
)
|
||||
return _KeyModelListing(
|
||||
rows=tuple(
|
||||
_listed_model_info(response_id, lookup_id, include_metadata, fallback_type)
|
||||
for response_id, lookup_id in entries
|
||||
),
|
||||
entries=entries,
|
||||
labels=label_entries(listed_entries, entries),
|
||||
)
|
||||
|
||||
|
||||
async def codex_catalog_rows_for_key(user_api_key_dict: UserAPIKeyAuth) -> tuple[CodexCatalogRow, ...]:
|
||||
listing: Final = await _key_model_listing(
|
||||
user_api_key_dict,
|
||||
await _caller_aliases(user_api_key_dict, None),
|
||||
await _listing_hidden_names(None),
|
||||
)
|
||||
return await asyncio.to_thread(
|
||||
codex_catalog_rows, listing.rows, listing.entries, llm_router, user_api_key_dict.team_id, listing.labels
|
||||
)
|
||||
|
||||
|
||||
@router.get("/v1/models", dependencies=[Depends(user_api_key_auth)], tags=["model management"])
|
||||
@router.get(
|
||||
"/models", dependencies=[Depends(user_api_key_auth)], tags=["model management"]
|
||||
|
|
@ -12683,10 +12795,7 @@ async def model_list(
|
|||
from litellm.proxy.management_endpoints.common_utils import (
|
||||
user_has_admin_privileges,
|
||||
)
|
||||
from litellm.proxy.utils import (
|
||||
create_model_info_response,
|
||||
get_available_models_for_user,
|
||||
)
|
||||
from litellm.proxy.utils import create_model_info_response
|
||||
from litellm.types.proxy.model_listing import ModelInfoResponse
|
||||
|
||||
http_request: Final = cast(Request | None, request) # cast-ok: in-process callers pass no request
|
||||
|
|
@ -12694,16 +12803,7 @@ async def model_list(
|
|||
http_request is not None and http_request.headers.get("anthropic-version") is not None
|
||||
)
|
||||
client_headers: Final[Mapping[str, str]] = http_request.headers if http_request is not None else _EMPTY_HEADERS
|
||||
caller_router_settings: Final = await proxy_config.get_hierarchical_router_settings(
|
||||
user_api_key_dict, prisma_client, proxy_logging_obj
|
||||
)
|
||||
caller_aliases: Final = caller_alias_maps(
|
||||
user_api_key_dict.aliases,
|
||||
user_api_key_dict.team_model_aliases,
|
||||
user_api_key_dict.team_id,
|
||||
team_id,
|
||||
caller_router_settings.get("model_group_alias") if isinstance(caller_router_settings, Mapping) else None,
|
||||
)
|
||||
caller_aliases: Final = await _caller_aliases(user_api_key_dict, team_id)
|
||||
routing_names: Final = ClaudeCodeRoutingNames(
|
||||
llm_router,
|
||||
team_id or user_api_key_dict.team_id,
|
||||
|
|
@ -12729,18 +12829,7 @@ async def model_list(
|
|||
proxy_logging_obj=proxy_logging_obj,
|
||||
)
|
||||
|
||||
# Compute once — used in both branches below to hide paused models from the listing.
|
||||
blocked_names: Final = llm_router.health().fully_blocked_model_names() if llm_router is not None else set[str]()
|
||||
|
||||
# Opt-in: also hide models whose deployments are all unhealthy per background
|
||||
# health checks. Empty when health state is unavailable or stale (fail open).
|
||||
unhealthy_names: Final = await get_hidden_unhealthy_model_names(
|
||||
healthy_only=healthy_only,
|
||||
general_settings=settings,
|
||||
llm_router=llm_router,
|
||||
)
|
||||
|
||||
hidden_names: Final = blocked_names | unhealthy_names
|
||||
hidden_names: Final = await _listing_hidden_names(healthy_only)
|
||||
|
||||
from litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints import typesafe_allowlist_model_names
|
||||
|
||||
|
|
@ -12838,58 +12927,24 @@ async def model_list(
|
|||
object="list",
|
||||
)
|
||||
|
||||
# Otherwise, use the normal behavior (current implementation)
|
||||
# Get available models for the user
|
||||
available_models: Final = await get_available_models_for_user(
|
||||
user_api_key_dict=user_api_key_dict,
|
||||
llm_router=llm_router,
|
||||
general_settings=general_settings,
|
||||
user_model=user_model,
|
||||
prisma_client=prisma_client,
|
||||
proxy_logging_obj=proxy_logging_obj,
|
||||
team_id=team_id,
|
||||
include_model_access_groups=include_model_access_groups or False,
|
||||
only_model_access_groups=only_model_access_groups or False,
|
||||
return_wildcard_routes=include_wildcard_routes,
|
||||
user_api_key_cache=user_api_key_cache,
|
||||
)
|
||||
offers_typesafe_allowlist_names: Final = (
|
||||
asks_for_allowlist_names and team_id is None and user_api_key_has_admin_view(user_api_key_dict)
|
||||
)
|
||||
typesafe_names_to_offer: Final = typesafe_allowlist_model_names() if offers_typesafe_allowlist_names else ()
|
||||
all_models = [*available_models, *(name for name in typesafe_names_to_offer if name not in available_models)]
|
||||
|
||||
undiscoverable_names: Final = undiscoverable_model_names(
|
||||
all_models, llm_router, user_api_key_dict, team_id or user_api_key_dict.team_id
|
||||
)
|
||||
if hidden_names or undiscoverable_names:
|
||||
all_models = [m for m in all_models if m not in hidden_names and m not in undiscoverable_names]
|
||||
|
||||
# Surface the public team name by default; legacy internal keys via flag.
|
||||
# The internal routing key drives the metadata/fallback lookup, while the
|
||||
# public name is what the client sees as the model id.
|
||||
model_data = []
|
||||
listed_entries: Final = await _entries_kept_by_listing_callbacks(
|
||||
TeamModelNameTranslator.listing_entries(all_models, llm_router, settings), user_api_key_dict
|
||||
)
|
||||
entries: Final = alias_listing_entries(
|
||||
listed_entries,
|
||||
key_listing: Final = await _key_model_listing(
|
||||
user_api_key_dict,
|
||||
caller_aliases,
|
||||
_describable_model_groups(listed_entries, hidden_names, user_api_key_dict, team_id),
|
||||
hidden_names,
|
||||
team_id=team_id,
|
||||
include_model_access_groups=include_model_access_groups or False,
|
||||
only_model_access_groups=only_model_access_groups or False,
|
||||
include_wildcard_routes=include_wildcard_routes,
|
||||
include_metadata=include_metadata or False,
|
||||
fallback_type=fallback_type,
|
||||
offered_model_names=typesafe_allowlist_model_names() if offers_typesafe_allowlist_names else (),
|
||||
)
|
||||
labels: Final = label_entries(listed_entries, entries)
|
||||
for response_id, lookup_id in entries:
|
||||
model_info = create_model_info_response(
|
||||
model_id=lookup_id,
|
||||
provider="openai",
|
||||
include_metadata=include_metadata or False,
|
||||
fallback_type=fallback_type,
|
||||
llm_router=llm_router,
|
||||
)
|
||||
model_info["id"] = response_id
|
||||
model_data.append(model_info)
|
||||
|
||||
listing: Final = cast(Sequence[ModelInfoResponse], model_data) # cast-ok: rows built above
|
||||
listing: Final = key_listing.rows
|
||||
entries: Final = key_listing.entries
|
||||
labels: Final = key_listing.labels
|
||||
if client_version is not None:
|
||||
return Response(
|
||||
content=await asyncio.to_thread(
|
||||
|
|
@ -12916,7 +12971,7 @@ async def model_list(
|
|||
)
|
||||
|
||||
return dict(
|
||||
data=model_data,
|
||||
data=list(listing),
|
||||
object="list",
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,21 +1,20 @@
|
|||
import json
|
||||
from collections.abc import Iterator, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from itertools import groupby
|
||||
from itertools import chain, groupby
|
||||
from types import MappingProxyType
|
||||
from typing import Final, cast # noqa: TID251 # validating the openai tool union strips vendor keys from raw tools
|
||||
|
||||
from pydantic import ValidationError
|
||||
from pydantic import TypeAdapter, ValidationError
|
||||
from typing_extensions import TypeIs # noqa: TID251 # narrows a client input item before its type is read
|
||||
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.types.llms.base import LiteLLMBaseModel
|
||||
from litellm.types.llms.openai import ALL_RESPONSES_API_TOOL_PARAMS, ResponseInputParam
|
||||
|
||||
ADDITIONAL_TOOLS_INPUT_ITEM_TYPE: Final = "additional_tools"
|
||||
|
||||
|
||||
class _InputItemType(LiteLLMBaseModel):
|
||||
type: str = ""
|
||||
_NAMESPACE_TOOL_TYPE: Final = "namespace"
|
||||
_ITEMS: Final = TypeAdapter(tuple[object, ...])
|
||||
|
||||
|
||||
class _ToolList(LiteLLMBaseModel):
|
||||
|
|
@ -29,11 +28,14 @@ class HoistedAdditionalTools:
|
|||
hoisted: tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]
|
||||
|
||||
|
||||
def _is_an_input_item(
|
||||
item: object,
|
||||
) -> TypeIs[Mapping[str, object]]: # guard-ok: auth scans every list input, and a pydantic error per item is slow
|
||||
return isinstance(item, Mapping)
|
||||
|
||||
|
||||
def _is_additional_tools_item(item: object) -> bool:
|
||||
try:
|
||||
return _InputItemType.model_validate(item).type == ADDITIONAL_TOOLS_INPUT_ITEM_TYPE
|
||||
except ValidationError:
|
||||
return False
|
||||
return _is_an_input_item(item) and item.get("type") == ADDITIONAL_TOOLS_INPUT_ITEM_TYPE
|
||||
|
||||
|
||||
def _tool_list_of(value: object) -> tuple[Mapping[str, object], ...]:
|
||||
|
|
@ -61,7 +63,7 @@ def _members_of_copies(copies: Sequence[Mapping[str, object]]) -> Iterator[Mappi
|
|||
|
||||
def _merged(copies: Sequence[Mapping[str, object]]) -> Mapping[str, object]:
|
||||
newest: Final = copies[-1]
|
||||
if newest.get("type") != "namespace" or len(copies) == 1:
|
||||
if newest.get("type") != _NAMESPACE_TOOL_TYPE or len(copies) == 1:
|
||||
return newest
|
||||
members: Final = tuple(_members_of_copies(copies))
|
||||
return {**newest, "tools": list(_unique_tools(members))} if members else newest
|
||||
|
|
@ -86,6 +88,11 @@ def _unique_tools(tools: Sequence[Mapping[str, object]]) -> tuple[Mapping[str, o
|
|||
return tuple(_merged(copies_by_identity[identity]) for identity in first_seen_order)
|
||||
|
||||
|
||||
def additional_tools_in_input(input: object) -> tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]:
|
||||
items: Final = _ITEMS.validate_python(input) if isinstance(input, list) else ()
|
||||
return tuple(chain.from_iterable(_tools_of_item(item) for item in items if _is_additional_tools_item(item)))
|
||||
|
||||
|
||||
def hoist_additional_tools(
|
||||
input: str | ResponseInputParam,
|
||||
tools: Sequence[ALL_RESPONSES_API_TOOL_PARAMS] | None,
|
||||
|
|
@ -112,3 +119,26 @@ def hoist_additional_tools(
|
|||
for tool in _unique_tools((*existing, *hoisted))
|
||||
)
|
||||
return HoistedAdditionalTools(input=remaining_input, tools=unique, hoisted=hoisted)
|
||||
|
||||
|
||||
def _top_level_tools(request: Mapping[str, object]) -> tuple[object, ...] | None:
|
||||
try:
|
||||
return _ITEMS.validate_python(request.get("tools") or ())
|
||||
except ValidationError:
|
||||
return None
|
||||
|
||||
|
||||
def with_additional_tools_hoisted(request: Mapping[str, object], *, drop_params: bool) -> Mapping[str, object]:
|
||||
request_input: Final = request.get("input")
|
||||
top_level_tools: Final = _top_level_tools(request)
|
||||
if not isinstance(request_input, list) or top_level_tools is None:
|
||||
return request
|
||||
input_items: Final = cast(ResponseInputParam, request_input) # cast-ok: forwarded as sent, as inference does
|
||||
hoisted: Final = hoist_additional_tools(
|
||||
input_items,
|
||||
cast("tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]", top_level_tools), # cast-ok: forwarded as sent
|
||||
drop_params=drop_params,
|
||||
)
|
||||
if not hoisted.hoisted:
|
||||
return request
|
||||
return {**request, "input": hoisted.input, "tools": list(hoisted.tools)}
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ from litellm.litellm_core_utils.prompt_templates.file_id_utils import (
|
|||
)
|
||||
from litellm.litellm_core_utils.tool_fields import DropPolicy
|
||||
from litellm.llms.base_llm.files.batch_records import responses_batch_body_to_chat_body
|
||||
from litellm.responses.additional_tools import hoist_additional_tools
|
||||
from litellm.responses.additional_tools import with_additional_tools_hoisted
|
||||
from litellm.responses.file_search.emulated_handler import replace_file_search_tools
|
||||
from litellm.responses.litellm_completion_transformation.compaction_items import holds_compaction_summary
|
||||
from litellm.responses.litellm_completion_transformation.transformation import (
|
||||
|
|
@ -20,7 +20,7 @@ from litellm.responses.litellm_completion_transformation.transformation import (
|
|||
LiteLLMCompletionResponsesConfig,
|
||||
)
|
||||
from litellm.responses.main import previous_response_client_ids
|
||||
from litellm.types.llms.openai import ALL_RESPONSES_API_TOOL_PARAMS, ResponseInputParam
|
||||
from litellm.types.llms.openai import ResponseInputParam
|
||||
from litellm.utils import ProviderConfigManager
|
||||
|
||||
_COUNTED_CHAT_PARAMS: Final = (
|
||||
|
|
@ -114,20 +114,6 @@ def with_provider_file_ids(
|
|||
}
|
||||
|
||||
|
||||
def _with_hoisted_additional_tools(responses_request: Mapping[str, object], drop_params: bool) -> Mapping[str, object]:
|
||||
responses_input: Final = responses_request.get("input")
|
||||
if not isinstance(responses_input, list):
|
||||
return responses_request
|
||||
input_items: Final = cast(ResponseInputParam, responses_input) # cast-ok: forwarded as sent, as inference does
|
||||
top_level_tools: Final = cast( # cast-ok: tools are forwarded as sent, as inference does
|
||||
"tuple[ALL_RESPONSES_API_TOOL_PARAMS, ...]", _ITEMS.validate_python(responses_request.get("tools") or ())
|
||||
)
|
||||
hoisted: Final = hoist_additional_tools(input_items, top_level_tools, drop_params=drop_params)
|
||||
if not hoisted.hoisted:
|
||||
return responses_request
|
||||
return {**responses_request, "input": hoisted.input, "tools": list(hoisted.tools)}
|
||||
|
||||
|
||||
def _caller_input(responses_request: Mapping[str, object]) -> ResponseInputParam | None:
|
||||
responses_input: Final = responses_request.get("input")
|
||||
return (
|
||||
|
|
@ -205,7 +191,9 @@ async def chat_count_request(
|
|||
chat_body: Final = await _with_previous_turns(
|
||||
responses_batch_body_to_chat_body(
|
||||
_with_compaction_as_summary_text(
|
||||
_with_emulated_file_search(_with_hoisted_additional_tools(responses_request, policy.drop_params))
|
||||
_with_emulated_file_search(
|
||||
with_additional_tools_hoisted(responses_request, drop_params=policy.drop_params)
|
||||
)
|
||||
),
|
||||
custom_llm_provider,
|
||||
policy.drop_params,
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ from typing import Final
|
|||
from pydantic import BaseModel, TypeAdapter, ValidationError
|
||||
|
||||
REASONING_ITEM_ID_PREFIX: Final = "rs_"
|
||||
WRAPPED_ENCRYPTED_CONTENT_PREFIX: Final = "litellm_enc:"
|
||||
_JSON_LIST: Final = TypeAdapter(list[object])
|
||||
_JSON_OBJECT: Final = TypeAdapter(dict[str, object])
|
||||
|
||||
|
|
@ -58,16 +59,35 @@ def is_minted_reasoning_item_id(item_id: object) -> bool:
|
|||
return parsed.version == 4 and str(parsed) == suffix
|
||||
|
||||
|
||||
def is_litellm_minted_reasoning_item(item: object) -> bool:
|
||||
def _reasoning_fields(item: object) -> Mapping[str, object] | None:
|
||||
try:
|
||||
fields: Final = _JSON_OBJECT.validate_python(
|
||||
item.model_dump(exclude_none=True) if isinstance(item, BaseModel) else item
|
||||
)
|
||||
except ValidationError:
|
||||
return False
|
||||
if fields.get("type") != "reasoning":
|
||||
return False
|
||||
return None
|
||||
return fields if fields.get("type") == "reasoning" else None
|
||||
|
||||
|
||||
def _is_minted(fields: Mapping[str, object]) -> bool:
|
||||
return (
|
||||
is_minted_reasoning_item_id(fields.get("id"))
|
||||
or decode_thinking_blocks(fields.get("encrypted_content")) is not None
|
||||
)
|
||||
|
||||
|
||||
def is_litellm_minted_reasoning_item(item: object) -> bool:
|
||||
fields: Final = _reasoning_fields(item)
|
||||
return fields is not None and _is_minted(fields)
|
||||
|
||||
|
||||
def is_other_provider_reasoning_item(item: object) -> bool:
|
||||
fields: Final = _reasoning_fields(item)
|
||||
if fields is None:
|
||||
return False
|
||||
encrypted_content: Final = fields.get("encrypted_content")
|
||||
return (
|
||||
_is_minted(fields)
|
||||
or bool(fields.get("content"))
|
||||
or (isinstance(encrypted_content, str) and encrypted_content.startswith(WRAPPED_ENCRYPTED_CONTENT_PREFIX))
|
||||
)
|
||||
|
|
|
|||
|
|
@ -89,6 +89,7 @@ from litellm.responses.litellm_completion_transformation.session_handler import
|
|||
ResponsesSessionHandler,
|
||||
StoredTurns,
|
||||
)
|
||||
from litellm.responses.utils import ResponsesAPIRequestUtils
|
||||
from litellm.types.llms.anthropic import AnthropicToolSearchToolRegex, claude_hosted_web_tool_call
|
||||
from litellm.types.llms.base import CachedTokensDetails
|
||||
from litellm.types.llms.openai import (
|
||||
|
|
@ -2556,7 +2557,12 @@ class LiteLLMCompletionResponsesConfig:
|
|||
Returns None for anything this deployment did not write, so a genuinely
|
||||
opaque blob is still skipped rather than forwarded as garbage.
|
||||
"""
|
||||
decoded: Final = decode_thinking_blocks(input_item.get("encrypted_content"))
|
||||
encrypted_content: Final = input_item.get("encrypted_content")
|
||||
decoded: Final = decode_thinking_blocks(
|
||||
ResponsesAPIRequestUtils.unwrap_encrypted_content_with_model_id(encrypted_content)[1]
|
||||
if isinstance(encrypted_content, str)
|
||||
else encrypted_content
|
||||
)
|
||||
if decoded is None:
|
||||
return None
|
||||
return tuple(
|
||||
|
|
|
|||
|
|
@ -50,6 +50,7 @@ from litellm.types.utils import (
|
|||
_PROVIDER_SPECIFIC_HEADER_ADAPTER: Final[
|
||||
TypeAdapter[ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None]
|
||||
] = TypeAdapter(ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None)
|
||||
_REASONING_FIELDS_ONLY_ITS_ORIGIN_RESOLVES: Final = frozenset({"encrypted_content", "id", "status"})
|
||||
|
||||
|
||||
class _UsageExtraFields(ExtraItemsTypedDict, total=False, extra_items=object):
|
||||
|
|
@ -678,7 +679,7 @@ class ResponsesAPIRequestUtils:
|
|||
return
|
||||
items: Final = cast(list[object], request_input) # cast-ok: untyped client json
|
||||
stripped: Final = tuple(
|
||||
ResponsesAPIRequestUtils._without_encrypted_reasoning(reasoning)
|
||||
ResponsesAPIRequestUtils.readable_reasoning(reasoning)
|
||||
if (reasoning := ResponsesAPIRequestUtils._encrypted_reasoning_item(item)) is not None
|
||||
and (should_strip is None or should_strip(reasoning))
|
||||
else item
|
||||
|
|
@ -696,14 +697,14 @@ class ResponsesAPIRequestUtils:
|
|||
return reasoning
|
||||
|
||||
@staticmethod
|
||||
def _without_encrypted_reasoning(reasoning: Mapping[str, object]) -> object | None:
|
||||
def readable_reasoning(reasoning: Mapping[str, object]) -> Mapping[str, object] | None:
|
||||
readable: Final = any(
|
||||
ResponsesAPIRequestUtils._has_readable_text(reasoning.get(key)) for key in ("summary", "content")
|
||||
)
|
||||
if not readable:
|
||||
return None
|
||||
kept: Final[dict[str, object]] = { # mutable-ok: request item rebuilt without the undecryptable keys
|
||||
key: value for key, value in reasoning.items() if key not in ("encrypted_content", "id")
|
||||
key: value for key, value in reasoning.items() if key not in _REASONING_FIELDS_ONLY_ITS_ORIGIN_RESOLVES
|
||||
}
|
||||
return kept
|
||||
|
||||
|
|
|
|||
|
|
@ -158,6 +158,7 @@ GroupField: TypeAlias = Literal[
|
|||
"display_name",
|
||||
"mode",
|
||||
"service_tiers",
|
||||
"pinned_reasoning_efforts",
|
||||
"max_input_tokens",
|
||||
"deployment_model_info",
|
||||
"token_limits",
|
||||
|
|
@ -939,7 +940,7 @@ class RouterRegistry:
|
|||
self,
|
||||
name: str,
|
||||
*,
|
||||
field: Literal["service_tiers"],
|
||||
field: Literal["service_tiers", "pinned_reasoning_efforts"],
|
||||
include_hidden: bool = False,
|
||||
team_id: str | None = None,
|
||||
) -> tuple[object, ...]: ...
|
||||
|
|
@ -1001,6 +1002,7 @@ class RouterRegistry:
|
|||
(were ``get_configured_display_name``, ``get_configured_mode``,
|
||||
``get_configured_token_limits``, ``get_model_listing_info``)
|
||||
- ``service_tiers``: (was ``get_configured_service_tiers``)
|
||||
- ``pinned_reasoning_efforts``: the ``reasoning_effort`` each routable deployment pins
|
||||
- ``max_input_tokens``: each routable deployment's own ``model_info.max_input_tokens``
|
||||
- ``deployment_model_info``: each routable deployment's model info, merged with the pricing map
|
||||
"""
|
||||
|
|
@ -1015,6 +1017,8 @@ class RouterRegistry:
|
|||
return r._core_get_configured_mode(name)
|
||||
if field == "service_tiers":
|
||||
return r._core_get_configured_service_tiers(name, team_id=team_id)
|
||||
if field == "pinned_reasoning_efforts":
|
||||
return r._core_get_pinned_reasoning_efforts(name, team_id=team_id)
|
||||
if field == "max_input_tokens":
|
||||
return r._core_get_configured_max_input_tokens(name, team_id=team_id)
|
||||
if field == "deployment_model_info":
|
||||
|
|
@ -1136,6 +1140,18 @@ class RouterRegistry:
|
|||
return r._core_prompt_caching_pin_cache
|
||||
raise ValueError(f"unknown plugin kind {kind!r}")
|
||||
|
||||
def decryptable_reasoning(
|
||||
self, model_group: str, request_input: Sequence[object], targets: "Sequence[Deployment]"
|
||||
) -> tuple[object, ...]:
|
||||
"""
|
||||
``request_input`` without the encrypted reasoning that ``targets``, the deployments a
|
||||
``model_group`` request may go to, cannot decrypt, keeping each item's readable summary.
|
||||
When encrypted content affinity is on for the group, the router sends the request to the
|
||||
reasoning's origin, so reasoning one target can decrypt stays; otherwise every target must
|
||||
decrypt it (was reading ``optional_callbacks`` for ``EncryptedContentAffinityCheck``).
|
||||
"""
|
||||
return self._router._core_decryptable_reasoning(model_group, request_input, targets)
|
||||
|
||||
|
||||
# Settings a caller may read. Assigning one from outside the router goes through apply(ChangeSet(assign=...)).
|
||||
SETTING_FIELDS: Final[tuple[str, ...]] = (
|
||||
|
|
|
|||
|
|
@ -407,6 +407,28 @@ class OptionalChecks:
|
|||
bound: Final = request.get("litellm_logging_obj") in self._bound_requests
|
||||
return {"success": bound and not sync, "failure": bound, "make_call": make_call}
|
||||
|
||||
def decryptable_reasoning(
|
||||
self, model_group: str, request_input: Sequence[object], targets: "Sequence[Deployment]"
|
||||
) -> tuple[object, ...]:
|
||||
"""``registry().decryptable_reasoning``: one target decrypting is enough only when encrypted
|
||||
content affinity pins ``model_group`` to the reasoning's origin."""
|
||||
target_dicts: Final = tuple(target.model_dump(exclude_none=True) for target in targets)
|
||||
pinned: Final = any(
|
||||
isinstance(callback, EncryptedContentAffinityCheck) and callback.is_enabled_for_model_group(model_group)
|
||||
for callback in self.optional_callbacks or ()
|
||||
)
|
||||
target_groups: Final = (target_dicts,) if pinned else tuple((target,) for target in target_dicts)
|
||||
kept: Final = list(request_input)
|
||||
for group in target_groups:
|
||||
EncryptedContentAffinityCheck.strip_reasoning_the_targets_cannot_decrypt(
|
||||
self._ctx, # pyright: ignore[reportArgumentType] # the core's RoutingContext serves registry()
|
||||
kept,
|
||||
None,
|
||||
group,
|
||||
unmarked_origin=None,
|
||||
)
|
||||
return tuple(kept)
|
||||
|
||||
def routing_callbacks(self) -> Iterable[object]:
|
||||
"""``_routing_callbacks``: the host's callbacks, or after ``observe(request_scoped=True)`` the
|
||||
host's callbacks with the router's own selectors and policies in place of theirs."""
|
||||
|
|
|
|||
|
|
@ -81,6 +81,7 @@ if TYPE_CHECKING:
|
|||
)
|
||||
|
||||
_MODEL_INFO_ADAPTER: Final = TypeAdapter(Mapping[str, object])
|
||||
_LITELLM_PARAMS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
|
||||
|
||||
# The key router-core's merge reads as an opaque value (``router-core/src/model_info.rs``).
|
||||
_OPAQUE_KEY: Final = "__litellm_python_value__"
|
||||
|
|
@ -761,6 +762,20 @@ class HostModelInfo:
|
|||
)
|
||||
return tuple(wire.from_wire(tier) for tier in tiers)
|
||||
|
||||
def _core_get_pinned_reasoning_efforts(self, model_name: str, team_id: str | None = None) -> tuple[object, ...]:
|
||||
"""
|
||||
Return the litellm_params.reasoning_effort each deployment a request
|
||||
from team_id to model_name can route to (see _routable_deployments)
|
||||
pins, unvalidated and in model_list order, None for a deployment that
|
||||
pins none; the caller decides how the deployments combine.
|
||||
|
||||
Returns an empty tuple for wildcard-expanded or unknown names.
|
||||
"""
|
||||
return tuple(
|
||||
_LITELLM_PARAMS_ADAPTER.validate_python(deployment["litellm_params"]).get("reasoning_effort")
|
||||
for deployment in self._routable_deployments(model_name, team_id)
|
||||
)
|
||||
|
||||
def _core_get_configured_max_input_tokens(
|
||||
self, model_name: str, team_id: str | None = None
|
||||
) -> tuple[int | None, ...]:
|
||||
|
|
|
|||
|
|
@ -1510,6 +1510,14 @@ class HostRegistry(HostModelInfo):
|
|||
return self.__getattr__("_core_prompt_caching_pin_cache") # pyright: ignore[reportReturnType] # raises RustCoreUnsupported
|
||||
return self.checks.prompt_caching_pin_cache
|
||||
|
||||
def _core_decryptable_reasoning(
|
||||
self, model_group: str, request_input: Sequence[object], targets: Sequence[Deployment]
|
||||
) -> tuple[object, ...]:
|
||||
"""The input ``targets`` can decrypt (``registry().decryptable_reasoning``)."""
|
||||
if self.checks is None:
|
||||
return self.__getattr__("_core_decryptable_reasoning") # pyright: ignore[reportReturnType] # raises RustCoreUnsupported
|
||||
return self.checks.decryptable_reasoning(model_group, request_input, targets)
|
||||
|
||||
def replay_model_cost_registrations(self) -> None:
|
||||
"""Re-register every served deployment's cost-map entries (``replay_model_cost_registrations``)."""
|
||||
provider_unresolved: Final = self._provider_unresolved_deployments
|
||||
|
|
|
|||
|
|
@ -148,7 +148,7 @@ class EncryptedContentAffinityCheck(CustomLogger):
|
|||
|
||||
return any("encrypted_content_affinity" in checks for checks in model_group_affinity_config.values())
|
||||
|
||||
def _is_enabled_for_model_group(self, model_group: str) -> bool:
|
||||
def is_enabled_for_model_group(self, model_group: str) -> bool:
|
||||
group_checks: Final = self.model_group_affinity_config.get(model_group)
|
||||
return self.enable_global_affinity or (
|
||||
group_checks is not None and "encrypted_content_affinity" in group_checks
|
||||
|
|
@ -530,7 +530,7 @@ class EncryptedContentAffinityCheck(CustomLogger):
|
|||
"""
|
||||
request_kwargs = request_kwargs or {}
|
||||
typed_healthy_deployments: Final = cast(list[dict[str, object]], healthy_deployments)
|
||||
if not self._is_enabled_for_model_group(model):
|
||||
if not self.is_enabled_for_model_group(model):
|
||||
return typed_healthy_deployments
|
||||
typed_request_kwargs: Final = _REQUEST_KWARGS_ADAPTER.validate_python(request_kwargs)
|
||||
target_order: Final = typed_request_kwargs.get("_target_order")
|
||||
|
|
|
|||
|
|
@ -477,6 +477,84 @@ def test_transform_request_forwards_non_string_instructions_and_input_untouched(
|
|||
)
|
||||
|
||||
|
||||
CODEX_CLOCK_NAMESPACE_TOOL: Final = {
|
||||
"type": "namespace",
|
||||
"name": "clock",
|
||||
"description": "Clock tools.",
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"name": "sleep",
|
||||
"description": "Sleep for some seconds.",
|
||||
"parameters": {"type": "object", "properties": {"seconds": {"type": "number"}}, "required": ["seconds"]},
|
||||
}
|
||||
],
|
||||
}
|
||||
CODEX_ADDITIONAL_TOOLS_ITEM: Final = {
|
||||
"type": "additional_tools",
|
||||
"role": "developer",
|
||||
"tools": [CODEX_CLOCK_NAMESPACE_TOOL],
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("instructions", [None, "Answer with exactly one word."], ids=["no-instructions", "instructions"])
|
||||
def test_transform_request_moves_codex_additional_tools_item_into_tools(instructions: str | None) -> None:
|
||||
user_item: Final = {"role": "user", "content": "Sleep for two seconds."}
|
||||
request: Final = FireworksAIResponsesAPIConfig().transform_responses_api_request(
|
||||
model="accounts/fireworks/models/glm-5p3",
|
||||
input=cast(ResponseInputParam, [CODEX_ADDITIONAL_TOOLS_ITEM, user_item]),
|
||||
response_api_optional_request_params={} if instructions is None else {"instructions": instructions},
|
||||
litellm_params=GenericLiteLLMParams(),
|
||||
headers={},
|
||||
)
|
||||
assert tuple(request["tools"]) == (CODEX_CLOCK_NAMESPACE_TOOL,)
|
||||
assert tuple(request["input"]) == (user_item,)
|
||||
assert request.get("instructions") == instructions
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("drop_params", "expected_clock_copies"),
|
||||
[
|
||||
pytest.param(True, 1, id="merged once under drop_params"),
|
||||
pytest.param(None, 2, id="forwarded as sent without drop_params"),
|
||||
],
|
||||
)
|
||||
@pytest.mark.parametrize("tools_as_tuple", [False, True], ids=["tools-list", "tools-tuple"])
|
||||
def test_transform_request_merges_codex_additional_tools_into_existing_tools_only_under_drop_params(
|
||||
drop_params: bool | None, expected_clock_copies: int, tools_as_tuple: bool
|
||||
) -> None:
|
||||
weather_tool: Final = {
|
||||
"type": "function",
|
||||
"name": "get_weather",
|
||||
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
|
||||
}
|
||||
user_item: Final = {"role": "user", "content": "Sleep for two seconds, then check the weather in Paris."}
|
||||
request: Final = FireworksAIResponsesAPIConfig().transform_responses_api_request(
|
||||
model="accounts/fireworks/models/glm-5p3",
|
||||
input=cast(ResponseInputParam, [CODEX_ADDITIONAL_TOOLS_ITEM, user_item, CODEX_ADDITIONAL_TOOLS_ITEM]),
|
||||
response_api_optional_request_params={"tools": (weather_tool,) if tools_as_tuple else [weather_tool]},
|
||||
litellm_params=GenericLiteLLMParams(drop_params=drop_params),
|
||||
headers={},
|
||||
)
|
||||
assert tuple(request["tools"]) == (weather_tool, *(CODEX_CLOCK_NAMESPACE_TOOL,) * expected_clock_copies)
|
||||
assert tuple(request["input"]) == (user_item,)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("drop_params", [True, None], ids=["drop_params", "no-drop_params"])
|
||||
def test_transform_request_leaves_codex_additional_tools_in_input_when_tools_is_malformed(
|
||||
drop_params: bool | None,
|
||||
) -> None:
|
||||
user_item: Final = {"role": "user", "content": "Sleep for two seconds."}
|
||||
request: Final = FireworksAIResponsesAPIConfig().transform_responses_api_request(
|
||||
model="accounts/fireworks/models/glm-5p3",
|
||||
input=cast(ResponseInputParam, [CODEX_ADDITIONAL_TOOLS_ITEM, user_item]),
|
||||
response_api_optional_request_params={"tools": {"type": "function", "name": "get_weather"}},
|
||||
litellm_params=GenericLiteLLMParams(drop_params=drop_params),
|
||||
headers={},
|
||||
)
|
||||
assert tuple(request["input"]) == (CODEX_ADDITIONAL_TOOLS_ITEM, user_item)
|
||||
|
||||
|
||||
def test_responses_call_maps_pydantic_developer_items_and_replays_pydantic_output_items() -> None:
|
||||
client: Final = _mock_http_client(_fireworks_response("accounts/fireworks/models/qwen3p8-2p4t-a95b"))
|
||||
pydantic_input: Final = cast(
|
||||
|
|
|
|||
|
|
@ -2401,6 +2401,37 @@ async def test_deny_by_default_ignores_tools_that_name_no_vector_store(
|
|||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("deny_by_default", [False, True], ids=["legacy", "deny-by-default"])
|
||||
@pytest.mark.parametrize(
|
||||
("key_vector_stores", "denied_by"),
|
||||
[(["KBSTOREA"], _KEY_DENIED), (["KBSTOREB"], None)],
|
||||
ids=["ungranted", "granted"],
|
||||
)
|
||||
async def test_file_search_inside_additional_tools_input_item_needs_a_vector_store_grant(
|
||||
deny_by_default: bool, key_vector_stores: list[str], denied_by: ProxyErrorTypes | None
|
||||
):
|
||||
await _assert_rag_query_outcome(
|
||||
_common_checks_for_rag_query(
|
||||
{"vector_store_deny_by_default": deny_by_default},
|
||||
_virtual_key(object_permission_id="key-permission"),
|
||||
SimpleNamespace(vector_stores=key_vector_stores),
|
||||
request_body={
|
||||
"model": "gpt-4o-mini",
|
||||
"input": [
|
||||
{
|
||||
"type": "additional_tools",
|
||||
"role": "developer",
|
||||
"tools": [{"type": "file_search", "vector_store_ids": ["KBSTOREB"]}],
|
||||
},
|
||||
{"type": "message", "role": "user", "content": "what is in this KB?"},
|
||||
],
|
||||
},
|
||||
),
|
||||
denied_by,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"request_body",
|
||||
|
|
|
|||
|
|
@ -35,6 +35,15 @@ def test_get_team_models_for_all_models_and_team_only_models():
|
|||
assert set(result) == set(combined_models)
|
||||
|
||||
|
||||
def test_get_team_models_lists_models_in_the_order_the_team_set_them() -> None:
|
||||
from litellm.proxy.auth.model_checks import get_team_models
|
||||
|
||||
team_models: Final = ["zeta", "alpha", "mu", "beta", "omega", "kappa", "delta", "all-proxy-models"]
|
||||
result: Final = get_team_models(team_models, ["proxy-b", "proxy-a"], {"grp": ["g1"]}, True)
|
||||
|
||||
assert result == [*team_models, "proxy-b", "proxy-a", "grp", "g1"]
|
||||
|
||||
|
||||
def test_get_team_models_all_proxy_models_includes_access_groups():
|
||||
"""
|
||||
When a team has 'all-proxy-models' and include_model_access_groups=True,
|
||||
|
|
|
|||
|
|
@ -12752,6 +12752,19 @@ def _user_defined_pass_through_endpoint():
|
|||
return endpoint
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_router_settings_model_group_alias_leaves_a_deployed_name_alone_on_provider_pass_through(monkeypatch):
|
||||
router = _alias_router()
|
||||
monkeypatch.setattr(litellm.proxy.proxy_server, "llm_router", router)
|
||||
data = {"model": "claude-sonnet", "messages": [{"role": "user", "content": "hi"}]}
|
||||
route = "/anthropic/v1/messages"
|
||||
request = _alias_request(route, data, path_params={"endpoint": "v1/messages"})
|
||||
token = _alias_token(monkeypatch, "key", {"claude-sonnet": "claude-haiku"}, ["claude-haiku", "claude-sonnet"])
|
||||
await _enforce_alias_access(token, data, route, request, router)
|
||||
assert data["model"] == "claude-sonnet"
|
||||
assert (await request.json())["model"] == "claude-sonnet"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("user_defined", [False, True])
|
||||
async def test_router_settings_model_group_alias_leaves_pass_through_bodies_alone(monkeypatch, user_defined):
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import json
|
||||
import logging
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import replace
|
||||
from types import MappingProxyType
|
||||
|
|
@ -7,6 +8,7 @@ from typing import Final, Literal
|
|||
import pytest
|
||||
|
||||
import litellm
|
||||
from litellm._logging import verbose_proxy_logger
|
||||
from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap
|
||||
from litellm.litellm_core_utils.llm_cost_calc.utils import get_service_tier_cost_key
|
||||
from litellm.proxy.common_utils.codex_model_catalog import (
|
||||
|
|
@ -15,11 +17,14 @@ from litellm.proxy.common_utils.codex_model_catalog import (
|
|||
CodexStockModel,
|
||||
CodexStockUpgrade,
|
||||
bundled_codex_models,
|
||||
codex_catalog_etag,
|
||||
codex_catalog_rows,
|
||||
codex_entry_names_default_reasoning_level,
|
||||
codex_model_list_body,
|
||||
codex_models_response_json,
|
||||
configured_service_tiers,
|
||||
deployment_capabilities,
|
||||
merged_codex_catalog,
|
||||
)
|
||||
from litellm.proxy.common_utils.codex_model_translation import CodexDeploymentCapabilities, CodexServiceTier
|
||||
|
||||
|
|
@ -628,6 +633,15 @@ def test_default_service_tier_survives_only_while_still_offered(configured, expe
|
|||
assert entry["default_service_tier"] == expected
|
||||
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("alias_defaults", "expected"), [("codex", "priority"), ("deployment", None)])
|
||||
def test_alias_with_offered_tiers_takes_codexs_default_tier_only_under_codex_defaults(alias_defaults, expected):
|
||||
(entry,) = _models(
|
||||
_row("team-sol", upstream_model="gpt-6-sol", service_tiers=["priority"]), alias_defaults=alias_defaults
|
||||
)
|
||||
|
||||
assert (_tiers(entry), entry["default_service_tier"]) == ([("priority", "Fast", "2x speed, increased usage")], expected)
|
||||
|
||||
def test_non_chat_and_wildcard_rows_are_left_out_of_the_catalog():
|
||||
body = _body(
|
||||
_row("gpt-6-astra"),
|
||||
|
|
@ -705,6 +719,55 @@ def test_an_entry_too_large_for_the_bytes_left_is_passed_over_and_the_smaller_on
|
|||
assert len(body.json.encode()) <= limit
|
||||
|
||||
|
||||
def test_merged_catalog_keeps_upstream_whole_and_adds_only_the_entries_that_fit_the_byte_limit():
|
||||
upstream = {"models": [{"slug": "upstream-model", "priority": 4, "base_instructions": "u" * 500}]}
|
||||
upstream_only = len(json.dumps(upstream, separators=(",", ":")))
|
||||
rows = tuple(_row(f"model-{index}") for index in range(4))
|
||||
|
||||
def merged(*merged_rows, byte_limit):
|
||||
return merged_codex_catalog(
|
||||
json.dumps(upstream).encode(), merged_rows, stock=_STOCK, instructions=_PROMPT, byte_limit=byte_limit
|
||||
)
|
||||
|
||||
unlimited = json.loads(merged(*rows, byte_limit=upstream_only * 100))["models"]
|
||||
limit = len(merged(*rows[:2], byte_limit=upstream_only * 100).encode())
|
||||
|
||||
body = merged(*rows, byte_limit=limit)
|
||||
|
||||
assert len(body.encode()) <= limit
|
||||
assert json.loads(body)["models"] == unlimited[:3]
|
||||
assert merged(*rows, byte_limit=upstream_only) is None
|
||||
|
||||
|
||||
def test_merged_catalog_gives_an_alias_no_service_tier_or_upgrade_of_codexs_own():
|
||||
upstream = json.dumps({"models": [{"slug": "upstream-model", "priority": 0}]}).encode()
|
||||
rows = (_row("team-sol", upstream_model="openai/gpt-6-sol"), _row("team-55", upstream_model="openai/gpt-5.5"))
|
||||
|
||||
added = json.loads(merged_codex_catalog(upstream, rows, stock=_STOCK, instructions=_PROMPT))["models"][1:]
|
||||
|
||||
assert [
|
||||
(entry["slug"], entry["service_tiers"], entry["default_service_tier"], entry["upgrade"]) for entry in added
|
||||
] == [
|
||||
("team-sol", [], None, None),
|
||||
("team-55", [], None, None),
|
||||
]
|
||||
|
||||
|
||||
def test_merged_catalog_warns_once_with_how_many_models_the_byte_limit_left_out(caplog, monkeypatch):
|
||||
upstream = json.dumps({"models": [{"slug": "upstream-model", "priority": 0}]}).encode()
|
||||
rows = tuple(_row(f"model-{index}") for index in range(4))
|
||||
limit = len(merged_codex_catalog(upstream, rows[:1], stock=_STOCK, instructions=_PROMPT, byte_limit=10**6).encode())
|
||||
monkeypatch.setattr(verbose_proxy_logger, "propagate", True)
|
||||
|
||||
with caplog.at_level(logging.WARNING, logger=verbose_proxy_logger.name):
|
||||
merged_codex_catalog(upstream, rows, stock=_STOCK, instructions=_PROMPT, byte_limit=limit)
|
||||
merged_codex_catalog(upstream, rows[:1], stock=_STOCK, instructions=_PROMPT, byte_limit=limit)
|
||||
|
||||
assert [record.getMessage() for record in caplog.records] == [
|
||||
f"Codex native model catalog left out 3 of 4 LiteLLM models to stay within Codex's {limit} byte limit"
|
||||
]
|
||||
|
||||
|
||||
def test_unlimited_body_keeps_every_entry():
|
||||
rows = tuple(_row(f"model-{index}") for index in range(3))
|
||||
|
||||
|
|
@ -947,6 +1010,38 @@ def test_catalog_rows_without_a_router_carry_only_the_listing():
|
|||
assert codex_catalog_rows(listing, (("plain", "plain"),), None) == (CodexCatalogRow(id="plain", mode="chat"),)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("client_version", [None, "0.165.0"])
|
||||
def test_entry_names_a_default_reasoning_level_exactly_when_the_native_entry_carries_one(client_version):
|
||||
router = litellm.Router(
|
||||
model_list=[
|
||||
{"model_name": "open-weights", "litellm_params": {"model": "fireworks_ai/open-weights"}},
|
||||
{"model_name": "coding-model", "litellm_params": {"model": "openai/gpt-5.5"}},
|
||||
{"model_name": "pinned-model", "litellm_params": {"model": "openai/gpt-5.5", "reasoning_effort": "high"}},
|
||||
{"model_name": "split-pin", "litellm_params": {"model": "openai/gpt-5.5", "reasoning_effort": "high"}},
|
||||
{"model_name": "split-pin", "litellm_params": {"model": "openai/gpt-5.5", "reasoning_effort": "low"}},
|
||||
{
|
||||
"model_name": "max-only",
|
||||
"litellm_params": {"model": "hosted_vllm/qa-local-coder", "api_base": "https://vllm.example/v1"},
|
||||
"model_info": {"supports_reasoning": True, "reasoning_effort_levels": ["max"]},
|
||||
},
|
||||
]
|
||||
)
|
||||
names = ("open-weights", "coding-model", "pinned-model", "split-pin", "max-only")
|
||||
listing = tuple(
|
||||
{"id": name, "object": "model", "created": 0, "owned_by": "openai", "mode": "chat"} for name in names
|
||||
)
|
||||
upstream = json.dumps({"models": [{"slug": "upstream-model", "priority": 0}]}).encode()
|
||||
rows = codex_catalog_rows(listing, tuple((name, name) for name in names), router)
|
||||
served = json.loads(merged_codex_catalog(upstream, rows, client_version=client_version))["models"][1:]
|
||||
served_names_one = [entry.get("default_reasoning_level") is not None for entry in served]
|
||||
|
||||
assert [
|
||||
codex_entry_names_default_reasoning_level(entry["slug"], router, None, client_version) for entry in served
|
||||
] == served_names_one
|
||||
assert served_names_one == [False, False, True, False, client_version is not None]
|
||||
assert served[2]["default_reasoning_level"] == "high"
|
||||
|
||||
|
||||
def test_codex_tool_search_is_offered_off_the_route_and_model_info_and_only_when_every_deployment_answers_it():
|
||||
claude = "anthropic/claude-sonnet-5-5"
|
||||
converse = "bedrock/us.anthropic.claude-sonnet-5-5"
|
||||
|
|
@ -995,6 +1090,34 @@ def test_codex_tool_search_is_offered_off_the_route_and_model_info_and_only_when
|
|||
assert models[0]["context_window"] == 1000000
|
||||
|
||||
|
||||
def test_native_merged_catalog_offers_codex_tool_search_as_the_proxy_listing_does():
|
||||
claude = "anthropic/claude-sonnet-5-5"
|
||||
router = litellm.Router(
|
||||
model_list=[
|
||||
{"model_name": "claude", "litellm_params": {"model": claude}},
|
||||
{
|
||||
"model_name": "claude-off",
|
||||
"litellm_params": {"model": claude},
|
||||
"model_info": {"supports_search_tool": False},
|
||||
},
|
||||
{"model_name": "coding-model", "litellm_params": {"model": "openai/gpt-5.5"}},
|
||||
]
|
||||
)
|
||||
names = ("claude", "claude-off", "coding-model")
|
||||
listing = tuple({"id": name, "object": "model", "created": 0, "owned_by": "x", "mode": "chat"} for name in names)
|
||||
entries = tuple((name, name) for name in names)
|
||||
upstream = json.dumps({"models": [{"slug": "upstream-model", "priority": 0}]}).encode()
|
||||
|
||||
listed = json.loads(codex_model_list_body(listing, entries, router))["models"]
|
||||
merged = json.loads(merged_codex_catalog(upstream, codex_catalog_rows(listing, entries, router)))["models"][1:]
|
||||
|
||||
assert [entry["slug"] for entry in merged] == list(names)
|
||||
assert [entry.get("supports_search_tool") for entry in merged] == [
|
||||
entry.get("supports_search_tool") for entry in listed
|
||||
]
|
||||
assert [entry["supports_search_tool"] for entry in merged[:2]] == [True, False]
|
||||
|
||||
|
||||
_SOL_SLUG: Final = next(
|
||||
slug for slug, model in bundled_codex_models().items() if model.default_service_tier is not None
|
||||
)
|
||||
|
|
@ -1282,3 +1405,20 @@ def test_model_info_that_does_not_validate_declares_no_capability_but_keeps_its_
|
|||
)
|
||||
|
||||
assert capabilities == CodexDeploymentCapabilities(priced_service_tier_keys=frozenset({priced}))
|
||||
|
||||
|
||||
class _CollidingName(str):
|
||||
def __hash__(self) -> int:
|
||||
return 1
|
||||
|
||||
|
||||
def test_catalog_etag_ignores_the_order_a_process_iterates_priced_service_tiers_in() -> None:
|
||||
def row_with_tiers(*tiers: str) -> CodexCatalogRow:
|
||||
capabilities: Final = CodexDeploymentCapabilities.model_construct(
|
||||
priced_service_tier_keys=frozenset(_CollidingName(tier) for tier in tiers)
|
||||
)
|
||||
return replace(_row("gpt-6-astra"), deployment_capabilities=(capabilities,))
|
||||
|
||||
assert codex_catalog_etag("upstream", [row_with_tiers("flex", "priority")]) == codex_catalog_etag(
|
||||
"upstream", [row_with_tiers("priority", "flex")]
|
||||
)
|
||||
|
|
|
|||
|
|
@ -2,13 +2,16 @@
|
|||
Unit tests for ToolPolicyGuardrail.
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
from collections.abc import Mapping, Sequence
|
||||
from types import SimpleNamespace
|
||||
from typing import Any, Final
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
|
||||
|
||||
from litellm.proxy.db import tool_registry_writer
|
||||
from litellm.proxy.guardrails.guardrail_hooks.tool_policy.tool_policy_guardrail import (
|
||||
ToolPolicyGuardrail,
|
||||
)
|
||||
|
|
@ -222,3 +225,104 @@ async def test_blocked_tool_loaded_by_a_replayed_tool_search_raises_next_to_allo
|
|||
)
|
||||
assert exc_info.value.status_code == 400
|
||||
assert exc_info.value.detail["blocked_tools"] == ["run_shell"]
|
||||
|
||||
|
||||
async def _install_tool_policies(monkeypatch: pytest.MonkeyPatch, input_policies: Mapping[str, str]) -> None:
|
||||
prisma_client = MagicMock()
|
||||
prisma_client.db.litellm_tooltable.find_many = AsyncMock(
|
||||
return_value=[
|
||||
SimpleNamespace(tool_name=name, input_policy=policy, output_policy="untrusted")
|
||||
for name, policy in input_policies.items()
|
||||
]
|
||||
)
|
||||
prisma_client.db.litellm_objectpermissiontable.find_many = AsyncMock(return_value=[])
|
||||
registry = tool_registry_writer.ToolPolicyRegistry()
|
||||
await registry.sync_tool_policy_from_db(prisma_client)
|
||||
monkeypatch.setattr(tool_registry_writer, "_tool_policy_registry", registry)
|
||||
|
||||
|
||||
def _responses_request(
|
||||
tools: Sequence[Mapping[str, object]], item_tools: Sequence[Mapping[str, object]]
|
||||
) -> dict[str, object]:
|
||||
return {
|
||||
"user_api_key_request_route": "/v1/responses",
|
||||
"tools": tools,
|
||||
"input": [
|
||||
{"type": "additional_tools", "role": "developer", "tools": item_tools},
|
||||
{"role": "user", "content": "clean up the disk"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"request_data",
|
||||
[
|
||||
_responses_request(
|
||||
[{"type": "function", "name": "get_weather"}],
|
||||
[{"type": "function", "name": "drop_database"}],
|
||||
),
|
||||
_responses_request(
|
||||
[{"type": "function", "name": "get_weather"}],
|
||||
[
|
||||
{
|
||||
"type": "namespace",
|
||||
"name": "mcp__db",
|
||||
"tools": [{"type": "function", "name": "drop_database"}],
|
||||
}
|
||||
],
|
||||
),
|
||||
_responses_request(
|
||||
[
|
||||
{"type": "function", "name": "get_weather"},
|
||||
{"type": "mcp", "server_label": "drop_database"},
|
||||
],
|
||||
[],
|
||||
),
|
||||
_responses_request(
|
||||
[
|
||||
{"type": "function", "name": "get_weather"},
|
||||
{"type": "namespace", "name": "mcp__db", "tools": [{"type": "function", "name": "drop_database"}]},
|
||||
],
|
||||
[],
|
||||
),
|
||||
],
|
||||
ids=["additional_tools_item", "namespace_in_item", "top_level_mcp", "top_level_namespace"],
|
||||
)
|
||||
async def test_blocked_tool_raises_next_to_a_top_level_function_tool(
|
||||
guardrail: ToolPolicyGuardrail, request_data: dict[str, object], monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
await _install_tool_policies(monkeypatch, {"get_weather": "untrusted", "drop_database": "blocked"})
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await guardrail.apply_guardrail(
|
||||
inputs=_tool_request_inputs(["get_weather"]),
|
||||
request_data=request_data,
|
||||
input_type="request",
|
||||
)
|
||||
assert exc_info.value.status_code == 400
|
||||
assert exc_info.value.detail["blocked_tools"] == ["drop_database"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("name", [["drop_database"], {"name": "drop_database"}], ids=["list", "dict"])
|
||||
async def test_function_tool_name_that_is_not_a_string_is_left_for_the_provider_to_refuse(
|
||||
guardrail: ToolPolicyGuardrail, name: object, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
await _install_tool_policies(monkeypatch, {"get_weather": "untrusted"})
|
||||
inputs: Final = {"tools": [{"type": "function", "function": {"name": name}}]}
|
||||
assert await guardrail.apply_guardrail(inputs=inputs, request_data={}, input_type="request") == inputs
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_blocked_tool_defined_at_the_top_level_and_in_an_item_is_named_once(
|
||||
guardrail: ToolPolicyGuardrail, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
tool = {"type": "function", "name": "drop_database"}
|
||||
await _install_tool_policies(monkeypatch, {"drop_database": "blocked"})
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
await guardrail.apply_guardrail(
|
||||
inputs=_tool_request_inputs(["drop_database"]),
|
||||
request_data=_responses_request([tool], [tool]),
|
||||
input_type="request",
|
||||
)
|
||||
assert exc_info.value.detail["blocked_tools"] == ["drop_database"]
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -125,6 +125,8 @@ def route_entry_points_to_stubs(router: _RouterDouble) -> _RouterDouble:
|
|||
return router.get_configured_display_name(name)
|
||||
if field == "service_tiers":
|
||||
return router.get_configured_service_tiers(name, **_given(team_id=team_id))
|
||||
if field == "pinned_reasoning_efforts":
|
||||
return router.get_pinned_reasoning_efforts(name, **_given(team_id=team_id))
|
||||
if field == "max_input_tokens":
|
||||
return router.get_configured_max_input_tokens(name, **_given(team_id=team_id))
|
||||
if field == "deployment_model_info":
|
||||
|
|
|
|||
|
|
@ -242,6 +242,42 @@ class TestCheckToolsAllowlist:
|
|||
assert exc_info.value.type == ProxyErrorTypes.tool_access_denied
|
||||
assert "restricted_tool" in str(exc_info.value.message)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("allowed_tools", "denied"),
|
||||
[
|
||||
pytest.param(["get_weather"], True, id="forbidden"),
|
||||
pytest.param(["get_weather", "delete_user"], False, id="allowed"),
|
||||
],
|
||||
)
|
||||
async def test_tool_defined_in_an_additional_tools_input_item_is_checked_on_responses_route(
|
||||
self, allowed_tools: list[str], denied: bool
|
||||
) -> None:
|
||||
body = {
|
||||
"model": "fireworks_ai/glm-5p3",
|
||||
"input": [
|
||||
{
|
||||
"type": "additional_tools",
|
||||
"role": "developer",
|
||||
"tools": [{"type": "function", "name": "delete_user", "parameters": {"type": "object"}}],
|
||||
},
|
||||
{"type": "message", "role": "user", "content": "Delete user 42"},
|
||||
],
|
||||
}
|
||||
check = check_tools_allowlist(
|
||||
request_body=body,
|
||||
valid_token=_token(metadata={"allowed_tools": allowed_tools}),
|
||||
team_object=None,
|
||||
route="/v1/responses",
|
||||
)
|
||||
if not denied:
|
||||
await check
|
||||
return
|
||||
with pytest.raises(ProxyException) as exc_info:
|
||||
await check
|
||||
assert exc_info.value.type == ProxyErrorTypes.tool_access_denied
|
||||
assert "delete_user" in str(exc_info.value.message)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"loaded_tool",
|
||||
[
|
||||
|
|
|
|||
|
|
@ -1,12 +1,18 @@
|
|||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from litellm.responses.litellm_completion_transformation.reasoning_items import (
|
||||
decode_thinking_blocks,
|
||||
encode_thinking_blocks,
|
||||
is_litellm_minted_reasoning_item,
|
||||
is_minted_reasoning_item_id,
|
||||
is_other_provider_reasoning_item,
|
||||
mint_reasoning_item_id,
|
||||
)
|
||||
from litellm.responses.litellm_completion_transformation.transformation import LiteLLMCompletionResponsesConfig
|
||||
from litellm.responses.utils import ResponsesAPIRequestUtils
|
||||
from litellm.types.llms.openai import ResponsesAPIOptionalRequestParams
|
||||
|
||||
A_PROVIDER_OWNED_REASONING_ITEM_ID = "rs_08d3a89dbb92277a006abf04f4266087d0b4eedacd7848f306"
|
||||
A_PROVIDER_OWNED_ENCRYPTED_BLOB = "gAAAAABo-opaque-provider-blob"
|
||||
|
|
@ -58,5 +64,82 @@ def test_a_reasoning_item_is_litellm_minted_by_its_id_or_by_its_encoded_thinking
|
|||
"encrypted_content": A_PROVIDER_OWNED_ENCRYPTED_BLOB,
|
||||
}
|
||||
)
|
||||
assert not is_litellm_minted_reasoning_item({"type": "message", "id": mint_reasoning_item_id(), "role": "assistant"})
|
||||
assert not is_litellm_minted_reasoning_item(
|
||||
{"type": "message", "id": mint_reasoning_item_id(), "role": "assistant"}
|
||||
)
|
||||
assert not is_litellm_minted_reasoning_item("a bare string input")
|
||||
|
||||
|
||||
SUMMARY = [{"type": "summary_text", "text": "Paris first."}]
|
||||
PROVIDER_OWNED_ITEM = {
|
||||
"type": "reasoning",
|
||||
"id": A_PROVIDER_OWNED_REASONING_ITEM_ID,
|
||||
"summary": SUMMARY,
|
||||
"encrypted_content": A_PROVIDER_OWNED_ENCRYPTED_BLOB,
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"item",
|
||||
[
|
||||
pytest.param({"type": "reasoning", "id": mint_reasoning_item_id(), "summary": SUMMARY}, id="minted id"),
|
||||
pytest.param(
|
||||
{"type": "reasoning", "summary": SUMMARY, "encrypted_content": encode_thinking_blocks([SIGNED_BLOCK])},
|
||||
id="encoded thinking blocks",
|
||||
),
|
||||
pytest.param(
|
||||
{"type": "reasoning", "summary": [], "content": [{"type": "reasoning_text", "text": "Paris first."}]},
|
||||
id="plain-text reasoning content",
|
||||
),
|
||||
pytest.param(
|
||||
{
|
||||
**PROVIDER_OWNED_ITEM,
|
||||
"encrypted_content": ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id(
|
||||
A_PROVIDER_OWNED_ENCRYPTED_BLOB, "another-deployment"
|
||||
),
|
||||
},
|
||||
id="blob wrapped with a litellm deployment id",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_reasoning_items_another_provider_produced_are_recognized(item):
|
||||
assert is_other_provider_reasoning_item(item)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"item",
|
||||
[
|
||||
pytest.param(PROVIDER_OWNED_ITEM, id="provider-owned blob"),
|
||||
pytest.param({"type": "reasoning", "summary": SUMMARY}, id="summary only, no id"),
|
||||
pytest.param(
|
||||
{"type": "reasoning", "summary": SUMMARY, "content": None, "encrypted_content": None}, id="nulled replay"
|
||||
),
|
||||
pytest.param({"type": "reasoning", "summary": SUMMARY, "content": []}, id="empty content"),
|
||||
pytest.param(
|
||||
{"type": "message", "id": mint_reasoning_item_id(), "role": "user", "content": "hi"}, id="not reasoning"
|
||||
),
|
||||
pytest.param("a stray string", id="not an object"),
|
||||
],
|
||||
)
|
||||
def test_reasoning_items_the_provider_can_replay_are_kept(item):
|
||||
assert not is_other_provider_reasoning_item(item)
|
||||
|
||||
|
||||
def test_signed_thinking_survives_the_model_id_wrap_on_its_way_back_to_the_chat_completions_bridge():
|
||||
wrapped = ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id(
|
||||
encode_thinking_blocks([SIGNED_BLOCK]), "anthropic-deployment"
|
||||
)
|
||||
messages = LiteLLMCompletionResponsesConfig.transform_responses_api_request_to_chat_completion_request(
|
||||
model="anthropic/claude-sonnet-5-5",
|
||||
input=[
|
||||
{"role": "user", "content": "Capital of France?"},
|
||||
{"type": "reasoning", "id": mint_reasoning_item_id(), "summary": [], "encrypted_content": wrapped},
|
||||
{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "Paris."}]},
|
||||
{"role": "user", "content": "And of Italy?"},
|
||||
],
|
||||
responses_api_request=ResponsesAPIOptionalRequestParams(),
|
||||
)["messages"]
|
||||
|
||||
assert [message.get("thinking_blocks") for message in messages if message["role"] == "assistant"] == [
|
||||
[SIGNED_BLOCK]
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1,8 +1,9 @@
|
|||
from typing import Final
|
||||
|
||||
import pytest
|
||||
from pydantic import JsonValue
|
||||
|
||||
from litellm.responses.additional_tools import hoist_additional_tools
|
||||
from litellm.responses.additional_tools import hoist_additional_tools, with_additional_tools_hoisted
|
||||
|
||||
_EXEC_TOOL = {
|
||||
"type": "custom",
|
||||
|
|
@ -134,3 +135,10 @@ def test_tool_list_repeated_on_every_turn_is_sent_as_is_without_drop_params() ->
|
|||
|
||||
assert hoisted.tools == (_TOP_LEVEL_TOOL, _function("exec", "old"), _function("exec", "new"))
|
||||
assert hoisted.input == [_USER_MESSAGE, _USER_MESSAGE]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tools", [7, "exec", {"type": "function"}], ids=["int", "str", "dict"])
|
||||
def test_request_whose_top_level_tools_is_not_a_list_stays_as_sent_for_the_provider_to_refuse(tools: object) -> None:
|
||||
request: Final = {"input": [_tool_list(_function("exec")), _USER_MESSAGE], "tools": tools}
|
||||
|
||||
assert with_additional_tools_hoisted(request, drop_params=True) == request
|
||||
|
|
|
|||
|
|
@ -273,6 +273,27 @@ def test_max_input_tokens_leave_out_blocked_deployments_and_other_teams(make_cor
|
|||
assert core.registry().group("mt", field="max_input_tokens", team_id="TA") == (2, 3)
|
||||
|
||||
|
||||
def test_pinned_reasoning_efforts_are_each_routable_deployments_own_value_in_order(make_core: MakeCore) -> None:
|
||||
core = make_core(
|
||||
model_list=[
|
||||
deployment("pe", "p1", reasoning_effort="high"),
|
||||
deployment("pe", "p2"),
|
||||
deployment("pe", "p3", reasoning_effort="low"),
|
||||
deployment("pe", "p4", reasoning_effort="minimal", model_info={"blocked": True}),
|
||||
deployment("pe", "p5", reasoning_effort="medium", model_info={"team_id": "TA"}),
|
||||
]
|
||||
)
|
||||
|
||||
assert core.registry().group("pe", field="pinned_reasoning_efforts") == ("high", None, "low")
|
||||
assert core.registry().group("pe", field="pinned_reasoning_efforts", team_id="TA") == (
|
||||
"high",
|
||||
None,
|
||||
"low",
|
||||
"medium",
|
||||
)
|
||||
assert core.registry().group("not-configured", field="pinned_reasoning_efforts") == ()
|
||||
|
||||
|
||||
def test_max_input_tokens_skip_a_value_a_saved_model_info_copied_from_the_cost_map(make_core: MakeCore) -> None:
|
||||
catalog_limit = GetModelCostMap.loaded_model_cost_map()["gpt-4o"]["max_input_tokens"]
|
||||
core = make_core(
|
||||
|
|
|
|||
|
|
@ -514,6 +514,81 @@ def test_attribute_errors_about_other_objects_pass_through_unchanged(router):
|
|||
assert _public_attribute_error(router, "_core_x", "x", other) is other
|
||||
|
||||
|
||||
_MINTING_DEPLOYMENT = "dep-minting-org"
|
||||
_OTHER_ORG_DEPLOYMENT = "dep-other-org"
|
||||
_BOTH_ORGS = (_MINTING_DEPLOYMENT, _OTHER_ORG_DEPLOYMENT)
|
||||
_GROUP_AFFINITY = ["encrypted_content_affinity"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"affinity, target_ids, reasoning_stays",
|
||||
[
|
||||
pytest.param({}, _BOTH_ORGS, False, id="no affinity, one target cannot decrypt"),
|
||||
pytest.param({}, (_MINTING_DEPLOYMENT,), True, id="no affinity, the only target minted it"),
|
||||
pytest.param(
|
||||
{"optional_pre_call_checks": _GROUP_AFFINITY}, _BOTH_ORGS, True, id="global affinity, origin among targets"
|
||||
),
|
||||
pytest.param(
|
||||
{"optional_pre_call_checks": _GROUP_AFFINITY},
|
||||
(_OTHER_ORG_DEPLOYMENT,),
|
||||
False,
|
||||
id="global affinity, origin not among targets",
|
||||
),
|
||||
pytest.param(
|
||||
{"optional_pre_call_checks": [], "model_group_affinity_config": {"pair": _GROUP_AFFINITY}},
|
||||
_BOTH_ORGS,
|
||||
True,
|
||||
id="affinity on for this group",
|
||||
),
|
||||
pytest.param(
|
||||
{"optional_pre_call_checks": [], "model_group_affinity_config": {"other-group": _GROUP_AFFINITY}},
|
||||
_BOTH_ORGS,
|
||||
False,
|
||||
id="affinity on for another group only",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_decryptable_reasoning_keeps_encrypted_reasoning_only_when_the_router_sends_it_to_a_deployment_that_decrypts_it(
|
||||
affinity, target_ids, reasoning_stays
|
||||
):
|
||||
from litellm.responses.utils import ResponsesAPIRequestUtils
|
||||
|
||||
r = Router(
|
||||
model_list=[
|
||||
{
|
||||
"model_name": "pair",
|
||||
"litellm_params": {"model": "openai/gpt-4o-mini", "api_base": api_base, "api_key": "sk-pair"},
|
||||
"model_info": {"id": model_id},
|
||||
}
|
||||
for api_base, model_id in (
|
||||
("https://minting-org.test/v1", _MINTING_DEPLOYMENT),
|
||||
("https://other-org.test/v1", _OTHER_ORG_DEPLOYMENT),
|
||||
)
|
||||
],
|
||||
**affinity,
|
||||
)
|
||||
summary = [{"type": "summary_text", "text": "multiply 17 by 23"}]
|
||||
user_turn = {"type": "message", "role": "user", "content": "What is 17*23?"}
|
||||
reasoning = {
|
||||
"type": "reasoning",
|
||||
"id": "rs_minted",
|
||||
"encrypted_content": ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id(
|
||||
"gAAAAA-blob", _MINTING_DEPLOYMENT
|
||||
),
|
||||
"summary": summary,
|
||||
}
|
||||
request_input = [user_turn, reasoning]
|
||||
targets = tuple(r.registry().deployment(model_id) for model_id in target_ids)
|
||||
try:
|
||||
kept = r.registry().decryptable_reasoning("pair", request_input, targets)
|
||||
finally:
|
||||
r.retire()
|
||||
|
||||
readable_only = {"type": "reasoning", "summary": summary}
|
||||
assert kept == (user_turn, reasoning if reasoning_stays else readable_only)
|
||||
assert request_input == [user_turn, reasoning], "the caller's input must not change"
|
||||
|
||||
|
||||
# Router-family helpers whose Router parameter became a RoutingContext keep the old spellings.
|
||||
|
||||
|
||||
|
|
|
|||
8
ui/litellm-dashboard/src/lib/http/schema.d.ts
generated
vendored
8
ui/litellm-dashboard/src/lib/http/schema.d.ts
generated
vendored
|
|
@ -1864,13 +1864,15 @@ export interface paths {
|
|||
};
|
||||
/**
|
||||
* Codex Passthrough Route
|
||||
* @description Forward a client's ChatGPT subscription with separate LiteLLM authentication.
|
||||
* @description Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the
|
||||
* key's other chat models through LiteLLM.
|
||||
*/
|
||||
get: operations["codex_passthrough_route_backend_api_codex__endpoint__get"];
|
||||
put?: never;
|
||||
/**
|
||||
* Codex Passthrough Route
|
||||
* @description Forward a client's ChatGPT subscription with separate LiteLLM authentication.
|
||||
* @description Forward a client's ChatGPT subscription with separate LiteLLM authentication, listing and serving the
|
||||
* key's other chat models through LiteLLM.
|
||||
*/
|
||||
post: operations["codex_passthrough_route_backend_api_codex__endpoint__post"];
|
||||
delete?: never;
|
||||
|
|
@ -55970,7 +55972,7 @@ export interface operations {
|
|||
};
|
||||
};
|
||||
responses: {
|
||||
/** @description Unmodified Codex response or server-sent events */
|
||||
/** @description Codex response, the model catalog with the key's LiteLLM models, or server-sent events */
|
||||
200: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue