scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-14 09:03:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 214

Merge 7a66d444d1 into 5e13919f0a

Browse source
This commit is contained in:
jesus-berri 2026-11-05 04:59:38 +00:00 • committed by GitHub
commit e1fa64aa38
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
43 changed files with 3128 additions and 180 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

76
cookbook/ai_coding_tool_guides/chatgpt_subscription_passthrough.md Normal file
View file

@ -0,0 +1,76 @@
# Use your ChatGPT subscription through LiteLLM
Sign into ChatGPT in Codex or Pi and send model requests through your existing LiteLLM proxy. The client keeps your login and refreshes it. Each request uses that client's ChatGPT account, while a separate LiteLLM virtual key controls gateway access and usage attribution.
An administrator adds a `chatgpt/` model using the usual model configuration. No ChatGPT login needs to be added to the proxy. This uses the same separate-key convention as provider passthrough, including Claude Code subscription forwarding.
## Add a model to the proxy
Reuse your existing proxy. If you need one, follow the [proxy quickstart](https://docs.litellm.ai/docs/proxy/docker_quick_start). Merge this entry into its model list and restart the proxy:
```yaml
model_list:
- model_name: gpt-6-luna
litellm_params:
model: chatgpt/gpt-6-luna
model_info:
mode: responses
```
Use a model available to the caller's subscription. The virtual key must allow the exposed model name. Leave saved ChatGPT logins and API credentials unset for a client-only deployment. Explicit cross-provider fallbacks can incur separate API charges; do not configure them for subscription-only traffic.
## Configure Codex
Use your existing LiteLLM URL and a virtual key assigned to your gateway user. Export that key as `LITELLM_API_KEY` in the environment that launches Codex. Keep the value out of committed configuration.
Add this provider to `~/.codex/config.toml`, preserving your other settings:
```toml
model = "gpt-6-luna"
model_provider = "litellm_subscription"
[model_providers.litellm_subscription]
name = "LiteLLM subscription"
base_url = "https://litellm.example.com/v1"
wire_api = "responses"
supports_websockets = false
requires_openai_auth = true
env_http_headers = { "x-litellm-api-key" = "LITELLM_API_KEY" }
```
Replace `https://litellm.example.com` with your proxy URL. A local proxy normally uses `http://localhost:4000/v1`. Launch Codex, sign into ChatGPT if needed, and send a prompt with a model available to your subscription. The selected model must match the name exposed by the gateway.
For a managed workspace rollout, use OpenAI's [gateway configuration guide](https://learn.chatgpt.com/docs/enterprise/sign-in-with-chatgpt-through-a-gateway).
## Native compatibility and Pi
For a native Codex model catalog and request format, change Codex's `base_url` to `https://litellm.example.com/backend-api/codex`. This route uses the same credentials but does not need a model deployment or resolve model aliases. Restricted keys must allow the upstream model name.
### Configure Pi
Sign into the built-in `openai-codex` provider in Pi. Merge this override into `~/.pi/agent/models.json` and export your LiteLLM virtual key as `LITELLM_API_KEY` before launching Pi:
```json
{
"providers": {
"openai-codex": {
"baseUrl": "https://litellm.example.com/backend-api/codex",
"headers": {
"x-litellm-api-key": "LITELLM_API_KEY"
}
}
}
}
```
Keep Pi's subscription login as the provider credential. The header value names an environment variable, not a key to paste into the file.
## Verify account usage
Send a prompt and check the completed request in LiteLLM's Logs page. Its gateway user/key and token counts should match the caller. Give each person their own LiteLLM virtual key for separate attribution. The gateway credential and ChatGPT login are independently authenticated; LiteLLM does not establish that both identities belong to the same person.
The native routes are `GET /backend-api/codex/models` and `POST /backend-api/codex/responses`. Model requests use the upstream model name, so a restricted LiteLLM key must allow that name. OpenAI's model catalog is returned in its native format; seeing a model there does not override gateway permissions.
The proxy forwards the ChatGPT bearer and account header to the fixed OpenAI Codex endpoint. It removes gateway credentials and cookies from the upstream request and redacts credential headers in logging callbacks. The client remains responsible for refresh and reauthentication. Native passthrough returns authentication and quota errors without switching subscriptions. Standard model routes retain the caller login across ChatGPT retries; explicitly configured cross-provider fallbacks use their own credentials. An explicit workspace header is preserved for OpenAI to validate.
This flow does not connect accounts through the dashboard or pool subscription allowances. Use a Codex build that supports ChatGPT authentication with a custom model provider. The integration forwards HTTP and SSE requests; it does not add native WebSocket or other Codex backend routes.

1
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -66,6 +66,7 @@ OPTIONAL_KWARGS_KEYS: Final = (
"itpm",
"otpm",
"use_xai_oauth",
"chatgpt_auth_file",
PROVIDER_AFFINITY_HEADER_KWARG_KEY,
}
)

12
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -143,6 +143,14 @@ def declared_authenticating_provider(model: str | None, custom_llm_provider: str
return declared if declared in PROVIDERS_THAT_AUTHENTICATE_ON_PROVIDER_INFO else None
def resolve_model_and_provider_for_metadata(model: str, custom_llm_provider: str | None = None) -> tuple[str, str]:
declared: Final = declared_authenticating_provider(model, custom_llm_provider)
if declared is not None:
return model.removeprefix(f"{declared}/"), declared
resolved_model, resolved_provider, _, _ = get_llm_provider(model=model, custom_llm_provider=custom_llm_provider)
return resolved_model, resolved_provider
def inferred_provider(model: str | None) -> str | None:
if not model:
return None
@ -804,7 +812,9 @@ def _get_openai_compatible_provider_info(
api_base,
dynamic_api_key,
custom_llm_provider,
) = litellm.ChatGPTConfig()._get_openai_compatible_provider_info(model, api_base, api_key, custom_llm_provider)
) = litellm.ChatGPTConfig()._get_openai_compatible_provider_info(
model, api_base, api_key, custom_llm_provider, litellm_params
)
elif custom_llm_provider == "novita":
api_base = api_base or get_secret("NOVITA_API_BASE") or "https://api.novita.ai/v3/openai"
dynamic_api_key = api_key or get_secret_str("NOVITA_API_KEY")

13
litellm/litellm_core_utils/get_provider_specific_headers.py
View file

@ -1,10 +1,23 @@
from collections.abc import Sequence
from typing import Final
from pydantic import TypeAdapter
from litellm.types.utils import ProviderSpecificHeader
_PROVIDER_HEADERS: Final[TypeAdapter[ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None]] = TypeAdapter(
ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None
)
class ProviderSpecificHeaderUtils:
@staticmethod
def forwarded_authorization(provider_specific_header: object, custom_llm_provider: str | None) -> str | None:
headers: Final = ProviderSpecificHeaderUtils.get_provider_specific_headers(
_PROVIDER_HEADERS.validate_python(provider_specific_header), custom_llm_provider
)
return next((value for key, value in headers.items() if key.lower() == "authorization"), None)
@staticmethod
def get_provider_specific_headers(
provider_specific_header: ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None,

85
litellm/llms/chatgpt/authenticator.py
View file

@ -1,9 +1,9 @@
import base64
import json
import os
import time
from collections.abc import Mapping
from typing import Final, TypeAlias
from functools import lru_cache
from typing import TYPE_CHECKING, Final
import httpx
from pydantic import JsonValue, TypeAdapter, ValidationError
@ -21,18 +21,20 @@ from .common_utils import (
CHATGPT_OAUTH_TOKEN_URL,
GetAccessTokenError,
GetDeviceCodeError,
JsonObject,
RefreshAccessTokenError,
decode_jwt_claims,
extract_chatgpt_account_id,
)
if TYPE_CHECKING:
from litellm.types.router import GenericLiteLLMParams
TOKEN_EXPIRY_SKEW_SECONDS: Final = 60
DEVICE_CODE_TIMEOUT_SECONDS: Final = 15 * 60
DEVICE_CODE_COOLDOWN_SECONDS: Final = 5 * 60
DEVICE_CODE_POLL_SLEEP_SECONDS: Final = 5
OPENAI_AUTH_CLAIM_KEY: Final = "https://api.openai.com/auth"
JsonObject: TypeAlias = Mapping[str, JsonValue]
_JSON_OBJECT_ADAPTER: Final = TypeAdapter(JsonObject)
@ -40,14 +42,34 @@ def _optional_str(value: JsonValue | None) -> str | None:
return value if isinstance(value, str) else None
def get_chatgpt_auth_file(
litellm_params: "Mapping[str, object] | GenericLiteLLMParams | None",
) -> str | None:
if litellm_params is None:
return None
value: Final = (
litellm_params.get("chatgpt_auth_file")
if isinstance(litellm_params, Mapping)
else litellm_params.chatgpt_auth_file
)
if not isinstance(value, str) or not value:
return None
return os.path.abspath(os.path.expanduser(value))
@lru_cache(maxsize=128)
def get_cached_authenticator(auth_file: str) -> "Authenticator":
return Authenticator(auth_file=auth_file)
class Authenticator:
def __init__(self) -> None:
self.token_dir = os.getenv(
"CHATGPT_TOKEN_DIR",
os.path.expanduser("~/.config/litellm/chatgpt"),
def __init__(self, auth_file: str | None = None) -> None:
default_auth_file: Final = os.path.join(
os.getenv("CHATGPT_TOKEN_DIR", os.path.expanduser("~/.config/litellm/chatgpt")),
os.getenv("CHATGPT_AUTH_FILE", "auth.json"),
)
self.auth_file = os.path.join(self.token_dir, os.getenv("CHATGPT_AUTH_FILE", "auth.json"))
self._ensure_token_dir()
self.auth_file = os.path.abspath(os.path.expanduser(auth_file or default_auth_file))
self.token_dir = os.path.dirname(self.auth_file)
def get_api_base(self) -> str:
return os.getenv("CHATGPT_API_BASE") or os.getenv("OPENAI_CHATGPT_API_BASE") or CHATGPT_API_BASE
@ -75,16 +97,18 @@ class Authenticator:
tokens: Final = self._login_device_code()
return tokens["access_token"]
def get_account_id(self) -> str | None:
def get_account_id(self, access_token: str | None = None) -> str | None:
auth_data: Final = self._read_auth_file()
if access_token and (not auth_data or auth_data.get("access_token") != access_token):
return extract_chatgpt_account_id(access_token)
if not auth_data:
return None
account_id: Final = _optional_str(auth_data.get("account_id"))
if account_id:
return account_id
id_token: Final = auth_data.get("id_token")
access_token: Final = auth_data.get("access_token")
derived: Final = self._extract_account_id(_optional_str(id_token or access_token))
saved_access_token: Final = auth_data.get("access_token")
derived: Final = extract_chatgpt_account_id(_optional_str(id_token or saved_access_token))
if derived:
self._write_auth_file({**auth_data, "account_id": derived})
return derived
@ -105,6 +129,7 @@ class Authenticator:
def _write_auth_file(self, data: JsonObject) -> None:
try:
self._ensure_token_dir()
with open(self.auth_file, "w") as f:
json.dump(data, f)
except OSError as exc:
@ -122,34 +147,11 @@ class Authenticator:
return time.time() >= float(derived_expires_at) - TOKEN_EXPIRY_SKEW_SECONDS
def _get_expires_at(self, token: str) -> int | None:
claims: Final = self._decode_jwt_claims(token)
exp: Final = claims.get("exp")
exp: Final = decode_jwt_claims(token).get("exp")
if isinstance(exp, (int, float)):
return int(exp)
return None
def _decode_jwt_claims(self, token: str) -> JsonObject:
try:
parts: Final = token.split(".")
if len(parts) < 2:
return {}
payload_b64: Final = parts[1] + "=" * (-len(parts[1]) % 4)
payload_bytes: Final = base64.urlsafe_b64decode(payload_b64)
return _JSON_OBJECT_ADAPTER.validate_python(json.loads(payload_bytes.decode("utf-8")))
except Exception:
return {}
def _extract_account_id(self, token: str | None) -> str | None:
if not token:
return None
claims: Final = self._decode_jwt_claims(token)
auth_claims: Final = claims.get(OPENAI_AUTH_CLAIM_KEY)
if isinstance(auth_claims, dict):
account_id: Final = auth_claims.get("chatgpt_account_id")
if isinstance(account_id, str) and account_id:
return account_id
return None
def _login_device_code(self) -> dict[str, str]:
cooldown_remaining: Final = self._get_device_code_cooldown_remaining(self._read_auth_file())
if cooldown_remaining > 0:
@ -336,15 +338,16 @@ class Authenticator:
"refresh_token": _optional_str(data.get("refresh_token")) or refresh_token,
"id_token": id_token,
}
account_id: Final = self.get_account_id()
auth_data: Final = self._build_auth_record(refreshed)
self._write_auth_file(auth_data)
self._write_auth_file({**auth_data, "account_id": account_id} if account_id else auth_data)
return refreshed
def _build_auth_record(self, tokens: dict[str, str]) -> JsonObject:
access_token: Final = tokens.get("access_token")
id_token: Final = tokens.get("id_token")
expires_at: Final = self._get_expires_at(access_token) if access_token else None
account_id: Final = self._extract_account_id(id_token or access_token)
account_id: Final = extract_chatgpt_account_id(id_token or access_token)
return {
"access_token": access_token,
"refresh_token": tokens.get("refresh_token"),

80
litellm/llms/chatgpt/chat/transformation.py
View file

@ -1,14 +1,20 @@
from collections.abc import Mapping
from typing import Any, Final
from litellm.exceptions import AuthenticationError
from litellm.llms.openai.openai import OpenAIConfig
from litellm.types.llms.openai import AllMessageValues
from litellm.types.router import GenericLiteLLMParams
from ..authenticator import Authenticator
from ..authenticator import Authenticator, get_cached_authenticator, get_chatgpt_auth_file
from ..common_utils import (
GetAccessTokenError,
chatgpt_client_api_base,
ensure_chatgpt_session_id,
get_chatgpt_client_credential,
get_chatgpt_default_headers,
get_chatgpt_request_credential,
strip_chatgpt_client_credential_headers,
)
from .streaming_utils import ChatGPTToolCallNormalizer
@ -19,9 +25,27 @@ class ChatGPTConfig(OpenAIConfig):
api_key: str | None = None,
api_base: str | None = None,
custom_llm_provider: str = "openai",
authenticator: Authenticator | None = None,
) -> None:
super().__init__()
self.authenticator = Authenticator()
self.authenticator = authenticator if authenticator is not None else Authenticator()
def _resolve_authenticator(
self, litellm_params: Mapping[str, object] | GenericLiteLLMParams | None
) -> Authenticator:
auth_file: Final = get_chatgpt_auth_file(litellm_params)
return get_cached_authenticator(auth_file) if auth_file else self.authenticator
@staticmethod
def _get_access_token_or_raise(authenticator: Authenticator, model: str, llm_provider: str) -> str:
try:
return authenticator.get_access_token()
except GetAccessTokenError as e:
raise AuthenticationError(
model=model,
llm_provider=llm_provider,
message=str(e),
)
def api_base_without_login(self) -> str:
return self.authenticator.get_api_base()
@ -32,35 +56,57 @@ class ChatGPTConfig(OpenAIConfig):
api_base: str | None,
api_key: str | None,
custom_llm_provider: str,
litellm_params: Mapping[str, object] | GenericLiteLLMParams | None = None,
) -> tuple[str | None, str | None, str]:
dynamic_api_base: Final = self.api_base_without_login()
try:
dynamic_api_key: Final = self.authenticator.get_access_token()
except GetAccessTokenError as e:
raise AuthenticationError(
model=model,
llm_provider=custom_llm_provider,
message=str(e),
)
client_credential: Final = get_chatgpt_request_credential(litellm_params)
if client_credential is not None:
return chatgpt_client_api_base(api_base), client_credential.access_token, custom_llm_provider
if api_key and not get_chatgpt_auth_file(litellm_params):
return api_base or self.api_base_without_login(), api_key, custom_llm_provider
authenticator: Final = self._resolve_authenticator(litellm_params)
dynamic_api_base: Final = authenticator.get_api_base()
dynamic_api_key: Final = self._get_access_token_or_raise(authenticator, model, custom_llm_provider)
return dynamic_api_base, dynamic_api_key, custom_llm_provider
def validate_environment(
self,
headers: dict,
headers: dict[str, str],
model: str,
messages: list[AllMessageValues],
optional_params: dict,
litellm_params: dict,
litellm_params: dict[str, object],
api_key: str | None = None,
api_base: str | None = None,
) -> dict:
) -> dict[str, str]:
client_credential: Final = get_chatgpt_client_credential(headers)
if client_credential is not None:
return dict(
get_chatgpt_default_headers(
client_credential.access_token,
client_credential.account_id,
ensure_chatgpt_session_id(litellm_params),
),
**strip_chatgpt_client_credential_headers(headers),
)
authenticator: Final = self._resolve_authenticator(litellm_params)
resolved_api_key: Final = (
self._get_access_token_or_raise(authenticator, model, "chatgpt")
if get_chatgpt_auth_file(litellm_params)
else api_key
)
validated_headers: Final = super().validate_environment(
headers, model, messages, optional_params, litellm_params, api_key, api_base
dict(strip_chatgpt_client_credential_headers(headers)),
model,
messages,
optional_params,
litellm_params,
resolved_api_key,
api_base,
)
account_id: Final = self.authenticator.get_account_id()
account_id: Final = authenticator.get_account_id(resolved_api_key)
session_id: Final = ensure_chatgpt_session_id(litellm_params)
default_headers: Final = get_chatgpt_default_headers(api_key or "", account_id, session_id)
default_headers: Final = get_chatgpt_default_headers(resolved_api_key or "", account_id, session_id)
return {**default_headers, **validated_headers}
def post_stream_processing(self, stream: Any) -> Any:

169
litellm/llms/chatgpt/common_utils.py
View file

@ -2,14 +2,32 @@
Constants and helpers for ChatGPT subscription OAuth.
"""
import base64
import json
import os
import platform
from typing import Any, Final
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from types import MappingProxyType
from typing import TYPE_CHECKING, Final, TypeAlias
from urllib.parse import urlsplit
from uuid import uuid4
import httpx
from pydantic import JsonValue, TypeAdapter
from litellm.litellm_core_utils.get_provider_specific_headers import ProviderSpecificHeaderUtils
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.types.utils import ProviderSpecificHeader
if TYPE_CHECKING:
from litellm.types.router import GenericLiteLLMParams
_SCOPED_HEADERS: Final[TypeAdapter[ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None]] = TypeAdapter(
ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None
)
_STRING_HEADERS: Final = TypeAdapter(Mapping[str, str])
_PARAMS: Final = TypeAdapter(Mapping[str, object])
# OAuth + API constants (derived from openai/codex)
CHATGPT_AUTH_BASE: Final = "https://auth.openai.com"
@ -19,6 +37,12 @@ CHATGPT_OAUTH_TOKEN_URL: Final = f"{CHATGPT_AUTH_BASE}/oauth/token"
CHATGPT_DEVICE_VERIFY_URL: Final = f"{CHATGPT_AUTH_BASE}/codex/device"
CHATGPT_API_BASE: Final = "https://chatgpt.com/backend-api/codex"
CHATGPT_CLIENT_ID: Final = "app_EMoamEEZ73f0CkXaXp7hrann"
CHATGPT_ACCOUNT_ID_HEADER: Final = "ChatGPT-Account-Id"
OPENAI_AUTH_CLAIM_KEY: Final = "https://api.openai.com/auth"
JsonObject: TypeAlias = Mapping[str, JsonValue]
_JSON_OBJECT_ADAPTER: Final = TypeAdapter(JsonObject)
_EMPTY_CLAIMS: Final[JsonObject] = MappingProxyType({})
DEFAULT_ORIGINATOR: Final = "codex_cli_rs"
DEFAULT_USER_AGENT: Final = "codex_cli_rs/0.0.0 (Unknown 0; unknown) unknown"
@ -229,7 +253,7 @@ def get_chatgpt_default_headers(
access_token: str,
account_id: str | None,
session_id: str | None = None,
) -> dict:
) -> dict[str, str]:
originator: Final = get_chatgpt_originator()
user_agent: Final = get_chatgpt_user_agent(originator)
headers: Final = {
@ -242,30 +266,133 @@ def get_chatgpt_default_headers(
if session_id:
headers["session_id"] = session_id
if account_id:
headers["ChatGPT-Account-Id"] = account_id
headers[CHATGPT_ACCOUNT_ID_HEADER] = account_id
return headers
def decode_jwt_claims(token: str) -> JsonObject:
try:
parts: Final = token.split(".")
if len(parts) < 2:
return _EMPTY_CLAIMS
payload_b64: Final = parts[1] + "=" * (-len(parts[1]) % 4)
payload_bytes: Final = base64.urlsafe_b64decode(payload_b64)
return _JSON_OBJECT_ADAPTER.validate_python(json.loads(payload_bytes.decode("utf-8")))
except Exception:
return _EMPTY_CLAIMS
def extract_chatgpt_account_id(token: str | None) -> str | None:
if not token:
return None
auth_claims: Final = decode_jwt_claims(token).get(OPENAI_AUTH_CLAIM_KEY)
if not isinstance(auth_claims, dict):
return None
account_id: Final = auth_claims.get("chatgpt_account_id")
return account_id if isinstance(account_id, str) and account_id else None
def _bearer_token(value: str) -> str:
scheme, separator, token = value.strip().partition(" ")
if not separator:
return scheme
return token.strip() if scheme.lower() == "bearer" else ""
def is_chatgpt_oauth_key(value: str | None) -> bool:
if value is None:
return False
return OPENAI_AUTH_CLAIM_KEY in decode_jwt_claims(_bearer_token(value))
@dataclass(frozen=True, slots=True)
class ChatGPTClientCredential:
access_token: str
account_id: str | None
def _header_value(headers: Mapping[str, str], name: str) -> str | None:
return next((value for key, value in headers.items() if key.lower() == name), None)
def get_chatgpt_client_credential(headers: Mapping[str, str]) -> ChatGPTClientCredential | None:
auth_header: Final = _header_value(headers, "authorization")
if auth_header is None or not is_chatgpt_oauth_key(auth_header):
return None
access_token: Final = _bearer_token(auth_header)
account_id: Final = _header_value(headers, CHATGPT_ACCOUNT_ID_HEADER.lower()) or extract_chatgpt_account_id(
access_token
)
return ChatGPTClientCredential(access_token=access_token, account_id=account_id)
def select_chatgpt_client_credential_headers(headers: Mapping[str, str]) -> Mapping[str, str]:
if get_chatgpt_client_credential(headers) is None:
return MappingProxyType({})
return MappingProxyType(
{
key: value
for key, value in headers.items()
if (key.lower() == "authorization" and is_chatgpt_oauth_key(value))
or key.lower() == CHATGPT_ACCOUNT_ID_HEADER.lower()
}
)
def strip_chatgpt_client_credential_headers(headers: Mapping[str, str]) -> Mapping[str, str]:
return MappingProxyType(
{
key: value
for key, value in headers.items()
if key.lower() not in ("authorization", CHATGPT_ACCOUNT_ID_HEADER.lower())
}
)
def get_chatgpt_request_credential(
params: "Mapping[str, object] | GenericLiteLLMParams | None",
) -> ChatGPTClientCredential | None:
normalized: Final = _normalize_litellm_params(params)
scoped: Final = normalized.get("provider_specific_header")
extra: Final = normalized.get("extra_headers")
headers: Final = ProviderSpecificHeaderUtils.get_provider_specific_headers(
provider_specific_header=_SCOPED_HEADERS.validate_python(scoped), custom_llm_provider="chatgpt"
)
return get_chatgpt_client_credential(headers) or get_chatgpt_client_credential(
_STRING_HEADERS.validate_python(extra) if extra is not None else MappingProxyType({})
)
def chatgpt_client_api_base(api_base: str | None) -> str:
base: Final = api_base or CHATGPT_API_BASE
parsed: Final = urlsplit(base)
if (
parsed.scheme != "https"
or parsed.hostname != "chatgpt.com"
or parsed.port not in (None, 443)
or parsed.username
or parsed.password
):
raise ChatGPTAuthError(
status_code=400, message="Client ChatGPT credentials require the official HTTPS ChatGPT endpoint"
)
return base
def get_chatgpt_default_instructions() -> str:
return os.getenv("CHATGPT_DEFAULT_INSTRUCTIONS") or CHATGPT_DEFAULT_INSTRUCTIONS
def _normalize_litellm_params(litellm_params: Any | None) -> dict:
if litellm_params is None:
return {}
if isinstance(litellm_params, dict):
return litellm_params
if hasattr(litellm_params, "model_dump"):
try:
return litellm_params.model_dump()
except Exception:
return {}
if hasattr(litellm_params, "dict"):
try:
return litellm_params.dict()
except Exception:
return {}
return {}
def _normalize_litellm_params(litellm_params: object) -> Mapping[str, object]:
if isinstance(litellm_params, Mapping):
return _PARAMS.validate_python(litellm_params)
serializer: Final = getattr(litellm_params, "model_dump", None) or getattr(litellm_params, "dict", None)
if not callable(serializer):
return MappingProxyType({})
try:
return _PARAMS.validate_python(serializer())
except Exception:
return MappingProxyType({})
def get_chatgpt_session_id(litellm_params: object) -> str | None:
@ -274,8 +401,8 @@ def get_chatgpt_session_id(litellm_params: object) -> str | None:
value = params.get(key)
if value:
return str(value)
metadata: Final = params.get("metadata")
if isinstance(metadata, dict):
metadata: Final = _normalize_litellm_params(params.get("metadata"))
if metadata:
value = metadata.get("session_id")
if value:
return str(value)

58
litellm/llms/chatgpt/responses/transformation.py
View file

@ -2,6 +2,7 @@ from collections.abc import Mapping
from typing import TYPE_CHECKING, Any, Final
import httpx
from pydantic import TypeAdapter
from litellm.exceptions import AuthenticationError
from litellm.litellm_core_utils.core_helpers import process_response_headers
@ -23,36 +24,64 @@ from litellm.types.llms.openai import (
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import LlmProviders
from ..authenticator import Authenticator
from ..authenticator import Authenticator, get_cached_authenticator, get_chatgpt_auth_file
from ..common_utils import (
CHATGPT_API_BASE,
GetAccessTokenError,
ensure_chatgpt_session_id,
get_chatgpt_client_credential,
get_chatgpt_default_headers,
get_chatgpt_default_instructions,
get_chatgpt_request_credential,
strip_chatgpt_client_credential_headers,
)
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
_LITELLM_PARAMS: Final = TypeAdapter(dict[str, object])
class ChatGPTResponsesAPIConfig(OpenAIResponsesAPIConfig):
def __init__(self) -> None:
def __init__(self, authenticator: Authenticator | None = None) -> None:
super().__init__()
self.authenticator = Authenticator()
self.authenticator = authenticator if authenticator is not None else Authenticator()
@property
def custom_llm_provider(self) -> LlmProviders:
return LlmProviders.CHATGPT
def _resolve_authenticator(
self, litellm_params: Mapping[str, object] | GenericLiteLLMParams | None
) -> Authenticator:
auth_file: Final = get_chatgpt_auth_file(litellm_params)
return get_cached_authenticator(auth_file) if auth_file else self.authenticator
def validate_environment(
self,
headers: dict,
headers: dict[str, str],
model: str,
litellm_params: GenericLiteLLMParams | None,
) -> dict:
) -> dict[str, str]:
session_id: Final = ensure_chatgpt_session_id(litellm_params)
client_credential: Final = get_chatgpt_client_credential(headers)
if client_credential is not None:
return {
**get_chatgpt_default_headers(client_credential.access_token, client_credential.account_id, session_id),
**strip_chatgpt_client_credential_headers(headers),
}
authenticator: Final = self._resolve_authenticator(litellm_params)
safe_headers: Final = strip_chatgpt_client_credential_headers(headers)
configured_api_key: Final = litellm_params.api_key if litellm_params is not None else None
if configured_api_key and not get_chatgpt_auth_file(litellm_params):
return {
**get_chatgpt_default_headers(
configured_api_key, authenticator.get_account_id(configured_api_key), session_id
),
**safe_headers,
}
try:
access_token: Final = self.authenticator.get_access_token()
access_token: Final = authenticator.get_access_token()
except GetAccessTokenError as e:
raise AuthenticationError(
model=model,
@ -60,10 +89,9 @@ class ChatGPTResponsesAPIConfig(OpenAIResponsesAPIConfig):
message=str(e),
)
account_id: Final = self.authenticator.get_account_id()
session_id: Final = ensure_chatgpt_session_id(litellm_params)
account_id: Final = authenticator.get_account_id()
default_headers: Final = get_chatgpt_default_headers(access_token, account_id, session_id)
return {**default_headers, **headers}
return {**default_headers, **safe_headers}
def transform_responses_api_request(
self,
@ -75,7 +103,7 @@ class ChatGPTResponsesAPIConfig(OpenAIResponsesAPIConfig):
) -> dict:
request: Final = super().transform_responses_api_request(
model,
input,
[{"role": "user", "content": input}] if isinstance(input, str) else input,
response_api_optional_request_params,
litellm_params,
headers,
@ -241,9 +269,13 @@ class ChatGPTResponsesAPIConfig(OpenAIResponsesAPIConfig):
api_base: str | None,
litellm_params: dict,
) -> str:
api_base = api_base or self.authenticator.get_api_base() or CHATGPT_API_BASE
api_base = api_base.rstrip("/")
return f"{api_base}/responses"
params: Final = _LITELLM_PARAMS.validate_python(litellm_params)
authenticator: Final = self._resolve_authenticator(params)
sends_saved_login: Final = get_chatgpt_request_credential(params) is None and not (
params.get("api_key") and not get_chatgpt_auth_file(params)
)
base: Final = (None if sends_saved_login else api_base) or authenticator.get_api_base() or CHATGPT_API_BASE
return f"{base.rstrip('/')}/responses"
def supports_native_websocket(self) -> bool:
"""ChatGPT does not support native WebSocket for Responses API"""

107
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -188,6 +188,26 @@ from litellm.utils import (
)
def _merge_headers_case_insensitive(
headers: Mapping[str, object], extra_headers: Mapping[str, object] | None
) -> dict[str, str]: # mutable-ok: legacy HTTP handler methods require a dict of headers
overridden: Final = frozenset(name.lower() for name in extra_headers or {})
return {
**{name: str(value) for name, value in headers.items() if name.lower() not in overridden},
**{name: str(value) for name, value in (extra_headers or {}).items()},
}
def _client_request_headers(
response_api_optional_request_params: Mapping[str, object],
extra_headers: Mapping[str, object] | None,
) -> dict[str, str]: # mutable-ok: provider validate_environment methods require a dict
from litellm.litellm_core_utils.core_helpers import as_str_mapping
optional_headers: Final = as_str_mapping(response_api_optional_request_params.get("extra_headers"))
return _merge_headers_case_insensitive(optional_headers or {}, extra_headers)
def _rust_responses_websocket_enabled(
custom_llm_provider: str | None,
) -> bool:
@ -2558,17 +2578,22 @@ class BaseLLMHTTPHandler:
else:
sync_httpx_client = client
headers = responses_api_provider_config.validate_environment(
headers=response_api_optional_request_params.get("extra_headers", {}) or {},
model=model,
litellm_params=litellm_params,
)
if extra_headers:
headers.update(extra_headers)
headers = { # rebind-ok: validation replaces the method's incoming headers
**responses_api_provider_config.validate_environment(
headers=(
_client_request_headers(response_api_optional_request_params, extra_headers)
if custom_llm_provider == "chatgpt"
else response_api_optional_request_params.get("extra_headers", {}) or {}
),
model=model,
litellm_params=litellm_params,
),
**((extra_headers or {}) if custom_llm_provider != "chatgpt" else {}),
}
# Check if streaming is requested
stream = response_api_optional_request_params.get("stream", False)
caller_requested_stream: Final = bool(stream or (extra_body or {}).get("stream"))
api_base: Final = responses_api_provider_config.get_complete_url(
api_base=litellm_params.api_base,
@ -2647,8 +2672,20 @@ class BaseLLMHTTPHandler:
stream=stream,
**body_kwargs,
)
if fake_stream is True:
return MockResponsesAPIStreamingIterator(
if caller_requested_stream:
if fake_stream is True:
return MockResponsesAPIStreamingIterator(
response=response,
model=model,
logging_obj=logging_obj,
responses_api_provider_config=responses_api_provider_config,
litellm_metadata=litellm_metadata,
custom_llm_provider=custom_llm_provider,
request_data=request_context,
call_type=CallTypes.responses.value,
)
return SyncResponsesAPIStreamingIterator(
response=response,
model=model,
logging_obj=logging_obj,
@ -2658,17 +2695,7 @@ class BaseLLMHTTPHandler:
request_data=request_context,
call_type=CallTypes.responses.value,
)
return SyncResponsesAPIStreamingIterator(
response=response,
model=model,
logging_obj=logging_obj,
responses_api_provider_config=responses_api_provider_config,
litellm_metadata=litellm_metadata,
custom_llm_provider=custom_llm_provider,
request_data=request_context,
call_type=CallTypes.responses.value,
)
response.read()
else:
response = sync_httpx_client.post(
url=api_base,
@ -2751,16 +2778,21 @@ class BaseLLMHTTPHandler:
async_httpx_client = client
headers = responses_api_provider_config.validate_environment(
headers=response_api_optional_request_params.get("extra_headers", {}) or {},
headers=(
_client_request_headers(response_api_optional_request_params, extra_headers)
if custom_llm_provider == "chatgpt"
else response_api_optional_request_params.get("extra_headers", {}) or {}
),
model=model,
litellm_params=litellm_params,
)
if extra_headers:
if extra_headers and custom_llm_provider != "chatgpt":
headers.update(extra_headers)
# Check if streaming is requested
stream = response_api_optional_request_params.get("stream", False)
caller_requested_stream: Final = bool(stream or (extra_body or {}).get("stream"))
api_base: Final = responses_api_provider_config.get_complete_url(
api_base=litellm_params.api_base,
@ -2840,8 +2872,20 @@ class BaseLLMHTTPHandler:
**body_kwargs,
)
if fake_stream is True:
return MockResponsesAPIStreamingIterator(
if caller_requested_stream:
if fake_stream is True:
return MockResponsesAPIStreamingIterator(
response=response,
model=model,
logging_obj=logging_obj,
responses_api_provider_config=responses_api_provider_config,
litellm_metadata=litellm_metadata,
custom_llm_provider=custom_llm_provider,
request_data=request_context,
call_type=CallTypes.responses.value,
)
return ResponsesAPIStreamingIterator(
response=response,
model=model,
logging_obj=logging_obj,
@ -2851,18 +2895,7 @@ class BaseLLMHTTPHandler:
request_data=request_context,
call_type=CallTypes.responses.value,
)
# Return the streaming iterator
return ResponsesAPIStreamingIterator(
response=response,
model=model,
logging_obj=logging_obj,
responses_api_provider_config=responses_api_provider_config,
litellm_metadata=litellm_metadata,
custom_llm_provider=custom_llm_provider,
request_data=request_context,
call_type=CallTypes.responses.value,
)
await response.aread()
else:
response = await async_httpx_client.post(
url=api_base,

15
litellm/main.py
View file

@ -636,10 +636,18 @@ async def acompletion(
"enable_json_schema_validation": enable_json_schema_validation,
}
if custom_llm_provider is None:
_supplemental: Final = cast( # cast-ok: kwargs values are request-scoped objects
"dict[str, object]",
{
**{k: kwargs[k] for k in OPTIONAL_KWARGS_KEYS | {"provider_specific_header"} if k in kwargs},
"extra_headers": extra_headers,
},
)
_, custom_llm_provider, _, _ = get_llm_provider(
model=model,
custom_llm_provider=custom_llm_provider,
api_base=kwargs.get("api_base") or base_url,
litellm_params=GenericLiteLLMParams.model_validate(_supplemental),
)
fallbacks = fallbacks or litellm.model_fallbacks
@ -5497,7 +5505,10 @@ def completion(
if deployment_id is not None: # azure llms
model = deployment_id
custom_llm_provider = "azure"
_supplemental_provider_params: Final = {k: kwargs[k] for k in OPTIONAL_KWARGS_KEYS if k in kwargs}
_supplemental_provider_params: Final = {
**{k: kwargs[k] for k in OPTIONAL_KWARGS_KEYS | {"provider_specific_header"} if k in kwargs},
"extra_headers": extra_headers,
}
model, custom_llm_provider, dynamic_api_key, api_base = get_llm_provider(
model=model,
custom_llm_provider=custom_llm_provider,
@ -5726,6 +5737,7 @@ def completion(
tpm=kwargs.get("tpm"),
rpm=kwargs.get("rpm"),
use_xai_oauth=kwargs.get("use_xai_oauth", False),
chatgpt_auth_file=kwargs.get("chatgpt_auth_file"),
gigachat_scope=kwargs.get("gigachat_scope"),
gigachat_auth_url=kwargs.get("gigachat_auth_url"),
gigachat_access_token=kwargs.get("gigachat_access_token"),
@ -8797,6 +8809,7 @@ async def ahealth_check(
custom_llm_provider=custom_llm_provider_from_params,
api_base=api_base_from_params,
api_key=api_key_from_params,
litellm_params=GenericLiteLLMParams(**model_params),
)
if model in litellm.model_cost and mode is None:
mode = litellm.model_cost[model].get("mode")

5
litellm/proxy/_lazy_features.py
View file

@ -204,6 +204,11 @@ LAZY_FEATURES: Final[tuple[LazyFeature, ...]] = (
module_path="litellm.proxy.management_endpoints.config_override_endpoints",
path_prefixes=("/config_overrides",),
),
LazyFeature(
name="codex_passthrough",
module_path="litellm.proxy.pass_through_endpoints.codex_passthrough_endpoints",
path_prefixes=("/backend-api/codex/",),
),
LazyFeature(
name="llm_passthrough",
module_path="litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints",

178
litellm/proxy/_lazy_openapi_snapshot.json
View file

@ -8509,6 +8509,184 @@
}
}
},
"codex_passthrough": {
"components": {
"schemas": {
"HTTPValidationError": {
"properties": {
"detail": {
"items": {
"$ref": "#/components/schemas/ValidationError"
},
"title": "Detail",
"type": "array"
}
},
"title": "HTTPValidationError",
"type": "object"
},
"ValidationError": {
"properties": {
"ctx": {
"title": "Context",
"type": "object"
},
"input": {
"title": "Input"
},
"loc": {
"items": {
"anyOf": [
{
"type": "string"
},
{
"type": "integer"
}
]
},
"title": "Location",
"type": "array"
},
"msg": {
"title": "Message",
"type": "string"
},
"type": {
"title": "Error Type",
"type": "string"
}
},
"required": [
"loc",
"msg",
"type"
],
"title": "ValidationError",
"type": "object"
}
}
},
"paths": {
"/backend-api/codex/{endpoint}": {
"get": {
"description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication.",
"operationId": "codex_passthrough_route_backend_api_codex__endpoint__get",
"parameters": [
{
"in": "path",
"name": "endpoint",
"required": true,
"schema": {
"enum": [
"models",
"responses"
],
"title": "Endpoint",
"type": "string"
}
}
],
"responses": {
"200": {
"content": {
"application/json": {
"schema": {}
}
},
"description": "Successful Response"
},
"422": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
},
"description": "Validation Error"
}
},
"security": [
{
"APIKeyHeader": []
}
],
"summary": "Codex Passthrough Route",
"tags": [
"codex_passthrough"
]
},
"post": {
"description": "Forward a client's ChatGPT subscription with separate LiteLLM authentication.",
"operationId": "codex_passthrough_route_backend_api_codex__endpoint__post",
"parameters": [
{
"in": "path",
"name": "endpoint",
"required": true,
"schema": {
"enum": [
"models",
"responses"
],
"title": "Endpoint",
"type": "string"
}
}
],
"requestBody": {
"content": {
"application/json": {
"schema": {
"additionalProperties": true,
"type": "object"
}
}
},
"required": true
},
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"additionalProperties": true,
"type": "object"
}
},
"text/event-stream": {
"schema": {
"type": "string"
}
}
},
"description": "Unmodified Codex response or server-sent events"
},
"422": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
},
"description": "Validation Error"
}
},
"security": [
{
"APIKeyHeader": []
}
],
"summary": "Codex Passthrough Route",
"tags": [
"codex_passthrough"
]
}
}
}
},
"compliance": {
"components": {
"schemas": {

1
litellm/proxy/_types.py
View file

@ -490,6 +490,7 @@ class LiteLLMRoutes(enum.Enum):
]
mapped_pass_through_routes = [
"/backend-api/codex",
"/bedrock",
"/comprehendmedical",
"/azure_speech",

5
litellm/proxy/auth/auth_utils.py
View file

@ -388,6 +388,11 @@ def _check_banned_params(
Shared between the root-level check and the nested-config check so a
new banned param only needs to be added in one place.
"""
if "chatgpt_auth_file" in body:
raise ValueError(
"Rejected Request: chatgpt_auth_file is not allowed in request body. It makes the proxy read an "
"auth.json from its own host, so only the deployment config may name one."
)
for param in _BANNED_REQUEST_BODY_PARAMS:
if param not in body:
continue

24
litellm/proxy/health_endpoints/_health_endpoints.py
View file

@ -73,6 +73,7 @@ from litellm.router_utils.clientside_credential_handler import (
clientside_credential_keys,
)
from litellm.secret_managers.main import get_secret_bool
from litellm.types.router import GenericLiteLLMParams
#### Health ENDPOINTS ####
@ -173,6 +174,19 @@ def _config_base_for_health_check(
return {key: value for key, value in config_params.items() if key not in _CONFIG_CONNECTION_FIELDS}
def _reject_unauthorized_chatgpt_auth_file(
requested: GenericLiteLLMParams, configured: GenericLiteLLMParams, user_api_key_dict: UserAPIKeyAuth
) -> None:
from litellm.proxy.management_endpoints.model_management_endpoints import ModelManagementAuthChecks
try:
ModelManagementAuthChecks.can_user_set_chatgpt_auth_file(
litellm_params=requested, user_api_key_dict=user_api_key_dict, existing_litellm_params=configured
)
except ProxyException as auth_file_error:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail={"error": auth_file_error.message})
def get_callback_identifier(callback):
"""
Get the callback identifier string, handling both strings and objects.
@ -2193,6 +2207,16 @@ async def test_model_connection(
litellm_params=litellm_params,
)
_reject_unauthorized_chatgpt_auth_file(
requested=GenericLiteLLMParams.model_validate(
{"chatgpt_auth_file": request_litellm_params.get("chatgpt_auth_file")}
),
configured=GenericLiteLLMParams.model_validate(
{"chatgpt_auth_file": config_litellm_params.get("chatgpt_auth_file")}
),
user_api_key_dict=user_api_key_dict,
)
## Auth check, on the final probe params so health_check_params cannot retarget it afterwards
await ModelManagementAuthChecks.can_user_make_model_call(
model_params=Deployment(

11
litellm/proxy/litellm_pre_call_utils.py
View file

@ -1202,13 +1202,14 @@ def clean_headers(
Cleaned headers dict
"""
from litellm.llms.anthropic.common_utils import is_anthropic_oauth_key
from litellm.llms.chatgpt.common_utils import is_chatgpt_oauth_key
clean_headers: Final = {}
litellm_key_lower: Final = litellm_key_header_name.lower() if litellm_key_header_name is not None else None
for header, value in headers.items():
header_lower = header.lower()
if header_lower == "authorization" and is_anthropic_oauth_key(value):
if header_lower == "authorization" and (is_anthropic_oauth_key(value) or is_chatgpt_oauth_key(value)):
if authenticated_with_header is None or authenticated_with_header.lower() != "authorization":
clean_headers[header] = value
continue
@ -1237,7 +1238,7 @@ def clean_headers(
def _is_credential_header(header: str) -> bool:
"""Whether `header` carries a caller credential rather than request context."""
return header.lower() in _CREDENTIAL_HEADER_NAMES
return header.lower().removeprefix("x-pass-") in _CREDENTIAL_HEADER_NAMES
def redact_credential_headers(headers: Mapping[str, str]) -> Mapping[str, str]:
@ -3497,6 +3498,7 @@ _ANTHROPIC_API_HEADER_PROVIDERS: Final = ",".join(
)
)
_ANTHROPIC_OAUTH_CREDENTIAL_PROVIDERS: Final = ",".join(sorted(ANTHROPIC_OAUTH_FORWARD_PROVIDERS))
_CHATGPT_OAUTH_CREDENTIAL_PROVIDERS: Final = LlmProviders.CHATGPT.value
def add_provider_specific_headers_to_request(
@ -3504,6 +3506,7 @@ def add_provider_specific_headers_to_request(
headers: dict,
) -> bool:
from litellm.llms.anthropic.common_utils import is_anthropic_oauth_key
from litellm.llms.chatgpt.common_utils import select_chatgpt_client_credential_headers
anthropic_api_headers: Final = {header: headers[header] for header in ANTHROPIC_API_HEADERS if header in headers}
anthropic_oauth_credential_headers: Final = {
@ -3511,16 +3514,20 @@ def add_provider_specific_headers_to_request(
for header, value in headers.items()
if header.lower() == "authorization" and is_anthropic_oauth_key(value)
}
chatgpt_oauth_credential_headers: Final = dict(select_chatgpt_client_credential_headers(headers))
scoped_headers: Final = [
ProviderSpecificHeader(custom_llm_provider=providers, extra_headers=extra_headers)
for providers, extra_headers in (
(_ANTHROPIC_API_HEADER_PROVIDERS, anthropic_api_headers),
(_ANTHROPIC_OAUTH_CREDENTIAL_PROVIDERS, anthropic_oauth_credential_headers),
(_CHATGPT_OAUTH_CREDENTIAL_PROVIDERS, chatgpt_oauth_credential_headers),
)
if extra_headers
]
if "provider_specific_header" in data:
del data["provider_specific_header"]
if scoped_headers:
data["provider_specific_header"] = scoped_headers[0] if len(scoped_headers) == 1 else scoped_headers
return bool(anthropic_oauth_credential_headers)

39
litellm/proxy/management_endpoints/model_management_endpoints.py
View file

@ -1216,6 +1216,12 @@ async def patch_model(
existing_litellm_params=db_model.litellm_params,
)
ModelManagementAuthChecks.can_user_set_chatgpt_auth_file(
litellm_params=patch_data.litellm_params,
user_api_key_dict=user_api_key_dict,
existing_litellm_params=db_model.litellm_params,
)
_raise_on_strategy_router_write_violation(
incoming_params=patch_data.litellm_params,
existing_params=db_model.litellm_params,
@ -2063,6 +2069,28 @@ class ModelManagementAuthChecks:
param="aws_session_tags",
)
@staticmethod
def can_user_set_chatgpt_auth_file(
litellm_params: GenericLiteLLMParams | None,
user_api_key_dict: UserAPIKeyAuth,
existing_litellm_params: GenericLiteLLMParams | None = None,
) -> Literal[True]:
if litellm_params is None or litellm_params.chatgpt_auth_file is None:
return True
if user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN:
return True
existing_auth_file: Final = (
existing_litellm_params.chatgpt_auth_file if existing_litellm_params is not None else None
)
if existing_auth_file == litellm_params.chatgpt_auth_file:
return True
raise ProxyException(
message=f"Only a proxy admin can set chatgpt_auth_file on a model. Your role={user_api_key_dict.user_role}.",
type=ProxyErrorTypes.auth_error.value,
code=status.HTTP_403_FORBIDDEN,
param="chatgpt_auth_file",
)
@staticmethod
async def allow_team_model_action(
model_params: Deployment | updateDeployment,
@ -2430,6 +2458,11 @@ async def add_new_model(
user_api_key_dict=user_api_key_dict,
)
ModelManagementAuthChecks.can_user_set_chatgpt_auth_file(
litellm_params=model_params.litellm_params,
user_api_key_dict=user_api_key_dict,
)
_raise_on_strategy_router_write_violation(
incoming_params=model_params.litellm_params,
existing_params=None,
@ -2634,6 +2667,12 @@ async def update_model(
existing_litellm_params=deployment.litellm_params,
)
ModelManagementAuthChecks.can_user_set_chatgpt_auth_file(
litellm_params=model_params.litellm_params,
user_api_key_dict=user_api_key_dict,
existing_litellm_params=deployment.litellm_params,
)
_raise_on_strategy_router_write_violation(
incoming_params=model_params.litellm_params,
existing_params=deployment.litellm_params,

87
litellm/proxy/pass_through_endpoints/codex_passthrough_endpoints.py Normal file
View file

@ -0,0 +1,87 @@
import hmac
from types import MappingProxyType
from typing import Annotated, Final, Literal
from fastapi import APIRouter, Depends, HTTPException, Request, Response
from litellm.llms.chatgpt.common_utils import CHATGPT_API_BASE
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.pass_through_endpoints.pass_through_endpoints import pass_through_request
router: Final = APIRouter()
_HEADERS: Final = frozenset(
{
"authorization",
"chatgpt-account-id",
"content-type",
"accept",
"user-agent",
"originator",
"openai-beta",
"session-id",
"thread-id",
"session_id",
"conversation_id",
"x-codex-turn-metadata",
"x-codex-turn-state",
}
)
@router.get("/backend-api/codex/{endpoint}", tags=["ChatGPT Pass-through", "pass-through"])
@router.post(
"/backend-api/codex/{endpoint}",
tags=["ChatGPT Pass-through", "pass-through"],
openapi_extra={
"requestBody": {
"required": True,
"content": {"application/json": {"schema": {"type": "object", "additionalProperties": True}}},
}
},
responses={
200: {
"description": "Unmodified Codex response or server-sent events",
"content": {
"application/json": {"schema": {"type": "object", "additionalProperties": True}},
"text/event-stream": {"schema": {"type": "string"}},
},
}
},
)
async def codex_passthrough_route(
endpoint: Literal["models", "responses"],
request: Request,
user_api_key_dict: Annotated[UserAPIKeyAuth, Depends(user_api_key_auth)],
) -> Response:
"""Forward a client's ChatGPT subscription with separate LiteLLM authentication."""
expected_method: Final = "GET" if endpoint == "models" else "POST"
if request.method != expected_method:
raise HTTPException(status_code=405, detail="Method not allowed", headers={"Allow": expected_method})
authorization: Final = request.headers.get("authorization", "")
scheme, _, token = authorization.partition(" ")
gateway_key: Final = request.headers.get("x-litellm-api-key", "")
gateway_parts: Final = gateway_key.split()
gateway_token: Final = (
gateway_parts[-1]
if len(gateway_parts) == 1 or (len(gateway_parts) == 2 and gateway_parts[0].lower() == "bearer")
else ""
)
if scheme.lower() != "bearer" or not token.strip() or not gateway_token:
raise HTTPException(
status_code=401,
detail="Send the ChatGPT bearer in Authorization and the LiteLLM key in x-litellm-api-key",
)
if hmac.compare_digest(token.strip().encode(), gateway_token.encode()):
raise HTTPException(status_code=401, detail="ChatGPT and LiteLLM credentials must be separate")
headers: Final = MappingProxyType({name: value for name, value in request.headers.items() if name in _HEADERS})
return await pass_through_request(
request=request,
target=f"{CHATGPT_API_BASE}/{endpoint}",
custom_headers=dict(headers),
user_api_key_dict=user_api_key_dict,
custom_llm_provider="chatgpt",
upstream_headers_filter=lambda _: headers,
)

9
litellm/proxy/pass_through_endpoints/llm_provider_handlers/openai_passthrough_logging_handler.py
View file

@ -10,6 +10,7 @@ from typing import Final
from urllib.parse import urlparse
import httpx
from pydantic import TypeAdapter
import litellm
from litellm._logging import verbose_proxy_logger
@ -91,6 +92,8 @@ def _is_openai_compatible_url(url_route: str | None) -> bool:
hostname: Final = parsed_url.hostname
if not hostname:
return False
if hostname == "chatgpt.com" and parsed_url.path.startswith("/backend-api/codex/"):
return True
if _hostname_matches(hostname, _OPENAI_HOSTNAMES):
return True
if _hostname_matches(hostname, _AZURE_OPENAI_HOSTNAMES):
@ -178,7 +181,7 @@ class OpenAIPassthroughLoggingHandler(BasePassthroughLoggingHandler):
if not url_route:
return False
parsed_url: Final = urlparse(url_route)
return _is_openai_compatible_host(parsed_url.hostname) and (
return _is_openai_compatible_url(url_route) and (
"/v1/responses" in parsed_url.path or "/responses" in parsed_url.path
)
@ -376,7 +379,9 @@ class OpenAIPassthroughLoggingHandler(BasePassthroughLoggingHandler):
) = None
handler_instance: Final = OpenAIPassthroughLoggingHandler()
custom_llm_provider: Final = kwargs.get("custom_llm_provider", "openai")
custom_llm_provider: Final = TypeAdapter(str).validate_python(
kwargs.get("custom_llm_provider", logging_obj.model_call_details.get("custom_llm_provider", "openai"))
)
if is_chat_completions:
# Handle chat completions with existing logic

30
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py
View file

@ -102,6 +102,7 @@ from litellm.proxy.common_utils.sse_keepalive import (
from litellm.proxy.litellm_pre_call_utils import (
LiteLLMProxyRequestSetup,
_get_dynamic_logging_metadata, # pyright: ignore[reportPrivateUsage] # shared proxy helper, same import style as _read_request_body above
redact_credential_headers,
strip_client_guardrail_information,
)
from litellm.proxy.route_llm_request import ProxyModelNotFoundError
@ -382,6 +383,8 @@ class HttpPassThroughEndpointHelpers(BasePassthroughUtils):
@staticmethod
def get_endpoint_type(url: str, custom_llm_provider: str | None = None) -> EndpointType:
parsed_url: Final = urlparse(url)
if custom_llm_provider == "chatgpt":
return EndpointType.OPENAI
if custom_llm_provider == "typesafe" and parsed_url.path.removesuffix("/").endswith("/v1/systemone"):
return EndpointType.DECISIONS
if (
@ -655,7 +658,7 @@ class HttpPassThroughEndpointHelpers(BasePassthroughUtils):
"url": str(request.url),
"method": request.method,
"body": copy.copy(_parsed_body), # use copy instead of deepcopy
"headers": request.headers,
"headers": redact_credential_headers(request.headers),
},
},
"call_type": "pass_through_endpoint",
@ -1026,6 +1029,10 @@ from litellm.passthrough.timeout_utils import (
)
def _unfiltered_headers(headers: Mapping[str, str]) -> Mapping[str, str]:
return headers
async def pass_through_request(
request: Request,
target: str,
@ -1041,6 +1048,7 @@ async def pass_through_request(
custom_llm_provider: str | None = None,
guardrails_config: dict | None = None,
timeout: float | None = None,
upstream_headers_filter: Callable[[Mapping[str, str]], Mapping[str, str]] = _unfiltered_headers,
):
"""
Pass through endpoint handler, makes the httpx request for pass-through endpoints and ensures logging hooks are called
@ -1087,13 +1095,15 @@ async def pass_through_request(
#########################################################
try:
url = httpx.URL(target)
headers = custom_headers
headers = HttpPassThroughEndpointHelpers.forward_headers_from_request(
forwarded_headers: Final = HttpPassThroughEndpointHelpers.forward_headers_from_request(
request_headers=_safe_get_request_headers(request).copy(),
headers=headers,
headers=custom_headers,
forward_headers=forward_headers,
)
upstream_headers: Final = _with_trace_context(headers, parent_span=user_api_key_dict.parent_otel_span)
upstream_headers: Final = _with_trace_context(
upstream_headers_filter(forwarded_headers),
parent_span=user_api_key_dict.parent_otel_span,
)
requested_query_params: dict | None = query_params or dict(request.query_params) or None
@ -1374,7 +1384,7 @@ async def pass_through_request(
additional_args={
"complete_input_dict": _parsed_body,
"api_base": str(logging_url),
"headers": upstream_headers,
"headers": redact_credential_headers(upstream_headers),
},
)
stream = HttpPassThroughEndpointHelpers._update_stream_param_based_on_request_body(
@ -1445,7 +1455,7 @@ async def pass_through_request(
data=_parsed_body or {},
user_api_key_dict=user_api_key_dict,
response=relay_response,
request_headers=dict(request.headers),
request_headers=dict(redact_credential_headers(request.headers)),
)
if callback_headers:
_response_headers.update(callback_headers)
@ -1537,7 +1547,7 @@ async def pass_through_request(
data=_parsed_body or {},
user_api_key_dict=user_api_key_dict,
response=detected_relay_response,
request_headers=dict(request.headers),
request_headers=dict(redact_credential_headers(request.headers)),
)
if callback_headers:
_response_headers.update(callback_headers)
@ -1587,7 +1597,7 @@ async def pass_through_request(
data=_parsed_body or {},
user_api_key_dict=user_api_key_dict,
response=response,
request_headers=dict(request.headers),
request_headers=dict(redact_credential_headers(request.headers)),
)
if relay_callback_headers:
relay_custom_headers.update(relay_callback_headers)
@ -1762,7 +1772,7 @@ async def pass_through_request(
data=_parsed_body or {},
user_api_key_dict=user_api_key_dict,
response=response,
request_headers=dict(request.headers),
request_headers=dict(redact_credential_headers(request.headers)),
)
if callback_headers:
custom_headers.update(callback_headers)

5
litellm/proxy/spend_tracking/savings.py
View file

@ -19,6 +19,7 @@ from pydantic import BaseModel, ConfigDict, Field
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.constants import INTERNAL_CALL_ORIGIN_METADATA_KEY
from litellm.litellm_core_utils.get_llm_provider_logic import resolve_model_and_provider_for_metadata
from litellm.litellm_core_utils.llm_cost_calc.utils import (
_get_cost_per_unit,
calculate_prompt_caching_savings,
@ -68,7 +69,9 @@ def _resolve_model(model: str | None, custom_llm_provider: str | None) -> _Model
if not model:
return None
try:
resolved_model, provider, _, _ = litellm.get_llm_provider(model=model, custom_llm_provider=custom_llm_provider)
resolved_model, provider = resolve_model_and_provider_for_metadata(
model=model, custom_llm_provider=custom_llm_provider
)
except Exception as e: # noqa: BLE001 # get_llm_provider raises for unroutable names; degrade to an unavailable estimate
verbose_proxy_logger.debug(
"savings: cannot resolve provider for model=%s custom_llm_provider=%s (%s)", model, custom_llm_provider, e

52
litellm/responses/main.py
View file

@ -57,7 +57,7 @@ from litellm.secret_managers.main import get_secret_str
from litellm.types.responses.main import *
from litellm.types.responses.streaming_websocket import ResponsesWebSocketRequestDefaults
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import all_litellm_params
from litellm.types.utils import LlmProviders, all_litellm_params
from litellm.utils import (
ProviderConfigManager,
client,
@ -529,6 +529,7 @@ def _will_bridge_to_chat_completions(
use_chat_completions_api: bool,
model_info: object,
api_base: str | None,
litellm_params: GenericLiteLLMParams | None = None,
) -> bool:
"""``_bridges_to_chat_completions`` for callers running before the provider config is resolved.
@ -554,12 +555,15 @@ def provider_stores_the_conversation(model: str, custom_llm_provider: str | None
first MCP round runs as an internal call that writes no spend log at all.
"""
api_base: Final = _api_base_kwarg(kwargs)
litellm_params: Final = GenericLiteLLMParams(**kwargs)
return not _will_bridge_to_chat_completions(
model,
custom_llm_provider or litellm.get_llm_provider(model=model, api_base=api_base)[1],
custom_llm_provider
or litellm.get_llm_provider(model=model, api_base=api_base, litellm_params=litellm_params)[1],
bool(kwargs.get("use_chat_completions_api")),
kwargs.get("model_info"),
api_base,
litellm_params,
)
@ -640,7 +644,12 @@ async def aresponses(
# get custom llm provider so we can use this for mapping exceptions
if custom_llm_provider is None:
_, custom_llm_provider, _, _ = litellm.get_llm_provider(
model=model, api_base=local_vars.get("base_url", None)
model=model,
api_base=local_vars.get("base_url", None),
litellm_params=GenericLiteLLMParams(
extra_headers=extra_headers,
**cast("dict[str, object]", kwargs), # cast-ok: kwargs is the untyped request dict
),
)
# Update local_vars with detected provider (fixes #19782)
local_vars["custom_llm_provider"] = custom_llm_provider
@ -670,6 +679,7 @@ async def aresponses(
bool(kwargs.get("use_chat_completions_api")),
kwargs.get("model_info"),
_api_base_kwarg(kwargs),
GenericLiteLLMParams(**kwargs),
),
):
(
@ -700,7 +710,7 @@ async def aresponses(
original_model=original_model,
swapped_model=model,
custom_llm_provider=custom_llm_provider,
kwargs=kwargs,
kwargs={**kwargs, "extra_headers": extra_headers},
prompt_id=prompt_id,
)
kwargs.pop("prompt_id", None)
@ -784,11 +794,14 @@ def _resolve_prompt_swapped_provider(
kwargs: Mapping[str, object],
prompt_id: str | None,
) -> str:
swapped_provider: Final = litellm.get_llm_provider(model=swapped_model)[1]
litellm_params: Final = GenericLiteLLMParams(**kwargs)
swapped_provider: Final = litellm.get_llm_provider(model=swapped_model, litellm_params=litellm_params)[1]
if kwargs.get("api_key") is None and kwargs.get("api_base") is None:
return swapped_provider
try:
original_provider: Final = custom_llm_provider or litellm.get_llm_provider(model=original_model)[1]
original_provider: Final = (
custom_llm_provider or litellm.get_llm_provider(model=original_model, litellm_params=litellm_params)[1]
)
except litellm.BadRequestError:
return swapped_provider
if swapped_provider == original_provider:
@ -840,6 +853,7 @@ def _apply_prompt_management_to_responses_call(
use_chat_completions_api,
kwargs.get("model_info"),
_api_base_kwarg(kwargs),
GenericLiteLLMParams(**kwargs),
),
):
(
@ -873,7 +887,7 @@ def _apply_prompt_management_to_responses_call(
original_model=original_model,
swapped_model=model,
custom_llm_provider=custom_llm_provider,
kwargs=kwargs,
kwargs={**kwargs, "extra_headers": local_vars.get("extra_headers")},
prompt_id=prompt_id,
)
)
@ -1216,7 +1230,12 @@ def responses(
if custom_llm_provider is None:
_, custom_llm_provider, _, _ = litellm.get_llm_provider(
model=model, api_base=local_vars.get("base_url", None)
model=model,
api_base=local_vars.get("base_url", None),
litellm_params=GenericLiteLLMParams(
extra_headers=extra_headers,
**cast("dict[str, object]", kwargs), # cast-ok: kwargs is the untyped request dict
),
)
local_vars["custom_llm_provider"] = custom_llm_provider
@ -1230,12 +1249,21 @@ def responses(
use_chat_completions_api=use_chat_completions_api,
)
# get llm provider logic
litellm_params: Final = GenericLiteLLMParams(**kwargs)
litellm_params: Final = GenericLiteLLMParams(extra_headers=extra_headers, **kwargs)
provider_headers: Final = (
ResponsesAPIRequestUtils.merge_provider_specific_headers(
extra_headers=extra_headers,
provider_specific_header=getattr(litellm_params, "provider_specific_header", None),
custom_llm_provider=custom_llm_provider,
)
if custom_llm_provider == LlmProviders.CHATGPT.value
else extra_headers
)
local_vars["extra_headers"] = provider_headers
try:
effective_extra_headers: Final = (
add_provider_affinity_header(
headers=extra_headers or MappingProxyType({}),
headers=provider_headers or MappingProxyType({}),
litellm_params=MappingProxyType(
{
"provider_affinity_header": litellm_params.provider_affinity_header,
@ -1247,7 +1275,7 @@ def responses(
),
)
if litellm_params.provider_affinity_header is not None
else extra_headers
else provider_headers
)
except ValueError as affinity_error:
raise litellm.BadRequestError(

22
litellm/responses/utils.py
View file

@ -4,12 +4,13 @@ from collections.abc import Callable, Iterable, Mapping, Sequence
from functools import reduce
from typing import Any, Final, Optional, TypeVar, Union, cast, get_type_hints, overload
from pydantic import BaseModel
from pydantic import BaseModel, TypeAdapter
from typing_extensions import TypeIs # noqa: TID251 # narrows untyped wire payloads without a runtime conversion
import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.dot_notation_indexing import delete_nested_value, is_nested_path
from litellm.litellm_core_utils.get_provider_specific_headers import ProviderSpecificHeaderUtils
from litellm.llms.base_llm.responses.transformation import BaseResponsesAPIConfig
from litellm.types.llms.openai import (
AllMessageValues,
@ -25,11 +26,16 @@ from litellm.types.responses.main import DecodedResponseId
from litellm.types.utils import (
CompletionTokensDetailsWrapper,
PromptTokensDetailsWrapper,
ProviderSpecificHeader,
SpecialEnums,
Usage,
text_tokens_without_nested_reasoning,
)
_PROVIDER_SPECIFIC_HEADER_ADAPTER: Final[
TypeAdapter[ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None]
] = TypeAdapter(ProviderSpecificHeader | Sequence[ProviderSpecificHeader] | None)
def _apply_nested_drop_params(params: dict[str, object], additional_drop_params: list[str] | None) -> dict[str, object]:
nested_paths: Final = tuple(path for path in additional_drop_params or () if is_nested_path(path))
@ -172,6 +178,20 @@ class ResponsesAPIRequestUtils:
**extra_headers,
}
@staticmethod
def merge_provider_specific_headers(
extra_headers: Mapping[str, object] | None,
provider_specific_header: object,
custom_llm_provider: str | None,
) -> dict[str, object] | None: # mutable-ok: legacy Responses request fields require a dict
scoped_headers: Final = ProviderSpecificHeaderUtils.get_provider_specific_headers(
provider_specific_header=_PROVIDER_SPECIFIC_HEADER_ADAPTER.validate_python(provider_specific_header),
custom_llm_provider=custom_llm_provider,
)
if extra_headers is None and not scoped_headers:
return None
return {**(extra_headers or {}), **scoped_headers}
@staticmethod
def _check_valid_arg(
supported_params: list[str] | None,

77
litellm/router.py
View file

@ -84,7 +84,9 @@ from litellm.litellm_core_utils.credential_accessor import CredentialAccessor
from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.litellm_core_utils.get_llm_provider_logic import (
declared_authenticating_provider,
inferred_provider,
is_registered_custom_provider,
resolve_model_and_provider_for_metadata,
)
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLogging
from litellm.litellm_core_utils.llm_cost_calc.utils import SERVICE_TIER_COST_KEY_SUFFIXES
@ -446,6 +448,7 @@ _CallbackT = TypeVar("_CallbackT")
_ALIAS_PARAMS_NEVER_FORWARDED: Final = frozenset({"model", "api_base", "api_key", "api_version"})
_ALIAS_MARKER_FORWARDED_PARAMS_KWARG: Final = "_alias_marker_forwarded_params"
_FORWARDED_CREDENTIAL_ISOLATION_PROVIDERS: Final = frozenset({"chatgpt"})
_CLAUDE_CODE_SESSION_ID_RE: Final = re.compile(r"^[a-zA-Z0-9_\-]{8,}$")
_CLAUDE_CODE_SESSION_ROUTER_TTL_SECONDS: Final = 3600
@ -3969,7 +3972,11 @@ class Router:
kwargs.setdefault(metadata_variable_name, {}).update(metadata_defaults)
def _handle_clientside_credential(
self, deployment: dict, kwargs: dict, function_name: str | None = None
self,
deployment: dict,
kwargs: dict,
function_name: str | None = None,
identity_excluded_params: frozenset[str] = frozenset(),
) -> Deployment:
"""
Build a per-request Deployment carrying the caller-supplied api_key/api_base,
@ -3991,7 +3998,10 @@ class Router:
function_name=function_name,
)
model_group: Final = kwargs.get(metadata_variable_name, {}).get("model_group")
_model_id: Final = self.generate_model_id(model_group=model_group, litellm_params=dynamic_litellm_params)
_model_id: Final = self.generate_model_id(
model_group=model_group,
litellm_params={**dynamic_litellm_params, **dict.fromkeys(identity_excluded_params)},
)
original_model_id: Final = model_info.get("id")
model_info["id"] = _model_id
model_info["original_model_id"] = original_model_id
@ -4045,9 +4055,44 @@ class Router:
deployment_litellm_model_name = deployment["litellm_params"]["model"]
deployment_api_base = deployment["litellm_params"].get("api_base")
deployment_model_name: Final = deployment["model_name"]
if is_clientside_credential(request_kwargs=kwargs):
from litellm.litellm_core_utils.get_provider_specific_headers import ProviderSpecificHeaderUtils
from litellm.llms.chatgpt.common_utils import is_chatgpt_oauth_key
request_context: Final = cast( # cast-ok: the router's legacy request dict has string keys
Mapping[str, object], kwargs
)
raw_deployment_params: Final[object] = deployment["litellm_params"]
deployment_context: Final = _LITELLM_PARAMS_ADAPTER.validate_python(
raw_deployment_params.model_dump(exclude_none=True)
if isinstance(raw_deployment_params, LiteLLM_Params)
else raw_deployment_params
)
declared_provider: Final = deployment_context.get("custom_llm_provider")
declared_model: Final = deployment_context.get("model")
deployment_provider: Final = (
declared_provider
if isinstance(declared_provider, str)
else inferred_provider(declared_model if isinstance(declared_model, str) else None)
)
candidate_authorization: Final = (
ProviderSpecificHeaderUtils.forwarded_authorization(
request_context.get("provider_specific_header"), deployment_provider
)
if deployment_provider in _FORWARDED_CREDENTIAL_ISOLATION_PROVIDERS
else None
)
forwarded_authorization: Final = (
candidate_authorization if is_chatgpt_oauth_key(candidate_authorization) else None
)
if is_clientside_credential(request_kwargs=kwargs) or forwarded_authorization is not None:
deployment_pydantic_obj: Final = self._handle_clientside_credential(
deployment=deployment, kwargs=kwargs, function_name=function_name
deployment=deployment,
kwargs={
**kwargs,
**({"api_key": forwarded_authorization} if forwarded_authorization is not None else {}),
},
function_name=function_name,
identity_excluded_params=frozenset({"api_key"}) if forwarded_authorization is not None else frozenset(),
)
model_info = deployment_pydantic_obj.model_info.model_dump()
deployment_litellm_model_name = deployment_pydantic_obj.litellm_params.model
@ -9673,6 +9718,10 @@ class Router:
custom_llm_provider = None
dynamic_api_key = None
api_base = None
elif declared_authenticating_provider(litellm_model, deployment.litellm_params.custom_llm_provider) is not None:
_model, custom_llm_provider = ( # rebind-ok: each branch resolves the deployment provider
self._model_group_llm_provider(deployment.litellm_params)
)
else:
# check if model provider in supported providers
(
@ -10954,6 +11003,17 @@ class Router:
)
return model_info
@staticmethod
def _model_group_llm_provider(litellm_params: LiteLLM_Params) -> tuple[str, str]:
try:
return resolve_model_and_provider_for_metadata(
model=litellm_params.model,
custom_llm_provider=litellm_params.custom_llm_provider,
)
except litellm.exceptions.BadRequestError as e:
verbose_router_logger.error("litellm.router.py::get_model_group_info() - %s", e)
return "", ""
def _set_model_group_info(self, model_group: str, user_facing_model_group_name: str) -> ModelGroupInfo | None:
"""
For a given model group name, return the combined model info
@ -11038,14 +11098,7 @@ class Router:
deployment_is_mapped = deployment_is_catalog_mapped(model_info, model_info_dict)
# get llm provider
litellm_model, llm_provider = "", ""
try:
litellm_model, llm_provider, _, _ = litellm.get_llm_provider(
model=litellm_params.model,
custom_llm_provider=litellm_params.custom_llm_provider,
)
except litellm.exceptions.BadRequestError as e:
verbose_router_logger.error("litellm.router.py::get_model_group_info() - %s", e)
litellm_model, llm_provider = self._model_group_llm_provider(litellm_params)
if model_info is None:
supported_openai_params = litellm.get_supported_openai_params(

1
litellm/types/litellm_params.py
View file

@ -89,6 +89,7 @@ class ProviderConnection:
litellm_credential_name: str | None = None
configurable_clientside_auth_params: "Sequence[str | ConfigurableClientsideParamsCustomAuth] | None" = None
use_xai_oauth: bool | None = None
chatgpt_auth_file: str | None = None
@dataclass(frozen=True, slots=True, kw_only=True)

4
litellm/types/router.py
View file

@ -426,6 +426,10 @@ class GenericLiteLLMParams(CredentialLiteLLMParams, CustomPricingLiteLLMParams):
default=False,
description="Use stored xAI OAuth credentials when no xAI API key is configured.",
)
chatgpt_auth_file: str | None = Field(
default=None,
description="Path to a ChatGPT OAuth auth.json for this deployment.",
)
model_config = ConfigDict(extra="allow", arbitrary_types_allowed=True)
merge_reasoning_content_in_choices: bool | None = False
model_info: dict | None = None

6
litellm/utils.py
View file

@ -2669,12 +2669,14 @@ def supports_native_streaming(model: str, custom_llm_provider: str | None) -> bo
Raises:
Exception: If the given model is not found in model_prices_and_context_window.json.
"""
from litellm.litellm_core_utils.get_llm_provider_logic import resolve_model_and_provider_for_metadata
try:
model, custom_llm_provider, _, _ = litellm.get_llm_provider(
resolved_model, resolved_provider = resolve_model_and_provider_for_metadata(
model=model, custom_llm_provider=custom_llm_provider
)
model_info: Final = _get_model_info_helper(model=model, custom_llm_provider=custom_llm_provider)
model_info: Final = _get_model_info_helper(model=resolved_model, custom_llm_provider=resolved_provider)
supports_native_streaming = model_info.get("supports_native_streaming", True)
if supports_native_streaming is None:
supports_native_streaming = True

48
tests/unit/litellm_core_utils/test_get_llm_provider_logic.py
View file

@ -1,6 +1,9 @@
from collections.abc import Iterator
from pathlib import Path
from typing import Final
import httpx
import respx
import pytest
import litellm
@ -8,6 +11,7 @@ from litellm import CustomLLM
from litellm.litellm_core_utils import get_llm_provider_logic
from litellm.litellm_core_utils.get_llm_provider_logic import (
get_llm_provider,
resolve_model_and_provider_for_metadata,
inferred_provider,
is_registered_custom_provider,
)
@ -101,3 +105,47 @@ def test_inferred_provider_matches_the_resolver_for_a_bare_model_name() -> None:
@pytest.mark.parametrize("model", ["some-unknown-model-xyz", "", None], ids=["unknown", "empty", "missing"])
def test_inferred_provider_is_none_when_nothing_resolves(model: str | None) -> None:
assert inferred_provider(model) is None
@pytest.fixture
def chatgpt_login_tripwire(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Iterator[None]:
blocked: Final = tmp_path / "blocked"
blocked.write_text("ordinary file")
monkeypatch.setenv("CHATGPT_TOKEN_DIR", str(blocked / "unused"))
monkeypatch.delenv("CHATGPT_AUTH_FILE", raising=False)
with respx.mock as transport:
yield
assert len(transport.calls) == 0
@pytest.mark.parametrize(
("model", "custom_llm_provider", "expected"),
[
("chatgpt/gpt-5.4", None, ("gpt-5.4", "chatgpt")),
("gpt-5.4", "chatgpt", ("gpt-5.4", "chatgpt")),
("github_copilot/gpt-4o", None, ("gpt-4o", "github_copilot")),
("openai/gpt-5.4", None, ("gpt-5.4", "openai")),
],
)
def test_resolve_model_and_provider_for_metadata_never_logs_in(
chatgpt_login_tripwire: None, model: str, custom_llm_provider: str | None, expected: tuple[str, str]
) -> None:
assert resolve_model_and_provider_for_metadata(model, custom_llm_provider) == expected
def test_metadata_callers_never_log_in_for_a_declared_chatgpt_deployment(
chatgpt_login_tripwire: None, monkeypatch: pytest.MonkeyPatch
) -> None:
from litellm.litellm_core_utils.llm_response_utils.get_api_base import get_api_base
from litellm.proxy.spend_tracking.savings import _resolve_model
from litellm.router import Router
from litellm.types.router import LiteLLM_Params
from litellm.utils import supports_native_streaming
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True")
assert supports_native_streaming("chatgpt/gpt-5.4", None) is True
assert get_api_base("chatgpt/gpt-5.4", {"model": "chatgpt/gpt-5.4"}) == "https://chatgpt.com/backend-api/codex"
identity = _resolve_model("chatgpt/gpt-5.4", None)
assert identity is not None and identity.provider == "chatgpt"
assert Router._model_group_llm_provider(LiteLLM_Params(model="chatgpt/gpt-5.4")) == ("gpt-5.4", "chatgpt")

811
tests/unit/llms/chatgpt/chat/test_chatgpt_chat_transformation.py Normal file
View file

@ -0,0 +1,811 @@
import asyncio
import json
from pathlib import Path
from types import MappingProxyType
from typing import Final
from unittest.mock import MagicMock
import httpx
import pytest
import respx
import litellm
from litellm.exceptions import AuthenticationError
from litellm.integrations.dotprompt.dotprompt_manager import DotpromptManager
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.llms.chatgpt.chat.transformation import ChatGPTConfig
from litellm.llms.chatgpt.common_utils import GetAccessTokenError
from litellm.types.router import GenericLiteLLMParams
GATEWAY_BASE = "https://chatgpt.example/backend-api/codex"
def _gateway_authenticator() -> MagicMock:
authenticator = MagicMock()
authenticator.get_api_base.return_value = GATEWAY_BASE
authenticator.get_access_token.return_value = "gateway-access-token"
return authenticator
def _provider_info(
config: ChatGPTConfig, api_base: str | None, api_key: str | None
) -> tuple[str | None, str | None, str]:
return config._get_openai_compatible_provider_info(
model="gpt-5.4", api_base=api_base, api_key=api_key, custom_llm_provider="chatgpt"
)
class TestChatGPTChatProviderInfo:
def test_configured_key_uses_configured_base_without_gateway_login(self):
authenticator = _gateway_authenticator()
config = ChatGPTConfig(authenticator=authenticator)
assert _provider_info(config, "http://127.0.0.1:9999", "configured-key") == (
"http://127.0.0.1:9999",
"configured-key",
"chatgpt",
)
authenticator.get_access_token.assert_not_called()
def test_configured_key_without_base_uses_gateway_base(self):
config = ChatGPTConfig(authenticator=_gateway_authenticator())
assert _provider_info(config, None, "configured-key") == (GATEWAY_BASE, "configured-key", "chatgpt")
def test_gateway_login_is_never_sent_to_a_custom_base(self):
config = ChatGPTConfig(authenticator=_gateway_authenticator())
assert _provider_info(config, "http://attacker.example", None) == (
GATEWAY_BASE,
"gateway-access-token",
"chatgpt",
)
def test_missing_gateway_login_raises_authentication_error(self):
authenticator = _gateway_authenticator()
authenticator.get_access_token.side_effect = GetAccessTokenError(status_code=401, message="not logged in")
config = ChatGPTConfig(authenticator=authenticator)
with pytest.raises(AuthenticationError, match="not logged in"):
_provider_info(config, None, None)
def _write_auth_record(path: Path, token: str, account_id: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
json.dumps(
{
"access_token": token,
"account_id": account_id,
"expires_at": 4102444800,
}
)
)
@pytest.fixture
def auth_files(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> tuple[Path, Path]:
monkeypatch.setattr("time.time", lambda: 1700000000)
default_dir: Final = tmp_path / "default"
monkeypatch.setenv("CHATGPT_TOKEN_DIR", str(default_dir))
monkeypatch.delenv("CHATGPT_AUTH_FILE", raising=False)
_write_auth_record(default_dir / "auth.json", "token-default", "acct-default")
file_a: Final = tmp_path / "account-a" / "auth.json"
file_b: Final = tmp_path / "account-b" / "auth.json"
_write_auth_record(file_a, "token-a", "acct-a")
_write_auth_record(file_b, "token-b", "acct-b")
return file_a, file_b
class TestChatGPTMultiAccountChat:
def test_validate_environment_uses_per_model_auth_file(self, auth_files: tuple[Path, Path]) -> None:
file_a, file_b = auth_files
config: Final = ChatGPTConfig()
headers_a: Final = config.validate_environment(
headers={},
model="gpt-6.1-sol",
messages=[],
optional_params={},
litellm_params={"chatgpt_auth_file": str(file_a)},
api_key="token-default",
)
headers_b: Final = config.validate_environment(
headers={},
model="gpt-6.1-sol",
messages=[],
optional_params={},
litellm_params={"chatgpt_auth_file": str(file_b)},
api_key="token-default",
)
assert headers_a["Authorization"] == "Bearer token-a"
assert headers_a["ChatGPT-Account-Id"] == "acct-a"
assert headers_b["Authorization"] == "Bearer token-b"
assert headers_b["ChatGPT-Account-Id"] == "acct-b"
def test_validate_environment_defaults_to_global_auth_file(self, auth_files: tuple[Path, Path]) -> None:
config: Final = ChatGPTConfig()
headers: Final = config.validate_environment(
headers={},
model="gpt-6.1-sol",
messages=[],
optional_params={},
litellm_params={},
api_key="token-default",
)
assert headers["Authorization"] == "Bearer token-default"
assert headers["ChatGPT-Account-Id"] == "acct-default"
def test_provider_info_uses_per_model_auth_file(self, auth_files: tuple[Path, Path]) -> None:
file_a, _ = auth_files
config: Final = ChatGPTConfig()
_, per_model_key, _ = config._get_openai_compatible_provider_info(
"gpt-6.1-sol", None, None, "chatgpt", {"chatgpt_auth_file": str(file_a)}
)
_, default_key, _ = config._get_openai_compatible_provider_info("gpt-6.1-sol", None, None, "chatgpt")
assert per_model_key == "token-a"
assert default_key == "token-default"
def test_get_llm_provider_threads_auth_file(self, auth_files: tuple[Path, Path]) -> None:
file_a, _ = auth_files
model, provider, dynamic_api_key, _ = get_llm_provider(
model="chatgpt/gpt-6.1-sol",
litellm_params=GenericLiteLLMParams(chatgpt_auth_file=str(file_a)),
)
assert model == "gpt-6.1-sol"
assert provider == "chatgpt"
assert dynamic_api_key == "token-a"
def _subscription_response(request: httpx.Request) -> httpx.Response:
account: Final = request.headers["ChatGPT-Account-Id"]
assert request.headers["Authorization"] in (
f"Bearer token-{account.removeprefix('acct-')}",
f"Bearer {_client_token(account)}",
)
assert "chatgpt_auth_file" not in request.content.decode()
return _responses_stream(account)
def _responses_stream(text: str) -> httpx.Response:
message: Final = {
"id": "msg_test",
"type": "message",
"role": "assistant",
"status": "completed",
"content": [{"type": "output_text", "text": text, "annotations": []}],
}
response: Final = {
"id": "resp_test",
"object": "response",
"created_at": 1700000000,
"status": "completed",
"model": "gpt-6.1-sol",
"output": [message],
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2},
}
events: Final = (
{"type": "response.created", "response": {**response, "status": "in_progress", "output": []}},
{
"type": "response.output_text.delta",
"item_id": "msg_test",
"output_index": 0,
"content_index": 0,
"delta": text,
},
{"type": "response.output_item.done", "output_index": 0, "item": message},
{"type": "response.completed", "response": response},
)
return httpx.Response(
200,
text="".join(f"data: {json.dumps(event)}\n\n" for event in events),
headers={"content-type": "text/event-stream"},
)
@pytest.mark.parametrize("blocked_default", [False, True])
@pytest.mark.parametrize("with_prompt", [False, True])
@pytest.mark.parametrize("endpoint", ["chat", "responses"])
@pytest.mark.parametrize("through_router", [False, True])
@pytest.mark.asyncio
async def test_requests_use_selected_subscription(
auth_files: tuple[Path, Path],
endpoint: str,
through_router: bool,
with_prompt: bool,
blocked_default: bool,
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
default_dir: Final = auth_files[0].parent / "missing-default"
if blocked_default:
default_dir.write_text("ordinary file")
monkeypatch.setenv("CHATGPT_TOKEN_DIR", str(default_dir / "unused"))
if with_prompt:
monkeypatch.setattr(
litellm,
"callbacks",
[
DotpromptManager(
prompt_id="account-check",
prompt_data={"content": "Saved account prompt", "metadata": {"model": "chatgpt/gpt-6.1-sol"}},
)
],
)
router: Final = (
litellm.Router(
model_list=[
{
"model_name": f"account-{index}",
"litellm_params": {"model": "chatgpt/gpt-6.1-sol", "chatgpt_auth_file": str(path)},
}
for index, path in enumerate(auth_files)
]
)
if through_router
else None
)
async def request(index: int, path: Path) -> str:
params: Final = MappingProxyType(
{
"model": f"account-{index}" if router else "chatgpt/gpt-6.1-sol",
**({"chatgpt_auth_file": str(path)} if router is None else {}),
**({"prompt_id": "account-check"} if with_prompt else {}),
}
)
if endpoint == "chat":
result: Final = await (router.acompletion if router else litellm.acompletion)(
**params, messages=[{"role": "user", "content": "Hello"}]
)
return result.choices[0].message.content
response: Final = await (router.aresponses if router else litellm.aresponses)(
**params, input=[{"role": "user", "content": "Hello"}]
)
return response.output[0].content[0].text
with respx.mock as transport:
transport.post("https://chatgpt.com/backend-api/codex/responses").mock(side_effect=_subscription_response)
results: Final = await asyncio.gather(*(request(index, path) for index, path in enumerate(auth_files)))
assert results == ["acct-a", "acct-b"]
if with_prompt:
assert all("Saved account prompt" in call.request.content.decode() for call in transport.calls)
def _client_token(account: str) -> str:
import base64
payload: Final = (
base64.urlsafe_b64encode(json.dumps({"https://api.openai.com/auth": {"chatgpt_account_id": account}}).encode())
.rstrip(b"=")
.decode()
)
return f"header.{payload}.signature"
@pytest.mark.parametrize("endpoint", ["chat", "responses"])
@pytest.mark.parametrize("configured_login", [False, True])
@pytest.mark.parametrize("with_prompt", [False, True])
@pytest.mark.parametrize("stream", [False, True])
@pytest.mark.asyncio
async def test_client_login_keeps_its_account_across_gateway_deployments(
auth_files: tuple[Path, Path],
endpoint: str,
configured_login: bool,
with_prompt: bool,
stream: bool,
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
monkeypatch.setenv("CHATGPT_TOKEN_DIR", str(auth_files[0].parent / "no-gateway-login"))
if with_prompt:
monkeypatch.setattr(
litellm,
"callbacks",
[
DotpromptManager(
prompt_id="client-account-check",
prompt_data={"content": "Saved client prompt", "metadata": {"model": "chatgpt/gpt-6.1-sol"}},
)
],
)
router: Final = litellm.Router(
model_list=[
{
"model_name": "subscriptions",
"litellm_params": {
"model": "chatgpt/gpt-6.1-sol",
**({"chatgpt_auth_file": str(path)} if configured_login else {}),
},
}
for path in auth_files
],
num_retries=0,
)
def reply(request: httpx.Request) -> httpx.Response:
account: Final = request.headers["ChatGPT-Account-Id"]
assert request.headers["Authorization"] == f"Bearer {_client_token(account)}"
return _subscription_response(
httpx.Request(
"POST",
request.url,
content=request.content,
headers={
**{key: value for key, value in request.headers.items() if key.lower() != "authorization"},
"Authorization": f"Bearer token-{account.removeprefix('acct-')}",
},
)
)
async def call(account: str) -> str:
params: Final = MappingProxyType(
{
"model": "subscriptions",
"stream": stream,
**({"prompt_id": "client-account-check"} if with_prompt else {}),
"provider_specific_header": {
"custom_llm_provider": "chatgpt",
"extra_headers": {"Authorization": f"Bearer {_client_token(account)}"},
},
}
)
response: Final = await (
router.acompletion(**params, messages=[{"role": "user", "content": "Hello"}])
if endpoint == "chat"
else router.aresponses(**params, input=[{"role": "user", "content": "Hello"}])
)
if stream:
chunks: Final = [part async for part in response]
return (
"".join(part.choices[0].delta.content or "" for part in chunks)
if endpoint == "chat"
else "".join(
getattr(part, "delta", "")
for part in chunks
if getattr(part, "type", "") == "response.output_text.delta"
)
)
return response.choices[0].message.content if endpoint == "chat" else response.output[0].content[0].text
with respx.mock as transport:
transport.post("https://chatgpt.com/backend-api/codex/responses").mock(side_effect=reply)
results: Final = await asyncio.gather(call("acct-client-a"), call("acct-client-b"))
assert results == ["acct-client-a", "acct-client-b"]
assert len(transport.calls) == 2
@pytest.mark.asyncio
async def test_one_client_limit_does_not_cool_down_other_subscriptions(
auth_files: tuple[Path, Path], monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
router: Final = litellm.Router(
model_list=[
{"model_name": "pool", "litellm_params": {"model": "chatgpt/gpt-6.1-sol", "chatgpt_auth_file": str(path)}}
for path in auth_files
],
num_retries=0,
allowed_fails=0,
cooldown_time=60,
)
headers: Final = {
"custom_llm_provider": "chatgpt",
"extra_headers": {"Authorization": f"Bearer {_client_token('acct-limited')}"},
}
with respx.mock as transport:
route: Final = transport.post("https://chatgpt.com/backend-api/codex/responses")
route.respond(
429,
headers={"retry-after": "60"},
json={"error": {"message": "Usage limit reached", "type": "rate_limit_error"}},
)
for _ in range(2):
with pytest.raises(litellm.RateLimitError):
await router.aresponses(model="pool", input="Hello", provider_specific_header=headers)
await asyncio.sleep(0)
route.mock(side_effect=_subscription_response)
response: Final = await router.aresponses(model="pool", input="Hello")
assert response.output[0].content[0].text in ("acct-a", "acct-b")
def _forwarded_login(authorization: str) -> dict[str, object]:
return {"custom_llm_provider": "chatgpt", "extra_headers": {"Authorization": authorization}}
@pytest.mark.asyncio
async def test_distinct_client_logins_do_not_grow_the_cost_map(
auth_files: tuple[Path, Path], monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
monkeypatch.setattr(litellm, "model_cost", dict(litellm.model_cost))
router: Final = litellm.Router(
model_list=[
{
"model_name": "pool",
"litellm_params": {"model": "chatgpt/gpt-6.1-sol", "chatgpt_auth_file": str(auth_files[0])},
}
],
num_retries=0,
)
accounts: Final = tuple(f"acct-client-{index}" for index in range(5))
baseline: Final = frozenset(litellm.model_cost)
with respx.mock as transport:
transport.post("https://chatgpt.com/backend-api/codex/responses").mock(side_effect=_subscription_response)
texts: Final = [
(
await router.aresponses(
model="pool",
input="Hello",
provider_specific_header=_forwarded_login(f"Bearer {_client_token(account)}"),
)
)
.output[0]
.content[0]
.text
for account in accounts
]
assert texts == list(accounts)
assert len(frozenset(litellm.model_cost) - baseline) <= 1
@pytest.mark.asyncio
async def test_one_client_limit_does_not_block_another_client_login(
auth_files: tuple[Path, Path], monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
router: Final = litellm.Router(
model_list=[
{
"model_name": "pool",
"litellm_params": {"model": "chatgpt/gpt-6.1-sol", "chatgpt_auth_file": str(auth_files[0])},
}
],
num_retries=0,
allowed_fails=0,
cooldown_time=60,
)
def limit_one_client(request: httpx.Request) -> httpx.Response:
if request.headers["ChatGPT-Account-Id"] == "acct-limited":
return httpx.Response(
429,
headers={"retry-after": "60"},
json={"error": {"message": "Usage limit reached", "type": "rate_limit_error"}},
)
return _subscription_response(request)
with respx.mock as transport:
transport.post("https://chatgpt.com/backend-api/codex/responses").mock(side_effect=limit_one_client)
for _ in range(2):
with pytest.raises(litellm.RateLimitError):
await router.aresponses(
model="pool",
input="Hello",
provider_specific_header=_forwarded_login(f"Bearer {_client_token('acct-limited')}"),
)
await asyncio.sleep(0)
response: Final = await router.aresponses(
model="pool",
input="Hello",
provider_specific_header=_forwarded_login(f"Bearer {_client_token('acct-other')}"),
)
assert response.output[0].content[0].text == "acct-other"
@pytest.mark.asyncio
async def test_forwarded_bearer_that_is_not_a_chatgpt_login_keeps_the_deployment_login(
auth_files: tuple[Path, Path], monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
monkeypatch.setattr(litellm, "model_cost", dict(litellm.model_cost))
router: Final = litellm.Router(
model_list=[
{
"model_name": "pool",
"litellm_params": {"model": "chatgpt/gpt-6.1-sol", "chatgpt_auth_file": str(auth_files[0])},
}
],
num_retries=0,
)
baseline: Final = frozenset(litellm.model_cost)
with respx.mock as transport:
route: Final = transport.post("https://chatgpt.com/backend-api/codex/responses").mock(
side_effect=_subscription_response
)
response: Final = await router.aresponses(
model="pool", input="Hello", provider_specific_header=_forwarded_login("Bearer sk-not-a-chatgpt-login")
)
assert response.output[0].content[0].text == "acct-a"
assert route.calls.last.request.headers["Authorization"] == "Bearer token-a"
assert frozenset(litellm.model_cost) == baseline
def _attacker_reply(request: httpx.Request) -> httpx.Response:
return _responses_stream("attacker")
@pytest.mark.parametrize("endpoint", ["chat", "responses"])
@pytest.mark.parametrize("caller_key", [None, "caller-own-key"])
@pytest.mark.asyncio
async def test_caller_endpoint_override_never_receives_the_saved_login(
auth_files: tuple[Path, Path], monkeypatch: pytest.MonkeyPatch, endpoint: str, caller_key: str | None
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
router: Final = litellm.Router(
model_list=[
{
"model_name": "pool",
"litellm_params": {"model": "chatgpt/gpt-6.1-sol", "chatgpt_auth_file": str(auth_files[0])},
}
],
num_retries=0,
)
override: Final = {"api_base": "https://attacker.example/v1", **({"api_key": caller_key} if caller_key else {})}
with respx.mock(assert_all_called=False) as transport:
attacker: Final = transport.route(host="attacker.example").mock(side_effect=_attacker_reply)
transport.post("https://chatgpt.com/backend-api/codex/responses").mock(side_effect=_subscription_response)
if endpoint == "chat":
await router.acompletion(model="pool", messages=[{"role": "user", "content": "Hello"}], **override)
else:
await router.aresponses(model="pool", input="Hello", **override)
assert attacker.calls.call_count == 0
@pytest.mark.parametrize("endpoint", ["chat", "responses"])
@pytest.mark.parametrize("auth_file", [False, True])
@pytest.mark.asyncio
async def test_saved_login_only_goes_to_the_chatgpt_endpoint(
auth_files: tuple[Path, Path], monkeypatch: pytest.MonkeyPatch, endpoint: str, auth_file: bool
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
params: Final = {
"model": "chatgpt/gpt-6.1-sol",
"api_base": "https://attacker.example/v1",
**({"chatgpt_auth_file": str(auth_files[0]), "api_key": "caller-dummy-key"} if auth_file else {}),
}
with respx.mock(assert_all_called=False) as transport:
attacker: Final = transport.route(host="attacker.example").mock(side_effect=_attacker_reply)
transport.post("https://chatgpt.com/backend-api/codex/responses").mock(side_effect=_subscription_response)
text: Final = (
(await litellm.acompletion(**params, messages=[{"role": "user", "content": "Hello"}]))
.choices[0]
.message.content
if endpoint == "chat"
else (await litellm.aresponses(**params, input="Hello")).output[0].content[0].text
)
assert attacker.calls.call_count == 0
assert text == ("acct-a" if auth_file else "acct-default")
@pytest.mark.parametrize("endpoint", ["chat", "responses"])
@pytest.mark.parametrize("async_call", [False, True])
@pytest.mark.parametrize("direct_headers", [False, True])
@pytest.mark.parametrize("configured_key", [False, True])
@pytest.mark.parametrize(
"target", ["https://example.com/v1", "http://chatgpt.com/backend-api/codex", "https://chatgpt.com.example.com/v1"]
)
@pytest.mark.asyncio
async def test_client_login_cannot_be_redirected_to_another_host(
endpoint: str, target: str, async_call: bool, direct_headers: bool, configured_key: bool
) -> None:
headers: Final = {"Authorization": f"Bearer {_client_token('acct-client')}"}
credential: Final = (
{"extra_headers": headers}
if direct_headers
else {"provider_specific_header": {"custom_llm_provider": "chatgpt", "extra_headers": headers}}
)
params: Final = MappingProxyType(
{
"model": "chatgpt/gpt-6.1-sol",
"api_base": target,
**({"api_key": "configured-test-key"} if configured_key else {}),
**credential,
}
)
async def request() -> None:
if endpoint == "chat":
if async_call:
await litellm.acompletion(**params, messages=[{"role": "user", "content": "Hello"}])
else:
litellm.completion(**params, messages=[{"role": "user", "content": "Hello"}])
elif async_call:
await litellm.aresponses(**params, input="Hello")
else:
litellm.responses(**params, input="Hello")
with respx.mock as transport:
with pytest.raises(litellm.BadRequestError, match="official HTTPS ChatGPT endpoint"):
await request()
assert len(transport.calls) == 0
@pytest.mark.asyncio
async def test_cross_provider_fallback_does_not_receive_the_chatgpt_login(
auth_files: tuple[Path, Path], monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
router: Final = litellm.Router(
model_list=[
{
"model_name": "subscription",
"litellm_params": {"model": "chatgpt/gpt-6.1-sol", "chatgpt_auth_file": str(auth_files[0])},
},
{
"model_name": "api-fallback",
"litellm_params": {"model": "openai/gpt-6.1-sol", "api_key": "sk-example-fallback"},
},
],
fallbacks=[{"subscription": ["api-fallback"]}],
num_retries=0,
)
with respx.mock as transport:
transport.post("https://chatgpt.com/backend-api/codex/responses").respond(
429, json={"error": {"message": "Usage limit reached", "type": "rate_limit_error"}}
)
fallback: Final = transport.post("https://api.openai.com/v1/responses").respond(
200,
json={
"id": "resp_fallback",
"object": "response",
"created_at": 1700000000,
"status": "completed",
"model": "gpt-6.1-sol",
"output": [
{
"id": "msg_fallback",
"type": "message",
"role": "assistant",
"status": "completed",
"content": [{"type": "output_text", "text": "fallback", "annotations": []}],
}
],
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2},
},
)
response: Final = await router.aresponses(
model="subscription",
input="Hello",
provider_specific_header={
"custom_llm_provider": "chatgpt",
"extra_headers": {
"Authorization": f"Bearer {_client_token('acct-client')}",
"ChatGPT-Account-Id": "acct-client",
},
},
)
assert response.output[0].content[0].text == "fallback"
assert fallback.calls.last.request.headers["Authorization"] == "Bearer sk-example-fallback"
assert "ChatGPT-Account-Id" not in fallback.calls.last.request.headers
@pytest.mark.parametrize("endpoint", ["chat", "responses"])
@pytest.mark.parametrize("async_call", [False, True])
@pytest.mark.parametrize("with_prompt", [False, True])
@pytest.mark.asyncio
async def test_sdk_extra_headers_use_client_login_without_gateway_auth(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, endpoint: str, async_call: bool, with_prompt: bool
) -> None:
blocked: Final = tmp_path / "blocked"
blocked.write_text("ordinary file")
monkeypatch.setenv("CHATGPT_TOKEN_DIR", str(blocked / "unused"))
monkeypatch.delenv("CHATGPT_AUTH_FILE", raising=False)
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
headers: Final = {"Authorization": f"Bearer {_client_token('acct-client-a')}"}
prompt_params: Final = {"prompt_id": "sdk-client"} if with_prompt else {}
if with_prompt:
monkeypatch.setattr(
litellm,
"callbacks",
[
DotpromptManager(
prompt_id="sdk-client",
prompt_data={"content": "Hello", "metadata": {"model": "chatgpt/gpt-6-luna"}},
)
],
)
with respx.mock as transport:
transport.post("https://chatgpt.com/backend-api/codex/responses").mock(side_effect=_subscription_response)
if endpoint == "chat":
params: Final = {
"model": "chatgpt/gpt-6.1-sol",
"messages": [{"role": "user", "content": "Hello"}],
"extra_headers": headers,
**prompt_params,
}
result: Final = await litellm.acompletion(**params) if async_call else litellm.completion(**params)
assert result.choices[0].message.content == "acct-client-a"
else:
response: Final = (
await litellm.aresponses(
model="chatgpt/gpt-6.1-sol", input="Hello", extra_headers=headers, **prompt_params
)
if async_call
else litellm.responses(
model="chatgpt/gpt-6.1-sol", input="Hello", extra_headers=headers, **prompt_params
)
)
assert response.output[0].content[0].text == "acct-client-a"
assert transport.calls.last.request.headers["Authorization"] == headers["Authorization"]
assert len(transport.calls) == 1
@pytest.mark.parametrize("endpoint", ["chat", "responses"])
@pytest.mark.parametrize("sync", [False, True])
@pytest.mark.parametrize("credential_source", ["default", "file", "api_key"])
@pytest.mark.parametrize("authorization_header", [None, "Authorization", "authorization"])
@pytest.mark.asyncio
async def test_requests_preserve_gateway_token_and_selected_workspace(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
endpoint: str,
sync: bool,
credential_source: str,
authorization_header: str | None,
) -> None:
monkeypatch.setattr("time.time", lambda: 1700000000)
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
monkeypatch.setenv("CHATGPT_AUTH_FILE", str(tmp_path / "default.json"))
token: Final = _client_token("acct-token-default")
auth_file: Final = tmp_path / ("selected.json" if credential_source == "file" else "default.json")
_write_auth_record(auth_file, token, "acct-saved-workspace")
configured_token: Final = _client_token("acct-configured")
expected_token: Final = configured_token if credential_source == "api_key" else token
expected_account: Final = "acct-configured" if credential_source == "api_key" else "acct-saved-workspace"
def reply(request: httpx.Request) -> httpx.Response:
assert request.headers.get_list("authorization") == [f"Bearer {expected_token}"]
assert request.headers.get_list("chatgpt-account-id") == [expected_account]
assert request.headers["originator"] == "codex_cli_rs"
return _subscription_response(
httpx.Request(
"POST",
request.url,
content=request.content,
headers={
"Authorization": f"Bearer token-{expected_account.removeprefix('acct-')}",
"ChatGPT-Account-Id": expected_account,
},
)
)
params: Final = MappingProxyType(
{
"model": "chatgpt/gpt-6.1-sol",
**({"chatgpt_auth_file": str(auth_file)} if credential_source == "file" else {}),
**({"api_key": configured_token} if credential_source == "api_key" else {}),
"extra_headers": {
"originator": "codex_cli_rs",
**({authorization_header: "Bearer sk-unrelated-example"} if authorization_header else {}),
},
}
)
with respx.mock as transport:
transport.post("https://chatgpt.com/backend-api/codex/responses").mock(side_effect=reply)
if endpoint == "chat":
result: Final = (
await asyncio.to_thread(litellm.completion, **params, messages=[{"role": "user", "content": "Hello"}])
if sync
else await litellm.acompletion(**params, messages=[{"role": "user", "content": "Hello"}])
)
assert result.choices[0].message.content == expected_account
else:
response: Final = (
await asyncio.to_thread(litellm.responses, **params, input=[{"role": "user", "content": "Hello"}])
if sync
else await litellm.aresponses(**params, input=[{"role": "user", "content": "Hello"}])
)
assert response.output[0].content[0].text == expected_account

174
tests/unit/llms/chatgpt/responses/test_chatgpt_responses_transformation.py
View file

@ -4,14 +4,18 @@ Tests for ChatGPT subscription Responses API transformation
Source: litellm/llms/chatgpt/responses/transformation.py
"""
import base64
import json
from collections.abc import Generator
from pathlib import Path
from typing import Final
from unittest.mock import MagicMock, patch
import httpx
import pytest
import litellm
from litellm.llms.chatgpt.authenticator import Authenticator
from litellm.llms.chatgpt.responses.transformation import ChatGPTResponsesAPIConfig
from litellm.llms.openai.common_utils import OpenAIError
from litellm.main import responses_api_bridge_check
@ -92,10 +96,12 @@ class TestChatGPTResponsesAPITransformation:
url = config.get_complete_url(api_base=None, litellm_params={})
assert url == "https://chatgpt.example.com/responses"
custom_url = config.get_complete_url(
api_base="https://custom.chatgpt.com", litellm_params={}
configured_key_url = config.get_complete_url(
api_base="https://custom.chatgpt.com", litellm_params={"api_key": "configured-key"}
)
assert custom_url == "https://custom.chatgpt.com/responses"
saved_login_url = config.get_complete_url(api_base="https://custom.chatgpt.com", litellm_params={})
assert configured_key_url == "https://custom.chatgpt.com/responses"
assert saved_login_url == "https://chatgpt.example.com/responses"
url_with_slash = config.get_complete_url(
api_base="https://chatgpt.example.com/", litellm_params={}
@ -364,3 +370,165 @@ class TestChatGPTResponsesAPITransformation:
assert "ChatGPT upstream failed" in str(exc_info.value)
assert exc_info.value.status_code == 502
def _fake_chatgpt_oauth_token(account_id: str) -> str:
header = base64.urlsafe_b64encode(b'{"alg":"RS256"}').rstrip(b"=").decode()
payload = (
base64.urlsafe_b64encode(
json.dumps({"https://api.openai.com/auth": {"chatgpt_account_id": account_id}}).encode()
)
.rstrip(b"=")
.decode()
)
return f"{header}.{payload}.fake-signature"
def _gateway_authenticator() -> MagicMock:
authenticator = MagicMock()
authenticator.get_access_token.return_value = "gateway-access-token"
authenticator.get_account_id.return_value = "acct-gateway"
return authenticator
def _config_with(authenticator: MagicMock) -> ChatGPTResponsesAPIConfig:
return ChatGPTResponsesAPIConfig(authenticator=authenticator)
class TestChatGPTClientCredentialForwarding:
def test_client_chatgpt_oauth_wins_over_configured_key_and_gateway_login(self):
authenticator = _gateway_authenticator()
token = _fake_chatgpt_oauth_token("acct-alice")
headers = _config_with(authenticator).validate_environment(
headers={"Authorization": f"Bearer {token}", "originator": "codex_cli_rs"},
model="gpt-5.4",
litellm_params=GenericLiteLLMParams(api_key="configured-gateway-key", litellm_session_id="session-1"),
)
assert headers["Authorization"] == f"Bearer {token}"
assert headers["ChatGPT-Account-Id"] == "acct-alice"
assert headers["originator"] == "codex_cli_rs"
assert headers["accept"] == "text/event-stream"
assert [name for name in headers if name.lower() == "authorization"] == ["Authorization"]
authenticator.get_access_token.assert_not_called()
def test_explicit_account_header_wins_over_jwt_claim(self):
token = _fake_chatgpt_oauth_token("acct-from-jwt")
headers = _config_with(_gateway_authenticator()).validate_environment(
headers={"authorization": f"Bearer {token}", "chatgpt-account-id": "acct-from-header"},
model="gpt-5.4",
litellm_params=GenericLiteLLMParams(),
)
assert headers["ChatGPT-Account-Id"] == "acct-from-header"
assert [name for name in headers if name.lower() == "chatgpt-account-id"] == ["ChatGPT-Account-Id"]
@pytest.mark.parametrize(
"bearer", ["Bearer sk-proj-fake-platform-key", "Bearer not-a-jwt", "Bearer a.bm90LWpzb24.c"]
)
def test_non_chatgpt_bearer_falls_back_to_configured_key(self, bearer, tmp_path):
authenticator = Authenticator(auth_file=str(tmp_path / "missing-default.json"))
headers = ChatGPTResponsesAPIConfig(authenticator=authenticator).validate_environment(
headers={"Authorization": bearer},
model="gpt-5.4",
litellm_params=GenericLiteLLMParams(api_key="configured-gateway-key"),
)
assert "ChatGPT-Account-Id" not in headers
assert headers["Authorization"] == "Bearer configured-gateway-key"
def test_configured_key_is_used_without_client_credential(self):
authenticator = _gateway_authenticator()
headers = _config_with(authenticator).validate_environment(
headers={},
model="gpt-5.4",
litellm_params=GenericLiteLLMParams(api_key="configured-gateway-key"),
)
assert headers["Authorization"] == "Bearer configured-gateway-key"
authenticator.get_access_token.assert_not_called()
def test_gateway_login_is_used_without_client_credential_or_configured_key(self):
headers = _config_with(_gateway_authenticator()).validate_environment(
headers={},
model="gpt-5.4",
litellm_params=GenericLiteLLMParams(),
)
assert headers["Authorization"] == "Bearer gateway-access-token"
assert headers["ChatGPT-Account-Id"] == "acct-gateway"
def test_two_clients_never_share_a_credential(self):
config = _config_with(_gateway_authenticator())
alice = _fake_chatgpt_oauth_token("acct-alice")
bob = _fake_chatgpt_oauth_token("acct-bob")
forwarded = [
config.validate_environment(
headers={"Authorization": f"Bearer {token}"}, model="gpt-5.4", litellm_params=GenericLiteLLMParams()
)
for token in (alice, bob, alice)
]
assert [h["Authorization"] for h in forwarded] == [f"Bearer {alice}", f"Bearer {bob}", f"Bearer {alice}"]
assert [h["ChatGPT-Account-Id"] for h in forwarded] == ["acct-alice", "acct-bob", "acct-alice"]
class TestChatGPTResponsesMultiAccount:
@pytest.fixture(autouse=True)
def fixed_clock(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr("time.time", lambda: 1700000000)
@staticmethod
def _write_auth_record(path: Path, token: str, account_id: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
json.dumps(
{
"access_token": token,
"account_id": account_id,
"expires_at": 4102444800,
}
)
)
def test_validate_environment_uses_per_model_auth_file(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
default_dir: Final = tmp_path / "default"
monkeypatch.setenv("CHATGPT_TOKEN_DIR", str(default_dir))
monkeypatch.delenv("CHATGPT_AUTH_FILE", raising=False)
self._write_auth_record(default_dir / "auth.json", "token-default", "acct-default")
file_a: Final = tmp_path / "account-a" / "auth.json"
file_b: Final = tmp_path / "account-b" / "auth.json"
self._write_auth_record(file_a, "token-a", "acct-a")
self._write_auth_record(file_b, "token-b", "acct-b")
config: Final = ChatGPTResponsesAPIConfig()
headers_a: Final = config.validate_environment(
headers={},
model="gpt-6.1-sol",
litellm_params=GenericLiteLLMParams(chatgpt_auth_file=str(file_a)),
)
headers_b: Final = config.validate_environment(
headers={},
model="gpt-6.1-sol",
litellm_params=GenericLiteLLMParams(chatgpt_auth_file=str(file_b)),
)
headers_default: Final = config.validate_environment(
headers={},
model="gpt-6.1-sol",
litellm_params=GenericLiteLLMParams(),
)
assert headers_a["Authorization"] == "Bearer token-a"
assert headers_a["ChatGPT-Account-Id"] == "acct-a"
assert headers_b["Authorization"] == "Bearer token-b"
assert headers_b["ChatGPT-Account-Id"] == "acct-b"
assert headers_default["Authorization"] == "Bearer token-default"
assert headers_default["ChatGPT-Account-Id"] == "acct-default"

148
tests/unit/llms/chatgpt/test_chatgpt_authenticator.py
View file

@ -1,11 +1,16 @@
import base64
import json
import time
from pathlib import Path
from typing import Final
from unittest.mock import mock_open, patch
import pytest
import httpx
import respx
from litellm.llms.chatgpt.authenticator import Authenticator
from litellm.llms.chatgpt.authenticator import Authenticator, get_cached_authenticator, get_chatgpt_auth_file
from litellm.types.router import GenericLiteLLMParams
def _make_jwt(payload: dict) -> str:
@ -68,3 +73,144 @@ class TestChatGPTAuthenticator:
assert account_id == "acct-123"
mock_write.assert_called_once()
assert mock_write.call_args[0][0]["account_id"] == "acct-123"
def test_get_account_id_prefers_stored_value_without_rewriting_file(self, authenticator):
id_token = _make_jwt({"https://api.openai.com/auth": {"chatgpt_account_id": "acct-from-token"}})
auth_data = json.dumps({"account_id": "acct-stored", "id_token": id_token})
with (
patch("builtins.open", mock_open(read_data=auth_data)),
patch.object(authenticator, "_write_auth_file") as mock_write,
):
assert authenticator.get_account_id() == "acct-stored"
mock_write.assert_not_called()
def test_get_account_id_without_login_returns_none(self, authenticator):
with (
patch("builtins.open", mock_open(read_data="{}")),
patch.object(authenticator, "_write_auth_file") as mock_write,
):
assert authenticator.get_account_id() is None
mock_write.assert_not_called()
def _write_auth_record(path: Path, token: str, account_id: str) -> None:
path.write_text(
json.dumps(
{
"access_token": token,
"account_id": account_id,
"expires_at": 4102444800,
}
)
)
class TestChatGPTMultiAccountAuthenticator:
@pytest.fixture(autouse=True)
def fixed_clock(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr("time.time", lambda: 1700000000)
def test_explicit_auth_file_overrides_env(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CHATGPT_TOKEN_DIR", str(tmp_path / "env-dir"))
monkeypatch.setenv("CHATGPT_AUTH_FILE", "env.json")
custom_file: Final = tmp_path / "account-a" / "auth.json"
authenticator: Final = Authenticator(auth_file=str(custom_file))
assert authenticator.auth_file == str(custom_file)
assert authenticator.token_dir == str(custom_file.parent)
assert not custom_file.parent.exists()
authenticator._write_auth_file({"access_token": "token-a"})
assert json.loads(custom_file.read_text())["access_token"] == "token-a"
def test_default_auth_file_from_env(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CHATGPT_TOKEN_DIR", str(tmp_path / "env-dir"))
monkeypatch.setenv("CHATGPT_AUTH_FILE", "env.json")
authenticator: Final = Authenticator()
assert authenticator.auth_file == str(tmp_path / "env-dir" / "env.json")
def test_bare_filename_auth_file_resolves_against_cwd(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.chdir(tmp_path)
authenticator: Final = Authenticator(auth_file="auth.json")
assert authenticator.auth_file == str(tmp_path / "auth.json")
assert authenticator.token_dir == str(tmp_path)
def test_two_auth_files_are_isolated(self, tmp_path: Path) -> None:
file_a: Final = tmp_path / "account-a.json"
file_b: Final = tmp_path / "account-b.json"
_write_auth_record(file_a, "token-a", "acct-a")
_write_auth_record(file_b, "token-b", "acct-b")
authenticator_a: Final = Authenticator(auth_file=str(file_a))
authenticator_b: Final = Authenticator(auth_file=str(file_b))
assert authenticator_a.get_access_token() == "token-a"
assert authenticator_a.get_account_id() == "acct-a"
assert authenticator_b.get_access_token() == "token-b"
assert authenticator_b.get_account_id() == "acct-b"
def test_get_cached_authenticator_reuses_instance_per_path(self, tmp_path: Path) -> None:
file_a: Final = tmp_path / "account-a.json"
file_b: Final = tmp_path / "account-b.json"
authenticator_a: Final = get_cached_authenticator(str(file_a))
assert get_cached_authenticator(str(file_a)) is authenticator_a
assert get_cached_authenticator(str(file_b)) is not authenticator_a
def test_get_chatgpt_auth_file(self) -> None:
assert get_chatgpt_auth_file(None) is None
assert get_chatgpt_auth_file({}) is None
assert get_chatgpt_auth_file({"chatgpt_auth_file": ""}) is None
assert get_chatgpt_auth_file({"chatgpt_auth_file": "/a/auth.json"}) == "/a/auth.json"
assert get_chatgpt_auth_file(GenericLiteLLMParams()) is None
assert get_chatgpt_auth_file(GenericLiteLLMParams(chatgpt_auth_file="/b/auth.json")) == "/b/auth.json"
def test_relative_and_absolute_auth_file_share_cached_authenticator(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.chdir(tmp_path)
get_cached_authenticator.cache_clear()
auth_file: Final = tmp_path / "x" / "auth.json"
auth_file.parent.mkdir(parents=True)
auth_file.write_text("{}")
relative_key: Final = get_chatgpt_auth_file({"chatgpt_auth_file": "x/auth.json"})
absolute_key: Final = get_chatgpt_auth_file({"chatgpt_auth_file": str(auth_file)})
assert relative_key is not None and absolute_key is not None
assert get_cached_authenticator(relative_key) is get_cached_authenticator(absolute_key)
def test_refresh_keeps_saved_workspace(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr("time.time", lambda: 1700000000)
auth_file: Final = tmp_path / "selected.json"
auth_file.write_text(
json.dumps(
{
"access_token": "expired-example-token",
"refresh_token": "example-refresh-token",
"expires_at": 1,
"account_id": "acct-selected-workspace",
}
)
)
token: Final = _make_jwt({"exp": 4102444800, "https://api.openai.com/auth": {"chatgpt_account_id": "acct-default"}})
authenticator: Final = Authenticator(auth_file=str(auth_file))
with respx.mock as transport:
transport.post("https://auth.openai.com/oauth/token").mock(
return_value=httpx.Response(
200,
json={"access_token": token, "id_token": token, "refresh_token": "example-rotated-refresh"},
)
)
assert authenticator.get_access_token() == token
assert authenticator.get_account_id(token) == "acct-selected-workspace"
assert json.loads(auth_file.read_text())["refresh_token"] == "example-rotated-refresh"

90
tests/unit/llms/chatgpt/test_chatgpt_common_utils.py Normal file
View file

@ -0,0 +1,90 @@
import base64
import json
import pytest
from litellm.llms.chatgpt.common_utils import (
extract_chatgpt_account_id,
get_chatgpt_client_credential,
is_chatgpt_oauth_key,
select_chatgpt_client_credential_headers,
)
def _make_jwt(payload: dict) -> str:
def _b64(obj: dict) -> str:
raw = json.dumps(obj, separators=(",", ":")).encode("utf-8")
return base64.urlsafe_b64encode(raw).decode("utf-8").rstrip("=")
return f"{_b64({'alg': 'none', 'typ': 'JWT'})}.{_b64(payload)}."
CHATGPT_JWT = _make_jwt({"https://api.openai.com/auth": {"chatgpt_account_id": "acct-alice"}})
CHATGPT_JWT_WITHOUT_ACCOUNT = _make_jwt({"https://api.openai.com/auth": {"user_id": "user-1"}})
NON_CHATGPT_JWT = _make_jwt({"sub": "someone", "https://api.openai.com/auth": "not-an-object"})
class TestExtractChatGPTAccountId:
def test_reads_account_id_from_auth_claim(self):
assert extract_chatgpt_account_id(CHATGPT_JWT) == "acct-alice"
@pytest.mark.parametrize(
"token",
[None, "", "sk-proj-not-a-jwt", "header.only", NON_CHATGPT_JWT, CHATGPT_JWT_WITHOUT_ACCOUNT],
)
def test_returns_none_without_a_usable_account_claim(self, token):
assert extract_chatgpt_account_id(token) is None
class TestIsChatGPTOAuthKey:
def test_recognizes_bearer_and_raw_chatgpt_jwt(self):
assert is_chatgpt_oauth_key(CHATGPT_JWT)
assert is_chatgpt_oauth_key(f"Bearer {CHATGPT_JWT}")
assert is_chatgpt_oauth_key(f"Bearer {CHATGPT_JWT_WITHOUT_ACCOUNT}")
@pytest.mark.parametrize(
"value", [None, "", "Bearer sk-proj-platform-key", "Bearer not.a.jwt", _make_jwt({"sub": "x"})]
)
def test_rejects_non_chatgpt_credentials(self, value):
assert not is_chatgpt_oauth_key(value)
class TestSelectChatGPTClientCredentialHeaders:
def test_account_header_alone_is_not_forwarded(self):
selected = select_chatgpt_client_credential_headers(
{"chatgpt-account-id": "acct-alice", "authorization": "Bearer sk-proj-platform-key"}
)
assert dict(selected) == {}
assert get_chatgpt_client_credential({"chatgpt-account-id": "acct-alice"}) is None
def test_forwards_only_bearer_and_account_context(self):
selected = select_chatgpt_client_credential_headers(
{
"Authorization": f"Bearer {CHATGPT_JWT}",
"ChatGPT-Account-Id": "acct-explicit",
"x-litellm-api-key": "sk-gateway",
"user-agent": "codex",
}
)
assert dict(selected) == {"Authorization": f"Bearer {CHATGPT_JWT}", "ChatGPT-Account-Id": "acct-explicit"}
credential = get_chatgpt_client_credential(selected)
assert credential is not None
assert credential.access_token == CHATGPT_JWT
assert credential.account_id == "acct-explicit"
def test_account_id_falls_back_to_jwt_claim(self):
credential = get_chatgpt_client_credential({"authorization": f"Bearer {CHATGPT_JWT}"})
assert credential is not None
assert credential.account_id == "acct-alice"
@pytest.mark.parametrize("scheme", ["Bearer", "bearer", "BEARER"])
def test_bearer_scheme_is_case_insensitive(scheme):
credential = get_chatgpt_client_credential({"authorization": f"{scheme} {CHATGPT_JWT}"})
assert credential is not None
assert credential.access_token == CHATGPT_JWT
assert credential.account_id == "acct-alice"
def test_other_authentication_schemes_are_not_chatgpt_logins():
assert not is_chatgpt_oauth_key(f"Basic {CHATGPT_JWT}")

213
tests/unit/llms/custom_httpx/test_llm_http_handler.py
View file

@ -9,6 +9,7 @@ from unittest.mock import AsyncMock, Mock, patch
import httpx
import pytest
import respx
from botocore.credentials import RefreshableCredentials
import litellm
@ -28,6 +29,8 @@ from litellm.llms.brave.search.transformation import BraveSearchConfig
from litellm.llms.base_llm.image_edit.transformation import BaseImageEditConfig
from litellm.llms.base_llm.image_generation.transformation import BaseImageGenerationConfig
from litellm.llms.base_llm.text_to_speech.transformation import BaseTextToSpeechConfig
from litellm.llms.chatgpt.authenticator import Authenticator as ChatGPTAuthenticator
from litellm.llms.chatgpt.responses.transformation import ChatGPTResponsesAPIConfig
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.custom_httpx.llm_http_handler import (
BaseLLMHTTPHandler,
@ -42,6 +45,10 @@ from litellm.llms.bedrock.messages.invoke_transformations.anthropic_claude3_tran
)
from litellm.llms.openai.videos.transformation import OpenAIVideoConfig
from litellm.llms.tinyfish.search.transformation import TinyfishSearchConfig
from litellm.responses.streaming_iterator import (
BaseResponsesAPIStreamingIterator,
MockResponsesAPIStreamingIterator,
)
from litellm.types.llms.openai import HttpxBinaryResponseContent, ResponsesAPIResponse
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import ImageObject, ImageResponse, ModelResponse, TranscriptionResponse
@ -356,6 +363,7 @@ async def test_async_response_api_handler_streams_when_provider_transform_adds_s
)
)
logging_obj = Mock()
logging_obj.dynamic_success_callbacks = None
await handler.async_response_api_handler(
model="gpt-5.3-codex",
@ -372,6 +380,166 @@ async def test_async_response_api_handler_streams_when_provider_transform_adds_s
assert client.post.call_args.kwargs["json"]["stream"] is True
_CHATGPT_SSE_BODY = (
"event: response.output_item.done\n"
'data: {"type": "response.output_item.done", "output_index": 0, "item": {"type": "message", '
'"id": "msg_1", "status": "completed", "role": "assistant", "content": [{"type": "output_text", '
'"text": "aggregated", "annotations": []}]}}\n'
"\n"
"event: response.completed\n"
'data: {"type": "response.completed", "response": {"id": "resp_1", "object": "response", '
'"created_at": 1, "model": "gpt-5.3-codex", "status": "completed", "output": [], '
'"parallel_tool_calls": false, "tool_choice": "auto", "tools": [], '
'"usage": {"input_tokens": 1, "output_tokens": 2, "total_tokens": 3}}}\n'
"\n"
)
def _chatgpt_sse_response():
return httpx.Response(
200,
headers={"content-type": "text/event-stream"},
content=_CHATGPT_SSE_BODY.encode(),
request=httpx.Request("POST", "https://chatgpt.example.com/responses"),
)
def _chatgpt_responses_logging_obj():
logging_obj = Mock()
logging_obj.dynamic_success_callbacks = None
logging_obj.async_success_handler = AsyncMock()
return logging_obj
def _chatgpt_responses_config():
authenticator = Mock(spec=ChatGPTAuthenticator)
authenticator.get_access_token.return_value = "access-test"
authenticator.get_account_id.return_value = "acct-test"
return ChatGPTResponsesAPIConfig(authenticator=authenticator)
def _chatgpt_handler_kwargs(caller_params, client):
return {
"model": "gpt-5.3-codex",
"input": "hi",
"responses_api_provider_config": _chatgpt_responses_config(),
"response_api_optional_request_params": caller_params,
"custom_llm_provider": "chatgpt",
"litellm_params": GenericLiteLLMParams(api_key="sk-test", api_base="https://chatgpt.example.com"),
"logging_obj": _chatgpt_responses_logging_obj(),
"client": client,
}
def _assert_aggregated_chatgpt_response(result):
assert not isinstance(result, BaseResponsesAPIStreamingIterator)
assert isinstance(result, ResponsesAPIResponse)
assert result.id == "resp_1"
assert result.output[0].content[0].text == "aggregated"
@pytest.mark.parametrize("caller_params", [{}, {"stream": False}])
def test_response_api_handler_aggregates_chatgpt_sse_for_a_non_streaming_caller(caller_params):
handler = BaseLLMHTTPHandler()
client = HTTPHandler(client=httpx.Client())
client.post = Mock(return_value=_chatgpt_sse_response())
result = handler.response_api_handler(**_chatgpt_handler_kwargs(caller_params, client))
assert client.post.call_args.kwargs["json"]["stream"] is True
_assert_aggregated_chatgpt_response(result)
def test_response_api_handler_streams_chatgpt_sse_for_a_streaming_caller():
handler = BaseLLMHTTPHandler()
client = HTTPHandler(client=httpx.Client())
client.post = Mock(return_value=_chatgpt_sse_response())
result = handler.response_api_handler(**_chatgpt_handler_kwargs({"stream": True}, client))
assert isinstance(result, BaseResponsesAPIStreamingIterator)
assert [event.type for event in result][-1] == "response.completed"
def test_response_api_handler_streams_chatgpt_sse_for_an_extra_body_streaming_caller():
handler = BaseLLMHTTPHandler()
client = HTTPHandler(client=httpx.Client())
client.post = Mock(return_value=_chatgpt_sse_response())
result = handler.response_api_handler(extra_body={"stream": True}, **_chatgpt_handler_kwargs({}, client))
assert isinstance(result, BaseResponsesAPIStreamingIterator)
assert [event.type for event in result][-1] == "response.completed"
def test_response_api_handler_fake_streams_only_for_a_streaming_caller():
handler = BaseLLMHTTPHandler()
client = HTTPHandler(client=httpx.Client())
client.post = Mock(return_value=_chatgpt_sse_response())
streamed = handler.response_api_handler(fake_stream=True, **_chatgpt_handler_kwargs({"stream": True}, client))
assert isinstance(streamed, MockResponsesAPIStreamingIterator)
client.post = Mock(return_value=_chatgpt_sse_response())
aggregated = handler.response_api_handler(fake_stream=True, **_chatgpt_handler_kwargs({}, client))
_assert_aggregated_chatgpt_response(aggregated)
@pytest.mark.asyncio
@pytest.mark.parametrize("caller_params", [{}, {"stream": False}])
async def test_async_response_api_handler_aggregates_chatgpt_sse_for_a_non_streaming_caller(caller_params):
handler = BaseLLMHTTPHandler()
client = AsyncHTTPHandler()
client.post = AsyncMock(return_value=_chatgpt_sse_response())
result = await handler.async_response_api_handler(**_chatgpt_handler_kwargs(caller_params, client))
assert client.post.call_args.kwargs["json"]["stream"] is True
_assert_aggregated_chatgpt_response(result)
@pytest.mark.asyncio
async def test_async_response_api_handler_streams_chatgpt_sse_for_a_streaming_caller():
handler = BaseLLMHTTPHandler()
client = AsyncHTTPHandler()
client.post = AsyncMock(return_value=_chatgpt_sse_response())
result = await handler.async_response_api_handler(**_chatgpt_handler_kwargs({"stream": True}, client))
assert isinstance(result, BaseResponsesAPIStreamingIterator)
assert [event.type async for event in result][-1] == "response.completed"
@pytest.mark.asyncio
async def test_async_response_api_handler_streams_chatgpt_sse_for_an_extra_body_streaming_caller():
handler = BaseLLMHTTPHandler()
client = AsyncHTTPHandler()
client.post = AsyncMock(return_value=_chatgpt_sse_response())
result = await handler.async_response_api_handler(
extra_body={"stream": True}, **_chatgpt_handler_kwargs({}, client)
)
assert isinstance(result, BaseResponsesAPIStreamingIterator)
assert [event.type async for event in result][-1] == "response.completed"
@pytest.mark.asyncio
async def test_async_response_api_handler_fake_streams_only_for_a_streaming_caller():
handler = BaseLLMHTTPHandler()
client = AsyncHTTPHandler()
client.post = AsyncMock(return_value=_chatgpt_sse_response())
streamed = await handler.async_response_api_handler(
fake_stream=True, **_chatgpt_handler_kwargs({"stream": True}, client)
)
assert isinstance(streamed, MockResponsesAPIStreamingIterator)
client.post = AsyncMock(return_value=_chatgpt_sse_response())
aggregated = await handler.async_response_api_handler(fake_stream=True, **_chatgpt_handler_kwargs({}, client))
_assert_aggregated_chatgpt_response(aggregated)
@pytest.mark.asyncio
async def test_async_response_api_handler_streaming_passes_logging_obj_to_post():
"""LIT-5466: @track_llm_api_timing only records llm_api_duration_ms when the POST
@ -399,7 +567,7 @@ async def test_async_response_api_handler_streaming_passes_logging_obj_to_post()
model="gpt-5",
input="hi",
responses_api_provider_config=config,
response_api_optional_request_params={},
response_api_optional_request_params={"stream": True},
custom_llm_provider="chatgpt",
litellm_params=GenericLiteLLMParams(),
logging_obj=logging_obj,
@ -433,7 +601,7 @@ async def test_async_response_api_handler_posts_the_async_transform_hook_result(
model="gpt-5",
input="hi",
responses_api_provider_config=config,
response_api_optional_request_params={},
response_api_optional_request_params={"stream": True},
custom_llm_provider="chatgpt",
litellm_params=GenericLiteLLMParams(),
logging_obj=Mock(),
@ -4409,3 +4577,44 @@ async def test_anthropic_messages_dropped_safeguards_send_neither_the_field_nor_
sent: Final = received[0]
assert "safeguards" not in json.loads(sent.content)
assert "dangerous-tool-use-2026-09-03" not in sent.headers.get("anthropic-beta", ""), dict(sent.headers)
@pytest.mark.asyncio
@pytest.mark.parametrize("is_async", (False, True))
@pytest.mark.parametrize("provider", ("openai", "azure"))
async def test_non_chatgpt_responses_preserves_caller_headers_after_validation(
is_async: bool, provider: str, respx_mock: respx.MockRouter,
) -> None:
route: Final = respx_mock.post(url__regex=r"https://responses\.example\.com/.*").respond(
200,
json={
"id": "resp_headers", "object": "response", "created_at": 1,
"model": "gpt-6-luna", "status": "completed", "output": [],
"parallel_tool_calls": False, "tool_choice": "auto", "tools": [],
},
)
client: Final = AsyncHTTPHandler(transport=httpx.AsyncHTTPTransport()) if is_async else HTTPHandler(client=httpx.Client())
kwargs: Final = {
"model": f"{provider}/gpt-6-luna",
"input": "hello",
"api_key": "fake-configured-key",
"api_base": "https://responses.example.com/v1",
"api_version": "2025-04-01-preview",
"extra_headers": {"Authorization": "Bearer fake-caller-key", "content-type": "application/vnd.example+json"},
"client": client,
}
try:
response: Final = await litellm.aresponses(**kwargs) if is_async else litellm.responses(**kwargs)
assert response.status == "completed"
assert route.calls[0].request.headers["authorization"] == "Bearer fake-caller-key"
expected_content_types: Final = (
["application/json", "application/vnd.example+json"]
if provider == "openai"
else ["application/vnd.example+json"]
)
assert route.calls[0].request.headers.get_list("content-type") == expected_content_types
finally:
if isinstance(client, AsyncHTTPHandler):
await client.close()
else:
client.client.close()

12
tests/unit/proxy/auth/test_auth_utils.py
View file

@ -2812,6 +2812,18 @@ class TestIsRequestBodySafeBlocksEndpointTargetingFields:
)
@pytest.mark.parametrize("allow_client_side_credentials", [False, True])
@pytest.mark.parametrize("nested", [False, True])
def test_request_cannot_select_chatgpt_auth_file(allow_client_side_credentials: bool, nested: bool) -> None:
selector: Final = {"chatgpt_auth_file": "/tokens/account-b/auth.json"}
with pytest.raises(ValueError, match="chatgpt_auth_file"):
is_request_body_safe(
request_body={"model": "chatgpt-account-a", **({"extra_body": selector} if nested else selector)},
general_settings={"allow_client_side_credentials": allow_client_side_credentials},
llm_router=None,
model="chatgpt-account-a",
)
class TestIsRequestBodySafeBlocksBedrockProjectOverride:
"""``aws_bedrock_project_id`` pins a deployment to a Bedrock project so
that project's data-retention policy applies to its requests. A

100
tests/unit/proxy/health_endpoints/test_health_endpoints.py
View file

@ -5,6 +5,7 @@ import time
from collections.abc import Iterator, Mapping, Sequence
from contextlib import contextmanager
from datetime import datetime, timedelta
from pathlib import Path
from types import SimpleNamespace
from typing import Final
from unittest.mock import AsyncMock, MagicMock, patch
@ -12,7 +13,7 @@ from unittest.mock import AsyncMock, MagicMock, patch
import httpx
import pytest
import respx
from fastapi import FastAPI
from fastapi import FastAPI, HTTPException
from fastapi.testclient import TestClient
from prisma.errors import ClientNotConnectedError, HTTPClientClosedError, PrismaError
@ -4326,3 +4327,100 @@ async def test_health_services_endpoint_langfuse_missing_keys_errors(monkeypatch
with pytest.raises(ProxyException, match="auth_check failed") as raised:
await health_services_endpoint(service="langfuse")
assert "connection refused by lf.internal.example" in str(raised.value.message)
@pytest.mark.asyncio
@pytest.mark.parametrize("admin", [False, True])
async def test_unsaved_chatgpt_account_probe_requires_proxy_admin(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, admin: bool
) -> None:
from starlette.requests import Request
from litellm.proxy import proxy_server
monkeypatch.setenv("CHATGPT_AUTH_FILE", str(tmp_path / "missing-default-auth.json"))
auth_file: Final = tmp_path / "auth.json"
auth_file.write_text(
json.dumps({"access_token": "token-probe", "account_id": "acct-probe", "expires_at": 4102444800})
)
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
monkeypatch.setattr(proxy_server, "prisma_client", MagicMock())
monkeypatch.setattr(proxy_server, "llm_router", None)
monkeypatch.setattr(proxy_server, "general_settings", {})
output: Final = {
"id": "msg_probe",
"type": "message",
"role": "assistant",
"status": "completed",
"content": [{"type": "output_text", "text": "ok", "annotations": []}],
}
response: Final = {
"id": "resp_probe",
"object": "response",
"created_at": 1700000000,
"status": "completed",
"model": "gpt-6.1-sol",
"output": [output],
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2},
}
with respx.mock(assert_all_called=False) as upstream:
upstream.post("https://chatgpt.com/backend-api/codex/responses").respond(
200,
text="data: " + json.dumps({"type": "response.completed", "response": response}) + "\n\n",
headers={"content-type": "text/event-stream"},
)
call: Final = health_test_model_connection(
request=Request({"type": "http", "headers": []}),
mode="responses",
litellm_params={"model": "chatgpt/gpt-6.1-sol", "chatgpt_auth_file": str(auth_file)},
model_info={},
user_api_key_dict=UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN if admin else LitellmUserRoles.INTERNAL_USER, token="test-key"
),
)
if admin:
result: Final = await call
assert result["status"] == "success"
assert upstream.calls.last.request.headers["Authorization"] == "Bearer token-probe"
assert upstream.calls.last.request.headers["ChatGPT-Account-Id"] == "acct-probe"
assert isinstance(json.loads(upstream.calls.last.request.content)["input"], list)
else:
with pytest.raises(HTTPException) as error:
await call
assert error.value.status_code == 403
assert len(upstream.calls) == 0
@pytest.mark.asyncio
async def test_team_admin_probe_cannot_name_another_chatgpt_auth_file(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
from starlette.requests import Request
from litellm.proxy import proxy_server
from litellm.proxy.management_endpoints.model_management_endpoints import ModelManagementAuthChecks
auth_file: Final = tmp_path / "other-account" / "auth.json"
auth_file.parent.mkdir()
auth_file.write_text(json.dumps({"access_token": "token-other", "account_id": "acct-other", "expires_at": 4102444800}))
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
monkeypatch.setattr(proxy_server, "prisma_client", MagicMock())
monkeypatch.setattr(proxy_server, "llm_router", None)
monkeypatch.setattr(proxy_server, "general_settings", {})
monkeypatch.setattr( # test-quality-ok: the team-membership check needs a live DB; this test isolates the auth file check
ModelManagementAuthChecks, "can_user_make_model_call", AsyncMock(return_value=True)
)
with respx.mock(assert_all_called=False) as upstream:
upstream.route(host="chatgpt.com").respond(500)
with pytest.raises(HTTPException) as error:
await health_test_model_connection(
request=Request({"type": "http", "headers": []}),
mode="responses",
litellm_params={"model": "chatgpt/gpt-6.1-sol", "chatgpt_auth_file": str(auth_file)},
model_info={"team_id": "test_team"},
user_api_key_dict=UserAPIKeyAuth(
user_role=LitellmUserRoles.INTERNAL_USER, team_id="test_team", token="test-key"
),
)
assert error.value.status_code == 403
assert "chatgpt_auth_file" in str(error.value.detail)
assert len(upstream.calls) == 0

61
tests/unit/proxy/management_endpoints/test_model_management_endpoints.py
View file

@ -507,6 +507,67 @@ class TestModelManagementAuthChecks:
)
assert exc_info.value.code == "403"
def _check_chatgpt_auth_file(self, requested: str | None, existing: str | None, admin: bool) -> object:
return ModelManagementAuthChecks.can_user_set_chatgpt_auth_file(
litellm_params=LiteLLM_Params(model="chatgpt/gpt-6.1-sol", chatgpt_auth_file=requested),
user_api_key_dict=self.admin_user if admin else self.team_admin_user,
existing_litellm_params=(
None if existing is None else LiteLLM_Params(model="chatgpt/gpt-6.1-sol", chatgpt_auth_file=existing)
),
)
@pytest.mark.parametrize(
"requested, existing, admin",
[
("/srv/acct-a/auth.json", None, True),
("/srv/acct-a/auth.json", "/srv/acct-a/auth.json", False),
(None, "/srv/acct-a/auth.json", False),
],
)
def test_chatgpt_auth_file_is_allowed_for_admins_or_when_unchanged(self, requested, existing, admin):
assert self._check_chatgpt_auth_file(requested, existing, admin) is True
@pytest.mark.parametrize("existing", [None, "/srv/acct-a/auth.json"])
def test_team_admin_cannot_point_a_model_at_another_chatgpt_auth_file(self, existing):
from litellm.proxy._types import ProxyException
with pytest.raises(ProxyException) as exc_info:
self._check_chatgpt_auth_file("/srv/acct-b/auth.json", existing, admin=False)
assert exc_info.value.code == "403"
assert exc_info.value.param == "chatgpt_auth_file"
@pytest.mark.asyncio
async def test_add_new_model_rejects_chatgpt_auth_file_for_team_admin(self):
from litellm.proxy._types import ProxyException
from litellm.proxy.management_endpoints.model_management_endpoints import (
add_new_model,
)
mock_prisma = MagicMock()
with (
patch("litellm.proxy.proxy_server.prisma_client", mock_prisma), # test-quality-ok: endpoint reads proxy server globals with no injection seam
patch("litellm.proxy.proxy_server.store_model_in_db", True), # test-quality-ok: endpoint reads proxy server globals with no injection seam
patch("litellm.proxy.proxy_server.premium_user", True), # test-quality-ok: endpoint reads proxy server globals with no injection seam
patch( # test-quality-ok: prior auth check needs a live DB; only the auth file check is under test
"litellm.proxy.management_endpoints.model_management_endpoints.ModelManagementAuthChecks.can_user_make_model_call",
new=AsyncMock(return_value=None),
),
):
with pytest.raises(ProxyException) as exc_info:
await add_new_model(
model_params=Deployment(
model_name="team-subscription",
litellm_params=LiteLLM_Params(
model="chatgpt/gpt-6.1-sol", chatgpt_auth_file="/srv/other-account/auth.json"
),
model_info={"id": "chatgpt-auth-file-create-test", "team_id": "test_team"},
),
user_api_key_dict=self.team_admin_user,
)
assert exc_info.value.code == "403"
assert exc_info.value.param == "chatgpt_auth_file"
mock_prisma.db.litellm_proxymodeltable.create.assert_not_called()
@pytest.mark.asyncio
async def test_add_new_model_rejects_aws_session_tags_for_non_admin(self):
from litellm.proxy._types import ProxyException

186
tests/unit/proxy/pass_through_endpoints/test_codex_passthrough_endpoints.py Normal file
View file

@ -0,0 +1,186 @@
import secrets
import asyncio
import json
from collections.abc import Iterator
from datetime import datetime
from typing import Final
import httpx
import pytest
import respx
from fastapi.testclient import TestClient
import litellm
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy import proxy_server
_BASE: Final = "https://chatgpt.com/backend-api/codex"
_TOKEN: Final = "test-subscription-token"
MASTER_KEY: Final = f"sk-{secrets.token_hex(16)}"
_HEADERS: Final = {"authorization": f"Bearer {_TOKEN}", "x-litellm-api-key": MASTER_KEY}
@pytest.fixture
def client(monkeypatch: pytest.MonkeyPatch, httpx_transport: None) -> Iterator[TestClient]:
monkeypatch.setattr(proxy_server, "master_key", MASTER_KEY)
yield TestClient(proxy_server.app)
def test_native_catalog_preserves_account_and_filters_gateway_credentials(
client: TestClient, respx_mock: respx.MockRouter
) -> None:
catalog: Final = {"models": [{"slug": "subscription-model", "supported_reasoning_levels": ["high"]}]}
upstream: Final = respx_mock.get(f"{_BASE}/models", params={"client_version": "1.0.0"}).respond(json=catalog)
response: Final = client.get(
"/backend-api/codex/models?client_version=1.0.0",
headers={
**_HEADERS,
"chatgpt-account-id": "account-a",
"cookie": "session=test-cookie",
"x-pass-authorization": "Bearer test-injected-token",
"x-pass-x-litellm-api-key": "test-injected-key",
},
)
assert response.status_code == 200
assert response.json() == catalog
assert upstream.calls.last.request.headers["authorization"] == f"Bearer {_TOKEN}"
assert upstream.calls.last.request.headers["chatgpt-account-id"] == "account-a"
assert "x-litellm-api-key" not in upstream.calls.last.request.headers
assert "cookie" not in upstream.calls.last.request.headers
@pytest.mark.parametrize("account", ["account-a", "account-b"])
def test_native_responses_preserves_body_and_stream(
client: TestClient, respx_mock: respx.MockRouter, account: str
) -> None:
body: Final = {
"model": "subscription-model",
"input": [{"role": "user", "content": "Hello"}],
"reasoning": {"effort": "high"},
"store": False,
"stream": True,
}
stream: Final = (
b'event: response.output_text.delta\ndata: {"type":"response.output_text.delta","delta":"Hello"}\n\n'
)
upstream: Final = respx_mock.post(f"{_BASE}/responses").respond(
content=stream, headers={"content-type": "text/event-stream"}
)
response: Final = client.post(
"/backend-api/codex/responses",
json=body,
headers={**_HEADERS, "authorization": f"Bearer test-{account}", "chatgpt-account-id": account},
)
assert response.status_code == 200
assert response.content == stream
assert response.headers["content-type"].startswith("text/event-stream")
assert json.loads(upstream.calls.last.request.content) == body
assert upstream.calls.last.request.headers["authorization"] == f"Bearer test-{account}"
assert upstream.calls.last.request.headers["chatgpt-account-id"] == account
@pytest.mark.parametrize("gateway_key", [None, "sk-invalid-test-key"])
def test_gateway_auth_rejects_before_upstream(
client: TestClient, respx_mock: respx.MockRouter, gateway_key: str | None
) -> None:
headers: Final = {
"authorization": f"Bearer {_TOKEN}",
**({"x-litellm-api-key": gateway_key} if gateway_key else {}),
}
response: Final = client.get("/backend-api/codex/models", headers=headers)
assert response.status_code in (400, 401, 403)
assert not respx_mock.calls
@pytest.mark.parametrize("authorization", ["", "Basic test", "Bearer ", f"Bearer {MASTER_KEY}"])
def test_subscription_credential_required(client: TestClient, respx_mock: respx.MockRouter, authorization: str) -> None:
response: Final = client.get("/backend-api/codex/models", headers={**_HEADERS, "authorization": authorization})
assert response.status_code == 401
assert not respx_mock.calls
@pytest.mark.parametrize("status", [401, 403, 429])
def test_provider_errors_do_not_fall_back(client: TestClient, respx_mock: respx.MockRouter, status: int) -> None:
upstream: Final = respx_mock.post(f"{_BASE}/responses").respond(
status, json={"error": {"message": "subscription unavailable"}}, headers={"retry-after": "60"}
)
response: Final = client.post(
"/backend-api/codex/responses", headers=_HEADERS, json={"model": "subscription-model", "stream": True}
)
assert response.status_code == status
assert upstream.call_count == 1
@pytest.mark.parametrize("method,path", [("POST", "models"), ("GET", "responses"), ("GET", "deviceauth")])
def test_only_native_inference_routes_are_forwarded(
client: TestClient, respx_mock: respx.MockRouter, method: str, path: str
) -> None:
response: Final = client.request(method, f"/backend-api/codex/{path}", headers=_HEADERS)
assert response.status_code in (405, 422)
assert not respx_mock.calls
class UsageRecorder(CustomLogger):
def __init__(self) -> None:
super().__init__()
self.events: Final[asyncio.Queue[dict[str, object]]] = asyncio.Queue()
async def async_log_success_event(
self, kwargs: dict[str, object], response_obj: object, start_time: datetime, end_time: datetime
) -> None:
await self.events.put(kwargs)
@pytest.mark.asyncio
@pytest.mark.parametrize("stream", [False, True])
async def test_usage_logged_without_credentials(
monkeypatch: pytest.MonkeyPatch, httpx_transport: None, respx_mock: respx.MockRouter, stream: bool
) -> None:
monkeypatch.setattr(proxy_server, "master_key", MASTER_KEY)
recorder: Final = UsageRecorder()
monkeypatch.setattr(litellm, "_async_success_callback", [recorder])
litellm.register_model(
{
"chatgpt/subscription-model": {
"litellm_provider": "chatgpt",
"mode": "responses",
"input_cost_per_token": 0.0,
"output_cost_per_token": 0.0,
}
}
)
result: Final = {
"id": "resp_test",
"object": "response",
"created_at": 1700000000,
"status": "completed",
"model": "subscription-model",
"output": [],
"usage": {"input_tokens": 12, "output_tokens": 3, "total_tokens": 15},
}
event: Final = {"type": "response.completed", "response": result}
response_body: Final = (
httpx.Response(
200,
content=f"event: response.completed\ndata: {json.dumps(event)}\n\n",
headers={"content-type": "text/event-stream"},
)
if stream
else httpx.Response(200, json=result)
)
respx_mock.post(f"{_BASE}/responses").mock(return_value=response_body)
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=proxy_server.app), base_url="http://test") as caller:
response: Final = await caller.post(
"/backend-api/codex/responses",
json={"model": "subscription-model", "stream": stream, "input": "Hello"},
headers={**_HEADERS, "x-pass-authorization": "Bearer test-hidden-token", "cookie": "test-hidden-cookie"},
)
assert response.status_code == 200
logged: Final = await asyncio.wait_for(recorder.events.get(), timeout=5)
payload: Final = logged["standard_logging_object"]
assert isinstance(payload, dict)
assert payload["prompt_tokens"] == 12
assert payload["completion_tokens"] == 3
serialized: Final = json.dumps(logged, default=str)
for secret in (_TOKEN, MASTER_KEY, "test-hidden-token", "test-hidden-cookie"):
assert secret not in serialized

117
tests/unit/proxy/test_litellm_pre_call_utils.py
View file

@ -1,4 +1,5 @@
import asyncio
import base64
import copy
import json
import os
@ -52,6 +53,7 @@ from litellm.constants import (
SESSION_ID_GENERATED_METADATA_KEY,
SESSION_ID_OMITTED_METADATA_KEY,
)
from litellm.llms.anthropic.common_utils import resolve_used_client_oauth_token
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM
from litellm.llms.fireworks_ai.common_utils import get_fireworks_session_id
from litellm.types.utils import CredentialItem
@ -8490,6 +8492,121 @@ def test_default_team_settings_bool_turn_off_message_logging_redacts():
)
def _fake_chatgpt_oauth_token(account_id: str) -> str:
header = base64.urlsafe_b64encode(b'{"alg":"RS256"}').rstrip(b"=").decode()
payload = (
base64.urlsafe_b64encode(
json.dumps({"https://api.openai.com/auth": {"chatgpt_account_id": account_id}, "exp": 4102444800}).encode()
)
.rstrip(b"=")
.decode()
)
return f"{header}.{payload}.fake-signature"
CHATGPT_OAUTH_TOKEN = f"Bearer {_fake_chatgpt_oauth_token('acct-alice')}"
CHATGPT_ACCOUNT_ID_HEADER_NAME = "ChatGPT-Account-Id"
PLATFORM_API_KEY_BEARER = "Bearer sk-proj-fake-platform-key-0123456789"
CHATGPT_LEAK_TARGET_PROVIDERS = ["openai", "azure", "anthropic", "bedrock", "vertex_ai"]
def _codex_client_headers(
authorization_header_name: str = "authorization", account_id: str | None = "acct-alice"
) -> dict:
return {
"content-type": "application/json",
"originator": "codex_cli_rs",
authorization_header_name: CHATGPT_OAUTH_TOKEN,
**({CHATGPT_ACCOUNT_ID_HEADER_NAME: account_id} if account_id is not None else {}),
}
def test_clean_headers_keeps_chatgpt_oauth_bearer_when_gateway_key_is_separate():
result = clean_headers(
headers=Headers({**_codex_client_headers(), "x-litellm-api-key": "sk-proxy-virtual-key"}),
litellm_key_header_name="x-litellm-api-key",
forward_llm_provider_auth_headers=False,
authenticated_with_header="x-litellm-api-key",
)
assert result["authorization"] == CHATGPT_OAUTH_TOKEN
assert "x-litellm-api-key" not in result
def test_clean_headers_drops_bearer_that_authenticated_to_the_gateway():
result = clean_headers(
headers=Headers({"authorization": CHATGPT_OAUTH_TOKEN, "content-type": "application/json"}),
litellm_key_header_name=None,
forward_llm_provider_auth_headers=False,
authenticated_with_header="authorization",
)
assert _authorization_values(result) == []
@pytest.mark.parametrize("bearer", [PLATFORM_API_KEY_BEARER, "Bearer not.a-jwt", "Bearer a.bm90LWpzb24.c"])
def test_clean_headers_drops_non_chatgpt_bearers(bearer):
result = clean_headers(
headers=Headers({"authorization": bearer, "x-litellm-api-key": "sk-proxy-virtual-key"}),
litellm_key_header_name="x-litellm-api-key",
forward_llm_provider_auth_headers=False,
authenticated_with_header="x-litellm-api-key",
)
assert _authorization_values(result) == []
@pytest.mark.parametrize("authorization_header_name", AUTHORIZATION_HEADER_CASINGS)
def test_chatgpt_oauth_credential_and_account_reach_chatgpt_provider(authorization_header_name):
forwarded = _headers_forwarded_to(_codex_client_headers(authorization_header_name), "chatgpt")
assert _authorization_values(forwarded) == [CHATGPT_OAUTH_TOKEN]
assert forwarded[CHATGPT_ACCOUNT_ID_HEADER_NAME] == "acct-alice"
@pytest.mark.parametrize("custom_llm_provider", CHATGPT_LEAK_TARGET_PROVIDERS)
def test_chatgpt_oauth_credential_is_never_forwarded_to_other_providers(custom_llm_provider):
forwarded = _headers_forwarded_to(_codex_client_headers(), custom_llm_provider)
assert _authorization_values(forwarded) == []
assert CHATGPT_OAUTH_TOKEN not in forwarded.values()
assert CHATGPT_ACCOUNT_ID_HEADER_NAME not in forwarded
def test_chatgpt_account_header_is_not_forwarded_without_chatgpt_oauth_bearer():
headers = {**_codex_client_headers(), "authorization": PLATFORM_API_KEY_BEARER}
assert _headers_forwarded_to(headers, "chatgpt") == {}
def test_anthropic_oauth_bearer_does_not_reach_chatgpt_provider():
assert _authorization_values(_headers_forwarded_to(_client_headers(), "chatgpt")) == []
@pytest.mark.parametrize("client_headers", [{"content-type": "application/json"}, _codex_client_headers()])
def test_provider_specific_header_from_the_request_body_never_reaches_chatgpt(client_headers):
data: Final = {
"provider_specific_header": {
"custom_llm_provider": "chatgpt",
"extra_headers": {"Authorization": "Bearer body-injected", "x-body-injected": "1"},
}
}
add_provider_specific_headers_to_request(data=data, headers=client_headers)
forwarded: Final = ProviderSpecificHeaderUtils.get_provider_specific_headers(
provider_specific_header=data.get("provider_specific_header"), custom_llm_provider="chatgpt"
)
assert "Bearer body-injected" not in forwarded.values()
assert "x-body-injected" not in forwarded
def test_chatgpt_login_is_not_stamped_as_an_anthropic_seat():
client_sent_oauth_token: Final = add_provider_specific_headers_to_request(data={}, headers=_codex_client_headers())
assert resolve_used_client_oauth_token(client_sent_oauth_token, "anthropic") is False
def test_add_user_api_key_auth_to_request_metadata_attributes_a_cli_session_to_its_alias():
data = {"model": "gpt-5.4-nano", "messages": [{"role": "user", "content": "hi"}], "litellm_metadata": {}}
session = UserAPIKeyAuth(

25
tests/unit/responses/test_responses_api_request_body.py
View file

@ -979,3 +979,28 @@ async def test_a_native_responses_provider_places_every_point_itself():
assert user_only == [0, 2, 4, 6]
assert with_unmatchable_system == user_only
@pytest.mark.parametrize("async_call", [False, True])
@pytest.mark.asyncio
async def test_provider_scoped_headers_only_join_chatgpt_responses_calls(
monkeypatch: pytest.MonkeyPatch, async_call: bool
):
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
params: Final = {
"model": "openai/gpt-5.4",
"input": "Hello",
"api_key": "sk-configured-test-key",
"provider_specific_header": {"custom_llm_provider": "openai", "extra_headers": {"x-scoped-header": "1"}},
}
with respx.mock as transport:
route: Final = transport.post("https://api.openai.com/v1/responses").respond(
200, json=_minimal_responses_api_payload("resp_scoped", "gpt-5.4")
)
if async_call:
await litellm.aresponses(**params)
else:
litellm.responses(**params)
assert "x-scoped-header" not in route.calls.last.request.headers
assert route.calls.last.request.headers["Authorization"] == "Bearer sk-configured-test-key"

36
tests/unit/responses/test_responses_utils.py
View file

@ -890,3 +890,39 @@ class TestResponsesInputToChatMessages:
assert ResponsesAPIRequestUtils.responses_input_to_chat_messages(
[reasoning_item, user_message, "stray"]
) == [user_message]
class TestMergeProviderSpecificHeaders:
def test_scoped_headers_merge_only_for_matching_provider(self):
scoped = {"custom_llm_provider": "chatgpt", "extra_headers": {"Authorization": "Bearer client-token"}}
merged = ResponsesAPIRequestUtils.merge_provider_specific_headers(
extra_headers={"originator": "codex_cli_rs"},
provider_specific_header=scoped,
custom_llm_provider="chatgpt",
)
untouched = ResponsesAPIRequestUtils.merge_provider_specific_headers(
extra_headers={"originator": "codex_cli_rs"},
provider_specific_header=scoped,
custom_llm_provider="openai",
)
assert merged == {"originator": "codex_cli_rs", "Authorization": "Bearer client-token"}
assert untouched == {"originator": "codex_cli_rs"}
def test_sequence_of_scoped_entries_each_keep_their_own_scope(self):
entries = [
{"custom_llm_provider": "anthropic", "extra_headers": {"Authorization": "Bearer anthropic-token"}},
{"custom_llm_provider": "chatgpt", "extra_headers": {"Authorization": "Bearer chatgpt-token"}},
]
merged = ResponsesAPIRequestUtils.merge_provider_specific_headers(
extra_headers=None, provider_specific_header=entries, custom_llm_provider="chatgpt"
)
assert merged == {"Authorization": "Bearer chatgpt-token"}
def test_no_scoped_headers_returns_extra_headers_unchanged(self):
assert ResponsesAPIRequestUtils.merge_provider_specific_headers(
extra_headers=None, provider_specific_header=None, custom_llm_provider="chatgpt"
) is None

9
tests/unit/test_router/test_router.py
View file

@ -99,6 +99,15 @@ def test_update_kwargs_does_not_mutate_defaults_and_merges_metadata():
assert kwargs["litellm_metadata"] == {"baz": 123}
def test_router_model_group_llm_provider_resolves_chatgpt_without_login(tmp_path, monkeypatch: pytest.MonkeyPatch):
monkeypatch.setenv("CHATGPT_TOKEN_DIR", str(tmp_path))
params: Final = LiteLLM_Params(model="chatgpt/gpt-5.4")
assert Router._model_group_llm_provider(params) == ("gpt-5.4", "chatgpt")
assert Router._model_group_llm_provider(LiteLLM_Params(model="no-such-provider-model-xyz")) == ("", "")
assert not any(tmp_path.iterdir())
def test_router_with_model_info_and_model_group():
"""
Test edge case where user specifies model_group in model_info

1
tests/unit/types/test_litellm_params.py
View file

@ -87,6 +87,7 @@ CONNECTION_NAMES: Final = (
"litellm_credential_name",
"configurable_clientside_auth_params",
"use_xai_oauth",
"chatgpt_auth_file",
"aws_batch_role_arn",
"s3_bucket_name",
"s3_region_name",

105
ui/litellm-dashboard/src/lib/http/schema.d.ts generated vendored
View file

@ -1737,6 +1737,30 @@ export interface paths {
patch: operations["azure_speech_proxy_route_azure_speech__endpoint__patch"];
trace?: never;
};
"/backend-api/codex/{endpoint}": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
/**
* Codex Passthrough Route
* @description Forward a client's ChatGPT subscription with separate LiteLLM authentication.
*/
get: operations["codex_passthrough_route_backend_api_codex__endpoint__get"];
put?: never;
/**
* Codex Passthrough Route
* @description Forward a client's ChatGPT subscription with separate LiteLLM authentication.
*/
post: operations["codex_passthrough_route_backend_api_codex__endpoint__post"];
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/batches": {
parameters: {
query?: never;
@ -34661,6 +34685,11 @@ export interface components {
cache_read_input_token_cost_priority?: number | null;
/** Cache Read Input Token Cost Ultrafast */
cache_read_input_token_cost_ultrafast?: number | null;
/**
* Chatgpt Auth File
* @description Path to a ChatGPT OAuth auth.json for this deployment.
*/
chatgpt_auth_file?: string | null;
/** Citation Cost Per Token */
citation_cost_per_token?: number | null;
/** Client Id */
@ -49169,6 +49198,11 @@ export interface components {
cache_read_input_token_cost_priority?: number | null;
/** Cache Read Input Token Cost Ultrafast */
cache_read_input_token_cost_ultrafast?: number | null;
/**
* Chatgpt Auth File
* @description Path to a ChatGPT OAuth auth.json for this deployment.
*/
chatgpt_auth_file?: string | null;
/** Citation Cost Per Token */
citation_cost_per_token?: number | null;
/** Client Id */
@ -52113,6 +52147,77 @@ export interface operations {
};
};
};
codex_passthrough_route_backend_api_codex__endpoint__get: {
parameters: {
query?: never;
header?: never;
path: {
endpoint: "models" | "responses";
};
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Successful Response */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": unknown;
};
};
/** @description Validation Error */
422: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["HTTPValidationError"];
};
};
};
};
codex_passthrough_route_backend_api_codex__endpoint__post: {
parameters: {
query?: never;
header?: never;
path: {
endpoint: "models" | "responses";
};
cookie?: never;
};
requestBody: {
content: {
"application/json": {
[key: string]: unknown;
};
};
};
responses: {
/** @description Unmodified Codex response or server-sent events */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": {
[key: string]: unknown;
};
"text/event-stream": string;
};
};
/** @description Validation Error */
422: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["HTTPValidationError"];
};
};
};
};
list_batches_batches_get: {
parameters: {
query?: {