diff --git a/examples/model_offerings/ARCHITECTURE.md b/examples/model_offerings/ARCHITECTURE.md index c40d0ad6022..9ef9c05bd0d 100644 --- a/examples/model_offerings/ARCHITECTURE.md +++ b/examples/model_offerings/ARCHITECTURE.md @@ -4,7 +4,7 @@ Automatic supplier metadata is the normal path for AUTO offerings. Selecting an The schema still requires an explicit `source`. Normal automatic usage selects `source: auto` without copying a capability card; it does not mean an omitted `source` field is accepted -This walkthrough is verified against runtime commit `ec1e681d95dca0e6660e21fba9060ec9e3c7afde`. Relative source links follow the code in the same checkout. The feature extends native LiteLLM adapters, types, authentication, listing and Router paths; referencing those paths does not mean every implementation in them originated in this PR. See the [setup guide](README.md) for configuration and lifecycle instructions +This walkthrough includes the upstream sync through `dddbaa803d6e9fcc862ea7a5412f771cb1ae040e`. Relative source links follow the code in the same checkout, and the PR records validation against its published head. The feature extends native LiteLLM adapters, types, authentication, listing and Router paths; referencing those paths does not mean every implementation in them originated in this PR. See the [setup guide](README.md) for configuration and lifecycle instructions ```text supplier connection -> inventory -> native normalization -> offering resolution @@ -177,6 +177,8 @@ It chooses the budget's effort key from request effort first, then deployment re The helper returns `max_output_tokens` for Responses calls and `max_tokens` for the other supported calls. It does not inject a reasoning effort just because a default was advertised. An explicit caller budget wins over this configured default-budget policy, and conflicting deployment budget parameters are removed in that policy path. Native/provider request translation still determines the eventual wire format +The selector reads `reasoning_effort` and `reasoning.effort`; it is not a second implementation of every provider's thinking controls. In particular, its budget choice precedes later native request-level effort rounding. Do not assume an effort-specific default budget is recomputed for a different, unsupported effort that upstream subsequently rounds, or that every `output_config`/thinking-budget shape supplies this selector's effort key + For the manual example, an omitted effort uses the base requested budget of `65536`; explicit `max` selects `131072`. Neither value means the physical output ceiling is known. This helper is defaulting, not a provider-independent quota or proof that a chosen budget is feasible An advertised reasoning default, a client electing to send that default, an explicit deployment request setting and a provider adapter's own default behavior are different things. The offering manager itself does not turn every advertised default into an outgoing request parameter. With no explicit effort setting or provider-specific injection, omission leaves the supplier's behavior in charge @@ -199,6 +201,10 @@ Variant syntax and UI conversion belong to the client, not this LiteLLM PR. A cl For inference, the public alias resolves to a selected native deployment. The Router applies request-default budget policy and dispatches through the selected provider adapter. The [OpenRouter chat adapter](../../litellm/llms/openrouter/chat/transformation.py) preserves explicit gateway `max`/`xhigh`, and the [Messages bridge normalization](../../litellm/llms/anthropic/pass_through/utils.py) does not reduce OpenRouter effort using unrelated backend flags. Native Responses and Responses-to-chat paths retain their API-specific request shapes +Upstream's [deployment reasoning resolver](../../litellm/router_utils/reasoning_effort_capability.py) now owns deployment-declared effort handling across chat, Responses and Messages. An accepted declared value is kept. A ranked request outside the declared set can be rounded according to native up/down policy, or dropped under native `drop_params` behavior; explicit `default` resolves to omission. Cost-map-only choices do not activate deployment rounding. Supplier choices resolved into an AUTO deployment's own `model_info` do count as its declaration. The external offering schema does not add a parallel rounding framework or expose every native Router setting + +Consequently, preserving accepted OpenRouter gateway efforts is not an all-provider or all-input passthrough guarantee. The fork still avoids the provider adapter's old unconditional max-to-xhigh rename and legacy undeclared OpenRouter approximation, while reusing upstream's declared-effort resolver. Other provider adapters keep their native request semantics. The shared native [deployment-pricing resolver](../../litellm/router_utils/deployment_pricing.py) also owns base-model key precedence for listing; the former fork-only key lookup is removed + The offering guard merges the exact configured `only`/`allow_fallbacks: false` body and checks it again at deployment dispatch. Caller connection, deployment, provider, fallback, tier and parameter-dropping overrides cannot replace it. Pinned response-cache reads/writes are disabled to prevent output from another endpoint satisfying the request. No `require_parameters` setting or fallback supplier is silently added. Unsupported pinned API surfaces and native batches are rejected rather than bypassing selection ## 9. Refresh, failure and present limitations diff --git a/examples/model_offerings/README.md b/examples/model_offerings/README.md index 206d5b21ee4..e32c77b7fb5 100644 --- a/examples/model_offerings/README.md +++ b/examples/model_offerings/README.md @@ -102,7 +102,7 @@ AUTO still checks model membership. Pinned AUTO offerings additionally discover Reasoning choices and defaults come from OpenRouter's model-level reasoning contract and are advertised only when every eligible endpoint declares the effort control. An endpoint's more restrictive declared choices narrow that set. This describes OpenRouter's accepted gateway options, not an independently discovered backend-native effort enum. An explicit empty choice list remains empty. Explicit null expands only to OpenRouter's documented gateway efforts, while omission does not declare choices. Mandatory reasoning excludes the disable value, and a disabled-by-default model does not claim an active default effort. Boolean reasoning support alone never supplies choices. Endpoint capability guarantees are combined conservatively, without using status, uptime or performance to change routing or availability -Every explicit `model_info` field remains authoritative. Existing handwritten context/output limits continue overriding discovery until the operator removes them. MANUAL offerings remain entirely handwritten. Requests without an effort leave the supplier default unchanged; explicit gateway effort values, including `max` and `xhigh`, are forwarded without backend-flag approximation on chat and translated Messages paths +Every explicit `model_info` field remains authoritative. Existing handwritten context/output limits continue overriding discovery until the operator removes them. MANUAL offerings remain entirely handwritten. An advertised default does not itself inject request effort. Accepted declared OpenRouter gateway efforts, including `max` and `xhigh`, are preserved without the old provider rename or backend-flag approximation. Upstream's native deployment resolver owns unsupported-value rounding/default/drop behavior across providers and routes; this is not a promise to forward every arbitrary effort unchanged Successful empty endpoint inventories clear derived endpoint metadata. Failed refreshes retain last-known endpoint metadata only for the same connection identity; an initial failure leaves unknown fields omitted, never replaced with another supplier's limits. Endpoint discovery does not disable an otherwise selected model or broaden its policy. Metadata refreshes publish atomically, and requests already running retain their previous snapshot. Policy edits still change the affected offering's private deployment ID diff --git a/litellm/llms/anthropic/pass_through/utils.py b/litellm/llms/anthropic/pass_through/utils.py index dcaa8e3f6b9..f466d1d3078 100644 --- a/litellm/llms/anthropic/pass_through/utils.py +++ b/litellm/llms/anthropic/pass_through/utils.py @@ -210,8 +210,6 @@ def normalize_reasoning_effort_value( handled where the thinking block is built. A deployment accepting no tier at all keeps the chain's floor, which is what every deployment degraded to before there was anything to ask. """ - if custom_llm_provider == "openrouter" or custom_llm_provider is None and model.startswith("openrouter/"): - return effort chain: Final = _EFFORT_DEGRADATION_CHAIN.get(effort) if chain is None: return effort @@ -223,6 +221,8 @@ def normalize_reasoning_effort_value( if deployment_rounds_reasoning_effort(deployment_model_info): return effort + if custom_llm_provider == "openrouter" or custom_llm_provider is None and model.startswith("openrouter/"): + return effort from litellm.utils import get_model_info try: diff --git a/litellm/proxy/proxy_server.py b/litellm/proxy/proxy_server.py index 4bbb14fc0df..2d8d6c1115a 100644 --- a/litellm/proxy/proxy_server.py +++ b/litellm/proxy/proxy_server.py @@ -167,8 +167,8 @@ from litellm.proxy.common_utils.token_count_settings import ( resolve_token_count_settings, ) from litellm.proxy.management_helpers.auto_router_availability import AutoRouterCatalogEntry, build_auto_router_catalog -from litellm.proxy.model_discovery import discover_model_metadata from litellm.proxy.management_helpers.model_group_custom_fields import model_group_custom_fields +from litellm.proxy.model_discovery import discover_model_metadata from litellm.router_utils.access_windows import access_windows_config_error from litellm.router_utils.add_retry_fallback_headers import ( get_fallback_errors_from_headers, diff --git a/litellm/router.py b/litellm/router.py index 0f83dcc9bcc..35283da6485 100644 --- a/litellm/router.py +++ b/litellm/router.py @@ -290,7 +290,6 @@ from litellm.router_utils.handle_error import ( send_llm_exception_alert, ) from litellm.router_utils.health_state_cache import DeploymentHealthCache -from litellm.router_utils.model_request_defaults import deployment_params_with_request_defaults, model_request_defaults from litellm.router_utils.kubernetes_pod_discovery import ( KubernetesPodDiscovery, async_resolve_pods_after, @@ -298,6 +297,7 @@ from litellm.router_utils.kubernetes_pod_discovery import ( resolve_pods_after, resolve_pods_after_bound, ) +from litellm.router_utils.model_request_defaults import deployment_params_with_request_defaults, model_request_defaults from litellm.router_utils.pre_call_checks.continuation_prefill_check import ( MID_STREAM_CONTINUATION_MARKER, ContinuationOutput, @@ -451,7 +451,6 @@ from litellm.utils import ( lookup_model_info_for_registration, model_info_with_pricing_overrides, provider_rejectable_params, - resolve_api_key_secret_reference, set_live_deployment_replay, ) @@ -11541,7 +11540,7 @@ class Router: deployment alone, which is what lets a refresh rebuild the same entries. """ model_info: Final = dict( - _MODEL_INFO_ADAPTER.validate_python(deployment.model_info.model_dump(exclude_none=True)) + _MODEL_INFO_ADAPTER.validate_python(deployment.model_info.model_dump(mode="json", exclude_none=True)) ) for field in CustomPricingLiteLLMParams.model_fields: field_value = deployment.litellm_params.get(field) diff --git a/tests/unit/test_model_prices_and_context_window.py b/tests/unit/test_model_prices_and_context_window.py index ce9847dca33..13e9bd38a5c 100644 --- a/tests/unit/test_model_prices_and_context_window.py +++ b/tests/unit/test_model_prices_and_context_window.py @@ -736,27 +736,6 @@ def test_unregistered_provider_guard_flags_only_labels_nobody_registered(): ] -def test_schema_and_model_info_preserve_independent_total_context(committed_schema, monkeypatch): - entry = { - "litellm_provider": "openai", - "context_window": 1200, - "max_input_tokens": 1100, - "max_output_tokens": 200, - "max_tokens": 200, - } - # The trusted-base generated catalog schema permits optional unknown properties. - # Runtime metadata supplies and validates the separate total-context contract. - assert build_validator(committed_schema).is_valid({"fixture-model": entry}) - monkeypatch.setattr(litellm, "model_cost", {"fixture-model": entry}) - info = litellm.get_model_info("fixture-model") - assert (info["context_window"], info["max_input_tokens"], info["max_output_tokens"], info["max_tokens"]) == ( - 1200, - 1100, - 200, - 200, - ) - - def test_native_model_info_and_lookup_preserve_reasoning_capabilities_and_request_defaults( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -766,6 +745,7 @@ def test_native_model_info_and_lookup_preserve_reasoning_capabilities_and_reques "context_window": 1200, "max_input_tokens": 1100, "max_output_tokens": 200, + "max_tokens": 200, "supports_function_calling": True, "supports_parallel_function_calling": False, "supports_reasoning": True, diff --git a/ui/litellm-dashboard/src/lib/http/schema.d.ts b/ui/litellm-dashboard/src/lib/http/schema.d.ts index 262e8b6deae..451bb9600e7 100644 --- a/ui/litellm-dashboard/src/lib/http/schema.d.ts +++ b/ui/litellm-dashboard/src/lib/http/schema.d.ts @@ -30205,6 +30205,11 @@ export interface components { * @description When true, `/models`, `/v1/models/{id}` and `/model/info` hide models whose backing deployments are all unhealthy, for every caller, without needing `healthy_only=true` per request. Requires `background_health_checks: true`, and keeps deployment health state cached without turning on `enable_health_check_routing`, so routing is unaffected. With no health state nothing is hidden. Hiding is presentation-only, a hidden model can still be called. */ model_list_healthy_only?: boolean | null; + /** + * Model Offerings Path + * @description Path to the authoritative external offering file, reloaded without restarting the proxy + */ + model_offerings_path?: string | null; /** * Otel * @description [BETA] OpenTelemetry support - this might change, use with caution. @@ -31799,6 +31804,8 @@ export interface components { model_info: components["schemas"]["litellm__types__router__ModelInfo"]; /** Model Name */ model_name: string; + /** @description Gateway-owned supplier selection, excluded from public model metadata */ + provider_policy?: components["schemas"]["OpenRouterProviderPolicy"] | null; } & { [key: string]: unknown; }; @@ -38600,6 +38607,21 @@ export interface components { /** Model Name */ model_name: string; }; + /** ModelRequestDefaults */ + ModelRequestDefaults: { + /** + * Output Token Budget + * @description Default requested output budget, not a physical model limit + */ + output_token_budget?: number | null; + /** + * Output Token Budget By Reasoning Effort + * @description Requested output budgets keyed by exact reasoning effort + */ + output_token_budget_by_reasoning_effort?: { + [key: string]: number; + } | null; + }; /** ModelResponse */ ModelResponse: { /** Choices */ @@ -39867,6 +39889,22 @@ export interface components { */ type: "openIdConnect"; }; + /** OpenRouterProviderPolicy */ + OpenRouterProviderPolicy: { + /** + * Allow Fallbacks + * @default false + * @constant + */ + allow_fallbacks: false; + /** Only */ + only: string[]; + /** + * Provider + * @constant + */ + provider: "openrouter"; + }; /** OpenSourceClassifierConfig */ OpenSourceClassifierConfig: { /** @@ -50210,6 +50248,16 @@ export interface components { litellm__proxy___types__ModelInfo: { /** Base Model */ base_model: ("gpt-4-1106-preview" | "gpt-4-32k" | "gpt-4" | "gpt-3.5-turbo-16k" | "gpt-3.5-turbo" | "text-embedding-ada-002") | null; + /** + * Context Window + * @description Maximum combined input and generated output tokens + */ + context_window?: number | null; + /** + * Default Reasoning Effort + * @description Effort used when the request omits it + */ + default_reasoning_effort?: string | null; /** Discoverable */ discoverable?: boolean | null; /** Id */ @@ -50219,6 +50267,16 @@ export interface components { * @default 0 */ input_cost_per_token: number | null; + /** + * Max Input Tokens + * @description Independent maximum input tokens + */ + max_input_tokens?: number | null; + /** + * Max Output Tokens + * @description Independent maximum generated output tokens + */ + max_output_tokens?: number | null; /** * Max Tokens * @default 2048 @@ -50231,6 +50289,43 @@ export interface components { * @default 0 */ output_cost_per_token: number | null; + /** + * Reasoning Effort Levels + * @description Exact effort values this route accepts + */ + reasoning_effort_levels?: string[] | null; + /** @description Configured request defaults, independent of supplier capability limits */ + request_defaults?: components["schemas"]["ModelRequestDefaults"] | null; + /** + * Supported Endpoints + * @description Inference endpoints exposed by this route + */ + supported_endpoints?: string[] | null; + /** + * Supported Modalities + * @description Accepted input modalities + */ + supported_modalities?: string[] | null; + /** + * Supported Output Modalities + * @description Supported output modalities + */ + supported_output_modalities?: string[] | null; + /** + * Supports Function Calling + * @description Whether this model accepts tool calls + */ + supports_function_calling?: boolean | null; + /** + * Supports Parallel Function Calling + * @description Whether this model accepts parallel tool calls + */ + supports_parallel_function_calling?: boolean | null; + /** + * Supports Reasoning + * @description Whether this model supports reasoning controls + */ + supports_reasoning?: boolean | null; } & { [key: string]: unknown; }; @@ -50281,6 +50376,11 @@ export interface components { cache_creation_input_token_cost_above_1hr?: number | null; /** Cache Read Input Token Cost */ cache_read_input_token_cost?: number | null; + /** + * Context Window + * @description Maximum combined input and generated output tokens + */ + context_window?: number | null; /** Cost Per Ptu Per Hour */ cost_per_ptu_per_hour?: number | null; /** Created At */ @@ -50292,6 +50392,11 @@ export interface components { * @default false */ db_model: boolean; + /** + * Default Reasoning Effort + * @description Effort used when the request omits it + */ + default_reasoning_effort?: string | null; /** Discoverable */ discoverable?: boolean | null; /** Enable Tag Filtering */ @@ -50308,6 +50413,16 @@ export interface components { internal_router_model?: boolean | null; /** Max Embedding Batch Size */ max_embedding_batch_size?: number | null; + /** + * Max Input Tokens + * @description Independent maximum input tokens + */ + max_input_tokens?: number | null; + /** + * Max Output Tokens + * @description Independent maximum generated output tokens + */ + max_output_tokens?: number | null; /** * Member Auto Router * @default false @@ -50329,6 +50444,43 @@ export interface components { ptu_shares?: { [key: string]: number; } | null; + /** + * Reasoning Effort Levels + * @description Exact effort values this route accepts + */ + reasoning_effort_levels?: string[] | null; + /** @description Configured request defaults, independent of supplier capability limits */ + request_defaults?: components["schemas"]["ModelRequestDefaults"] | null; + /** + * Supported Endpoints + * @description Inference endpoints exposed by this route + */ + supported_endpoints?: string[] | null; + /** + * Supported Modalities + * @description Accepted input modalities + */ + supported_modalities?: string[] | null; + /** + * Supported Output Modalities + * @description Supported output modalities + */ + supported_output_modalities?: string[] | null; + /** + * Supports Function Calling + * @description Whether this model accepts tool calls + */ + supports_function_calling?: boolean | null; + /** + * Supports Parallel Function Calling + * @description Whether this model accepts parallel tool calls + */ + supports_parallel_function_calling?: boolean | null; + /** + * Supports Reasoning + * @description Whether this model supports reasoning controls + */ + supports_reasoning?: boolean | null; /** Team Id */ team_id?: string | null; /** Team Public Model Name */