fix(router): align metadata with native registration and reasoning paths

This commit is contained in:
J.H.S. Schlepers 2026-10-26 08:21:38 +01:00
parent 4ed36dc9dc
commit 23ae9fbe17
7 changed files with 166 additions and 29 deletions

View file

@ -4,7 +4,7 @@ Automatic supplier metadata is the normal path for AUTO offerings. Selecting an
The schema still requires an explicit `source`. Normal automatic usage selects `source: auto` without copying a capability card; it does not mean an omitted `source` field is accepted
This walkthrough is verified against runtime commit `ec1e681d95dca0e6660e21fba9060ec9e3c7afde`. Relative source links follow the code in the same checkout. The feature extends native LiteLLM adapters, types, authentication, listing and Router paths; referencing those paths does not mean every implementation in them originated in this PR. See the [setup guide](README.md) for configuration and lifecycle instructions
This walkthrough includes the upstream sync through `dddbaa803d6e9fcc862ea7a5412f771cb1ae040e`. Relative source links follow the code in the same checkout, and the PR records validation against its published head. The feature extends native LiteLLM adapters, types, authentication, listing and Router paths; referencing those paths does not mean every implementation in them originated in this PR. See the [setup guide](README.md) for configuration and lifecycle instructions
```text
supplier connection -> inventory -> native normalization -> offering resolution
@ -177,6 +177,8 @@ It chooses the budget's effort key from request effort first, then deployment re
The helper returns `max_output_tokens` for Responses calls and `max_tokens` for the other supported calls. It does not inject a reasoning effort just because a default was advertised. An explicit caller budget wins over this configured default-budget policy, and conflicting deployment budget parameters are removed in that policy path. Native/provider request translation still determines the eventual wire format
The selector reads `reasoning_effort` and `reasoning.effort`; it is not a second implementation of every provider's thinking controls. In particular, its budget choice precedes later native request-level effort rounding. Do not assume an effort-specific default budget is recomputed for a different, unsupported effort that upstream subsequently rounds, or that every `output_config`/thinking-budget shape supplies this selector's effort key
For the manual example, an omitted effort uses the base requested budget of `65536`; explicit `max` selects `131072`. Neither value means the physical output ceiling is known. This helper is defaulting, not a provider-independent quota or proof that a chosen budget is feasible
An advertised reasoning default, a client electing to send that default, an explicit deployment request setting and a provider adapter's own default behavior are different things. The offering manager itself does not turn every advertised default into an outgoing request parameter. With no explicit effort setting or provider-specific injection, omission leaves the supplier's behavior in charge
@ -199,6 +201,10 @@ Variant syntax and UI conversion belong to the client, not this LiteLLM PR. A cl
For inference, the public alias resolves to a selected native deployment. The Router applies request-default budget policy and dispatches through the selected provider adapter. The [OpenRouter chat adapter](../../litellm/llms/openrouter/chat/transformation.py) preserves explicit gateway `max`/`xhigh`, and the [Messages bridge normalization](../../litellm/llms/anthropic/pass_through/utils.py) does not reduce OpenRouter effort using unrelated backend flags. Native Responses and Responses-to-chat paths retain their API-specific request shapes
Upstream's [deployment reasoning resolver](../../litellm/router_utils/reasoning_effort_capability.py) now owns deployment-declared effort handling across chat, Responses and Messages. An accepted declared value is kept. A ranked request outside the declared set can be rounded according to native up/down policy, or dropped under native `drop_params` behavior; explicit `default` resolves to omission. Cost-map-only choices do not activate deployment rounding. Supplier choices resolved into an AUTO deployment's own `model_info` do count as its declaration. The external offering schema does not add a parallel rounding framework or expose every native Router setting
Consequently, preserving accepted OpenRouter gateway efforts is not an all-provider or all-input passthrough guarantee. The fork still avoids the provider adapter's old unconditional max-to-xhigh rename and legacy undeclared OpenRouter approximation, while reusing upstream's declared-effort resolver. Other provider adapters keep their native request semantics. The shared native [deployment-pricing resolver](../../litellm/router_utils/deployment_pricing.py) also owns base-model key precedence for listing; the former fork-only key lookup is removed
The offering guard merges the exact configured `only`/`allow_fallbacks: false` body and checks it again at deployment dispatch. Caller connection, deployment, provider, fallback, tier and parameter-dropping overrides cannot replace it. Pinned response-cache reads/writes are disabled to prevent output from another endpoint satisfying the request. No `require_parameters` setting or fallback supplier is silently added. Unsupported pinned API surfaces and native batches are rejected rather than bypassing selection
## 9. Refresh, failure and present limitations

View file

@ -102,7 +102,7 @@ AUTO still checks model membership. Pinned AUTO offerings additionally discover
Reasoning choices and defaults come from OpenRouter's model-level reasoning contract and are advertised only when every eligible endpoint declares the effort control. An endpoint's more restrictive declared choices narrow that set. This describes OpenRouter's accepted gateway options, not an independently discovered backend-native effort enum. An explicit empty choice list remains empty. Explicit null expands only to OpenRouter's documented gateway efforts, while omission does not declare choices. Mandatory reasoning excludes the disable value, and a disabled-by-default model does not claim an active default effort. Boolean reasoning support alone never supplies choices. Endpoint capability guarantees are combined conservatively, without using status, uptime or performance to change routing or availability
Every explicit `model_info` field remains authoritative. Existing handwritten context/output limits continue overriding discovery until the operator removes them. MANUAL offerings remain entirely handwritten. Requests without an effort leave the supplier default unchanged; explicit gateway effort values, including `max` and `xhigh`, are forwarded without backend-flag approximation on chat and translated Messages paths
Every explicit `model_info` field remains authoritative. Existing handwritten context/output limits continue overriding discovery until the operator removes them. MANUAL offerings remain entirely handwritten. An advertised default does not itself inject request effort. Accepted declared OpenRouter gateway efforts, including `max` and `xhigh`, are preserved without the old provider rename or backend-flag approximation. Upstream's native deployment resolver owns unsupported-value rounding/default/drop behavior across providers and routes; this is not a promise to forward every arbitrary effort unchanged
Successful empty endpoint inventories clear derived endpoint metadata. Failed refreshes retain last-known endpoint metadata only for the same connection identity; an initial failure leaves unknown fields omitted, never replaced with another supplier's limits. Endpoint discovery does not disable an otherwise selected model or broaden its policy. Metadata refreshes publish atomically, and requests already running retain their previous snapshot. Policy edits still change the affected offering's private deployment ID

View file

@ -210,8 +210,6 @@ def normalize_reasoning_effort_value(
handled where the thinking block is built. A deployment accepting no tier at all keeps the
chain's floor, which is what every deployment degraded to before there was anything to ask.
"""
if custom_llm_provider == "openrouter" or custom_llm_provider is None and model.startswith("openrouter/"):
return effort
chain: Final = _EFFORT_DEGRADATION_CHAIN.get(effort)
if chain is None:
return effort
@ -223,6 +221,8 @@ def normalize_reasoning_effort_value(
if deployment_rounds_reasoning_effort(deployment_model_info):
return effort
if custom_llm_provider == "openrouter" or custom_llm_provider is None and model.startswith("openrouter/"):
return effort
from litellm.utils import get_model_info
try:

View file

@ -167,8 +167,8 @@ from litellm.proxy.common_utils.token_count_settings import (
resolve_token_count_settings,
)
from litellm.proxy.management_helpers.auto_router_availability import AutoRouterCatalogEntry, build_auto_router_catalog
from litellm.proxy.model_discovery import discover_model_metadata
from litellm.proxy.management_helpers.model_group_custom_fields import model_group_custom_fields
from litellm.proxy.model_discovery import discover_model_metadata
from litellm.router_utils.access_windows import access_windows_config_error
from litellm.router_utils.add_retry_fallback_headers import (
get_fallback_errors_from_headers,

View file

@ -290,7 +290,6 @@ from litellm.router_utils.handle_error import (
send_llm_exception_alert,
)
from litellm.router_utils.health_state_cache import DeploymentHealthCache
from litellm.router_utils.model_request_defaults import deployment_params_with_request_defaults, model_request_defaults
from litellm.router_utils.kubernetes_pod_discovery import (
KubernetesPodDiscovery,
async_resolve_pods_after,
@ -298,6 +297,7 @@ from litellm.router_utils.kubernetes_pod_discovery import (
resolve_pods_after,
resolve_pods_after_bound,
)
from litellm.router_utils.model_request_defaults import deployment_params_with_request_defaults, model_request_defaults
from litellm.router_utils.pre_call_checks.continuation_prefill_check import (
MID_STREAM_CONTINUATION_MARKER,
ContinuationOutput,
@ -451,7 +451,6 @@ from litellm.utils import (
lookup_model_info_for_registration,
model_info_with_pricing_overrides,
provider_rejectable_params,
resolve_api_key_secret_reference,
set_live_deployment_replay,
)
@ -11541,7 +11540,7 @@ class Router:
deployment alone, which is what lets a refresh rebuild the same entries.
"""
model_info: Final = dict(
_MODEL_INFO_ADAPTER.validate_python(deployment.model_info.model_dump(exclude_none=True))
_MODEL_INFO_ADAPTER.validate_python(deployment.model_info.model_dump(mode="json", exclude_none=True))
)
for field in CustomPricingLiteLLMParams.model_fields:
field_value = deployment.litellm_params.get(field)

View file

@ -736,27 +736,6 @@ def test_unregistered_provider_guard_flags_only_labels_nobody_registered():
]
def test_schema_and_model_info_preserve_independent_total_context(committed_schema, monkeypatch):
entry = {
"litellm_provider": "openai",
"context_window": 1200,
"max_input_tokens": 1100,
"max_output_tokens": 200,
"max_tokens": 200,
}
# The trusted-base generated catalog schema permits optional unknown properties.
# Runtime metadata supplies and validates the separate total-context contract.
assert build_validator(committed_schema).is_valid({"fixture-model": entry})
monkeypatch.setattr(litellm, "model_cost", {"fixture-model": entry})
info = litellm.get_model_info("fixture-model")
assert (info["context_window"], info["max_input_tokens"], info["max_output_tokens"], info["max_tokens"]) == (
1200,
1100,
200,
200,
)
def test_native_model_info_and_lookup_preserve_reasoning_capabilities_and_request_defaults(
monkeypatch: pytest.MonkeyPatch,
) -> None:
@ -766,6 +745,7 @@ def test_native_model_info_and_lookup_preserve_reasoning_capabilities_and_reques
"context_window": 1200,
"max_input_tokens": 1100,
"max_output_tokens": 200,
"max_tokens": 200,
"supports_function_calling": True,
"supports_parallel_function_calling": False,
"supports_reasoning": True,

View file

@ -30205,6 +30205,11 @@ export interface components {
* @description When true, `/models`, `/v1/models/{id}` and `/model/info` hide models whose backing deployments are all unhealthy, for every caller, without needing `healthy_only=true` per request. Requires `background_health_checks: true`, and keeps deployment health state cached without turning on `enable_health_check_routing`, so routing is unaffected. With no health state nothing is hidden. Hiding is presentation-only, a hidden model can still be called.
*/
model_list_healthy_only?: boolean | null;
/**
* Model Offerings Path
* @description Path to the authoritative external offering file, reloaded without restarting the proxy
*/
model_offerings_path?: string | null;
/**
* Otel
* @description [BETA] OpenTelemetry support - this might change, use with caution.
@ -31799,6 +31804,8 @@ export interface components {
model_info: components["schemas"]["litellm__types__router__ModelInfo"];
/** Model Name */
model_name: string;
/** @description Gateway-owned supplier selection, excluded from public model metadata */
provider_policy?: components["schemas"]["OpenRouterProviderPolicy"] | null;
} & {
[key: string]: unknown;
};
@ -38600,6 +38607,21 @@ export interface components {
/** Model Name */
model_name: string;
};
/** ModelRequestDefaults */
ModelRequestDefaults: {
/**
* Output Token Budget
* @description Default requested output budget, not a physical model limit
*/
output_token_budget?: number | null;
/**
* Output Token Budget By Reasoning Effort
* @description Requested output budgets keyed by exact reasoning effort
*/
output_token_budget_by_reasoning_effort?: {
[key: string]: number;
} | null;
};
/** ModelResponse */
ModelResponse: {
/** Choices */
@ -39867,6 +39889,22 @@ export interface components {
*/
type: "openIdConnect";
};
/** OpenRouterProviderPolicy */
OpenRouterProviderPolicy: {
/**
* Allow Fallbacks
* @default false
* @constant
*/
allow_fallbacks: false;
/** Only */
only: string[];
/**
* Provider
* @constant
*/
provider: "openrouter";
};
/** OpenSourceClassifierConfig */
OpenSourceClassifierConfig: {
/**
@ -50210,6 +50248,16 @@ export interface components {
litellm__proxy___types__ModelInfo: {
/** Base Model */
base_model: ("gpt-4-1106-preview" | "gpt-4-32k" | "gpt-4" | "gpt-3.5-turbo-16k" | "gpt-3.5-turbo" | "text-embedding-ada-002") | null;
/**
* Context Window
* @description Maximum combined input and generated output tokens
*/
context_window?: number | null;
/**
* Default Reasoning Effort
* @description Effort used when the request omits it
*/
default_reasoning_effort?: string | null;
/** Discoverable */
discoverable?: boolean | null;
/** Id */
@ -50219,6 +50267,16 @@ export interface components {
* @default 0
*/
input_cost_per_token: number | null;
/**
* Max Input Tokens
* @description Independent maximum input tokens
*/
max_input_tokens?: number | null;
/**
* Max Output Tokens
* @description Independent maximum generated output tokens
*/
max_output_tokens?: number | null;
/**
* Max Tokens
* @default 2048
@ -50231,6 +50289,43 @@ export interface components {
* @default 0
*/
output_cost_per_token: number | null;
/**
* Reasoning Effort Levels
* @description Exact effort values this route accepts
*/
reasoning_effort_levels?: string[] | null;
/** @description Configured request defaults, independent of supplier capability limits */
request_defaults?: components["schemas"]["ModelRequestDefaults"] | null;
/**
* Supported Endpoints
* @description Inference endpoints exposed by this route
*/
supported_endpoints?: string[] | null;
/**
* Supported Modalities
* @description Accepted input modalities
*/
supported_modalities?: string[] | null;
/**
* Supported Output Modalities
* @description Supported output modalities
*/
supported_output_modalities?: string[] | null;
/**
* Supports Function Calling
* @description Whether this model accepts tool calls
*/
supports_function_calling?: boolean | null;
/**
* Supports Parallel Function Calling
* @description Whether this model accepts parallel tool calls
*/
supports_parallel_function_calling?: boolean | null;
/**
* Supports Reasoning
* @description Whether this model supports reasoning controls
*/
supports_reasoning?: boolean | null;
} & {
[key: string]: unknown;
};
@ -50281,6 +50376,11 @@ export interface components {
cache_creation_input_token_cost_above_1hr?: number | null;
/** Cache Read Input Token Cost */
cache_read_input_token_cost?: number | null;
/**
* Context Window
* @description Maximum combined input and generated output tokens
*/
context_window?: number | null;
/** Cost Per Ptu Per Hour */
cost_per_ptu_per_hour?: number | null;
/** Created At */
@ -50292,6 +50392,11 @@ export interface components {
* @default false
*/
db_model: boolean;
/**
* Default Reasoning Effort
* @description Effort used when the request omits it
*/
default_reasoning_effort?: string | null;
/** Discoverable */
discoverable?: boolean | null;
/** Enable Tag Filtering */
@ -50308,6 +50413,16 @@ export interface components {
internal_router_model?: boolean | null;
/** Max Embedding Batch Size */
max_embedding_batch_size?: number | null;
/**
* Max Input Tokens
* @description Independent maximum input tokens
*/
max_input_tokens?: number | null;
/**
* Max Output Tokens
* @description Independent maximum generated output tokens
*/
max_output_tokens?: number | null;
/**
* Member Auto Router
* @default false
@ -50329,6 +50444,43 @@ export interface components {
ptu_shares?: {
[key: string]: number;
} | null;
/**
* Reasoning Effort Levels
* @description Exact effort values this route accepts
*/
reasoning_effort_levels?: string[] | null;
/** @description Configured request defaults, independent of supplier capability limits */
request_defaults?: components["schemas"]["ModelRequestDefaults"] | null;
/**
* Supported Endpoints
* @description Inference endpoints exposed by this route
*/
supported_endpoints?: string[] | null;
/**
* Supported Modalities
* @description Accepted input modalities
*/
supported_modalities?: string[] | null;
/**
* Supported Output Modalities
* @description Supported output modalities
*/
supported_output_modalities?: string[] | null;
/**
* Supports Function Calling
* @description Whether this model accepts tool calls
*/
supports_function_calling?: boolean | null;
/**
* Supports Parallel Function Calling
* @description Whether this model accepts parallel tool calls
*/
supports_parallel_function_calling?: boolean | null;
/**
* Supports Reasoning
* @description Whether this model supports reasoning controls
*/
supports_reasoning?: boolean | null;
/** Team Id */
team_id?: string | null;
/** Team Public Model Name */