mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
5f2986a1f3
* feat(complexity_router): calibrate the classifier rubric with worked examples The built-in rubric stated its tier boundaries as prose alone, and prose calibrated to consumer chat puts "non-trivial code, multi-step technical work" at the top of the scale. That is the median request in developer and agent traffic, so ordinary engineering read as top-tier and the router paid for the most expensive model on it. Adds calibration examples to the rubric, selected by a new classifier_llm_config.rubric preset. The agentic preset (now the default) anchors routine installs, builds, multi-file edits, and standard debugging at MEDIUM; the chat preset omits those anchors for deployments serving only conversational traffic. Both share the same tier criteria, the trust-boundary paragraph, and the context-window closing line, so this moves where the boundary sits without changing the taxonomy. Both presets render byte-identical to the strings a prompt sweep scored, and a test pins that, so the measured accuracy describes what a router sends. * feat(ui): pick the classifier rubric preset on an auto-router Adds a Rubric dropdown to the auto-router's classification panel, so the agentic and chat presets are selectable rather than config-file only. The prompt editor prefills from the selected preset, since prefilling agentic text for a router on chat would show examples its classifier never receives. The picker is disabled while a custom prompt is set, and the payload builder drops the preset in that case: a custom prompt is the classifier's whole system role, so the backend rejects the two together. The builder records the default preset explicitly, so a later change to which preset is default cannot silently move an existing router. * fix(complexity_router): mark an unchosen rubric preset with None, not model_fields_set The mutual-exclusion check read model_fields_set to tell an explicit preset from the default. That flag does not survive serialization, and this config is dumped and handed straight back to ComplexityRouter by /auto_router/test_routing, where a dump re-states every field. So a custom-prompt classifier saved fine and then failed validation on preview, rejecting on the second pass what it accepted on the first. The preset is now optional, with None meaning the default, matching how None already means the built-in rubric for system_prompt on the same model. The default lives in one place, DEFAULT_RUBRIC_PRESET, resolved where the prompt is assembled. The dashboard stops sending a copy of the default it displays, so a router nobody configured follows the default rather than pinning today's value, and UI-built routers behave the same as hand-written config. Regenerates schema.d.ts, which was left stale by an earlier description edit. * feat(complexity_router): grandfather existing routers onto the uncalibrated rubric An unset preset now means LEGACY, the rubric exactly as it shipped before calibration examples existed, so upgrading cannot move the tier decisions or the bill of a router that is already running. Config-file routers get this for free since they name no preset, and a stored config that never had one reads the same way. New routers still get the calibrated rubric: switching a classifier to LLM stamps the agentic preset, because a classifier being configured for the first time has no prior tier behaviour to preserve. The picker offers legacy so an existing router's state is representable and opening the form cannot silently upgrade it. Each preset is pinned byte-identical to the text the prompt sweep scored, legacy included, which is what proves an existing router's prompt did not move. Also collapses the preset data from a NamedTuple with group wrappers and per-preset frozensets into plain text blocks in a MappingProxyType, matching how the tier criteria next to it are already stored: 21 lines of prompt text no longer cost 190 lines of constructors. Tiers are format placeholders so tier_labels still reach the examples. * refactor(complexity_router): name the field classification_rubric `rubric` alone did not say what it selects, and the field sits beside `system_prompt`, which genuinely is the whole classification prompt. The name now says which of the two an operator is reaching for: the rubric the built-in prompt is assembled from, not the prompt itself. Renames the config field, the query param, the enum, and the dashboard label to match, and moves the preset text to classification_rubrics.py. * test(ui): set the preset the mutual-exclusion case is meant to drop The rename left classification_classification_rubric in the custom-prompt case, so its input never carried a preset and the assertion held for the wrong reason: it proved an absent preset stays absent, not that a set one is dropped. A normalizer that forwards the preset whenever one is set passed with the typo and fails without it. tsc reports the typo as TS2353; the earlier sweep grepped for the source file and not the test, so it went unseen. * test(ui): scope the role-gate assertions to each page's own endpoint The memory, workflows, and guardrails-monitor page tests asserted that a denied role fires no request at all. Their names, and the assertion on the very next line, say the intent is narrower: the page must not fetch its own data. Resolving whether a caller is an org admin goes through /organization/list for every role, since deciding org-admin-for-any-org needs the list, and the route scopes rows per caller. That legitimate request fails a blanket no-fetch assertion, so all three files went red on staging for a reason unrelated to what they test. Drops the blanket assertion and keeps the scoped one. Bypassing the gate in memory/page.tsx still fails five tests, so the narrower assertion continues to catch a genuinely broken gate. * fix(complexity_router): document that an unset rubric keeps the legacy prompt The field said 'Leave unset for agentic' while an omitted rubric resolves to LEGACY, so the OpenAPI schema an operator reads promised calibrated routing where they got the uncalibrated one. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _experimental/mcp_server | ||
| a2a | ||
| agent_endpoints | ||
| analytics_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| batches_endpoints | ||
| client | ||
| common_utils | ||
| config_resolvers | ||
| credential_endpoints | ||
| db | ||
| discovery_endpoints | ||
| enterprise_billing | ||
| experimental/mcp_server | ||
| fine_tuning_endpoints | ||
| google_endpoints | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| logging_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| memory | ||
| middleware | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| proxy_server | ||
| public_endpoints | ||
| rag_endpoints | ||
| realtime_endpoints | ||
| response_api_endpoints | ||
| shutdown | ||
| spend_tracking | ||
| test_configs | ||
| types_utils | ||
| ui_crud_endpoints | ||
| utils | ||
| vector_store_endpoints | ||
| vector_store_files_endpoints | ||
| video_endpoints | ||
| __init__.py | ||
| conftest.py | ||
| test_aiohttp_cleanup_closed.py | ||
| test_aiohttp_session_recovery.py | ||
| test_api_key_masking_in_errors.py | ||
| test_audio_speech_prometheus_hooks.py | ||
| test_batch_expiry.py | ||
| test_batch_metadata_none_fix.py | ||
| test_batch_retrieve_bedrock.py | ||
| test_batch_x_litellm_model_encoding.py | ||
| test_blocked_response_usage.py | ||
| test_budget_reservation.py | ||
| test_caching_routes.py | ||
| test_chat_completion_metadata.py | ||
| test_common_request_processing.py | ||
| test_component_allowlists.py | ||
| test_cors_config.py | ||
| test_custom_proxy.py | ||
| test_dynamic_mcp_route.py | ||
| test_empty_model_list.py | ||
| test_enforce_user_param.py | ||
| test_fallback_management_endpoints.py | ||
| test_fastapi_offline_routes.py | ||
| test_filter_models_by_team_access_group.py | ||
| test_health_check_functions.py | ||
| test_health_check_max_tokens.py | ||
| test_langfuse_passthrough_security.py | ||
| test_lazy_openapi_snapshot.py | ||
| test_litellm_pre_call_utils.py | ||
| test_max_budget_env_var.py | ||
| test_mcp_asgi_response.py | ||
| test_model_based_routing_files_batches.py | ||
| test_model_dump_with_preserved_fields.py | ||
| test_model_id_header_propagation.py | ||
| test_model_info_default_limits.py | ||
| test_model_level_guardrails.py | ||
| test_model_list_healthy_only.py | ||
| test_modify_response_streaming_passthrough.py | ||
| test_openapi_schema_validation.py | ||
| test_plugin_routes.py | ||
| test_pricing_field_strip.py | ||
| test_prometheus_cleanup.py | ||
| test_provider_url_destination_guard.py | ||
| test_proxy_cli.py | ||
| test_proxy_logging_hook_detection.py | ||
| test_proxy_server.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_pyroscope.py | ||
| test_read_model_list.py | ||
| test_redis_auth_cache_flag.py | ||
| test_response_model_sanitization.py | ||
| test_route_a2a_models.py | ||
| test_route_llm_request.py | ||
| test_sensitive_route_auth.py | ||
| test_shared_health_check.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||
| test_team_org_move.py | ||
| test_tools_allowlist_enforcement.py | ||
| test_update_llm_router_resilience.py | ||