mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
1f4acbb924
* feat(complexity_router): custom classifier plugins via classifier_type 'plugin' Adds a third classification mode where an operator-supplied hook decides the tier instead of the heuristic scorer or the LLM classifier. The hook implements an async classify(context) returning a tier name (built-in value, tier_labels label, or tier_definitions name) or None to decline; failures, timeouts, and unknown tiers fall back exactly like a failed LLM classifier. The context carries the request messages and metadata, including caller identity, so a plugin can route by team, spend, or any business rule. The plugin resolves from a dotted path at proxy startup with a load-time check that classify is a coroutine function, and is closed off over HTTP like the routing plugins list. Routing decisions record the new classifier_plugin cause. tier_definitions now accepts classifier_type 'plugin' alongside 'llm'. * fix(proxy): resolve plugin dotted paths in _delete_deployment before hashing ids The db-sync reconcile re-reads the raw config and hashes litellm_params to compute which ids the config wants served, but the router's ids were hashed from the resolved params where plugin dotted paths are live instances. The mismatched ids made the reconcile evict every plugin-bearing auto-router one sync after startup, on any proxy with a database connected. This also affected the existing routing plugins list, not just the new classifier plugin. Resolving the plugins in _delete_deployment the same way load_config does makes both sides hash the same canonical form. A plugin module broken on disk at reconcile time skips cleanup instead of evicting valid deployments, matching how a get_config failure is handled * fix(complexity_router): treat non-string plugin verdicts as declines, centralize the empty-mapping sentinel A hook returning a non-string raised inside resolve_classified_tier outside the plugin exception boundary, failing the request instead of falling back. Also moves the read-only empty mapping to constants.py per repo convention and moves the classifier plugin product docs out of the package README for the docs repo * refactor(complexity_router): rename the plugin classifier mode to classifier_type 'custom' The mode value now names the operator's intent while classifier_plugin keeps naming the mechanism; routing decisions keep the classifier_plugin cause * refactor(proxy): pin plugin-bearing deployment ids from the raw params instead of resolving in the reconcile Replaces the previous approach of re-running plugin resolution inside _delete_deployment, which imported operator modules on every reconcile cycle and skipped the whole cleanup pass when any one module was broken on disk. load_config now stamps model_info.id from the raw litellm_params before resolution swaps dotted paths for live instances, so the reconcile's raw-config hash matches by construction and needs no resolution at all: a broken module cannot stall cleanup for unrelated models, and any future param-transforming resolution is covered by the same pin. _generate_model_id becomes a staticmethod so the pin can run before the Router exists; its statically dead non-string key branches are removed. Also documents candidate_models as an informational snapshot for classifier plugins, unlike the narrowing surface RoutingPlugin filters * fix(router): restore _generate_model_id key handling, align classifier context with the routing-plugin pattern The staticmethod conversion accidentally dropped the non-string-key branches from _generate_model_id, a silent hash change for any params with non-string keys; they are restored verbatim. The classifier plugin context now follows the Router-level routing-plugin recipe exactly: structured messages come from resolve_structured_messages over the raw messages, and the metadata key comes from the shared get_metadata_variable_name_from_kwargs helper, which also replaces the duplicated inline sniff in _pick_model_for_tier. This removes the raw-or-resolved fallback where a plugin could silently receive resolved messages when a call site forgot to pass the raw ones * refactor(router): make generate_model_id public, guard classifier context construction Two modules legitimately hash deployment ids with the same helper now (Router and the proxy's config-load pin), so the private name was lying about its audience and the cross-module call needed a pyright suppression; renaming it public restores the static safety net. The classifier plugin's RoutingContext construction moves inside the failure boundary, matching the LLM path where litellm-side prompt building also falls back rather than failing the request, and a prompt-only call with no message list is now covered by a test |
||
|---|---|---|
| .. | ||
| .litellm_cache | ||
| auto_router | ||
| example_config_yaml | ||
| test_configs | ||
| test_model_response_typing | ||
| azure_fine_tune.jsonl | ||
| azure_speech.mp3 | ||
| batch_job_results_furniture.jsonl | ||
| cache_unit_tests.py | ||
| conftest.py | ||
| create_mock_standard_logging_payload.py | ||
| data_map.txt | ||
| eagle.wav | ||
| example.jsonl | ||
| gettysburg.wav | ||
| large_text.py | ||
| model_cost.json | ||
| openai_batch_completions.jsonl | ||
| openai_batch_completions_router.jsonl | ||
| speech_vertex.mp3 | ||
| stream_chunk_testdata.py | ||
| test_acompletion.py | ||
| test_acompletion_fallbacks.py | ||
| test_acooldowns_router.py | ||
| test_add_function_to_prompt.py | ||
| test_aim_guardrails.py | ||
| test_alangfuse.py | ||
| test_amazing_vertex_completion.py | ||
| test_anthropic_prompt_caching.py | ||
| test_arize_ai.py | ||
| test_arize_phoenix.py | ||
| test_assistants.py | ||
| test_async_fn.py | ||
| test_auth_utils.py | ||
| test_azure_anthropic_sync_post.py | ||
| test_azure_openai.py | ||
| test_azure_perf.py | ||
| test_basic_python_version.py | ||
| test_batch_completion_return_exceptions.py | ||
| test_batch_completions.py | ||
| test_blocked_user_list.py | ||
| test_braintrust.py | ||
| test_budget_manager.py | ||
| test_cache_preset_key.py | ||
| test_caching.py | ||
| test_caching_handler.py | ||
| test_caching_ssl.py | ||
| test_class.py | ||
| test_completion.py | ||
| test_completion_cost.py | ||
| test_completion_with_retries.py | ||
| test_config.py | ||
| test_cost_calc.py | ||
| test_custom_callback_input.py | ||
| test_custom_llm.py | ||
| test_custom_logger.py | ||
| test_disk_cache_unit_tests.py | ||
| test_docker_no_network_on_deploy.py | ||
| test_dual_cache.py | ||
| test_dynamic_rate_limit_handler.py | ||
| test_embedding.py | ||
| test_exceptions.py | ||
| test_fake_openai_endpoint.py | ||
| test_file_types.py | ||
| test_function_call_parsing.py | ||
| test_function_calling.py | ||
| test_function_setup.py | ||
| test_gcs_bucket.py | ||
| test_gcs_cache_unit_tests.py | ||
| test_gemini_reasoning_content.py | ||
| test_get_llm_provider.py | ||
| test_get_model_file.py | ||
| test_get_model_info.py | ||
| test_get_optional_params_embeddings.py | ||
| test_get_optional_params_functions_not_supported.py | ||
| test_google_ai_studio_gemini.py | ||
| test_guardrails_ai.py | ||
| test_helicone_integration.py | ||
| test_http_parsing_utils.py | ||
| test_img_resize.py | ||
| test_langchain_ChatLiteLLM.py | ||
| test_least_busy_routing.py | ||
| test_litellm_max_budget.py | ||
| test_llm_guard.py | ||
| test_load_test_router_s3.py | ||
| test_loadtest_router.py | ||
| test_logging.py | ||
| test_longer_context_fallback.py | ||
| test_lowest_cost_routing.py | ||
| test_lowest_latency_routing.py | ||
| test_lunary.py | ||
| test_max_tpm_rpm_limiter.py | ||
| test_mem_leak.py | ||
| test_mem_usage.py | ||
| test_mock_request.py | ||
| test_model_alias_map.py | ||
| test_multiple_deployments.py | ||
| test_no_top_level_test_invocations.py | ||
| test_ollama.py | ||
| test_ollama_local.py | ||
| test_ollama_local_chat.py | ||
| test_openai_moderations_hook.py | ||
| test_opik.py | ||
| test_pass_through_endpoints.py | ||
| test_profiling_router.py | ||
| test_prometheus_service.py | ||
| test_prompt_caching.py | ||
| test_prompt_injection_detection.py | ||
| test_provider_specific_config.py | ||
| test_pydantic.py | ||
| test_pydantic_namespaces.py | ||
| test_redis_batch_optimizations.py | ||
| test_register_model.py | ||
| test_responses_stream_cache_keys.py | ||
| test_router.py | ||
| test_router_batch_completion.py | ||
| test_router_budget_limiter.py | ||
| test_router_caching.py | ||
| test_router_client_init.py | ||
| test_router_cooldown_handlers.py | ||
| test_router_custom_routing.py | ||
| test_router_debug_logs.py | ||
| test_router_fallback_handlers.py | ||
| test_router_fallbacks.py | ||
| test_router_get_deployments.py | ||
| test_router_max_parallel_requests.py | ||
| test_router_pattern_matching.py | ||
| test_router_retries.py | ||
| test_router_timeout.py | ||
| test_router_utils.py | ||
| test_router_with_fallbacks.py | ||
| test_rules.py | ||
| test_sagemaker.py | ||
| test_sagemaker_nova_integration.py | ||
| test_scheduler.py | ||
| test_secret_detect_hook.py | ||
| test_spend_calculate_endpoint.py | ||
| test_stream_chunk_builder.py | ||
| test_streaming.py | ||
| test_supabase_integration.py | ||
| test_team_config.py | ||
| test_text_completion.py | ||
| test_timeout.py | ||
| test_together_ai.py | ||
| test_tpm_rpm_routing_v2.py | ||
| test_ui_sso_helper_utils.py | ||
| test_unit_test_caching.py | ||
| test_update_spend.py | ||
| test_validate_environment.py | ||
| test_wandb.py | ||
| user_cost.json | ||
| vertex_ai.jsonl | ||
| vertex_batch_completions.jsonl | ||
| vertex_key.json | ||
| whitelisted_bedrock_models.txt | ||