mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-03 07:23:33 +00:00
df50c7572a
Adds the litellm-router-core crate, its step-driven PyO3 bridge (RouterCoreHandle / RouterCall) and RustCore (litellm/router_rust/), served behind LITELLM_RUST through the ROUTER route rule. The host call-out is the only boundary: every attempt goes through Host::invoke, there is no native executor path, and RustCore never builds, wraps or forwards to PythonCore, so router_python_core.py stays deletable. Attempt preparation lives in litellm/router_attempt.py so it outlives router_python_core.py. RustCore matches PythonCore's behavior, including its known bugs (fixed in follow-up PRs). Routing decisions (strategy picks, routing groups, PTU, pre-routing, pre-call checks, fallback walk, health routing, access groups, weights, cooldowns, allowed fails, usage) live in the crate; work that needs live Python objects (request shaping, endpoints, streaming, model info facts, callbacks, alerting, prompt management) lives in the host-side litellm/router_rust/ package (steps/, invoke/, surface/) driven by Rust steps. The crate owns the registry. It runs the deployment build loop (secret references, ids, organization fan-out, install order) and asks the host for what needs Python, one deployment at a time so errors surface where Python raised them: secrets, validate, provider and added steps. It holds the model list, the index maps and the pattern routers with the Python core's history rules (host_index.rs), and answers the registry lookups (lookup.rs). The Python views of that state in litellm/router_rust/surface/ (model_list, index(), names(), patterns()) read the handle, and every write to them becomes a changeset. Model info, deployment model info, configured limits and model group info are merged in the crate (model_info.rs, group_info.rs) from catalog rows and facts the host answers on a model_info step asked at query time, so price reloads and register_model stay visible. The usage cache key format is defined once, in the crate. The crate records each attempt's outcome itself rather than through litellm callbacks. A streamed attempt is deferred: it counts its request when the stream opens, and the host's stream wrappers report the success with the final usage, or a failure after the first chunk, when the stream ends, as the Python core's callbacks do. tests/unit/test_router/core characterizes PythonCore's public behavior and runs every test on both cores; the remaining Rust differences (the router's own outcome callbacks) are listed in rust_parity_pending.txt. |
||
|---|---|---|
| .. | ||
| azure_client_usage_test.py | ||
| ban_constant_numbers.py | ||
| ban_copy_deepcopy_kwargs.py | ||
| bedrock_pricing.py | ||
| callback_manager_test.py | ||
| check_data_replace_usage.py | ||
| check_e2e_no_raw_requests.py | ||
| check_endpoint_coverage.py | ||
| check_fastuuid_usage.py | ||
| check_get_model_cost_key_performance.py | ||
| check_guardrail_apply_decorator.py | ||
| check_licenses.py | ||
| check_migrations_lock_timeout.py | ||
| check_migrations_no_data_rewrites.py | ||
| check_no_none_string_error_fields.py | ||
| check_no_publicly_known_master_key.py | ||
| check_prisma_binary_cache.py | ||
| check_provider_folders_documented.py | ||
| check_py310_typing_imports.py | ||
| check_spanattributes_value_usage.py | ||
| check_telemetry_imports.py | ||
| check_unbounded_in_lists.py | ||
| check_unsafe_enterprise_import.py | ||
| check_workflow_job_name_collisions.py | ||
| check_workflow_startup_safety.py | ||
| code_qa_check_tests.py | ||
| enforce_llms_folder_style.py | ||
| ensure_async_clients_test.py | ||
| info_log_check.py | ||
| liccheck.ini | ||
| license_cache.json | ||
| litellm_logging_code_coverage.py | ||
| log.txt | ||
| memory_test.py | ||
| prevent_key_leaks_in_exceptions.py | ||
| recursive_detector.py | ||
| router_code_coverage.py | ||
| router_enforce_line_length.py | ||
| router_litellm_globals_check.py | ||
| router_surface_allowlist.json | ||
| router_surface_check.py | ||
| test_aio_http_image_conversion.py | ||
| test_ban_set_verbose.py | ||
| test_chat_completion_imports.py | ||
| test_check_no_none_string_error_fields.py | ||
| test_check_no_publicly_known_master_key.py | ||
| test_e2e_idp_stack.py | ||
| test_e2e_junit_report.py | ||
| test_e2e_metadata.py | ||
| test_generate_provider_endpoints_support.py | ||
| test_merge_smoke.py | ||
| test_no_hardcoded_secrets.py | ||
| test_provider_cache.py | ||
| test_provider_replay_harness.py | ||
| test_proxy_types_import.py | ||
| test_router_strategy_async.py | ||
| test_unit_passed_gate.py | ||
| test_workflow_job_name_collisions.py | ||
| unbounded_in_baseline.txt | ||