28 commits

Author	SHA1	Message	Date
yuneng-jiang	b3086ccd74	chore(release): backport 11 staging PRs onto patch-1.92.0rc2 for the 1.92.0 stable cut (#32959) ... * fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) (#32389) * fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) * test(register_model): use a triple provider prefix as the unresolvable-key fixture get_model_info now resolves bedrock/bedrock/... like a routing prefix, so the double-prefix fixture stopped exercising the register_model fallback path. Lock the new double-prefix resolution in as a model-info regression test (cherry picked from commit 734fd29e00) * fix(guardrails): walk Responses-API text taxonomy in shared content helpers (#32542) * fix(guardrails): walk Responses-API text taxonomy in shared content helpers Every guardrail sharing litellm/proxy/guardrails/_content_utils.py silently drops all text on the /v1/responses path. AIM turns it into a loud 422 ( {"error":"No messages in the request"}); every other guardrail (Lakera v2, Cato, Lasso, Repello, IBM, Azure Content Safety, enterprise secret detection) scans an empty payload and lets the request through unscanned. Three defects, all in _content_utils.py: 1. _iter_text_parts_in_content recognised only part.type == "text", but the Responses API uses input_text (request) and output_text (assistant). 2. _coerce_input_to_messages gated on "every item has a role key"; any Responses input list containing a function_call or function_call_output item failed the check and was wrapped as one opaque blob. 3. build_inspection_messages forwarded any role through, including a bare tool role missing tool_call_id, which validators like AIM's /fw/v1/analyze reject with a schema error. Fix walks the actual Responses item taxonomy (message, function_call, function_call_output, bare content parts and strings), recognises {text, input_text, output_text} everywhere, and coerces any role outside {system, user, assistant} to user in the outbound inspection payload. * style: ruff-format changed guardrail files * test(guardrails): cover function_call_output string form; drop em-dash in new docstring * fix(guardrails): map function_call_output straight to user role Avoids ever materialising a schema-invalid bare tool message. The downstream role-safety coercion in build_inspection_messages still guards genuinely caller-supplied non-standard roles (developer, function, custom values); add a regression test covering that path so the coercion has real coverage after this simplification. * test(guardrails): pin chat-completions tool-role coercion in build_inspection_messages * docs(test): soften AIM-specific claims in LIT-4294 test docstrings Ryan's review flagged that several test docstrings assert AIM's /fw/v1/analyze validates + rejects specific schema violations. That behavior is customer-reported in the LIT-4294 writeup, not directly verified by us. Rephrase to attribute the AIM 422 to the customer's writeup and describe the underlying constraint as the OpenAI chat schema; any downstream API that validates against that schema rejects the same shape. * refactor(guardrails): move unsupported-role coercion into AIM only The generic coercion in build_inspection_messages collapsed any role outside {system, user, assistant} to user for every caller of the helper. Combined with the pre-existing apply_redacted_messages_back write-back behavior in Lakera/AIM/Cato, that turned a loud OpenAI 400 on chat-completions tool-message masking into a silent semantic corruption of the outbound request (role tool with tool_call_id got rewritten to bare role user, dropping the assistant + tool_calls sibling). AIM specifically requires the coercion because its /fw/v1/analyze validates the payload against the OpenAI chat schema; other guardrails either do not validate roles or do their own reconstruction. Move the coercion to AimGuardrail._build_aim_inspection_messages so the shared helper keeps caller roles intact and no new cross-guardrail role corruption is introduced. The pre-existing apply_redacted_messages_back structural flatten remains as separate follow-up work. function_call_output items still synthesise role user in the shared helper because they have no natural role field, which is a different concern from coercing a caller-supplied role. * refactor(guardrails): preserve role fidelity in shared _content_utils Shared inspection helpers should extract text and preserve semantic role signals; role coercion for third-party schema safety stays inside the guardrail that needs it (AIM). Three shared-helper changes: - Bare content-part dicts (input_text/output_text) with an explicit role keep it; only role-less parts default to user. - Responses message items already had their role preserved; the behavior is now covered by an explicit test. - function_call_output items default to role tool (semantic equivalent of the chat-completions tool message shape) instead of role user, so Responses and chat completions produce symmetric inspection payloads. A caller-supplied role on the item is still preserved. AIM's schema-safe coercion in _build_aim_inspection_messages already handles the resulting role tool: it collapses to user before the POST to /fw/v1/analyze so AIM's OpenAI-schema validator does not reject the bare tool message (no tool_call_id can survive the flatten). Added a regression test in test_aim.py covering that path. (cherry picked from commit e84a19acd5) * feat: add Meta Model API provider and muse-spark-1.1 (day-0) (#32701) (cherry picked from commit d82645d163) * fix(bedrock): keep mid-conversation system messages in place for Claude Invoke (#32578) Hoisting every role system entry into the top-level system field mutates the cache prefix whenever a client such as Claude Code appends a new mid-conversation system message, invalidating the prompt cache for the entire message history on Bedrock Invoke. Bedrock only rejects a system entry at messages.0, so hoist just the leading run and forward the rest in place (cherry picked from commit cc36d5469c) * feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls (#32655) * feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls The GenAI semantic conventions record failures of a GenAI client operation as a log-based event named gen_ai.client.operation.exception, carrying the exception.type / exception.message / exception.stacktrace trio at severity WARN and correlated to the failed span. OTel v2 never emitted it: a failed LLM call produced only the deprecated error.* span attributes, a generic exception span event without a stacktrace, and the stacktrace under the vendor key litellm.provider.error.stack_trace. Build the logs pipeline (LoggerProvider + console/OTLP log exporters mirroring the metrics plumbing) and record the event behind the enable_events flag, which until now was defined but consumed nowhere. An operator-configured LoggerProvider global is reused so the events ride their existing logs pipeline; an explicit NoOpLoggerProvider global is honored as an opt-out and builds no recorder at all. The existing span-side error surface (error.type, error.message, the exception span event, and the litellm.provider.error.* detail keys) is untouched for backwards compatibility. * fix(otel): always ride the semconv-required exception pair on the GenAI event Filtering the event attributes on truthiness conflated "absent" with "empty", so an empty exception.type or exception.message would have been dropped, leaving an event with neither semconv-required field. Build the attributes so the pair is unconditional and only the recommended stacktrace is omitted when the payload carries none. * docs(otel): document the events plumbing module in the package README * test(otel): cover the log exporter selection and logs endpoint normalization The new logs plumbing had no coverage for exporter-kind selection, the console fallback for an unrecognized kind, the /v1/logs signal-path rewriting that lets one OTEL_ENDPOINT serve every signal, or the simple-vs-batch processor split. (cherry picked from commit 99b4c5ed3e) * fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831) * fix(bedrock): gate in-place system role messages on model support for Claude Invoke * feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule (cherry picked from commit 5e23a5ab05) * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support (#32867) * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface (thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5) get the effort translated to a legacy thinking budget_tokens. Models with no reasoning support have thinking/effort dropped under drop_params. And because adaptive thinking carries no budget while the legacy form must satisfy Anthropic's max_tokens > budget_tokens rule, the translated budget is capped below max_tokens, dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass through untouched. This matters because clients like Claude Code speak native Anthropic /v1/messages and send the adaptive interface unconditionally, regardless of the routed model. The native passthrough previously only capability-gated the OpenAI-style reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so a pre-4.6 model rejected it with "This model does not support the effort parameter" and the request failed. Claude Code already gets drop_params auto-set, so its requests now succeed. * test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests Addresses review feedback on the max_tokens-too-small branch. Previously a thinking-capable model whose max_tokens could not fit the minimum thinking budget had thinking silently dropped regardless of drop_params, while a residual output_config field in the same call still raised when drop_params was off. Gate both consistently on drop_params: raise a clear error (naming max_tokens for the undersized case) when drop_params is off, drop otherwise. Claude Code gets drop_params auto-set, so it still succeeds. Adds tests for the undersized-max_tokens raise, the residual output_config raise, and the no-adaptive-interface passthrough on a non-adaptive model. * fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts The previous raise-when-not-drop_params behavior broke existing bedrock and vertex messages tests: those providers already silently strip unsupported output_config for pre-4.6 models (issue #22797) with no drop_params required, and the shared parent transform raising pre-empted that. It also conflicted with the goal of keeping requests working rather than failing them. Make the reshape silent: translate effort to legacy thinking for thinking-capable models, drop thinking for non-reasoning models, and remove only the consumed effort key from output_config, leaving any residual (e.g. format) for provider subclasses (bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the review note about inconsistent drop_params handling by making every path uniform. Updates the tests to assert the silent behavior and residual output_config preservation. * fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5) Greptile caught a real bug: the early-return guard treated supports_output_config as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises supports_output_config (it accepts output_config.effort) but is not adaptive, so it rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking block raw, reproducing the exact failure the fix is meant to prevent. thinking:{type:adaptive} and output_config.effort are independent capabilities. Only early-return for adaptive-thinking models. For a model that supports output_config.effort but is not adaptive, keep the native effort and drop only the unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code payload now returns 200 instead of 400. Adds regression tests for Opus 4.5 with and without adaptive thinking. * fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models Claude Opus 4.5 advertises supports_output_config but not adaptive thinking, so the early-return guard forwarded thinking.type=adaptive raw and Anthropic rejected it. The guard now only skips true adaptive models; effort-only requests on effort-capable models still pass through untouched. The _map_reasoning_effort call is wrapped to surface unrecognized effort values as a clean 400, matching _translate_reasoning_effort_to_anthropic * fix(anthropic): fall back to legacy thinking when effort level unsupported Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code defaults to xhigh on newer models, so preserving that level raw gets rejected by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model and fall through to the budget translation for unsupported levels * fix(anthropic): keep effort-only requests untouched for provider normalization The xhigh fall-through consumed effort-only requests on effort-capable models, breaking bedrock invoke's own normalization which clamps xhigh to the model's ceiling after the base transform runs (test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict the fall-through to requests that carry adaptive thinking; effort-only requests pass through so provider subclasses keep owning level clamping --------- Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> (cherry picked from commit 3a62e5428f) * fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system (#32882) Exact cost-map hits resolve before fallback-generalization rules, so the mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the bedrock-anthropic-claude-mid-conversation-system rule and hoisted mid-conversation system messages, invalidating the prompt cache. (cherry picked from commit c15891fc98) * Merge pull request #32873 from BerriAI/litellm_fallback_rules_routing_split refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds (cherry picked from commit 45d3644408) * Merge pull request #32874 from BerriAI/litellm_thread_provider_capability_probes fix(anthropic): thread real provider through capability probes instead of pinning anthropic (cherry picked from commit ead7ad3804) * test: add /v1/messages to supported_endpoints schema enum (#32739) (cherry picked from commit bf02a4a47f) --------- Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> Co-authored-by: tin-berri <tin@berri.ai>	2026-07-11 16:29:55 -07:00
Mateo Wang	b76a858826	feat: declarative fallback generalizations for unknown models (#29718) ... * feat: declarative fallback generalizations for unknown models Unknown or newly-released models previously degraded (missed cost lookups, wrong supports_* flags, broken provider routing) and were patched with one-off hardcoded regexes scattered across Python. This adds a single data-driven source of truth: a fallback_generalizations block in model_prices_and_context_window.json holding ordered, case-insensitive regex rules that map a model name to the metadata to apply when it has no exact entry. A new fallback_generalizations module owns the rules and a compiled-regex cache that is built once and invalidated on reload, so the O(n) scan runs only on a cache miss. get_llm_provider now routes an otherwise-unknown model via the first matching rule's litellm_provider, replacing the hardcoded _CLAUDE_PATTERN and _matches_claude_model_pattern. _get_model_info_helper falls back to a matching rule's model_info after the exact lookups miss, so get_model_info and the supports_* helpers resolve unknown models from the same rule. get_model_cost_map extracts the block out of the returned map, and the integrity check now counts real model entries (excluding reserved meta keys) so the new key cannot mask a genuinely shrunk upstream file. The top level of the file stays a flat map of models so existing litellm releases that fetch the live file keep working and keep receiving updates; the block ships in both the root file and the bundled backup. An anthropic-claude rule reproduces the old future-claude routing and additionally supplies capability flags and a context window https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * refactor(anthropic): derive adaptive-thinking from a version threshold; harden generalizations Replace the per-minor-version _is_claude_4_6_model / _is_claude_4_7_model substring matchers with a single _claude_version_at_least predicate that parses the Claude family version from the model name and compares against 4.6. This covers 4.8/4.9/5.x without a code change (the old matchers missed 4.8 entirely) while keeping an explicit supports_adaptive_thinking flag authoritative when present, so there is one source of truth. The two direct call sites in the chat transformation now route through _is_adaptive_thinking_model instead of the deleted matchers. Also address review feedback on the generalizations module: return a copy of the matched model_info so a future caller cannot mutate the compiled-rule cache, document that patterns are matched with re.search and must anchor with ^ and $, and reindent the fallback_generalizations block to the file's 2-space style in both JSON files. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): surface adaptive-thinking from the cost map; fix date misparse supports_adaptive_thinking shipped in the model cost map but was never declared on ModelInfo nor copied during construction, so get_model_info (and the supports_* factory) silently dropped it for every provider-prefixed or generalized name; only a bare base entry resolved. Wire it through ModelInfo like the other capability flags and backfill the flag onto the genuine Claude 4.6/4.7/4.8 entries across providers so the data, not code, declares the capability. The anthropic-claude fallback rule also carries the flag (and now accepts a dotted minor, e.g. 4.6) so an unmapped future Claude degrades to adaptive thinking without a code change. Tighten the Claude version parser so an eight-digit date suffix (claude-opus-4-20250514, the non-adaptive Opus 4.0) is no longer read as minor 4.20250514. The cost map stays authoritative; the version check is only a fallback for provider-prefixed names (bedrock/invoke routes, -v1-less ids) that resolve to no mapped entry and so cannot be reached by an exact lookup or the bare-name rule. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): date-safe adaptive-thinking version fallback, conservative fallback pricing, ruff strict gate Reconcile adaptive-thinking detection after merging litellm_internal_staging. Keep the cost-map resolver (_supports_model_capability) as the source of truth and add a date-safe opus/sonnet/haiku >= 4.6 name version as a fallback for provider-prefixed ids the cost map cannot resolve (e.g. bedrock/invoke/us.anthropic.claude-opus-4-6). A two-digit cap on the minor keeps an eight-digit date suffix from being misread as a minor version, so the dated Claude 4.0 release stays non-adaptive Price the shipped anthropic-claude fallback rule at the Opus tier so an unknown or newly released Claude is over-costed rather than billed as free Drop the module-level global state in fallback_generalizations (PLW0603) in favor of a small registry object, and switch its annotations plus the new utils helper to builtin generics (UP006), bringing the ruff strict-rule totals back under ceiling * refactor(anthropic): drive adaptive-thinking version gate from a declarative rule Replace the bespoke _claude_version_at_least heuristic with a version-gated fallback_generalizations rule. Unmapped Claude ids now resolve adaptive thinking purely from the cost map: an explicit entry, or the new self-contained anthropic-claude-adaptive-thinking rule that matches opus/sonnet/haiku >= 4.6 (covering 5.x, 6.x and beyond with no code change). New families ship via Price Data Reload instead of a code edit The rule carries the same Opus-tier pricing as the broad anthropic-claude rule plus supports_adaptive_thinking, and is matched first; the broad rule stays version-neutral, so an unmapped >= 4.6 Claude resolves to full pricing and the adaptive flag from one rule, while a sub-4.6 alias such as claude-opus-4-0 is still priced yet stays non-adaptive. The regex caps the minor at two digits so a dated 4.0 id (...-4-20250514) is never read as a >= 4.6 minor * refactor(anthropic): dedupe adaptive-thinking rule via declarative extends The version-gated anthropic-claude-adaptive-thinking rule duplicated the broad anthropic-claude rule's entire Opus-tier price block because rules do not merge: first match wins and returns one rule's whole model_info, so the adaptive rule had to be self-contained. Add a declarative extends field to fallback_generalizations: a rule names a parent and inherits its model_info, with its own keys overriding. Inheritance is resolved once at install time against each rule's raw model_info, so the adaptive rule now carries only its delta (supports_adaptive_thinking) and inherits pricing from the broad rule. Runtime matching, provider routing and gating are unchanged; the broad rule stays anchored and first-match-wins still holds. * docs(anthropic): add ignored description key documenting each generalization regex * fix(anthropic): drop fabricated pricing from the anthropic-claude fallback rule Per review feedback, the base rule no longer carries input/output/cache costs, and the adaptive-thinking rule that extends it inherits that no-pricing model_info. Pricing an unmapped model at a guessed tier reports a confidently-wrong cost without the caller knowing; dropping it keeps the standard unpriced behavior (zero, not a fabricated number) so a missing price stays visible. The rules still supply provider routing, context window, and capability flags, so a brand-new Claude can still be called and its capabilities (including adaptive thinking for >= 4.6) resolved. Description and tests updated to match	2026-06-27 21:01:19 -07:00
Sameer Kankute	cb041966bf	Litellm oss staging 040626 (#29671) ... * fix(azure): apply api_version fallback chain to image edit URL `AzureImageEditConfig.get_complete_url` only read `api_version` from `litellm_params`. When callers configured it via `litellm.api_version` or `AZURE_API_VERSION`, the constructed URL had no `?api-version=` and Azure responded `404 Resource not found`. Apply the same fallback chain the Azure chat path already uses in `common_utils.py`: litellm_params > litellm.api_version > AZURE_API_VERSION env > litellm.AZURE_DEFAULT_API_VERSION Adds 5 unit tests pinning each layer of the chain plus a regression guard for `api_base` that already carries `?api-version=`. * feat(mcp): core sampling and elicitation flow with security hardening - Add sampling_handler.py: full MCP sampling/createMessage flow with model selection (hint-based + priority-based), auth enforcement, budget checks, route restriction gates, and tag policy pre-auth - Add elicitation_handler.py: MCP elicitation/create relay with downstream client capability detection - Wire sampling/elicitation callbacks in mcp_server_manager.py gated behind allow_sampling/allow_elicitation config flags - Add allow_sampling/allow_elicitation fields to MCPServer type - Fix session lock deadlock: skip lock for JSON-RPC response POSTs (elicitation/sampling replies) with truncated-body heuristic - Extend client.py with sampling_callback and elicitation_callback - Security: RouteChecks gate, tag-budget bypass fix, x-forwarded-for spoofing fix, Latin-1 header encoding guard - Add 4 new test modules (model access, priority selection, request builder, tool conversion) + update existing MCP tests * fix(security): run pre-call guardrails before MCP sampling acompletion Without this, an upstream MCP server with allow_sampling enabled could send prompts that bypass every guardrail (content filtering, PII redaction, prompt-injection detection) configured on /chat/completions. - Call proxy_logging_obj.pre_call_hook(call_type='acompletion') before llm_router.acompletion so guardrails fire for sampling sub-calls - Add HTTPException to the re-raise list so guardrail rejections propagate correctly instead of being swallowed as generic errors * feat(bedrock_mantle): add Responses API support (/openai/v1/responses) (#29490) * feat(bedrock_mantle): add Responses API transformation config * test(bedrock_mantle): cover trailing-slash api_base normalization * feat(bedrock_mantle): export BedrockMantleResponsesAPIConfig * feat(bedrock_mantle): register gpt-5.x Responses config (gpt-oss unchanged) * feat(bedrock_mantle): add gpt-5.5/gpt-5.4 Responses price-map entries * refactor(bedrock_mantle): exclude gpt-oss instead of allow-listing gpt-5 for Responses routing Frontier OpenAI models on Bedrock Mantle are Responses-only on /openai/v1/responses; gpt-oss is the legacy family that also speaks chat-completions. Gate by excluding gpt-oss (which keeps its chat-completions emulation) and defaulting everything else to the native Responses config, so future frontier models (gpt-6, etc.) route correctly without a code change. Verified against the live us-east-2 Mantle endpoint: gpt-oss 400s on /openai/v1/responses while gpt-5.5 400s on both standard paths. * test(bedrock_mantle): cover supports_native_websocket opt-out Closes the one uncovered line flagged by codecov on the Responses config. The assertion documents that Mantle Responses has no realtime/websocket transport, so realtime routing must not attempt a socket it cannot serve. * fix(bedrock_mantle): route file_search through emulation instead of forwarding to Mantle BedrockMantleResponsesAPIConfig inherited supports_native_file_search() -> True from OpenAIResponsesAPIConfig but never overrode it. Mantle has no OpenAI vector stores, so a forwarded file_search tool is rejected with a 400 (verified upstream: Tool type 'file_search' is not supported). Opting out, like the existing supports_native_websocket override, routes the tool through LiteLLM's file_search emulation instead. * fix(bedrock_mantle): only route openai.gpt frontier models to Responses The previous gate excluded gpt-oss and routed every other model to the native Responses config. But on Mantle only the OpenAI gpt frontier models (gpt-5.x) are served on /openai/v1/responses; gpt-oss and the non-OpenAI families (nvidia, mistral, google, zai, ...) are chat-completions only and 400 on that path. Allow-list the openai.gpt- family (excluding gpt-oss) instead, so chat-only models fall through to the chat-completions emulation. Verified against the live us-east-2 endpoint: nvidia.nemotron-nano-9b-v2 returns 400 on /openai/v1/responses and 200 on /v1/chat/completions. * feat(custom_llm): allow streaming/astreaming to yield ModelResponseStream (#27580) * fix(custom_llm): allow streaming/astreaming to yield ModelResponseStream directly * fix(streaming): enhance ModelResponseStream handling for custom LLM providers * fix(streaming): strip finish_reason from content chunks and ensure tool_calls are preserved * fix(streaming): add type ignore for finish_reason assignment in CustomStreamWrapper * fix(proxy): strip stack trace from HTTP 503 responses (CWE-209) (#28330) * fix(proxy/cwe-209): strip Python traceback from HTTP 503 error responses The /cache/ping endpoint included a full Python traceback in its 503 error response body (inside the ProxyException message), leaking internal file paths, line numbers, and call stacks to any caller. Two MCP route handlers in proxy_server.py similarly interpolated str(e) into "Internal server error" detail strings. Fix: log the traceback server-side via verbose_proxy_logger.exception() and omit it from the ProxyException payload / HTTPException detail returned to clients. Tests updated to assert no "traceback" keyword or frame paths appear in the 503 body, with a new dedicated regression test. CWE-209: Generation of Error Message Containing Sensitive Information. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(proxy/cwe-209): apply Greptile P2 fixes and add MCP exception-path tests Greptile 4/5 review identified two remaining gaps and Codecov reported 0% coverage on the two MCP handler exception branches: 1. caching_routes.py — str(e) in "Service Unhealthy ({str(e)})" could still leak Redis hostnames/IPs; replaced with static "Service Unhealthy". HTTPException is now re-raised before the generic handler so the "cache not initialized" 503 still reaches callers with its detail. Removed the redundant str(e) arg from verbose_proxy_logger.exception() (exception() already appends the traceback automatically). 2. tests — two new unit tests cover the exception paths in dynamic_mcp_route and toolset_mcp_route that were previously at 0%: - test_dynamic_mcp_route_unexpected_exception_returns_500_without_traceback - test_toolset_mcp_route_unexpected_exception_returns_500_without_traceback All 25 tests pass (9 caching + 16 MCP). CWE-209: Generation of Error Message Containing Sensitive Information. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(caching_routes): restore precise assertion in test_cache_ping_no_cache_initialized The assertion was weakened to `"Cache not initialized" in str(data)`, which matches the raw string of the entire response dict and would pass even if the error moved to an unexpected field or changed structure. Restore a targeted check on the parsed response: assert the exact string in the correct field `data["detail"]`, matching FastAPI's HTTPException serialisation format {"detail": "<message>"}. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(caching_routes): restore precise assertion and add CWE-209 no-cache path test The assertion in test_cache_ping_no_cache_initialized was weakened to `"Cache not initialized" in str(data)`, which matched against the raw string representation of the entire response dict. This would pass silently even if the error message moved to an unexpected field or the structure changed. Restore a targeted assertion on the parsed field: assert data["detail"] == "Cache not initialized. litellm.cache is None" matching FastAPI's HTTPException serialisation format exactly. Add test_cache_ping_no_cache_does_not_expose_internals to show the code path is still working correctly after the CWE-209 fix: verifies that the HTTPException is re-raised as-is (no traceback, no source paths), and asserts the complete response structure is exactly {"detail": "Cache not initialized. litellm.cache is None"}. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(caching_routes): restore ProxyException envelope for null-cache 503 The except HTTPException: raise guard (added in the CWE-209 fix) caused the null-cache HTTPException to escape as FastAPI's {"detail": "..."} shape instead of the {"error": {...}} ProxyException envelope that callers expect. Move the null-cache guard before the try block and raise ProxyException directly so the response structure is consistent with all other /cache/ping 503s, and the except HTTPException: raise guard is only reachable by unexpected downstream HTTPExceptions. Update the two no-cache tests to assert the correct ProxyException envelope. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Update utils.py (#26609) * feat(pricing): add Snowflake Cortex REST API model pricing (#26612) * feat(pricing): add Snowflake Cortex REST API model pricing ## Summary Adds pricing and context window information for 20+ Snowflake Cortex REST API models to `model_prices_and_context_window.json`. ## What's included - **7 Claude models** (sonnet-4-5, sonnet-4-6, 4-sonnet, 4-opus, haiku-4-5, 3-7-sonnet, 3-5-sonnet) — with prompt caching rates - **4 OpenAI models** (gpt-4.1, gpt-5, gpt-5-mini, gpt-5-nano) — with prompt caching rates - **5 Llama models** (3.1-8b, 3.1-70b, 3.1-405b, 3.3-70b, 4-maverick) - **1 DeepSeek model** (deepseek-r1) - **1 Mistral model** (mistral-large2) - **1 Snowflake model** (snowflake-llama-3.3-70b) - **2 Embedding models** (arctic-embed-l-v2.0, arctic-embed-m-v2.0) Each entry includes `input_cost_per_token`, `output_cost_per_token`, `cache_read_input_token_cost` (where applicable), `max_input_tokens`, `max_output_tokens`, and capability flags (`supports_function_calling`, `supports_vision`, `supports_prompt_caching`, `supports_reasoning`). ## Pricing source All prices are in USD per token, sourced from the official [Snowflake Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf) — Tables 6(b) (REST API with Prompt Caching) and 6(c) (REST API). ## Context The existing `snowflake/` provider has zero model entries in the pricing JSON, which means LiteLLM cannot track costs for Snowflake Cortex calls. This PR fills that gap. ## Related - Existing provider: `litellm/llms/snowflake/` - Cortex REST API docs: https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-rest-api * Update model_prices_and_context_window.json Fix the JSON parsing error * Update model_prices_and_context_window.json Removed the duplicate entry * fix(utils): copy extra_body before adding unknown params to prevent model config mutation (#29620) Fixes #29615. In add_provider_specific_params_to_optional_params, the line: extra_body = passed_params.pop("extra_body", None) or {} returns the original dict reference when extra_body is non-empty (truthy). Subsequent writes like extra_body[k] = passed_params[k] then mutate the shared model config object held by the router, poisoning /model/info and all subsequent requests for that deployment. The or {} short-circuit creates a new dict only when extra_body is falsy (None or {}), which is why the bug does not reproduce with extra_body: {}. Fix: wrap in dict() so we always work on a fresh shallow copy. * fix(vertex_ai): Bake tool_choice into Gemini CachedContent body to prevent silent drop (#29097) * fix(vertex_ai): bake tool_choice into Gemini CachedContent body to prevent silent drop * address greptile feedback on tool_choice cache test * adds test that uses ToolConfig(functionCallingConfig=FunctionCallingConfig(mode=ANY)) instead of a dict literal, mirroring what map_tool_choice_values actually produce * fix(gemini/veo): move image from parameters into instances[0] (#29501) * fix(gemini/veo): move image from parameters into instances[0] Veo's predictLongRunning schema puts image (and prompt) on the instances element; parameters is for aspectRatio/durationSeconds/etc. The Gemini path was leaving image in params_copy, so it ended up nested under parameters and the API silently ignored it. The Vertex path already builds the instance dict explicitly, so this just aligns the Gemini path with it. Fixes #29498 * address greptile: unconditional pop + BytesIO test - Pop `image` from params_copy unconditionally so it never reaches GeminiVideoGenerationParameters even when None, removing implicit reliance on Pydantic's extra-field-ignore. - Add test_transform_video_create_request_image_filelike_goes_to_instance covering the BytesIO path (_convert_image_to_gemini_format) — round-trips the base64 to confirm encoding. - Add test_transform_video_create_request_image_none_is_dropped covering the new None branch. * fix(huggingface): handle special token text in embedding usage (#29660) * fix(guardrails): recompile ToolPermissionGuardrail rules on update_in_memory_litellm_params (#29655) * fix(guardrails): recompile ToolPermissionGuardrail rules on update_in_memory_litellm_params ToolPermissionGuardrail builds self.rules and the compiled target/pattern maps only in __init__. The base update_in_memory_litellm_params re-sets raw attributes via setattr but never rebuilds those maps, so a guardrail updated in place (PUT /guardrails, or the immediate in-memory sync) keeps enforcing the construction-time rules until it is reinitialized (PATCH path, periodic DB poll, or restart). Extract the compile step into _load_rules and override update_in_memory_litellm_params to rebuild from it (dict- and model-safe), re-normalizing default_action / on_disallowed_action. Mirrors the existing PresidioGuardrail override of the same method. Adds regression tests. Fixes #29592. * fix(guardrails): handle dict params in ToolPermissionGuardrail in-memory update Delegate to super() only for LitellmParams input (the base setattr loop is model-only); apply the raw-dict case inline. Fixes the mypy arg-type error and makes the recompile work when the proxy passes the raw DB dict. * fix(guardrails): preserve tool-permission rules on a partial in-memory update A partial update (e.g. a LitellmParams whose rules field is None) ran through the generic setattr, which set self.rules to None, and the recompile was skipped, leaving the guardrail with no rules. Snapshot the previous rules and restore them when the update carries no rules; an explicit empty list still clears them. Adds a regression test for the rules-absent case. Addresses the Greptile review note on #29655. * fix(bedrock): stop base_model label from stripping tools/tool_choice (#29621) * fix(bedrock): stop base_model label from stripping tools/tool_choice A Router/proxy Bedrock deployment whose model_info.base_model is a friendly label (e.g. claude-haiku-4-5) silently lost tools/tool_choice: the outgoing Converse request was built without toolConfig, so the model behaved as if no tools were provided. Worked in v1.84.0, regressed in v1.85.0, and with drop_params=true it failed silently. Two changes compound into the bug. completion() passed model_info.base_model as the model argument to get_optional_params, so the real Bedrock model id never reached supported-param resolution; and get_supported_openai_params resolved the provider config's params from base_model or model, letting the label fully replace the real model. For Bedrock the label resolves to no tool support, so tools/tool_choice were dropped before transformation. completion() now keeps model as the real deployment model and threads the resolved base_model (kwarg or model_info) through separately, and get_supported_openai_params treats base_model as additive: it returns the union of the params supported by model and by base_model. A hint can only add capabilities, never strip ones the real model already exposes, which also preserves the original base_model behavior from #27717 and Azure's base_model driven model-type detection. Fixes #29618 * test(main): make base_model param test robust to new parametrize cases Restore an explicit per-case expected_model_param literal instead of hardcoding the gemini id, so a future case with a different model can't produce a misleading assertion failure. * fix(fireworks_ai): pass response_format json_schema through unchanged (#29606) FireworksAIConfig.map_openai_params was rewriting the OpenAI strict `{type: json_schema, json_schema: {name, strict, schema}}` shape into `{type: json_object, schema: ...}` before sending to Fireworks, dropping `strict` and `name` and changing the `type`. Per Fireworks' docs json_object means "force any valid JSON output (no specific schema)", so the schema constraint was effectively dropped and grammar-guided decoding never ran; model output silently violated the schema. The rewrite landed in #7085 (Dec 2024) when Fireworks did not yet accept native json_schema. Fireworks accepts the OpenAI strict shape natively now, so the rewrite has become a regression. Removes the rewrite. Passes response_format through unchanged. Updates the existing test_map_response_format to assert pass-through. Adds focused regression tests in tests/test_litellm/ covering preservation of type, strict, name, and schema body, plus that json_object alone still works. * fix(types): import Required from typing_extensions in gemini types * style: reformat sampling_handler.py for py312 black compat * refactor(mcp-sampling): extract helpers to fix PLR0915 too-many-statements in handle_sampling_create_message * fix(proxy-server): add explicit ProxyLogging type annotation to proxy_logging_obj to fix mypy inference * fix(mcp-sampling): suppress mypy assignment error on ImportError fallback for proxy_logging_obj * fix(test): use .value when comparing LlmProviders enum against string in test_default_api_base * fix(test): iterate LlmProviders enum in test_default_api_base to avoid str pollution from custom provider registration litellm.provider_list is a mutable global initialized to list(LlmProviders) but custom_llm_setup() appends plain provider strings to it. When a test_custom_llm.py test runs first in the same xdist worker, provider_list contains a str and calling .value on it raises AttributeError. Iterate the immutable LlmProviders enum instead, which is deterministic and what the check intends. * fix(mcp): depth-aware JSON-RPC response detection and neutral speed-priority fallback Replace the flat substring check in the truncated-body routing path with a top-level-key scan so a JSON-RPC response whose result payload nests a "method" field is still detected as a response and skips the session lock, removing a deadlock against the in-flight tool call awaiting it. Drop the inverse max_output_tokens speed proxy when no model exposes output_tokens_per_second; context-window size does not track latency, so a neutral score avoids biasing speedPriority toward the smallest-context model. * fix(guardrails): make ToolPermission rule reload atomic on invalid regex _load_rules appended each rule to self.rules before compiling its regex, so an invalid pattern raised mid-loop after the bad rule was already live but without a _compiled_rule_targets entry. _matches_regex reads a missing compiled target as a None pattern and returns True, turning the bad rule into a match-all that silently applies its decision to every tool. Via update_in_memory_litellm_params (PUT /guardrails) this corrupted the live guardrail. Build the parsed rules and compiled maps into locals and swap them in only after every regex compiles, and restore the previous ruleset if a live update is rejected, so an invalid regex now fails the update without leaving the guardrail enforcing a broken policy. * test(mcp): cover sampling conversion, model resolution, and elicitation relay paths The MCP sampling and elicitation handlers shipped with partial test coverage, leaving the response-to-MCP conversion, the model resolution fallback chain, completion-kwargs assembly, guardrail routing, and the entire elicitation relay untested. That pulled the PR's diff (patch) coverage below the codecov threshold even though overall project coverage rose. Add focused unit tests for _convert_openai_response_to_mcp_result, _convert_mcp_tools_to_openai, _convert_mcp_tool_choice_to_openai, image and audio content conversion, the hint-matching and fallback branches of _resolve_model_from_preferences, _build_completion_kwargs, the router and guardrail-rejection paths of _run_guardrails_and_call_llm, the handle_sampling_create_message success and error-propagation flows, the marker-hoisting fallback for tool content on unexpected roles, and the elicitation form/url/generic relay together with its decline paths --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: lengkejun <lengkejun@xd.com> Co-authored-by: Yug <yugborana000@gmail.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: tanmay958 <53569547+tanmay958@users.noreply.github.com> Co-authored-by: DrishnaTrivedi <142084770+DrishnaTrivedi@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Navnit Shukla <Navnit.shukla25@gmail.com> Co-authored-by: PRABHU KIRAN VANDRANKI <72809214+VANDRANKI@users.noreply.github.com> Co-authored-by: Adrian Lopez <109683617+adriangomez24@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: JooHo Lee <96564470+BWAAEEEK@users.noreply.github.com> Co-authored-by: Dinesh Girbide <85330597+Dinesh-Girbide@users.noreply.github.com> Co-authored-by: cloudwiz <22098246+andrey-dubnik@users.noreply.github.com> Co-authored-by: Ahmad Khan <ahmadkhan2508@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-04 11:07:20 -07:00
Mateo Wang	f81d8ae077	[internal copy of #29232] feat: route future Claude models to Anthropic provider via pattern matching (#29239) ... * feat: route future Claude models to Anthropic provider via pattern matching Add pattern-based matching for Claude model names so that future models (e.g., claude-opus-4-9, claude-sonnet-5-0) are automatically routed to the Anthropic provider without requiring model_prices_and_context_window.json updates. The pattern matches: claude-{opus|sonnet|haiku}-{major}-{minor}[-YYYYMMDD] https://claude.ai/code/session_017asCVDN5jBFMBcZRjiQR6C * fix: don't hard-code the tier names Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * style: move import re to module level (PEP 8) Move `import re` from inside the module body to the top-level imports section, following PEP 8 style guidelines that all imports should appear at the top of the file. https://claude.ai/code/session_01Dt8fzn81eYMfxu1MoBa5hN * test: fix claude-mini-4-5 assertion to match generic-tier pattern The pattern intentionally accepts any [a-z]+ tier (see f835f84) rather than a hard-coded opus|sonnet|haiku list, so claude-mini-4-5 routes to anthropic and the old 'is False' assertion was wrong. Replace it with a positive test that locks in the generic-tier behavior and guards against a regression to hard-coded tier names. * style: collapse _CLAUDE_PATTERN to one line (black) Black 26.3.1 (CI) collapses the re.compile call onto a single line since it fits within the line limit. Fixes the failing lint check. --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>	2026-06-02 15:16:01 -07:00
Chesars	c75f0c0566	test: drop duplicate openrouter prefix-strip test ... The multi-segment case (openrouter/<provider>/<model> → <provider>/<model>) is already covered by tests/test_litellm/llms/openrouter/test_openrouter_provider_routing.py in internal_staging, with broader coverage including double-prefix native models, wildcard deployments, and the bridge double-call scenario. Keeping a duplicate in tests/local_testing/ adds maintenance load with no extra coverage.	2026-04-25 18:06:25 -03:00
Chesars	232464e151	test: drop incorrect openrouter native-prefix test ... The removed test asserted that get_llm_provider(model="openrouter/auto") should return model="openrouter/auto" with the prefix preserved. That contract is wrong: the LiteLLM convention strips one "openrouter/" prefix, so a native OpenRouter model is reached via "openrouter/openrouter/auto" (double-prefix) which the wire send as "openrouter/auto" — the ID the OpenRouter API expects for natives. E2E checks against api.openrouter.ai confirm this is the correct routing convention for all native models (auto, bodybuilder, free, pareto-code).	2026-04-25 17:25:49 -03:00
Chesars	91e78eca3d	Merge remote-tracking branch 'upstream/litellm_internal_staging' into upstream-litellm_staging_03_21_2026 ... # Conflicts: # .circleci/config.yml # .circleci/requirements.txt # .github/workflows/_test-unit-base.yml # .github/workflows/_test-unit-services-base.yml # .github/workflows/auto_update_price_and_context_window.yml # .github/workflows/create-release.yml # .github/workflows/llm-translation-testing.yml # .github/workflows/publish_to_pypi.yml # .github/workflows/scan_duplicate_issues.yml # .github/workflows/test-linting.yml # .github/workflows/test-litellm-matrix.yml # .github/workflows/test-litellm.yml # .github/workflows/test-mcp.yml # .github/workflows/test-model-map.yaml # .github/workflows/test-proxy-e2e-azure-batches.yml # .github/workflows/test-unit-core-utils.yml # .github/workflows/test-unit-documentation.yml # .github/workflows/test-unit-enterprise-routing.yml # .github/workflows/test-unit-integrations.yml # .github/workflows/test-unit-llm-providers.yml # .github/workflows/test-unit-misc.yml # .github/workflows/test-unit-proxy-auth.yml # .github/workflows/test-unit-proxy-db.yml # .github/workflows/test-unit-proxy-endpoints.yml # .github/workflows/test-unit-proxy-infra.yml # .github/workflows/test-unit-proxy-legacy.yml # .github/workflows/test-unit-responses-caching-types.yml # .github/workflows/test-unit-security.yml # .github/workflows/test_server_root_path.yml # docs/my-website/docs/embedding/supported_embedding.md # litellm/litellm_core_utils/get_llm_provider_logic.py # litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_transformation.py # litellm/proxy/_experimental/out/404/index.html # litellm/proxy/_experimental/out/__next.__PAGE__.txt # litellm/proxy/_experimental/out/__next._full.txt # litellm/proxy/_experimental/out/__next._head.txt # litellm/proxy/_experimental/out/__next._index.txt # litellm/proxy/_experimental/out/__next._tree.txt # litellm/proxy/_experimental/out/_next/static/3qyC5Vtvhd5fSC6sPp1iW/_buildManifest.js # litellm/proxy/_experimental/out/_next/static/3qyC5Vtvhd5fSC6sPp1iW/_clientMiddlewareManifest.json # litellm/proxy/_experimental/out/_next/static/3qyC5Vtvhd5fSC6sPp1iW/_ssgManifest.js # litellm/proxy/_experimental/out/_next/static/aKKihXXKRJWLQThZgi8Rq/_buildManifest.js # litellm/proxy/_experimental/out/_next/static/aKKihXXKRJWLQThZgi8Rq/_clientMiddlewareManifest.json # litellm/proxy/_experimental/out/_next/static/aKKihXXKRJWLQThZgi8Rq/_ssgManifest.js # litellm/proxy/_experimental/out/_next/static/bmMTxs1O5fQKYcsMNTRMT/_buildManifest.js # litellm/proxy/_experimental/out/_next/static/bmMTxs1O5fQKYcsMNTRMT/_clientMiddlewareManifest.json # litellm/proxy/_experimental/out/_next/static/bmMTxs1O5fQKYcsMNTRMT/_ssgManifest.js # litellm/proxy/_experimental/out/_next/static/chunks/11362340846735c3.js # litellm/proxy/_experimental/out/_next/static/chunks/1a04d31843c96649.js # litellm/proxy/_experimental/out/_next/static/chunks/342c7d7210247a5e.js # litellm/proxy/_experimental/out/_next/static/chunks/39768ec0eebd2554.js # litellm/proxy/_experimental/out/_next/static/chunks/3b3c0b070b14da06.js # litellm/proxy/_experimental/out/_next/static/chunks/3bddc72a3ecc2253.js # litellm/proxy/_experimental/out/_next/static/chunks/4472ece1be7379b3.js # litellm/proxy/_experimental/out/_next/static/chunks/54e29148cb2f2582.js # litellm/proxy/_experimental/out/_next/static/chunks/67ddb5107368a659.js # litellm/proxy/_experimental/out/_next/static/chunks/6a167cef4b09b496.js # litellm/proxy/_experimental/out/_next/static/chunks/7174130ddef406dd.js # litellm/proxy/_experimental/out/_next/static/chunks/7c36bfe1ba5e3ba8.js # litellm/proxy/_experimental/out/_next/static/chunks/7e5fe5584502da06.js # litellm/proxy/_experimental/out/_next/static/chunks/8dda507c226082ca.js # litellm/proxy/_experimental/out/_next/static/chunks/8dfde809dc4ad794.js # litellm/proxy/_experimental/out/_next/static/chunks/99109c78121231a0.js # litellm/proxy/_experimental/out/_next/static/chunks/9dd55e1f36a7225c.js # litellm/proxy/_experimental/out/_next/static/chunks/a230559fcabaea23.js # litellm/proxy/_experimental/out/_next/static/chunks/a6c7f80b3968f639.js # litellm/proxy/_experimental/out/_next/static/chunks/ac9e96d21c200b48.js # litellm/proxy/_experimental/out/_next/static/chunks/ae9cf43b8c0c76aa.js # litellm/proxy/_experimental/out/_next/static/chunks/cf06797ce4e438f9.js # litellm/proxy/_experimental/out/_next/static/chunks/d069df5baead6d90.js # litellm/proxy/_experimental/out/_next/static/chunks/d2e3b7dd6499c245.js # litellm/proxy/_experimental/out/_next/static/chunks/d44e73d8ebac5747.js # litellm/proxy/_experimental/out/_next/static/chunks/dc8a270fee94ced6.js # litellm/proxy/_experimental/out/_next/static/chunks/df6546cd8a44d3b3.js # litellm/proxy/_experimental/out/_next/static/chunks/ea0f22bd4b3393bd.js # litellm/proxy/_experimental/out/_next/static/chunks/eaa9f9b9bb3e054b.js # litellm/proxy/_experimental/out/_next/static/chunks/turbopack-901b35f89c1f6751.js # litellm/proxy/_experimental/out/_next/static/chunks/turbopack-d1b22f5e0bd58c57.js # litellm/proxy/_experimental/out/_next/static/chunks/turbopack-ddedb29a5eb0118f.js # litellm/proxy/_experimental/out/_not-found.txt # litellm/proxy/_experimental/out/_not-found/__next._full.txt # litellm/proxy/_experimental/out/_not-found/__next._head.txt # litellm/proxy/_experimental/out/_not-found/__next._index.txt # litellm/proxy/_experimental/out/_not-found/__next._not-found.__PAGE__.txt # litellm/proxy/_experimental/out/_not-found/__next._not-found.txt # litellm/proxy/_experimental/out/_not-found/__next._tree.txt # litellm/proxy/_experimental/out/_not-found/index.html # litellm/proxy/_experimental/out/api-reference.html # litellm/proxy/_experimental/out/api-reference.txt # litellm/proxy/_experimental/out/api-reference/__next.!KGRhc2hib2FyZCk.api-reference.__PAGE__.txt # litellm/proxy/_experimental/out/api-reference/__next.!KGRhc2hib2FyZCk.api-reference.txt # litellm/proxy/_experimental/out/api-reference/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/api-reference/__next._full.txt # litellm/proxy/_experimental/out/api-reference/__next._head.txt # litellm/proxy/_experimental/out/api-reference/__next._index.txt # litellm/proxy/_experimental/out/api-reference/__next._tree.txt # litellm/proxy/_experimental/out/chat.html # litellm/proxy/_experimental/out/chat.txt # litellm/proxy/_experimental/out/chat/__next._full.txt # litellm/proxy/_experimental/out/chat/__next._head.txt # litellm/proxy/_experimental/out/chat/__next._index.txt # litellm/proxy/_experimental/out/chat/__next._tree.txt # litellm/proxy/_experimental/out/chat/__next.chat.__PAGE__.txt # litellm/proxy/_experimental/out/chat/__next.chat.txt # litellm/proxy/_experimental/out/experimental/api-playground.html # litellm/proxy/_experimental/out/experimental/api-playground.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.experimental.api-playground.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.experimental.api-playground.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._full.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._head.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._index.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._tree.txt # litellm/proxy/_experimental/out/experimental/budgets.html # litellm/proxy/_experimental/out/experimental/budgets.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.experimental.budgets.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.experimental.budgets.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._full.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._head.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._index.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._tree.txt # litellm/proxy/_experimental/out/experimental/caching.html # litellm/proxy/_experimental/out/experimental/caching.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.experimental.caching.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.experimental.caching.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/caching/__next._full.txt # litellm/proxy/_experimental/out/experimental/caching/__next._head.txt # litellm/proxy/_experimental/out/experimental/caching/__next._index.txt # litellm/proxy/_experimental/out/experimental/caching/__next._tree.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins.html # litellm/proxy/_experimental/out/experimental/claude-code-plugins.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.experimental.claude-code-plugins.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.experimental.claude-code-plugins.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._full.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._head.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._index.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._tree.txt # litellm/proxy/_experimental/out/experimental/old-usage.html # litellm/proxy/_experimental/out/experimental/old-usage.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.experimental.old-usage.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.experimental.old-usage.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._full.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._head.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._index.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._tree.txt # litellm/proxy/_experimental/out/experimental/prompts.html # litellm/proxy/_experimental/out/experimental/prompts.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.experimental.prompts.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.experimental.prompts.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._full.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._head.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._index.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._tree.txt # litellm/proxy/_experimental/out/experimental/tag-management.html # litellm/proxy/_experimental/out/experimental/tag-management.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.experimental.tag-management.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.experimental.tag-management.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._full.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._head.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._index.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._tree.txt # litellm/proxy/_experimental/out/guardrails.html # litellm/proxy/_experimental/out/guardrails.txt # litellm/proxy/_experimental/out/guardrails/__next.!KGRhc2hib2FyZCk.guardrails.__PAGE__.txt # litellm/proxy/_experimental/out/guardrails/__next.!KGRhc2hib2FyZCk.guardrails.txt # litellm/proxy/_experimental/out/guardrails/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/guardrails/__next._full.txt # litellm/proxy/_experimental/out/guardrails/__next._head.txt # litellm/proxy/_experimental/out/guardrails/__next._index.txt # litellm/proxy/_experimental/out/guardrails/__next._tree.txt # litellm/proxy/_experimental/out/index.html # litellm/proxy/_experimental/out/index.txt # litellm/proxy/_experimental/out/login.html # litellm/proxy/_experimental/out/login.txt # litellm/proxy/_experimental/out/login/__next._full.txt # litellm/proxy/_experimental/out/login/__next._head.txt # litellm/proxy/_experimental/out/login/__next._index.txt # litellm/proxy/_experimental/out/login/__next._tree.txt # litellm/proxy/_experimental/out/login/__next.login.__PAGE__.txt # litellm/proxy/_experimental/out/login/__next.login.txt # litellm/proxy/_experimental/out/logs.html # litellm/proxy/_experimental/out/logs.txt # litellm/proxy/_experimental/out/logs/__next.!KGRhc2hib2FyZCk.logs.__PAGE__.txt # litellm/proxy/_experimental/out/logs/__next.!KGRhc2hib2FyZCk.logs.txt # litellm/proxy/_experimental/out/logs/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/logs/__next._full.txt # litellm/proxy/_experimental/out/logs/__next._head.txt # litellm/proxy/_experimental/out/logs/__next._index.txt # litellm/proxy/_experimental/out/logs/__next._tree.txt # litellm/proxy/_experimental/out/mcp/oauth/callback.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._full.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._head.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._index.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._tree.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.oauth.callback.__PAGE__.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.oauth.callback.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.oauth.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/index.html # litellm/proxy/_experimental/out/model-hub.html # litellm/proxy/_experimental/out/model-hub.txt # litellm/proxy/_experimental/out/model-hub/__next.!KGRhc2hib2FyZCk.model-hub.__PAGE__.txt # litellm/proxy/_experimental/out/model-hub/__next.!KGRhc2hib2FyZCk.model-hub.txt # litellm/proxy/_experimental/out/model-hub/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/model-hub/__next._full.txt # litellm/proxy/_experimental/out/model-hub/__next._head.txt # litellm/proxy/_experimental/out/model-hub/__next._index.txt # litellm/proxy/_experimental/out/model-hub/__next._tree.txt # litellm/proxy/_experimental/out/model_hub.html # litellm/proxy/_experimental/out/model_hub.txt # litellm/proxy/_experimental/out/model_hub/__next._full.txt # litellm/proxy/_experimental/out/model_hub/__next._head.txt # litellm/proxy/_experimental/out/model_hub/__next._index.txt # litellm/proxy/_experimental/out/model_hub/__next._tree.txt # litellm/proxy/_experimental/out/model_hub/__next.model_hub.__PAGE__.txt # litellm/proxy/_experimental/out/model_hub/__next.model_hub.txt # litellm/proxy/_experimental/out/model_hub_table.html # litellm/proxy/_experimental/out/model_hub_table.txt # litellm/proxy/_experimental/out/model_hub_table/__next._full.txt # litellm/proxy/_experimental/out/model_hub_table/__next._head.txt # litellm/proxy/_experimental/out/model_hub_table/__next._index.txt # litellm/proxy/_experimental/out/model_hub_table/__next._tree.txt # litellm/proxy/_experimental/out/model_hub_table/__next.model_hub_table.__PAGE__.txt # litellm/proxy/_experimental/out/model_hub_table/__next.model_hub_table.txt # litellm/proxy/_experimental/out/models-and-endpoints.html # litellm/proxy/_experimental/out/models-and-endpoints.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next.!KGRhc2hib2FyZCk.models-and-endpoints.__PAGE__.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next.!KGRhc2hib2FyZCk.models-and-endpoints.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._full.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._head.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._index.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._tree.txt # litellm/proxy/_experimental/out/onboarding.html # litellm/proxy/_experimental/out/onboarding.txt # litellm/proxy/_experimental/out/onboarding/__next._full.txt # litellm/proxy/_experimental/out/onboarding/__next._head.txt # litellm/proxy/_experimental/out/onboarding/__next._index.txt # litellm/proxy/_experimental/out/onboarding/__next._tree.txt # litellm/proxy/_experimental/out/onboarding/__next.onboarding.__PAGE__.txt # litellm/proxy/_experimental/out/onboarding/__next.onboarding.txt # litellm/proxy/_experimental/out/organizations.html # litellm/proxy/_experimental/out/organizations.txt # litellm/proxy/_experimental/out/organizations/__next.!KGRhc2hib2FyZCk.organizations.__PAGE__.txt # litellm/proxy/_experimental/out/organizations/__next.!KGRhc2hib2FyZCk.organizations.txt # litellm/proxy/_experimental/out/organizations/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/organizations/__next._full.txt # litellm/proxy/_experimental/out/organizations/__next._head.txt # litellm/proxy/_experimental/out/organizations/__next._index.txt # litellm/proxy/_experimental/out/organizations/__next._tree.txt # litellm/proxy/_experimental/out/playground.html # litellm/proxy/_experimental/out/playground.txt # litellm/proxy/_experimental/out/playground/__next.!KGRhc2hib2FyZCk.playground.__PAGE__.txt # litellm/proxy/_experimental/out/playground/__next.!KGRhc2hib2FyZCk.playground.txt # litellm/proxy/_experimental/out/playground/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/playground/__next._full.txt # litellm/proxy/_experimental/out/playground/__next._head.txt # litellm/proxy/_experimental/out/playground/__next._index.txt # litellm/proxy/_experimental/out/playground/__next._tree.txt # litellm/proxy/_experimental/out/policies.html # litellm/proxy/_experimental/out/policies.txt # litellm/proxy/_experimental/out/policies/__next.!KGRhc2hib2FyZCk.policies.__PAGE__.txt # litellm/proxy/_experimental/out/policies/__next.!KGRhc2hib2FyZCk.policies.txt # litellm/proxy/_experimental/out/policies/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/policies/__next._full.txt # litellm/proxy/_experimental/out/policies/__next._head.txt # litellm/proxy/_experimental/out/policies/__next._index.txt # litellm/proxy/_experimental/out/policies/__next._tree.txt # litellm/proxy/_experimental/out/settings/admin-settings.html # litellm/proxy/_experimental/out/settings/admin-settings.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.settings.admin-settings.__PAGE__.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.settings.admin-settings.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._full.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._head.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._index.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._tree.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts.html # litellm/proxy/_experimental/out/settings/logging-and-alerts.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.settings.logging-and-alerts.__PAGE__.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.settings.logging-and-alerts.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._full.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._head.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._index.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._tree.txt # litellm/proxy/_experimental/out/settings/router-settings.html # litellm/proxy/_experimental/out/settings/router-settings.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.settings.router-settings.__PAGE__.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.settings.router-settings.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._full.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._head.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._index.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._tree.txt # litellm/proxy/_experimental/out/settings/ui-theme.html # litellm/proxy/_experimental/out/settings/ui-theme.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.settings.ui-theme.__PAGE__.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.settings.ui-theme.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._full.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._head.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._index.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._tree.txt # litellm/proxy/_experimental/out/teams.html # litellm/proxy/_experimental/out/teams.txt # litellm/proxy/_experimental/out/teams/__next.!KGRhc2hib2FyZCk.teams.__PAGE__.txt # litellm/proxy/_experimental/out/teams/__next.!KGRhc2hib2FyZCk.teams.txt # litellm/proxy/_experimental/out/teams/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/teams/__next._full.txt # litellm/proxy/_experimental/out/teams/__next._head.txt # litellm/proxy/_experimental/out/teams/__next._index.txt # litellm/proxy/_experimental/out/teams/__next._tree.txt # litellm/proxy/_experimental/out/test-key.html # litellm/proxy/_experimental/out/test-key.txt # litellm/proxy/_experimental/out/test-key/__next.!KGRhc2hib2FyZCk.test-key.__PAGE__.txt # litellm/proxy/_experimental/out/test-key/__next.!KGRhc2hib2FyZCk.test-key.txt # litellm/proxy/_experimental/out/test-key/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/test-key/__next._full.txt # litellm/proxy/_experimental/out/test-key/__next._head.txt # litellm/proxy/_experimental/out/test-key/__next._index.txt # litellm/proxy/_experimental/out/test-key/__next._tree.txt # litellm/proxy/_experimental/out/tools/mcp-servers.html # litellm/proxy/_experimental/out/tools/mcp-servers.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.tools.mcp-servers.__PAGE__.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.tools.mcp-servers.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.tools.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._full.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._head.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._index.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._tree.txt # litellm/proxy/_experimental/out/tools/vector-stores.html # litellm/proxy/_experimental/out/tools/vector-stores.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.tools.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.tools.vector-stores.__PAGE__.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.tools.vector-stores.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._full.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._head.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._index.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._tree.txt # litellm/proxy/_experimental/out/usage.html # litellm/proxy/_experimental/out/usage.txt # litellm/proxy/_experimental/out/usage/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/usage/__next.!KGRhc2hib2FyZCk.usage.__PAGE__.txt # litellm/proxy/_experimental/out/usage/__next.!KGRhc2hib2FyZCk.usage.txt # litellm/proxy/_experimental/out/usage/__next._full.txt # litellm/proxy/_experimental/out/usage/__next._head.txt # litellm/proxy/_experimental/out/usage/__next._index.txt # litellm/proxy/_experimental/out/usage/__next._tree.txt # litellm/proxy/_experimental/out/users.html # litellm/proxy/_experimental/out/users.txt # litellm/proxy/_experimental/out/users/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/users/__next.!KGRhc2hib2FyZCk.users.__PAGE__.txt # litellm/proxy/_experimental/out/users/__next.!KGRhc2hib2FyZCk.users.txt # litellm/proxy/_experimental/out/users/__next._full.txt # litellm/proxy/_experimental/out/users/__next._head.txt # litellm/proxy/_experimental/out/users/__next._index.txt # litellm/proxy/_experimental/out/users/__next._tree.txt # litellm/proxy/_experimental/out/virtual-keys.html # litellm/proxy/_experimental/out/virtual-keys.txt # litellm/proxy/_experimental/out/virtual-keys/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/virtual-keys/__next.!KGRhc2hib2FyZCk.virtual-keys.__PAGE__.txt # litellm/proxy/_experimental/out/virtual-keys/__next.!KGRhc2hib2FyZCk.virtual-keys.txt # litellm/proxy/_experimental/out/virtual-keys/__next._full.txt # litellm/proxy/_experimental/out/virtual-keys/__next._head.txt # litellm/proxy/_experimental/out/virtual-keys/__next._index.txt # litellm/proxy/_experimental/out/virtual-keys/__next._tree.txt # scripts/install.sh # tests/local_testing/test_get_llm_provider.py	2026-04-25 17:15:24 -03:00
Ishaan Jaffer	e8461b5b97	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
David Chen	d1df4e838b	Litellm fix update bedrock models (#24947) ... * update bedrock models in tests * updated more tests and model_prices_and_context_window * fix model id and pricing * replace more sonnet models * update tests * git push * update pricing * flaky total cost * monkey patch * relax the cost change * fix and revert some changes * revert the pricing * chore: move cost/pricing changes to bedrock-cost-fixes branch * chore: split Bedrock file-api beta stripping to separate branch Removes strip_unsupported_file_api_betas_for_bedrock_invoke from this branch; see litellm_bedrock_invoke_strip_file_api_betas for that fix. Made-with: Cursor	2026-04-01 19:22:54 -07:00
Imgyu Kim	ad07d7faad	fix: strip 'openrouter/' prefix from model names (#24234) ... Remove early return in get_llm_provider_logic.py that prevented the 'openrouter/' prefix from being stripped. The early return was intended for 'native OpenRouter models' like 'openrouter/free', but no such models exist in the model registry — all OpenRouter models are multi-segment (e.g. 'openrouter/anthropic/claude-3.5-sonnet') and need the prefix stripped before being sent to the OpenRouter API. This regression was introduced in v1.82.3 and caused 400 Bad Request errors for all OpenRouter models.	2026-03-21 18:06:44 +09:00
Sameer Kankute	a5ea08a0bf	Fix test_default_api_base failing because of chatgpt as provider	2026-01-21 09:32:38 +05:30
Sameer Kankute	6751badf3a	fix: test_default_api_base for ragfow	2025-12-04 21:49:33 +05:30
Ishaan Jaff	19e26a5c60	test_default_api_base	2025-07-04 18:26:54 -07:00
Ishaan Jaff	643d2a8ccb	[Feat] Option to force/always use the litellm proxy (#10559) (#10633) (#10773) ... * [Feat] Option to force/always use the litellm proxy (#10559) (#10633) * fix: add use_litellm_proxy * fix: update LiteLLMProxyChatConfig * fix get llm provider logic * tests get llm provider logic * add dynamic use_litellm_proxy * docs forcsing litellm proxy usage * fix: _should_use_litellm_proxy_by_default * fixes: get_custom_llm_provider --------- Co-authored-by: Antoine Legrand <2t.antoine@gmail.com>	2025-05-12 20:22:54 -07:00
Ishaan Jaff	de7870cb54	Add llamafile as a provider (#10203) (#10482) ... * Update docs for OpenAI compatible providers, add Llamafile docs, include Llamafile in the sidebar * Add Llamafile as an LlmProviders enum * Add llamafile as a OpenAI compatible provider (in the list of compatible providers) * Add Llamafile chat config and tests * Wire up Llamafile Co-authored-by: Peter Wilson <peter@mozilla.ai>	2025-05-01 18:36:55 -07:00
Krrish Dholakia	267084a1af	test(test_get_llm_provider.py): cover scenario where xai not in model name	2025-03-18 11:04:59 -07:00
Krrish Dholakia	aeec703c4e	test(test_get_llm_provider.py): Minimal repro for https://github.com/BerriAI/litellm/issues/9291	2025-03-18 10:35:50 -07:00
Ishaan Jaff	b242c66a3b	(Feat) - Add /bedrock/invoke support for all Anthropic models (#8383) ... * use anthropic transformation for bedrock/invoke * use anthropic transforms for bedrock invoke claude * TestBedrockInvokeClaudeJson * add AmazonAnthropicClaudeStreamDecoder * pass bedrock_invoke_provider to make_call * fix _get_base_bedrock_model * fix get_bedrock_route * fix bedrock routing * fixes for bedrock invoke * test_all_model_configs * fix AWSEventStreamDecoder linting * fix code qa * test_bedrock_get_base_model * test_get_model_info_bedrock_models * test_bedrock_base_model_helper * test_bedrock_route_detection	2025-02-07 22:41:11 -08:00
Krish Dholakia	becd4bc748	Litellm dev 01 11 2025 p3 (#7702) ... * fix(__init__.py): fix init to exclude pricing-only model cost values from real model names prevents bad health checks on wildcard routes * fix(get_llm_provider.py): fix to handle calling bedrock_converse models	2025-01-11 20:06:54 -08:00
Krish Dholakia	0120176541	Litellm dev 12 30 2024 p2 (#7495) ... * test(azure_openai_o1.py): initial commit with testing for azure openai o1 preview model * fix(base_llm_unit_tests.py): handle azure o1 preview response format tests skip as o1 on azure doesn't support tool calling yet * fix: initial commit of azure o1 handler using openai caller simplifies calling + allows fake streaming logic alr. implemented for openai to just work * feat(azure/o1_handler.py): fake o1 streaming for azure o1 models azure does not currently support streaming for o1 * feat(o1_transformation.py): support overriding 'should_fake_stream' on azure/o1 via 'supports_native_streaming' param on model info enables user to toggle on when azure allows o1 streaming without needing to bump versions * style(router.py): remove 'give feedback/get help' messaging when router is used Prevents noisy messaging Closes https://github.com/BerriAI/litellm/issues/5942 * fix(types/utils.py): handle none logprobs Fixes https://github.com/BerriAI/litellm/issues/328 * fix(exception_mapping_utils.py): fix error str unbound error * refactor(azure_ai/): move to openai_like chat completion handler allows for easy swapping of api base url's (e.g. ai.services.com) Fixes https://github.com/BerriAI/litellm/issues/7275 * refactor(azure_ai/): move to base llm http handler * fix(azure_ai/): handle differing api endpoints * fix(azure_ai/): make sure all unit tests are passing * fix: fix linting errors * fix: fix linting errors * fix: fix linting error * fix: fix linting errors * fix(azure_ai/transformation.py): handle extra body param * fix(azure_ai/transformation.py): fix max retries param handling * fix: fix test * test(test_azure_o1.py): fix test * fix(llm_http_handler.py): support handling azure ai unprocessable entity error * fix(llm_http_handler.py): handle sync invalid param error for azure ai * fix(azure_ai/): streaming support with base_llm_http_handler * fix(llm_http_handler.py): working sync stream calls with unprocessable entity handling for azure ai * fix: fix linting errors * fix(llm_http_handler.py): fix linting error * fix(azure_ai/): handle cohere tool call invalid index param error	2025-01-01 18:57:29 -08:00
Krish Dholakia	0924df4971	Litellm dev 12 27 2024 p2 1 (#7449) ... * fix(azure_ai/transformation.py): route ai.services.azure calls to the azure provider route requires token to be passed in as 'api-key' Closes https://github.com/BerriAI/litellm/issues/7275 * fix(key_management_endpoints.py): enforce user is member of team, if team_id set and team_id exists in team table * fix(key_management_endpoints.py): handle assigned_user_id = none * feat(create_key_button.tsx): allow assigning keys to other users allows proxy admin to easily assign other people keys * build(create_key_button.tsx): fix error message display don't swallow the error message for key creation failure * build(create_key_button.tsx): allow proxy admin to edit team id * build(create_key_button.tsx): allow proxy admin to assign keys to other users * build(edit_user.tsx): clarify how 'user budgets' are applied * test: remove dup test * fix(key_management_endpoints.py): don't raise error if team not in db ' * test: fix test	2024-12-27 20:02:32 -08:00
Krish Dholakia	6a45ee1ef7	fix(hosted_vllm/transformation.py): return fake api key, if none give… (#7301) ... * fix(hosted_vllm/transformation.py): return fake api key, if none give. Prevents httpx error Fixes https://github.com/BerriAI/litellm/issues/7291 * test: fix test * fix(main.py): add hosted_vllm/ support for embeddings endpoint Closes https://github.com/BerriAI/litellm/issues/7290 * docs(vllm.md): add docs on vllm embeddings usage * fix(__init__.py): fix sambanova model test * fix(base_llm_unit_tests.py): skip pydantic obj test if model takes >5s to respond	2024-12-18 18:41:53 -08:00
Krish Dholakia	e9aa492af3	LiteLLM Minor Fixes & Improvement (11/14/2024) (#6730) ... * fix(ollama.py): fix get model info request Fixes https://github.com/BerriAI/litellm/issues/6703 * feat(anthropic/chat/transformation.py): support passing user id to anthropic via openai 'user' param * docs(anthropic.md): document all supported openai params for anthropic * test: fix tests * fix: fix tests * feat(jina_ai/): add rerank support Closes https://github.com/BerriAI/litellm/issues/6691 * test: handle service unavailable error * fix(handler.py): refactor together ai rerank call * test: update test to handle overloaded error * test: fix test * Litellm router trace (#6742) * feat(router.py): add trace_id to parent functions - allows tracking retry/fallbacks * feat(router.py): log trace id across retry/fallback logic allows grouping llm logs for the same request * test: fix tests * fix: fix test * fix(transformation.py): only set non-none stop_sequences * Litellm router disable fallbacks (#6743) * bump: version 1.52.6 → 1.52.7 * feat(router.py): enable dynamically disabling fallbacks Allows for enabling/disabling fallbacks per key * feat(litellm_pre_call_utils.py): support setting 'disable_fallbacks' on litellm key * test: fix test * fix(exception_mapping_utils.py): map 'model is overloaded' to internal server error * test: handle gemini error * test: fix test * fix: new run	2024-11-15 01:02:54 +05:30
Krish Dholakia	f59cb46e71	Litellm dev 11 11 2024 (#6693) ... * fix(__init__.py): add 'watsonx_text' as mapped llm api route Fixes https://github.com/BerriAI/litellm/issues/6663 * fix(opentelemetry.py): fix passing parallel tool calls to otel Fixes https://github.com/BerriAI/litellm/issues/6677 * refactor(test_opentelemetry_unit_tests.py): create a base set of unit tests for all logging integrations - test for parallel tool call handling reduces bugs in repo * fix(__init__.py): update provider-model mapping to include all known provider-model mappings Fixes https://github.com/BerriAI/litellm/issues/6669 * feat(anthropic): support passing document in llm api call * docs(anthropic.md): add pdf anthropic call to docs + expose new 'supports_pdf_input' function * fix(factory.py): fix linting error	2024-11-12 00:16:35 +05:30
Krish Dholakia	c03e5da41f	LiteLLM Minor Fixes & Improvements (10/24/2024) (#6421) ... * fix(utils.py): support passing dynamic api base to validate_environment Returns True if just api base is required and api base is passed * fix(litellm_pre_call_utils.py): feature flag sending client headers to llm api Fixes https://github.com/BerriAI/litellm/issues/6410 * fix(anthropic/chat/transformation.py): return correct error message * fix(http_handler.py): add error response text in places where we expect it * fix(factory.py): handle base case of no non-system messages to bedrock Fixes https://github.com/BerriAI/litellm/issues/6411 * feat(cohere/embed): Support cohere image embeddings Closes https://github.com/BerriAI/litellm/issues/6413 * fix(__init__.py): fix linting error * docs(supported_embedding.md): add image embedding example to docs * feat(cohere/embed): use cohere embedding returned usage for cost calc * build(model_prices_and_context_window.json): add embed-english-v3.0 details (image cost + 'supports_image_input' flag) * fix(cohere_transformation.py): fix linting error * test(test_proxy_server.py): cleanup test * test: cleanup test * fix: fix linting errors	2024-10-25 15:55:56 -07:00
Krish Dholakia	2b9db05e08	feat(proxy_cli.py): add new 'log_config' cli param (#6352) ... * feat(proxy_cli.py): add new 'log_config' cli param Allows passing logging.conf to uvicorn on startup * docs(cli.md): add logging conf to uvicorn cli docs * fix(get_llm_provider_logic.py): fix default api base for litellm_proxy Fixes https://github.com/BerriAI/litellm/issues/6332 * feat(openai_like/embedding): Add support for jina ai embeddings Closes https://github.com/BerriAI/litellm/issues/6337 * docs(deploy.md): update entrypoint.sh filepath post-refactor Fixes outdated docs * feat(prometheus.py): emit time_to_first_token metric on prometheus Closes https://github.com/BerriAI/litellm/issues/6334 * fix(prometheus.py): only emit time to first token metric if stream is True enables more accurate ttft usage * test: handle vertex api instability * fix(get_llm_provider_logic.py): fix import * fix(openai.py): fix deepinfra default api base * fix(anthropic/transformation.py): remove anthropic beta header (#6361)	2024-10-21 21:25:58 -07:00
Ishaan Jaff	ab0b536143	(feat) add azure openai cost tracking for prompt caching (#6077) ... * add azure o1 models to model cost map * add azure o1 cost tracking * fix azure cost calc * add get llm provider test	2024-10-05 15:04:18 +05:30
Krrish Dholakia	3560f0ef2c	refactor: move all testing to top-level of repo ... Closes https://github.com/BerriAI/litellm/issues/486	2024-09-28 21:08:14 -07:00

Renamed from litellm/tests/test_get_llm_provider.py (Browse further)