mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-09 22:31:41 +00:00
31 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
6a9dcb5ce6 | test: allow dashscope domain in qwen alias default api_base check | ||
|
yuneng-jiang
|
6a0d03914c
|
test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
|
||
|
ryan-crabbe-berri
|
680bcfd8aa
|
test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731)
* test(lint): ban blind pytest.raises(Exception) with ruff B017 A bare pytest.raises(Exception) accepts whatever the body throws. The TypeError a refactor introduces satisfies it exactly as well as the rejection the test was written for, so the crash reads as a pass and the test never goes red. All 111 existing sites are narrowed here. A runtime probe recorded the concrete exception each one actually catches, and each site now names that type. Where the code under test genuinely raises a bare Exception, the site pins a stable slice of the message with match= instead. Two sites tell on themselves. The shared responses-API cancel test raises "custom_llm_provider is required but passed as None" rather than talking to a provider at all, because cancel_responses takes a provider, not a model. And test_bedrock_guardrails_with_streaming was the only test in its file still passing without AWS credentials, because the NoCredentialsError boto3 raised long before the guardrail ran satisfied the blind raises. * fix(test): widen the openai batch-dispatch assertion to OpenAIError The narrowed NotFoundError only holds where OPENAI_API_KEY is set. Without one the SDK raises OpenAIError while building the client, long before any 404, so CI went red. OpenAIError covers both and still rejects a TypeError from a refactor. |
||
|
mateo-berri
|
77885779ca | refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds | ||
|
Mateo Wang
|
b76a858826
|
feat: declarative fallback generalizations for unknown models (#29718)
* feat: declarative fallback generalizations for unknown models Unknown or newly-released models previously degraded (missed cost lookups, wrong supports_* flags, broken provider routing) and were patched with one-off hardcoded regexes scattered across Python. This adds a single data-driven source of truth: a fallback_generalizations block in model_prices_and_context_window.json holding ordered, case-insensitive regex rules that map a model name to the metadata to apply when it has no exact entry. A new fallback_generalizations module owns the rules and a compiled-regex cache that is built once and invalidated on reload, so the O(n) scan runs only on a cache miss. get_llm_provider now routes an otherwise-unknown model via the first matching rule's litellm_provider, replacing the hardcoded _CLAUDE_PATTERN and _matches_claude_model_pattern. _get_model_info_helper falls back to a matching rule's model_info after the exact lookups miss, so get_model_info and the supports_* helpers resolve unknown models from the same rule. get_model_cost_map extracts the block out of the returned map, and the integrity check now counts real model entries (excluding reserved meta keys) so the new key cannot mask a genuinely shrunk upstream file. The top level of the file stays a flat map of models so existing litellm releases that fetch the live file keep working and keep receiving updates; the block ships in both the root file and the bundled backup. An anthropic-claude rule reproduces the old future-claude routing and additionally supplies capability flags and a context window https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * refactor(anthropic): derive adaptive-thinking from a version threshold; harden generalizations Replace the per-minor-version _is_claude_4_6_model / _is_claude_4_7_model substring matchers with a single _claude_version_at_least predicate that parses the Claude family version from the model name and compares against 4.6. This covers 4.8/4.9/5.x without a code change (the old matchers missed 4.8 entirely) while keeping an explicit supports_adaptive_thinking flag authoritative when present, so there is one source of truth. The two direct call sites in the chat transformation now route through _is_adaptive_thinking_model instead of the deleted matchers. Also address review feedback on the generalizations module: return a copy of the matched model_info so a future caller cannot mutate the compiled-rule cache, document that patterns are matched with re.search and must anchor with ^ and $, and reindent the fallback_generalizations block to the file's 2-space style in both JSON files. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): surface adaptive-thinking from the cost map; fix date misparse supports_adaptive_thinking shipped in the model cost map but was never declared on ModelInfo nor copied during construction, so get_model_info (and the supports_* factory) silently dropped it for every provider-prefixed or generalized name; only a bare base entry resolved. Wire it through ModelInfo like the other capability flags and backfill the flag onto the genuine Claude 4.6/4.7/4.8 entries across providers so the data, not code, declares the capability. The anthropic-claude fallback rule also carries the flag (and now accepts a dotted minor, e.g. 4.6) so an unmapped future Claude degrades to adaptive thinking without a code change. Tighten the Claude version parser so an eight-digit date suffix (claude-opus-4-20250514, the non-adaptive Opus 4.0) is no longer read as minor 4.20250514. The cost map stays authoritative; the version check is only a fallback for provider-prefixed names (bedrock/invoke routes, -v1-less ids) that resolve to no mapped entry and so cannot be reached by an exact lookup or the bare-name rule. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): date-safe adaptive-thinking version fallback, conservative fallback pricing, ruff strict gate Reconcile adaptive-thinking detection after merging litellm_internal_staging. Keep the cost-map resolver (_supports_model_capability) as the source of truth and add a date-safe opus/sonnet/haiku >= 4.6 name version as a fallback for provider-prefixed ids the cost map cannot resolve (e.g. bedrock/invoke/us.anthropic.claude-opus-4-6). A two-digit cap on the minor keeps an eight-digit date suffix from being misread as a minor version, so the dated Claude 4.0 release stays non-adaptive Price the shipped anthropic-claude fallback rule at the Opus tier so an unknown or newly released Claude is over-costed rather than billed as free Drop the module-level global state in fallback_generalizations (PLW0603) in favor of a small registry object, and switch its annotations plus the new utils helper to builtin generics (UP006), bringing the ruff strict-rule totals back under ceiling * refactor(anthropic): drive adaptive-thinking version gate from a declarative rule Replace the bespoke _claude_version_at_least heuristic with a version-gated fallback_generalizations rule. Unmapped Claude ids now resolve adaptive thinking purely from the cost map: an explicit entry, or the new self-contained anthropic-claude-adaptive-thinking rule that matches opus/sonnet/haiku >= 4.6 (covering 5.x, 6.x and beyond with no code change). New families ship via Price Data Reload instead of a code edit The rule carries the same Opus-tier pricing as the broad anthropic-claude rule plus supports_adaptive_thinking, and is matched first; the broad rule stays version-neutral, so an unmapped >= 4.6 Claude resolves to full pricing and the adaptive flag from one rule, while a sub-4.6 alias such as claude-opus-4-0 is still priced yet stays non-adaptive. The regex caps the minor at two digits so a dated 4.0 id (...-4-20250514) is never read as a >= 4.6 minor * refactor(anthropic): dedupe adaptive-thinking rule via declarative extends The version-gated anthropic-claude-adaptive-thinking rule duplicated the broad anthropic-claude rule's entire Opus-tier price block because rules do not merge: first match wins and returns one rule's whole model_info, so the adaptive rule had to be self-contained. Add a declarative extends field to fallback_generalizations: a rule names a parent and inherits its model_info, with its own keys overriding. Inheritance is resolved once at install time against each rule's raw model_info, so the adaptive rule now carries only its delta (supports_adaptive_thinking) and inherits pricing from the broad rule. Runtime matching, provider routing and gating are unchanged; the broad rule stays anchored and first-match-wins still holds. * docs(anthropic): add ignored description key documenting each generalization regex * fix(anthropic): drop fabricated pricing from the anthropic-claude fallback rule Per review feedback, the base rule no longer carries input/output/cache costs, and the adaptive-thinking rule that extends it inherits that no-pricing model_info. Pricing an unmapped model at a guessed tier reports a confidently-wrong cost without the caller knowing; dropping it keeps the standard unpriced behavior (zero, not a fabricated number) so a missing price stays visible. The rules still supply provider routing, context window, and capability flags, so a brand-new Claude can still be called and its capabilities (including adaptive thinking for >= 4.6) resolved. Description and tests updated to match |
||
|
Sameer Kankute
|
cb041966bf
|
Litellm oss staging 040626 (#29671)
* fix(azure): apply api_version fallback chain to image edit URL
`AzureImageEditConfig.get_complete_url` only read `api_version` from
`litellm_params`. When callers configured it via `litellm.api_version`
or `AZURE_API_VERSION`, the constructed URL had no `?api-version=` and
Azure responded `404 Resource not found`.
Apply the same fallback chain the Azure chat path already uses in
`common_utils.py`:
litellm_params > litellm.api_version > AZURE_API_VERSION env >
litellm.AZURE_DEFAULT_API_VERSION
Adds 5 unit tests pinning each layer of the chain plus a regression
guard for `api_base` that already carries `?api-version=`.
* feat(mcp): core sampling and elicitation flow with security hardening
- Add sampling_handler.py: full MCP sampling/createMessage flow with
model selection (hint-based + priority-based), auth enforcement,
budget checks, route restriction gates, and tag policy pre-auth
- Add elicitation_handler.py: MCP elicitation/create relay with
downstream client capability detection
- Wire sampling/elicitation callbacks in mcp_server_manager.py
gated behind allow_sampling/allow_elicitation config flags
- Add allow_sampling/allow_elicitation fields to MCPServer type
- Fix session lock deadlock: skip lock for JSON-RPC response POSTs
(elicitation/sampling replies) with truncated-body heuristic
- Extend client.py with sampling_callback and elicitation_callback
- Security: RouteChecks gate, tag-budget bypass fix, x-forwarded-for
spoofing fix, Latin-1 header encoding guard
- Add 4 new test modules (model access, priority selection, request
builder, tool conversion) + update existing MCP tests
* fix(security): run pre-call guardrails before MCP sampling acompletion
Without this, an upstream MCP server with allow_sampling enabled could
send prompts that bypass every guardrail (content filtering, PII
redaction, prompt-injection detection) configured on /chat/completions.
- Call proxy_logging_obj.pre_call_hook(call_type='acompletion') before
llm_router.acompletion so guardrails fire for sampling sub-calls
- Add HTTPException to the re-raise list so guardrail rejections
propagate correctly instead of being swallowed as generic errors
* feat(bedrock_mantle): add Responses API support (/openai/v1/responses) (#29490)
* feat(bedrock_mantle): add Responses API transformation config
* test(bedrock_mantle): cover trailing-slash api_base normalization
* feat(bedrock_mantle): export BedrockMantleResponsesAPIConfig
* feat(bedrock_mantle): register gpt-5.x Responses config (gpt-oss unchanged)
* feat(bedrock_mantle): add gpt-5.5/gpt-5.4 Responses price-map entries
* refactor(bedrock_mantle): exclude gpt-oss instead of allow-listing gpt-5 for Responses routing
Frontier OpenAI models on Bedrock Mantle are Responses-only on /openai/v1/responses;
gpt-oss is the legacy family that also speaks chat-completions. Gate by excluding
gpt-oss (which keeps its chat-completions emulation) and defaulting everything else
to the native Responses config, so future frontier models (gpt-6, etc.) route
correctly without a code change. Verified against the live us-east-2 Mantle endpoint:
gpt-oss 400s on /openai/v1/responses while gpt-5.5 400s on both standard paths.
* test(bedrock_mantle): cover supports_native_websocket opt-out
Closes the one uncovered line flagged by codecov on the Responses config.
The assertion documents that Mantle Responses has no realtime/websocket
transport, so realtime routing must not attempt a socket it cannot serve.
* fix(bedrock_mantle): route file_search through emulation instead of forwarding to Mantle
BedrockMantleResponsesAPIConfig inherited supports_native_file_search()
-> True from OpenAIResponsesAPIConfig but never overrode it. Mantle has no
OpenAI vector stores, so a forwarded file_search tool is rejected with a
400 (verified upstream: Tool type 'file_search' is not supported). Opting
out, like the existing supports_native_websocket override, routes the tool
through LiteLLM's file_search emulation instead.
* fix(bedrock_mantle): only route openai.gpt frontier models to Responses
The previous gate excluded gpt-oss and routed every other model to the
native Responses config. But on Mantle only the OpenAI gpt frontier models
(gpt-5.x) are served on /openai/v1/responses; gpt-oss and the non-OpenAI
families (nvidia, mistral, google, zai, ...) are chat-completions only and
400 on that path. Allow-list the openai.gpt- family (excluding gpt-oss)
instead, so chat-only models fall through to the chat-completions emulation.
Verified against the live us-east-2 endpoint: nvidia.nemotron-nano-9b-v2
returns 400 on /openai/v1/responses and 200 on /v1/chat/completions.
* feat(custom_llm): allow streaming/astreaming to yield ModelResponseStream (#27580)
* fix(custom_llm): allow streaming/astreaming to yield ModelResponseStream directly
* fix(streaming): enhance ModelResponseStream handling for custom LLM providers
* fix(streaming): strip finish_reason from content chunks and ensure tool_calls are preserved
* fix(streaming): add type ignore for finish_reason assignment in CustomStreamWrapper
* fix(proxy): strip stack trace from HTTP 503 responses (CWE-209) (#28330)
* fix(proxy/cwe-209): strip Python traceback from HTTP 503 error responses
The /cache/ping endpoint included a full Python traceback in its 503 error
response body (inside the ProxyException message), leaking internal file
paths, line numbers, and call stacks to any caller. Two MCP route handlers
in proxy_server.py similarly interpolated str(e) into "Internal server
error" detail strings.
Fix: log the traceback server-side via verbose_proxy_logger.exception()
and omit it from the ProxyException payload / HTTPException detail returned
to clients. Tests updated to assert no "traceback" keyword or frame paths
appear in the 503 body, with a new dedicated regression test.
CWE-209: Generation of Error Message Containing Sensitive Information.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy/cwe-209): apply Greptile P2 fixes and add MCP exception-path tests
Greptile 4/5 review identified two remaining gaps and Codecov reported
0% coverage on the two MCP handler exception branches:
1. caching_routes.py — str(e) in "Service Unhealthy ({str(e)})" could
still leak Redis hostnames/IPs; replaced with static "Service Unhealthy".
HTTPException is now re-raised before the generic handler so the
"cache not initialized" 503 still reaches callers with its detail.
Removed the redundant str(e) arg from verbose_proxy_logger.exception()
(exception() already appends the traceback automatically).
2. tests — two new unit tests cover the exception paths in
dynamic_mcp_route and toolset_mcp_route that were previously at 0%:
- test_dynamic_mcp_route_unexpected_exception_returns_500_without_traceback
- test_toolset_mcp_route_unexpected_exception_returns_500_without_traceback
All 25 tests pass (9 caching + 16 MCP).
CWE-209: Generation of Error Message Containing Sensitive Information.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(caching_routes): restore precise assertion in test_cache_ping_no_cache_initialized
The assertion was weakened to `"Cache not initialized" in str(data)`, which
matches the raw string of the entire response dict and would pass even if the
error moved to an unexpected field or changed structure.
Restore a targeted check on the parsed response: assert the exact string in
the correct field `data["detail"]`, matching FastAPI's HTTPException
serialisation format {"detail": "<message>"}.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(caching_routes): restore precise assertion and add CWE-209 no-cache path test
The assertion in test_cache_ping_no_cache_initialized was weakened to
`"Cache not initialized" in str(data)`, which matched against the raw string
representation of the entire response dict. This would pass silently even if
the error message moved to an unexpected field or the structure changed.
Restore a targeted assertion on the parsed field:
assert data["detail"] == "Cache not initialized. litellm.cache is None"
matching FastAPI's HTTPException serialisation format exactly.
Add test_cache_ping_no_cache_does_not_expose_internals to show the code path
is still working correctly after the CWE-209 fix: verifies that the HTTPException
is re-raised as-is (no traceback, no source paths), and asserts the complete
response structure is exactly {"detail": "Cache not initialized. litellm.cache is None"}.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(caching_routes): restore ProxyException envelope for null-cache 503
The except HTTPException: raise guard (added in the CWE-209 fix) caused
the null-cache HTTPException to escape as FastAPI's {"detail": "..."} shape
instead of the {"error": {...}} ProxyException envelope that callers expect.
Move the null-cache guard before the try block and raise ProxyException
directly so the response structure is consistent with all other /cache/ping
503s, and the except HTTPException: raise guard is only reachable by
unexpected downstream HTTPExceptions.
Update the two no-cache tests to assert the correct ProxyException envelope.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Update utils.py (#26609)
* feat(pricing): add Snowflake Cortex REST API model pricing (#26612)
* feat(pricing): add Snowflake Cortex REST API model pricing
## Summary
Adds pricing and context window information for 20+ Snowflake Cortex REST API models to `model_prices_and_context_window.json`.
## What's included
- **7 Claude models** (sonnet-4-5, sonnet-4-6, 4-sonnet, 4-opus, haiku-4-5, 3-7-sonnet, 3-5-sonnet) — with prompt caching rates
- **4 OpenAI models** (gpt-4.1, gpt-5, gpt-5-mini, gpt-5-nano) — with prompt caching rates
- **5 Llama models** (3.1-8b, 3.1-70b, 3.1-405b, 3.3-70b, 4-maverick)
- **1 DeepSeek model** (deepseek-r1)
- **1 Mistral model** (mistral-large2)
- **1 Snowflake model** (snowflake-llama-3.3-70b)
- **2 Embedding models** (arctic-embed-l-v2.0, arctic-embed-m-v2.0)
Each entry includes `input_cost_per_token`, `output_cost_per_token`, `cache_read_input_token_cost` (where applicable), `max_input_tokens`, `max_output_tokens`, and capability flags (`supports_function_calling`, `supports_vision`, `supports_prompt_caching`, `supports_reasoning`).
## Pricing source
All prices are in USD per token, sourced from the official [Snowflake Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf) — Tables 6(b) (REST API with Prompt Caching) and 6(c) (REST API).
## Context
The existing `snowflake/` provider has zero model entries in the pricing JSON, which means LiteLLM cannot track costs for Snowflake Cortex calls. This PR fills that gap.
## Related
- Existing provider: `litellm/llms/snowflake/`
- Cortex REST API docs: https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-rest-api
* Update model_prices_and_context_window.json
Fix the JSON parsing error
* Update model_prices_and_context_window.json
Removed the duplicate entry
* fix(utils): copy extra_body before adding unknown params to prevent model config mutation (#29620)
Fixes #29615. In add_provider_specific_params_to_optional_params, the line:
extra_body = passed_params.pop("extra_body", None) or {}
returns the original dict reference when extra_body is non-empty (truthy).
Subsequent writes like extra_body[k] = passed_params[k] then mutate the
shared model config object held by the router, poisoning /model/info and
all subsequent requests for that deployment.
The or {} short-circuit creates a new dict only when extra_body is falsy
(None or {}), which is why the bug does not reproduce with extra_body: {}.
Fix: wrap in dict() so we always work on a fresh shallow copy.
* fix(vertex_ai): Bake tool_choice into Gemini CachedContent body to prevent silent drop (#29097)
* fix(vertex_ai): bake tool_choice into Gemini CachedContent body to prevent silent drop
* address greptile feedback on tool_choice cache test
* adds test that uses ToolConfig(functionCallingConfig=FunctionCallingConfig(mode=ANY)) instead of a dict literal, mirroring what map_tool_choice_values actually produce
* fix(gemini/veo): move image from parameters into instances[0] (#29501)
* fix(gemini/veo): move image from parameters into instances[0]
Veo's predictLongRunning schema puts image (and prompt) on the
instances element; parameters is for aspectRatio/durationSeconds/etc.
The Gemini path was leaving image in params_copy, so it ended up
nested under parameters and the API silently ignored it.
The Vertex path already builds the instance dict explicitly, so this
just aligns the Gemini path with it.
Fixes #29498
* address greptile: unconditional pop + BytesIO test
- Pop `image` from params_copy unconditionally so it never reaches
GeminiVideoGenerationParameters even when None, removing implicit
reliance on Pydantic's extra-field-ignore.
- Add test_transform_video_create_request_image_filelike_goes_to_instance
covering the BytesIO path (_convert_image_to_gemini_format) — round-trips
the base64 to confirm encoding.
- Add test_transform_video_create_request_image_none_is_dropped covering
the new None branch.
* fix(huggingface): handle special token text in embedding usage (#29660)
* fix(guardrails): recompile ToolPermissionGuardrail rules on update_in_memory_litellm_params (#29655)
* fix(guardrails): recompile ToolPermissionGuardrail rules on update_in_memory_litellm_params
ToolPermissionGuardrail builds self.rules and the compiled target/pattern
maps only in __init__. The base update_in_memory_litellm_params re-sets raw
attributes via setattr but never rebuilds those maps, so a guardrail updated
in place (PUT /guardrails, or the immediate in-memory sync) keeps enforcing
the construction-time rules until it is reinitialized (PATCH path, periodic
DB poll, or restart).
Extract the compile step into _load_rules and override
update_in_memory_litellm_params to rebuild from it (dict- and model-safe),
re-normalizing default_action / on_disallowed_action. Mirrors the existing
PresidioGuardrail override of the same method. Adds regression tests.
Fixes #29592.
* fix(guardrails): handle dict params in ToolPermissionGuardrail in-memory update
Delegate to super() only for LitellmParams input (the base setattr loop is
model-only); apply the raw-dict case inline. Fixes the mypy arg-type error
and makes the recompile work when the proxy passes the raw DB dict.
* fix(guardrails): preserve tool-permission rules on a partial in-memory update
A partial update (e.g. a LitellmParams whose rules field is None) ran through
the generic setattr, which set self.rules to None, and the recompile was
skipped, leaving the guardrail with no rules. Snapshot the previous rules and
restore them when the update carries no rules; an explicit empty list still
clears them. Adds a regression test for the rules-absent case.
Addresses the Greptile review note on #29655.
* fix(bedrock): stop base_model label from stripping tools/tool_choice (#29621)
* fix(bedrock): stop base_model label from stripping tools/tool_choice
A Router/proxy Bedrock deployment whose model_info.base_model is a friendly
label (e.g. claude-haiku-4-5) silently lost tools/tool_choice: the outgoing
Converse request was built without toolConfig, so the model behaved as if no
tools were provided. Worked in v1.84.0, regressed in v1.85.0, and with
drop_params=true it failed silently.
Two changes compound into the bug. completion() passed model_info.base_model
as the model argument to get_optional_params, so the real Bedrock model id
never reached supported-param resolution; and get_supported_openai_params
resolved the provider config's params from base_model or model, letting the
label fully replace the real model. For Bedrock the label resolves to no tool
support, so tools/tool_choice were dropped before transformation.
completion() now keeps model as the real deployment model and threads the
resolved base_model (kwarg or model_info) through separately, and
get_supported_openai_params treats base_model as additive: it returns the
union of the params supported by model and by base_model. A hint can only add
capabilities, never strip ones the real model already exposes, which also
preserves the original base_model behavior from #27717 and Azure's base_model
driven model-type detection.
Fixes #29618
* test(main): make base_model param test robust to new parametrize cases
Restore an explicit per-case expected_model_param literal instead of
hardcoding the gemini id, so a future case with a different model can't
produce a misleading assertion failure.
* fix(fireworks_ai): pass response_format json_schema through unchanged (#29606)
FireworksAIConfig.map_openai_params was rewriting the OpenAI strict
`{type: json_schema, json_schema: {name, strict, schema}}` shape into
`{type: json_object, schema: ...}` before sending to Fireworks, dropping
`strict` and `name` and changing the `type`. Per Fireworks' docs json_object
means "force any valid JSON output (no specific schema)", so the schema
constraint was effectively dropped and grammar-guided decoding never ran;
model output silently violated the schema.
The rewrite landed in #7085 (Dec 2024) when Fireworks did not yet accept
native json_schema. Fireworks accepts the OpenAI strict shape natively now,
so the rewrite has become a regression.
Removes the rewrite. Passes response_format through unchanged. Updates the
existing test_map_response_format to assert pass-through. Adds focused
regression tests in tests/test_litellm/ covering preservation of type,
strict, name, and schema body, plus that json_object alone still works.
* fix(types): import Required from typing_extensions in gemini types
* style: reformat sampling_handler.py for py312 black compat
* refactor(mcp-sampling): extract helpers to fix PLR0915 too-many-statements in handle_sampling_create_message
* fix(proxy-server): add explicit ProxyLogging type annotation to proxy_logging_obj to fix mypy inference
* fix(mcp-sampling): suppress mypy assignment error on ImportError fallback for proxy_logging_obj
* fix(test): use .value when comparing LlmProviders enum against string in test_default_api_base
* fix(test): iterate LlmProviders enum in test_default_api_base to avoid str pollution from custom provider registration
litellm.provider_list is a mutable global initialized to list(LlmProviders) but custom_llm_setup() appends plain provider strings to it. When a test_custom_llm.py test runs first in the same xdist worker, provider_list contains a str and calling .value on it raises AttributeError. Iterate the immutable LlmProviders enum instead, which is deterministic and what the check intends.
* fix(mcp): depth-aware JSON-RPC response detection and neutral speed-priority fallback
Replace the flat substring check in the truncated-body routing path with a
top-level-key scan so a JSON-RPC response whose result payload nests a
"method" field is still detected as a response and skips the session lock,
removing a deadlock against the in-flight tool call awaiting it.
Drop the inverse max_output_tokens speed proxy when no model exposes
output_tokens_per_second; context-window size does not track latency, so a
neutral score avoids biasing speedPriority toward the smallest-context model.
* fix(guardrails): make ToolPermission rule reload atomic on invalid regex
_load_rules appended each rule to self.rules before compiling its regex, so an
invalid pattern raised mid-loop after the bad rule was already live but without
a _compiled_rule_targets entry. _matches_regex reads a missing compiled target
as a None pattern and returns True, turning the bad rule into a match-all that
silently applies its decision to every tool. Via update_in_memory_litellm_params
(PUT /guardrails) this corrupted the live guardrail.
Build the parsed rules and compiled maps into locals and swap them in only after
every regex compiles, and restore the previous ruleset if a live update is
rejected, so an invalid regex now fails the update without leaving the guardrail
enforcing a broken policy.
* test(mcp): cover sampling conversion, model resolution, and elicitation relay paths
The MCP sampling and elicitation handlers shipped with partial test
coverage, leaving the response-to-MCP conversion, the model resolution
fallback chain, completion-kwargs assembly, guardrail routing, and the
entire elicitation relay untested. That pulled the PR's diff (patch)
coverage below the codecov threshold even though overall project
coverage rose.
Add focused unit tests for _convert_openai_response_to_mcp_result,
_convert_mcp_tools_to_openai, _convert_mcp_tool_choice_to_openai, image
and audio content conversion, the hint-matching and fallback branches of
_resolve_model_from_preferences, _build_completion_kwargs, the router and
guardrail-rejection paths of _run_guardrails_and_call_llm, the
handle_sampling_create_message success and error-propagation flows, the
marker-hoisting fallback for tool content on unexpected roles, and the
elicitation form/url/generic relay together with its decline paths
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: lengkejun <lengkejun@xd.com>
Co-authored-by: Yug <yugborana000@gmail.com>
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: tanmay958 <53569547+tanmay958@users.noreply.github.com>
Co-authored-by: DrishnaTrivedi <142084770+DrishnaTrivedi@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Navnit Shukla <Navnit.shukla25@gmail.com>
Co-authored-by: PRABHU KIRAN VANDRANKI <72809214+VANDRANKI@users.noreply.github.com>
Co-authored-by: Adrian Lopez <109683617+adriangomez24@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: JooHo Lee <96564470+BWAAEEEK@users.noreply.github.com>
Co-authored-by: Dinesh Girbide <85330597+Dinesh-Girbide@users.noreply.github.com>
Co-authored-by: cloudwiz <22098246+andrey-dubnik@users.noreply.github.com>
Co-authored-by: Ahmad Khan <ahmadkhan2508@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Mateo Wang
|
f81d8ae077
|
[internal copy of #29232] feat: route future Claude models to Anthropic provider via pattern matching (#29239)
* feat: route future Claude models to Anthropic provider via pattern matching
Add pattern-based matching for Claude model names so that future models
(e.g., claude-opus-4-9, claude-sonnet-5-0) are automatically routed to
the Anthropic provider without requiring model_prices_and_context_window.json
updates.
The pattern matches: claude-{opus|sonnet|haiku}-{major}-{minor}[-YYYYMMDD]
https://claude.ai/code/session_017asCVDN5jBFMBcZRjiQR6C
* fix: don't hard-code the tier names
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* style: move import re to module level (PEP 8)
Move `import re` from inside the module body to the top-level imports
section, following PEP 8 style guidelines that all imports should
appear at the top of the file.
https://claude.ai/code/session_01Dt8fzn81eYMfxu1MoBa5hN
* test: fix claude-mini-4-5 assertion to match generic-tier pattern
The pattern intentionally accepts any [a-z]+ tier (see
|
||
|
Chesars
|
c75f0c0566 |
test: drop duplicate openrouter prefix-strip test
The multi-segment case (openrouter/<provider>/<model> → <provider>/<model>) is already covered by tests/test_litellm/llms/openrouter/test_openrouter_provider_routing.py in internal_staging, with broader coverage including double-prefix native models, wildcard deployments, and the bridge double-call scenario. Keeping a duplicate in tests/local_testing/ adds maintenance load with no extra coverage. |
||
|
Chesars
|
232464e151 |
test: drop incorrect openrouter native-prefix test
The removed test asserted that get_llm_provider(model="openrouter/auto") should return model="openrouter/auto" with the prefix preserved. That contract is wrong: the LiteLLM convention strips one "openrouter/" prefix, so a native OpenRouter model is reached via "openrouter/openrouter/auto" (double-prefix) which the wire send as "openrouter/auto" — the ID the OpenRouter API expects for natives. E2E checks against api.openrouter.ai confirm this is the correct routing convention for all native models (auto, bodybuilder, free, pareto-code). |
||
|
Chesars
|
91e78eca3d |
Merge remote-tracking branch 'upstream/litellm_internal_staging' into upstream-litellm_staging_03_21_2026
# Conflicts: # .circleci/config.yml # .circleci/requirements.txt # .github/workflows/_test-unit-base.yml # .github/workflows/_test-unit-services-base.yml # .github/workflows/auto_update_price_and_context_window.yml # .github/workflows/create-release.yml # .github/workflows/llm-translation-testing.yml # .github/workflows/publish_to_pypi.yml # .github/workflows/scan_duplicate_issues.yml # .github/workflows/test-linting.yml # .github/workflows/test-litellm-matrix.yml # .github/workflows/test-litellm.yml # .github/workflows/test-mcp.yml # .github/workflows/test-model-map.yaml # .github/workflows/test-proxy-e2e-azure-batches.yml # .github/workflows/test-unit-core-utils.yml # .github/workflows/test-unit-documentation.yml # .github/workflows/test-unit-enterprise-routing.yml # .github/workflows/test-unit-integrations.yml # .github/workflows/test-unit-llm-providers.yml # .github/workflows/test-unit-misc.yml # .github/workflows/test-unit-proxy-auth.yml # .github/workflows/test-unit-proxy-db.yml # .github/workflows/test-unit-proxy-endpoints.yml # .github/workflows/test-unit-proxy-infra.yml # .github/workflows/test-unit-proxy-legacy.yml # .github/workflows/test-unit-responses-caching-types.yml # .github/workflows/test-unit-security.yml # .github/workflows/test_server_root_path.yml # docs/my-website/docs/embedding/supported_embedding.md # litellm/litellm_core_utils/get_llm_provider_logic.py # litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_transformation.py # litellm/proxy/_experimental/out/404/index.html # litellm/proxy/_experimental/out/__next.__PAGE__.txt # litellm/proxy/_experimental/out/__next._full.txt # litellm/proxy/_experimental/out/__next._head.txt # litellm/proxy/_experimental/out/__next._index.txt # litellm/proxy/_experimental/out/__next._tree.txt # litellm/proxy/_experimental/out/_next/static/3qyC5Vtvhd5fSC6sPp1iW/_buildManifest.js # litellm/proxy/_experimental/out/_next/static/3qyC5Vtvhd5fSC6sPp1iW/_clientMiddlewareManifest.json # litellm/proxy/_experimental/out/_next/static/3qyC5Vtvhd5fSC6sPp1iW/_ssgManifest.js # litellm/proxy/_experimental/out/_next/static/aKKihXXKRJWLQThZgi8Rq/_buildManifest.js # litellm/proxy/_experimental/out/_next/static/aKKihXXKRJWLQThZgi8Rq/_clientMiddlewareManifest.json # litellm/proxy/_experimental/out/_next/static/aKKihXXKRJWLQThZgi8Rq/_ssgManifest.js # litellm/proxy/_experimental/out/_next/static/bmMTxs1O5fQKYcsMNTRMT/_buildManifest.js # litellm/proxy/_experimental/out/_next/static/bmMTxs1O5fQKYcsMNTRMT/_clientMiddlewareManifest.json # litellm/proxy/_experimental/out/_next/static/bmMTxs1O5fQKYcsMNTRMT/_ssgManifest.js # litellm/proxy/_experimental/out/_next/static/chunks/11362340846735c3.js # litellm/proxy/_experimental/out/_next/static/chunks/1a04d31843c96649.js # litellm/proxy/_experimental/out/_next/static/chunks/342c7d7210247a5e.js # litellm/proxy/_experimental/out/_next/static/chunks/39768ec0eebd2554.js # litellm/proxy/_experimental/out/_next/static/chunks/3b3c0b070b14da06.js # litellm/proxy/_experimental/out/_next/static/chunks/3bddc72a3ecc2253.js # litellm/proxy/_experimental/out/_next/static/chunks/4472ece1be7379b3.js # litellm/proxy/_experimental/out/_next/static/chunks/54e29148cb2f2582.js # litellm/proxy/_experimental/out/_next/static/chunks/67ddb5107368a659.js # litellm/proxy/_experimental/out/_next/static/chunks/6a167cef4b09b496.js # litellm/proxy/_experimental/out/_next/static/chunks/7174130ddef406dd.js # litellm/proxy/_experimental/out/_next/static/chunks/7c36bfe1ba5e3ba8.js # litellm/proxy/_experimental/out/_next/static/chunks/7e5fe5584502da06.js # litellm/proxy/_experimental/out/_next/static/chunks/8dda507c226082ca.js # litellm/proxy/_experimental/out/_next/static/chunks/8dfde809dc4ad794.js # litellm/proxy/_experimental/out/_next/static/chunks/99109c78121231a0.js # litellm/proxy/_experimental/out/_next/static/chunks/9dd55e1f36a7225c.js # litellm/proxy/_experimental/out/_next/static/chunks/a230559fcabaea23.js # litellm/proxy/_experimental/out/_next/static/chunks/a6c7f80b3968f639.js # litellm/proxy/_experimental/out/_next/static/chunks/ac9e96d21c200b48.js # litellm/proxy/_experimental/out/_next/static/chunks/ae9cf43b8c0c76aa.js # litellm/proxy/_experimental/out/_next/static/chunks/cf06797ce4e438f9.js # litellm/proxy/_experimental/out/_next/static/chunks/d069df5baead6d90.js # litellm/proxy/_experimental/out/_next/static/chunks/d2e3b7dd6499c245.js # litellm/proxy/_experimental/out/_next/static/chunks/d44e73d8ebac5747.js # litellm/proxy/_experimental/out/_next/static/chunks/dc8a270fee94ced6.js # litellm/proxy/_experimental/out/_next/static/chunks/df6546cd8a44d3b3.js # litellm/proxy/_experimental/out/_next/static/chunks/ea0f22bd4b3393bd.js # litellm/proxy/_experimental/out/_next/static/chunks/eaa9f9b9bb3e054b.js # litellm/proxy/_experimental/out/_next/static/chunks/turbopack-901b35f89c1f6751.js # litellm/proxy/_experimental/out/_next/static/chunks/turbopack-d1b22f5e0bd58c57.js # litellm/proxy/_experimental/out/_next/static/chunks/turbopack-ddedb29a5eb0118f.js # litellm/proxy/_experimental/out/_not-found.txt # litellm/proxy/_experimental/out/_not-found/__next._full.txt # litellm/proxy/_experimental/out/_not-found/__next._head.txt # litellm/proxy/_experimental/out/_not-found/__next._index.txt # litellm/proxy/_experimental/out/_not-found/__next._not-found.__PAGE__.txt # litellm/proxy/_experimental/out/_not-found/__next._not-found.txt # litellm/proxy/_experimental/out/_not-found/__next._tree.txt # litellm/proxy/_experimental/out/_not-found/index.html # litellm/proxy/_experimental/out/api-reference.html # litellm/proxy/_experimental/out/api-reference.txt # litellm/proxy/_experimental/out/api-reference/__next.!KGRhc2hib2FyZCk.api-reference.__PAGE__.txt # litellm/proxy/_experimental/out/api-reference/__next.!KGRhc2hib2FyZCk.api-reference.txt # litellm/proxy/_experimental/out/api-reference/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/api-reference/__next._full.txt # litellm/proxy/_experimental/out/api-reference/__next._head.txt # litellm/proxy/_experimental/out/api-reference/__next._index.txt # litellm/proxy/_experimental/out/api-reference/__next._tree.txt # litellm/proxy/_experimental/out/chat.html # litellm/proxy/_experimental/out/chat.txt # litellm/proxy/_experimental/out/chat/__next._full.txt # litellm/proxy/_experimental/out/chat/__next._head.txt # litellm/proxy/_experimental/out/chat/__next._index.txt # litellm/proxy/_experimental/out/chat/__next._tree.txt # litellm/proxy/_experimental/out/chat/__next.chat.__PAGE__.txt # litellm/proxy/_experimental/out/chat/__next.chat.txt # litellm/proxy/_experimental/out/experimental/api-playground.html # litellm/proxy/_experimental/out/experimental/api-playground.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.experimental.api-playground.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.experimental.api-playground.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._full.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._head.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._index.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._tree.txt # litellm/proxy/_experimental/out/experimental/budgets.html # litellm/proxy/_experimental/out/experimental/budgets.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.experimental.budgets.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.experimental.budgets.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._full.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._head.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._index.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._tree.txt # litellm/proxy/_experimental/out/experimental/caching.html # litellm/proxy/_experimental/out/experimental/caching.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.experimental.caching.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.experimental.caching.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/caching/__next._full.txt # litellm/proxy/_experimental/out/experimental/caching/__next._head.txt # litellm/proxy/_experimental/out/experimental/caching/__next._index.txt # litellm/proxy/_experimental/out/experimental/caching/__next._tree.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins.html # litellm/proxy/_experimental/out/experimental/claude-code-plugins.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.experimental.claude-code-plugins.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.experimental.claude-code-plugins.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._full.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._head.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._index.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._tree.txt # litellm/proxy/_experimental/out/experimental/old-usage.html # litellm/proxy/_experimental/out/experimental/old-usage.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.experimental.old-usage.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.experimental.old-usage.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._full.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._head.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._index.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._tree.txt # litellm/proxy/_experimental/out/experimental/prompts.html # litellm/proxy/_experimental/out/experimental/prompts.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.experimental.prompts.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.experimental.prompts.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._full.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._head.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._index.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._tree.txt # litellm/proxy/_experimental/out/experimental/tag-management.html # litellm/proxy/_experimental/out/experimental/tag-management.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.experimental.tag-management.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.experimental.tag-management.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._full.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._head.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._index.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._tree.txt # litellm/proxy/_experimental/out/guardrails.html # litellm/proxy/_experimental/out/guardrails.txt # litellm/proxy/_experimental/out/guardrails/__next.!KGRhc2hib2FyZCk.guardrails.__PAGE__.txt # litellm/proxy/_experimental/out/guardrails/__next.!KGRhc2hib2FyZCk.guardrails.txt # litellm/proxy/_experimental/out/guardrails/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/guardrails/__next._full.txt # litellm/proxy/_experimental/out/guardrails/__next._head.txt # litellm/proxy/_experimental/out/guardrails/__next._index.txt # litellm/proxy/_experimental/out/guardrails/__next._tree.txt # litellm/proxy/_experimental/out/index.html # litellm/proxy/_experimental/out/index.txt # litellm/proxy/_experimental/out/login.html # litellm/proxy/_experimental/out/login.txt # litellm/proxy/_experimental/out/login/__next._full.txt # litellm/proxy/_experimental/out/login/__next._head.txt # litellm/proxy/_experimental/out/login/__next._index.txt # litellm/proxy/_experimental/out/login/__next._tree.txt # litellm/proxy/_experimental/out/login/__next.login.__PAGE__.txt # litellm/proxy/_experimental/out/login/__next.login.txt # litellm/proxy/_experimental/out/logs.html # litellm/proxy/_experimental/out/logs.txt # litellm/proxy/_experimental/out/logs/__next.!KGRhc2hib2FyZCk.logs.__PAGE__.txt # litellm/proxy/_experimental/out/logs/__next.!KGRhc2hib2FyZCk.logs.txt # litellm/proxy/_experimental/out/logs/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/logs/__next._full.txt # litellm/proxy/_experimental/out/logs/__next._head.txt # litellm/proxy/_experimental/out/logs/__next._index.txt # litellm/proxy/_experimental/out/logs/__next._tree.txt # litellm/proxy/_experimental/out/mcp/oauth/callback.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._full.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._head.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._index.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._tree.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.oauth.callback.__PAGE__.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.oauth.callback.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.oauth.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/index.html # litellm/proxy/_experimental/out/model-hub.html # litellm/proxy/_experimental/out/model-hub.txt # litellm/proxy/_experimental/out/model-hub/__next.!KGRhc2hib2FyZCk.model-hub.__PAGE__.txt # litellm/proxy/_experimental/out/model-hub/__next.!KGRhc2hib2FyZCk.model-hub.txt # litellm/proxy/_experimental/out/model-hub/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/model-hub/__next._full.txt # litellm/proxy/_experimental/out/model-hub/__next._head.txt # litellm/proxy/_experimental/out/model-hub/__next._index.txt # litellm/proxy/_experimental/out/model-hub/__next._tree.txt # litellm/proxy/_experimental/out/model_hub.html # litellm/proxy/_experimental/out/model_hub.txt # litellm/proxy/_experimental/out/model_hub/__next._full.txt # litellm/proxy/_experimental/out/model_hub/__next._head.txt # litellm/proxy/_experimental/out/model_hub/__next._index.txt # litellm/proxy/_experimental/out/model_hub/__next._tree.txt # litellm/proxy/_experimental/out/model_hub/__next.model_hub.__PAGE__.txt # litellm/proxy/_experimental/out/model_hub/__next.model_hub.txt # litellm/proxy/_experimental/out/model_hub_table.html # litellm/proxy/_experimental/out/model_hub_table.txt # litellm/proxy/_experimental/out/model_hub_table/__next._full.txt # litellm/proxy/_experimental/out/model_hub_table/__next._head.txt # litellm/proxy/_experimental/out/model_hub_table/__next._index.txt # litellm/proxy/_experimental/out/model_hub_table/__next._tree.txt # litellm/proxy/_experimental/out/model_hub_table/__next.model_hub_table.__PAGE__.txt # litellm/proxy/_experimental/out/model_hub_table/__next.model_hub_table.txt # litellm/proxy/_experimental/out/models-and-endpoints.html # litellm/proxy/_experimental/out/models-and-endpoints.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next.!KGRhc2hib2FyZCk.models-and-endpoints.__PAGE__.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next.!KGRhc2hib2FyZCk.models-and-endpoints.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._full.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._head.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._index.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._tree.txt # litellm/proxy/_experimental/out/onboarding.html # litellm/proxy/_experimental/out/onboarding.txt # litellm/proxy/_experimental/out/onboarding/__next._full.txt # litellm/proxy/_experimental/out/onboarding/__next._head.txt # litellm/proxy/_experimental/out/onboarding/__next._index.txt # litellm/proxy/_experimental/out/onboarding/__next._tree.txt # litellm/proxy/_experimental/out/onboarding/__next.onboarding.__PAGE__.txt # litellm/proxy/_experimental/out/onboarding/__next.onboarding.txt # litellm/proxy/_experimental/out/organizations.html # litellm/proxy/_experimental/out/organizations.txt # litellm/proxy/_experimental/out/organizations/__next.!KGRhc2hib2FyZCk.organizations.__PAGE__.txt # litellm/proxy/_experimental/out/organizations/__next.!KGRhc2hib2FyZCk.organizations.txt # litellm/proxy/_experimental/out/organizations/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/organizations/__next._full.txt # litellm/proxy/_experimental/out/organizations/__next._head.txt # litellm/proxy/_experimental/out/organizations/__next._index.txt # litellm/proxy/_experimental/out/organizations/__next._tree.txt # litellm/proxy/_experimental/out/playground.html # litellm/proxy/_experimental/out/playground.txt # litellm/proxy/_experimental/out/playground/__next.!KGRhc2hib2FyZCk.playground.__PAGE__.txt # litellm/proxy/_experimental/out/playground/__next.!KGRhc2hib2FyZCk.playground.txt # litellm/proxy/_experimental/out/playground/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/playground/__next._full.txt # litellm/proxy/_experimental/out/playground/__next._head.txt # litellm/proxy/_experimental/out/playground/__next._index.txt # litellm/proxy/_experimental/out/playground/__next._tree.txt # litellm/proxy/_experimental/out/policies.html # litellm/proxy/_experimental/out/policies.txt # litellm/proxy/_experimental/out/policies/__next.!KGRhc2hib2FyZCk.policies.__PAGE__.txt # litellm/proxy/_experimental/out/policies/__next.!KGRhc2hib2FyZCk.policies.txt # litellm/proxy/_experimental/out/policies/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/policies/__next._full.txt # litellm/proxy/_experimental/out/policies/__next._head.txt # litellm/proxy/_experimental/out/policies/__next._index.txt # litellm/proxy/_experimental/out/policies/__next._tree.txt # litellm/proxy/_experimental/out/settings/admin-settings.html # litellm/proxy/_experimental/out/settings/admin-settings.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.settings.admin-settings.__PAGE__.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.settings.admin-settings.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._full.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._head.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._index.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._tree.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts.html # litellm/proxy/_experimental/out/settings/logging-and-alerts.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.settings.logging-and-alerts.__PAGE__.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.settings.logging-and-alerts.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._full.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._head.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._index.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._tree.txt # litellm/proxy/_experimental/out/settings/router-settings.html # litellm/proxy/_experimental/out/settings/router-settings.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.settings.router-settings.__PAGE__.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.settings.router-settings.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._full.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._head.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._index.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._tree.txt # litellm/proxy/_experimental/out/settings/ui-theme.html # litellm/proxy/_experimental/out/settings/ui-theme.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.settings.ui-theme.__PAGE__.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.settings.ui-theme.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._full.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._head.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._index.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._tree.txt # litellm/proxy/_experimental/out/teams.html # litellm/proxy/_experimental/out/teams.txt # litellm/proxy/_experimental/out/teams/__next.!KGRhc2hib2FyZCk.teams.__PAGE__.txt # litellm/proxy/_experimental/out/teams/__next.!KGRhc2hib2FyZCk.teams.txt # litellm/proxy/_experimental/out/teams/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/teams/__next._full.txt # litellm/proxy/_experimental/out/teams/__next._head.txt # litellm/proxy/_experimental/out/teams/__next._index.txt # litellm/proxy/_experimental/out/teams/__next._tree.txt # litellm/proxy/_experimental/out/test-key.html # litellm/proxy/_experimental/out/test-key.txt # litellm/proxy/_experimental/out/test-key/__next.!KGRhc2hib2FyZCk.test-key.__PAGE__.txt # litellm/proxy/_experimental/out/test-key/__next.!KGRhc2hib2FyZCk.test-key.txt # litellm/proxy/_experimental/out/test-key/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/test-key/__next._full.txt # litellm/proxy/_experimental/out/test-key/__next._head.txt # litellm/proxy/_experimental/out/test-key/__next._index.txt # litellm/proxy/_experimental/out/test-key/__next._tree.txt # litellm/proxy/_experimental/out/tools/mcp-servers.html # litellm/proxy/_experimental/out/tools/mcp-servers.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.tools.mcp-servers.__PAGE__.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.tools.mcp-servers.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.tools.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._full.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._head.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._index.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._tree.txt # litellm/proxy/_experimental/out/tools/vector-stores.html # litellm/proxy/_experimental/out/tools/vector-stores.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.tools.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.tools.vector-stores.__PAGE__.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.tools.vector-stores.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._full.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._head.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._index.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._tree.txt # litellm/proxy/_experimental/out/usage.html # litellm/proxy/_experimental/out/usage.txt # litellm/proxy/_experimental/out/usage/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/usage/__next.!KGRhc2hib2FyZCk.usage.__PAGE__.txt # litellm/proxy/_experimental/out/usage/__next.!KGRhc2hib2FyZCk.usage.txt # litellm/proxy/_experimental/out/usage/__next._full.txt # litellm/proxy/_experimental/out/usage/__next._head.txt # litellm/proxy/_experimental/out/usage/__next._index.txt # litellm/proxy/_experimental/out/usage/__next._tree.txt # litellm/proxy/_experimental/out/users.html # litellm/proxy/_experimental/out/users.txt # litellm/proxy/_experimental/out/users/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/users/__next.!KGRhc2hib2FyZCk.users.__PAGE__.txt # litellm/proxy/_experimental/out/users/__next.!KGRhc2hib2FyZCk.users.txt # litellm/proxy/_experimental/out/users/__next._full.txt # litellm/proxy/_experimental/out/users/__next._head.txt # litellm/proxy/_experimental/out/users/__next._index.txt # litellm/proxy/_experimental/out/users/__next._tree.txt # litellm/proxy/_experimental/out/virtual-keys.html # litellm/proxy/_experimental/out/virtual-keys.txt # litellm/proxy/_experimental/out/virtual-keys/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/virtual-keys/__next.!KGRhc2hib2FyZCk.virtual-keys.__PAGE__.txt # litellm/proxy/_experimental/out/virtual-keys/__next.!KGRhc2hib2FyZCk.virtual-keys.txt # litellm/proxy/_experimental/out/virtual-keys/__next._full.txt # litellm/proxy/_experimental/out/virtual-keys/__next._head.txt # litellm/proxy/_experimental/out/virtual-keys/__next._index.txt # litellm/proxy/_experimental/out/virtual-keys/__next._tree.txt # scripts/install.sh # tests/local_testing/test_get_llm_provider.py |
||
|
Ishaan Jaffer
|
e8461b5b97
|
style: run black formatter on files from main merge | ||
|
David Chen
|
d1df4e838b
|
Litellm fix update bedrock models (#24947)
* update bedrock models in tests * updated more tests and model_prices_and_context_window * fix model id and pricing * replace more sonnet models * update tests * git push * update pricing * flaky total cost * monkey patch * relax the cost change * fix and revert some changes * revert the pricing * chore: move cost/pricing changes to bedrock-cost-fixes branch * chore: split Bedrock file-api beta stripping to separate branch Removes strip_unsupported_file_api_betas_for_bedrock_invoke from this branch; see litellm_bedrock_invoke_strip_file_api_betas for that fix. Made-with: Cursor |
||
|
Imgyu Kim
|
ad07d7faad |
fix: strip 'openrouter/' prefix from model names (#24234)
Remove early return in get_llm_provider_logic.py that prevented the 'openrouter/' prefix from being stripped. The early return was intended for 'native OpenRouter models' like 'openrouter/free', but no such models exist in the model registry — all OpenRouter models are multi-segment (e.g. 'openrouter/anthropic/claude-3.5-sonnet') and need the prefix stripped before being sent to the OpenRouter API. This regression was introduced in v1.82.3 and caused 400 Bad Request errors for all OpenRouter models. |
||
|
Sameer Kankute
|
a5ea08a0bf | Fix test_default_api_base failing because of chatgpt as provider | ||
|
Sameer Kankute
|
6751badf3a | fix: test_default_api_base for ragfow | ||
|
Ishaan Jaff
|
19e26a5c60 | test_default_api_base | ||
|
Ishaan Jaff
|
643d2a8ccb
|
[Feat] Option to force/always use the litellm proxy (#10559) (#10633) (#10773)
* [Feat] Option to force/always use the litellm proxy (#10559) (#10633) * fix: add use_litellm_proxy * fix: update LiteLLMProxyChatConfig * fix get llm provider logic * tests get llm provider logic * add dynamic use_litellm_proxy * docs forcsing litellm proxy usage * fix: _should_use_litellm_proxy_by_default * fixes: get_custom_llm_provider --------- Co-authored-by: Antoine Legrand <2t.antoine@gmail.com> |
||
|
Ishaan Jaff
|
de7870cb54
|
Add llamafile as a provider (#10203) (#10482)
* Update docs for OpenAI compatible providers, add Llamafile docs, include Llamafile in the sidebar * Add Llamafile as an LlmProviders enum * Add llamafile as a OpenAI compatible provider (in the list of compatible providers) * Add Llamafile chat config and tests * Wire up Llamafile Co-authored-by: Peter Wilson <peter@mozilla.ai> |
||
|
Krrish Dholakia
|
267084a1af | test(test_get_llm_provider.py): cover scenario where xai not in model name | ||
|
Krrish Dholakia
|
aeec703c4e | test(test_get_llm_provider.py): Minimal repro for https://github.com/BerriAI/litellm/issues/9291 | ||
|
Ishaan Jaff
|
b242c66a3b
|
(Feat) - Add /bedrock/invoke support for all Anthropic models (#8383)
* use anthropic transformation for bedrock/invoke * use anthropic transforms for bedrock invoke claude * TestBedrockInvokeClaudeJson * add AmazonAnthropicClaudeStreamDecoder * pass bedrock_invoke_provider to make_call * fix _get_base_bedrock_model * fix get_bedrock_route * fix bedrock routing * fixes for bedrock invoke * test_all_model_configs * fix AWSEventStreamDecoder linting * fix code qa * test_bedrock_get_base_model * test_get_model_info_bedrock_models * test_bedrock_base_model_helper * test_bedrock_route_detection |
||
|
Krish Dholakia
|
becd4bc748
|
Litellm dev 01 11 2025 p3 (#7702)
* fix(__init__.py): fix init to exclude pricing-only model cost values from real model names prevents bad health checks on wildcard routes * fix(get_llm_provider.py): fix to handle calling bedrock_converse models |
||
|
Krish Dholakia
|
0120176541
|
Litellm dev 12 30 2024 p2 (#7495)
* test(azure_openai_o1.py): initial commit with testing for azure openai o1 preview model * fix(base_llm_unit_tests.py): handle azure o1 preview response format tests skip as o1 on azure doesn't support tool calling yet * fix: initial commit of azure o1 handler using openai caller simplifies calling + allows fake streaming logic alr. implemented for openai to just work * feat(azure/o1_handler.py): fake o1 streaming for azure o1 models azure does not currently support streaming for o1 * feat(o1_transformation.py): support overriding 'should_fake_stream' on azure/o1 via 'supports_native_streaming' param on model info enables user to toggle on when azure allows o1 streaming without needing to bump versions * style(router.py): remove 'give feedback/get help' messaging when router is used Prevents noisy messaging Closes https://github.com/BerriAI/litellm/issues/5942 * fix(types/utils.py): handle none logprobs Fixes https://github.com/BerriAI/litellm/issues/328 * fix(exception_mapping_utils.py): fix error str unbound error * refactor(azure_ai/): move to openai_like chat completion handler allows for easy swapping of api base url's (e.g. ai.services.com) Fixes https://github.com/BerriAI/litellm/issues/7275 * refactor(azure_ai/): move to base llm http handler * fix(azure_ai/): handle differing api endpoints * fix(azure_ai/): make sure all unit tests are passing * fix: fix linting errors * fix: fix linting errors * fix: fix linting error * fix: fix linting errors * fix(azure_ai/transformation.py): handle extra body param * fix(azure_ai/transformation.py): fix max retries param handling * fix: fix test * test(test_azure_o1.py): fix test * fix(llm_http_handler.py): support handling azure ai unprocessable entity error * fix(llm_http_handler.py): handle sync invalid param error for azure ai * fix(azure_ai/): streaming support with base_llm_http_handler * fix(llm_http_handler.py): working sync stream calls with unprocessable entity handling for azure ai * fix: fix linting errors * fix(llm_http_handler.py): fix linting error * fix(azure_ai/): handle cohere tool call invalid index param error |
||
|
Krish Dholakia
|
0924df4971
|
Litellm dev 12 27 2024 p2 1 (#7449)
* fix(azure_ai/transformation.py): route ai.services.azure calls to the azure provider route requires token to be passed in as 'api-key' Closes https://github.com/BerriAI/litellm/issues/7275 * fix(key_management_endpoints.py): enforce user is member of team, if team_id set and team_id exists in team table * fix(key_management_endpoints.py): handle assigned_user_id = none * feat(create_key_button.tsx): allow assigning keys to other users allows proxy admin to easily assign other people keys * build(create_key_button.tsx): fix error message display don't swallow the error message for key creation failure * build(create_key_button.tsx): allow proxy admin to edit team id * build(create_key_button.tsx): allow proxy admin to assign keys to other users * build(edit_user.tsx): clarify how 'user budgets' are applied * test: remove dup test * fix(key_management_endpoints.py): don't raise error if team not in db ' * test: fix test |
||
|
Krish Dholakia
|
6a45ee1ef7
|
fix(hosted_vllm/transformation.py): return fake api key, if none give… (#7301)
* fix(hosted_vllm/transformation.py): return fake api key, if none give. Prevents httpx error Fixes https://github.com/BerriAI/litellm/issues/7291 * test: fix test * fix(main.py): add hosted_vllm/ support for embeddings endpoint Closes https://github.com/BerriAI/litellm/issues/7290 * docs(vllm.md): add docs on vllm embeddings usage * fix(__init__.py): fix sambanova model test * fix(base_llm_unit_tests.py): skip pydantic obj test if model takes >5s to respond |
||
|
Krish Dholakia
|
e9aa492af3
|
LiteLLM Minor Fixes & Improvement (11/14/2024) (#6730)
* fix(ollama.py): fix get model info request Fixes https://github.com/BerriAI/litellm/issues/6703 * feat(anthropic/chat/transformation.py): support passing user id to anthropic via openai 'user' param * docs(anthropic.md): document all supported openai params for anthropic * test: fix tests * fix: fix tests * feat(jina_ai/): add rerank support Closes https://github.com/BerriAI/litellm/issues/6691 * test: handle service unavailable error * fix(handler.py): refactor together ai rerank call * test: update test to handle overloaded error * test: fix test * Litellm router trace (#6742) * feat(router.py): add trace_id to parent functions - allows tracking retry/fallbacks * feat(router.py): log trace id across retry/fallback logic allows grouping llm logs for the same request * test: fix tests * fix: fix test * fix(transformation.py): only set non-none stop_sequences * Litellm router disable fallbacks (#6743) * bump: version 1.52.6 → 1.52.7 * feat(router.py): enable dynamically disabling fallbacks Allows for enabling/disabling fallbacks per key * feat(litellm_pre_call_utils.py): support setting 'disable_fallbacks' on litellm key * test: fix test * fix(exception_mapping_utils.py): map 'model is overloaded' to internal server error * test: handle gemini error * test: fix test * fix: new run |
||
|
Krish Dholakia
|
f59cb46e71
|
Litellm dev 11 11 2024 (#6693)
* fix(__init__.py): add 'watsonx_text' as mapped llm api route Fixes https://github.com/BerriAI/litellm/issues/6663 * fix(opentelemetry.py): fix passing parallel tool calls to otel Fixes https://github.com/BerriAI/litellm/issues/6677 * refactor(test_opentelemetry_unit_tests.py): create a base set of unit tests for all logging integrations - test for parallel tool call handling reduces bugs in repo * fix(__init__.py): update provider-model mapping to include all known provider-model mappings Fixes https://github.com/BerriAI/litellm/issues/6669 * feat(anthropic): support passing document in llm api call * docs(anthropic.md): add pdf anthropic call to docs + expose new 'supports_pdf_input' function * fix(factory.py): fix linting error |
||
|
Krish Dholakia
|
c03e5da41f
|
LiteLLM Minor Fixes & Improvements (10/24/2024) (#6421)
* fix(utils.py): support passing dynamic api base to validate_environment Returns True if just api base is required and api base is passed * fix(litellm_pre_call_utils.py): feature flag sending client headers to llm api Fixes https://github.com/BerriAI/litellm/issues/6410 * fix(anthropic/chat/transformation.py): return correct error message * fix(http_handler.py): add error response text in places where we expect it * fix(factory.py): handle base case of no non-system messages to bedrock Fixes https://github.com/BerriAI/litellm/issues/6411 * feat(cohere/embed): Support cohere image embeddings Closes https://github.com/BerriAI/litellm/issues/6413 * fix(__init__.py): fix linting error * docs(supported_embedding.md): add image embedding example to docs * feat(cohere/embed): use cohere embedding returned usage for cost calc * build(model_prices_and_context_window.json): add embed-english-v3.0 details (image cost + 'supports_image_input' flag) * fix(cohere_transformation.py): fix linting error * test(test_proxy_server.py): cleanup test * test: cleanup test * fix: fix linting errors |
||
|
Krish Dholakia
|
2b9db05e08
|
feat(proxy_cli.py): add new 'log_config' cli param (#6352)
* feat(proxy_cli.py): add new 'log_config' cli param Allows passing logging.conf to uvicorn on startup * docs(cli.md): add logging conf to uvicorn cli docs * fix(get_llm_provider_logic.py): fix default api base for litellm_proxy Fixes https://github.com/BerriAI/litellm/issues/6332 * feat(openai_like/embedding): Add support for jina ai embeddings Closes https://github.com/BerriAI/litellm/issues/6337 * docs(deploy.md): update entrypoint.sh filepath post-refactor Fixes outdated docs * feat(prometheus.py): emit time_to_first_token metric on prometheus Closes https://github.com/BerriAI/litellm/issues/6334 * fix(prometheus.py): only emit time to first token metric if stream is True enables more accurate ttft usage * test: handle vertex api instability * fix(get_llm_provider_logic.py): fix import * fix(openai.py): fix deepinfra default api base * fix(anthropic/transformation.py): remove anthropic beta header (#6361) |
||
|
Ishaan Jaff
|
ab0b536143
|
(feat) add azure openai cost tracking for prompt caching (#6077)
* add azure o1 models to model cost map * add azure o1 cost tracking * fix azure cost calc * add get llm provider test |
||
|
Krrish Dholakia
|
3560f0ef2c |
refactor: move all testing to top-level of repo
Closes https://github.com/BerriAI/litellm/issues/486 |
Renamed from litellm/tests/test_get_llm_provider.py (Browse further)