mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-12 08:42:18 +00:00
8 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
norkelbyte
|
5898a42f5a
|
fix(openrouter)!: list OpenRouter's live catalog for openrouter/* when check_provider_endpoint is on (#48820)
* fix(openrouter): list OpenRouter's live catalog for openrouter/* when check_provider_endpoint is on * fix(openrouter): list models off the event loop and keep only chat-callable catalog ids The model-list builders are sync, and with check_provider_endpoint on they call OpenRouter's catalog with a 10 second timeout and no caching of failures. They ran on the event loop inside /v1/models, /model/info, /model_group/info and the per-user listing, so one slow catalog froze every request on the worker. Those calls now run in a worker thread with asyncio.to_thread File, batch and RAG credential lookups only need each name's deployment, and a wildcard that is itself a deployment sits in the list ahead of its expansions and resolves its own credentials. get_complete_model_list takes expand_deployed_wildcard_routes, which that lookup sets to False, so a deployed wildcard never fetches its provider catalog there. A key wildcard wider than any deployment, like anthropic/* over a deployed anthropic/claude-*, still expands so it finds the matching deployment's credentials. Every other caller keeps the default The pattern router's model lookup tolerates a pattern removed between its two reads of the pattern table, which a deployment delete on the event loop can now do while a listing runs in a thread Listing now reads only OpenRouter's models/user catalog. OpenRouter's own /models list carries no embedding models, the OpenAI-format /v1/models has no mode field so chat pickers cannot tell embeddings apart, /model_group/info labels them mode chat, and one failing catalog emptied the whole list. Ids ending in :batch are dropped because chat completions answers 404 for them, while :free and every other variant stay. An api_base that already ends in /chat/completions, which chat accepts as is, has that suffix stripped before the catalog route is joined * fix(proxy): keep model listing off the shared thread pool and out of credential lookups Listings run on their own thread pool so many slow catalogs cannot starve the default executor that chat and Responses calls share. File and batch credential lookups match deployed names against a key's wildcard grants instead of listing any provider catalog. The token counter keeps OpenRouter deployments on their previous path, and a catch-all "*" over provider/* lists bare ids that route back through it * test(proxy): give the off-loop listing tests a generous deadlock guard and typed ids * fix(proxy): keep routed models in wildcard model info, skip key-only OpenRouter catalogs, cache failed listings briefly * fix(utils): cache failed model listings under their own key so a late failure never hides a good listing * fix(proxy): remember only timed-out provider listings so a provider that failed fast is listed again * fix(proxy): ask each provider catalog once per worker, keep deployed names out of catch-all ids, and match file credentials by name Listings that start together for one deployment now wait for the first catalog request. A catch-all wildcard no longer lists a name a deployment serves by name. A key wildcard only finds file and batch credentials on deployments named under it, so a bare name keeps the default credentials as before. * fix(proxy): keep the listing failure log line the same for timed-out listings * refactor(proxy): fold the cached listing lookup back into the provider listing function * fix(proxy): list each model id once and keep a named deployment's row over a wildcard's * fix(proxy): drop a wildcard's model info row when named deployments serve every name it lists --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mika-lt
|
cfb77ecc3a
|
fix(proxy): list only routable models for a wildcard alias over a provider org (#49152)
* fix(proxy): list only routable models for a wildcard alias over a provider org A wildcard alias over an org-scoped deployment (together_ai/* over together_ai/nosuchorg/*) listed the provider's whole catalog on /v1/models and /model/info, and every listed id routed to a model the provider 404s. A skipped deployment path segment now counts as a routing prefix only when the routed model resolves to a catalog id through the get_model_info lookup order, so responses/, Bedrock converse/, and region paths list as before. Together's model list now also carries the cost map's together_ai rows, skipping the together-ai-* pricing tiers, so an alias over a real org lists the models Together serves today. * fix(proxy): keep flat catalog ids under routes the cost map can't parse, drop retired Together rows * fix(proxy): check skipped path segments only on Together so Hugging Face provider routes keep listing --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3c1eb73a06
|
fix(proxy)!: list every priced Vertex family under a vertex_ai/* grant (#41959)
* fix(proxy): list every priced Vertex family under a vertex_ai/* grant A vertex_ai/* key listed none of the 68 cost map rows priced under the Mistral, Llama, Qwen, AI21, OpenAI, image, video and embedding labels because their sets were never part of the vertex_ai union, and Qwen rows had no ingestion branch at all. Wildcard expansion also dropped a meta/ or openai/ publisher segment as if it were a provider prefix, and the wildcard health check would have picked $0-per-token rows first once the union grew, so it now sorts paid rows ahead of unpriced ones. The VERTEX_FAMILIES_A_VERTEX_WILDCARD_GRANT_DOES_NOT_LIST allowlist is gone so the cost map guard enforces every label from now on. * fix(health): route every wildcard health-check pick through the listed provider * fix(health-check): keep per-character-billed rows ahead of rows priced $0 everywhere in the wildcard pick * fix(health-check): break $0-per-token ties toward per-character-billed rows in the wildcard pick * fix(health): only health-check ids the router sends to the provider The wildcard health check picker now skips ids that get_llm_provider routes to a different provider or to none, so the longer vertex_ai union cannot pull partner rows into the check. Drops the forced custom_llm_provider and the per-character tie-break from earlier revisions, and pins the replicate and wandb wildcard listing to each row's full owner id. * fix(health): match wildcard health picks against the provider's own route The previous check kept a bare id only when get_llm_provider routed it to the exact provider name, which dropped every ai21 row because bare ai21 ids route to the ai21_chat alias, so ai21/* health checks sent no request. Compare the bare id's route with the route of the provider-prefixed id instead, so aliases match. Resolve routes with inferred_provider, which returns the declared provider for github_copilot and chatgpt without starting their device-code login. * fix(health): keep wildcard health picks that only route through a saved custom_llm_provider Rank picks routed to the checked provider first and ids routed to no provider second, each by cost, and skip only ids routed to a different provider, so a wildcard saved with custom_llm_provider still sends its health request * refactor(vertex_ai): mark the new Vertex registry sets Final * chore: re-run CI against current main --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mrunmayee Daware
|
2196943195
|
fix(proxy): match partial wildcards against prefix-stripped model ids (#46390)
* fix(proxy): match partial wildcards against prefix-stripped model ids (issue #46384) Refactor model checks to improve filtering logic and type hints. * test(proxy): partial wildcard listing regression coverage (issue #46384) * fix(proxy): drop unneeded annotation on suffix list (LIT001) * fix(proxy): annotate suffix list with reasoned mutable-ok (basedpyright) Change suffix_appended_wildcard_models to a typed list. * Add lock timeout before creating telemetry report table Set local lock timeout to 10 seconds before creating the telemetry report table. * fix(proxy): list alias wildcards as routable ids, not raw catalog ids An alias whose suffix collides with a catalog sub-prefix (e.g. public "cloudflare/@cf/*" over deployment "cloudflare/*") took the partial-filter branch and listed raw catalog ids. Routing substitutes the captured suffix into the deployment pattern, so those listed ids routed to targets that do not exist. Only use filter semantics when the deployment repeats the wildcard suffix (or no deployment is available); aliases prepend their prefix to provider-stripped catalog ids so every listed id routes back to a real catalog model. Addresses Greptile review on #46390. * fix(proxy): derive partial wildcard listings from the deployment's routing pattern List an id for a partial wildcard only when routing it through the deployment's pattern lands on a catalog model, so aliases, catch-alls, and key filters over a wildcard deployment list ids that answer when called. /model/info expands a bare * row the same way /v1/models does. * fix(proxy): drop fragment ids for a catch-all over a partial wildcard and route bare aliases through the deployment pattern A * catch-all over a partial deployment now lists no catalog ids, a bare alias like my-cf/* over cloudflare/@cf/* names ids by the deployment pattern, an alias whose wildcard count differs from its deployment lists nothing, and /model/info rows show the model each listed id routes to * fix(proxy): keep listed ids on /model/info alias rows and clean up test-registered models * fix(proxy): list catalog models through routing-prefix wildcard deployments * fix(proxy): keep namespace and bare catch-all wildcards on routable listings A leading deployment segment the model name repeats, or the provider name itself, is no longer read as a routing segment, so same-name namespace wildcards and a key filter over `*` -> `openai/*` stop listing ids that route to no model. A bare `*` deployment under a prefixed name keeps the legacy listing. `*` over a partial deployment now lists the names that route through it, dropping empty ids. * fix(proxy): list nothing for a catch-all over a mid-word wildcard deployment * fix(proxy): list only wildcard ids that route back to their deployment and catalog model A listed id now has to round-trip through the alias pattern to the catalog model it names, and the router's wildcard match for it has to pick this deployment, so an id a more specific sibling wildcard serves is no longer listed under the catch-all on /v1/models or /model/info * fix(proxy): list wildcard aliases that route at a word boundary and ids a sibling serves with the same model --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
9cf09c18fe
|
fix(providers): route Azure Responses past dated api versions, count Grok reasoning tokens on azure_ai, refresh provider sets after register_model, fix gpt-6.1-sol Azure rows (#46377)
* fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version * fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider * fix(utils): refresh provider model sets after register_model * fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row * test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version * fix(utils): keep router deployment ids out of provider model sets register_model now refreshes the provider sets, so the per-request and router registrations keyed by a deployment id put those opaque ids into wildcard model listings. Those callers now pass add_to_provider_model_sets=False. * fix(azure): carry responses_api_version through the chat to Responses bridge completion() dropped responses_api_version from the litellm_params it hands the Responses bridge, so azure/responses/<model> on a dated chat api_version still built the 404ing /openai/responses URL. Also cover the auto-router marker never forwarding the new param. * fix(azure): stop listing a wrong set of reasoning efforts when none is rejected The message said only low, medium and high are supported, which is wrong for models that also take xhigh such as gpt-6.1-sol. * test(azure): use the latest grok and gpt-5.4 rows in the new azure tests * fix(proxy): list each wildcard-expanded model once A provider set can hold a model both bare and provider-prefixed, which now happens after any priced request registers its openai/<model> key, so the openai/* listing showed that model twice. * fix(azure): route Responses past dated GA api_versions by default A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before * chore(azure_ai): mark the grok chunk_parser dict as an override contract * fix(azure): rewrite only the route's /openai segment when moving a call to v1 The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses. Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too, so the rewrite now only touches the last /openai path segment. The Responses version resolver no longer duplicates the builder's default for an unset api_version. Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite now has one test for the dated GA default and one where only responses_api_version can make the call work * test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params * fix(utils): list ai21 chat and vertex code models under their provider wildcard register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too. * fix(main): keep caller-chosen model names out of provider wildcard listings --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
43f07d0bd8
|
fix(bedrock): list the account's invocable models behind bedrock/* when check_provider_endpoint is on (#43981)
* fix(bedrock): list the account's invocable models behind bedrock/* when check_provider_endpoint is on * fix(bedrock): sign the listing query RFC 3986 style and raise BedrockError on a failed listing * test(utils): keep test_utils.py at main's formatting, only the Bedrock discovery test is new * fix(bedrock): chain BedrockModelInfo's constructor so the passthrough config keeps its event parser * fix(bedrock): list discovered ids without the provider prefix so partial wildcards filter get_valid_models returns bare vendor ids for every other provider and for the static bedrock catalog, and the proxy's wildcard expansion adds the provider prefix itself. The lister prefixed its ids, so a partial wildcard like bedrock/anthropic.* never matched the proxy's filter and listed uncallable bedrock/anthropic.bedrock/<id> entries * test(bedrock): integration cells for wildcard discovery through the proxy and the SDK * fix(proxy): keep a partial wildcard a filter when its deployment repeats the prefix The wildcard expansion guessed filter-or-alias by whether any provider id carried the prefix. A deployment such as model_name bedrock/anthropic.* over model bedrock/anthropic.* can only ever route names that share the prefix, so when the account has no on-demand anthropic.* id (every Anthropic model behind an inference profile) the guess fell into the alias branch and listed bedrock/anthropic.<every id>, none of them callable. A deployment whose model repeats the suffix now always filters, and lists nothing when nothing matches * fix(proxy): prefix every discovered id under a custom wildcard prefix that starts a vendor id * test(bedrock): Final and read-only annotations in the wildcard discovery tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mayuri
|
9506f9e58b
|
fix(proxy): admit litellm_proxy/hosted_vllm in provider-endpoint discovery (#38617)
* fix(proxy): admit litellm_proxy/hosted_vllm in provider-endpoint discovery get_provider_models gated all endpoint discovery on membership in the static litellm.models_by_provider dict, which litellm_proxy and hosted_vllm are intentionally absent from since their model list only exists behind the provider's own endpoint. GET /v1/models returned the literal wildcard string instead of the expanded model list for these providers. Now falls through to ProviderConfigManager, which already knows about them, before giving up. * fix(tests): stop patching an SDK internal in the discovery regression test TQ008 flagged patching litellm.proxy.auth.model_checks.get_valid_models. Assert the gate's own return value instead of mocking past it. * chore: drop redundant comment per repo convention * test(proxy): cover litellm_proxy wildcard model discovery end to end --------- Signed-off-by: mayuriphad <163738104+mayuriphad@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
39e31958f8
|
test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy (#43998)
* test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): stub HIBP through respx by disabling the aiohttp transport Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): share the httpx transport fixture across proxy unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): restore proxy globals without a missing-value sentinel Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): package moved dirs and stub the login breach check at the HTTP boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): isolate the mcp server manager per test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
Renamed from tests/test_litellm/proxy/auth/test_model_checks.py (Browse further)