mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-19 00:01:29 +00:00
47bba14336
* fix(passthrough): parse Bedrock stream spend incrementally instead of buffering the whole response Bedrock pass-through streaming kept every relayed chunk in memory until EOF and then decoded, parsed and translated the whole stream again for spend logging. Large or concurrent streams could exhaust proxy worker memory. Sync and async passthrough wrappers now hand each chunk to a provider stream collector as it is relayed. Bedrock decodes event-stream frames incrementally, folds consecutive text deltas, and keeps only what stream_chunk_builder needs for usage, tool calls and metadata. Text deltas are no longer retained in the Bedrock and Anthropic stream decoders either. Providers without a collector keep the previous raw-bytes behavior. Collector failures are isolated so spend tracking can never interrupt the customer stream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(passthrough): assert the spend payload the collector builds instead of mock internals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(passthrough): type the Bedrock collector helpers by the collector protocol instead of asserting the class Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| main.py | ||
| README.md | ||
| timeout_utils.py | ||
| utils.py | ||
This makes it easier to pass through requests to the LLM APIs.
E.g. Route to VLLM's /classify endpoint:
SDK (Basic)
import litellm
response = litellm.llm_passthrough_route(
model="hosted_vllm/papluca/xlm-roberta-base-language-detection",
method="POST",
endpoint="classify",
api_base="http://localhost:8090",
api_key=None,
json={
"model": "swapped-for-litellm-model",
"input": "Hello, world!",
}
)
print(response)
SDK (Router)
import asyncio
from litellm import Router
router = Router(
model_list=[
{
"model_name": "roberta-base-language-detection",
"litellm_params": {
"model": "hosted_vllm/papluca/xlm-roberta-base-language-detection",
"api_base": "http://localhost:8090",
}
}
]
)
request_data = {
"model": "roberta-base-language-detection",
"method": "POST",
"endpoint": "classify",
"api_base": "http://localhost:8090",
"api_key": None,
"json": {
"model": "roberta-base-language-detection",
"input": "Hello, world!",
}
}
async def main():
response = await router.allm_passthrough_route(**request_data)
print(response)
if __name__ == "__main__":
asyncio.run(main())
PROXY
- Setup config.yaml
model_list:
- model_name: roberta-base-language-detection
litellm_params:
model: hosted_vllm/papluca/xlm-roberta-base-language-detection
api_base: http://localhost:8090
- Run the proxy
litellm proxy --config config.yaml
# RUNNING on http://localhost:4000
- Use the proxy
curl -X POST http://localhost:4000/vllm/classify \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <your-api-key>" \
-d '{"model": "roberta-base-language-detection", "input": "Hello, world!"}' \
How to add a provider for passthrough
See VLLMModelInfo for an example.
- Inherit from BaseModelInfo
from litellm.llms.base_llm.base_utils import BaseLLMModelInfo
class VLLMModelInfo(BaseLLMModelInfo):
pass
- Register the provider in the ProviderConfigManager.get_provider_model_info
from litellm.utils import ProviderConfigManager
from litellm.types.utils import LlmProviders
provider_config = ProviderConfigManager.get_provider_model_info(
model="my-test-model", provider=LlmProviders.VLLM
)
print(provider_config)