diff --git a/docs/my-website/docs/providers/vertex_partner.md b/docs/my-website/docs/providers/vertex_partner.md index 856f054b8e6..fc2f4d3d8b1 100644 --- a/docs/my-website/docs/providers/vertex_partner.md +++ b/docs/my-website/docs/providers/vertex_partner.md @@ -3,7 +3,7 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Vertex AI - Anthropic, DeepSeek, Model Garden +# Vertex AI - Partner Models ## Supported Partner Providers @@ -16,7 +16,6 @@ import TabItem from '@theme/TabItem'; | AI21 (Jamba) | `vertex_ai/jamba-*` | [Vertex AI - AI21 Models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/ai21) | | Qwen | `vertex_ai/qwen/*` | [Vertex AI - Qwen Models](https://cloud.google.com/vertex-ai/generative-ai/docs/maas/qwen) | | OpenAI (GPT-OSS) | `vertex_ai/openai/gpt-oss-*` | [Vertex AI - GPT-OSS Models](https://console.cloud.google.com/vertex-ai/publishers/openai/model-garden/) | -| Model Garden | `vertex_ai/openai/{MODEL_ID}` or `vertex_ai/{MODEL_ID}` | [Vertex Model Garden](https://cloud.google.com/model-garden?hl=en) | ## Vertex AI - Anthropic (Claude) @@ -793,112 +792,3 @@ curl http://0.0.0.0:4000/v1/chat/completions \ - -## Model Garden - -:::tip - -All OpenAI compatible models from Vertex Model Garden are supported. - -::: - -#### Using Model Garden - -**Almost all Vertex Model Garden models are OpenAI compatible.** - - - - - -| Property | Details | -|----------|---------| -| Provider Route | `vertex_ai/openai/{MODEL_ID}` | -| Vertex Documentation | [Model Garden LiteLLM Inference](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb), [Vertex Model Garden](https://cloud.google.com/model-garden?hl=en) | -| Supported Operations | `/chat/completions`, `/embeddings` | - - - - -```python -from litellm import completion -import os - -## set ENV variables -os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811" -os.environ["VERTEXAI_LOCATION"] = "us-central1" - -response = completion( - model="vertex_ai/openai/", - messages=[{ "content": "Hello, how are you?","role": "user"}] -) -``` - - - - - - -**1. Add to config** - -```yaml -model_list: - - model_name: llama3-1-8b-instruct - litellm_params: - model: vertex_ai/openai/5464397967697903616 - vertex_ai_project: "my-test-project" - vertex_ai_location: "us-east-1" -``` - -**2. Start proxy** - -```bash -litellm --config /path/to/config.yaml - -# RUNNING at http://0.0.0.0:4000 -``` - -**3. Test it!** - -```bash -curl --location 'http://0.0.0.0:4000/chat/completions' \ - --header 'Authorization: Bearer sk-1234' \ - --header 'Content-Type: application/json' \ - --data '{ - "model": "llama3-1-8b-instruct", # 👈 the 'model_name' in config - "messages": [ - { - "role": "user", - "content": "what llm are you" - } - ], - }' -``` - - - - - - - - - - - - -```python -from litellm import completion -import os - -## set ENV variables -os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811" -os.environ["VERTEXAI_LOCATION"] = "us-central1" - -response = completion( - model="vertex_ai/", - messages=[{ "content": "Hello, how are you?","role": "user"}] -) -``` - - - - diff --git a/docs/my-website/docs/providers/vertex_self_deployed.md b/docs/my-website/docs/providers/vertex_self_deployed.md new file mode 100644 index 00000000000..f7e7f181f58 --- /dev/null +++ b/docs/my-website/docs/providers/vertex_self_deployed.md @@ -0,0 +1,180 @@ +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Vertex AI - Self Deployed Models + +Deploy and use your own models on Vertex AI through Model Garden or custom endpoints. + +## Model Garden + +:::tip + +All OpenAI compatible models from Vertex Model Garden are supported. + +::: + +### Using Model Garden + +**Almost all Vertex Model Garden models are OpenAI compatible.** + + + + + +| Property | Details | +|----------|---------| +| Provider Route | `vertex_ai/openai/{MODEL_ID}` | +| Vertex Documentation | [Model Garden LiteLLM Inference](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb), [Vertex Model Garden](https://cloud.google.com/model-garden?hl=en) | +| Supported Operations | `/chat/completions`, `/embeddings` | + + + + +```python +from litellm import completion +import os + +## set ENV variables +os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811" +os.environ["VERTEXAI_LOCATION"] = "us-central1" + +response = completion( + model="vertex_ai/openai/", + messages=[{ "content": "Hello, how are you?","role": "user"}] +) +``` + + + + + + +**1. Add to config** + +```yaml +model_list: + - model_name: llama3-1-8b-instruct + litellm_params: + model: vertex_ai/openai/5464397967697903616 + vertex_ai_project: "my-test-project" + vertex_ai_location: "us-east-1" +``` + +**2. Start proxy** + +```bash +litellm --config /path/to/config.yaml + +# RUNNING at http://0.0.0.0:4000 +``` + +**3. Test it!** + +```bash +curl --location 'http://0.0.0.0:4000/chat/completions' \ + --header 'Authorization: Bearer sk-1234' \ + --header 'Content-Type: application/json' \ + --data '{ + "model": "llama3-1-8b-instruct", # 👈 the 'model_name' in config + "messages": [ + { + "role": "user", + "content": "what llm are you" + } + ], + }' +``` + + + + + + + + + + + + +```python +from litellm import completion +import os + +## set ENV variables +os.environ["VERTEXAI_PROJECT"] = "hardy-device-38811" +os.environ["VERTEXAI_LOCATION"] = "us-central1" + +response = completion( + model="vertex_ai/", + messages=[{ "content": "Hello, how are you?","role": "user"}] +) +``` + + + + + +## Gemma Models (Custom Endpoints) + +Deploy Gemma models on custom Vertex AI prediction endpoints with OpenAI-compatible format. + +| Property | Details | +|----------|---------| +| Provider Route | `vertex_ai/gemma/{MODEL_NAME}` | +| Vertex Documentation | [Vertex AI Prediction](https://cloud.google.com/vertex-ai/docs/predictions/get-predictions) | +| Required Parameter | `api_base` - Full prediction endpoint URL | + +### Usage + + + + +**1. Add to config.yaml** + +```yaml +model_list: + - model_name: gemma-model + litellm_params: + model: vertex_ai/gemma/gemma-3-12b-it-1759525599171 + api_base: https://ENDPOINT.us-central1-PROJECT.prediction.vertexai.goog/v1/projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID:predict + vertex_project: "my-project-id" + vertex_location: "us-central1" +``` + +**2. Start proxy** + +```bash +litellm --config /path/to/config.yaml +``` + +**3. Test it** + +```bash +curl http://0.0.0.0:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-1234" \ + -d '{ + "model": "gemma-model", + "messages": [{"role": "user", "content": "What is machine learning?"}], + "max_tokens": 100 + }' +``` + + + + + +```python +from litellm import completion + +response = completion( + model="vertex_ai/gemma/gemma-3-12b-it-1759525599171", + messages=[{"role": "user", "content": "What is machine learning?"}], + api_base="https://ENDPOINT.us-central1-PROJECT.prediction.vertexai.goog/v1/projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID:predict", + vertex_project="my-project-id", + vertex_location="us-central1", +) +``` + + +