mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-08 03:08:45 +00:00
e3f087315d
* feat(terraform): add display_name to litellm_model resource and model data sources
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): persist display_name on update and read /model/info data envelope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(terraform): drop PATCH /model/{model_id}/update from endpoint audit allowlist
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): surface external display_name removal as drift on refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(terraform): rerun after uv download timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
242 lines
10 KiB
Markdown
242 lines
10 KiB
Markdown
# litellm_model Resource
|
|
|
|
Manages a LiteLLM model configuration. This resource allows you to create, update, and delete model configurations in your LiteLLM instance.
|
|
|
|
## Example Usage
|
|
|
|
### Basic OpenAI Model
|
|
|
|
```hcl
|
|
resource "litellm_model" "gpt4" {
|
|
model_name = "gpt-4-proxy"
|
|
custom_llm_provider = "openai"
|
|
model_api_key = var.openai_api_key
|
|
base_model = "gpt-4"
|
|
tier = "paid"
|
|
mode = "chat"
|
|
|
|
input_cost_per_million_tokens = 30.0
|
|
output_cost_per_million_tokens = 60.0
|
|
}
|
|
```
|
|
|
|
### Advanced Model with All Features
|
|
|
|
```hcl
|
|
resource "litellm_model" "advanced_gpt4" {
|
|
model_name = "gpt-4-advanced"
|
|
custom_llm_provider = "openai"
|
|
model_api_key = var.openai_api_key
|
|
model_api_base = "https://api.openai.com/v1"
|
|
api_version = "2023-05-15"
|
|
base_model = "gpt-4"
|
|
tier = "paid"
|
|
team_id = "team-123"
|
|
mode = "chat"
|
|
reasoning_effort = "medium"
|
|
thinking_enabled = true
|
|
thinking_budget_tokens = 1024
|
|
merge_reasoning_content_in_choices = true
|
|
tpm = 100000
|
|
rpm = 1000
|
|
|
|
# Cost configuration (per million tokens)
|
|
input_cost_per_million_tokens = 30.0 # $0.03 per 1k tokens = $30 per million
|
|
output_cost_per_million_tokens = 60.0 # $0.06 per 1k tokens = $60 per million
|
|
}
|
|
```
|
|
|
|
### AWS Bedrock Model with Cross-Account Access
|
|
|
|
```hcl
|
|
resource "litellm_model" "bedrock_claude" {
|
|
model_name = "bedrock-claude-proxy"
|
|
custom_llm_provider = "bedrock"
|
|
base_model = "anthropic.claude-3-sonnet-20240229-v1:0"
|
|
tier = "paid"
|
|
mode = "chat"
|
|
|
|
# AWS configuration with cross-account access
|
|
aws_access_key_id = var.aws_access_key_id
|
|
aws_secret_access_key = var.aws_secret_access_key
|
|
aws_region_name = "us-east-1"
|
|
aws_session_name = "litellm-cross-account-session"
|
|
aws_role_name = "arn:aws:iam::123456789012:role/LiteLLMCrossAccountRole"
|
|
|
|
input_cost_per_million_tokens = 3.0
|
|
output_cost_per_million_tokens = 15.0
|
|
}
|
|
```
|
|
|
|
### Anthropic Model
|
|
|
|
```hcl
|
|
resource "litellm_model" "claude" {
|
|
model_name = "claude-proxy"
|
|
custom_llm_provider = "anthropic"
|
|
model_api_key = var.anthropic_api_key
|
|
base_model = "claude-3-sonnet-20240229"
|
|
tier = "paid"
|
|
mode = "chat"
|
|
|
|
input_cost_per_million_tokens = 3.0
|
|
output_cost_per_million_tokens = 15.0
|
|
}
|
|
```
|
|
|
|
### Azure OpenAI Model
|
|
|
|
```hcl
|
|
resource "litellm_model" "azure_gpt4" {
|
|
model_name = "azure-gpt4-proxy"
|
|
custom_llm_provider = "azure"
|
|
model_api_key = var.azure_openai_key
|
|
model_api_base = var.azure_openai_endpoint
|
|
api_version = "2023-12-01-preview"
|
|
base_model = "gpt-4"
|
|
tier = "paid"
|
|
mode = "chat"
|
|
|
|
input_cost_per_million_tokens = 30.0
|
|
output_cost_per_million_tokens = 60.0
|
|
}
|
|
```
|
|
|
|
## Argument Reference
|
|
|
|
The following arguments are supported:
|
|
|
|
* `model_name` - (Required) string. The name of the model configuration used to identify the model in API calls.
|
|
|
|
* `custom_llm_provider` - (Required) string. The LLM provider for this model (e.g., "openai", "anthropic", "azure", "bedrock").
|
|
|
|
* `model_api_key` - (Optional) string (Sensitive). The API key for the underlying model provider. Sensitive attributes are hidden from Terraform output but still stored in plaintext in the state file; prefer storing provider secrets in a `litellm_credential` and referencing it via `litellm_credential_name`, and secure your state backend.
|
|
|
|
* `model_api_base` - (Optional) string. The base URL for the model provider's API.
|
|
|
|
* `api_version` - (Optional) string. The API version to use for the model provider.
|
|
|
|
* `base_model` - (Required) string. The actual model identifier from the provider (e.g., "gpt-4", "claude-2").
|
|
|
|
* `pricing_base_model` - (Optional) string. A pricing key fed to `model_info.base_model` **independently of routing**. When set, `litellm_params.model` still routes via `base_model`, but LiteLLM looks up cost against this key. Useful when the routing/deployment name differs from the cost-map key — e.g. an Azure deployment routed as `azure/gpt-4.1` whose real tier is Data Zone: set `pricing_base_model = "us/gpt-4.1-2025-04-14"` so it is billed at the Data Zone rate. When unset, `base_model` drives pricing as before.
|
|
|
|
* `litellm_credential_name` - (Optional) string. Name of a LiteLLM credential to use for this model.
|
|
|
|
* `tier` - (Optional) string. The usage tier for this model. Valid values are `"free"` or `"paid"`. Default: `"free"`.
|
|
|
|
* `team_id` - (Optional) string. Associate the model with a specific team.
|
|
|
|
* `display_name` - (Optional) string. Human-readable name stored in `model_info.display_name` and returned as `display_name` by `/v1/models`, so clients such as Claude Code and Claude Desktop show it in their model picker instead of `model_name`. When unset, clients fall back to `model_name`.
|
|
|
|
* `mode` - (Optional) string. The intended use of the model. Valid values are:
|
|
* `completion`
|
|
* `embedding`
|
|
* `image_generation`
|
|
* `chat`
|
|
* `moderation`
|
|
* `audio_transcription`
|
|
* `audio_speech`
|
|
* `rerank`
|
|
|
|
* `tpm` - (Optional) integer. Tokens per minute limit for this model.
|
|
|
|
* `rpm` - (Optional) integer. Requests per minute limit for this model.
|
|
|
|
* `reasoning_effort` - (Optional) string. Configures the model's reasoning effort level. Valid values are:
|
|
* `low`
|
|
* `medium`
|
|
* `high`
|
|
|
|
* `thinking_enabled` - (Optional) boolean. Enables the model's thinking capability. Default: `false`.
|
|
|
|
* `thinking_budget_tokens` - (Optional) integer. Sets the token budget for the model's thinking capability. Default: `1024`. Note: this field is only relevant when `thinking_enabled = true`.
|
|
|
|
* `merge_reasoning_content_in_choices` - (Optional) boolean. When set to `true`, merges reasoning content into the model's choices.
|
|
|
|
* `input_cost_per_million_tokens` - (Optional) float. Cost per million input tokens. The provider converts this to a per-token cost sent to the API.
|
|
|
|
* `output_cost_per_million_tokens` - (Optional) float. Cost per million output tokens. The provider converts this to a per-token cost sent to the API.
|
|
|
|
* `input_cost_per_pixel` - (Optional) float. Cost applied per input pixel for models that charge by image size.
|
|
|
|
* `output_cost_per_pixel` - (Optional) float. Cost applied per output pixel for image-generation models.
|
|
|
|
* `input_cost_per_second` - (Optional) float. Cost applied per input second for audio/transcription models.
|
|
|
|
* `output_cost_per_second` - (Optional) float. Cost applied per output second for audio/transcription models.
|
|
|
|
* `vertex_project` - (Optional) string. Vertex AI project id (for `custom_llm_provider = "vertex"`).
|
|
|
|
* `vertex_location` - (Optional) string. Vertex AI location (e.g., `us-central1`).
|
|
|
|
* `vertex_credentials` - (Optional) string. Vertex credentials (JSON string or path depending on your setup).
|
|
|
|
* `additional_litellm_params` - (Optional) map(string). A map of arbitrary additional parameters that will be merged into the `litellm_params` object sent to the LiteLLM API. This is intended for provider-specific or experimental options not exposed as dedicated arguments.
|
|
|
|
Conversion and behavior rules (how the provider handles values):
|
|
* When values in the map are strings the provider will attempt to coerce them:
|
|
* `"true"` / `"false"` (strings) -> boolean true / false
|
|
* Numeric strings are parsed first as integers; if integer parsing fails, parsed as floats (e.g., `"16384"` -> 16384, `"0.75"` -> 0.75)
|
|
* JSON strings (starting with `[` or `{`) are parsed as JSON objects/arrays
|
|
* Non-convertible strings remain strings
|
|
* Non-string map values (if supplied) are passed through unchanged.
|
|
* The provider merges these keys into the `litellm_params` payload sent to the API.
|
|
* Note: the remote API may not echo back all custom parameters; this provider preserves `additional_litellm_params` in state when present in configuration.
|
|
|
|
**Special parameter: `additional_drop_params`**
|
|
* When `additional_drop_params` is provided as a JSON array string, it specifies parameters to remove from the final `litellm_params` before sending to the API
|
|
* This allows you to override or remove built-in parameters if needed
|
|
* The `additional_drop_params` key itself is not included in the final parameters
|
|
|
|
Example showing booleans, integers, floats, strings, and parameter dropping:
|
|
|
|
```hcl
|
|
resource "litellm_model" "with_additional" {
|
|
model_name = "custom-model"
|
|
custom_llm_provider = "openai"
|
|
model_api_key = var.openai_api_key
|
|
base_model = "gpt-4"
|
|
mode = "chat"
|
|
|
|
additional_litellm_params = {
|
|
"use_fine_tune" = "true" # becomes boolean true
|
|
"max_context" = "16384" # becomes integer 16384
|
|
"scale" = "0.75" # becomes float 0.75
|
|
"note" = "for testing" # stays string
|
|
"complex_config" = "{\"nested\": {\"value\": 42}}" # parsed as JSON object
|
|
"additional_drop_params" = "[\"reasoningEffort\"]" # removes reasoningEffort parameter
|
|
}
|
|
}
|
|
```
|
|
|
|
### AWS-specific Configuration
|
|
|
|
* `aws_access_key_id` - (Optional) string (Sensitive). AWS access key ID for AWS-based models.
|
|
|
|
* `aws_secret_access_key` - (Optional) string (Sensitive). AWS secret access key for AWS-based models. As with `model_api_key`, the value is stored in plaintext in the state file; prefer a `litellm_credential` referenced via `litellm_credential_name` and secure your state backend.
|
|
|
|
* `aws_region_name` - (Optional) string. AWS region name for AWS-based models.
|
|
|
|
* `aws_session_name` - (Optional) string (Sensitive). AWS session name for cross-account access scenarios.
|
|
|
|
* `aws_role_name` - (Optional) string (Sensitive). AWS IAM role name for cross-account access scenarios.
|
|
|
|
## Attribute Reference
|
|
|
|
In addition to the arguments above, the following attributes are exported:
|
|
|
|
* `id` - The ID of the model configuration.
|
|
|
|
## Import
|
|
|
|
Model configurations can be imported using the model ID:
|
|
|
|
```shell
|
|
terraform import litellm_model.gpt4 <model-id>
|
|
```
|
|
|
|
Note: The model ID is generated when the model is created and is different from the `model_name`.
|
|
|
|
## Security Note
|
|
|
|
When using this resource, ensure that sensitive information such as API keys and AWS credentials are stored securely. It's recommended to use environment variables or a secure secret management solution rather than hardcoding these values in your Terraform configuration files.
|