LLM Providers

Complete reference for configuring all 27 LLM providers in OpenFang

OpenFang supports 27 LLM providers spanning cloud APIs, self-hosted solutions, and specialized coding assistants. Each provider is configured via environment variables for API keys and optional TOML settings for custom endpoints.

Provider List

OpenFang supports these providers out of the box:

Anthropic

Claude Opus/Sonnet/Haiku

OpenAI

GPT-4o, GPT-4, o1, o3

Google

Gemini 2.0 Flash/Pro

Groq

Ultra-fast inference

DeepSeek

Coding specialists

xAI

Grok models

OpenRouter

Multi-provider routing

Mistral

European open models

Together AI

Open source models

Fireworks

Fast open models

Cohere

Command R+

Perplexity

Search-augmented LLMs

AI21 Labs

Jamba models

Cerebras

Fastest inference

SambaNova

Enterprise inference

Hugging Face

70k+ models

Replicate

Cloud model hosting

Ollama

Local models

vLLM

Self-hosted inference

LM Studio

Desktop inference

Qwen

Alibaba models

MiniMax

Chinese frontier

Zhipu AI

GLM models

Moonshot

Kimi long context

Qianfan

Baidu ERNIE

Bedrock

AWS hosted models

GitHub Copilot

Copilot API

Quick Setup

1. Anthropic (Claude)

Environment

export ANTHROPIC_API_KEY="sk-ant-api03-..."

[default_model]
provider = "anthropic"
model = "claude-sonnet-4-20250514"
api_key_env = "ANTHROPIC_API_KEY"

Available Models: claude-opus-4-20250514, claude-sonnet-4-20250514, claude-haiku-4.5-20251001, claude-opus-4.6, claude-sonnet-4.6

API Key: Get from console.anthropic.com

---

2. OpenAI

Environment

export OPENAI_API_KEY="sk-proj-..."

[default_model]
provider = "openai"
model = "gpt-4o"
api_key_env = "OPENAI_API_KEY"

Available Models: gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-4, o1, o1-mini, o3-mini, gpt-3.5-turbo

API Key: Get from platform.openai.com

---

3. Google Gemini

Environment

export GEMINI_API_KEY="AIza..."
# or
export GOOGLE_API_KEY="AIza..."

[default_model]
provider = "gemini"
model = "gemini-2.0-flash-exp"
api_key_env = "GEMINI_API_KEY"

Available Models: gemini-2.0-flash-exp, gemini-2.0-flash-thinking-exp, gemini-1.5-pro, gemini-1.5-flash, gemini-exp-1206

API Key: Get from aistudio.google.com

---

4. Groq

Environment

export GROQ_API_KEY="gsk_..."

[default_model]
provider = "groq"
model = "llama-3.3-70b-versatile"
api_key_env = "GROQ_API_KEY"

Available Models: llama-3.3-70b-versatile, llama-3.1-70b-versatile, llama-3.2-90b-vision-preview, mixtral-8x7b-32768

API Key: Get from console.groq.com

Free Tier: 30 requests/minute, 7,000 requests/day

---

5. DeepSeek

Environment

export DEEPSEEK_API_KEY="sk-..."

[default_model]
provider = "deepseek"
model = "deepseek-chat"
api_key_env = "DEEPSEEK_API_KEY"

Available Models: deepseek-chat, deepseek-reasoner, deepseek-coder

API Key: Get from platform.deepseek.com

---

6. Ollama (Local)

Setup

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull llama3.2

[default_model]
provider = "ollama"
model = "llama3.2:latest"
# No API key needed for local Ollama

# Optional: custom endpoint
[provider_urls]
ollama = "http://gpu-server.local:11434"

No API Key Required: Ollama runs locally on your machine.

Available Models: Any model from ollama.com/library - llama3.2, mistral, codellama, qwen2.5, etc.

---

7. OpenRouter

Environment

export OPENROUTER_API_KEY="sk-or-..."

[default_model]
provider = "openrouter"
model = "anthropic/claude-sonnet-4"
api_key_env = "OPENROUTER_API_KEY"

API Key: Get from openrouter.ai/keys

Benefits: Access 100+ models through a single API, automatic failover, cost optimization

---

8. xAI (Grok)

Environment

export XAI_API_KEY="xai-..."

[default_model]
provider = "xai"
model = "grok-beta"
api_key_env = "XAI_API_KEY"

Available Models: grok-beta, grok-vision-beta

API Key: Get from console.x.ai

---

Chinese Providers

Qwen (Alibaba)

Environment

export DASHSCOPE_API_KEY="sk-..."

[default_model]
provider = "qwen"
model = "qwen-plus"
api_key_env = "DASHSCOPE_API_KEY"

Models: qwen-max, qwen-plus, qwen-turbo, qwen3-235b-a22b, qwen-vl-plus, qwen-coder-plus

---

Zhipu AI (GLM)

Environment

export ZHIPU_API_KEY="..."

[default_model]
provider = "zhipu"
model = "glm-4-plus"
api_key_env = "ZHIPU_API_KEY"

Models: glm-4-plus, glm-4-flash, glm-5-20250605, glm-4v-plus, codegeex-4

---

MiniMax

Environment

export MINIMAX_API_KEY="..."

[default_model]
provider = "minimax"
model = "MiniMax-M2.5"
api_key_env = "MINIMAX_API_KEY"

Models: MiniMax-M2.5, MiniMax-M2.1, minimax-text-01, abab6.5-chat

Context: Up to 1M tokens

---

Moonshot (Kimi)

Environment

export MOONSHOT_API_KEY="sk-..."

[default_model]
provider = "moonshot"
model = "moonshot-v1-128k"
api_key_env = "MOONSHOT_API_KEY"

Models: moonshot-v1-128k, moonshot-v1-32k, moonshot-v1-8k

---

Baidu Qianfan (ERNIE)

Environment

export QIANFAN_API_KEY="..."

[default_model]
provider = "qianfan"
model = "ernie-4.5-8k"
api_key_env = "QIANFAN_API_KEY"

Models: ernie-4.5-8k, ernie-4.0-turbo-8k, ernie-speed-128k

---

Self-Hosted & Local

vLLM

Start vLLM Server

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --port 8000

[default_model]
provider = "vllm"
model = "meta-llama/Llama-3.3-70B-Instruct"

[provider_urls]
vllm = "http://localhost:8000/v1"

No API Key Required for local deployment.

---

LM Studio

config.toml

[default_model]
provider = "lmstudio"
model = "llama-3.3-70b"

[provider_urls]
lmstudio = "http://localhost:1234/v1"

1. Download LM Studio
2. Load a model (e.g., Llama 3.3 70B)
3. Start local server (default port 1234)
4. Configure OpenFang to use it

---

Enterprise & Cloud

AWS Bedrock

Environment

export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_REGION="us-east-1"

[default_model]
provider = "bedrock"
model = "bedrock/anthropic.claude-sonnet-4-6"
api_key_env = "AWS_ACCESS_KEY_ID"

Available Models: bedrock/anthropic.claude-opus-4-6, bedrock/anthropic.claude-sonnet-4-6, bedrock/amazon.nova-pro-v1:0, bedrock/meta.llama3-3-70b-instruct-v1:0

---

GitHub Copilot

Environment

export GITHUB_TOKEN="ghp_..."

[default_model]
provider = "github-copilot"
model = "gpt-4o-copilot"
api_key_env = "GITHUB_TOKEN"

Requires: Active GitHub Copilot subscription

---

Additional Providers

Mistral AI

export MISTRAL_API_KEY="..."

Models: mistral-large, mistral-medium, mistral-small, codestral

---

Together AI

export TOGETHER_API_KEY="..."

Models: Open source models like Llama, Mixtral, Qwen via fast inference

---

Fireworks AI

export FIREWORKS_API_KEY="..."

Models: Ultra-fast inference for Llama, Mixtral, DeepSeek

---

Cohere

export COHERE_API_KEY="..."

Models: command-r-plus, command-r, command

---

Perplexity

export PERPLEXITY_API_KEY="pplx-..."

Models: sonar, sonar-pro - search-augmented responses

---

AI21 Labs

export AI21_API_KEY="..."

Models: jamba-1.5-large, jamba-1.5-mini

---

Cerebras

export CEREBRAS_API_KEY="..."

Models: World’s fastest inference (2,000+ tokens/sec)

---

SambaNova

export SAMBANOVA_API_KEY="..."

Models: Enterprise-grade inference platform

---

Hugging Face

export HF_API_KEY="hf_..."

Models: 70,000+ models via Inference API

---

Replicate

export REPLICATE_API_TOKEN="r8_..."

Models: Cloud-hosted open source models

---

Custom Provider URLs

Override any provider’s base URL:

[provider_urls]
openai = "https://api.openai.com/v1"           # Default
anthropic = "https://my-proxy.com/anthropic"   # Custom proxy
ollama = "http://192.168.1.100:11434"          # Remote Ollama
vllm = "http://gpu-cluster.internal:8000/v1"   # Internal vLLM

Provider Status Check

Check which providers are configured:

# List all providers and their status
openfang providers status

# Test a specific provider
openfang providers test anthropic

# List models from a provider
openfang models list --provider anthropic

Multi-Provider Strategy

Cost-Optimized

[default_model]
provider = "groq"  # Free tier
model = "llama-3.3-70b-versatile"
api_key_env = "GROQ_API_KEY"

[[fallback_providers]]
provider = "ollama"  # Local fallback
model = "llama3.2:latest"

[default_model]
provider = "anthropic"
model = "claude-opus-4-20250514"
api_key_env = "ANTHROPIC_API_KEY"

[[fallback_providers]]
provider = "openai"
model = "gpt-4o"
api_key_env = "OPENAI_API_KEY"

[default_model]
provider = "ollama"
model = "llama3.2:latest"

[[fallback_providers]]
provider = "lmstudio"
model = "mistral-7b"

Troubleshooting

Authentication Errors

# Verify API key is set
echo $ANTHROPIC_API_KEY

# Test provider connection
openfang providers test anthropic

Rate Limits

Configure fallback providers to handle rate limits:

[[fallback_providers]]
provider = "groq"
model = "llama-3.3-70b-versatile"

Model Not Found

# List all models from provider
openfang models list --provider anthropic

# Check if provider is configured
openfang providers status

Next Steps

Model Configuration

Configure model routing and fallbacks

Channel Setup

Connect messaging platforms