For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Learn how to configure Ollama models in kagent.

Configuring Ollama

Ollama allows you to run LLMs locally on your computer or in a Kubernetes cluster. Configuring Ollama in kagent follows the same pattern as for other providers.

Let’s give an example of how to run Ollama on a Kubernetes cluster:

1. Create a namespace for Ollama deployment and service:

kubectl create ns ollama

2. Create the deployment and service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ollama
spec:
  selector:
    matchLabels:
      name: ollama
  template:
    metadata:
      labels:
        name: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - name: http
          containerPort: 11434
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: ollama
spec:
  type: ClusterIP
  selector:
    name: ollama
  ports:
  - port: 80
    name: http
    targetPort: http
    protocol: TCP

You can run kubectl get pod -n ollama and wait until the pod has started.

Once the pod has started, you can port-forward to the Ollama service and use ollama run [model-name] to download/run the model. You can download Ollama binary here.

As kagent relies on calling tools, make sure you’re using a model that allows function calling.

Let’s assume we’ve downloaded the llama3 model, you can then use the following ModelConfig to configure the model:

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: llama3-model-config
  namespace: kagent
spec:
  apiKeySecretKey: OPENAI_API_KEY
  apiKeySecret: kagent-openai
  model: llama3
  provider: Ollama
  ollama:
    host: http://ollama.ollama.svc.cluster.local