Configuring Ollama

Ollama allows you to run LLMs locally on your computer or in a Kubernetes cluster. Configuring Ollama in kagent follows the same pattern as for other providers.

Let's give an example of how to run Ollama on a Kubernetes cluster:

1. Create a namespace for Ollama deployment and service:

kubectl create ns ollama

2. Create the deployment and service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ollama
spec:
  selector:
    matchLabels:
      name: ollama
  template:
    metadata:
      labels:
        name: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - name: http
          containerPort: 11434
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: ollama
spec:
  type: ClusterIP
  selector:
    name: ollama
  ports:
  - port: 80
    name: http
    targetPort: http
    protocol: TCP

You can run kubectl get pod -n ollama and wait until the pod has started.

Once the pod has started, you can port-forward to the Ollama service and use ollama run [model-name] to download/run the model. You can download Ollama binary here.

As kagent relies on calling tools, make sure you're using a model that allows function calling.

Let's assume we've downloaded the llama3 model, you can then use the following ModelConfig to configure the model:

apiVersion: kagent.dev/v1alpha1
kind: ModelConfig
metadata:
  name: llama3-model-config
  namespace: kagent
spec:
  apiKeySecretKey: OPENAI_API_KEY
  apiKeySecretRef: kagent-openai
  model: llama3
  provider: Ollama
  ollama:
    host: http://ollama.ollama.svc.cluster.local
  # Required: Specify model capabilities
  modelInfo:
    vision: false
    functionCalling: true
    jsonOutput: true
    family: "unknown"
    structuredOutput: true
    multipleSystemMessages: false

Since Ollama supports various models with different capabilities, you must specify the model's capabilities using the modelInfo field. For detailed information about model capabilities and how to configure them, see Custom Models. Make sure to set functionCalling: true if your model supports it, as this is required for using tools with the agent.