03 — llm.py side-by-side: Ollama vs OpenAI

Duration: 15 min Prerequisites: chapter 02 (env ready), having seen Project 1 — chapter 08.

What’s identical (don’t re-read if you did Project 1)

Both files share byte-for-byte the same:

• SQL_TOOL (the JSON-schema dict describing query_sql),
• SYSTEM_PROMPT (rules for the LLM),
• db.safe_query() (the SELECT-only DB layer),
• the shape of the agent loop (ask(question, conn) → {answer, sql_calls}).

This was a deliberate design choice: we want the only surface where the two apps differ to be the 5–6 LLM call sites.

The 4 differences in one table

Line	Ollama (csv-llm-ollama/llm.py)	OpenAI (csv-llm-openai/llm.py)
Client init	client = ollama.Client(host=host)	client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
First call	response = client.chat(model=model, messages=messages, tools=[SQL_TOOL])	response = client.chat.completions.create(model=model, messages=messages, tools=[SQL_TOOL])
Read tool_calls	msg = response["message"]; tcs = msg.get("tool_calls") or []	msg = response.choices[0].message; tcs = msg.tool_calls or []
Append tool result	messages.append({"role":"tool","name":"query_sql","content":out})	messages.append({"role":"tool","tool_call_id":tc.id,"content":out})

That’s the whole diff. Imports + 4 statements.

The full Ollama llm.py (Project 1)

csv-llm-ollama/llm.py

import ollama
import json
import sys
sys.path.insert(0, "../csv-llm-shared")
import db

SQL_TOOL = { ... }            # see Project 1 chapter 08
SYSTEM_PROMPT = """..."""     # same

def ask(question, conn, model="llama3.1:8b", host="http://127.0.0.1:11434"):
    client = ollama.Client(host=host)                                # ← (A)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user",   "content": question},
    ]
    sql_calls = []

    response = client.chat(model=model, messages=messages,           # ← (B)
                           tools=[SQL_TOOL])
    msg = response["message"]                                         # ← (C)
    messages.append(msg)

    for tc in (msg.get("tool_calls") or []):                          # ← (C)
        args = tc["function"]["arguments"]
        if isinstance(args, str):
            args = json.loads(args)
        sql = args["sql"]
        try:
            result_df = db.safe_query(conn, sql)
            tool_content = result_df.to_csv(index=False)
        except ValueError as e:
            tool_content = f"ERROR: {e}"
        sql_calls.append({"sql": sql, "result": tool_content})
        messages.append({                                              # ← (D)
            "role": "tool",
            "name": "query_sql",
            "content": tool_content,
        })

    if sql_calls:
        response = client.chat(model=model, messages=messages,        # ← (B)
                               tools=[SQL_TOOL])

    return {
        "answer":    response["message"]["content"],                  # ← (C)
        "sql_calls": sql_calls,
    }

The full OpenAI llm.py (Project 2)

csv-llm-openai/llm.py

import os
import json
from dotenv import load_dotenv
from openai import OpenAI
import sys
sys.path.insert(0, "../csv-llm-shared")
import db

load_dotenv()

SQL_TOOL = { ... }            # same as Project 1
SYSTEM_PROMPT = """..."""     # same as Project 1

def ask(question, conn, model="gpt-4o-mini", api_key=None):
    client = OpenAI(api_key=api_key or os.getenv("OPENAI_API_KEY"))   # ← (A)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user",   "content": question},
    ]
    sql_calls = []

    response = client.chat.completions.create(model=model,            # ← (B)
                                              messages=messages,
                                              tools=[SQL_TOOL])
    msg = response.choices[0].message                                 # ← (C)
    messages.append(msg.model_dump(exclude_none=True))

    for tc in (msg.tool_calls or []):                                 # ← (C)
        args = json.loads(tc.function.arguments)
        sql = args["sql"]
        try:
            result_df = db.safe_query(conn, sql)
            tool_content = result_df.to_csv(index=False)
        except ValueError as e:
            tool_content = f"ERROR: {e}"
        sql_calls.append({"sql": sql, "result": tool_content})
        messages.append({                                              # ← (D)
            "role": "tool",
            "tool_call_id": tc.id,
            "content": tool_content,
        })

    if sql_calls:
        response = client.chat.completions.create(model=model,        # ← (B)
                                                  messages=messages,
                                                  tools=[SQL_TOOL])

    return {
        "answer":    response.choices[0].message.content,             # ← (C)
        "sql_calls": sql_calls,
    }

The 4 differences explained

(A) Client construction

# Ollama
client = ollama.Client(host="http://127.0.0.1:11434")

# OpenAI
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

Ollama needs a host (where the local server runs). OpenAI needs an API key (where the SaaS auth happens). Both default sensibly so you usually pass nothing.

(B) The chat call

# Ollama
client.chat(model=..., messages=..., tools=[SQL_TOOL])

# OpenAI
client.chat.completions.create(model=..., messages=..., tools=[SQL_TOOL])

Different paths, same arguments. The tools= parameter takes the same JSON-schema dict in both.

(C) Response shape

# Ollama: dict
msg = response["message"]
tool_calls = msg.get("tool_calls") or []
content = response["message"]["content"]

# OpenAI: typed Pydantic-ish object
msg = response.choices[0].message
tool_calls = msg.tool_calls or []
content = response.choices[0].message.content

OpenAI returns a typed object; Ollama returns a plain dict. Both have the same fields, just accessed differently. The choices[0] is because OpenAI can return multiple completions (we ask for one).

(D) Tool result message format

# Ollama
{"role": "tool", "name": "query_sql", "content": tool_content}

# OpenAI
{"role": "tool", "tool_call_id": tc.id, "content": tool_content}

OpenAI requires the tool_call_id to match the call it made. Ollama uses the function name to disambiguate. If you forget the tool_call_id with OpenAI you get a 400 error.

A unifying wrapper?

You could absolutely write a Backend ABC with Ollama and OpenAI subclasses to hide these 4 lines behind a common interface. We deliberately don’t, because the goal of this course is to see the differences. In production, picking one is usually fine.

What about tools parameter shape?

A common question: does the tools=[SQL_TOOL] parameter need to be reshaped between Ollama and OpenAI?

No — both use the same {type: "function", function: {name, description, parameters}} JSON-schema. This is intentional: OpenAI invented the shape, Ollama copied it precisely to stay compatible.

Takeaways

• 4 lines change: client, call, response read, tool result format.
• The system prompt, tool spec, and agent loop shape are identical.
• OpenAI uses a typed object; Ollama uses a dict — that’s the biggest reading shift.
• OpenAI requires tool_call_id; Ollama uses name. Forget it → 400 error.

Next: The OpenAI agent loop with SQL tool →