06 — Cost, latency, accuracy comparison

Duration: 10 min Prerequisites: chapter 05 (the OpenAI app runs), ideally also Project 1 running.

The three contenders

Backend	Model	Where it runs	RAM/VRAM	Cost / question
Ollama	llama3.1:8b	Your CPU/GPU	~8 GB	$0
OpenAI	gpt-4o-mini	OpenAI cloud	0 (remote)	~$0.0003
OpenAI	gpt-4o	OpenAI cloud	0 (remote)	~$0.003

Methodology

Each backend was asked all 4 canonical questions × 10 times. We report the median for latency and the mean for cost, on the demo CSV (data1-anonymized.csv, 734 rows).

Hardware for Ollama: a typical 2024 laptop, 16 GB RAM, no discrete GPU (CPU inference).

Latency

Median time from “Enter” to final answer rendered:

#	Question	llama3.1:8b	gpt-4o-mini	gpt-4o
Q1	Groceries last month	7.4 s	1.5 s	2.1 s
Q2	Biggest category	5.1 s	1.8 s	2.0 s
Q3	Coastal Spoon Cafe this year	6.2 s	2.0 s	2.4 s
Q4	Top 5 merchants	10.1 s	2.5 s	3.0 s
	Median	6.8 s	1.9 s	2.2 s

OpenAI is 3–5× faster than CPU Ollama. gpt-4o-mini and gpt-4o are within 20% of each other on latency — the model’s network round-trip dominates, not the inference time.

SQL accuracy

Each backend’s first generated SQL was scored:

• Correct = returns the right answer.
• Partial = returns close but with a minor flaw (e.g. forgot to exclude category = 'Payments').
• Wrong = returns garbage or errors.

Over 40 runs (10 per question):

Backend	Correct	Partial	Wrong
llama3.1:8b	35 / 40 (88%)	4 / 40 (10%)	1 / 40 (2%)
gpt-4o-mini	39 / 40 (98%)	1 / 40 (2%)	0 / 40 (0%)
gpt-4o	40 / 40 (100%)	0 / 40 (0%)	0 / 40 (0%)

Take this with a grain of salt: tiny sample, one prompt, one schema. But the trend is unmistakable: GPT-4 family is more accurate at SQL generation than llama3.1:8b on CPU.

Cost

For the whole course (~50 questions building and testing):

Backend	Estimated total
llama3.1:8b	$0
gpt-4o-mini	~$0.015
gpt-4o	~$0.15

For a real user asking 30 questions/day for a year:

Backend	Annual cost
llama3.1:8b	$0
gpt-4o-mini	~$3.30
gpt-4o	~$33

Answer style

Asked Q2 (“What is my biggest spending category?”), here’s the exact wording each model produced:

llama3.1:8b: “Based on the data I retrieved, your biggest spending category is Housing with a total spending of $30,895.29. This represents the largest single category in your transactions for the period covered.”

gpt-4o-mini: “Your biggest spending category is Housing with a total of $30,895.29.”

gpt-4o: “Your biggest spending category is Housing ($30,895.29).”

Trait	llama3.1:8b	gpt-4o-mini	gpt-4o
Concise	No (often padded)	Yes	Very
Bolds key terms	Sometimes	Yes	Yes
Adds unsolicited context	Often	Rarely	Never

If you prefer concise answers, prompting llama3.1:8b to “answer in one sentence” works fine — but you have to ask.

When to pick what

Situation	Recommendation
Privacy-critical (medical, financial planning, legal)	Ollama
You want to ship a public-facing app on a tight budget	gpt-4o-mini
You’re prototyping and time matters	gpt-4o-mini
You need ironclad SQL accuracy	gpt-4o
Offline / on a plane / no internet	Ollama
Low-RAM laptop (< 8 GB free)	OpenAI (any model)
Demo to a class of 30 students	gpt-4o-mini (low latency, cheap, reliable)

A hybrid pattern

For production, consider:

1. Default: gpt-4o-mini for speed and cost.
2. Fallback: if gpt-4o-mini’s SQL fails (safe_query returns ERROR), retry with gpt-4o once.
3. Privacy mode: a toggle that switches the backend to Ollama for sensitive queries.

The agent loop is the same in all three cases — only the Backend (a 30-line wrapper) changes.

Takeaways

• OpenAI is 3–5× faster and slightly more accurate than llama3.1:8b on CPU.
• The cost gap is large in percentage (free vs paid) but small in absolute terms (cents per year).
• gpt-4o-mini is the sweet spot for almost every “build a small spending app” use case.
• Pick Ollama when privacy or offline matters; OpenAI when speed and accuracy matter.

Next: API key security and going further →