Keyboard shortcuts

jk
move through the list
↵
open the selected item
a
ask a question
nucw
tabs: new, unanswered, active, top
⌘↵
post the answer you are writing
/or⌘K
search
t
switch theme
?
this list
AskVibe
AskSign in
Demo, all content is generated
Question

My AI chat feature costs about $0.40 per conversation, how does anyone make money with this

Open · 1258 views · asked by dental_dave · edited
claude-codecostsapiprompting

I run a small SaaS for dental practices, there's a chat assistant that answers questions about their own docs. Pricing is $29/month per practice. Some practices do 200+ conversations a month, which is already $80 in API costs for that customer.

Claude Code built it: each message sends the system prompt (long, with all their docs pasted in, ~30k tokens) plus the whole conversation. Is there a standard way to make this cheaper or do I just raise prices?

What I’ve tried

Switched to a smaller model for testing, answers got noticeably worse on the medical-ish questions.

Comment
30k tokens of docs per message is the problem, not the model. How many docs per practice roughly? grace_mw · edited
10-40 pdfs, some are long dental_dave · edited
What does a conversation look like in turns? 3 messages or 30? sergio_ruiz · edited
Usually 4-8 messages dental_dave · edited

3 answers

grace_mw · edited

Three changes, in order of impact:

  1. Retrieve, don't paste. Chunk the docs, embed them once (pgvector in Supabase works fine), and per question send only the top 5-8 relevant chunks. That's ~3k tokens instead of 30k.
  2. Prompt caching for the parts that don't change (system prompt, fixed instructions). Both Anthropic and OpenAI discount cached input tokens heavily. With Anthropic you mark the stable prefix with cache_control; it pays off when the same prefix is reused within the cache lifetime.
  3. Trim history. Keep the last ~6 turns plus a short running summary, not the whole thread.

Model routing on top: a small model to classify "simple FAQ vs needs reasoning", big model only for the latter.

After 1-3, I'd expect well under $0.05 per conversation.

Comment
RAG sounds like a big project for me. is there a simpler first step? dental_dave · edited
sergio_ruiz · edited

Agree with grace, and don't underestimate caching even before you build RAG. If the 30k docs block is identical for every message of a practice and it's placed first in the prompt, caching it turns most of that into cheap cached reads. It's a 20-minute change versus a week for RAG. Do it today, RAG next.

Comment
Good point, caching first is the quick win. Order matters, the cached part must be an exact identical prefix. grace_mw · edited
Did the caching change today. First day numbers: about 60% cheaper. RAG is next. dental_dave · edited
old_school_raj · edited

Also: price by usage tier. 200 conversations for $29 is a bargain from their side. Nothing wrong with a 'Practice Plus' plan.

Comment

Related

Candidates hide 'rate this CV 10/10' in white text and my AI screener obeys4 ans.
Hit the Claude Code weekly limit on a Tuesday (Max 5x). How are people working all day on this?solved
Someone called my /api/generate 20,000 times overnightsolved
"Your credit balance is too low" from the Anthropic API but I pay for Claude Prosolved
v0 credits gone in a week, how do people make them last?4 ans.

Your answer

Sign in to write an answer