13 Comments
User's avatar
Simon Says's avatar

I found the framework in this piece resonates with my journaled and analyzed observations during SF Tech Week, which is building the privileged contexts for customers is the gap that model labs have left for startups to deliver the OUTCOME that customers want from AI.

And the quality of that outcome would accumulate over use cases if the harness is backed by 1. Efficient and accurate RAG, 2. Logs and recursive evals that improve success rate and lower intervention rate, 3. Model routing learned on cost, including failed attempts, and reliability (success rate).

I dive deeper about the implementation in the section, #How to ride it, in my article here:

https://simongydsays.substack.com/p/riding-the-exponential-of-ai-what

Would love to have more input and discussion!

DH's avatar

Your deep dive article/conference report was amazing! Well done.

Simon Says's avatar

Thanks for reading!! Hope there would be more deep dive to follow up 🤞🏽

Mitchell Kosowski's avatar

Counting failed attempts in the cost per completed unit is the part I think most build-vs-buy math misses. Internal prototypes tend to get budgeted on the demo, while retries, evals and human review show up later, so the “cheap” alternative often isn’t.

Nothing Is Accidental's avatar

The four to six hours a week Harvey reports is the number that would settle the pricing question, and the Repsol pilot doesn't say what it's measured against. If some of those hours now go to checking Harvey's edits, the saving is thinner than it reads. Is that figure net of review time?

Vvinit's avatar

On the other hand, from the perspective of a frontier model team, I am curious what framework would they leverage to make a decision when entering a product domain: market size, depth of context building needed, security and governance. What else comes to mind?

Tanya Trushnikova's avatar

The demo is the fun part. Making it work reliably on a random Tuesday afternoon is where the actual business begins.

Abhishek Pan's avatar

That benchmark against the customer's best alternative only means something if it runs on the customer's own documents. A vendor eval built from earlier customers' contracts can score well on clause types the new buyer barely uses, while an in-house prototype is usually built around their templates. Ask for a sample of their real cases before the pilot and score both on it.

Dan Kobina's avatar

I think that the moat will eventually just become what it always has been... I see a future where the strongest AI companies won't necessarily have the smartest models, they'll be the ones that understand their customers' workflows so deeply that replacing them becomes more expensive than keeping them.

Nan Deng's avatar

The premium I would not price as access is the preparation of context the buyer already owns. Policies, CRM records, and past contracts are theirs, but the post's own list of what makes them usable is permissions, normalization, retrieval, and ongoing validation. I would store the accepted, edited, and rejected redline as the version the next review reads, not as a prompt the next session can drop, because a direct frontier-model prompt can reproduce one draft and cannot reproduce which edit was kept. The pricing-benchmark case is the other half: years of figures across about 3,000 companies are worth renting only if a query returns the validated row, not a summary the model is free to smooth.

Rameez MeeraSahib's avatar

Really liked this, especially the point about the customer’s best alternative.

I wrote about the same shift recently from a slightly different angle: what should the customer actually pay for once the software starts doing the work?

My conclusion was that pricing should move closer to completed work, but only as far as attribution remains credible.

A Seat, a Token, or a Job Done? — https://rmeerasahib.substack.com/p/ai-pricing-seat-token-job-done

Raj Maiti's avatar

Verification will be an integral part of the stack

Rogue4Gay's avatar

Your proposal is way too complicated and fragments AI into multiple vendors or products.

Not what I'm looking for in my small business.

Larger enterprises could possibly support but I doubt the IT department would agree with the fragmentation of AI tools into use models.