Humans Can't Prompt

In the early days of Google I visited their office in Palo Alto. Back then they had a real-time display of search queries along with a map of the world showing where the searches originated. I’ll never forget, walking into the office and seeing this on the screen:

“My wife says we can only get one dog, which one should we get?”

Now, I’m a fairly technical user. I knew the “right” way to use Google. I prided myself with advanced searches like “orioles -baseball site:nytimes.com filetype:pdf.” I understood how traditional search worked (e.g. keyword lookup in an inverted index), there was no way something like “my wife says we can only get one dog, which one should we get?” would generate a good result. 

Back then, search required the exact word or phrase to appear on a given page; there was zero chance that a web page with that exact phrase existed, and a bajillion pages with those words in different order. It would be so easy for a product manager to write this off as “stupid user doesn’t understand search”. 

But, what stuck with me was that Google did not consider this user error! Instead, they considered it core to their mission to connect this person with useful information. In other words: user behavior should drive product design. NOT the other way around. 

That single idea has been the core of my design philosophy since, and has driven one of the key ways in which Teammates is being built:

Humans Can’t Prompt!

As we onboarded the earliest Teammates customers, we gave them white-glove treatment and were seeing amazing results. But as we started handing the reins to customers and letting them drive, we had a sobering realization: a lot of their success hinged on prompts. And specifically, prompts that we would write with/for them –  dozens of beautifully hand-crafted lines, developed from our AI experience and trial-and-error iteration. 

“It’s easy!” we would say. “Just describe what you want in plain English.” But then we watched customer after customer struggle. We had made waaaay too many assumptions:

1. We assumed people actually knew what they wanted (they don’t)
2. We assumed people would be able to self-identify ambiguities and under-specified requirements in their own writing (they can’t)
3. We assumed people were patient enough to iterate (they aren’t)

It’s not just us – every company and product is integrating AI into their products, the vast majority with a natural language interface. And the quality of the results is tightly coupled to the ability of the user to prompt well – to get inside the “head” of the LLM. Consider this documentation from a popular AI agent: 

Playbooks can immediately unlock [Agent’s] ability to contribute in a wide range of areas, but today require skill to write. Similar to prompt engineering, writing playbooks requires trial and error. The fruit of this labor, though, is a document which unlocks the ability to independently tackle complex work.

Is “prompt engineering” and “trial and error” a reasonable expectation of a customer? We don’t think so. And going further, (for the prompt nerds reading this) learning to add magic phrases like “you are a helpful assistant”, “think step by step”, or delineating data with Markdown and XML <tags> is NOT okay to ask a normal person.

‍

Being able to prompt well is difficult; it requires learning, practice, and frankly, actually giving a shit. Plus, it’s ever changing across different models. Just like it was back when I wanted to search for .pdfs of Orioles stats.

Humans just can’t prompt. But more importantly: they shouldn't have to!

Humans Can’t Prompt… But Computers Can!

At Teammates, in addition to our No Flowcharts design principle, we also have a “No Prompts” design principle. 

Wait what? Aren’t you an AI company? How can you do this without prompts? Look, of course there are prompts under the hood (serverless computing has plenty of servers, after all 😉). But they are no longer something that our customers need to write. Instead, we have built an interface layer and interaction pattern that enables Teammates TO WRITE THEIR OWN PROMPTS.

Think of it like a compiler: software engineers write high level code like Python, then a compiler turns it into machine instructions. Similarly, our customers have interactive 2-way chat experiences with a teammate, and the “prompt compiler” translates the chat (along with their past work history, memory, context, handbooks, and intuition) into a proper prompt.

Let’s see what this looks like in practice. Consider the following set of instructions from an early version of Teammates (which, to be clear, worked well):

‍

Yeah, I couldn’t read the whole thing either, and I wrote it! Now compare this legacy experience to our new experience:

At this point the teammate gets to work, making as much progress as possible. But quickly they are stymied:

TEAMMATE (via Slack) Hi @Ben, I got an “unauthorized” message when querying HubSpot. I don’t think I have access. Can you enable it for me?

My bad. Fixed. Try again.

This process goes on for a while, especially the very first time through while the teammate is “learning” and the user is “teaching”, But at the end of the experience, all the logic for how to query HubSpot for meetings and map custom objects and everything else is encapsulated in a “handbook” for future reference – no “prompt” needed!

‍

A Whole New Way to Think About Software

I’ll be honest – when I laid out the “No Prompts” company mandate, I had no idea if it would work. I just knew the alternatives – flowcharts, massive prompts, and low-code tools wasn’t the future we needed. Not only did it work, but it turns out our Teammates are much better at prompting than we ever were!

‍

‍

‍