In this document i attempt to discuss what i believe to be the biggest pain point between new users and AI.
this document is also a collaborative document between myself and GEMINI.
i provide the observations, GEMINI provides the logic and technical terms.
i originally posted it on my blog. but now i am posting it here to invite discussion on its contents
This is an interesting framing.
I like the idea of treating prompt engineering not merely as better wording, but as a protocol for transferring intent between the user and the model.
That seems especially useful for new users, because many failures in AI interaction come not from the model being unable to answer, but from the intent not being represented clearly enough in the exchange.
Framing this as a âprotocolâ also makes the interaction feel less like a trick or technique, and more like a shared structure for thinking with AI.
One small addition I would make is: if the intent is still unclear, the model should ask.
good.thank you
Sorry to say so but I do think coding logic into prompting will end up being a dead end road. Instead of creating a protocol for prompting which sorry again, it is just better wording, we should move into separating logic and reusable knowledge into a different architecture layer, a cognitive runtime, where we can have a clear idea of the which and whys of the process follow for a given result instead of relying on blackboxes for everything. That is the way into a real usable secure deployment of agents in real settings. Check this more detail argument for that Beyond Prompting: Decoupling Cognition from Execution in LLM-based Agents through the ORCA Framework
seperating logic and reueable knowledge alows one to make better prompt based tools. which is great.
the problem with your overall responce is that it assumes that LLMs are just chat boxes, and they are not.
shifting things in to a cognitive runetime is simply a codebase architecture, which whie we are talking about programs, this does not take in to account how agents actually work.
while there is room for backround logic routines agents still need INTRUCTIONS. and these instructions come in the form of conversation. and where there is conversation there is room for intent ambiguity. and there are 2 paths to follow from here
at its very nature, a LLM is a black box. as it breaks down words in to semantic context, and then generates a response out of semantic context. the fact that this works at all is amazing, the fact that it works reliably and consistently is a mundane miracle.
there is only so much that can be done directly with algorithms.
my point here is that the path forward involves many diverging paths, not just 1.
Hi Lance, fair point. I think whatever effor we do to make prompting more clear to avoid missresponses is welcome. At the same time, I would not trust an LLM black box with all its so called hallucintatios (a fancy euphemism for a evidently wrong response) more than I trust a human. We should use LLM ¨thinking¨ capabilities to allow it to orchestrate and operate but ain doing so, we should narrow the decision space enough to get a 99,999% accuracy. I like having a trained pilot driving my plane but I dont want him to fly without following strict procedures that were built on years of experience, no matter how good he seems to be. In the same way we should provide curated procedures so LLMs does not come up with hallucinated solutions that are too far from a safety envelope. There is always room for calling that creativite and use it for our benefit but when thinking on agentic AI driving real processes, we should be more cautious. I would not like a blackbox desinging solutions from first principles every time i ask for a task and hoping the prompt is good enough to avoid miss solutions.
narrowing the decision space is one of the things that my prompt engineering series is focused on. howeverâŚ
in order to understand why 99% accuracy my not be possible, its important to understand where those hallucinations come from.
and the first step to doing that is understanding that there are 2 intended out put vectors for a conversation
definite response is something like if you want an email made, and you already know the target, the subject, the tone, the specified contents, etc.
explored responce is something i deal with more often when you are exploring a topic of conversation and you dont know where it is going to go, like asking a question you dont know the answer to.
most hallucinations seem to come in the second category from my observations.
one might call that evidence, and it is, but i dont do empirical studies, so the evidence is still observational from me.
but in one of my prompt engineering papers i point out that in exploritory conversation one what is known as conversational entropy.
a useful way of understanding and measureing that entropy is imagining that before you start a conversation with an LLM that entropy is at 99%.
one of the goals of prompting at all is narowing the entropy vector.
bad promopting takes dozens of prompts to move from 99% to 70%
most conversations want to be around or under 30%.
the direct outputs happen around 10% for example.
what does all this mean? the AI has access to ALOT of information. that is not a question. however, what it also has access to is every word in the english language (for example) and no built in direction for conversation. that is actually the minimum that the human in the conversation provides. how does this information tie together for this conversation.
when the conversation is fresh, and the entropy is high, the halucinations are more likely to happen, then the halucinations are more likely. as the entropy lowers the likelyhood of halucination.
and now we look at the most prominant reason halucinations happen is due to agents being trained for completion in the short term.
what this looks like is - answer to question
not - correct answer to question
now, in the work i do, i had to develop guardrails, because strict rules turned it in to an execution agent and not a conversation/ exploration agent.
and this matters for me because most my work is either creative (so i only know then intended direction, not necessarily the exact content) or im answering questions that have no exsisting answer (something else that does not have a predefined output). because of this, in most scenarios, a pure execution agent is of no use.
in the back round a pure execution agent can handle handoffs and the like, but for pure exploration (like Protocol of intent) or new problem solving, i need an agent with room for exploration. if if there is no conversational exploration, that cannot happen.
so those guardrails look something like - âdone do this, do that; and here is 3 reasons whyâ
i have been useing this system successfully for months. and not with 1 agent. i have been coordinating single projects across 5 agents, 4 of which have completely different architectures and training regiments.
so it works, and it works for me. does the process i have still have issues? yes, however those issues can be isolated and addressed. which is arguably more important, and what i have been building to on this responce.
knowing that an agent can and will hallucinate is important, but what is of equal importance is being able to identify what type of hallucination it is, so that one can then mitigate those types of halucinations.
a system that fails is almost guaranteed after a certain level of complexity. the real issue is whether it will fail in understandable ways that arent catastrophic.
I think probably a combination of approaches is the best way to goi. Most of the time the exploratory part of the request is mixed with many deterministic ones, or those that are borderline exploratory. Channelling the deterministic ones into proven execution workflows and separating them from the more exploratory ones is probably better. Also, even into the exploratory ones, you can have the need of tracking, meaning you need to undertand why the recommended selected response is produced by the model and in some cases you can also decompose it into certain parts that are more understandable for a human auditing the process. I think even in those cases using a better prompting will aid haveing more robust implementations
âChannelling the deterministic ones into proven execution workflows and separating them from the more exploratory ones is probably better.â
this is actually how i develop my prompt engineering based tools.
any time i have to resolve a complex problem i try and outline a solution, if that works, i record that. and if a similar/ same situation comes up again, i try the previous solution, if that works, that gets adopted as a tool. over time the tools get updated, refined. they slowly become more comprehensive.
these tools eventually become a part of an opperational logic for problem solving or even best practices so better ways of doing things depending on your workflow.
âmeaning you need to undertand why the recommended selected response is produced by the model and in some cases you can also decompose it into certain parts that are more understandable for a human auditing the processâ
it is interesting to observe how responces differ across models for specific complex imputs.
tthis is why i have a series on prompt engineering - to explore these observations, and try to explain them better in a way that isnt loaded with jargon.