I’ve noticed that prompts that work perfectly in ChatGPT sometimes fail miserably when I try them in Claude 3 or Llama 2. Is prompt engineering model-specific, or are there universal rules that apply to all Large Language Models? I'm trying to build a tool that is model-agnostic.
3 answers
Prompting is definitely model-specific to an extent! For example, Anthropic's Claude models are known to be much better at following long, XML-formatted instructions. They also tend to be "chattier" and require more specific instructions to stay concise. GPT-4, however, is better at following dense, technical instructions without as much structure. If you want to be model-agnostic, focus on the "Chain of Thought" method and clear delimiters, as these are foundational to the Transformer architecture that both models share. However, for peak performance, you'll always need a little bit of model-specific tuning.
Kimberly, do you think the "XML tagging" style that Claude loves will eventually become the standard for GPT-4 as well, or will they diverge further?
From my tests, Claude handles long-form context much better. If your prompt is over 2,000 words, Claude is usually the winner for accuracy.
That matches my experience too, Megan. Claude's large context window and "needle in a haystack" performance make it great for analyzing huge documents.
Raymond, we are actually already seeing GPT-4 improve its performance when using structured tags! While it doesn't "require" them as much as Claude does, it certainly helps the model parse data more accurately. I think the industry is moving toward a more structured, markdown-heavy prompting style because it's easier for both the AI and the human engineers to read. So, learning that structured style now is a very safe bet for the future.