Prompting Claude for RAG

Claude (especially 3.5 Sonnet and Opus) is uniquely suited for RAG because of its high reasoning capability and vast context window. However, to unlock its full potential, you must follow Anthropic's specific prompting best practices.

Use XML Tags

Claude is explicitly trained to use XML tags for structure. This is the most effective way to provide RAG context.

You are a helpful assistant. Use the following context to answer the query.

<context>
  <document id="budget_2024">
    The budget for Q1 is $2.5M.
  </document>
</context>

User: What is the Q1 budget?

The "Put Instructions at the Bottom" Rule

For very long contexts, Claude performs better when the most important instructions appear after the data.

Negative Constraints

Claude is very good at following negative instructions, which is vital for preventing hallucinations.

• "If the answer is not in the context, say 'I cannot find that information' and do not use your internal knowledge."

System Prompt vs. User Message

• System Prompt: Set the persona, the rules of the RAG system, and the formatting requirements.
• User Message: Provide the retrieved context and the final user query.

Multimodal Prompting

When sending images, explain the relationship between the image and the text.

• "The following image is a screenshot of the dashboard mentioned in the text above. Use both to explain the CPU spike."

Implementation Code (Python)

import anthropic

client = anthropic.Anthropic()

message = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1000,
    system="You are a technical support agent. Use the provided XML context only.",
    messages=[
        {"role": "user", "content": f"<context>{formatted_docs}</context>\n\nQuery: {user_query}"}
    ]
)

Exercises

1. Compare a "Native Prompt" (no tags) vs. an "XML Prompt" for a 50-document set. Which one does Claude follow better?
2. Why should you avoid using the phrase "According to the context" in your system prompt? (Hint: It makes the output sound robotic).
3. How would you prompt Claude to output ONLY valid JSON?