Prompt Compression

When you interact with an AI model, every word you send counts as part of the input — and models have limits on how much input they can process at once (called a context window). Prompt compression tackles this by trimming your input down to only what the model truly needs to produce a good answer.

There are two main approaches:

• Hard compression: Edits the actual text — removing filler words, cutting redundant sentences, or summarizing long passages into shorter ones. The result is still readable by humans.
• Soft compression: Converts text into compact numerical representations (latent vectors) that the model understands but humans can't read.

The goal is always the same: use fewer tokens without losing the meaning that matters.

‍

Understanding this concept matters for three practical reasons. First, fewer tokens mean lower API costs — especially at scale, savings add up fast. Second, shorter inputs lead to faster responses, which improves user experience. Third, it enables you to fit more useful information within a model's context window, making it essential for long conversations, multi-step AI agent workflows, and applications that process large documents.

You can apply this technique whenever your prompts are long, repetitive, or packed with context the model may not fully need. Here's how to get started:

1. Identify redundancy: Look for repeated instructions, overly detailed examples, or background information that doesn't directly serve the task.
2. Summarize context: If you're passing in long documents, extract only the most relevant paragraphs or key facts before sending them to the model.
3. Prune low-value tokens: Remove filler phrases, pleasantries, and formatting that don't affect the output quality.
4. Use tools: Libraries like LLMLingua or Microsoft's prompt compression utilities can automatically score and remove less important tokens for you.

A good rule of thumb: if the compressed version still produces an equally accurate answer, the compression works.

Before compression (87 tokens):

"You are a helpful customer support assistant for an online bookstore. You are friendly, professional, and knowledgeable. Please read the following customer message carefully and provide a detailed, helpful response. The customer says: 'I ordered a book three days ago and the tracking page still shows no updates. I'm worried my order might be lost. Can you help me check the status?'"

After compression (39 tokens):

"Bookstore support assistant. Customer says: 'Ordered a book three days ago, tracking shows no updates, worried order is lost. Help check status?' Respond helpfully."

The compressed version removes redundant instructions and filler while preserving the role, the customer's concern, and the expected action — cutting token usage by over 50% with no loss in response quality.

‍