Developing an LLM-Powered Content Summarizer (Hands-on Project)

Introduction: Your First Practical LLM Application!

Welcome to an exciting chapter where we’ll put all your any-llm knowledge into action! So far, we’ve explored the foundations of any-llm, learned how to connect to various providers, handle different output types, and manage asynchronous operations. Now, it’s time to build something tangible and incredibly useful: an LLM-powered content summarizer.

In this chapter, you’ll learn how to design, implement, and refine a Python application that can distill lengthy articles or documents into concise summaries using the any-llm library. This project will solidify your understanding of prompt engineering, API interaction, error handling, and basic application structure. Get ready to transform raw text into digestible insights with the power of large language models!

To get the most out of this chapter, make sure you’re comfortable with:

Installing any-llm-sdk and configuring API keys for your chosen LLM provider (e.g., OpenAI, Mistral, Ollama).
Making basic any_llm.completion calls.
Understanding the difference between synchronous and asynchronous calls.

Let’s dive in and build something awesome!

Core Concepts: Crafting a Smart Summarizer

Before we start coding, let’s understand the core ideas behind building an effective summarizer. It’s more than just asking an LLM to “summarize this.”

1. The Art of Prompt Engineering for Summarization

The quality of your summary hinges almost entirely on your prompt. A well-crafted prompt guides the LLM to produce the desired output. For summarization, we typically want:

Conciseness: Get to the point without unnecessary fluff.
Accuracy: Reflect the original content faithfully.
Coherence: Read naturally and flow well.
Key Information: Capture the most important aspects.

Think of the prompt as a set of instructions you give to a very smart, but sometimes literal, assistant. You need to be clear, specific, and provide context.

2. Handling Input and Output

Our summarizer needs to:

Receive Input: This could be a string of text directly, or perhaps read from a file.
Interact with any-llm: Send the prompt and the content to the LLM.
Process Output: Take the LLM’s response, extract the summary, and present it clearly to the user.

3. The Summarizer’s Workflow

Here’s a simple mental model of our application’s flow. Visualizing it helps us plan our code!

flowchart TD A[Start Application] --> B{Get Content to Summarize}; B -->|User Input/File| C[Prepare Prompt & Content]; C --> D{Call any-llm API}; D -->|Summary Response| E[Extract Summary]; E --> F[Display Summary]; F --> G[End Application];

Figure 13.1: Basic workflow for our LLM-powered content summarizer.

This diagram shows how our application will take content, prepare it, send it to any-llm, and then display the result. Simple, right? Let’s turn this into code!

Step-by-Step Implementation: Building Our Summarizer

We’ll build our summarizer incrementally, adding features and explanations as we go.

Step 1: Initial Setup and Basic Summarization

First, let’s create a new Python file, say summarizer_app.py. We’ll start with the absolute basics: importing any-llm and making a simple summary call.

Action: Create a file named summarizer_app.py and add the following code.

# summarizer_app.py

import os
from any_llm import completion

# --- Configuration ---
# It's good practice to get API keys from environment variables for security.
# Make sure you've set one of these in your environment, e.g.,
# export OPENAI_API_KEY="sk-YOUR_KEY_HERE"
# or export MISTRAL_API_KEY="YOUR_KEY_HERE"
# For local models via Ollama, you might not need an API key,
# but you'd specify 'ollama' as the provider.

# Choose your preferred LLM provider. Examples: "openai", "mistral", "ollama"
# As of late 2025, 'openai' and 'mistral' are popular cloud choices,
# while 'ollama' is excellent for local models.
LLM_PROVIDER = os.environ.get("LLM_PROVIDER", "openai") # Default to openai

# Choose a model that works with your provider.
# For OpenAI: "gpt-4o", "gpt-3.5-turbo"
# For Mistral: "mistral-large-latest", "mistral-medium"
# For Ollama: "llama3", "phi3" (ensure it's downloaded locally)
LLM_MODEL = os.environ.get("LLM_MODEL", "gpt-4o")

# --- Test Content ---
sample_text = """
The quick brown fox jumps over the lazy dog. This is a classic pangram,
a sentence containing every letter of the alphabet at least once.
Pangrams are often used to display typefaces or for testing equipment.
They are a fun linguistic curiosity that demonstrates the versatility
of language.
"""

def get_summary(text: str) -> str:
    """
    Generates a summary for the given text using any-llm.
    """
    # Our first, simple prompt.
    prompt = f"Please summarize the following text concisely:\n\n{text}"

    try:
        response = completion(
            model=LLM_MODEL,
            provider=LLM_PROVIDER,
            prompt=prompt,
            temperature=0.3, # A lower temperature makes the output more deterministic (less creative)
            max_tokens=100,  # Limit the length of the summary
        )
        return response.text
    except Exception as e:
        return f"An error occurred during summarization: {e}"

if __name__ == "__main__":
    print(f"--- Summarizing with {LLM_PROVIDER}/{LLM_MODEL} ---")
    summary = get_summary(sample_text)
    print("\nOriginal Text:")
    print(sample_text)
    print("\n--- Summary ---")
    print(summary)
    print("-----------------\n")

Explanation:

import os and from any_llm import completion: We import necessary modules. os helps us access environment variables for secure API key management.
Configuration (LLM_PROVIDER, LLM_MODEL): We define constants for our chosen LLM provider and model. Notice how we use os.environ.get to allow these to be overridden by environment variables, making our script flexible. Remember to set your API key as an environment variable (e.g., export OPENAI_API_KEY="sk-...") before running this!
sample_text: A short string we’ll use to test our summarizer.
get_summary(text: str) -> str: This function encapsulates our any-llm call.
- prompt: This is our instruction to the LLM. It’s simple for now.
- completion(...): We call the any-llm completion function.
  - model: Specifies which LLM model to use.
  - provider: Tells any-llm which service to connect to (e.g., OpenAI, Mistral).
  - prompt: Our carefully crafted instruction.
  - temperature=0.3: This parameter controls the randomness of the output. Lower values (like 0.3) make the output more focused and less “creative,” which is usually desired for summarization.
  - max_tokens=100: Limits the length of the LLM’s response, helping us keep summaries concise.
- try...except: A basic error handling block to catch any issues during the API call.
if __name__ == "__main__":: This block runs when the script is executed directly, calling our get_summary function and printing the results.

Action: Run your script from the terminal.

# First, set your API key (replace with your actual key and chosen provider)
export OPENAI_API_KEY="sk-..."
# You can also set the provider and model explicitly if you want to override the default
# export LLM_PROVIDER="mistral"
# export LLM_MODEL="mistral-large-latest"

python summarizer_app.py

You should see a concise summary of the sample_text printed to your console. How cool is that? You’ve just built your first LLM-powered application!

Step 2: Making the Summarizer Interactive (Reading from a File)

Hardcoding text is fine for testing, but a real summarizer needs to process external content. Let’s modify our script to read text from a file.

Action:

Create a new text file named article.txt in the same directory as summarizer_app.py.

Paste some longer text into article.txt. For example:

# article.txt
Artificial intelligence (AI) is rapidly transforming various industries,
from healthcare to finance and entertainment. In healthcare, AI
algorithms assist in diagnosing diseases, developing new drugs, and
personalizing treatment plans. For instance, AI can analyze medical
images with high accuracy, often surpassing human capabilities in
detecting subtle anomalies.

In the financial sector, AI is employed for fraud detection, algorithmic
trading, and risk assessment. Machine learning models can identify
suspicious transactions in real-time, protecting consumers and
institutions. The entertainment industry leverages AI for content
recommendation engines, special effects, and even generating music or
scripts. Netflix's recommendation system, for example, is a testament
to AI's power in personalizing user experience.

However, the rapid advancement of AI also brings ethical concerns.
Issues such as data privacy, algorithmic bias, and job displacement
require careful consideration. Regulatory bodies and researchers are
working to establish guidelines and best practices to ensure AI
development is responsible and beneficial to society. The future of AI
promises continued innovation, but also demands a thoughtful approach
to its integration into our daily lives.

Modify your summarizer_app.py to include a new function read_text_from_file and update the if __name__ == "__main__": block.

# summarizer_app.py (Updated)

import os
from any_llm import completion

# ... (Configuration and sample_text remain the same) ...

def read_text_from_file(filepath: str) -> str | None:
    """
    Reads content from a specified file.
    """
    try:
        with open(filepath, 'r', encoding='utf-8') as f:
            return f.read()
    except FileNotFoundError:
        print(f"Error: File not found at {filepath}")
        return None
    except Exception as e:
        print(f"Error reading file {filepath}: {e}")
        return None

# ... (get_summary function remains the same) ...

if __name__ == "__main__":
    print(f"--- Summarizing with {LLM_PROVIDER}/{LLM_MODEL} ---")

    # Let's try summarizing the content from our article.txt
    file_path = "article.txt"
    content_to_summarize = read_text_from_file(file_path)

    if content_to_summarize:
        print(f"\nOriginal Text from '{file_path}':")
        # Print only the first 300 characters to avoid flooding the console
        print(content_to_summarize[:300] + "..." if len(content_to_summarize) > 300 else content_to_summarize)

        print("\n--- Generating Summary ---")
        summary = get_summary(content_to_summarize)
        print("\n--- Summary ---")
        print(summary)
        print("-----------------\n")
    else:
        print("Could not read content to summarize. Exiting.")

Explanation:

read_text_from_file(filepath: str) -> str | None: This new function takes a file path, attempts to open and read it, and returns the content. It includes try...except blocks to handle FileNotFoundError and other potential issues during file reading.
The if __name__ == "__main__": block is updated to:
- Define file_path as “article.txt”.
- Call read_text_from_file to get the content.
- Only proceed with summarization if content_to_summarize is not None.
- We also added a snippet to print only the beginning of the original text to keep the console output manageable.

Action: Run your script again.

python summarizer_app.py

You should now get a summary of the content from article.txt.

Step 3: Enhancing the Prompt for Better Summaries

Our current prompt, “Please summarize the following text concisely,” is a good start, but we can make it much more effective by using a “system” role and more specific instructions. This is where prompt engineering shines!

LLMs often respond better when given a clear role (system message) and then specific instructions for the user’s input.

Action: Modify your get_summary function to use a more sophisticated prompt structure.

# summarizer_app.py (Updated with improved prompt)

import os
from any_llm import completion
from any_llm.types import Message # Import Message type for structured prompts

# ... (Configuration, sample_text, read_text_from_file remain the same) ...

def get_summary(text: str) -> str:
    """
    Generates a summary for the given text using any-llm with a structured prompt.
    """
    # Using a structured list of messages for better prompt engineering
    messages = [
        Message(role="system", content="You are an expert summarizer. Your task is to provide a concise, accurate, and coherent summary of the given text, focusing on the main ideas and key details. The summary should be easy to understand and no longer than 150 words."),
        Message(role="user", content=f"Please summarize the following article:\n\n{text}")
    ]

    try:
        response = completion(
            model=LLM_MODEL,
            provider=LLM_PROVIDER,
            messages=messages, # Use 'messages' instead of 'prompt'
            temperature=0.4, # Slightly higher temperature for a bit more natural language, but still focused
            max_tokens=180,  # Adjust max_tokens to allow for the 150-word instruction + buffer
        )
        return response.text
    except Exception as e:
        return f"An error occurred during summarization: {e}"

# ... (if __name__ == "__main__": block remains the same) ...

Explanation:

from any_llm.types import Message: We import the Message type, which allows us to construct prompts with different roles (system, user, assistant).
messages = [...]: Instead of a single prompt string, we now use a list of Message objects.
- The system message sets the context and persona for the LLM (“You are an expert summarizer…”). This is crucial for guiding its behavior.
- The user message provides the actual content to be summarized and reiterates the task.
- We also added a word count limit (150 words) directly into the system prompt, which is a powerful way to control output length.
messages=messages: We pass this list to the completion function instead of the prompt argument.
temperature and max_tokens are slightly adjusted to accommodate the more detailed instructions and potentially longer desired summary (150 words is more than 100 tokens usually).

Action: Run the script again and compare the summary quality.

python summarizer_app.py

You should observe a more refined and potentially better-structured summary, closer to the instructions given in the system prompt. This highlights the power of good prompt engineering!

Step 4: Adding Basic Command-Line Argument Support

To make our summarizer even more practical, let’s allow users to specify the input file directly from the command line.

Action: Update your summarizer_app.py to use argparse.

# summarizer_app.py (Updated with argparse)

import os
import argparse # Import argparse
from any_llm import completion
from any_llm.types import Message

# ... (Configuration, sample_text, read_text_from_file, get_summary functions remain the same) ...

if __name__ == "__main__":
    # Setup argument parser
    parser = argparse.ArgumentParser(description="An LLM-powered content summarizer using any-llm.")
    parser.add_argument("filepath", type=str,
                        help="Path to the text file to be summarized.")
    parser.add_argument("--provider", type=str, default=LLM_PROVIDER,
                        help=f"LLM provider to use (default: {LLM_PROVIDER}). E.g., openai, mistral, ollama.")
    parser.add_argument("--model", type=str, default=LLM_MODEL,
                        help=f"LLM model to use (default: {LLM_MODEL}). E.g., gpt-4o, mistral-large-latest, llama3.")

    args = parser.parse_args()

    # Override defaults with command-line arguments if provided
    LLM_PROVIDER = args.provider
    LLM_MODEL = args.model

    print(f"--- Summarizing with {LLM_PROVIDER}/{LLM_MODEL} ---")

    content_to_summarize = read_text_from_file(args.filepath)

    if content_to_summarize:
        print(f"\nOriginal Text from '{args.filepath}':")
        print(content_to_summarize[:300] + "..." if len(content_to_summarize) > 300 else content_to_summarize)

        print("\n--- Generating Summary ---")
        summary = get_summary(content_to_summarize)
        print("\n--- Summary ---")
        print(summary)
        print("-----------------\n")
    else:
        print("Could not read content to summarize. Exiting.")

Explanation:

import argparse: This module makes it easy to write user-friendly command-line interfaces.
parser = argparse.ArgumentParser(...): We create an argument parser object.
parser.add_argument(...): We define three arguments:
- filepath: A required positional argument for the file to summarize.
- --provider: An optional argument to specify the LLM provider, with a default.
- --model: An optional argument to specify the LLM model, with a default.
args = parser.parse_args(): This parses the command-line arguments.
LLM_PROVIDER = args.provider and LLM_MODEL = args.model: We update our configuration variables based on user input.
The rest of the logic remains similar, but now uses args.filepath instead of a hardcoded string.

Action: Try running your summarizer with the new command-line interface.

# Summarize article.txt using the default provider/model
python summarizer_app.py article.txt

# Summarize article.txt using Mistral (if configured)
# export MISTRAL_API_KEY="your-mistral-key"
python summarizer_app.py article.txt --provider mistral --model mistral-large-latest

# Summarize article.txt using Ollama (if running locally and model downloaded)
# Ensure Ollama server is running and 'llama3' model is pulled
python summarizer_app.py article.txt --provider ollama --model llama3

This makes our summarizer much more flexible and user-friendly!

Mini-Challenge: Customize Your Summarizer!

You’ve built a solid foundation. Now, let’s make it even better!

Challenge: Modify the get_summary function’s system prompt to produce summaries that are:

Focused on a specific aspect: For example, “Summarize this article focusing only on the ethical implications of AI.”
Formatted in a specific way: E.g., “Provide the summary as a bulleted list of 3 key points.”

Hint: All the changes should primarily happen within the Message(role="system", ...) content string. Experiment with different phrasings to see how the LLM responds! Remember to adjust max_tokens if your desired format requires more space.

What to observe/learn:

How subtle changes in the system prompt can drastically alter the summary’s focus and format.
The importance of clear, unambiguous instructions for LLMs.
The power of any-llm to switch providers and models to see which one performs best for your specific prompting style.

Common Pitfalls & Troubleshooting

Even with any-llm simplifying things, working with LLMs can present a few common hurdles.

API Key Not Set (KeyError or ValueError):
- Problem: You forgot to set the environment variable for your chosen provider’s API key (e.g., OPENAI_API_KEY, MISTRAL_API_KEY). The completion function will likely raise an error because it can’t authenticate.
- Solution: Double-check that you’ve run export YOUR_PROVIDER_API_KEY="sk-..." in your terminal before running the Python script. For a more robust solution in a production environment, consider using a .env file and a library like python-dotenv.
- any-llm’s Role: any-llm tries to auto-detect the provider if an environment variable is found, but it still needs some key for cloud providers.
Token Limit Exceeded (TokenLimitExceeded or similar API error):
- Problem: The input text combined with the prompt is too long for the chosen LLM model’s context window.
- Solution:
  - For summarization, this is common with very long articles. You might need to implement a chunking strategy, where you break the input text into smaller segments, summarize each segment, and then potentially summarize the summaries. This is an advanced topic but crucial for production-grade summarizers.
  - Choose a model with a larger context window (e.g., gpt-4o typically has a larger context than gpt-3.5-turbo).
  - Make your prompt more concise if possible.
- any-llm’s Role: any-llm passes the content to the provider. The token limit is a constraint of the underlying LLM itself.
Generic or Poor Quality Summaries:
- Problem: The summary isn’t specific enough, misses key points, or is simply not what you expected.
- Solution: This is almost always a prompt engineering issue.
  - Be more specific: Instead of “summarize,” try “Summarize the main arguments of this article in three bullet points for a non-technical audience.”
  - Add examples (few-shot prompting): For complex tasks, providing one or two good input-summary pairs in your prompt can dramatically improve results.
  - Iterate: Experiment! Change a few words in the prompt, run it, and see the difference. Keep refining until you get the desired output.
- any-llm’s Role: any-llm faithfully relays your prompt to the LLM. The intelligence of the summary comes from your prompt and the underlying model.

Summary: Your Summarizer is Ready!

Congratulations! You’ve successfully built a functional and flexible content summarizer using Mozilla’s any-llm.

Here are the key takeaways from this chapter:

Practical Application: You moved from theoretical concepts to a real-world, useful application.
Prompt Engineering is King: The quality of your summary is directly proportional to the clarity and specificity of your prompt. System messages are powerful!
Modular Design: Breaking down the application into functions (like read_text_from_file and get_summary) makes it easier to manage and extend.
Flexibility with any-llm: By simply changing configuration variables or command-line arguments, you can switch between different LLM providers and models without altering your core summarization logic.
Robustness: Basic error handling and command-line argument parsing make your application more user-friendly and resilient.

This project is just the beginning. You can expand it by adding features like:

Chunking for very long documents.
Different summarization styles (extractive vs. abstractive).
Output to different formats (JSON, Markdown).
A simple web interface using Flask or FastAPI.

Keep experimenting, keep building, and continue to leverage the power of any-llm to create innovative AI applications!

References

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.

Developing an LLM-Powered Content Summarizer (Hands-on Project)

Table of Contents

Introduction: Your First Practical LLM Application!

Core Concepts: Crafting a Smart Summarizer

1. The Art of Prompt Engineering for Summarization

2. Handling Input and Output

3. The Summarizer’s Workflow

Step-by-Step Implementation: Building Our Summarizer

Step 1: Initial Setup and Basic Summarization

Step 2: Making the Summarizer Interactive (Reading from a File)

Step 3: Enhancing the Prompt for Better Summaries

Step 4: Adding Basic Command-Line Argument Support

Mini-Challenge: Customize Your Summarizer!

Common Pitfalls & Troubleshooting

Summary: Your Summarizer is Ready!

References