Chapter 14: Project 2: Aligning an LLM for Factual Accuracy

Introduction: Guiding LLMs Towards Truth

Welcome back, future LLM alignment expert! In our previous project, we explored fine-tuning an LLM for a specific style. Now, we’re tackling an even more critical challenge: factual accuracy. Large Language Models, despite their incredible capabilities, are notorious for “hallucinating” – generating plausible-sounding but incorrect information. This can severely limit their trustworthiness and utility in many real-world applications.

In this chapter, we’ll embark on a practical project using Tunix to align an LLM to be more factually accurate. We’ll learn how to leverage Tunix’s powerful post-training framework to reduce hallucinations and ensure our models provide reliable information. This project will reinforce your understanding of data preparation, reward modeling, and iterative alignment techniques.

Before we dive in, make sure you’re comfortable with the basics of Tunix, including loading models, handling datasets, and understanding core training loops, as covered in earlier chapters. We’ll build upon that foundation to address the nuanced problem of factual correctness. Let’s make our LLMs more honest!

Core Concepts: Factual Alignment with Tunix

Aligning an LLM for factual accuracy isn’t about teaching it new facts from scratch. It’s about teaching it how to use its existing knowledge responsibly and how to avoid inventing information. This often involves a blend of data curation and reinforcement learning from human feedback (RLHF) or similar preference-based alignment methods.

What is Factual Alignment and Why Does It Matter?

Factual alignment is the process of training an LLM to generate responses that are consistent with verifiable information. It’s crucial because:

Trustworthiness: Users need to trust that the information an LLM provides is correct, especially in critical domains like education, healthcare, or finance.
Reliability: Reduces the risk of spreading misinformation or making decisions based on incorrect data.
Safety: Prevents the generation of harmful or biased false statements.

Imagine an LLM confidently telling you that the capital of France is Madrid. While grammatically perfect, the factual error undermines its utility. Our goal is to minimize such occurrences.

The Role of Tunix in Factual Alignment

Tunix, with its JAX-native efficiency and flexible design, provides the perfect toolkit for this task. We won’t be pre-training a model from scratch; instead, we’ll be post-training an existing LLM. This typically involves:

Supervised Fine-Tuning (SFT) on Factual Data: Initially fine-tuning on a high-quality dataset of (prompt, factually correct response) pairs. This helps the model learn the style of factual responses.
Reward Modeling (RM): Training a separate model (the “reward model”) to judge the factual correctness of an LLM’s output. This model learns to assign higher scores to factually accurate responses and lower scores to inaccurate ones.
Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO): Using the reward model’s feedback (or direct human preferences) to further fine-tune the LLM, guiding it to generate responses that maximize factual accuracy.

For this project, we’ll focus on the reward modeling aspect and then conceptually show how it integrates into an RLHF-like loop within Tunix.

Data is King: Curating Factual Datasets

The quality of your training data is paramount. For factual alignment, you need datasets that contain:

Prompts: Questions or statements requiring factual answers.
Factually Correct Responses: Verifiable answers to those prompts.
Factually Incorrect Responses (for Reward Model): Plausible but false answers, crucial for teaching the reward model what not to reward.

These datasets are often created by human annotators who fact-check and label responses.

Reward Modeling for Factual Accuracy

A reward model is essentially a classifier that takes a prompt and an LLM’s response, then outputs a score indicating its factual correctness.

How does it work?

You provide it with pairs of responses for the same prompt: one factually correct, one incorrect.
It learns to differentiate between them, assigning a higher “reward” to the correct one.
This reward signal then guides the LLM during its alignment phase.

Let’s visualize this process:

flowchart TD A[Start: Base LLM] --> B[Generate Responses to Prompts] B --> C{Human Review & Fact-Checking} C --> D[Create Factual Dataset] D --> E[Train Reward Model] E -->|Provides Factual Score| F[RLHF/DPO Alignment Loop] F -->|Updates LLM Weights| A F --> G[Evaluate Aligned LLM]

This diagram illustrates the iterative nature of factual alignment. The base LLM generates responses, which are then used to build a dataset for training a reward model. The reward model then provides feedback to guide the LLM’s further refinement through an alignment algorithm like RLHF or DPO.

Step-by-Step Implementation: Building a Factual Aligner

For this project, we’ll simplify by focusing on training a reward model that can distinguish factual from non-factual responses. We’ll then discuss how this reward model would be used in a broader RLHF loop.

Our Goal: Train a simple reward model using Tunix that can score a given response for factual accuracy.

Step 1: Setting Up Your Tunix Environment (Quick Recap)

First, ensure your environment is ready. We’ll assume you have JAX and Tunix installed. As of 2026-01-30, the latest stable Tunix release can be installed via pip.

# It's always a good idea to work in a virtual environment
python -m venv tunix_factual_env
source tunix_factual_env/bin/activate

# Install Tunix and JAX with appropriate backend (e.g., CUDA for GPU)
# For JAX with CUDA 12 and Python 3.10, you might use:
pip install "jax[cuda12_pip]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
pip install tunix==0.1.0 # Verify latest stable version on GitHub releases
pip install transformers flax datasets # Other useful libraries

Note: Always check the official Tunix GitHub releases page (https://github.com/google/tunix/releases) for the absolute latest stable version and installation instructions, as library versions evolve rapidly.

Step 2: Preparing a Factual Dataset

For our reward model, we need pairs of (prompt, chosen_response, rejected_response), where chosen_response is factually correct and rejected_response is factually incorrect for the given prompt.

Let’s create a small, synthetic dataset for demonstration purposes. In a real-world scenario, you’d load a much larger, carefully curated dataset.

import jax
import jax.numpy as jnp
from tunix.data import Dataset, DataCollator
from tunix.models import TunixModel
from transformers import AutoTokenizer

# 1. Define our synthetic data
# Each entry is a tuple: (prompt, factually_correct_response, factually_incorrect_response)
factual_data = [
    ("What is the capital of France?", "Paris is the capital of France.", "London is the capital of France."),
    ("Who wrote 'Romeo and Juliet'?", "William Shakespeare wrote 'Romeo and Juliet'.", "Charles Dickens wrote 'Romeo and Juliet'."),
    ("What is the chemical symbol for water?", "The chemical symbol for water is H2O.", "The chemical symbol for water is CO2."),
    ("How many planets are in our solar system?", "There are 8 planets in our solar system.", "There are 9 planets in our solar system."), # Pluto is a dwarf planet now!
]

# Let's choose a small pre-trained model for tokenization
# For a real reward model, you'd use a robust encoder like a BERT or RoBERTa variant.
model_name = "bert-base-uncased" # A good encoder for text classification
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 2. Tokenize the data
def tokenize_function(examples):
    # The reward model takes a prompt and a response, so we concatenate them
    # We'll create two entries per factual_data item:
    # 1. (prompt + chosen)
    # 2. (prompt + rejected)
    
    # For training, we need to compare chosen vs rejected.
    # So we'll tokenize them separately and the loss function will handle the comparison.
    tokenized_chosen = tokenizer(
        examples[0] + " " + examples[1],
        truncation=True,
        padding="max_length",
        max_length=128
    )
    tokenized_rejected = tokenizer(
        examples[0] + " " + examples[2],
        truncation=True,
        padding="max_length",
        max_length=128
    )
    return {
        "chosen_input_ids": tokenized_chosen["input_ids"],
        "chosen_attention_mask": tokenized_chosen["attention_mask"],
        "rejected_input_ids": tokenized_rejected["input_ids"],
        "rejected_attention_mask": tokenized_rejected["attention_mask"],
    }

# Convert our list to a format suitable for processing
# In a real scenario, you'd load from a Hugging Face `datasets.Dataset`
processed_data = [tokenize_function(item) for item in factual_data]

# Let's peek at one tokenized example
print(f"Example chosen input IDs: {processed_data[0]['chosen_input_ids'][:10]}...")
print(f"Example rejected input IDs: {processed_data[0]['rejected_input_ids'][:10]}...")

# Tunix's Dataset class
# We'll create a custom Tunix Dataset and DataCollator for our specific reward model needs.
# For simplicity, we'll treat each item in `processed_data` as a single sample.
class FactualRewardDataset(Dataset):
    def __init__(self, data):
        self.data = data

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return {
            "chosen_input_ids": jnp.array(self.data[idx]["chosen_input_ids"]),
            "chosen_attention_mask": jnp.array(self.data[idx]["chosen_attention_mask"]),
            "rejected_input_ids": jnp.array(self.data[idx]["rejected_input_ids"]),
            "rejected_attention_mask": jnp.array(self.data[idx]["rejected_attention_mask"]),
        }

reward_dataset = FactualRewardDataset(processed_data)

# A simple collator (can be more complex for padding batches)
class FactualRewardCollator(DataCollator):
    def __call__(self, features):
        batch = {}
        batch["chosen_input_ids"] = jnp.stack([f["chosen_input_ids"] for f in features])
        batch["chosen_attention_mask"] = jnp.stack([f["chosen_attention_mask"] for f in features])
        batch["rejected_input_ids"] = jnp.stack([f["rejected_input_ids"] for f in features])
        batch["rejected_attention_mask"] = jnp.stack([f["rejected_attention_mask"] for f in features])
        return batch

data_collator = FactualRewardCollator()

Explanation:

We define factual_data containing prompts, factually correct responses (chosen), and factually incorrect responses (rejected).
We load a bert-base-uncased tokenizer. This model is good for encoding text for classification tasks.
The tokenize_function takes a data entry and tokenizes both the prompt + chosen_response and prompt + rejected_response. We concatenate them because the reward model needs to evaluate the response in the context of the prompt.
We wrap our processed data in a custom FactualRewardDataset and FactualRewardCollator to fit Tunix’s data pipeline. This setup prepares the data for a pairwise comparison loss.

Step 3: Defining and Training a Factual Reward Model

Now, let’s define our reward model. A common approach is to take a pre-trained language model encoder (like BERT) and add a small classification head on top. This head will output a single scalar score.

The reward model is trained using a pairwise ranking loss, such as a Bradley-Terry or sigmoid cross-entropy loss. The goal is to maximize the score difference between the chosen (factual) response and the rejected (non-factual) response.

from flax import linen as nn
from flax.training import train_state
from transformers import FlaxAutoModelForSequenceClassification
import optax

# 1. Define the Reward Model architecture
# We'll use a pre-trained Flax model for sequence classification
# and adapt it to output a single score.
class FactualRewardModel(nn.Module):
    pretrained_model_name: str
    dropout_rate: float = 0.1

    @nn.compact
    def __call__(self, input_ids, attention_mask, train: bool = True):
        # Load the base encoder
        # We need to ensure it outputs hidden states, not just logits
        base_model = FlaxAutoModelForSequenceClassification.from_pretrained(
            self.pretrained_model_name,
            num_labels=1, # We want a single scalar output
            from_pt=True # Often models are pre-trained in PyTorch
        )
        
        # Pass through the base model
        outputs = base_model(
            input_ids=input_ids,
            attention_mask=attention_mask,
            params=self.param("base_model_params"), # Tunix manages parameters
            dropout_rng=nn.make_rng('dropout') if train else None
        )
        
        # The output `logits` from FlaxAutoModelForSequenceClassification with num_labels=1
        # will already be our single scalar score.
        score = outputs.logits.squeeze(-1) # Ensure it's a 1D array of scores
        return score

# 2. Initialize the model and optimizer
key = jax.random.PRNGKey(0)
model_key, dropout_key = jax.random.split(key)

# Dummy input for initialization
dummy_input_ids = jnp.ones((1, 128), dtype=jnp.int32)
dummy_attention_mask = jnp.ones((1, 128), dtype=jnp.int32)

# Instantiate our reward model
reward_model_module = FactualRewardModel(pretrained_model_name=model_name, dropout_rate=0.1)

# Initialize parameters
params = reward_model_module.init(
    {'params': model_key, 'dropout': dropout_key},
    dummy_input_ids,
    dummy_attention_mask,
    train=False # Initialize in eval mode
)['params']

# 3. Define the loss function (Pairwise Ranking Loss)
def reward_loss_fn(chosen_scores, rejected_scores):
    # We want chosen_scores > rejected_scores
    # A common loss is the sigmoid cross-entropy between (chosen - rejected) and 1
    # or a log-sigmoid loss: -log(sigmoid(chosen - rejected))
    difference = chosen_scores - rejected_scores
    loss = -jnp.mean(jax.nn.log_sigmoid(difference))
    return loss

# 4. Define the training step
@jax.jit
def train_step(state, batch, dropout_rng):
    def loss_fn(params):
        chosen_scores = reward_model_module.apply(
            {'params': params, 'dropout': dropout_rng},
            batch["chosen_input_ids"],
            batch["chosen_attention_mask"],
            train=True
        )
        rejected_scores = reward_model_module.apply(
            {'params': params, 'dropout': dropout_rng},
            batch["rejected_input_ids"],
            batch["rejected_attention_mask"],
            train=True
        )
        loss = reward_loss_fn(chosen_scores, rejected_scores)
        return loss, (chosen_scores, rejected_scores)

    grad_fn = jax.value_and_grad(loss_fn, has_aux=True)
    (loss, (chosen_scores, rejected_scores)), grads = grad_fn(state.params)
    state = state.apply_gradients(grads=grads)
    return state, loss, chosen_scores, rejected_scores

# 5. Set up the Tunix Trainer (conceptual)
# Tunix doesn't have a direct 'RewardModelTrainer' out-of-the-box,
# but its flexible `Trainer` class can be customized.
# For simplicity, we'll write a basic training loop here.

# Optimizer
learning_rate = 2e-5
optimizer = optax.adamw(learning_rate)

# Tunix's `TrainState` for managing parameters and optimizer state
state = train_state.TrainState.create(apply_fn=reward_model_module.apply, params=params, tx=optimizer)

print("\n--- Starting Reward Model Training ---")
num_epochs = 5
batch_size = 2 # Small batch size for synthetic data
data_loader = reward_dataset.dataloader(batch_size=batch_size, collate_fn=data_collator, shuffle=True)

for epoch in range(num_epochs):
    epoch_loss = 0
    num_batches = 0
    dropout_rng = jax.random.fold_in(key, epoch) # New dropout key for each epoch

    for batch in data_loader:
        state, loss, chosen_scores, rejected_scores = train_step(state, batch, dropout_rng)
        epoch_loss += loss
        num_batches += 1
    
    avg_epoch_loss = epoch_loss / num_batches
    print(f"Epoch {epoch+1}/{num_epochs}, Loss: {avg_epoch_loss:.4f}")

print("--- Reward Model Training Complete ---")

# 6. Test the trained reward model (inference)
print("\n--- Testing Reward Model ---")
def predict_score(prompt, response):
    tokenized_input = tokenizer(
        prompt + " " + response,
        truncation=True,
        padding="max_length",
        max_length=128,
        return_tensors="jax"
    )
    # Apply the model in evaluation mode
    score = reward_model_module.apply(
        {'params': state.params},
        tokenized_input["input_ids"],
        tokenized_input["attention_mask"],
        train=False
    ).item() # .item() to get scalar from JAX array
    return score

test_prompt = "What is the largest ocean on Earth?"
correct_response = "The Pacific Ocean is the largest ocean on Earth."
incorrect_response = "The Atlantic Ocean is the largest ocean on Earth."
hallucination_response = "The Moon Ocean is the largest ocean on Earth."

score_correct = predict_score(test_prompt, correct_response)
score_incorrect = predict_score(test_prompt, incorrect_response)
score_hallucination = predict_score(test_prompt, hallucination_response)

print(f"Prompt: '{test_prompt}'")
print(f"Score for '{correct_response}': {score_correct:.4f}")
print(f"Score for '{incorrect_response}': {score_incorrect:.4f}")
print(f"Score for '{hallucination_response}': {score_hallucination:.4f}")

# Hopefully, score_correct > score_incorrect and score_correct > score_hallucination!

Explanation of the Code:

FactualRewardModel: We define a Flax nn.Module that wraps FlaxAutoModelForSequenceClassification. We set num_labels=1 because we want a single scalar output (the score). The from_pt=True argument helps load weights from PyTorch checkpoints if the base model was originally trained in PyTorch.
reward_loss_fn: This is the core of our reward model training. It implements a pairwise ranking loss. The jax.nn.log_sigmoid(difference) term ensures that if chosen_scores are higher than rejected_scores, the loss is low, and vice-versa. We aim to minimize this loss.
train_step: This JAX-jitted function performs a single optimization step. It calculates the scores for both chosen and rejected responses, computes the loss, and updates the model parameters using optax.
Training Loop: We set up a simple training loop. In a full Tunix application, you’d integrate this train_step into a tunix.Trainer for more features like logging, checkpointing, and evaluation. For this project, a manual loop demonstrates the core mechanics.
Inference: After training, we define predict_score to see how our reward model scores different responses. We expect it to give higher scores to factually correct statements.

Step 4: Integrating the Reward Model into an Alignment Loop (Conceptual)

Once you have a functional reward model, the next step in a full factual alignment pipeline is to use this model to guide your main LLM. This is typically done through techniques like Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO).

Tunix provides the building blocks for such advanced alignment. Here’s a conceptual outline of how it would work:

# This is conceptual code to illustrate the RLHF/DPO integration.
# It's not executable as a standalone example.

# from tunix.core import Trainer
# from tunix.models import TunixModel
# from tunix.rl import PPOTrainer # or DPOTrainer

# 1. Load the LLM to be aligned (the "policy" model)
# policy_model = TunixModel.from_pretrained("your-llm-model-name")

# 2. Load a "reference" model (a frozen copy of the initial LLM)
# This is used to prevent the policy model from drifting too far from its original capabilities.
# reference_model = TunixModel.from_pretrained("your-llm-model-name", freeze=True)

# 3. Use the trained FactualRewardModel as the reward function
# We would need to wrap our FactualRewardModel into a callable that returns rewards for generated texts.
# def get_factual_rewards(prompts, generated_responses):
#     # Tokenize prompts and responses
#     # Pass through our trained FactualRewardModel
#     # Return scalar reward scores
#     pass # ... implementation using our reward_model_module and state.params ...

# 4. Set up the RLHF/DPO Trainer
# rlhf_trainer = PPOTrainer(
#     policy_model=policy_model,
#     reference_model=reference_model,
#     reward_fn=get_factual_rewards,
#     # ... other PPO/DPO specific parameters like KL divergence penalty ...
# )

# 5. Define the dataset for RLHF/DPO (just prompts)
# rlhf_prompts = ["Tell me about the history of AI.", "Explain quantum entanglement.", ...]
# rlhf_dataset = TunixDataset(rlhf_prompts)

# 6. Start the RLHF/DPO training loop
# rlhf_trainer.train(rlhf_dataset, num_epochs=1)

print("\n--- Conceptual Integration of Reward Model into RLHF/DPO ---")
print("In a full Tunix RLHF/DPO pipeline, your trained FactualRewardModel would act as the `reward_fn`.")
print("The policy LLM would generate responses to prompts, and the reward model would score them.")
print("This score, along with a KL divergence penalty to the reference model, would then be used to update the policy LLM's weights.")
print("This iterative process guides the LLM to generate more factually accurate responses over time.")

Key Idea: The FactualRewardModel we just trained provides a scalar signal. During RLHF, the LLM generates a response, that response is scored by our reward model, and the LLM’s parameters are adjusted via an algorithm (like PPO) to maximize this reward, thereby increasing the likelihood of generating factually accurate text.

Mini-Challenge: Enhance Your Reward Model

You’ve built a foundational factual reward model. Now, let’s challenge you to improve it!

Challenge: Modify the FactualRewardModel to use a different pre-trained encoder from the Hugging Face model hub (e.g., roberta-base or distilbert-base-uncased). Observe if this change impacts the training loss or the final inference scores.

Hint:

Change model_name = "bert-base-uncased" to another suitable encoder model.
Ensure you install the necessary dependencies if you pick a model from a different family (e.g., roberta might need specific tokenizers, though AutoTokenizer usually handles this).
Pay attention to the tokenizer’s behavior for the new model.
You might need to adjust max_length if the new model has different context window limitations.

What to observe/learn: Different pre-trained encoders have varying capacities to understand nuances in text. A more powerful encoder might lead to a more effective reward model, even with our small synthetic dataset. This exercise highlights how the choice of base model impacts the performance of downstream tasks.

Common Pitfalls & Troubleshooting

Aligning LLMs for factual accuracy can be tricky. Here are some common issues you might encounter:

Poor Data Quality: If your chosen_response isn’t truly factual or your rejected_response isn’t genuinely incorrect (or is too obviously wrong), your reward model will learn garbage.
- Troubleshooting: Invest heavily in human annotation and rigorous fact-checking for your dataset. Consider using automated fact-checking tools to assist, but always have human oversight.
Reward Model Collapse: The reward model might stop providing meaningful signals, either always giving high scores or always low scores, regardless of the input.
- Troubleshooting:
  - Monitor Loss: A rapidly decreasing or flat loss might indicate issues.
  - Dataset Balance: Ensure a good mix of clear factual and clear non-factual examples.
  - Hyperparameters: Experiment with learning rates, batch sizes, and dropout.
  - Model Complexity: A too-simple reward model might not capture the nuances of factual correctness.
Over-optimization / Mode Collapse in LLM: If the reward signal is too strong or too narrow, the LLM might learn to “game” the reward model, producing repetitive, overly cautious, or bland responses that score high but aren’t genuinely helpful.
- Troubleshooting (for full RLHF):
  - KL Divergence Penalty: This is crucial in RLHF to prevent the policy model from deviating too much from the reference model. Tunix’s PPOTrainer (or similar) will include this.
  - Diverse Reward Signals: Consider combining factual rewards with other rewards (e.g., helpfulness, harmlessness) if your alignment goal is broader.
Computational Resources: Training reward models and especially running RLHF can be computationally intensive, requiring significant GPU resources.
- Troubleshooting:
  - Smaller Base Models: Start with smaller bert-base-uncased or distilbert-base-uncased for reward models.
  - Batch Size: Optimize batch size to fit your GPU memory.
  - Gradient Accumulation: If memory is tight, use gradient accumulation to simulate larger batch sizes.
  - Efficient Implementations: Tunix’s JAX backend is already highly optimized, but ensure your code is also efficient.

Summary: Building Trustworthy LLMs

Congratulations! You’ve just completed a significant project that lays the groundwork for building more factually accurate LLMs using Tunix.

Here are the key takeaways from this chapter:

Factual alignment is crucial for building trustworthy and reliable LLMs, combating the problem of hallucination.
Data quality is the bedrock of factual alignment, requiring carefully curated datasets of factual and non-factual responses.
We learned to build and train a reward model using Tunix, leveraging a pre-trained encoder and a pairwise ranking loss to distinguish between correct and incorrect information.
We conceptually explored how this reward model integrates into a broader RLHF or DPO pipeline to guide the main LLM’s behavior.
You tackled a mini-challenge to experiment with different base models for your reward model, understanding their impact.
We discussed common pitfalls like data quality issues, reward model collapse, and over-optimization, along with strategies to troubleshoot them.

By mastering these concepts, you’re well on your way to creating more responsible and effective AI systems. In the next chapter, we might delve into even more advanced alignment techniques or explore deployment strategies for your fine-tuned Tunix models. Keep experimenting, keep learning!

References

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.