Welcome back, compression adventurers! In the previous chapters, we laid the groundwork for understanding what OpenZL is and why it’s a game-changer for structured data. We learned that OpenZL isn’t just another generic compressor; it’s a smart framework that wants to understand your data’s shape to compress it more effectively.

But how do we tell OpenZL about our data’s structure? That’s precisely what we’ll uncover in this chapter! We’ll dive into SDDL (Simple Data Description Language), OpenZL’s dedicated language for describing data schemas. Think of SDDL as the blueprint you provide to OpenZL, detailing every room, wall, and window of your data house.

By the end of this chapter, you’ll be able to:

  • Understand the purpose and importance of SDDL in the OpenZL ecosystem.
  • Define basic and complex data types using SDDL syntax.
  • Create your first .sddl file to describe real-world data.
  • Identify common pitfalls and troubleshoot your SDDL definitions.

Ready to draw some blueprints? Let’s get started!

Core Concepts: Speaking Your Data’s Language

OpenZL’s superpower lies in its “format-aware” compression. This means it doesn’t just treat your data as a stream of undifferentiated bytes; it knows if it’s an integer, a floating-point number, a string, or a complex structure like a list of sensor readings. This knowledge allows OpenZL to apply specialized, highly efficient codecs.

What is SDDL?

SDDL, or the Simple Data Description Language, is a domain-specific language (DSL) designed specifically for OpenZL to define the schema of your structured data. It’s concise, human-readable, and focuses purely on data types and their arrangement.

Why is it separate from the data itself? Imagine you have a spreadsheet. The data in the cells is one thing, but the column headers, the data types assigned to each column (e.g., “Date,” “Number,” “Text”), and how rows are structured – that’s the schema. SDDL is how you write down that schema for OpenZL.

Why SDDL Matters for OpenZL

SDDL is the bridge between your raw, structured data and OpenZL’s intelligent compression engine. Without an SDDL definition, OpenZL would have to guess your data’s structure, which defeats the purpose of format-aware compression.

Here’s why it’s so crucial:

  1. Enables Specialized Codecs: Knowing a field is an int32 allows OpenZL to use integer-specific compression algorithms (like delta encoding or variable-byte encoding), which are far more efficient than generic byte-stream compressors.
  2. Improves Compression Ratio: By understanding the data types and relationships, OpenZL can identify redundancies and patterns that a generic compressor would miss.
  3. Boosts Performance: When OpenZL knows what to expect, it can allocate resources and process data much faster during both compression and decompression.
  4. Reduces Ambiguity: It removes any guesswork, ensuring your data is interpreted and compressed exactly as intended.

Consider a simple analogy: You’re trying to pack a suitcase. If you just throw everything in (generic compression), it’s messy and inefficient. But if you have a packing list (SDDL schema) that says “shirts go here, socks there, fragile items in a padded box,” you can pack much more efficiently and protect your items better.

Let’s visualize this flow:

graph TD A[Structured Data] --> B{SDDL Definition File} B -->|Provides Schema| C[OpenZL Framework] C -->|Generates Optimal Plan| D[Specialized Codecs] D --> E[Compressed Output] E -->|\1| F[Original Data]

Figure 4.1: The role of SDDL in the OpenZL compression workflow.

Basic SDDL Syntax: Types and Fields

SDDL uses a syntax similar to many C-family languages, making it familiar to developers. The fundamental building block is the struct keyword, which allows you to define a composite data type made up of various fields.

Each field has a name and a type.

// This is a comment in SDDL
struct MyFirstType {
    // field_name type;
    int32 my_integer_field;
    float64 my_float_field;
}

Let’s break down this tiny snippet:

  • //: Denotes a single-line comment.
  • struct MyFirstType: This declares a new structured data type named MyFirstType. Think of it as defining a template for a record.
  • int32 my_integer_field;: Inside the struct, we define fields. my_integer_field is the name of the field, and int32 is its type. The semicolon ; terminates the field definition.

SDDL Primitive Types

SDDL supports a range of fundamental data types, much like programming languages. These are the “atoms” from which all your complex data structures are built.

Here are some common primitive types:

  • Integers:
    • int8, uint8: Signed/unsigned 8-bit integers.
    • int16, uint16: Signed/unsigned 16-bit integers.
    • int32, uint32: Signed/unsigned 32-bit integers.
    • int64, uint64: Signed/unsigned 64-bit integers.
    • Why so many? To precisely describe the range of your integer data, allowing OpenZL to pick the most efficient representation.
  • Floating-point numbers:
    • float32: Single-precision floating-point.
    • float64: Double-precision floating-point.
  • Boolean:
    • bool: Represents true/false values.
  • Text:
    • string: For variable-length text data.

Composite Types: Arrays and Nested Structs

Real-world data is rarely just a single integer or a float. It often involves collections of items or complex objects nested within each other. SDDL handles this with array and by allowing structs to be nested.

  • Arrays: To define a list or sequence of items of the same type, you use array<Type>.
    struct DataPacket {
    array<uint8> raw_bytes; // An array of 8-bit unsigned integers
    array<float32> sensor_readings; // A list of float readings
}
  • Nested Structs: You can include one struct definition within another, building up complex hierarchies.
    struct Point {
    float32 x;
    float32 y;
}

struct Rectangle {
    Point top_left; // A field of type Point
    Point bottom_right; // Another field of type Point
    string color;
}
    Here, Rectangle contains two Point structs, demonstrating how you can model relationships and complex objects.

Step-by-Step Implementation: Building Your First SDDL File

Let’s put these concepts into practice. We’ll create an SDDL file to describe a common type of structured data: sensor readings from an IoT device.

Scenario: IoT Sensor Data

Imagine an IoT device that periodically sends batches of sensor data. Each individual reading includes a timestamp, temperature, and the sensor’s unique ID. We want to define an SDDL schema for this.

Step 1: Create Your SDDL File for a Single Reading

First, let’s define the structure for a single sensor reading. Create a new file named sensor_data.sddl in your project directory.

// sensor_data.sddl

// Defines the structure for a single sensor reading.
struct SensorReading {
    int64 timestamp;     // Unix timestamp in milliseconds (e.g., 1706284800000)
    float32 temperature; // Temperature in Celsius (e.g., 23.5)
    uint16 sensor_id;   // Unique identifier for the sensor (e.g., 101)
}

Explanation:

  • // sensor_data.sddl: A comment indicating the filename, good practice for clarity.
  • struct SensorReading: We define a new data type called SensorReading.
  • int64 timestamp;: This field will store the time the reading was taken. We use int64 because Unix timestamps can be large numbers, especially when in milliseconds.
  • float32 temperature;: The temperature value. float32 provides sufficient precision for most sensor applications.
  • uint16 sensor_id;: A unique ID for the sensor. uint16 (unsigned 16-bit integer) is often enough for thousands of sensors.

This SensorReading struct is now a blueprint for what a single reading looks like.

Step 2: Add an Array of Readings to Form a Batch

In reality, devices often send multiple readings together in a batch to save network overhead. Let’s update our sensor_data.sddl to reflect this by creating a SensorBatch that contains an array of SensorReadings.

Modify your sensor_data.sddl file to include the new SensorBatch struct:

// sensor_data.sddl

// Defines the structure for a single sensor reading.
struct SensorReading {
    int64 timestamp;     // Unix timestamp in milliseconds (e.g., 1706284800000)
    float32 temperature; // Temperature in Celsius (e.g., 23.5)
    uint16 sensor_id;   // Unique identifier for the sensor (e.g., 101)
}

// Defines a batch of sensor readings, typically sent together.
struct SensorBatch {
    array<SensorReading> readings; // An array holding multiple individual SensorReading structs
    string batch_id;               // A unique identifier for this specific batch (e.g., "batch-20260126-A")
    int64 batch_timestamp;         // When this batch was assembled/sent
}

Explanation:

  • We’ve added a new struct SensorBatch.
  • array<SensorReading> readings;: This is the key part! It tells OpenZL that readings is a collection (an array) where each element conforms to our SensorReading blueprint.
  • string batch_id;: A text field to identify the batch.
  • int64 batch_timestamp;: A timestamp for when the entire batch was created or sent, distinct from individual reading timestamps.

Now, you have a complete SDDL file describing a batch of sensor data. This file can be provided to OpenZL, allowing it to understand and optimize compression based on this precise schema.

Mini-Challenge: Your First SDDL Schema!

It’s your turn to apply what you’ve learned!

Challenge: Define an SDDL schema for a UserProfile. This profile should include:

  1. A unique user_id (a large unsigned integer).
  2. The user’s username (a string).
  3. Their age (a small unsigned integer).
  4. A list of their favorite_colors (an array of strings).
  5. A boolean indicating if they are an is_active user.

Hint: Remember to use appropriate primitive types and the array keyword for collections. Think about the smallest possible types that can still hold the expected data range.

What to observe/learn: This challenge reinforces your understanding of defining structs, choosing appropriate primitive types, and using arrays for collections. It’s a great way to solidify your grasp of SDDL’s core syntax.

(Take a moment to write your user_profile.sddl file before moving on!)

Click for Solution (after you've tried it!)
// user_profile.sddl

// Defines the structure for a user's profile.
struct UserProfile {
    uint64 user_id;             // Unique identifier for the user (e.g., 1234567890123456)
    string username;            // User's chosen username (e.g., "CoderCat")
    uint8 age;                  // User's age (e.g., 30, uint8 is sufficient for ages 0-255)
    array<string> favorite_colors; // A list of colors the user likes (e.g., ["blue", "green"])
    bool is_active;             // True if the user is currently active, false otherwise
}

Common Pitfalls & Troubleshooting

As with any new language, you might encounter a few hiccups when writing SDDL. Here are some common issues and how to tackle them:

  1. Syntax Errors (Missing Semicolons, Misspellings):

    • Mistake: Forgetting a semicolon ; at the end of a field definition, or misspelling a keyword like struct or a type like int32.
    • Debugging Tip: SDDL parsers (which OpenZL will use internally) are usually quite good at reporting the exact line number where an error occurred. Pay close attention to these messages. Double-check your syntax against the examples. SDDL is case-sensitive!

  2. Incorrect Type Selection (Type Mismatches):

    • Mistake: Using a type that’s too small for your data (e.g., uint8 for a sensor_id that can exceed 255) or unnecessarily large (e.g., int64 for a uint8 value).
    • Debugging Tip: Understand the range and precision of each primitive type. If your sensor_id could go up to 100,000, uint16 (max 65,535) won’t cut it; you’d need uint32. While using larger types “just in case” might seem safe, it can reduce compression efficiency. Strive for the smallest type that accurately represents your data.

  3. Over-complex Schemas:

    • Mistake: Trying to model every minute detail or creating deeply nested structures that might not be strictly necessary for compression optimization.
    • Best Practice: Keep your SDDL schemas focused on the data structure that is most relevant for OpenZL to understand and compress. While SDDL can handle complexity, sometimes simplifying the schema can lead to a more robust and easier-to-manage compression pipeline. Consider what specific fields and their types will benefit most from specialized codecs.

Summary

Fantastic work! You’ve successfully taken your first steps into defining data schemas with SDDL, a crucial skill for leveraging OpenZL’s full power.

Here’s what we covered:

  • SDDL is OpenZL’s blueprint language for describing structured data.
  • It enables format-aware compression, leading to better ratios and speeds.
  • You learned to define struct types, which are collections of fields.
  • We explored primitive types like int64, float32, string, and bool.
  • You now know how to create composite types using array<Type> and by nesting structs.
  • We put this into practice by building a sensor_data.sddl file and tackling a mini-challenge.
  • You’re aware of common pitfalls like syntax errors and type mismatches.

In the next chapter, we’ll take these SDDL definitions and learn how OpenZL uses them to create intelligent compression plans. This is where the magic truly happens, as OpenZL maps your data’s blueprint to its powerful suite of codecs!

References

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.