Chapter 6: Getting Data Ready: Basic Data Manipulation in Python

Introduction: Shaping the Raw Material

Welcome back, future AI explorer! In our previous chapters, we’ve journeyed through the fascinating world of AI and Machine Learning, understanding the core concepts of how machines “learn” and why data is their lifeblood. We also took our first exciting steps into Python programming, learning about variables, data types, and basic operations. You’re doing great!

Now, it’s time to get our hands a little dirty (in a good way!) with that precious data. Imagine you’re a chef, and you’ve just received a basket full of fresh ingredients. Before you can cook a delicious meal, you need to wash, peel, chop, and prepare everything, right? Data is no different. Raw data, straight from its source, is rarely in the perfect shape for a machine learning model. It might have missing pieces, incorrect values, or be organized in a way that’s hard for our algorithms to understand.

This chapter is all about becoming a data preparation pro. We’ll learn how to use a super popular and powerful Python library called Pandas to inspect, clean, and transform our data. Think of Pandas as your ultimate kitchen knife set for data – it makes slicing, dicing, and arranging data surprisingly easy and fun. By the end of this chapter, you’ll have the foundational skills to take raw data and start shaping it into something a machine learning model can truly learn from. Let’s get started!

Core Concepts: Why Data Needs a Makeover

Before we dive into the code, let’s understand why data manipulation is so critical.

What is Data Manipulation? The Chef’s Analogy

Data manipulation is the process of cleaning, transforming, and organizing raw data into a more suitable format for analysis or machine learning.

Think of it this way:

You want to bake a cake, but all you have are whole eggs, a block of butter, a sack of flour, and a bag of sugar. You can’t just throw them all in a bowl and expect a cake!

• You need to crack the eggs.
• Melt the butter.
• Measure the flour and sugar precisely.
• Mix them in the right order.

In the world of AI, our “ingredients” are data points. Our “cake” is a machine learning model that makes accurate predictions. If our data isn’t prepared correctly:

• Missing values are like missing ingredients – the recipe won’t work.
• Incorrect data types (e.g., text where numbers are expected) are like using salt instead of sugar.
• Irrelevant information is like adding random dirt to your batter.

All these issues lead to a “bad cake” – a machine learning model that performs poorly, makes inaccurate predictions, or simply refuses to run. Data manipulation ensures our model gets the best possible “ingredients.”

Meet Pandas: Your Data Sidekick

To handle data effectively in Python, especially tabular data (like spreadsheets), we use a library called Pandas.

• What is it? Pandas is an open-source Python library that provides high-performance, easy-to-use data structures and data analysis tools. It’s built on top of another library called NumPy (which you might explore later), making it very efficient.
• Why use it? Pandas makes working with complex datasets feel intuitive. It allows you to load data from various sources (CSV, Excel, databases), clean it, transform it, and prepare it for more advanced analysis or machine learning tasks with just a few lines of code.

The Pandas DataFrame: Your Data Table

The most important data structure in Pandas is the DataFrame.

Imagine a DataFrame as a super-powered spreadsheet or a table in a database.

It has:

• Rows: Each row usually represents a single observation or record (e.g., one customer, one house, one sensor reading).
• Columns: Each column represents a specific attribute or “feature” of those observations (e.g., customer name, house price, sensor temperature).
• Index: Each row also has a unique label, called an index, which helps you quickly find specific rows. By default, this is just a sequence of numbers (0, 1, 2…).

Here’s a mental picture of a DataFrame:

See? Just like a table! This structure is incredibly common in data science and machine learning.

The Pandas Series: A Single Column

While a DataFrame is a 2-dimensional table, a Series is a 1-dimensional array.

Think of a Series as a single column from your DataFrame.

It’s essentially a list of values, but with an associated index. If you pick out the ‘Age’ column from our example DataFrame, that’s a Series:

Understanding DataFrames and Series is key, as most of our data manipulation will involve working with these structures.

Step-by-Step Implementation: Getting Hands-On with Pandas

Alright, time to roll up our sleeves and write some code! We’ll go step-by-step, building our understanding incrementally.

Step 1: Setting Up Pandas

First, we need to make sure Pandas is installed in your Python environment. If you’re using Anaconda or Miniconda (which we recommended in Chapter 4 for setting up your environment), Pandas usually comes pre-installed. If not, or if you’re using a plain Python installation, you can install it using pip (Python’s package installer).

As of January 2026, Pandas version 2.2.0 is a stable and widely used release.

Open your terminal or command prompt (or a new cell in your Jupyter Notebook/VS Code) and run:

pip install pandas==2.2.0

This command tells pip to install the Pandas library, specifically requesting version 2.2.0. If you already have it, it might just confirm it’s up to date.

Once installed, we need to import it into our Python script or notebook so we can use its functions. It’s a common convention to import Pandas as pd.

import pandas as pd
print(f"Pandas version: {pd.__version__}")

Explanation:

• import pandas as pd: This line brings the Pandas library into our current Python session. We use as pd to create a shorter alias, pd, so we don’t have to type pandas. every time we use a function from the library. This is standard practice and saves a lot of typing!
• print(f"Pandas version: {pd.__version__}"): This is a little check to confirm which version of Pandas you’re using. It’s always good to be aware of your tool versions!

Step 2: Creating Your First DataFrame

Let’s create a simple DataFrame from scratch using a Python dictionary. Each key in the dictionary will become a column name, and its value will be a list of data for that column.

# Create a dictionary representing our data
data = {
    'Name': ['Alice', 'Bob', 'Clara', 'David', 'Eve'],
    'Age': [24, 30, 28, 35, 22],
    'City': ['New York', 'London', 'Paris', 'Tokyo', 'Berlin'],
    'Score': [85, 92, 78, 95, 88]
}

# Convert the dictionary into a Pandas DataFrame
df = pd.DataFrame(data)

# Let's see our new DataFrame!
print(df)

Explanation:

1. data = {...}: We define a standard Python dictionary.
   • 'Name', 'Age', 'City', 'Score' are the keys of the dictionary. These will become the column names in our DataFrame.
   • The values associated with each key are Python lists. Each list contains the data for that specific column. Notice how each list has the same number of items (5 in this case) – this is crucial for creating a rectangular table.
2. df = pd.DataFrame(data): This is the magic line! We call the DataFrame() constructor from the pd (Pandas) library and pass our data dictionary to it. Pandas automatically understands that the dictionary keys are column names and the lists are column values, assembling them into a DataFrame.
3. print(df): This displays the entire DataFrame. You’ll see it nicely formatted, just like our mental picture of a table, complete with an index on the left (0, 1, 2, 3, 4).

Step 3: Inspecting Your Data – Getting to Know It

Once you have a DataFrame, the first thing you usually want to do is get a sense of what’s inside. Pandas offers several handy functions for this.

Add these lines to your script (after the DataFrame creation):

print("\n--- First 3 Rows ---")
print(df.head(3))

print("\n--- Data Info ---")
df.info()

print("\n--- Descriptive Statistics ---")
print(df.describe())

Explanation:

• df.head(3): The .head() method is incredibly useful. It shows you the first few rows of your DataFrame. By default, it shows 5 rows, but you can pass a number (like 3) to specify how many rows you want to see. This is great for a quick peek without overwhelming your screen.
• df.info(): This method provides a concise summary of your DataFrame. It tells you:
  • The number of entries (rows).
  • The number of columns.
  • Each column’s name, how many non-null values it contains (useful for finding missing data!), and its data type (e.g., object for text, int64 for integers).
  • Memory usage. This is vital for understanding your data’s structure and identifying potential data quality issues.
• df.describe(): This method generates descriptive statistics of the numerical columns in your DataFrame. It calculates things like:
  • count: Number of non-null values.
  • mean: The average value.
  • std: Standard deviation (how spread out the data is).
  • min/max: Smallest and largest values.
  • 25%, 50% (median), 75%: Quartiles, showing the distribution of data. This gives you a quick numerical overview of your data’s characteristics.

Step 4: Selecting Data – Picking What You Need

Often, you don’t need to work with the entire DataFrame. You might want to focus on specific columns or rows.

Add these lines:

print("\n--- Select 'Name' Column ---")
print(df['Name'])

print("\n--- Select 'Name' and 'City' Columns ---")
print(df[['Name', 'City']])

print("\n--- Select Row with Index 1 (Bob) ---")
print(df.loc[1])

print("\n--- Select Rows with Index 0 and 3 ---")
print(df.loc[[0, 3]])

Explanation:

• df['Name']: To select a single column, you use square brackets with the column name as a string. Notice the output is a Pandas Series!
• df[['Name', 'City']]: To select multiple columns, you pass a list of column names inside the square brackets. This returns a DataFrame containing only those columns.
• df.loc[1]: The .loc[] accessor is used for label-based indexing. Here, 1 refers to the index label of the row. It returns a Series representing that row’s data.
• df.loc[[0, 3]]: You can also pass a list of index labels to .loc[] to select multiple specific rows. This returns a DataFrame.

Step 5: Filtering Data – Finding Specific Records

Filtering allows you to select rows based on certain conditions, like finding all people older than 25 or everyone from London.

Add these lines:

print("\n--- People Older Than 25 ---")
print(df[df['Age'] > 25])

print("\n--- People from London or Paris ---")
print(df[(df['City'] == 'London') | (df['City'] == 'Paris')])

Explanation:

• df['Age'] > 25: This part creates a boolean Series. It checks each ‘Age’ value and returns True if it’s greater than 25, and False otherwise.
  • [False, True, True, True, False]
• df[...]: When you pass this boolean Series back into the DataFrame’s square brackets, Pandas acts like a filter. It only keeps the rows where the corresponding boolean value is True.
• df[(df['City'] == 'London') | (df['City'] == 'Paris')]: Here we’re combining two conditions.
  • (df['City'] == 'London'): Checks if the city is ‘London’.
  • (df['City'] == 'Paris'): Checks if the city is ‘Paris’.
  • |: This is the logical OR operator in Python (for Pandas boolean series). It means “either this condition is true OR that condition is true.”
  • For logical AND, you would use &.
  • Important: Always wrap individual conditions in parentheses when combining them, as & and | have higher precedence than comparison operators.

Step 6: Adding and Modifying Columns

Data isn’t static! You’ll often need to add new information or update existing data.

Add these lines:

print("\n--- Adding a new 'Grade' column ---")
df['Grade'] = ['A', 'B', 'C', 'A+', 'B']
print(df)

print("\n--- Modifying 'Score' for Alice (Index 0) ---")
df.loc[0, 'Score'] = 90
print(df)

print("\n--- Creating a 'Pass_Fail' column based on Score ---")
df['Pass_Fail'] = df['Score'].apply(lambda x: 'Pass' if x >= 80 else 'Fail')
print(df)

Explanation:

• df['Grade'] = ['A', 'B', 'C', 'A+', 'B']: To add a new column, you simply assign a list (or a Pandas Series) of values to a new column name (which becomes a new key in the DataFrame). The length of the list must match the number of rows in the DataFrame.
• df.loc[0, 'Score'] = 90: To modify a specific cell, you use .loc[] with both the row index label and the column name. This directly updates the value.
• df['Pass_Fail'] = df['Score'].apply(lambda x: 'Pass' if x >= 80 else 'Fail'): This is a slightly more advanced but very common way to create a new column based on existing data.
  • df['Score'].apply(...): The .apply() method is used on a Series (our ‘Score’ column) to apply a function to each element of that Series.
  • lambda x: 'Pass' if x >= 80 else 'Fail': This is a small, anonymous function (a lambda function) that takes one input x (which will be each score) and returns ‘Pass’ if x is 80 or more, otherwise ‘Fail’. This new Series of ‘Pass’/‘Fail’ values is then assigned to our new ‘Pass_Fail’ column.

Step 7: Handling Missing Data (A First Look)

Missing data is a common problem. It can be represented as NaN (Not a Number) in numerical columns or simply be empty. Machine learning models usually can’t handle missing values directly, so we need to deal with them.

Let’s first introduce some missing data into our DataFrame to simulate a real-world scenario.

import numpy as np # We'll use NumPy to represent missing values

# Let's create a DataFrame with some missing values
data_with_nan = {
    'Product': ['Laptop', 'Mouse', 'Keyboard', 'Monitor', 'Webcam'],
    'Price': [1200, 25, np.nan, 300, 50],
    'Stock': [10, 150, 75, np.nan, 120],
    'Category': ['Electronics', 'Electronics', 'Electronics', 'Electronics', 'Peripherals']
}
df_nan = pd.DataFrame(data_with_nan)
print("\n--- DataFrame with Missing Values ---")
print(df_nan)

print("\n--- Check for Missing Values ---")
print(df_nan.isnull().sum())

print("\n--- Drop Rows with ANY Missing Values ---")
df_dropped = df_nan.dropna()
print(df_dropped)

print("\n--- Fill Missing 'Price' with its Mean ---")
mean_price = df_nan['Price'].mean()
df_filled_price = df_nan.fillna({'Price': mean_price}) # Fill only 'Price' column
print(df_filled_price)

print("\n--- Fill Missing 'Stock' with its Median ---")
median_stock = df_nan['Stock'].median()
df_filled_stock = df_nan.fillna({'Stock': median_stock}) # Fill only 'Stock' column
print(df_filled_stock)

Explanation:

1. import numpy as np: NumPy is another fundamental Python library for numerical computing. Pandas uses it internally, and np.nan is the standard way to represent “Not a Number” (i.e., a missing value) in numerical data.
2. df_nan = pd.DataFrame(data_with_nan): We create a new DataFrame with np.nan values in ‘Price’ and ‘Stock’ columns.
3. df_nan.isnull().sum():
   • df_nan.isnull(): This creates a DataFrame of booleans, where True indicates a missing value.
   • .sum(): When called on a boolean Series/DataFrame, True is treated as 1 and False as 0. So, sum() counts the number of True values (missing values) in each column. This is a quick way to see how many missing entries you have per column.
4. df_dropped = df_nan.dropna(): The .dropna() method simply removes any row that contains at least one missing value. Be careful with this! If you have a lot of missing data, you might lose too much valuable information.
5. df_filled_price = df_nan.fillna({'Price': mean_price}) and df_filled_stock = df_nan.fillna({'Stock': median_stock}):
   • df_nan['Price'].mean(): Calculates the average of the non-missing ‘Price’ values.
   • df_nan['Stock'].median(): Calculates the median (middle value) of the non-missing ‘Stock’ values. Median is often preferred over mean for skewed data as it’s less affected by extreme outliers.
   • .fillna(): This method replaces missing values. We pass a dictionary {'ColumnName': value} to specify which value to use for missing entries in a particular column. This is one common strategy for handling missing data, known as imputation. Other strategies exist, but this is a great start!

Mini-Challenge: Your Turn to Manipulate!

Alright, you’ve learned a lot of basic data manipulation techniques. Now, it’s your turn to apply them!

Challenge:

1. Create a new Pandas DataFrame called student_data from the following dictionary:

   student_info = {
       'Student_ID': [101, 102, 103, 104, 105],
       'Name': ['Liam', 'Olivia', 'Noah', 'Emma', 'Ava'],
       'Major': ['CS', 'Biology', 'Math', 'CS', 'Physics'],
       'Grade': [88, 75, 92, np.nan, 81], # Note: np.nan for missing grade
       'Credits': [15, 12, 16, 15, 14]
   }
   

2. Inspect the DataFrame to identify any missing values.
3. Filter the DataFrame to show only students majoring in ‘CS’ (Computer Science).
4. Add a new column called 'Passed_Credits' which is True if ‘Credits’ is 15 or more, and False otherwise.
5. Fill the missing ‘Grade’ value with the median grade of all other students.

Hint: Remember to import numpy for np.nan! For filling the missing grade, first calculate the median of the ‘Grade’ column, then use .fillna().

import pandas as pd
import numpy as np

student_info = {
    'Student_ID': [101, 102, 103, 104, 105],
    'Name': ['Liam', 'Olivia', 'Noah', 'Emma', 'Ava'],
    'Major': ['CS', 'Biology', 'Math', 'CS', 'Physics'],
    'Grade': [88, 75, 92, np.nan, 81], # Note: np.nan for missing grade
    'Credits': [15, 12, 16, 15, 14]
}

student_data = pd.DataFrame(student_info)
print("--- Original Student Data ---")
print(student_data)

# 2. Inspect for missing values
print("\n--- Missing Values Check ---")
print(student_data.isnull().sum())

# 3. Filter for CS majors
cs_students = student_data[student_data['Major'] == 'CS']
print("\n--- CS Majors ---")
print(cs_students)

# 4. Add 'Passed_Credits' column
student_data['Passed_Credits'] = student_data['Credits'] >= 15
print("\n--- Student Data with Passed_Credits ---")
print(student_data)

# 5. Fill missing 'Grade' with median
median_grade = student_data['Grade'].median()
student_data_filled_grade = student_data.fillna({'Grade': median_grade})
print("\n--- Student Data with Missing Grade Filled (Median) ---")
print(student_data_filled_grade)

What to observe/learn:

• Did you successfully create the DataFrame?
• Could you identify the missing grade for Emma?
• Did your filtering correctly show only Liam and Emma?
• Was the Passed_Credits column correctly added, showing True for Liam, Noah, and Emma?
• Was Emma’s grade correctly filled with the median (which should be 88)?

If you successfully completed these steps, give yourself a pat on the back! You’re getting the hang of data manipulation!

Common Pitfalls & Troubleshooting

Working with data can sometimes throw curveballs. Here are a few common issues beginners face with Pandas:

1. KeyError when selecting columns:

   • Problem: You try df['name'] but the column is actually 'Name'. Python is case-sensitive!
   • Solution: Double-check your column names. Use df.columns to see the exact names of all columns in your DataFrame.
   • Example:

     # If column is 'Name', but you tried:
     # print(df['name']) # This would cause KeyError
     print(df['Name']) # Correct
     

2. Forgetting to assign changes back:

   • Problem: Many Pandas operations (like .dropna(), .fillna(), creating new columns from operations) return a new DataFrame or Series. If you don’t assign the result back to a variable (e.g., df = df.dropna()), your original DataFrame remains unchanged.
   • Solution: Always remember to reassign the result or use the inplace=True argument (though direct reassignment is often preferred for clarity in modern Pandas).
   • Example:

     # This will print the DataFrame without the dropped rows,
     # but df itself will still have them
     df.dropna()
     print(df) # Original df still has NaNs
     
     # Correct way to make the change permanent to df:
     df = df.dropna()
     print(df) # Now df has no NaNs
     

3. Incorrect boolean logic for filtering:

   • Problem: Using and or or instead of & or | when combining conditions for DataFrame filtering.
   • Solution: For element-wise logical operations on Pandas Series (which is what df['Age'] > 25 creates), you must use & (AND) and | (OR). Also, remember to wrap each individual condition in parentheses.
   • Example:

     # This will cause an error:
     # df[df['Age'] > 25 and df['City'] == 'London']
     
     # Correct way:
     df[(df['Age'] > 25) & (df['City'] == 'London')]
     

Summary: You’re a Data Sculptor!

Wow, you’ve covered a lot of ground in this chapter! You’ve officially started your journey into practical data handling, a cornerstone skill for anyone venturing into AI and Machine Learning.

Here are the key takeaways:

• Data Manipulation is Essential: Raw data needs cleaning, transforming, and organizing (like a chef’s ingredients) before it can be used effectively by ML models. “Garbage in, garbage out” is a real concern!
• Pandas is Your Tool: The Pandas library in Python is the industry standard for working with tabular data, making complex operations intuitive.
• DataFrames and Series: You learned about the two main Pandas data structures: the 2-dimensional DataFrame (like a spreadsheet) and the 1-dimensional Series (like a single column).
• Key Operations: You practiced fundamental data manipulation tasks:
  • Inspecting data (.head(), .info(), .describe()) to understand its shape and contents.
  • Selecting specific columns or rows (df['col'], df[['col1', 'col2']], df.loc[]).
  • Filtering data based on conditions (df[df['Age'] > 25]).
  • Adding and Modifying columns to enhance your dataset.
  • Basic Handling of Missing Data (.isnull().sum(), .dropna(), .fillna()).
• Incremental Learning: You built your code step-by-step, understanding each piece before moving on. This is the best way to learn complex programming concepts!

You now have a solid foundation in preparing data, which is a critical skill. Data scientists and ML engineers spend a significant amount of their time on these very tasks. You’re well on your way!

What’s Next?

In the upcoming chapters, we’ll continue to build on these skills. We’ll explore more advanced data cleaning techniques, learn how to visualize our data to uncover hidden patterns, and then finally, we’ll start building our very first simple machine learning models! Get ready for more exciting steps!

References

• Pandas Official Documentation
• Python Official Documentation
• NumPy Official Documentation

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.