Why Pandas?

If you work with data in Python, Pandas is simply indispensable. This library offers powerful data structures and analysis tools that make data manipulation much simpler and more intuitive.

What is Pandas?

Pandas is an open-source library that provides high-performance, easy-to-use data structures, especially for working with tabular data (like spreadsheets) and time series.

Basic Structures

DataFrame

The DataFrame is Pandas’ most important structure - think of it as an Excel table, but much more powerful.

import pandas as pd

# Creating a simple DataFrame
data = {
    'name': ['Ana', 'Bruno', 'Carlos', 'Diana'],
    'age': [25, 30, 35, 28],
    'city': ['Sao Paulo', 'Rio de Janeiro', 'Belo Horizonte', 'Curitiba'],
    'salary': [5000, 6000, 7000, 5500]
}

df = pd.DataFrame(data)
print(df)

Series

A Series is a single column of data, similar to an array or list, but with an associated index.

# Accessing a column as Series
ages = df['age']
print(type(ages))  # pandas.core.series.Series

Essential Operations

1. Reading Data

# Read CSV
df = pd.read_csv('data.csv')

# Read Excel
df = pd.read_excel('data.xlsx')

# Read JSON
df = pd.read_json('data.json')

# Read from URL
url = 'https://example.com/data.csv'
df = pd.read_csv(url)

2. Initial Exploration

# First rows
print(df.head())

# Last rows
print(df.tail())

# DataFrame information
print(df.info())

# Descriptive statistics
print(df.describe())

# Check null values
print(df.isnull().sum())

3. Data Selection

# Select one column
names = df['name']

# Select multiple columns
subset = df[['name', 'age']]

# Filter rows
over_30 = df[df['age'] > 30]

# Multiple filters
sp_over_30 = df[(df['city'] == 'Sao Paulo') & (df['age'] > 30)]

4. Data Manipulation

# Add new column
df['annual_salary'] = df['salary'] * 12

# Remove column
df = df.drop('city', axis=1)

# Sort data
df_sorted = df.sort_values('salary', ascending=False)

# Group and aggregate
avg_by_city = df.groupby('city')['salary'].mean()

Data Cleaning

Handling Null Values

# Remove rows with null values
df_clean = df.dropna()

# Fill null values
df['age'].fillna(df['age'].mean(), inplace=True)

# Replace specific values
df['city'].replace('Rio de Janeiro', 'RJ', inplace=True)

Type Conversion

# Convert to numeric
df['age'] = pd.to_numeric(df['age'], errors='coerce')

# Convert to datetime
df['date'] = pd.to_datetime(df['date'])

# Convert to category (saves memory)
df['city'] = df['city'].astype('category')

Advanced Operations

Merge and Join

# Data from two DataFrames
customers = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Ana', 'Bruno', 'Carlos']
})

orders = pd.DataFrame({
    'id': [1, 1, 2],
    'product': ['Book', 'Notebook', 'Pen'],
    'value': [30, 15, 5]
})

# Merge (similar to SQL JOIN)
result = pd.merge(customers, orders, on='id')

Pivot Tables

# Create pivot table
pivot = df.pivot_table(
    values='salary',
    index='city',
    columns='department',
    aggfunc='mean'
)

Apply - Custom Functions

# Apply function to a column
df['name_upper'] = df['name'].apply(lambda x: x.upper())

# Apply function to multiple columns
def calculate_bonus(row):
    if row['salary'] > 6000:
        return row['salary'] * 0.1
    return row['salary'] * 0.05

df['bonus'] = df.apply(calculate_bonus, axis=1)

Data Analysis with Pandas

Basic Statistics

# Mean, median, mode
print(df['salary'].mean())
print(df['salary'].median())
print(df['salary'].mode())

# Correlation between columns
correlation = df[['age', 'salary']].corr()

Aggregations

# Multiple aggregations
summary = df.groupby('city').agg({
    'salary': ['mean', 'min', 'max'],
    'age': 'mean'
})

Performance Tips

1. Use Categories for Repetitive Data

# Saves a lot of memory
df['category'] = df['category'].astype('category')

2. Read Only Necessary Columns

# Faster and uses less memory
df = pd.read_csv('data.csv', usecols=['name', 'age', 'salary'])

3. Use Chunks for Large Files

# Process in chunks
for chunk in pd.read_csv('large_data.csv', chunksize=10000):
    # Process each chunk
    process(chunk)

Data Export

# Save to CSV
df.to_csv('result.csv', index=False)

# Save to Excel
df.to_excel('result.xlsx', index=False)

# Save to JSON
df.to_json('result.json', orient='records')

Conclusion

Pandas is a powerful tool that dramatically simplifies working with data in Python. Mastering Pandas is essential for anyone working with data science, data analysis, or machine learning.

Next steps:

  • Practice with real datasets (Kaggle is great for this)
  • Combine Pandas with Matplotlib/Seaborn for visualizations
  • Explore time series operations
  • Learn about memory optimization

Keep practicing and you’ll soon be manipulating data like a pro!