8  Cleaning Data

8.1 Normalize column names

import pandas as pd

df = pd.DataFrame(columns=["City Name", " Total Value ", "YEAR"])

df.columns = (
    df.columns
    .str.strip()
    .str.lower()
    .str.replace(" ", "_")
)

df.columns

8.2 Replace special missing-value markers

df = pd.DataFrame({
    "value": ["10", "*", "N/D", "25.5"]
})

df["value"] = (
    df["value"]
    .replace({"*": pd.NA, "N/D": pd.NA})
    .pipe(pd.to_numeric, errors="coerce")
)

df

8.3 Remove duplicates

df = df.drop_duplicates()

8.4 Things to remember

• Keep the original raw data unchanged.
• Convert missing markers before converting data types.
• Validate row counts before and after cleaning.