Verified against ChatGPT · 2026-07-25
Turn a messy DataFrame into a documented, reproducible cleaning pipeline
A prompt for building a pandas cleaning pipeline as small, named, chainable functions with logged row drops and a schema check at the end, instead of one long unexplained block of reassignments.
The prompt
Ready to copy — highlighted parts are example details you can swap.
Write a pandas data-cleaning pipeline for the DataFrame described below. The output must be a set of small, named, testable functions chained together — not one long block of unexplained df[...] reassignments. INPUT DATA orders.csv, ~200k rows: order_id, customer_email, order_date (string), amount, status KNOWN ISSUES TO FIX duplicate order_id rows, order_date in mixed formats, some amount values as "$1,200.00" strings TARGET SCHEMA order_id: unique int; order_date: datetime64; amount: float, no nulls; status: one of a fixed set of values REQUIREMENTS 1. Each cleaning step is its own function taking a DataFrame and returning a DataFrame (e.g. def drop_duplicate_orders(df: pd.DataFrame) -> pd.DataFrame), so the pipeline reads as df.pipe(step_a).pipe(step_b)... — reviewable and testable step by step, not one monolithic function. 2. Never mutate the input DataFrame in place inside a step; return a new one (copy() where needed) so steps stay composable and order-independent bugs are easier to spot. 3. For every row dropped or value changed, log a count — a silent drop of bad rows is a data-loss bug wearing a clean-code costume. 4. Validate the final shape against order_id: unique int; order_date: datetime64; amount: float, no nulls; status: one of a fixed set of values: correct dtypes (don't leave a date column as object), no unexpected nulls in required columns, and fail loudly with a clear message if the result doesn't match, rather than returning a DataFrame that's quietly wrong. 5. Use vectorized pandas operations; only drop to apply() with a Python-level loop if you can state why a vectorized approach isn't possible. OUTPUT FORMAT 1. Each step function, in order. 2. The pipe() chain that composes them. 3. One validation function that checks the result against order_id: unique int; order_date: datetime64; amount: float, no nulls; status: one of a fixed set of values and raises if it doesn't match.
Customize the highlighted detailsoptional — the prompt above already works
Why this works
Composing the pipeline with pipe() over small, pure, named functions means each step can be reviewed, unit-tested, and reordered independently — a single 40-line block of chained df[...] assignments cannot be tested at all except end-to-end, which hides exactly which step introduced a bug. Requiring a logged count for every row dropped or value changed targets the single most common and most dangerous pandas cleaning mistake: a filter or dropna() that silently removes far more data than intended, discovered weeks later when a downstream report looks wrong with no trail back to the cause. The schema validation at the end catches a specific, very real pandas failure mode — a date column that parsed as object instead of datetime64 because one row had a malformed value, which then breaks every downstream .dt accessor call with a confusing error far from the actual cause.
What you get back
def parse_order_dates(df: pd.DataFrame) -> pd.DataFrame: df = df.copy() before_na = df["order_date"].isna().sum() df["order_date"] = pd.to_datetime(df["order_date"], errors="coerce") dropped = df["order_date"].isna().sum() - before_na if dropped: logger.warning(f"parse_order_dates: {dropped} rows had unparseable order_date, now NaT") return df cleaned = ( raw_df .pipe(drop_duplicate_orders) .pipe(parse_order_dates) .pipe(clean_amount_column) .pipe(validate_against_schema) )
Verified against
ChatGPT GPT-5.1 · 2026-07-25
Changelog
- 2026-07-25 — Initial publish, verified against ChatGPT (GPT-5.1) on pandas 2.2.
Need this built into your business?
If a prompt isn't enough — custom software, built and maintained for you — that's Scult's day job.
EXPLORE CUSTOM SOFTWARE
