iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
You can build a small data cleaning service by accepting a CSV upload in FastAPI, parsing it with pandas under explicit rules, returning the cleaned CSV with row counts in the response headers, and packaging the whole thing in a Docker image. The service is only as good as the rules you write down first, so this guide starts with those decisions and then builds the code around them.
Decide the contract before writing any code
A cleaning endpoint is easy to build and hard to trust if callers cannot predict what it will do to their data. Fix these decisions first, then write them into the code and the README so that every rule is visible.
| Decision | Choice used in this guide | Change it when |
|---|---|---|
| Accepted input | UTF-8 CSV with a header row and a .csv file name |
Callers send Excel files or other formats; each format needs its own parser |
| Upload size | 10 MB hard cap, enforced while the body is read | Your largest real file is bigger; raise the cap and the container memory limit together |
| Required columns | order_id and amount |
Your data contract uses different names or needs more columns |
| Missing values | Empty cells, NA, N/A and null are treated as missing; rows without an order_id or a numeric amount are dropped |
Downstream systems need those rows kept and flagged instead |
| Duplicates | The first row for each order_id is kept |
Repeated IDs are legitimate, for example one ID per line item |
| Malformed files | Rejected with HTTP 422 instead of skipping lines silently | You prefer partial output with a warning, which needs an explicit on_bad_lines setting and a report field |
| Output | Cleaned CSV, plus row counts in X-Rows-* headers |
Callers want JSON records instead of a file |
| Authentication and retention | Not included in this guide | The service is reachable by anyone who can reach it; add an API key or similar control and a written deletion policy before production use |
Project layout
Keep the application in a small package so that the Docker build copies only what the server needs. The layout is:
Free tools Windows power users keep installed
One-click scans. No signup required.
csv-cleaner/
app/
__init__.py (empty)
main.py
requirements.txt
Dockerfile
.dockerignore
docker-compose.yml (optional)Accept the upload safely
FastAPI receives uploaded files as form data, so install the parser it depends on:
#1 Best Overall
pip install fastapi uvicorn pandas python-multipartWithout python-multipart, a route that declares File() cannot receive uploads. The FastAPI request files guide covers the form-data mechanics.
Choosing between bytes and UploadFile
The same guide contrasts two parameter types. A parameter typed as bytes holds the entire file contents in memory. An UploadFile parameter instead uses what the guide calls a “spooled” file: the content stays in memory up to a size threshold and is then written to a temporary file on disk. It also exposes a file-like interface and metadata such as the file name and content type. For a service that may receive larger CSVs, UploadFile is the better declaration, and this guide uses it.
The upload is still read into a bytes object in the endpoint below, but only up to the cap plus one byte. That lets the service reject oversized files without buffering all of them. The trade-off is that the whole accepted file is in memory during parsing. If you need to handle files larger than available memory, use pd.read_csv(..., chunksize=...) and keep any cross-chunk state, such as a set of seen order_id values, explicitly in your code. Duplicate removal in that mode is more involved than the single-pass version shown here.
Why the extension check is not validation
Checking that the file name ends in .csv only screens out obvious mistakes. Real validation happens during parsing and in the required-column check. Do not rely on the client-supplied content type.
Parse with explicit rules
Pandas will guess a great deal if you let it. The pandas read_csv reference documents the controls that make parsing predictable. This guide uses these:
dtype=strkeeps every column as text. Identifiers such as00412keep their leading zeros, and you convert only the columns you intend to convert.keep_default_na=Falsetogether withna_values=[...]makes the missing-value list exactly the one you wrote, instead of pandas’ longer default list.encoding="utf-8-sig"reads UTF-8 and removes a byte-order mark if one is present. Plainutf-8leaves that invisible character at the start of the first column name, which then fails the required-column check with a confusing message.on_bad_lines="error"makes a row with the wrong number of fields raise a parser error. This is the default behavior, and it is set explicitly here so that the policy is visible in the code.
Handle missing values deliberately
How pandas represents a missing value depends on the column’s dtype. Numeric columns use NaN, object columns use None or NaN, and datetime columns use NaT. The pandas missing-data guide explains this, and isna() and notna() detect all of these forms. Because the service reads everything as text, a blank cell arrives as a missing value in an object column, and the numeric conversion happens afterwards.
One consequence matters: pd.to_numeric(..., errors="coerce") turns unparseable text such as n/a! into missing values. The code below then counts those rows with blank amounts in the same bucket. If you need to tell blank cells apart from invalid text, count them before the conversion.
Recommended Free Tools
Clean the data and report what changed
The cleaning function returns both the cleaned frame and a count for each rule, so callers can see what happened to their rows. Put this in app/main.py together with the endpoint in the next section.
from io import BytesIO
import pandas as pd
from fastapi import FastAPI, File, HTTPException, UploadFile
from fastapi.concurrency import run_in_threadpool
from fastapi.responses import Response
MAX_UPLOAD_BYTES = 10 * 1024 * 1024 # 10 MB
REQUIRED_COLUMNS = ["order_id", "amount"]
NA_VALUES = ["", "NA", "N/A", "null"]
app = FastAPI(title="CSV cleaning service")
def parse_csv(data: bytes) -> pd.DataFrame:
try:
return pd.read_csv(
BytesIO(data),
dtype=str,
keep_default_na=False,
na_values=NA_VALUES,
encoding="utf-8-sig",
on_bad_lines="error",
)
except UnicodeDecodeError:
raise HTTPException(status_code=422, detail="File must be UTF-8 encoded")
except pd.errors.EmptyDataError:
raise HTTPException(status_code=422, detail="File contains no header row")
except pd.errors.ParserError as exc:
raise HTTPException(status_code=422, detail=f"Malformed CSV: {exc}")
def clean_frame(df: pd.DataFrame):
df.columns = [str(c).strip().lower() for c in df.columns]
missing_cols = [c for c in REQUIRED_COLUMNS if c not in df.columns]
if missing_cols:
raise HTTPException(
status_code=422,
detail=f"Missing required columns: {', '.join(missing_cols)}",
)
rows_in = len(df)
df["order_id"] = df["order_id"].str.strip().replace("", pd.NA)
df["amount"] = pd.to_numeric(
df["amount"].str.replace(",", "", regex=False).str.strip(),
errors="coerce",
)
if "customer_email" in df.columns:
df["customer_email"] = df["customer_email"].str.strip().str.lower()
df = df.dropna(subset=["order_id", "amount"])
rows_dropped_missing = rows_in - len(df)
before_dedupe = len(df)
df = df.drop_duplicates(subset=["order_id"], keep="first")
rows_dropped_duplicate = before_dedupe - len(df)
report = {
"rows_in": rows_in,
"rows_dropped_missing": rows_dropped_missing,
"rows_dropped_duplicate": rows_dropped_duplicate,
"rows_out": len(df),
}
return df, report
def process(data: bytes):
df = parse_csv(data)
cleaned, report = clean_frame(df)
return cleaned.to_csv(index=False), reportWhat the rules do to a sample file
Suppose a caller uploads orders.csv with this content:
order_id,customer_email,amount
1001, Ana@Example.com ,"1,250.00"
1002,,89.5
1001, Ana@Example.com ,"1,250.00"
1003,bo@example.com,The service strips whitespace, lowercases the email addresses, removes thousands separators, and converts amounts to numbers. Row 1003 has no amount, so it is dropped as missing. The second row for 1001 is a duplicate, so it is dropped. The response body is:
order_id,customer_email,amount
1001,ana@example.com,1250.0
1002,,89.5The headers report X-Rows-In: 4, X-Rows-Dropped-Missing: 1, X-Rows-Dropped-Duplicate: 1 and X-Rows-Out: 2. Row 1002 keeps its amount even though its email is blank, because the email column is optional in this contract.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Wire up the endpoint
The endpoint reads at most one byte beyond the cap, rejects anything larger, and then runs the CPU-bound pandas work in a thread pool. Running pandas directly inside an async function would block the event loop, so other requests would wait while a large file is parsed.
@app.post("/clean")
async def clean_upload(file: UploadFile = File(...)):
if not (file.filename or "").lower().endswith(".csv"):
raise HTTPException(status_code=415, detail="Upload a file with a .csv extension")
data = await file.read(MAX_UPLOAD_BYTES + 1)
if len(data) > MAX_UPLOAD_BYTES:
raise HTTPException(status_code=413, detail="File exceeds the 10 MB limit")
csv_text, report = await run_in_threadpool(process, data)
headers = {
"X-Rows-In": str(report["rows_in"]),
"X-Rows-Dropped-Missing": str(report["rows_dropped_missing"]),
"X-Rows-Dropped-Duplicate": str(report["rows_dropped_duplicate"]),
"X-Rows-Out": str(report["rows_out"]),
"Content-Disposition": 'attachment; filename="cleaned.csv"',
}
return Response(content=csv_text, media_type="text/csv", headers=headers)Run and test it locally
Start the server without Docker first, so that parsing problems are easier to see:
uvicorn app.main:app --reload --port 8000Upload the sample file and save the cleaned output while printing the response headers:
curl -s -D - -F "file=@orders.csv" http://localhost:8000/clean -o cleaned.csvFastAPI also serves interactive documentation at http://localhost:8000/docs, where the same upload can be sent from the browser.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Containerize the service
Docker’s Python language guide covers the general pattern of a Python container with a pinned requirements file, which is the approach used here.
Pin the dependencies
Create requirements.txt from the environment you developed in, so that the image installs the same versions you ran:
pip freeze > requirements.txtReview the file and remove anything the service does not import. Commit it with the code, and reinstall from it in a fresh virtual environment before you build the image.
Write the Dockerfile
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1
PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ ./app/
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]Use the Python version you developed against, and change the tag to match it. The --host 0.0.0.0 flag matters: uvicorn bound to 127.0.0.1 inside the container is unreachable from the host, even with a published port.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Exclude local files from the build
.git
.venv/
__pycache__/
*.pyc
*.csvBuild and run
docker build -t csv-cleaner:1 .
docker run --rm -p 8000:8000 csv-cleaner:1Run the same curl command as before. The container is now the only runtime dependency the caller needs.
Optional: run with Docker Compose on one host
services:
cleaner:
build: .
ports:
- "8000:8000"
restart: unless-stopped
mem_limit: 512mStart it with docker compose up --build -d. The restart: unless-stopped policy brings the container back after a crash or host reboot. Set mem_limit from your largest realistic upload: parsing a CSV in pandas commonly needs several times the file’s size in memory, so a 10 MB file can require far more than 10 MB of RAM.
Deploy beyond a single host
The FastAPI Docker guide describes several ways to run a container image and notes that HTTPS is commonly handled outside the application container. Choose by how much infrastructure you want to operate:
- Docker Compose on one server. The least operational work. You own the host, its patching, and its restart behavior. TLS is usually terminated by a reverse proxy in front of the container.
- Kubernetes, Docker Swarm or Nomad. These handle replication, rolling updates and health-based restarts, but you must run the orchestrator. Align the number of replicas with the strategy the guide describes for your orchestration setup.
- A managed cloud service that runs container images. You supply an image and the provider runs it. Check the provider’s request body size limit, memory options and HTTPS handling before you rely on the 10 MB cap in this guide.
Whichever route you choose, a reverse proxy in front of the service often enforces its own limit before the request reaches FastAPI. Nginx, for example, defaults to client_max_body_size 1m, so a 2 MB upload can fail with a 413 from the proxy even though the application allows 10 MB. Raise the proxy limit to match the application’s cap.
If you run several uvicorn workers with --workers, each worker is a separate process with its own copy of the loaded data, so memory use grows with the worker count.
Quick Recap
Troubleshooting
- Uploads are rejected or the route fails at startup.
python-multipartis not installed in the environment the image was built from. Add it torequirements.txtand rebuild withdocker build --no-cacheif the old layer is cached. - HTTP 422 with “Malformed CSV”. The message names the line that failed to tokenize. Common causes are an unclosed quote, a stray comma in an unquoted field, or a row with an extra column. Fix the source file, or change the policy deliberately.
- HTTP 422 with “Missing required columns”. Check the header for case, extra spaces, and a leftover byte-order mark. The service lowercases and trims names, but it does not rename columns that differ in wording.
- Every amount is dropped. Currency symbols, spaces used as thousands separators, or a decimal comma will not convert. Extend the conversion step for the formats your callers send, and check the
X-Rows-Dropped-Missingheader after each change. - The container exits with code 137. The kernel stopped the process for exceeding its memory limit. Raise
mem_limit, lower the upload cap, or run fewer workers. - The host cannot reach port 8000. Confirm that the container was started with
-p 8000:8000and that uvicorn is bound to0.0.0.0.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
