Process a file too large to load into memory, without silently truncating it

Write a streaming processor for the large file described below. This must never load the whole file into memory at once — that's the entire reason this isn't just a five-line pandas.read_csv() call, which is exactly what tends to work fine on a sample file in dev and then OOM-kills the process the first time it runs against the real one.

FILE
A daily server access log, JSONL format, roughly 30GB per day, delivered to a local disk mount by an upstream log shipper.

PROCESSING NEEDED
Count requests per status code and per endpoint, and insert every request with status >= 500 into a database table for alerting.

MEMORY BUDGET
Runs as a scheduled job on a container capped at 512MB; the job is killed and restarted by the orchestrator if it exceeds that.

FAILURE RECOVERY
If killed partway through, the next run should resume from roughly where it left off rather than reprocessing the full 30GB file from the start, and must not double-count requests already inserted.

REQUIREMENTS
1. Read the file incrementally — line-by-line for a line-delimited format (CSV, JSONL, plain log lines) using the file object as an iterator, or with an incremental parser for a format that isn't naturally line-delimited (ijson for large JSON arrays, csv.reader over an open file handle rather than csv reading a fully-loaded string). Never call .read() or .readlines() on the whole file, and never pass the whole file into pandas.read_csv() without chunksize= if pandas is genuinely the right tool here.
2. If Count requests per status code and per endpoint, and insert every request with status >= 500 into a database table for alerting. requires aggregation (a running total, a count per category, a set of distinct values seen), keep only the aggregate state in memory, not the raw rows — state explicitly what the actual peak memory usage will be as a function of the number of distinct keys or categories, not the number of rows, since that's the number that actually determines whether this fits Runs as a scheduled job on a container capped at 512MB; the job is killed and restarted by the orchestrator if it exceeds that..
3. Process the file in bounded-size batches for anything that benefits from batching (a bulk database insert, a batched API call) — pick a batch size and justify it against Runs as a scheduled job on a container capped at 512MB; the job is killed and restarted by the orchestrator if it exceeds that., rather than either one-row-at-a-time (slow) or the whole file at once (defeats the entire point of streaming).
4. Match If killed partway through, the next run should resume from roughly where it left off rather than reprocessing the full 30GB file from the start, and must not double-count requests already inserted. explicitly: if the file needs to be resumable after a crash partway through, track and persist progress (a byte offset, a line number, or a natural checkpoint like "last fully-processed batch id") somewhere that survives a process restart, and state exactly what happens on restart — reprocessing from the last checkpoint, and whether that risks reprocessing a partially-committed batch.
5. Validate row-level data as it streams past, the same way a normal validation pass would, but without ever accumulating invalid rows in an in-memory list beyond what Runs as a scheduled job on a container capped at 512MB; the job is killed and restarted by the orchestrator if it exceeds that. allows — write them to a bounded rejects file/stream instead, flushed periodically, not held entirely in memory until the end.
6. State the actual big-O memory behavior of the design in one sentence: O(1) relative to file size if truly streaming, or name specifically what does grow with file size and why that's still an acceptable, bounded amount given Runs as a scheduled job on a container capped at 512MB; the job is killed and restarted by the orchestrator if it exceeds that..

OUTPUT FORMAT
1. The streaming processor.
2. The batching and checkpoint/resume logic.
3. One sentence stating the actual memory behavior as a function of input size, confirmed against Runs as a scheduled job on a container capped at 512MB; the job is killed and restarted by the orchestrator if it exceeds that..