Parallelize a CPU-bound batch job across cores without a pickling error

Parallelize the batch job below across multiple CPU cores. This needs to be multiprocessing specifically, not threading — confirm that's actually the right call given the workload before writing anything.

CURRENT JOB
A script that resizes and re-encodes 80,000 product images from a shared network drive, one at a time.

WORKLOAD SIZE
80,000 images per run, run nightly.

PER-ITEM WORK
Each image is decoded, resized with Pillow, re-encoded to WebP, and written back to disk — no network calls involved.

HARDWARE
A dedicated 16-core batch-processing VM, runs this job alone with nothing else scheduled on it.

REQUIREMENTS
1. Confirm this is genuinely CPU-bound before reaching for multiprocessing: if Each image is decoded, resized with Pillow, re-encoded to WebP, and written back to disk — no network calls involved. is actually I/O-bound (waiting on a network call or disk, not computing), say so explicitly and recommend threading or asyncio instead, since multiprocessing adds real overhead (process startup, inter-process serialization) that only pays off when the bottleneck is CPU cycles, not waiting.
2. Every argument passed into the worker function, and everything the worker function returns, must be picklable — the default way multiprocessing.Pool ships data between processes. Check Each image is decoded, resized with Pillow, re-encoded to WebP, and written back to disk — no network calls involved. for anything that isn't: an open file handle, a database connection, a lambda, a bound method on an unpicklable object. If something isn't picklable, redesign the worker to open that resource inside the worker process itself, not pass it in from the parent.
3. Choose a chunk size deliberately, not the library default of 1 — for 80,000 images per run, run nightly. many small, fast items, a chunk size of 1 makes the inter-process communication overhead dominate the actual work; compute a reasonable chunksize (total items divided by roughly 4x the process count) and say why.
4. Size the process pool to A dedicated 16-core batch-processing VM, runs this job alone with nothing else scheduled on it.'s actual core count via os.cpu_count(), not a hardcoded number that might exceed or badly underuse the machine this actually runs on, and leave at least one core free if this runs alongside other processes on the same machine.
5. Handle a single worker failure without losing the whole batch: use Pool.imap or imap_unordered with per-item exception handling inside the worker (catch, return a tagged failure result) rather than letting one bad item's exception propagate and kill the entire pool run silently.
6. If the workload benefits from sharing large read-only data across workers (a big lookup table, say), use an initializer function with Pool(initializer=..., initargs=...) so it's loaded once per worker process, not repeatedly pickled and sent per task.
7. State the platform this actually runs on and whether it matters here: Windows and macOS default to the spawn start method (each worker re-imports the module from scratch, so worker code must be importable and guarded behind if __name__ == "__main__"), while Linux defaults to fork — a script that only works because it happened to be tested on Linux can fail to start at all on Windows with a confusing pickling error that has nothing to do with the actual worker logic.

OUTPUT FORMAT
1. Confirmation this is genuinely CPU-bound, or the recommendation to use threading/asyncio instead if it isn't.
2. The worker function and the Pool-based orchestration code.
3. The chosen chunksize and pool size, with the arithmetic behind each.
4. How a single failed item is surfaced without killing the batch.
5. Confirmation the code is guarded for the start method A dedicated 16-core batch-processing VM, runs this job alone with nothing else scheduled on it.'s platform actually uses.