The previous article covered threading and the Global Interpreter Lock's fundamental limitation for CPU-bound work. python multiprocessing, via multiprocessing.Process and Pool, sidesteps that limitation entirely by running genuinely separate operating-system processes, each with its own Python interpreter and memory space. This article covers creating processes, process pools, sharing data safely across process boundaries, and the start-method details (including a genuinely important change in Python 3.14) that many introductory guides skip over.
Introduction: Sidestepping the GIL with Separate Processes
A quick recap
As covered in the previous multithreading article: threads share memory, but they're limited by the GIL to executing one thread's Python bytecode at a time, meaning threading provides no real speedup for CPU-bound work. multiprocessing sidesteps this entirely — each process gets its own independent Python interpreter, its own GIL, and its own separate memory space, allowing genuine parallel execution across multiple CPU cores simultaneously.
The core trade-off
This isn't free, and it's worth framing honestly upfront: processes trade memory isolation (each process can't directly see or modify another's variables) and cross-process communication overhead (getting data between processes requires explicit serialization) for genuine parallel execution across CPU cores — something threading, constrained by the GIL, simply cannot provide for CPU-bound work.
What this article covers
Creating and running processes, managing many of them cleanly with Pool, safely sharing data across the process boundary, and the start-method details — fork, spawn, forkserver — that shape how a child process actually comes into existence, including a genuinely significant default change that landed in Python 3.14.
Creating and Running Processes
The basic pattern
from multiprocessing import Process
def worker(name):
print(f"Worker {name} running")
if __name__ == "__main__":
p = Process(target=worker, args=("A",))
p.start()
p.join()This is deliberately, closely similar to the threading.Thread API covered in the previous article — Process(target=..., args=...), then .start() to begin execution and .join() to block until it finishes — making the knowledge genuinely transferable between the two.
The mandatory if name == "main": guard
This is essential, not optional, particularly on Windows and macOS (which, as covered in Section 5, default to the spawn start method): without this guard, a child process re-imports the main script from scratch as part of its own startup — and without the guard, that re-import would trigger the Process(...).start() call all over again inside the child process itself, which would spawn another child, which would do the same thing again, recursively, essentially forever.
from multiprocessing import Process
def worker():
print("Working")
# WITHOUT this guard, running this script directly on spawn-based platforms
# can trigger infinite recursive process creation
if __name__ == "__main__":
p = Process(target=worker)
p.start()
p.join()Practical example: independent memory
Here's a genuinely important demonstration of exactly how different this is from threading, where shared memory was the whole point:
from multiprocessing import Process
shared_list = [1, 2, 3]
def modify_list():
shared_list.append(4)
print(f"Inside child process: {shared_list}")
if __name__ == "__main__":
p = Process(target=modify_list)
p.start()
p.join()
print(f"In parent process: {shared_list}")Inside child process: [1, 2, 3, 4]
In parent process: [1, 2, 3]The child process's shared_list and the parent's shared_list are genuinely two separate objects living in two separate memory spaces, despite sharing the same variable name in the source code — the child's modification never touches the parent's copy at all. This is the single most important conceptual difference from threading to internalize: with threads, this same code would show the parent's list correctly updated to [1, 2, 3, 4], since threads genuinely share memory. Processes fundamentally don't, by default — Section 4 covers the explicit mechanisms required to actually share data between them.
Managing Multiple Processes with Pool
The problem with manual Process objects
Exactly mirroring the equivalent problem covered for threads, manually creating, tracking, starting, and joining individual Process objects gets genuinely unwieldy fast, for anything beyond a couple of parallel tasks.
multiprocessing.Pool
from multiprocessing import Pool
def square(n):
return n * n
if __name__ == "__main__":
with Pool(processes=4) as pool:
results = pool.map(square, [1, 2, 3, 4, 5])
print(results) # [1, 4, 9, 16, 25]Pool(processes=4) creates a fixed pool of 4 worker processes. pool.map() distributes the input list across those workers, collecting results in the original order, regardless of which worker actually finished which item first — genuinely convenient, and closely mirroring ThreadPoolExecutor.map() from the previous article.
A practical example with a real speedup
from multiprocessing import Pool, cpu_count
import time
def cpu_bound_task(n):
# a genuinely CPU-intensive computation
return sum(i * i for i in range(n))
if __name__ == "__main__":
numbers = [10_000_000] * 8
# Serial execution
start = time.time()
serial_results = [cpu_bound_task(n) for n in numbers]
print(f"Serial: {time.time() - start:.2f} seconds")
# Parallel execution, sized to the machine's actual CPU core count
start = time.time()
with Pool(processes=cpu_count()) as pool:
parallel_results = pool.map(cpu_bound_task, numbers)
print(f"Parallel: {time.time() - start:.2f} seconds")Serial: 8.10 seconds
Parallel: 1.35 secondsThis is exactly the kind of workload where multiprocessing genuinely delivers — a real, measurable speedup on a genuinely CPU-bound task, roughly proportional to the number of CPU cores actually available (cpu_count() returns the number of logical cores on the current machine, a sensible default size for the pool in most cases).
Sharing Data Between Processes
The core constraint
As demonstrated in Section 2: processes don't share memory by default. Any data that needs to move between processes has to be explicitly passed, typically via serialization (pickling) through a Queue or Pipe, or through dedicated, purpose-built shared-memory objects.
from multiprocessing import Process, Queue
def worker(q):
q.put("Result from child process")
if __name__ == "__main__":
q = Queue()
p = Process(target=worker, args=(q,))
p.start()
result = q.get()
p.join()
print(result) # Result from child processQueue (a near-clone of the queue.Queue mentioned in the previous threading article) is thread- and process-safe, and handles the necessary serialization transparently — anything placed into it via .put() is pickled before being sent across the process boundary, and unpickled automatically on .get().
Value and Array for simple shared primitives
For simple, single-value or array-shaped shared data, Value and Array are backed by actual shared memory — genuinely shared between processes, not copied and serialized on every access:
from multiprocessing import Process, Value, Lock
def increment(counter, lock):
for _ in range(100_000):
with lock:
counter.value += 1
if __name__ == "__main__":
counter = Value("i", 0) # 'i' = a shared integer, initialized to 0
lock = Lock()
processes = [Process(target=increment, args=(counter, lock)) for _ in range(4)]
for p in processes:
p.start()
for p in processes:
p.join()
print(counter.value) # correctly 400000Just like the threading example in the previous article, incrementing shared state from multiple processes without a lock is a genuine race condition — counter.get_lock() (or, as shown above, an explicitly created separate Lock) is needed for safe concurrent updates, for exactly the same underlying reason locking mattered for threads.
Manager() for complex objects
For sharing more complex, arbitrary Python objects — dictionaries, lists — across processes, Manager() provides proxy objects that behave like the real thing, but transparently handle the necessary inter-process communication behind the scenes:
from multiprocessing import Process, Manager
def worker(shared_dict, key, value):
shared_dict[key] = value
if __name__ == "__main__":
with Manager() as manager:
shared_dict = manager.dict()
processes = [
Process(target=worker, args=(shared_dict, f"key{i}", i))
for i in range(4)
]
for p in processes:
p.start()
for p in processes:
p.join()
print(dict(shared_dict)) # {'key0': 0, 'key1': 1, 'key2': 2, 'key3': 3}Manager() is considerably more flexible than Value/Array — supporting dicts, lists, and more — but genuinely slower, since every operation on a manager-backed object involves inter-process communication behind the scenes, rather than direct shared-memory access.
multiprocessing.shared_memory for high performance
For genuinely performance-sensitive scenarios — large NumPy arrays, big datasets — multiprocessing.shared_memory provides direct, raw shared-memory access without the pickling overhead that Queue or Manager() would otherwise incur for large objects. This is a considerably more advanced tool, generally reached for specifically once you've confirmed that serialization overhead is a genuine, measured bottleneck for a large-data workload — not something to default to for everyday, small-scale inter-process communication.
Start Methods and Choosing Between Multiprocessing and Threading
The three start methods
fork— the historical default on Linux (through Python 3.13). Fast, since it copies the parent process's memory via copy-on-write rather than rebuilding everything from scratch — but genuinely unsafe if the parent process has active threads at the moment of forking, which can lead to deadlocks in the child process. This risk is exactly whyforkhas been the source of real, hard-to-diagnose bugs in production systems that mix threading and multiprocessing.spawn— starts a completely fresh Python interpreter process from scratch. Safer and more predictable, but slower to start, and it requires every argument passed to a child process to be picklable, since the child doesn't inherit the parent's memory at all. This has been the default on Windows, and on macOS since Python 3.8.forkserver— a middle-ground approach: a single, clean server process is started early (before any threads exist), and it handles forking new worker processes on request, avoiding the fork-in-threads danger while retaining some of fork's speed advantage.
An important, current update: the Linux default changed in Python 3.14
This is worth getting right, since it's a genuinely significant, fairly recent change: as of Python 3.14, the default start method on Linux changed from fork to forkserver — directly motivated by the fork-with-threads danger described above, which had been a long-standing, well-documented source of subtle bugs. If you're running Python 3.14 or later on Linux, forkserver is now what you get by default, without any explicit configuration; on versions before 3.14, fork remains the default. You can always check or explicitly set the start method yourself, regardless of your Python version:
import multiprocessing as mp
print(mp.get_start_method()) # confirms what's actually in effect
mp.set_start_method("spawn") # explicitly override, if needed — must be called before creating any ProcessWhy spawn (and forkserver) require picklable arguments
Because spawn and forkserver don't inherit the parent's memory the way fork does, every argument passed to a child process's target function has to be serialized (pickled) to actually get there — meaning anything you pass as an argument (or a returned result, coming back) needs to be a genuinely picklable object. This rules out passing things like open file handles, database connections, or unpicklable custom objects directly as arguments — a real, practical constraint worth knowing about before you hit a confusing PicklingError partway through a larger project.
A quick decision framework
Tying directly back to the previous threading article's own closing guidance: CPU-bound work that stays in Python bytecode → multiprocessing, to genuinely bypass the GIL. I/O-bound work waiting on network or disk → threading (ideally via ThreadPoolExecutor) or asyncio (covered in a later article in this series), since I/O-bound work doesn't need multiple CPU cores in the first place — it needs to avoid blocking while waiting. Mixing the two — using both threading and multiprocessing together in the same application — is occasionally genuinely useful for specific, more advanced architectures, but rarely helps for straightforward cases, and adds real complexity that's worth avoiding unless you have a specific, well-understood reason to combine them.
A practical caution: benchmark before assuming multiprocessing helps
Multiprocessing has real, non-trivial overhead — process startup cost (particularly with spawn/forkserver, which start a genuinely fresh interpreter) and serialization cost for any data crossing the process boundary via Queue, Pipe, or Manager(). For small or already-fast tasks, this overhead can genuinely outweigh the parallelism benefit entirely — spinning up several processes to parallelize a task that would have finished in milliseconds serially can easily end up slower overall, once process-creation and serialization overhead are accounted for. Before committing multiprocessing to a piece of code, benchmark both the serial and parallel versions directly against each other, on data genuinely representative of your real workload — exactly the same "measure, don't assume" discipline that applies to performance optimization generally, and genuinely necessary here given how real and easy-to-underestimate multiprocessing's overhead can be.