Inter-Process Communication: Pipes, Sockets & Shared Memory

Each process has its own virtual address space. Process A cannot read Process B's variables directly; the OS enforces that boundary. But processes constantly need to cooperate: a shell connects commands with pipes, a web server spawns workers that share a cache, a daemon notifies clients of events. That cooperation requires Inter-Process Communication (IPC).

UNIX and Linux provide a rich set of IPC mechanisms. Each has different trade-offs in speed, complexity, and use case. Here is each one in turn.

Why Processes Can't Share Memory Directly#

When the OS creates a process, it gets its own page table. Two processes with a variable at virtual address 0x7fff1234 are looking at different physical frames. Writing to one does not affect the other.

This isolation is a feature: it's why a crashing process doesn't take down the whole system. IPC mechanisms are the deliberate, controlled exceptions to that rule.

Pipes (Anonymous, Unidirectional)#

A pipe is the simplest IPC mechanism. It's a kernel-managed buffer (typically 64 KB on Linux) with a write end and a read end. Data flows in one direction: write end → buffer → read end.

Key properties:

• Unidirectional: for bidirectional comms you need two pipes
• Anonymous: no name in the filesystem; only inherited by related processes (parent/child)
• Blocking: reading from an empty pipe blocks; writing to a full pipe blocks
• Automatic EOF: when all write ends are closed, read returns 0

Shell Pipes#

When you type ls | grep foo, the shell:

1. Creates a pipe
2. Forks two children
3. Connects ls's stdout to the pipe's write end (dup2)
4. Connects grep's stdin to the pipe's read end
5. Execs ls and grep in the two children

Python: subprocess.PIPE#

Named Pipes (FIFOs)#

Anonymous pipes only work between related processes (parent/child). A named pipe (FIFO) is a pipe with a filename in the filesystem. Any two processes, even completely unrelated ones, can open it by path.

FIFOs appear in ls -l with a p type flag. They behave like pipes once opened: kernel-buffered, unidirectional, blocking. The file path is just a rendezvous point; no data is stored on disk.

Message Queues#

Pipes pass a raw byte stream, so the receiver has no idea where one message ends and the next begins. Message queues deliver discrete, typed messages. The receiver gets one complete message per receive call.

POSIX message queues (mq_open, mq_send, mq_receive) are available on Linux. Python's multiprocessing.Queue offers a similar message-oriented API, though it is not built on POSIX message queues; internally it uses a pipe (see below).

multiprocessing.Queue serialises Python objects with pickle, puts them into a pipe, and handles the framing for you. For high-throughput work between many workers, see multiprocessing.JoinableQueue which supports task_done() / join() for drain-and-wait patterns.

Give the Queue a Limit#

Queue() with no argument has no limit. If producers put work in faster than the worker takes it out, the queue grows for as long as that lasts, and every item in it sits in the process's memory. Nothing warns you: memory climbs until the operating system kills the process, and every message still waiting in the queue is lost with it.

Queue(maxsize=20) caps it. Once 20 items are waiting, put() blocks (or put_nowait() raises queue.Full), so the producer is told to slow down or to try again later. Memory stays flat, the worker keeps going, and nothing already accepted is lost. Pushing back on the sender like this is called backpressure.

Shared Memory: The Fastest IPC#

All IPC mechanisms so far copy data: write it into a kernel buffer, then copy it out to the reader. Shared memory eliminates the copies entirely. A region of physical RAM is mapped into the virtual address space of two or more processes. One process writes to a virtual address; the other process can read it immediately, with zero kernel involvement after setup.

This makes shared memory the fastest IPC mechanism, with throughput limited only by memory bandwidth. The catch: nothing prevents both processes from writing simultaneously. You must use semaphores or mutexes to coordinate access.

Python: multiprocessing.shared_memory (3.8+)#

Python: mmap for shared memory#

Signals: Asynchronous Notifications#

Signals are the UNIX mechanism for asynchronous notification. They are not for transferring data. A signal is just a number that says "something happened."

Signal	Number	Default action	Meaning
SIGTERM	15	Terminate	Polite shutdown request
SIGKILL	9	Terminate (uncatchable)	Immediate kill
SIGINT	2	Terminate	Ctrl+C from terminal
SIGCHLD	17	Ignore	Child process state changed
SIGALRM	14	Terminate	Timer expired
SIGUSR1	10	Terminate	User-defined signal 1
SIGUSR2	12	Terminate	User-defined signal 2
SIGHUP	1	Terminate	Terminal closed (often: reload config)
SIGSEGV	11	Core dump	Segmentation fault

SIGKILL and SIGSTOP cannot be caught or ignored; the kernel handles them directly.

Run this, then from another terminal: kill -USR1 <pid> or kill -TERM <pid>.

Signals have important limitations: you can't queue multiple signals of the same type (a second SIGUSR1 before the first is handled may be lost), and signal handlers run in a restricted context (not all functions are async-signal-safe).

Unix Domain Sockets vs TCP Sockets for Local IPC#

Sockets are the most flexible IPC mechanism: they support bidirectional, full-duplex, stream or datagram communication. For local IPC you have two options:

Unix domain sockets (AF_UNIX)

• Address is a filesystem path (/tmp/app.sock)
• Data stays in kernel, with no TCP/IP stack overhead
• ~2× faster than loopback TCP for high-throughput local IPC
• Supports passing file descriptors between processes (SCM_RIGHTS)

TCP loopback (AF_INET to 127.0.0.1)

• Goes through the TCP/IP stack
• Easy to extend to remote machines later
• Works across containers / VMs; Unix sockets do not cross filesystem boundaries easily

Comparison: All IPC Mechanisms#

Mechanism	Speed	Complexity	Direction	Persistence	Best for
Anonymous pipe	Medium	Low	One-way	None	Parent ↔ child data stream
Named pipe (FIFO)	Medium	Low	One-way	Path exists	Unrelated processes, simple
Message queue	Medium	Medium	Both ways	Until deleted	Discrete jobs, task queues
Shared memory	Very fast	High (needs sync)	Both ways	Until unlinked	High-throughput, large data
Signal	Minimal	Low	One-way	None	Async events, notifications
Unix domain socket	Fast	Medium	Full-duplex	Until unlinked	General local IPC
TCP loopback	Moderate	Medium	Full-duplex	None	Cross-host extensible IPC

Choosing the Right Mechanism#

• One command to another in a shell script → pipe
• Daemon that receives jobs from multiple clients → Unix domain socket or message queue
• High-throughput data sharing between workers → shared memory + semaphore
• Notify a running process to reload its config → SIGUSR1
• Microservices on the same host → Unix domain socket (faster) or TCP (portable)
• Worker pool that processes tasks → multiprocessing.Queue

There is no single IPC tool. The right choice depends on whether you need data transfer or notification, how much data, whether you need synchronisation, and whether the processes are related or completely independent.