Linux Epoll, Kqueue & Event Loop Kernel Architecture Studio
Examine how the Linux kernel powers high-scale network servers (Nginx, Envoy, Node.js, Redis). Simulate the kernel red-black interest tree, contrast Level-Triggered vs Edge-Triggered buffer draining, and observe how EPOLLEXCLUSIVE eliminates multi-core thundering herd wakeups.
1. Kernel Epoll Set & Triggering Mode Selection
Ready List Length: 0 sockets
epoll_wait() Invocation Count: 0
Buffer Drain Status: All idle
Worker Wakeups: 0
2. Multi-Worker Thundering Herd Simulator (EPOLLEXCLUSIVE)
Simulate 4 worker processes listening on the same shared TCP listening socket. Observe CPU context switching spikes without EPOLLEXCLUSIVE versus optimal 1-worker wakeups with EPOLLEXCLUSIVE.
Worker 1: Idle
Worker 2: Idle
Worker 3: Idle
⚠️ 5 Fatal Traps in Linux Epoll & Event Loop Engineering
1. Blocking Socket in Edge-Triggered (ET) Mode Causing Event Loop Freeze
In Edge-Triggered mode, the application must loop read() until the buffer is empty. If the file descriptor was accidentally left in blocking mode (without O_NONBLOCK), the final read() blocks the entire thread waiting for packets that never arrive, completely freezing the event loop.
2. Buffer Underrun Starvation in ET Without EAGAIN Drain
If a worker in ET mode only reads 1,024 bytes and exits the loop without reading until EAGAIN, the kernel will NOT re-notify epoll_wait() on the next iteration because the state did not change. The remaining unconsumed bytes sit in the kernel buffer indefinitely.
3. Dangling File Descriptor Leaks via dup() and fork()
Calling close(fd) removes the descriptor from the epoll set only if no other file table references exist. If the process called fork() or dup(), the underlying kernel open file description remains referenced, keeping the dead socket in the epoll set and delivering ghost events.
4. Epoll Set Nesting Loops & Max Nesting Limits (EPOLL_MAX_NESTS)
Because an epoll file descriptor can itself be added to another epoll instance, malicious or buggy code can construct deep or cyclical polling graphs. The Linux kernel enforces a hard recursion limit (EPOLL_MAX_NESTS = 4); exceeding this triggers instant EINVAL errors.
5. Multi-Threaded Data Races in Level-Triggered Mode
In LT mode with a thread pool, if Thread 1 begins reading from a socket but has not drained it before Thread 2 calls epoll_wait(), the kernel immediately delivers the same descriptor to Thread 2. Both threads attempt to read concurrently, corrupting application protocol framing. Use EPOLLONESHOT to prevent concurrent delivery.