Files, descriptors, and I/O
To the kernel, an open file, pipe, socket, or terminal is a file descriptor: a small integer referring to an entry in a per-process table. Linux's uniform I/O model rests on this idea.
File descriptors
When a process opens a file, the kernel returns a file descriptor (fd), the lowest unused non-negative integer. Descriptors 0, 1, and 2 are standard input, output, and error. The fd indexes the process's descriptor table, which points to open file descriptions in the kernel holding the current offset and access mode. ls -l /proc/PID/fd lists a process's open descriptors.
open, read, write, close
open() takes a path and flags such as O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_TRUNC, and O_APPEND, plus permissions for new files. read(fd, buffer, n) reads up to n bytes and returns how many were read, 0 at end of file; write(fd, buffer, n) returns how many were written, which may be fewer than requested. close(fd) releases the descriptor. lseek() moves the file offset.
Short reads and writes
read and write may transfer fewer bytes than asked: reads from pipes, sockets, and terminals return whatever is available, and writes to pipes or sockets may be partial. Correct programs loop until all data is handled and treat EINTR, caused by signals, as a reason to retry. Ignoring short counts causes subtle data-loss bugs.
Buffering
Every read or write system call has overhead, so the C library's standard I/O functions (fopen, fprintf, fgets) buffer data in user space, issuing larger system calls. Output to a terminal is usually line-buffered and to a file fully buffered, which is why output can appear out of order or be lost if a program crashes before flushing. The kernel also buffers file data in its page cache.
Durability and fsync
When write() returns, data has usually reached the kernel's page cache, not the disk. fsync(fd) blocks until the file's data and metadata are safely on storage; fdatasync skips some metadata. Databases call fsync carefully to guarantee durability; to replace a file atomically, programs write a temporary file, fsync it, then rename() it over the original, since rename is atomic within a file system.
Duplication and pipes
dup2(oldfd, newfd) makes newfd refer to the same open file as oldfd; shells use it for redirection, for example duplicating a file's descriptor onto 1 before exec so the program's output goes to the file. pipe() creates a pair of descriptors, a read end and a write end; after fork, one process writes and the other reads, which is how shell pipelines are built.
Multiplexing I/O
Servers handling many connections cannot block on each one. poll() and select() wait for any of several descriptors to become ready; epoll, Linux's scalable interface, efficiently monitors thousands of descriptors and powers servers such as nginx. Non-blocking descriptors (O_NONBLOCK) return EAGAIN instead of waiting. Memory-mapped files via mmap() let programs access files as memory.
Worked example
How a shell runs ls > out.txt: it forks; in the child it calls open("out.txt", O_WRONLY | O_CREAT | O_TRUNC, 0666), getting fd 3; dup2(3, 1) makes fd 1 refer to out.txt; close(3); then execve("/usr/bin/ls", ...). ls writes to fd 1 as always, unaware that it now refers to a file. The parent shell waits for the child, then continues with its own fd 1 still attached to the terminal.
Key takeaways
File descriptors are small integers indexing a per-process table of open files, pipes, sockets, and terminals; 0, 1, and 2 are standard streams; /proc/PID/fd lists them. open, read, write, close, and lseek perform I/O with flags such as O_CREAT and O_APPEND. Reads and writes can be short, so loop and retry on EINTR. stdio buffers in user space, and the kernel's page cache buffers too. fsync forces data to storage; write-fsync-rename replaces files atomically. dup2 implements redirection and pipe() pipelines. poll, select, and epoll multiplex many descriptors; O_NONBLOCK and mmap offer other I/O styles.
Flashcard
Small integer referring to an open file
Check your understanding
Why do careful programs write a temporary file, call fsync, and then rename it over the original?