Real-time voice and video
Calls and video meetings must deliver media within a fraction of a second, coping with jitter and loss on a network that guarantees neither.
Constraints of conversation
In a conversation, delays above about 150 milliseconds one way start to be noticeable, and above about 400 milliseconds conversation becomes awkward, with people talking over each other. Unlike streaming, there is no time to buffer seconds of media or to retransmit lost packets over long paths, so real-time media usually travels over UDP.
RTP and RTCP
The Real-time Transport Protocol (RTP), standardised in RFC 3550, carries media in UDP packets with headers containing a payload type (identifying the codec), a sequence number (for detecting loss and reordering), a timestamp (for playout timing), and a source identifier. The companion RTCP sends periodic reports on loss, jitter, and delay, which senders use to adapt.
Jitter buffers
Packets leave the sender at even intervals but arrive with varying delays. A receiver's jitter buffer holds packets briefly and plays them at a steady pace using their timestamps. Fixed playout delays waste time when the network is calm; adaptive jitter buffers adjust the delay, typically during silences, trading a little extra latency for fewer late, discarded packets.
Handling loss
Lost or late audio and video packets cannot wait for retransmission. Forward error correction sends redundant information so lost packets can be reconstructed; Opus can include a low-bit-rate copy of previous audio in each packet. Packet loss concealment fills gaps by repeating or interpolating audio, and video decoders conceal missing blocks until the next key frame or a requested refresh.
Signalling with SIP
Before media flows, calls must be set up. The Session Initiation Protocol (SIP) locates users, negotiates codecs and addresses using the Session Description Protocol (SDP), and manages call setup and teardown. SIP is widely used in internet telephony and in carriers' voice-over-LTE (VoLTE) and Wi-Fi calling services.
WebRTC
WebRTC, standardised by the W3C and IETF and supported in all major browsers, provides real-time audio, video, and data channels between browsers and apps. Peers exchange session descriptions through any signalling channel, use ICE with STUN servers to discover addresses through NATs, fall back to TURN relays when direct paths fail, and encrypt media with DTLS-SRTP. Google Meet, Discord, and many others build on it.
Group calls and quality of service
Group video calls usually use a selective forwarding unit (SFU) server: each participant uploads streams once, often at several resolutions (simulcast), and the SFU forwards the appropriate one to each viewer, avoiding a mesh of connections. Networks can prioritise real-time traffic using differentiated services (DiffServ) markings, common in enterprise networks, though the public internet generally ignores them.
Worked example
A voice call sends 20 millisecond audio frames. Measured network delays vary from 40 to 110 milliseconds. A jitter buffer holding about 80 milliseconds means packets are played 120 milliseconds after sending, assuming the minimum delay of 40 ms plus 80 ms of buffering, so a packet delayed up to 110 ms still arrives in time. Add about 20 ms for encoding and 20 ms for audio capture and playback, and mouth-to-ear delay is about 160 ms, near the comfort limit.
Key takeaways
Conversation needs one-way delay under about 150 ms, so real-time media uses UDP without retransmission. RTP (RFC 3550) carries payload type, sequence numbers, timestamps, and source IDs; RTCP reports loss and jitter. Jitter buffers smooth variable delays, adapting during silences. FEC and loss concealment hide missing packets. SIP with SDP sets up calls, including VoLTE. WebRTC provides browser real-time media with ICE, STUN, TURN, and DTLS-SRTP encryption. SFUs with simulcast scale group calls; DiffServ prioritises media in managed networks.
Flashcard
Total delay from speaker to listener
Check your understanding
Why do video-call apps usually send media over UDP instead of TCP?