What Every Programmer Should Know About TLS and the Web PKI (Part 0)

December 30, 2023

What is a TLS certificate? What is X.509? Why do you sometimes see Your connection is not private (Chrome) and Warning: Potential Security Risk Ahead (Firefox) when you visit a website? Why do you need to obtain a TLS certificate when you deploy a website? And who are these people giving out certificates?

In this series, I will attempt to answer these questions and more. I will provide an over-simplified description of the Web's Public Key Infrastructure (PKI)—the dark corner of the Web concerned primarily with authenticating web servers to web clients like browsers. Concretely, I will discuss Transport Layer Security (TLS) certificates—their role, how they are created and distributed, and how they are validated. My goal is to provide you with an accurate (enough) model with which you can better understand the importance of TLS and TLS certificates in the Web today.

I have certainly made mistakes. Please email me any corrections at james@.

Prologue: Why do we need TLS?

I start with an overview of what TLS is and why we need it. This is required background for discussing TLS certificates and the Web PKI in general.

HTTP requests are insecure

The World Wide Web is built around the HTTP protocol. Web browsers like Firefox, Chrome, and Safari make HTTP requests to web servers on your behalf when you visit a website's URL (e.g., http://example.com/). In response, web servers return website data in the form of HTTP responses. HTTP responses often include HTML that the browsers parses and renders on the screen. More generally though, web clients and servers can include any kind of data in HTTP request and responses. For instance, programmatic Web APIs like http://api.example.com/ may return JSON data. And there are plenty of non-browser web clients out there, such as curl.

Web servers are programs running on a computer connected to the Internet. Specifically, a web server is a TCP server running on port 80 (typically, but not necessarily) that can respond to the HTTP-formatted messages placed in TCP packets sent by web clients. When a web client makes an HTTP request to http://example.com/, it first translates that URL into an IP address like using DNS, then establishes a TCP connection with the server advertising that IP address, on port 80. It then sends an HTTP request over the TCP connection and receives the HTTP response over the same connection.

Without TLS, the browser's HTTP (over TCP over IP) requests and the web server's HTTP responses are sent in the clear. This is a big security problem. It means a motivated attacker can intercept the connection, read the packets, and even modify packet contents. So an attacker that intercepts your connection to http://gmail.com/ could not only read all of your emails, but they could also modify your emails (not on Google's servers, just the emails that are rendered in your browser). Worse still, if you send an email, they can change (or just drop) your email, making Google think you said things you did not actually say. This class of attack is called a monster-in-the-middle (MITM) attack.

TLS secures HTTP requests

TLS prevents MITM attacks by adding security to HTTP connections. When you visit a website using TLS (which means you type https://example.com/, note the s for Secure), your web client establishes an encrypted communication channel with the web server. TLS stands for Transport Layer Security because it secures the transport layer, in this case TCP, that the application layer, in this case HTTP, uses.

To see how TLS works, recall that when your browser makes an HTTP request to http://example.com/, it first establishes a TCP connection to web server, then sends its HTTP request over that TCP connection. In contrast, when your browser makes an HTTPS request to https://example.com/, it still establishes a TCP connection, but before sending an HTTP request, it establishes a TLS connection atop that TCP connection. Only after the TLS connection has been established does it send the HTTP request over that newly secured channel. On the server side, HTTPS web servers (typically running on port 443) are still TCP servers, but they are also TLS servers. They know how to establish a TLS connection with an incoming client after a TCP connection with that client has been established.

The process of establishing a secure connection using TLS is centered around cryptographic key exchange. When the client tells the server that it wants to establish a TLS connection, the server sends an asymmetric public cryptographic key to the client. The client can use this public key to encrypt all subsequent packets such that only the holder of the corresponding private key (the server) can decrypt the packets.

However, while this is a simple and useful way to think about TLS, it is imprecise. In practice, the client actually uses the server's public key to securely agree upon a symmetric cryptographic key with the server, that both use to encrypt/decrypt subsequent packets. Using a new, session-based symmetric cryptographic key has a few benefits, namely: (1) now the server can also encrypt stuff it sends client, (2) symmetric encryption is faster than asymmetric encryption, and (3) if an attacker saves the encrypted communication between the client and server and then later compromises the server's private key, the recorded session still cannot be encrypted (see: forward secrecy).

As a result, an attacker listening in on a TLS-enabled HTTP (HTTPS) connection is unable to understand or tamper with the HTTP requests and responses. TLS is so important, or rather MITM attacks are so dangerous, that major browsers today will not allow you to visit an HTTP-only website unless you click through a big warning page. You can see an example of this here.

But you could be securely talking to a monster

Unfortunately, securing the connection is not enough. True, once your browser establishes a TLS connection with a web server, no attackers can read or modify packets sent over that connection. But how can your browser know that it is actually talking to the website you requested?

To illustrate the problem more clearly, let's return to the cryptographic key exchange in the TLS connection establishment step. More precisely, when you connect to https://example.com/, your browser sends an initial TLS message to example.com's web server called the ClientHello. The server then responds with its public key, which your browser uses this public key to encrypt communications. But consider that an attacker could intercept your ClientHello message. They could then send their own, false public key in response. Your browser would then use this public key to encrypt communications, but with the attacker, not with example.com. To see how disasterous this is, consider that the attacker could simulatneously open up their own secure connection to the real example.com and forward the packets it receives from your browser (which has no idea it is talking to an attacker) to the real web server, reading and modifying them as they please.

The missing piece is authentication. The browser needs some way of guaranteeing that the public key it receives in the establishment of the TLS connection (called the TLS handshake) actually belongs to the website you requested. More concretely, the browser should never establish a TLS connection unless it is sure that the server's public key actually belongs to the that website. Without authentication, TLS is virtually useless since, as we've seen, monster-in-the-middle attacks are just as possible (though slightly more complicated).

TLS certificates are an authentication tool

TLS certificates bind public keys to domain names. During the TLS handshake, after your browser sends its ClientHello, the web server actually sends an X.509 certificate containing its public key, not just a public key. Crucially, certificates also contain the domain(s) that the certificate's public key is valid for. So the browser can now validate public keys (or more precisely, certificates) by checking that the domain name you requested (example.com) is present in the certificate sent by the server. If so, the browser believes that the public key actually belongs to the website you requested, and it can use that public key to establish a secure connection. If not, it rejects the connection.

Of course, it's not that simple. You yourself could create a TLS certificate for example.com with your own public key. And your browser (correctly) would reject it. So, as we'll see, determining the validity of a given certificate is much more complicated than simply checking whether domain names match.

To find out more about certificate validity, stay tuned for the next post in the series, where I will also be discussing the fundamental problem with TLS certificates and the Web PKI.