Friday, May 15, 2009

PCI & Content Delivery Networks

Here's an interesting, but commonly overlooked, little security nugget.

If you are running an e-commerce application and rely on a Content Delivery Network (CDN), such as Akamai, beware how your customers' SSL tunnels start and stop.

I came across a scenario in which an organization-- who has passed several PCI Reports on Compliance (RoCs)-- used Akamai as a redirect for their www.[companyname].com e-commerce site. Akamai does their impressive geographical caching stuff by owning the "www" DNS record and responding with an IP based on where you are. They do great work. The organization hosts the web, application, and database servers in a state-of-the-art, expensive top five hosting facility. Since it's known that credit card data passes through the web, app, and database tiers, the organization has PCI binding language in their contract with the hosting provider, which requires the hosting provider to do the usual littany to protect credit cards (firewalls, IDS, biometrics-- must have a note from your mom before you can set foot on-site, that sort of thing). And the organization with the goods follows all appropriate PCI controls, obviously, as they have passed their RoC year after year since the origin of PCI.

Funny thing ... it wasn't until some questions came out about how SSL (TLS) really works under the hood before a big, bad hole was discovered. One of the IT managers was pursuing the concept of Extended Validation certs (even though EV certs are a stupid concept), and an "engineer" (use that term laughingly) pointed out that if they purchased the fancy certs and put them on the webservers in at the hosting provider, they would fail to turn their customers' address bars green. Why? Because of the content delivery network.

You see, SSL/TLS happens in the OSI model before HTTP does. That means a customer who wants to start an encrypted tunnel with "" must first look up the DNS entry, then attempt SSL/TLS with them over TCP port 443. This is important: the browser does NOT say "Hey, I want '', is that you? Okay ... NOW ... let's exchange keys and start a tunnel."

In this case, as Akamai hosts the "www" record for "", Akamai must be ready for HTTPS calls into their service. "But wait ... " (you're thinking) " ... Akamai just delivers static content like images or resource files. How can they handle the unique and dynamic behaviors of the application which is required on the other end of the SSL/TLS tunnel?" The answer to your question is: They can't.

On the one hand, the CDN could refuse to accept traffic on port 443 or just refuse to handshake SSL/TLS requests. But that would break transactions into your "" URLs.

On the other hand, the CDN could accept your customers' HTTPS requests, then serve as a proxy between your customers and your hosting providers' web servers. The entire transactions could be encrypted using HTTPS. But the problem is the CDN must act as a termination point for your customers' requests-- they must DECRYPT those requests. Then they pass those messages back to the hosting provider using a new-- and separate-- HTTPS tunnel.

Guess which option CNDs choose? That's right-- they don't choose to break customers HTTPS attempts. They proxy them. And how did this particular organization figure that out? Well, because an EV-SSL cert on their web server is never presented to their customer. The address bar stays the boring white color, because the customer sees the CDN's certificate, not the organization's.

Why is this statistically relevant? Because a malicious CDN-- or perhaps a malicious employee at a CDN-- could eavesdrop on their HTTPS proxies and save copies of your customers' credit card numbers (or any other confidential information) for their own benefit. The CDN gets to see the messages between the clients and the servers even if only for an instant-- the classic man-in-the-middle attack. An instant is long enough for a breach to occur.

The moral of this story? 1) Learn how the OSI model works. 2) Don't overlook anything. 3) PCI (or any other compliance regulation for that matter) is far from perfect.


Neil Turton said...

I take it PCI stands for Payment Card Industry rather than Peripheral Component Interconnect - it might be worth spelling that out. I can't believe I'm the only one who got confused by that.

Anyway, I must be missing something. Before the EV certificate issue came up, the web browser would say "I'm SSL version x.x, please send your certificate". The CDN would then say "Here's the certificate for". If it didn't then the browser would show a broken lock symbol. This means that the the CDN must have a separate IP address for each company it hosts. Given that, can't it just proxy the SSL session at the TCP level? That way you'd get end-to-end security.

That said, I take it they didn't. Like they didn't say "You have to use and direct the DNS entry for to your own servers".

My next question is: How did Akamai get hold of the SSL certificate for Is this just another case of SSL certificates being too lax to be useful?

Cheers, Neil.

pci said...

Thanks for the information.