How authentication on the web has evolved in the last 20 years - Part 2

Passwords are insecure, and in-memory session storage doesn't scale. The second part of this series describes how I have seen developers solving these two problems.

The era of in-memory session storage

When my team started using in-memory session storage to remove database bottleneck, unknowingly, my team had hit upon a solution that lasted for a long time (and continues to be used by many companies). The standard operating procedure with Memcached (and later Redis) was simple- The in-memory cache/database stored session and login information. Applications and services queried the session service on every request. Things were terrific; Facebook contributed a lot to Memcached.

Redis became very popular for storing sessions because it provided disk-backed storage. Even if the in-memory database crashed, all customers didn't have to log in again (which was the case with Memcached)

Facebook and the rest of the industry soon realized these session stores were the Achilles' heel for their evergrowing empire.

Password insecurity

My friend Sanjog reminded me that the excellent developer community solved the dictionary attack problem for /etc/password nearly 35 years ago with /etc/shadow. Unfortunately, I got to see many platforms that were vulnerable to this attack well after the year 2000.

Developers and security experts knew they had to do something about passwords. One of the earliest attempts I recall was from RSA (Wikipedia tells me that RSA tokens came out in 2006, I almost forgot it was called keyfob). They sold hardware dongles, and I saw some of my friends using them to access financial information. RSA hardware tokens were my first introduction to OTP (one-time-password).

I didn't use the RSA dongles myself, so I don't know if they were part of a multi-factor authentication paradigm, but I am sure they did inspire the industry.

Single Sign-On

Another way the industry has tried to get rid of password security (partially) is by introducing various single sign-on methods. OAuth2 and SAML are popular technologies that allow this in entirely different environments. B2C (business to customer) environments prefer OAuth2, and B2B (business to business) environments often prefer SAML integrations.

In 2002, I met a prominent Indian media conglomerate with 34 island properties and 34 ways to log in to each property; unfortunately, the single sign-on project didn't go through. Sometimes I think about how solving that problem in 2002 could have taught me so much.

Between 2003 and 2007, I had moved to telecom and only heard about OpenID from developer websites like Slashdot. For some reason, I thought Yahoo got the short end of the stick with the OpenID project, and their engineers were responsible for implementation. While OpenID is not very popular, and even big giants like Facebook are not part of the project, a similar system OAuth (and now OAuth2) has become extremely popular.

Until I looked up the details for OAuth for this post, I didn't know Twitter was a big player in the OAuth journey. I can still recall how early Twitter clients asked users to enter their user names and passwords and how bad apps could use the credentials for nefarious activities.

Sessions, move over. JWT to the rescue

At this point, the developers decided it was super costly to have all services (or centralized API gateways) hitting in-memory session stores for all requests. What's the best way to avoid this lookup?

Interestingly, the solution goes back to the naive implementation of sending user id, logged-in state, etc., in cookies or headers. Applications sign the payload and ensure someone can not tamper with the login id and logged-in status like they could in plain text cookies.

JWT or JSON Web Tokens standard first came out in 2010 officially. I remember failing an interview a couple of years later for not knowing about the technology. My knowledge was limited to HMAC, one of the available options for implementing JWT integrity.

With JWT, a token once issued by the login service is valid for a custom-defined duration, and everyone receiving the token can validate the token to allow access. It was surprising for me to know that the first version of Angular2 didn't even implement cookies, and JWT was the only way to authenticate by default.

JWT is not free of issues

JWT makes it challenging to implement features like logout and "log out of all devices" it's also tricky to invalidate selective tokens for blacklisting users. But competent engineers have found some great solutions.

JWT has tried to solve cookie protection by using HttpOnly cookies; this was an old trick that developers started using for security session cookies and is equally effective for JWT. My security friends tell me that XSS, XHR Response Chaining, and other exploits exist, but this is an excellent first-level defense.

I have had instances where JWT's (or cookies carrying them) became too big for our CDN or API gateway. It's natural for my team and me to think about opaque sessions at these times because large JWT sizes contribute to request latencies. The central session storage doesn't look too bad with API gateways in place.

We (the developers) found a way to store extra information with sessions; I see developers applying the same poor practice to JWT. It's very tempting to send PII (Personal Identifiable Information) in JWT, and for most compliance and security practitioners, PII in JWT becomes a nightmare.

Another thing that bothers me about JWT is that we have two choices to validate tokens: 1) Share JWT secret with all services so that they can verify the token. 2) Share JWT with CDN edge or API gateway so that JWT verification is possible outside of the application. I'm not too fond of either option. From what I have read so far, I like what Netflix has done with its passport mechanism.

OTP and its cousins

While my first brush with OTP was with RSA tokens, I now regularly see SMS OTPs, Email OTPs, and even URL OTPs. SMS or Email OTP is an excellent path to go password-less.

My favorites are HOTP and TOTP. Hardware tokens used HOTP or HMAC based OTP, and I have also used a variant of HOTP to simulate signed URLs.

I haven't used TOTP in production, but the technology and standard are fascinating.

Future looks interesting

I don't have much direct experience with biometric authentication, but I am confident we will see it being used in online services too. Biometric + multi-factor authentication that Edna Mode from The Incredibles used might be unusual, but I expect to see a combination.

While I can't claim to predict the future of authentication, I expect to see new methods that make authentication faster. Even newer methods that plug the security gaps, the faster methods exposed would follow, and the cycle would repeat.

Astute readers would point out that I am talking about authentication here but sometimes get into authorization territory. I wish I could keep them separate in writing or coding consistently.

While writing this article, a friend told me that there are now standards for device authentication- I haven't used FIDO2, WebAuthn-2, but they are a step in the right direction.

Tell me about your authentication stack. Are you using JWT or central session storage? Are you using password-based or password-less logins?