thoughts on hype cycles

Big Data. Internet of Things. Blockchain and crypto. Chatbots. Smart home voice assistants. Augmented reality. Generative AI. These are just a few of the hype cycles I've seen during 15 years in the tech industry.

Hype cycles are endemic in tech. Consumer devices and software are ubiquitous, and the companies that make them have market caps that reach into the trillions. 4.5 billion smartphone users, 3 billion Facebook users (yes, still), 14 billion YouTube videos, terabit transatlantic cables: tech operates on a scale that would have been unimaginable even a generation ago.

I mean, I remember the sound of the Internet over 33.6Kbps dial-up. A generation before that, my father used punchcards to program a room-sized computer at the University of Waterloo. A generation before that, Thomas Watson at IBM marvelled that he was able to sell 18 computers instead of the 5 he'd predicted.

With global scale comes the promise of nearly unlimited upside, the idea of making billions instead of mere millions. With nearly unlimited upside comes nearly unlimited FOMO. Hype Cycle Technology X lands, and suddenly every company and organisation needs an X strategy, regardless of whether X applies in any meaningful way to what they do. It's a sort of secular Pascal's Wager: if X does apply and we find its killer app, the rewards could be massive; if it doesn't, we sunk a (comparatively) small investment into it; how could we not make the attempt?

(At this point, I'll note that I use X throughout this post as a variable, meant to stand in for any hype cycle technology. It has absolutely nothing to do with the Social Network Formerly Known As Twitter.)

The ubiquity of digital technology also lends hype cycles the considerable force of social proof. For more widely-visible cycles, it seems like everyone's talking about it: your colleagues and management, sure, but also your friends and family outside of tech. Since the App Store opened its doors in 2008, I've had countless friends and family members approach me to talk about "this great idea for an app".

Of course, hype cycles do exist outside digital tech. Gartner conducts hype cycle research on other technical fields such as low-carbon energy technologies and environmental sustainability.

I don't work in, say, carbon sequestration - but it's a safe bet that if I did, I wouldn't get random well-meaning relatives asking if I have any great tips for carbon-neutral living.

Let's take a look at the Gartner hype cycle model, which describes five phases that hype cycle technologies go through:

If you're outside the tech industry, you're probably most familiar with the first three phases:

• Technology Trigger: new technology X lands on Earth, usually with a splashy press release, catchy proof-of-concept, or both.
• Peak of Inflated Expectations: the possibilities for X seem limitless. Well-respected pundits proclaim that X will completely transform society as we know it within the next 5-10 years. VCs scramble to throw money into X.
• Trough of Disillusionment: it turns out it's harder to apply X than first expected. Maybe X has some glaring downsides (privacy, cost, energy consumption). Maybe X is always a few last details away from full viability (self-driving cars, VR headsets). Maybe X just isn't applicable as widely as thought (3D printing, chatbots, blockchain).

The Trough of Disillusionment is where a lot of corporate research labs quietly pull funding and disband teams, and is the graveyard of purely hype-cycle-driven companies with no lasting value to offer.

Once the initial hype cycle has faded from public view, the last two phases offer an opportunity for enduring value.

• Slope of Enlightenment: the most promising technologies, teams, and companies survive the Trough and find real applications for X. Early adopters look to adopt those applications or fund further R&D.
• Plateau of Productivity: X reaches mainstream adoption. Late adopters get on board.

...which is all well and good, and in any case adequately explained elsewhere.

But now I'll ask a different question: what should you do about this as an engineer or technology leader? How should you think about it?

If you've been in this industry for any length of time, you've probably seen at least one hype cycle in action at work. Upper management starts talking about how we need to get ahead of the curve on blockchain, or chatbots, or Big Data, or generative AI, or whatever value of X is hot right now. Every team is asked to brief upwards as to how X holds great potential in their area. Money is splashed around to attract X experts to start an X innovation team / lab / hub / whatever. Even the most basic not-really-functional demos of X bring attention and praise from on high.

Meanwhile, you're shaking your head. How dumb can management be? Haven't they seen the exact same hype cycle dynamic play out again, and again, and again? Isn't it painfully obvious that X is near the Peak, and all this investment will turn out to be a fantastic waste of time, energy, and money? What about all the teams doing valuable work that isn't X-related; will they be rewarded adequately? What the hell are we going to do with teams worth of X experts when the X bubble bursts?

This cynicism isn't totally off-base, either. After all, from the Global Innovation 1000 study:

There is no long-term correlation between the amount of money a company spends on its innovation efforts and its overall financial performance...

If there's no correlation between innovation spending and outcomes, that suggests that many of the companies "getting in on X" are in fact wasting their money, while others get much more bang for their buck (or: euro, yen, kronor...)

And if you're our hypothetical engineer above - well, you've seen several hype cycles go by. Since most companies make poor investments in hype cycle tech, your median (or even p90) experience of hype cycle tech is staggering waste and unrealistic expectations. You may never have seen a company get this right!

The Innovator's Dilemma

So what's going on here? Applying the principle of charity here: what is the engineer missing that makes this behaviour make sense?

To be sure, it could be a bad case of executive groupthink. But let's contrast how the engineer thinks about hype cycle investing with how management thinks about it. In our scenario, let's say the company plans to invest $10M in X. If X is useful to the company (and they can figure out how), there could be a large upside.

The engineer sees it as a pretty straightforward bet: if you invest in X, you have a small chance of success - maybe X is useful after all, though this seems really, really unlikely to the engineer.

So it's a no-brainer, right? The expected value of investment is $-8.9M, which is less than zero, which makes it a bad deal.

Not to mention that it's really, really annoying to be passed over for promotions because you're not an X innovation team, or to see X experts hired at double your salary. The engineer might also factor these into the table as attrition costs, say another $5M worth:

Even worse: $-14.8M! Who would make this bet? Only deeply irrational executives, that's for sure.

The executive is operating on a fundamentally different set of assumptions, though:

• The Innovator's Dilemma: if X succeeds, and we're not invested, and X is disruptive to our business, it could be existential for the company.
• Skewed returns: some X will be especially high-value, and we don't know which one. Our innovation portfolio needs enough bets on different X to have a good chance of containing high-value outliers. (This is similar to VC thinking.)
• Some attrition is OK: the normal rate is 10-15%. Plus: new hires means new ideas, and it creates opportunities for promotion or internal transfer that can balance out the risk that people feel left out by X.
• Information has value: investing in X gives us information about X, and access to expertise about X. Even if our X plans don't work out now, tapping into X networks now could position us to buy a successful X startup later.

So their table looks more like this:

Due to the skewed returns at top, our investment now has an expected value of $1M, even though most of the time the return will be negative. Not only that, but if X is in that eventually useful category, there could be some future reward that's hard to estimate now.

The executive also factors in the downside risk of passing on the Next Big Thing, and concludes that the expected value of not investing is $-5.9M. Who would pass up this investment!?

So who's right?

To be fair, all these numbers are highly made up. Any attempt to estimate the value of an unproven technology, or of your company's efforts to harness that technology, will result in estimates with huge variance.

The engineer makes the key mistake of ignoring the last two phases of the hype cycle, and ignores the possibility (however small) that failing to act now makes the company irrelevant in 10-15 years. This is a classic Taleb distribution, in the face of which it makes sense to guard against ruin. They also place too much emphasis on potential attrition costs, and discount the reputation upside of making a bet on X.

The executive makes the key mistake of assigning almost limitless upside (and downside) to the disruptive category. In their mind, the numbers in this category may as well be infinite, in which case: why not invest even more than $10M? So almost any amount will appear to be a smart bet. In reality, this number is impossible to estimate with any accuracy until much, much later in the hype cycle - way out in late Slope and Plateau territory.

So both are wrong, though in different ways.

However: since this is a problem involving Taleb distributions, and VC-style skewed returns, and other such hallmarks of complex iterated games involving big systems - a competent executive here is probably less wrong. They're looking at the big picture, using sources of information the engineer doesn't have access to. They're also looking at the long term, and across multiple hype cycles for possible technologies X.

Of course, an incompetent executive is more dangerously wrong. They might invest way too much, and end up as one of those companies on the high-cost end of the lack of correlation with return. They might also fail to build a large enough innovation portfolio, in which case what investments they do make will fail more often than not, which discourages further innovation and creates a downward spiral. They might even do both! (For instance: blowing way too much cash on a select few hype cycles and ignoring others, without any purpose or strategy.)

For the engineer, the lesson is simple: as strange as it might seem, there are perfectly rational reasons to throw money at hype cycles. So what's a useful response to this reality?

One thing that remains true in engineering: you always need to be learning, at least a little bit. If there's an X hype cycle in full swing, chances are your company is hiring X experts (or trying to, at least) - so talk to them! What do they see in X? Where does X hold the most promise? What are the limitations of X? (Often technical experts will happily give this context!)

What else needs to be true to effectively make use of X? (Here I'm thinking skillsets, tools, ways of working, ways of organising teams.)

What is it like to work with X? What are the day-to-day tasks of someone who works on X? What's hard or tedious when doing practical work with X?

And remember: most hype cycle technologies eventually reach the Plateau of Productivity and become part of the tech landscape for the foreseeable future. A bit of curiosity, due diligence, and experimentation can help you decide your personal stance to each hype cycle. Some (non-exhaustive) options:

• career shift: you really want to be part of X. Maybe you'll take courses or take a master's degree, or seek mentorship, or jump into the area as a more junior hire.
• side project: you like the idea of X enough, but don't necessarily see it as a main career thing right now - maybe trying it out in a side project is a good way to test it out.
• add to toolkit: you're content to dabble with X, and see if there's anything in the approach / toolset / mindset that seems useful for your main work.
• not involved: maybe you're just not interested in X, for a variety of reasons - mismatch with your interests, moral / ethical considerations, already learning something else, etc.

For what it's worth: I actually think "not involved" is a completely reasonable default stance for 90+% of hype cycles, especially in the early stages. It's OK to focus your energies on a few topics you're interested in! Still, I like to ask curious questions whenever I get the chance, or even read up a tiny bit here and there - every now and then, this will convince me to shift from "not involved" to "add to toolkit" or "side project".