From the Big Short to the Big Freeze: Undifferentiated Data Woes

Originally published on LinkedIn Pulse.

Undifferentiated data fueled the mortgage crisis. Now it’s exacerbating Texas’s energy woes

The subprime mortgage crisis of 2007-2010 was, in large part, a failure of data analysis. By bundling subprime mortgage customers with different credit profiles together, and monetizing their debts in bulk through mortgage-backed securities, banks and other financial players obscured homeowners’ wildly differing risk profiles. The result: buyers of these securities did not fully understand the risk they were taking on until nonpayment started and everything bloomed into a fully blown global economic crisis.

Here in Texas, we’re now seeing a similar crisis unfold — not in the real estate markets, but in the energy sector. The blackouts that have rocked Texas in recent days are the result of unexpectedly cold weather, of course. While a lot of scrutiny is being placed on the generation outage, on a deeper level, just like the mortgage crisis, this is the result of using undifferentiated data to guide decision-making.

Without efficient demand analysis, we can’t price energy effectively. We also can’t send the right signals to energy producers and ERCOT to articulate current and anticipated future energy needs. And we can’t send the right signals to customers about how they should manage their energy usage during periods of high demand and systemic stress.

Innowatts CEO Siddhartha Sachdeva

Let me explain. As things stand, many energy providers today are “bundling” customers with different demand profiles to come up with load forecasts. In other words, energy consumption forecasts are based on average energy usage by average consumers. Often this works, more or less. But when you encounter unexpected events — such as a sudden plunge into Arctic temperatures — those averages aren’t as much use.

To cope with tumultuous times, energy providers need to use granular data, based not on broad historical averages but rather on bottom-up insights from smart meters and other localized data sources, such as weather and socio-economic trends. Only by understanding how different kinds of customers use energy, and how that varies according to factors such as weather or time of day, can we hope to develop rigorous, effective systems for energy management and eventually energy transition. 

To put this concept in context, Texas energy providers typically bucket residential customers into two broad groups: electric heat and non-electric heat (highlighted in blue in the graph below). Look closer, though, and it becomes apparent that there are actually at least 36 groups of different residential demand profiles present in this market.

The above costs are derived from trailing 3 year spot market prices

If you’re only looking at the electric heat and non-electric heat profiles, then the price separation between the two doesn’t seem that wide — just 19% difference between the two groups. When you look at the full spectrum of 36 groups, though, we can see that in reality the price spread is 107%. 

So what’s the impact of these two approaches? Well, if you’re only grouping your customers into two buckets, you’ll wind up with a significant number of customers being underpriced by an average of 36%. In these cases, energy providers are not including enough risk premium, and are relying too much on ERCOT spot markets. This can lead to huge collateral calls, which ultimately results in sudden bill increases for customers and default for energy providers. 

At the other end of the spectrum, some customer groups are overpriced by an average of 46%. When that happens, energy providers will lose these customers to smarter energy providers that have differentiated and customized pricing. That, in turn, skews the energy provider’s portfolio, since they’ll be left with a higher proportion of high cost-to-serve customers.

This is a simple example, but the reality is that a lack of granular data affects all aspects of energy generation and distribution. Without efficient demand analysis, we can’t price energy effectively. We also can’t send the right signals to energy producers and ERCOT to articulate current and anticipated future energy needs. And we can’t send the right signals to customers about how they should manage their energy usage during periods of high demand and systemic stress.

If we don’t understand demand drivers at the level of individual customers, we’ll never be able to cope with disruptions like the cold snap that impacted Texas. A granular understanding of customer needs can help you understand how demand patterns will change in unusual circumstances — but if you simply lump everyone together, you’ll be subject to unexpected shocks and spikes, with disastrous results. 

Of course, the blackouts that have rocked Texas aren’t solely due to poor data practices. It took a succession of errors — including subpar weatherization, incorrect pricing plans, lack of customer communication, poor maintenance, and bad planning — to spark this crisis. But with better data and more granular insights into energy demand patterns, it would have been far easier to make the right infrastructure and maintenance investments, smooth out energy-demand peaks, and make better decisions.

This isn’t the first time the U.S. has faced blackouts and failures of energy infrastructure. With the world warming and extreme weather growing more common, it certainly won’t be the last. We need to get to work now to build more resilient and sustainable energy systems — and that means not just tougher pipes and turbines, but also better data intelligence and analytics to help us make smarter decisions in a turbulent and rapidly changing world. The 2008 financial crisis taught us a hard lesson about the need for granular, well-differentiated data. It’s time for us to apply those lessons to the energy sector, and ensure we have the tools and data we need to effectively serve consumers.

Join our mailing list to to get notified of
new case studies, white papers, news and events.