The State of Dark Data in the Energy Industry

When we last wrote about dark data and its energy implications, we discussed how dark data is rarely leveraged to its fullest potential, as well as the roadblocks hindering its collection across industries. Let’s take a deeper dive into the current state of how dark data is used in the energy world, how lessons from another market might help energy, and understand how dark data can be harnessed for greater efficiency within the energy industry

In case you need a quick refresh, dark data refers to the data that organizations gather or have available to them but don’t actually use to its full business potential to generate revenue. That can include everything from email correspondence to log files to security video footage to that old filing cabinet full of contracts. All of those data points could potentially be harnessed to generate key insights, but instead , they sit idle and unused. With all that locked potential, dark data could play a much larger role in the energy industry. We’ll look at 3 different areas of the industry for their opportunity.


How Dark Data Will Improve the Grid


Dark or not, utilities and ISOs make data-driven energy decisions every day. Energy’s complex system requires incredible amounts of data on demand and supply to be optimized in real-time for smooth operations. Increasing sensitivity to grid security, questions about reliability due to the influx of distributed resources and a focus on resilience in the face of climate events are increasing the need for investment in new monitoring tools at the utility and ISO level. 

The growth of IoT-connected devices is also driving a big data boom in the energy industry. Smart meters, cloud computing, and other emerging technologies are making data collection and analysis easier than ever, leading to greater efficiency and a reduction of maintenance and operation costs along the energy value chain. But tons of useful information from these devices go underutilized. 

Take the example of smart meters. According to a study from the American Council for an Energy-Efficient Economy (ACEEE), many utilities that have deployed smart meter technologies have yet to maximize the value of dark that they collect on these devices. This prevents these utilities from providing optimization services and better managing demand on their own grid. 

Dark data will have to play a central role in optimizing grid operations and resources in the future. Other industries have tapped into the potential of complex data analysis — why not energy?


Data Usage in Mobility

Mobility, for example, has been greatly transformed by increased data availability. The emergence of data-driven ride-hailing apps, like Uber and Lyft, has fundamentally shifted how people get from point A to point B. Leveraging data from local traffic patterns, GPS, public transit routes, and internal data about the length of the journey, Uber and Lyft can provide better services to drivers and passengers. This is already a great gain in mobility data. Yet Lyft has taken a step further, using broader data (such as gender and socioeconomic status) and crossed data points to provide a mobility study during the COVID-19 pandemic. These kinds of insights ultimately help city managers and local officials with better decision-making processes.

Imagine for a moment that we were able to know more about the customer’s energy consumption behavior and better predict their demands, especially during a crisis.

Better Understanding Your Customer with Dark Data


We increasingly know more about residential energy customers with certain technologies becoming more widespread: smart meters, smart thermostats, smart home equipment, and increasingly EVs. Service providers like Tesla, Nest and Amazon help these customers monitor energy usage and display it in easy to understand interfaces. By collecting their real-time demand data and understanding more about consumption behaviors, these providers can sell services to utilities that manage load or even reduce demand to provide “grid services” as well as regular consumer value. 

But the dark non-energy data in this scenario is still not unlocked. Consider for a moment that each of these service providers don’t know whether they are servicing the same customer. Or take it even further and share that data with the energy supplier and utility servicing that customer as well. How might the ecosystem benefit from a richer understanding of the energy consumer? 

A holy grail for the industry is the ability to predict load. By sharing information about when customers get home from navigation apps or when they use smart devices based on smart-home applications, utilities and suppliers will be able to get more accurate pictures of needs. Something as simple as knowing whether a person got a new Nissan Leaf can be game changing for infrastructure planning.

Of course, we can use past trends to predict future trends, but harvesting unused data to balance supply and demand is key to ensuring energy reliability and efficiency. In the near future, using smart meter data and allowing more devices to share their data can become a critical component to energy efficiency and demand response but only if service providers have better control and understanding of that data.

Just as important in this equation is learning how to share all those data insights in a cohesive system, allowing IoT-enabled devices to communicate with greater efficiency, with a dash of AI. Currently, there’s no consensus on how connected devices communicate. Nor are there incentives for data owners to share their data. Data owners, suppliers, and producers should be incentivized to share their energy data, creating opportunity for deeper insights to be unearthed by other energy solution providers. 

We recently also discussed the potentially game-changing impact of Tesla’s new retail energy arm on how data access could make the next generation of retail energy suppliers.

dark data energy usage

Energy Retailers Should Keep Its Eye Towards Transformation

Today, data is leveraged across energy generation, trading, transmission, distribution, and consumption. There’s a general consensus on how this existing data can be further leveraged for future insights and giving customers greater services, as ongoing energy digitization and decarbonization creates vast opportunities for further data usage. But it’s hard to determine what data is missing or unavailable when most available energy data goes unused to begin with. 

Take the simple example of helping customers better understand the type of energy deals that they are signing up for with energy suppliers. Often this process is full of dark data! Customers might not fully understand or know key energy risks or components that might be exposed to market fluctuations. This isn’t merely a problem of transparency but also of wasted data.

Saavy customers and energy consultants will recognize that where there is risk there is opportunity. By sharing with customers details about the types of risks they might be exposed to in an energy contract, you have the ability to sell them on more advanced offerings like demand response, battery storage or on-site solar. This can build you credibility and you also serve as a one person army helping bring more data to light in the ecosystem. 

Innovative players in the energy space (like Dash Energy) are working to go beyond that, building tools that make extracting, aggregating, analyzing, and sharing data easier than ever before. The more data is created, the more important data digitization, analysis, efficiency, and modernization becomes. And the more that data can be utilized by the customer for their advantage.


The energy industry needs to leverage the tools at its disposal to make the most of the data it generates.


When data analysis is pushed to its fullest potential, it is a major driver for energy efficiency — we just have to keep pushing it to new levels of discovery and intelligence.

by | Sep 28, 2021

Templates, ebooks, articles…

Once a month, we’ll email you a list of resources and insightful information about the energy industry. 

The Henry Hub and its Role in Natural Gas Pricing

The Henry Hub and its Role in Natural Gas Pricing

Besides being a source of natural gas delivery, the Henry Hub plays an important role in the financial markets, too. Since 1990, the hub has been the physical delivery point for the world’s most-traded natural gas futures contracts on the New York Mercantile Exchange.

read more