April 2021
Why renewable energy data is still flowing the wrong way
To unlock the true value of digitization in energy, we need to rethink our approach to data exchange
Co-founder and CEO, AMMP Technologies
When starting AMMP three years ago, both the gap – and the opportunity – seemed clear. Despite a great deal of digitization in the world of (renewable) energy, most of the actual data lived in siloes – with few ways to easily access it, or get business value out of it. We wanted to work towards a future where the open exchange would allow a much greater level of value creation.
Indeed, over the past years we’ve seen a flurry of progress in this space. In the world of PV, for example, most major inverter vendors now provide data access via REST APIs, with many new ones having come online in the past year or so. Despite some kinks – like limited standardization among these APIs – this has been a step in the right direction.
Yet, REST APIs cannot underpin the world of truly smart energy that many of us foresee. To bring this smart future to life, we need a much livelier exchange of real-time energy data and events. We need to move past the request-response pattern inherent to REST APIs, and embrace publish/subscribe (“pub/sub”) data flow models.
This is the subject of a white paper we have published on this topic, and something I would like to briefly introduce in this post.
Assuming you use instant messaging, you probably already use pub/sub on a daily basis – albeit in an abstracted form. But it’s in the context of machine-to-machine communication where the pub/sub paradigm truly shines.
Here’s a crude analogy: Your home appears to have a leaky water pipe.
The “REST API version” of solving this problem involves a number of request-response actions. Such as googling “plumber near me”, checking out a few plumbers’ websites, calling the ones that seem competent, and finding someone who is available and can do the job for the right price.
On the other hand, the “pub/sub version” is to subscribe to a stream of data and notifications that contains all the latest plumber info: listing information, any price updates, availability, potentially even live plumber movements in your neighborhood (the…Uber for plumbers?).
For you, a human that needs their leaky pipe repaired, the “google and call around” approach probably seems just fine. In fact, you probably don’t want to be bombarded with up-to-the minute updates regarding your town’s plumbing scene.
But what if we could outsource the job to a machine? Would it not be better to have Siri or Alexa actually subscribe to that live stream of updates, parse the incoming information and instantly find and call over the best, fastest, cheapest, plumber to fix your leak?
These days, pub/sub protocols are rarely of interest to us humans – it’s just not the way we expect to work with information. Indeed, pub/sub emerged in applications where machines need to exchange real-time data with each other – applications such as financial trading, or telemetry for oil pipelines. The pub/sub pattern is hugely powerful when it comes to machine-to-machine interaction – and especially when we look at energy applications.
In practice, most renewable energy data currently flows based on one of two models: Push or Pull. Push involves a provider (a vendor or a plant) pushing new data to clients as it becomes available. Push has been quite popular historically, but is messy to set up, and difficult to scale. Pull, on the other hand, involves a client requesting data – in the most common case, from a REST API. This is more user-friendly – and is quickly growing in popularity – but has at least three fundamental limitations in terms of data flow.
While our white paper goes into detail on these, the most glaring drawback is that the Pull method can’t handle real-time data exchange. The client only receives data if they request it; there is no mechanism for a provider to notify the client that new data is available, or that a time-critical event has occurred. So while Pull and REST APIs will likely continue to be useful for tasks like monthly reporting, they are not the answer when it comes to making our energy systems truly smart.
In many ways, pub/sub combines the best of the Push and the Pull worlds – while going easy on complexity and tight on security. In pub/sub, the data providers connect to the broker, as do the clients. A provider can publish data to relevant topics on this broker – such as “system1/inverter3/power”. Clients subscribe to the set of topics they are interested in, and receive all incoming data for those topics as soon as data comes in. That’s basically all there is to it. The providers share the data they have, while clients get what they need – in real time, securely, and with minimum hassle.
Given the advantages of pub/sub, broader adoption appears inevitable – though the pace of this is still tepid. In the world of renewables and PV, so far Victron is the only major vendor to fully implement a pub/sub interface, with some others taking initial steps.
We recognize that progress is often gradual. Yet the value that pub/sub unlocks for both data providers and data users cannot be overstated. And the benefits of broader adoption for our ecosystem are likely to be immense. Already, influential institutions like the Eclipse Foundation are pushing for standardization on pub/sub communication in Industrial IoT – as part of a broader drive towards true interoperability in the space.
The tools are already in our hands. We just need to put them to use – in order to shape a cleaner, smarter, better tomorrow.
[For further details on this, including in-depth technical coverage, see the white paper.]
