The New Data Stack: Notes from the Frontline of Next-Generation ETRMs

AI is great. AI sucks. Most days, you feel both before lunch.

That was my honest opening when the conversation turned to artificial intelligence on yesterday’s Energy Trading Digitalisation panel — The New Data Stack: Real-Time Intelligence and the Next Generation of ETRMs. I shared the screen with Nate Branscombe (COO, Six One Commodities), Carl Lidholm (EVP Energy & Utilities, Hansen Technologies), Chris Rees (Partnerships Manager, Meteomatics), and Sajindra Jayasena (Chief AI, Digital & Information Officer, Targray).

Across forty-five minutes we covered architecture, AI, governance, and where ETRMs are actually heading. Here are the threads that kept coming back — and what they mean for anyone running, buying, or building energy trading software.

1. The data problem has changed shape

Energy firms have always had data challenges. What’s different now is not volume — it’s velocity, variability, and the cost of being wrong.

Sajindra opened the panel with this framing, and Carl backed it with hard numbers from the European market: intraday imbalance prices in Germany have hit five figures per MWh. The penalty for stale data is no longer theoretical. It is measured in real money, in real time.

Add to that the structural shifts — renewables turning weather into a first-class price signal, behind-the-meter flexibility entering the supply stack, auxiliary markets growing across Europe, and IoT telemetry flooding in from every connected asset — and the case for redesigning the data architecture writes itself. The systems most of us built ten or twenty years ago, with batch ETL and end-of-day risk runs, simply weren’t built for this.

2. Event-driven is a promise, not a feature

Event-driven architecture is the buzzword of the moment. React in real time. Process on trigger. The pitch writes itself.

My pushback on the panel was simple: the trigger fires when one signal arrives, but a real decision usually depends on five or ten others. If those haven’t landed yet, you didn’t make a fast good decision — you made a fast bad one. And as Carl pointed out, fast bad decisions in today’s market are punitively expensive.

The fix isn’t to abandon event-driven design. It’s to be honest about dependencies. Know which data has to be present before a given event is allowed to cascade. Sometimes the right answer is a scheduled run that guarantees completeness. Often the right answer is a hybrid — event-driven where it makes sense, scheduled where completeness matters more than latency. Sajindra noted that capital markets has been wrestling with this for a decade in systematic and high-frequency trading. Energy is now living the same problem.

3. AI is the world’s best power tool. Treat it like one.

Here’s the part where I have to be direct, because a lot of money is being wasted right now on AI initiatives that won’t survive contact with production.

AI is not a panacea. It is the best power tool we’ve ever had — it identifies patterns and synthesizes information at a speed humans cannot match. We use it across every part of our business at ennrgy.ai, in our products and to build our products. But the moment you let it run unsupervised, it runs amok. The firms succeeding with AI are the ones treating it as augmentation with governance, not turnkey replacement for judgment.

Chris made a great related point about MCP connectors changing how trading systems pull weather data — customers asking questions like “is there a day-similarity index for thundery showers?” and getting a meaningful answer in minutes. That’s the upside. The downside is the same architecture lets bad prompts pull bad data, fast, at scale.

And one trap nobody talks about enough: if you supercharge each silo of your organization with AI, you may be driving them in opposite directions. The integration problem doesn’t go away because you bolted GenAI onto each piece. It gets worse. Carl made the same point a few minutes later. End-to-end matters more than pointwise speed.

4. Governance isn’t a kill switch you bolt on

The kill-switch moment was the panel’s most quoted exchange. I told the story of a trading customer running three bots — two third-party, one in-house — who asked us, as the ETRM provider, to put a single physical toggle in line so they could kill all three at once when things went sideways. One throat to choke.

Nate’s reaction was sharp:

He’s right. And my comeback stands too: when what you deployed doesn’t have one built in, you have to build your own external kill switch. Reactive governance — building controls after deployment — is still the norm in our industry, and that’s a problem.

Sajindra brought up the under-discussed threat vector of prompt injection — the LLM equivalent of SQL injection, where malicious instructions can be embedded in data the model consumes. As more firms wire LLMs into their data and trading stacks, this becomes a real production risk. Guardrails, model-drift verification, circuit breakers, human-in-the-loop checkpoints, and layered approvals aren’t paperwork. They’re the price of admission.

5. The next-generation ETRM is a connected value chain

The ETRM started as a monolith. Then it became modular. The next step isn’t “more modular.” It’s connected end-to-end, with the feedback loop closed.

That means the value chain — from Chris’s weather forecasts feeding load and price models, through pre-trade analytics, through trade execution, all the way to billing and settlement — has to flow as one stream of information. Quickly and accurately. With humans in the loop at the right moments to provide judgment and oversight, not at every step.

Carl described it as monetizing the whole energy system — from dynamic tariffs and price signals to the end consumer all the way back to weather and market intelligence. Same picture, said different ways. The boundaries between forecasting, trading, settlement, and behind-the-meter asset management are getting thinner. The firms that close those gaps first will compound that advantage.

6. Speed has a ceiling

My favorite question came at the end. There’s a tradeoff between speed and control — how do firms balance real-time responsiveness with audit and risk oversight?

My answer: at some point, you’re chasing diminishing marginal returns. Markets move quickly, velocity matters, but there’s fast — and there’s irresponsibly fast. Being the first dog to catch the car isn’t a win. Sajindra put it more elegantly: there’s no point having a Formula One car if you can’t brake into the turn. Carl noted that being among the fastest is usually sufficient — you don’t need to be the absolute fastest if it means breaking your controls.

That feels like the right place to close. The pressure to move faster is real and won’t let up. But the firms that win the next five years won’t be the absolute fastest. They’ll be the ones who built the connected stack, kept humans in the right loops, and earned the right to automate — one governed step at a time.

Watch the full panel replay on the Energy Trading Digitalisation event site.