Stop trying to always use AI

Recently, I've been working with Candy on a few projects which use LLMs (Large Language Models) to help with processing.

One project has been to create a BESS (Battery Energy Storage System) data warehouse so that technicians working out in the field can get key telemetry data centralised on a single mobile application. This can give information in real time (current capacity, discharge rates, active/offline etc) and this data is also stored to give an historical picture of batteries which can then be used to look for demand/supply patterns, repetitive anomalies and reporting capabilities. This database is then connected to a front end app via hosted APIs which presents the information as clearly as possible and which Candy is currently building. I'll write about this application more in a longer post once we publish the app on GitHub as it's part of a bigger BESS renewable energy topic.

After building the data-model and the main APIs, I decided also to expose the database to a locally hosted AI model. Candy had already developed an application for Icelandic translation using a local LLM architecture and you can read about it here. Why not then, I thought, try and use the same architecture with this new application ?

I had already installed Ollama on my machine and had pulled down some of the open source Mistral LLMs and these ran reasonably well when called from the terminal.

In the Python code, I was able to add an additional series of functions that accessed Ollama easily thanks to a well documented Ollama Python library. It was surprisingly simple to first of all call the LLM from the main published API and then to get the LLM to connect to the database.

Please return me all of the batteries in the BESS database which are currently active ?

With this prompt, the LLM was able to return the relevant data in JSON format - all executed locally.

My satisfaction at the ease of doing this is, however, largely offset by the fact that I am honestly struggling to see how this sort of 'agentic' AI is really going to add any value to what our application is actually supposed to do.

Let me explain:

Firstly, this application is designed to give clear information to electrical grid engineers who are doing a stressful job. From a functional perspective, a chat-bot option seems incongruous and convoluted over just simple buttons.

Seondly, I'm running all of this locally on CPU. Candy and I are currently building a dedicated server but you might have noticed how seriously expensive all PC components have got recently (we certainly did) and to cut a long story short, the machine isn’t ready yet. What that means is that during testing on our personal computers, we experience the LLMs use far more during processing compared to just running a specific API call for the equivalent task. A bit of bench-marking on my fairly beefy Thinkpad running Linux gives me the following results:

• API Call to Database to return all data: 2 seconds
• API Call to LLM to call database and return the data : 10 minutes !!! And in that time you have to contend with your PC transforming itself into a neurotic version of Concorde.

I know, I could and probably should have spent 5 hours optimising the model, so it's not a perfect comparison - but the LLM is supposed to make my life easier, not more difficult , and I have other things to be getting on with in my life.

Likewise, I should also be running the model on a GPU. But here's the thing, GPU uses a load of extra electricity to run, it's really expensive to buy and burns out after a certain amount of processing - so my gut feel is that we should probably limit our use of this.

And then, after all that, it gave me the wrong answer !!

Please total the maximum discharge in MW of all the active batteries in the BESS database ?

The AI went off for 15 minutes, it then came back and presented me with the correct data but failed to add up the numbers properly answering 7.2 instead of 7.8. It turns out that LLMs aren't terribly good at math: simple addition in this case. Who knew ?

So this got me thinking more generally. We keep hearing about the 'agentic revolution' and how this is going to tear through the economy and destroy jobs etc. And please don't get me wrong, I suspect that there are use cases for this functionality. But perhaps, just perhaps, there are much fewer than we are being led to believe and that old-fashioned coding is still going to be a better way of doing a lot of tasks that are being earmarked for AI.

I'm leaving this LLM agentic function in the back-end of my application but I'm just going to add more APIs to deal with the heavy queries to the database to save on processing and my sanity. Candy might put an agent function somewhere in her front-end app but I'm wondering if we should put a health warning on it.