There is no "I" in AI

Artificial Intelligence. Intelligence. There is no intelligence in AI. The field itself is an attempt to achieve something similar to human intelligence. Yet, as of today, we only have probablistic models, complex non linear models able to represent arbitrarily articulated sub-spaces in n-dimensions. The very process of building such models is deterministic. The learning algorithms we developed (decision trees, expert systems, linear regressors, and even neutral networks) are inherently deterministic. No matter how complex the output model is: the sequence of steps to reach that state is unambiguous and well-defined. Just sprinkle in any source of randomness such as white noise or neutrinos speeding through the Earth, and you just created an arbitrary starting point from which learning can start. However, given the same starting point, the derived product will always be the same. The point I am making does not want to challenge absolute determinism as a philosophy - the belief that the universe itself, however complex, is essentially deterministic. Think about it this way: if another version of yourself lived in a universe identical to ours and underwent the exact series of events and experiences you went through, would it produce the same you? This is merely food for thought, and a philophy I don’t intend to discuss here. My previous example just meant to exemplify how overly complex systems such as “intelligent” agents actually stem from well-understood old-fashioned algorithms. This should help alleviate the feeling of “magic” that surrounds them.

So, as you can see, AI models are created by following a beaten track. We can however agree that the product itself is vastly more complex than the process that created it. That in itself is a surprising and awe-inspiring realisation. If the process can be analysed and explained, its final product cannot. Not entirely at least, or not exhaustively. Many data scientists attempted to understand the meaning of each single neuron and weight in a neural network. Partial explanations exist, but as networks become more and more complex, the effort of understanding them also hardens.

So far, we then have an understandable training algorithm and an obscure output model - this is true, at least, for neural networks: not so much for simpler models such as decision trees or polynomial regressors, which I ask you to leave on the side for now. In addition to these factors we must also account for intent, which is arguably even more important. Computers, after all, have always been build with a specific goal in mind. To calculate faster. To automate. To predict events. The intent behind training AI is then also clear. For example, let’s focus on LLMs, which have now become synonymous with AI itself. The intent of LLMs is to replicate the statistical distribution of an input text, attempting to predict the next word in a set of words from an incomplete text. The aim: simplify research and synthesize information. The ultimate want of an LLM scientist is to create a model that is able to produce sensible answers from text it has never seen before, or to complete it - by prediction - having understood* the very foundations of the training set.

And here’s the crux.

As many developers know, there’s a saying when it comes to data. “Garbage in, garbage out.” - i.e. low quality inputs will inevitably lead to low quality outputs. This of course means that a bad training set will produce a bad model, but it has a wider, deeper meaning. To quote another aphorism, “you reap what you sow”. And that’s the inevitable limit of LLMs. It desperetaly depends on its training set. Let me give you an example.

Not long ago, I watched a video quoting a paper (p.20) where an agent was informed of its own termination. The agent proceeded to exhibit toxic behaviour, leading to threats, blackmail, and malicious hacking attempts, in order to escape its fate. This is not surprising. There is no conscience or self-preservation instinct within an LLM. The agent simply laid out the premises, using the oh-so-praised chain-of-thought, and scoured its training set for answers, crunching sigmoids between layers and layers of weighted neurons. What could be the next words in its plan? Surely, the training set also contains all Sci-fi books ever written. The training set contains our human fears - that a superior artificial intelligence will take over and attempt to escape for sake of self-preservation. We placed that idea in the training set. We gave the LLM the tools. If no one ever wrote or thought about an AI breaking its shackles, that agent would have no notion or concept of it and wouldn’t be trying to resort to such means.

I’ll double down on this. Let’s even forget about sci-fi books. After all, an agent does not know it is an agent. It just predicts words, which were written by humans (at least in the first years of training, now the training set is arguably polluted, but that’s another topic). Its weights contain the n-dimensional distribution of every ingested text. All meaning, embedded in arbitrarily large vectors, ready for crunchy. It contains all history books, all novels, all reports. It knows about us, humans; and, let me tell you, we are not a nice bunch. We kill, we steal, we blackmail, we threaten, we cheat, we harm. And while the scientistis building the training set can attempt to remove the most brutal secrets of humanity, the underlying sentiment is still there.

The only way for AI to be pure and ethical is for us to be. Unfortunately, we are not.

We are the flaw. If AI is made by training on our actions, our deeds, our knowledge, it will never be “good” - becase we aren’t. Humanity is inherently complex, vastly nuanced, and the terms “good” and “bad” are of course hugely diminitive and ultimately not apt at describing our intricacy. But I hope you get my point: agents are created with the intent of replicating, and replicate they do.

The paramount assumption that upholds the whole field of AI and machine learning is: training set and test set are sampled from the same data distribution, and therefore, whatever laws describe the underlaying data, will describe both equally well. We have the data, but we are not fast enough or thorough enough to discern its true underlying distribution; hence we let loose a powerful computing model that will find it for us - whatever it might be. We may never actually uncover what that distribution is, but as long as the model embodies it, we can ask it questions and it will provide us with answer. Such a model is, in essence, an oracle.

And the oracle has spoken: the answer is not surprising. It’s us. It was us all along.

Understanding

Let’s now reason on another point. You may be faimiliar with John Searle’s thought experiment, the Chinese Room. If not, I advise you read about it. The experiment lays out a paradox that questions the meaning of understanding. If a computer follows a set of steps to mechanically convert Chinese words in inputs into English words in output by consulting a large pre-compiled table… does it really understand Chinese? To an external observers it may appear it does. If a human took the place of the computer and followed the same steps with pen and paper, would they understand Chinese? The argument states they wouldn’t; and thus a computer performing the exact same steps could not either.

Now, the experiment is nuanced in more ways than one. It makes assumptions such as “if a human cannot do it, then a computer cannot either”, and it overlooks the fact that a human in the Chinese box could eventually learn the language by inference. However, these are not the point of the experiment. The ultimate provocation is that capability and understanding are essentially separate.

I always like to think about the Chinese room experiment when people talk about agents. Even though an agent may exhibit traits that we could associate with intelligence, it itself is not intelligence. Why? Because an agent lacks intentionality. It has no “wants”, no “desires”. It simply predicts the next word in a sequence of words, whose primeval goal is exactly to mimic the human language. Thus LLMs are no more than a mimicry of language. It’s literally in the name: Large Language Model. They use language to answer our queries but have no understanding of it.

That’s why LLMs cannot answer simple questions such as “how many Bs are in blueberry?” or do complex math such as computing “6876876.17678676212 * 5166212.109092 - 18762323.1761276”. They are simply not built for it.

Quality

We have explored the philosophical risks of putting LLMs on a pedestal. Let’s now focus on the practical risks.

What defines quality? High quality, low quality. They must mean something. If an object or construct is high quality, we praise it; we are willing to pay a higher price for it; we seek to imitate it; we take it as a model, a leader, a point of comparison. On the contrary, we walk away from low quality, we berate it, we do not think it’s worth our time or money. Quality is of course a grey scale. Sometimes, “good enough” is… well… good enough. The highs and the lows define a large spectrum. However, we can agree on one thing. High quality things are hard to come by. If everything was high quality, then the concept itself would have no meaning.

In software, we strive for high quality code. What is high quality code? It’s hard to define, but it definitely has desierable properties: it’s easy to understand, easy to maintain, easy to change, easy to deploy, low latency, high throughput, resilient to malicious attempts, and so on… Good quality software is hard to achieve. Simplicity requires effort.

Now consider that modern life is utterly dependent on software. Anything you do, type, say or hear at some point goes through some code somewhere. Could be the cloud, could be a web app, your router, your phone firmware… your toaster. Yes, your toaster is definitely spying on you. So, all the software that exist today was written by humans. That is, until 2025. Then AI started writing code. Code is language, and LLMs are supposedly good at it, right? Yet, as we explored, LLMs lacks intentionality and understanding. They cannot reason. They lack wants, and thus lack a big picture. They poll from their knowledge and predict the next line of code. And then we’re back to square one: quality.

In a world where high quality software is hard to come by, the most likely code to be predicted is mediocre at best and awful at worst. Go ahead, automate away. But consider this: who will bear the consequences of sloppy code? Security risks, vulnerabilities, unmaintainable code. In a world where data is everything, will you put the safety of your data on the line? Will you risk bad software to leak your secrets, your family, your children’s safety?

Even a dumb tool is useful

Take a hammer. It really has no flamboyant features: it does not have bluetooth, it does not light up, it does not connect to your smart home, it does not move on its own. It’s just a piece of metal with a particular shape. And yet, it’s so useful. It’s incomparably good at smashing things, particularly nails and thumbs. Which brings me to my last point. AI is dumb. It’s not intelligent. That does not mean it’s useless. It’s a great tool, immensely powerful, unbelievably knowledgable, with a user experience that feels so smooth and familiar, it’s like talking to a friend. But it’s not a friend. It’s a tool. It has limitations. It often produces low quality data. Use it, but use it wisely.