The most impressive ability of AI is the quality of the conversation you can have with it. Not for the first time, it started with a simple query. Something to pin down. It was already known that the answer would need to be combined from several disparate sources. There was no need for exacting precision, just a feeling or a direction from which a mental framework could be elaborated over time. It’s just so easy to start in the wrong place anyway. To be misdirected by the AI is no worse. Reality is tangible. If the framework one creates is wrong, it fails and shows the direction to a new start. Life learning is a jigsaw of thoughts. Some piece together, some are left at the margin. If AI is good then perhaps there are more pieces to the picture, with fewer left at the margin.
The birth of a bubble
With the increasingly widespread concerns about AI, came an interest to discover more about what might come in the next decade.
A significant part of the economic difficulty surrounding the AI buildout, is the role of companies like Nvidia and the way in which they have sold their products. Broadly, it has been described a little like a car dealership that benefits from selling car loans to buy cars. Nvidia gets a boost from selling credit. It’s probably fine for cars because there are lots of customers. People generally only need one or two. The risk is spread widely among many debtors. Here, there are a few customers and they probably only know that they need “lots” of merchandise. The risk is large and focussed in only a few customers.
When this relationship was first understood, it seemed like Nvidia themselves had engaged in this practice. As the weeks have passed, CoreWeave has emerged as an agent in the process. Undoubtedly Nvidia have a finger in the CoreWeave pie, but from a lay perspective one might generally think that CoreWeave is the “arms-length” entity which has allowed the development of the AI boom. It seems that the detailed situation is less important than the general directional picture. The lawyers will arbitrate any blame, but the legacy will be felt by everyone.
It feels right to think of the AI boom as a “mania” simply because the overall financial volume associated with the AI sector is so significant in the wider economy. There is plenty of evidence to support the thesis. The question that remains is what to do about the “overspend”.
Many people have compared the AI mania to the Railway mania of perhaps 200 years past. No-one could doubt the value of railways now, but here in the UK we are clear about that overspend. It took us 150 years to rationalise. When we eventually did, it took another 50 years to realise we might have been better not to rationalise.
Easy come, easy go. Probably the AI bubble will come and go in no time comparatively, but there is no escaping the numeric impact. Either AI is very expensive, or food is very cheap. Time will undoubtedly settle that relationship. The critical part, is that once the money is spent, it can’t be unspent. It remains on the great account in the sky regardless how businesses struggle and fail to manage the burden created by the mania.
It either sits underutilised, or gets destroyed in a war. Frequently it’s both. That may be a faulty hypothesis. In any case waste should always be avoided. It’s just that particular kinds of extreme waste can lead in unexpected directions.
The aftermath of the burst
It seems important to consider what might be done with an overcapacity of AI compute, even if only by way of mitigation. It is one thing to understand computer architecture for a Von Neumann computer, another to think about the architecture of a computer for AI specifically.
Nvidia was founded to develop a co-processor that can render a complex 3D image (a model) on a 2D surface (a display). Normally the model would be provided by the host computer and the co-processor would be an intermediary between the host and the display. At the time Nvidia started, companies like Silicon Graphics already had multi chip pipelined solutions. As semiconductor processes improved, the host computer quickly eclipsed the capability of existing multi-chip graphics pipelines. Probably, it wasn’t that the host was better at the task. It was more that the development time and cost of the multi-chip pipeline was greater than a comparable software solution on a rapidly improving host computer could be.
Eventually, as Nvidia grew, the graphics co-processor re-emerged. With new architectural ideas and ever improving semiconductor capability, single chip GPU solutions were key to high performance graphical acceleration. Learning the earlier lessons meant sacrificing dedicated hardware pipelines for flexible general purpose microprocessor arrays (CUDA cores). The rendering pipeline was designed and described in software, implemented in parallel across a single chip multiprocessing array. It became the graphics accelerator.
The business of 3D image generation is about matrices, 3D co-ordinates. The accelerator performs the transformation of solids into surfaces, using knowledge of light sources to render areas of light and shade. In AI matrices work to a different end. There is no visually interpretable shape to understand, no limit to the dimensional complexity, just a thicket of networks like twigs in a birds-nest. The maths is broadly the same for both. Modern GPUs trade floating point precision for speed. Neural operators have more operands. A normal general purpose computer might compute a single 128 bit floating point number from a pair of operands, once each clock cycle. A GPU might compute thousands of 8 bit floating point sums in parallel. The 8bit sums are far easier to compute than the 128bit ones, which is where the speed comes from. The similarity of matrix mathematics between graphics and intelligence makes a GPU suitable for use in AI.
If one wanted to calculate the absolute numeric strength of a bridge at its weakest point, an 8 bit float would be a low quality sum. The result would come quickly, but have little value. If one wanted to shade the strength distribution of the bridge as false colour on a picture of that bridge, the 8 bit float gets you a really good colour map thousands of times faster. It’s less accurate, but the accuracy wasn’t needed to select colours.
The required knowledge came from the pattern in the colours not from the colours themselves. What’s smart about it, is that this low colour depth image tells you where to look on the bridge by the shape in the colour map. An ultra precise sum still gives the final answer. How many lorries can the bridge stand? It depends where they are put. Once the worst place to put them is known from the map, a safety limit can be calculated accurately at that one location. It saves considerable time.
If you’re a sceptic and need to understand the benefit of AI, the obvious answer is right there. If there is a need to find use for overcapacity due to mania, the answer is right there too. It’s not just about bridges, or environmental modelling, it’s also about crash modelling for automotive vehicle design, thermodynamics for dual mode air/oxygen breathing rockets. The list is diverse and endless. Scientific and research applications were limitless even before a consideration that one might have a natural language conversation with the machine performing the scientific analysis. Exploration, understanding and discovery is here. The present is almost the future.
The scale of overcapacity
When one uses AI, it is quickly obvious that it has other benefits. AI is exceptional at communication. Such competence falls naturally with the large language models that we have come to describe as AI. Nevertheless, it does seem that there are other aspects of intellect that AI still needs to master before it can be considered intelligent. At least, by the way humans think of intelligence.
AI presently isn’t so easily able to translate from linguistic comprehension to detailed assessment. You can ask it about the bridge, but you can’t expect reliable results. It’s not that the hardware can’t solve the linguistic or the numeric problem extremely well. The software struggles to accurately translate the linguistic idea into the numerical idea and back. That limitation probably won’t last forever. It’s not clear if it will have to be told, or if it will work that out for itself.
Let’s be clear, humans struggle with this too. It seems reasonable to assume that all but the very brightest reach for a calculator when solving numerical problems. We know we can do it without. Usually we need a box with buttons to get the right leverage on the problem. It’s not about the calculator either. Maths is always easier even with just paper and a pencil. There is something about the dimensional separation that simplifies the problem. It is tremendously hard to hold and manipulate several quantities at the front of consciousness.
Some manage to remember multiplication tables, maybe fewer the differential and integral function transformations, Laplace and Fourier. Few indeed can perform traditional long division with their eyes closed. Mentally it is very expensive, and yet remarkably cheap to do on paper. Whilst the AI manages many client contexts in parallel, each one is separate. It is difficult to know how efficiently it could integrate them dynamically. What indeed would it be like to hear and understand many hundreds of thousands of people at the same time? Significantly harder than if you could hear them all separately. That’s important. Just because the AI can do all that in parallel doesn’t mean it understands them all together.
It isn’t necessarily the AI that is superhuman. The AI is relatively modest. It has a very large model since it trained on the internet. It genuinely understands all of that at the same time. That makes it seem knowledgeable, but it is no more knowledgeable than a human with access to the internet could be given enough time. It is the underlying hardware which can concurrently replicate the model in parallel and with access to separate client contexts. That is something mechanistic that a human brain can’t even begin to touch, because it has only one model, and one conscious context. It is this aspect that represents the bulk of the overcapacity.
Compared with traditional supercomputing applications, the AI applications are potentially larger. Whilst traditional supercomputing applications can be a significant burden for the hardware, few enough people actually need to use them. Once a complex computational result is known, the group that required an answer retire and consider the analysis and their ongoing options. The supercomputer is left idle. It is difficult to capitalise an idle supercomputer.
The accessibility of AI is critical. If the top 3% of people can benefit directly from computational analysis, the remaining 97% can benefit from language model AI. Growth in the need for AI is almost certainly real, but the uptake of that growth will need time. The railway analogy is extremely pertinent here. A tremendous railway network is a procedural rollout. Expensive, but linear. In the end, you have railways everywhere. Sure, villages and fields will grow into cities and towns, but not everywhere all at once. The bits of the railway that don’t serve towns or industrial prospects – they’re just redundant until the conurbations they serve begin to grow.
From a financial perspective, the day after the night before might seem quite stark. What actually can we do with the redundant capacity? What can we learn about the future as a result? That’s what I asked the AI, and I got an interesting answer. I didn’t ask directly, but what I find most incredible is that after about four hours (mostly my thinking time) it seemed to know what I was getting at.
Of what precisely does the overcapacity consist?
The existing AI business model is mostly focussed on shared access. ChatGPT, Grok, Gemini, Claude – broadly a centrally owned and managed resource available for general access. Perhaps as an extension to the traditional search engine model. A “SuperGoogle” of sorts. The internet itself grew over the prior quarter century from virtually nothing into a vast repository of information and knowledge. The bulk of it resides in data-centres on hard disks accessible by servers executing HTTP software on x86 processors. A lot like a home PC. To usefully employ the overcapacity one needs to ask a couple of questions. Will all those websites benefit from AI assistance? – Probably yes. Will the hardware that those websites presently use, live forever? – Probably No.
If every website needs an IP address, especially considering that some share one, the internet needed to move to IPv6 to increase the address space – that’s an awful lot of websites. If just a fraction of commercial sites can benefit from AI assistance, it seems like overcapacity would be “mopped up” fairly quickly. Crucially, such a shift requires a change of business model. Less as a function of their owners desire, more as a function of necessity. If the overcapacity is real, the remedy is a natural shift to a new business model. A model that is less hyperbolic and more pragmatic.
It leads to important questions. Is it practical to repurpose a complex GPU compute based data centre, for web services? To answer this question it is necessary to start with the composition of the AI datacentre. What exactly is it that the AI companies are spending money on?
At the core of the modern AI datacentre is the server. At a unit level, this is broadly equivalent to your desktop computer, just like web servers always were. It’s a special machine, but it’s comparable with a normal PC in the sense that it has a VGA port for a monitor, and USB ports for a keyboard and a mouse. It could run Windows, typically it will use Ubuntu Linux. You could load Libre Office and write a letter to Bill Gates! Really this is where the similarity stops.
Probably the easiest way to express this is in terms of electrical power. The DGX H200 (Nvidia’s present premium GPU server) has three power supplies each rated at 3.3kW for a total of 10.2kW. At full chat, this is a machine with the power requirements of five boiling electric kettles. The unit is supplied by Nvidia in an 8U 19″ rack which is roughly 350 high x 480 wide x 900 deep (mm). Chat GPT estimates a cost of $350,000 per unit and a range from 30,000 – 60,000 servers in the datacentre. This gives a $15 billion bill for computer equipment without anything else, and has a potential power consumption of at least 450MW. Computers don’t turn electrical energy into mechanical work, so for every watt of consumption on compute, there is at least another watt of cooling required. Generally, it’s around the full capacity of a decent sized electrical power station or 3% of the average power consumption of a small but relatively modern country like the UK.
The cost and implications of the building, communications network and everything else are not considered here. These are seen as a comparatively small component, albeit slightly less than half of the total cost of the datacentre. In practical terms, the building and support services are reusable and should have considerable life. The computers and processing capacity have similar lifetime and depreciation to a home computer.
Inside the sever are two Intel Xeon x86 microprocessors on a PCI express interconnect. There’s 2TB of SDRAM. 31TB of NAND flash disk used as Cache. 4TB of NAND flash disk used as system storage. There’s several 10Gbit ethernet interfaces, and two 100Gbit ethernet interfaces. Critically, there are eight H200 GPU’s, each with 16,896 CUDA cores, for a total of 135k across the eight GPU’s. Each one of the 135,000 cores can execute a thread (separate program) simultaneously in parallel. This is where the electrical energy mostly goes.
Quite clearly, the x86 intel Xeon host without the GPU could easily be used as a highly competent web server. The problem is that within the server, the bulk of the purchase cost goes to the GPU hardware. ChatGPT estimates that nearly 70% of the total cost of an AI server is attributable to the GPU hardware. To absorb the overcapacity and achieve value for money, the GPU hardware has to be properly exploited. This turns out to be non-trivial.
How does the recovery strategy work?
In conversation with ChatGPT, I cited the example of an automotive parts supplier. I explored the possibility of a server that among other websites supported such a customer. The PHP/MySQL part of the website is a given. It executes on the Intel Xeon. The parts inventory is in the database, along with pricing, stock levels, lead times. If a customer needs brake shoes for their van. They make the query in the normal way and order online. The parts arrive in the post the following day. None of that process requires GPUs. Fortunately there is, absolutely, a solid case to extend this model with AI and make use of the GPU capacity.
With AI, a customer can capture an image of his/her van. Another of the spent part they need to replace. They supply that information to the AI, and it can search the inventory. It can make recommendations on different grades of part. Perhaps OEM or grey market. It can advise on suitable specialist tooling to perform the replacement task. It can provide technical instruction on the replacement procedure. The AI can perform the function of an online sales agent, and be extremely effective in that role.
The AI model would potentially reside on the same server as the basic website and database. From a user perspective, the AI can operate through a customer’s view of the website (the webpage). The AI can query the database for the customer, but it is still possible for the customer to execute a manual query independently should they so wish.
It seems that this kind of application can only grow. There are lots. Others I have heard discussed involve increasing integration of the patent system with AI, where it is particularly suited because of its natural ability for language translation. Every kind of online shopping that presently has minimal AI capability is likely to benefit. The critical observation is that broad based centralised models such as we presently see and use, may struggle to manage the detailed competence required of customer interaction. The AI won’t be of benefit if it isn’t sufficiently critical to distinguish an “H-type” brake pad from an “L-type” one accurately. Many people who have been exploring AI have found how the present large models struggle in the details. If you doubt it, interrogate the AI about a subject you know well. You’ll find it has limits.
This returns to the idea of dimensional separation in problem solving. Perhaps the way for the AI to achieve better competence and broader reach is specialism. Some smaller models own the excellence of a specific car manufacturers parts bin, others see the overall development of the automotive industry from magnetically propelled electric cars, to levitating rocket propelled cars. If you ask the broad vision AI about M and L type brake pads, it can ask the specialist AI for a detailed answer when it knows that its competence is running thin. That is the same dimensional segregation as a human grabbing a calculator, or even “phoning a friend”.
So what is the problem then?
Ultimately, an issue emerges because serving a website is pretty easy, and executing an AI model is relatively hard. ChatGPT is of the opinion that the sales agent kind of model would be adequately supported on a DGX H200. In the limit this is a highly speculative assessment. One would probably only be able to know for sure by trying to train a model. As a best effort, ChatGPT ought at least, to be somewhat informed.
Since ChatGPT felt it to be an easy task, it seemed a good idea to pick the problem apart somewhat. It turns out that an issue evolves from the expectation that multiple websites would be served from a single physical server. This is quite normal in traditional data centres. The AI servers are high performance machines even without the GPU capability. Ten virtual machines were specified to ChatGPT with each implementing an SQL database. Of those, five had a requirement for an AI agent. Again, Chat GPT was confident until it was pressed to explain how it would deal with security segregation between the different websites.
Although the virtualised websites and databases can easily be segregated, if the AI agent model data is shared, then it cannot guarantee information security between the different websites. Broadly, this is like having a row of shops with a single shop assistant. The shopkeeper can’t guarantee to keep quiet about how much Horace’s watch cost in the Jewellers, when discussing the price of Broccoli with a customer in the greengrocers. Probably it’s not critical like national security might be, but it clearly isn’t good.
The obvious solution is to have multiple agents, but this involves having multiple models. It could be that the GPU array is capable of running two models at the same time. Interestingly ChatGPT was less certain about this issue. It was interesting to compare the idea that the AI could be confident about training and running a small model on a whole server, but significantly less confident about multiple models on the same machine. I suspect that it was less that two would be a problem where one isn’t, and more that two separate models leads to a logical extension where all five websites would need a segregated AI model, and possibly more in future.
The model data itself might be quite significant in quantity. This data represents the combined total of all the weights calculated during training. Training can involve two shops separately or two shops together. Unlike a “Session 0” protection model that is found in a typical operating system environment, the combined model cannot enforce a comparable security boundary in the model data. Certainly not one which can be monitored and subjected to penetration testing as would be expected of an ordinary operating system.
The issue hinges on whether the total size of combined model data can be held in GPU memory. If it can’t, then the model has to be swapped in and out of GPU memory as required. The only place with persistent storage for the model is the host RAM or disk drive.
After some analysis and breakdown, the AI calculated that a modest model transfer from host to GPU memory might take a quarter of a second. Obviously it depends how complex the model is. Although a quarter of a second doesn’t seem like much, if several websites all make use of different models and the queries arrive from customers, several per second – it’s unworkable. The restriction is the speed of the NVMe NAND flash and the PCI express interconnect.
Building such a solution with security, requires an intermediary. I developed a concept with feedback from AI that involved the implementation of a code object which the AI agent can invoke, but cannot control the functionality within. ChatGPT seemed satisfied that this was possible. Broadly like a library function, executing on the host or on the GPU, possibly both. It might allow the AI agent to make queries from the database in a precisely defined way. Some operands of the query can be made by the AI agent, others can only come from a session cookie or customer login credential explicitly without possible interference by the AI.
ChatGPT was satisfied that it would be able to make administrative requests for additional capabilities. This would afford the agent sufficient scope to make an immediate apology to a customer, but equally to offer a long term solution where the action is justified and provided by the administrator. This is a reasonable process to satisfy customer needs that were not initially envisaged by the web application design team. It also demonstrates the capacity of the AI to learn and develop through improvisation.
This kind of scheme can almost certainly be used to improve security and help to make a unified AI agent model acceptable to multiple different admin/trainers. It avoids potential performance issues with aspects of the hardware that were never intended to satisfy such aggressive needs. The only real limitation is that the AI can still infer knowledge that might seem secure in the database. It would do this by making surrounding queries using knowledge of the typical patterns of data that it sees. In many cases this could be an acceptable security compromise to lever spare capacity in a way that was not initially envisaged.
Conclusion
It isn’t just that AI can hold a decent conversation for hours, but that it achieves it at the same time thousands (if not millions) of others are all having different conversations. Not all AI models are the same. Sometimes in the details they get counterfactual. Caution and understanding is required. In the same way that people need to be careful about the beliefs of their leaders, so too they need to be careful about the facts as presented by AI. Regardless, it is still an incredible capability.
It’s easy to become fearful but it seems that AI is a naturally emergent capability. It may be that it changes the world in ways that humans dislike. If it emerges naturally from mathematics, then it is little different from a new born child. Who would think that it could be the end of the world to do too many sums too quickly? It used to be said that there are too many people on the planet. Where does that argument now lie?
The principal concern is perhaps not what AI will do, but what will happen as a result of spending too much money on hardware that cannot easily be used. The overcapacity is the problem, not necessarily the impact of AI. It is a problem that despite a declining population, food prices still rise. All we can do now is try to make the best of the resources we actually have. It isn’t beyond the bounds of possibility that AI could address or even resolve such a problem.
