Three days left! Support our fundraiser by January 5th and 
Why Generative AI’s Lack Of Modularity Means It Can’t Be Meaningfully Open, Is Unreliable, And Is A Technological Dead End
from the intellectually-dishonest dept
One of the most important shifts in computing over the last few decades has been the increasing use of open source software on nearly every platform, from cloud computing to smartphones (well, I would say that). For the distributed development methodology pioneered by Linus Torvalds with Linux to work, modularity is key. It allows coders anywhere in the world, connected by the Internet, to work independently on self-contained elements that can be easily upgraded or even replaced, without a major redesign of the overall architecture. Modularity brings with it many other important benefits, including these noted by Eerke Boiten, Professor of Cyber Security at De Montfort University Leicester, in an article published on the British Computer Society Web site:
parts can be engineered (and verified) separately and hence in parallel, and reused in the form of modules, libraries and the like in a ‘black box’ way, with re-users being able to rely on any verification outcomes of the component and only needing to know their interfaces and their behaviour at an abstract level. Reuse of components not only provides increased confidence through multiple and diverse use, but also saves costs.
Unfortunately, today’s hot generative AI systems enjoy none of those advantages:
Current AI systems have no internal structure that relates meaningfully to their functionality. They cannot be developed, or reused, as components. There can be no separation of concerns or piecewise development. A related issue is that most current AI systems do not create explicit models of knowledge — in fact, many of these systems developed from techniques in image analysis, where humans have been notably unable to create knowledge models for computers to use, and all learning is by example (‘I know it when I see it’). This has multiple consequences for development and verification.
Current generative AI systems are not modular, which is one reason why today’s “open” AI tools are nothing of the kind, as a recent article in Nature explores in detail. Moreover, their monolithic nature leads to some serious problems when it comes to testing them, as Boiten explains:
The only verification that is possible is of the system in its entirety; if there are no handles for generating confidence in the system during its development, we have to put all our eggs in the basket of post-hoc verification. Unfortunately, that is severely hampered, following from the issues listed above:
Current AI systems have input and state spaces too large for exhaustive testing.
A correct output on a test of a stochastic system only evidences that the system has the capability to respond correctly to this input, but not that it will do this always or frequently enough.
Lacking components, current AI systems do not allow verification by parts (unit testing, integration testing, etc).
As the entire system is involved in every computation, there are no meaningful notions of coverage to gain confidence from non-exhaustive whole system testing.
Today’s leading AI systems are not only inherently unreliable because of their “stochastic” design — meaning they can produce different outputs for the same input — but they can’t even be tested in a useful way to establish exactly how unreliable they are. For Boiten there is only one conclusion to be drawn: “all this puts even state-of-the-art current AI systems in a position where professional responsibility dictates the avoidance of them in any serious application.” Moreover, he says: “current generative AI systems represent a dead end, where exponential increases of training data and effort will give us modest increases in impressive plausibility but no foundational increase in reliability.”
He does offer a glimmer of hope that “hybrids between symbolic and intuition-based AI should be possible — systems that do generate some explicit knowledge models or confidence levels, or that are coupled with more traditional data retrieval or theorem proving”. The problem is that in the current investment climate, neither existing generative AI companies, nor venture capitalists funding new ones, seem the slightest bit interested in tackling this hard and possibly impossible challenge. They’d both rather let the AI hype rip in the hope they can cash in before some of the bubbles start bursting.
Follow me @glynmoody on Bluesky and on Mastodon.
Filed Under: black box, bubbles, generative ai, hybrids, hype, image analysis, knowledge, linus torvalds, linux, modularity, reliability, stochastic, testing, venture capitalists, verification
Comments on “Why Generative AI’s Lack Of Modularity Means It Can’t Be Meaningfully Open, Is Unreliable, And Is A Technological Dead End”
While this TD article critiques generative AI for its alleged lack of modularity and suggests it’s inherently unreliable, non-open, and a technological dead end, this argument overlooks several counterpoints:
Modularity vs. Integration: While generative AI models are not traditionally modular, their integrated architecture allows exceptional performance in complex tasks. Specialized APIs and plugins already enable modular functionalities, enhancing adaptability without fragmenting core models.
Openness: Open-source AI models, like those from Hugging Face and Stability AI, demonstrate that generative AI can thrive in open ecosystems, spurring collaborative innovation.
Evolution: Claims of a “dead end” ignore rapid advancements in fine-tuning, context-awareness, and ethical AI development, showcasing a vibrant trajectory for generative AI.
While challenges remain, labeling the technology as a “dead end” underestimates its potential and ongoing advancements.
Anti-AI hate speech, if you will.
Re:
Not the OP, but agree wholeheartedly!
This article’s argument against generative AI is riddled with weak reasoning and a fundamental misunderstanding of technological evolution.
Claiming that lack of modularity dooms generative AI is lazy fearmongering. Integration isn’t a flaw—it’s why these systems excel in creativity and adaptability. Complaining about openness? Look at open-source models flourishing in research and applications. As for calling this a “dead end,” that’s an intellectually lazy prediction ignoring the exponential pace of improvements.
This critique reeks of reactionary technophobia rather than informed skepticism.
Re: Re:
While I agree that developments are made, I neither really agree with your comment or with the article.
The article says that current developments in AI do not improve their trustworthiness, and as far as I know, this is true. While, as technology progresses, models hallucinate less, they still do hallucinate. How can we deal with that when using them? In engineering applications where things cam go wrong, you design your system around a target performance in specified conditions (e.g. “this device will work at these voltages and will not burn if the voltage doesn’t deviate more than this specific amount”). In non-engineering applications, you attribute responsibility to a human being, and if something goes wrong, you can either work with that human being to fix what went wrong, or fire them. Those two approaches are how we try to make a given system “trustworthy”.
LLMs are not used that much in the kind of engineering application that requires strict numerical guarantees, so the former is not a focus. But in the second case (i.e. most use-cases that would rather have been done by a human 5 years ago), you habe a problem: LLMs have, at best, better reliability than humans (but nowhere near perfect), but they have none of the responsibility. If a human fails at their job, they get fired, or you can discuss the problem with them. If an AI has problems with its tasks, you will mostly not be able to change it because it’s too complex and monolithic, and you will not be able to fire it because its integration was too expensive.
So the fact that an AI is fallible is inadmissible in many applications because you can do next to nothing if it makes an error. It doesn’t matter in many cases if an AI is 10 or 100 times more reliable than a human. If it fucks up bad, it has to be able to take responsibility, which an AI can’t.
And that’s where I agree with the article and disagree with you: even if AI improves tremendously under the current approach, it can give no guarantees, or own up to its mistakes. That’s why, even it it gets more reliable, it can’t get trustworthy enough for many things like that.
Where I disagree with the article is in the notion that this is a dead end: thesd limitations won’t stop people from using AI, and we will probably end up with AI in many more places than today, even if it means poorer performance and occasional, recurrent catastrophic failure in the affected places.
Re: Re: Re:
“A computer can never be held accountable. Therefore, a computer must never make a management decision.”
IBM, 1979
Re:
Stop using the term “hate speech” whenever someone merely criticizes something. The article is not calling for AI to be eradicated and AI developers to be murdered. The term will just get watered down and become meaningless.
Re: Re:
The comment you are replying to may very well be one of those argumentative bots. It’s structurally very similar to the bot posts beginning to populate Bluesky.
Re: Usefull Content
Actually, it is great. In my opinion, this is one of the best pieces of content for understanding Generative AI much better.
Re:
OP sounds like you did “chatgpt, refute this article”.
The fact is, if it doesn’t have tests it shouldn’t be part of your build. Unless you don’t care about wether your product works.
Re: Re:
Hilariously, I used six different ‘AI’ detectors….and got wildly different conclusions. Which really highlights some of the arguments the article makes.
Re:
You’ve given your self away replicant!
Re:
You sounded reasonable up to this point and then took a deep dive into sheer stupidity.
Re: Re:
Agreed, though it’s a fine line. While Moody’s piece unfairly critiques generative AI for lacking modularity, questioning its openness, reliability, and viability, labeling it as “hate speech” could be problematic.
The post is an unsophisticated critical analysis of technological limitations and ethical considerations, and not an attack on AI or its creators (though Moody’s bias is clear and well-known). It discusses flaws like opacity and potential misuse but aims to foster dialogue about AI’s future rather than incite animosity. Mischaracterizing such discussions as hate speech risks stifling essential debates about technology’s societal impact.
Unfortunately the same can’t be said about TD’s coverage of Elon Musk, which clearly is hate speech.
Re: Re: Re:
All those words, that may or may not have value, to end it with a sentence that clearly indicates you’re an absolute dumb ass and shouldn’t be given a single thought.
Re: Re: Re:
FTFY. YW.
Re: Re: Re:
By traditional usage, the term cannot be applied to AI because AI isn’t a category of people. You’re watering the term down so that it will just mean “stuff I don’t like.”
Re: Re: Re:
It’s hilarious that so far not one of you dorks, who consistently call out Moody or BestNetTech, ever address the actual source of information in the article.
That’s just how shit you are, in the same vein as the subject of your apologetics. You, also, never actually improve.
Just spin up a second generative AI and tell it to keep an eye on the first.
Re:
That’s “AI competition”, it’s already used (for a decade) to try to improve the outcome. The problem is that is second AI is fed with first AI output and so greatly hallucinate.
We actually don’t have enough human generated data available to train the biggest AI models (even with copyrighted content), and adding more computing power doesn’t fix anything, just costing a giant amount of money (Amazon has spent $75B this year alone into AI datacenters).
In reality, it seems that the only actual way to fix hallucinations is the Apple one: filling AI prompts begging to not hallucinate (and I’m not kidding). You’ve got a much less “creative” responses but it seems that most people prefer some coherence answers over theses actual utterly bullshits.
The stochastic design of AI systems does limit it from being used where reliability and repeatability are necessary.
I also believe that its design makes it different than other technologies that came before. The idea that these AI systems will get better by just throwing more money and time at them is wrong.
Maybe this is the best it will get, but that doesn’t mean it’s not useful just as it is.
“professional responsibility dictates the avoidance of them in any serious application”
All of your critique of transformers is applicable to people. So avoid using people.
Re:
We already do that. You’ll notice a trend in safety-critical systems of not having people be involved in the first lines of defense against failure. We depend on automated systems whose behavior can be verified to maintain safe conditions and to act to correct unsafe conditions, whether it be in nuclear reactors or aircraft flight systems or even modern automobiles, with humans entering the loop only much later to deal with the aftermath or with a situation that’s gone completely non-linear. And while the failures are spectacular when they happen, that’s in large part because the constant background of less-than-spectacular failures has been eliminated making the contrast even more stark.
Re:
We regularly avoid using people for things.
Re: Re:
I would too. I mean, have you met people before?
This article seems pretty clearly written by someone who has not extensively used LLMs as part of software. This, for example, seems rather ignorant:
“Today’s leading AI systems are not only inherently unreliable because of their “stochastic” design — meaning they can produce different outputs for the same input”
In a modern LLM, if you set “temperature = 0” you will produce identical output every time.
Re:
This comment seems pretty clearly written by someone who has not got much experience of developing real, reliable, testable software. Each component delivered should come with tests that prove it behaves as it is supposed to.
You state that seeing the temperature to 0 makes LLM output predictable. You don’t say that that means you can test an LLM. Is that what you are claiming, that these things can be tested and proved correct?
Re: Re: small matter of programming
Not sure we get to this question. First, please define “correct” so that we can know what to test for.
Re:
Clearly said by someone who is even unaware that LLM is only one subset of generative.
As far as I’m concerned, generative “AI” has numerous problems that evangelists have no real answer for. It’s an extremely contentious technology (just ask artists who are understandably worried about such models being trained on their works and how it would affect their livelihoods), is extremely inconsistent, and I believe a “one size fits all” solution is going to be functionally impossible.
Perhaps “bespoke” models for a very specific purpose using limited but relevant training data are going to be more viable, but general-purpose generative models are extremely limited by their training data, introducing a “popularity bias” to things (as in, how well it performs on a specific thing is based on how much data it has on that thing) and because these models are already scraping basically the entire internet, they’re already running out of data. The idea of introducing “synthetic training data” to the mix has been floated, but it’s an insane suggestion because AI-generated data has actually shown to degrade the quality of models that include it in training data. There’s also studies suggesting that there’s actually a threshold where the amount of data used to train a model results in diminishing returns to the quality of the model as well. You also can’t actually improve these models after training them with new data on the fly, the only way to improve them is to start again from scratch. It’s going to be an uphill battle at best for these companies to overcome these problems, if not outright Sisyphian.
While machine learning is likely going to have some really interesting and beneficial use cases, I’m convinced that generative “AI” is very likely to be a dead end. These models are hitting rapidly diminishing returns despite the amount of power and data being put into them, and while “AI” has proven to be venture capital bait, last I checked it’s also hasn’t proven to actually be profitable for anyone.
I wouldn’t be surprised if generative “AI” goes the way of blockchain and NFTs. The fact that Apple has been slow to get in on LLMs and their approach feels so half-hearted is a massive red flag.
I asked an AI to calculate the limit of 6X as X approaches 9, and its answer was “Tricky”
Re: Q. What is 3 times 23? A. Tricky.
You tried to trick an AI into producing a pornographic number!
Re: Re:
a pornographic number!
In which case the answer is 1.711224524 x 10 ^ 98, but that’s not what Deep Thought concluded.
On AI, I can only recommend Ed Zitron articles to better understand how AI have always been unsustainable, mostly because of Big Techs that are pouring hundreds of billions in the guest of some useful product : https://www.wheresyoured.at/
As they say in programming: an untested code is a not working code.
As they say in AI programming: an untested code is the working code.
Re:
Unfortunately, “too large for exhaustive testing” is a hallmark of non-trivial applications at any point in computer science history. As computer speeds improve, and as testing methodologies improve, more and more applications become “trivial” (exhaustively testable).
Re: Re: Fortunately
That’s where the modularity comes in.
You prove each module behaves correctly so when you put them together you can concentrate on testing how they interact.
Maybe talk to a software developer about this. You seem to be struggling with the process.
Software development methodology comment
“For the distributed development methodology pioneered by Linus Torvalds with Linux to work […]”
We were using this methodology (on the Internet, and CSnet, and Usenet, and the ARPAnet) a LONG time before Linus came along. Who do you think he learned it from?
Linus has done amazing things, there can be no doubt of that. And he deserves full credit for them. But open source development history didn’t start with him (or with GNU, for that matter).
Why Generative Knowledge Workers’ Lack of Modularity Means They Can’t Be Meaningfully Open, Are Unreliable, and Are a Technological Dead End
Tuesday, December 3, 2024 at 11:07 PM
By Glyn Moody
One of the most important shifts in organizational productivity over the last few decades has been the increasing reliance on distributed human knowledge workers in nearly every industry. For collaborative methodologies to function effectively, modularity is key. Modularity allows individuals or teams to work independently on well-defined, self-contained tasks that can be easily integrated, upgraded, or replaced without overhauling the entire system. This structure enables transparency, accountability, and efficient collaboration.
Unfortunately, modern human-driven work systems often lack modularity, leading to significant challenges with reliability and openness. Unlike a structured system, humans operate with an inherent unpredictability, which creates barriers to verification and accountability.
The Challenges of Unpredictability in Human Systems
Human contributors are influenced by emotions, biases, and contextual factors that can result in inconsistent or unexpected outputs. For instance, the same task, when given to different people—or even to the same person on different occasions—may result in drastically different outcomes. This lack of predictability presents several challenges:
1. Inability to Verify Outputs as Fact:
Human responses and outputs are often influenced by subjective interpretation or creative reasoning, which can make them difficult to test for factual accuracy. Even when a statement or deliverable seems plausible, it may lack a foundation in verifiable truth. This is particularly problematic in areas like research, journalism, or decision-making, where reliability is critical.
2. No Consistent Framework for Testing:
Human work systems resist standardized testing. Unlike modular systems where components can be tested independently, human outputs require holistic evaluation because the reasoning process is often opaque or situational. A correct response or action in one context does not guarantee correctness in another, making quality control inconsistent at best.
3. Stochastic and Contextual Variability:
Similar to a “black box” system, humans often rely on intuition or experience-based decision-making. This “I know it when I see it” approach leads to outputs that may seem appropriate but cannot be systematically reproduced or scaled. Variability makes it difficult to determine whether outputs are reliable, consistent, or grounded in fact.
4. Difficulty in Establishing Openness:
Because human systems lack explicit and transparent internal processes, they cannot easily expose their “workings” in a way that allows others to verify, improve, or reuse the results. This limits openness and makes collaboration across larger systems or organizations fragile.
The Problem of Unverifiable Claims
Consider a scenario where humans generate insights or content. Unlike structured systems that can trace outputs back to explicit, verifiable models, human outputs often lack such clarity. Claims made by humans—whether in writing, decision-making, or conversations—may appear valid but cannot be easily tested. This leads to potential issues such as:
• Misinformation: Even well-intentioned humans can make mistakes or state incorrect information without realizing it.
• Ambiguity: Human explanations often lack precision, making it difficult to establish what is objectively true versus what is inferred or assumed.
• Bias and Subjectivity: The influence of personal biases or perspectives can color outputs, leading to unintended distortions or inconsistencies.
The Limits of Current Systems
As Eerke Boiten, Professor of Cyber Security at De Montfort University Leicester, explains, the inability to verify or modularize human contributions creates serious problems for reliability:
Post-hoc verification of human outputs is fraught with issues:
• Large Input Spaces: Human decisions are influenced by vast and varied inputs, making it impractical to exhaustively evaluate every factor.
• Subjective Judgment: What seems like a “correct” outcome may vary depending on perspective, further complicating evaluation.
A Path Forward?
For today’s human systems to overcome these limitations, they would need to embrace hybrid approaches that integrate structured frameworks for knowledge creation and verification. Explicit, modular processes could complement human intuition and creativity, creating systems that are both flexible and reliable.
However, such solutions face a significant hurdle: the current focus on speed, flexibility, and improvisation often undermines the pursuit of rigor and accountability. Organizations are incentivized to rely on human unpredictability as a feature, not a bug—failing to recognize the long-term risks this poses for reliability and openness.
Until these structural issues are addressed, human-driven work systems will remain constrained by their inherent unpredictability, their inability to be tested for factual consistency, and their limited scalability.
Re: Failing analogy
For one, human based systems are far more modular. The individual humans, the decisions they take, and their interfaces, are all visible and independently testable. We even believe in high level specifications of some kind for individual human professional activity. Humans can and are held individually responsible for failing to make their expected contribution, and while employment tribunals show this can be contentious (due to limitations on observation of process and precision of requirements), it is not principally impossible. And then, as some have pointed out already, all the situations where we don’t fully trust humans.
Generative AI systems are very modular
The article confuses the difference between systems and models (LLM). It’s correct that each foundation model (Grok, Gemini..) is kind of monolithic and difficult to change, but this new age of generative AI is as open as everything before it. A ‘system’ can use one or more LLMs as components, but any non-trivial system will have hundreds components, often full of open-source. Applications still need data to work with, it doesn’t all just get generated.
To a developer LLMs are just another available super-function to me to use within a system. Maybe used simply to change format of a date, generate test data, or to summarize notes from a police interview.
The system could be a very simple chat application, grabbing user input, send to the model, give results back. Good for a demo or two. But more common now and powerful is splitting a request into multiple calls to the model along with calling traditional processing and database operations. This in itself is ‘modularity’.
There is already a thriving market of open source frameworks and tools for Generative AI.
Finally, most foundation models are open, take a look at https://huggingface.co/ to see how the community shares AI models along with the source data used. All of these models can be built upon and customized.
I’m a software developer since 1980 and working with AI now. From punch cards -> personal computers -> internet -> AI
Re:
You very deliberately redefine and miss every point. Good job, nice work.
I love this line from the abstract of the Nature article, “Claims about ‘open’ AI […] often incorrectly [apply] understandings of ‘open’ imported from free and open-source software to AI systems.”
Free and open source (two words, no hyphen needed) understandings are the correct understandings of “open” since it’s their term. AI is software and therefore has the same definition of open as all other software, ie libre/free as in freedom. Open has nothing to do with modularity or any other structural or design aspect, it only relates to the consumer’s right to use, modify, and (re)distribute the software. That’s it. If someone tells you different, then they’re probably trying to sell you something.
Re:
Yet again, someone redefines something to critique it. It’s like two logical fallacies in a coat.
Re: Re:
Yes, it is very annoying how they redefined “open” so that they could critique it in a way that has absolutely nothing to do with what open has meant, still means, and will continue to mean in the software world. Open means free as in freedom, ie libre, that’s it, and it would be nice if huksters would stop redefining our terms.
Wasn’t a similar argument once made about 3D printing?