Lies, Damn Lies, and Statistics: Large Language Models Are Having a Moment But The Future of AI Is Hyperdimensional
If by now you haven’t heard about ChatGPT and all its various cousins, acquaintances and relations you might want to check some recent obituaries cause there’s a better than even chance you’re deceased. Notice just what happened? The suggestion that your very existence isn’t a fixed reality but rather a probabilistic artifact conditioned on the available written record? That’s all you need to know about Large Language Models (LLM). Oh, and if it turns out that the rumors of your demise have been stochastically exaggerated, have nice day.
Back in 2021, Emily M. Bender and colleagues published an article that clearly sets out the problem with existing Large Language Models and the underlying techniques that make them such a marvel to your average technology consumer. If you haven’t read “On the Dangers of Stochastic Parrots” take a few minutes and find out what it’s all about. Apart from the technological foibles of today’s language model methodologies, like the propensity to make things up (see AI: “What’s reality but a collective hunch?” – November 2017), lurks and even larger issue, the propensity of humans to subscribe to their output without the slightest bit of curiosity or skepticism. After all, what you see on the internet is real, honest, everybody says so. And therein lies the problem.
But don’t be lulled into blissful reclusivity. LLM’s are having a moment and it’s going to be a BIG one. The economic structure of the planet is likely to be appreciably impacted from this point forward in ways that few anticipate, especially economists since they don’t teach this stuff in fancy universities. It’s true that millions of people, mostly knowledge workers, will eventually lose their jobs. But most of them are lawyers so you probably won’t notice. It’s true that almost all your future conversations will take place with a machine. But you probably won’t miss people as they are highly over rated. It’s also true that this crap can write better code than Skynet and that some miscreant will try to use this stuff to take over the world. But if Pinky and The Brain can’t get it done neither will ChatGPT anytime soon. Finally, as Sam Altman, the CEO of OpenAI, recently pointed out, LLM’s have already crossed the horizon of diminishing returns as additional investments in computational prowess will only result in unacceptable increases in carbon foot prints without commensurate improvements in witty, grammatically immaculate banter.
LLM is peaking at a particularly interesting moment in the annals of technology. For the past few years pundits have ruminated over the future of the internet arguing not whether there would be a Web 3.0 but rather exactly what catalyst would cause it to coalesce. Would it be a richer, more robust version of blockchain (see Beyond the Cyber-Cryptoverse – February 2019)? Or would it be a layered stack of graphical metaverse components assembling and orchestrating your other life, the one you’d prefer to lead. Then along comes LLM and the Web 3.0 conversation has all but been silenced. The metaverse proponents have left the field and several prominent champions have struck their colors and abandoned their initiatives entirely including the likes of Microsoft and Disney, leaving Facebook pretty much alone with its brand new branding and nothing else to show for it (see Metaverse Schmetaverse – August 2021). Similarly, blockchain proponents have been left scratching their heads, convinced of its utility but unable to figure out how to fund it, let alone who might profit from its build out. Each of these visions share a similar problem, success depends on costless, viral promulgation and adoption of a common edifying infrastructure; how else can you create a trustworthy semantic utility? But putting that kind of toll on this kind of road is not going to generate any traffic.
In some ways this reflects the dilemma posed by intelligent assistants or agents. Apple got things started with Siri by building a complex, costly and proprietary infrastructure of bespoke ontologies (see Prophets on the SILK Road – October 2012). Then Google came along in 2018 with Duplex, a generalizable machine learning approach to the creation of ontological context using supervised, real-time training of recurrent, feed-forward neural networks. (see The Dawn of Agency – May 2018). Neither of these things went anywhere important. But in today’s post, Siri plays the part of Web 3.0 and Duplex plays the part of LLM. The strategic question being: does a generalizable approach to obtaining “meaning” obviate the need for common edifying infrastructure in the realization of a trustworthy semantic utility? Now, before you answer, think of all those stochastic parrots making stock picks, pasta recipes and crawling the internet to determine whether or not you’re still alive.
On second thought, don’t bother, here’s a better question.
Can ChatGPT or some yet to be published successor create its own common edifying infrastructure and imbue the written record with its own instructions, precautions and caveats and, if so, what would it be? In other words, can LLM create Web 3.0? Could subscribers to LLM’s play a roll in validating their veracity? Can consumers become a node in the model? And how would that be transcribed and encoded? Who and how many times would we need to agree to your demise in order to make it official? Ordinary people can back-prop too and would that make things better or worse? Remember, “on the internet, nobody knows you’re a dog”.
All of this notwithstanding, are we really prepared to say that sophisticated auto-complete bots are the final stop for AI? Not likely. In the scheme of AI things, LLM’s are just one technique.
As far back as the late 1990’s, disciples of Geoffrey Hinton including Tony Pate began to explore the notion of distributed representation of objects through multi-dimensional vector arrays. In rough terms this advanced the notion of “hashing” objects into complex arrays of numbers that could then be recognized and manipulated. This technique was then further refined by Pentti Kanerva in 2009 who proposed its technical implementation while also highlighting some important impediments to its practical realization, namely conventional computing architectures.
Let’s say that the immutable properties of an object can be described as a multi-dimensional array of values and that the variable attributes of an object can also be described as a multi-dimensional array of values, unique from those of the object. And the “essence” of this thing would be the product of the multiplication of the arrays of the object and its attributes. We are suddenly talking about a entity of potentially very rich and very large vectors, the conventional digital tokenization of which could consume a huge amount of computational resources, such that the “essence” of a car would be the multiplication of it’s make, model, color, speed, registration, location, distance to destination and need for fuel, etc. Sticking this into your average von Neumann computer would be like stuffing an enormous hair ball into your kitchen’s in-sink aerator, your next move will be to get your pipes replaced. But humans do this kind of thing all the time and still think about what’s for lunch.
For the past several years, the biomimetic neuromorphic branch of the AI tree has been navigating their way around the von Neumann bottle neck of moving large sets of data and instructions into and out of separate components by using memristor based in-memory computing (see AI: Waking the Neuromorphic – February 2018 and HPE Builds a Boat in Its Basement – December 2016). And it turns out, hyperdimensional computing (HDC) is better suited to memristor architectures for much the same reason. While far from perfected, the number of academic projects exercising this approach have been growing steadily over the last few years. A survey report by Khalifa University listed 21 academic hyperdimensional computing projects in 2021 employing memristor and FPGA architectures. The read outs from these projects have just started rolling in and the results seem more than promising, not the least of which are the nonvolatile, low energy aspects of in-memory vector manipulation and permutation using memristor technology. Several of these projects have even gone so far as to suggest that this technology is especially suited to edge applications, where access to cloud based resources might be sporadic or intermittent, kind of like the wetware humans are equipped with. This suggests that the future of AI might take a page from the assistant/agent architecture with its pre-frontal memristor nose out sniffing the weeds and its associative GPU memory tucked back in a cloud.
Perhaps the most interesting aspect of hyperdimensional cognition is one that has yet to be explored. While LLM’s employ complex models to “guess” what their response to any given prompt should be, HDC employs well understood mathematics to compound multidimensional objects into complex concepts, the former occasionally producing intractable confabulations and hallucinations, the latter leaving a trail of algebraic bread crumbs explicating its methods, processes and permutations. The implication being that for any given phenomenon or collection of phenomena, if you can articulate its initial state and propose or project its end state, its transition or metamorphosis can be calculated, in other words it can imagine cause and effect or, in AI jargon, it can articulate an invariant counterfactual ( see Stalking the Snark: Invariant Counterfactuals, Transcendental Deductions and the Future of AI – May 2019). So, instead of the hyperdimensionalized car we earlier proposed, let’s hyperdimensionalize a hydrogen atom undergoing fusion in a tokamak reactor and solve for sustainability.
“Voila!” – endless energy. (You can thank me later.)
Samuel Clemens, aka Mark Twain, has a number of lasting witticisms attributed to him including “Lies, Damn Lies and Statistics”, although he never really penned them. At the time, Clemens thought this one belonged to Benjamin Disraeli although there’s scant proof of that either. Regardless of its origin, this phrase has become an apt moniker for our current LLM moment. ChatGPT’s response to prompts are crafted by numerous stochastic models that anticipate the machination of the query as much as the ordination of its reply and no matter how familiar, clever and authentic it might seem, its answer is merely a guess. That’s all. And there is no getting around the fact that it is impressive, bordering on astounding, but in the larger scheme of things, it is but one of many techniques that will eventually make up what we call AI.
Hyperdimensional vectorization will be another technique and together with LLM’s they will be a beacon that on brighter days casts its own dark shadow. If we find that shadow to be ominous we should remember that it’s of our own making and the words of another well known aphorism Clemens never said “a lie is half way ’round the world before truth has pulled its boots on”.
And act accordingly.
Cover graphic courtesy of fragment of Figure 2.15 of Rumelhart, Hinton and Will recurrent network diagram from Distributed Representations and Nested Compositional Structure by Tony Pate copyright 1994. All other images, statistics, illustrations and citations, etc. derived and included under fair use/royalty free provisions.
