Google and OpenAI are Walmarts besieged by fruit stands

OpenAI may be synonymous with machine learning now, and Google is doing its best to catch up, but soon both will face a new threat: rapidly multiplying open source projects that push the state of the art and they leave the deep, but pocket ones. unwieldy corporations in the dust. This Zerg-like threat may not be existential, but it will certainly keep dominant players on the defensive.

The notion isn’t new by any means — in the fast-moving AI community, you’d expect to see these kinds of disruptions on a weekly basis — but the situation was put into perspective by a widely shared document purported to originate from Google . “We don’t have a moat, and neither does OpenAI,” the note says.

I won’t bore the reader with a lengthy summary of this perfectly readable and interesting piece, but the gist is that while GPT-4 and other proprietary models have gotten the lion’s share of attention and indeed revenue, the advantage they have gained. with funding and infrastructure it looks slimmer every day.

While OpenAI’s release pace may seem blistering by the standards of regular major software releases, GPT-3, ChatGPT, and GPT-4 were certainly on edge when compared to iOS or Photoshop versions. But they are still occurring on a scale of months and years.

What the note points out is that in March, a fairly rough model of Meta’s foundational language model, called LLaMA, was leaked. inside weeks, people playing on laptops and penny-per-minute servers had added core features like instruction tuning, multiple modes, and reinforcement learning from human feedback. OpenAI and Google were probably looking into the code as well, but they couldn’t – couldn’t – replicate the level of collaboration and experimentation that occurred in subreddits and Discords.

Could it really be that the titanic computational problem that seemed to pose an insurmountable obstacle (a moat) for challengers is already a relic of a different era of AI development?

Sam Altman already pointed out that we should expect diminishing returns when we throw parameters at the problem. Bigger isn’t always better, of course, but few would have guessed that smaller was.

GPT-4 is a Wal-Mart and nobody likes Wal-Mart

The business paradigm that OpenAI and others are pursuing right now is a direct descendant of the SaaS model. You have some high-value software or service, and you offer carefully gated access to it via an API or something similar. It’s a simple, proven approach that makes a lot of sense when you’ve invested hundreds of millions in developing a single, monolithic but versatile product like a large language model.

If GPT-4 generalizes well to answer questions about precedents in contract law, great, never mind that a large part of its “intellect” is devoted to being able to reproduce the style of every author who has ever published a work in English. GPT-4 is like a Wal-Mart. Nobody actually flight to go there, so the company makes sure there is no other option.

But customers are starting to wonder, why am I walking through 50 aisles of trash to buy some apples? Why am I contracting the services of the largest, most general-purpose AI model ever created if all I want to do is exercise some intelligence to match the language of this contract to a couple hundred others? At the risk of torturing the metaphor (to say nothing of the reader), if GPT-4 is the Wal-Mart you go to for apples, what happens when a fruit stand opens in the parking lot?

It didn’t take long in the AI ​​world to get a great language model running, in a very truncated form of course, on a Raspberry Pi. For a company like OpenAI, its jockey Microsoft, Google, or anyone else in the AI-as-a-service world, they’re effectively asking the whole premise of their business: that these systems are so hard to build and run that they have to – it for you In fact, it’s starting to look like these companies chose and designed a version of AI that fit their existing business model, not the other way around!

Once upon a time you had to offload the calculations involved in word processing to a central system – your terminal was just a screen. Of course, that was a different time, and we have been able to adapt the entire application on a personal computer for some time now. These processes have happened many times since then, as our devices have repeatedly and exponentially increased their computing power. These days when anything needs to be done on a supercomputer, everyone understands that it’s just a matter of time and optimization.

For Google and OpenAI, the moment came much faster than expected. And they weren’t the ones who did the optimization, and they may never do it at this rate.

Now, that doesn’t mean they’re out of luck. Google didn’t get to where it’s being the best, not for a long time anyway. Being a Wal-Mart has its perks. Businesses don’t want to have to find the custom solution that does the job they want 30% faster if they can get a decent price from their current supplier and not rock the boat too much. Never underestimate the value of inertia in business!

Of course, people are iterating on LLaMA so fast that they’re running out of camelids to name them. By the way, I would like to thank the developers for an excuse to scroll through hundreds of pictures of cute brown vicuñas instead of working. But few enterprise IT departments will build an implementation of Stability’s ongoing open source derivative of a quasi-legal filtered Meta model over OpenAI’s simple and efficient API. They have a business to run!

But at the same time, I stopped using Photoshop years ago to edit and create images because open source options like Gimp and Paint.net have become so good. At this point, the argument goes in another direction. How much to pay for photoshop? No way, we have a business to run!

What the anonymous authors at Google are clearly worried about is that the distance from the first situation to the second will be much shorter than anyone thought, and it doesn’t seem like anyone can do anything about it.

Except, the note argues: embrace it. Open, publish, collaborate, share, engage. As they conclude:

Google should establish itself as a leader in the open source community, taking the lead by cooperating with, rather than ignoring, the larger conversation. This probably means taking some awkward steps, such as publishing the model weights for small ULM variants. This necessarily means giving up some control over our models. But this commitment is inevitable. We cannot expect to drive innovation as much as to control it.