💯agreeing with
@DrJimFan. I think there will be a point of time we won't be able to distinguish if we interact with real humans or virtual agents and NPCs within virtual worlds.
Metaverse platforms are looking to drive further engagement in their virtual world, (re)inventing social interactions and the way we communicate with each other. The Creators Economy and Economy of Attention are closely connected and will further be.
Besides
@nvidia, companies like
@inworld_ai or
@virtualbeings among many others are building solutions for platforms like
@TheSandboxGame @decentraland @Roblox @FortniteGame and many others to integrate easily
Before we have a million robots in the physical world, we will first see a billion embodied agents in virtual worlds. Gaming is the second major area I'm dedicated to in 2024. AI and Gaming are born for each other, and their happy marriage is just getting started.
On one hand, open-ended games provide a "primordial soup" for generalist AI to emerge. An agent's capabilities are upper-bounded by the complexity of the world it lives in. Minecraft is a prime example. In 2023, we saw an explosion of new algorithms enabled by Minecraft. To name a few:
- MineCLIP (from our team's open-source framework MineDojo): learn reward functions by watching 100,000s of YouTube gameplay videos.
- VPT (OpenAI): imitate behavior by pseudo-labeling actions from YouTube.
- Voyager: in-context lifelong learning with an explicit Skill Library of code.
- STEVE-1: guide actor to follow commands in MineCLIP's latent space, inspired by DALLE-2
- And many more: DEPS, Jarvis-1, DreamerV3 ...
Besides Minecraft, there are a lot more games that require extremely advanced perception, agility, exploration, reasoning, and planning. We are just starting to scratch the surface.
I believe games (and simulation in general) will provide the next trillion high-quality tokens to train our foundation models. What's cool is that these tokens are actively selected by the agent itself through exploration. It can choose to experiment with things that maximally reduce its internal uncertainties - kind of like how human curiosity works.
On the other hand, AI will lead to a paradigm shift in the Gaming industry. This year, we see a surge of community interest in Stanford Smallville, where 25 AI agents inhabit a digital town. They go to work, gossip, organize socials, make new friends, and even fall in love.
It's an exciting experiment, but we still have not felt any impact on the real games out there. This is because our LLMs are too boring and too expensive. If you take a look at Smallville's chat log, you will find that the conversations are not fun at all. No parents talk to their kids in such polite manner. The unit economy of deploying so many agents also does not make sense at scale.
That being said, I believe 2024 is an inflection point. The Digital Westworld is coming, and will transform the industry once and for all. Games will feel truly alive. The characters will interact with humans and each other, form relationships, take consistent actions over their lifetime, and react in human-like ways. Each game will have infinite replay value, and each player will have unique and tailored experience.