AI systems of today outsource learning to humans. They excel at distilling the knowledge discovered by humans into neural networks. The real prize of AI is in algorithms that can discover knowledge themselves.

The end goal is a learning algorithm, not a set of weights.

@vinalgadhiya See the Oak Architecture and the STOMP progression

Cool, but not surprisingly! A small model that learns on the actual distribution of a task can easily outperform a large model that is aiming for zero-shot high performance on an unseen task.

This is because the world is infinitely complex, and no amount of large-scale learning will ever capture all the nuances of all tasks in the world. Even if we scaled data collection and model training by 100x, we would still not get zero-shot high performance on everything we cared about. This is also why the reliable way to improve performance on any specific benchmark is to use benchmark-specific data or benchmark-specific RL environments.

Once you accept the view that the world is infinitely complex and no amount of learning is sufficient, it becomes obvious that we need to make it as easy as possible for systems to learn on the distribution in deployment. There are many ways to do this; the way that Oak Lab is betting on is to develop efficient continual learning algorithms that make it possible to update large networks in deployment in real time.

The more I learn about the role of data vendors in model updates, the clearer it becomes that we have indeed only discovered a way to distill existing knowledge into networks.

There are many direct and indirect indicators of this. First, the only way to reliably improve performance on a benchmark is to hire people to solve that benchmark first and then use them to generate data for updating the models. Second, big labs keep hiring more and more domain experts and funnelling more and more money to data vendors.

Surely, if we had recreated anything close to human intelligence (or were even on a path that leads to human intelligence), we would not have to rely on networks of thousands of humans to improve models.

(None of this is to say that models are not useful; a mechanism that enables the distribution of human knowledge to millions of users through a simple interface is a huge productivity booster.)

The mistake here was overestimating the capabilities of existing learning algorithms.

It should be self-evident that if we want to manufacture robots using processes that are not exact, or if aspects of their bodies change over time, then there is no way around continual adaptation. Tendon-driven hands introduce both of these challenges.

When your robot bodies are not exact, then just finding a performant policy on one body is not enough. Maintaining performance over time and successfully transferring the same policy to multiple bodies are both non-trivial tasks. This is something that Keen's Physical Atari setup clearly demonstrated.

Historically, the robotics industry has sidestepped this issue by making robots with tight tolerances and by maintaining those tolerances. It seems like Figure is headed in that direction as well.

Ultimately, the right solution is to fix the learning algorithms, organize the knowledge of the robot as self-verifiable subproblems, and have the robot maintain the correctness of this knowledge on its own through continual learning. Everything else is just a band-aid.

I can’t say enough good things about John Carmack @ID_AA_Carmack and his Keen Technologies. But now Khurram Javed @kjaved_ and I have broken away to start our own startup and pursue a slightly different path toward understanding intelligence. Like Keen (and like Ineffable) we at Oak Lab @oaklab believe in reinforcement learning and that intelligence is created and maintained from run-time experience. But we think current deep learning methods are weak and inefficient, and need not more tweaks, but fundamentally new ideas and a thorough reworking before they can provide a solid foundation for achieving the more ambitious goals of AI.

Last month, @RichardSSutton and I left Keen to do our own thing. I had ~two absolutely wonderful years at Keen, and I learned a lot working with John, Gloria, Joseph, and the rest of the team. If you want to work on some of the foundational unsolved problems in AI, such as continual learning, then I would strongly recommend applying to Keen.

Going forward, Rich and I have founded a small company called Oak Lab. We have a fairly complete roadmap to building animal-like intelligence that learns purely from its own experience (the OaK architecture), and Oak Lab is going to follow this roadmap aggressively with a small, focused team. We will be sharing our progress often and aim to build a prototype of the complete OaK architecture in the next few years. A successful prototype will be closer to a baby learning in its first year than it will be to any of the current AI systems.

Our strategy is to demonstrate the limitations of current methods in simple settings, and then work out algorithms that overcome these limitations in a domain-independent way. Only after we have made sufficient progress on the core algorithms will we build large-scale artifacts.

If you are interested in learning about some of the specifics of our approach, then follow @oaklab. We will be sharing more details in the coming weeks and months.

One of those papers that are so timely and relevant that you wish you wrote it (link in the reply).

At some point in the last few years, researchers started proposing RL algorithms that are not compatible with the interface of the RL problem. These algorithms only work in a s for which human engineers can make fast simulators and iterate on those simulators to reduce the sim2real gap. These algorithms are often presented on equal footing with algorithms that learn directly from experience (e.g., papers comparing PQN with DQN).

I think this abuse happened because at some point the nuance that RL researchers use simulations as a proxy for real problems to speed up research was lost and for a subset of researchers, the simulators became the goal. Once simulators are the goal, then spinning up multiple copies of it is a no-brainer. Using dozens or hundreds of parallel environments allows one to side-step key unsolved research problems, but it also significantly limits the value of the resulting algorithms. It is no surprise that despite the strong performance of these algorithms on benchmarks, they have had limited economic value.

The biggest strength of RL, over other paradigms, is that removes the need for humans to figure out the right answers or a model of the environment; it’s important to distinguish between the powerful RL methods that learn from experience and would be hugely valuable from their handicapped counterparts that are only applicable on a small subset of problems.

Caveat: An RL algorithm that learns a model using its own experience is perfectly compaitable with the RL interface. It’s only when the method relies on humans to figure out the model for it that it loses its generality.

Paper link: arxiv.org/pdf/2606.28433

It's interesting that people who are less in the weeds, like Dwarkesh, see the argument for continual learning better than people at the frontier labs.

The world is multiple orders of magnitude more complex than any single agent that will ever exist. In a complex world, unforeseeable situations will always happen and learning is the only scalable way to deal with these situations. Any system that stops learning will be surpassed by systems that continue to learn.

The Physical Atari work is finally in a good state to share. I procrastinate by making nice UIs and visualizations, and I am quite happy with how the website for Physical Atari turned out (link in the reply). 



There are many ways to use this platform for research. For me, an be used to bring some rigour to research on RL applied to robotics.

It is common to use RL in robotics to build demos using sim-to-real or off-policy learning from human-behaviour datasets, but these methods are rarely clearly compared with one another in ways that increase our understanding of the area.



We tried sim-to-real on the Physical Atari system and found the performance to be abysmal. This is perhaps not surprising. Sim-to-real is not a paradigm that scales with compute and data. A human has to constantly be involved to reduce the sim-to-real gap by repeatedly redeploying policies.

I'm sure that, with enough iterations, this gap could be reduced on Physical Atari and sim-to-real could be made to work, but good sim-to-real transfer does not emerge simply from more data and compute.

Surprisingly, we found that even real-to-real transfer---in which policies learned on one robot body are evaluated on a different body---significantly degraded performance. It’s possible that due to differences in robot bodies (arising from manufacturing variation and wear over time), the best policies are learned by doing RL directly on the robot. Learning directly on a robot bakes the exact dynamics into the policies, which can be superior to training for much longer under incorrect dynamics.

The work will be presented at RLC 2026 in Montreal, and the associated paper is on the website.

Links to the paper, code, and instructions for building the robots here: keenagi.com/research/physi…

Yup!

The value of distilling human knowledge into autonomous systems pales in comparison to the value of systems that discover knowledge from experience.

We already know how to distill human knowledge into systems. The frontier of AI research is not doing more of what we already know.

That said, progress in AI applications might continue to be driven by human knowledge, at least until we discover better learning algorithms.

RL Ethics has a Predictive Semantics

I would like to try to explain the view of ethics and values that arises from my research in reinforcement learning in simple, layman’s terms that are accessible to all.

Reinforcement learning agents seek to maximize their reward over time, where reward is essentially pleasure minus pain. This is not quite hedonism, because the maximization takes into account all the consequences, long-term as well as short. A reinforcement learning agent might endure pain to get a larger pleasure later, or forego an immediate pleasure if it stored up later, greater pain. Formally, reward is a number at each time step, and the reinforcement learning agent seeks to maximize value—the sum of the rewards at future time steps. (This could be defined precisely with some math.)

The assignment of rewards to time steps is a free choice that defines the agent’s goal; different agents could have different rewards, and there is no basis (yet) for preferring one set of rewards over another. Value though is a different matter. Given a world and a way of generating rewards, the true values at each time step are fully determined. The rewards are primary, dependent on nothing else, whereas the values are secondary, following from the rewards (and the dynamics of the environment). In decision making, the agent should make the choice that leads to highest immediate value, not highest immediate reward.

If rewards are arbitrary, values follow from the rewards, and correct behavior follows from the values, then all seems straightforward. What about all the complexities and controversies of ethics? Some of these are still present, arising because the values, though well defined, are initially unknown and can be difficult to calculate or learn.

If the agent has knowledge of the world, then it may be able to calculate the values, but to do so exactly generally requires too much knowledge, computation, and memory. In practice, in new situations the calculation must be done partly at decide time, and cannot be done to completion without slowing down action selection too much.

In the absence of knowledge and computation, but given a generous allocation of memory and time, the agent can alternatively learn the values, again approximately. It is common for the agent to store an approximation to the world’s state’s values, and then to gradually improve these approximations—these predictions of subsequent rewards—by further experience. The stored approximate values are immediately available estimates of the desirability of situations; they are directly analogous to our intuitive sense of good and bad. They are ready for immediate use, but may only be rough approximations to the true values. They may be made more accurate with calculation (if the agent has knowledge) or learning (with more experience).

This completes the explication of the value system of the individual. Next we will go on to consider the value systems of groups. But the individual forms such an essential foundation that is never replaced, so let’s dwell on it a moment longer by reviewing its stark tenants: Each agent wants to get pleasure (reward) from the world. Pleasure is built-in to the agent and obvious when it happens, but when it will happen depends on the world and must be learned or calculated—and the world is too complex for either of these methods to yield answers that are completely correct. That is, every state of the world has a real, objective value (the amount of pleasure that will follow it), but estimates of its value are subjective. Forming better value estimates is a major cognitive task. They are a key intermediate step towards getting more pleasure from the world. Agents work on this all the time. It determines what they do.

If an agent lived alone, then this would be the end of our discussion of values and ethics. But people are not solo agents. Peoples’ worlds are comprised, in part, of other people, and this has many impacts of their attempts to estimate value and obtain reward. They live within groups of agents with whom they interact frequently and whom are major determinants of their success is obtaining reward. And thus, to achieve our reward, each of us must take into account, as best we are able, the rewards and values of those around us.
…

The most important insight is that it's alright, and perhaps obligatory, for the ultimate value to be hedonic (based on reward), as long as it is not "selfish" (disregarding the impact on others). The ultimate meaning of something being good, or right, or ethical, or moral, is that it will probably have a good outcome for the individual. Whether it will or not is extraordinarily difficult to calculate, so instead we use heuristics—approximations using features of a situation. The mistake is to think that those features are definitional rather that approximate predictive. The real definitional meaning of good is that it turns our well for us on average.

It is good to have a well-funded place that understands that intelligence is not about distilling human knowledge and skills into neural networks. Most of the current AI systems, while amazing, outsource discovery of knowledge to humans.

Systems that learn from human knowledge, in the limit, will be able to know and do everything that humans can do currently. These systems are immensely valuable, but they will be continually superseded by humans who can both use these systems and learn from their own experience.

Systems that learn from their own experience, in the limit, will be able to do things that no human can do now and no human would even be able to do.

The latter systems would be vastly more powerful.

The naive way to look at the future of AI is to look at numbers like network parameter count, transistor density, and benchmarks, and reason about the future solely from these numbers. This way prevails because it doesn't require thinking about the details of computer architectures and algorithms. It leads to reductionist arguments like "any marginal compute given to an adversary would be devastating because <insert model of the month> is dangerous."

The reality, as Jensen points out multiple times, is more complex. Unless there is an unprecedented breakthrough in chip manufacturing, the path to better AI is through better learning algorithms and architectures. This has been the trend for the last five years and will likely continue to be so. I know from my own work that you can get 100x to 1000x gains in computational efficiency by using better learning algorithms.

Jensen also correctly argues that an ASIC that runs a specific model efficiently is not a replacement for the CUDA stack. I have been working with non-traditional algorithms over the past few years (sparse event-driven neural networks), and the specialized ASICs (including Nvidia's Tensor Cores) are largely useless. CUDA cores, on the other hand, are extremely good at running these algorithms even though they were not designed for them. In a world where CUDA didn't exist, CPUs would be the best computing platform for discovering better algorithms, not TPUs.

You cannot achieve human-like learning at human-like energy consumption simply by improving chip manufacturing. The right algorithms running on the right chip made by the 7 nm process would vastly outperform the current algorithms running on the best chips made by the 2 nm process.

A bunch of us are organizing a workshop at RLC. If your goal is to develop algorithms that allow agents to learn from complex data streams without relying on human data and human designers, then this workshop would be a good fit.

All paths lead to real-time learning in deployment.

The sooner we take the real-time learning setting seriously and develop algorithms that can learn and plan continually in real time, the faster we will get to agents that can achieve goals in big worlds.

World models are all the rage these days, so it's worth reiterating a few points that are largely correct.

1. Yes, our agents need models. The primary use of these models is planning. Planning can be done in real time, to improve an immediate decision, or in the background when not much is going on, to improve future decisions.

2. Learning models that predict the next sensory percept, such as pixels, is insufficient. The models should predict agent state; agent state is a summary of the past observations.

3. Learning one-step models is insufficient. Models should be conditioned on sequences of actions (e.g., option models). Finding what sequences of actions they should be conditioned on is an unsolved problem.

Blender is easily my favorite free software. I wanted to make an overview of our Physical Atari platform for a paper. My first instinct was to use Illustrator with shapes representing different components, but then I realized I have access to 3D models of most components, and I lender instead.

An hour of fiddling with the lighting and layout, and I am very pleased with the results.

This is not all Blender. Modeling the wires would have been painful, so I added them in post with Nano Banana (Gemini). The image generation models are not nearly as steerable as coding models, but with enough context engineering, I was able to get the desired output. I had to draw the path of wires with a brush in Photoshop before passing the images to Gemini, and I had to generate each wire individually. Then I blended the generated images into the original image with some manual work in Photoshop.

I think the idea that these models will replace designers is overblown. At their current capability, they are very useful tools in the right hands (just like the coding models). That said, I have no doubt that these tools will make designers more productive. Adding these wires without Gemini would have taken me forever.

P.S. I tried ChatGPT's image generation too and found it to be not as steerable. Even when it succeeded, it changed the objects in subtle ways that made it impossible to blend in the changes into the original image.

Continual learning is turning into a buzzword. I have always been surprised by the tendency of people to use the term continual learning to mean anything except the obvious---the system continues to learn throughout its lifetime.

I suspect this tendency will get worse as now there are ever stronger incentives to redefine the term so one can label their work as continual learning.