Pinned Tweet
sharing an essay on the relation between compression and intelligence, in commemoration of Ray Solomonoff's 100th birthday greene.sh/compression-an…
8
41
383
43,037
a universe saturated with intelligence and a universe at maximum entropy may be indistinguishable from the perspective of a computationally-bounded observer
34
9
194
13,742
TIL von Neumann's self-replication architecture preceded our understanding of the mechanics of DNA (which turned out to be the same)
10
11
240
7,968
the economy of a world with greater than human intelligence will not be ran by humans
so just to clarify the whole cerebras drama:
- openai *is* using cerebras chips for ultrafast but they ran out because Jane Street bought up all their supply
- jane street’s paying cerebras $200M PER megawatt just to access super fast inference. that’s insane. cerebras cost is ~$26M per mw.
so cerebras has a good chip + is making bank?
1
0
18
2,901
Ryan Greene retweeted
We've been obsessed with bending the scaling laws recently and our new paper shows that looping with model growth improves the scaling exponent, leading to gain that compounds with compute!
Everyone assumes architectural changes only give constant factor gains and pretraining rom data (e.g. @dwarkesh_sp's recent post).
We found that model growth, looping, and boundary operators result in compute multipliers over standard transformers that grow exponentially with each OOM of compute.
- 1.55x at 1e20 FLOPs and 2.7x projected at 1e25.
- Matches GPT-3 13B on CORE with 20x less compute
w/ @charllechen, @akshayvegesna, @andrewgwils
🧵
Everyone assumes architectural changes only give constant factor gains and pretraining rom data (e.g. @dwarkesh_sp's recent post).
We found that model growth, looping, and boundary operators result in compute multipliers over standard transformers that grow exponentially with each OOM of compute.
- 1.55x at 1e20 FLOPs and 2.7x projected at 1e25.
- Matches GPT-3 13B on CORE with 20x less compute
w/ @charllechen, @akshayvegesna, @andrewgwils
🧵
12
49
448
76,726
Ryan Greene retweeted
I am hearing disturbing rumors that Ilya Sutskever has made the Boltzmann Machine computationally tractable.
4
2
49
7,112
an impressive jump that i have to assume is the result of being the first frontier model to have RL'd the public demo environments, which is a rather large confounder on what Chollet is trying to get these benchmarks to measure (out-of-distribution generalization)
On ARC-AGI-3, an evaluation where AI models must solve novel problems, Opus 5’s score is three times as high as the next best model.
20
23
351
44,459
i do not mean to cast any shade towards Anthropic (all of the labs are going to be doing this), just highlighting a flaw that has been prevalent throughout the ARC-AGI variations
3
1
48
2,581
oh, you’re a capabilities researcher? are you minimizing the description length of the digital universe, or embedding it with a will to power?
2
4
55
6,262
Schmidhuber's obsession with credit assignment is just another example of his great taste in the most important research directions :)
Now of course Schmidhuber is divisive and does spend a lot of time trying to appropriately assign credit in the history of AI and through this has become something of a punchline. But you have to admit that he has demonstrated incredible vision and foresight.
4
3
115
15,012
was not expecting b2b saas on the wikipedia article for "List of works based on dreams"
2
0
38
2,450
this is a fantastic book, highly recommend it and checking out Antikythera more generally
@ryangr great visualization! discussion of this paper was my favorite part of What Is Intelligence by Blaise Agüera y Arcas
whatisintelligence.antikythera.org/chapter-01/#vi…
3
0
8
2,062
for a little toy project i reproduced the quoted artificial life paper: a 2D grid of randomly generated Brainfuck programs breed and spontaneously evolve self-replicators, despite no explicit optimization functions
i mean this literally. given an infinite universe where self-replicating (sustaining) is possible; after enough time, it is *inevitable*, and once created, entropy will destroy all else: the universe becomes more and more selective for the self-replicating
62
78
1,172
174,654
every pixel is an instruction; each instruction has a unique color, while black represents a value on the tape that is raw data storage / not an instruction. every 8x8 section of pixels represents a single program
in this run, a self-replicator emerges relatively early on and soon takes over most of the grid, until a more efficient self-replicator evolves and takes over everything
in this run, a self-replicator emerges relatively early on and soon takes over most of the grid, until a more efficient self-replicator evolves and takes over everything
3
0
70
9,617
the entire program is just 300 lines of code, most of which is just boiler plate for parallelization and gif generation github.com/Rabrg/artifici…
12
6
89
28,617