Le Chonk's Tokenizer. LeChonkenizer?
![model_type: BPE
vocab_size: 204800
num_special_slots: 1000
num_named_special: 28
num_filler_special: 972
num_bpe_tokens: 203800
num_merges: 423969
merges_per_token: 2.08
ignore_merges: True
byte_fallback: False
normalizer: None
pre_tokenizer: ['Split', 'ByteLevel']
post_processor: ByteLevel
decoder: ByteLevel
regex: [^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}{1,3}(?=(?:\p{N}{3})*(?!\p{N}))|\p{N}{1,2}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+
special_tokens: [(0, '<unk>'), (1, '<s>'), (2, '</s>'), (3, '[INST]'), (4, '[/INST]'), (5, '[AVAILABLE_TOOLS]'), (6, '[/AVAILABLE_TOOLS]'), (7, '[TOOL_RESULTS]'), (8, '[/TOOL_RESULTS]'), (9, '[TOOL_CALLS]'), (10, '[IMG]'), (11, '<pad>'), (12, '[IMG_BREAK]'), (13, '[IMG_END]'), (14, '[PREFIX]'), (15, '[MIDDLE]'), (16, '[SUFFIX]'), (17, '[SYSTEM_PROMPT]'), (18, '[/SYSTEM_PROMPT]'), (19, '[TOOL_CONTENT]'), (24, '[AUDIO]'), (25,](/pic/orig/media%2FHVz0775XIAAVEXm.jpg)
Today, we release the weights of a new and improved Le Chonk, advancing the open frontier for agentic coding and cybersecurity.
💻 Built to build: Le Chonk ranks #1 on @cognition’s FrontierCode among open models, and sets a new benchmark for Pareto efficiency on Built to defend: State-of-the-art in cyberdefense, put through the paces by the world's leading cyber vendors, including @CrowdStrike, @cloudflare, @rubrik, @zscaler, and others.
Built for sovereignty: Make it yours. Download the weights on @huggingface , customize it with Mistral Forge, and deploy it wherever you need it.
Read more from our team and customers 👇:
1
0
12
428
The @huggingface version of our chess position evaluations dataset just crossed 1 billion rows. That's 1 billion @stockfishchess evaluations from games played on @lichess!
4
3
19
4,312
Interesting release by @jure Leskovec's group: RT-J (86M "relational transformer" ) and THE JOIN, a web-crawled corpus of 639 relational databases with 6K forecasting tasks. (data and models on @huggingface)
3
1
19
1,186
Can't get Schmidhubered if I Schmidhuber myself.
Everyone wants those AI scientists! But how possible is it for an LLM to generate novel research ideas rather than just optimizing hyperparameters (broadly construed)?
Building directly on the once novel work of @SchmidhuberAI!
1
0
4
281
Christopher Akiki retweeted
🚨 [Token][ization] Paper Alert 🚨
Tokenization is a wildly understudied area of language modeling despite it having effects across all of NLP. Over the past ~8 months, 32 (!) tokenizer researchers put together the most comprehensive survey of the field. Check it out!
Tokenization is a wildly understudied area of language modeling despite it having effects across all of NLP. Over the past ~8 months, 32 (!) tokenizer researchers put together the most comprehensive survey of the field. Check it out!
6
83
346
41,217
Christopher Akiki retweeted
ChessLFM went full circle and is now on the home page of @lichess
Super cool to see, Lichess is where I collected all the seed data. Based on my limited chess skills, I never thought I'd have an article there.
Super cool to see, Lichess is where I collected all the seed data. Based on my limited chess skills, I never thought I'd have an article there.
4
1
56
3,647
t-SNE on SigLIP embeddings of 1,360 @OReillyMedia cover animals. The penguin cluster is my favorite.
O'Reilly animals sorted by orientation using PCA.
3
16
126
7,235
O'Reilly animals sorted by orientation using PCA.
Image quilt of all @OReillyMedia book covers sorted by color.
15
189
2,429
197,041
Image quilt of all @OReillyMedia book covers sorted by color.
An image quilt of all @OReillyMedia animals.
1
10
179
206,070
Christopher Akiki retweeted
single 5090 running parametric umap on 8 million points. getting ready to push to 16 million then 150 million 🌌
2
4
19
1,143
The McManifold
(2216 burger recipes as 146-dimensional ingredient vectors)
(2216 burger recipes as 146-dimensional ingredient vectors)
Using burgers as a model system, a generative AI system rediscovers the classic Big Mac without explicit supervision and generates novel burgers optimized for deliciousness, sustainability, or nutrition, according to a paper in npj Science of Food. go.nature.com/3SU7JDQ
0
4
17
1,478
work in progress: 1 million synthetic personas from @NVIDIAAI's Nemotron-Personas-USA dataset.
1
0
9
559
