hi i wrote smth :D
kussshhhh.github.io/articles/open-…

only if you knew how connected things are

stay tf away from spirituality chopped people

i think people should get angry more at things they don't like

can't be discussing butterfly shit with caterpillars

my heart is too big to get rage baited but you can keep trying

yeah i have a gambling addiction. i can’t stop betting on myself

hi

thinking of doing smth

so it turned out this was basically just correct

its much easier to verify than to solve and this is the foundation all management runs on, thats why if you have good taste you would want people working under you.

so ideally one must keeping going higher and higher while making sure all that matters is according to their taste :D

hmm hmm

Woke up inside the cozy box and immediately started a tiny battle with the soft grey blanket 🐾

PSA to everyone doing neuron interp after the recent monosemantic neuron posts: the model's training recipe matters a lot.

e.g. GPT2 seems to have been trained WITH dropout (modern LLMs are not), which seems to increase redundancy and (maybe) polysemanticity but reduce superposition (see sources in comments).

Other choices probably also have an effect (e.g. SwiGLU vs GELU MLPs, optimizer, amount of weight decay, etc.) and I think we should be careful not to generalize too much.

Currently working on a project to characterize the effects of these choices on neuron interpretability (a lot of the existing work is done on older/toy models which I'm not sure are representative), feel free to reach out if interested!

@skayfi3 I didn't fully appreciate how mid these people were until they started to reply

in high dimensional spaces you are deterministic

wrote a technical blog on how to train nanogpt2 in moe style without a load balancer. do give it a read.

happy sunday you all! now i should log out, rest and sip some matcha.

its lowkey a war on people with the light still in their eyes

in high dimensional spaces, the lunches are free
~ kai