Wrote an annotated Triton implementation of Flash Attention 2. (Links in reply) This is based on the flash attention implementation by the Triton team. Changed it to support GQA and cleaned up a little bit. Check it out to read the code for forward and backward passes along with the math and derivations. Hope this helps understand transformer attention and flash attention better. There's about 60 more annotated deep learning paper implementations on this website.

Letting co-pilot comment on my pull request and then replying to those comments and resolving them makes me feel like a Schizophreniac . But honestly, some of the suggestions are legit useful, so I’m just gonna keep doing that.

docker is supposed to solve the "works on my machine" problem but often I find that it just adds another layer to the "works on my machine" problem esp if you use a mac

I mean, it’s your app.

Going to save this and reply to chat gpt posts on LinkedIn

> I made <thing> from scratch in Python! >look inside >import <thing>

this.

CS grads on suicide watch

OpenAI o3 is 2727 on Codeforces which is equivalent to the #175 best human competitive coder on the planet. This is an absolutely superhuman result for AI and technology at large.

No it isn’t. That’s the whole point

How to remove .env from git

“Oh thanks. Didn’t notice that”