Mencoba memulai podcast saya sendiri. Rencananya membahas segala sesuatu yang berhubungan dengan data. Episode perdana isunya cukup dekat dengan saya dan @yoseflaw: pendidikan ilmu data. open.spotify.com/episode/2fm…

Deadline is approaching If you have a cool work on SEA NLP, consider submitting to SEALP

🎙️ Recently had a fun chat about my PhD research with @aliakbars in his podcast! We covered on a high-level our (me my supervisors) recent TORS & SIGIR papers on #recsys fairness evaluation ⚖️ Check out other interesting eps in the podcast as well! The podcast is in Indonesian.

Eps. baru siniar Tentang Data! Kali ini membahas fairness in recommender systems bersama @theresia_v_r. Selamat mendengarkan! open.spotify.com/episode/2WP…

1/ What We Learned from 1,747 Responsible AI Papers: The Real-World Impact. Spoiler: it's impactful, but the translation to real-world use is limited. Paper presented today #aies2024 at 6 pm researchswinger.org/publicat… Work with @comarios @aliakbars Here's what we found:

🚨 New paper alert 🚨 Our @PNASNexus study reveals which jobs are most (and least) impacted by #AI after analyzing thousands of AI patents. Curious how AI affects your job? Search for your role in our tool to find out! 🔍 social-dynamics.net/aii 📰academic.oup.com/pnasnexus/a…

Curious how #AI will impact your career? Discover the future of your job with our interactive tool: social-dynamics.net/aii/ The tool is based on our latest research article published in @PNASNexus academic.oup.com/pnasnexus/a… work by @aliakbars @comarios @edytapbogucka)

📢 Sejauh ini sudah ada tiga orang yang mentoring sama saya. Terima kasih dan semoga bermanfaat! 👨🏻‍🏫 Kalau ada yang tertarik mentoring sama saya, sekarang bisa meluncur ke aliakbars.id/mentoring. Ada promo harga khusus untuk 30 menit x 2 sesi langsung!

What happened when MIT stopped paying Elsevier? Not much except they are saving a lot of money. "MIT is interested in collaborating with other libraries to reinvest these funds in community-controlled open publishing initiatives..." sparcopen.org/our-work/big-d…

For those with curious mind: apa yang terjadi kalau generative model dilatih dengan data yang dia hasilkan secara berulang2 (tanpa human feedback loop)? Modelnya "kolaps". nature.com/articles/s41586-0…

📢 Saya mau coba bikin 1:1 berbayar nih. Per sesi durasinya 30 menit. Slotnya terbatas karena menyesuaikan dengan kesibukan saya di sini. Sejauh ini sih bahkan yang 1:1 dari internasional juga puas. Pakai link di bawah ya! calendly.com/ali-septiandri/…

To help explain the weirdness of LLM Tokenization I thought it could be amusing to translate every token to a unique emoji. This is a lot closer to truth - each token is basically its own little hieroglyph and the LLM has to learn (from scratch) what it all means based on training data statistics. So have some empathy the next time you ask an LLM how many letters 'r' there are in the word 'strawberry', because your question looks like this: 👩🏿‍❤️‍💋‍👨🏻🧔🏼🤾🏻‍♀️🙍‍♀️🧑‍🦼‍➡️🧑🏾‍🦼‍➡️🤙🏻✌🏿🈴🧙🏽‍♀️📏🙍‍♀️🧑‍🦽🧎‍♀🍏💂 Play with it here :) colab.research.google.com/dr…

Adik2 mahasiswa, friendly reminder kalau sitasi pendapat si X, itu baiknya diambil dari tulisan si X langsung -- bukan dari tulisan Y yg di dalamnya ngutip tulisan X. Ada pengecualian, yes, tapi pakem dasarnya gitu. Ini hal sederhana tapi sering saya temukan pas review artikel.

Our CHIMERA team had a very successful couple of days visiting @HealthcareHubEX last week. We took part in a hackathon where we won the 'novelty of approach' category, and ECRs presented their research.

WEIRD #ICWSM! How Western, Educated, Industrialized, Rich, and Democratic is Social Computing Research? with @comarios @aliakbars there's bad & good news though... We recommend slightly expanding the current "paper checklist"! arxiv.org/pdf/2406.02090 #FAccT2024 #CHI2024

🌏 We are organising an Urban Recommender Systems workshop #UrbanRec at @ACMRecSys 2024. Join us in Bari, Italy, in October. The deadline for paper submissions is August 2nd! urbanrec.github.io/UrbanRec2…

Hiburan tengah minggu nih. Eps terbaru siniar Tentang Data membahas tentang tim data vs PM. Jadi sebetulnya PM itu harus bisa SQL ngga sih? Terus seberapa dalam tim data perlu punya product knowledge? Simak obrolan saya dan Gurun Nevada di eps kali ini! open.spotify.com/episode/6iE…

Akhirnya episode yang (hilang) ini bisa dirilis. Saya ngobrolin tentang antropologi dan studi etnografi sama Mas Ibnu Nadzir di episode terbaru siniar Tentang Data. Selamat mendengarkan! podcasters.spotify.com/pod/s…

# explaining llm.c in layman terms Training Large Language Models (LLMs), like ChatGPT, involves a large amount of code and complexity. For example, a typical LLM training project might use the PyTorch deep learning library. PyTorch is quite complex because it implements a very general Tensor abstraction (a way to arrange and manipulate arrays of numbers that hold the parameters and activations of the neural network), a very general Autograd engine for backpropagation (the algorithm that trains the neural network parameters), and a large collection of deep learning layers you may wish to use in your neural network. The PyTorch project is 3,327,184 lines of code in 11,449 files. On top of that, PyTorch is written in Python, which is itself a very high-level language. You have to run the Python interpreter to translate your training code into low-level computer instructions. For example the cPython project that does this translation is 2,437,955 lines of code across 4,306 files. I am deleting all of this complexity and boiling the LLM training down to its bare essentials, speaking directly to the computer in a very low-level language (C), and with no other library dependencies. The only abstraction below this is the assembly code itself. I think people find it surprising that, by comparison to the above, training an LLM like GPT-2 is actually only a ~1000 lines of code in C in a single file. I am achieving this compression by implementing the neural network training algorithm for GPT-2 directly in C. This is difficult because you have to understand the training algorithm in detail, be able to derive all the forward and backward pass of backpropagation for all the layers, and implement all the array indexing calculations very carefully because you don’t have the PyTorch tensor abstraction available. So it’s a very brittle thing to arrange, but once you do, and you verify the correctness by checking agains PyTorch, you’re left with something very simple, small and imo quite beautiful. Okay so why don’t people do this all the time? Number 1: you are giving up a large amount of flexibility. If you want to change your neural network around, in PyTorch you’d be changing maybe one line of code. In llm.c, the change would most likely touch a lot more code, may be a lot more difficult, and require more expertise. E.g. if it’s a new operation, you may have to do some calculus, and write both its forward pass and backward pass for backpropagation, and make sure it is mathematically correct. Number 2: you are giving up speed, at least initially. There is no fully free lunch - you shouldn’t expect state of the art speed in just 1,000 lines. PyTorch does a lot of work in the background to make sure that the neural network is very efficient. Not only do all the Tensor operations very carefully call the most efficient CUDA kernels, but also there is for example torch.compile, which further analyzes and optimizes your neural network and how it could run on your computer most efficiently. Now, in principle, llm.c should be able to call all the same kernels and do it directly. But this requires some more work and attention, and just like in (1), if you change anything about your neural network or the computer you’re running on, you may have to call different kernels, with different parameters, and you may have to make more changes manually. So TLDR: llm.c is a direct implementation of training GPT-2. This implementation turns out to be surprisingly short. No other neural network is supported, only GPT-2, and if you want to change anything about the network, it requires expertise. Luckily, all state of the art LLMs are actually not a very large departure from GPT-2 at all, so this is not as strong of a constraint as you might think. And llm.c has to be additionally tuned and refined, but in principle I think it should be able to almost match (or even outperform, because we get rid of all the overhead?) PyTorch, with not too much more code than where it is today, for most modern LLMs. And why I am working on it? Because it’s fun. It’s also educational, because those 1,000 lines of very simple C are all that is needed, nothing else. It's just a few arrays of numbers and some simple math operations over their elements like and *. And it might even turn out to be practically useful with some more work that is ongoing.