Now you can use bitsandbytes on AMD GPUs and Intel hardware. This is a big milestone and was a huge undertaking. @Titus_vK did an amazing job here. Eager to hear feedback! Let us know how it works for you.

Our Undergraduate Research Internship Program is now accepting applications for summer research internships. Rising college juniors & seniors who are passionate about technology and champion diversity and inclusion are encouraged to apply by 10/21: msft.it/6017meOVF

The OpenAI drama feels like what would happen if a Sociology department accidentally became a Fortune 500 company

Last week, the WSJ reported on the decline of tech jobs, particularly entry-level and recruiting roles, suggesting they might be gone for good. I initially thought it was exaggerated, but then saw a UC Berkeley CS professor mention that even his top students are struggling to get job offers. That’s concerning. My kid is far from college, but I’m curious—how are parents of high schoolers approaching career guidance for their kids in light of this?

🚨 Johns Hopkins @JHUCompSci is hiring faculty at all ranks! 1) Data Science and AI 🤖; 2) All other areas of CS 💻. We will double in size to become one of the largest departments in the coming years! 🚀 Note: early action and spousal placement. 👥 🧵 apply.interfolio.com/153420

Thoughts on OSI’s *draft* open-source AI definition v0.0.9: Key question: how much information about the data needs to be disclosed to get the open-source stamp? My initial reaction was that the entire dataset must be released, but now I think this is neither sufficient nor necessary.

We are proud to share this article, @yian_yin (PhD ’22), who has accomplished plenty since earning his PhD in 2022 from IEMS at @NorthwesternEng. He was listed in @Forbes 2024 ’30 Under 30’ in the science category. tinyurl.com/tt7j4yvy

[Backpropagation] by Hand✍️ [1] Forward Pass ↳ Given a multi layer perceptron (3 levels), an input vector X, predictions Y^{Pred} = [0.5, 0.5, 0], and ground truth label Y^{Target} = [0, 1, 0]. [2] Backpropagation ↳ Insert cells to hold our calculations. [3] Layer 3 - Softmax (blue) ↳ Calculate ∂L / ∂z3 directly using the simple equation: Y^{Pred} - Y^{Target} = [0.5, -0.5, 0]. ↳ This simple equation is the benefit of using Softmax and Cross Entropy Loss together. [4] Layer 3 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W3 and ∂L / ∂b3 by multiplying ∂L / ∂z3 and [ a2 | 1 ]. [5] Layer 2 - Activations (green) ↳ Calculate ∂L / ∂a2 by multiplying ∂L / ∂z3 and W3. [6] Layer 2 - ReLU (blue) ↳ Calculate ∂L / ∂z2 by multiplying ∂L / ∂a2 with 1 for positive values and 0 otherwise. [7] Layer 2 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W2 and ∂L / ∂b2 by multiplying ∂L / ∂z2 and [ a1 | 1 ]. [8] Layer 1 - Activations (green) ↳ Calculate ∂L / ∂a1 by multiplying ∂L / ∂z2 and W2. [9] Layer 1 - ReLU (blue) ↳ Calculate ∂L / ∂z1 by multiplying ∂L / ∂a1 with 1 for positive values and 0 otherwise. [10] Layer 1 - Weights (orange) & Biases (black) ↳ Calculate ∂L / ∂W1 and ∂L / ∂b1 by multiplying ∂L / ∂z1 and [ x | 1 ]. [11] Gradient Descent ↳ Update weights and biases (typically a learning rate is applied here). 💡 Matrix Multiplication is All You Need: Just like in the forward pass, backpropagation is all about matrix multiplications. You can definitely do everything by hand as I demonstrated in this exercise, albeit slow and imperfect. This is why GPU's ability to multiply matrices efficiently plays such an important role in the deep learning evolution. This is why NVIDIA is now close to $1 trillion in valuation. 💡Exploding Gradients: We can already see the gradients are getting larger as we back-propagate up, even in this simple 3-layer network. This motivates using methods like skip connections to handle exploding (or diminishing) gradients as in the ResNet. I did the calculations entirely by hand. Please let me know if you spot any error or have any questions!

learned something super interesting this week if you train GPT-2 to multiply, you can't even train it to multiply 4-digit numbers (30% accuracy) but if you use a really clever (and somewhat complex) training scheme, GPT-2 can generalize up to 20-digit numbers (100% accuracy) it turns out that the reason GPT-2 couldn't multiply four-digit numbers wasn't too few layers, too small of a hidden dimension, or bad training data. all that was fine. the learning algorithm itself was the issue the better model was built with a clever trick on regular training -- first using chain-of-thought, then progressively removing tokens as the model learns to solve problems in fewer and fewer steps what this tells me is that our standard model-training is suboptimal (supervised learning SGD Adam) these experiments come from this paper: From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step. it's only a couple months old and while implicit CoT is super cool, its insights are much more general: this is proof that our current way of doing optimization is not the best there's clearly a better learning algorithm out there

Problems with Perseus 4 and work towards Perseus 6 » Perseus Digital Library Updates sites.tufts.edu/perseusupdat… via @PerseusDigLib