"The model will converge anyway" - a compelling argument, and a costly misconception. Two models. Same everything. Model A, step 0: loss = 3.29 → starts learning immediately. Model B, step 0: loss = 27 → spends the first 9,000 steps just getting back to where A started. Model B didn't train for 10k steps. It trained for ~800. The rest was debt repayment. A bad initialization doesn't slow you down. It steals your training budget — silently, one "optimization" step at a time. 🧵

For a vocabulary of 27 characters (26 letters 1 special token), a uniform distribution means the probability of predicting the correct next character is exactly 1/27 = 0.037. This represents a baseline of zero information. We measure performance using Negative Log Likelihood (NLL). We use the negative log because probabilities are <= 1 (yielding negative logs), and we want to minimize a positive penalty. The step-by-step derivation for our initial loss: Step 1: P = 1/27 Step 2: ln(1/27) approx -3.2958 Step 3: Apply the negative sign -> 3.2958 This value serves as a critical Sanity Check. If your initial loss is significantly higher (e.g, 27), it indicates a "degenerate" initialization where the model is confidently wrong. This usually stems from high-magnitude logits that force the Softmax function to collapse onto a single incorrect class.