What kind of data should we prioritize during self-training? Confident ❌ Uncertain ✅ We’re excited to introduce 🤔EAST 😎— a novel weighting strategy that prioritizes uncertain data during self-training. EAST uses a mapping function with a tunable sharpness parameter to control how strongly uncertainty influences the weighting. 🔬 Empirically, we show that emphasizing uncertain data improves reasoning performance. 📄 Paper: arxiv.org/pdf/2503.23913 💻 Code: github.com/mandyyyyii/east

🎉Super excited to have presented our poster for "From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction" at ICLR 2024! Check out our webpage for our code, interactive visualization, and more! nima.sh/jmp/

Meme this!

SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models paper page: huggingface.co/papers/2307.1… Recent advances in large language models (LLMs) have demonstrated notable progress on many mathematical benchmarks. However, most of these benchmarks only feature problems grounded in junior and senior high school subjects, contain only multiple-choice questions, and are confined to a limited scope of elementary arithmetic operations. To address these issues, this paper introduces an expansive benchmark suite SciBench that aims to systematically examine the reasoning capabilities required for complex scientific problem solving. SciBench contains two carefully curated datasets: an open set featuring a range of collegiate-level scientific problems drawn from mathematics, chemistry, and physics textbooks, and a closed set comprising problems from undergraduate-level exams in computer science and mathematics. Based on the two datasets, we conduct an in-depth benchmark study of two representative LLMs with various prompting strategies. The results reveal that current LLMs fall short of delivering satisfactory performance, with an overall score of merely 35.80%. Furthermore, through a detailed user study, we categorize the errors made by LLMs into ten problem-solving abilities. Our analysis indicates that no single prompting strategy significantly outperforms others and some strategies that demonstrate improvements in certain problem-solving skills result in declines in other skills. We envision that SciBench will catalyze further developments in the reasoning abilities of LLMs, thereby ultimately contributing to scientific research and discovery.

Paper: arxiv.org/abs/2401.12255 Website: cnut1648.github.io/Model-Fin… We present a pilot study on LLM fingerprinting as a form of very lightweight instruction tuning. Model publisher specifies a confidential private key and implants it as an instruction backdoor that causes the LLM to generate specific text when the key is present. Results showed that this approach is lightweight and does not affect the normal behavior of the model. It also prevents publisher overclaiming, maintains robustness against fingerprint guessing and parameter-efficient training, and supports multi-stage fingerprinting akin to MIT License.