You misread. There had been multiple LLM projects within FAIR for years. Some were open sourced as research prototypes (e.g. OPT175B, Galactica, BlenderBot...). In mid-2022, FAIR started a large LLM project called Zetta, which was still going in late 2022 when ChatGPT came out. A small group at FAIR-Paris was working on theorem proving. They needed an LLM for their own purpose and thought Zetta was too big and not ready. They developed their own model, which eventually became Llama-1. What happened internally between Zetta and Llama is somewhat similar to what just happened between DeepSeek and the big US players: a small team of talented folks innovated and beat the large teams.

And finally @Yampeleg. You should just watch it. PS He read OPT175B logbook so we won't have to.

OPT175Bですらまだ試せていないのにLLaMAまでてきた🦙

webbigdata.jp/post-17639/ ちなみにFlanT5の論文でOPT175Bというメタがオープンソースにした大規模言語モデルがあるがそいつも明示的にはコードで学習していないっぽい。下の画像はOPT論文の学習データセット。Githubが乗ってない。 GoogleもMETAもコード学習した言語モデル出さないのはなぜか？

"Better" is not entirely true according to HELM. It is much better than OPT175B in "Toxicity" category (and better than GPT3 as well). And a little bit better in "Efficiency" and "Robustness". But much worse in "Accuracy", and "Bias".

Everyone's talking about #ChatGPT, but did you know there was a free, #opensource equivalent of #GPT3 called Bloom? It's better than OPT175B, and you can easily use it in Transformers or @huggingface's API inference! 🤯 Code in the screenshot's alt. 🔗 huggingface.co/bigscience/bl…

I'm blown away by how powerful #GPT3 and #OPT175B are! I didn't realize there was a free, #opensource version of GPT-3. Amazing what technology can do! #ChatGPT

Yeah - that definitely makes "publishing" these things more challenging. One thing i think does work very well is something like the OPT175B "Lab Logbook" publication model: x.com/PhilippHennig5/status/… this was absolutely invaluable!

METAオワコン雰囲気ある人にはあるが、エグいぐらいのAI技術を持っているのも一応認識した方がいい。大規模言語モデルNLLB200、OPT175b,映像生成AI MAKE A VIDEO ,トランスフォーマーモデル超えるかもなMEGA、ロボットハンド強化学習のAI PGDM,Speech2Speech翻訳 METAはメタバースをAIで加速するのでは

以下の推定によるとGPT3は10^12FLOPSで4トークン程度生成する。docs.google.com/document/d/1… とすると現在の最新のGPUでRTX4090なんかは10^14FLOPSいかないくらいはあるからGPU１個で推論計算は余裕そうなんだけど、OPT175Bは256GB VRAM要求するそうで、10台上記GPUないとダメだけどなんでだろう。気になる

米メタが巨大言語モデルを公開　未熟で危険な技術を世に放つ成否 #OPT175B #言語モデル #GPT3 #IT経営 #業界動向 dlvr.it/SRnj64

Unlike many other large language models, @Meta's OPT-175B will be available for free to all researchers or institutions that request access. #languagemodels #languagemodel #largelanguagemodel #meta #tech #gpt3 #OPT175B #AI #artificialintelligence

An open-source model as powerful as GPT-3! - mailchi.mp/6dae87ca23a9/an-o… #ai #opt #gpt #gpt3 #opt175b #meta #metaai #openai #artificialintelligence #machinelearning #datascience

🤖 Meta's new AI language model OPT-175B ‘has a higher toxicity rate” and it “appears to exhibit more stereotypical biases in almost all categories except for religion.” technologyreview.com/2022/05… #AI #OPT175B

Würde mich freuen, wenn ihr euch den Thread zum neuen AI Model von Meta (Facebook) anschaut, ich find dass so spannend! #OPT175B

Meta has released a huge new AI language model called OPT-175B and made it available to a broad array of researchers. It also released a technical report with some truly extraordinary findings about just how dangerous this machine can be. 🧵 #AI #OPT175B

OPT-175b: Open Pre-Trained #language model with 175 billion paramaters is now available to the #research community. #NLP #MachineLearning #OPT175B ai.facebook.com/blog/democra…

Schon wieder Advent? Einen Tag ist man im Konferenztunnel, schon steht das nächste große Sprachmodell in der Tür. Mein Kollege @MengeSonnentag hat eine lesenswerte Meldung zu #OPT175B von #Meta getickert 🫴 #LLM #AI #Zeitenwende