Does anyone know if there are any models specifically trained on data before GPT-3 was widely available? I know Sam has mentioned that GPT-4 could potentially be this but I think we need a site and a model like "before[.]ai" or similar that is specifically dedicated to continual training on ONLY datasets validated to be available before GPT-3 or some other clear line in the sand where we know models we're generating a significant volume of data on the internet. If we don't do this as some sort of public good, open-source, shared offering I think attempting to baseline understanding and separate raw human content from model generated content is going to be impossible. This is not about generating a large extremely capable model for free, but about human vs. AI/ML provenance. This seems like the sort of thing an academic institution or standards agency like NIST, etc. could help maintain with the support of large model vendors @sama @DarioAmodei @elonmusk

I’ve been experimenting with these extended memory agentic frameworks with the knowledge of what it takes to build a global scale systems. I’m finding them amazing for experimenting and prototyping but the debugging skills severely lacking. They appear to always weight training data over supplied evidence.

Probably 8-10 times now I’ve had it implement a spec or configuration based on guidance from 1-2 years ago and when provided the official latest spec it still doesn’t conform. I think there is a disconnect in valuing latest vendor/author supplied information over the volume of data available during training.

🚨 CAUGHT: My utility charged me for 834 kWh over 5.4 hours. 200A @ 240V = 48 kW (48 kWh/hr max) Over 5.4 hrs = 258 kWh max possible They charged 834 kWh = 3.2x IMPOSSIBLE! Could charge 7.7 Teslas! $102.61 bogus. gridsense.us #UtilityBilling