Pretty sure a lot of new ‘scraping’ is actually a person getting an AI to browse a single site for them at a time, not an automated crawl of everything. Smart companies should also have a AI-friendly page, simple and with all text in one place, if they want to stay relevant.
Easily solution: Allow scraping, be open like the internet was meant to be. Wikipedia surely is less funded than this place and allows all forms of scraping known to man.