This is interesting! But I’d point out that this is a feature of ML research in much of the physical sciences. We have mechanistic models that can be used as a causal, generative ground truth models & use them to further AI research. That was a message of my NeurIPS2016 keynote
Compiling explicit, hand-written algorithms into weights means that we can now generate groundtruth models with a known mechanism. Then we can evaluate existing interpretability tools by applying them to these well-behaving models and examine the resulting explanation.
2/