Hello everyone,
This tweet is about a side research project I’ve been busy with for the past two months
@mlnomads
Through this work, I aim to create a white-box deep learning, where you can trace the geometry/topology of your input from the start to the logits.
Before diving into details, please note: this work is still in progress. It needs large-scale validation, so don’t take anything as ultimate truth. Unfortunately, due to unforeseen circumstances, I can’t finish it alone, so I’m looking for your feedback.
As part of exploring contrastive learning (preprint: Simo Loss AFCL), I introduced a new operation: the ⵟ-product (yat), which combines notions of orthogonality/parallelism and distance. Unlike the dot product, it works in a pseudo-metric space, capturing direction and distance while naturally inducing non-linearity.
Using this, I developed a phyisics inspired "neuron", called the ⵟ-neuron, which is the backbone of the first version of Neural-Matter Networks (NMNs)
In NMNs, we no longer need activation functions. Non-linear behavior is achieved intrinsically through the ⵟ-product. The network operates with only softmax for class probability distribution for the output layer and attention (I also proposed softermax, which remove the need for exp operation).
The layer code in flax linen and pytorch are available in the repo, i am gonna share the next part of the blog soon, but before that, I would love to get your feedback on my thoughts and the current state of the work.
Also, during this work I used resources from both the Google Developer Expert program and Google AI/ML Developer Programs.
Thank you for all the support everyone.