The Idea: Build Neural Nets to do simple tasks (or multiple simple tasks). Then evolve the working NN into an LLM/Transformer (e.g., by adding GPT-2 components piece by piece) while preserving the abilities.
NN Specimen:
PARITY-calculatingNN_Schmidhuber1.py at http: https://huggingface.co/MartialTerran/Neural_Nets_Doing_Simple_Tasks/blob/main/PARITY-calculatingNN_Schmidhuber1.py
Inspired by: Aug 28, 2024 Youtube Interview of Juergen Schmidhuber at https://www.youtube.com/watch?v=DP454c1K_vQ See also (seven years ago) True Artificial Intelligence will change everything | Juergen Schmidhuber | TEDxLakeComo www.youtube =-Y7PLaxXUrs
[Jürgen Schmidhuber, the father of generative AI shares his groundbreaking work in deep learning and artificial intelligence. In this exclusive interview, he discusses the history of AI, some of his contributions to the field, and his vision for the future of intelligent machines. Schmidhuber offers unique insights into the exponential growth of technology and the potential impact of AI on humanity and the universe.]
In this interview, Schmidhuber stated that LLMs cannot compute "parity" of bits in a binary number sequence, but that "Recurrent" NNs (RNNs) can compute "parity".
So, I wanted to know whether a simple feed forward NNs can compute "parity". (If so, perhaps LLMs actually can compute "parity" if specifically trained to do so) By buidling an NN that can compute parity, we establish roughly the minimum amount of parameters that are reqired for such a task, and then we try to reproduce the ability within a Small Toy Transformer on the same scale, or scale up until the ability is restored. This vertical method of experimentation-in-evolution of AI can become a chapter test in a college course in Artificial Intelligence.
Method of the python script informed by:
Create a Basic Neural Network Model - Deep Learning with PyTorch 5 - https://www.youtube.com/watch?v=JHWqWIoac2I (2023-06-05)
and Building a Neural Network with PyTorch in 15 Minutes https://www.youtube.com/watch?v=mozBidd58VQ