Learn
https://www.youtube.com/watch?v=aircAruvnKk&ab_channel=3Blue1Brown ^ 3b1b vid on Neural Networks
https://www.youtube.com/watch?v=KNAWp2S3w94&list=PLQY2H8rRoyvwWuPiWnuTDBHe7I0fMSsfO&ab_channel=TensorFlow ^ TensorFlow YouTube -> Playlists -> ZeroToHero
https://www.youtube.com/@YannicKilcher/about
^ Good ML community formed around this YouTuber (links to Discord)
http://d2l.ai/ ^ Up-To-Date ML book/resource, uses PyTorch (got reco from above Discord)
https://www.simplilearn.com/keras-vs-tensorflow-vs-pytorch-article ^ Keras / PyTorch / TensorFlow (Theano dying)
CodeFusion: https://arxiv.org/abs/2310.17680v1 -- Applying SD + Transformers to code-gen (v3 has no paper)
Sits on: https://arxiv.org/abs/2212.11685 "Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise"
Unsorted
28 Oct 2023 Bojan https://discord.com/channels/714501525455634453/1045297868136779846/1167622249700917318
https://www.youtube.com/watch?v=THIIk7LR9_8&ab_channel=MITCBMM "Attention Approximates Sparse Distributed Memory " (https://arxiv.org/abs/2111.05498)
https://arxiv.org/abs/2310.08866 Adaptivity and Modularity for Efficient Generalization Over Task Complexity
-> https://arxiv.org/abs/2302.11529 Deep Modular Learning -- ?
-> https://openreview.net/forum?id=1EuxRTe0WN PonderNet -- varying compute-power per token? https://arxiv.org/abs/2107.05407
https://www.rwkv.com/ RNN Transformer-alternative that seems to generate GPTx-level responses fast and cheap
29 Oct
https://arxiv.org/abs/2307.03170 Focused Transformer: Contrastive Training for Context Scaling -- seems to be using a memory-lookup to supply the weights in an attention layer(?)
https://arxiv.org/abs/2310.00194 A Prefrontal Cortex-inspired Architecture for Planning in Large Language Models
Misc.
https://www.youtube.com/@Mutual_Information ^ Course on RL here
https://www.youtube.com/@deeplizard/videos ^ (tx Kake)
https://hackmd.io/@machine-learning/blog-post-cnnumpy-fast#III-Performance-of-fast-implementation ^ Amazing ConvNet-in-numpy
Random Forests
https://www.youtube.com/watch?v=J4Wdy0Wc_xQ&ab_channel=StatQuestwithJoshStarmer Beautful simple explanations from this YouTuber
Graph NNs
https://www.youtube.com/watch?v=GXhBEj1ZtE8&ab_channel=AlexFoo 3b1b-level exposé
Hinton
TODO: How do Hinton's Capsule Networks relate to xformers? https://analyticsindiamag.com/what-happened-with-capsule-neural-networks/
CapsNets: https://www.youtube.com/watch?v=pPN8d0E3900&ab_channel=Aur%C3%A9lienG%C3%A9ron
ForwardForward:
paper: https://www.cs.toronto.edu/~hinton/FFA13.pdf
First I watched https://www.youtube.com/watch?v=rVzDRfO2sgs&ab_channel=EdanMeyer (40min) which draws out some visuals.
Then I listened to https://www.youtube.com/watch?v=NWqy_b1OvwQ&ab_channel=EyeonAI (which unfortunately has none).
Then I downloaded the transcript and picked through it, googling a few things like "Boltzmann Machine" and tidied it up.
impls
- https://github.com/mohammadpz/pytorch_forward_forward (1.2k stars) -- nice diagrams
- https://github.com/JacksonWuxs/Forward-Forward-Network might be better
Also references https://github.com/Trel725/forward-forward/blob/main/forward-forward.ipynb
- https://github.com/rmwkwok/forward_forward_algorithm <-- last commit was 3 days ago, so this one's hot from the oven. Nice 'pen' colour-diags too. And thorough testing.
^ Great rundown on Twitter with pics!