HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
Paper
•
2503.04598
•
Published
•
16
Jaward Sesay
Jaward
AI & ML interests
I like to train large deep neural nets too 🧠🤖💥 | First Paper (AutoAgents: A Framework for Automatic Agent Generation) Accepted @ IJCAI 2024 | Role Model Karpathy
Recent Activity
posted
an
update
1 day ago
Lightweight (nanoGPT) implementation of hybrid norm - an intuitive normalization method that combines the strength of both pre-norm (i.e QKV-norm in MHA) and post-norm in the feed-forward network.
Code: https://github.com/Jaykef/ai-algorithms/blob/main/hybrid_normalization.ipynb
Organizations
Jaward's activity
upvoted
a
paper
1 day ago
upvoted
a
paper
6 days ago
upvoted
2
papers
about 1 month ago
upvoted
an
article
about 1 month ago
Article
Open-R1: a fully open reproduction of DeepSeek-R1
•
798
upvoted
a
paper
about 2 months ago
upvoted
a
collection
2 months ago
upvoted
4
papers
4 months ago
Multimodal Autoregressive Pre-training of Large Vision Encoders
Paper
•
2411.14402
•
Published
•
43
LLaVA-o1: Let Vision Language Models Reason Step-by-Step
Paper
•
2411.10440
•
Published
•
114
AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant
Paper
•
2410.18603
•
Published
•
32
GPT-4o System Card
Paper
•
2410.21276
•
Published
•
84
upvoted
4
papers
5 months ago
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
Paper
•
2410.06885
•
Published
•
44
Differential Transformer
Paper
•
2410.05258
•
Published
•
171
Emu3: Next-Token Prediction is All You Need
Paper
•
2409.18869
•
Published
•
94
MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models
Paper
•
2409.17481
•
Published
•
47
upvoted
a
collection
5 months ago
upvoted
a
paper
5 months ago
upvoted
3
papers
6 months ago