lilianweng.github.io6 months agoHow to Train Really Large Models on Many GPUs?[Updated on 2022-03-13: add expert choice routing.] [Updated on 2022-06-10]: Greg and I wrote a shorted and upgraded version of this post, published on OpenAI Blog: “Techniques for Training Large Neural Networks”Visit lilianweng.github.ioBookmarkAdd to collection