논문해외
Muon Makes the First Mixture-of-Experts Layer Harder to Balance
HF Daily Papers
Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW. Most balance the load with a bias on each expert's routing score, adjusted after every step.
원문의 저작권은 HF Daily Papers에 있습니다. 위 발췌는 원문 확인을 돕기 위한 짧은 인용입니다.