Three new papers accepted to TMLR!

We have three new papers from our group accepted to the Transactions on Machine Learning Research (TMLR).

In the first paper, “mL-BFGS: A Momentum-based L-BFGS for Distributed Large-scale Neural Network Optimization”, we propose a new momentum-based mechanism to stabilize L-BFGS in stochastic settings. Like the momentum in SGD, the momentum-based L-BFGS is almost cost-free but still captures second-order information to speed up the convergence. We evaluate momentum-based L-BFGS on standard benchmark datasets (including ImageNet) and models. Experiments show momentum-based L-BFGS enjoys a much faster wall-clock convergence rate than other quasi-Newton methods.

Authors: Yue Niu, Zalan Fabian, Sunwoo Lee, Mahdi Soltanolkotabi, Salman Avestimehr

---

In the second paper, “Overcoming Resource Constraints in Federated Learning: Large Models Can Be Trained with Only Weak Clients”, we design a new sub-model training method for federated learning with large models. The proposed method, PriSM, allows all clients to train small sub-models while still attaining a full model on the server side. Through such a training method, we significantly increase the global model’s performance without incurring massive communication and computation burdens on participating clients.

Authors: Yue Niu, Saurav Prakash, Souvik Kundu, Sunwoo Lee, Salman Avestimehr

---

In the third paper, “Revisiting Sparsity Hunting in Federated Learning: Why does the Sparsity Consensus Matters?“, we re-visit sparse training in federated learning and reveal that a consensus of sparsity pattern among all clients in still crucial in federated learning. By learning a sparse pattern beforehand, the proposed method, FLASH, enforces the same sparsity in all client models. With the same sparsity, not only does FLASH reduce overall communication costs, but also improves the overall model performance.

Authors: Sara Babakniya, Souvik Kundu, Saurav Prakash, Yue Niu, Salman Avestimehr