Offre en lien avec l’Action/le Réseau : – — –/– — –

Laboratoire/Entreprise : LIS, Aix-Marseille Univ.
Durée : 3 years (2021-2024)
Contact : caroline.chaux@univ-amu.fr
Date limite de publication : 2021-05-14

Contexte :
The data deluge and the recent trends in machine learning result in the explosion in the size of the models, with possibly hundreds of billions of parameters [Brown et al., 2020]. Consequences of this phenomenon include major concerns: the difficulty to control those models in terms of design, training, interpretation, security; the need for large computational resources, for training but also for making predictions; the environmental impact that reaches unsustainable levels [Strubell et al., 2019].

Sujet :
The objective of this PhD project is to propose frugal models that are able to handle large volumes of data with efficiency while being structured to provide a reduced time and space complexity. As opposed to distillation techniques [Hinton et al., 2015] that are applied after training, the target structures are intrinsic to the proposed models. Either in deep neural networks or in other machine learning models, the space and time
complexity is mainly due to the linear part of the models, involving large matrices or tensors of data and parameters. A key challenge is to reduce this particular aspect. Appart from the well-known low-rank approaches [Mishra et al., 2013], one promising strategy is to decompose a large N × N matrix into a product of sparse matrices. Such models, named Flexible Multilayer Sparse Approximations [Le Magoarou and Gribonval, 2016] or butterfly factorizations [Dao et al., 2019, Vahid et al., 2020], inherit the structures of the fast transforms like the fast Fourier or Hadamard transforms. Consequently, their typical complexity is in O(N logN) in space and in time for matrix-vector multiplications. Preliminary works have shown how to leverage such models to revisit K-means in a frugal way [Giffon et al., 2021] and to use them for compressing neural networks [Giffon, 2020]. However, such models have not been well controlled so far, both in their ability to model arbitrary data and in their training procedure [Cheukam Ngouonou, 2020, Le Quoc, 2020, Zheng, 2020].
The expected works should contribute to the following research directions:
• proposing new frugal models, e.g., based on sparse/butterfly factorizations;
• studying the properties of such models (expressivity, frugality);
• developing learning algorithms for those models, with techniques including convex, nonconvex and combinatorial optimization;
• deploying such models in machine learning models (neural networks, kernel machines, and so on), tasks and use cases.

Profil du candidat :
Applicants should have excellent general skills in maths and computer science, ideally with some expertise in machine learning, optimization and signal processing. Some science popularization tasks will also be attached to this position.

Formation et compétences requises :
Master degree in maths and/or computer science

Adresse d’emploi :
ECM/LIS, Bureau 511, Bât. Equerre, 38 rue F. Joliot Curie, F-13013 Marseille
France

Document attaché : 202104221240_2021_PhD_offer_frugal_ML_models.pdf