Journal of High Energy Physics(2024)SCI 2区SCI 1区
National Institute for Subatomic Physics | University of Warwick | University of Zurich | University of Liverpool | University of Bristol | Uppsala University | Centre National de la Recherche Scientifique | European Organization for Nuclear Research | University of Michigan–Ann Arbor | University of Cincinnati | National Laboratory of Frascati | TU Dortmund University | University of Glasgow | University of Manchester | University of Cambridge | Sorbonne Paris Cité | Universidade Federal do Rio de Janeiro | Laboratoire de Physique des 2 Infinis Irène Joliot-Curie | INFN Sezione di Firenze | INFN Sezione di Ferrara | Syracuse University | Institute of High Energy Physics | INFN Sezione di Roma II | Massachusetts Institute of Technology | University of Oxford | Université Savoie Mont Blanc | Heidelberg University | École Polytechnique Fédérale de Lausanne | Sorbonne Université | Universidade de Santiago de Compostela | Centro Brasileiro de Pesquisas Físicas | University of Edinburgh | RWTH Aachen University | INFN Sezione di Pisa | INFN Sezione di Genova | Universitat Ramon Llull | University of Padua | Institute of Nuclear Physics | Imperial College London | University of Chinese Academy of Sciences | Maastricht University | INFN Sezione di Milano Bicocca | University of Maryland | South China Normal University | INFN Sezione di Cagliari | Wuhan University | INFN Sezione di Milano | Institut National de Physique Nucléaire et de Physique des Particules | Institute for Nuclear Research | Aix-Marseille Université | Horia Hulubei National Institute for R and D in Physics and Nuclear Engineering | Universitat de Barcelona | INFN Sezione di Bari | Los Alamos National Laboratory | Hunan University | University of Groningen | Laboratoire de Physique Nucléaire et de Hautes Énergies | Central China Normal University | Eötvös Loránd University | University College Dublin | Cracow University of Technology | INFN Sezione di Perugia | University of Bonn | Monash University | INFN Sezione di Bologna | National Centre for Nuclear Research | Pontifical Catholic University of Rio de Janeiro | Tsinghua University | Instituto de Física Corpuscular | AGH University of Krakow | University of Birmingham | Lanzhou University | Central South University | INFN Sezione di Roma I | Universidad Nacional de Colombia | Rutherford Appleton Laboratory | Peking University | University of Ferrara | University of Genoa | University of Milano-Bicocca | Scuola Normale Superiore | École Polytechnique | University of Basilicata | Max Planck Institute for Nuclear Physics | University of Bari Aldo Moro | University of Milan | University of Siena | Universidad de Alcalá | Universitat de València | University of Bologna
- Pretraining has recently greatly promoted the development of natural language processing (NLP)
- We show that M6 outperforms the baselines in multimodal downstream tasks, and the large M6 with 10 parameters can reach a better performance
- We propose a method called M6 that is able to process information of multiple modalities and perform both single-modal and cross-modal understanding and generation
- The model is scaled to large model with 10 billion parameters with sophisticated deployment, and the 10 -parameter M6-large is the largest pretrained model in Chinese
- Experimental results show that our proposed M6 outperforms the baseline in a number of downstream tasks concerning both single modality and multiple modalities We will continue the pretraining of extremely large models by increasing data to explore the limit of its performance
