The developer asked for the base model, and Ant gave it to him! Six key training nodes are simultaneously open sourced

Zhidongxi Author | Yang Jingli Editor | Li Shuiqing Zhidongxi reported on August 20 that today, Ant Bailing open sourced Ling-3.0-tiny-base and Ling-3.0-flash-base. In addition to the final Base model weights, the team also simultaneously opened the pre-training and mid-term training weights of the two models, with a total of 6 checkpoints. Previously, Ling-3.0-flash was released on July 24 and open sourced on August 7; the lightweight model Ling-3.0-tiny was open sourced on August 11. Only nine days after Tiny was open sourced, Ant Bailing further opened up the Base version of two models without post-training. Ling-3.0-tiny-base has a total parameter volume of 7.9B and an activation parameter volume of 1.3B. It is suitable for code field training, budget-sensitive reinforcement learning research, and model behavior research. Ling-3.0-flash-base has a total parameter volume of 124B and an activation parameter volume of 5.1B. It can be used for continued training of models in code, complex reasoning, long context, and professional fields. This opening covers three stages: pre-training, mid-term training and WSM merger. Developers can choose different training starting points based on their own data, tasks and research goals. WSM (Warmup-Stable and Merge) is a learning rate scheme for large model pre-training. It uses multiple checkpoints to weight and merge, replacing the traditional learning rate attenuation, making the model more suitable for continuous pre-training and dynamic expansion of data. Since the two models adopt a unified training scheme, developers can first verify the training strategy at low cost on Ling-3.0-tiny-base, and then extend the effective method to Ling-3.0-flash-base. Ling-3.0-tiny-base open source address: Hugging Face: https://huggingface.co/inclusionAI/Ling