In February, Sogou founder Wang Xiaochuan stated on Weibo that “China wants its personal OpenAI.” The Chinese language entrepreneur is now inching nearer to his dream as his nascent startup Baichuan Intelligence rolled out its next-generation massive language mannequin Baichuan-13B at present.
Baichuan is being touted as one in all China’s most promising LLM builders, due to its founder’s storied previous as a pc science prodigy from Tsinghua College and founding the search engine supplier Sogou, which was later acquired by Tencent.
Wang stepped down from Sogou in late 2021. As ChatGPT took the world by storm, the entrepreneur launched Baichuan in April and rapidly pocketed $50 million in financing from a bunch of angel buyers.
Like different homegrown LLMs of China, Baichuan, a 13 billion-parameter mannequin based mostly on the Transformer structure (which additionally undergirds GPT), is skilled on Chinese language and English knowledge. (Parameters seek advice from variables that the mannequin makes use of to generate and analyze textual content.) The mannequin is open-source and optimized for industrial software, in line with its GitHub web page.
Baichuan-13 is skilled on 1.4 trillion tokens. As compared, Meta’s LLaMa makes use of 1 trillion tokens in its 13 billion-parameter mannequin. Wang beforehand stated in an interview that his startup was on observe to launch a large-scale mannequin corresponding to OpenAI’s GPT-3.5 by the tip of this yr.
Having began solely three months in the past, Baichuan has already achieved a notable pace of improvement. By the tip of April, the workforce had grown to 50 folks, and in June, it rolled out its first LLM, the pre-training mannequin Baichuan-7B which boasts 7 billion parameters.
Now, the foundational mannequin Baichuan-13B is obtainable free of charge to teachers and builders who’ve obtained official approval to make use of it for industrial functions. Importantly, within the age of U.S. AI chip sanctions on China, the mannequin provides variations that may run on consumer-grade {hardware}, together with Nvidia’s 3090 graphic playing cards.
Different Chinese language companies which have invested closely in massive language fashions embody the search engine large Baidu; Zhipu.ai, a derivative of Tsinghua College led by Professor Tang Jie; in addition to the analysis institute IDEA led by Harry Shum, who co-founded Microsoft Analysis Asia.
China’s massive language fashions are quickly rising because the nation prepares to implement among the world’s most stringent AI rules. As reported by the Monetary Occasions, China is anticipated to attract up rules for generative AI with a specific deal with content material, indicating stepped-up management than the foundations launched in April. Firms might also have to receive a license earlier than launching massive language fashions, which may decelerate China’s efforts to compete with the U.S. within the nascent business.