Countdown to ERNIE: Chinese Media on Baidu’s AI Chatbot
“ERNIE will be an idiot...” “Fundamentally, I cannot imagine a world where Baidu has had a robot more powerful than ChatGPT all this time but has managed to keep it a secret.”
Mar 14
7
A March 9 report from The Wall Street Journal said that Baidu employees are deeply uneasy with the rush to launch its ChatGPT competitor ERNIE on March 16 via a presentation you can stream at 2am EST. Some Baidu employees even told the reporters they have been selling company stock ahead of the launch in a show of pessimism.
Baidu, best known for its search engine, was one of China’s first major Internet giants. In recent years, however, the company has performed poorly, with valuation contracting and executives departing in droves. CEO Robin Li (李彦宏) is betting it all on its ChatGPT copycat, but will it be able to hold a candle to GPT4? Below are some translated excerpts from Chinese media, which discuss:
How Baidu built its LLM ERNIE;
The challenges of eliminating “anti-Chinese tendencies” from a chatbot;
Baidu’s cloud computing blueprint;
What Chinese companies want to do with ERNIE;
And how a cat with an American flag led to Baidu banning the word “patriotism” from its image-generation AI.
ChinaTalk is a reader-supported publication. To receive new posts and support our work, consider becoming a free or paid subscriber.
Subscribed
Baidu’s Decade-Long AI Bid
An overview of Baidu’s longstanding AI ambitions, from Daily Financial News 每日财报:
Documents show that as early as the beginning of 2010, Baidu had already conducted sorting and planning for the generally intelligent technology NLP (Natural Language Processing), and established its Natural Language Processing Department. In March 2019, Baidu launched the large model ERNIE, which broke the ninety-point barrier for the first time and topped the globally authoritative GLUE Benchmark, making history.
In September 2021, Baidu released PLATO-XL, the world’s first 100 billion-parameter conversational large model, which in one fell swoop surpassed Facebook’s Blender, Google’s Meena, and Microsoft’s DialoGPT. In becoming the world’s first large pre-trained generative model for Chinese-English dialogue — with 100 billion parameters — PLATO-XL once again set a new standard for the quality of open-domain dialogue and opened up the imagination for conversation models.
In addition, during a recent earnings call, Robin Li also said, “Baidu’s ERNIE model is a very localized LLM for the Chinese market, which means that Baidu’s current research and development of ERNIE bot will be more suitable for Chinese and the Chinese market than models developed overseas.”
As a result, after the financial report was released, many major banks raised Baidu’s target stock price. Among them, Credit Suisse maintained a “buy” rating on Baidu Group, with a target price raised from HKD204 to HK 207; Bank of America reiterated its “buy” rating and a target price of HKD230; Jefferies Group raised Baidu’s target price from USD 210 to USD 214.
How Chinese is ERNIE?
Will ERNIE’s Chinese be better than ChatGPT’s? Huxiu (虎嗅), a tech news outlet, explores how Baidu, after abandoning bilingual Chinese-English training for ERNIE, is trying to build a uniquely Chinese chatbot:
An AI industry insider told Huxiu, “Currently, it is clear that large models are the future direction of AI development, but how to implement large models and their commercial value will take at least five to eight years to verify.”
However, Robin Li is quite confident about this. He pointed out in an internal memo, “AI technology has reached a turning point, and all industries are inevitably being changed. The Chinese AI market is about to experience explosive demand growth, and the release of its commercial value will be unprecedented and exponential.”
Compared to ChatGPT, ERNIE bot has two natural advantages for Chinese users. The first is compliance. [Jordan: see my piece on this dimension here.] On the one hand, OpenAI currently does not have an official interface for Chinese users.
On the other hand, many of the answers provided by ChatGPT are not suitable for Chinese users because its training data is mainly based on the ideology of European and American countries, and may include anti-China or anti-Chinese tendencies in some of the answers.
The other advantage is understanding and training in Chinese. ChatGPT is an LLM trained on the GPT model, while ERNIE’s large model uses models that are more suitable for the Chinese language, as well as data that is tailored for Chinese training. Li believes that Baidu has a deep understanding of Chinese and Chinese culture through its massive Chinese language database, which is one of the important advantages of Baidu’s LLM research and development.
Jordan: I’m skeptical. The base model did ingest alot of Chinese text but ChatGPT is really, really good at Chinese despite having very little HLRF in Mandarin.
The Detractors
飞总聊IT (literally “Boss Fei Talks IT”) is a popular tech insider account on WeChat. Boss Fei does not mince words: “ERNIE is going to be an idiot.”
Baidu has a product called Wenxin Yige, which uses AI to draw pictures. There are also several foreign AI drawing products, but Baidu’s product is relatively unique. In early February of this year, an incident was reported online. Someone entered “a patriotic cat” on Wenxin Yige and had it draw a picture. The cat that was drawn had an American flag in the background. Simply put, Wenxin Yige understood patriotism as love for the United States.
I don’t like to get involved in non-technical issues, so let’s focus on the technology. From a technical point of view, Baidu’s ERNIE large model had a problem with the data training set, which resulted in default patriotism being equated with love for the United States. I cannot judge the originality of the ERNIE large model, but allow me to question its training dataset.
How did Baidu fix this problem? They set “patriotism,” “patriotic cat,” and all other related words you can imagine as banned words.
If you can’t input these words at the source, you won’t be able to draw pictures of patriotic cats with American flags in the background.
I asked DALL-E for “patriotic cat watercolor” and it was much more politically correct! Horto!
Technically speaking, it appears that the semantic problem of this large model has not been fixed or is not easy to fix, so Baidu can patch it only from the user input end to prevent some input that may expose problems with the LLM. This is the technical problem exposed by Wenxin Yige regarding the ERNIE large model.
…
Since we only see the landing of drawing applications and not chatting applications, we can tentatively judge the initial training priority and focus of the ERNIE model, which is AI drawing, not chatting.
However, after ChatGPT emerged, Baidu realized that chatbots could also be popular. Therefore, Baidu needs some time to feed ERNIE with a lot of language material to improve its chatting intelligence, which it is not good at.
This also explains why the hype started in February and the official announcement was not until March 16. Because they needed 996 and 007 overtime, and took advantage of this time to quickly feed ERNIE with language materials and train it to become better at chatting.
We can at least conclude that when ChatGPT was just popular, the ERNIE model was not trained with chatting as its core training direction. Through the patriotic cat in February, we can also see that the ERNIE model had some semantic problems even in the drawing training dataset.
In this regard, it is very difficult for me to have confidence in ERNIE bot, which was rushed to launch on March 16. Can it possibly be more amazing than ChatGPT? Can something that was rushed to completion be better than something that has benefitted from long-term investment and was developed from years of quietly burning money? I find it hard to believe.
But Baidu has reasons to make its own ERNIE bot. Professor Yiran Chen [of Duke University] said that Chinese internet companies are all very good at commercialization. When no one else has a LLM yet, at least Baidu has a rushed ERNIE model. Launching a rushed product still puts it ahead of other Chinese companies. If it gets started with commercialization now and manages to hold the top position among Chinese ChatGPT copycats — and especially if it gets press coverage and boosts stock prices — it will have been worth it.
Commercialization is Chinese companies’ home turf.
But fundamentally, I cannot imagine a world where Baidu has had a robot more powerful than ChatGPT all this time but has managed to keep it a secret. It would have been impossible to resist making money and attention off of it, and I don’t see Baidu burning money for all this time just to perfect its output.
So? Of course it doesn’t exist.
How is ERNIE Trained?
The Paper reported on Baidu’s technical investment in AI training:
Billions of dollars in investment have laid the foundation for Baidu’s advantage in NLP development. “The three core elements of AI are computing power, algorithms, and data. We believe that Baidu has a leading advantage in all three areas. Baidu has laid out a comprehensive plan at the chip, framework, model, and application levels.” Bai Yang, chief internet industry analyst at China International Capital Corporation, said Baidu has the computing power, algorithms, and data required for artificial intelligence and has a comprehensive advantage in developing ERNIE in China and even globally.
In terms of algorithms, Baidu’s ERNIE large model is similar to OpenAI’s GPT model, and has been released and iterated for several generations since 2019. It has extended from natural language understanding to a multimodal model, capable of visual, document, graphic, and speech functions. Therefore, the ERNIE model series that ERNIE bot is based on already has strong generalized abilities and performance.
Baidu’s AI strategic roadmap.
In terms of computing power, Baidu’s self-developed AI chip Kunlun has been deployed in multiple scenarios with tens of thousands of chips, and it has had strong practical experience in search engineering. Therefore, the underlying computing power of ERNIE is guaranteed. In addition, Baidu also has multiple cloud computing availability zones and a large-scale supercomputing cluster, laying the infrastructure for large model training. Currently, Baidu has three cloud computing centers — in Yangquan, Xushui, and Dingxing. Among them, the Yangquan intelligent cloud data center can support 160,000 servers in its first phase. The entire Baidu Yangquan cloud computing center is expected to support 240,000 servers, providing different computing needs such as high concurrency, high elasticity, and high precision for conversational large model applications.
In terms of data, the ChatGPT model’s training mainly uses text databases from the Internet, including up to 570GB of data obtained from network text, Wikipedia, and articles. Baidu’s search business has a strong first-mover advantage in accumulating massive structured and unstructured data in real-world data and user-demand understanding. These large-scale structured and unstructured data are expected to support the full pre-training of the ERNIE bot.
The Supporters
In the run-up to the launch, quite a few big and small names alike across various sectors in China have announced that they will be integrating with ERNIE bot. Integration gives them early access to the technology, but details on how ERNIE is affecting their business operations are still sparse.
Next up, we detail the seven different companies and government bureaus who have already announced partnerships with ERNIE.
KPMG is planning to apply Baidu’s leading intelligent dialogue technology achievements in various business areas such as finance, automotive, manufacturing, consumer goods, medicine, and education. This move marks KPMG’s priority access to leading AI technology, and also signifies the beginning of conversational language model technology providing empowerment for professional services and consulting in China. KPMG will be able to conduct internal testing using ERNIE bot, integrate the technical capabilities of ERNIE bot, and deepen cooperation with Baidu in product research and development, standard setting, and other fields. With the assistance of Baidu’s technical team, KPMG will create a joint solution through technology-sharing, training, and joint marketing to enhance competitiveness, create full-scenario AIGC-based intelligent solutions and services for enterprise users, and lead the transformation and upgrading of various industries relying on smart interconnection and innovative interconnection.
Industrial Bank will use its Financial Technology Research Institute as a carrier to conduct internal testing and integrate the technology capabilities of “Wenxin Yiyuan” in various financial scenarios such as smart branches, intelligent services, intelligent risk control, intelligent operations, intelligent marketing, and intelligent investment research. It will provide users with AI-powered solutions and services for all-scenario intelligent finance, and create an “AI everywhere” smart Industrial Bank.
With the assistance of Baidu’s technical team, a joint solution will be developed to strengthen competitiveness through technology sharing, training empowerment, and joint marketing. A one-stop, customized, intelligent, multi-scenario, and full life-cycle professional service will be provided to customers to meet their advanced needs for professional wealth management. With the support of intelligent interconnection and innovative interconnection, it will lead the transformation and upgrading of the wealth management industry.
Changan Automobile will work with Baidu Apollo to create a large-scale artificial intelligence interaction based on intelligent automotive scenarios, integrating the comprehensive capabilities of ERNIE bot and ushering in a new era of intelligent automotive experience.
The Paper/Pengpai News (and dozens of other newspapers and broadcast stations):
The Paper will fully experience ERNIE bot’s entire suite of capabilities, and apply Baidu’s leading intelligent conversation technology achievements to create a comprehensive series of products and services within the content ecosystem.
ECNU’s metaverse research group will fully integrate with ERNIE bot via the Baidu Intelligent Cloud, thus applying Baidu’s leading intelligent conversation technology achievements to create scenarios facing basic education.
And the culture and tourism offices of a lot of local governments, for some reason:
The Cultural and Tourism Commission of Wuxi County will apply Baidu’s leading intelligent conversation technology achievements in the field of cultural tourism. It will promote a large number of high-quality cultural and tourism resources, such as the largest alpine grassland in southern China and snow-covered Hongchiba, the “National Historical and Cultural Famous Town” Ningchang ancient town, the “deepest valley in Asia” Lanying Grand Canyon, the “only ancient county town that was not submerged in the Three Gorges reservoir area” Daning Ancient City, and the “National Key Cultural Relics Protection Unit” Jingzhuba Hanging Coffins group, as well as the mysterious Wu culture and salt culture, to promote the internationalization of “Spirit of Heaven and Earth, Wander in Wuxi” tourism.
