The technical foundation of ChatGPT is a large language model (LLM), which at its heart is a “next word” prediction engine: Given a preceding word sequence, it constructs a probability distribution for the immediate next word based on a training text corpus.
The corpus used to train such a large-scale language model typically consists of documents collected from the cyber and physical worlds, including Web pages, books, periodicals, one-time publications, e-mails and instant messages, etc.
During training, each document in the corpus is scanned, word by word, from the beginning to the end.
When word X is scanned, the words preceding X serve as a contextual sequence. X is treated as the prediction target, and a training data pair “contextual text sequence, prediction target” is established.
As the scan goes on, myriad such pairs are thus formed. From these training data pairs, a neural network is used to learn the mathematical correlation model between the contextual text sequences and the words immediately following them. The beauty of this particular approach of building language models is that its training data does not require manual labeling.
Although ChatGPT is based on word-by-word prediction, the quality of its responses to user prompts is surprisingly high, because most of the sentences it produces are grammatically correct, semantically relevant, structurally fluent and sometimes even featuring (not necessarily correct) fresh new ideas.
As far as reading comprehension is concerned, ChatGPT seems able to extract key ideas from each individual article, to compare and contrast the similar and different ideas present in multiple articles, and even synthesize novel ideas for situations that are similar, but not exactly identical, to those explored in the training corpus.
Exactly because ChatGPT generates each word in response to a user prompt by consulting with the “text sequence to word” prediction model, narratives included in the response can sometimes contain factual errors or even be completely fabricated.
For example, asked about The Eagles’ works, ChatGPT might quote a fragment of the lyrics of their famous song Hotel California, and the actual quotation included in the response might turn out to be ChatGPT’s own fabrication.
Nevertheless, it is still quite remarkable that just by simply extracting and applying the co-occurrence relationships between text sequences and words in the training text corpus, ChatGPT is able to respond to a wide variety of user prompts with often jaw-dropping quality.
This seems to validate the famous saying by linguist John Firth: “You shall know a word by the company it keeps.”
ChatGPT’s underlying language model is closely tied with its training text corpus. That is why when the question “What is the relationship between China and Taiwan?” is input into ChatGPT in traditional and simplified characters, the answers it outputs are diametrically different.
This suggests that, if future Taiwanese ChatGPT-based applications need a Chinese LLM, they cannot depend on one developed by China, for ideological considerations and national security concerns.
However, the training material that OpenAI uses to train its Chinese LLM might not be sufficiently comprehensive or frequently refreshed. For example, if one intends to use ChatGPT to create scripts for Taiwanese TV dramas, then its underlying LLM must be augmented with additional training based on Taiwanese dialogue data.
Similarly, if one wants to apply ChatGPT to analyzing Taiwanese court judgements to automatically identify abnormal or inconsistent ones, the underlying LLM must be further trained on a corpus made up of past court judgements. These cases suggest that Taiwan should own its LLM to guarantee it is fully localized and always kept up to date.
It is expected that ChatGPT-based applications will pop up all over the place in Taiwan. If they are all built on OpenAI’s ChatGPT, the economic cost associated with the application programming interface calls to OpenAI is going to be enormous, especially when accumulated over multiple decades.
If the government develops its own Chinese LLM based on text materials of Taiwanese origin and make it available for domestic artificial intelligence (AI) text application developers, this infrastructural investment would form the backbone of, and make a gargantuan contribution to, the effective development of its digital industry in the coming decades.
Chiueh Tzi-cker is a joint appointment professor in the Institute of Information Security at National Tsing Hua University.
Taiwan’s semiconductor industry gives it a strategic advantage, but that advantage would be threatened as the US seeks to end Taiwan’s monopoly in the industry and as China grows more assertive, analysts said at a security dialogue last week. While the semiconductor industry is Taiwan’s “silicon shield,” its dominance has been seen by some in the US as “a monopoly,” South Korea’s Sungkyunkwan University academic Kwon Seok-joon said at an event held by the Center for Strategic and International Studies. In addition, Taiwan lacks sufficient energy sources and is vulnerable to natural disasters and geopolitical threats from China, he said.
After reading the article by Hideki Nagayama [English version on same page] published in the Liberty Times (sister newspaper of the Taipei Times) on Wednesday, I decided to write this article in hopes of ever so slightly easing my depression. In August, I visited the National Museum of Ethnology in Osaka, Japan, to attend a seminar. While there, I had the chance to look at the museum’s collections. I felt extreme annoyance at seeing that the museum had classified Taiwanese indigenous peoples as part of China’s ethnic minorities. I kept thinking about how I could make this known, but after returning
What value does the Chinese Nationalist Party (KMT) hold in Taiwan? One might say that it is to defend — or at the very least, maintain — truly “blue” qualities. To be truly “blue” — without impurities, rejecting any “red” influence — is to uphold the ideology consistent with that on which the Republic of China (ROC) was established. The KMT would likely not object to this notion. However, if the current generation of KMT political elites do not understand what it means to be “blue” — or even light blue — their knowledge and bravery are far too lacking
Taipei’s population is estimated to drop below 2.5 million by the end of this month — the only city among the nation’s six special municipalities that has more people moving out than moving in this year. A city that is classified as a special municipality can have three deputy mayors if it has a population of more than 2.5 million people, Article 55 of the Local Government Act (地方制度法) states. To counter the capital’s shrinking population, Taipei Mayor Chiang Wan-an (蔣萬安) held a cross-departmental population policy committee meeting on Wednesday last week to discuss possible solutions. According to Taipei City Government data, Taipei’s