Bio

You have undoubtedly come across the term GPT-3. It seems that we are finally beginning to realise how deeply AI is penetrating everyday life. But what exactly is GPT-3, and which challenges does it bring? In this blog, I want to explain what GPT-3 is and discuss one of the educational challenges associated with the model.

What Is GPT-3?

GPT-3 is a language-processing model. Most people are more familiar with AI models based on image processing, such as facial recognition. In both cases, however, we are dealing with machine-learning models trained on enormous quantities of data.

Once trained, the algorithm can generate many different kinds of text, ranging from poetry and essays to programming code. Producing such texts requires a high level of linguistic sophistication. The texts generated by the model are coherent and grammatically correct. It is therefore not always easy to distinguish between a text generated by the model and one written by a human being.

The development can certainly be described as spectacular, although it is not entirely new. For some time, a wide range of applications has been available to support us in working with text. A well-known example is DeepL, which is capable of producing translations of a high linguistic quality.

GPT-3 therefore builds on developments from previous years, although the quality of its output has improved considerably compared with earlier versions. Whereas GPT-2 was trained using 1.5 billion parameters, GPT-3 was trained using 175 billion parameters.

In addition, OpenAI, the company that developed the model, has made it available to the wider public. This means that anyone wishing to generate text can use the model. Its users may include school students, marketing professionals, developers of spam and malware, and programmers more generally.

The Challenge

The new technology also creates new challenges across a wide range of domains.

For example, when the model is used to generate spam emails, spam filters may find it increasingly difficult to identify and remove them from inboxes. If a large proportion of social-media content is generated automatically and subsequently liked or promoted through algorithms, the nature of the content presented on social media may change radically.

It is impossible to provide an overview here of all the potential challenges associated with GPT-3. I will therefore focus on one of the educational challenges created by the model’s general availability.

Students can use the model to write essays or generate poetry. This may seem like the worst possible nightmare for teachers and lecturers who want to teach students to think critically and to write linguistically correct texts. A different approach is therefore required.

A Critical Perspective Based on How AI Is Developed

Anyone who understands how an AI model is trained and how its output is generated can immediately formulate several critical observations.

1. The Training Data

Let us first consider the data used to train the model. Approximately 90 per cent of GPT-3’s training data is in English. This means that the model’s output will also be limited in certain respects, because relatively little content from other linguistic regions is included.

The output is also directly connected to the training data in other ways. The internet data used to train the model contains many forms of bias, including political bias and discrimination against particular groups. These biases can easily be reproduced in the output.

Put more simply: what occurs frequently in the data is reinforced.

Some examples illustrate this problem.

The system may fail to adequately represent male nurses because the percentage of male nurses in the data is considerably lower than that of female nurses. This example is reminiscent of the well-known Amazon case, in which training an algorithm on existing human-resources data resulted in discrimination against women during recruitment.

The system is also limited to the content already available on the wider internet and often to content that is considered mainstream. New knowledge, however, is frequently not mainstream and may therefore be absent from the generated output.

2. The Model Does Not Truly Understand Its Output

The model does not genuinely understand the content it produces.

A deep-learning model such as GPT-3 is fed billions of pages of text. This includes, for example, all the pages on Wikipedia, as well as large quantities of other text available online.

One characteristic of such a deep-learning model is that there is no direct human oversight in the sense that no person continuously indicates whether the model’s output is correct. The model primarily searches for patterns and relationships.

One could therefore argue that coherence often takes precedence over factual accuracy. Put differently, GPT-3 is not a subject-matter expert.

This becomes apparent when, for example, you ask the model to generate a person’s curriculum vitae. I experimented with generating my own CV. The first result stated that I was a publisher of educational books. It is true that there is a major publishing company called Folens, but it seemed rather difficult to connect that company to my first name.

In a second attempt, I added KU Leuven and VIVES in an effort to steer the system in a particular direction. The model then presented me as a doctor of computer science who had studied at both Vrije Universiteit Brussel and KU Leuven.

Neither output contained any meaningful degree of accuracy.

3. Creativity

A final consideration concerns creativity.

I do not agree with the claim, frequently made in commentaries, that the model is incapable of creativity. Creativity can also involve combining existing information in a new way. This is a form of creativity that the model can perform.

What the model cannot do, however, is arrive at genuinely new insights through critical reflection. This is an essential consideration when working with GPT-3.

A New Approach

The greatest challenge for education therefore lies in engaging critically with generated texts.

There is little point in excluding new technologies of this kind from education. It is characteristic of our time that new ways of interacting with information continue to emerge. This relationship with information has changed fundamentally since the rise of the internet, while the quantity of available content and information continues to grow.

The main challenge is learning how to assess the value and reliability of that information.

As briefly illustrated above, even limited knowledge of how AI models are trained can encourage a more critical way of using them. For me, this is where the greatest educational challenge lies.

The relevant criteria are not fundamentally different from the research skills that have already been integrated into education:

Examining the methodology. This includes critically evaluating, verifying and further developing the knowledge produced by the model.

Examining the origin of the sources being used. In this context, this means considering the data from which the text was generated.