The AI revolution: The potential and risks for research and researcher training
Justin Zobel, Pro Vice-Chancellor (Graduate & International Research), University of Melbourne
On 30 November 2022, the release of ChatGPT triggered a wave of global interest in artificial intelligence (AI). Observers quickly identified the power of such tools to transform a range of fields and their potential to disrupt journalism, media, the creative arts, and academia.
Many observers treated AI as an entirely new phenomenon, but the use of AI technologies in everyday products is ubiquitous. For example, it is a key component of search, online shopping, dictation in word processing, and autocompletion and word suggestion in text messaging. It is also used – controversially – in social applications such as policing and border security. In research and industry use of AI has been significant since the 2000s, for tasks such as pattern analysis in large data sets, drug discovery, and disease diagnosis and outcome prediction. Some of these uses have been transformative and have led to knowledge that otherwise would have been nearly impossible to discover.
A common factor to all of these applications is that the AI is undertaking tasks previously thought to require human expertise, and to a large extent is not programmed in the traditional sense, with a fixed set of rules, but acquires its capability by inspection of massive volumes of data.
This same characteristic is the underpinning of the generative AI of which ChatGPT is an exemplar. The data used includes truly vast volumes of text, from which it can infer connections between words, syntax, and usages. Such large language models had been available for years, but the generative use of LLMs, at this scale, was new. At a high level, however, it isn’t complex. One word suggests another, a sentence type suggests the sentence type that follows, and so on; a prompt triggers a cascade of outputs by following chains of connection, without reference to meanings, concepts, or the outside world.
Yet, despite entirely lacking a grounding in facts or reasoning, the output of these tools can be extraordinarily convincing. They seem intelligent, rational, well-informed, and – critically – to understand. Even experts in the field waver on the question as to whether true intelligence is being exhibited.
My perspective, though, is that they are no more than pattern-matching machines regurgitating a sophisticated mash-up of the data they’ve been fed. Their plausibility is a psychological trick like that used by scammers and frauds: an articulate fluidity persuades us that something richer is happening underneath, but it is illusory.
That same fluidity is perhaps what led to the initial explosion of interest. There is no question that generative AI is utterly transformative, both in itself and in how interaction with computers is perceived. We can hold a computer-mediated dialogue with a synthesis (albeit a very flawed synthesis) of what seems to be all of human knowledge. We can seek knowledge, translate text, generate plans, write poems, and draft articles; as well as write code and generate pictures.
When ChatGPT was released, the implications for students undertaking coursework study were obvious. We want to assess a student’s capability and thus it is their words and thinking that we want to see, not a machine’s.
The implications for research have been less clear. Some researchers are comfortable using generative AI to create text for their papers, as they regard paper writing as rote work that is unrelated to the merits of the research outcomes. A broad consensus quickly emerged among publishers that this use of such tools is acceptable if it is declared, effectively deferring concerns to referees.
Other concerns relate to practices that are more obviously unsavoury. To begin with, these tools can be used for outright fraud.
Another form of fraud is the use of generative AI to review papers. This undermines the integrity of the process and is not a substitute for human assessment. An AI cannot interrogate the robustness of the method, the quality of the insights, the logic, the correctness of citations, and the accuracy of the literature review. For these reasons, too, the use of generative AI for literature summaries is highly questionable.
The AI technologies that have a longer history of practical use, such as data interpretation tools, have separate risks such as lack of interpretability and lack of quantifiability of error.
But one particular risk is shared: that these tools are readily used without understanding their limitations. Generative AI often produces nonsense, but so do analytic tools, for example discovering spurious associations that are then characterised as cause and effect. There’s also the problem of bias: not just reflecting the biases in the underlying data but amplifying them.
There are further risks for higher degree by research (HDR) candidates. As for coursework students, it is troubling if they submit a thesis that they haven’t written, but also troubling if their learning or development is compromised – and this can happen in many ways. Slick, superficial summaries of a field discourage deep learning; the use of generated text is a form of cognitive offloading that diminishes skills acquisition and knowledge retention; and the use of such text in communication with mentors can conceal a lack of intellectual development.
Fundamentally, though, generative AI is an extraordinary technology. Some constructive uses have already emerged, such as for authoring programs, but in other respects it is challenging our practices in ways that we do not yet know how to address. We do not yet even know how to answer the critical question of what it will mean to be AI literate.
It may be that we can adapt by making small tweaks to how we undertake and report research; but perhaps our practices will completely change, with, say, new understandings of what comprises a publication or new requirements for certification of how the research was undertaken. In my view, some dramatic changes are inevitable but they can bring with them the promise of accelerated discovery and dissemination of knowledge.
- Date