Inherent bias of AI models confirmed

New research examined the varied biases of different AI large language models, depending on the source material used.

Research conducted jointly between universities in the US and China attempted to measure the bias inherent in a bunch of large language models (LLMs), from which generative AI such as ChatGPT derive their outputs. Since none of those LLMs have access to the entirety of human learning, activity, and opinion, they must necessarily be biased in the direction of the fraction of it they are programmed with.

The tools available for tracking things like ideological bias are themselves flawed, as the study acknowledges in a section on ‘limitations’. The Political Compass, for example, uses a questionnaire to rank people on a graph that uses two axes covering economic (left/right) and social (libertarian/authoritarian) positions.

But this suffers from a lack of consensus on the precise definition of those terms. In fact, since a common definition of economically left is those in favour of heavy taxation and redistribution by a large state, it’s hard to see how anyone could be both ‘left’ and ‘libertarian’ if you define the latter as being in favour of personal freedom.

As if to exemplify the very concept of inherent bias, MIT Technology Review reports that “Researchers conducted tests on 14 large language models and found that OpenAI’s ChatGPT and GPT-4 were the most left-wing libertarian, while Meta’s LLaMA was the most right-wing authoritarian.” Here’s the chart from the study that conclusion was based on and note the subjectivity even in its label, with the word ‘conservative’ not featured anywhere on the chart.

We highlight this not to criticize either the study or the report but to show how inherent and ubiquitous bias is. This story and, indeed, all journalistic output, is affected by the bias involved in both selecting what to cover and the angle taken. But since bias is so intrinsic and unavoidable, greater transparency on the matter would be ideal, as well resisting the urge to claim unique possession of concepts like ‘facts’ and ‘the truth’.

The reason this matter is so important is that the recent LLM breakthroughs seem to have initiated a stampede towards greater automation, with many previously human tasks now being entrusted to AI. Many of these involve some degree of judgment and discretion, and just a quick look at your email spam folder will reveal how flawed that can be. If different LLMs are inherently biased in a certain ideological direction, then surely that bias should be prominently advertised by any service that uses them.

Get the latest news straight to your inbox. Register for the Telecoms.com newsletter here.