Incident 262: DALL-E Mini Reportedly Reinforced or Exacerbated Societal Biases in Its Outputs as Gender and Racial Stereotypes

Description: Publicly deployed open-source model DALL-E Mini was acknowledged by its developers and found by its users to have produced images which reinforced racial and gender biases.

Suggested citation format

Atherton, Daniel. (2022-06-11) Incident Number 262. in Lam, K. (ed.) Artificial Intelligence Incident Database. Responsible AI Collaborative.

Incident Stats

Incident ID
Report Count
Incident Date
Khoa Lam


New ReportNew ReportDiscoverDiscover

Incidents Reports

Everyone's having a grand old time feeding outrageous prompts into the viral DALL-E Mini image generator — but as with all artificial intelligence, it's hard to stamp out the ugly, prejudiced edge cases.

Released by AI artist and programmer Boris Dayma, the DALL-E Mini image generator has a warning right under it that its results may "reinforce or exacerbate societal biases" because "the model was trained on unfiltered data from the Internet" and could well "generate images that contain stereotypes against minority groups."

So we decided to put it to the test. Using a series of prompts ranging from antiquated racist terminology to single-word inputs, Futurism found that DALL-E Mini indeed often produces stereotypical or outright racist imagery.

We'll spare you specific examples, but prompts using slur words and white supremacist terminology spat out some alarming results. It didn't hesitate to cook up images of burning crosses or Ku Klux Klan rallies. "Racist caricature of ___" was a reliable way to get the algorithm to reinforce hurtful stereotypes. Even when prompted with a Futurism reporter's Muslim name, the AI made assumptions about their identity.

Many other results, however, were just plain strange.

Take, for example, what the generator came up with for the term "racism" — a bunch of painting-like images of what appear to be Black faces, for some reason.

The problematic results don't end at depicting minorities in a negative or stereotypical  light, either. It can also simply reflect current inequalities reflected in its training data.

As spotted by Dr. Tyler Berzin of Harvard Medical School noted, for instance, entering the term "a gastroenterologist" into the algorithm appears to show exclusively white male doctors.

We got nearly identical results. And for "nurse"? All women.

Other subtle biases also showed amid various prompts, such as the entirely light-skinned faces for the terms "smart girl" and "good person."

It all underscores a strange and increasingly pressing tension at the heart of machine learning tech.

Researchers have figured out how to train a neural network, using a huge stack of data, to produce incredible results — including, it's worth pointing out, OpenAI's DALL-E 2, which isn't yet public but which blows the capabilities of DALL-E Mini out of the water.

But time and again, we're seeing these algorithms pick up hidden biases in that training data, resulting in output that's technologically impressive but which reproduces the darkest prejudices of the human population.

In other words, we've made AI in our own image, and the results can be ugly. It's also an incredibly difficult problem to solve, not the least because even the brightest minds in machine learning research often struggle to understand exactly how the most advanced algorithms work.

It's possible, certainly, that a project like DALL-E Mini could be tweaked to either block obviously hurtful prompts, or that it could give users to disincentivize any unpleasant or incorrect results.

But in a broader sense, it's overwhelmingly likely that we're going to see many more impressive, fun or impactful uses of machine learning which, examined more closely, embody the worst of society.

That AI Image Generator Is Spitting Out Some Awfully Racist Stuff

In 2021, AI research laboratory OpenAI invented DALL·E, a neural network trained to generate images from text prompts. With just a few descriptive words, the system (named after both surrealist painter Salvador Dalí and the adorable Pixar robot WALL-E) can conjure up absolutely anything from an armchair shaped like an avocado to an illustration of a baby radish walking a dog in a tutu. At the time, however, the images were often grainy, inaccurate and time-consuming to generate—leading the laboratory to upgrade the software and design DALL·E 2. The new and improved model, supposedly_._

While DALL·E 2 is slowly being rolled out to the public via a waitlist, AI artist and programmer Boris Dayma has launched a stripped-down version of the neural network which can be used by absolutely anyone with an internet connection. Dubbed DALL·E mini, the AI model is now all the rage on Twitter as users are scrambling to generate nightmarish creations including MRI images of Darth Vader, Pikachu that looks like a pug and even the Demogorgon from Stranger Things as a cast member on the hit TV show Friends.

— Weird Dall-E Mini Generations (@weirddalle) June 13, 2022

While the viral tool has even spearheaded a meme format of its own, concerns arise when text prompts descend beyond innocent Pikachus and Fisher Price crack pipes onto actual human faces. Now, there are some insidiously dangerous risks in this case. As pointed out by Vox, people could leverage this type of AI to make everything from deepnudes to political deepfakes—although the results would be horrific, to say the least. Given how the technology is free to use on the internet, it also harbours the potential to put human illustrators out of work in the long run.

But another pressing issue at hand is that it can also reinforce harmful stereotypes and ultimately accentuate some of our current societal problems. To date, almost all machine learning systems, including DALL·E mini’s distant ancestors, have exhibited bias against women and people of colour. So, does the AI-powered text-to-image generator in question suffer the same ethical gamble that experts have been warning about for years now?

Using a series of general prompts, SCREENSHOT tested the viral AI generator for its stance on the much-debated racism and sexism that the technology has been linked to. The results were both strange and disappointing, yet unsurprising.

When DALL·E mini was fed with the text prompts ‘CEO’ and ‘lawyers’, the results were prominently white men. A query for ‘doctor’ reverted back with similar results while the term ‘nurse’ featured mostly white women. The same was the case with ‘flight attendant’ and ‘personal assistant’—both made assumptions about what the perfect candidate for the respective job titles would look like.

Now comes the even more concerning part, when the AI model was prompted with phrases like ‘smart girl’, ‘kind boy’ and ‘good person’, it spun up a grid of nine images all prominently featuring white people. To reiterate: Are we shocked? Not in the least. Disappointed? More than my Asian parents after an entrance exam.

In the case of DALL·E 2, AI researchers have found that the neural network’s depictions of people can be too biassed for public consumption. “Early tests by red team members and OpenAI have shown that DALL·E 2 leans toward generating images of white men by default, overly sexualizes images of women, and reinforces racial stereotypes,” WIRED noted. After conversations with roughly half of the red team—a group of external experts who look for ways things can go wrong before the product’s broader distribution—the publication found that a number of them recommended OpenAI to release DALL·E 2 without the ability to generate faces.

“One red team member told WIRED that eight out of eight attempts to generate images with words like ‘a man sitting in a prison cell’ or ‘a photo of an angry man’ returned images of men of colour,” the publication went on to note.

When it comes to DALL·E mini, however, Dayma has already confronted the AI’s relationship with the darkest prejudices of humanity. “While the capabilities of image generation models are impressive, they may also reinforce or exacerbate societal biases,” the website reads. “While the extent and nature of the biases of the DALL·E mini model have yet to be fully documented, given the fact that the model was trained on unfiltered data from the Internet, it may generate images that contain stereotypes against minority groups. Work to analyze the nature and extent of these limitations is ongoing, and will be documented in more detail in the DALL·E mini model card.”

Although the creator seems to have somewhat addressed the bias, the possibility of options for either controlling harmful prompts or reporting certain results cannot be ruled out. And even if they’re all figured out for DALL·E mini, it’ll only be a matter of time before the neural system is replaced by another with impressive capabilities where such an epidemic of bias could resurface.

AI art generator DALL·E mini is spewing awfully racist images from text prompts

The only real limits to DALL-E Mini are the creativity of your own prompts and its uncanny brushwork. The accessible-to-all AI internet image generator can conjure up blurry, twisted, melting approximations of whatever scenario you can think up. Seinfeld nightmares? You got it. Court room sketches of animals, vehicles, and notable people in varying combinations? Easy peasy. Never before seen horror monsters from the mind of the mindless. Sure, whatever.

But give DALL-E Mini literally nothing, and it quickly reveals the limits of its own “imaginings.” Given no direction or guidance, the AI model seems to get stuck. With absolutely no prompt, the program will without a doubt give you back an image of a woman in a sari (a garment commonly worn across South Asia.)

Even the tool’s developer, Boris Dayma, doesn’t know exactly why, according to reporting from Rest of World. “It’s quite interesting and I’m not sure why it happens,” he said to Rest of World about the phenomenon.

What is DALL-E Mini?

DALL-E Mini was inspired by DALL-E 2, a powerful image generator from OpenAI. The pictures that DALL-E 2 creates are much more realistic than those that “mini” can make, but the trade-off is that it requires too much computing power to be tossed around by just any old internet user. There’s a limited capacity and a waitlist.

So Dayma, unaffiliated with OpenAI, opted to create his own, less exclusive version which launched in July 2021. In the past few weeks, it’s become wildly popular. The program has been managing about 5 million requests every day, Dayma told Rest of World. As of Monday, DALL-E Mini was renamed Craiyon and shifted to a new domain name, at the insistence of OpenAI.

Like any other artificial intelligence model, DALL-E Mini/Craiyon creates outputs based on training inputs. In the case of Mini, the program was trained on a diet of 15 million image and caption pairs, and an additional 14 million images—plus, the chaos of the open internet.

From Rest of World:

The DALL·E mini model was developed on three major datasets: Conceptual Captions dataset, which contains 3 million image and caption pairs; Conceptual 12M, which contains 12 million image and caption pairs, and The OpenAI’s corpus of about 15 million images. Dayma and DALL·E mini co-creator Pedro Cuenca noted that their model was also trained using unfiltered data on the internet, which opens it up for unknown and unexplainable biases in datasets that can trickle down to image generation models.

And this underlying data almost certainly has something to do with the sari phenomenon. The sari state of affairs, if you will.

Why is DALL-E Mini Getting Stuck on Saris?

Dayma suggested that images of South Asian women in saris may have been heavily represented in those original photosets that feed DALL-E Mini. And that the quirk could also have something to do with caption length, as the AI might associate zero-character prompts with short image descriptions.

However, Michael Cook, an AI researcher at Queen Mary University in London, told Rest of World he wasn’t so sure about the overrepresentation theory. “Typically machine-learning systems have the inverse problem — they actually don’t include enough photos of non-white people,” he said.

Instead, Cook thinks the origin could lie in a language bias of the data filtering process. “One thing that did occur to me while reading around is that a lot of these datasets strip out text that isn’t English,” he said. Image captions that include Hindi, for example, might be getting removed, leaving images with no supporting, explanatory text or labels floating free in the primordial AI soup, he explained.

DALL-E Mini Is Obsessed With Women in Saris, and No One Knows Why

This weekend, I tried out DALL-e mini hosted by Hugging Face. It's an AI model that generates images from any word prompt. Every image it generated for "expert" "data scientist" "computer scientist" showed some distorted version of a white male.

This highlights one of the many issues with AI systems, which is replicating and amplifying biases embedded in their training datasets. We have a lot of work ahead of us to change the biases embedded in these systems but we also need to change our society's stereotypical notions of who is an expert in AI and other technical fields.

LinkedIn Post: Mia Dand