Stability.AI, the company that developed Stable Diffusion, launched a new version of the AI model in late November. A spokesperson says that the original model was released with a safety filter, which Lensa does not appear to have used, as it would remove these outputs. One way Stable Diffusion 2.0 filters content is by removing images that are repeated often. The more often something is repeated, such as Asian women in sexually graphic scenes, the stronger the association becomes in the AI model. 

Caliskan has studied CLIP (Contrastive Language Image Pretraining), which is a system that helps Stable Diffusion generate images. CLIP learns to match images in a data set to descriptive text prompts. Caliskan found that it was full of problematic gender and racial biases.

“Women are associated with sexual content, whereas men are associated with professional, career-related content in any important domain such as medicine, science, business, and so on,” Caliskan says. 

Funnily enough, my Lensa avatars were more realistic when my pictures went through male content filters. I got avatars of myself wearing clothes (!) and in neutral poses. In several images, I was wearing a white coat that appeared to belong to either a chef or a doctor. 

But it’s not just the training data that is to blame. The companies developing these models and apps make active choices about how they use the data, says Ryan Steed, a PhD student at Carnegie Mellon University, who has studied biases in image-generation algorithms. 

“Someone has to choose the training data, decide to build the model, decide to take certain steps to mitigate those biases or not,” he says.  

The app’s developers have made a choice that male avatars get to appear in space suits, while female avatars get cosmic G-strings and fairy wings. 

A spokesperson for Prisma Labs says that “sporadic sexualization” of photos happens to people of all genders, but in different ways.