AI Can Recreate What We've Seen Based On Our Brain Activity

Recently, researchers from Osaka University published a study describing a method for using artificial intelligence to recreate images from the minds of humans based solely on their brain activity. The results were uncanny replications of what the people had seen: a clock tower, a plane in flight, a teddy bear, a train. The AI models didn’t just replicate the object, they knew the correct perspective, scale, and placement of the object. Some say this is an early example of AI reading minds. Whether you consider this mind reading or not, it has huge implications for the future. 

This is not the first-time scientists have tried to recreate images using AI and brain scans, however it’s the first time they have succeeded. In 2011, researchers from UC Berkeley used AI to read fMRI brain scans to recreate movie trailers watched by study subjects. While the results showed some correlation, they were blurry and often impossible to decipher. This is where the current breakthrough differs. The method used by the Osaka researchers resulted in images that were clear and almost exactly the same as the images that were seen by the study subjects. Their process involved something new – an AI image generator called Stable Diffusion. 

What is Stable Diffusion?

Stable Diffusion, created by Stability AI, is one of several AI image generators released publicly in 2022. You may have heard of another similar AI tool called Dall-e, created by OpenAI, the company behind ChatGPT. These AI models create images and art in real-time based on text prompts. The prompts can be farfetched, obscure, and detailed. Nothing phases these models. And they aren’t cutting and pasting together images that already exist online. They are creating unique images of people and things that don’t exist anywhere else. Here’s an example.

Prompt: “A cat riding a horse in space in anime style.” 

Stable Diffusion:

What is fMRI?

You may be familiar with MRIs (Magnetic Resonance Imaging). They are the large, tube-shaped magnets that create detailed images of our organs and tissues and can reveal ailments like tumors, damaged ligaments, injuries to the spine, joints, and organs. The difference between an MRI and an fMRI (Functional Magnetic Resonance Imaging) is that MRIs produce scans of the body’s anatomy, while fMRIs produce scans that measure changes in blood flow in the brain to detect brain activity. An fMRI creates a three dimensional map of the brain that illustrates which parts are active when. These scans help scientists understand how the brain works when it comes to perception, attention, memory, and decision-making. And they show us what’s happening in our brains when our bodies move, talk, and see. 

Applying Stable Diffusion to fMRI Scans 

The Osaka researchers took brain scans from four people who lay in an fMRI scanner as they viewed 10,000 images over the course of a year and fed them into two AI models along with the related images. They trained one model to find correlations between brain activity and the corresponding images and they trained the second model to find correlations between brain activity and the text captions that accompanied the photos. Based on the brain scans, the first model was able to recreate the perspective and layout of what the person had seen, but the object in the image was blurry and indecipherable. However, when the second AI model was applied, it was able to recognize what object the subjects were seeing by using the text captions from the training images. Once trained, these two models were able to recreate rough images that were not part of their original training, based on fMRI brain scans. But this was not much of an improvement over previous experiments.

Then (and this is key), the researchers added one more step, feeding the blurry images and related text captions produced by their two AI models into Stable Diffusion. This combined use of textual and visual information was a novel approach that allowed Stable Diffusion to recreate the images seen by the study subjects with striking clarity.  

Implications

While this study was very small and the technique is far from ready for public use, it has the potential to unlock the mysteries of the human brain. Experts say that it could one day help people who cannot communicate, such as individuals who outwardly appear unresponsive but may still be conscious; it may recreate our dreams, memories, and thoughts in images and movies, and it may reveal how animals see the world. Ultimately, these kinds of AI techniques will usher in a new wave of brain-machine interfaces and help us understand the inner workings of the mind.