Representation of visual memory created by Mary Leonard, Graphic Design Specialist at UPenn.
By Alexandra Brodin
Scheie Vision Summer 2020
Imagine you are walking down the street, when you see a familiar face. Perhaps you are unable to remember where you met the person, but you just know you have seen him or her before. Most of us have experienced this routine phenomenon, known as visual familiarity memory, but what is happening inside the brain?
Nicole Rust, PhD, Associate Professor of Psychology at the University of Pennsylvania (UPenn), and her research team are working to answer this question. Dr. Rust became interested in studying visual familiarity memory while conducting her postdoctoral research at the Massachusetts Institute of Technology, and she continues to pursue this work as a vision scientist at UPenn.
Familiarity—the kind of memory described in the scenario above—is distinctive from another type of memory known as recollection, which is the ability to recall specific facts. In our scenario, if we were able to remember where we met the person (at work, for example), that memory would constitute recollection. Familiarity memory describes the sense that we have seen an image before, as opposed to memories of factual information.
How We Recognize Familiar Images
It is not fully understood how familiarity memory works, although the neurological areas involved in the process have been identified. Seeing an image causes neural signals to fire in the retina, which travel to the brain via the optic nerve and propagate through the visual system, eventually reaching a section of the brain called the inferotemporal (IT) cortex.
“We often regard IT cortex as the highest stage of the visual system because it is the last visual brain area that neural signals travel through before they become multi-sensory,” Dr. Rust explained. The integration of information across the different senses happens outside of the visual system, in association cortices (the areas of the brain that support cognitive functions, such as memory).
One of the association cortices thought to be important for visual familiarity memory is the perirhinal cortex. Perirhinal cortex is part of the medial temporal lobe, which also includes the hippocampus. Together, the medial temporal lobe structures are considered to be the brain’s memory system.
Current research in familiarity memory traces its roots to a 1973 study by psychologist Lionel Standing, PhD. In this study, Dr. Standing asked participants to perform a simple visual familiarity task. First, the participants were asked to flip through a stack of thousands of images, looking at each picture for a few seconds. Two days later, Dr. Standing and his team tested the subjects’ memory by presenting each participant with two images: one they had seen in the stack, and one novel image. The participants were then asked to indicate which image was familiar.
The findings suggested that humans are able to remember images with incredible accuracy. They also demonstrated a quality known as vast capacity. This means that there was no discernable cap on the number of images the participants could remember—after seeing as many as 10,000 images, the rate of remembering did not change. Further studies have shown that people are still excellent at recognizing familiar images when the corresponding novel image is similar in content (for example, two images of the same object, but from slightly different viewpoints).
Dr. Rust and her team are working to describe the neurological mechanisms behind these behavioral observations. One hypothesis she has investigated suggests that novel images produce a high firing response in the neurons of the IT cortex, whereas familiar images produce a lower firing response. This effect is known as repetition suppression. The difference between these two firing responses was hypothesized to be the signal for familiarity memory.
Locating the Signal for Familiarity Memory
In March 2018, Dr. Rust and her team published a paper in eLife, in which they set out to verify the hypothesis that repetition suppression is in fact the memory signal by studying neural responses to familiar and novel images in animal models. They used geometric representations (vectors on a graph) to represent the neural response patterns that resulted from seeing different images.
Each image we see produces a unique neural response pattern in the IT cortex. “For example, you can show an image of a pineapple and get a particular pattern of responses across neurons in the IT cortex,” Dr. Rust explained. “Let’s say neuron two responded really vigorously and neuron one did not, so that you get that specific pattern. And now let’s say you show a different image, maybe an apple, and now neuron one starts to respond vigorously and neuron two, less so. That’s a different pattern.”
When images contain similar content, the response patterns tend to look alike. “All the pineapples are reflected as similar neural response patterns, and all the apples have similar patterns,” Dr. Rust continued. “So, that’s the identity representation, what we are looking at. And I think about that as vision.”
This relationship between unique response patterns and object identity has long been established—in fact, Dr. Rust worked on this problem in her postdoctoral training. The goal of her 2018 study was to investigate whether repetition suppression is the signal for familiarity memory. Her team’s results suggested that this hypothesis is correct: a novel image will produce a more vigorous response in the IT cortex, which is represented graphically by a response pattern with a larger magnitude. The converse is true for images that had been seen before. Consistent with the hypothesis that repetition suppression is in fact the signal for memory, they also found that the amount of repetition suppression predicted how well subjects remembered seeing the images, and how quickly they forgot them with time.
Memorability: Complicating the Signal
Although familiarity memory is highly accurate and robust, not all images are equally memorable—that is, an image’s “memorability” also plays a role in whether subjects are able to remember a familiar image. The term memorability is a measure of how easy it is for average viewers to remember an image they have seen before. Memorability can be represented numerically as the fraction or percentage of subjects who will remember seeing a particular image, such that each image can be assigned a specific memorability score.
In their August 2019 eLife publication, Dr. Rust and her team sought to describe the neurological underpinnings of image memorability. “We know that we all tend to find the same images more and less memorable,” Dr. Rust explained. “In terms of the fraction of memorability variation shared across different people, that fraction is large.”
In the past, neuroscientists have tried to account for memorability by determining what features in an image correlate with a higher rate of remembering. Some of these features are well established. For example, images that contain people, atypical content, or color (as opposed to black and white images) tend to be more memorable. However, a large proportion (about 25%) of the factors that drive image memorability remains unexplained.
Faced with this gap, Dr. Rust and her team decided to approach the question of memorability by looking at the neurological signals associated with this phenomenon. Using memorability scores, they observed neurological responses to images of high and low memorability. “The simple answer is that images that are more memorable produce more vigorous responses in the IT cortex the first time that you look at them,” Dr. Rust explained.
This finding complicates the idea of repetition suppression, which suggests that familiar images that produce the lowest firing responses are the ones we remember best. However, Dr. Rust’s research on memorability suggests that a low firing response in the IT cortex may also signal that an image is less memorable.
Dr. Rust and her team are researching how the findings from these two key studies fit together. There are two overarching possibilities: either the signal they attributed to familiarity memory (the difference in the response patterns associated with novel and familiar images) was not the correct signal, or that signal is much more complex than they originally thought.
One possible explanation is that there may be different subsets of neurons associated with different signals. That is, one subset of neurons in the IT cortex may respond in terms of image familiarity versus novelty, and another subset may respond in terms of image memorability. In the future, Dr. Rust plans to continue teasing out the relationship between these two seemingly conflicting neural signals.
Implications of Familiarity Memory
Memory deficiencies play a major role in neurodegenerative conditions such as dementia and Alzheimer’s disease, which are often devastating to patients and their families. Research on the mechanisms of visual familiarity memory may lead to the development of new or better treatments for brain disorders that affect memory.
Another possible intervention pertains to how familiarity and novelty may factor in learning processes. Dr. Rust published an article last year in Current Opinion in Neurobiology that explores the role of visual novelty in learning. Children and young animals are innately drawn to novelty, and they tend to show more interest in exploring new things. However, classic models of learning have largely centered on the importance of external rewards, reinforcement, or some combination of the two. Dr. Rust’s research suggests that traditional learning models could potentially be expanded to include a novelty-based system as well. The concept of novelty is also being explored for its ability to drive learning in artificial intelligence technology.
The brain’s familiarity memory system is highly complex, and many questions remain to drive further exploration. The answers may have important impacts on our deepening understanding of the relationship between the eyes and the brain.