Though the mug is perceived as a homogenous blue, each pixel shows slight variations in reflected wavelengths, due to changes in illumination. Source: Brainard, D. H., & Maloney, L. T. (2011). Surface color perception and equivalent illumination models. Journal of Vision, 11(5), 1-1. doi:10.1167/11.5.1. Association for Research in Vision and Ophthalmology is copyright holder.
By Rebecca Salowe
Scheie Vision Annual Report 2019
David Brainard, PhD, the RRL Professor of Psychology and Director of the Vision Research Center at University of Pennsylvania (UPenn), is fascinated by color constancy. His longstanding R01 grant on this topic has yielded numerous discoveries and publications in journals such as Journal of Vision, Annual Review of Vision Science, Ophthalmology, PLoS Computational Biology, and Annual Review of Neuroscience.
Color constancy refers to our ability to perceive colors as relatively constant over varying illuminations (i.e. light sources). For example, a red apple will still look red on a sunny day or cloudy day – or in a grocery store or a home.
However, when we look at the apple, the resultant image on the retina is based not only on its intrinsic red, but also on the spectrum of the light source striking it. By this logic, the apple would shift in color depending on the surrounding illumination. Thankfully, our brain employs the process of color constancy, which “undoes” the surrounding illumination and allows us to perceive objects as a consistent color in different settings.
“It’s one of many examples of where the image on the retina does not correctly, or immediately, predict what we see,” explained Dr. Brainard. “For example, a similar illusion is how a phone looks the same shape when you pick it up and rotate it. Color constancy is why we can say, ‘this is a red shirt’ and not ‘this is a red shirt under this specific light.’”
We use the process of color constancy many times over the course of the day. Small differences in color provide us with essential information about an object’s properties, such as distinguishing fresh fish from old fish. Even simple tasks such as putting together a matching outfit in your closet, or interpreting a stop sign on a cloudy versus sunny day, involve the process of color constancy.
Sometimes, color constancy fails or can be manipulated. For example, the produce section in a supermarket intentionally shines bright lights on the produce. “These lights are designed to defeat our color constancy and to make the food look ripe and fresh,” said Dr. Brainard. “This is why you may buy a tomato, thinking it looks very ripe, then take it home and wonder why you bought that tomato.”
Dressing rooms in retail stores provide another example. Here, the lighting is often altered to make clothing look as flattering as possible, encouraging purchases. A dress that looked very appealing in the fitting room may look less enticing in the regular lighting of home, for this reason.
Though it is ever-present in our lives, color constancy has historically been a difficult field to study. “I have spent a fair fraction of my research career learning how to measure, model, and understand color constancy in more detail,” said Dr. Brainard. “This area has been studied since the mid-1800s, and has a long history – which is a way of saying that we have not nailed it yet.”
For his current R01, Dr. Brainard returned to the basic question of why color constancy is an important area to study: it influences decision-making. Previous studies have focused more on how subjects judge the appearance of an object, without taking the extra step to evaluate subsequent choices. In these experiments, subjects adjusted the color of an object seen under one illumination to match the color of a reference object at a different illumination. However, this task is not representative of everyday life. We do not, for example, adjust the reflectance properties of a tomato to look extra red or tasty; instead, we evaluate the color of the tomato and make a decision about buying it.
Dr. Brainard’s lab took this extra leap, devising experimental methods that directly test how evaluation of object appearance affects decision-making. For example, in one study, healthy participants were asked to match colors of images shown in different illuminations. The subjects viewed computer renderings of an object, such as the cube shown below, which has a reference button and two comparison buttons. The reference was seen under one illuminant, while the comparisons were under another. The subject then decided which comparison button best matched the color of the reference object.
“We force you to choose, and over lots of trials, we vary the available choices in a careful way,” Dr. Brainard said. “The key is that subjects are not asked what the image looks like, or to adjust the color; they are just asked to pick the one that matches most closely. This forces them to do a task that they do in real life. How well you do will depend on how good your constancy is.”
Subjects achieved 100% constancy if they correctly matched the button with identical reflectance properties, but different illumination, to the reference button. In contrast, 0% constancy meant that the subject chose a button that had a different surface reflectance that compensated for illumination differences, making it reflect the same light as the reference. The experiment also probed choices between the 0% and 100% options; these indicated partial constancy.
The results of this experiment, as well as other similar setups, showed that individuals have a wide range of color constancy. At the moment, what drives these differences is not fully understood. It is possible that variations in cone spectral sensitivity or number of cones in different classes contribute to these differences.
In 2015, the viral internet sensation of #theDress brought this topic into popular culture. Viewers passionately debated whether the photographed dress was blue/black or white/gold in color. At first, many assumed that the lighting of the room or glow of the computer screen was responsible for these differences in color perception. However, it quickly became clear that this was not the case. “In my classes, I’ve put the image onto a projector, so everyone is seeing the dress on the same screen and in the same lighting,” said Dr. Brainard. “There is still the same debate.”
Most researchers now agree that differences in the application of color constancy to the dress contribute to these variations. “When different people look at this photo, their color constancy goes one way or the other,” said Dr. Brainard. “Some people process it as if the illumination were yellow, and thus discount the yellowness and get a blue/black look; other people process the illumination as bluer and see the dress as white/gold.”
In addition to examining how color constancy influences object selection, Dr. Brainard has also studied how the material of an object affects choices. “We vary color and material and ask subjects which was more similar to the reference,” said Dr. Brainard. “In one case, the color differed, and in one case, the material differed, and we worked on measuring how the two attributes are weighed.”
The results were surprising, showing a large variation from person to person. This suggests that some people rely more heavily on color, while others rely more on material, when evaluating object properties.
Some experiments also probed if providing subjects with feedback on their choices helped to improve future color-based selections. They did find that feedback helped subjects to be more conscious of their choices.
Much of Dr. Brainard's research to date has focused on quantifying how color is perceived and used. Going forward, Dr. Brainard hopes to extend his research program to gain insight into the neural mechanisms that process the retinal image. This research will help us understand how this processing provides the basis for the color constancy we enjoy.