Can Mathematical Tools Illuminate Artistic Style?

March 1, 2005

Using a wavelet-related image processing tool developed for the detection of doctored digital photographs, the Dartmouth researchers analyzed eight drawings attributed to Bruegel (circles) and five acknowledged imitations (squares). As shown here, they were ultimately successful in separating the imitations from the authentic Bruegels from the authentic Bruegels.

Sara Robinson

Mathematicians often wring their hands and lament the lack of stories about mathematical research in the popular press. Yet in December, dozens of media outlets, including The New York Times and Wired, ran lengthy news stories on a rather technical paper involving mathematical tools for digital image processing. Why did this story attract attention when so many others have not?

It is not that the researchers solved a longstanding open problem. The result is a novel application of existing tools, but it involves no new mathematics. "The most interesting part is the marriage of the tools and the application," says Dan Rockmore, a professor of mathematics and the initiator of the project.

It helped that the paper appeared in the Proceedings of the National Academy of Sciences, one of a handful of general science journals that science reporters read regularly. But the primary reason behind the media frenzy was that the research connected mathematics to something far more digestible to the public-in this case, art. The authors---three Dartmouth researchers, Siwei Lyu, Dan Rockmore, and Hany Farid---had used image processing techniques to distinguish authentic Bruegel drawings from imitations.

"Art and science seem to be on different intellectual axes: Art is all warmth---beauty, emotion, and the soul---whereas science is cold---cool, rational thought," Rockmore says. "So when one can illuminate the other, then, as in any unexpected connection, there is a sense of surprise---and this is newsworthy."
Even researchers with an unusual and appealing project need both savvy and persistence to bring their work to the attention of audiences outside their own research communities. Rockmore and Farid are ideally suited for the task.

Rockmore has an eclectic array of interests across the arts and sciences. He has made his career in mathematical research (computational harmonic analysis and signal processing), while pursuing outside interests in art and art history. He makes frequent pilgrimages to art exhibits and socializes with art historians. He also keeps up with and, from time to time, contributes to the popular press. His essays and book and film reviews have appeared in newspapers and magazines, and he is an occasional commentator on Vermont Public Radio. He has even produced a documentary about mathematicians, and has another two mathematics-inspired films in the works.

Farid, trained as an applied mathematician, now works in computer science, applying existing mathematical tools to nonstandard situations. "I live at the crossroads in applied math and engineering," he says. "I don't think of novel tools, I just find a fit with exciting

A Private Tour: Genesis of a Collaboration

For Rockmore, the project began in the fall of 2001 at a dinner party hosted by his friend Ellen Handy, a professor of art history at City College of New York. At the party, Rockmore was introduced to Nadine Orenstein, the curator of a show of drawings by the Flemish artist Pieter Bruegel then running at the Metropolitan Museum of Art. Tickets were hard to come by, and the crowds made it difficult to see the drawings. Orenstein thus invited Rockmore to view the show on a Monday, when the museum is closed to the general public, but open to staff, researchers, and curators. He accepted eagerly.

On the chosen Monday, Orenstein took the time to show Rockmore around the exhibit, a focus of which was comparing Bruegel's work with that of his many imitators. (Bruegel, like Rembrandt, was often copied by admirers and forgers.) At first, it was hard to tell the difference between the real Bruegels and the imitations, Rockmore says, but as Orenstein pointed out the subtle differences in style, he began to see what she was seeing.

The drawings were characterized by delicate lines and shading, and, as Orenstein showed, the shapes and textures of the individual pen strokes betrayed the hand of the artist. "What art historians pay attention to is the lines: darkness or width, or how jagged they are," Rockmore says. "This is the style of the artist in the eye of the connoisseur." Intrigued, Rockmore thought immediately of the science of digital image processing, which enables researchers to project images into a space in which the thickness and orientation of lines can be extracted precisely.

"The tools of digital image processing might highlight the same points as the experts, or they might reveal different things," he thought. "Perhaps artistic style could be precisely quantified."

Back at Dartmouth, Rockmore chatted about his experience with Farid, whose specialty is digital image processing. Farid says that he, too, had thought off and on about applying his digital image processing techniques to art authentication. It would be a long shot, he thought, but worth a try because it would be "very cool" if it actually worked.

Farid's research has focused on mathematical tools for detecting doctored photographs. Each technique for altering an image will change some of its statistical properties, he explains. When two images are spliced together into one, for example, the final image has to be resampled and interpolated onto a new lattice, a process that introduces new correlations between pixels.

Another thread of his research, more relevant to art, distinguishes between computer graphics images and photographic images. This problem became practically important in 2002, when the Supreme Court ruled that computer-
generated child pornography is legal, whereas actual photographs are not.

Orenstein was happy to provide Rockmore, Farid, and Farid's graduate student Siwei Lyu with high-quality 35mm slides of the drawings from the exhibit. With a wide variety of drawings to choose from, the researchers decided to focus on landscapes. They culled eight similar drawings attributed to Bruegel and five acknowledged imitations, and scanned them at a resolution of 2400 dpi. After cropping the images to focus on a central region and converting them to grayscale, they subdivided the images into 64 256 x 256-pixel regions.

Getting the Tools to Work

Wavelets and similar transforms are useful for this application because they are capable of capturing precisely the features that make an artist's drawing (or painting) style distinct: the length, orientation, and thickness of the pen (or brush) strokes.

The decomposition, originally developed for photographs, is based on "separable quadrature mirror filters." These functions are similar to wavelets but technically distinct because the associated transformation is not invertible. The method transforms an image into frequency space and then splits it into multiple scales and orientations. Each image is represented as a vector of length 72. The first 36 values are coefficients that directly reflect the energy at each frequency at different scales and orientations, and the second 36 are error statistics, measuring the degree to which each coefficient can be described as a simple function (called a linear predictor) of its spatial, orientation, and scale neighbors. The error statistics thus reflect the smoothness of the style.

With images from the Bruegel show, Lyu went to work, applying the model developed for digital photographs to the regions. "At first," Farid recalls, "the project was a colossal disaster." A straightforward application of the tool did not separate the imitations from the authentic Bruegels. Lyu worked for a few months, tweaking the linear predictor, but eventually the researchers put art authentication on hold and turned to other work.

Returning to the Bruegel project many months later, Farid again toyed with the predictor; at one point, it seemed that the Bruegels were beginning to cluster together. With this encouragement, the three researchers joined forces again, and this time they succeeded. The key, Farid explains, was to focus on the optimally predictive neighborhood of each point in the frequency space. "Let's say that the master did all his pen strokes at an angle of 37 degrees," he says. "Then you should look for redundancy at that angle."

As a further test of their methods, the researchers turned to a painting in Dartmouth's Hood Museum: Madonna With Child, by Pietro di Cristoforo Vannucci (Perugino). As with many Renaissance paintings, art historians believe that Perugino painted a portion of the work and that apprentices did the rest. By uncovering statistical differences in the brushstroke styles in the six faces in the painting, Rockmore, Farid, and Lyu would provide further evidence for this theory. After photographing the painting, scanning it in, and performing an analysis similar to that done for the Bruegel drawings, the researchers found that three of the faces clustered together in the transform space, while the points corresponding to the remaining faces were distinct. From this, the researchers concluded that at least four artists were involved in the painting, a view consistent with those of several art historians.

A common pitfall in research of this type is to create a model that is optimized for a particular data set but does not work on any others. The researchers emphasize that the PNAS paper is only a first step; to validate the model, they must also test it on works by other artists. The Perugino was not an ideal test case, because there is no clear consensus on the number of artists involved in the painting.

Still, Rockmore notes, the results are encouraging because the underlying model was developed for a very different problem. "The amazing thing about our results is that the tools weren't designed for drawings," he says. "Often in these learning algorithms, if you train stuff well enough, it works; in this case it wasn't trained."

Getting Published

The next task was to submit the paper to an appropriate journal. Because the most interesting aspect of the research was the application rather than the method, the researchers hoped to expose it to a broader audience than just mathematicians and engineers. Their first thought was to try for Nature or Science, but neither journal was interested. On appealing the rejection from Science, they were told that the work was too technical and too narrow.

In the end, the researchers turned to the Proceedings of the National Academy of Sciences, which has a special track for member-supported submissions. Rockmore was acquainted with NAS member David Donoho, a professor of statistics at Stanford. Donoho liked the work, although he warned that it is generally hard to get mathematics papers into PNAS. Indeed, the paper encountered resistance at first, but was eventually accepted for publication (and appeared in the issue of December 7).

"It really was a lesson on how the top science publications are predisposed against mathematics," Rockmore says. "They said it was too technical, yet when I open up Science or Nature, the articles on medicine or chemistry seem very technical to me." It was also a lesson on how persistence pays off.

Detail from a Bruegel drawing (top left) and the extractions (clockwise from top right) of its horizontal, vertical, and diagonal elements.

The Next Step

The researchers' next step will be to expand their data set. Getting access to famous works of art isn't easy, but since the media blitz about the paper, several curators and conservators have expressed interest in making their collections available for testing. As a result of discussions now under way with the Met, the researchers hope to receive digital photographs of the museum's Rembrandt collection: To date, only 14 of the 21 paintings have been authenticated. They also hope to gain access to photographs from a Botticelli exhibit that is scheduled to run at the Met in the fall of 2005. Meanwhile, they are analyzing a drawing owned by Harvard that was recently attributed to Bruegel.

As is often the case with news about science, the media outlets didn't get all the details right. The stories included few, if any, technical details, but several news accounts portrayed the work as a salvo in the battle of man against machine, which Farid and Rockmore consider unfortunate. "Most art experts don't see it that way, because the method would not replace what they do," Rockmore says. Conservators, whose job it is to determine the provenance of paintings, already use technical tools, such as chemical analyses of paint and infrared imaging, he points out. The Dartmouth model, if it works on other data sets, would simply provide them with yet another tool for their arsenal.

Sara Robinson is a freelance writer based in Los Angeles, California.

Donate · Contact Us · Site Map · Join SIAM · My Account
Facebook Twitter Youtube linkedin google+