New Google tool lets you experience synesthesia

Dive into the mind of the abstract artist who could hear colors with “Play a Kandinsky.”

When Russian artist Vasily Kandinsky was born, the first computer hadn’t even been built yet.

Now, more than 75 years after Kandinsky’s death, an online tool powered by AI is allowing people to experience his paintings the same way he did: as music.

Hear Colors, See Sounds

Kandinsky had a neurological condition called synesthesia, in which the stimulation of one sense triggers another.

In Kandinsky’s case, seeing certain colors caused him to hear certain sounds and feel certain emotions. Yellow sounded like trumpets and felt cheeky, for example, while red sounded like a violin and felt restless.

“When Kandinsky painted, two senses worked systematically together: hearing and sight,” Serge Lasvignes, president of the Centre Pompidou, an art museum, library, and research center in Paris, wrote in a Google blog post.

“Colors and shapes translated into sounds, harmonies, and vibrations made up lines and patterns,” he continued.

Connecting colors and sounds is just one of as many as 80 different ways to experience synesthesia, though. Other people associate letters with colors or words with flavors.

No one knows for sure how many people have synesthesia — depending on your source, it could be one in 2,000 people, one in 300 people, or (to some extent) everyone — but famous “synesthetes” include musician Kanye West, author Vladimir Nabokov, and composer Richard Wagner.

For people who don’t experience synesthesia naturally, the concept can be hard to understand — how can a color have a sound? — so Google Arts & Culture and the Centre Pompidou collaborated on “Play a Kandinsky.”

This online tool lets users hear what Kandinsky might have heard when he painted his masterpiece, Yellow-Red-Blue, in 1925 — and all of the music in the experiment was written by AI.

Recreating a Masterpiece

The first step to creating “Play a Kandinsky” was understanding exactly how Kandinsky experienced synesthesia, so the researchers studied his writings about his experiences.

Next, they needed the music.

By inviting the machine to express itself, the music kind of emerged by itself.


Antoine Bertin

Experimental musicians NSDOS and Antoine Bertin trained a Google AI to compose music for parts of Yellow-Red-Blue based on what they’d learned about Kandinsky’s synesthesia and the phonographs in the artist’s personal collection.

“We trained Google Transformer, a neural network that was developed by the Google Magenta team, on music from Kandinsky’s time and invited this machine to generate new scores to form a new perspective on what the painting may sound like from today’s point of view,” Bertin said.

“By inviting the machine to express itself, the music kind of emerged by itself,” he added. “And it felt that, as artists, we were more the intermediary from Kandinsky’s time to today rather than the interpreters of the theory ourselves, and that was very exciting.”

Your Chance to Experience Synesthesia

Now, anyone who wants to experience synesthesia the way Kandinsky might have can visit the “Play a Kandinsky” experiment, which is broken into four chapters.

The first chapter introduces the concept of synesthesia and how Kandinsky experienced it.

In the second, users click on one of three shapes — a yellow rectangle, a red cross, or a blue circle — to hear a snippet of music similar to what Kandinsky might have heard and felt when he looked at that shape.

When Kandinsky painted, two senses worked systematically together: hearing and sight.


Serge Lasvignes

The third chapter is meant to let users experience synesthesia the way Kandinsky might have while looking at Yellow-Red-Blue.

Google broke the painting down into seven regions, and by clicking on one, users trigger a musical movement based on the shapes and colors in that region.

The region with a yellow rectangle at its center, for example, triggers the “Yellow Movement,” described as “simple and bright shapes which echo earthly madness and passionate joy.”

In the final chapter, users make their own musical movement based on Yellow-Red-Blue.

They start by choosing two emotions from a list, and then listen to those emotions conveyed through the painting. By clicking on various parts of the painting, they can add or subtract additional sounds.

Whether “Play a Kandinsky” achieves its goal of allowing users to experience synesthesia the way the artist might have is hard to say.

However, it does make the abstract concept easier to grasp, and bringing AI into the experiment could entice a new audience to explore the works of one of the greatest artists of a time before digital technology.

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at tips@freethink.com.

Related
Farmers can fight invasive insects with AI and a robotic arm
As the invasive spotted lanternfly threatens to expand its range, Carnegie Mellon researchers are developing a robot to fight back.
Google unveils AI try-on feature for shopping
Google’s AI-powered virtual try-on feature lets shoppers see what an article of clothing would look like on a wide range of models.
GitHub CEO says Copilot will write 80% of code “sooner than later”
GitHub CEO Thomas Dohmke goes in depth to answer questions about how AI-powered development will change the future of innovation itself.
No, AI probably won’t kill us all – and there’s more to this fear campaign than meets the eye
A dose of scepticism is warranted when considering the AI doomsayer narrative — there are commercial incentives to manufacture fear of AI.
AI is riding to the rescue on wildfires
AI-powered systems designed to detect, confirm, and detail wildfires at the earliest possible time may help firefighters tame infernos in the West.
Up Next
virtual patients
Subscribe to Freethink for more great stories