Derya Akkaynak remembers clearly the day she saw the sea without water.
A late summer rain had forced her to abandon her ritual of a Sunday motorcycle ride, and, consigned to the inside, she decided to tweak the algorithm she had been working on for the last three and a half years. She input the changes that had been rattling around her mind — it just hadn’t worked so far — and the image she had spent months staring at, an image made muted, alien, by the real-world Instagram filter of the ocean, suddenly burst into life.
Stripped by her algorithm of the water and its warping effects — bane of underwater photography — the corals along the bottom of the image became a dazzling array of pinks and purples, blues pied by yellows, a breathtaking display that causes excitement in her voice even a year after that rainy August day in 2018.
“My first thought,” Akkaynak says, “was why is everything so colorful?'”
Underwater Images Without the Water
Underwater photography and videography is at the mercy of the water. The underwater world can look vastly different from place to place. Your underwater photos could be shot in the ethanol-clear waters of Hawaii, the grey haze of the Atlantic, or in the iced tea opacity of a rushing river. Even underwater images from the same location will be maddeningly variable.
One day, silt could create a small wall of particles, cutting off your vision; the next, an algae bloom could change your world from blue to green. The time of day will have an impact, as the angle of the sun changes; even the angle of the camera. The water interacts with the light in unusual ways, muting colors and warping perceptions. All of this makes underwater pictures and videos harder to analyze for scientists.
Growing up in Turkey, Akkaynak — her first name means “ocean” — spent months of her childhood in the small fishing village of Cesmealti on the Aegean Sea. There, she would spend her days swimming around and poking at the marine life, trying to understand how and why they do what they do.
After a stint as an aerospace engineer, she realized she needed to return to the ocean. While studying the remarkable camouflage abilities of octopuses, she became frustrated with the quality of the underwater images she was working with.
“Any image you take underwater, it’s just degraded by this layer of green or blue or, if you’re in a murky place, brown,” Akkaynak says. It’s hard enough to pick out an octopus underwater. Akkaynak was vexed by the water’s impact and wished she could eliminate it.
A few years later Sea-thru was born, an imaging algorithm that makes underwater images appear like they would on land. Developed in the Marine Imaging Lab of Dr. Tali Treibitz at the University of Haifa in Israel — the university is currently in the process of commercializing it — Sea-thru does more than just color correct.
“Many people say, ‘But I can do this in Photoshop!” Akkaynak laughs. In addition to being performed by an algorithm rather than a person, so it is more objective and efficient than Photoshop, her physics-based algorithm does not need a color chart, and it does not simply layer adjustments on top of the image until it pleases a subjective eye test. It uses the physical properties of water and light to reverse engineer the impact of the water on the raw image data.
Essentially, the algorithm understands what water conditions are doing to the light, and, by working backwards, it reverses those conditions. The end result is as if the water was never there.
Beyond Making Underwater Photography Look Better
Sea-thru’s images are arresting. Colors pop, the movements of marine life become clearer, edges and shapes take on sharper focus. But the algorithm was designed to do more than just make underwater photography and videography more pleasing to the eye; it was designed to make these underwater images easier for scientists to analyze. By automatically adjusting, cleaning, and standardizing imaging data, Sea-thru may be able to save scientists a ton of time.
“That was my goal,” Akkaynak says. “To help speed up the pace of marine science.” Additionally, Sea-thru may solve the fundamental problem of recovery color and contrast (and find those damn octopuses).
Alex Jordan has been putting Sea-thru to the test. A researcher at the Max Planck Institute of Animal Behavior, he studies how social behavior evolves and is impacted by the environment. Jordan does this by studying populations of social spiders (your author’s new nightmare) and cichlids in Lake Tanganyika.
The fish in the African Great Lakes are famous for their incredible diversity, making them perfect for Jordan’s research. One type of cichlid in the lake will be incredibly social, and another, closely related type, a lone wolf, allowing for Jordan and his team to analyze and compare. They do this by filming hours and hours and hours of video footage.
That footage is cleaned up then fed to a machine learning algorithm, which will help pick out the little tics, motions, and nuances of the cichlid’s behavior. With its superb pattern finding ability, the algorithm can then help Jordan better understand what’s happening in cichlid’s social lives.
Machine learning requires clean, consistent data to learn from — which underwater images are not. If you want the algorithm to track this fish and not that fish, the water conditions can make it tough. Say your underwater images one day are of a black and white fish in a brown world; the algorithm can learn that. Then the next day an algae bloom hits, and your fish is now green and gray in a yellow world!
“Remember that dress illusion that went around a year or two ago?” Jordan says. “It’s kind of exactly like that. You’re trying to train a model to detect a difference, but then day to day to day those differences themselves are different. And it just causes us incredible headaches.”
Sea-thru standardizes that data and eliminates the water conditions that make underwater images difficult for algorithms and humans to analyze.
Marine biologists aren’t the only researchers who need their underwater images clear as day. Archeologists rely on underwater photography to document and research important finds, and use video to survey and capture artifacts, shipwrecks, and other remnants of humanity scattered under the sea.
Zoë J. Wood is a professor of computer science at Cal Poly whose research focuses on computer graphics and modeling. Working with engineer Christopher Clark at Harvey Mudd College and archeologist Timmy Gambin from the University of Malta, Wood helped develop algorithms for identifying and mapping shipwrecks off Malta’s coast. Two years ago, they found a rare British plane from World War II, now home to crustaceans and baby fish.
The researchers want to not only excavate ancient wrecks, but also be able to compile high-resolution, 3D models of wrecks and use that information to keep tabs on their condition year after year. Using autonomous underwater vehicles (AUVs) — swimming drones — they take underwater images, which can then be stitched together to create models of the ship.
The images Wood is working with face the same challenges as Jordan’s; the light coming through the water means the images are not as detailed and accurate, leading to less accurate modeling.
“We have hours and hours of blurry blue video,” Wood laughs.
Sea-thru could help their research “massively,” Gambin says. It could improve the 3D models and reduce image processing time.
The algorithm may make another intersection of humanity and the sea clearer, as well. Jordan sees potential in combining Sea-thru with underwater photography and videography to create environmental awareness. Remove the polluted waters from a coral reef, for example, and the true beauty of what could be may shine through.
A Different View of the Sea
Seeing his cichlids and their environment clearly for the first time was a special moment, Jordan says. There is red sand at the bottom of Lake Tanganyika, which always comes up in his boots. Watching the fish swim against this red backdrop for the first time was moving.
“It’s really nice to see the fish in their true color in their environment,” Jordan says. It was like capturing absolutely perfect water conditions.
While Akkanyak’s goal was to help scientists analyze their data better, freeing them up for more research, the eerie beauty of the underwater images Sea-thru creates cannot be denied.
She returned to the sea, the setting of her youth, and has made it disappear.
