Multimer: Better Decisions Based on Human Signals

Think of where you were fourteen years ago. A lot has changed, right? Technology has changed. Information flow has changed. The way we relate to and understand each other has, in many significant ways, changed.

Fourteen years ago, people used Discmans. Much has changed!

Fourteen years ago, my co-founder Ilias Koen and I met as students at an art and technology program in New York City. Back then, people still used portable compact disc players to listen to music on the subway. (Yes, the Discman.) Since then, we’ve helped each other build robots that draw, gloves that fly swans, and bike helmets that read brains. All of our projects, from grad school to the three companies we have co-founded, have been driven by a deep curiosity over the ways that data can unveil hidden relationships between our natural environment, our built landscapes, and ourselves as individual people. Our curiosity has been anchored in New York City, but it’s a curiosity that encompasses the world, its ecosystems, its major municipalities— especially now that the majority of the world’s population lives in cities.

“Exploring relationships between our natural environment, our built landscapes, and ourselves:” a still from a Unity interactive about tectonic plates in the Pacific Northwest. Interactive by Ilias Koen for The DuKode Studio.

In the past fourteen years, the convergence of several mature technologies and communication protocols — GPS, the Internet, cellular networks, Bluetooth, LEDs, touch screens — has empowered societies such that an individual person can generate massive amounts of new data each day — data that reflects emotions, thoughts, tasks, goals, dreams — through her computer, phone, and electronic devices. This is the kind of data that, by volume alone, often falls under the umbrella of “Big Data.”

An image of “big data” (Internet usage) from UN Global Pulse, our recent community partner. They focus on the use of big data to serve the United Nations’ Sustainable Development Goals. We admire and hope to contribute to this effort of “Data for Good.”

There is another kind of big data that has been unlocked by the advances of the past fourteen years. We’ve nicknamed this data “Human Signals.” Human signals are basically the kinds of physiological signals that were once measured only in hospitals or specialized clinics: heart rates, brainwaves, muscle movement, respiration, sweat, steps — the list goes on. Our ability to collect and analyze this data has been limited. Until now.

Some of our diverse data collectors capturing their brainwave data with our biosensor system MindRider, http://mindriderdata.com

Now, thanks to that convergence of technologies from the past fourteen years, we can measure human signal data not just from one individual, but from a whole population, and not just at one moment in time, but for days, weeks, months, and more. You may even own a device that measures some of these human signals, which have played a key role in understanding how our environments affect us. Being able to collect such large amounts of data, in such high spatial, temporal, and spectral resolution allows us to ask bigger questions, such as: when you average out the variance between individuals, days, and experiences, what does the real emotional temperature of an environment look like? What does the accurate mood map of a region look like?

A visualization of the first major MindRider dataset, depicted in 3D to show levels of pure attention (red) near bridges and tall buildings. This data was collected by 10 cyclists who rode the major streets of Manhattan in 2014. Learn more at http://book.multimerdata.com

These are some of the questions that drive us as we continue to develop our our location-aware biosensor MindRider and build our analytics platform Multimer. We think that their answers could have a profound impact not only on placemaking, wayfinding, and sensemaking, but on how we conceive of, report on, and plan these processes. They could have a profound impact on how media is made, and on how it is consumed. The past fourteen years — and the past fourteen decades — have shown us that changing technologies can have profound, unforeseen impacts on media — and vice versa.

A visualization of the first statistically significant MindRider data study from 2016. Dozens of cyclists and a few pedestrians contributed data for all the streets of Manhattan south of Central Park. Red indicates levels of pure attention. Explore this dataset at http://collected.multimerdata.com
Our first Multimer Experience Map, based on 2016 MindRider data. The MindRider data was gridded and classified to show basic mental experiences of interest to our partner organizations. http://categorized.multimerdata.com
The Multimer “Best Neighborhoods” map, which compares the MindRider data (and its initial “Multimer Experience Map” classification) against relevant external datasets including parks, landmarks, access to public transit, and residential zoning. This algorithm is based on established qualitative criteria from the APA. http://bestneighborhoods.multimerdata.com

We understand that profound, unforeseen impacts can yield great rewards — and great risks. While we have been working with biosensors for years, and have been formally analyzing the resulting data since 2014, we acknowledge that human signal data, like most human data, poses risks around privacy, bias, misrepresentation, and many other ethics issues. We’re honored to be supported by Matter, especially at this crucial time for media and the flow of global information. Many of the ethical risks we face in collecting and analyzing human signal data are the same kinds of risks that all new media technologies face, and only at this crucial time are we all beginning to realize this.

As a federally-funded program, we are required to adhere to the same human subjects trainings and review boards required for medical and clinical trials. To that end, we are also grateful to be supported by the National Science Foundation’s SBIR program, Urban-X/SOSV, and the E14 Fund at MIT Media Lab, where this project began way back in 2011 with my early prototype of a brain-reading bike helmet. The “drunken walk of the entrepreneur” never happens alone.

MindRider circa 2014 (back when it was still a helmet). Want to see the very first MindRIder prototype circa 2012, as covered by WIRED?

Since building that first prototype, we have been privileged to work with and measure data from many, many kinds of people, including our old friend and new data engineer Tommy Mitchell, hardware designer Yapah Berry, research coordinator Tania van Bergen, and recent partnerships manager Natalia Villegas. And while the insights from human signal data drive our research, it is the humans themselves — the diverse, rambunctious group of people that move through and work in the areas we measure — that drive our work. We believe that technology only works for the people it serves when it is made from and by the diversity of people that it represents. If nothing else, it is this belief with which we hope to make a lasting impact. This is what we fundamentally mean when we say better decisions based on human signals.

DuKorp’s team past and present wearing MindRiders past and present. Photos by Ben Tudhope for DuKorp and Lindsay Dill for the League of American Cyclists.