Ideas & Insights / All Perspectives / Ideas & Insights

How Digital Jedis Battle Big Data: A True Story

Digital Humanitarians

The overflow of information generated during disasters can be as paralyzing to humanitarian response efforts as the lack of information. Computers, mobile phones, social media, mainstream news, earth-based sensors, humanitarian drones and orbiting satellites generate vast volumes of data during major disasters. Making sense of this flash flood of information, “Big Data”, is proving an impossible challenge for traditional humanitarian organizations. They are more adept at dealing with information scarcity than overflow. So these organizations are turning to Digital Humanitarians for help.

“The first ‘battle’ between Digital Jedis and Big Data began on January 12, 2010, following the devastating earthquake that struck Haiti. Within hours, Digital Jedis mobilized online, launching a ‘Crisis Map’ pinpointing the damage and resulting needs across the Haitian capital of Port-au-Prince.”

Who exactly are these Digital Humanitarians? They are volunteers and professionals from the world over and from all walks of life. What do they share in common? The desire to make a difference, and they do by rapidly mobilizing online in collaboration with international humanitarian organizations. In virtually real-time, they make sense of “Big Data” to support relief efforts worldwide. How? They craft and leverage ingenious crowdsourcing solutions with trail-blazing insights from artificial intelligence. My new book, Digital Humanitarians, charts the sudden and spectacular rise of these “Digital Jedis” by sharing their remarkable, real-life stories, highlighting how their humanity coupled with innovative solutions to Big Data is changing humanitarian response forever. Digital Humanitarians will make you think differently about what it means to be humanitarian and will invite you to join the journey online.

The first “battle” between Digital Jedis and Big Data began on January 12, 2010, following the devastating earthquake that struck Haiti. Within hours, Digital Jedis mobilized online, launching a “Crisis Map” pinpointing the damage and resulting needs across the Haitian capital of Port-au-Prince. They manually monitored social media and mainstream news around the clock, searching for relevant and map-able reports from Haiti. Within a week, thousands of digital volunteers from dozens of countries around the world had come together online to map the latest reports from Port-au-Prince; the majority of these Digital Jedis were Haitians living abroad. Meanwhile, other volunteers crowdsourced the analysis of satellite imagery to create the most detailed street map of Haiti ever made. Ten days into these digital humanitarian efforts, the head of FEMA, Craig Fugate, referred to these crisis maps as the most detailed and updated maps available to the humanitarian community. Weeks later, the U.S. Marine Corps emailed these Digital Jedis to thank them for their efforts, noting that their maps had helped them save hundreds of lives.

This first battle with Big Data taught Digital Jedis some important lessons. Perhaps the most obvious one was this: crowdsourcing alone would not win the Big Data battles of the future. More than 20 million tweets were generated during Hurricane Sandy in 2012, for example. Manually reading all these tweets—to look for the most important ones—would take one person about 60,000 hours or just over 6 years. So what Digital Jedis needed were next generation humanitarian technologies powered by crowdsourcing and artificial intelligence. This is where AIDR comes in—Artificial Intelligence for Disaster Response. AIDR uses crowdsourcing to train algorithms that automatically identify relevant tweets during disasters. (Incidentally, AIRS—Artificial Intelligence for Resilience Societies—uses the same approach as AIDR and will soon be piloted by The Rockefeller Foundation’s 100 Resilient Cities initiative).

Obviously, tweets are not the only source of Big Data during disasters, which is why MicroMappers exists. This platform seeks to combine crowdsourcing with artificial intelligence to automatically identify relevant features in pictures, satellite imagery and even aerial imagery. While still experimental, this new platform should go a long way to support Digital Jedis during future humanitarian crises. But AIDR and MicroMappers only “solve” the needle-in-the-haystack problem. How are we to know whether the tweets and pictures that are automatically identified as important during disasters are actually credible? This is where the prototypes Verily and TweetCred come in. Verily uses time-critical crowdsourcing to verify and counter rumors during disasters. Meanwhile, TweetCred uses artificial intelligence to score the credibility of crisis-related tweets. Note that each of these new prototypes are directly applicable to social good projects more broadly. In other words, their value is not limited to disaster response only, as evidenced by the recent use of MicroMappers for wildlife protection (pictured below).

While AIDR, MicroMappers, Verily and TweetCred sound promising—I’m obviously biased since my team and I at QCRI are the ones developing these free and open source solutions—what is far more important are the scientific methodologies driving these platforms—namely human computing (crowdsourcing) and machine learning (artificial intelligence). This is one of the more important messages in my new book: focus on the science. Speaking of messages, the most important one that I hope readers will take from the book is that the story of Digital Humanitarians is first and foremost a human story. Hope, in a word, is the main reason I wrote this book. In a world seemingly overrun by heart-wrenching news headlines and daily reminders on social media about all the ugly and cruel ways that technologies are being used, I felt the pressing need to share a different narrative; a narrative about how selfless volunteers from all walks of life, from all ages, nationalities, creeds use digital technologies to help complete strangers on the other side of the planet.

I’ve had the privilege of witnessing this digital goodwill first hand and repeatedly over the years. This goodwill is what continues to restore my faith in humanity and what gives me hope, even when things are tough and not going well. And so, I wrote Digital Humanitarians first and foremost to share this hope more widely. We each have agency and we can change the world for the better. I’ve seen this and witnessed the impact first hand. So if readers come away with a renewed sense of hope and agency after reading the book, I will have achieved my main objective.


Patrick Meier is an internationally recognized thought-leader on humanitarian technology and innovation. His new book “Digital Humanitarians” has already been endorsed by Harvard, MIT, Stanford, Oxford, UN, World Bank and the Red Cross. Patrick wrote the first draft of the book while a Residential Fellow at The Rockefeller Foundation’s Bellagio Center. He currently directs QCRI’s Social Innovation Program where he develops “Next Generation Humanitarian Technologies” in partnership with international humanitarian organizations. Patrick has a PhD from The Fletcher School, Pre-Doc from Stanford and an MA from Columbia. His work has appeared in the New York Times, Washington Post, CNN, BBC, Forbes, Times, Wired, and Mashable. Patrick’s influential blog iRevolutions has received over 1.5 million hits. Follow on Twitter at: @patrickmeier.

Leave a comment