Op-Eds & White Papers

Nonnecke, B., Prabhakar, T., Brown, C., & Crittenden, C. (July 2017). Inclusive AI: Technology and Policy for a Diverse Urban Future. CITRIS and the Banatao Institute.

Nonnecke, B. (Jan. 2017). Revolutionary Code: Why we need more conscious and collaborative social movement technologies. Context: By New America.

Nonnecke, B., Bruch, M., & Crittenden, C. (June 2016). IoT & Sustainability: Practice, Policy, and Promise. CITRIS and the Banatao Institute.

Research Articles

Malasakit 1.0: A Participatory Online Platform for Crowdsourcing Disaster Risk Reduction Strategies in the Philippines

Nonnecke, B., Mohanty, S., Lee, A., Lee, J., Beckman, S., Mi., J., Krishnan, S., Roxas, R., Oco, N., Crittenden, C., & Goldberg, K. (2017, October). Proceedings of the IEEE Global Humanitarian Tech Conference (GHTC). San Jose, CA.


We describe Malasakit 1.0 (meaning “sincere care” in Filipino), a customizable participatory assessment platform that collects and streamlines quantitative and qualitative analyses and insights of disaster risk reduction (DRR) strategies. While supervised classification approaches offer opportunity to understand qualitative textual suggestions, those methods break down in areas like the Philippines, home to hundreds of dialects and regional language nuances in varying socioeconomic contexts. Instead, Malasakit uses dimensionality reduction and peer-to-peer evaluation on qualitative textual suggestions to identify locally appropriate DRR strategies. We present results from 12 field tests conducted in eight distinct geographic locations in the Philippines. 998 participants provided 7,249 evaluations on flood and typhoon preparedness and 2,675 peer-to-peer ratings on 907 textual suggestions for how local government could improve DRR strategies. Results suggest that female participants are more confident than males in their community’s ability to recover from a major typhoon. High-rated textual suggestions focus on issuing immediate early warnings and cleaning drainages to reduce flooding. Malasakit can be accessed at

Multistakeholderism in Praxis: The Case of the Regional and National Internet Governance Forum (IGF) Initiatives

Epstein, D. & Nonnecke, B. (2016). Policy & Internet8(2), 148-173.


The growing phenomena of regional and national Internet Governance Forum (IGF) initiatives offer an opportunity to look into how various interpretations of the multistakeholder model play out in different cultural, political, and economic settings. The variety of ways in which the multistakeholderism is enacted are expressed through the organizational structures and procedures of these events, their funding mechanisms, their agendas and formats, the kind of participation they attract and enable, and their potential influence on the national, regional, or global Internet governance debates. This article is a systematic attempt to map out regional and national IGF initiatives with an emphasis on how the multistakeholder model is playing out in various contexts. This analysis builds on existing dispersed documentation of these initiatives, transcripts from meetings (such as global IGF interregional dialogs), and interviews with individuals engaged in facilitation of regional and national IGF initiatives. The goal of this exercise is to offer an empirically grounded framework for thinking about the emerging models of multistakeholder governance.

The Transformative Effects of Multistakeholderism in Internet Governance: A Case Study of the East Africa Internet Governance Forum

Nonnecke, B. (2016). Telecommunications Policy, 40(4), 343-352.


Internet governance issues tend to be contentious, multifaceted, and interconnected among various stakeholders. Since Internet governance transcends national and sectoral boundaries, the Internet Governance Forum (IGF) was formed as a potentially effective and legitimate multi-stakeholder governance network that could better ensure cooperation. An evaluation of the impacts of the IGF model in national and regional contexts, particularly developing country contexts, has gained limited attention. This study is one of the first to explore the function of the IGF model in a regional context by investigating the impacts of the East African Community’s (EAC) East Africa Internet Governance Forum (EAIGF), the first regional IGF established globally.

International regime theory, the theory of elite competition, and a modified version of the actor-centered institutionalism model were used to examine the power of stakeholders, including impacts of institutional endowments on stakeholder influence within the EAIGF and impacts of participation in the EAIGF on member country Internet governance tactics. Data were collected through semi-structured in-depth interviews with EAIGF participants, observations of the 2012 EAIGF meeting, and qualitative document analysis of EAIGF documents and related Internet governance policy documents from EAC member states. Findings reveal the EAIGF promotes capacity building, knowledge sharing, and policy transfer within the region and has contributed to a shift in the balance of power in policymaking processes from state-centric to multi-stakeholder. However, the forum reinforced the influence of states and stakeholders with stronger institutional endowments on the framing of policy issues within the forum. Thus, policy decisions attributed to the forum are likely reflective of the preferences of stronger stakeholders in the forum.

Personalized Telehealth in the Future: A Global Research Agenda

Dinesen, B., Nonnecke, B., Lindeman, D., Toft, E., Kidholm, K., Jethwani, K., Young, H., Spindler, H., Oestergaard, C., Southard, J., Gutierrez, M., Anderson, N., Albert, N., Han, J., Nesbitt, N. (2016). Journal of Medical Internet Research.

As telehealth plays an even greater role in global health care delivery, it will be increasingly important to develop a strong evidence base of successful, innovative telehealth solutions that can lead to scalable and sustainable telehealth programs. This paper has two aims: (1) to describe the challenges of promoting telehealth implementation to advance adoption and (2) to present a global research agenda for personalized telehealth within chronic disease management. Using evidence from the United States and the European Union, this paper provides a global overview of the current state of telehealth services and benefits, presents fundamental principles that must be addressed to advance the status quo, and provides a framework for current and future research initiatives within telehealth for personalized care, treatment, and prevention. A broad, multinational research agenda can provide a uniform framework for identifying and rapidly replicating best practices, while concurrently fostering global collaboration in the development and rigorous testing of new and emerging telehealth technologies. In this paper, the members of the Transatlantic Telehealth Research Network offer a 12-point research agenda for future telehealth applications within chronic disease management.

Case Study: The California Report Card Version 1.0

Nonnecke, B., Aitamurto, T., Catterson, D., Crittenden, C., Garland, C., Huang, A., Krishnan, S., Nelimarkka, M., Newsom, G., Patel, J., Scott, J., & Goldberg, K. (forthcoming Summer 2016). Case Study: The California Report Card Version 1.0. In E. Gordon & P. Mihailidis (Eds.), Civic Media: Technology, Design, Practice (pp. 235-239). Cambridge, MA: MIT Press.


The California Report Card (CRC) v1.0 is an experimental platform developed by UC Berkeley in collaboration with the California Office of Lt. Gov. Gavin Newsom that streamlines public input by openly encouraging suggestions from a broad range of participants, and combining peer-to-peer review with statistical models to identify and highlight the most insightful ideas. The full case study is available at

DevCAFE 1.0: A participatory platform for assessing development initiatives in the field

Nonnecke, B.*, Krishnan, S.*, Patel, J., Zhou, M., Byaruhanga, L., Masinde, D., Meneses, M., Martin del Camp, A., Crittenden, C., & Goldberg, K. (2015, October). Proceedings of the IEEE Global Humanitarian Tech Conference (GHTC). Seattle, WA. [.pdf].

Ribbon AwardRecipient of the Best Paper Award at the 2015 IEEE Global Humanitarian Tech Conference. 



The design and assessment of development initiatives is increasingly participatory, where decision makers consider feedback from affected populations. While digital data collection facilitates faster and more reliable analysis, existing data collection tools are not optimized for unstructured qualitative (textual) data and peer-to- peer participant collaboration. In this paper, we propose a system called the Development Collaborative Assessment and Feedback Engine version 1.0 (DevCAFE), a customizable participatory assessment platform that collects and integrates quantitative assessment, qualitative feedback and peer-to-peer collaborative filtering. DevCAFE incorporates a library of statistical analyses for researchers to quickly identify quantitative and qualitative trends while collecting field data. DevCAFE can run on any mobile device with a web-browser and can work with or without Internet connectivity. We present results from two pilot projects: (1) 137 participants evaluating family planning education trainings at three Nutrition Education Centers in rural Uganda, and (2) 4,518 participants evaluating policy priorities for elected leaders in the June 2015 Mexico mid-term elections. DevCAFE collected over 19,000 peer-to-peer ratings of 336 submitted ideas. Feedback gathered through DevCAFE enabled targeted reforms to the family planning efforts in Uganda and the need for increased government attention to public safety in Mexico. Case studies and interactive demos are available at:

Motivating and Prioritizing Ongoing Student Feedback During MOOCs and Large On-Campus Courses Using Collaborative Filtering

Zhou, M., Cliff, A., Krishnan, S., Nonnecke, B., Crittenden, C., Uchino, K., & Goldberg, K. (2015, September). Paper presented at ACM Special Interest Group for Information Technology Education Conference. Chicago, IL.


We explore a way for instructors to collect and manage student feedback throughout a course rather than waiting for end-of-course evaluations. While qualitative (textual) input is useful, the high volume of input generated in large courses can be difficult for instructors to manage in a timely manner. Platforms such as Piazza are managed by students and teaching assistants and do not provide anonymous feedback to instructors. To collect ongoing feedback and efficiently identify potentially valuable suggestions while maintaining student confidentiality, we developed the MOOC Collaborative Assessment and Feedback Engine (M-CAFE) v1.0. This mobile-friendly platform encourages students to check in weekly to numerically assess their own performance, provide textual suggestions about how the course might be improved, and rate suggestions from other students. For instructors, M-CAFE displays trends and highlights potentially valuable suggestions using collaborative filtering. We describe case studies with two edX MOOCs and with an on-campus undergraduate course. We analyze data and system performance based on over 400 textual suggestions with over 5000 ratings from over 4000 student visits. Initial results suggest that ongoing course evaluation and collaborative filtering can provide valuable and timely feedback to instructors.

A Mobile Platform for Participatory Assessment of Development Programs and a Case Study in Uganda

Nonnecke, B.,  Krishnan, S.,  Akula, A., Cliff, A., Huang, A., Lin, A., Nehama, S., Byaruhanga, L., Masinde, D., Crittenden, C.,  and Goldberg, K. (2015, May). Paper presented at the Humanitarian Technology Conference: Science, Systems and Global Impact, Cambridge, MA.


Development organizations need timely and reliable feedback on the efficacy of program interventions. Traditional assessment methodologies tend to be costly, time-consuming, and lack interaction: participants provide responses but seldom have access to the data or a role in its interpretation. We developed the Collaborative Assessment and Feedback Engine (CAFE) to engage communities from developing regions in collective assessment of local conditions, needs, and outcomes of development programs (see Figure 1).

The CAFE platform collects data in three phases: (1) Quantitative assessment, where participants submit responses on a scale from 0 to 10, “Strongly Disagree” to “Strongly Agree”, establishing baseline participant demographics and opinions. (2) Ideation, where participants provide a textual response to an open-ended question. (3) Collaborative evaluation, where participants evaluate the responses of others. CAFE provides researchers and participants with rapid, preliminary insights while in the field and can be used without Internet connectivity by taking advantage of ad-hoc networks (e.g., mobile devices networked to a laptop).

CAFE uses peer review to filter the most insightful ideas and Principal Component Analysis (PCA) to group participants based on their answers in the quantitative assessment phase. Insights from the statistical analysis are displayed to facilitate the discussion and incentivize participation. For example, median scores from the quantitative assessment and anonymous textual responses from the ideation phase are displayed immediately.

We present a case study on use of CAFE to assess family planning and reproductive health (FPRH) trainings at three health centers in Uganda in June 2014. Data were collected through CAFE from 137 women, including demographics, quantitative assessments of FPRH knowledge, and qualitative feedback on how to improve FPRH education strategies. Our data suggest that PCA can extract insights for further cross-sectional demographic analysis. For example, CAFE allowed us to identify that more tailored and locally relevant trainings are necessary to increase efficacy of FPRH outreach at each center. Our preliminary results suggest CAFE can collect reliable, direct and timely feedback on the efficacy of development programs.

M-CAFE: Managing MOOC Student Feedback with Collaborative Filtering

Zhou, M., Cliff, A., Huang, A., Krishnan, S., Nonnecke, B., Uchino, K., Joseph, S., Fox, A., Goldberg, K. (2015, March).  Proceedings of the 2nd ACM Conference on Learning At Scale (L@S 2015), Vancouver, BC, Canada


Ongoing student feedback on course content and assignments can be valuable for MOOC instructors in the absence of face-to-face-interaction. To collect ongoing feedback and scalably identify valuable suggestions, we built the MOOC Collaborative Assessment and Feedback Engine (M-CAFE). This mobile platform allows MOOC students to numerically assess the course, their own performance, and provide textual suggestions about how the course could be improved on a weekly basis. M-CAFE allows students to visualize how they compare with their peers and read and evaluate what others have suggested, providing peer-to-peer collaborative filtering. We evaluate MCAFE based on data from two EdX MOOCs.

Comparing three online civic engagement platforms using the “Spectrum of Public Participation” framework

Nelimarkka, M., Nonnecke, B., Krishnan, S., Aitamurto, A., Huang, A., Newsom, G., Gregory, C., Patel, J., Catterson, D., Crittenden, C., Scott, J., Garland, C., & Goldberg, G. (Sept. 2014). Paper presented at University of Oxford: Internet, Policy, and Politics Conference on Crowdsourcing for Politics and Policy (IPP2014). Oxford, UK. 

Online civic engagement platforms accessed via desktops or mobile devices can provide new opportunities for the public to express views and insights, consider the views of others, assist in identifying innovative ideas and new approaches to public policy issues, and directly engage with elected leaders. Existing platforms vary widely in their approaches to: assessment, engagement, ideation, evaluation, and deliberation.

We consider three online platforms: the Living Voters Guide, including its earlier iterations and Reflect; the Open Town Hall; and the California Report Card. We compare them using the International Association of Public Participation’s “Spectrum of Public Participation” framework. Using a 10-point scale, we evaluate the user interface of each platform in terms of how well it supports the Spectrum’s levels of civic engagement (inform, consult, involve, collaborate, and empower). Results suggest how user interface design affects civic engagement and suggest opportunities for future work.

Moving beyond dichotomy: Comparing composite telecommunications regulatory governance indices

Martin, B. & Jayakar, K. (2013). Telecommunications Policy, 37(9), 691-701.


Despite an apparent consensus about the importance of the quality of telecommunications regulatory agencies, there is no agreement among researchers about how to measure it. While dichotomous coding of de jure independence has served as a proxy to measure whether an agency’s regulatory governance is transparent, non-arbitrary, and free from political influence, we view measuring multiple components of regulatory governance including incorporating measures of regulatory independence into a composite index as providing a more nuanced understanding of facilitating or inhibiting factors. This paper compares composite telecommunications regulatory independence indices and regulatory governance indices available in the literature in order to construct more parsimonious indices. Using a methodology labeled “qualitative meta-synthesis” based on synthesizing previously published indices, we construct six different indices using combinations of 32 different variables, and different weights. Data from the OECD database are used to re-create the five original indices from the literature as well as our own six composites. Some of these indices (original and composites) were found to be negatively correlated with independent measures of regulatory governance such as the World Bank’s Government Effectiveness Index and the Rule of Law Index; this may be attributable to the fact that countries, and especially those with poor overall governance standards, may need to put in place stronger telecommunications governance institutions in order to attract telecommunications investment to the country. The analysis suggests that a parsimonious index of as few as seven variables is capable of measuring the quality of telecommunications governance in a country, at the same time making the selection of variables and their weighting in the index more systematic than in previously available indices.

Influences of media on social movements: Problematizing hyperbolic inferences about impacts

Olorunnisola, A. & Martin, B. (2013). Telematics & Informatics, 30(3), 275-288.

– Most Downloaded Telematics & Informatics Article for 90 days (As of July 2013)


Pronouncements about the value of information and communication technology (ICT)  (hereafter traditional, new, and social media) to social movements – hyperbolic in popular media references to new and social media (e.g., Facebook revolution, Twitter revolution, etc.) – invite scholarly inquiries that critically assess the implications of these assumptions for African countries. Sensing the tendency toward technological determinism, a position which Castells warns is fraught with failure to recognize complex interactions between society and technology; authors examined popular press vis-à-vis scholarly assumptions about the value of media during social movements. Questions that critically analyze the roles and power of old versus new media in social movements should be posed particularly about 21st century iterations with citizens increasingly doubling as creators and disseminators of news and information. For example: To what extent do various media comparatively facilitate or constrain activists in social movements? How have new ICTs assisted citizen activists in circumventing the power and reach of traditional media? How have the roles of traditional versus new media in social movements been framed in the popular press and academic journals? What contextual factors (e.g., communal networks; third-party- and foreign-interventions, digital divide, etc) may be accountable for the take-off and successes of social movements? In a continent fraught with cultural, political, and socio-economic divisions of historic proportions, authors critically assessed cases across Africa of variegated employment of old (i.e., radio, newspaper, television) and new media platforms (i.e., Facebook, Twitter, mobile telephone text messaging) by four social movements spanning 35 years. Assessments underscore citizen empowerment and multiplier capabilities of new media but affirm the value of contextual factors that minimize hyperbolic assumptions about the contribution of new media to the formation and progression of social movements.

Voicing dissent from Kenya to Egypt: How new ICTs can either be “liberation” or “repression” technologies

Martin, B. & Olorunnisola, A. (2013). In A. Olorunnisola & A. Douai (Eds.), New media influences on social and political change in Africa. Hershey, Pennsylvania: IGI Global.


Participants in varying but recent citizen-led social movements in Kenya, Iran, Tunisia, and Egypt found new voices by employing new ICTs. In some cases new ICTs were used to mobilize others to join and/or to encourage use of violence against other ethnicities. In nearly all cases the combined use of new ICTs kept the world informed of developments as ensuing protests progressed. In most cases, the use of new ICTs as alternative media motivated international actors’ intervention in averting or resolving ensuing crises. Foregoing engagements have also induced state actions such as appropriation of Internet and mobile phone SMS for counter-protest message dissemination and/or termination of citizens’ access. Against the background of the sociology and politics of social movements and a focus on the protests in Kenya and Egypt, this paper broaches critical questions about recent social movements and processes: To what extent have the uses of new ICTs served as alternative platforms for positive citizens’ communication? When is use of new ICTs convertible into “weapons of mass destruction”? When does state repression or take-over of ICTs constitute security measures and when is such action censorship? In the process, the paper appraises the roles of local and international third parties to the engagement while underscoring conceptual definitions whose usage in studies of this kind should be conscientiously employed. Authors offer suggestions for future investigations.

Regulatory Governance in African telecommunications: Testing the Resource Curse Hypothesis

Jayakar, K. & Martin, B. (2012). Telecommunications Policy, 36(9), 691-703.


This paper examines regulatory governance in the context of African telecommunications. Though there is already a substantial literature devoted to the regulatory practices in developing countries, it generally conceptualizes the quality of regulation as an exogenous policy variable that affects the performance of the telecommunications sector or treats it as a normative recommendation to improve performance.

In this paper, the quality of governance is approached from a different perspective, investigating how it might be related to the structural economic characteristics of the developing country. It applies to telecommunications the theory known in development economics as the paradox of plenty or the resource curse, by which countries richly endowed with natural resources lag behind in the development of governance institutions. An inverse correlation is thus expected between natural resource dependence and the quality of regulatory governance, both generally in the economy and specifically in telecommunications.

To measure regulatory governance, both previously-published general indices, as well as sector-specific telecommunications governance indices developed for this paper are used. These governance index values were then compared to several measures of natural resource dependency in 53 African countries. The results indicate that, while governance standards in the overall economy behaved as expected by theory, the sector-specific telecommunications governance did not. Several speculations are offered why telecommunications governance acted contrary to resource dependency theory.

Hitting or Missing African UAS Objectives? Evaluating the World Bank’s Universal Access and Service (UAS) Policy Recommendations for Developing Countries

Martin, B. (2012). Communications and Strategies, 86(2)


Most African countries have historically lagged in telecommunications development. Recent modifications to universal access and service (UAS) policies have helped to develop an environment capable of unprecedented mobile telecommunications growth. The World Bank Information for Development Program (InfoDev) developed “The ICT Regulation Toolkit” (ICTRT) to serve as a “best practices” guide for ICT policymakers in developing countries. The ICTRT recommends that developing countries include roll-out obligations in telecommunications licenses, develop a UAS fund to manage disbursements of subsidies needed for investment, focus on commercial solutions rather than UAS policy solutions, take advantage of new technological efficiencies to achieve UAS objectives, include social factors in the design of UAS policies, and align UAS objectives with national social programs to foster co-achievement of long-term development goals. By conducting a critical analysis of Uganda’s adoption of the InfoDev ICTRT recommendations, this paper contextualizes the beneficial and detrimental impacts of policy transfer. The paper concludes with supply and demand-side UAS recommendations that should be incorporated into the InfoDev ICTRT.

Mobile Phones and Rural Livelihoods: Diffusion, Uses, and Perceived Impacts Among Farmers in Rural Uganda

Martin, B. & Abbott, E. (2011). Information Technologies & International Development, 7(4), 17-34.


To successfully use mobile phones to aid development efforts, understanding the impact of the social structure on mobile phone adoption, uses, perceived impacts, and reinvention of uses is invaluable. Interviews were conducted with 90 mobile phone-owning holders of small- to medium-sized farms—50 women and 40 men—actively involved in agricultural development-based farm groups in Kamuli District, Uganda. Respondents indicated use of the mobile phone for coordinating access to agricultural inputs, market information, to monitor financial transactions, and to consult with agricultural experts. Over time, the number and variety of agricultural uses increased among all users, indicating that adoption occurs for a few key purposes, but that uses will be added or reinvented to fit changing needs. This study identified a number of unique uses, including storing local market trends in the calendar, using the speakerphone function for group consultation with agricultural experts, and taking photos of agricultural demonstrations.