The Use of IVR to Support Monitoring and Evaluation of Media Interventions: A Case Study of the VOTO System in Rwanda

This report "explore[s] the potential that Interactive Voice Response (IVR) systems have as both a distribution and monitoring and evaluation (M&E) tool, noting the limitations the current state of IVR technology imposes."

These findings of the United Stated Institute of Peace (USIP)-funded grant "Amplifying Peace: Testing Mobile Interaction in Rwanda" describe the use of mobile IVR technology for those engaged in peacebuilding. The research, implemented by the Center for Advanced Research in Global Communication (CARGC) investigated the "usefulness of the platform for information dissemination and information collection, and evaluated the platform as a tool to improve monitoring and evaluation [M&E]."

According to the report, IVR can address language and literacy barriers because it is an audio tool and can be delivered through mobile phones. It also "provides the possibility of collecting large quantities of qualitative and quantitative data for very little cost, by asking questions of mobile phone users rather than having to interview respondents face-to-face. It offers benefits beyond SMS data as it can collect voice data.... This pilot study used IVR both to distribute content and to collect data from listeners, through a survey that listeners heard after the end of the program."

The study focused on Radio La Benevolencija (RLB), an infrastructure of radio programmes for peacebuilding in the Great Lakes region of Africa. It sought to demonstrate how mobile phone technologies can "1) amplify the reach of radio programs; and 2) streamline (M&E) of radio programs" in Rwanda. RLB created a new segment of its radio drama New Dawn for this research  and chose to use the platform VOTO, offering an interactive SMS and voice call platform, which is Internet based, "requiring no software installation or equipment purchasing (save a mobile phone to connect to the platform)". The information is stored on a cloud server and is downloadable in excel format from the server over an interactive client interface. The system was developed to make calls free to the caller - both a toll-free number on one network and a callback system on another were needed.

A focus group discussion (FGD) was conducted to troubleshoot problems; then, the New Dawn content was promoted using radio and paper flyers. "Token prizes (an RLB T-shirt or umbrella) were offered to the first 100 listeners."

 Findings showed that IVR has high potential to be used both as a distribution tool and as an M&E tool:

• IVR technologies have the potential to significantly increase the reach of radio programmes.
• Feedback from focus groups suggests that "people are not only willing but even excited to be able to provide program feedback through the IVR system."
• The data collected was useful for measuring audience demographics, receiving feedback on the programmes, and understanding programme impact. 
• "The system successfully collected both quantitative and qualitative data." 
• "The system was also easy for the public to use, and so could be implemented on a wide scale with no end-user training required." 

However, limitations included: "The technical limitations of the VOTO system and the resulting cost of gathering the data turned out to be too great to warrant a recommendation of the platform for conducting joint quantitative and qualitative M&E at this time, at least for organizations without in-house expertise in computer programming." Dropped calls were an issue; unfinished surveys thought to be finished caused difficult-to-manage data sets and expensive "data cleaning". Call costs were high for the organisation.

The study concludes that using an IVR system to collect qualitative and quantitative data is possible, especially for improving the project and understanding impact, even given limitations of survey length. It highlights respondent eagerness to leave open-ended feedback and suggestions. It recommends:

• a cost reduction arrangement with a local telecomm operator; direct communication with the IVR service platform managers;
• a question about the group size of listeners per call;
• short media content as calls are often dropped;
• an opportunity to skip directly to the survey at the start of the call;
• formative research at the outset of the project to understand who has mobile phones and if they can use the IVR system easily; and
• publication and circulation of any findings on IVR usage for content dissemination and M&E.