By Megan Noel, SC4CCM M&E Technical Advisor
It’s a great measure of success when an mHealth system like cStock gains Ministry of Health support to scale up nationwide. This happened for cStock, the community health worker (CHW)-driven mobile and web-based LMIS developed by the SC4CCM project in Malawi. But even systems with full stakeholder support need to be continuously assessed and managed for data quality, in order to remain relevant and viable for and by users.
Two sides of the data quality equation for cStock are user inputs and system outputs. The project team worked hard to ensure correct system outputs, and even hoped its robust time trend data could replace an endline survey. However, as cStock became more established in Malawi, supply chain professionals there described varying levels of trust in and willingness to use data generated by the system. The project needed a better understanding of data quality related to user inputs, so SC4CCM embarked on a Routine Data Quality Assessment (RDQA) of cStock at the end of 2013, adapting an existing tool developed by MEASURE Evaluation.
Because cStock data is generated at the community level, the project adapted the RDQA tool for assessors/auditors to visit a sample of CHWs (called health surveillance assistants or HSAs in Malawi). Several data collection teams of two people each deployed to six districts (three experienced with cStock, and three recently trained), armed with recent cStock report print outs. For each HSA, stock levels were counted and adjusted using register entries to find an accurate reporting day figure. This figure was compared with the figure actually reported by the HSA (using cStock and/or an alternate paper-based reporting system), yielding a verification factor or measure of quality per HSA, product, and reporting method (cStock or paper). Verification factors were analyzed for the full group of six districts, and separately for the three experienced and unexperienced districts.
Results from the RDQA showed very similar results across the original and new districts, and for cStock compared to the paper-based system. In terms of user input quality, about a third of records sent to cStock were 100% accurate, another third had minor data quality issues, and the final third had major quality issues. The RDQA activity revealed that stock counts at the end of the month were not regularly performed due to logistical challenges faced by HSAs to easily access their medicines. Recommendations were made based on the findings, and results shared with stakeholders and system users.
mHealth systems like cStock can benefit greatly from conducting an RDQA or other data quality assessment in the early or mid stages of system design and scale up. The MEASURE Evaluation tool is highly adaptable and is a useful model for assessing all kinds of information systems. Moreover, the RDQA helped identify strategies for quality improvement, and the results satisfied the appetite of stakeholders and users for transparency and credibility around the system. I found the activity particularly useful because it showed that the quality of data from cStock compares with the traditional paper system, and while there are some specific things that can be done to improve quality, supply chain staff can count on data from cStock to make decisions about resupply of medicines for CHWs.
Read more about SC4CCM’s RDQA.
