Getting started: For the SNT team

This section is for members of the Subnational Tailoring (SNT) team who are leading or closely engaged with data-driven analysis. It outlines your responsibilities in the SNT process, how to engage effectively with analysts, and key areas to supervise during the analysis lifecycle.

Key responsibilities related to analysis

The SNT team should expect to:

• Provide context to the analysis, set its aim and objectives, and identify the key questions
• Ensure that there is an analysis plan to respond to the key questions with clear, useful outputs
• Take an active role in the analysis process
• Select the analysis team
• Identify appropriate data sources for SNT needs
• Decide on the operational unit for analysis
• Act as central data coordinator by extracting and/or assembling data, then sharing with the analysis team
• Respond to analysis team questions on data, including getting answers and clarifications from stakeholders outside the SNT team such as district officials as needed
• Interact frequently with the analysis team to give direction and feedback during active analysis periods
• Understand the analysis
• Contribute expertise to strengthening analysis approaches and validating results
• Decide which data sources and analysis products will be put to which use

The SNT team may find it helpful to:

• Request an analysis plan in advance from the analysis team
• Request additional explanations from the analysis team
• Request technical training from the analysis team

Recruiting an analysis team

Multiple skill sets are required for SNT analysis. Some or all of these skills may exist already in the national malaria program. Usually these skills do not exist all in the same individual, and multiple analysts are needed to carry out all the analysis tasks of SNT. The minimum set of skills required for a full SNT analysis exercise generally include:

• Data collection and management
• Data visualization
• Basic descriptive and statistical analysis
• Geostatistical analysis
• Mathematical modeling
• Health economics
• Costing and budget management

Very few individuals have all of these skills, so the analysis team should be built to complement existing capacity in the NMP. Whether internal or external, analysts should be familiar with SNT goals and ready to take instruction from the SNT team.

If an external analysis team will be recruited to support the SNT process by complementing existing capacities in the national malaria program, the recruited team should not only together possess all the technical capacities, but also be familiar with the concepts of SNT such that they know what analyses to conduct. The analysis team must understand that they are the hands executing the tasks set to them by the SNT team, which retains all decision-making power.

Working with the analysis team

As the governing body of the SNT process, the SNT team is responsible for ensuring data and methods are appropriate and that results are valid. This means that the SNT team must take a strong supervisory role to the analysis team. The SNT team must feel comfortable asking the analysis team for explanations, requesting changes, and making decisions on validity and use of analysis results.

The SNT team should request that analysts keep good records of their work. This may include storing all their process and results details through a growing PowerPoint slide deck (or other living document), which is shared with the SNT team at each update. The living document should also include records of discussions with the SNT team and their conclusions, such that the complete record of analysis exists in a single place. This record, while extensive, should explain in a clear and logical way what was done, what was decided, and why. Analysts should also keep minutes of SNT team discussions pertaining to their work and disseminate minutes after meetings along with clear action items and assignees.

The SNT team should feel free to invite experts to meetings when their areas of expertise are being discussed (for example, invite entomologists when vector control and entomological data are reviewed and presented), so that the analysis process is as validated and appropriate for decision-making as possible.

Checklist of analysis items for the SNT team’s review

The questions that the SNT team should ask of their analysts vary for each step of the SNT process. At any given step in the analysis, the SNT team generally provides guidance, facilitates data access and proper interpretation, validates outputs, and translates evidence generated in the course of the SNT exercise into decisions:

• Guidance: SNT team should expect to review and validate the analysis plan or proposed approach, and provide key decisions that determine the course of analysis
• Data source: SNT team should expect to facilitate sharing of a validated and approved dataset with the analysis team
• Consultation: SNT team should expect the analysis team to have detailed questions for the SNT team
• Validation: SNT team should expect to review and discuss analytical outputs regularly and validate the results for decision-making
• Translation: SNT team should expect to make decisions based on the evidence generated through the SNT process

The different steps of the analysis process included in this library include specific questions or areas of review that the SNT team should expect to be raised during the SNT process. These should be taken as a starting point, not as a comprehensive list of what needs to be discussed with the SNT team. The list below provides a summary of these considerations:

Warning

This list is partial and items will be added as more pages of the code library are completed.

Operational unit for decision-making

• Guidance: Before any analysis begins, the SNT team must decide on the operational unit for decision-making.
• Guidance: The SNT team should indicate which source (shapefile, master facility list, DHIS2, etc) to use as the official version of admin unit names.
• Data source: The analysis team should use official national shapefiles at this operational unit level, and larger, for analysis.
• Consultation: If the official shapefile has issues such as overlapping admin units, inexact duplicates made for planning or other purposes, or other problems, the SNT team should be consulted for guidance on which method of resolution is appropriate.
• Validation: Visualized shapefiles should be reviewed and validated by the SNT team at the beginning of SNT.

Health facility lists

• Guidance: The SNT team should provide guidance on local naming conventions and abbreviations used for health facility names.
• Guidance: If there are differences in facility names between the master health facility list and the list included in the routine surveillance database (DHIS2 or other), the SNT team should be consulted for guidance on how to resolve these differences. This may include validating potential fuzzy name matches, resolving differences in admin unit assignments, or deciding how to handle facilities that appear in one list but not the other.
• Consultation: If a facility coordinates dataset has missing names or coordinates, duplicate facility names with different coordinates, coordinates outside the facility’s admin unit boundaries, or other issues, the SNT team or master facility list focal point should be consulted for clarifications.
• Consultation: The method used to define facility activity status should be discussed and validated with the SNT team. In some cases, a Health Facility Master List may be appropriate; in others, indicator-based definitions may be more reliable. The final approach should reflect how malaria services are delivered and reported within the national system.

Routine epidemiological surveillance data

Preparing routine data for analysis

• Data source: The SNT team (for example, the NMP data manager or HMIS representative) should share a valid extraction of routine surveillance data, for example from DHIS2 if that is the country’s HMIS platform, with the analysis team, along with a data dictionary.
• Guidance: The analysis team should work closely with the HMIS focal person on the SNT team to tunderstand data element names, confirm variable definitions, ensure that all necessary data fields are extracted, and that subsequent calculations are performed correctly. For example:
  • Are admissions included in outpatients?
  • Are data on pregnant women included in the data for adults?
  • Is there double counting between RDT and microscopy results? If yes, is it appropriate to only use RDT results?
  • Are data from the private sector included here? If so, what percentage of the private sector reports into DHIS2?
  • Are data from commmunity health workers included in their assigned health facility data or are data separate?
  • Have any variables been included or adapted throughout the years? If so, how should they be treated throughout the time series?
  • And others.
• Consultation: The SNT team should expect to be consulted to provide context around data practice. For example:
  • If certain health facilities are sentinel data collection points for malaria deaths, there may be more facilities reporting deaths than malaria hospitalizations. The SNT team should explain such situations to the analysis team.
  • Reporting practices around ANC, IPTp, and other multiple-dose routine interventions, whether they are reported based on (gestational) age of the patient or by the dose number the patient has received.
  • Are zeroes are entered or are data fields are left blank when they are zero? Has this practice changed throughout the time series?
  • And others.
• Guidance: The SNT team should confirm which specific data elements from DHIS2 should be summed to obtain the correct total. If a total also exists as a variable in DHIS2 and is not equal to the summed data elements, the SNT team should provide guidance to the analysis team on how to manage incoherent totals.
• Guidance: The SNT team should indicate which calculation method is appropriate when the presumed cases column is not already included.
• Guidance: If the DHIS2 dataset contains duplicate health facility-month records with different data, these must be resolved before analysis is possible SNT team should provide guidance to the analysis team on how to manage these duplicates.
• Guidance: The SNT team should indicate which facility types are inpatient vs outpatient and inform the analysis team which types are expected to report which indicators. This information helps ensure that calculated reporting rates of indicators are relevant.

Quality control of routine data

• Consultation: The analysis team should not remove or correct outliers without consulting the SNT team. Individual outliers should be investigated with an SNT team focal person who should provide context, reaching out to district focal people if needed, to understand if the outlier is an error or if it instead reflects actual reality at the health facility.
• Validation: The analysis team should validate the outlier detection method with the SNT team: the SNT team should review outlier detection results and select the method (if any) where observations categorized as outliers are most likely true outliers. If no method is performing well, the SNT team should guide the analysis team in how to proceed.
• Consultation: The SNT team should explain clinical case management and referral practices to the analysis team so that relevant data coherency checks can be identified, data elements are correctly used, and their results properly interpreted. If practice has changed during the analysis time period, the SNT team should indicate when and/or where these changes occurred so that years where a certain coherency check is irrelevant can be excluded.
• Validation: Results of coherency checks should be discussed with the SNT team to review patterns and validate findings to ensure conclusions from coherency review are aligned with program practices and transmission realities, regardless of when the coherency check is performed (before or after outlier correction and/or imputation of missing data).
• Translation: Tables of incoherent records by health facility-month should be shared with the SNT team for follow-up.
• Guidance: The SNT team should confirm which method, if any, should be used to correct outliers, and which method, if any, should be used to impute missing data. For both of these tasks, the analysis team should present to the SNT team the extent and nature of the data issues that might require correction, and the resulting impact of the proposed correction method.
• Translation: The SNT team should ensure that there is a plan to follow-up with the core surveillance focal points in the country to integrate all the corrections applied to the database, and make a plan to review, follow-up and minimize all the issues identified with the database in the future.

Routine data reporting rates

• Guidance: If there is no database that tracks monthly active or inactive status of health facilities, the SNT team should provide guidance to the analysis team on how monthly reporting status should be used to determine active/inactive status. For example, after 6 months of non-reporting of all indicators, the facility should be considered inactive.
• Consultation: When calculating reporting rates per indicator, the SNT team should be consulted on which types of facilities should be considered as expected to report that indicator. For example, when calculating reporting rates for hospital admissions, it may be appropriate to exclude outpatient clinics.
• Guidance: When calculating reporting rates to use to estimate unreported cases, the SNT team should inform the analysis team whether to use weighted or unweighted reporting rates. It may be helpful for the analysis team to prepare both versions so they can be reviewed side by side.

Population denominators

• Data source: Population data must be sourced from the official dataset designated by the SNT team.
• Guidance: If population estimates (such as WorldPop population rasters) are used in analysis, for example to estimate the proportion of the population living >5km from the nearest health facility, or to generate population-weighted averages when aggregating rasters to admin units, sources of estimated data must be approved by the SNT team. The analysis team should review the estimated population data and present to the SNT team for validation before use. The SNT team should be on the lookout for known local population patterns that may not be adequately reflected in the population estimate.
• Data source: If population growth rates are to be applied by the analysis team, the SNT team is responsible for sourcing and approving any growth parameters.

Climate data

• Data source: For climate data, the SNT team should decide whether to use data from national meteorological or weather stations, or if global sources should be used instead. If national data are to be used, the SNT team should facilitate access and be available to answer any clarification questions the analysis team may have such that data is used properly.
• Validation: Whatever the data source, the SNT team should review and validate it before use in later analyses. If climate data for analysis is coming from global rasters, the SNT team may still want to compare it to whatever local meteorological data is available.

Estimating treatment-seeking rate

• Guidance: The SNT team should inform the analysis team which sources of care to include for the public and private sectors, as definitions may vary by country.
• Guidance: When estimating treatment-seeking rates for malaria from household survey data, it is possible to use as denominator all recent fevers or only recent fevers that were also positive by RDT at time of the survey. Each method has strengths and weaknesses. The SNT team should provide guidance on which method is to be used. The SNT team may request the analysis team to try multiple methods so that this decision is made after review and discussion of both results.
• Guidance: If an estimate of treatment-seeking rate is needed for years without surveys, the SNT team should specify which interpolation method they prefer.
• Validation: Finally, after treatment-seeking rates are generated, the SNT team should review the output maps to ensure that the care-seeking estimates align with local realities and programmatic knowledge. If the SNT team finds that a map or region has treatment-seeking estimates that are unrealistic, they should provide guidance to the analysis team on how to proceed.

When working with estimates from geospatial models

• Data source: The SNT team is responsible for deciding whether to use modeled estimates for an indicator, for example parasite prevalence. If a modeled estimate is to be used, the SNT team should indicate where it is coming from: whether the estimate should be generated by the analysis team or a pre-existing estimate should be used.
• Consultation: Regardless of the source of the estimate, the SNT team should expect to be informed of the modeling approach used to generate those estimates, including covariates used, their data sources, and key assumptions. If the estimate is being generated by the analysis team, the SNT team should be consulted to inform the modeling approach.
• Validation: All geospatial modeled outputs, whether pre-existing or generated by the analysis team as part of the current SNT, must be reviewed and validated by the SNT team before they are used in subsequent analyses.

Mortality data

• Guidance: The SNT team must decide, most likely after reviewing all options, what source of malaria mortality or proxy malaria mortality data they which to use, if any, for decision-making. This may include all-cause under-5 mortality estimates from surveys or rasters, malaria mortality from routine data, or another source.
• Consultation: Thresholds for continuous indicators should be discussed with the SNT team and outputs shared to support transparent and reproducible threshold decisions.
• Validation: Geospatial model mortality estimates should be discussed and validated before they are used in any subsequent analysis, for example against other national sources of mortality data.

Presenting results

• Guidance: The SNT team must ensure that any analysis results (maps, plots, table, etc.) clearly present all information needed for downstream use in decision-making. This may include giving the analysis team feedback so that the presentation of these results is improved.
• Guidance: The SNT team should inform the analysis team what thresholds (bins) are appropriate when mapping quantitative data. A single indicator may need different thresholds for different purposes.

Tip

This checklist is meant to help the SNT team engage with analysis, not replace deep thinking. Always consider what’s appropriate for your context and don’t hesitate to push for improvements.