A. PURPOSE OF THE MEETING

The Consultative Expert Meeting on Rapid Assessment Procedures and Their Application to Population Programmes was held at the headquarters of the United Nations Population Fund (UNFPA), New York, 6-8 December 1995.

The purpose of the meeting was as follows:

• To review the range of rapid assessment procedures and their state of practical development;

• To evaluate the appropriateness of the use of rapid assessment procedures at various stages in the programme/project cycle;

• To examine prospects for using rapid assessment procedures in reproductive health programmes, including family planning and population programme efforts; and

• To examine the practical implications of quantitative and qualitative rapid assessment applications for training requirements, staff qualifications, documentation procedures and dissemination strategies.

The meeting's agenda appears as Annex A, the list of participants is attached as Annex B, the composition of working groups is attached as Annex C, and a list of technical papers appears as Annex D.

Mr. M. Nizamuddin, Deputy Director, Technical and Evaluation Division, UNFPA, opened the meeting. Mr. Jyoti S. Singh, Deputy Executive Director (Programme), UNFPA, set out in his opening address (Annex E) the work to be undertaken at the meeting.

Mr. Singh noted that the International Conference on Population and Development (ICPD) Programme of Action identifies, inter alia, the need for comprehensive, reliable and comparable data, both qualitative and quantitative. Such data would establish linkages among various sectors involved in population and development programmes: population, education, poverty, family well-being, environment and development. Mr. Singh also drew attention to the importance of disaggregating data by gender.

A central principle of the ICPD Programme of Action is the promotion of human well-being through informed choice. The goals of the Programme of Action include the following:

• Making reproductive health accessible to all individuals of appropriate ages through the primary health-care system;

• Narrowing the gap between average infant and child mortality rates in developing and developed regions of the world;

• Reducing maternal mortality;

• Providing universal access to primary or equivalent education, especially for girls, no later than by the year 2015; and

• Ensuring the full participation of women in sustainable development.

B. BACKGROUND

The goal of rapid assessment exercises is to generate programmatically useful information. This may be done through various methods and techniques. In reproductive health, the application of this approach appears appropriate to foster better understanding of the needs for reproductive health services. These include the specific needs of select groups (adolescents, for example) and quality-of-care issues that affect the use of family planning and reproductive health services.

In April 1995, a Consultative Meeting identified priorities for UNFPA activities in which rapid assessment procedures have potential value. These were:

• Designing and strengthening an information system for planning, implementing and monitoring reproductive health programmes;

• Developing and strengthening approaches and methods to link and analyse reproductive health data;

• Developing national and subnational information bases and enhancing national capabilities in the analysis of information; and

• Improving data utilization by promoting and encouraging their use by health planners and managers.

It was anticipated that these steps might ensure the adoption and continued use of reproductive health information bases in a wider international context.

C. TYPES OF RAPID ASSESSMENTS

One objective of the meeting was to determine the extent to which rapid assessment procedures might be adopted to facilitate reaching ICPD goals by permitting the rapid compilation and analysis of relevant information for policy and programme purposes. As a prelude to answering this question, the meeting examined various approaches to rapid assessments, such as community diagnosis, rapid rural appraisals, management information systems methods, needs assessments, situational analyses, rapid anthropological assessments, rapid epidemiological assessments and so on. The discussion in the plenary session focused initially on methodological issues, including quantitative versus qualitative techniques and probability versus non-probability sampling, and on what combinations of techniques are appropriate.

Introductory papers concerning the collection of qualitative and quantitative data were presented, followed by a series of presentations on country and programme experiences in the use of rapid assessment procedures in family planning, reproductive health and other public health programmes.

Participants stressed the importance of triangulation to enhance the reliability and generalizability of findings. Triangulation aims at maximizing data validity and reliability by the use of multiple methods, often also with the involvement of research teams with individuals having different research skills.

Participants also remarked on the importance of involving end-users of data from the beginning of the process.

The meeting reached consensus that rapid assessments might include any relatively quick, low-cost method. Rapid assessment procedures should usually take 6-12 weeks from implementation to the availability of results, with emphasis on the speed with which results are fed back into policy-making. Rapid assessment procedures may provide decision makers with explanatory information related to sociocultural behaviour, although the kind of information to be collected would depend on organizational or programmatic questions and on the unit of analysis (e.g., system, community, facility or client).

Participants agreed that, for UNFPA's purposes, the essential elements were that:

• The research be conducted in a focused, efficient and timely fashion, collecting only the information required to meet the needs identified by the agency commissioning the research; and

• A range of qualitative and quantitative methods be used, appropriate to the research questions and objectives of the study.

Participants noted that quantitative and qualitative approaches serve different purposes and should be regarded as complementary rather than competing. They recognized that problems could arise in reconciling information obtained through different methods, given specialized professional backgrounds and the compartmentalization of fields of expertise. Participants emphasized the value of using interdisciplinary teams to conduct rapid assessments.

Rapid assessment procedures should be viewed as a technique for generating "quick" data for planning and policy-making rather than for building databases. They should be used to increase understanding of community situations and cultural sensitivities and to provide information for the development of information, education and communication (IEC) strategies. Discussions noted the significance to population and reproductive health programmes of disaggregating data by age, ethnicity and cultural group.

Questions that arose in the discussion included the following:

• What applications are relevant to population and development projects and programmes, especially at the field level?

• How can rapid assessment procedures be used for programme planning and policy formulation?

• How can rapid assessment procedures be used to improve delivery of reproductive health services; to anticipate and increase programme capacity to deal with new concerns (e.g., the introduction of new technology); to ensure the quality of service over time; and to the address needs of adolescents and other groups (e.g., men, specific minority groups and refugee communities)?

• To what extent are community organization and mobilization prerequisites to effective rapid assessment procedures and "ownership" of results?

• How can the results of rapid assessment procedures best be disseminated and utilized?

D. METHODOLOGICAL CONSIDERATIONS

Modified Cluster Sampling (MCS) was suggested as an alternative to the cluster survey methodology used in the Expanded Programme on Immunization (EPI) of the World Health Organization (WHO).1/ The MCS method would ameliorate the problem of statistical bias associated with the survey method.

Another method discussed was the multiple indicator cluster surveys used by the United Nations Children's Fund (UNICEF) to monitor mid-decade goals.2/ It was suggested that the cost reduction and time saved by using such methods might be only marginal compared with those entailed in using standard statistical procedures, including surveys; moreover, the complexity and duration of such undertakings would be substantially unchanged. However, since rapidly collected quantitative data continue to be needed for some purposes, such surveys merit further attention.

UNICEF has used rapid assessment procedures as a technique for quick feedback, finding them useful not so much for designing projects but for identifying where corrections should be made in the mid-course implementation phase of projects. Rapid assessment procedures are now so popular and accepted within the organization that they are being widely used without a good understanding of the methodology and without adequate training for those conducting them. Attention to the questions of who should initiate, manage and use rapid assessment procedures will increase the value of their results. The importance of quality control at various stages, the need for simple guidelines on how to conduct rapid assessment procedures and a basic understanding of how projects can make good use of these procedures were emphasized.

Participants identified quality-of-care and quality improvement issues as key areas in which rapid assessment procedures could be important tools to improve the level of services. Such techniques as exit interviews and client flow analysis, although they would not provide statistically viable data, could provide a wealth of relevant information where only anecdotal evidence had been available previously.

The issue of the relationship of assessments to quality improvement was addressed in a presentation on the Client-Oriented Provider Efficient (COPE) self-assessment technique for improving family planning services, adopted by the Association for Voluntary Surgical Contraception (AVSC). To ensure quality, everyone involved with COPE studies must understand what quality entails. Participants added a cautionary note that it is not possible to ensure quality through "end-of-the-line" inspection. The presentation regarding experiences applying the COPE technique emphasized the importance of involving the community in improving quality, stressing that community organization and mobilization are prerequisites to effective rapid assessment procedures and to ownership of the study results.

E. NATIONAL EXPERIENCES

The discussion of national experiences drew attention to the benefits and disadvantages of rapid assessment procedures. The experience from Zambia in Rapid Evaluation Methodology (REM) highlighted the value of rapid assessments in providing quickly processed results to appraise health providers' performance and to analyse the views of those in the community about the services they received. Difficulties experienced in undertaking REM were related to inadequate time to collect the information, the intensive nature of questionnaire development and the pressure to complete analysis of the data. Issues of special importance included the needs for ensuring political commitment from the highest levels of the organization, adequate (and dedicated) budgets for this process, sufficient preparation and training, and recognition of the multidisciplinary nature of such approaches and of the importance of building REM into the health development process.

Experience in Pakistan pointed to the benefits of using rapid assessment procedures to obtain population-related data that would otherwise have been difficult to collect. Structured interviews included questions on the efficiency of the national population programme, characteristics of those who came for services, and the various influences of local leaders, cultural factors, political preferences, religious resistance and other social factors (for example, the influence of mothers-in-law). Such data provided a better, deeper perspective of the programme.

F. RAPID ASSESSMENTS AND INFORMATION, EDUCATION AND COMMUNICATION ACTIVITIES

There was wide agreement on the suitability of rapid assessment procedures for IEC activities. Rapid assessment procedures are an appropriate formative research tool, especially for analysing why people do not act even when they have the knowledge and understanding to make needed decisions. Rapid methods can be used to identify community needs, monitor the quality of interpersonal communication and counselling and develop IEC strategies. They can improve client-provider relationships, improve the quality of IEC materials and messages, provide leads to follow-up research and contribute to consciousness-raising efforts. The application of rapid assessment procedures in IEC also reinforces the empowerment process by emphasizing the listening part of the consultation process between service providers and beneficiaries. Challenges confronting IEC researchers include: convincing the gatekeepers and overcoming specialization (e.g., sociocultural research, data collection and IEC), negative perceptions of qualitative research and the underrepresentation of IEC in multidisciplinary research teams. Rapid assessment procedures constitute a promising methodology for the post-ICPD agenda to increase the community's and, especially, women's share in decision-making and access to information, to identify alternative delivery systems for underserved groups and to assess advocacy needs.

Because rapid assessment procedures entail compromises or trade-offs among time, quality and cost of the research, participants stressed the necessity of formulating clear aims in using rapid assessment procedures. The need to generate data quickly would be the impetus to undertake such efforts where "it is better to get some information quickly than none at all." It should be specified whether results will be used, for example, as a source of baseline data or as operational research, in a continued activity or a one-time-only activity.

UNFPA's objective was stated as an exploration of the application of rapid assessment procedures as a technique to supplement and complement, rather than replace, survey methodologies. The aim is to improve the quality of data in population programmes, not to generate from rapid assessment procedures large databases with data that might not be used productively, a problem common to standard data collection procedures, including surveys, censuses and some qualitative research.

The meeting formed two working groups, one to discuss a research agenda for methodological work and another to discuss the appropriateness of rapid assessment procedures at different stages of programmes and the practical implications of using rapid assessment procedures. The following chapters highlight their discussions.

A. SAMPLING AND OTHER METHODOLOGICAL CONSIDERATIONS

Working Group A examined various facets of the methodology of using rapid assessment procedures, including sampling methods and techniques for validating the information collected.

The group noted that qualitative rapid assessment procedures need to address issues of observer and courtesy bias, recall bias and the limitations of information collected through a case-study approach.

In the discussion on representativeness, the group concluded that rapid assessments seeking to generate data representative of a larger universe need to use probability samples 3/ and employ validity checks.

Discussion focused on the validity and generalizability of results from non-random methods. The EPI two-stage cluster sampling and its modified form, MCS, need to be tested along with alternatives such as "snowball sampling"4/, segment sampling 5/ and other purposive sampling methods.

Recommending that quantitative rapid assessment procedures be validated through field research, the group stressed that:

• Different research questions demand different methods. What, how many, and where questions require quantitative methods; how and why questions require qualitative methods. If the question is why, qualitative and quantitative approaches may generally be used without the requisite of probability samples. Rapid assessment methods can be used before surveys to generate hypotheses and after surveys to explore the why questions that arise from particular findings;

• Different programmes require different mixes of methods, and operational questions at different levels will also require different mixes of methods. A variety of qualitative methods has been designed to collect different kinds of information. Discussion of these differences needs to be explicit in any manual that is developed;

• Rapid assessment procedures require a mix of methods to permit the triangulation of data to enhance the reliability and validity of findings;

Iteration -- the continuity of the process of the data collection and interpretation, in which data are constantly analysed and new research questions generated in the light of new findings -- is needed at two levels. First, iteration is necessary within a rapid assessment procedure. When qualitative research is being conducted in a discrete setting -- a given community or facility -- a research team needs to discuss its work on a regular basis, such as at the end of each day, to plan the direction of the next day's enquiries. Second, iteration is required between qualitative and quantitative teams. For example, preliminary qualitative research would be required to develop or modify instruments designed to collect quantifiable data, whereas a review of quantitative findings would direct qualitative research enquiries. This would need to be well explained in a manual.

Rapid assessment procedures can be used in different settings to ensure variability within a population, but the full replication of such a method in different sites is unnecessary. For example, in a country with 16 provinces, focus groups and other qualitative methods might be used in four distinctive and discrete areas. The generalizability and/or applicability of findings could be established through a meeting of, for example, district health officers and NGO programme staff from other areas.

Rapid assessment procedures, although especially suited to the local community or facility level, can be adapted to other levels. For example, the use of key informant interviewing, reviews of current data, and meetings can provide baseline data for policy development.

B. TRAINING AND THE DEVELOPMENT OF GUIDELINES

The group considered training an important issue, especially for qualitative research methods, as the validity of results can be greatly affected by a researcher's skills. The group agreed that more work -- at minimum, a review of current approaches to training -- would be helpful in framing recommendations to potential users of rapid assessment procedures.

Short questionnaires and other methodological tools designed for rapid assessment procedures require pre-testing before field use to clarify their potential benefits and limitations. Guidelines for rapid assessment procedures adapted to family planning and reproductive and sexual health would be necessary, with examples of the range of methods, their use in the field, the presentation of results and the use of the results for policy development, planning and programme purposes.

The group argued that rapid assessment procedures need to be documented both in a users' manual and in reports, the latter to allow others to make independent assessments of the quality of data collected. A manual can explain the difference between rapid research (of any kind) and the rapid assessment procedure approach and might address the topic of using existing data in programme development and planning. Other topics that could be treated in separate publications include:

• Rapid assessment procedures in family planning services;

• Rapid assessment procedures for reproductive and sexual health;

• Quality of care for reproductive health; and

• Training for rapid assessment procedure studies.

C. FEEDBACK

Guidelines need to include feedback mechanisms to encourage the use of findings from rapid assessment procedures. Mechanisms need to be identified to ensure that the agency commissioning the study participates in the identification of research issues and in the discussion of findings. The agency also needs to develop the means by which findings are fed back into the institution, through strategic planning, changes in operating procedures etc.

Community members should be involved in the collection of data, as appropriate, and in problem-solving in response to the study's findings. Participatory research methods, such as COPE, are ideal for this purpose.

The group recommended the following:

• Conducting repeat studies, as appropriate, using multiple methods of data collection, in multiple settings, at different times;

• Replicating methods and procedures to establish their reliability;

• Spelling out the limitations on rapid assessment methods and interpretations of findings;

• Preparing a manual or handbook to address a range of ethical issues, including confidentiality and community ownership or control of data; and

• Tailoring the approach to the aims of each rapid assessment procedure and including in the manual a range of alternatives that can be adopted or modified by individual agencies and researchers.

The group noted that inadequate use has been made of extant data. There is a need for a manual to discuss how data might be used and interpreted, particularly data from demographic and health surveys and national census material as well as facility-level fertility data and district-level family planning returns. The manual should also discuss the interpretation of such information as fertility rates and birth-order data and the potential use of this information to develop or enhance programmes. In addition, extant anthropological and other social and cultural information needs to be used.

The kind of institution conducting rapid assessment procedures will affect the type of study that can be undertaken, a relationship that needs further exploration.

The group concluded that:

• Clear plans for the use of data should be made when a rapid assessment procedure is commissioned;

• UNFPA should develop case-studies for the use of rapid assessment procedures, which might be generated by developing or testing a manual or guidelines for rapid assessment procedures; and

• It is unnecessary to invest large amounts of funding in testing methods or in conducting major multicountry field trials. However, some testing of tools and approaches would be essential as part of the preparation of a manual or set of manuals.

Working Group B discussed the appropriateness and practical implications of using rapid assessment procedures at different stages of population programmes and projects.

Unlike conventional knowledge, attitudes and practice (KAP) studies, rapid assessment procedures take into account contextual factors and reveal the complexity of community and individual decision-making in reproduction. Hence, rapid assessment procedures should be given appropriate emphasis in population programmes. Rapid assessment procedures may be especially suited for exploring sensitive issues in reproductive and sexual health, such as IEC research on unsafe abortion, adolescent sexuality, female genital mutilation, domestic violence and sexual abuse, commercial sex and emergency reproductive health needs. Rapid assessment procedures also facilitate monitoring the quality of reproductive health services.

Regarding the implications of increased use of rapid assessment procedures for population and reproductive health programmes, the group discussed a possible training strategy, the use of rapid assessment procedures for monitoring and evaluation, the analysis and reporting of results and the institutionalization of rapid assessment procedures in UNFPA.

A. TRAINING STRATEGY

Field office staff and national staff should be trained in rapid assessment procedures. All programme managers need some kind of orientation regarding the potential uses of rapid assessment procedures, although not necessarily training per se. A prerequisite for such training efforts would be UNFPA's development of a training plan at regional and national levels.

The training of trainers could be undertaken at the regional level, possibly in cooperation with regional Centres of Excellence, and the primary audience would seem to be the regional advisers of the Country Support Teams (CSTs). However, training at the country level should be undertaken as soon as possible. It would be best to gather participants from a variety of organizational backgrounds and academic disciplines.

At the country level, important audiences for training were identified as follows: managers of national programmes (e.g., ministries of health and other social sectors from central and district levels), heads of national research institutes and NGOs. Persons with previous experience in using rapid assessment procedures would seem to be best qualified for training. Available resources in the regions should be identified.

To overcome the possible skepticism of staff towards qualitative methods and resistance to changing habits, training should start with a participatory assessment of advantages and disadvantages of usual approaches and a discussion of alternatives. Training would explain where rapid assessment procedures are useful in the programmes and where they are not. For example, rapid assessment procedures are not recommended for disease prevalence or maternal mortality surveillance. Trainees should become knowledgeable about the techniques and benefits of triangulation, about the range of possible methods and about the comparative advantages of several specific methods in relation to the research questions asked. Training should discourage researchers from trying to answer too many research questions at once through rapid assessment procedures and from trying to collect more data than can be used.

Ideally, training should precede an actual data collection exercise and include the development of a viable plan. One-time training is not enough, however; follow-up and supervision are needed after the initial training.

After training in rapid assessment procedures assessment procedures, the following gains are expected:

• Staff will become better interviewers;

• They will be able to incorporate anthropological research components in any type of study; and

• They will better internalize monitoring procedures. Training in rapid assessment procedures should also help clarify differences between ideal behaviour and reality.

On the question of integrating rapid assessment procedures with other procedures or using them as a stand-alone methodology, the group agreed that rapid assessment procedures should be seen as a complement to other methods, not a substitute. Rapid assessment procedures are useful in generating new ideas before the design of quantitative studies.

B. MONITORING AND EVALUATION

Although not all operational research needs to be rapid, rapid assessment procedures can be used as a tool to answer some operational questions, as illustrated by such methods as COPE, situation analysis (The Population Council) or the "Village Monograph" approach of the Centre d'Etude et de Recherches sur les Populations Africaines et Asiatiques (CERPAA). Rapid assessment procedures can also be used to monitor the quality of existing information-gathering systems (e.g., processing of records). It would be valuable to document cases in which such procedures were used to improve programmes.

The basic features which should be emphasized are rapidity, simplicity, cost-effectiveness and immediate feedback. Rapid assessment procedures help staff perceive monitoring and evaluation as an integral part of the programme and not as a totally separate undertaking. If implementation staff use rapid assessment procedures as a continuous monitoring tool, a certain level of standardization is recommended, with some flexibility in adaptations to local situations. Coordination and supervision by experienced persons are crucial. If rapid assessment procedures are used for baseline research, a core set of indicators would be helpful to allow comparability across regions; in addition, culture-specific indicators may be useful.

To sustain the evaluation process, appropriate methodologies and technologies should be employed so that community members can use the methods on their own and receive immediate feedback.

C. ANALYSIS AND REPORTING

For analysing and reporting results of rapid assessment procedures, the group recognized that it is easier to communicate results of quantitative than qualitative studies. When both quantitative and qualitative methods are used, they should be integrated to answer common research questions. For an analysis of rapid assessment procedures results, it is important to begin with a clear conceptual framework, which should be reflected in the report. The analysis should be organized according to programme objectives. Quantitative data should be organized along standard indicators. Qualitative data should be presented in ways that are easy to read. Whenever possible, complex results should be presented in tables or boxes, along the lines of content-analysis schemes. Qualitative methods are especially helpful in identifying typical behaviour patterns or categories and themes, and manuals on rapid assessment procedures might include examples of such presentations.

Reports of findings should emphasize that rapid assessment procedures do not pretend to be generalizable or statistically representative. Reports should also present unanticipated but important issues. Conclusions should be action-oriented, in the spirit of applied research and consistent with identified needs.

To increase the involvement of the user community, cooperation between researchers and users of results is recommended at all stages of the rapid assessment procedures process. The group recommended that quantitative results be fed back to the community on the spot (using portable pocket computers, for instance, or simple manual computation on pocket cards). This would permit users to examine additional issues raised by the findings and would raise awareness about the preparation of local planning initiatives. In Zambia, for example, multitier and multisectoral committees at district and central levels organized the dissemination of findings through workshops that served to develop action plans.

D. INSTITUTIONALIZATION IN UNFPA

To institutionalize rapid assessment procedures, the group recommended that UNFPA organize consultations with concerned parties, including field practitioners and academics, as appropriate; the dissemination of documentation alone on rapid assessment procedures will not suffice. The standardization of terminology among United Nations agencies to ensure a shared language and understanding of rapid assessment procedures would also be helpful.

1/Cluster survey methodology is synonymous with the term "two-stage cluster sampling", a form of survey sampling in which the first stage consists of selecting a set of comparatively small areas, such as villages -- defined as "clusters" -- and the second stage consists of selecting households within those clusters. The selection procedures often do not conform strictly to scientific sampling methodology and thus may yield biased results. In modified cluster sampling, the procedures in both the first and second stages are based on strict probability techniques to ensure that the results are not biased. See A. Turner, R. Magnani and M. Shuaib, "A not quite as quick but much cleaner alternative to the expanded programme on immunization (EPI) cluster survey design", International Journal of Epidemiology, 25 (1996):198-203.

2/Multiple indicator cluster surveys is the name of a survey programme developed by UNICEF in 1994. It uses the modified cluster sampling methodology in developing countries for surveying numerous indicators to assist in monitoring progress towards the goals of the 1990 World Summit for Children. The name has since been shortened to "multiple indicator surveys (MIS)." See UNICEF, Monitoring Progress Toward the Goals of the World Summit for Children: A Practical Handbook for Multiple-Indicator Surveys (New York, UNICEF, 1995).

3/Probability sampling is based on scientific, mathematical procedures in which all members of a given target population (e.g., households or persons) have known, non-zero chances of being selected in the sample, and in a way such that the numerical probabilities of each sample element can be calculated to use in inflating the survey or study results to represent the total population. See L. Kish, Survey Sampling (New York, Wiley, 1965).

4/Snowball sampling is a form of sampling in which the initial set of sample individuals is asked to identify others with the same characteristic. The latter are then interviewed and asked to identify further individuals and so on. This type of sampling is useful when rare events or rare population subgroups are being studied. For example, in a survey of individuals with a rare disease, one individual in the survey would be asked to supply names of others with the same disease that she might know about. See United Nations, Sampling Rare and Elusive Populations (New York, United Nations Statistics Division, 1993).

5/Segment sampling is synonymous with cluster sampling. See note 1 above.