CHAPTER TWO -- DEVELOPMENT AND CHARACTERISTICS OF RAPID ASSESSMENT PROCEDURES

The application of anthropology in applied research is problematic because of a perception of the extended period of time necessary to generate data. As a consequence, applied anthropologists and researchers from a number of other disciplinary fields (agriculture, particularly) have, over the past 10 to 15 years, developed specific procedures and techniques for use in different cultural and geographic settings for programmatic purposes by people who may not necessarily possess high-level academic skills. This work has resulted in several manuals, handbooks, common protocol and field research guides for health research, known by various acronyms (RAP, RRA, FES, REM etc.). This report avoids the use of acronyms to depict rapid assessment procedures, in general, precisely because of the associations of terminology, acronyms and particular approaches and packages of methods.

As elaborated below, interviews, focus groups and observations are the primary means by which perceptions and beliefs are generally documented. However, an essential feature of rapid assessments, which explains the approaches of existing protocols and manuals, is the tailoring of specific methods to meet study objectives; the choice of method is necessarily determined by the questions to be answered.

A. ANTHROPOLOGICAL APPROACHES TO APPLIED RESEARCH

An early and highly influential manual was developed in the 1980s for primary health care and nutrition, 15 based on preliminary work conducted in Guatemala, with field tests in an additional 15 countries.16 Two manuals developed about the same time, concerned with water supply and sanitation, also had anthropological input.17. Concurrently, other anthropologists developed various approaches to conduct, produce and disseminate timely applied research in an interdisciplinary setting, e.g., Rapid Rural Appraisals (RRAs) for agricultural development and nutritional research,18 needs assessments 19 community diagnoses, situational analyses, focused ethnographic studies (FES), targeted intervention research (TIR) and rapid evaluation methods (REMs).20

Other approaches, also usually known by their acronyms, emphasize participatory research or partnerships between researchers and community members. These participatory approaches include Participatory Rural Appraisals (PRAs), Participatory Learning Methods (PALM), Agroecosystem Research (AER), Farming Systems Research (FSR), Participatory Action Research (PAR), Rapid Rural Systems Analysis (RRSA), Méthode Accelerée de Recherche Participative (MARP), Promotion of the Role of Women in Water and Environmental Sanitation Services (PROWWESS) and so on.21 They stress the importance of action research 22 and the role of community members in initiating, collecting, analysing and using their own data, or the collaboration of community members with others (government officers or researchers), so that they can control the production and flow of knowledge and shape policy. Vijaya Shrestra captures the participatory nature of these approaches in her definition of rapid assessment as "a semi-structured process of learning with and from village people about their needs, problems/conditions, local resources, expertise, capabilities, experience and pertinent social information." 23 Shrestra's definition relates to her application of the approach to water and sanitation in western Nepal, conducted with the operational goals of ensuring access to a safe and adequate water supply and of promoting locally appropriate health education and sanitation activities. The general emphasis, however, is on the engagement of researchers with the study population and the importance given to the perceptions of that population, a feature common to all rapid anthropological procedures and approaches.

A number of recent manuals have focused on specific diseases or health issues.24 They all emphasize the collection of population-based information, although they vary in their purposes and in the relevance of the range of social, cultural, economic and political factors that might influence health status or outcome. In all the manuals, qualitative methods have been tailored to ensure the collection of information central to the commissioning agency, department or control programme, and/or directed to the development of culturally appropriate interventions, health education material and so on. The focused ethnographic study adopted for acute respiratory infections (ARIs),25 for example, examines the local terminology for coughing and the ability of caretakers to recognize and respond to symptoms of acute disease (e.g., fast breathing, chest indrawing), while manuals on sexually transmitted diseases (STDs) and HIV/AIDS explore the social and cultural context of infection (sex work, patterns of relationships and partnering, education policies, government approaches) as well as the particularity of risk-related practice.26

The manuals all introduce various qualitative research methods to facilitate the exploration of relationships, interactions and perceptions. They are also used to supplement objective information with subjective insights 27 and to describe the social and cultural context in which programmes and policies are implemented. The combined use of such methods as in-depth interviewing, focus groups and observation allows triangulation and, hence, maximizes the internal validity of data to enhance insight and understanding of the basis of behaviours. The iterative nature of anthropological methods (discussed below) allows the researcher using the manual to add categories during the course of data collection and to pursue unexpected links.28

B. SAMPLING FRAMES AND STUDY POPULATIONS

At the core of the reluctance to adapt rapid assessment procedures to population programmes is a concern about coverage. Rapid assessments are, by design if not by definition, studies which opt for timely, focused and qualitative information at the expense of "scientific" sureness of results through strong probability sampling. Rapid assessments using ethnographic methods enable researchers to examine specific research questions closely. For example, a small study of clients' perspectives of MCH services may result in important insights into the use and/or operation of such services more widely. The findings cannot, however, be generalized to the population as a whole.

Several problems associated with such focused studies have been identified. These include those related to the accuracy of findings within a given study, their representativeness for the larger population, cultural inappropriateness, and their subjectivity as a result of using data collection methods drawing predominantly from anthropology, in which context the reliability of data relates especially to the individual researcher's skills in triangulating information.29

A central concern, however, relates to the selection of the study population. This has been the concern particularly for rapid epidemiological assessments, in which the goal is to yield data generalizable to the wider population. Rapid epidemiological assessments have predominantly followed the approach to sampling used to assess immunization coverage, as developed for the specific needs of the World Health Organization (WHO) Expanded Programme on Immunization (EPI). This sampling procedure uses two-stage cluster sampling, with non-probability sampling first of 30 areas or villages of the total geographic area under study, followed by the random sampling of seven households per village, often by spinning a bottle in the centre of the town, then taking the first and six contiguous houses in the direction following the bottle point.

A number of researchers, particularly statisticians and epidemiologists, have criticized this method of recruitment. While remaining sympathetic to the goal of rapid assessments, they argue the need for scientific rigour even in face of pragmatic needs for programme and policy purposes. A. G. Turner and colleagues, for example, argue that the failure of EPI studies to adhere to standard rapid sampling relates to non-probability, hence:

  • It is not possible to assess sampling areas or establish confidence limits of the data;

  • Sampling weights cannot be calculated;

  • Sampling bias can occur, for example, where immunized children are clustered in the first place; and

  • Problems exist in expanding the method for a multipurpose study because of the need for different populations according to issues to be included in the study.

    Turner and colleagues have attempted to modify this method to increase the reliability and validity of the sample. They propose a middle ground between EPI cluster sampling and random sampling by modifying methods of the selection of households to allow for the calculation of probability. Their alternative is a four-stage sampling procedure, which, in practice, includes:

  • A sketch map to select sample clusters;

  • The creation of segments of equal size;

  • The random selection of single segment from each cluster; and

  • Interviews of all eligible people in all households in the segment.30

    M. Anker and colleagues have also sought to modify the EPI method to enhance reliability. Their model provides for stratification at the first level of sampling, then random selection with a probability proportional to the size of the population covered. They also emphasize the range of methods necessary for rapid assessments: this includes consultation with national and other health staff; checks and reviews of data quality, collection and management; logistical prearrangement for fieldwork; and adequate computer support to ensure the timely production of information.31

    In a more conventional academic study, S. Gujral and colleagues sampled three to seven villages from all of 11 blocks of their study district in western India and interviewed the head of household as a proxy to gather information on all children in the study villages.32 O. Razum's rapid assessment of immunization coverage using a modified cluster sampling method provides a further example which diverges from the conventional household approach to the recruitment of population. At the same time, it supports a common-sense approach to sampling and data collection and to maximizing efficiency through the identification of a relevant population for the sampling frame. Razum argues that in areas with high early school attendance, sampling young children would produce less bias than household selection. In Zimbabwe, he was able to gain from the Ministry of Education the latest grade one enrolment lists of all primary schools in the study district, from which a cumulative enrolment list was prepared. Thirty schools were selected through proportional randomized sampling, and in each of these schools a grade 1 pupil was selected by drawing a random number. The selected pupil took the enumerators home. When a randomly selected pupil was absent from school on the day of the survey, a sibling or friend was asked to take the enumerators to the home of the selected pupil. The household closest to the pupil's home was the starting household of the cluster. From that stage, the procedure resembled the standard coverage survey.33

    C. FIELD TRIALS

    The published protocol and manuals for rapid assessment procedures and needs assessments are the products of often extensive field trials. The topics and techniques have been tried out in different settings to fine-tune research tools and to identify sources of difficulty or confusion. As already noted, the manual for anthropological assessments of primary health care and nutrition programmes written by Scrimshaw and Hurtado was developed in Guatemala.34 It was tested in 15 other countries, with the involvement of researchers already familiar with the language and culture of the study area working with a limited list of objectives and guidelines for the collection of community, household and health service data.35

    Similarly, the formative research for the ARI FES (focused ethnographic study for acute respiratory infections), conducted in the Philippines, was aimed both at documenting cultural and behavioural aspects of ARI and at exploring the range of research methods that might be included in a manual.36 Follow-up studies took place in Africa, Europe (Turkey), Asia and Latin America, supported by workshops to assist the researchers with data collection techniques, data management and analysis.37 The Hygiene Evaluation Procedure handbook adopted a similar process, with field development in Ethiopia, Kenya and the United Republic of Tanzania, field-testing of the draft protocol in Afghanistan and Kerala, India, and completion of the contents of the handbook at a peer evaluation workshop held in London, April 1996.38

    The development of procedures for family planning programmes will need to identify the suitability of various methods in the context of different programme issues, national contexts, and different times in the course of programmes. A prior task will be to develop a comprehensive framework, with sufficient flexibility to encourage its selective use and so to ensure the timely and appropriate collection of information.

    D. ITERATION AND TRIANGULATION

    Many of the methods advocated in rapid assessment manuals derive from anthropology, although the methods are increasingly common in a wide range of disciplines and professions, including social work, sociology, health education, nursing and psychology.39 The approaches typically involve the combined collection of primary data using qualitative methods and the collection and analysis of quantitative data, including data from clinic-based or demographic surveys and record reviews. The choice of methods is developed in the context of specific programme needs and health issues, involving a mix considered to be the most appropriate to the particular problem or the research objectives.

    The mix of methods in rapid assessment manuals reflects the importance of iteration and the emphasis placed on triangulation.

    Iteration

    Iteration refers to the continuity of the process of the collection and interpretation of data whereby data are constantly analysed and new research questions generated in the light of new findings.

    In rapid assessments, this approach may be adopted formally, for example, by field staff who meet regularly to report findings and to clarify further directions and priorities for the enquiry. This ensures that only relevant data are collected and that both non-issues and new and emerging issues are identified early in the assessment. For example, preliminary interviews with national family planning programme staff may indicate that minority groups are reluctant to use family planning services because of lack of knowledge about such services or because of the role of children in subsistence agriculture. Focus-group discussions in the field indicate that women know that there are means by which they can space and limit numbers of children and that they would like to do so. However, their access to services is poor and provider-client relationships discourage continuation. The assessment team, on this basis, would cease to ask about knowledge of family planning or the value of children. Instead, they would use interviews to explore further the question of access, for example, in terms of clinic hours, location of clinics, transport services and cost. Through observations and interviews, the team would also examine more closely provider-client interactions. Iteration allows emerging issues to be clarified promptly to ensure that the study's relevance is maintained.40 Box 2 elaborates this approach with an example of an EPI study designed as an applied health research project, conducted in Zimbabwe's Chimanimani District.

    Iteration also allows researchers to identify points of "theoretical saturation" -- that is, when no new data are being collected -- in preference to numerical coverage/probability sampling. This is important in rapid assessments in which research time needs to be allocated expediently, although the notion of theoretical saturation is also a general principle governing anthropological and other qualitative research. This cannot be predicted: in some cases, many interviews and/or focus groups may need to be conducted before no new data are forthcoming; in other cases, the topic may be exhausted quickly. Iteration also enables research team members to react to the data they have collected: to cross-check with participants, to follow up questions as they arise and to provide further focus to the study.41

    Triangulation

    The use of a mix of methods whereby triangulation is possible -- as is characteristic of rapid assessments -- allows researchers to focus on a small community or facility and to examine more closely, for example, discrepancies between lay and professional interpretations of events and encounters, differences between illness and disease and so on.

    Box 2. An iterative research design to assess service quality

    Leading question Research instruments
    Aim
    1. Is there a service quality problem
    in the EPI of the district?
    . Routine service data
    . Previous survey data
    . Informal observations and discussions
    . To formulate an initial presumption
    2. What is the true dimension of the quality problem?
    . Coverage survey . To assess the dimension of the problem
    . To test the validity of routine service data
    3. Where exactly does the quality problem lie?
    . Detailed analysis of the survey data . To delimit the area for further research: availability
    access/utilization
    adherence to standards
    4. What causes the quality problems on the side of the providers?
    . Open discussions with nurses (workshop) . To obtain the point of view of the nurses
    5. What causes the quality problems on the side of the users?
    . Focus groups with
    mothers
    . To obtain the point of view of the users
    Source: Oliver Razum, n.d. [c.1994], Improving Service Quality through Action Research as applied in the Expanded Programme on Immunization (EPI) (Heidelberg: Institute of Tropical Hygiene and Public Health and the MCH/FP Health Systems Research Network in the Southern African Region),
    p. 93, Table 22.

    Triangulation aims at maximizing the validity and reliability of the data by the use of multiple methods, often also with the involvement of research teams with individuals having different research skills.42 In the malaria manual, for example, Agyepong and colleagues advocate the inclusion of researchers with qualitative and clinical skills, and, depending on the aim of the assessment, epidemiological and/or entomological skills.43 In reproductive health, team members with anthropological expertise might be complemented by others with clinical skills, for example, in studies conducted in clinical settings or in which issues of folk diagnosis and treatment of reproductive tract or other infections may be important. The involvement of team members with local knowledge and language skills may improve rapport with the study community and increase the validity of data.

    Triangulation also occurs through cross-checking information by interviewing a wide range of people, by checking reported events in published documentation and by observation.44 Access to services, for instance, needs to be considered in terms of the formal provision of services, client and non-client perceptions of the services, and direct and indirect costs of services.

    E. CONSULTATION

    The successful conduct and use of findings of rapid assessments depends partly on the involvement of the commissioning agency or department. Its input in identifying programme-related needs and key research questions helps encourage the use of findings in the development or improvement of policy, programmes or specific services, thus ensuring that the data collected are used to maximum effect. In addition, service providers at national and agency levels as well as at the local level are part of the target population, and rapid assessment procedures may be used for internal purposes as well as for clients of services.

    Briefing by the commissioning agency is, therefore, a critical component of the procedure. For rapid assessments related to population and/or reproductive health, such briefings might be extended to include non-governmental organizations (NGOs) providing relevant health services as well as sections of the Government concerned with both policy and programmes. This approach allows the identification of relevant published documents. In addition, an appropriate consultation and briefing before the rapid assessment assists researchers in identifying other agencies and persons whose input might be relevant and provides the team with formal assistance, practical and political, to conduct the study.

    One approach to this kind of consultation is that of the TIR framework developed by D. Helitzer-Allen and colleagues for STDs/HIV.45 This approach identifies five levels of analysis -- individual, social network, organizational, community and public policy -- which help identify relevant behavioural, structural and administrative determinants. The aim is to discriminate between issues affecting the development of policy and those affecting the delivery of services. Many of these can be addressed programmatically and, therefore, depending on objectives, may need to be the central issues of enquiry of a rapid assessment. In the context of population programmes, these might include political and cultural issues which affect the acceptability, content, adoption and implementation of programmes as well as administrative and technical factors.

    The purpose of including programmatic questions in a needs analysis is to ensure the collection of appropriate information for programme planners; the approach discourages the collection of extraneous information and so maximizes the time set aside for the assessment. 46 Compromise is implicit in this approach and underlies the relevance of consultation. Rapid assessments used as policy and management tools -- as opposed to those used in development stages of research projects -- need to be clearly focused.