Rapid evidence assessment on women’s empowerment interventions within the food system: a meta-analysis

Miriam Berretta; Meital Kupfer; Shannon Shisler; Charlotte Lane

doi:10.1186/s40066-023-00405-9

Rapid evidence assessment on women’s empowerment interventions within the food system: a meta-analysis

Berretta, Miriam; Kupfer, Meital; Shisler, Shannon; Lane, Charlotte 2023-05-26 00:00:00 Background Women’s empowerment interventions represent a key opportunity to improve nutrition-related out- comes. Still, cross-contextual evidence on the factors that cause poorer nutrition outcomes for women and girls and how women’s empowerment can improve nutrition outcomes is scant. We rapidly synthesized the available evidence regarding the impacts of interventions that attempt to empower women and/or girls to access, participate in and take control of components of the food system. Methodology We considered outcomes related to food security; food affordability and availability; dietary quality and adequacy; anthropometrics; iron, zinc, vitamin A, and iodine status; and measures of wellbeing. We also sought to understand factors affecting implementation and sustainability, including equity. We conducted a rapid evidence assessment, based on the systematic literature search of key academic databases and gray literature sources from the regular maintenance of the living Food System and Nutrition Evidence Gap Map. We included impact evaluations and systematic reviews of impact evaluations that considered the women’s empowerment interventions in food systems and food security and nutrition outcomes. We conducted an additional search for supplementary, qualitative data related to included studies. Conclusion Overall, women’s empowerment interventions improve nutrition-related outcomes, with the largest effects on food security and food affordability and availability. Diet quality and adequacy, anthropometrics, effects were smaller, and we found no effects on wellbeing. Insights from the qualitative evidence suggest that women’s empowerment interventions best influenced nutritional outcomes when addressing characteristics of gender- transformative approaches, such as considering gender and social norms. Policy-makers should consider improving women’s social capital so they can better control and decide how to feed their families. Qualitative evidence sug- gests that multi-component interventions seem to be more sustainable than single-focus interventions, combining a livelihoods component with behavioral change communication. Researchers should consider issues with inconsistent data and reporting, particularly relating to seasonal changes, social norms, and time between rounds of data col- lection. Future studies on gender-transformative approaches should carefully consider contextual norms and avoid stereotyping women into pre-decided roles, which may perpetuate social norms. Keywords Women’s empowerment, Review, Food system, Meta-analysis *Correspondence: Miriam Berretta mberretta@3ieimpact.org Full list of author information is available at the end of the article © The Author(s) 2023. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. The Creative Commons Public Domain Dedication waiver (http:// creat iveco mmons. org/ publi cdoma in/ zero/1. 0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 2 of 52 and improve the nutrition of women and their commu- Introduction nities directly and indirectly. Women can improve their Most research on women within food systems focuses on own and their children’s nutritional status when they their roles as caregivers and cooks [1]. However, women have the socio-economic power and social capital to are key actors within food systems, serving as produc- make decisions on food and non-food expenditures and ers, processors, distributors, vendors, and consumers. the ability to take care of themselves and their families Often living in more vulnerable conditions than men [3]. By giving women more control and self-determina- due to societal norms, women face negative, differential tion, women’s empowerment interventions are expected access to affordable, nutritious foods relative to men. to have larger impacts than similar interventions that do Gendered food systems interact with gender equality and not incorporate an empowerment approach. Women’s equity at individual and systemic (community) levels, empowerment interventions may allow women to make as well as in formal (traditions and economic roles) and the choices that are most likely to benefit them while informal (household norms) ways, also referred to as the addressing the broader social and cultural context. As a four quadrants of change (Fig. 1). To achieve food sys- result, women’s empowerment interventions represent a tems transformation, women will need to have adequate key opportunity to improve nutrition-related outcomes, agency and control over resources. Social norms, policies, and women’s empowerment has been highlighted as a and governance structures must be fair and equitable to critical, crosscutting theme for food systems transfor- allow women access to food and livelihood opportuni- mation [4]. However, cross-contextual evidence on the ties. However, many food systems and nutrition inter- factors that cause poorer nutrition outcomes for women ventions are criticized as disempowering because they and how women’s empowerment can improve nutritional can entrench stereotypes by targeting women and girls outcomes is still scant [2]. explicitly in the roles of caregivers or cooks. Gender-transformative approaches (GTA) acknowl- Improvements in women’s empowerment are expected edge the equal role that all genders have in women’s to facilitate women’s interactions with the food system Fig. 1 Theory of change, from Njuki et al. [2] B erretta et al. Agriculture & Food Security (2023) 12:13 Page 3 of 52 empowerment and thus target men as agents of change to environment and dietary measures, a subset of the fac- transform structural barriers and social norms [5]. While tors presented in Fig. 1. Measures of wellbeing are also many women’s empowerment interventions include GTA considered due to their direct link with women’s empow- approaches, women’s empowerment and GTA differ erment. The interventions we identified primarily relate mainly in the following aspects (adapted from [6]): to behavior change communication, skills training, and asset transfers. Interventions were often complex and integrated other components, such as microcredits, self- • Approaches to women’s empowerment often focus help groups, and provision of vitamins supplements. only on women. GTA, on the other hand, aim to They often targeted men as well as women, making them address broader social contexts and avoid essential gender-transformative. zing men and women. • A central element of GTA is intersectionality, i.e., Objectives and research questions considering the interconnections between different The objective of this work was to rapidly synthesize the social identities, such as gender, race, ethnicity, or available evidence regarding the impacts of interventions geographic location. that attempt to empower women and/or girls to access, participate in and take control of components of the food For our purposes, women’s empowerment interven- system. Outcomes considered are limited to measures of tions within the food system are defined as “efforts tar - the food environment and diet. This fills the synthesis gap geted at increasing women’s abilities to make decisions identified by Moore et al. [1]. We also sought to under - regarding the purchase and consumption of healthy stand factors affecting implementation and sustainabil - foods” based on 3ie’s Food Systems and Nutrition Evi- ity, including equity. We specified the following research dence Gap Map [1]. Moore et al. [1] determined that, as questions a priori (Appendix 1): of January 2022, there were 21 evaluations of the impacts of interventions that target women’s abilities to make decision regarding the purchase of healthy foods, for 1. What are the effects of women’s empowerment inter - example by improving decision-making on household ventions within the food system on food availabil- expenditures. However, these studies had not been syn- ity, accessibility, and affordability, of healthy diets or thesized to determine average treatment effects and key nutritional status? contextual factors driving to impact. In this rapid evi- 2. Are there any unintended consequences of such dence assessment, we focus on 10 of those studies which interventions? looked at specific outcomes related to food security, food 3. Do effects vary by context, approach to empower - affordability and availability, diet quality and adequacy, ment, or other moderators? anthropometrics, iron, zinc, vitamin A, iodine status, and measures of well-being. Methodology This rapid evidence assessment provides a novel syn - To respond to these research questions, we conducted a thesis of the available evidence on the impacts of inter- rapid evidence assessment (REA). As far as possible this ventions to support women’s empowerment within the REA is based on the rigorous methodologies adopted food system, contributing to the literature base on both in a systematic review [9]. However, due to time and women’s empowerment and food systems. It is expected resource limitations, the search and screening process to support policymakers, experts, and stakeholders and the data extraction process were shortened [10]. in making evidence-informed decisions regarding the These abbreviated steps allowed for the rapid nature of implementation and design of such interventions. Stake- this rapid evidence assessment. The protocol for the REA holders can use this work to understand how to better was developed a priori in February 2021 and is provided integrate gender-transformative approaches as one char- in Appendix 1. acteristic of feminist development policies, to improve nutritional outcomes in the project and study design Search and screening based on the EGM by Moore et al. [1] process while acknowledging and moving past the use of We did not conduct a new search for impact evaluations, stereotypes. but relied on an existing, open-source evidence gap map In this rapid assessment, we run a meta-analysis and a (EGM) by Moore et al. [1]. The EGM includes all impact barriers and facilitators analysis of interventions on the evaluations and systematic reviews of impact evaluations economic and social empowerment of women with the of interventions within the food system which measure goal of providing them the means and ability to affect outcomes related to food security and nutrition in low- dietary decisions; [7, 8]. As a result, we focus on food and middle-income countries (Appendix 7). Because the Berretta et al. Agriculture & Food Security (2023) 12:13 Page 4 of 52 Table 1 PICOS Criteria Included Excluded Participants People of any age and gender residing in low- and middle-income countries High-income countries Intervention(s) Interventions aimed at increasing women’s empowerment and giving women the All else capabilities to make decisions on the purchase and consumption of a healthy diet Comparison Business as usual, including pipeline and waitlist controls No comparator An alternate intervention Outcome(s) Food security All else Food affordability and availability Diet quality and adequacy Anthropometrics Iron, zinc, vitamin A, and iodine status Measures of well-being Study designs Experimental, quasi-experimental, systematic reviews and cost evidence Efficacy trials Before-after with no control group Cross-sectional studies, etc. search conducted by Moore et al. [1] was not specifically environment (food security and food affordability and focused on women’s empowerment, rather it included availability), diet (diet quality and adequacy, anthropo- women’s empowerment among a variety of other topics, metrics, and micronutrient status), or well-being. Table 1 it is possible that some articles may have been missed. presents the population, interventions, comparisons, out- However, there is no reason to believe that there would comes, and study designs (PICOS), modified from Moore have been any systematic bias in the types of articles et al. [1], employed by this REA. that were omitted or that this would have meaningfully Although we did not perform any new searches for affected results. impact evaluations for this rapid evidence assessment, we conducted a targeted search in Google Scholar look- ing for the qualitative papers related to included studies • The search by Moore et al. [1] was extensive and sys - to allow us to investigate how impacts were achieved. The tematic, covering 12 academic databases and 13 gray search included the name of the program or intervention, literature sources (Appendix 7). Single screening with if available, as well as the country the intervention took safety first was used at both title and abstract and full place in. Eligible qualitative study designs were [11]: text stages. A machine learning classifier was applied to automatically exclude studies with a low prob- ability of inclusion. Although the original search was • A qualitative study collecting primary data using complete in May 2020, the search is continuously mixed methods or quantitative methods of data col- updated with studies added to the EGM through lection and analysis and reporting some information January 2022 considered for this REA. As of January on all the following: the research question, proce- 2022, over 160,000 articles were screened for inclu- dures for collecting data, procedures for analyzing sion in the EGM and 2,647 studies were included data, and information on sampling and recruitment, Appendix 7. including at least two sample characteristics. • A descriptive quantitative study collecting primary Because this REA is based on the search by Moore et al. data using quantitative methods of data collection [1], the same criteria for eligible populations, compara- and descriptive quantitative analysis and reporting tors, and study designs employed by Moore et al. [1] were some information on all the following: the research used for this REA. Moore et al. [1] included interventions question, procedures for collecting data, procedures which targeted women’s empowerment within food sys- for analyzing data, and information on sampling and tems. Women’s empowerment interventions which func- recruitment, including at least two sample character- tioned outside the food system, such as those related to istics. economic empowerment outside of the food system, were • A process evaluation assessing whether an interven- not included. From the 21 studies on women’s empower- tion is being implemented as intended and what is ment interventions included in their EGM, we selected felt to be working well and why. Process evaluations the ten studies evaluating outcomes related to the food may include the collection of qualitative and quanti- B erretta et al. Agriculture & Food Security (2023) 12:13 Page 5 of 52 Table 2 Included outcomes and indicator extracted for evidence synthesis Outcome Indicators* Food security Preferred outcomes: food security indexes and composite scores Secondary outcome: skipped meals Tertiary outcome: reports of insufficient food Food affordability and availability Preferred outcome: per capita food consumption in monetary units Secondary outcome: per capita food consumption in weight Other measures, such as the cost of a food basket, will be considered if these are not available Diet quality and adequacy Preferred outcomes: composite diet scores such as the nutrient rich food index Secondary outcome: dietary diversity and other food variety measures Tertiary outcome: intake of specific foods Anthropometrics Preferred outcomes: body mass index, weight for length, length for age, weight for age Other measures, such as MUAC and ponderal index, will be considered if these are not available Micronutrient (iron, zinc, vitamin A, iodine) status Preferred outcome: measures of content in blood/tissue (ex. hemoglobin levels) Secondary outcome: intake in weight (grams, micrograms, etc.) Tertiary outcome: intake in percentage relative to recommended intake Other measures will be considered Well-being Preferred outcome: perceived well-being Secondary outcome: anxiety *Indicators are listed by preference based on a priori specification. Such a priori specification reduces bias by preventing subjective reporting of outcomes by the team conducting the Rapid Evidence Assessment. Most indicators were ultimately not found in the studies tative data from different stakeholders to cover sub - specified, the model with the most control variables was jective issues, such as perceptions of intervention used. success or more objective issues, such as how an Two team members extracted bibliographic, geo- intervention was operationalized. They might also be graphic information, methods, and substantive data. used to collect organizational information. Substantive data were related to interventions, selected outcomes, population (including gender/age disaggrega- While the identification of qualitative evidence was tion, when available), and effect sizes. Discrepancies were limited to studies linked to the included impact evalua- reconciled through a discussion between the two team tions, the process of data extraction, critical appraisal, members. Qualitative information on barriers and facili- and evidence synthesis was independent. tators to implementation, sustainability and equity impli- cations, and other considerations for practitioners was Data extraction extracted by a single reviewer. Data extraction templates were modified from 3ie’s Included quantitative impact evaluations were standard coding protocol for systematic reviews, reflect - appraised by two independent team members using a ing another shortened step for the purposes of making critical appraisal tool (Appendix 3). Qualitative stud- this assessment rapid (Appendix 2). The primary modi - ies linked to included impact evaluations were critically fication to the tool was a restriction on the number and appraised by a single reviewer using a mixed methods type of outcomes considered. The outcomes considered appraisal tool developed by CASP [12] and applied in were broad and could be measured using a variety of Snilstveit et al. [11] (Appendix 3). indicators. To restrict the number of outcomes extracted, we specified preferred and secondary indicators of inter - Synthesis approach est a priori (Table 2). This limited the analysis to be We provide a narrative summary of the papers identified. conducted to only the specified outcomes. Composite This includes an overall description of the literature and a measures were always preferred over disaggregated ones. general synthesis of findings. Key information from each If multiple analyses were presented considering the same study, such as intervention type, study design, country, out- outcome (ex. Univariate analysis and a regression with comes, measurement type, effect sizes, and confidence rat - control variables), the data from the model preferred ing is summarized in tables. Results from meta-analyses and by the author was extracted. If no preferred model was associated forest plots are presented in the section on the Berretta et al. Agriculture & Food Security (2023) 12:13 Page 6 of 52 FREQUENCIES OF BIAS IN RCT S Low ROB Some concerns REPORTING BIA S 4 2 OUTCOME MEASUREMENT BIAS 6 PERFORMANCE BIA S 4 2 DEVIATIONS FROM INTENDED 5 1 INTERVENTION S CONFOUNDIN G SELECTION BIA S 4 2 UNIT OF ANALYSIS 5 1 A SSIGNMENT MECHANIS M 6 Fig. 2 Risk of Bias of the included randomized control trials findings. Qualitative information is summarized in a section [23]. We also summarize the findings of each study, on implications for implementation and sustainability. including narratively reporting on individual effects, in Table 3. For all outcomes except micronutrient status, Meta‑analysis the metrics were determined to be sufficiently similar to In addition to presenting individual effect estimates for warrant a joint analysis in addition to the presentation of all six outcomes, we conducted five meta-analyses to pro - individual effects. vide summary effect estimates on the five outcomes for To compare the effect sizes, we converted all of them which we had sufficient data. This meta-analysis provides to a single metric, Cohen’s d. We then converted all additional value relative to presenting the individual Cohen’s d to Hedges g to correct for small sample sizes. effect estimates by presenting a summary effect estimate. We chose the appropriate formulae for effect size cal - Meta-analyzed effects have the benefit of being sup - culations in reference to, and dependent upon, the data ported by a broader (Figs. 2 and 3), potentially more gen- provided in included studies. For example, for studies eralizable evidence base than individual point estimates. reporting means (X) and pooled standard deviation (SD) Previous works have statistically synthesized similar evi- for treatment (T) and control or comparison (C) at fol- dence, for instance, on food security and food affordabil - low-up only, we used the following formula: ity and availability [13, 14], anthropometrics measures X − X Tp+1 Cp+1 [14, 16, 17] micronutrients status [18–20], diet quality d = SD and adequacy [21, 22], Because only ten studies were included, meta-analysis If the study did not report the pooled standard deviation, was conducted at the outcome (column 1, Table 2), not it is possible to calculate it using the following formula: the indicator level (column 2, Table 2). However, due to 2 2 variations in the indicators used and their interpreta- (n − 1)SD + (n − 1)SD Tp+1 Cp+1 Tp+1 Cp+1 SD = tion, we also present the standardized effect estimates for p+1 n + n − 2 Tp+1 Cp+1 each study in each forest plot (Figs. 4, 5, 6, 7 and 8) and Appendix 6. The decision to conduct meta-analysis was where the intervention was expected to change the made on a case-by-case basis after considering if the indi- standard deviation of the outcome variable, we used the cators adequately captured the same underlying concept standard deviation of the control group only:For studies SOURCES OF BIAS B erretta et al. Agriculture & Food Security (2023) 12:13 Page 7 of 52 FREQUENC IE S OF BI AS IN QE D Fig. 3 Risk of bias of the included quasi-experimental included studies Fig. 4 Forest plot showing the effect of empowerment interventions on food security outcomes X X −X p+1 Tp+1 Cp+1 reporting means (X) and standard deviations (SD) for d = = For studies reporting SD SD p+1 p+1 treatment and control or comparison groups at baseline mean differences between treatment and control, stand - (p) and follow-up (p + 1): ard error (SE) and sample size (n): X −X p+1 p d = For studies reporting mean differences SD p+1 p+1 (∆X) between treatment and control and standard devia - d = √ SD n tion (SD) at follow-up (p + 1): Berretta et al. Agriculture & Food Security (2023) 12:13 Page 8 of 52 Fig. 5 Forest plot showing the effect of empowerment interventions on food affordability/availability outcomes Fig. 6 Forest plot showing the effect of empowerment interventions on diet quality and adequacy For studies reporting regression results, we followed the 1 1 approach suggested by Keef and Roberts (2004) using the d = + n n T C regression coefficient and the pooled standard deviation of the outcome. Where the pooled standard deviation of where n denotes the sample size of treatment group and the outcome was not unavailable, we used the regression control. We used the following where total sample size coefficients and standard errors or t-statistics to do the information (N) is available only (as suggested in Polanin following, where sample size information is available in [34]): each group: B erretta et al. Agriculture & Food Security (2023) 12:13 Page 9 of 52 Fig. 7 Forest plot showing the effect of empowerment interventions on weight relative to height Fig. 8 Forest plot showing the effect of empowerment interventions on wellbeing 2t 4 d = T.INV.2T (exact p value, (n − 1)) d = √ Var = + N 4N where outcomes were reported in proportions of individ- When necessary, we calculated the t statistic (t) by uals, we calculated the Cox-transformed log odds ratio dividing the coefficient by the standard error. If the effect size [35]: authors only report confidence intervals and no standard √ error, we calculated the standard error from the confi - d = Log Odds Ratio v × dence intervals using the following: where OR is the odds ratio calculated from the two-by- SD = N × (upper limit - lower limit) 3.92 two frequency table. We fitted a random effects meta-analyses model when If the study did not report the standard error, but did we identified two or more studies that we assessed to be report t, we extracted and used this as reported by the sufficiently similar. We assessed heterogeneity using the authors. If an exact p value was reported but no standard DerSimonian–Laird estimator by calculating the Q sta- error or t, we used the following Excel function to deter- 2 2 tistic, I , and τ to provide an estimate of the amount of mine the t-value. variability in the distribution of the true effect sizes [23]. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 10 of 52 Table 3 Summary of included studies Study reference and country Study design Intervention Authors’ interpretation of effects Ahmed et al. [24] [Bangladesh] Randomized controlled trial The Transfer Modality Research Initiative ( TMRI) provided: All four interventions (Cash transfer, food transfer, cash + food, Cash or food transfers, with or without nutrition behavior change cash + BCC) increased monthly per capita food consumption, communication (BCC) for rural women living in poverty daily per capita intake caloric, and food consumption score. The effects are slightly higher for cash + BCC, particularly from the food consumption score Bandiera et al. [25] [Bangladesh] Randomized controlled trial The Targeting the Ultra-Poor program provided: Food security among women improved, but there was no effect on (a) livestock assets and skills transfers for the poorest women. mental health status Women were offered a menu of assets to support income gener - ating activities. Assets included livestock and goods for small-scale retail operations, tree nurseries and vegetable growing (b) each asset was offered with a package of complementary training and support Blakstad et al. [26] [ Tanzania] Randomized controlled trial The Homestead Food Production Program provided: Household dietary diversity score increased, but there was no effect (a) agricultural training for women and inputs to promote home- on food security stead food production (b) nutrition and public health counseling for women to improve diet and health-related behaviors Bonuedi et al. [27] [Sierra Leone] Quasi-experimental design The Pro-Resilience Action (PROACT ) project provided: LANN, combined with the cash crop intervention, improved dietary (a) the LANN was a participatory community-based intervention diversity and food consumption among women and children. involving nutrition education, behavioral change communica- LANN alone did not have any effect tion and awareness creation on the benefits of consuming diverse diets, proper child feeding and water, sanitation, and hygiene ( WASH) practices, and sustainable agriculture and natural resource management in rural areas (b) a cash crop, income-oriented intervention aimed at enhancing economic access to nutritious foods. It included a nutrition pro- gram directed at improving nutrition knowledge and stimulating nutrition-sensitive spending and allocation of other household resources Deninger et al. [28] [India] Quasi-experimental design The District Poverty Initiative in India: The creation of SHGs had mixed effects on food consumption (RS/ Supported new self-help groups for women living in poverty year), energy intake per capita (kcal/day), and protein intake p.c. in India by training leaders and accountants from new self- (g/day) among poor, non-poor, and poorest of the poor. The three help groups in basic management and accounting. The SHGs outcomes improved among the poor. Energy intake increased also combined savings generation and micro-lending with social for the poorest of the poor, but the other two outcomes were not mobilization significant for them. None of the outcomes improved among the non-poor Emran et al. [29] [Bangladesh] Quasi-experimental design The Targeting the Ultra-Poor ( TUP) program provided: The probability of having two meals a day, the probability of having (a) health, education, and training for poor women, including sufficient food to meet the household’s needs, and grain stock trainings in livestock and poultry rearing; fruit, vegetable, and increased. The highest impacts were reported on the first two herb cultivation; operation of tree nurseries; and village vending outcomes (b) vitamin A supplements for children under five B erretta et al. Agriculture & Food Security (2023) 12:13 Page 11 of 52 Table 3 (continued) Study reference and country Study design Intervention Authors’ interpretation of effects Haque et al. [30] [Bangladesh] Randomized controlled trial The Suchana project provided: The Suchana project increased food consumption during preg- Training on agriculture, aquaculture, and market development, nancy, the consumption of vitamin A capsules after last delivery, including challenging the gender barriers to agriculture, health and the consumption of at least 100 IFA tablets during pregnancy. and nutrition practices among the beneficiary women, husbands, Greater impacts were reported for the first two outcomes and other household members Heckert et al. [31] [Burkina Faso] Randomized controlled trial The Enhanced Homestead Food Production (E-HFP) program The E-HFP program reduced wasting but had null effects on hemo - provided: globin levels among children (a) agricultural assets (b) behavior change communication on agricultural activities, optimal infant and young child feeding, health, hygiene, and care practices Marquis et al. [32] [Ghana] Quasi-experimental design The Enhancing Child Nutrition through Animal Source Food The program had a positive effect on height-for-age z score, a Management (ENAM) program provided: negative effect on weight-for-age z score, and a null effect on BMI- (a) microcredit loans for-age z score of preschool-aged children (b) weekly nutrition, technical, and entrepreneurship training on viable income-generation activities Pan et al. [33] [Uganda] Quasi-experimental design This large-scale agricultural extension program for smallholder The program reduced meals skipped, worries about insufficient women farmers provided: food, and limited variety of food among smallholder women. It (a) training through model farmers increased per capita food consumption (b) easier access to and affordability of seeds sold through farmers serving as community agriculture promoters Berretta et al. Agriculture & Food Security (2023) 12:13 Page 12 of 52 We were unable to explore heterogeneity using modera- Anthropometric measures, micronutrient status, and tor analyses due to the small number of included studies. well-being outcomes were less common (n = 2 each). We found nine qualitative reports related to seven Qualitative synthesis interventions. Additional qualitative information was not The meta-analysis conducted with the quantitative data found for the remaining interventions. The qualitative has been complemented by a thematic synthesis utiliz- components of the main studies and additional studies ing the extracted qualitative data. Qualitative data were were minimal and primarily focused on contextual infor- synthesized thematically by a single team member and mation from the researchers. Many of the qualitative reviewed by two other team members. Themes consid - studies used focus group discussions or key informant ered related to non-nutrition impacts, barriers and facili- interviews to better understand participants’ lived reali- tators to impact, and cost evidence. ties. Qualitative data contextualized results of empower- ment interventions and food and nutrition security based Results on the differing intervention locations and intersect - Characteristics of the included studies ing social, cultural and gender norms that influence the We included ten studies retrieved through the systematic impacts on nutrition and other key outcomes. search done for the Food Systems and Nutrition Evidence All the randomized controlled trials except Blakstad Gap Map, conducted in January 2022 (Table 3). An addi- et al. [26] have an overall rating of ‘some concerns’, mainly tional, low-quality systematic review was identified and due to reporting bias, performance bias, and selec- excluded from analysis. Four of the ten included stud- tion bias (Fig. 7; Appendix 5). Deininger and Liu [28] ies were implemented in Bangladesh, while the remain- also encountered issues related to deviation from the ing studies where in Burkina Faso, Ghana, India, Sierra intended interventions and the unit of analysis did not Leone, Tanzania, and Uganda. The four studies in Bang - correspond to the unit of randomization. ladesh represent unique evaluations of a cash transfer Two quasi-experimental studies were rated as having program, an agricultural training program, and two fully a low risk of bias (Fig. 8; [32, 33]), one study as having independent evaluations of Targeting-Ultra-Poor pro- ‘some concerns’ [29], and one as having a high risk of bias gram (TUP) with a time gap of eight years and some- [27]. The major sources of bias were related to reporting what different intervention designs. More information bias, spill-over, cross-over and contamination, perfor- on study characteristics can be found in Additional file 1: mance bias, and confounding. Table S1. Randomized controlled trials (n = 4) and difference-in- What are the effects of women’s empowerment difference were the most common designs (n = 4). Half interventions on food environment, diet, and well‑being of the studies using difference-in-difference also used outcomes? statistical matching (n = 2). One study used statistical Standardized effects are reported in Table 7 in Appendix matching alone and one used regression discontinuity 6, calculated as outlined in the Methodology section. The to identify counterfactuals. Nine additional qualitative meta-analysis results of the random effects model are papers associated with seven interventions were also reported in Table 4. We could not run a meta-analysis on identified and included. micronutrient status because the two studies looking at it Almost all studies provided training (n = 8). Some also measured different underlying concepts which could not provided asset transfers (n = 6) and behavior change be meaningfully combined. communication (n = 3; Tables 3, 6 in Appendix 6, and Additional file 1: Table S1). Behavior change communi- Effect of women’s empowerment interventions on food cation interventions generally communicated messages security outcomes is promising about women’s empowerment and women’s roles within Our analysis of the effects of women’s empowerment their communities. Often, they targeted men, making interventions suggests they improved food security them gender-transformative. Training and educational outcomes overall ( µ = 0.24 [95% CI: 0.001 to 0.47 ], interventions focused on agriculture and/or nutrition, p = 0.048 , Fig . 4). Women receiving these interventions but some also considered entrepreneurship and water, had a 59.5% chance of having food security scores above sanitation, and hygiene. Asset transfers were largely the mean in the control group. There was significant vari - related to cash or agricultural inputs, including livestock. ation in the size of the effect, ranging from 0.07 in Tanza - Food affordability and availability outcomes were the nia, to 0.67 in Bangladesh. most common (n = 5). Diet quality and adequacy and We included four studies which reported the following food security outcomes were also common (n = 4 each). indicators: food security index (whether the household B erretta et al. Agriculture & Food Security (2023) 12:13 Page 13 of 52 Table 4 Meta-analytical results Outcomes Specific outcomes indicators # of included effects Overall effect size [95% CI] Estimated percentile change Heterogeneity of Range of effects (total number of compared to control group overall effect (Q and beneficiaries) [95% CI] I^2) Food security Food security index—whether 4 (12,545) 0.24* [0.00; 0.47] 9.5% [0; 18.1%] 111.16***, 97.3% 0.07 to 0.67 HH had surplus food or deficit, enough food to eat, and could afford to eat two meals a day; Food availability: sufficient food to meet the household’s needs; Household food insecu- rity access scale; Skip meals Food affordability/availability Food consumption score; 6 (12,972) 0.23** [0.09; 0.38] 9.1% [3.6%; 14.8%] 187.27***, 91.99% − 0.11 to 0.49 Total food consumption expenditure in the 12 months preceding the survey (Food production and market purchases) (LOG)-Household; Food consumption (RS/year); Grain stocks (kg); Per capita food consumption Diet quality and adequacy Household dietary diversity; 4 (16,025.5) 0.09** [0.06, 0.12] 3.6% [2.4%; 4.8%] 0.53***, 0% 0.076 to 0.14 Additional food consumed during pregnancy; Protein intake p.c. (g/day) in the total population; Variety of foods consumed Weight relative to length Weight-for-length z score; BMI- 2 (1156.6) 0.12* [0.00, 0.23] 4.8% [0; 9.1%] 0.12, 0% 0.06 to 0.12 for-age z score Well-being outcomes Mental health index; Worry 2 (10,100) 0.08 [0.01; 0.15] 3.2% [0.4%; 6%] 2.9*, 65.6% − 0.11 to 0.04 about insufficient food * is p < 0.05, ** is p < 0.01 and *** is p < 0.001. For more information on the results and the studies please see Additional file 1: Table S1 Berretta et al. Agriculture & Food Security (2023) 12:13 Page 14 of 52 had surplus food or deficit, enough food to eat, and could Effect of women’s empowerment interventions on diet quality afford to eat two meals a day), household food insecurity and adequacy outcomes is promising assessment scale (HFIAS), skipped meals, and food avail- Our analysis of the effects of women’s empowerment able to meet a household’s needs of two meals a day [25, interventions suggests they improved diet quality and 26, 29, 33]. All studies provided training or education, adequacy ( µ = 0.09 [95% CI: 0.06 to 0.12] , p < 0.01, mostly related to agriculture. Three also provided some Fig. 6). Women receiving these interventions had a 53.6% form of asset transfer [25, 29, 33]. chance of having diet quality and adequacy scores above Two studies were assessed as having some concerns the mean in the control group. The variations among the related to risk of bias [25, 29] and two were assessed as range of effects were not as high as for other outcomes, low risk of bias [26, 33]. ranging from 0.08 in India to 0.14 in Sierra Leone. Four studies reported impacts related to diet quality Effect of women’s empowerment interventions on food and adequacy, such as dietary diversity and amount of affordability and availability outcomes is promising food or protein consumed [27, 28, 30, 33]. All four stud- Our analysis of the effects of women’s empowerment ies employed training/education interventions focused interventions suggests they improved the availability on agriculture [27, 30, 33] or enterprise/accountability and affordability of food ( µ = 0.23 [95% CI: 0.09 to 0.38] [28]. Two studies also transferred assets [27, 33], and one p < 0.01, Fig. 5). Women receiving these interventions included a behavioral change communication component had a 59.1% chance of having food affordability and avail - [27]. ability scores above the mean in the control group. There One study was scored as low risk of bias [33], two were was significant variation in the size of the effect, ranging scored as having some concerns [28, 30], and one was from 0.08 in Uganda, to 0.49 in Bangladesh. rated as high risk of bias [27]. Food affordability and availability was measured in five included studies, per capita food consumption, food con- Effect of women’s empowerment interventions sumption per capita (Rs/year), total food consumption on anthropometrics is promising but there is a lack expenditure (food production and market purchases in of evidence the 12 months preceding the survey), and grain stock (kg) Our analysis of the effects of women’s empowerment [24, 26, 2829, 33]. We included two estimates for Ahmed interventions suggests they improved measures of et al. as the results were reported for independent sam- weight relative to height ( µ = 0.12[ 95% CI: 0.002to0.23] , ples from the North and South of Bangladesh, without an p = 0.046 Fig. 7). Children of women receiving these overall estimate for all the areas. interventions had a 54.8% chance of having anthropo- All studies but Deininger and Liu [28] included assets metrics scores above the mean in the control group. transfer, such as cash, cash crops [24, 27], or livestock, Two studies reported impacts on anthropomet- seeds, or vitamin A supplements [29, 33]. All studies, ric measures of children based on WHO z-scores [31, except Ahmed et al. [24] included trainings or education 32]. Both studies transferred agricultural [32] or finan - on nutrition [27], or agriculture [29, 33], or enterprise/ cial assets [32]. The Heckert and colleagues’ study also accountability [28]. Two studies also included a behavior included a behavioral change communication strategy, change communication component [24, 27]. while Marquis and colleagues included entrepreneur- Ahmed and colleagues also reported increases in ship training. Marquis et al. [32] also report a decrease monthly food consumption per capita in both northern in weigh-for-age (g = − 0.42 [95% CI: − 0.77 to − 0.06]) and southern regions of their intervention area (North and an increase in height-for-age (g = 0.40 [95% CI: 0.04 areas: g = 0.32 [95% CI: 0.27 to 0.38]; South areas: g = 0.22 to 0.75]). Heckert and colleagues were scored as having [95% CI: 0.16 to 0.27]) and per capita daily intake caloric some concerns about bias while Marquis et al. [32] had (North areas: g = 0.22 [95% CI: 0.17 to 0.28]; South areas: low risk of bias. g = 0.09 [95% CI: 0.043 to 0.15]). Three other intervention arms (provision of food, cash, or food plus cash) were Effect of women’s empowerment interventions also evaluated. However, we were not able to include on micronutrient status is promising but there is a lack them in the meta-analysis as they were not comparable to of evidence the other studies. All three reported similar impacts. Two studies considered the effects of women’s empow - Only Bonuedi et al. were assessed as having a high risk erment interventions on micronutrient status, but these of bias, the remaining studies have either some concerns could not be meaningfully combined in a meta-analysis [24, 28, 29] or low risk of bias [33]. because they measured different underlying concepts. B erretta et al. Agriculture & Food Security (2023) 12:13 Page 15 of 52 Haque et al. found that Suchana’s gender-transformative interventions allowed women to accumulate savings and approach, which encompassed a portfolio of agriculture spend more judiciously, rather than consistently respond- and entrepreneurship trainings, increased the consump- ing to immediate needs. tion of iron, folic acid tablets (g = 0.25 [95% CI:0.21 to Two interventions which combined training with 0.28]). Heckert et al. evaluated an agricultural education improved accessibility of agricultural assets increased and behavior change communication strategy, but they opportunities for paid work. The agricultural interven - found no effect on hemoglobin levels (g = − 0.10 [95% CI: tion in Uganda resulted in an increase in work for wages − 0.03 to 0.23]). Both studies were rated as having some and freed up off-farm work times for the entire house - concerns about bias. hold, including women [33]. Similarly, because of the TUP program, the labor market choices of household Effects of women’s empowerment interventions on mental members aside from the targeted woman also shifted well‑being outcomes is not significant and there is a lack [25]. However, women themselves did not have increased of evidence labor participation. Women in the program spent most Our analysis of the effects of women’s empowerment of their time at home and were generally not employed interventions shows no effect on mental health outcomes outside of the home [38]. In fact, women reported that ( µ = 0.08[ 95% CI: 0.01to0.14], p = 0.088 , Fig . 8). Bandi- they preferred to stay at home due to low pay and social era et al. [25] reported a mental health index constructed stigma in workplaces. based on self-reported happiness and mental anxiety, Similarly, two interventions focusing on household while Pan et al. [33] measured the level of worries regard- farming for improved nutritional outcomes were labor ing insufficient food. Both studies evaluated assets trans - and time intensive, which resulted in high attrition [26]. fer interventions, such as livestock, seeds, vegetables This additional labor was an increased burden on women growing, and specific trainings which accompanied to and took away from their time to acquire and prepare the transfers. Pan et al. [33] paper was assessed as hav- food for their families [27]. When data collection coin- ing a low risk of bias, while Bandiera et al. [25] paper was cided with harvest months in Sierra Leone, women’s assessed as having some concerns related to performance involvement in the farming activities increased their time bias. constraints and adversely affected caregiving practices. Implications Barriers and facilitators Implications for non‑nutrition outcomes Restrictive social norms preventing women from being Authors of many of these studies concluded that the able to take advantage of the interventions as intended interventions accomplished their goals of supporting was a common barrier. Structural gender barriers act as women’s empowerment, often by introducing gender- a driver of inequality in the household and community, transformative approaches which challenged traditional as specified in Njuki et al. theory of change (Fig. 1). In social norms. The Enhanced Homestead Food Produc - highly patriarchal societies, such as Sierra Leone, deeply tion (E-HFP) program in Burkina Faso included a gen- entrenched social and cultural norms marginalize der-transformative approach in which it improved men’s women, restrict their decision-making and exclude them perceptions of women as farm managers and increased from accessing or controlling household resources [27]. respect and communication in agri-business activities Single-focus interventions that only targeted nutrition or [31]. The accompanying behavior change communication value chain inputs without behavior change communica- intervention allowed mothers to better communicate tion related to social norms were not able to fully real- with men to improve familial support and adopt positive ize potential impacts because entrenched norms were nutrition behaviors, such as improved feeding practices. significant barriers to long-lasting change [33]. Even if Similarly, the Suchana program in Bangladesh resulted women were given the tools to work outside the home in improvements in women’s empowerment and mater- or own assets, they were often blocked from leveraging nal healthcare practices using a gender-transformative these tools by norms that dictate how women can act and approach [30]. Women became more confident to dis - work [33]. Gender-transformative approaches address cuss issues around food and management of household this social barrier by including men to ensure that the full resources with their partners [27]. Self-help group partic- impacts of interventions can be leveraged and realized as ipation improved social awareness and leadership skills. intended. Women mobilized to protest child marriage and violence In the TUP program, asset transfers that were intended against women in their communities [37]. The Target - for women members of households were controlled by ing-Ultra-Poor program (TUP) in Bangladesh increased men due to social norms [39]. Social norms delineated saving and borrowing opportunities for women. These what type of assets women were allowed to own. Larger Berretta et al. Agriculture & Food Security (2023) 12:13 Page 16 of 52 livestock, like cattle, were automatically perceived to over some of the transferred assets [39]. Interventions belong to men because they were higher in value and which took place at the home and approached women as traded more often. Their sale required an adult male’s caregivers and providers may have further perpetuated consent, which restricted women’s ability to own and the stereotype of women within these roles [37]. manage them. Restrictions almost always came from Unfortunately, the long time needed to change social jealous or violent husbands. When the TUP transferred norms was a barrier to these interventions achieving small livestock such as poultry, that women more often impact in the short period in which they were evalu- owned, it was easily controlled by women [39]. Reli- ated. The theory of change from women’s empowerment gious norms also played a role in restricting women’s interventions to improved nutrition outcomes assumes public movements. Care responsibilities were reinforced a change in social norms, which requires a significant by conservative social norms for women in Bangladesh, amount of time (Fig. 1). Change within the food sys- where women were demarcated as primary caregivers in tem is a dynamic process which often depends on other the home [37]. changes outside the scope of these interventions. Moreo- In some contexts, community and men’s support also ver, change processes are not straightforward and can facilitated improvements in outcomes, demonstrating be accompanied by setbacks, sometimes occurring par- the importance of gender-transformative approaches allel to positive effects. Behavior change communica - that actively challenge gender norms and power inequi- tion can be slow to expand women’s empowerment and ties between genders. In the Homestead Food Production households’ social status and networks [24]. Impacts intervention in Tanzania, women who lived near neigh- often become apparent in the long-term when founda- bors who also grew crops at home had higher dietary tional improvements consolidate and are dependent on diversity [26]. Participants who were close to markets internal and external factors. Food and nutrition security were able to access, trade and procure food and related and women’s empowerment may need to be achieved in items easier than those who were farther away [25]. If stages, according to different resources and opportuni - husbands and other men in the household or community ties [33]. For example, in India, the District Poverty Ini- were more receptive to change, then progress was more tiative fostered group formation and supported more visible with women in the TUP [37]. If a husband was mature groups, which could have significant economic more open to his wife engaging in out-of-house activities, benefits in the long term [28]. Because the study utilized livelihood strategies were more successful. data from three and six years after group formation, the Multi-component interventions may leverage synergistic research implies there may have been impacts on capital effects to have greater impacts than the individual com - endowments and economic effects on individuals and the ponents would have [27]. Complementary program arms group itself. Authors of evaluations that occurred within can reinforce each other in achieving desired results and 12 months of the interventions’ end indicated that a more reduce implementation costs to achieve the same objec- comprehensive understanding of women’s empowerment tives [27]. The asset-based component of the PROACT and nutritional outcomes would require longer-term and program in Sierra Leone had little effect. However, when more frequent data collection [26, 31]. combined with a behavior change communication com- Specific characteristics of the target group can affect ponent, it increased women’s decision-making power, impacts and may explain heterogeneity in results. House- shifting women’s roles in the household, and expand- hold decisions regarding assets and nutrition were ing women’s ability to work outside the house. Behavior shaped by local ecological and economic conditions [24]. change communication components of the TMRI pro- In India, target groups that were the poorest saw the gram in Bangladesh combined with the incentive of asset largest asset accumulation and empowerment improve- transfers allowed women’s sustained participation and ments. This resulted in the poorest benefitting both achieved an overall improvement in household indicators socially and economically [28]. Interventions which lev- over the course of the program [38]. erage existing groups may experience high attrition if the Interventions which do not address equity can be groups themselves experience attrition. For example, the less successful and re-enforced social norms. Often, Enhancing Child Nutrition through Animal Source Food entrenched norms and roles were not acknowledged Management program targeted microcredit groups, and within included interventions [40]. Failure to address experienced significant attrition among those who were these norms may have resulted in some interventions not benefiting from the loan program [32]. This may not being unsuccessful. This was seen in the Bangladesh asset have been observed if the intervention targeted women transfer program which did not address norms around directly and did not work through the microcredit group. livestock ownership and resulted in men gaining control B erretta et al. Agriculture & Food Security (2023) 12:13 Page 17 of 52 Cost information as food security and food affordability and availability. Cost reporting was low (n = 3). When studies reported Impacts seem to reduce along the causal chain. Some of cost data, either through cost per participant or cost ben- the more final outcomes, such as anthropometric and efit analysis, the benefits generally outweighed the costs. well-being measures, can take years to meaningfully The District Poverty Initiative in India found that net change. As such, modest early effects may imply longer- present value of benefits from the project were approxi - term change. mately $1,690 million, significantly more than the project Insights from the qualitative evidence suggest that cost of $110 million. Even if benefits only lasted for one women’s empowerment interventions best influenced year the estimated benefits still significantly exceed pro - food environment and diet outcomes when gender ject costs, with a benefit–cost ratio of 1.5 to 1 [28]. The and social norms were considered. However, often, TUP program in Bangladesh also showed that average entrenched norms and roles were not acknowledged in benefits, including increased household welfare, were these interventions [40]. When community, and espe- 3.21 times larger than costs. Big push programs, like the cially male support, was found, it may have facilitated TUP, required large investment. However, in this case, impact. Including gender-transformative approaches in it resulted in cost-effective and sustainable change in women’s empowerment interventions may be essential household welfare, including nutrition [37]. to challenge and overcome existing social norms which Multi-component interventions can be cost-effective often prevent the achievement of intended impacts. because they combine complementary initiatives, such Such transformative approaches may be necessary to as interventions targeting nutrition and social norms. allow women to fully benefit from ongoing interventions. This was seen in PROACT where impacts were only Restrictive social norms may prevent women from taking achieved once a behavior change component was added full advantage of the interventions and reduce potential to the asset transfer [27]. Similarly, when added to an impacts. asset transfer program, the TMRI women’s empower- Although women’s empowerment interventions are ment behavior change communication component costs promising approaches for improving measures of the $50 per beneficiary per year, which is a relatively low cost food environment and diet, interventions may need to compared to stand-alone behavior change communica- move beyond women’s empowerment interventions tion interventions [24]. Low-cost additional activities include GTA and gain the buy-in of men and the com- can have greater impact than expected, especially when munity. This can result in increased power of women in integrated with other components. The training of model household decision-making while also sensitizing men to farmers in Uganda improved cultivation methods at rela- women’s pursuits of work outside of the home [41]. GTA tively low cost when compared with the cost of inputs, require cultural and social adaptation to local contexts such as a high-yield and drought-resistant seeds. Both through strengthened local partnerships and capacities training and the provision of inputs improved women’s while considering intersectionality, e.g., by considering efficiency in household gardens [33]. However, when cal - different interconnections between gender, socioeco - culating costs, the additional cost of such labor should nomic class, and caste divisions. GTA and intersection- not be ignored, especially because these costs are often ality, both characteristics of feminist development policy, born by the women that these interventions are trying to are crucial to progress on gender equality and leverage help [26]. the full potential of policies and interventions. Similarly, interventions should attempt to improve women’s social Discussion capital so they can better control and decide how to Overall, our analyses suggest women’s empowerment acquire and prepare food for their families [39]. Focus- interventions can improve measures of the food environ- ing on the duration of interventions is also important. ment and diet. We find significant and positive effects Long-term interventions may be needed to account for on food security (0.24 [95%CI: 0.00 to 0.47], n = 4), food slow processes, such as changing social norms. Multi- component interventions, which combine a livelihoods affordability and availability ( µ = 0.023[95% CI: 0.06 to component (asset transfer or financial services) with 0.38] , n = 6), and diet quality and adequacy ( µ = 0.09 behavioral change communication and advocacy, may be [95% CI: 0.06 to 0.12 ], n = 4). With two studies consider- more effective than interventions focusing on just liveli - ing outcomes related to weight-for-length ( µ = 0.12 [95% hoods or behavioral change. CI: 0.00 to 0.23 ]) and wellbeing ( µ = 0.08 [95% CI: 0.01 With ten included studies, the evidence base is small, to 0.15 ]) each, the evidence is too limited to draw con- which can reduce generalizability. Variation in the meas- clusions. Although impacts on diet quality and adequacy, ures considered in the meta-analysis may drive het- anthropometrics, and well-being were positive, they were erogeneity in results. However, the overall quality of the smaller than impacts on more proximate outcomes, such Berretta et al. Agriculture & Food Security (2023) 12:13 Page 18 of 52 evidence is fair with most of the studies (n = 6) rated as implemented in Sub-Saharan Africa or South Asia, leav- having ‘some concerns’ regarding bias. Three studies ing evidence gaps in Central America, South America, were assessed as having ‘low risk of bias.’ Given the low and Central Asia. Most studies were implemented in con- number of studies available and potential biases, the texts that were particularly patriarchal and restrictive for results should be interpreted with some caution. women, meaning that results in more egalitarian socie- Although the evidence was generally of high quality, ties may be different. Although we were able to run a five we had some concerns related to reporting, performance, meta-analysis, interpretation of the results is limited due and selection bias of the randomized controlled trials. to the low number of studies and variation in the indica- Within the quasi-experimental studies, we found issues tors synthesized. Cost data will also be needed to deter- related to reporting bias, spill-over, cross-over and con- mine if these impacts are cost-effective. To determine the tamination, performance bias, and confounding. Some sustainability of impacts over time, future studies should authors reported issues with incomplete or low-quality have longer intervention periods to ensure accurate cap- data, for instance, incomplete children’s health or vacci- ture of perceived impacts. Qualitative data can add rich nation records. Moreover, some children aged out dur- depth to quantitative findings by adding context, expe - ing the evaluation period making the data inconsistent. riences and meaning to the lived experiences of project Other studies did not collect data across seasons, an participants. Mixed-methods studies should focus on essential element when collecting data on agriculture identifying impacts and then using qualitative research to outcomes, which can act differently across seasons. Short interrogate how these impacts were achieved. Studies in interventions and short data collection periods might places with caste divisions, such as India or Bangladesh, also prevent impacts from being identified. These limita - could have benefited from a disaggregation in the experi - tions could result in findings being somewhat unreliable. ences and outcomes of women and households from dif- ferent castes. Future studies should try to avoid outcome Strengths, limitations & future directions measurement bias, reporting bias, spill-over, cross-over The interventions considered in this analysis were multi- and contamination, performance bias, confounding, and faceted, often considering two or three components: selection bias. Future studies should also ensure that data behavior change communication, training, and asset collection is representative of different seasons and con - transfers. As such, it is not possible to determine which textual changes, to avoid incomplete or insufficient data of these approaches is most effective. Future work can [26, 30, 32]. isolate the effects of these different pathways, as done by Due to the rapid nature of this work, results should be Bonuedi et al. [27], to determine which of these compo- interpreted with caution. The studies included in this nents is most effective. review are those found through the systematic search for The meta-analyses presented here combine disparate the EGM produced by Moore et al. [1] as of January 2022. indicators of broad concepts. The combined analysis of It is possible that a more sensitive and targeted search these different indicators is justified because they meas - strategy would identify additional studies. Moreover, the ure the same underlying concept. However, the variation REA is limited in the scope of interventions included. in indicator used by each study may explain the heteroge- Only those which take place within the food system are neity in results. For example, the analysis on food security considered; interventions functioning outside of the food combines a food security index, household food insecu- system may influence nutrition outcomes but have not rity assessment scale, number skipped meals, and indi- been considered. cator of whether food is available to meet a household’s needs of two meals a day. The framing of food attributes as positive versus negative can affect attitudes toward Appendices food [42], so framing questions around food security and insecurity may produce different results. As such, indi - Appendix 1: Rapid Evidence Assessment viudal effect estimates should also be considered and are on Women’s Empowerment in Food Systems reported within each forest plot and in Appendix 6. Sum- Interventions – Protocol maries of the effects identified by each study are provided Background in Table 3. Future work should move toward standardiz- The problem, condition, or issue ing measurement to allow for better comparability. Some Women are key actors within food systems, serving as of such efforts already exist, but should be further sup - producers, wage workers, traders, processors, and con- ported to allow for stronger synthesis [43, 44]. sumers. Women also face differential outcomes related to Given the limited evidence base, more research accessing and affording nutritious foods or a healthy diet. is needed in this field broadly. All the studies were Some evidence shows that women—often living in more B erretta et al. Agriculture & Food Security (2023) 12:13 Page 19 of 52 vulnerable conditions than men due to societal norms— Methodology can improve their own and their children’s nutritional To respond to these research questions, we will conduct status when they have socio-economic power to make a rapid evidence assessment, based on a systematic lit- decisions on food and non-food expenditures (especially erature search of key academic databases. Literature will accessing resources) and can take care of themselves and be screened for quality and summarized visually and in their families [3]. As a result, women’s empowerment a narrative format. A rapid evidence assessment is based interventions represent a key opportunity to improve upon the rigorous methodology adopted in a systematic nutrition-related outcomes. There is substantial agree - review; however, many steps are shortened [10]. ment about pathways to improve women’s empowerment in food systems. However, cross-contextual evidence Criteria for including and excluding studies in the review on the factors that cause poorer nutrition outcomes for (PICOS) women, and how women’s empowerment can improve nutritional outcomes is still scant [2]. Criteria Included Excluded The interventions Participants People of any age High-income countries We will include interventions that integrate activities and gender residing in low- and middle- to empower women and/or girls to access, participate income countries and take control in components of the food system, (L&MICs) for example improving decision-making on household Intervention(s) Interventions aimed All else expenditures. We have extracted relevant papers from at increasing women’s empowerment and the Food Systems and Nutrition evidence gap map that giving women the have any intervention component relating to women’s capabilities to make empowerment. decisions on the pur- chase and consump- tion of a healthy diet Expected theories of change Comparison Business as usual, No comparator Our theory of change is based on the pathways devel- including pipeline and oped by Njuki et al. [2] to presume that women’s empow- waitlist controls An alternate interven- erment can lead to improved nutrition with a variety of tion other influencing factors. Gendered food systems inter - Outcome(s) Food affordability, All else act with gender equality and inequality in a four-dimen- accessibility, and avail- sional space: individual, systemic, formal, and informal. ability Iron, zinc, vitamin A, and iodine status Rationale for the review Anthropometric This rapid evidence assessment is expected to inform measures Diet quality and decisions regarding gender and women’s empowerment adequacy in nutrition and food systems interventions. Given that Measures of well- women’s empowerment has been highlighted as a criti- being cal, crosscutting theme for the transformation of the food Study designs Experimental, quasi- Efficacy trials, before- experimental, system- after with no control system [4], key decision-makers have indicated interest in atic reviews and cost group, cross-sectional this area. Researchers can use this work to better under- evidence studies and so on stand how to intertwine gender-sensitive or -transforma- Types of study participants tive interventions for improved nutritional outcomes. Only studies which consider populations in low- and Research questions middle-income countries (as defined using the World Bank Country and Lending Groups classification in first 1. What are the effects of women’s empowerment inter - year of intervention or if not available then Publication ventions within the food system on the availabil- year) will be considered. The exception to this is if a ity, accessibility, and affordability of healthy diets or country held high-income status for only one year before nutritional status? reverting to L&MIC status. These will be included even if 2. Are there any unintended consequences of such the intervention began in the high-income year. As of the interventions? writing of this protocol, this applies to Argentina (2014, 3. Do effects vary by context, approach to empower - 2017), Venezuela (2014), Mauritius (2019), and Roma- ment, or other moderators? nia (2019). If the study is conducted in a high-income Berretta et al. Agriculture & Food Security (2023) 12:13 Page 20 of 52 country but measures impact on people, firms, or institu - Types of outcome measures tions in an L&MIC, it can be included. For example, we The table below outlines outcome indicators that will would not exclude a study that measures impact of New be extracted. These outcomes can be measured using Zealand’s immigration visa lottery on residents of Tonga. a variety of indicators. We have indicated the preferred outcomes and alternate outcomes which could be used if Types of interventions preferred outcomes are not reported. Composite meas- Eligible interventions were identified during the devel - ures will always be preferred over disaggregated ones. opment of the Food Systems and Nutrition Evidence Outcome Indicators Gap Map [1]. The map defined women’s empowerment interventions as “efforts targeted at increasing women’s Food security Preferred outcomes: food security abilities to make decisions regarding the purchase and indexes and composite scores Secondary outcome: skipped meals consumption of healthy foods.” After completing the Tertiary outcome: reports of insuf- search, we found that these interventions were primar- ficient food ily related to agriculture skills training, asset transfers, Food affordability Preferred outcome: per capita food microcredit, and behavior change. consumption in monetary units Secondary outcome: per capita food consumption in weight Citation Intervention Other measures, such as cost of a food basket, will be considered if Ahmed et al. [24] The intervention consists of two treatment arms: these are not available cash or food transfers, with or without nutri- Food availability/accessibility Preferred outcomes: food assets, tion behavior change communication (BCC), to production (community gardens,) women living in poverty in rural Bangladesh and stores Bandiera et al. [25] The intervention is a nationwide asset transfer Other measures, such as distance “plus” program in Bangladesh. The intervention and accessibility to markets transfers livestock assets and skills to the poorest women Diet quality and adequacy Preferred outcomes: composite diet scores such as the nutrient rich food Bonuedi et al. [27] The intervention is two-pronged: (1) cash crop index and (2) nutrition components. (1) Included farmer Secondary outcome: dietary diver- field schools (FFS), productive inputs, and value sity and other food variety measures chain linkages. (2) Included gender-sensitive nutri- Tertiary outcome: intake of specific tion behavior change and awareness creation foods Choudhury et al. [45] Suchana improves nutrition service delivery, nutri- Anthropometrics Preferred outcomes: body mass tion governance, and the knowledge of women index, weight for length, length for and girls regarding gender norms and gender- age, weight for age based violence that can impact mother and child Other measures, such as MUAC and nutrition ponderal index, will be considered if Deininger et al. [28] The intervention is self-help groups for women these are not available living in poverty in India Iron, zinc, vitamin A, and iodine Preferred outcome: measures of Emran et al. [29] This is an asset transfer “plus” intervention, status content in blood/tissue (ex. hemo- bundling asset transfers with capacity building globin levels) (health, education, and training) for poor women Secondary outcome: intake in with the goal of helping them graduate to the weight (grams, micrograms, etc.) standard micro-credit program of BRAC Tertiary outcome: intake in percent- Heckert et al. [31] The intervention is the Enhanced Homestead age relative to recommended intake Food Production (E-HFP) program, a nutrition- and Other measures will be considered gender-sensitive agriculture training program Well-being Preferred outcome: perceived well- Marquis et al. [32] This is a microcredit “plus” intervention that being provides microcredit loans and weekly sessions of Secondary outcome: anxiety nutrition and entrepreneurship education for 179 women with children 2–5 years of age Mosha et al. [26] The agricultural training and provision of inputs Types of comparators intervention includes the provision of small agricultural inputs to women, garden training support, and nutrition and health counselling to • Business as usual, including pipeline and waitlist con- improve food security trols Pan et al. [33] A large-scale agricultural extension program for • An alternate intervention smallholder women farmers to improve food security in Uganda • Studies with no comparator are excluded B erretta et al. Agriculture & Food Security (2023) 12:13 Page 21 of 52 Types of study design targeted searches to identify qualitative studies and pro- Experimental, quasi-experimental, systematic review, cess evaluations of the included interventions. and cost evidence will be considered. The following study designs will be included. Selection of studies Screening Because we are utilizing the results of the Food systems EGM, there is no search and screening pro- • Randomized controlled trial cess to select the studies. Rather, within the FSN EGM, • Regression discontinuity design we selected ten studies that have women’s empowerment • Controlled before-and-after studies, including interventions associated with the relevant outcomes. – Propensity-weighted multiple regression Data extraction and coding procedures Data extraction – Instrumental variable templates will be modified from 3ie’s repository cod - – Fixed effects models ing protocol and the coding protocols typically used for – Difference-in-differences (and any mathematical systematic reviews (Appendix 2). This includes biblio - equivalents) graphic, geographic information and substantive data, as – Matching techniques well as standardized methods information. In addition, two members of the team will extract data independently on interventions, outcomes, population (including gen- • Interrupted time series der/age disaggregation, when available), and effect sizes • Systematic reviews that include a quantitative or nar- corresponding to the outcomes indicated above, and any rative synthesis discrepancies will be reconciled. On interventions, out- comes, population (including gender/age disaggregation, Ex-post cost-effectiveness analyses will be included, when available), and effect sizes corresponding to the provided that they are associated with an included outcomes indicated above, and any discrepancies will be impact evaluation. reconciled. Qualitative information on barriers and facili- tators to implementation, sustainability and equity impli- Date, language, and form of publication cations, and other considerations for practitioners will All proceeding restrictions are from the EGM. also be extracted. • Date: 2000 Critical appraisal All the included quantitative impact • Language: English evaluations will be appraised by two independent mem- bers of the team using a critical appraisal tool (Appen- dix 1.1 and 1.2). Qualitative studies linked to included Search strategy impact evaluations will also be critically appraised. We will not perform any new searches for this REA. Instead, we will look at the ten studies of women’s Qualitative search and appraisal In addition to qualita- empowerment interventions identified in the Food Sys - 1 tive evidence from the included studies to assess factors tems and Nutrition ’living’ EGM, updated every four that determine or hinder the effectiveness of interven - months (last update December 2021). We specifically tions using a combination of qualitative synthesis, we will searched for interventions using women’s empowerment conduct a basic search on the programs in each of the ten within the food system implemented in low- and middle- papers, looking for the following relevant papers [11]: income countries. This EGM was developed through a systematic search and screening process equal to that of • A qualitative study collecting primary data using a systematic review. However, because interventions had mixed- methods or quantitative methods of data col- to function within the food system to be included, many lection and analysis and reporting some information women’s empowerment interventions, such as those on all of the following: the research question, proce- related to self-help groups broadly, were not included. dures for collecting data, procedures for analyzing Ultimately, the EGM includes ten evaluations of women’s data, and information on sampling and recruitment, empowerment interventions which considered outcomes including at least two sample characteristics. related to food availability, accessibility, and affordabil - • A descriptive quantitative study collecting primary ity and nutritional status. We will conduct additional data using quantitative methods of data collection and descriptive quantitative analysis and report some https:// gapma ps. 3ieim pact. org/ evide nce- maps/ food- syste ms- and- nutri tion- information on all of the following: the research evide nce- gap- map. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 22 of 52 question, procedures for collecting data, procedures available literature and a general synthesis of findings. for analyzing data, and information on sampling and Key information from each study, such as intervention recruitment, including at least two sample character- type, study design, country, outcomes, measurement istics. type, effect sizes, and confidence rating will be summa - • A process evaluation assessing whether an interven- rized in a table. Results from meta-analyses and their tion is being implemented as intended and what is associated forest plots will be presented when the data felt to be working well, and why. Process evaluations is sufficient. Qualitative information will be summa - may include the collection of qualitative and quanti- rized narratively in a practitioner’s brief to support pro- tative data from different stakeholders to cover sub - ject design and implementation. An updated theory of jective issues, such as perceptions of intervention change will be developed based on the combination of success or more objective issues, such as how an qualitative and quantitative data. intervention was operationalized. They might also be used to collect organizational information. Limitations Due to the rapid nature of this work, results should be interpreted more cautiously than those of a systematic While the identification of qualitative evidence is lim - review. Relying on the existing Food Systems and Nutri- ited to studies linked to the included impact evaluations, tion EGM may result in some relevant studies being the process of data extraction, critical appraisal, and evi- omitted from this evidence assessment. The small num - dence synthesis is independent. ber of studies which are expected to be retrieved through We will assess the quality of included qualitative stud- this REA may restrict the possibility of using meta-analy- ies, process evaluations, and descriptive quantitative sis and our ability to draw generalizable conclusions. studies using a mixed methods appraisal tool developed by CASP [12] and applied in Snilstveit et al. [46]. This Appendix 2: Data extraction tool tool is in Appendix 1.3. The meta-analysis conducted with the quantitative data will thus be complemented by a thematic synthesis utilizing the extracted qualitative Variable group Variable Label data. Publication info Record type Analytical approach for quantitative data If sufficient Record Title data is available, we will conduct meta-analysis to provide Record authors summary effect estimates. We will choose the appropri - Publication year ate formulae for effect size calculations in reference to, URL link and dependent upon, the data provided in included stud- Intervention and implementation Intervention ies. We will conduct random effects meta-analyses when considerations Intervention details we identify two or more studies that we assess to be suf- Unintended consequences ficiently similar. We will assess heterogeneity by calculat - Barriers and facilitators to imple- 2 2 ing the Q statistic, I , and τ to provide an estimate of the mentation amount of variability in the distribution of the true effect Evaluation considerations Study design sizes [23]. We will explore heterogeneity through the use Covariates of moderator analyses if the data allow. We will also test Outcomes for the presence of publication bias if at least 10 studies Sustainability and financial con- Sustainability comments siderations are included in the analysis. Cost effectiveness comments Other Other Data presentation Confidence rating (srr only) We will provide a narrative summary of the papers iden- tified. This will include an overall description of the B erretta et al. Agriculture & Food Security (2023) 12:13 Page 23 of 52 Quantitative data extraction tool Variable level Explanation Intervention codes Variable level Explanation Intervention description Use this open answer field to enter, in the author’s own words, a Study ID (DEP) This is the study ID from DEP (e.g., description of the intervention, up 17347) to a paragraph or so; more detail information will be preferred. Be Study ID (EPPI) This is the study ID from EPPI selective and concise with the reviewer. It should match the study excerpts being transcribed here ID from the Outcome Mapping as to ensure accurate and precise Sheet (e.g., 41504196) descriptions of the intervention. Estimate ID The estimate ID will provide a Include page numbers with every specific number for each effect size excerpt extracted. Do this for each extracted and should include the Treatment arm original study number, underscore, Intervention Record the intervention for the cor- then the unique ID number (e.g., responding effect size SC-SR1_1, SC-SR1_2 and so on) Exposure to intervention (in How long is the intervention expo- Evaluation design 0 = Experimental Design (e.g., RCT ), months) sure itself? 1 = Quasi-Experimental Design Evaluation period (in months) The total number of months How counterfactual is chosen Free text (e.g., random control trial, elapsed between the end of an propensity score matching, etc.)— intervention and the point at which Multiple codes are ok an outcome measure is taken post Analysis type for this effect size Free text, what type of analysis was intervention, or as a follow-up meas- used (Regression, 2SLS, ANCOVA, urement. If less than one month, etc.)- Multiple codes are ok use decimals (e.g., measurement Estimate type Type of data for this effect size: immediately after the intervention 1 = Continuous—means and SDs, end would be coded as 0, one week 2 = Continuous—mean differ - would be 0.25, etc.) ence and SD, 3 = Dichotomous Post-intervention or change from 0 = Post-intervention, 1 = Change outcome—proportions, 4 = Regres- baseline? from baseline sion data Outcome Codes Comparison 1 = No intervention (service delivery Outcome description Use this open answer field to as usual), 2 = Other intervention, enter, in the author’s own words, 3 = Pipeline (waitlist) control (still a description of the outcome. Be service delivery as usual) selective and concise with the Describe comparison group Free text, describe the comparison excerpts being transcribed here group as to ensure accurate and precise Country Select the countries in which the descriptions of the outcome. study was conducted (drop down Include page numbers with every menu). There is a multi-country excerpt extracted. Do this for each option for situations when there outcome are more than 15 countries, and no Outcome Record the outcome for the cor- disaggregated effects provided for responding effect size each country Eec ff t Size Data Extraction Subgroup Is this analysis of a subgroup? Reverse Sign (i.e., decrease is Record no if an increase is good, 0 = no, 1 = yes good) record yes if a decrease is good and If yes to subgroup, describe Free text, describe the subgroup the sign needs to be reversed if applicable (e.g., boys, girls). If no Unit of analysis What is the unit of analysis? UOA subgroup, type N/A for this effect size: 1 = Individual, Source Note the page number, table num- 2 = Household, 3 = Group (e.g., ber, column, and row you used to community organization), 4 = Vil- extract the data lage, 5 = Other, 6 = Not clear Treatment effect 1 = Intention to Treat (ITT ), 2 = Aver- Mean_t Outcome mean for the treatment age Treatment Eec ff t on the Treated group (ATET ), 3 = Average Treatment Eec ff t Sd_t Outcome standard deviation for (ATE) 4 = Local Average Treatment treatment group Eec ff t (LATE) Mean_c Outcome mean for the comparison group Berretta et al. Agriculture & Food Security (2023) 12:13 Page 24 of 52 Variable level Explanation Variable level Explanation Sd_c Outcome standard deviation for g THIS IS FOR SENIOR QUANT LEAD TO control group FILL OUT Mean_overall_diff Overall mean difference (treat - Var(d) THIS IS FOR SENIOR QUANT LEAD TO ment—control) FILL OUT Diff se Standard error of the overall mean se(d) THIS IS FOR SENIOR QUANT LEAD TO difference FILL OUT Diff _t t statistic of mean difference CI_l THIS IS FOR SENIOR QUANT LEAD TO FILL OUT Odds ratio Odds ratio reported in the study CI_u THIS IS FOR SENIOR QUANT LEAD TO OR_se Odds ratio standard error reported FILL OUT in the study Remove THIS IS FOR PROJECT MANAGER TO Risk ratio Risk ratio reported in study FILL OUT RR_se Risk ratio standard error Formula Used THIS IS FOR SENIOR QUANT LEAD TO Reg_coeff Report the regression coefficient of FILL OUT the treatment effect g_1 THIS IS FOR SENIOR QUANT LEAD TO Reg_SE Report the associated standard error FILL OUT of the regression coefficient g_rev THIS IS FOR SENIOR QUANT LEAD TO Reg_t Report the associated t statistic of FILL OUT the effect size (coefficient/SE) g THIS IS FOR SENIOR QUANT LEAD TO Exact p value Exact p value if given, if not, record FILL OUT as written in the manuscript (e.g., vi THIS IS FOR SENIOR QUANT LEAD TO p < 0.001, or p> 0.05) FILL OUT Clust_t Number of clusters—treatment wi THIS IS FOR SENIOR QUANT LEAD TO group FILL OUT Clust_c Number of clusters—control group ywi THIS IS FOR SENIOR QUANT LEAD TO Clust_T Number of clusters—total sample FILL OUT n_t Sample size—treatment group 95ci_lower THIS IS FOR SENIOR QUANT LEAD TO FILL OUT n_c Sample size—control group 95ci_upper THIS IS FOR SENIOR QUANT LEAD TO n_T Sample size—total sample FILL OUT Periods (1 if cross-sectional) Record how many periods of evalu- ation there are (e.g., cross section is cilow_3sf THIS IS FOR SENIOR QUANT LEAD TO 1, panel data with 3 measurements FILL OUT is 3) cihigh_3sf THIS IS FOR SENIOR QUANT LEAD TO Does the sample size need to be Often in panel data, models will FILL OUT corrected? report number of observations ci THIS IS FOR SENIOR QUANT LEAD TO rather than number of participants. FILL OUT In this column you will indicate wb_g THIS IS FOR SENIOR QUANT LEAD TO "Yes" if the sample size needs to be FILL OUT divided by the number of periods, and "No" if either it is cross-sectional Checked THIS IS FOR EFFECT SIZE RELIABILITY data, or if the authors have already CHECKER TO FILL OUT divided the number of observations ROB Category THIS IS FOR SENIOR QUANT LEAD by the number of panel assess- OR PM TO FILL OUT ments and thus no correction is necessary Treatment variable Record the treatment variable as Appendix 3: Critical appraisal tools written in the model (e.g., the vari- Appraisal of risk of bias for impact evaluations using RCT able name the author uses, such as ("Intervention x Time") designs Dataset Record if data comes from an identi- The following table provides a provisional tool to guide fied dataset the risk of bias assessment for quantitative impact Coder Record your name evaluations. Notes Record any notes important for the team n_T_revised THIS IS FOR SENIOR QUANT LEAD TO FILL OUT sp THIS IS FOR SENIOR QUANT LEAD TO FILL OUT d THIS IS FOR SENIOR QUANT LEAD TO FILL OUT B erretta et al. Agriculture & Food Security (2023) 12:13 Page 25 of 52 Provisional risk of bias assessment tool (RCT) General ID EPPI ID General Study first author Open answer General Time taken to com- Minutes plete assessment General Design type: What 1 = Randomized controlled trial – type of study design (RCT ) (random assignment to is used? households/individuals) or quasi- RCT 2 = Cluster-RC T (quasiRC T ) General Methods used for 1 = Statistical matching (PSM, CEM, – analysis: Which covariate matching) 2 = Difference- methods are used in-differences (DID) estimation to control for methods 3 = IV-regression (2stage selection bias and least squares or bivariate probit) confounding? 4 = Heckman selection model 5 = Fixed effects regression 6 = Covariate adjusted estimation 7 = Propensity-weighted regression 8 = Comparison of means= Other (please state) General Design and analysis Open answer Briefly describe the study design and method description analysis method undertaken by the authors General Study population Open answer Provide any details in the paper that describe how the study population was selected, covering: a) How is the population selected? what is the sampling strategy to recruit participants from that popula- tion into the study? b) What are the characteristics of that study participants? Was this a pilot program aimed at being scaled up? d) Were there spe- cific factors of success or failure in the implementation? General Type of comparison 1 = No intervention Indicate type of comparison group group (Service delivery as usual) 2 = Other intervention 3 = Pipeline (waitlist) control (still service deliv- ery as usual) General Type of comparison Open answer group (If other) General Ethical clearance Open answer Provide any details of ethical research clearances granted. Report unclear if this information is not available General Study registration Open answer Provide any details of study registra- tion, including registry IDs, etc. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 26 of 52 General ID EPPI ID 1: Assignment Assignment 1 = Yes, 2 = Probably a) The authors describe a random Score “Yes” if all criterion a), b), c) mechanism— mechanism: Was the Yes, 3 = Probably No, 4 component in sequence generation/ and d) are satisfied Assessment allocation or identi- = No, 8 = Unclear randomization method (e.g Score "Probably Yes" if only fication mechanism lottery, coin toss, criterion a) and b) are not satis- random or as good random number generator) and fied OR if only criteria c) is not as random? assignment is performed for all units satisfied at the start of the study centrally or Score “Unclear” if d) is not satis- using a method concealed from par- fied because no balance table is ticipants and intervention delivery reported b) If public lottery Score "Probably No" if d) is not is used for the sequence generation, satisfied because there is no authors provide detail on the exact balance table reported and settings and participants attending there is evidence suggesting a the lottery problem in the randomization, c) If a special such as baseline coefficients randomization procedure is used to in a diff-in-diff regression table ensure balance, it is well described are very different or sample size and justified given the study setting is too small for the procedure (stratification, pairwise matching, used (using stratification when unique random draw, multiple ran- there are less than two units for dom draws, etc.) each intervention and control d) A balance table is reported sug- group in each strata can lead to gesting that allocation was random imbalance) between all groups including sub- Score “No” if d) is not satis- group receiving different treatment fied because there are large within control or treatment groups imbalances concerning a large (if the comparison is relevant for this number of variables, providing assessment) evidence that the assignment was not random. If this is scored as no, use the NRS tool 1: Assignment Assignment justifi- Open answer Justification for coding decision mechanism— cation (Include a brief summary of justifica- Justification tion for rating, mentioning your response to all sub-questions, cite relevant pages) 2: Unit of analy- Unit of analysis: 1 = Yes 2 = No 3 = Not reported/ Score "Yes" if UoA = UoR OR if sis—Assess- Is unit of analysis unclear 4 = Not applicable UoA ≠ UoR and standard errors are ment in cluster alloca- clustered at the UoR level OR data is tion addressed collapsed to the UoR level in standard error Score "Not reported/unclear" if calculation? not enough information is provided on the way the standard errors were calculated or what the unit of analysis is Score "Not applicable" if it is not a cluster RCT Score "No" otherwise B erretta et al. Agriculture & Food Security (2023) 12:13 Page 27 of 52 General ID EPPI ID 3: Selection Selection bias Was 1 = Yes, 2 = Probably Score "Yes" if there is no attrition or bias-Assess- any differential Yes, 3 = Probably No, 4 attrition falls into the green zone and ment selection into or out = No, 8 = Unclear the study establishes that attrition is of the study (attri- randomly distributed (e.g., by present- tion bias) ade- ing balance by key characteristics quately resolved? across groups) AND if survey respond- ents were randomly sampled Score "Probably yes" if attrition falls into the green zone AND if survey respondents were randomly sampled Score "Unclear" if there is an attrition problem but no information provided on the relationship between attrition and treatment status, OR if there is not enough information on how the population surveyed was sampled Score "Probably no" if there is attrition which is likely to be related to the intervention OR is some indication that the survey respondents were purposely sampled in a way that might have led the sampling to be dif- ferent between treatment and control groups, or attrition falls into the yellow zone Score "No" if attrition falls into the red zone 3: Selection Selection bias justi- Open answer Justification for coding decision bias-Justifica- fication (Include a brief summary of justifica- tion tion for rating, mentioning your response to all sub-questions, cite relevant pages) 4: Confound- Confounding and 1 = Yes, 2 = Probably a) Baseline characteristics are similar in Score “Yes” if criterion a) and b) ing- Assess- group equivalence: Yes, 3 = Probably No, 4 magnitude; are satisfied; ment Was the method of = No, 8 = Unclear b) Unbalanced covariates at the indi- Score "Probably yes" if a) is not analysis executed vidual and cluster level are controlled satisfied but b) is satisfied and adequately to in adjusted analysis; c) Adjustments imbalances are small in magni- ensure compa- to the randomization were taken into tude OR if only a) is satisfied rability of groups account in the analysis (stratum fixed Score “Unclear” if no balance throughout the effects, pairwise matching variables)? table is provided or if imbal- study and prevent (Bruhn and McKenzie ances are controlled for but they confounding 2009) are very large in magnitude and assignment mechanism is not coded as "Yes" or "Probably yes" Score "Probably no" if a) and b) are not satisfied and the magni- tude of imbalances are small Score “No” if a) and b) are not satisfied and the magnitude of imbalances are large, and covari- ates are clear determinant of the outcomes 4: Confound- Confounding justi- Open answer Justification for coding decision ing-Justifica- fication (Include a brief summary of justifica- tion tion for rating, mentioning your response to all sub-questions, cite relevant pages) Berretta et al. Agriculture & Food Security (2023) 12:13 Page 28 of 52 General ID EPPI ID 5: Deviations Deviations from 1 = Yes, 2 = Probably Yes, 3 = Prob- a) There were no implementation Score “Yes” if criterion a), b), c) from intended intended interven- ably No, 4= No, 8 = Unclear issues that might have led the control and d) are satisfied; interven- tions: Spillovers, participants to receive the treatment Score "Probably yes" if there is tions—Assess- crossovers, and con- (implementer’s mistake) no obvious problem but there ment tamination: was the b) The intervention is unlikely to spillo- is no information reported on study adequately ver to comparisons (e.g., participants potential risks related to spill protected against and non-participants are geographi- overs, contamination, or survey spillovers, crosso- cally and/or socially separated from effects in the control group OR if vers, and contami- one another and general equilibrium there were issues with spillovers nation? effects are not likely) or the potential but they were controlled for or effects of spill overs were measured measured (e.g., variation in the % of unit within a Score “Unclear” if spillovers, cluster receiving the treatment) crossovers, survey effects and/ There is no risk of contamination by or contamination are not external programs: the treatment and addressed clearly comparisons are isolated from other Score "Probably no" if any of the interventions which might explain criterion a), b), c) or d) are not changes in outcomes satisfied but the scale of the d) There is nothing in the surveys issue is not clear that might have given the control Score “No” if any of the criterion participants an idea of what the other a), b), c) or d) are not satisfied group might receive OR they did but and happened at a large scale in there is no risk that this has changed the study their behaviors; AND the survey process did not reveal information to the control group that they did not have before (e.g., the study aims to measure increase in take up of a service or product that participants might not know about) Authors might put something in place in the design of the study that allows to control for that survey effect (e.g., a pure control with no monitoring except baseline end line) 5: Deviations Deviations justifica- Open answer Justification for coding decision from intended tion (Include a brief interven- summary of justification for rating, tions—Justifi- mentioning your response to all sub- cation questions, cite relevant pages) For example, intervention groups are geographically separated, authors use intention to treat estimation or instrumental variables to account for non-adherence, and survey questions are not likely to expose individuals in the control group to information about desirable behaviors (‘survey effects’) B erretta et al. Agriculture & Food Security (2023) 12:13 Page 29 of 52 General ID EPPI ID 6. Performance Performance bias: 1 = Yes, 2 = Probably Yes, 3 = Prob- a) The authors state explicitly that the Score “Yes” if either criterion a) or bias -Assess- Was the process ably No, 4 process of monitoring the interven- b) are satisfied; ment of monitoring = No, 8 = Unclear tion and outcome measurement is Score "Probably yes" if the study individuals unlikely blinded and conducted in the same is based on data collected dur- to introduce moti- frequency for treatment and control ing a trial and there is no obvi- vation bias among groups, or argue convincingly why it is ous issue with the monitoring participants? not likely that being monitored could processes, but authors do not affect the performance of participants mention potential risks in treatment and comparison groups Score “Unclear” if it is not clear in different ways (such as resulting in whether the authors use Hawthorne or John Henry effects) an appropriate method to b) The outcome is based on data prevent Hawthorne and John collected in the context of a survey, Henry Eec ff ts (e.g., blinding of and not associated with a particular outcomes and, or enumera- intervention trial, or data are collected tors, other methods to ensure from administrative records or in the consistent monitoring across context of a retrospective (ex post) groups) evaluation Hawthorne effects may result where participants know that they are being observed and John Henry Eec ff ts may result from participant knowledge of being compared Score "Probably no" if there was imbalance in the frequency of monitoring in intervention groups, which might have influ- enced participants’ behaviors Score "No" if neither criterion a) or b) are satisfied 6. Performance Performance bias Open answer Justification for coding decision bias-Justifica- justification (Include a brief summary of justifica- tion tion for rating, mentioning your response to all sub-questions, cite relevant pages) 7. Outcome Outcome measure- 1 = Yes, 2 = Probably a) Outcome assessors are blinded, or Score “Yes” if criterion a), b), c) measurement ment bias: Was the Yes, 3 = Probably No, 4 the outcome measures are not likely and d) are satisfied: bias - study free from = No, 8 = Unclear to be biased by their judgment Score "Probably yes" if there Assessment biases in outcome b) For self-reported outcomes: is a small risk related to any of measurement? respondents in the intervention group a), b), c) or d) and there is no are not more likely to have accurate more information provided to answers due to recall bias; justify the absence of bias OR if c) For self-reported outcomes: there was a high risk of bias, but respondents do not have incentives to authors have either controlled it over/under report something related in their design or measured to their performance or actions, OR it with a placebo outcome researchers put in place mechanisms Score “Unclear” if it there is a to reduce the risk of reporting bias high risk related to any of a), b), (researchers not strongly involved in c) or d) and there is no more the implementation of the program information provided to justify and it is clear that their answers to the absence of bias the survey will not affect what they Score "Probably no" if there are receive in future) OR authors high risk related to a), b), c) or d) have measured the risks of bias and it is clear that authors were through not able to control for this bias falsification tests or measuring the Score “No” if there is evidence effect on placebo outcomes in cases of bias where there was a risk of reporting bias d) Timing issue: the data collec- tion period did not differ between intervention and comparison group; the baseline data is not likely to be affected by the beginning of the inter - vention or affects a small percentage of the study participants Berretta et al. Agriculture & Food Security (2023) 12:13 Page 30 of 52 General ID EPPI ID 7. Outcome Outcome measure- Open answer Justification for coding decision measurement ment justification (Include a brief summary of justifica- bias-Justifica- tion for rating, mentioning your tion response to all sub-questions, cite relevant pages) 8. Reporting Analysis reporting: 1 = Yes, 2 = Probably a) A pre-analysis plan or trial protocol Score "Yes" if all the criterion bias-Assess- Was the study free Yes, 3 = Probably No, 4 is published and referred to or the trial a), b), c), d), and e) are satisfied; ment from selective analy- = No, 8 = Unclear was preregistered, or the outcomes Score "Probably yes" if all the sis reporting? were preregistered; conditions are met except a), or b) Authors report results correspond- if all the conditions are met but ing to the outcomes announced there is some element missing in the method section (there is no that could have helped under- outcome reporting bias); stand the results c) Authors report results of unadjusted better (e); analysis and intention to treat (ITT ) Score "Unclear" if there is not estimation, alongside any adjusted enough information to deter- and treatment-on-the treated/com- mine that there is an analysis plier average-causal effects analysis.) missing; Score "Probably no" if d) Authors use the appropriate analy- any of the criterion b), c) or d) sis method (use baseline data when are not satisfied; Score "No" if available), and different treatment any of the criterion b), c) or d) arms are are not satisfied and there is differentiated in the analysis evidence that the analysis results e) Authors have reported all the analy- would be different because sis which could help understand the large imbalances were not con- results and no other bias is assessed as trolled for, compliance was very unclear due to the low and ITT estimation was not lack of an important analysis (e.g., a reported or different treatment balance table or a subgroup analysis) arms were pooled 8. Reporting Analysis reporting Open answer Justification for coding decision bias-Justifica- justification (Include a brief summary of justifica- tion tion for rating, mentioning your response to all sub-questions, cite relevant pages) 9. Other bias- Other risks of bias 1 = Yes, 4 = No Assessment Is the study free from other sources of bias? 9. Other bias- Other bias justifica- Open answer Justification for coding decision Justification tion 10. Blinding- Blinding of partici- 1 = Yes 2 = No 8 = unclear If there is no information, code NO. If observers- pants? 9 = N/A there is information but it is ambigu- Assessment ous, code UNCLEAR 10. Blinding— Blinding of outcome 1 = Yes 2 = No 8 = unclear If there is no information, code NO. If observers— assessors? 9 = N/A there is information but it is ambigu- Assessment ous, code UNCLEAR 10. Blinding- Blinding of data 1 = Yes 2 = No 8 = unclear If there is no information, code NO. If analysts- analysts? 9 = N/A there is information but it is ambigu- Assessment ous, code UNCLEAR 10. Blinding- Method(s) used to Open answer (including describe Describe method(s) used to blind method(s) blind method of placebo control) No 9 = N/A 11. External External validity Open answer a) What do authors say about external Include all information that can validity-Assess- validity? help assess the external validity ment of the results B erretta et al. Agriculture & Food Security (2023) 12:13 Page 31 of 52 Summary of justification for rating, mentioning your response to all sub-questions, cite relevant pages). Appraisal of risk of bias for impact evaluations using quasi‑experimental designs Risk of bias assessment tool (QED) Code Question Coding Criteria Decision‑rules General ID EPPI ID General Time taken to complete Minutes assessment General Study first author Open answer General Outcomes assessed Open answer General Study design: What type of 1 = Natural experi- study design is used? ment: randomized or as-if randomized 2 = Natural experi- ment: regression discontinuity (RD) 3 = CBA (non-rand- omized assignment with treat- ment and contempo- raneous comparison group, baseline, and end line data col- lection) – individual repeated measure- ment 4 = CBA pseudo panel (repeated measurement for groups but different individuals) 5 = Interrupted time series (with or without contemporaneous control group) 6 = Panel data, but no baseline (pre-test) 7 = Comparison group with end line data only General Methods used for analysis: 1 = Statistical – Which methods are used matching (PSM, CEM, to control for selection bias covariate matching) and confounding? 2 = Difference-in- differences (DID) estimation methods 3 = IV-regression (2-stage least squares or bivariate probit) 4 = Heckman selection model 5 = Fixed effects regression6 = Covari- ate adjusted estima- tion 7 = Propensity- weighted regression 8= Comparison of means = Other (please state) Berretta et al. Agriculture & Food Security (2023) 12:13 Page 32 of 52 Code Question Coding Criteria Decision‑rules General Study population Open answer Provide any details in the paper that describe how the study population was selected, covering: a) How is the population selected? what is the sampling strategy to recruit participants from that population into the study? b) What are the characteristics of that study participants? c) Was this a pilot program aimed at being scaled up? d) Were there specific factors of success or failure in the implementation? General Ethical clearance Open answer Provide any details of ethical research clear- ances granted. Report unclear if this informa- tion is not available 1: Selec- 1—Mechanism of assign- 1 = Yes, 2 = Probably tion bias- ment: was the allocation or Yes, Assess- identification mechanism 3 = Probably No, ment able to 4 = No, control for selection bias? 8 = Unclear 1: Selec- For regression discontinuity Open answer a) Allocation is made based on a predeter- Score “Yes” if criteria a), b), c) are all tion designs mined discontinuity on a continuous variable satisfied bias-Jus- (Regression discontinuity design) and blinded Score "Probably Yes" if there are tification to participants or; minor differences in between b) if not blinded, individuals reasonably cannot both sides of the cut-off point but affect the assignment variable in response to authors convincingly argue that knowledge of the participation decision rule; the differences are unlikely to affect c) and the sample size immediately at both the outcome, OR individuals are sides of the cutoff point is sufficiently large to not blinded and there are low risk equate groups on average of them affecting the assignment, but the authors do not mention it Score “Unclear” if it is unclear whether participants can affect it in response to knowledge of the allocation mechanism Score "Probably No" if there are differences between individuals on both sides of the cut-off point, and there are doubts that the differences are due to individuals altering the assignment OR the participants are blinded but there is evidence that the decisions that determined the discontinuity is based on differences between the two groups or differences in time Score “No” if the sample size is not sufficient OR there is evidence that participants altered the assignment variable prior to assignment. If the research has serious concerns with the validity of the assignment process or the group equivalence completely fails, we recommend assessing risk of bias of the study using the relevant questions for the appropriate methods of analy- sis (cross-sectional regressions, difference-in-difference, etc.) rather than the RDDs questions B erretta et al. Agriculture & Food Security (2023) 12:13 Page 33 of 52 Code Question Coding Criteria Decision‑rules 1: Selec- For assignment-based Open answer a) Participants and non-participants are either Score “Yes” if a) or b) and c) are tion nonrandomised program matched based on all relevant characteristics satisfied bias-Jus- placement and self-selec- explaining participation and outcomes, or; Score "Probably yes" if a) or b) are tification tion (studies using a match- b) all relevant characteristics are accounted addressed for but there is some ing strategy or regression for.** and the data set used contains relevant doubt related to c), OR authors analysis, excluding variables that are measured in a relevant way combined statistical matching and IV ) (i.e., they were not collected for a different difference-in-difference to cope purpose initially and therefore are good proxy with unobservable differences, OR for some characteristics) they only did statistical match- **Accounting for and matching on all relevant ing and there were clear rules for characteristics is usually only feasible when the selection into the program (no program allocation rule is known and there are self-selection) no errors of targeting. It is unlikely that studies Score “Unclear ” if · it is not clear not based on randomization or regression whether all relevant characteristics discontinuity can score “YES” on this criterion. (only relevant time-varying char- There are different ways in which covariates acteristics in the case of panel data can be taken into account. Differences across regressions) are controlled groups in observable characteristics can be Score "Probably no" if only a statisti- considered as covariates in the framework of cal matching was done and there a regression analysis or can be assessed by was self-selection into the program testing equality of means between groups. Score “No” if relevant characteristics Differences in unobservable characteristics are omitted from the analysis can be taken into account using instrumental variables (see also question 1.d) or proxy vari- ables in the framework of a regression analysis, or using a fixed effects or difference-in-differ - ences model if the only characteristics which are unobserved are time-invariant 1: Selec- For identification based on Open answer Score “Yes” if an appropriate instrumental vari- tion an instrumental variable (IV able is used which is exogenously generated: bias-Jus- estimation) for example, due to a ‘natural’ experiment or tification random allocation Score "Probably yes" if there is less evidence (no balance table showing differences between the intervention and comparison group) Score “Unclear” if the exogeneity of the instru- ment is unclear (both externally as well as why the variable should not enter by itself in the outcome equation) Score "Probably no" if there is evidence that enrolment in the program is correlated with a variable that might also influence outcome and on the instrumental variable Score “No” if it is clear that the instrument is not exogenous and affect the outcome through other channels than the program 2: Con- 2—Group equivalence: was 1 = Yes, 2 = Probably found- the method of analysis exe- Yes, ing- cuted adequately to ensure 3 = Probably No, Assess- comparability of groups 4 = No, 8 = Unclear ment throughout the study and prevent confounding? Berretta et al. Agriculture & Food Security (2023) 12:13 Page 34 of 52 Code Question Coding Criteria Decision‑rules 2: Con- For regression discontinuity Open answer a) The interval for selection of treatment and Score "Yes if criterion a), b), c) and found- design control group is reasonably small OR authors d) are addressed ing-Justi- have weighted the matches on their distance Score "Probably yes" if b) is not fication to the cutoff point; and addressed but c) is b) the mean of the covariates of the individu- addressed and differences in als immediately at both sides of the cut-off means are not large point (selected sample of participants and Score “Unclear” if insufficient details non-participants) are overall not statistically are provided on controls; or if different based on t-test or insufficient details are provided on ANOVA for equality of means; cluster controls c) Significant differences in covariates of the Score "Probably no" if b) is not individuals have been controlled in multi- addressed (absence of a difference variate analysis; and for cluster assignment, test or balance table) and there are authors control for external cluster-level doubt regarding the continuity on factors that might confound the impact of the both sides of the cut-off point (a) program Score “No” otherwise 2: Con- For non-randomized trials Open answer a) The authors use a difference-in-differences Score "Yes, if a, b, c, d (if relevant) is found- using difference-in-differ - (or fixed effects) multivariate estimation addressed and baseline imbalances ing- Justi- ences methods of analysis method; between groups were relatively fication b) the authors control for a comprehensive low OR the method was combined set of individual time-varying characteristics, by a statistical matching and for cluster assignment, authors control Score "Probably yes" if all possible for external cluster-level factors that might variables are controlled for and confound the impact of the program**; the selection into the program c) and the attrition rate is sufficiently low and was done according to clear rules, similar in treatment and control, or the study but baseline imbalances between assesses that dropouts are random draws from groups were very large the sample (for example, by examining correla- Score “Unclear” if insufficient details tion with determinants of outcomes, in both are provided; or if insufficient treatment and comparison groups); details are provided on cluster **Knowing controls allocation rules for the program – or even Score "Probably no" if some time- whether the non-participants were individuals varying characteristics are not that refused to participate in the program, as controlled for and the program was opposed to individuals that were not given self-selected by the intervention the opportunity to participate in the program groups – can help in the assessment of whether the Score “No” if any of the criterion is covariates accounted for in the regression not addressed capture all the relevant characteristics that explain differences between treatment and comparison groups B erretta et al. Agriculture & Food Security (2023) 12:13 Page 35 of 52 Code Question Coding Criteria Decision‑rules 2: Con- For statistical matching Open answer a) Matching is either on baseline characteristics Score "Yes, if a, b, c, and d (if rel- found- studies including pro- or time-invariant characteristics which cannot evant) are addressed ing-Justi- pensity scores (PSM) and be affected by participation in the program; Score "Probably yes" if the selection fication covariate matching** and the variables used to match are relevant into the program was done accord- **Matching strategies are (for example, demographic and socio-eco- ing to clear rules, which are used sometimes complemented nomic factors) to explain both participation for the matching but there are with difference-indifference and the outcome (so that there can be no slight imbalances remaining after only uses in the estima- evident differences across groups in variables matching tion the common support that might explain outcomes); and, for cluster Score “Unclear” if relevant variables region of the sample size, assignment, authors control for external are not included in the matching reducing the likelihood of cluster-level factors that might confound the equation, or if matching is based existence of time variant impact of the program on characteristics collected at end unobservable differences b) in addition, for PSM Rosenbaum’s test line; or if insufficient details are across groups affecting suggests the results are not sensitive to the provided on cluster controls outcome of interest and existence of hidden bias; and, Score "Probably no" if the program removing biases aris- c) with the exception of Kernel matching, the was self-selected by the interven- ing from time-invariant means of the individual covariates are equated tion groups or participants OR if unobservable characteris- for treatment and comparison groups after the selection into the program was tics, regression estimation matching; done according to clear rules but methods. This combination d) different matching methods including vary- there is no baseline data available approach is superior since it ing sample sizes gelds the same results and to match the participants or groups authors consider the use of control observa- on tions multiple times against the same treat- Score “No” if matching was done ment in their standard error calculation based on variables that are likely to be affected by the program or any other scenario that affect a), b) c) or d) 2: Con- For regression-based stud- Open answer a) The study controls for relevant confounders Score "Yes if a, b, c and d are found- ies using cross-sectional that may be correlated with both participa- addressed ing-Justi- data (excluding IV ) tion and explain outcomes (for example, Score "Probably yes" if all criteria fication demographic and socio-economic factors at are addressed but authors did not individual and community report the Hausman test level) using multivariate methods with appro- (b) priate proxies for unobservable covariates, and, Score “Unclear” if relevant for cluster assignment, authors control particu- confounders are controlled but larly for external cluster-level factors that might appropriate proxy variables or confound the impact of the program; statistical tests are not reported; or b) and a Hausman test with an appropriate if insufficient details are provided instrument suggests there is no evidence of on cluster controls endogeneity**; Score "Probably no" if any of the c) and none of the covariate controls can be criterion other than b) is not affected by participation; addressed d) and either, only those observations in the Score “No" if none of the criterion region of common support for participants are addressed and non-participants in terms of covariates are used, or the distributions of covariates are balanced for the entire sample population across groups; **The Hausman test explores endogeneity in the framework of regression by comparing whether the OLS and the IV approaches geld significantly different estimations. However, it plays a different role in the different meth- ods of analysis. While in the OLS regression framework the Hausman test mainly explores endogeneity and therefore is related with the validity of the method, in IV approaches it explores whether the author has chosen the best available strategy for addressing causal attribution (since in the absence of endogene- ity OLS gelds more precise estimators) and therefore is more related with analysis report- ing bias Berretta et al. Agriculture & Food Security (2023) 12:13 Page 36 of 52 Code Question Coding Criteria Decision‑rules 2: Con- For identification based on Open answer a) The instrumenting equation is significant Score "Yes, if a, b, c, d (if relevant) is found- an instrumental variable (IV at the level of F ≥ 10 (or if an F test is not addressed ing-Justi- estimation) reported, the authors report and assess Score "Probably yes" if one of the fication whether the R-squared (goodness of fit) of the tests required for criterion a) or participation equation is sufficient for appro - b) is not reported but the other priate identification); b) the identified instru- is, and the rest of the criterion are ments are individually significant (p ≤ 0.01); for addressed, and the instrument is Heckman models, the identifiers are reported convincing and significant (p ≤ 0.05); Score “UNCLEAR” if relevant con- c) where at least two instruments are used, founders are the authors report on an over-identifying test controlled for but appropriate (p ≤ 0.05 is required to reject the null hypoth- statistical tests are not reported; or esis); and none of the covariate controls can if insufficient details are provided be affected by participation and the study, and on cluster controls authors convincingly assesses qualitatively why Score "Probably no" if exogeneity the instrument only affects the outcome via of the instrument is not convinc- participation. If the instrument is the random ing and appropriate tests are not assignment of the treatment, the reviewer reported should also assess the quality and success of Score “No” otherwise if any of the the randomization procedure in part a) tests required for criterion a), b) or d) and, for cluster assignment, authors c) are reported and not satisfied particularly control for external cluster-level factors that might confound the impact of the program (for example, weather, infrastructure, community fixed effects, and so forth) through multivariable analysis 3: Perfor- 3—Performance bias: 1 = Yes, 2 = Probably a) For data collected in the context of a Score “Yes” if either criterion a) or b) mance was the process of being Yes, particular are satisfied; bias- observed free from motiva- 3 = Probably No, intervention trial (randomized or nonran- Score "Probably yes" if the study Assess- tion bias? 4 = No, domised assignment), the authors state is based on survey data collected ment 8 = Unclear explicitly that the process of monitoring the during a trial and there is no intervention and outcome measurement is obvious issue with the monitoring blinded, or argue convincingly why it processes, but authors do not men- is not likely that being monitored could affect tion potential risks the performance of participants in treatment Score “Unclear” if it is not clear and comparison groups in different ways whether the authors use an (such as resulting in Hawthorne or John Henry appropriate method to prevent effects) Hawthorne and John Henry Eec ff ts b) The study is based on data collected in the (e.g., blinding of outcomes and, context of a survey, and not associated with a or enumerators, other methods particular to ensure consistent monitoring intervention trial, or data are collected from across groups) administrative records or in the context of a Hawthorne effects may result retrospective (ex post) evaluation where participants know that they are being observed and John Henry Eec ff ts may result from par - ticipant knowledge of being com- pareScore "Probably no" if there was imbalance in the frequency of monitoring in intervention groups, which might have influenced participants’ behaviors Score "No" if both criterion a) and b) are not satisfied 3: Perfor- Performance bias-Justifi- Open answer Justification for coding decision (Include a brief mance cation summary of justification for rating, mention- bias-Jus- ing your response to all sub-questions, cite tification relevant pages) B erretta et al. Agriculture & Food Security (2023) 12:13 Page 37 of 52 Code Question Coding Criteria Decision‑rules 4: Spillo- 4—Spillovers, crossovers, 1 = Yes, 2 = Probably a) There were no implementation issues that Score “Yes” if criterion a), b), c) and vers, and contamination: was the Yes, might have led the control participants to d) are satisfied; crosso- study adequately protected 3 = Probably No, receive the treatment (implementer’s mistake) Score "Probably yes" if there is no vers, and against spillovers, crosso- 4 = No, The intervention is unlikely to spillover to obvious problem but there is no contam- vers, and contamination? 8 = Unclear comparisons (e.g., participants and non- information reported on potential ination- participants are geographically and/or socially risks related to spill overs, Assess- separated from one another and general equi- contamination, or survey effects in ment librium effects are not likely) or the potential the control group OR if there were effects of spill overs were measured (e.g., vari- issues with spillovers but they were ation in the % of unit within a cluster receiving controlled for or measured the treatment) Score “Unclear” if spillovers, crosso- c) There is no risk of contamination by external vers, survey effects and/or contami- programs: the treatment and comparisons are nation are not addressed clearly isolated from other interventions which might Score "Probably no" if any of the explain changes in outcomes criterion a), b), c) or d) are not b) There is nothing in the surveys that might satisfied but the scale of the issue have given the control participants an idea of is not clear what the other group might receive OR they Score “No” if any of the criterion did but there is no risk that this has changed a), b), c) or d) are not satisfied and their behaviors; AND the survey process did happened at a large scale in the not reveal information to the control group study that they did not have before (e.g., the study aims to measure increase in take up of a service or product that participants might not know about) Authors might put something in place in the design of the study that allows to control for that survey effect (e.g., a pure control with no monitoring except baseline end line) 4: Spillo- Spillovers, crossovers, and Open answer Justification for coding decision (Include a brief vers, contamination-Justification summary of justification for rating, mention- crosso- ing your response to all sub-questions, cite vers, and relevant pages) contam- ination- Justifica- tion 5: 5—Outcome measurement 1 = Yes, 2 = Probably a) Outcome assessors are blinded, or the Score “Yes” if criterion a), b), c) and Outcome bias Yes, outcome measures are not likely to be biased d) are satisfied: measure- 3 = Probably No, by their judgment Score "Probably yes" if there is ment 4 = No, b) For self-reported outcomes: respondents in a small risk related to any of a), bias- 8 = Unclear the intervention group are not more likely to b), c) or d) and there is no more Assess- have accurate answers due to recall bias; information provided to justify the ment c) For self-reported outcomes: absence of bias OR if there was a respondents do not have incentives to over/ high risk of bias, but authors have under report something related to their perfor- either controlled it in their design mance or actions, OR researchers put in place or measured mechanisms to reduce the risk of reporting it with a placebo outcome bias (researchers not strongly involved in the Score “Unclear” if it there is a high implementation of the program and it is clear risk related to any of a), b), c) or d) that their answers to the survey will not affect and there is no more information what they receive in future) OR authors have provided to justify the absence measured the risks of bias through falsifica- of bias tion tests or measuring the effect on placebo Score "Probably no" if there are outcomes in cases where there was a risk of high risk related to a), b), c) or d) reporting bias and it is clear that authors were not d) Timing issue: the data collection able to control for this bias period did not differ between intervention and Score “No” if there is evidence of comparison group; the baseline data is not bias likely to be affected by the beginning of the intervention or affects a small percentage of the study participants Berretta et al. Agriculture & Food Security (2023) 12:13 Page 38 of 52 Code Question Coding Criteria Decision‑rules 5: Outcome measurement Open answer Justification for coding decision (Include a brief Outcome bias-Justification summary of justification for rating, mention- measure- ing your response to all sub-questions, cite ment relevant pages) bias-Jus- tification 6: Report- 6—Selective analysis 1 = Yes, 2 = Probably a) a pre-analysis plan is published, especially Score “Yes” if a), b), c) and d) are ing bias- reporting: was the study Yes, for prospective NRS, but it should also be for satisfied OR if a) is not met and it is Assess- free from selective analysis 3 = Probably No, retrospective studies b) authors use ‘common’ a retrospective NRS ment reporting? 4 = No, methods of estimation (i.e., credible analysis Score "Probably Yes" if authors 8 = Unclear method to deal with attribution given the data combined methods and reported available); c) There is no evidence that out- relevant tests (d) only for one comes were selectively reported (e.g., results method OR if all the criteria are met for all relevant outcomes in the methods sec- except for a) and it is a prospec- tion are reported in the results section); tive NRS d) Requirements for specific methods of Score "Unclear" if intended out- analysis: comes not specified in the paper - For PSM and covariate matching: (a) Where OR if any of the requirements for d) over 10% are not reported of participants fail to be matched, sensitivity Score "Probably No" if b) is analysis is used to re-estimate results using addressed, but authors did not different matching methods (Kernel Matching present results for all outcomes techniques); (b) For matching with replace- announced in the method section ment, no single observation in the control OR did not meet requirement d) group is matched with a large number of although reported observations in the treatment group.—For IV Score “No” if authors use uncom- (including Heckman) models, (a) The authors mon or less rigorous estimation test and report the results of a Hausman test methods such as failure to conduct for exogeneity (p ≤ 0.05 is required to reject multivariate analysis for outcomes the null hypothesis of exogeneity); (b) the coef- equations OR if some important ficient of the selectivity correction term (Rho) outcomes are subsequently omit- is significantly different from zero (P < 0.05) ted from the results or the signifi- (Heckman approach) cance and magnitude of important - For studies using multivariate regression anal- outcomes was not assessed ysis, authors conduct appropriate specification tests (e.g., testing robustness of results to the inclusion of additional variables, or (very rare) reporting results of multicollinearity test, etc.) 6: Report- Analysis reporting bias— Open answer Justification for coding decision (Include a brief ing Justification summary of justification for rating, mention- bias-Jus- ing your response to all sub-questions, cite tification relevant pages) 7: Other 7—Other risks of bias: Is 1 = Yes, 4 = No Score “Yes” if the reported results do not sug- bias- the study free from other gest any other sources of bias. Score “No” if Assess- sources of bias? other potential threats to validity are present, ment and note these here (e.g., coherence of results, survey instruments used are not reported) 7: Other Other risks of bias-Justifi- Open answer Justification for coding decision (Include a brief bias-Jus- cation summary of justification for rating, mention- tification ing your response to all sub-questions, cite relevant pages) 8: 8—External validity Open answer Open answer- what do authors say about External external validity if anything? validity B erretta et al. Agriculture & Food Security (2023) 12:13 Page 39 of 52 Qualitative analysis tool Study type Methodological appraisal criteria Response Yes No Comment Screening questions: Configurative assessment: assessing ‘fatal flaws’ • Study reports primary data and applied methods (Dixon-Woods 2005) • Study states clear research questions and objectives Configurative ‘fatal flaws’ • Study states clear research design, which is appropriate to address the stated research ques- based on Pawson (2003) tion and objectives (Purposivity) TAPUS framework • The findings of the study are based on collected data, which justify the knowledge claims (Accuracy) Screening question based on abstract and/or superficial reading of full text: Further appraisal is not feasible or appro ‑ priate when the answer is ‘No’ to any of the above screening questions! 1. Qualitative and descrip- I. RESEARCH IS DEFENSIBLE IN DESIGN (providing a research strategy that addresses the tive quantitative, and question) process evaluations Appraisal indicators: Bullet Is the research design clearly specified and appropriate for aims and objectives of the research? Consider whether i. there is a discussion of the rationale for the study design ii. the research question is clear, and suited to the inquiry iii. there are convincing arguments for different features of the study design iv. limitations of the research design and implications for the research evidence are discussed Defensi- Arguable Critical Not Worth to con- ble defen- tinue: sible II. RESEARCH FEATURES AN APPROPRIATE SAMPLE (following an adequate strategy for selection of participants) Appraisal indicators: Consider whether i. there is a description of study location and how/why it was chosen ii. the researcher has explained how the participants were selected iii. the selected participants were appropriate to collect rich and relevant data iv. reasons are given why potential participants chose not take part in study Appropriate sample Functional sample Critical sample Flawed sample Worth to continue: III. RESEARCH IS RIGOROUS IN CONDUCT (Providing a systematic and transparent account of the research process) Appraisal indicators: Consider whether i. researchers provide a clear account/description of the process by which data was collected (e.g., for interview method, is there an indication of how interviews were conducted? /procedures for collection or recording of data?) ii. researchers demonstrate that data collection targeted depth, detail, and richness of information (e.g., interview/observation schedule) iii. there is evidence of how descriptive analytical categories, classes, labels, etc. have been gener- ated and used iv. presentation of data distinguishes clearly between the data, the analytical frame used, and the interpretation v. methods were modified during the study; and if so, has the researcher explained how and why? Rigorous Considerate conduct Critical Flawed conduct Worth to continue: conduct conduct IV. RESEARCH FINDINGS ARE CREDIBLE IN CLAIM/BASED ON DATA (Providing well-founded and plausible arguments based on the evidence generated) Appraisal indicators: Consider whether i. there is a clear description of the form of the original data ii. sufficient amount of data is presented to support interpretations and findings/conclusions Berretta et al. Agriculture & Food Security (2023) 12:13 Page 40 of 52 Study type Methodological appraisal criteria Response Yes No Comment iii. the researchers explain how the data presented were selected from the original sample to feed into the analysis process (i.e., commentary and cited data relate; there is an analytical context to cited data, not simply repeated description; is there an account of frequency of presented data?) iv. there is a clear and transparent link between data, interpretation, and findings/conclusion? v. there is evidence (of attempts) to give attention to negative cases/outliers, etc.? Credible Arguable Doubtful claims Not credible If findings not credible, can claims claims data still be used? V. REASEARCH ATTENDS TO CONTEXTS (Describing the contexts and particulars of the study) Appraisal indicators: Consider whether i. there is an adequate description of the contexts of data sources and how they are retained and portrayed? ii. participants’ perspectives/observations are placed in personal contexts iii. appropriate consideration is given to how findings relate to the contexts (how findings are influenced by or influence the context) iv. the study makes any claims (implicit or explicit) that infer generalization (if yes, comment on appropriateness) Context Context considered Context men- No context attention central tioned` VI. RESEARCH IS REFLECTIVE (Assessing what factors might have shaped the form and output of research) Appraisal indicators: Consider whether i. appropriate consideration is given to how findings relate to researchers’ influence/own role during analysis and selection of data for presentation ii. researchers have attempted to validate the credibility of findings (e.g., triangulation, respondent validation, more than one analyst) iii. researchers explain their reaction to critical events that occurred during the study iv. researchers discuss ideological perspectives/values/philosophies and their impact on the meth- odological or other substantive content of the research (implicit/explicit) Reflection Consideration Acknowledgment Unreflective research NB: Can override previous exclusion! B erretta et al. Agriculture & Food Security (2023) 12:13 Page 41 of 52 Study type Methodological appraisal criteria Response Yes No Comment OVERALL CRITICAL APPRAISAL DECISION Decision rule: – a single critical appraisal judgment in any of the 6 appraisal domains leads to a criti- cal overall judgment – 2 or more high critical appraisal judgements in any of the 6 appraisal domains lead to an overall high risk of bias / low-quality rating – 2 or more moderate critical appraisal judge- ments in any of the 6 appraisal domains lead to an overall moderate risk of bias / moderate quality rating – which means that for a study to be rated of low risk of bias / high quality at least 5 appraisal domains need be rated as of low critical appraisal High-quality Moderate-quality Low-quality Critical Empirical research Empirical research (study generates new evidence Empirical research (study generates new evidence quality (study generates new relevant to the review question and complies with relevant to the review question and complies with Empirical evidence relevant to reasonable methodological criteria to ensure reliability minimum methodological criteria to ensure reliability research the review question and empirical grounding of the evidence) and empirical grounding of the evidence) (the and complies with all evidence methodological criteria generated to ensure reliability and by the empirical grounding of study does the evidence) not comply with mini- mum meth- odological criteria to ensure reli- ability and empirical ground- ing of the evidence) Sources used in this section (in alphabetical order); Campbell et al. [9]; CASP (2006); CRD (2009); Dixon-Woods et al. (2004); Dixon- Woods et al. (2006); Greenhalgh and Brown (2014); Harden et al. (2004); Harden et al. (2009); Harden and Gough (2012); Mays and Pope (1995); Pluye et al. (2011); Spencer et al. 2006; Thomas et al. (2003); SCIE (2010) Berretta et al. Agriculture & Food Security (2023) 12:13 Page 42 of 52 Study type Methodological appraisal Response criteria Yes No Comment /confidence judgment 2. Mixed-methods I. RESEARCH INTEGRATION/ Sequential explanatory design SYNTHESIS OF METHODS The quantitative component is (Assessing the value-added of followed by the qualitative. The the mixed methods approach) purpose is to explain quantitative Applied mixed methods design: results using qualitative findings. Sequential explanatory design E.g., the quantitative results guide Sequential explorative design the selection of qualitative data Triangulation design sources and data collection, and Embedded design the qualitative findings contribute Appraisal indicators: to the interpretation of quantita- Consider whether tive results i. the rationale for integrating Sequential exploratory design the qualitative and quantitative qualitative component is followed methods to answer the research by the quantitative. The purpose question is explained is to explore, develop and test an [DEFENSIBLE] instrument (or taxonomy), or a ii. mixed methods research design conceptual framework (or theo- is relevant to address the qualita- retical model). E.g., the qualitative tive and quantitative research findings inform the quantitative questions, or the qualitative and data collection, and the quantita- quantitative aspects of the mixed tive results allow a generalization methods research question of the qualitative findings [DEFENSIBLE] Triangulation designs the qualita- tive and quantitative components iii. there is evidence that data gath- are concomitant. The purpose is to ered by both research methods examine the same phenomenon was brought together to inform by interpreting qualitative and new findings to answer the mixed quantitative results (bringing data methods research question (e.g., analysis together at the interpreta- form a complete picture, synthe- tion stage), or by integrating quali- size findings, configuration) tative and quantitative datasets [CREDIBLE] (e.g., data on same cases), or by iv. the approach to data integra- transforming data (e.g., quantiza- tion is transparent and rigorous in tion of qualitative data) considering all findings from both Embedded/convergent design the qualitative and quantitative The qualitative and quantitative module (danger of cherry-picking) components are concomitant. The [RIGOROUS] purpose is to support a qualita- v appropriate consideration is tive study with a quantitative given to the limitations associ- sub-study (measures), or to better ated with this integration, e.g., understand a specific issue of a the divergence of qualitative and quantitative study using a qualita- quantitative data (or results)? tive sub-study, e.g., the efficacy [REFLEXIVE] or the implementation of an intervention based on the views of participants B erretta et al. Agriculture & Food Security (2023) 12:13 Page 43 of 52 Study type Methodological appraisal Response criteria Yes No Comment /confidence judgment For mixed methods research studies, each component undergoes its individual critical appraisal first. Since qualitative studies are either included or excluded, no combined risk of bias assessment is facilitated, and the assigned risk of bias from the quantitative component similarly holds for the mixed methods research The above appraisal indicators only refer to the applied mixed methods design. If this design is not found to comply with each of the four mixed methods appraisal criteria below, then the quantitative/qualitative components will individually be included in the review: Mixed-methods critical appraisal: Qualitative critical appraisal: Quantitative critical appraisal: 1. Research is defensible in Include/Exclude 1. Low risk of bias design 2. Risk of bias 2. Research is rigorous in 3. High risk of bias conduct 4. Critical risk of bias 3. Research is credible in claim 4. Research is reflective Combined appraisal: Include / Exclude mixed methods findings judged with ____________________________ risk of bias Section based on Pluye et al. (2011). Further sources consulted (in alphabetical order): Creswell and Clark (2007); Crow (2013); Long (2005); O’Cathain et al. (2008); O’Cathain (2010); Pluye and Hong (2014); Sirriyeh et al. (2011) For the qualitative studies, we use a slightly different language to scale the critical appraisal assessments as compared to the quantitative studies. The far right rating column always reflects a ‘critical’ appraisal judgment (i.e., ‘unreflective research’ above) with judgements moving further to the left on a scale from high to low critical appraisal Detailed results for diet quality and adequacy Appendix 4: Additional meta‑analysis results We included a total of k = 4 studies in the analysis. The Detailed results for food security observed outcomes ranged from 0.08 to 0.14 . The esti - A total of k = 4 studies were included in the analysis. mated average outcome based on the random effects The observed outcomes ranged from 0.07 to 0.67 , with model was µ = 0.09 (95% CI: 0.06 to 0.12 ). Therefore, the majority of estimates being positive (100%). The esti - the average outcome differed significantly from zero mated average outcome based on the random effects ( z = 5.64 , p < 0.01 ). According to the Q-test, there was model was µ = 0.24 (95% CI: 0.00 to 0.47 ). Therefore, no significant amount of heterogeneity in the true out - the average outcome differed significantly from zero 2 2 comes (Q (3) = 0.53 , p = 0.91 , τ = 0.00 , I = 0.00%). ( z = 1.97 , p = 0.05 . According to the Q-test, the true An examination of the studentized residuals revealed outcomes appear to be heterogeneous (Q (3) = 111.16 , 2 2 that none of the studies had a value larger than ±2.50 and p < 0.01 , τ = 0.06 , I = 97.30%). hence there was no indication of outliers in the context of An examination of the studentized residuals revealed this model. that one study [25] had a value larger than ±2.50 and may be a potential outlier in the context of this model. Detailed results for anthropometric measures We included a k = 2 studies in the analysis. The estimated Detailed results for food affordability/availability average outcome based on the random effects model We included a total of k = 6 studies were included in was µ = 0.12(95% CI: 0.00to0.23 ). Therefore, the average the analysis. The observed outcomes ranged from 0.08 to outcome did not differ significantly from zero ( z = 1.99 , 0.49 , with the majority of estimates being positive (100%). p = 0.05 ). According to the Q-test, there was no sig- The estimated average outcome based on the random nificant amount of heterogeneity in the true outcomes effects model was µ = 0.23 (95% CI: 0.09 to 0.38 ). There - 2 2 (Q (1) = 0.12 , p = 0.73 , τ = 0.00 I = 0.00% ). Given the fore, the average outcome differed significantly from zero small number of studies, this result should be interpreted ( z = 3.19 , p < 0.01 ). According to the Q-test, the true with caution. outcomes appear to be heterogeneous (Q (15) = 187.27 , 2 2 p < 0.01 , τ = 0.02 , I = 91.99%). Detailed results for well‑being outcomes An examination of the studentized residuals revealed We included a k = 2 studies in the analysis. The esti - that one study (Ahmed et al. 2019 had a value larger than mated average outcome based on the random effects ±2.96 and may be a potential outlier in the context of this model was µ = 0.08(95% CI: 0.01to0.15 ). Therefore, the model. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 44 of 52 average outcome did not differ significantly from zero Appendix 5: Detailed risk of bias ( z = 2.11 , p = 0.034 ). According to the Q-test, there was See Tables 5 and 6 significant amount of heterogeneity in the true outcomes The nine additional qualitative studies were assessed. 2 2 (Q (1) = 2.90 , p = 0.08 , τ = 0.00 I = 65.57% ). Given Five [37, 38, 39, 40, 48] were found to be high quality, the small number of studies, this result should be inter- with the remaining four [41, 49, 5051] marked as medium preted with caution. quality according to the assessment tool. The main Table 5 Risk of bias in experimental studies Author Overall (Year) Score Some Some Heckert concer Low Low Low Low Low Low Low conce (2019) ns Risk Risk Risk Risk Risk Risk Risk rns Deining Some Some Some Some er concer Low concer concer Low concer Low Low Low (2009) ns Risk ns ns Risk ns Risk Risk Risk Some Some Some Haque concer Low Low concer Low Low Low Low conce (2021) ns Risk Risk ns Risk Risk Risk Risk rns Blaksta d Low Low Low Low Low Low Low Low Low (2020) ROB Risk Risk Risk Risk Risk Risk Risk Risk Bandier Some Some a concer Low Low Low Low Low concer Low Low (2017) ns Risk Risk Risk Risk Risk ns Risk Risk Some Some Ahmed concer Low Low Low Low Low concer Low Low (2019) ns Risk Risk Risk Risk Risk ns Risk Risk Assignment mechanism Unit of analysis Selection bias Confounding Deviations from intended Performance bias Outcome measurement Reporting bias B erretta et al. Agriculture & Food Security (2023) 12:13 Page 45 of 52 Table 6 Risk of bias in quasi-experimental studies Author Overall (Year) Score Low Low Low Low Low Pan (2015) Low ROB risk risk Low risk risk risk risk Marquis Low Low Low Low Low (2015) Low ROB risk risk Low risk risk risk risk Emran Some Low Low Some Low Low Low (2009) concerns risk risk concerns risk risk risk Bonuedi Low High High Low High (2020) High ROB risk risk High risk risk risk risk factor differentiating high and medium quality quali - Below is the list of databases and organizational web- tative studies was the level of rigor and detail provided sites searched in the FSN EGM. This online Appendix in the methods. Triangulating data by interviewing dif- provides more detailed information about the search ferent population groups in a given community allowed strategy: https:// www. 3ieim pact. org/ sites/ defau lt/ files/ for different perspectives, making qualitative studies 2021- 01/ EGM16- Online- appen dix-B- Search- strat egy. pdf more rigorous. Sometimes the male head of household was interviewed along with the woman beneficiary, as Academic databases well as other community members, which can affect the We conducted electronic searches of the following data- information reported. Studies were high quality if they bases of published sources: triangulated data, used ethical methods (i.e., did not add additional burden onto women’s time) and added rich • MEDLINE contextual layers to quantitative findings in other studies • EMBASE or the same study. • Cochrane Controlled Trials Register (CENTRAL) • CINAHL • CAB Global Health • CAB Abstracts Appendix 6: Eec ff t estimates from included studies • Agricola See Table 7 • PsychINFO Appendix 7: Food system EGM framework • Africa-Wide Information and search strategy • Academic Search Complete See Table 8 • Scopus The complete Food system EGM framework can be • Campbell Library found at this link: https:// www. 3ieim pact. org/ sites/ defau lt/ files/ 2021- 01/ EGM16- Online- appen dix-A- Addit ional- metho ds- detail. pdf Website searched Selection bias Confounding Performance bias Spillovers, crossovers, and Outcome measurement bias Reporting bias Berretta et al. Agriculture & Food Security (2023) 12:13 Page 46 of 52 ff Table 7 Eect estimates from included studies in REA† First author Year Country Intervention type Evaluation/synthesis method Outcome Standardized effect Sample size estimate (Confidence Interval) Food security Bandiera 2017 Bangladesh Training/education Asset Randomized control trial and Food security index—whether 0.07 (0.03; 0.12) 6732 transfer difference-in-difference HH had surplus food or deficit, enough food to eat, and could afford to eat two meals a day* Emran 2009 Bangladesh Training/education Asset Difference-in-difference and Meals twice a day 0.6 (0.5; 0.7) 1569 transfer statistical matching Food availability: Sufficient food 0.66 (0.56; 0.76) 1569 to meet the household’s needs* Blackstad 2020 Tanzania Training/education Randomized control trial Household food insecurity 0.07 (− 0.05; 0.2) 876 access scale* Pan 2015 Uganda Training/education Asset Regression discontinuity Skip meals* − 0.13 (− 0.2; − 0.06) 3368 transfer Food affordability and avail- ability Ahmed 2019 Bangladesh Asset transfer (Cash) Randomized control trial Per capita monthly food con- 0.13 (0.08; 0.19) 5000 sumption—North Per capita monthly food con- 0.08 (0.03; 0.14) 5000 sumption—South Per capita daily intake caloric— 0.07 (0.02; 0.13) 5000 North Per capita daily intake caloric— 0.02 (− 0.03; 0.08) 5000 South Food consumption score— 0.17 (0.12; 0.23) 5000 North Food consumption score 0.07 (0.02; 0.13) 5000 − South Asset transfer (Food) Per capita monthly food con- 0.11 (0.06; 0.17) 5000 sumption—North Per capita monthly food con- 0.07 (0.02; 0.13) 5000 sumption—South Per capita daily intake caloric— 0.13 (0.08; 0.19) 5000 North Per capita daily intake caloric— 0.01 (− 0.05; 0.06) 5000 South Food consumption score— 0.24 (0.18; 0.29) 5000 North B erretta et al. Agriculture & Food Security (2023) 12:13 Page 47 of 52 Table 7 (continued) First author Year Country Intervention type Evaluation/synthesis method Outcome Standardized effect Sample size estimate (Confidence Interval) Food consumption score— 0.12 (0.07; 0.18) 5000 South Assets transfer (Cash and food) Per capita monthly food con- 0.11 (0.06; 0.17 5000 sumption—North Per capita monthly food con- 0.11 (0.06; 0.17) 5000 sumption—South Per capita daily intake caloric— 0.07 (0.02; 0.13) 5000 North Per capita daily intake caloric— 0.03 (− 0.03; 0.09) 5000 South Food consumption score— 0.16 (0.11; 0.22) 5000 North Food consumption score— 0.11 (0.06; 0.17) 5000 South Behavior change communica- Per capita monthly food con- 0.32 (0.27; 0.38) 5000 tion sumption—North Asset transfer (Cash) Per capita monthly food con- 0.22 (0.17; 0.28) 5000 sumption—South Per capita daily intake caloric— 0.22 (0.17; 0.28) 5000 North Per capita daily intake caloric— 0.10 0.04; 0.15) 5000 South Food consumption score— 0.49 (0.44; 0.55) 5000 North* Food consumption score— 0.28 (0.23; 0.34) 5000 South* Bonuedi 2020 Sierra Leone Behavior change communica- Statistical matching Total food consumption − 0.04 (− 0.18, 0.09) 836 tion Training/education Assets expenditure in the 12 months transfer preceding the survey (Food production and market pur- chases) (LOG)-Household* Total food consumption 0.22 (0.09; 0.36) 836 expenditure in the 12 months preceding the survey (Food production and market pur- chases) (LOG)-Household* Berretta et al. Agriculture & Food Security (2023) 12:13 Page 48 of 52 Table 7 (continued) First author Year Country Intervention type Evaluation/synthesis method Outcome Standardized effect Sample size estimate (Confidence Interval) Deininger 2009 India Training/education Difference-in-difference and Food consumption (RS/year)— 0.09 (− 0.03; 0.2) 2199 statistical matching All groups* Energy intake p.c. (kcal/day)— 0.02 (− 0.09; 0.14) 2199 All groups Food consumption (RS/year)- 0.19 (− 0.08; 0.47) 404 POP (Poorest of the poor) Food consumption (RS/year)- 0.42 (0.06; 0.77) 243 Poor Food consumption (RS/year)- − 0.11 (− 0.54; 0.33) 157 Non-poor Energy intake p.c. (kcal/day)- 0.36 (0.09; 0.64) 404 POP (Poorest of the poor) Energy intake p.c. (kcal/day)- 0.59 (0.23; 0.95) 243 Poor Energy intake p.c. (kcal/day)- − 0.08 (− 0.52; 0.36) 157 Non-poor Emran 2009 Bangladesh Assets transfer Training/educa- Difference-in-difference and Grain stocks (kg)* 0.22 (0.12; 0.32) 1569 tion statistical matching Pan 2015 Uganda Training/education Assets Regression discontinuity Per capita food consumption* 0.08 (0.01; 0.15) 3368 transfer Diet quality and adequacy Bonuedi 2020 Sierra Leone Behavior change communica- Propensity score matching Household dietary diversity* 0.14 (0.00, 0.27) 836 tion Training/education Women’s dietary diversity 0.10 (− 0.05, 0.26) 636 Children’s dietary diversity − 0.05 (− 0.21, 0.11) 575 Behavior change communica- Household dietary diversity 0.23 (0.09, 0.26) 836 tion Training/education Assets transfer Women’s dietary diversity 0.31 (0.15, 0.46) 636 Children’s dietary diversity 0.29 (0.12, 0.45 575 Haque 2021 Bangladesh Training/education Randomized control trial Additional food consumed dur- 0.09 (0.05, 0.13) 10722 ing pregnancy* Deninger 2009 India Training/education Difference-in-difference and Protein intake p.c. (g/day) in the 0.08 (− 0.04, 0.19) 1099.5 statistical matching total population* Protein intake p.c. (g/day) 0.32 (0.05, 0.60) 202 among the poor of the poor B erretta et al. Agriculture & Food Security (2023) 12:13 Page 49 of 52 Table 7 (continued) First author Year Country Intervention type Evaluation/synthesis method Outcome Standardized effect Sample size estimate (Confidence Interval) Protein intake p.c. (g/day) 0.66 (0.30, 1.02) 121.5 among the poor Protein intake p.c. (g/day) 0.20 (− 0.24, 0.64) 78.5 among the non-poor Pan 2015 Uganda Training/education Asset Regression discontinuity Variety of foods consumed* 0.09 (0.02, 0.15) 3368 transfer Anthropometrics Heckert 2019 Burkina Faso Behavior change communica- Randomized control trial Weight-for-length z score* 0.12 (0.00,0.25) 1035 tion Asset transfer Marquis 2015 Ghana Training/education Asset Difference-in-difference BMI-for-age z score* 0.06 (− 0.30, 0.41) 121.6 transfer Weight-for-age z-sore − 0.42 (− 0.77, − 0.06) 121.6 Height-for-age z score 0.40 (0.04,0.75) 121.6 Micronutrient status Haque 2021 Bangladesh Training/education Randomized control trial Consumption of at least 100 IFA 0.25 (0.21, 0.28) 10722 tablets during pregnancy Received vitamin a capsule 0.20 (0.16, 0.24) 10722 after last delivery Heckert 2019 Burkina Faso Behavior change communica- Randomized control trial Change in hemoglobin (g/dL) 0.10 (− 0.02, 0.23) 1035 tion Asset transfer Wellbeing outcomes Bandiera 2017 Bangladesh Training/education Asset Randomized control trial and Mental health index 0.04 (− 0.00, 0.09) 6732 transfer difference-in-difference Pan 2015 Uganda Training/education Asset Regression discontinuity Worry about insufficient food − 0.11 (− 0.18, − 0.04) 3368 transfer *Indicates estimates that were used in meta-analysis. Only one outcome per study per analysis was included to maintain independence of observations. Outcomes were selected based on comparability with other studies †Some studies appear multiple times because they report data related to multiple outcomes Berretta et al. Agriculture & Food Security (2023) 12:13 Page 50 of 52 Table 8 PICOS summary of criteria for the inclusion and exclusion of studies Criteria Inclusion criteria Exclusion criteria Population Program participants that were located in a L&MIC in the first year Studies focused on niche populations, such as athletes or the of implementation3 military Impact evaluations with at least one effect size for an L&MIC Efficacy studies, unless they were completed in a sufficiently real- country population world setting Studies focused on the prevention of clinical conditions Studies targeting participants with a clinical condition Studies focused on high-income country migrant populations in L&MICs and vice versa Intervention Interventions that directly intervene on an aspect of the food Interventions not in the food system or interventions targeting system within its three primary domains: the food supply chain, drivers of the food system without an explicit food system focus the food environment and consumer behavior Studies evaluating multiple food systems interventions Unconditional cash transfer programs Interventions focused on the financing of a food systems interven- tion Comparisons Appropriate comparisons included: business as usual, an alterna- Studies that did not justify and make use of an appropriate com- tive treatment, no treatment or an early-versus-late comparison parison group (where those that took part in earlier years are compared to those that took part in later years) Source: 3ie 2020 The cutoff at the year 2000 was made arbitrarily to make the volume of search results more manageable Gray literature sites searched • Abdul Latif Jamee l Pover ty Actio n Lab To identify relevant gray literature, we searched the fol- • Globa l Devel opmen t Netwo rk lowing databases (some of which contain a mixture of • World Bank Devel opmen t Impac t Evalu ation (DIME) published and gray literature):and Impac t Evalu ation Polic y Papers • nter- Ameri can Devel opmen t Bank • Cente r for Globa l Devel opment • Google Scholar • Cente r for Effec tive Globa l Actio n (CEGA) • EconLit • Depar tment for Inter natio nal Devel opmen t Resea rch • ENN-Network for Devel opmen t (R4D) • IDEAS/RePEc • Innovative Methods and Metrics for Agriculture and • USAID Nutrition Actions grantee database • Inter natio nal Food Polic y Resea rch Insti tute • WHO Global Index Medicus • CIGAR • Gray Literature Report • Food and Agric ultur e Organ ization of t he Unite d • Social Science Research Network (SSRN)Natio ns (FAO) • Eldis • High Level Pane l of Exper ts on Food Secur ity and • EpistemonikosNutri tion • 3ie Development Evidence Portal • World Food Progr amme • Registry of International Development Impact Evalu • Actio n Again st Hunger • UNICEF ations (RIDIE) • Unite d Natio ns Evalu ation Group • Oxfam Policy & Practice • Asian Devel opmen t Bank • World Agrof orest ry Centr e (ICRAF) Below is a list of organizational websites we manually • Inter natio nal Lives tock Resea rch Insti tute (ILRI) searched for additional related studies. • Nutri tion Inter natio nal • AgEco n Searc h (Unive rsity of Minne sota) • Innov ation s for Pover ty Action B erretta et al. Agriculture & Food Security (2023) 12:13 Page 51 of 52 fss_ briefs_ review_ evide nce_ gender_ equal ity. pdf? seque nce= 3& isAll Supplementary Information owed=y The online version contains supplementary material available at https:// doi. 3. WHO. Understanding the women’s empowerment pathway. Brief #4. org/ 10. 1186/ s40066- 023- 00405-9. Improving nutrition through agriculture technical brief series. Arlington: Additional file 1. It contains additional information about the included USAID/Strengthening Partnerships, Results, and Innovations in Nutrition studies in terms of Intervention type,Detailed intervention, Evaluation Globally (SPRING) Project. 2014. method, Hypotheses mechanisms of action, Impacts Barriers and facili- 4. United Nations Food Systems Summit. Chapter 2 key inputs from summit tatorsto impact, implementation, and evaluation, Equity consideration, workstreams action tracks. 2021. https:// foods ystems. commu nity/ food- Sources of bias, Risk of bias, Eec ff tiveness,ans Conclusions.syste ms- summit- compe ndium/ action- tracks/. Accessed 27 Jan 2022. 5. Cole SM, Kantor P, Sarapura S, Rajaratnam S. Gender-transformative approaches to address inequalities in food, nutrition and economic Acknowledgements outcomes in aquatic agricultural systems. 2015. Not applicable. 6. Wong, F, Vos A, Pyburn R, Newton J. Implementing gender transformative approaches in agriculture. A Discussion Paper for the European Commis- Author contributions sion. 2019. MB contributed to extract the effect sizes and analyze them through the 7. Cheung J, Gursel D, Kirchner MJ, Scheyer V. Practicing feminist foreign meta-analysis. She was a major contributor in writing the manuscript. MK policy in the everyday: a toolkit. Germany; 2021. searched the additional qualitative studies, extracted the data and analyzed 8. Thompson L. Defining feminist foreign policy. Washington: International them. They were a major contributor in writing the manuscript. CL contrib- Center for Research on Women; 2019. p. 1–7. uted extract the effect sizes and analyze them through the meta-analysis. She 9. Campbell Collaboration. (2017). Campbell systematic reviews: Policies was a major contributor in reviewing and writing the manuscript, as well as and guidelines. in ensuring its overall quality. SS ensured the meta-analysis were conducted 10. Barends, E., Rousseau, D. M. & Briner, R. B. CEBMa Guideline for Rapid following the highest standard and corrected any mistakes. She was a major Evidence Assessments in Management and Organizations. Amsterdam. contributor in reviewing the manuscript, as well as in ensuring its overall qual- 2017. https:// www. cebma. org/ wp- conte nt/ uploa ds/ CEBMa- REA- Guide ity. All authors read and approved the final manuscript. line. pdf 11. Snilstveit B. Systematic reviews: from ‘bare bones’ reviews to policy Funding relevance. J Dev Eec ff t. 2012;4(3):388–408. https:// doi. org/ 10. 1080/ 19439 This project has been commissioned and funded by Germany’s Federal Min- 342. 2012. 709875. istry for Economic Cooperation and Development (BMZ) through Deutsche 12. CASP (2018). Qualitative Checklist. [online] Available at: https://casp-uk. Gesellschaft für Internationale Zusammenarbeit (GIZ) through its “Knowledge net/images/checklist/documents/CASP-Qualitative-Studies-Checklist/ for Nutrition” program. CASP-Qualitative-Checklist-2018_fillable_form.pdf Accessed: 1st March Availability of data and materials 13. Korth M, Stewart R, Langer L, Madinga N, Rebelo Da Silva N, Zaranyika H, All the data used to support finding in this study derive from the included van Rooyen C, de Wet T. What are the impacts of urban agriculture pro- study, please see the full list in the Reference section. grams on food security in low and middle-income countries: a systematic review. Environ Evid. 2014;3(1):1–10. 14. Stewart R, Langer L, Da Silva NR, Muchiri E, Zaranyika H, Erasmus Y, Declarations Randall N, Rafferty S, Korth M, Madinga N, de Wet T. The effects of train- ing, innovation and new technology on African smallholder farmers’ Ethics approval and consent to participate economic outcomes and food security: a systematic review. Campbell This is not applicable to this manuscript as we did not use primary data. Syst Rev. 2015;11(1):1–224. 15. Goudet SM, Bogin BA, Madise NJ, Griffiths PL. Nutritional interventions for Consent for publication preventing stunting in children (birth to 59 months) living in urban slums This is not applicable as we did not include any individual person’s data in any in low-and middle-income countries (LMIC). Cochrane Database Syst Rev. form. 2019. https:// doi. org/ 10. 1002/ 14651 858. CD011 695. pub2. 16. Sguassero Y, de Onis M, Bonotti AM, Carroli G. Community-based supple- Competing interests mentary feeding for promoting the growth of children under five years The authors declare that they have no competing interests. of age in low and middle income countries. Cochrane Database Syst Rev. 2012;2012(6):CD005039. https:// doi. org/ 10. 1002/ 14651 858. CD005 039. Author details pub3. The International Initiative for Impact Evaluation, Synthesis, and Reviews 17. Visser ME, Schoonees A, Ezekiel CN, Randall NP, Naude CE. Agricultural Office (SRO), London, UK. The International Initiative for Impact Evaluation, and nutritional education interventions for reducing aflatoxin exposure Washington, DC, USA. to improve infant and child growth in low-and middle-income countries. Cochrane Database Syst Rev. 2020. https:// doi. org/ 10. 1002/ 14651 858. Received: 20 June 2022 Accepted: 10 January 2023 CD013 376. pub2. 18. Shah D, Sachdev HS, Gera T, De-Regil LM, Peña-Rosas JP. Fortification of staple foods with zinc for improving zinc status and other health outcomes in the general population. Cochrane Database Syst Rev. 2016. https:// doi. org/ 10. 1002/ 14651 858. CD010 697. pub2. 19. Suchdev PS, Peña-Rosas JP, De-Regil LM. Multiple micronutrient powders References for home (point-of-use) fortification of foods in pregnant women. 1. Moore N, Lane C, Storhaug I, Franich A, Rolker H, et al. The effects of Cochrane Database Syst Rev. 2015. https:// doi. org/ 10. 1002/ 14651 858. food systems interventions on food security and nutrition outcomes in CD011 158. pub2. low- and middle-income countries. International Initiative for Impact 20. Suchdev PS, Jefferds MED, Ota E, da Silva LK, De-Regil LM. Home forti- Evaluation (3ie); 2021. fication of foods with multiple micronutrient powders for health and 2. Njuki J, Eissler S, Malapit H, Meinzen-Dick R, Bryan E, Quisumbing A. A nutrition in children under two years of age. Cochrane Database Syst Rev. review of evidence on gender equality, women’s empowerment, and 2020;2(2):CD008959. https:// doi. org/ 10. 1002/ 14651 858. CD008 959. pub3. food systems: Food Systems Summit Brief Prepared by Research Partners 21. Gera T, Sachdev HS, Boy E. Eec ff t of iron-fortified foods on hematologic of the Scientific Group for the Food Systems Summit. 2021. https:// and biological outcomes: systematic review of randomized controlled bonnd oc. ulb. uni- bonn. de/ xmlui/ bitst ream/ handle/ 20. 500. 11811/ 9132/ trials. Am J Clin Nutr. 2012;96(2):309–24. https:// doi. org/ 10. 3945/ ajcn. 111. 031500. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 52 of 52 22. Ota E, Hori H, Mori R, Tobe-Gai R, Farrar D. Antenatal dietary education 45. Choudhury N, Raihan MJ, Ahmed ST, Islam KE, Self V, Rahman S, Schofield and supplementation to increase energy and protein intake. Cochrane L, Hall A, Ahmed T. The evaluation of Suchana, a large-scale development Database Syst Rev. 2015. https:// doi. org/ 10. 1002/ 14651 858. CD000 032. program to prevent chronic undernutrition in north-eastern Bangladesh. pub3. BMC Public Health. 2020;20:1–9. 23. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to meta- 46. Snilstveit, B, Stevenson, J, Langer, L, da Silva, N, Rabat, Z, Nduku, P, Polanin, analysis. New York: John Wiley & Sons; 2021. J, Shemilt, I, Eyers, J, Ferraro, PJ, Incentives for climate mitigation in the 24. Ahmed A, Hoddinott J, Roy S. Food transfers, cash transfers, behavior land use sector – the effects of payment for environmental services (PES) change communication and child nutrition. Intl Food Policy Res Inst. on environmental and socio-economic outcomes in low- and middle- 2019;1868. income countries: a mixed-method systematic review 3ie Systematic 25. Bandiera O, Burgess R, Das N, Gulesci S, Rasul I, Sulaiman M. Labor Review 44. London: International Initiative for Impact Evaluation (3ie). markets and poverty in village economies. Q J Econ. 2017;132(2):811–70. 2019. https:// doi. org/ 10. 23846/ SR000 44 https:// doi. org/ 10. 1093/ qje/ qjx003. 47. Critical Appraisal Skills Programme. (2006) 10 questions to help you 26. Blakstad MM, Mosha D, Bellows AL, Canavan CR, Chen JT, Mlalama makesense of qualitative research. Public Health Resource Unit. K, et al. Home gardening improves dietary diversity, a cluster-rand- Retrieved fromhttp:// www. biome dcent ral. com/ conte nt/ suppl ement ary/ omized controlled trial among Tanzanian women. Matern Child Nutr. 2046‐4053‐3‐139‐S8. pdf 2021;17(2):e13096. https:// doi. org/ 10. 1111/ mcn. 13096. 48. Das N, Yasmin R, Ara J, Kamruzzaman M, Davis P, Behrman J, et al. How do 27. Bonuedi I, Kornher L, Gerber N. Making cash crop value chains nutrition- intrahousehold dynamics change when assets are transferred to women? sensitive: evidence from a quasi-experiment in rural Sierra Leone. SSRN Evidence from BRACCs challenging the frontiers of poverty reduction Electron J. 2020. https:// doi. org/ 10. 2139/ ssrn. 36039 18. targeting the ultra-poor program in Bangladesh. SSRN Electron J. 2013. 28. Deininger K, Liu Y. Economic and social impacts of self-help groups in https:// doi. org/ 10. 2139/ ssrn. 24057 12. India. World Bank Group; 2009. 49. Huda K, Kaur S. ‘It was as if we were drowning’: shocks, stresses and safety 29. Emran MS, Robano V, Smith SC. Assessing the frontiers of ultra-poverty nets in India. Gend Dev. 2011;19(2):213–27. https:// doi. org/ 10. 1080/ reduction: Evidence from CFPR/TUP, an innovative program in Bangla-13552 074. 2011. 592632. desh. TUP, An Innovative Program in Bangladesh. 2009. 50. Olney DK, Dillon A, Ruel MT, Nielsen J. Lessons learned from the evalua- 30. Haque MA, Choudhury N, Ahmed ST, Farzana FD, Ali M, Naz F, et al. The tion of Helen Keller International’s enhanced homestead food production large-scale community-based programme ‘Suchana ’improved maternal program. Achieving a nutrition revolution for Africa: The road to healthier healthcare practices in north-eastern Bangladesh: findings from a cluster diets and optimal nutrition. 2016;67–81. randomized pre-post study. Matern Child Nutr. 2022. https:// doi. org/ 10. 51. van den Bold M, Dillon A, Olney D, Ouedraogo M, Pedehombga A, 1111/ mcn. 13258. Quisumbing A. Can integrated agriculture-nutrition programmes change 31. Heckert J, Olney DK, Ruel MT. Is women’s empowerment a pathway to gender norms on land and asset ownership? Evidence from Burkina Faso. improving child nutrition outcomes in a nutrition-sensitive agriculture J Dev Stud. 2015;51(9):1155–74. https:// doi. org/ 10. 1080/ 00220 388. 2015. program?: Evidence from a randomized controlled trial in Burkina Faso. 10360 36. Soc Sci Med. 2019;233:93–102. https:// doi. org/ 10. 1016/j. socsc imed. 2019. 05. 016. Publisher’s Note 32. Marquis GS, Colecraft EK, Sakyi-Dawson O, Lartey A, Ahunu BK, Birks KA, Springer Nature remains neutral with regard to jurisdictional claims in pub- et al. An integrated microcredit, entrepreneurial training, and nutri- lished maps and institutional affiliations. tion education intervention is associated with better growth among preschool-aged children in rural Ghana. J Nutr. 2015;145(2):335–43. https:// doi. org/ 10. 3945/ jn. 114. 194498. 33. Pan Y, Smith SC, Sulaiman M. Agricultural extension and technology adoption for food security: evidence from Uganda. Am J Agric Econ. 2018;100(4):1012–31. https:// doi. org/ 10. 1093/ ajae/ aay012. 34. Polanin, J.R. and Snilstveit, B., 2016. Converting between effect sizes. Campbell Systematic Reviews, 12(1), pp.1-13. 35. Sánchez-Meca, J., Marín-Martínez, F., & Chacón-Moscoso, S. (2003). Eec ff t- size indices for dichotomized outcomes in meta-analysis. Psychological methods, 8(4), 448. 37. Kabeer N, Datta S. Randomized control trials and qualitative impacts: what do they tell us about the immediate and long-term assessments of productive safety nets for women in extreme poverty in West Bengal? (No. 19–199). Working Paper Series. 2020 38. Roy S, Hidrobo M, Hoddinott J, Ahmed A. Transfers, behavior change communication, and intimate partner violence: postprogram evidence from rural Bangladesh. Rev Econ Stat. 2019;101(5):865–77. https:// doi. org/ 10. 1162/ rest_a_ 00791. 39. Roy S, Ara J, Das N, Quisumbing AR. Flypaper effects” in transfers targeted to women: evidence from BRAC’s “Targeting the Ultra Poor” program in Bangladesh. J Dev Econ. 2015;117:1–19. 40. Kieran C, Gray B, Gash M. Understanding gender norms in rural Burkina Faso: a qualitative assessment. 2018. 41. Hagan LL, Aryeetey R, Colecraft EK, Marquis GS, Nti AC, University of Ghana, et al. Microfinance with education in rural Ghana: Men’s perception of household level impact. Afr J Food Agric Nutr Dev. 2012;12(49):5776–88. https:// doi. org/ 10. 18697/ ajfand. 49. enam7. 42. Dolgopolova I, Li B, Pirhonen H, Roosen J. The effect of attribute framing on consumers’ attitudes and intentions toward food: a Meta-analysis. Bio- based Appl Econ J. 2021;10:253–64. 43. Data4Diets’ (2021), International Dietary Data Expansion Project. https:// inddex. nutri tion. tufts. edu/ data4 diets. Accessed 4 Sep 2022. 44. DQQ Tools & Data’ (2021), Global Diet Quality Project. https:// www. globa ldiet quali ty. org/ dqq. Accessed 11 Apr 2022. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Agriculture & Food Security Springer Journals http://www.deepdyve.com/lp/springer-journals/rapid-evidence-assessment-on-women-s-empowerment-interventions-within-wm6UY6Qpdc

Loading next page...

References (53)

Isaac Bonuedi, Lukas Kornher, Nicolas Gerber (2020)
Making Cash Crop Value Chains Nutrition-Sensitive: Evidence from a Quasi-Experiment in Rural Sierra Leone
Agronomy eJournal
M. Emran, Virginia Robano, Stephen Smith (2009)
Assessing the Frontiers of Ultra-Poverty Reduction: Evidence from CFPR/TUP, An Innovative Program in Bangladesh
Development Economics
M Borenstein, LV Hedges, JP Higgins, HR Rothstein (2021)
10.1002/9781119558378
Introduction to meta-analysis
N. Kabeer, Sanchari Datta (2020)
Randomized control trials and qualitative impacts: what do they tell us about the immediate and long-term assessments of productive safety nets for women in extreme poverty in West Bengal?
LSE Research Online Documents on Economics
Mahesh Patel (1989)
An introduction to meta-analysis.
Health policy, 11 1
J. Heckert, Deanna Olney, M. Ruel (2019)
Is women's empowerment a pathway to improving child nutrition outcomes in a nutrition-sensitive agriculture program?: Evidence from a randomized controlled trial in Burkina Faso
Social Science & Medicine (1982), 233
Qualitative Checklist
(2006)
10 questions to help you makesense of qualitative research. Public Health Resource Unit
O. Bandiera, R. Burgess, N. Das, Selim Gulesci, Imran Rasul, M. Sulaiman (2016)
Labor Markets and Poverty in Village Economies
Randomized Social Experiments eJournal
N. Choudhury, M. Raihan, S. Ahmed, K. Islam, Vanessa Self, S. Rahman, Lilly Schofield, Andrew Hall, T. Ahmed (2020)
The evaluation of Suchana, a large-scale development program to prevent chronic undernutrition in north-eastern Bangladesh
BMC Public Health, 20
(2021)
Practicing feminist foreign policy in the everyday: a toolkit
S. Goudet, B. Bogin, N. Madise, P. Griffiths (2015)
Nutritional interventions for preventing stunting in children (birth to 59 months) living in urban slums in low- and middle-income countries (LMIC).
The Cochrane database of systematic reviews, 6
Yao Pan, Stephen Smith, M. Sulaiman (2018)
Agricultural Extension and Technology Adoption for Food Security: Evidence from Uganda
Agricultural & Natural Resource Economics eJournal
A. Akhter, J. Hoddinott, Shalini Roy (2019)
Food Transfers, Cash Transfers, Behavior Change Communication and Child Nutrition: Evidence From Bangladesh
Food Laws
Mara Bold, A. Dillon, Deanna Olney, Marcellin Ouedraogo, Abdoulaye Pedehombga, A. Quisumbing (2015)
Can Integrated Agriculture-Nutrition Programmes Change Gender Norms on Land and Asset Ownership? Evidence from Burkina Faso
The Journal of Development Studies, 51
K. Huda, S. Kaur (2011)
‘It was as if we were drowning’: shocks, stresses and safety nets in India
Gender & Development, 19
P. Suchdev, J. Peña-Rosas, L. De-Regil (2015)
Multiple micronutrient powders for home (point-of-use) fortification of foods in pregnant women.
The Cochrane database of systematic reviews, 6
Deanna Olney, A. Dillon, M. Ruel, J. Nielsen (2016)
Lessons Learned from the Evaluation of Helen Keller International’s Enhanced Homestead Food Production Program
Global Diet Quality Project
PS Suchdev, MED Jefferds, E Ota, LK da Silva, LM De-Regil (2020)
Home fortification of foods with multiple micronutrient powders for health and nutrition in children under two years of age
Cochrane Database Syst Rev, 2
Shalini Roy, J. Ara, N. Das, A. Quisumbing (2015)
“Flypaper effects” in transfers targeted to women: Evidence from BRAC's “Targeting the Ultra Poor” program in Bangladesh
Journal of Development Economics, 117
I. Dolgopolova, Bingqing Li, Helena Pirhonen, J. Roosen (2022)
The effect of attribute framing on consumers’ attitudes and intentions toward food: A Meta-analysis
Bio-based and Applied Economics
N. Moore, Charlotte Lane, I. Storhaug, A. Franich, H. Rolker, J. Furgeson, T. Sparling, Birte Snilstveit, Innovative Methods, Metrics Agriculture, Nutrition Actions (2021)
The effects of food systems interventions on food security and nutrition outcomes in low- and middle-income countries
T. Gera, H. Sachdev, E. Boy (2012)
Effect of iron-fortified foods on hematologic and biological outcomes: systematic review of randomized controlled trials.
The American journal of clinical nutrition, 96 2
(2014)
the women’s empowerment pathway
Feng Li (2005)
An Introduction to Metaanalysis
Journal of The Northwest Normal University
(2018)
Understanding gender norms in rural Burkina Faso: a qualitative assessment
Ruth Stewart, Laurenz Langer, N. Silva, E. Muchiri, Hazel Zaranyika, Yvonne Erasmus, N. Randall, S. Rafferty, Marcel Korth, Nolizwe Madinga, T. Wet (2015)
The Effects of Training, Innovation and New Technology on African Smallholder Farmers' Economic Outcomes and Food Security: A Systematic Review
Campbell Systematic Reviews, 11
(2019)
Defining feminist foreign policy. Washington: International Center for Research on Women
J. Polanin, Birte Snilstveit (2016)
Converting between effect sizes
Campbell Systematic Reviews, 12
Md Haque, N. Choudhury, S. Ahmed, F. Farzana, M. Ali, Farina Naz, M. Raihan, S. Rahman, T. Siddiqua, A. Faruque, T. Ahmed (2021)
The large‐scale community‐based programme ‘Suchana’ improved maternal healthcare practices in north‐eastern Bangladesh: Findings from a cluster randomized pre‐post study
Maternal & Child Nutrition, 18
L. Hagan, R. Aryeetey, Esi Colecraft, G. Marquis, Ac Nti, A. Danquah (2012)
Microfinance with education in rural Ghana: Men's perception of household level impact
African Journal of Food, Agriculture, Nutrition and Development
Chapter 2 key inputs from summit workstreams action tracks
M. Blakstad, Dominic Mosha, A. Bellows, C. Canavan, Jarvis Chen, Killian Mlalama, Ramadhani Noor, J. Kinabo, H. Masanja, W. Fawzi (2020)
Home gardening improves dietary diversity, a cluster‐randomized controlled trial among Tanzanian women
Maternal & Child Nutrition, 17
G. Marquis, Esi Colecraft, O. Sakyi‐Dawson, A. Lartey, B. Ahunu, Katherine Birks, L. Butler, M. Reddy, H. Jensen, E. Huff-Lonergan (2015)
An integrated microcredit, entrepreneurial training, and nutrition education intervention is associated with better growth among preschool-aged children in rural Ghana.
The Journal of nutrition, 145 2
M. Visser, A. Schoonees, C. Ezekiel, N. Randall, C. Naude (2019)
Agricultural and nutritional education interventions for reducing aflatoxin exposure to improve infant and child growth in low- and middle-income countries.
The Cochrane database of systematic reviews, 4
S. Cole, P. Kantor, Silvia Sarapura, S. Rajaratnam (2014)
Gender-transformative approaches to address inequalities in food, nutrition and economic outcomes in aquatic agricultural systems
Yanina Sguassero, M. Onís, A. Bonotti, G. Carroli (2012)
Community-based supplementary feeding for promoting the growth of children under five years of age in low and middle income countries.
The Cochrane database of systematic reviews, 6
(2017)
Campbell systematic reviews: Policies and guidelines
K. Deininger, Yanyan Liu (2009)
Economic and Social Impacts of Self-Help Groups in India
Monetary Economics
N. Das, Rabeya Yasmin, J. Ara, M. Kamruzzaman, P. Davis, Julia Behrman, Shalini Roy, A. Quisumbing (2013)
How Do Intrahousehold Dynamics Change When Assets Are Transferred to Women? Evidence from BRAC’s Challenging the Frontiers of Poverty Reduction — Targeting the Ultra Poor Program in Bangladesh
Agricultural & Natural Resource Economics eJournal
(2019)
Implementing gender transformative approaches in agriculture. A Discussion Paper for the European Commission
(2003)
Effectsize indices for dichotomized outcomes in meta-analysis
Shalini Roy, Melissa Hidrobo, J. Hoddinott, Akhter Ahmed (2017)
Transfers, Behavior Change Communication, and Intimate Partner Violence: Postprogram Evidence from Rural Bangladesh
Review of Economics and Statistics, 101
Marcel Korth, Ruth Stewart, Laurenz Langer, Nolizwe Madinga, N. Silva, Hazel Zaranyika, C. Rooyen, T. Wet (2014)
What are the impacts of urban agriculture programs on food security in low and middle-income countries: a systematic review
Environmental Evidence, 3
(2017)
CEBMa Guideline for Rapid Evidence Assessments in Management and Organizations
E. Ota, H. Hori, R. Mori, Ruoyan Tobe-Gai, D. Farrar (2015)
Antenatal dietary education and supplementation to increase energy and protein intake.
The Cochrane database of systematic reviews, 6
D. Shah, H. Sachdev, T. Gera, L. De-Regil, J. Peña-Rosas (2016)
Fortification of staple foods with zinc for improving zinc status and other health outcomes in the general population.
The Cochrane database of systematic reviews, 6
International Dietary Data Expansion Project. https:// inddex. nutri tion. tufts. edu/ data4 diets
J. Njuki, Sarah Eissler, H. Malapit, R. Meinzen-Dick, E. Bryan, A. Quisumbing (2021)
A review of evidence on gender equality, women’s empowerment, and food systems
SSRN Electronic Journal
Energy intake p.c. (kcal/day)-POP (Poorest of the poor)
L. De-Regil, P. Suchdev, G. Vist, S. Walleser, J. Peña-Rosas (2011)
Home fortification of foods with multiple micronutrient powders for health and nutrition in children under two years of age.
The Cochrane database of systematic reviews, 9
Birte Snilstveit (2012)
Systematic reviews: from ‘bare bones’ reviews to policy relevance
Journal of Development Effectiveness, 4

Publisher: Springer Journals
Copyright: Copyright © The Author(s) 2023
eISSN: 2048-7010
DOI: 10.1186/s40066-023-00405-9
Publisher site: See Article on Publisher Site

Abstract

Background Women’s empowerment interventions represent a key opportunity to improve nutrition-related out- comes. Still, cross-contextual evidence on the factors that cause poorer nutrition outcomes for women and girls and how women’s empowerment can improve nutrition outcomes is scant. We rapidly synthesized the available evidence regarding the impacts of interventions that attempt to empower women and/or girls to access, participate in and take control of components of the food system. Methodology We considered outcomes related to food security; food affordability and availability; dietary quality and adequacy; anthropometrics; iron, zinc, vitamin A, and iodine status; and measures of wellbeing. We also sought to understand factors affecting implementation and sustainability, including equity. We conducted a rapid evidence assessment, based on the systematic literature search of key academic databases and gray literature sources from the regular maintenance of the living Food System and Nutrition Evidence Gap Map. We included impact evaluations and systematic reviews of impact evaluations that considered the women’s empowerment interventions in food systems and food security and nutrition outcomes. We conducted an additional search for supplementary, qualitative data related to included studies. Conclusion Overall, women’s empowerment interventions improve nutrition-related outcomes, with the largest effects on food security and food affordability and availability. Diet quality and adequacy, anthropometrics, effects were smaller, and we found no effects on wellbeing. Insights from the qualitative evidence suggest that women’s empowerment interventions best influenced nutritional outcomes when addressing characteristics of gender- transformative approaches, such as considering gender and social norms. Policy-makers should consider improving women’s social capital so they can better control and decide how to feed their families. Qualitative evidence sug- gests that multi-component interventions seem to be more sustainable than single-focus interventions, combining a livelihoods component with behavioral change communication. Researchers should consider issues with inconsistent data and reporting, particularly relating to seasonal changes, social norms, and time between rounds of data col- lection. Future studies on gender-transformative approaches should carefully consider contextual norms and avoid stereotyping women into pre-decided roles, which may perpetuate social norms. Keywords Women’s empowerment, Review, Food system, Meta-analysis *Correspondence: Miriam Berretta mberretta@3ieimpact.org Full list of author information is available at the end of the article © The Author(s) 2023. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. The Creative Commons Public Domain Dedication waiver (http:// creat iveco mmons. org/ publi cdoma in/ zero/1. 0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 2 of 52 and improve the nutrition of women and their commu- Introduction nities directly and indirectly. Women can improve their Most research on women within food systems focuses on own and their children’s nutritional status when they their roles as caregivers and cooks [1]. However, women have the socio-economic power and social capital to are key actors within food systems, serving as produc- make decisions on food and non-food expenditures and ers, processors, distributors, vendors, and consumers. the ability to take care of themselves and their families Often living in more vulnerable conditions than men [3]. By giving women more control and self-determina- due to societal norms, women face negative, differential tion, women’s empowerment interventions are expected access to affordable, nutritious foods relative to men. to have larger impacts than similar interventions that do Gendered food systems interact with gender equality and not incorporate an empowerment approach. Women’s equity at individual and systemic (community) levels, empowerment interventions may allow women to make as well as in formal (traditions and economic roles) and the choices that are most likely to benefit them while informal (household norms) ways, also referred to as the addressing the broader social and cultural context. As a four quadrants of change (Fig. 1). To achieve food sys- result, women’s empowerment interventions represent a tems transformation, women will need to have adequate key opportunity to improve nutrition-related outcomes, agency and control over resources. Social norms, policies, and women’s empowerment has been highlighted as a and governance structures must be fair and equitable to critical, crosscutting theme for food systems transfor- allow women access to food and livelihood opportuni- mation [4]. However, cross-contextual evidence on the ties. However, many food systems and nutrition inter- factors that cause poorer nutrition outcomes for women ventions are criticized as disempowering because they and how women’s empowerment can improve nutritional can entrench stereotypes by targeting women and girls outcomes is still scant [2]. explicitly in the roles of caregivers or cooks. Gender-transformative approaches (GTA) acknowl- Improvements in women’s empowerment are expected edge the equal role that all genders have in women’s to facilitate women’s interactions with the food system Fig. 1 Theory of change, from Njuki et al. [2] B erretta et al. Agriculture & Food Security (2023) 12:13 Page 3 of 52 empowerment and thus target men as agents of change to environment and dietary measures, a subset of the fac- transform structural barriers and social norms [5]. While tors presented in Fig. 1. Measures of wellbeing are also many women’s empowerment interventions include GTA considered due to their direct link with women’s empow- approaches, women’s empowerment and GTA differ erment. The interventions we identified primarily relate mainly in the following aspects (adapted from [6]): to behavior change communication, skills training, and asset transfers. Interventions were often complex and integrated other components, such as microcredits, self- • Approaches to women’s empowerment often focus help groups, and provision of vitamins supplements. only on women. GTA, on the other hand, aim to They often targeted men as well as women, making them address broader social contexts and avoid essential gender-transformative. zing men and women. • A central element of GTA is intersectionality, i.e., Objectives and research questions considering the interconnections between different The objective of this work was to rapidly synthesize the social identities, such as gender, race, ethnicity, or available evidence regarding the impacts of interventions geographic location. that attempt to empower women and/or girls to access, participate in and take control of components of the food For our purposes, women’s empowerment interven- system. Outcomes considered are limited to measures of tions within the food system are defined as “efforts tar - the food environment and diet. This fills the synthesis gap geted at increasing women’s abilities to make decisions identified by Moore et al. [1]. We also sought to under - regarding the purchase and consumption of healthy stand factors affecting implementation and sustainabil - foods” based on 3ie’s Food Systems and Nutrition Evi- ity, including equity. We specified the following research dence Gap Map [1]. Moore et al. [1] determined that, as questions a priori (Appendix 1): of January 2022, there were 21 evaluations of the impacts of interventions that target women’s abilities to make decision regarding the purchase of healthy foods, for 1. What are the effects of women’s empowerment inter - example by improving decision-making on household ventions within the food system on food availabil- expenditures. However, these studies had not been syn- ity, accessibility, and affordability, of healthy diets or thesized to determine average treatment effects and key nutritional status? contextual factors driving to impact. In this rapid evi- 2. Are there any unintended consequences of such dence assessment, we focus on 10 of those studies which interventions? looked at specific outcomes related to food security, food 3. Do effects vary by context, approach to empower - affordability and availability, diet quality and adequacy, ment, or other moderators? anthropometrics, iron, zinc, vitamin A, iodine status, and measures of well-being. Methodology This rapid evidence assessment provides a novel syn - To respond to these research questions, we conducted a thesis of the available evidence on the impacts of inter- rapid evidence assessment (REA). As far as possible this ventions to support women’s empowerment within the REA is based on the rigorous methodologies adopted food system, contributing to the literature base on both in a systematic review [9]. However, due to time and women’s empowerment and food systems. It is expected resource limitations, the search and screening process to support policymakers, experts, and stakeholders and the data extraction process were shortened [10]. in making evidence-informed decisions regarding the These abbreviated steps allowed for the rapid nature of implementation and design of such interventions. Stake- this rapid evidence assessment. The protocol for the REA holders can use this work to understand how to better was developed a priori in February 2021 and is provided integrate gender-transformative approaches as one char- in Appendix 1. acteristic of feminist development policies, to improve nutritional outcomes in the project and study design Search and screening based on the EGM by Moore et al. [1] process while acknowledging and moving past the use of We did not conduct a new search for impact evaluations, stereotypes. but relied on an existing, open-source evidence gap map In this rapid assessment, we run a meta-analysis and a (EGM) by Moore et al. [1]. The EGM includes all impact barriers and facilitators analysis of interventions on the evaluations and systematic reviews of impact evaluations economic and social empowerment of women with the of interventions within the food system which measure goal of providing them the means and ability to affect outcomes related to food security and nutrition in low- dietary decisions; [7, 8]. As a result, we focus on food and middle-income countries (Appendix 7). Because the Berretta et al. Agriculture & Food Security (2023) 12:13 Page 4 of 52 Table 1 PICOS Criteria Included Excluded Participants People of any age and gender residing in low- and middle-income countries High-income countries Intervention(s) Interventions aimed at increasing women’s empowerment and giving women the All else capabilities to make decisions on the purchase and consumption of a healthy diet Comparison Business as usual, including pipeline and waitlist controls No comparator An alternate intervention Outcome(s) Food security All else Food affordability and availability Diet quality and adequacy Anthropometrics Iron, zinc, vitamin A, and iodine status Measures of well-being Study designs Experimental, quasi-experimental, systematic reviews and cost evidence Efficacy trials Before-after with no control group Cross-sectional studies, etc. search conducted by Moore et al. [1] was not specifically environment (food security and food affordability and focused on women’s empowerment, rather it included availability), diet (diet quality and adequacy, anthropo- women’s empowerment among a variety of other topics, metrics, and micronutrient status), or well-being. Table 1 it is possible that some articles may have been missed. presents the population, interventions, comparisons, out- However, there is no reason to believe that there would comes, and study designs (PICOS), modified from Moore have been any systematic bias in the types of articles et al. [1], employed by this REA. that were omitted or that this would have meaningfully Although we did not perform any new searches for affected results. impact evaluations for this rapid evidence assessment, we conducted a targeted search in Google Scholar look- ing for the qualitative papers related to included studies • The search by Moore et al. [1] was extensive and sys - to allow us to investigate how impacts were achieved. The tematic, covering 12 academic databases and 13 gray search included the name of the program or intervention, literature sources (Appendix 7). Single screening with if available, as well as the country the intervention took safety first was used at both title and abstract and full place in. Eligible qualitative study designs were [11]: text stages. A machine learning classifier was applied to automatically exclude studies with a low prob- ability of inclusion. Although the original search was • A qualitative study collecting primary data using complete in May 2020, the search is continuously mixed methods or quantitative methods of data col- updated with studies added to the EGM through lection and analysis and reporting some information January 2022 considered for this REA. As of January on all the following: the research question, proce- 2022, over 160,000 articles were screened for inclu- dures for collecting data, procedures for analyzing sion in the EGM and 2,647 studies were included data, and information on sampling and recruitment, Appendix 7. including at least two sample characteristics. • A descriptive quantitative study collecting primary Because this REA is based on the search by Moore et al. data using quantitative methods of data collection [1], the same criteria for eligible populations, compara- and descriptive quantitative analysis and reporting tors, and study designs employed by Moore et al. [1] were some information on all the following: the research used for this REA. Moore et al. [1] included interventions question, procedures for collecting data, procedures which targeted women’s empowerment within food sys- for analyzing data, and information on sampling and tems. Women’s empowerment interventions which func- recruitment, including at least two sample character- tioned outside the food system, such as those related to istics. economic empowerment outside of the food system, were • A process evaluation assessing whether an interven- not included. From the 21 studies on women’s empower- tion is being implemented as intended and what is ment interventions included in their EGM, we selected felt to be working well and why. Process evaluations the ten studies evaluating outcomes related to the food may include the collection of qualitative and quanti- B erretta et al. Agriculture & Food Security (2023) 12:13 Page 5 of 52 Table 2 Included outcomes and indicator extracted for evidence synthesis Outcome Indicators* Food security Preferred outcomes: food security indexes and composite scores Secondary outcome: skipped meals Tertiary outcome: reports of insufficient food Food affordability and availability Preferred outcome: per capita food consumption in monetary units Secondary outcome: per capita food consumption in weight Other measures, such as the cost of a food basket, will be considered if these are not available Diet quality and adequacy Preferred outcomes: composite diet scores such as the nutrient rich food index Secondary outcome: dietary diversity and other food variety measures Tertiary outcome: intake of specific foods Anthropometrics Preferred outcomes: body mass index, weight for length, length for age, weight for age Other measures, such as MUAC and ponderal index, will be considered if these are not available Micronutrient (iron, zinc, vitamin A, iodine) status Preferred outcome: measures of content in blood/tissue (ex. hemoglobin levels) Secondary outcome: intake in weight (grams, micrograms, etc.) Tertiary outcome: intake in percentage relative to recommended intake Other measures will be considered Well-being Preferred outcome: perceived well-being Secondary outcome: anxiety *Indicators are listed by preference based on a priori specification. Such a priori specification reduces bias by preventing subjective reporting of outcomes by the team conducting the Rapid Evidence Assessment. Most indicators were ultimately not found in the studies tative data from different stakeholders to cover sub - specified, the model with the most control variables was jective issues, such as perceptions of intervention used. success or more objective issues, such as how an Two team members extracted bibliographic, geo- intervention was operationalized. They might also be graphic information, methods, and substantive data. used to collect organizational information. Substantive data were related to interventions, selected outcomes, population (including gender/age disaggrega- While the identification of qualitative evidence was tion, when available), and effect sizes. Discrepancies were limited to studies linked to the included impact evalua- reconciled through a discussion between the two team tions, the process of data extraction, critical appraisal, members. Qualitative information on barriers and facili- and evidence synthesis was independent. tators to implementation, sustainability and equity impli- cations, and other considerations for practitioners was Data extraction extracted by a single reviewer. Data extraction templates were modified from 3ie’s Included quantitative impact evaluations were standard coding protocol for systematic reviews, reflect - appraised by two independent team members using a ing another shortened step for the purposes of making critical appraisal tool (Appendix 3). Qualitative stud- this assessment rapid (Appendix 2). The primary modi - ies linked to included impact evaluations were critically fication to the tool was a restriction on the number and appraised by a single reviewer using a mixed methods type of outcomes considered. The outcomes considered appraisal tool developed by CASP [12] and applied in were broad and could be measured using a variety of Snilstveit et al. [11] (Appendix 3). indicators. To restrict the number of outcomes extracted, we specified preferred and secondary indicators of inter - Synthesis approach est a priori (Table 2). This limited the analysis to be We provide a narrative summary of the papers identified. conducted to only the specified outcomes. Composite This includes an overall description of the literature and a measures were always preferred over disaggregated ones. general synthesis of findings. Key information from each If multiple analyses were presented considering the same study, such as intervention type, study design, country, out- outcome (ex. Univariate analysis and a regression with comes, measurement type, effect sizes, and confidence rat - control variables), the data from the model preferred ing is summarized in tables. Results from meta-analyses and by the author was extracted. If no preferred model was associated forest plots are presented in the section on the Berretta et al. Agriculture & Food Security (2023) 12:13 Page 6 of 52 FREQUENCIES OF BIAS IN RCT S Low ROB Some concerns REPORTING BIA S 4 2 OUTCOME MEASUREMENT BIAS 6 PERFORMANCE BIA S 4 2 DEVIATIONS FROM INTENDED 5 1 INTERVENTION S CONFOUNDIN G SELECTION BIA S 4 2 UNIT OF ANALYSIS 5 1 A SSIGNMENT MECHANIS M 6 Fig. 2 Risk of Bias of the included randomized control trials findings. Qualitative information is summarized in a section [23]. We also summarize the findings of each study, on implications for implementation and sustainability. including narratively reporting on individual effects, in Table 3. For all outcomes except micronutrient status, Meta‑analysis the metrics were determined to be sufficiently similar to In addition to presenting individual effect estimates for warrant a joint analysis in addition to the presentation of all six outcomes, we conducted five meta-analyses to pro - individual effects. vide summary effect estimates on the five outcomes for To compare the effect sizes, we converted all of them which we had sufficient data. This meta-analysis provides to a single metric, Cohen’s d. We then converted all additional value relative to presenting the individual Cohen’s d to Hedges g to correct for small sample sizes. effect estimates by presenting a summary effect estimate. We chose the appropriate formulae for effect size cal - Meta-analyzed effects have the benefit of being sup - culations in reference to, and dependent upon, the data ported by a broader (Figs. 2 and 3), potentially more gen- provided in included studies. For example, for studies eralizable evidence base than individual point estimates. reporting means (X) and pooled standard deviation (SD) Previous works have statistically synthesized similar evi- for treatment (T) and control or comparison (C) at fol- dence, for instance, on food security and food affordabil - low-up only, we used the following formula: ity and availability [13, 14], anthropometrics measures X − X Tp+1 Cp+1 [14, 16, 17] micronutrients status [18–20], diet quality d = SD and adequacy [21, 22], Because only ten studies were included, meta-analysis If the study did not report the pooled standard deviation, was conducted at the outcome (column 1, Table 2), not it is possible to calculate it using the following formula: the indicator level (column 2, Table 2). However, due to 2 2 variations in the indicators used and their interpreta- (n − 1)SD + (n − 1)SD Tp+1 Cp+1 Tp+1 Cp+1 SD = tion, we also present the standardized effect estimates for p+1 n + n − 2 Tp+1 Cp+1 each study in each forest plot (Figs. 4, 5, 6, 7 and 8) and Appendix 6. The decision to conduct meta-analysis was where the intervention was expected to change the made on a case-by-case basis after considering if the indi- standard deviation of the outcome variable, we used the cators adequately captured the same underlying concept standard deviation of the control group only:For studies SOURCES OF BIAS B erretta et al. Agriculture & Food Security (2023) 12:13 Page 7 of 52 FREQUENC IE S OF BI AS IN QE D Fig. 3 Risk of bias of the included quasi-experimental included studies Fig. 4 Forest plot showing the effect of empowerment interventions on food security outcomes X X −X p+1 Tp+1 Cp+1 reporting means (X) and standard deviations (SD) for d = = For studies reporting SD SD p+1 p+1 treatment and control or comparison groups at baseline mean differences between treatment and control, stand - (p) and follow-up (p + 1): ard error (SE) and sample size (n): X −X p+1 p d = For studies reporting mean differences SD p+1 p+1 (∆X) between treatment and control and standard devia - d = √ SD n tion (SD) at follow-up (p + 1): Berretta et al. Agriculture & Food Security (2023) 12:13 Page 8 of 52 Fig. 5 Forest plot showing the effect of empowerment interventions on food affordability/availability outcomes Fig. 6 Forest plot showing the effect of empowerment interventions on diet quality and adequacy For studies reporting regression results, we followed the 1 1 approach suggested by Keef and Roberts (2004) using the d = + n n T C regression coefficient and the pooled standard deviation of the outcome. Where the pooled standard deviation of where n denotes the sample size of treatment group and the outcome was not unavailable, we used the regression control. We used the following where total sample size coefficients and standard errors or t-statistics to do the information (N) is available only (as suggested in Polanin following, where sample size information is available in [34]): each group: B erretta et al. Agriculture & Food Security (2023) 12:13 Page 9 of 52 Fig. 7 Forest plot showing the effect of empowerment interventions on weight relative to height Fig. 8 Forest plot showing the effect of empowerment interventions on wellbeing 2t 4 d = T.INV.2T (exact p value, (n − 1)) d = √ Var = + N 4N where outcomes were reported in proportions of individ- When necessary, we calculated the t statistic (t) by uals, we calculated the Cox-transformed log odds ratio dividing the coefficient by the standard error. If the effect size [35]: authors only report confidence intervals and no standard √ error, we calculated the standard error from the confi - d = Log Odds Ratio v × dence intervals using the following: where OR is the odds ratio calculated from the two-by- SD = N × (upper limit - lower limit) 3.92 two frequency table. We fitted a random effects meta-analyses model when If the study did not report the standard error, but did we identified two or more studies that we assessed to be report t, we extracted and used this as reported by the sufficiently similar. We assessed heterogeneity using the authors. If an exact p value was reported but no standard DerSimonian–Laird estimator by calculating the Q sta- error or t, we used the following Excel function to deter- 2 2 tistic, I , and τ to provide an estimate of the amount of mine the t-value. variability in the distribution of the true effect sizes [23]. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 10 of 52 Table 3 Summary of included studies Study reference and country Study design Intervention Authors’ interpretation of effects Ahmed et al. [24] [Bangladesh] Randomized controlled trial The Transfer Modality Research Initiative ( TMRI) provided: All four interventions (Cash transfer, food transfer, cash + food, Cash or food transfers, with or without nutrition behavior change cash + BCC) increased monthly per capita food consumption, communication (BCC) for rural women living in poverty daily per capita intake caloric, and food consumption score. The effects are slightly higher for cash + BCC, particularly from the food consumption score Bandiera et al. [25] [Bangladesh] Randomized controlled trial The Targeting the Ultra-Poor program provided: Food security among women improved, but there was no effect on (a) livestock assets and skills transfers for the poorest women. mental health status Women were offered a menu of assets to support income gener - ating activities. Assets included livestock and goods for small-scale retail operations, tree nurseries and vegetable growing (b) each asset was offered with a package of complementary training and support Blakstad et al. [26] [ Tanzania] Randomized controlled trial The Homestead Food Production Program provided: Household dietary diversity score increased, but there was no effect (a) agricultural training for women and inputs to promote home- on food security stead food production (b) nutrition and public health counseling for women to improve diet and health-related behaviors Bonuedi et al. [27] [Sierra Leone] Quasi-experimental design The Pro-Resilience Action (PROACT ) project provided: LANN, combined with the cash crop intervention, improved dietary (a) the LANN was a participatory community-based intervention diversity and food consumption among women and children. involving nutrition education, behavioral change communica- LANN alone did not have any effect tion and awareness creation on the benefits of consuming diverse diets, proper child feeding and water, sanitation, and hygiene ( WASH) practices, and sustainable agriculture and natural resource management in rural areas (b) a cash crop, income-oriented intervention aimed at enhancing economic access to nutritious foods. It included a nutrition pro- gram directed at improving nutrition knowledge and stimulating nutrition-sensitive spending and allocation of other household resources Deninger et al. [28] [India] Quasi-experimental design The District Poverty Initiative in India: The creation of SHGs had mixed effects on food consumption (RS/ Supported new self-help groups for women living in poverty year), energy intake per capita (kcal/day), and protein intake p.c. in India by training leaders and accountants from new self- (g/day) among poor, non-poor, and poorest of the poor. The three help groups in basic management and accounting. The SHGs outcomes improved among the poor. Energy intake increased also combined savings generation and micro-lending with social for the poorest of the poor, but the other two outcomes were not mobilization significant for them. None of the outcomes improved among the non-poor Emran et al. [29] [Bangladesh] Quasi-experimental design The Targeting the Ultra-Poor ( TUP) program provided: The probability of having two meals a day, the probability of having (a) health, education, and training for poor women, including sufficient food to meet the household’s needs, and grain stock trainings in livestock and poultry rearing; fruit, vegetable, and increased. The highest impacts were reported on the first two herb cultivation; operation of tree nurseries; and village vending outcomes (b) vitamin A supplements for children under five B erretta et al. Agriculture & Food Security (2023) 12:13 Page 11 of 52 Table 3 (continued) Study reference and country Study design Intervention Authors’ interpretation of effects Haque et al. [30] [Bangladesh] Randomized controlled trial The Suchana project provided: The Suchana project increased food consumption during preg- Training on agriculture, aquaculture, and market development, nancy, the consumption of vitamin A capsules after last delivery, including challenging the gender barriers to agriculture, health and the consumption of at least 100 IFA tablets during pregnancy. and nutrition practices among the beneficiary women, husbands, Greater impacts were reported for the first two outcomes and other household members Heckert et al. [31] [Burkina Faso] Randomized controlled trial The Enhanced Homestead Food Production (E-HFP) program The E-HFP program reduced wasting but had null effects on hemo - provided: globin levels among children (a) agricultural assets (b) behavior change communication on agricultural activities, optimal infant and young child feeding, health, hygiene, and care practices Marquis et al. [32] [Ghana] Quasi-experimental design The Enhancing Child Nutrition through Animal Source Food The program had a positive effect on height-for-age z score, a Management (ENAM) program provided: negative effect on weight-for-age z score, and a null effect on BMI- (a) microcredit loans for-age z score of preschool-aged children (b) weekly nutrition, technical, and entrepreneurship training on viable income-generation activities Pan et al. [33] [Uganda] Quasi-experimental design This large-scale agricultural extension program for smallholder The program reduced meals skipped, worries about insufficient women farmers provided: food, and limited variety of food among smallholder women. It (a) training through model farmers increased per capita food consumption (b) easier access to and affordability of seeds sold through farmers serving as community agriculture promoters Berretta et al. Agriculture & Food Security (2023) 12:13 Page 12 of 52 We were unable to explore heterogeneity using modera- Anthropometric measures, micronutrient status, and tor analyses due to the small number of included studies. well-being outcomes were less common (n = 2 each). We found nine qualitative reports related to seven Qualitative synthesis interventions. Additional qualitative information was not The meta-analysis conducted with the quantitative data found for the remaining interventions. The qualitative has been complemented by a thematic synthesis utiliz- components of the main studies and additional studies ing the extracted qualitative data. Qualitative data were were minimal and primarily focused on contextual infor- synthesized thematically by a single team member and mation from the researchers. Many of the qualitative reviewed by two other team members. Themes consid - studies used focus group discussions or key informant ered related to non-nutrition impacts, barriers and facili- interviews to better understand participants’ lived reali- tators to impact, and cost evidence. ties. Qualitative data contextualized results of empower- ment interventions and food and nutrition security based Results on the differing intervention locations and intersect - Characteristics of the included studies ing social, cultural and gender norms that influence the We included ten studies retrieved through the systematic impacts on nutrition and other key outcomes. search done for the Food Systems and Nutrition Evidence All the randomized controlled trials except Blakstad Gap Map, conducted in January 2022 (Table 3). An addi- et al. [26] have an overall rating of ‘some concerns’, mainly tional, low-quality systematic review was identified and due to reporting bias, performance bias, and selec- excluded from analysis. Four of the ten included stud- tion bias (Fig. 7; Appendix 5). Deininger and Liu [28] ies were implemented in Bangladesh, while the remain- also encountered issues related to deviation from the ing studies where in Burkina Faso, Ghana, India, Sierra intended interventions and the unit of analysis did not Leone, Tanzania, and Uganda. The four studies in Bang - correspond to the unit of randomization. ladesh represent unique evaluations of a cash transfer Two quasi-experimental studies were rated as having program, an agricultural training program, and two fully a low risk of bias (Fig. 8; [32, 33]), one study as having independent evaluations of Targeting-Ultra-Poor pro- ‘some concerns’ [29], and one as having a high risk of bias gram (TUP) with a time gap of eight years and some- [27]. The major sources of bias were related to reporting what different intervention designs. More information bias, spill-over, cross-over and contamination, perfor- on study characteristics can be found in Additional file 1: mance bias, and confounding. Table S1. Randomized controlled trials (n = 4) and difference-in- What are the effects of women’s empowerment difference were the most common designs (n = 4). Half interventions on food environment, diet, and well‑being of the studies using difference-in-difference also used outcomes? statistical matching (n = 2). One study used statistical Standardized effects are reported in Table 7 in Appendix matching alone and one used regression discontinuity 6, calculated as outlined in the Methodology section. The to identify counterfactuals. Nine additional qualitative meta-analysis results of the random effects model are papers associated with seven interventions were also reported in Table 4. We could not run a meta-analysis on identified and included. micronutrient status because the two studies looking at it Almost all studies provided training (n = 8). Some also measured different underlying concepts which could not provided asset transfers (n = 6) and behavior change be meaningfully combined. communication (n = 3; Tables 3, 6 in Appendix 6, and Additional file 1: Table S1). Behavior change communi- Effect of women’s empowerment interventions on food cation interventions generally communicated messages security outcomes is promising about women’s empowerment and women’s roles within Our analysis of the effects of women’s empowerment their communities. Often, they targeted men, making interventions suggests they improved food security them gender-transformative. Training and educational outcomes overall ( µ = 0.24 [95% CI: 0.001 to 0.47 ], interventions focused on agriculture and/or nutrition, p = 0.048 , Fig . 4). Women receiving these interventions but some also considered entrepreneurship and water, had a 59.5% chance of having food security scores above sanitation, and hygiene. Asset transfers were largely the mean in the control group. There was significant vari - related to cash or agricultural inputs, including livestock. ation in the size of the effect, ranging from 0.07 in Tanza - Food affordability and availability outcomes were the nia, to 0.67 in Bangladesh. most common (n = 5). Diet quality and adequacy and We included four studies which reported the following food security outcomes were also common (n = 4 each). indicators: food security index (whether the household B erretta et al. Agriculture & Food Security (2023) 12:13 Page 13 of 52 Table 4 Meta-analytical results Outcomes Specific outcomes indicators # of included effects Overall effect size [95% CI] Estimated percentile change Heterogeneity of Range of effects (total number of compared to control group overall effect (Q and beneficiaries) [95% CI] I^2) Food security Food security index—whether 4 (12,545) 0.24* [0.00; 0.47] 9.5% [0; 18.1%] 111.16***, 97.3% 0.07 to 0.67 HH had surplus food or deficit, enough food to eat, and could afford to eat two meals a day; Food availability: sufficient food to meet the household’s needs; Household food insecu- rity access scale; Skip meals Food affordability/availability Food consumption score; 6 (12,972) 0.23** [0.09; 0.38] 9.1% [3.6%; 14.8%] 187.27***, 91.99% − 0.11 to 0.49 Total food consumption expenditure in the 12 months preceding the survey (Food production and market purchases) (LOG)-Household; Food consumption (RS/year); Grain stocks (kg); Per capita food consumption Diet quality and adequacy Household dietary diversity; 4 (16,025.5) 0.09** [0.06, 0.12] 3.6% [2.4%; 4.8%] 0.53***, 0% 0.076 to 0.14 Additional food consumed during pregnancy; Protein intake p.c. (g/day) in the total population; Variety of foods consumed Weight relative to length Weight-for-length z score; BMI- 2 (1156.6) 0.12* [0.00, 0.23] 4.8% [0; 9.1%] 0.12, 0% 0.06 to 0.12 for-age z score Well-being outcomes Mental health index; Worry 2 (10,100) 0.08 [0.01; 0.15] 3.2% [0.4%; 6%] 2.9*, 65.6% − 0.11 to 0.04 about insufficient food * is p < 0.05, ** is p < 0.01 and *** is p < 0.001. For more information on the results and the studies please see Additional file 1: Table S1 Berretta et al. Agriculture & Food Security (2023) 12:13 Page 14 of 52 had surplus food or deficit, enough food to eat, and could Effect of women’s empowerment interventions on diet quality afford to eat two meals a day), household food insecurity and adequacy outcomes is promising assessment scale (HFIAS), skipped meals, and food avail- Our analysis of the effects of women’s empowerment able to meet a household’s needs of two meals a day [25, interventions suggests they improved diet quality and 26, 29, 33]. All studies provided training or education, adequacy ( µ = 0.09 [95% CI: 0.06 to 0.12] , p < 0.01, mostly related to agriculture. Three also provided some Fig. 6). Women receiving these interventions had a 53.6% form of asset transfer [25, 29, 33]. chance of having diet quality and adequacy scores above Two studies were assessed as having some concerns the mean in the control group. The variations among the related to risk of bias [25, 29] and two were assessed as range of effects were not as high as for other outcomes, low risk of bias [26, 33]. ranging from 0.08 in India to 0.14 in Sierra Leone. Four studies reported impacts related to diet quality Effect of women’s empowerment interventions on food and adequacy, such as dietary diversity and amount of affordability and availability outcomes is promising food or protein consumed [27, 28, 30, 33]. All four stud- Our analysis of the effects of women’s empowerment ies employed training/education interventions focused interventions suggests they improved the availability on agriculture [27, 30, 33] or enterprise/accountability and affordability of food ( µ = 0.23 [95% CI: 0.09 to 0.38] [28]. Two studies also transferred assets [27, 33], and one p < 0.01, Fig. 5). Women receiving these interventions included a behavioral change communication component had a 59.1% chance of having food affordability and avail - [27]. ability scores above the mean in the control group. There One study was scored as low risk of bias [33], two were was significant variation in the size of the effect, ranging scored as having some concerns [28, 30], and one was from 0.08 in Uganda, to 0.49 in Bangladesh. rated as high risk of bias [27]. Food affordability and availability was measured in five included studies, per capita food consumption, food con- Effect of women’s empowerment interventions sumption per capita (Rs/year), total food consumption on anthropometrics is promising but there is a lack expenditure (food production and market purchases in of evidence the 12 months preceding the survey), and grain stock (kg) Our analysis of the effects of women’s empowerment [24, 26, 2829, 33]. We included two estimates for Ahmed interventions suggests they improved measures of et al. as the results were reported for independent sam- weight relative to height ( µ = 0.12[ 95% CI: 0.002to0.23] , ples from the North and South of Bangladesh, without an p = 0.046 Fig. 7). Children of women receiving these overall estimate for all the areas. interventions had a 54.8% chance of having anthropo- All studies but Deininger and Liu [28] included assets metrics scores above the mean in the control group. transfer, such as cash, cash crops [24, 27], or livestock, Two studies reported impacts on anthropomet- seeds, or vitamin A supplements [29, 33]. All studies, ric measures of children based on WHO z-scores [31, except Ahmed et al. [24] included trainings or education 32]. Both studies transferred agricultural [32] or finan - on nutrition [27], or agriculture [29, 33], or enterprise/ cial assets [32]. The Heckert and colleagues’ study also accountability [28]. Two studies also included a behavior included a behavioral change communication strategy, change communication component [24, 27]. while Marquis and colleagues included entrepreneur- Ahmed and colleagues also reported increases in ship training. Marquis et al. [32] also report a decrease monthly food consumption per capita in both northern in weigh-for-age (g = − 0.42 [95% CI: − 0.77 to − 0.06]) and southern regions of their intervention area (North and an increase in height-for-age (g = 0.40 [95% CI: 0.04 areas: g = 0.32 [95% CI: 0.27 to 0.38]; South areas: g = 0.22 to 0.75]). Heckert and colleagues were scored as having [95% CI: 0.16 to 0.27]) and per capita daily intake caloric some concerns about bias while Marquis et al. [32] had (North areas: g = 0.22 [95% CI: 0.17 to 0.28]; South areas: low risk of bias. g = 0.09 [95% CI: 0.043 to 0.15]). Three other intervention arms (provision of food, cash, or food plus cash) were Effect of women’s empowerment interventions also evaluated. However, we were not able to include on micronutrient status is promising but there is a lack them in the meta-analysis as they were not comparable to of evidence the other studies. All three reported similar impacts. Two studies considered the effects of women’s empow - Only Bonuedi et al. were assessed as having a high risk erment interventions on micronutrient status, but these of bias, the remaining studies have either some concerns could not be meaningfully combined in a meta-analysis [24, 28, 29] or low risk of bias [33]. because they measured different underlying concepts. B erretta et al. Agriculture & Food Security (2023) 12:13 Page 15 of 52 Haque et al. found that Suchana’s gender-transformative interventions allowed women to accumulate savings and approach, which encompassed a portfolio of agriculture spend more judiciously, rather than consistently respond- and entrepreneurship trainings, increased the consump- ing to immediate needs. tion of iron, folic acid tablets (g = 0.25 [95% CI:0.21 to Two interventions which combined training with 0.28]). Heckert et al. evaluated an agricultural education improved accessibility of agricultural assets increased and behavior change communication strategy, but they opportunities for paid work. The agricultural interven - found no effect on hemoglobin levels (g = − 0.10 [95% CI: tion in Uganda resulted in an increase in work for wages − 0.03 to 0.23]). Both studies were rated as having some and freed up off-farm work times for the entire house - concerns about bias. hold, including women [33]. Similarly, because of the TUP program, the labor market choices of household Effects of women’s empowerment interventions on mental members aside from the targeted woman also shifted well‑being outcomes is not significant and there is a lack [25]. However, women themselves did not have increased of evidence labor participation. Women in the program spent most Our analysis of the effects of women’s empowerment of their time at home and were generally not employed interventions shows no effect on mental health outcomes outside of the home [38]. In fact, women reported that ( µ = 0.08[ 95% CI: 0.01to0.14], p = 0.088 , Fig . 8). Bandi- they preferred to stay at home due to low pay and social era et al. [25] reported a mental health index constructed stigma in workplaces. based on self-reported happiness and mental anxiety, Similarly, two interventions focusing on household while Pan et al. [33] measured the level of worries regard- farming for improved nutritional outcomes were labor ing insufficient food. Both studies evaluated assets trans - and time intensive, which resulted in high attrition [26]. fer interventions, such as livestock, seeds, vegetables This additional labor was an increased burden on women growing, and specific trainings which accompanied to and took away from their time to acquire and prepare the transfers. Pan et al. [33] paper was assessed as hav- food for their families [27]. When data collection coin- ing a low risk of bias, while Bandiera et al. [25] paper was cided with harvest months in Sierra Leone, women’s assessed as having some concerns related to performance involvement in the farming activities increased their time bias. constraints and adversely affected caregiving practices. Implications Barriers and facilitators Implications for non‑nutrition outcomes Restrictive social norms preventing women from being Authors of many of these studies concluded that the able to take advantage of the interventions as intended interventions accomplished their goals of supporting was a common barrier. Structural gender barriers act as women’s empowerment, often by introducing gender- a driver of inequality in the household and community, transformative approaches which challenged traditional as specified in Njuki et al. theory of change (Fig. 1). In social norms. The Enhanced Homestead Food Produc - highly patriarchal societies, such as Sierra Leone, deeply tion (E-HFP) program in Burkina Faso included a gen- entrenched social and cultural norms marginalize der-transformative approach in which it improved men’s women, restrict their decision-making and exclude them perceptions of women as farm managers and increased from accessing or controlling household resources [27]. respect and communication in agri-business activities Single-focus interventions that only targeted nutrition or [31]. The accompanying behavior change communication value chain inputs without behavior change communica- intervention allowed mothers to better communicate tion related to social norms were not able to fully real- with men to improve familial support and adopt positive ize potential impacts because entrenched norms were nutrition behaviors, such as improved feeding practices. significant barriers to long-lasting change [33]. Even if Similarly, the Suchana program in Bangladesh resulted women were given the tools to work outside the home in improvements in women’s empowerment and mater- or own assets, they were often blocked from leveraging nal healthcare practices using a gender-transformative these tools by norms that dictate how women can act and approach [30]. Women became more confident to dis - work [33]. Gender-transformative approaches address cuss issues around food and management of household this social barrier by including men to ensure that the full resources with their partners [27]. Self-help group partic- impacts of interventions can be leveraged and realized as ipation improved social awareness and leadership skills. intended. Women mobilized to protest child marriage and violence In the TUP program, asset transfers that were intended against women in their communities [37]. The Target - for women members of households were controlled by ing-Ultra-Poor program (TUP) in Bangladesh increased men due to social norms [39]. Social norms delineated saving and borrowing opportunities for women. These what type of assets women were allowed to own. Larger Berretta et al. Agriculture & Food Security (2023) 12:13 Page 16 of 52 livestock, like cattle, were automatically perceived to over some of the transferred assets [39]. Interventions belong to men because they were higher in value and which took place at the home and approached women as traded more often. Their sale required an adult male’s caregivers and providers may have further perpetuated consent, which restricted women’s ability to own and the stereotype of women within these roles [37]. manage them. Restrictions almost always came from Unfortunately, the long time needed to change social jealous or violent husbands. When the TUP transferred norms was a barrier to these interventions achieving small livestock such as poultry, that women more often impact in the short period in which they were evalu- owned, it was easily controlled by women [39]. Reli- ated. The theory of change from women’s empowerment gious norms also played a role in restricting women’s interventions to improved nutrition outcomes assumes public movements. Care responsibilities were reinforced a change in social norms, which requires a significant by conservative social norms for women in Bangladesh, amount of time (Fig. 1). Change within the food sys- where women were demarcated as primary caregivers in tem is a dynamic process which often depends on other the home [37]. changes outside the scope of these interventions. Moreo- In some contexts, community and men’s support also ver, change processes are not straightforward and can facilitated improvements in outcomes, demonstrating be accompanied by setbacks, sometimes occurring par- the importance of gender-transformative approaches allel to positive effects. Behavior change communica - that actively challenge gender norms and power inequi- tion can be slow to expand women’s empowerment and ties between genders. In the Homestead Food Production households’ social status and networks [24]. Impacts intervention in Tanzania, women who lived near neigh- often become apparent in the long-term when founda- bors who also grew crops at home had higher dietary tional improvements consolidate and are dependent on diversity [26]. Participants who were close to markets internal and external factors. Food and nutrition security were able to access, trade and procure food and related and women’s empowerment may need to be achieved in items easier than those who were farther away [25]. If stages, according to different resources and opportuni - husbands and other men in the household or community ties [33]. For example, in India, the District Poverty Ini- were more receptive to change, then progress was more tiative fostered group formation and supported more visible with women in the TUP [37]. If a husband was mature groups, which could have significant economic more open to his wife engaging in out-of-house activities, benefits in the long term [28]. Because the study utilized livelihood strategies were more successful. data from three and six years after group formation, the Multi-component interventions may leverage synergistic research implies there may have been impacts on capital effects to have greater impacts than the individual com - endowments and economic effects on individuals and the ponents would have [27]. Complementary program arms group itself. Authors of evaluations that occurred within can reinforce each other in achieving desired results and 12 months of the interventions’ end indicated that a more reduce implementation costs to achieve the same objec- comprehensive understanding of women’s empowerment tives [27]. The asset-based component of the PROACT and nutritional outcomes would require longer-term and program in Sierra Leone had little effect. However, when more frequent data collection [26, 31]. combined with a behavior change communication com- Specific characteristics of the target group can affect ponent, it increased women’s decision-making power, impacts and may explain heterogeneity in results. House- shifting women’s roles in the household, and expand- hold decisions regarding assets and nutrition were ing women’s ability to work outside the house. Behavior shaped by local ecological and economic conditions [24]. change communication components of the TMRI pro- In India, target groups that were the poorest saw the gram in Bangladesh combined with the incentive of asset largest asset accumulation and empowerment improve- transfers allowed women’s sustained participation and ments. This resulted in the poorest benefitting both achieved an overall improvement in household indicators socially and economically [28]. Interventions which lev- over the course of the program [38]. erage existing groups may experience high attrition if the Interventions which do not address equity can be groups themselves experience attrition. For example, the less successful and re-enforced social norms. Often, Enhancing Child Nutrition through Animal Source Food entrenched norms and roles were not acknowledged Management program targeted microcredit groups, and within included interventions [40]. Failure to address experienced significant attrition among those who were these norms may have resulted in some interventions not benefiting from the loan program [32]. This may not being unsuccessful. This was seen in the Bangladesh asset have been observed if the intervention targeted women transfer program which did not address norms around directly and did not work through the microcredit group. livestock ownership and resulted in men gaining control B erretta et al. Agriculture & Food Security (2023) 12:13 Page 17 of 52 Cost information as food security and food affordability and availability. Cost reporting was low (n = 3). When studies reported Impacts seem to reduce along the causal chain. Some of cost data, either through cost per participant or cost ben- the more final outcomes, such as anthropometric and efit analysis, the benefits generally outweighed the costs. well-being measures, can take years to meaningfully The District Poverty Initiative in India found that net change. As such, modest early effects may imply longer- present value of benefits from the project were approxi - term change. mately $1,690 million, significantly more than the project Insights from the qualitative evidence suggest that cost of $110 million. Even if benefits only lasted for one women’s empowerment interventions best influenced year the estimated benefits still significantly exceed pro - food environment and diet outcomes when gender ject costs, with a benefit–cost ratio of 1.5 to 1 [28]. The and social norms were considered. However, often, TUP program in Bangladesh also showed that average entrenched norms and roles were not acknowledged in benefits, including increased household welfare, were these interventions [40]. When community, and espe- 3.21 times larger than costs. Big push programs, like the cially male support, was found, it may have facilitated TUP, required large investment. However, in this case, impact. Including gender-transformative approaches in it resulted in cost-effective and sustainable change in women’s empowerment interventions may be essential household welfare, including nutrition [37]. to challenge and overcome existing social norms which Multi-component interventions can be cost-effective often prevent the achievement of intended impacts. because they combine complementary initiatives, such Such transformative approaches may be necessary to as interventions targeting nutrition and social norms. allow women to fully benefit from ongoing interventions. This was seen in PROACT where impacts were only Restrictive social norms may prevent women from taking achieved once a behavior change component was added full advantage of the interventions and reduce potential to the asset transfer [27]. Similarly, when added to an impacts. asset transfer program, the TMRI women’s empower- Although women’s empowerment interventions are ment behavior change communication component costs promising approaches for improving measures of the $50 per beneficiary per year, which is a relatively low cost food environment and diet, interventions may need to compared to stand-alone behavior change communica- move beyond women’s empowerment interventions tion interventions [24]. Low-cost additional activities include GTA and gain the buy-in of men and the com- can have greater impact than expected, especially when munity. This can result in increased power of women in integrated with other components. The training of model household decision-making while also sensitizing men to farmers in Uganda improved cultivation methods at rela- women’s pursuits of work outside of the home [41]. GTA tively low cost when compared with the cost of inputs, require cultural and social adaptation to local contexts such as a high-yield and drought-resistant seeds. Both through strengthened local partnerships and capacities training and the provision of inputs improved women’s while considering intersectionality, e.g., by considering efficiency in household gardens [33]. However, when cal - different interconnections between gender, socioeco - culating costs, the additional cost of such labor should nomic class, and caste divisions. GTA and intersection- not be ignored, especially because these costs are often ality, both characteristics of feminist development policy, born by the women that these interventions are trying to are crucial to progress on gender equality and leverage help [26]. the full potential of policies and interventions. Similarly, interventions should attempt to improve women’s social Discussion capital so they can better control and decide how to Overall, our analyses suggest women’s empowerment acquire and prepare food for their families [39]. Focus- interventions can improve measures of the food environ- ing on the duration of interventions is also important. ment and diet. We find significant and positive effects Long-term interventions may be needed to account for on food security (0.24 [95%CI: 0.00 to 0.47], n = 4), food slow processes, such as changing social norms. Multi- component interventions, which combine a livelihoods affordability and availability ( µ = 0.023[95% CI: 0.06 to component (asset transfer or financial services) with 0.38] , n = 6), and diet quality and adequacy ( µ = 0.09 behavioral change communication and advocacy, may be [95% CI: 0.06 to 0.12 ], n = 4). With two studies consider- more effective than interventions focusing on just liveli - ing outcomes related to weight-for-length ( µ = 0.12 [95% hoods or behavioral change. CI: 0.00 to 0.23 ]) and wellbeing ( µ = 0.08 [95% CI: 0.01 With ten included studies, the evidence base is small, to 0.15 ]) each, the evidence is too limited to draw con- which can reduce generalizability. Variation in the meas- clusions. Although impacts on diet quality and adequacy, ures considered in the meta-analysis may drive het- anthropometrics, and well-being were positive, they were erogeneity in results. However, the overall quality of the smaller than impacts on more proximate outcomes, such Berretta et al. Agriculture & Food Security (2023) 12:13 Page 18 of 52 evidence is fair with most of the studies (n = 6) rated as implemented in Sub-Saharan Africa or South Asia, leav- having ‘some concerns’ regarding bias. Three studies ing evidence gaps in Central America, South America, were assessed as having ‘low risk of bias.’ Given the low and Central Asia. Most studies were implemented in con- number of studies available and potential biases, the texts that were particularly patriarchal and restrictive for results should be interpreted with some caution. women, meaning that results in more egalitarian socie- Although the evidence was generally of high quality, ties may be different. Although we were able to run a five we had some concerns related to reporting, performance, meta-analysis, interpretation of the results is limited due and selection bias of the randomized controlled trials. to the low number of studies and variation in the indica- Within the quasi-experimental studies, we found issues tors synthesized. Cost data will also be needed to deter- related to reporting bias, spill-over, cross-over and con- mine if these impacts are cost-effective. To determine the tamination, performance bias, and confounding. Some sustainability of impacts over time, future studies should authors reported issues with incomplete or low-quality have longer intervention periods to ensure accurate cap- data, for instance, incomplete children’s health or vacci- ture of perceived impacts. Qualitative data can add rich nation records. Moreover, some children aged out dur- depth to quantitative findings by adding context, expe - ing the evaluation period making the data inconsistent. riences and meaning to the lived experiences of project Other studies did not collect data across seasons, an participants. Mixed-methods studies should focus on essential element when collecting data on agriculture identifying impacts and then using qualitative research to outcomes, which can act differently across seasons. Short interrogate how these impacts were achieved. Studies in interventions and short data collection periods might places with caste divisions, such as India or Bangladesh, also prevent impacts from being identified. These limita - could have benefited from a disaggregation in the experi - tions could result in findings being somewhat unreliable. ences and outcomes of women and households from dif- ferent castes. Future studies should try to avoid outcome Strengths, limitations & future directions measurement bias, reporting bias, spill-over, cross-over The interventions considered in this analysis were multi- and contamination, performance bias, confounding, and faceted, often considering two or three components: selection bias. Future studies should also ensure that data behavior change communication, training, and asset collection is representative of different seasons and con - transfers. As such, it is not possible to determine which textual changes, to avoid incomplete or insufficient data of these approaches is most effective. Future work can [26, 30, 32]. isolate the effects of these different pathways, as done by Due to the rapid nature of this work, results should be Bonuedi et al. [27], to determine which of these compo- interpreted with caution. The studies included in this nents is most effective. review are those found through the systematic search for The meta-analyses presented here combine disparate the EGM produced by Moore et al. [1] as of January 2022. indicators of broad concepts. The combined analysis of It is possible that a more sensitive and targeted search these different indicators is justified because they meas - strategy would identify additional studies. Moreover, the ure the same underlying concept. However, the variation REA is limited in the scope of interventions included. in indicator used by each study may explain the heteroge- Only those which take place within the food system are neity in results. For example, the analysis on food security considered; interventions functioning outside of the food combines a food security index, household food insecu- system may influence nutrition outcomes but have not rity assessment scale, number skipped meals, and indi- been considered. cator of whether food is available to meet a household’s needs of two meals a day. The framing of food attributes as positive versus negative can affect attitudes toward Appendices food [42], so framing questions around food security and insecurity may produce different results. As such, indi - Appendix 1: Rapid Evidence Assessment viudal effect estimates should also be considered and are on Women’s Empowerment in Food Systems reported within each forest plot and in Appendix 6. Sum- Interventions – Protocol maries of the effects identified by each study are provided Background in Table 3. Future work should move toward standardiz- The problem, condition, or issue ing measurement to allow for better comparability. Some Women are key actors within food systems, serving as of such efforts already exist, but should be further sup - producers, wage workers, traders, processors, and con- ported to allow for stronger synthesis [43, 44]. sumers. Women also face differential outcomes related to Given the limited evidence base, more research accessing and affording nutritious foods or a healthy diet. is needed in this field broadly. All the studies were Some evidence shows that women—often living in more B erretta et al. Agriculture & Food Security (2023) 12:13 Page 19 of 52 vulnerable conditions than men due to societal norms— Methodology can improve their own and their children’s nutritional To respond to these research questions, we will conduct status when they have socio-economic power to make a rapid evidence assessment, based on a systematic lit- decisions on food and non-food expenditures (especially erature search of key academic databases. Literature will accessing resources) and can take care of themselves and be screened for quality and summarized visually and in their families [3]. As a result, women’s empowerment a narrative format. A rapid evidence assessment is based interventions represent a key opportunity to improve upon the rigorous methodology adopted in a systematic nutrition-related outcomes. There is substantial agree - review; however, many steps are shortened [10]. ment about pathways to improve women’s empowerment in food systems. However, cross-contextual evidence Criteria for including and excluding studies in the review on the factors that cause poorer nutrition outcomes for (PICOS) women, and how women’s empowerment can improve nutritional outcomes is still scant [2]. Criteria Included Excluded The interventions Participants People of any age High-income countries We will include interventions that integrate activities and gender residing in low- and middle- to empower women and/or girls to access, participate income countries and take control in components of the food system, (L&MICs) for example improving decision-making on household Intervention(s) Interventions aimed All else expenditures. We have extracted relevant papers from at increasing women’s empowerment and the Food Systems and Nutrition evidence gap map that giving women the have any intervention component relating to women’s capabilities to make empowerment. decisions on the pur- chase and consump- tion of a healthy diet Expected theories of change Comparison Business as usual, No comparator Our theory of change is based on the pathways devel- including pipeline and oped by Njuki et al. [2] to presume that women’s empow- waitlist controls An alternate interven- erment can lead to improved nutrition with a variety of tion other influencing factors. Gendered food systems inter - Outcome(s) Food affordability, All else act with gender equality and inequality in a four-dimen- accessibility, and avail- sional space: individual, systemic, formal, and informal. ability Iron, zinc, vitamin A, and iodine status Rationale for the review Anthropometric This rapid evidence assessment is expected to inform measures Diet quality and decisions regarding gender and women’s empowerment adequacy in nutrition and food systems interventions. Given that Measures of well- women’s empowerment has been highlighted as a criti- being cal, crosscutting theme for the transformation of the food Study designs Experimental, quasi- Efficacy trials, before- experimental, system- after with no control system [4], key decision-makers have indicated interest in atic reviews and cost group, cross-sectional this area. Researchers can use this work to better under- evidence studies and so on stand how to intertwine gender-sensitive or -transforma- Types of study participants tive interventions for improved nutritional outcomes. Only studies which consider populations in low- and Research questions middle-income countries (as defined using the World Bank Country and Lending Groups classification in first 1. What are the effects of women’s empowerment inter - year of intervention or if not available then Publication ventions within the food system on the availabil- year) will be considered. The exception to this is if a ity, accessibility, and affordability of healthy diets or country held high-income status for only one year before nutritional status? reverting to L&MIC status. These will be included even if 2. Are there any unintended consequences of such the intervention began in the high-income year. As of the interventions? writing of this protocol, this applies to Argentina (2014, 3. Do effects vary by context, approach to empower - 2017), Venezuela (2014), Mauritius (2019), and Roma- ment, or other moderators? nia (2019). If the study is conducted in a high-income Berretta et al. Agriculture & Food Security (2023) 12:13 Page 20 of 52 country but measures impact on people, firms, or institu - Types of outcome measures tions in an L&MIC, it can be included. For example, we The table below outlines outcome indicators that will would not exclude a study that measures impact of New be extracted. These outcomes can be measured using Zealand’s immigration visa lottery on residents of Tonga. a variety of indicators. We have indicated the preferred outcomes and alternate outcomes which could be used if Types of interventions preferred outcomes are not reported. Composite meas- Eligible interventions were identified during the devel - ures will always be preferred over disaggregated ones. opment of the Food Systems and Nutrition Evidence Outcome Indicators Gap Map [1]. The map defined women’s empowerment interventions as “efforts targeted at increasing women’s Food security Preferred outcomes: food security abilities to make decisions regarding the purchase and indexes and composite scores Secondary outcome: skipped meals consumption of healthy foods.” After completing the Tertiary outcome: reports of insuf- search, we found that these interventions were primar- ficient food ily related to agriculture skills training, asset transfers, Food affordability Preferred outcome: per capita food microcredit, and behavior change. consumption in monetary units Secondary outcome: per capita food consumption in weight Citation Intervention Other measures, such as cost of a food basket, will be considered if Ahmed et al. [24] The intervention consists of two treatment arms: these are not available cash or food transfers, with or without nutri- Food availability/accessibility Preferred outcomes: food assets, tion behavior change communication (BCC), to production (community gardens,) women living in poverty in rural Bangladesh and stores Bandiera et al. [25] The intervention is a nationwide asset transfer Other measures, such as distance “plus” program in Bangladesh. The intervention and accessibility to markets transfers livestock assets and skills to the poorest women Diet quality and adequacy Preferred outcomes: composite diet scores such as the nutrient rich food Bonuedi et al. [27] The intervention is two-pronged: (1) cash crop index and (2) nutrition components. (1) Included farmer Secondary outcome: dietary diver- field schools (FFS), productive inputs, and value sity and other food variety measures chain linkages. (2) Included gender-sensitive nutri- Tertiary outcome: intake of specific tion behavior change and awareness creation foods Choudhury et al. [45] Suchana improves nutrition service delivery, nutri- Anthropometrics Preferred outcomes: body mass tion governance, and the knowledge of women index, weight for length, length for and girls regarding gender norms and gender- age, weight for age based violence that can impact mother and child Other measures, such as MUAC and nutrition ponderal index, will be considered if Deininger et al. [28] The intervention is self-help groups for women these are not available living in poverty in India Iron, zinc, vitamin A, and iodine Preferred outcome: measures of Emran et al. [29] This is an asset transfer “plus” intervention, status content in blood/tissue (ex. hemo- bundling asset transfers with capacity building globin levels) (health, education, and training) for poor women Secondary outcome: intake in with the goal of helping them graduate to the weight (grams, micrograms, etc.) standard micro-credit program of BRAC Tertiary outcome: intake in percent- Heckert et al. [31] The intervention is the Enhanced Homestead age relative to recommended intake Food Production (E-HFP) program, a nutrition- and Other measures will be considered gender-sensitive agriculture training program Well-being Preferred outcome: perceived well- Marquis et al. [32] This is a microcredit “plus” intervention that being provides microcredit loans and weekly sessions of Secondary outcome: anxiety nutrition and entrepreneurship education for 179 women with children 2–5 years of age Mosha et al. [26] The agricultural training and provision of inputs Types of comparators intervention includes the provision of small agricultural inputs to women, garden training support, and nutrition and health counselling to • Business as usual, including pipeline and waitlist con- improve food security trols Pan et al. [33] A large-scale agricultural extension program for • An alternate intervention smallholder women farmers to improve food security in Uganda • Studies with no comparator are excluded B erretta et al. Agriculture & Food Security (2023) 12:13 Page 21 of 52 Types of study design targeted searches to identify qualitative studies and pro- Experimental, quasi-experimental, systematic review, cess evaluations of the included interventions. and cost evidence will be considered. The following study designs will be included. Selection of studies Screening Because we are utilizing the results of the Food systems EGM, there is no search and screening pro- • Randomized controlled trial cess to select the studies. Rather, within the FSN EGM, • Regression discontinuity design we selected ten studies that have women’s empowerment • Controlled before-and-after studies, including interventions associated with the relevant outcomes. – Propensity-weighted multiple regression Data extraction and coding procedures Data extraction – Instrumental variable templates will be modified from 3ie’s repository cod - – Fixed effects models ing protocol and the coding protocols typically used for – Difference-in-differences (and any mathematical systematic reviews (Appendix 2). This includes biblio - equivalents) graphic, geographic information and substantive data, as – Matching techniques well as standardized methods information. In addition, two members of the team will extract data independently on interventions, outcomes, population (including gen- • Interrupted time series der/age disaggregation, when available), and effect sizes • Systematic reviews that include a quantitative or nar- corresponding to the outcomes indicated above, and any rative synthesis discrepancies will be reconciled. On interventions, out- comes, population (including gender/age disaggregation, Ex-post cost-effectiveness analyses will be included, when available), and effect sizes corresponding to the provided that they are associated with an included outcomes indicated above, and any discrepancies will be impact evaluation. reconciled. Qualitative information on barriers and facili- tators to implementation, sustainability and equity impli- Date, language, and form of publication cations, and other considerations for practitioners will All proceeding restrictions are from the EGM. also be extracted. • Date: 2000 Critical appraisal All the included quantitative impact • Language: English evaluations will be appraised by two independent mem- bers of the team using a critical appraisal tool (Appen- dix 1.1 and 1.2). Qualitative studies linked to included Search strategy impact evaluations will also be critically appraised. We will not perform any new searches for this REA. Instead, we will look at the ten studies of women’s Qualitative search and appraisal In addition to qualita- empowerment interventions identified in the Food Sys - 1 tive evidence from the included studies to assess factors tems and Nutrition ’living’ EGM, updated every four that determine or hinder the effectiveness of interven - months (last update December 2021). We specifically tions using a combination of qualitative synthesis, we will searched for interventions using women’s empowerment conduct a basic search on the programs in each of the ten within the food system implemented in low- and middle- papers, looking for the following relevant papers [11]: income countries. This EGM was developed through a systematic search and screening process equal to that of • A qualitative study collecting primary data using a systematic review. However, because interventions had mixed- methods or quantitative methods of data col- to function within the food system to be included, many lection and analysis and reporting some information women’s empowerment interventions, such as those on all of the following: the research question, proce- related to self-help groups broadly, were not included. dures for collecting data, procedures for analyzing Ultimately, the EGM includes ten evaluations of women’s data, and information on sampling and recruitment, empowerment interventions which considered outcomes including at least two sample characteristics. related to food availability, accessibility, and affordabil - • A descriptive quantitative study collecting primary ity and nutritional status. We will conduct additional data using quantitative methods of data collection and descriptive quantitative analysis and report some https:// gapma ps. 3ieim pact. org/ evide nce- maps/ food- syste ms- and- nutri tion- information on all of the following: the research evide nce- gap- map. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 22 of 52 question, procedures for collecting data, procedures available literature and a general synthesis of findings. for analyzing data, and information on sampling and Key information from each study, such as intervention recruitment, including at least two sample character- type, study design, country, outcomes, measurement istics. type, effect sizes, and confidence rating will be summa - • A process evaluation assessing whether an interven- rized in a table. Results from meta-analyses and their tion is being implemented as intended and what is associated forest plots will be presented when the data felt to be working well, and why. Process evaluations is sufficient. Qualitative information will be summa - may include the collection of qualitative and quanti- rized narratively in a practitioner’s brief to support pro- tative data from different stakeholders to cover sub - ject design and implementation. An updated theory of jective issues, such as perceptions of intervention change will be developed based on the combination of success or more objective issues, such as how an qualitative and quantitative data. intervention was operationalized. They might also be used to collect organizational information. Limitations Due to the rapid nature of this work, results should be interpreted more cautiously than those of a systematic While the identification of qualitative evidence is lim - review. Relying on the existing Food Systems and Nutri- ited to studies linked to the included impact evaluations, tion EGM may result in some relevant studies being the process of data extraction, critical appraisal, and evi- omitted from this evidence assessment. The small num - dence synthesis is independent. ber of studies which are expected to be retrieved through We will assess the quality of included qualitative stud- this REA may restrict the possibility of using meta-analy- ies, process evaluations, and descriptive quantitative sis and our ability to draw generalizable conclusions. studies using a mixed methods appraisal tool developed by CASP [12] and applied in Snilstveit et al. [46]. This Appendix 2: Data extraction tool tool is in Appendix 1.3. The meta-analysis conducted with the quantitative data will thus be complemented by a thematic synthesis utilizing the extracted qualitative Variable group Variable Label data. Publication info Record type Analytical approach for quantitative data If sufficient Record Title data is available, we will conduct meta-analysis to provide Record authors summary effect estimates. We will choose the appropri - Publication year ate formulae for effect size calculations in reference to, URL link and dependent upon, the data provided in included stud- Intervention and implementation Intervention ies. We will conduct random effects meta-analyses when considerations Intervention details we identify two or more studies that we assess to be suf- Unintended consequences ficiently similar. We will assess heterogeneity by calculat - Barriers and facilitators to imple- 2 2 ing the Q statistic, I , and τ to provide an estimate of the mentation amount of variability in the distribution of the true effect Evaluation considerations Study design sizes [23]. We will explore heterogeneity through the use Covariates of moderator analyses if the data allow. We will also test Outcomes for the presence of publication bias if at least 10 studies Sustainability and financial con- Sustainability comments siderations are included in the analysis. Cost effectiveness comments Other Other Data presentation Confidence rating (srr only) We will provide a narrative summary of the papers iden- tified. This will include an overall description of the B erretta et al. Agriculture & Food Security (2023) 12:13 Page 23 of 52 Quantitative data extraction tool Variable level Explanation Intervention codes Variable level Explanation Intervention description Use this open answer field to enter, in the author’s own words, a Study ID (DEP) This is the study ID from DEP (e.g., description of the intervention, up 17347) to a paragraph or so; more detail information will be preferred. Be Study ID (EPPI) This is the study ID from EPPI selective and concise with the reviewer. It should match the study excerpts being transcribed here ID from the Outcome Mapping as to ensure accurate and precise Sheet (e.g., 41504196) descriptions of the intervention. Estimate ID The estimate ID will provide a Include page numbers with every specific number for each effect size excerpt extracted. Do this for each extracted and should include the Treatment arm original study number, underscore, Intervention Record the intervention for the cor- then the unique ID number (e.g., responding effect size SC-SR1_1, SC-SR1_2 and so on) Exposure to intervention (in How long is the intervention expo- Evaluation design 0 = Experimental Design (e.g., RCT ), months) sure itself? 1 = Quasi-Experimental Design Evaluation period (in months) The total number of months How counterfactual is chosen Free text (e.g., random control trial, elapsed between the end of an propensity score matching, etc.)— intervention and the point at which Multiple codes are ok an outcome measure is taken post Analysis type for this effect size Free text, what type of analysis was intervention, or as a follow-up meas- used (Regression, 2SLS, ANCOVA, urement. If less than one month, etc.)- Multiple codes are ok use decimals (e.g., measurement Estimate type Type of data for this effect size: immediately after the intervention 1 = Continuous—means and SDs, end would be coded as 0, one week 2 = Continuous—mean differ - would be 0.25, etc.) ence and SD, 3 = Dichotomous Post-intervention or change from 0 = Post-intervention, 1 = Change outcome—proportions, 4 = Regres- baseline? from baseline sion data Outcome Codes Comparison 1 = No intervention (service delivery Outcome description Use this open answer field to as usual), 2 = Other intervention, enter, in the author’s own words, 3 = Pipeline (waitlist) control (still a description of the outcome. Be service delivery as usual) selective and concise with the Describe comparison group Free text, describe the comparison excerpts being transcribed here group as to ensure accurate and precise Country Select the countries in which the descriptions of the outcome. study was conducted (drop down Include page numbers with every menu). There is a multi-country excerpt extracted. Do this for each option for situations when there outcome are more than 15 countries, and no Outcome Record the outcome for the cor- disaggregated effects provided for responding effect size each country Eec ff t Size Data Extraction Subgroup Is this analysis of a subgroup? Reverse Sign (i.e., decrease is Record no if an increase is good, 0 = no, 1 = yes good) record yes if a decrease is good and If yes to subgroup, describe Free text, describe the subgroup the sign needs to be reversed if applicable (e.g., boys, girls). If no Unit of analysis What is the unit of analysis? UOA subgroup, type N/A for this effect size: 1 = Individual, Source Note the page number, table num- 2 = Household, 3 = Group (e.g., ber, column, and row you used to community organization), 4 = Vil- extract the data lage, 5 = Other, 6 = Not clear Treatment effect 1 = Intention to Treat (ITT ), 2 = Aver- Mean_t Outcome mean for the treatment age Treatment Eec ff t on the Treated group (ATET ), 3 = Average Treatment Eec ff t Sd_t Outcome standard deviation for (ATE) 4 = Local Average Treatment treatment group Eec ff t (LATE) Mean_c Outcome mean for the comparison group Berretta et al. Agriculture & Food Security (2023) 12:13 Page 24 of 52 Variable level Explanation Variable level Explanation Sd_c Outcome standard deviation for g THIS IS FOR SENIOR QUANT LEAD TO control group FILL OUT Mean_overall_diff Overall mean difference (treat - Var(d) THIS IS FOR SENIOR QUANT LEAD TO ment—control) FILL OUT Diff se Standard error of the overall mean se(d) THIS IS FOR SENIOR QUANT LEAD TO difference FILL OUT Diff _t t statistic of mean difference CI_l THIS IS FOR SENIOR QUANT LEAD TO FILL OUT Odds ratio Odds ratio reported in the study CI_u THIS IS FOR SENIOR QUANT LEAD TO OR_se Odds ratio standard error reported FILL OUT in the study Remove THIS IS FOR PROJECT MANAGER TO Risk ratio Risk ratio reported in study FILL OUT RR_se Risk ratio standard error Formula Used THIS IS FOR SENIOR QUANT LEAD TO Reg_coeff Report the regression coefficient of FILL OUT the treatment effect g_1 THIS IS FOR SENIOR QUANT LEAD TO Reg_SE Report the associated standard error FILL OUT of the regression coefficient g_rev THIS IS FOR SENIOR QUANT LEAD TO Reg_t Report the associated t statistic of FILL OUT the effect size (coefficient/SE) g THIS IS FOR SENIOR QUANT LEAD TO Exact p value Exact p value if given, if not, record FILL OUT as written in the manuscript (e.g., vi THIS IS FOR SENIOR QUANT LEAD TO p < 0.001, or p> 0.05) FILL OUT Clust_t Number of clusters—treatment wi THIS IS FOR SENIOR QUANT LEAD TO group FILL OUT Clust_c Number of clusters—control group ywi THIS IS FOR SENIOR QUANT LEAD TO Clust_T Number of clusters—total sample FILL OUT n_t Sample size—treatment group 95ci_lower THIS IS FOR SENIOR QUANT LEAD TO FILL OUT n_c Sample size—control group 95ci_upper THIS IS FOR SENIOR QUANT LEAD TO n_T Sample size—total sample FILL OUT Periods (1 if cross-sectional) Record how many periods of evalu- ation there are (e.g., cross section is cilow_3sf THIS IS FOR SENIOR QUANT LEAD TO 1, panel data with 3 measurements FILL OUT is 3) cihigh_3sf THIS IS FOR SENIOR QUANT LEAD TO Does the sample size need to be Often in panel data, models will FILL OUT corrected? report number of observations ci THIS IS FOR SENIOR QUANT LEAD TO rather than number of participants. FILL OUT In this column you will indicate wb_g THIS IS FOR SENIOR QUANT LEAD TO "Yes" if the sample size needs to be FILL OUT divided by the number of periods, and "No" if either it is cross-sectional Checked THIS IS FOR EFFECT SIZE RELIABILITY data, or if the authors have already CHECKER TO FILL OUT divided the number of observations ROB Category THIS IS FOR SENIOR QUANT LEAD by the number of panel assess- OR PM TO FILL OUT ments and thus no correction is necessary Treatment variable Record the treatment variable as Appendix 3: Critical appraisal tools written in the model (e.g., the vari- Appraisal of risk of bias for impact evaluations using RCT able name the author uses, such as ("Intervention x Time") designs Dataset Record if data comes from an identi- The following table provides a provisional tool to guide fied dataset the risk of bias assessment for quantitative impact Coder Record your name evaluations. Notes Record any notes important for the team n_T_revised THIS IS FOR SENIOR QUANT LEAD TO FILL OUT sp THIS IS FOR SENIOR QUANT LEAD TO FILL OUT d THIS IS FOR SENIOR QUANT LEAD TO FILL OUT B erretta et al. Agriculture & Food Security (2023) 12:13 Page 25 of 52 Provisional risk of bias assessment tool (RCT) General ID EPPI ID General Study first author Open answer General Time taken to com- Minutes plete assessment General Design type: What 1 = Randomized controlled trial – type of study design (RCT ) (random assignment to is used? households/individuals) or quasi- RCT 2 = Cluster-RC T (quasiRC T ) General Methods used for 1 = Statistical matching (PSM, CEM, – analysis: Which covariate matching) 2 = Difference- methods are used in-differences (DID) estimation to control for methods 3 = IV-regression (2stage selection bias and least squares or bivariate probit) confounding? 4 = Heckman selection model 5 = Fixed effects regression 6 = Covariate adjusted estimation 7 = Propensity-weighted regression 8 = Comparison of means= Other (please state) General Design and analysis Open answer Briefly describe the study design and method description analysis method undertaken by the authors General Study population Open answer Provide any details in the paper that describe how the study population was selected, covering: a) How is the population selected? what is the sampling strategy to recruit participants from that popula- tion into the study? b) What are the characteristics of that study participants? Was this a pilot program aimed at being scaled up? d) Were there spe- cific factors of success or failure in the implementation? General Type of comparison 1 = No intervention Indicate type of comparison group group (Service delivery as usual) 2 = Other intervention 3 = Pipeline (waitlist) control (still service deliv- ery as usual) General Type of comparison Open answer group (If other) General Ethical clearance Open answer Provide any details of ethical research clearances granted. Report unclear if this information is not available General Study registration Open answer Provide any details of study registra- tion, including registry IDs, etc. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 26 of 52 General ID EPPI ID 1: Assignment Assignment 1 = Yes, 2 = Probably a) The authors describe a random Score “Yes” if all criterion a), b), c) mechanism— mechanism: Was the Yes, 3 = Probably No, 4 component in sequence generation/ and d) are satisfied Assessment allocation or identi- = No, 8 = Unclear randomization method (e.g Score "Probably Yes" if only fication mechanism lottery, coin toss, criterion a) and b) are not satis- random or as good random number generator) and fied OR if only criteria c) is not as random? assignment is performed for all units satisfied at the start of the study centrally or Score “Unclear” if d) is not satis- using a method concealed from par- fied because no balance table is ticipants and intervention delivery reported b) If public lottery Score "Probably No" if d) is not is used for the sequence generation, satisfied because there is no authors provide detail on the exact balance table reported and settings and participants attending there is evidence suggesting a the lottery problem in the randomization, c) If a special such as baseline coefficients randomization procedure is used to in a diff-in-diff regression table ensure balance, it is well described are very different or sample size and justified given the study setting is too small for the procedure (stratification, pairwise matching, used (using stratification when unique random draw, multiple ran- there are less than two units for dom draws, etc.) each intervention and control d) A balance table is reported sug- group in each strata can lead to gesting that allocation was random imbalance) between all groups including sub- Score “No” if d) is not satis- group receiving different treatment fied because there are large within control or treatment groups imbalances concerning a large (if the comparison is relevant for this number of variables, providing assessment) evidence that the assignment was not random. If this is scored as no, use the NRS tool 1: Assignment Assignment justifi- Open answer Justification for coding decision mechanism— cation (Include a brief summary of justifica- Justification tion for rating, mentioning your response to all sub-questions, cite relevant pages) 2: Unit of analy- Unit of analysis: 1 = Yes 2 = No 3 = Not reported/ Score "Yes" if UoA = UoR OR if sis—Assess- Is unit of analysis unclear 4 = Not applicable UoA ≠ UoR and standard errors are ment in cluster alloca- clustered at the UoR level OR data is tion addressed collapsed to the UoR level in standard error Score "Not reported/unclear" if calculation? not enough information is provided on the way the standard errors were calculated or what the unit of analysis is Score "Not applicable" if it is not a cluster RCT Score "No" otherwise B erretta et al. Agriculture & Food Security (2023) 12:13 Page 27 of 52 General ID EPPI ID 3: Selection Selection bias Was 1 = Yes, 2 = Probably Score "Yes" if there is no attrition or bias-Assess- any differential Yes, 3 = Probably No, 4 attrition falls into the green zone and ment selection into or out = No, 8 = Unclear the study establishes that attrition is of the study (attri- randomly distributed (e.g., by present- tion bias) ade- ing balance by key characteristics quately resolved? across groups) AND if survey respond- ents were randomly sampled Score "Probably yes" if attrition falls into the green zone AND if survey respondents were randomly sampled Score "Unclear" if there is an attrition problem but no information provided on the relationship between attrition and treatment status, OR if there is not enough information on how the population surveyed was sampled Score "Probably no" if there is attrition which is likely to be related to the intervention OR is some indication that the survey respondents were purposely sampled in a way that might have led the sampling to be dif- ferent between treatment and control groups, or attrition falls into the yellow zone Score "No" if attrition falls into the red zone 3: Selection Selection bias justi- Open answer Justification for coding decision bias-Justifica- fication (Include a brief summary of justifica- tion tion for rating, mentioning your response to all sub-questions, cite relevant pages) 4: Confound- Confounding and 1 = Yes, 2 = Probably a) Baseline characteristics are similar in Score “Yes” if criterion a) and b) ing- Assess- group equivalence: Yes, 3 = Probably No, 4 magnitude; are satisfied; ment Was the method of = No, 8 = Unclear b) Unbalanced covariates at the indi- Score "Probably yes" if a) is not analysis executed vidual and cluster level are controlled satisfied but b) is satisfied and adequately to in adjusted analysis; c) Adjustments imbalances are small in magni- ensure compa- to the randomization were taken into tude OR if only a) is satisfied rability of groups account in the analysis (stratum fixed Score “Unclear” if no balance throughout the effects, pairwise matching variables)? table is provided or if imbal- study and prevent (Bruhn and McKenzie ances are controlled for but they confounding 2009) are very large in magnitude and assignment mechanism is not coded as "Yes" or "Probably yes" Score "Probably no" if a) and b) are not satisfied and the magni- tude of imbalances are small Score “No” if a) and b) are not satisfied and the magnitude of imbalances are large, and covari- ates are clear determinant of the outcomes 4: Confound- Confounding justi- Open answer Justification for coding decision ing-Justifica- fication (Include a brief summary of justifica- tion tion for rating, mentioning your response to all sub-questions, cite relevant pages) Berretta et al. Agriculture & Food Security (2023) 12:13 Page 28 of 52 General ID EPPI ID 5: Deviations Deviations from 1 = Yes, 2 = Probably Yes, 3 = Prob- a) There were no implementation Score “Yes” if criterion a), b), c) from intended intended interven- ably No, 4= No, 8 = Unclear issues that might have led the control and d) are satisfied; interven- tions: Spillovers, participants to receive the treatment Score "Probably yes" if there is tions—Assess- crossovers, and con- (implementer’s mistake) no obvious problem but there ment tamination: was the b) The intervention is unlikely to spillo- is no information reported on study adequately ver to comparisons (e.g., participants potential risks related to spill protected against and non-participants are geographi- overs, contamination, or survey spillovers, crosso- cally and/or socially separated from effects in the control group OR if vers, and contami- one another and general equilibrium there were issues with spillovers nation? effects are not likely) or the potential but they were controlled for or effects of spill overs were measured measured (e.g., variation in the % of unit within a Score “Unclear” if spillovers, cluster receiving the treatment) crossovers, survey effects and/ There is no risk of contamination by or contamination are not external programs: the treatment and addressed clearly comparisons are isolated from other Score "Probably no" if any of the interventions which might explain criterion a), b), c) or d) are not changes in outcomes satisfied but the scale of the d) There is nothing in the surveys issue is not clear that might have given the control Score “No” if any of the criterion participants an idea of what the other a), b), c) or d) are not satisfied group might receive OR they did but and happened at a large scale in there is no risk that this has changed the study their behaviors; AND the survey process did not reveal information to the control group that they did not have before (e.g., the study aims to measure increase in take up of a service or product that participants might not know about) Authors might put something in place in the design of the study that allows to control for that survey effect (e.g., a pure control with no monitoring except baseline end line) 5: Deviations Deviations justifica- Open answer Justification for coding decision from intended tion (Include a brief interven- summary of justification for rating, tions—Justifi- mentioning your response to all sub- cation questions, cite relevant pages) For example, intervention groups are geographically separated, authors use intention to treat estimation or instrumental variables to account for non-adherence, and survey questions are not likely to expose individuals in the control group to information about desirable behaviors (‘survey effects’) B erretta et al. Agriculture & Food Security (2023) 12:13 Page 29 of 52 General ID EPPI ID 6. Performance Performance bias: 1 = Yes, 2 = Probably Yes, 3 = Prob- a) The authors state explicitly that the Score “Yes” if either criterion a) or bias -Assess- Was the process ably No, 4 process of monitoring the interven- b) are satisfied; ment of monitoring = No, 8 = Unclear tion and outcome measurement is Score "Probably yes" if the study individuals unlikely blinded and conducted in the same is based on data collected dur- to introduce moti- frequency for treatment and control ing a trial and there is no obvi- vation bias among groups, or argue convincingly why it is ous issue with the monitoring participants? not likely that being monitored could processes, but authors do not affect the performance of participants mention potential risks in treatment and comparison groups Score “Unclear” if it is not clear in different ways (such as resulting in whether the authors use Hawthorne or John Henry effects) an appropriate method to b) The outcome is based on data prevent Hawthorne and John collected in the context of a survey, Henry Eec ff ts (e.g., blinding of and not associated with a particular outcomes and, or enumera- intervention trial, or data are collected tors, other methods to ensure from administrative records or in the consistent monitoring across context of a retrospective (ex post) groups) evaluation Hawthorne effects may result where participants know that they are being observed and John Henry Eec ff ts may result from participant knowledge of being compared Score "Probably no" if there was imbalance in the frequency of monitoring in intervention groups, which might have influ- enced participants’ behaviors Score "No" if neither criterion a) or b) are satisfied 6. Performance Performance bias Open answer Justification for coding decision bias-Justifica- justification (Include a brief summary of justifica- tion tion for rating, mentioning your response to all sub-questions, cite relevant pages) 7. Outcome Outcome measure- 1 = Yes, 2 = Probably a) Outcome assessors are blinded, or Score “Yes” if criterion a), b), c) measurement ment bias: Was the Yes, 3 = Probably No, 4 the outcome measures are not likely and d) are satisfied: bias - study free from = No, 8 = Unclear to be biased by their judgment Score "Probably yes" if there Assessment biases in outcome b) For self-reported outcomes: is a small risk related to any of measurement? respondents in the intervention group a), b), c) or d) and there is no are not more likely to have accurate more information provided to answers due to recall bias; justify the absence of bias OR if c) For self-reported outcomes: there was a high risk of bias, but respondents do not have incentives to authors have either controlled it over/under report something related in their design or measured to their performance or actions, OR it with a placebo outcome researchers put in place mechanisms Score “Unclear” if it there is a to reduce the risk of reporting bias high risk related to any of a), b), (researchers not strongly involved in c) or d) and there is no more the implementation of the program information provided to justify and it is clear that their answers to the absence of bias the survey will not affect what they Score "Probably no" if there are receive in future) OR authors high risk related to a), b), c) or d) have measured the risks of bias and it is clear that authors were through not able to control for this bias falsification tests or measuring the Score “No” if there is evidence effect on placebo outcomes in cases of bias where there was a risk of reporting bias d) Timing issue: the data collec- tion period did not differ between intervention and comparison group; the baseline data is not likely to be affected by the beginning of the inter - vention or affects a small percentage of the study participants Berretta et al. Agriculture & Food Security (2023) 12:13 Page 30 of 52 General ID EPPI ID 7. Outcome Outcome measure- Open answer Justification for coding decision measurement ment justification (Include a brief summary of justifica- bias-Justifica- tion for rating, mentioning your tion response to all sub-questions, cite relevant pages) 8. Reporting Analysis reporting: 1 = Yes, 2 = Probably a) A pre-analysis plan or trial protocol Score "Yes" if all the criterion bias-Assess- Was the study free Yes, 3 = Probably No, 4 is published and referred to or the trial a), b), c), d), and e) are satisfied; ment from selective analy- = No, 8 = Unclear was preregistered, or the outcomes Score "Probably yes" if all the sis reporting? were preregistered; conditions are met except a), or b) Authors report results correspond- if all the conditions are met but ing to the outcomes announced there is some element missing in the method section (there is no that could have helped under- outcome reporting bias); stand the results c) Authors report results of unadjusted better (e); analysis and intention to treat (ITT ) Score "Unclear" if there is not estimation, alongside any adjusted enough information to deter- and treatment-on-the treated/com- mine that there is an analysis plier average-causal effects analysis.) missing; Score "Probably no" if d) Authors use the appropriate analy- any of the criterion b), c) or d) sis method (use baseline data when are not satisfied; Score "No" if available), and different treatment any of the criterion b), c) or d) arms are are not satisfied and there is differentiated in the analysis evidence that the analysis results e) Authors have reported all the analy- would be different because sis which could help understand the large imbalances were not con- results and no other bias is assessed as trolled for, compliance was very unclear due to the low and ITT estimation was not lack of an important analysis (e.g., a reported or different treatment balance table or a subgroup analysis) arms were pooled 8. Reporting Analysis reporting Open answer Justification for coding decision bias-Justifica- justification (Include a brief summary of justifica- tion tion for rating, mentioning your response to all sub-questions, cite relevant pages) 9. Other bias- Other risks of bias 1 = Yes, 4 = No Assessment Is the study free from other sources of bias? 9. Other bias- Other bias justifica- Open answer Justification for coding decision Justification tion 10. Blinding- Blinding of partici- 1 = Yes 2 = No 8 = unclear If there is no information, code NO. If observers- pants? 9 = N/A there is information but it is ambigu- Assessment ous, code UNCLEAR 10. Blinding— Blinding of outcome 1 = Yes 2 = No 8 = unclear If there is no information, code NO. If observers— assessors? 9 = N/A there is information but it is ambigu- Assessment ous, code UNCLEAR 10. Blinding- Blinding of data 1 = Yes 2 = No 8 = unclear If there is no information, code NO. If analysts- analysts? 9 = N/A there is information but it is ambigu- Assessment ous, code UNCLEAR 10. Blinding- Method(s) used to Open answer (including describe Describe method(s) used to blind method(s) blind method of placebo control) No 9 = N/A 11. External External validity Open answer a) What do authors say about external Include all information that can validity-Assess- validity? help assess the external validity ment of the results B erretta et al. Agriculture & Food Security (2023) 12:13 Page 31 of 52 Summary of justification for rating, mentioning your response to all sub-questions, cite relevant pages). Appraisal of risk of bias for impact evaluations using quasi‑experimental designs Risk of bias assessment tool (QED) Code Question Coding Criteria Decision‑rules General ID EPPI ID General Time taken to complete Minutes assessment General Study first author Open answer General Outcomes assessed Open answer General Study design: What type of 1 = Natural experi- study design is used? ment: randomized or as-if randomized 2 = Natural experi- ment: regression discontinuity (RD) 3 = CBA (non-rand- omized assignment with treat- ment and contempo- raneous comparison group, baseline, and end line data col- lection) – individual repeated measure- ment 4 = CBA pseudo panel (repeated measurement for groups but different individuals) 5 = Interrupted time series (with or without contemporaneous control group) 6 = Panel data, but no baseline (pre-test) 7 = Comparison group with end line data only General Methods used for analysis: 1 = Statistical – Which methods are used matching (PSM, CEM, to control for selection bias covariate matching) and confounding? 2 = Difference-in- differences (DID) estimation methods 3 = IV-regression (2-stage least squares or bivariate probit) 4 = Heckman selection model 5 = Fixed effects regression6 = Covari- ate adjusted estima- tion 7 = Propensity- weighted regression 8= Comparison of means = Other (please state) Berretta et al. Agriculture & Food Security (2023) 12:13 Page 32 of 52 Code Question Coding Criteria Decision‑rules General Study population Open answer Provide any details in the paper that describe how the study population was selected, covering: a) How is the population selected? what is the sampling strategy to recruit participants from that population into the study? b) What are the characteristics of that study participants? c) Was this a pilot program aimed at being scaled up? d) Were there specific factors of success or failure in the implementation? General Ethical clearance Open answer Provide any details of ethical research clear- ances granted. Report unclear if this informa- tion is not available 1: Selec- 1—Mechanism of assign- 1 = Yes, 2 = Probably tion bias- ment: was the allocation or Yes, Assess- identification mechanism 3 = Probably No, ment able to 4 = No, control for selection bias? 8 = Unclear 1: Selec- For regression discontinuity Open answer a) Allocation is made based on a predeter- Score “Yes” if criteria a), b), c) are all tion designs mined discontinuity on a continuous variable satisfied bias-Jus- (Regression discontinuity design) and blinded Score "Probably Yes" if there are tification to participants or; minor differences in between b) if not blinded, individuals reasonably cannot both sides of the cut-off point but affect the assignment variable in response to authors convincingly argue that knowledge of the participation decision rule; the differences are unlikely to affect c) and the sample size immediately at both the outcome, OR individuals are sides of the cutoff point is sufficiently large to not blinded and there are low risk equate groups on average of them affecting the assignment, but the authors do not mention it Score “Unclear” if it is unclear whether participants can affect it in response to knowledge of the allocation mechanism Score "Probably No" if there are differences between individuals on both sides of the cut-off point, and there are doubts that the differences are due to individuals altering the assignment OR the participants are blinded but there is evidence that the decisions that determined the discontinuity is based on differences between the two groups or differences in time Score “No” if the sample size is not sufficient OR there is evidence that participants altered the assignment variable prior to assignment. If the research has serious concerns with the validity of the assignment process or the group equivalence completely fails, we recommend assessing risk of bias of the study using the relevant questions for the appropriate methods of analy- sis (cross-sectional regressions, difference-in-difference, etc.) rather than the RDDs questions B erretta et al. Agriculture & Food Security (2023) 12:13 Page 33 of 52 Code Question Coding Criteria Decision‑rules 1: Selec- For assignment-based Open answer a) Participants and non-participants are either Score “Yes” if a) or b) and c) are tion nonrandomised program matched based on all relevant characteristics satisfied bias-Jus- placement and self-selec- explaining participation and outcomes, or; Score "Probably yes" if a) or b) are tification tion (studies using a match- b) all relevant characteristics are accounted addressed for but there is some ing strategy or regression for.** and the data set used contains relevant doubt related to c), OR authors analysis, excluding variables that are measured in a relevant way combined statistical matching and IV ) (i.e., they were not collected for a different difference-in-difference to cope purpose initially and therefore are good proxy with unobservable differences, OR for some characteristics) they only did statistical match- **Accounting for and matching on all relevant ing and there were clear rules for characteristics is usually only feasible when the selection into the program (no program allocation rule is known and there are self-selection) no errors of targeting. It is unlikely that studies Score “Unclear ” if · it is not clear not based on randomization or regression whether all relevant characteristics discontinuity can score “YES” on this criterion. (only relevant time-varying char- There are different ways in which covariates acteristics in the case of panel data can be taken into account. Differences across regressions) are controlled groups in observable characteristics can be Score "Probably no" if only a statisti- considered as covariates in the framework of cal matching was done and there a regression analysis or can be assessed by was self-selection into the program testing equality of means between groups. Score “No” if relevant characteristics Differences in unobservable characteristics are omitted from the analysis can be taken into account using instrumental variables (see also question 1.d) or proxy vari- ables in the framework of a regression analysis, or using a fixed effects or difference-in-differ - ences model if the only characteristics which are unobserved are time-invariant 1: Selec- For identification based on Open answer Score “Yes” if an appropriate instrumental vari- tion an instrumental variable (IV able is used which is exogenously generated: bias-Jus- estimation) for example, due to a ‘natural’ experiment or tification random allocation Score "Probably yes" if there is less evidence (no balance table showing differences between the intervention and comparison group) Score “Unclear” if the exogeneity of the instru- ment is unclear (both externally as well as why the variable should not enter by itself in the outcome equation) Score "Probably no" if there is evidence that enrolment in the program is correlated with a variable that might also influence outcome and on the instrumental variable Score “No” if it is clear that the instrument is not exogenous and affect the outcome through other channels than the program 2: Con- 2—Group equivalence: was 1 = Yes, 2 = Probably found- the method of analysis exe- Yes, ing- cuted adequately to ensure 3 = Probably No, Assess- comparability of groups 4 = No, 8 = Unclear ment throughout the study and prevent confounding? Berretta et al. Agriculture & Food Security (2023) 12:13 Page 34 of 52 Code Question Coding Criteria Decision‑rules 2: Con- For regression discontinuity Open answer a) The interval for selection of treatment and Score "Yes if criterion a), b), c) and found- design control group is reasonably small OR authors d) are addressed ing-Justi- have weighted the matches on their distance Score "Probably yes" if b) is not fication to the cutoff point; and addressed but c) is b) the mean of the covariates of the individu- addressed and differences in als immediately at both sides of the cut-off means are not large point (selected sample of participants and Score “Unclear” if insufficient details non-participants) are overall not statistically are provided on controls; or if different based on t-test or insufficient details are provided on ANOVA for equality of means; cluster controls c) Significant differences in covariates of the Score "Probably no" if b) is not individuals have been controlled in multi- addressed (absence of a difference variate analysis; and for cluster assignment, test or balance table) and there are authors control for external cluster-level doubt regarding the continuity on factors that might confound the impact of the both sides of the cut-off point (a) program Score “No” otherwise 2: Con- For non-randomized trials Open answer a) The authors use a difference-in-differences Score "Yes, if a, b, c, d (if relevant) is found- using difference-in-differ - (or fixed effects) multivariate estimation addressed and baseline imbalances ing- Justi- ences methods of analysis method; between groups were relatively fication b) the authors control for a comprehensive low OR the method was combined set of individual time-varying characteristics, by a statistical matching and for cluster assignment, authors control Score "Probably yes" if all possible for external cluster-level factors that might variables are controlled for and confound the impact of the program**; the selection into the program c) and the attrition rate is sufficiently low and was done according to clear rules, similar in treatment and control, or the study but baseline imbalances between assesses that dropouts are random draws from groups were very large the sample (for example, by examining correla- Score “Unclear” if insufficient details tion with determinants of outcomes, in both are provided; or if insufficient treatment and comparison groups); details are provided on cluster **Knowing controls allocation rules for the program – or even Score "Probably no" if some time- whether the non-participants were individuals varying characteristics are not that refused to participate in the program, as controlled for and the program was opposed to individuals that were not given self-selected by the intervention the opportunity to participate in the program groups – can help in the assessment of whether the Score “No” if any of the criterion is covariates accounted for in the regression not addressed capture all the relevant characteristics that explain differences between treatment and comparison groups B erretta et al. Agriculture & Food Security (2023) 12:13 Page 35 of 52 Code Question Coding Criteria Decision‑rules 2: Con- For statistical matching Open answer a) Matching is either on baseline characteristics Score "Yes, if a, b, c, and d (if rel- found- studies including pro- or time-invariant characteristics which cannot evant) are addressed ing-Justi- pensity scores (PSM) and be affected by participation in the program; Score "Probably yes" if the selection fication covariate matching** and the variables used to match are relevant into the program was done accord- **Matching strategies are (for example, demographic and socio-eco- ing to clear rules, which are used sometimes complemented nomic factors) to explain both participation for the matching but there are with difference-indifference and the outcome (so that there can be no slight imbalances remaining after only uses in the estima- evident differences across groups in variables matching tion the common support that might explain outcomes); and, for cluster Score “Unclear” if relevant variables region of the sample size, assignment, authors control for external are not included in the matching reducing the likelihood of cluster-level factors that might confound the equation, or if matching is based existence of time variant impact of the program on characteristics collected at end unobservable differences b) in addition, for PSM Rosenbaum’s test line; or if insufficient details are across groups affecting suggests the results are not sensitive to the provided on cluster controls outcome of interest and existence of hidden bias; and, Score "Probably no" if the program removing biases aris- c) with the exception of Kernel matching, the was self-selected by the interven- ing from time-invariant means of the individual covariates are equated tion groups or participants OR if unobservable characteris- for treatment and comparison groups after the selection into the program was tics, regression estimation matching; done according to clear rules but methods. This combination d) different matching methods including vary- there is no baseline data available approach is superior since it ing sample sizes gelds the same results and to match the participants or groups authors consider the use of control observa- on tions multiple times against the same treat- Score “No” if matching was done ment in their standard error calculation based on variables that are likely to be affected by the program or any other scenario that affect a), b) c) or d) 2: Con- For regression-based stud- Open answer a) The study controls for relevant confounders Score "Yes if a, b, c and d are found- ies using cross-sectional that may be correlated with both participa- addressed ing-Justi- data (excluding IV ) tion and explain outcomes (for example, Score "Probably yes" if all criteria fication demographic and socio-economic factors at are addressed but authors did not individual and community report the Hausman test level) using multivariate methods with appro- (b) priate proxies for unobservable covariates, and, Score “Unclear” if relevant for cluster assignment, authors control particu- confounders are controlled but larly for external cluster-level factors that might appropriate proxy variables or confound the impact of the program; statistical tests are not reported; or b) and a Hausman test with an appropriate if insufficient details are provided instrument suggests there is no evidence of on cluster controls endogeneity**; Score "Probably no" if any of the c) and none of the covariate controls can be criterion other than b) is not affected by participation; addressed d) and either, only those observations in the Score “No" if none of the criterion region of common support for participants are addressed and non-participants in terms of covariates are used, or the distributions of covariates are balanced for the entire sample population across groups; **The Hausman test explores endogeneity in the framework of regression by comparing whether the OLS and the IV approaches geld significantly different estimations. However, it plays a different role in the different meth- ods of analysis. While in the OLS regression framework the Hausman test mainly explores endogeneity and therefore is related with the validity of the method, in IV approaches it explores whether the author has chosen the best available strategy for addressing causal attribution (since in the absence of endogene- ity OLS gelds more precise estimators) and therefore is more related with analysis report- ing bias Berretta et al. Agriculture & Food Security (2023) 12:13 Page 36 of 52 Code Question Coding Criteria Decision‑rules 2: Con- For identification based on Open answer a) The instrumenting equation is significant Score "Yes, if a, b, c, d (if relevant) is found- an instrumental variable (IV at the level of F ≥ 10 (or if an F test is not addressed ing-Justi- estimation) reported, the authors report and assess Score "Probably yes" if one of the fication whether the R-squared (goodness of fit) of the tests required for criterion a) or participation equation is sufficient for appro - b) is not reported but the other priate identification); b) the identified instru- is, and the rest of the criterion are ments are individually significant (p ≤ 0.01); for addressed, and the instrument is Heckman models, the identifiers are reported convincing and significant (p ≤ 0.05); Score “UNCLEAR” if relevant con- c) where at least two instruments are used, founders are the authors report on an over-identifying test controlled for but appropriate (p ≤ 0.05 is required to reject the null hypoth- statistical tests are not reported; or esis); and none of the covariate controls can if insufficient details are provided be affected by participation and the study, and on cluster controls authors convincingly assesses qualitatively why Score "Probably no" if exogeneity the instrument only affects the outcome via of the instrument is not convinc- participation. If the instrument is the random ing and appropriate tests are not assignment of the treatment, the reviewer reported should also assess the quality and success of Score “No” otherwise if any of the the randomization procedure in part a) tests required for criterion a), b) or d) and, for cluster assignment, authors c) are reported and not satisfied particularly control for external cluster-level factors that might confound the impact of the program (for example, weather, infrastructure, community fixed effects, and so forth) through multivariable analysis 3: Perfor- 3—Performance bias: 1 = Yes, 2 = Probably a) For data collected in the context of a Score “Yes” if either criterion a) or b) mance was the process of being Yes, particular are satisfied; bias- observed free from motiva- 3 = Probably No, intervention trial (randomized or nonran- Score "Probably yes" if the study Assess- tion bias? 4 = No, domised assignment), the authors state is based on survey data collected ment 8 = Unclear explicitly that the process of monitoring the during a trial and there is no intervention and outcome measurement is obvious issue with the monitoring blinded, or argue convincingly why it processes, but authors do not men- is not likely that being monitored could affect tion potential risks the performance of participants in treatment Score “Unclear” if it is not clear and comparison groups in different ways whether the authors use an (such as resulting in Hawthorne or John Henry appropriate method to prevent effects) Hawthorne and John Henry Eec ff ts b) The study is based on data collected in the (e.g., blinding of outcomes and, context of a survey, and not associated with a or enumerators, other methods particular to ensure consistent monitoring intervention trial, or data are collected from across groups) administrative records or in the context of a Hawthorne effects may result retrospective (ex post) evaluation where participants know that they are being observed and John Henry Eec ff ts may result from par - ticipant knowledge of being com- pareScore "Probably no" if there was imbalance in the frequency of monitoring in intervention groups, which might have influenced participants’ behaviors Score "No" if both criterion a) and b) are not satisfied 3: Perfor- Performance bias-Justifi- Open answer Justification for coding decision (Include a brief mance cation summary of justification for rating, mention- bias-Jus- ing your response to all sub-questions, cite tification relevant pages) B erretta et al. Agriculture & Food Security (2023) 12:13 Page 37 of 52 Code Question Coding Criteria Decision‑rules 4: Spillo- 4—Spillovers, crossovers, 1 = Yes, 2 = Probably a) There were no implementation issues that Score “Yes” if criterion a), b), c) and vers, and contamination: was the Yes, might have led the control participants to d) are satisfied; crosso- study adequately protected 3 = Probably No, receive the treatment (implementer’s mistake) Score "Probably yes" if there is no vers, and against spillovers, crosso- 4 = No, The intervention is unlikely to spillover to obvious problem but there is no contam- vers, and contamination? 8 = Unclear comparisons (e.g., participants and non- information reported on potential ination- participants are geographically and/or socially risks related to spill overs, Assess- separated from one another and general equi- contamination, or survey effects in ment librium effects are not likely) or the potential the control group OR if there were effects of spill overs were measured (e.g., vari- issues with spillovers but they were ation in the % of unit within a cluster receiving controlled for or measured the treatment) Score “Unclear” if spillovers, crosso- c) There is no risk of contamination by external vers, survey effects and/or contami- programs: the treatment and comparisons are nation are not addressed clearly isolated from other interventions which might Score "Probably no" if any of the explain changes in outcomes criterion a), b), c) or d) are not b) There is nothing in the surveys that might satisfied but the scale of the issue have given the control participants an idea of is not clear what the other group might receive OR they Score “No” if any of the criterion did but there is no risk that this has changed a), b), c) or d) are not satisfied and their behaviors; AND the survey process did happened at a large scale in the not reveal information to the control group study that they did not have before (e.g., the study aims to measure increase in take up of a service or product that participants might not know about) Authors might put something in place in the design of the study that allows to control for that survey effect (e.g., a pure control with no monitoring except baseline end line) 4: Spillo- Spillovers, crossovers, and Open answer Justification for coding decision (Include a brief vers, contamination-Justification summary of justification for rating, mention- crosso- ing your response to all sub-questions, cite vers, and relevant pages) contam- ination- Justifica- tion 5: 5—Outcome measurement 1 = Yes, 2 = Probably a) Outcome assessors are blinded, or the Score “Yes” if criterion a), b), c) and Outcome bias Yes, outcome measures are not likely to be biased d) are satisfied: measure- 3 = Probably No, by their judgment Score "Probably yes" if there is ment 4 = No, b) For self-reported outcomes: respondents in a small risk related to any of a), bias- 8 = Unclear the intervention group are not more likely to b), c) or d) and there is no more Assess- have accurate answers due to recall bias; information provided to justify the ment c) For self-reported outcomes: absence of bias OR if there was a respondents do not have incentives to over/ high risk of bias, but authors have under report something related to their perfor- either controlled it in their design mance or actions, OR researchers put in place or measured mechanisms to reduce the risk of reporting it with a placebo outcome bias (researchers not strongly involved in the Score “Unclear” if it there is a high implementation of the program and it is clear risk related to any of a), b), c) or d) that their answers to the survey will not affect and there is no more information what they receive in future) OR authors have provided to justify the absence measured the risks of bias through falsifica- of bias tion tests or measuring the effect on placebo Score "Probably no" if there are outcomes in cases where there was a risk of high risk related to a), b), c) or d) reporting bias and it is clear that authors were not d) Timing issue: the data collection able to control for this bias period did not differ between intervention and Score “No” if there is evidence of comparison group; the baseline data is not bias likely to be affected by the beginning of the intervention or affects a small percentage of the study participants Berretta et al. Agriculture & Food Security (2023) 12:13 Page 38 of 52 Code Question Coding Criteria Decision‑rules 5: Outcome measurement Open answer Justification for coding decision (Include a brief Outcome bias-Justification summary of justification for rating, mention- measure- ing your response to all sub-questions, cite ment relevant pages) bias-Jus- tification 6: Report- 6—Selective analysis 1 = Yes, 2 = Probably a) a pre-analysis plan is published, especially Score “Yes” if a), b), c) and d) are ing bias- reporting: was the study Yes, for prospective NRS, but it should also be for satisfied OR if a) is not met and it is Assess- free from selective analysis 3 = Probably No, retrospective studies b) authors use ‘common’ a retrospective NRS ment reporting? 4 = No, methods of estimation (i.e., credible analysis Score "Probably Yes" if authors 8 = Unclear method to deal with attribution given the data combined methods and reported available); c) There is no evidence that out- relevant tests (d) only for one comes were selectively reported (e.g., results method OR if all the criteria are met for all relevant outcomes in the methods sec- except for a) and it is a prospec- tion are reported in the results section); tive NRS d) Requirements for specific methods of Score "Unclear" if intended out- analysis: comes not specified in the paper - For PSM and covariate matching: (a) Where OR if any of the requirements for d) over 10% are not reported of participants fail to be matched, sensitivity Score "Probably No" if b) is analysis is used to re-estimate results using addressed, but authors did not different matching methods (Kernel Matching present results for all outcomes techniques); (b) For matching with replace- announced in the method section ment, no single observation in the control OR did not meet requirement d) group is matched with a large number of although reported observations in the treatment group.—For IV Score “No” if authors use uncom- (including Heckman) models, (a) The authors mon or less rigorous estimation test and report the results of a Hausman test methods such as failure to conduct for exogeneity (p ≤ 0.05 is required to reject multivariate analysis for outcomes the null hypothesis of exogeneity); (b) the coef- equations OR if some important ficient of the selectivity correction term (Rho) outcomes are subsequently omit- is significantly different from zero (P < 0.05) ted from the results or the signifi- (Heckman approach) cance and magnitude of important - For studies using multivariate regression anal- outcomes was not assessed ysis, authors conduct appropriate specification tests (e.g., testing robustness of results to the inclusion of additional variables, or (very rare) reporting results of multicollinearity test, etc.) 6: Report- Analysis reporting bias— Open answer Justification for coding decision (Include a brief ing Justification summary of justification for rating, mention- bias-Jus- ing your response to all sub-questions, cite tification relevant pages) 7: Other 7—Other risks of bias: Is 1 = Yes, 4 = No Score “Yes” if the reported results do not sug- bias- the study free from other gest any other sources of bias. Score “No” if Assess- sources of bias? other potential threats to validity are present, ment and note these here (e.g., coherence of results, survey instruments used are not reported) 7: Other Other risks of bias-Justifi- Open answer Justification for coding decision (Include a brief bias-Jus- cation summary of justification for rating, mention- tification ing your response to all sub-questions, cite relevant pages) 8: 8—External validity Open answer Open answer- what do authors say about External external validity if anything? validity B erretta et al. Agriculture & Food Security (2023) 12:13 Page 39 of 52 Qualitative analysis tool Study type Methodological appraisal criteria Response Yes No Comment Screening questions: Configurative assessment: assessing ‘fatal flaws’ • Study reports primary data and applied methods (Dixon-Woods 2005) • Study states clear research questions and objectives Configurative ‘fatal flaws’ • Study states clear research design, which is appropriate to address the stated research ques- based on Pawson (2003) tion and objectives (Purposivity) TAPUS framework • The findings of the study are based on collected data, which justify the knowledge claims (Accuracy) Screening question based on abstract and/or superficial reading of full text: Further appraisal is not feasible or appro ‑ priate when the answer is ‘No’ to any of the above screening questions! 1. Qualitative and descrip- I. RESEARCH IS DEFENSIBLE IN DESIGN (providing a research strategy that addresses the tive quantitative, and question) process evaluations Appraisal indicators: Bullet Is the research design clearly specified and appropriate for aims and objectives of the research? Consider whether i. there is a discussion of the rationale for the study design ii. the research question is clear, and suited to the inquiry iii. there are convincing arguments for different features of the study design iv. limitations of the research design and implications for the research evidence are discussed Defensi- Arguable Critical Not Worth to con- ble defen- tinue: sible II. RESEARCH FEATURES AN APPROPRIATE SAMPLE (following an adequate strategy for selection of participants) Appraisal indicators: Consider whether i. there is a description of study location and how/why it was chosen ii. the researcher has explained how the participants were selected iii. the selected participants were appropriate to collect rich and relevant data iv. reasons are given why potential participants chose not take part in study Appropriate sample Functional sample Critical sample Flawed sample Worth to continue: III. RESEARCH IS RIGOROUS IN CONDUCT (Providing a systematic and transparent account of the research process) Appraisal indicators: Consider whether i. researchers provide a clear account/description of the process by which data was collected (e.g., for interview method, is there an indication of how interviews were conducted? /procedures for collection or recording of data?) ii. researchers demonstrate that data collection targeted depth, detail, and richness of information (e.g., interview/observation schedule) iii. there is evidence of how descriptive analytical categories, classes, labels, etc. have been gener- ated and used iv. presentation of data distinguishes clearly between the data, the analytical frame used, and the interpretation v. methods were modified during the study; and if so, has the researcher explained how and why? Rigorous Considerate conduct Critical Flawed conduct Worth to continue: conduct conduct IV. RESEARCH FINDINGS ARE CREDIBLE IN CLAIM/BASED ON DATA (Providing well-founded and plausible arguments based on the evidence generated) Appraisal indicators: Consider whether i. there is a clear description of the form of the original data ii. sufficient amount of data is presented to support interpretations and findings/conclusions Berretta et al. Agriculture & Food Security (2023) 12:13 Page 40 of 52 Study type Methodological appraisal criteria Response Yes No Comment iii. the researchers explain how the data presented were selected from the original sample to feed into the analysis process (i.e., commentary and cited data relate; there is an analytical context to cited data, not simply repeated description; is there an account of frequency of presented data?) iv. there is a clear and transparent link between data, interpretation, and findings/conclusion? v. there is evidence (of attempts) to give attention to negative cases/outliers, etc.? Credible Arguable Doubtful claims Not credible If findings not credible, can claims claims data still be used? V. REASEARCH ATTENDS TO CONTEXTS (Describing the contexts and particulars of the study) Appraisal indicators: Consider whether i. there is an adequate description of the contexts of data sources and how they are retained and portrayed? ii. participants’ perspectives/observations are placed in personal contexts iii. appropriate consideration is given to how findings relate to the contexts (how findings are influenced by or influence the context) iv. the study makes any claims (implicit or explicit) that infer generalization (if yes, comment on appropriateness) Context Context considered Context men- No context attention central tioned` VI. RESEARCH IS REFLECTIVE (Assessing what factors might have shaped the form and output of research) Appraisal indicators: Consider whether i. appropriate consideration is given to how findings relate to researchers’ influence/own role during analysis and selection of data for presentation ii. researchers have attempted to validate the credibility of findings (e.g., triangulation, respondent validation, more than one analyst) iii. researchers explain their reaction to critical events that occurred during the study iv. researchers discuss ideological perspectives/values/philosophies and their impact on the meth- odological or other substantive content of the research (implicit/explicit) Reflection Consideration Acknowledgment Unreflective research NB: Can override previous exclusion! B erretta et al. Agriculture & Food Security (2023) 12:13 Page 41 of 52 Study type Methodological appraisal criteria Response Yes No Comment OVERALL CRITICAL APPRAISAL DECISION Decision rule: – a single critical appraisal judgment in any of the 6 appraisal domains leads to a criti- cal overall judgment – 2 or more high critical appraisal judgements in any of the 6 appraisal domains lead to an overall high risk of bias / low-quality rating – 2 or more moderate critical appraisal judge- ments in any of the 6 appraisal domains lead to an overall moderate risk of bias / moderate quality rating – which means that for a study to be rated of low risk of bias / high quality at least 5 appraisal domains need be rated as of low critical appraisal High-quality Moderate-quality Low-quality Critical Empirical research Empirical research (study generates new evidence Empirical research (study generates new evidence quality (study generates new relevant to the review question and complies with relevant to the review question and complies with Empirical evidence relevant to reasonable methodological criteria to ensure reliability minimum methodological criteria to ensure reliability research the review question and empirical grounding of the evidence) and empirical grounding of the evidence) (the and complies with all evidence methodological criteria generated to ensure reliability and by the empirical grounding of study does the evidence) not comply with mini- mum meth- odological criteria to ensure reli- ability and empirical ground- ing of the evidence) Sources used in this section (in alphabetical order); Campbell et al. [9]; CASP (2006); CRD (2009); Dixon-Woods et al. (2004); Dixon- Woods et al. (2006); Greenhalgh and Brown (2014); Harden et al. (2004); Harden et al. (2009); Harden and Gough (2012); Mays and Pope (1995); Pluye et al. (2011); Spencer et al. 2006; Thomas et al. (2003); SCIE (2010) Berretta et al. Agriculture & Food Security (2023) 12:13 Page 42 of 52 Study type Methodological appraisal Response criteria Yes No Comment /confidence judgment 2. Mixed-methods I. RESEARCH INTEGRATION/ Sequential explanatory design SYNTHESIS OF METHODS The quantitative component is (Assessing the value-added of followed by the qualitative. The the mixed methods approach) purpose is to explain quantitative Applied mixed methods design: results using qualitative findings. Sequential explanatory design E.g., the quantitative results guide Sequential explorative design the selection of qualitative data Triangulation design sources and data collection, and Embedded design the qualitative findings contribute Appraisal indicators: to the interpretation of quantita- Consider whether tive results i. the rationale for integrating Sequential exploratory design the qualitative and quantitative qualitative component is followed methods to answer the research by the quantitative. The purpose question is explained is to explore, develop and test an [DEFENSIBLE] instrument (or taxonomy), or a ii. mixed methods research design conceptual framework (or theo- is relevant to address the qualita- retical model). E.g., the qualitative tive and quantitative research findings inform the quantitative questions, or the qualitative and data collection, and the quantita- quantitative aspects of the mixed tive results allow a generalization methods research question of the qualitative findings [DEFENSIBLE] Triangulation designs the qualita- tive and quantitative components iii. there is evidence that data gath- are concomitant. The purpose is to ered by both research methods examine the same phenomenon was brought together to inform by interpreting qualitative and new findings to answer the mixed quantitative results (bringing data methods research question (e.g., analysis together at the interpreta- form a complete picture, synthe- tion stage), or by integrating quali- size findings, configuration) tative and quantitative datasets [CREDIBLE] (e.g., data on same cases), or by iv. the approach to data integra- transforming data (e.g., quantiza- tion is transparent and rigorous in tion of qualitative data) considering all findings from both Embedded/convergent design the qualitative and quantitative The qualitative and quantitative module (danger of cherry-picking) components are concomitant. The [RIGOROUS] purpose is to support a qualita- v appropriate consideration is tive study with a quantitative given to the limitations associ- sub-study (measures), or to better ated with this integration, e.g., understand a specific issue of a the divergence of qualitative and quantitative study using a qualita- quantitative data (or results)? tive sub-study, e.g., the efficacy [REFLEXIVE] or the implementation of an intervention based on the views of participants B erretta et al. Agriculture & Food Security (2023) 12:13 Page 43 of 52 Study type Methodological appraisal Response criteria Yes No Comment /confidence judgment For mixed methods research studies, each component undergoes its individual critical appraisal first. Since qualitative studies are either included or excluded, no combined risk of bias assessment is facilitated, and the assigned risk of bias from the quantitative component similarly holds for the mixed methods research The above appraisal indicators only refer to the applied mixed methods design. If this design is not found to comply with each of the four mixed methods appraisal criteria below, then the quantitative/qualitative components will individually be included in the review: Mixed-methods critical appraisal: Qualitative critical appraisal: Quantitative critical appraisal: 1. Research is defensible in Include/Exclude 1. Low risk of bias design 2. Risk of bias 2. Research is rigorous in 3. High risk of bias conduct 4. Critical risk of bias 3. Research is credible in claim 4. Research is reflective Combined appraisal: Include / Exclude mixed methods findings judged with ____________________________ risk of bias Section based on Pluye et al. (2011). Further sources consulted (in alphabetical order): Creswell and Clark (2007); Crow (2013); Long (2005); O’Cathain et al. (2008); O’Cathain (2010); Pluye and Hong (2014); Sirriyeh et al. (2011) For the qualitative studies, we use a slightly different language to scale the critical appraisal assessments as compared to the quantitative studies. The far right rating column always reflects a ‘critical’ appraisal judgment (i.e., ‘unreflective research’ above) with judgements moving further to the left on a scale from high to low critical appraisal Detailed results for diet quality and adequacy Appendix 4: Additional meta‑analysis results We included a total of k = 4 studies in the analysis. The Detailed results for food security observed outcomes ranged from 0.08 to 0.14 . The esti - A total of k = 4 studies were included in the analysis. mated average outcome based on the random effects The observed outcomes ranged from 0.07 to 0.67 , with model was µ = 0.09 (95% CI: 0.06 to 0.12 ). Therefore, the majority of estimates being positive (100%). The esti - the average outcome differed significantly from zero mated average outcome based on the random effects ( z = 5.64 , p < 0.01 ). According to the Q-test, there was model was µ = 0.24 (95% CI: 0.00 to 0.47 ). Therefore, no significant amount of heterogeneity in the true out - the average outcome differed significantly from zero 2 2 comes (Q (3) = 0.53 , p = 0.91 , τ = 0.00 , I = 0.00%). ( z = 1.97 , p = 0.05 . According to the Q-test, the true An examination of the studentized residuals revealed outcomes appear to be heterogeneous (Q (3) = 111.16 , 2 2 that none of the studies had a value larger than ±2.50 and p < 0.01 , τ = 0.06 , I = 97.30%). hence there was no indication of outliers in the context of An examination of the studentized residuals revealed this model. that one study [25] had a value larger than ±2.50 and may be a potential outlier in the context of this model. Detailed results for anthropometric measures We included a k = 2 studies in the analysis. The estimated Detailed results for food affordability/availability average outcome based on the random effects model We included a total of k = 6 studies were included in was µ = 0.12(95% CI: 0.00to0.23 ). Therefore, the average the analysis. The observed outcomes ranged from 0.08 to outcome did not differ significantly from zero ( z = 1.99 , 0.49 , with the majority of estimates being positive (100%). p = 0.05 ). According to the Q-test, there was no sig- The estimated average outcome based on the random nificant amount of heterogeneity in the true outcomes effects model was µ = 0.23 (95% CI: 0.09 to 0.38 ). There - 2 2 (Q (1) = 0.12 , p = 0.73 , τ = 0.00 I = 0.00% ). Given the fore, the average outcome differed significantly from zero small number of studies, this result should be interpreted ( z = 3.19 , p < 0.01 ). According to the Q-test, the true with caution. outcomes appear to be heterogeneous (Q (15) = 187.27 , 2 2 p < 0.01 , τ = 0.02 , I = 91.99%). Detailed results for well‑being outcomes An examination of the studentized residuals revealed We included a k = 2 studies in the analysis. The esti - that one study (Ahmed et al. 2019 had a value larger than mated average outcome based on the random effects ±2.96 and may be a potential outlier in the context of this model was µ = 0.08(95% CI: 0.01to0.15 ). Therefore, the model. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 44 of 52 average outcome did not differ significantly from zero Appendix 5: Detailed risk of bias ( z = 2.11 , p = 0.034 ). According to the Q-test, there was See Tables 5 and 6 significant amount of heterogeneity in the true outcomes The nine additional qualitative studies were assessed. 2 2 (Q (1) = 2.90 , p = 0.08 , τ = 0.00 I = 65.57% ). Given Five [37, 38, 39, 40, 48] were found to be high quality, the small number of studies, this result should be inter- with the remaining four [41, 49, 5051] marked as medium preted with caution. quality according to the assessment tool. The main Table 5 Risk of bias in experimental studies Author Overall (Year) Score Some Some Heckert concer Low Low Low Low Low Low Low conce (2019) ns Risk Risk Risk Risk Risk Risk Risk rns Deining Some Some Some Some er concer Low concer concer Low concer Low Low Low (2009) ns Risk ns ns Risk ns Risk Risk Risk Some Some Some Haque concer Low Low concer Low Low Low Low conce (2021) ns Risk Risk ns Risk Risk Risk Risk rns Blaksta d Low Low Low Low Low Low Low Low Low (2020) ROB Risk Risk Risk Risk Risk Risk Risk Risk Bandier Some Some a concer Low Low Low Low Low concer Low Low (2017) ns Risk Risk Risk Risk Risk ns Risk Risk Some Some Ahmed concer Low Low Low Low Low concer Low Low (2019) ns Risk Risk Risk Risk Risk ns Risk Risk Assignment mechanism Unit of analysis Selection bias Confounding Deviations from intended Performance bias Outcome measurement Reporting bias B erretta et al. Agriculture & Food Security (2023) 12:13 Page 45 of 52 Table 6 Risk of bias in quasi-experimental studies Author Overall (Year) Score Low Low Low Low Low Pan (2015) Low ROB risk risk Low risk risk risk risk Marquis Low Low Low Low Low (2015) Low ROB risk risk Low risk risk risk risk Emran Some Low Low Some Low Low Low (2009) concerns risk risk concerns risk risk risk Bonuedi Low High High Low High (2020) High ROB risk risk High risk risk risk risk factor differentiating high and medium quality quali - Below is the list of databases and organizational web- tative studies was the level of rigor and detail provided sites searched in the FSN EGM. This online Appendix in the methods. Triangulating data by interviewing dif- provides more detailed information about the search ferent population groups in a given community allowed strategy: https:// www. 3ieim pact. org/ sites/ defau lt/ files/ for different perspectives, making qualitative studies 2021- 01/ EGM16- Online- appen dix-B- Search- strat egy. pdf more rigorous. Sometimes the male head of household was interviewed along with the woman beneficiary, as Academic databases well as other community members, which can affect the We conducted electronic searches of the following data- information reported. Studies were high quality if they bases of published sources: triangulated data, used ethical methods (i.e., did not add additional burden onto women’s time) and added rich • MEDLINE contextual layers to quantitative findings in other studies • EMBASE or the same study. • Cochrane Controlled Trials Register (CENTRAL) • CINAHL • CAB Global Health • CAB Abstracts Appendix 6: Eec ff t estimates from included studies • Agricola See Table 7 • PsychINFO Appendix 7: Food system EGM framework • Africa-Wide Information and search strategy • Academic Search Complete See Table 8 • Scopus The complete Food system EGM framework can be • Campbell Library found at this link: https:// www. 3ieim pact. org/ sites/ defau lt/ files/ 2021- 01/ EGM16- Online- appen dix-A- Addit ional- metho ds- detail. pdf Website searched Selection bias Confounding Performance bias Spillovers, crossovers, and Outcome measurement bias Reporting bias Berretta et al. Agriculture & Food Security (2023) 12:13 Page 46 of 52 ff Table 7 Eect estimates from included studies in REA† First author Year Country Intervention type Evaluation/synthesis method Outcome Standardized effect Sample size estimate (Confidence Interval) Food security Bandiera 2017 Bangladesh Training/education Asset Randomized control trial and Food security index—whether 0.07 (0.03; 0.12) 6732 transfer difference-in-difference HH had surplus food or deficit, enough food to eat, and could afford to eat two meals a day* Emran 2009 Bangladesh Training/education Asset Difference-in-difference and Meals twice a day 0.6 (0.5; 0.7) 1569 transfer statistical matching Food availability: Sufficient food 0.66 (0.56; 0.76) 1569 to meet the household’s needs* Blackstad 2020 Tanzania Training/education Randomized control trial Household food insecurity 0.07 (− 0.05; 0.2) 876 access scale* Pan 2015 Uganda Training/education Asset Regression discontinuity Skip meals* − 0.13 (− 0.2; − 0.06) 3368 transfer Food affordability and avail- ability Ahmed 2019 Bangladesh Asset transfer (Cash) Randomized control trial Per capita monthly food con- 0.13 (0.08; 0.19) 5000 sumption—North Per capita monthly food con- 0.08 (0.03; 0.14) 5000 sumption—South Per capita daily intake caloric— 0.07 (0.02; 0.13) 5000 North Per capita daily intake caloric— 0.02 (− 0.03; 0.08) 5000 South Food consumption score— 0.17 (0.12; 0.23) 5000 North Food consumption score 0.07 (0.02; 0.13) 5000 − South Asset transfer (Food) Per capita monthly food con- 0.11 (0.06; 0.17) 5000 sumption—North Per capita monthly food con- 0.07 (0.02; 0.13) 5000 sumption—South Per capita daily intake caloric— 0.13 (0.08; 0.19) 5000 North Per capita daily intake caloric— 0.01 (− 0.05; 0.06) 5000 South Food consumption score— 0.24 (0.18; 0.29) 5000 North B erretta et al. Agriculture & Food Security (2023) 12:13 Page 47 of 52 Table 7 (continued) First author Year Country Intervention type Evaluation/synthesis method Outcome Standardized effect Sample size estimate (Confidence Interval) Food consumption score— 0.12 (0.07; 0.18) 5000 South Assets transfer (Cash and food) Per capita monthly food con- 0.11 (0.06; 0.17 5000 sumption—North Per capita monthly food con- 0.11 (0.06; 0.17) 5000 sumption—South Per capita daily intake caloric— 0.07 (0.02; 0.13) 5000 North Per capita daily intake caloric— 0.03 (− 0.03; 0.09) 5000 South Food consumption score— 0.16 (0.11; 0.22) 5000 North Food consumption score— 0.11 (0.06; 0.17) 5000 South Behavior change communica- Per capita monthly food con- 0.32 (0.27; 0.38) 5000 tion sumption—North Asset transfer (Cash) Per capita monthly food con- 0.22 (0.17; 0.28) 5000 sumption—South Per capita daily intake caloric— 0.22 (0.17; 0.28) 5000 North Per capita daily intake caloric— 0.10 0.04; 0.15) 5000 South Food consumption score— 0.49 (0.44; 0.55) 5000 North* Food consumption score— 0.28 (0.23; 0.34) 5000 South* Bonuedi 2020 Sierra Leone Behavior change communica- Statistical matching Total food consumption − 0.04 (− 0.18, 0.09) 836 tion Training/education Assets expenditure in the 12 months transfer preceding the survey (Food production and market pur- chases) (LOG)-Household* Total food consumption 0.22 (0.09; 0.36) 836 expenditure in the 12 months preceding the survey (Food production and market pur- chases) (LOG)-Household* Berretta et al. Agriculture & Food Security (2023) 12:13 Page 48 of 52 Table 7 (continued) First author Year Country Intervention type Evaluation/synthesis method Outcome Standardized effect Sample size estimate (Confidence Interval) Deininger 2009 India Training/education Difference-in-difference and Food consumption (RS/year)— 0.09 (− 0.03; 0.2) 2199 statistical matching All groups* Energy intake p.c. (kcal/day)— 0.02 (− 0.09; 0.14) 2199 All groups Food consumption (RS/year)- 0.19 (− 0.08; 0.47) 404 POP (Poorest of the poor) Food consumption (RS/year)- 0.42 (0.06; 0.77) 243 Poor Food consumption (RS/year)- − 0.11 (− 0.54; 0.33) 157 Non-poor Energy intake p.c. (kcal/day)- 0.36 (0.09; 0.64) 404 POP (Poorest of the poor) Energy intake p.c. (kcal/day)- 0.59 (0.23; 0.95) 243 Poor Energy intake p.c. (kcal/day)- − 0.08 (− 0.52; 0.36) 157 Non-poor Emran 2009 Bangladesh Assets transfer Training/educa- Difference-in-difference and Grain stocks (kg)* 0.22 (0.12; 0.32) 1569 tion statistical matching Pan 2015 Uganda Training/education Assets Regression discontinuity Per capita food consumption* 0.08 (0.01; 0.15) 3368 transfer Diet quality and adequacy Bonuedi 2020 Sierra Leone Behavior change communica- Propensity score matching Household dietary diversity* 0.14 (0.00, 0.27) 836 tion Training/education Women’s dietary diversity 0.10 (− 0.05, 0.26) 636 Children’s dietary diversity − 0.05 (− 0.21, 0.11) 575 Behavior change communica- Household dietary diversity 0.23 (0.09, 0.26) 836 tion Training/education Assets transfer Women’s dietary diversity 0.31 (0.15, 0.46) 636 Children’s dietary diversity 0.29 (0.12, 0.45 575 Haque 2021 Bangladesh Training/education Randomized control trial Additional food consumed dur- 0.09 (0.05, 0.13) 10722 ing pregnancy* Deninger 2009 India Training/education Difference-in-difference and Protein intake p.c. (g/day) in the 0.08 (− 0.04, 0.19) 1099.5 statistical matching total population* Protein intake p.c. (g/day) 0.32 (0.05, 0.60) 202 among the poor of the poor B erretta et al. Agriculture & Food Security (2023) 12:13 Page 49 of 52 Table 7 (continued) First author Year Country Intervention type Evaluation/synthesis method Outcome Standardized effect Sample size estimate (Confidence Interval) Protein intake p.c. (g/day) 0.66 (0.30, 1.02) 121.5 among the poor Protein intake p.c. (g/day) 0.20 (− 0.24, 0.64) 78.5 among the non-poor Pan 2015 Uganda Training/education Asset Regression discontinuity Variety of foods consumed* 0.09 (0.02, 0.15) 3368 transfer Anthropometrics Heckert 2019 Burkina Faso Behavior change communica- Randomized control trial Weight-for-length z score* 0.12 (0.00,0.25) 1035 tion Asset transfer Marquis 2015 Ghana Training/education Asset Difference-in-difference BMI-for-age z score* 0.06 (− 0.30, 0.41) 121.6 transfer Weight-for-age z-sore − 0.42 (− 0.77, − 0.06) 121.6 Height-for-age z score 0.40 (0.04,0.75) 121.6 Micronutrient status Haque 2021 Bangladesh Training/education Randomized control trial Consumption of at least 100 IFA 0.25 (0.21, 0.28) 10722 tablets during pregnancy Received vitamin a capsule 0.20 (0.16, 0.24) 10722 after last delivery Heckert 2019 Burkina Faso Behavior change communica- Randomized control trial Change in hemoglobin (g/dL) 0.10 (− 0.02, 0.23) 1035 tion Asset transfer Wellbeing outcomes Bandiera 2017 Bangladesh Training/education Asset Randomized control trial and Mental health index 0.04 (− 0.00, 0.09) 6732 transfer difference-in-difference Pan 2015 Uganda Training/education Asset Regression discontinuity Worry about insufficient food − 0.11 (− 0.18, − 0.04) 3368 transfer *Indicates estimates that were used in meta-analysis. Only one outcome per study per analysis was included to maintain independence of observations. Outcomes were selected based on comparability with other studies †Some studies appear multiple times because they report data related to multiple outcomes Berretta et al. Agriculture & Food Security (2023) 12:13 Page 50 of 52 Table 8 PICOS summary of criteria for the inclusion and exclusion of studies Criteria Inclusion criteria Exclusion criteria Population Program participants that were located in a L&MIC in the first year Studies focused on niche populations, such as athletes or the of implementation3 military Impact evaluations with at least one effect size for an L&MIC Efficacy studies, unless they were completed in a sufficiently real- country population world setting Studies focused on the prevention of clinical conditions Studies targeting participants with a clinical condition Studies focused on high-income country migrant populations in L&MICs and vice versa Intervention Interventions that directly intervene on an aspect of the food Interventions not in the food system or interventions targeting system within its three primary domains: the food supply chain, drivers of the food system without an explicit food system focus the food environment and consumer behavior Studies evaluating multiple food systems interventions Unconditional cash transfer programs Interventions focused on the financing of a food systems interven- tion Comparisons Appropriate comparisons included: business as usual, an alterna- Studies that did not justify and make use of an appropriate com- tive treatment, no treatment or an early-versus-late comparison parison group (where those that took part in earlier years are compared to those that took part in later years) Source: 3ie 2020 The cutoff at the year 2000 was made arbitrarily to make the volume of search results more manageable Gray literature sites searched • Abdul Latif Jamee l Pover ty Actio n Lab To identify relevant gray literature, we searched the fol- • Globa l Devel opmen t Netwo rk lowing databases (some of which contain a mixture of • World Bank Devel opmen t Impac t Evalu ation (DIME) published and gray literature):and Impac t Evalu ation Polic y Papers • nter- Ameri can Devel opmen t Bank • Cente r for Globa l Devel opment • Google Scholar • Cente r for Effec tive Globa l Actio n (CEGA) • EconLit • Depar tment for Inter natio nal Devel opmen t Resea rch • ENN-Network for Devel opmen t (R4D) • IDEAS/RePEc • Innovative Methods and Metrics for Agriculture and • USAID Nutrition Actions grantee database • Inter natio nal Food Polic y Resea rch Insti tute • WHO Global Index Medicus • CIGAR • Gray Literature Report • Food and Agric ultur e Organ ization of t he Unite d • Social Science Research Network (SSRN)Natio ns (FAO) • Eldis • High Level Pane l of Exper ts on Food Secur ity and • EpistemonikosNutri tion • 3ie Development Evidence Portal • World Food Progr amme • Registry of International Development Impact Evalu • Actio n Again st Hunger • UNICEF ations (RIDIE) • Unite d Natio ns Evalu ation Group • Oxfam Policy & Practice • Asian Devel opmen t Bank • World Agrof orest ry Centr e (ICRAF) Below is a list of organizational websites we manually • Inter natio nal Lives tock Resea rch Insti tute (ILRI) searched for additional related studies. • Nutri tion Inter natio nal • AgEco n Searc h (Unive rsity of Minne sota) • Innov ation s for Pover ty Action B erretta et al. Agriculture & Food Security (2023) 12:13 Page 51 of 52 fss_ briefs_ review_ evide nce_ gender_ equal ity. pdf? seque nce= 3& isAll Supplementary Information owed=y The online version contains supplementary material available at https:// doi. 3. WHO. Understanding the women’s empowerment pathway. Brief #4. org/ 10. 1186/ s40066- 023- 00405-9. Improving nutrition through agriculture technical brief series. Arlington: Additional file 1. It contains additional information about the included USAID/Strengthening Partnerships, Results, and Innovations in Nutrition studies in terms of Intervention type,Detailed intervention, Evaluation Globally (SPRING) Project. 2014. method, Hypotheses mechanisms of action, Impacts Barriers and facili- 4. United Nations Food Systems Summit. Chapter 2 key inputs from summit tatorsto impact, implementation, and evaluation, Equity consideration, workstreams action tracks. 2021. https:// foods ystems. commu nity/ food- Sources of bias, Risk of bias, Eec ff tiveness,ans Conclusions.syste ms- summit- compe ndium/ action- tracks/. Accessed 27 Jan 2022. 5. Cole SM, Kantor P, Sarapura S, Rajaratnam S. Gender-transformative approaches to address inequalities in food, nutrition and economic Acknowledgements outcomes in aquatic agricultural systems. 2015. Not applicable. 6. Wong, F, Vos A, Pyburn R, Newton J. Implementing gender transformative approaches in agriculture. A Discussion Paper for the European Commis- Author contributions sion. 2019. MB contributed to extract the effect sizes and analyze them through the 7. Cheung J, Gursel D, Kirchner MJ, Scheyer V. Practicing feminist foreign meta-analysis. She was a major contributor in writing the manuscript. MK policy in the everyday: a toolkit. Germany; 2021. searched the additional qualitative studies, extracted the data and analyzed 8. Thompson L. Defining feminist foreign policy. Washington: International them. They were a major contributor in writing the manuscript. CL contrib- Center for Research on Women; 2019. p. 1–7. uted extract the effect sizes and analyze them through the meta-analysis. She 9. Campbell Collaboration. (2017). Campbell systematic reviews: Policies was a major contributor in reviewing and writing the manuscript, as well as and guidelines. in ensuring its overall quality. SS ensured the meta-analysis were conducted 10. Barends, E., Rousseau, D. M. & Briner, R. B. CEBMa Guideline for Rapid following the highest standard and corrected any mistakes. She was a major Evidence Assessments in Management and Organizations. Amsterdam. contributor in reviewing the manuscript, as well as in ensuring its overall qual- 2017. https:// www. cebma. org/ wp- conte nt/ uploa ds/ CEBMa- REA- Guide ity. All authors read and approved the final manuscript. line. pdf 11. Snilstveit B. Systematic reviews: from ‘bare bones’ reviews to policy Funding relevance. J Dev Eec ff t. 2012;4(3):388–408. https:// doi. org/ 10. 1080/ 19439 This project has been commissioned and funded by Germany’s Federal Min- 342. 2012. 709875. istry for Economic Cooperation and Development (BMZ) through Deutsche 12. CASP (2018). Qualitative Checklist. [online] Available at: https://casp-uk. Gesellschaft für Internationale Zusammenarbeit (GIZ) through its “Knowledge net/images/checklist/documents/CASP-Qualitative-Studies-Checklist/ for Nutrition” program. CASP-Qualitative-Checklist-2018_fillable_form.pdf Accessed: 1st March Availability of data and materials 13. Korth M, Stewart R, Langer L, Madinga N, Rebelo Da Silva N, Zaranyika H, All the data used to support finding in this study derive from the included van Rooyen C, de Wet T. What are the impacts of urban agriculture pro- study, please see the full list in the Reference section. grams on food security in low and middle-income countries: a systematic review. Environ Evid. 2014;3(1):1–10. 14. Stewart R, Langer L, Da Silva NR, Muchiri E, Zaranyika H, Erasmus Y, Declarations Randall N, Rafferty S, Korth M, Madinga N, de Wet T. The effects of train- ing, innovation and new technology on African smallholder farmers’ Ethics approval and consent to participate economic outcomes and food security: a systematic review. Campbell This is not applicable to this manuscript as we did not use primary data. Syst Rev. 2015;11(1):1–224. 15. Goudet SM, Bogin BA, Madise NJ, Griffiths PL. Nutritional interventions for Consent for publication preventing stunting in children (birth to 59 months) living in urban slums This is not applicable as we did not include any individual person’s data in any in low-and middle-income countries (LMIC). Cochrane Database Syst Rev. form. 2019. https:// doi. org/ 10. 1002/ 14651 858. CD011 695. pub2. 16. Sguassero Y, de Onis M, Bonotti AM, Carroli G. Community-based supple- Competing interests mentary feeding for promoting the growth of children under five years The authors declare that they have no competing interests. of age in low and middle income countries. Cochrane Database Syst Rev. 2012;2012(6):CD005039. https:// doi. org/ 10. 1002/ 14651 858. CD005 039. Author details pub3. The International Initiative for Impact Evaluation, Synthesis, and Reviews 17. Visser ME, Schoonees A, Ezekiel CN, Randall NP, Naude CE. Agricultural Office (SRO), London, UK. The International Initiative for Impact Evaluation, and nutritional education interventions for reducing aflatoxin exposure Washington, DC, USA. to improve infant and child growth in low-and middle-income countries. Cochrane Database Syst Rev. 2020. https:// doi. org/ 10. 1002/ 14651 858. Received: 20 June 2022 Accepted: 10 January 2023 CD013 376. pub2. 18. Shah D, Sachdev HS, Gera T, De-Regil LM, Peña-Rosas JP. Fortification of staple foods with zinc for improving zinc status and other health outcomes in the general population. Cochrane Database Syst Rev. 2016. https:// doi. org/ 10. 1002/ 14651 858. CD010 697. pub2. 19. Suchdev PS, Peña-Rosas JP, De-Regil LM. Multiple micronutrient powders References for home (point-of-use) fortification of foods in pregnant women. 1. Moore N, Lane C, Storhaug I, Franich A, Rolker H, et al. The effects of Cochrane Database Syst Rev. 2015. https:// doi. org/ 10. 1002/ 14651 858. food systems interventions on food security and nutrition outcomes in CD011 158. pub2. low- and middle-income countries. International Initiative for Impact 20. Suchdev PS, Jefferds MED, Ota E, da Silva LK, De-Regil LM. Home forti- Evaluation (3ie); 2021. fication of foods with multiple micronutrient powders for health and 2. Njuki J, Eissler S, Malapit H, Meinzen-Dick R, Bryan E, Quisumbing A. A nutrition in children under two years of age. Cochrane Database Syst Rev. review of evidence on gender equality, women’s empowerment, and 2020;2(2):CD008959. https:// doi. org/ 10. 1002/ 14651 858. CD008 959. pub3. food systems: Food Systems Summit Brief Prepared by Research Partners 21. Gera T, Sachdev HS, Boy E. Eec ff t of iron-fortified foods on hematologic of the Scientific Group for the Food Systems Summit. 2021. https:// and biological outcomes: systematic review of randomized controlled bonnd oc. ulb. uni- bonn. de/ xmlui/ bitst ream/ handle/ 20. 500. 11811/ 9132/ trials. Am J Clin Nutr. 2012;96(2):309–24. https:// doi. org/ 10. 3945/ ajcn. 111. 031500. Berretta et al. Agriculture & Food Security (2023) 12:13 Page 52 of 52 22. Ota E, Hori H, Mori R, Tobe-Gai R, Farrar D. Antenatal dietary education 45. Choudhury N, Raihan MJ, Ahmed ST, Islam KE, Self V, Rahman S, Schofield and supplementation to increase energy and protein intake. Cochrane L, Hall A, Ahmed T. The evaluation of Suchana, a large-scale development Database Syst Rev. 2015. https:// doi. org/ 10. 1002/ 14651 858. CD000 032. program to prevent chronic undernutrition in north-eastern Bangladesh. pub3. BMC Public Health. 2020;20:1–9. 23. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to meta- 46. Snilstveit, B, Stevenson, J, Langer, L, da Silva, N, Rabat, Z, Nduku, P, Polanin, analysis. New York: John Wiley & Sons; 2021. J, Shemilt, I, Eyers, J, Ferraro, PJ, Incentives for climate mitigation in the 24. Ahmed A, Hoddinott J, Roy S. Food transfers, cash transfers, behavior land use sector – the effects of payment for environmental services (PES) change communication and child nutrition. Intl Food Policy Res Inst. on environmental and socio-economic outcomes in low- and middle- 2019;1868. income countries: a mixed-method systematic review 3ie Systematic 25. Bandiera O, Burgess R, Das N, Gulesci S, Rasul I, Sulaiman M. Labor Review 44. London: International Initiative for Impact Evaluation (3ie). markets and poverty in village economies. Q J Econ. 2017;132(2):811–70. 2019. https:// doi. org/ 10. 23846/ SR000 44 https:// doi. org/ 10. 1093/ qje/ qjx003. 47. Critical Appraisal Skills Programme. (2006) 10 questions to help you 26. Blakstad MM, Mosha D, Bellows AL, Canavan CR, Chen JT, Mlalama makesense of qualitative research. Public Health Resource Unit. K, et al. Home gardening improves dietary diversity, a cluster-rand- Retrieved fromhttp:// www. biome dcent ral. com/ conte nt/ suppl ement ary/ omized controlled trial among Tanzanian women. Matern Child Nutr. 2046‐4053‐3‐139‐S8. pdf 2021;17(2):e13096. https:// doi. org/ 10. 1111/ mcn. 13096. 48. Das N, Yasmin R, Ara J, Kamruzzaman M, Davis P, Behrman J, et al. How do 27. Bonuedi I, Kornher L, Gerber N. Making cash crop value chains nutrition- intrahousehold dynamics change when assets are transferred to women? sensitive: evidence from a quasi-experiment in rural Sierra Leone. SSRN Evidence from BRACCs challenging the frontiers of poverty reduction Electron J. 2020. https:// doi. org/ 10. 2139/ ssrn. 36039 18. targeting the ultra-poor program in Bangladesh. SSRN Electron J. 2013. 28. Deininger K, Liu Y. Economic and social impacts of self-help groups in https:// doi. org/ 10. 2139/ ssrn. 24057 12. India. World Bank Group; 2009. 49. Huda K, Kaur S. ‘It was as if we were drowning’: shocks, stresses and safety 29. Emran MS, Robano V, Smith SC. Assessing the frontiers of ultra-poverty nets in India. Gend Dev. 2011;19(2):213–27. https:// doi. org/ 10. 1080/ reduction: Evidence from CFPR/TUP, an innovative program in Bangla-13552 074. 2011. 592632. desh. TUP, An Innovative Program in Bangladesh. 2009. 50. Olney DK, Dillon A, Ruel MT, Nielsen J. Lessons learned from the evalua- 30. Haque MA, Choudhury N, Ahmed ST, Farzana FD, Ali M, Naz F, et al. The tion of Helen Keller International’s enhanced homestead food production large-scale community-based programme ‘Suchana ’improved maternal program. Achieving a nutrition revolution for Africa: The road to healthier healthcare practices in north-eastern Bangladesh: findings from a cluster diets and optimal nutrition. 2016;67–81. randomized pre-post study. Matern Child Nutr. 2022. https:// doi. org/ 10. 51. van den Bold M, Dillon A, Olney D, Ouedraogo M, Pedehombga A, 1111/ mcn. 13258. Quisumbing A. Can integrated agriculture-nutrition programmes change 31. Heckert J, Olney DK, Ruel MT. Is women’s empowerment a pathway to gender norms on land and asset ownership? Evidence from Burkina Faso. improving child nutrition outcomes in a nutrition-sensitive agriculture J Dev Stud. 2015;51(9):1155–74. https:// doi. org/ 10. 1080/ 00220 388. 2015. program?: Evidence from a randomized controlled trial in Burkina Faso. 10360 36. Soc Sci Med. 2019;233:93–102. https:// doi. org/ 10. 1016/j. socsc imed. 2019. 05. 016. Publisher’s Note 32. Marquis GS, Colecraft EK, Sakyi-Dawson O, Lartey A, Ahunu BK, Birks KA, Springer Nature remains neutral with regard to jurisdictional claims in pub- et al. An integrated microcredit, entrepreneurial training, and nutri- lished maps and institutional affiliations. tion education intervention is associated with better growth among preschool-aged children in rural Ghana. J Nutr. 2015;145(2):335–43. https:// doi. org/ 10. 3945/ jn. 114. 194498. 33. Pan Y, Smith SC, Sulaiman M. Agricultural extension and technology adoption for food security: evidence from Uganda. Am J Agric Econ. 2018;100(4):1012–31. https:// doi. org/ 10. 1093/ ajae/ aay012. 34. Polanin, J.R. and Snilstveit, B., 2016. Converting between effect sizes. Campbell Systematic Reviews, 12(1), pp.1-13. 35. Sánchez-Meca, J., Marín-Martínez, F., & Chacón-Moscoso, S. (2003). Eec ff t- size indices for dichotomized outcomes in meta-analysis. Psychological methods, 8(4), 448. 37. Kabeer N, Datta S. Randomized control trials and qualitative impacts: what do they tell us about the immediate and long-term assessments of productive safety nets for women in extreme poverty in West Bengal? (No. 19–199). Working Paper Series. 2020 38. Roy S, Hidrobo M, Hoddinott J, Ahmed A. Transfers, behavior change communication, and intimate partner violence: postprogram evidence from rural Bangladesh. Rev Econ Stat. 2019;101(5):865–77. https:// doi. org/ 10. 1162/ rest_a_ 00791. 39. Roy S, Ara J, Das N, Quisumbing AR. Flypaper effects” in transfers targeted to women: evidence from BRAC’s “Targeting the Ultra Poor” program in Bangladesh. J Dev Econ. 2015;117:1–19. 40. Kieran C, Gray B, Gash M. Understanding gender norms in rural Burkina Faso: a qualitative assessment. 2018. 41. Hagan LL, Aryeetey R, Colecraft EK, Marquis GS, Nti AC, University of Ghana, et al. Microfinance with education in rural Ghana: Men’s perception of household level impact. Afr J Food Agric Nutr Dev. 2012;12(49):5776–88. https:// doi. org/ 10. 18697/ ajfand. 49. enam7. 42. Dolgopolova I, Li B, Pirhonen H, Roosen J. The effect of attribute framing on consumers’ attitudes and intentions toward food: a Meta-analysis. Bio- based Appl Econ J. 2021;10:253–64. 43. Data4Diets’ (2021), International Dietary Data Expansion Project. https:// inddex. nutri tion. tufts. edu/ data4 diets. Accessed 4 Sep 2022. 44. DQQ Tools & Data’ (2021), Global Diet Quality Project. https:// www. globa ldiet quali ty. org/ dqq. Accessed 11 Apr 2022.

Journal

Agriculture & Food Security – Springer Journals

Published: May 26, 2023

Keywords: Women’s empowerment; Review; Food system; Meta-analysis

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Rapid evidence assessment on women’s empowerment interventions within the food system: a meta-analysis

Rapid evidence assessment on women’s empowerment interventions within the food system: a meta-analysis

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Rapid evidence assessment on women’s empowerment interventions within the food system: a meta-analysis

Rapid evidence assessment on women’s empowerment interventions within the food system: a meta-analysis

References (53)

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies