In Search of Evidence for Impact on De-Institutionalization: A Systematic Review of the Current Status, Gaps and Future Directions of Translatable Mental Health Research on Alternative Care

What should count as ‘evidence’ in the transition from institutionalized care to alternative family-based care arrangements for children and youth without available parents? How is evidence on de-institutionalization commonly presented with the goal to have ‘impact’ and what are the gaps stalling translation of DI findings to policy or practice of alternative family-based care? Core characteristics of independent and intersubjective scientific evidence, as produced through an empirical or meta study, are reproducibility: would the same data lead to similar if not identical conclusions? and replicability: would another study with similar methods lead to similar if not identical conclusions? These are explained elsewhere [1]. We therefore propose to use as criteria for ‘evidence for impact’ reproducibility and replicability. Independent researchers should be able to reproduce and replicate a study that presents evidence of impact, whether it is quantitative, mixed-methods, qualitative, or a meta-study. Transparency in (pre-)reporting the process and published products of a study is a key condition for evidence to have impact.

In our process model of research programs, the methodological differentiation between the context of discovery and the context of justification can be made, in line with the work of Popper [2] and Lakatos [3]. To make progress, research programs need fruitful hypotheses that can be falsified by empirical data. Without fallibility, empirical hypotheses are empty of meaning as they fail to exclude any incompatible observations and thus would always be ‘true’, or better: circular. In the context of discovery, researchers are trying to find bold hypotheses about the social or physical reality that are open to falsification. In this exploratory stage of developing such hypotheses ‘anything goes’ [4], that is, in the context of discovery, a multitude of qualitative or quantitative methods to generate such bold hypotheses are allowed if and only if in the long run they are subjected to transparent, systematic, and stringent tests to check whether they survive several efforts to undermine them with falsifying empirical data. In this process model replication studies are crucial as tests to see how robust the original hypotheses are. In the context of justification, this robustness is examined with narrative or quantitative syntheses or meta-analyses, their replication, and combination in so-called umbrella reviews of meta-analytic results. Evidence for impact can only be found in the context of justification, where guarantees of the truth-value of findings are established and ready to be applied in policy or practice. Policymakers, professionals, parents, and, not least, children have the right to receive from researchers advice and support based on intersubjective scientific insights instead of subjective intuitions.

Against this outline of what we consider to be evidence for impact, we conducted a systematic scoping review of DI studies done in the last five years. We aim to review this recent literature to chart the landscape of DI studies, note its strengths and challenges, and present some successful examples of various types of (correlational, qualitative, experimental, and mixed method) DI studies. We also want to make clear what gaps between available evidence and the translation of that evidence to policy or practice can be observed in this field of inquiry. Our approach might best be characterized as a systematic scoping review charting the domain of de-institutionalization research. Because of the heterogeneous set of studies, a quantitative synthesis did not seem possible, but we tried to make some of the search and coding steps transparent by providing inter rater agreements.

We searched for pertinent papers on Web of Science from 1 June 2018 to 1 March 2024 (the years following those that had been searched in the context of the Lancet Commission on Institutionalisation and De-institutionalization [5,6] using the following string of terms: TS = ((de-institutionalization OR de-institutionalization OR de-institutionalization OR de-institutionalization OR “alternative care” OR “family-based care” OR “family-type care” OR “care reform”) AND (child* OR adolec* OR infan* OR orphan*)) and Preprint Citation Index (Exclude—Database) and 2018–2024 (Publication Years). Except for the Preprint Citation Index, we searched all other collections of Web of Science (Web of Science Core Collection, Grants Index, KCI- Korean Journal Database, MEDLINE, ProQuest Dissertation & Theses Citation Index, and the Scientific Electronic Library Online SciELO Citation Index). The Web of Science collection covers a range of sources, including grey literature (dissertation theses, etc.), books, and more. We found 639 potentially relevant hits (Figure 1).

The screening process, outlined in the flowchart (Figure 1), started with excluding non-empirical articles (such as reviews, editorials, and policy reviews), non-primary research studies based on secondary data analysis, and studies not focusing on de-institutionalization or the target population of children or adolescents. From the 639 publications, 61 reviews, and 37 editorials were excluded using Web of Science, resulting in 541 remaining articles. Subsequently, five duplicates were identified and removed, leaving a total of 536 publications to undergo screening based on title and abstract. A total of 315 articles were excluded that comprised papers not pertaining to de-institutionalization (k = 246), not reporting empirical data (k = 42), secondary analysis studies (k = 11), and articles not focusing on children or youth (k = 16). As a result, 221 articles were eligible for further evaluation based on their full content.

Training of coding was conducted on the 50 publications collected in a previous search that had been coded by MJBK and MHvIJ, and served as training material for EA. After the training phase, the coding system was expanded with categories for Participant and Public Involvement [7] (PPI: present; absent; unclear) and PPI type of participants (care users; care leavers; professional caregivers; family members; other). Added to the original category for type of participant was “legal guardian’ and for data-analysis ‘policy framework analysis’ was added. The resulting coding system included the following categories: country of study; design; data-collection method; participant age; type of participant; sample size; sampling method; data-analysis; recommendations; focus on children with disabilities; PPI (see Supplementary Materials Table S1 for the coding system). For intercoder reliability, MHvIJ and EA independently coded another 30 papers with the final coding system. The overall intercoder agreement was good, with all percentages of agreement above 88% except for PPI (52%), but some categories had insufficient data or insufficient variance to compute agreement (age and sampling method). EA coded the remaining papers.

We refrained from a formal study quality assessment with standardized tools as the majority of studies included qualitative designs or mixed methods that are more difficult to characterize with quantitative quality tools for which inter rater reliabilities might be computed.

Regions. DI studies covered a large number of countries (67 in total) across most continents. The numbers of DI studies in Africa (61) are in the same range as DI studies conducted in Western European countries (56), whereas the number of DI papers from Eastern Europe (12), North America (18), and South America (20) are much lower, and Asian DI studies fall in the middle (36). In the current set of DI papers publications from China and Japan were virtually absent, and work in Australia and the Middle East was similarly almost lacking. Surely these are remarkable gaps on the world map because the underrepresented countries certainly do have institutions where children and youth are growing up, and careful reflection on conditions and policies to transition to family-based care would be required in the interest of the institutionalized children.

Research designs. The large majority of the studies were qualitative, k = 87, of which 4 were case studies (see Figure 2). Other qualitative approaches were historical, policy framework, or legal analysis, and phenomenological, grounded theory, or thematic analysis studies. In this qualitative category, semi-structured interviews were popular as a method of data collection, as well as focus groups. Surprisingly few studies employed the preeminent qualitative method of participant observation, which is often used in cross-cultural research [8]. Quantitative DI studies were present in sufficient numbers to illustrate their feasibility in the complicated area of DI, but their numbers are modest (k = 43) compared to the qualitative work. Again, this field of inquiry differs from mainstream developmental science, in which qualitative studies occupy a rather small niche and quantitative methods dominate the published literature. A mixture of quantitative and qualitative approaches within the same study was found in several papers (k = 29). Participant and Public Involvement [9] still appears almost absent, although in recent years the call to include youth, caregivers, and care professionals in DI studies sounds louder. It should be noted, however, that we have not been able to code PPI with sufficient inter coder agreement to be sure of quantitative estimates.

Sampling. The type of sampling was only minimally reported in many papers. When reported, the large majority of DI studies seemed to use some type of ‘convenience’ sampling (‘purposive’, ‘snowball’), k = 131, with no intention to strive for a replicable or representative data-collection approach (k = 14). Important exceptions are the Bucharest Early Intervention Project [10] with numerous papers on the effects of alternative family-based care (i.e., foster-family care) on formerly institutionalized infants. Important for DI are BEIP publications on conditions that predict more versus less successful foster care arrangements for children’s development into adolescence and emerging adulthood [11]. DI studies on population-wide administrative data tracing the development of formerly institutionalized children in various alternative care arrangements are still scarce but potentially fruitful for policy-related best-evidence DI recommendations.

Participants. Although a substantial number of studies included the target group of children in their research (k = 69, see Figure 3), two salient gaps in sampling are the under-representations of the youngest children (k = 17 in the range of 0–12 years of age) and children with disabilities (k = 18). Participants were mostly typically developing adolescents or adults (k = 139), but note that in many papers, detailed information about the age or developmental status of participants is missing. As no child should be left behind in institutional settings, the lack of DI studies on children with atypical (cognitive, motor, or neuro-) development is striking. Mostly professionals (including social workers, policymakers, NGOs) were targeted, whereas (birth, foster, or kin) parents and care leavers were also represented. Adults are more easily to reach and to participate in studies as informants or respondents than young children, which might be one of the reasons for the neglect of the youngest age categories.

Sample size. The number of participants in DI studies ranged from 1 to 5351 with a majority of studies covering a small number of participants (fewer than 20 participants in 40 studies), often recruited through convenience sampling. In quantitative developmental, biomedical, and neurobiological animal and human research, small samples turned out to be one of the most important causes for the replication crisis. It has been shown that small samples have a high risk of producing impressive results that, in the long run, cannot be replicated. The translational value of such findings is low and may even be counterproductive for application to policy or practice by creating collateral damage for the targeted children [9].

Policy Recommendations. The majority of the studies mentioned policy implications of their results (k = 158). Most of these studies suggested policy changes (k = 123), whereas k = 35 papers mentioned research gaps. In only 22 papers, no recommendations were presented. For the responsible translation of research into policy or practice, replicated results are indispensable [9,12]. The dependence of the sector on information collected with designs, sampling procedures, sample size, and data-collection and analysis methods that are difficult to reproduce makes it harder to bridge the gap between research and responsible application.

Our systematic scoping review of recent DI studies showed some substantive gaps to be filled. DI studies have not yet covered all regions and continents of the globe. Regions that are underrepresented in an absolute sense, irrespective of the number of children in alternative care, are China, Japan, the Middle East, Australia and New Zealand, and Eastern Europe. Furthermore, in remarkably few DI studies, young children from 0–6 years of age and children with disabilities are included. Some impressive (quasi-)experimental studies have been conducted among these groups, e.g., King and colleagues’ [11] Bucharest study starting in infancy and Hearst et al.’s [13] study on Zambian families with children with disabilities. These examples illustrate the feasibility of DI research in rather difficult circumstances and deserve to be replicated in other regions or countries. Sampling in this field of inquiry is unfortunately, mostly convenience recruitment of participants with the risk of dependent data as ‘birds of a feather flock together’ (whether it is snowballing or purposive sampling). It also tends to lead to small, underpowered studies that nevertheless include overconfident practical or policy guidelines or advice. Larger studies with more representative sampling are badly needed.

For translation to policy or practice, independent replication of studies on de-institutionalization is required. “One study is no study” if the goal is to apply results in real life settings where the development of physical and mental health is at stake. Three conditions are necessary for the translation of scientific findings into policy or practice. First, empirical results should be replicated in several independent studies that meta-analytically show an overall robust effect. Second, the change in the caregiving environment should be ethically defensible, for example, in line with the UN Convention on the Rights of the Child [14] which is the universal expression of consensus on the rights of the child. Third, from a cost-effectiveness perspective, the proposed change should prove to be a good use of scarce resources compared to care-as-usual or alternative care options [9]. Of course, this epistemological and methodological model is controversial as it privileges quantitative and experimental designs in the confirmatory stage of a research program, and positions qualitative and mixed methods research in the hypotheses-generating, exploratory stage. This is a necessary methodological distinction to ensure the best evidence-based foundations for translating findings into policy or practice. This model does not undervalue the qualitative studies because they are considered to aim at discoveries as a crucial starting point for confirmatory research [9].

Our impression of the studies collected for the current review is a dearth of replicated and replicable DI investigations. The number of studies with very small samples (fewer than 20 participants) amounts to k = 40, and small studies run the risk of inflated and thus non-replicable results. For example, in one study interviews were conducted with 7 experts and 10 care-leavers of institutions recruited through convenience sampling [15]. The authors differentiated results depending on gender and age, leaving very small sub-groups, but concluded that the developmental pathways of male informants looked normal, and the experiences of female care leavers were somewhat traumatising. Another example is a study reporting on interviews with 28 adolescents in institutional care recruited through convenience sampling [16]. Their qualitative approach suggested that the participants were hardened by early adversities and had acquired resilience that facilitated long-term adjustment. Therefore, the authors conclude that institutional care would hold the potential to offer what at-risk residents need.

Many studies failed to examine the intercoder reliability of data collection or data analysis and might not be reproducible by independent researchers. This is the case not only for qualitative but also for quantitative research. Replication research is crucial to make progress in the field. Doing a replication study promotes critical understanding of previous research, increases one’s own methodological rigor, provides a test of potential false positives in earlier studies, and may reveal moderating factors that explain differences in results. Replication research enables meta-analyses, a necessary step on the way to translation.

To close the gap between scientific goals and ethical desiderata, PPI [7,9] has been promoted in the biomedical and mental health sciences. PPI covers a spectrum from patients’ or policymakers’ involvement in outlining relevant but uncharted areas of research to action research that integrates research and activism in all stages of the applied research cycle. In DI studies, PPI is rarely implemented, although the idea is gaining traction that youth in alternative care should have a stronger voice in decisions about research and policy related to alternative care. One of the rare examples of PPI in DI research is the study by Frimpong-Manso and colleagues [17] in Ghana, who adopted ‘practice research’ to co-create scientific knowledge and its translation to practice. In close cooperation with five practicing social workers, research questions were refined, interviews were developed and conducted with a convenience sample of 25 professionals from NGOs and institutions, and data were analysed together. One of the policy recommendations was to turn residential settings into respite care centers with short-term stays. Yet, five practitioners seem not sufficient, considering the complexity of ethical issues. More theory and research are needed to bridge the gap between what is (scientifically) the case in the domain of alternative care and the arguments for what should be striven for (ethical grounding), with proposed changes in policy or practice.

The third requirement for responsible translation is a defensible cost-benefit, cost-effectiveness, or cost-utility balance of various alternative care arrangements, including large and small institutional care, foster care, adoption, kinship care, or kafalah; this economic approach was mentioned in [13,18]. Of course, the effectiveness of the policy change, intervention, or treatment should be established, preferably on the basis of meta-analytic evidence and compared to other policy or intervention domains. The additional costs of an intervention should be computed compared to care-as-usual or alternative options, in the short- but also long-run. Scalability should be estimated if a policy or intervention is thought to address the needs of millions of families and children [19]. The combination of these three dimensions might lead to an educated guess about the cost-benefit balance.

In the Rogers and Karunan [18] study, the authors refer to Carter’s [20] estimates of costs involved in various alternative care arrangements. Carter concluded that community residential/small group home care would cost approximately half that of state institutional care; foster care approximately one fifth to one third, and family support/social service provision one eighth of state institutional care [20]. These computations are, however, outdated, not only because they are more than 20 years old and only based on Eastern European provisions, but also because, in health economics, more sophisticated models have been developed. Such models have been applied by Wilson-Barthes and colleagues [21,22] to the cost-utility balance among institutional care, living on the street, and being raised in alternative family-based care.

In their health economics computations on data of the OSCAR’s Health and Well-Being Project in Kenya, family-based care would be more cost-effective compared to living on the street [22], and less cost-effective compared to institutional care [21]. The health economics models, however, suffer from a lack of valid developmental input. For example, Wilson-Barthes et al. [21] used a 10-items Children’s Depression Inventory-Short Form or CDI-SF [23] to measure depression-free days and convert these to Quality-Adjusted Life Years or QALYs, a cornerstone of cost-effectiveness computations and comparisons. Yet, the CDI-SF was not validated in Kenya, and the conversion to QALYs had not been psychometrically or econometrically tested. In general, such short questionnaires for complex constructs like children’s well-being and development may be flawed by response biases that are partially rooted in genetic differences [24]. Much more work must be done to make the cost-effectiveness models work in this complicated domain of alternative care [25]. But such work will be critical to convince policymakers not only to initiate but also to complete the difficult transitions involved in DI.

Limitations. First, we were not able to conduct a standardized study quality rating, as most studies were qualitative or used a mixture of qualitative and quantitative methods. This restricts the validity of our conclusions that should be considered grounded hypotheses for further, confirmatory research [9]. Quantification of qualitative studies is problematic, as we showed in a previous methodological study [26]. Second, we did not assess actual impact outcomes of the studies in our scoping review. In our epistemological and methodological model, the actual impact on child development cannot be measured in a reliable and valid way, as responsible translation relies on replicable findings that are multiple times tested by independent research groups, across socio-economic and cultural contexts, and a variety of participant samples to examine the context-dependence of preliminary results. Third, we found some impressive (quasi-)experimental studies on DI in complex contexts, such as King and colleagues’ [11] Bucharest study and Hearst et al.’s [13] study on Zambian families with children with disabilities. We noted that these examples illustrate the feasibility of DI research under extremely complicated conditions and deserve to be replicated in other contexts, regions, or countries. The current research domain, however, is not yet ready for responsible translation to policy or practice. More confirmatory work must be done to avoid premature and potentially iatrogenic collateral damage to the vulnerable children without available parents. Practitioners, professionals and policymakers must take care arrangement decisions and implement policies and treatments based on their wisdom, insights and experiences and on some general guidelines for ‘good-enough’ child care (Triple S, see below) instead of relying on preliminary hypotheses still to be examined in confirmatory research (see [27] for our Cooperative Practitioners-Researchers model for translation of scientific input and practical expertise into effective treatment).

We started our paper with a description of the concept of DI based on definitions of institutional versus family-based care as provided by iCARE [28]. This seems a simple issue, and consensus between stakeholders appears almost self-evident. But in this sector, nothing seems more complicated than reaching such a consensus [29], and the complex iCARE treatise confirms that it does more than bridge divergent views and positions. Here, we stipulate that the DI concept semantically consists of three components, the target population, the institutional arrangement, and the DI destination, with minimally acceptable alternative care settings for the target population. We propose to settle on (contestable but pragmatic) stipulations or working definitions of these three components as follows: children and adolescents are defined as any human beings below age 18 years; institutions are stipulated to be 24/7 professional or non-kin group care regulating day and night routines regardless of group size, duration or quality of the institutions; de-institutionalization is the process of transitioning away from institutions toward alternative family-based care arrangements which can be foster or kinship care, adoption or kafalah, or reunification with the family of origin. Living in large or small institutions, boarding schools, or residential madrasahs, or living on the street are excluded from this type of alternative (family-based) settings.

More than 50 years of attachment and emotion regulation research on child and adolescent development has established a firm evidence base for three minimal requirements for ‘good-enough’ care arrangements. Children and their caregivers need Safe, Stable, and Shared care (Triple S) [9,30]. Safe arrangements imply care without physical or psychological abuse or neglect (but not necessarily secure attachments); stable arrangements provide continuous care and availability of a small core set of caregivers, avoiding as much as possible break-ups of caregiver—child relationships; and shared care implies the availability of a network of support figures for parents and children to fall back upon in times of need. A crucial question for DI is what kinds of alternative care arrangements can satisfy these three requirements, backed by replicated evidence, ethical grounding, and a positive cost-effectiveness balance. Instead of inflow, stock, and outflow numbers of institutionalized children to monitor DI progress, far more crucial and helpful will be quantitative and qualitative DI indicators that periodically assess maltreatment rates (safety), number of alternative care break-ups (stability), and size of the supportive social network around both child and family (shared care) [31].

Against the background of the discussed gaps in DI studies and the challenges of improving research quality, we will now turn to the next steps in developing and prioritizing DI research projects.

First, the transparency and replicability of DI studies can be improved with several measures that have so far been rarely used in this sector. Pre-registration should always be considered. In the biomedical and psychological sciences, pre-registration of study design and analysis prior to data-collection has been shown to be protective against inflated and non-replicable results [32]. Various steps in data-collection, coding, and analysis should be performed by a research team instead of a single researcher to facilitate establishing intercoder agreement and a transparent process of data-analysis. Such research teams should be diverse in terms of cultural background. Reports of the study should be transparent about the assumptions behind the design and about the limitations of the findings for theory testing and practical application. Financial conflicts of interest should be avoided or made explicit in publications. Investigators should be independent from stakeholders financing (parts of) the study when designing, conducting, and reporting their research: academic freedom is paramount [33].

Second, for DI studies in Low and Middle Income Countries (LMICs), local students and researchers should be enabled to (co-)author publications about the results or to profit otherwise from a DI project through research training or education in line with the Cape Town Statement on Fostering Research Integrity Through the Promotion of Fairness, Equity, and Diversity [34]. The Cape Town Statement emphasizes open science that guarantees full free access of researchers and professionals in LMICs to the scientific literature. The declaration also stresses the privileged access to their own data for some years to prevent non-LMIC scientists from profiting from faster access with more resources than their peers in LMICs. Collecting raw data in LMICs and then exporting them to academia in so-called WEIRD (Western, Educated, Industrialized, Rich, and Democratic [35] countries without local students, professionals, and researchers benefiting from the work should not be allowed.

Third, ‘Patient and Public Involvement’ (PPI) is increasingly considered important and helpful to make research more responsive to the needs and priorities of participants, patients, and the public. PPI may improve the quality and relevance of research by making sure that the research questions lead to actionable answers. Involving participants with lived experiences, workers in institutions, kin, foster, and adoptive parents in the research process allows those who are the subjects of the translational studies on de-institutionalization to have a voice in shaping the research agenda and addressing translational issues [36]. Individuals with lived experience, policymakers, and other stakeholders should be invited to be involved in defining the aim of a study and in the interpretation, ethical evaluation, and potential translation of the results to policy or practice. Other stages of the research cycle, which include creating testable hypotheses, connecting them to valid methods of data-collection and analysis, and the replicable report of the study, require independent research expertise [9].

Fourth, for evidence to have a positive impact on children’s lives, it is important that high-quality evidence, as outlined above, is taken up and used by governmental and civil society policymakers and practitioners, the media, and children and families with lived experience of care. Planning studies to produce such evidence should include a plan for ways to ensure that the evidence is available to those who can and will use it to make a difference, being convinced of the cost-effectiveness of the proposed changes. This could be managers changing their policies or program methods (including scaling up or closing); government agencies improving their culture of evidence use, so that there are discussions of evidence or promotion of studies to fill gaps that have been noticed; and collaborative development of global evidence-based guidelines for good practice. Our suggestions for DI studies creating more evidence for impact on governments, policymakers, sponsors, and practitioners are summarized in Table 1, presenting helpful and unhelpful strategies to achieve such evidence.

In their paper on ‘Global priority for the care of orphans and other vulnerable children: transcending problem definition challenges’, Shawar and Shiffman [29] conclude: “In order to potentially become a more potent force for advancing global priority, children’s care proponents within international organizations, donor agencies, and non-governmental agencies working across countries will need to better manage their disagreements around de-institutionalization as a care reform strategy”. The alternative care reform sector is a complex puzzle with numerous separate pieces that must be integrated into a picture of the whole landscape and the ways to facilitate alternative family-based care for vulnerable children. When they succeed in joining forces, creating a basis for reliable, transparent, and replicable DI research, their efforts to make DI reform sustainable and successful in improving the lives of (alternative) parents and their vulnerable children will not be in vain.

The following supporting information can be found at https://www.sciepublish.com/article/pii/1176, Table S1: Coding system.

The current report has been developed as part of The Evidence for Impact Working Group (E4I) and has been guided by its Terms of Reference focussing on “Strengthening the common understanding and positioning on evidence” described as “independent and objective” evidence in this sector. Better Care Network provided supported Ella Asnin for her contributions to the current paper. The content of this paper is the sole responsibility of the authors.

M.J.B.-K.: Conceptualization, Methodology, Investigation, Validation, Writing—Original Draft, Writing—Review & Editing, Visualization, Supervision; E.A.: Investigation, Writing—Original Draft, Writing—Review & Editing, Visualization; M.H.v.I.: Conceptualization, Methodology, Investigation, Validation, Writing—Original Draft, Writing—Review & Editing Supervision, Project Administration.

Not applicable.

Not applicable.

Not applicable.

This research was funded by Better Care Network grant number 00268, September 27, 2024

This paper is based on a report developed as part of The Evidence for Impact Working Group (E4I) of the Transforming Children’s Care Collaborative (https://www.transformcare4children.org/ accessed on 2 May 2024). The report has been guided by the Working Groups’ Terms of Reference focussing on “Strengthening the common understanding and positioning on evidence” described as “independent and objective” evidence in this sector. Better Care Network supported EA for her contributions to the current paper. The content of this paper is the sole responsibility of the authors.