Why don't you provide information on costs, effect sizes & benefits?
As a What Works Centre we are committed to assessing the effectiveness of programmes and approaches to Early Intervention and also to developing clear accounting of costs and benefits.
This information is in its infancy because so few evaluations have been conducted in the UK and so few have been long term. There are a number of other issues:
- There is no single value metric on which all benefits can be measured. This is partly because the outcomes of Early Intervention programmes are multiple and diverse, and partly because many of these outcomes do not have an obvious monetisable or marketable value, such as attachment or mental health. In future work, we will assess the extent to which well-being provides a single currency for assessment of benefits, to place alongside other estimates (such as social return on investment, fiscal savings, and so on.)
- Individual study effect sizes are not easily comparable, in part because of the diversity and multiplicity of outcomes and in part because of the diversity of methods used. This complicates the ostensibly simpler exercise of cost-effectiveness analysis. Nonetheless, as we build up the database behind the Guidebook we will increasingly include comparative data.
- Since we are assessing programmes for a commissioning audience, much of the information about costs is commercially sensitive and subject to negotiation between commissioners and providers. Programme delivery costs will also include important locally variable elements (such as salaries or rent).
These issues are not insurmountable, but require additional work before the information can be included in the Library entries. In this first version of the Guidebook, we have assessed programmes in terms of the strength of the evidence that they have been effective. In future iterations we will add information also about effect sizes, costs and benefits.
Do you validate programmes?
No. We do not validate programmes.
Currently, the programmes included in our Library have been assessed by other clearing houses. The current ratings given to the programmes in our Library instead represent an objective assessment of the strength of their evidence, based on the judgement of other authoritative organisations against broadly similar standards of evidence.
We do not view these assessments as a validation, accreditation or recommendation of any given programme. There are intense debates about what constitutes strong or weak evidence. While informative, any assessment of a programme’s effectiveness needs careful interpretation, since it is never possible to know for certain if an intervention will be effective at a specific time or place. So while we provide an assessment about the strength of a programme’s evaluation evidence, this information should not be interpreted as a kite mark or endorsement of a specific programme. And it should never be used as a replacement ongoing monitoring and evaluation as programmes are being implemented.
Do you only look at programmes?
The Programmes Library only provides information about specific programmes. However, in other parts of the Guidebook we include systems tools and other information about effective systems and practices that we have identified through our ‘What Works’ reviews.
How did you choose the nine EIF outcomes?
In our first assessment of the evidence for the first iteration of our EI Guidebook, we have classified programmes in a new way, in terms of nine domains of outcome that represent a broad range of pathways by which early intervention programmes can provide social value by:
- impacting on the family and home environment
- impacting directly on early child development
- impacting on the journey through childhood to adulthood in terms of measurable outcomes such as criminality, violence and abuse, neglect of children, poor attachment to the labour market or ill health.
Domain 1 describes the immediate benefit of programmes that improve parenting, family interactions and the home environment. Domain 2 describes the immediate benefit of programmes that support early child development, for example social and emotional skills. Domain 3, breaks down into seven further outcome categories that are beneficial to children and society, all of which can be understood in negative and positive terms. These nine outcome domains have been chosen to reflect the very broad range of ways in which Early Intervention can make a positive difference to children, families and communities.
These nine outcome domains also map onto the ways in which the policy and practice environment interacts with children and families, cutting across issues of mental health and wellbeing (health sector activity), education and skills (schools and colleges), crime and anti-social behaviour (policing and justice) and so on. In this way the Guidebook provides a unique resource that maps the logic and impact of Early Intervention.
Are you just another clearing house?
No. Programme assessment is just one part of our mission. We also provide valuable advice on how to commission, develop and implement effective services for children and their families and we work with our Pioneering EI Places to improve the evidence base. Much of this information can be found on the Implementation side of the Guidebook and we intend to add to this section substantially over the coming months.
The Early Intervention Foundation also provides important advice to our 20 Pioneering Places, who we work with directly to determine:
- How to target services and reach families
- How to work out what help families need
- How to measure if programmes or interventions are working
- Early intervention’s impact on mainstream services and the Voluntary and Communities Sector operate.
Through our work with our Pioneering Places, we also anticipate that we will learn important lessons that will translate into advice that can be offered more widely to other communities, academics and policy makers. More information about our work with Pioneering Places can be accessed here.
The information that we gain through programme assessment and our work with our Pioneering Places will also inform and improve our ability to advocate for Early Intervention on the national level.
Can you help us evaluate our programme?
We are not currently resourced to offer bespoke evaluation advice across the sector. At the moment this advice is necessarily limited to our Pioneering Places. Over the coming months, we will add a substantial amount of information on how to evaluate and develop early intervention programmes.
Should I only invest in interventions with established (Level 4) evidence?
There are several advantages in implementing interventions with established evidence. First, interventions with this level of evidence will have evidence showing consistent benefits in multiple places with multiple populations. But this is no guarantee that an intervention will work again in a new setting or authority, particularly if it is not delivered well. Established evidence does, however, increase the likelihood that it will work.
Interventions with established evidence also tend to be more developed, meaning that the original providers have worked through various issues that could hamper effective implementation. In particular, interventions with established evidence are more likely to be able to provide assistance to Places in setting up and ‘installing’ the intervention to maximise its effectiveness. This assistance often includes guidance on how to recruit and supervise the best practitioners for delivering the interventions, methods for targeting and referring children and families to the interventions and methods for monitoring the intervention’s effectiveness as it is being installed. Interventions with established evidence are also more likely to providing better training materials and staff development support.
However, commissioners should never invest in an intervention on the basis of its evidence only.
The decision to invest in an intervention should be informed by many factors, including the extent to which it:
- Is actually needed (i.e. does population-level data suggest that children and families within the community would specifically benefit from it?)
- Fills a priority gap in provision, and does not overlap with other existing services that are potentially effective (in which case, it may be better to evaluate and monitor these services)
- Is feasible to implement in your local area taking into account any constraints and challenges, e.g. budgetary constraints, workforce skills, geography, political priorities
- Can be commissioned, implemented, and monitored robustly as part of a clear commissioning and delivery plan
- Compares favourably to other options that may be available on all of the above considerations, when pros and cons of different possible solutions to your priority outcomes and population needs are weighed up
- Can be implemented in a cost effective way.
Please see the Implementation side of the Guidebook to learn more about effective commissioning practices.
How do I get my programme in the Guidebook?
The current version of the Guidebook includes a small number of programmes which exemplify the range of outcomes through which Early Intervention can provide benefits for children of different ages. The Guidebook will continue to grow through the inclusion of other programmes. The Guidebook will expand as we undertake reviews on specific topics of interest and we include some of the best-rated programmes from these reviews in the Programme Library.
How often will you be updating the Programmes Library?
We will be updating the Programmes Library over the course of the next year by adding some of the programmes which we have rated in our “Foundations for Life: What Works to Support Parent Child Interaction in the Early Years” review. We also intend to update the Programmes Library based on the outputs of future reviews.
What are the EIF standards of evidence based on?
Interventions vary greatly in terms of the amount and quality of their evaluation evidence. A number of internationally recognised organisations (e.g. WHO, NICE, NESTA) have therefore developed broadly similar ‘standards of evidence’ to assess whether an intervention does or does not work.
In this first version of the Guidebook, we have classified programmes in terms of the strength of the evidence that they have been effective, based on the assessments of other authoritative clearinghouses. The notion of strength of evidence combines issues of the quality of evaluation with the general issue of whether positive effects have been found in the evaluation evidence.
For the purposes of the Guidebook, we currently assess programmes on a scale of six gradings, starting from a rating of ‘negative’ and then numerically running from 0 to 4. This strength of evidence scale is broadly based on the NESTA evidence standards with the addition of 0 being assigned to interventions that are not based on any specified theory evaluation evidence and the negative rating assigned to programmes for which there is strong and consistent evidence that the approach is harmful, or provides no observable benefits to children or families. Programmes receiving a 4 have been assessed to be effective through multiple randomised controlled trials (RCTs). More information about EIF’s Standards of Evidence can be viewed here. More information about the relative merits of RCTs can be viewed here.
The EIF Standards of Evidence enable us to assess the quality of evidence of effectiveness behind any planned Early Intervention approach. In this respect, they can be applied to any form of programme, including specific interventions at the local level or broad, national programmes of activity. The EIF Standards can also be applied to all forms of public, private and social expenditure.
We are currently moving towards communicating our standards of evidence in a different manner to better reflect the nature of the, though the criteria by which we assess programmes has not changed. This new communication is used in our “Foundations for Life: What Works to Support Parent Child Interaction in the Early Years” report. The way we communicate the evidence ratings expressed in the Guidebook may therefore change in the future.
Why are there only 50 programmes and how did you choose them?
There are literally thousands of early interventions offered across the UK. While some of these have established evidence of effectiveness and are offered in many communities, the majority are in the formative stages of their development and are only available in one or two places.
Identifying and confirming the details of all of these programmes within EIF’s first year would have been impossible. But we had to start somewhere. So we consulted 15 authoritative clearing houses to identify 50 interventions that represented a range of children’s ages, outcomes and levels of evidence for commissioners to consider. More information about the clearing houses we consulted can be accessed here.
We felt that it was important to provide commissioners with a range of possibilities, but also wanted to make sure that the programmes we included were feasible to deliver within the UK. This meant that the First 50 programmes in our Library needed to also have evidence of having been implemented within the UK.
It is important that commissioners understand that exclusion from the Guidebook does not mean a programme does not work. The First 50 were chosen from a wider database which is still relatively small at 1500 programmes.
Over the coming year we will be adding further programmes to the Programme Library as we review them. Programmes whose evidence was rated at levels 3 and 4 by EIF in our “Foundations for Life: What Works to Support Parent Child Interaction in the Early Years” review will be added to the Programme Library during this time.