Published on January 22, 2014
Improving Futures Evaluation and Learning Evidence Briefing Presentation Laurie Day and James Ronicle, Ecorys UK 21st January 2014
Part 1: Evaluation Overview & Methodology Laurie Day, Evaluation Director
Improving Futures Key Indicators • Launched by the Big Lottery Fund in March 2011 • Total £26 million grant funding distributed between 26 pilot projects across the UK • Aiming to test different approaches to improve outcomes for children in families with multiple and complex needs • Requirements: – led by VCOs partnership with statutory services – include mechanisms to engage 'hardest to reach' families; and. – oldest child 5-10 years at entry stage
Aims of the Evaluation Measuring Impact Evaluation aims 1. To measure the effectiveness, impact and outcomes of the 26 projects; and the programme as a whole 2. To support the projects with measuring progress 3. To capture and share learning across the programme, and to disseminate to policymakers & practitioners across the UK Scope • Consortium led by Ecorys, with Ipsos MORI, Family Lives and University of Nottingham • Five year evaluation (Oct 2011 - Dec 2016) • Evaluation and learning
Evaluation Framework Programme Effectiveness Programme Impacts Programme-level Effectiveness of programme design, development and implementation Net improvements to children‟s life chances Net risk reduction for families with complex needs Net attributable savings Project-level Effectiveness of governance; partnerships; strategy and planning; user involvement and delivery Quality of provision Sustainability Strategic Benefits Strategic influence over UK policy Leverage over external resources Synergy with other programmes Knowledge transfer Replication of effective practice Programme Outcomes Children Adults Families Health and wellbeing Parenting skills and confidence Emotional and behavioural development Health and wellbeing Family functioning and relationships Educational outcomes Safeguarding and social care Crime and ASB Educational outcomes Safeguarding and social care Crime and ASB Financial wellbeing and security Secure and safe environment Social networks and belonging
Methodology Measuring Impact Programme-Level Evaluation • Literature review • Project-level evaluation Monitoring data collection (IFMIS): secure online system (practitioner-led) • Co-production of local evaluation plans: support with identifying project outcome measures & data sources • Bi-annual monitoring against local grant milestones, and qualitative feedback on lessons learned • Longitudinal survey of families: baseline; +12 mths, and +24 mths • Stakeholder telephone survey: key local stakeholders (2013 & 15) • Support for self-evaluation: advice on methods, tools and training • Case study research: mix of longitudinal (n=6) & snapshot (n=14) • • Family Advisory Panel Enhanced evaluation activities: CBA case studies; impact case studies; participatory research case studies
IFMIS: Overview Measuring Impact Key considerations • Need to gather comparable data: measuring changes to families‟ status over time • 26 local projects, with different local data systems Format Secure online database Data entry by project workers, based upon data recorded within service plan (or equivalent) Three mandatory data entry points: entry, exit, + 6mths post-exit (with optional interim measure)
Key Indicators • Child, adult and family indicators, based on a review of the research literature, plus socio-demographic data • Problems and strengths; hard and softer measures • Three indicator types : dynamic (subject to change), eventbased (time-specific), and status (interpretive only) • Record type and amount (hours) of support received
Measuring Impact Creating a family record – example
Measuring Impact example Measuring Impact Adding and updating indicators – Impact
Measuring Impact and Outcomes Measuring Impact 1. Desk research to develop key indicators (risk factors & strengths) 2. Embed within monitoring framework (IFMIS) 3. Measure short-term change over time (monitoring data) 4. Measure longer-term outcomes, up to +2yrs (survey) 5. Test predictive „strength‟ of indicators (compare survey & MI) 6. Scale-up to estimate programme-wide effects
Counterfactual Measuring Impact • Need to establish „additionality‟ of the programme • Counterfactual analysis: explore alternative scenario (no intervention). Two approaches: 1. Self-reporting of impacts: all projects (exit questionnaire and via Ipsos MORI survey) 2. Quasi-experimental design: 4-5 impact case study projects: compare outcomes between participants and matched comparison group (“difference-in-differences”)
Evaluation Activities Completed to Date 1. IFMIS system live and fully embedded 2. Bespoke evaluation plans in place for all 26 projects 3. Baseline survey of families underway 4. First local stakeholder survey underway 5. Six „trailblazer‟ case study visits completed (England, Wales, and Scotland) 6. Participatory action research training delivered
Learning and dissemination strand Measuring Impact Policy Makers, Practitioners, Wider Stakeholders Policy Briefings Good Practice Publications Project Exchanges Policy Workshops Webinars Improving Shared Learning Environment Futures Wider Stakeholders Policy Makers, Practitioners, Projects Face to Face Learning Events Electronic Newsletters Press Releases
Learning activities completed to date Measuring Impact Internal 1. 2. 3. 4. National launch event for the programme Secure online portal for projects Project networking events and webinars Newsletter External 1. Launch of evaluation web-site: http://www.improvingfutures.org/ 2. Social media activities 3. Policy round table session
Part 2: Key Findings from Year 1 of the Evaluation James Ronicle, Evaluation Project Manager
Introduction - Progress in Establishing Projects - Supporting Families - Who are the Improving Futures Families? - Early Outcomes for Families - Conclusions and Next Steps - Implications and Lessons Learnt for Future BIG Programmes
Progress in Establishing Projects • Focus to date has been on building local partnerships and delivery teams. • Projects still in early stages of delivery & models are evolving • Number of families supported by projects are lower than anticipated. However, referral numbers picked-up considerably during 2013. • Projects face more challenging financial climate than when launched. • Projects faced barriers to partnership working due to heightened sense of competition & widespread restructuring • Despite this, projects building strong local partnerships: - Engaging public services - Collaborations between VCSs - Engaging „micro‟ VCSs • Consequently, projects sharing best practice & early signs becoming embedded within local structures for children & families.
Supporting Families: Supporting Families: Examples of Improving Futures Projects Examples of Projects The Neighbourhood Alliance, Sunderland • • Led by Foundation of Light, Sunderland Football Club Developed „Neighbourhood Menu‟ of 45 „micro- VCSs‟ to support families Family Pathways, Lewisham • • • Focuses on children demonstrating destructive or anti-social behaviour Runs „Roots of Empathy‟; classroom-based programme where children spend time with babies to strengthen emotional competence Delivers training for practitioners on promoting attachment between parents & children BIG Manchester Tyne Gateway, N & S Tyneside • • • Created co-located team to tackle „toxic trio‟ of substance misuse, mental health & domestic violence Provides three areas of support: • Holistic family support through Key Worker • Peer and community engagement • Spot purchasing of specialist support Recruits Family Entrepreneurs as „barefoot professionals‟: • Outreach • Support in assessing family needs • Support in delivering services • Direct support to families
Supporting Families: Identification and Engagement • Primary schools have provided a hub for many projects (59% of families engaged through schools) • Challenges around reaching out to ‘harder to engage’ families who do not traditionally participate • Previous negative experiences of contact with services having an impact on willingness to engage • Some challenges around 5-10 age criteria • Greater number of families with complex needs than anticipated, in some projects
Supporting Families: Packages of Support • Replicating many aspects of good practice : • • • „Key Worker‟ model Holistic support packages Evidence-based parenting programmes: • Incredible Years, Triple P, Roots of Empathy • Innovative finance mechanisms: • • Family budgets Spot purchasing • Peer support • However, not all projects providing ‘holistic’ support • Key success factors: • • • „Out of hours‟ provision Frequent Key Worker contact Hands-on approach with referrals to other services
Who are the Improving Futures Families? Measuring Impact Demographics • Free School Meals significantly above national averages (77%) • Minority ethnic background – above UK averages (at 31%). Variation from previous family programmes (FIPs: 88% white, and FPs: 77% White-British) • High proportion of lone parent families – approaching two thirds, at entry stage (62%). • Predominately female adult profile (75%)
Who Are the Improving Futures Families? Family Risk Factors Measuring Impact • ‘Upper-middle’ in continuum of need. More prevalent than general population; less prevalent than other intensive intervention programmes. Figure 1: Family risk factors affecting 10% or more of the cohort Base: 891 families
Who Are the Improving Futures Families? Measuring Impact Adult indicators Risks (-) Strengths (+) • • Parenting difficulties the most prevalent issue: parenting anxiety or frustration (56%), problems with discipline / boundaries (49%) Over half of adults within the cohort (51%) demonstrating awareness of how to keep children safe from harm, and around one third setting appropriate boundaries (34%) • Well over one third with suspected or reported stress or anxiety (41% - twice the population average) • Far lower prevalence of adult ASB or crime indicators than some other programmes (2.1% ASB, compared 79% for FIPs). Fair levels of face-to-face contact with school staff and positive relationships, ranging from under half for all adults (45%) to just over half for lone parents (52%) •
Who Are the Improving Futures Families? Measuring Impact Child indicators Risks (-) Strengths (+) • Child behavioural problems by far • the most prevalent risk, when lowlevel (30%) and persistent disruptive behaviour (16%) are combined • One in five children with suspected or reported stress or anxiety (22%); mirroring similar problems in adults within the cohort, and one in ten under-achieving at school • Lower propensity for project workers to report risk factors for children; could indicate recording difficulties • Most children adopt some aspects of a healthy lifestyle: well over half (61.5%) attend regular medical appointments; a fair proportion participate in play (37%), and exercise or sport activities (27%) Over one third of children have supportive peer friendships at school (43%), whilst just over one quarter have regular contact with friends outside school (27%).
Who Are the Improving Futures Families? Measuring Impact Towards some typologies… • Factor analysis of IFMIS data (in SPSS) – clusters of variables that are strongly correlated with each other but relatively uncorrelated with variables in other clusters • Iterative approach to find the most robust and realistic groupings • Eight typologies identified on this basis...
Who Are the Improving Futures Families? Measuring Impact Towards some typologies (2) • Typology 1 - Families with strong structures and behaviours • Typology 2 – Low skilled families with financial and housing difficulties • Typology 3 – Families with significant discipline and behavioural issues • Typology 4 – Transient families with domestic abuse, mental health and substance misuse • Typology 5 – Families engaged in their community • Typology 6 – Families known to the police for minor disorder • Typology 7 – Socially excluded families with health problems • Typology 8 – Families known to the police for more serious incidents
Early Outcomes for Families 1. Projects able to achieve „quick wins‟ for families: 1. Resolving financial problems 2. Improving housing circumstances 2. Parents reporting improved well-being: 1. Confidence & self-esteem 2. Reduced Anxiety 3. Parenting capacity 3. Improved behaviour and attendance at school
Conclusions and Next Steps Evaluation Conclusions •Improving Futures is becoming integrated into wider local support infrastructure for children and families •The programme is strengthening links between statutory and VCS organisations •Projects showcased diverse range of practices for working with families at different levels of need. Some are replicating good practice, others developing good practice •Too early to offer robust conclusions about impact of programme, but positive early signs that projects making different to families‟ lives. Areas for Development •Reaching families not in contact with mainstream services •Taking stock of ‘whole family’ approaches
Implications / Lessons Learned for Future Programmes Evaluation design • Need to strike a balance between a) freedom for grant-holders to innovate, and b) setting parameters for measuring impact Programme design • Value of statutory (DCS) endorsement of VCS bids • Development phase critical for building multi-agency partnerships (…but make next stages conditional on early success?) Key areas of added value / potential legacy… • • • • Understanding dynamics of change for families over time What works with the 5-10 age group, and VCS-school links New forms of service commissioning (esp. neighbourhood level) Models of engagement and support for BME families
Canvas Prints at Affordable Prices make you smile.Visit http://www.shopcanvasprint...
30 Días en Bici en Gijón organiza un recorrido por los comercios históricos de la ...
Con el fin de conocer mejor el rol que juega internet en el proceso de compra en E...
With three established projects across the country and seven more in the pipeline,...
Retailing is not a rocket science, neither it's walk-in-the-park. In this presenta...
Briefing Paper 125 Building capabilities in the voluntary sector: A summary of what the evidence tells us Rob Macmillan and Angela Ellis Paine, with Helen ...
The Big Lottery Fund awards grants to UK organisations to help improve their ... Big Stories. Case studies and stories about some of the projects we have ...
The Big Lottery Fund. Find out more about the Big Lottery Fund. Read about us. Our people. Read about our Board and National Committee members. Meet our ...
... of the Big Lottery Fund be handed over to Big ... Big Lottery Fund’s outreach and evidence ... briefing service is funded by The Big Lottery ...
Big Lottery Fund Head of Knowledge Candidate Briefing Pack ... all, ensuring that new knowledge is relevant and builds on evidence we already have.
... the Big Lottery Fund has been delivering two ... • The data helped to evidence the progress of individual projects ... briefing paper word ...
The Big Lottery Fund. Find out more about the Big Lottery Fund. Read about us. Our people. Read about our Board and National Committee members. Meet our ...
Communications Director Candidate Briefing Pack August 2015 . I am delighted that you are interested in joining the Big Lottery Fund. ... evidence based ...
Ensuring local VCS groups don’t lose ... to the Big Lottery Fundi. ... Equity and Evidence of Need’ 5. Lottery money should not be ...