SROI Methodology and Report 2024 Costello Medical

SROI Model Methodology

11 APRIL 2024

Costello Medical

UK | US | Singapore | China

www.costellomedical.com

Social return on investment (SROI)

An SROI analysis is an outcomes-based framework that enables organisations to understand, quantify and monetise the social, environmental and economic value created through their services. To assign a financial value to social and environmental outcomes that lack a well-defined market value, financial proxies, derived from published literature, are used. The final output of an SROI analysis is an SROI ratio, indicating the value of an organisation’s services (expressed in cost units, £) delivered per £1 investment.

There are seven principles behind the SROI analysis method, as set out by the SROI Network.1 These are described in Table 1

Table 1. Seven principles of the SROI method1

Principle Description

Involvement of stakeholders

Understand what changes

Value the things that matter

Only include what is material

Do not over-claim

Be transparent

Verify results

The consultation of well-defined stakeholder groups throughout the analysis ensures that the value of the organisation’s outcomes, and the way in which they are measured, are informed by those directly affected.

Robust analyses include an evaluation of how both positive and negative change may be created by an organisation, including those which are unintentional.

Financial proxies are used to assign a value to outcomes generated by the organisation, for outcomes that do not have a well-defined market value or financial worth.

Critical selection of the inputs and outcomes to be included in the model is crucial to give a true and fair representation of the value generated by an organisation.

A correct evaluation of outcomes involves recognition of the other factors which may contribute to the outcome, beyond the organisation’s services.

An honest account should be provided of when the SROI analysis may be considered accurate. The methodology and results of the analysis should be reported to and discussed with stakeholders.

Independent validation by appropriate parties may help to remove subjectivity.

Abbreviations: SROI: social return on investment.

Calculating SROI

The SROI ratio is calculated as a ratio of the net benefits generated through an organisation’s activities and the net investment required to run these activities, and represents the value of their services over a defined period of time. The simple equation applied to calculate the SROI is as follows:

Total

SROI =

Total

Any SROI ratio with a value greater than one indicates that more value is being generated than invested. However, there is currently no agreement for what constitutes a “good” SROI ratio, as methodological differences prevent robust comparison of SROI analyses conducted by different parties. In particular, the SROI ratio estimates yielded by these analyses may vary based on the type of outcomes being included and excluded, the considered time horizon, and the level of conservatism applied while making subjective assumptions.

Objectives

The objective of this SROI analysis was to identify the value generated by Noise Solution services and activities. This report reflects the methodology of the updated SROI analyses, which builds on the original SROI analysis carried out in 2019 by incorporating an updated impact map and alternative inputs for quantifying outcomes. Details of the original SROI model can be found in a separate report. It is hoped that the identified evidence can be used, in conjunction with qualitative evaluations, to support future funding of Noise Solution.

The SROI ratio produced indicates the value generated by Noise Solution per £1 of external investment. The focus of this ratio is placed on social value, as opposed to purely financial. In addition, whilst the conversion of social value to monetary units is analytically useful and helps to convey social value in a widely accepted and tangible format, it was not possible to monetise all the anticipated social outcomes of Noise Solution’s work. Furthermore, whilst the most appropriate model inputs were used wherever possible, some assumptions were required due to limitations in the available data.

The SROI ratio presented in this report therefore provides a conservative and provisional estimate, rather than a comprehensive outlook, of Noise Solution’s impact. The ratio is designed to complementan evaluative assessment of the social value generated by Noise Solution, as well as to identify gaps in Noise Solution’s internal evaluative data.

This report describes the methodology of the updated SROI analysis conducted and details all financial proxies used to monetise the socioeconomic outcomes produced by Noise Solution’s work. The seven principles underpinning the SROI framework, described above, have been adhered to throughout the analysis and this report:1

 Involvement of stakeholders: The impact map of outcomes generated by Noise Solution were primarily informed by discussions with participants, immediate family/guardians and mentors in the original analysis, with support from other stakeholders within Noise Solution. Where possible, data used in the model were informed by responses from users of Noise Solution services

 Understand what changes: Key outcomes captured by the model were determined, where possible, by looking at relevant changes in users of Noise Solution services. Expert opinions from Noise Solution stakeholders were also used to inform the Noise Solution outcomes

 Value the things that matter: Informed by published financial proxies, the SROI model considers not only the benefits and costs to Noise Solution service users, but also to the wider population through governmental-related costs

 Only include what is material: It is crucial to accurately establish impact in SROI analyses, by estimating how much of an outcome would have happened anyway and what proportion of an outcome experienced by a stakeholder can be isolated as stemming from Noise Solution activities.1 Model results were therefore adjusted for deadweight, attribution and temporal drop-off. Please see the section on Adjustment for further details on these three adjustment factors

 Do not over-claim: Where estimates and assumptions were made as part of this SROI analysis, these were conservative and clearly stated

 Be transparent: All sources used to inform both outcomes and financial proxies were recorded and are detailed in this report. All decisions (and the accompanying rationale) likely to affect the outcome of the analysis are also detailed in the SROI model

 Verify results: The SROI framework is subjective, and as a result multiple sources were used to justify the inclusion and exclusion of all outcomes and costs. Where possible, discussions with Noise Solution were held to verify outcomes and their association with Noise Solution services

Scope and stakeholders

Time Horizon

The SROI focuses on participant outcomes that are generated within two years of completing the programme. Longer-term outcomes that occur beyond two years of interacting with Noise Solution are not explored in the analysis. This approach is rationalised by the fact that many model outcomes depend on participation in Noise Solution services (e.g. reduced behavioural problems, increased social capital). Complex interactions between internal and external factors mean that estimates of impact made over longer time horizons are unlikely to be accurate. In addition, a single time horizon across the model facilitates interpretation, compared with a model using varying time horizons across inputs. A duration of two years therefore balanced these requirements to maximise both accuracy and interpretability of estimates.

It should be noted that some outputs are estimated to deliver a different impact in the first and subsequent years after Noise Solution participation. For example, the impact of Noise Solution on productivity loss of guardians includes a partial impact in the year of Noise Solution and a subsequent full impact the following year, due to the time allocated by guardians to attend sessions with participants. The total impact is therefore quantified as an average of the two years.

The model also includes the functionality to account for ‘drop-off’ in value from Noise Solution services throughout the year (see section on Adjustment). For example, it would be reasonable to assume that the impact of the programme on a participant would be lower after 12 months of completion than after one month. This functionality allows for more granular and conservative estimates of the social value produced by Noise Solution.

Stakeholders

Several stakeholder groups are considered in the SROI model. Discussions with Simon Glenister, Director of Noise Solution, and Damien Ribbans, Operation Director, assisted in directing the scope of the analyses. Stakeholder groups include:

 Programme participants

 Immediate family members (e.g. guardians and siblings)

 Other related services: private and government-funded health/social/educational services (including employees)

Musicians tutoring Noise Solution participants and Noise Solution were also considered in the impact mapping stage (see Impact map), however it was decided that they were not relevant for inclusion in the SROI model. This was due to a lack of outcome materiality, with some outcomes relating to these stakeholder groups being difficult to quantify/monetise and other outcomes difficult to accurately attribute to Noise Solution. For example, the supporting of local community and economy by hiring musician tutors. It was therefore decided that their inclusion may over claim Noise Solution’s benefits.

In the original SROI analyses, interviews with one person from each of the key stakeholder groups were conducted. The interviews aimed to identify and confirm the key outcomes of interest for the SROI model, as well as identify outcomes to consider for future data collection and any relevant stakeholders who were not originally considered in the model. In the updated analyses, the original research was complemented with qualitative analyses of feedback received from Noise Solution participants, conducted by Noise Solution employees.

Population Inputs

Noise Solution participants were defined as individuals who had completed the programme. Data on 374 previous participants of Noise Solution were obtained and used in the model to inform age demographic inputs, including median ages and the proportion of participants under 18, and wellbeing. During the Noise Solution programme, questionnaires reflecting the Short Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS) are completed by participants to measure their wellbeing. A significant improvement in wellbeing is defined by an increase of at least one in the SWEMWBS score, as per Noise Solution’s suggestion.

Population inputs can continue to be updated by Noise Solution to increase representation of the participant population.

Investments

The investment per participant for Noise Solution has been calculated as the cost of participating in the 10-week programme (£2,075). This was deemed an appropriate valuation of the investment, as Noise Solution is costed on a cost-recovery basis, and therefore covers all necessary investment costs incurred.

Activities and outcomes

Impact map

An updated impact map for the programme was developed collaboratively by Noise Solution and Costello Medical. The impact map details the relationship between the resources (inputs) that are used by Noise Solution to deliver their activities, and the direct and tangible results of these activities (outputs). Outcomes are the consequences of the activities on relevant stakeholder groups. Relevant outcomes are listed on the impact map, separated into categories based on the timepoint at which they are expected to be achieved:

 Immediate: Achieved immediately up until three months of engaging with Noise Solution, and when maintained, may result in medium-term or sustained outcomes. Outcomes in this category include developing new skills and improvements in self-esteem and self-confidence

 Short-term: Achieved within three to six months of engaging with Noise Solution, and when maintained, may result in a sustained outcome. Outcomes falling into this category include increased educational attainment and willingness to try new activities and tasks

 Medium-term outcomes: Achieved in six to twelve months, and possibly sustained for a longer period. Outcomes in this category include increased labour productivity from guardians and reduced need for additional support in other areas through increased resilience and independence

 Long-term outcomes: Achieved from 12 months, for example reduced financial burden on families and reduced isolation

This relationship is called a theory of change and aims to summarise the impact Noise Solution has on society. The finalised impact map is presented in Figure 1.

Figure 1. Impact map

Evaluating and monetising outcomes

The outcomes presented in the impact maps were subsequently assessed to determine whether they could be modelled robustly. Attempts were made to capture and quantify the vast majority of these outcomes; however, this was not possible in all cases for a number of reasons:

 Some shorter term outcomes overlapped with longer term outcomes, such as ‘Increased labour productivity of families’ and ‘Reduced financial burden’. In this case, modelling both outcomes would have led to double counting in the value generated

 Some long-term outcomes affected broad and ill-defined groups of stakeholders, such as ‘Increased awareness of self’. Monetising these outcomes would introduce considerable uncertainty to the findings

 Despite attempts to monetise all outcomes, suitable financial proxies could not be found for some. One such outcome was ‘Improved sense of hope’

The following sub-sections detail the outcomes that have been included in the SROI model. For each outcome considered, the parameters and the respective data sources are described, including the indicators and financial proxies used in the model. An indicator is used to determine whether change has taken place. A proxy is a value that is deemed to be close to the desired indicator, for which exact data might be unavailable.1 A summary of the outcomes valued in the SROI analysis is provided in Table 2.

For each outcome, multiple estimates and assumptions have been made. These have been described below. To decrease the uncertainty in the model, inputs could be replaced by data collected by Noise Solution in the future. Inputs for which this is applicable have been highlighted throughout the input tables using orange shading.

Table 2. Summary of outcomes valued in the SROI analysis

Impact on participant’s education

Impact on participant’s health

Impact on participant’s family

Outcome

Re-entry into education, employment or training

Reduced behavioural issues/school conduct problems

Reduced/avoided school exclusions

Reduced need for additional support in education and related services

Increased social capital/reduced isolation

Diverted from acute mental health admission

Disengagement from mental health services

Increased labour productivity from guardians (increased/returned to work)

Reduced stress and burden for family members/guardians

Abbreviations: SROI: social return on investment.

For all financial proxies, the costs have been adjusted for inflation using the customer price index (CPI). The calculation for adjusting the costs has been provided below:

CPI (current year )

Inflated cost =

CPI ( year of financial proxy ) x financial proxy

Costs were inflated to the 2023 price due to the 2024 CPI having not yet been published at the time of the SROI analysis. All costs provided in this report are those from original sources and are unadjusted for inflation; please check the model for inflated costs.

Impact on participant’s education

Re-entry into education, employment or training

As participants show re-engagement with education, training or employment, there is an expected reduced burden on government social services that would otherwise be spent to support these youth. The proposed indicator for this outcome was the proportion of Noise Solution participants who are not in education, employment or training (NEET) prior to Noise Solution, and those who have shed their NEET status following Noise Solution. It was assumed that individuals with NEET status would not be in any form of education, employment or training for the entirety of a given year. Data on truancy of Noise Solution participants are currently not collected. Therefore, estimates from Noise Solution were used; these have been presented in Table 3

Table 3. Estimated proportion of Noise Solution participants with NEET status

Abbreviations: NEET: Not in education, employment or training.

The financial proxy used for this outcome was the weekly job seekers allowance (JSA) for individuals aged over 16. Allowances vary for individuals aged 16–24 and over 25, therefore the proportion of participants in each age category was calculated using a Noise Solution dataset of past participants (Table 4) and different costs assigned to each category (Table 5).2 The proportions of participants moving between age groups during the time horizon were not considered. It was assumed that individuals under 16 years old would not be receiving JSA if they are NEET, therefore they were not accounted for in this outcome. Other costs included in the model, including guardian labour productivity, were assumed to already capture the associated costs of having NEET status under 16.

Table 4. Age of Noise Solution participants

Table 5. Financial proxy for NEET status

Abbreviations: NEET: Not in education, employment or training.

Reduced behavioural issues/school conduct problems

As participants develop sense of competency in a new area, and increased engagement in pro-social activities and behaviours in the short-term, there is expected to be a reduction in school conduct problems, such as detentions. These outcomes from the impact map have been calculated by estimating the proportion of Noise Solution participants that would receive behavioural complaints that disrupt classroom teaching, both pre- and post-participation in Noise Solution. As this outcome is specific to school behaviour, it is applied to schoolaged participants (under 18) with non-NEET status only.

In the absence of any data identified in the literature or collected by Noise Solution, the reduction in school conduct problems has been based on assumptions. It has been assumed that before Noise Solution, 80% of school-aged, non-NEET participants receive behavioural complaints at school. Post Noise Solution, it is assumed that participants who show a significant improvement in wellbeing have a 65% reduction in receiving behavioural complaints, whilst all other non-NEET participants receive the same number of complaints. These inputs are summarised in Table 6. Participants who shed NEET status and return to school are also considered following Noise Solution. It is assumed that these individuals return to school with the same reduction in behavioural problems as individuals who were in school.

Table 6. Estimated behavioural problems in non-NEET participants in school

Noise Solution estimate for reduction of behavioural complaints in school-aged, nonNEET participants

Abbreviations: NEET: not in education, employment or training; SWEMWBS: Short Warwick-Edinburgh Mental Wellbeing Scale.

The financial proxy used to calculate the value of this outcome was the cost of a school-based social and emotional learning programme, as reported in the PSSRU 2021, in addition to the annual cost of teaching time salary that is lost to management of classroom behavioural issues per disruptive child.3, 4 These values are summarised in Table 7.

The Time and Cost of Classroom Behavior Management (Gormley, 2021)4

Assuming 3 seconds of disruption per occurrence, teachers spend 10.95 hours responding to a single referred child’s behaviour each year. Based on average England (excluding London/fringe) teaching salary of £28,0005 (assumed 37.5 hours/week)

Abbreviations: PSSRU: Personal and Social Services Research Unit.

Reduced/avoided school exclusion

Outcomes relating to young people re-engaging with education, training, employment or voluntary placements are included throughout the impact map. As participants are expected to gain confidence, competency and achievement in the short-term, and hence re-engagement in the medium-term, there is an expected re-engagement (either in mainstream schooling, or other means) of a proportion of those participants who were previously excluded from schooling. This outcome has been calculated as the proportion of participants who have been excluded before Noise Solution, and the proportion of participants who remain excluded after Noise Solution, and has only been applied to school-aged participants. These inputs have been estimated by Noise Solution and are presented in Table 8. It has been estimated that of the ~45% of participants that are expelled/suspended from school, there is a 1:1 split between exclusions and suspensions. This estimate has been used due to a lack of appropriate input data, however should be updated

Table 7. Financial proxy for managing school behavioural problems

to better reflect the proportion of Noise Solution participants who are excluded from school based on data captured at Noise Solution referral.

Estimate from Noise Solution – ~40% of participants with prior exclusions re-engage with education, employment or voluntary placements

To estimate associated costs, a report published by the New Philanthropy Council that estimates the total cost of school exclusions in the UK was reviewed.6 Specifically, costs to the education system and social services were extracted. Other costs included in the report that were not considered in the SROI model were: impact on earnings, cost to the NHS and cost of higher crime. These were not included as they were either captured elsewhere in the model (e.g., mental health treatment costs), or likely to come to bear outside the duration scope of the model (e.g., physical health costs). The cost of managing the process of an exclusion was applied in the model as a one-off cost in the first year and not considered in subsequent years. A summary of the costs associated with school exclusions is provided in Table 9.

Table 9. Financial proxy for school exclusions, per exclusion

Financial Proxy Value

Cost of managing the process of an exclusion (one-off cost) £831.00

Source/Justification

Misspent Youth Report:

Reduced need for additional support in education and related services

Another outcome identified in the impact map is the reduced need for additional support in education and other related services. This outcome has been calculated by estimating the use of educational support services pre- and post-participation in Noise Solution. As this outcome is specific to schools, it is only calculated for school-aged, non-NEET participants.

Three educational services were captured in the model; educational psychologist, special needs support and classroom assistance, based on sources cited in literature.7, 8 The proportion of participants in contact with an educational psychologist, and/or with special educational needs (SEN), is based on the Mental Health of Children and Young People 2017 survey, stratified by type of behavioural disorder (see Table 10).7 In the absence of estimates for classroom assistants, this was assumed equal to the proportion of children accessing SEN support. Of all Noise Solution participants, it was assumed that 80% had either an emotional, conduct or hyperactive disorder, distributed equally. The remaining 20% were assumed to not have a disorder. It is assumed that the prevalence of disorders is irrespective of the participants age. Mean use of each service was derived from Romeo 2006, a paper estimating the cost of severe antisocial behaviour in children, which collected service use data for children who had been referred to mental health services in the UK (Table 10).8

Following Noise Solution, it was assumed that participants who had a meaningful improvement in wellbeing, measured through SWEMWBS, reduced their use of each service by 65%. Those without a meaningful improvement in SWEMWBS continued with the usage of services as before. The impact of individuals who shed NEET status are also considered post-Noise Solution, and are assumed to follow the same distributions in terms of disorders and service use as individuals who did not have NEET status at the start of the program.

Table 8. Estimated school exclusions of participants

A summary of the inputs derived from literature are presented in Table 10. The resulting estimated proportion of Noise Solution participants receiving support before and following Noise Solutions is presented in Table 11.

Table 10. Education and related service use, as reported in literature

Abbreviations: SEN: special education needs.

Table 11. Estimated use of support in education and related services by participants

individuals who receive selected support according to behavioural disorder, as reported in Table 10

Reduction in support services is applied at the cohort level, so that 65% of Noise Solution participants who have a significant improvement in wellbeing cease use of support services. This is equivalent to each of these participants reducing use of services by 65% each, if service use has a 1:1 ratio with disorder intensity. Participants who shed NEET status have the same reduction in use of education support and related services

Abbreviations: NEET: not in education, employment or training.

Costs for each service were derived from the PSSRU and the Education Endowment Foundation.3, 9 The cost of an educational psychologist was assumed equal to a paediatric occupational therapy visit. Special needs

support and classroom assistance hourly rates were calculated based on the mean wage of a teaching assistant in the UK in 2023 (£23,000), assuming 195 working days and 6.5 hour days.10

Table 12. Financial proxy for additional support in education and related services

Abbreviations: PSSRU: Personal and Social Services Research Unit; SEN: special education needs.

Impact on participant’s health

Increased social capital/reduced isolation

A further outcome identified in the impact map is the increased social capital as a result of Noise Solution participation and thereby reduced isolation. The value of this outcome was approximated by calculating the proportion of individuals experiencing loneliness based on data published by the ONS.11 This assumes that all participants who join Noise Solution have the same experience of loneliness as someone with a probable mental health disorder. The proportion of individuals experiencing loneliness was reported in individuals aged 11–16 and 17–22, as presented in Table 13. The proportion of individuals experiencing loneliness before Noise Solution was calculated assuming all participants aged 16 and under have the same likelihood of loneliness as one another, and that all participants aged 17 and over have the same likelihood of loneliness as one another. The proportion of participants who no longer experience loneliness following Noise Solution has been estimated by Noise Solution based on discussions with participants who took part in the programme.

Assumes that all participants in Noise Solution have the same experience of loneliness as someone with a mental health disorder and participants aged 16 and under have the same likelihood of experiencing loneliness

ONS, 2023; Proportion of individuals experiencing loneliness by mental health of child, 2023; aged 17–22

Assumes that all participants in Noise Solution have the same experience of loneliness as someone with a mental health disorder and participants aged 17 and over have the same likelihood of experiencing loneliness

A weighted average of the above inputs based on the age distribution of Noise Solution participants

Table 13. Estimated loneliness of participants

Abbreviations: ONS: Office for National Statistics.

The financial proxy associated with this outcome was based on an analysis by the Department for Digital, Culture, Media and Sport which estimated the costs associated with loneliness stemming from the impact on subjective wellbeing, health and productivity.12 For individuals aged 17 and over, the total cost considered the impact of health, productivity and wellbeing for individuals with moderate to severe loneliness. The report did not consider the cost for individuals under 16, therefore it was assumed that the cost of mild loneliness, which does not consider productivity loss, would be an appropriate proxy for individuals 16 and under. The costs have been summarised in Table 14.

Table

Loneliness monetisation report, 202012

Assumes loneliness impacts subjective wellbeing, health and productivity

Loneliness monetisation report, 202012

Assumes cost of moderate to severe loneliness in individuals aged 16 and under is proportionate to mild loneliness in individuals aged 17 and over, to account for the lack of impact on productivity

Diverted from acute mental health admission

In-patient admission to mental health services was identified as an outcome impacted by Noise Solution. The proportion of patients with previous in-patient mental health admission prior to Noise Solution was estimated by Noise Solution, as was the proportion diverted from in-patient admission in the year following the intervention. These proportions were assumed to apply equally to participants irrespective of age.

This outcome has been monetised by valuing the cost of in-patient admission for children, which is under the responsibility of CAHMS, and for adults. For children, a report in 2017 reported the average cost of CAMHS inpatient admission per person. For adults, the average cost per person was estimated using the average cost in-patient admission per day (£446.00), the average length of in-patient admission (49 days) and the costs following discharge from acute medical units (£2,431.00).13-15 The costs have been summarised in Table 16.

14. Financial proxy for loneliness

Table 15. Estimated in-patient mental health admission

Table

Financial Proxy Value

Cost of average CAMHS in-patient admission

Source/Justification

Children's Commissioner: Children's Mental Healthcare in England14

Average cost of CAMHS in-patient admission from 2017 Cost

East London NHS Foundation Trust, 2020; daily cost of in-patient mental health admission (£446.00)15

The strategy unit, 2019; average length of mental health in-patient admission (49 days)16

PSSRU, 2022; post-discharge costs (£2,431.00)13

Abbreviations: CAMHS: Children and Adolescent Mental Health Services; NHS: National Health Service; PSSRU: Personal and Social Services Research Unit.

Disengagement from mental health services

In addition to in-patient mental health admissions, it is also expected that individuals who attend Noise Solution will disengage from community mental health services as a result of improved wellbeing. In the absence of any data identified in the literature or collected by Noise Solution, it was estimated that 35% of participants use community mental health services prior to Noise Solution. Following Noise Solution, it is estimated that this reduces to 15%. These inputs have been summarised in Table 17

As with in-patient mental health admissions, different providers are used for children and adult services. For children, the outcome was monetised by valuing the cost of CAHMS community support per child per year, as reported by the NHS.17 For adults, the cost was estimated using the average cost per attendance of adults requiring mental health support to day care, provided by local authority social services (£58.00) or the private and voluntary sector (£41.00), with the assumption that the adult would attend one or the other, with equal likelihood of each, for the average length of recovery (three sessions). The cost of delivering behavioural activation, a treatment for depression provided in a group setting, was also considered, with the assumption that the adult would attend a complete programme (12 sessions) once a year (£228.00) in addition to local authority/voluntary services.13 These costs have been summarised in Table 18.

16. Financial proxy for in-patient mental health admission

Table 17. Estimated engagement of participants with mental health services

Table 18. Financial proxy for community mental health services

Cost of CAMHS community mental health services, per year

Cost of adult community mental health visits, per year

Abbreviations: CAMHS: Children and Adolescent Mental Health Services.

Impact on participant’s family

NHS Benchmarking Network: Benchmarking Mental Health17

Average cost of a year of CAMHS community mental health services from 2017

PSSRU, 202213

Cost per attendance to day care, provided by social services (£58.00) or voluntary or private sector (41.00); assumed attendance of three sessions

Cost per attendance to behavioural activation programme (12 sessions), once a year (228.00)

Reduced stress and burden for family members/guardians

An outcome identified in the impact map is the reduction in stress and reduced burden experienced by family members, resulting from the participant’s behavioural problems. This outcome estimates the proportion of guardians and siblings experiencing increased stress/caregiver burden, before and after Noise Solution, and applies the cost of a counselling intervention, to represent the value with and without Noise Solution. It is assumed that guardians and siblings are affected by the participant’s behaviour, regardless of the participant’s age.

Separate calculations were made for guardians and siblings, each of which are described in turn. The proportion of siblings affected by the participant’s behaviour pre Noise Solution was based on reported data from the Mental Health of Children and Adolescents 1999 study.18 Specifically, the answers to ‘child’s problems cause difficulties with other family members’ for children with emotional, conduct and hyperactive disorders was considered. The proportion of siblings affected ranged from 19–35% for individuals with behavioural disorders. Given that not all Noise Solution participants are anticipated to have behavioural disorders, a conservative estimate was used whereby 20% of all siblings are disrupted by the participant’s behaviour. After Noise Solution, it was anticipated that individuals who have a significant improvement in wellbeing, measured through SWEMWBS, would have a reduction in behavioural problems that would remove the stress and burden experienced by siblings. The average number of siblings per participant was calculated using data reported by the ONS.19 The inputs used to value the reduced stress and burden for siblings are described in Table 19.

Table

Proportion

Proportion of siblings affected by participant's behaviour,

Conservative estimate based on proportion of siblings affected by child’s behaviour according to different behavioural disorders; reported in Mental Health of Children and Adolescents, 199918

Proportion of participants experiencing increased (>1 score increase) SWEMWBS score assumed to experience reduced incidence of behavioural issues

All siblings of participants who do not show a significant improvement in wellbeing remain affected, if already in an affected state

19. Estimated stress and burden for siblings

Abbreviations: ONS: Office for National Statistics; SWEMWBS: Short Warwick-Edinburgh Mental Wellbeing Scale.

For guardians, it was assumed that prior to Noise Solution, all guardians showed increased stress resultant of child’s behaviour, in the absence of relevant data. Stress and caregiver burden post Noise Solution was calculated in the same way as for siblings, with a proportional reduction based on Noise Solution participants showing a meaningful improvement in wellbeing. The average number of guardians per participant were calculated based on the proportion of families with one or two guardians using data published from the ONS.20 Table 20 summarises the described inputs used to calculate the proportion of guardians who are impacted by the participant’s behaviour, pre and post Noise Solution.

Table 20.

The model assumes that all guardians of participants who do not show a significant improvement in wellbeing maintain affected, if already in an affected state

Abbreviations: ONS: Office for National Statistics; SWEMWBS: Short Warwick-Edinburgh Mental Wellbeing Scale.

The financial proxy associated with this outcome was the average cost per counselling intervention, as reported in the PSSRU 2021.3 Specifically, the cost applied was for interventions ranging from 9 to 12 counselling sessions (i.e. short- to medium-term) over 12 hours of intervention, at a value of £1,217 per counselling intervention (i.e. per family member affected). It should be noted that this cost is applied as a representative approximation of the ‘value’ of reduced stress and caregiver burden, rather than the actual cost that would be expected to be incurred by family members/guardians experiencing increased stress. The cost has been summarised in Table 21.

Table 21.

Abbreviations: PSSRU: Personal and Social Services Research Unit.

Increased labour productivity from guardians

As participants show increased engagement in pro-social activities and behaviour in the short-term, there is anticipated increased labour productivity from guardians due to increased/return to work. The value of this outcome was approximated by calculating the average number of days absent from work pre, during and post Noise Solution, and applying a pro-rata average wage for each day absent, across both guardians. This outcome is assumed only to impact participants who live with their guardians and/or need adult supervision. Therefore, the outcome was only considered for participants aged 18 and under.

Estimated stress and burden for guardians

Financial proxy for stress and burden for family members/guardians

In the absence of data from Noise Solution, it was assumed that pre Noise Solution, one guardian per participant was absent one day per month, resulting in an annual estimate of 12 days. For the year including and immediately after the 10-week duration of Noise Solution, in which participants attend 10 two-hour sessions, it was assumed that a guardian attended the first five sessions with participants and that this time represented lost productivity in addition to the 12 day background rate. This additional productivity loss was pro-rata adjusted to reflect the proportion of sessions that participants failed to attend (15.8%), assuming that non-attendance was equally likely between guardian-attended and guardian-non-attended sessions. In the case of non-attendance, it is assumed that there is no additional loss of productivity for the guardian above the base rate of productivity loss, i.e. the guardian would have been absent from work on the given day irrespective of a planned Noise Solution session and that the attendance time is absorbed in the 12 day background rate.

For both the year of the program, and the years following that, it was assumed that the guardians of participants who show a significant improvement in wellbeing no longer have any absent days from work related to caring for the participant, while guardians of other participants remain at pre Noise Solution levels. These inputs are summarised in Table 22. Productivity loss relating to care of the participant was applied in addition to productivity loss due to attendance of sessions with participants in the year of the program. In subsequent years, only productivity loss relating to the care of the participant is considered, as guardians no longer attend the course with the participant.

Table 22. Estimated loss of productivity of guardians per participant

Assumes that guardians attend half of Noise Solution sessions and that non-attended sessions do not result in loss of productivity for guardians

Assumes that the guardians of participants who show a significant improvement in wellbeing no longer have any absent days from work, while other guardians of participants remain at pre-Noise Solution levels

Assumes that the guardians of participants who show a significant improvement in

This outcome has been monetised by valuing the loss of productivity resulting from guardians being absent from work. The financial proxy for this outcome is the national average weekly wage in the UK (Table 23).21

Table 23. Financial proxy for labour productivity

Adjustment

To avoid overestimating the impact of Noise Solution, the following adjustments were applied in the SROI model:

 Deadweight: Deadweight involves the recognition that some stakeholders might experience outcomes of interest, even if Noise Solution did not intervene. Deadweight was built into the final calculations for each outcome modelled, by considering the number of stakeholders who would have experienced the outcome in a situation where Noise Solution had not provided the service due to other influences

 Attribution: Attribution involves the recognition that outcomes achieved may not entirely be due to Noise Solution’s services alone. Service users, for example, may be engaged with several organisations, providing overlapping support. Through use of a percentage, attribution estimates the proportion of impact created entirely by Noise Solution, after deducting the contribution of other services. Only this is accounted for when calculating the final value of the service in question. Please see below for further information

 Drop-off: Drop-off involves the recognition that outcomes may not last indefinitely. The impact of an outcome may decrease over time, or benefits may be more likely to derive from other factors. Drop-off was accounted for by applying monthly percentages to each outcome; please see below for further information on how drop-off was calculated, and for drop-off factors

 Discounting: Discounting involves the recognition that outcomes that occur in the future are valued less than those incurred immediately. In the base case, a 3.5% discount rate is assumed across all variables for year two onwards, based on the HM Treasury Green Book for discounting in the public sector.22 The model includes the flexibility for the user to vary this to 5% (more conservative) or 0% (less conservative) for all of the outcomes, or alternatively, the user can adjust it for each outcome individually

Attribution

The default values for attribution included in the model were informed through discussion with stakeholders. From the interviews conducted in the original SROI analysis, it was understood that many participants engaging with Noise Solution simultaneously receive additional support (such as counselling or therapy). However, Noise Solution is understood to act as an enabler to the effectiveness of these alternative interventions, and therefore remains a key component to the overall success of the participant. Furthermore, the majority of participants who are referred to Noise Solution have already tried alternative interventions that have not delivered the satisfactory results. As a result, it is estimated that the proportion of outcomes observed attributable to other interventions is expected to be relatively low. In the absence of data, 25% of each outcome has been attributed to another intervention, and therefore not included in the final base case SROI calculation. The model includes the flexibility for the user to vary this to 50% (more conservative) or 0% (less conservative) for all of the outcomes, or alternatively, the user can adjust it for each outcome individually.

Drop-off

All outcomes included in the model are expected to last two years. Estimates for drop-off after one year have been based on long-term data reported in the literature where possible. For outcomes where no data have been identified for estimating drop-off, it has assumed to be 50% - common in SROI practice.1 The model includes the flexibility for the user to vary this to 50% or 75% for all inputs (more conservative), or 0% (less conservative). Alternatively, the user can adjust this input for each outcome individually. In addition, the model also includes the flexibility to calculate the SROI assuming each outcome lasts for one year only. If this option is selected, drop-off is no longer relevant, and therefore not applied in the model. The drop-off used in the base case are described in Table 24.

based on Impetus 2017 where 52% of NEET people remained NEET the following year.23 In the absence of data, this outcome has been conservatively assumed to only last two years.

In absence of alternative data, dropoff has been aligned with a SROI publication of a UK school-based wellbeing programme, which assumes 50% each year.

Table 24. Annual drop-off for each outcome

Sensitivity Analyses

A one-way sensitivity analysis was conducted on the base case results, to identify the inputs which the SROI was most sensitive to. Sensitivity analysis was conducted through varying the inputs used to calculate each outcome one at a time, with an upper and lower value. The impact on the SROI ratio was subsequently recorded, in order to identify the top ten variables that had the greatest impact on the SROI. The results of the sensitivity analysis are presented in a tornado plot in the model.

Assumptions and limitations

General limitations of SROI methodology

Every economic model is a simplification of events happening in the real world, and thus is subject to assumptions and limitations. However, while assumptions and limitations are an inherent part of economic modelling, the model itself can still produce useful and reliable results if limitations are addressed, or results are interpreted within the constraints of those limitations.

SROI analyses are subject to several inherent limitations. First, SROI methodology attempts to quantify and monetise the non-tangible impact of an organisation. The process of assigning financial values to nonfinancial outcomes is inevitably associated with a degree of uncertainty.1 Therefore, it is important to be cautious when drawing detailed conclusions from the analysis. Moreover, the lack of a universally accepted approach to SROI modelling means that few meaningful conclusions can be drawn from the comparison of different organisations’ final SROI ratios. Nonetheless, when used in the right context, SROI analysis is a valuable tool to highlight the breadth and depth of impact of the activities of Noise Solution on its stakeholders in a granular and semi-quantitative way.

Limitations of the Noise Solution SROI model

There was a lack of sufficient evaluative data to fully capture the impact of the Noise Solution programme. In most cases, exact inputs were not available and input-specific assumptions had to be made. Where assumptions have been made for model input values, there were informed by comparable figures, published evidence and discussions with Noise Solution. Therefore, any model input that relies on assumptions is presumed to be close to the true value and any impact on model results is anticipated to be minor. Furthermore, conservative values were used to minimise the risk of inflating the SROI results, and assumptions were validated by Noise Solution. Noise Solution are currently working to update the information captured about their participants in referrals and their customer relationship management (CRM) platform, in order to better understand the journey of the individuals that join their programme. Once this data becomes available, the SROI model can better reflect the population.

The lack of sufficient data also meant that certain outcomes and stakeholder groups had to be excluded from the analysis. For example, a significant improvement in wellbeing has been observed from participation in the programme. However, the paucity of available quantitative data and a lack of a robust way to incorporate this in the SROI model led to exclusion of this outcome from the analysis. Consequently, these limitations mean that the final SROI values are likely to be an underestimate due to the use of conservative assumptions and the exclusion of certain outcomes.

Other limitations of the model include its limited ability to capture the value that beneficiaries of the programme were able to pass on. For example, participants of Noise Solution may have been able to share their experience with peers facing a similar situation and therefore introduce them to the programme. It is expected that impact generation occurred beyond that captured in the model, and this impact is likely to continue to grow beyond the model time horizon as knowledge and expertise continues to spread. However, quantification of the value of these relationships comes with inherent uncertainty and would also require an element of forecasting, which was beyond the scope of this analysis.

References

1. Jeremy Nicholls EL, Eva Neitzert, Tim Goodspeed. A Guide to Social Return on Investment. In: Network TS, ed, 2012.

2. Department for Work and Pensions. Benefit and pension rates 2023 to 2024. Available at https://www.gov.uk/government/publications/benefit-and-pension-rates-2023-to-2024/benefit-andpension-rates-2023-to-2024. Last accessed: February 2024. 2023.

3. PSSRU. Unit costs of health and social care 2021. Available at: https://www.pssru.ac.uk/projectpages/unit-costs/unit-costs-of-health-and-social-care-2021/ [Last accessed: February 2024]. 2021.

4. Gormley MJ. The Time and Cost of Classroom Behavior Management. Presented at the National Association of School Psychologists Annual Convention. 2021.

5. Gov.uk. National pay scales for eligible teaching and education leadership occupation codes. Available at: https://www.gov.uk/government/publications/national-pay-scales-for-eligible-teaching-andeducation-jobs/national-pay-scales-for-eligible-teaching-and-education-leadership-occupation-codes Last accessed: February 2024. 2023.

6. Brookes M, Goodall E, Heady L. Misspent Youth - The Costs of Truancy and Exclusion. New Philanthropy Capital 2007.

7. Collinson D. The mental health of children and young people in England. In: National Centre of Social Research, ONS, eds, 2017.

8. Romeo R, Knapp M, Scott S. Economic cost of severe antisocial behaviour in children - and who pays it. British Journal of Psychiatry 2006;188:547-553.

9. Education Endowment Foundation. Teaching and learning toolkit. 2018.

10. National Careers Service. Special educational needs (SEN) teaching assistant. Available at: https://nationalcareers.service.gov.uk/job-profiles/special-educational-needs-(sen)-teaching-assistant. Last accessed: February 2024. 2023.

11. ONS. Mental Health of Children and Young People in England, 2023: Wave 4 follow up to the 2017 survey. 2023.

12. Department for Digital C, Media & Sport. Loneliness monetisation report. 2020.

13. PSSRU. Unit costs of health and social care 2022. Available at: https://www.pssru.ac.uk/projectpages/unit-costs/ [Last accessed: February 2024]. 2022.

14. Children's Commissioner. Briefing: Children’s Mental Healthcare in England. 2017.

15. Trust ELNF. FOI DA3620. 2021.

16. The strategy unit. Exploring Mental Health Inpatient Capacity across Sustainability and Transformation Partnerships in England. 2019.

17. NHS. Benchmarking Mental Health. 2017.

18. Meltzer H, Gatward R. The mental health of children and adolescents in Great Britan. In: Statistics OfN, ed, 1999.

19. ONS. Families with dependent children by number of children, UK, 1996 to 2017. 2018.

20. ONS. Families and households. 2023.

21. ONS. Employee earnings in the UK: 2023. Available at: https://www.beta.ons.gov.uk/employmentandlabourmarket/peopleinwork/earningsandworkinghours/ bulletins/annualsurveyofhoursandearnings/2023#:~:text=Median%20weekly%20earnings%20for %20full-time%20employees%20was%20%C2%A3682,highest%20growth%20since%20comparable %20records%20began%20in%201997. Last accessed: February 2024. 2023.

22. HM Treasury. The Green Book: Central Government Guidance on Appraisal and Evaluation. 2018. 23. Impetus. Youth Jobs Index. 2017.