Evaluation of Public Health Network Cymru (2017) by Public Health Network Cymru

Evaluation of Public Health Network Cymru (2017) Dr Benjamin Gray, Public Health Research Data Coordinator

Get in contact You can contact Public Health Network Cymru in a variety of ways Email us

publichealth.network@wales.nhs.uk

Call us 02920 104450 Write to us Public Health Network Cymru, Public Health Wales, Floor 5, Number 2 Capital Quarter, Tyndall Street, Cardiff, CF10 4BZ Social media twitter: @PHNetworkCymru facebook: Publichealthnetworkcymru

Executive Summary This report is the first comprehensive evaluation of Public Health Network Cymru (‘the Network’) since the consolidation of the four former topic-specific networks (physical activity; nutrition; mental health promotion; sexual health) and provides an up-to-date overview of the work of the Network, building on previous evaluations undertaken in 2008 and 2013. The evaluation looked to ask and answer a number of key questions in relation to the main activities of the Network. These were: • • • •

How well is the Network performing against its aims and objectives? What are the member and stakeholder perspectives of the new website and Ebulletin? What does the Network need to do differently to increase engagement? How well is the Network performing compared to the previous evaluation in 2013?

The data required to inform this evaluation was collected via three primary approaches; (i) a member questionnaire designed to gather information on member demographics and views on some of the main activities of the Network; (ii) interviews with stakeholders to establish how well the Network are meeting the key themes of the aims and objectives; and (iii) use of existing event feedback captured from delegates. Overall, the work of the Network is positively received by both the members and the stakeholders. It is important to appreciate that the Network has upwards of 1000 members and these numbers increase each month. Therefore, in Wales especially the potential of the Network cannot be underestimated. The Ebulletin and website are the two main sources where members access information to help inform professional practice, whilst information on upcoming events appear to be of greatest interest to members. Engagement is an issue for the Network which is clearly demonstrable from the underutilisation of the forum resources on the new website. Although somewhat promisingly there does appear to be an appetite amongst members to become more ‘active’ and/or contribute to future Ebulletins. Undertaking this evaluation has resulted in seven recommendations to be considered. These recommendations build on the things that the Network already does well, such as the Ebulletin, further developments of methods to extend the reach and learning of events (e.g. podcasts, livestreaming) and also to create the opportunity for members to become more ‘active’. This evaluation and future implementation of recommendations align to the relevant action (2017/18) and milestone detailed in Strategic Priority 5A of Public Health Wales ‘Our Strategic Plan’ 2017-2020 and Public Health Wales’ Long Term Strategy (Strategic Priority: Building and mobilising knowledge, capacity and capability to drive health and well-being across Wales).

Contents 1.0

Introduction

1.1

Overview of Public Health Network Cymru

1.2

Overview of main activities and changes since the launch of the Network

1.2.1

The Website

1.2.2

The Ebulletin

1.2.3

Events

1.3

Purpose of this evaluation

2.0

Methods

2.1

Member Questionnaire

2.2

Stakeholder Perspectives

2.3

Use of Existing Event Feedback

3.0

Results

3.1

Demographics of Questionnaire Respondents

3.2

Member Opinions and Behaviours

3.3

Perspectives on Website (Members and Stakeholders)

3.4

Perspectives on Ebulletin (Members and Stakeholders)

3.5

Events

3.6

Overall Performance

4.0

Key Questions (Discussion)

4.1

Performance against Aims and Objectives

4.1.1

Educate and inform the wider public health workforce on developments related to public health practice in Wales

4.1.2

Highlight and showcase examples of good practice in Public Health delivered throughout 21 Wales

Contents 4.1.3 4.1.4 4.1.5 4.1.6 4.1.7 4.2

Encourage interactivity and the exchange of knowledge and ideas amongst the public health workforce in Wales Facilitate opportunities for debate and discussion about contemporary public health issues amongst the various stakeholders within Wales and beyond Promote opportunities for collaboration amongst stakeholders on new and emerging public health issues The Website Inform and promote local, national and international public health related policy, practice and evidence Increase awareness of the work of Public Health Wales to the wider workforce What are the member perspectives of the new website and Ebulletin?

22 22 22 23 23 23

4.2.1

New website

4.2.2

Ebulletin

4.3

What does the network need to do differently to increase engagement?

4.4

Performance compared to previous evaluation (2013)

5.0

Conclusions

5.1

Summary of Main Findings

5.2

Limitations

5.3

Recommendations

Appendices

Appendix A1. Public Health Network Cymru Website Analytics (July - October 17)

Appendix A2. Example Wordle (Behaviour Change and Addictive Behaviours)

Appendix A3. Example Wordle (Adverse Childhood Experiences: What it means for you)

1.0 Introduction 1.1 Overview of Public Health Network Cymru Public Health Network Cymru (‘the Network’) was launched in May 2015 following the consolidation and further development of four former topic-focussed public health networks (physical activity; nutrition; mental health promotion; sexual health). Now covering a full range of public health topics, the network aims to be the first place that people consider when they are seeking up-to-date news, emerging developments or peer advice on public health issues in Wales. The Network strives to be the place where practitioners from all sectors can find relevant, timely, easily understood information, tools and guidance. Public Health Network Cymru has the following overarching objectives; • Educate and inform the wider public health workforce on developments related to public health practice in Wales; • Highlight and showcase examples of good practice in Public Health delivered throughout Wales; • Encourage interactivity and the exchange of knowledge and ideas amongst the public health workforce in Wales; • Facilitate opportunities for debate and discussion about contemporary public health issues amongst the various stakeholders within Wales and beyond; • Promote opportunities for collaboration amongst stakeholders on new and emerging public health issues; • Inform and promote local, national and international public health related policy, practice and evidence; • Increase awareness of the work of Public Health Wales to the wider workforce. As of November 2017, the Network had 1085 individual members who had enrolled through the website (https://www.publichealthnetwork.cymru/en) and the Network also had 1077 Twitter followers of their handle @PHNetworkCymru. 1.2 Overview of main activities and changes since the launch of the Network 1.2.1 The Website As part of the evolution of the Network from the previous ‘networks’ a new website (https://www.publichealthnetwork.cymru) was developed that encompassed all public health related topics rather than focussing just on the four previous topics, therefore replacing the pre-existing individual networks own websites. This newly designed website was developed over an 18-24 month period with feedback gathered from members and stakeholders. The new Network website also houses five affiliate websites; (i) (Public Health Wales) Research and Development Community, (ii) International Health Coordination Centre (IHCC), (iii) Wales Health Impact Assessment Support Unit (WHIASU), (iv) Making Every Contact Count (MECC) and, (v) National Centre for Population Health and Wellbeing Research (NCPHWR).

1.2.2 The Ebulletin The conception of the Network also resulted in a change to the monthly newsletters. Instead of three separate newsletters being sent to members, each month all members are now e-mailed a link to a copy of an Ebulletin which focusses on a different theme or topic. For example, previous issues of the Ebulletin have focussed on subjects ranging from World Mental Health Day to Homelessness etc. In the weeks prior to the publication of an Ebulletin, members are emailed and invited to contribute to that month’s theme or more broadly to contribute. The Ebulletins are published in both Welsh and English language and in addition to a ‘simple’ PDF version, the Ebulletins are also available in a ‘high-quality’ format which has been published using ISSUU which is a digital publishing platform. An example of a ‘high-quality’ Ebulletin can be accessed at https://issuu.com/phwprid/docs/phnc_nov_bulletin_english_hq. 1.2.3 Events Events which include conferences, seminars and roadshows (which travel around locations in Wales) are a staple component of the work of the Network. Since the inception of the Network a number of events have taken place throughout Wales and these have ranged from many topics and areas. The topics for the events are either chosen through a poll by the Network members or are deemed to be an emerging or import public health topic by one of the Network team. Examples of some of these events are listed below; • • • • • •

#TechniHealth – Health Promotion in the Digital Era Adverse Childhood Experiences: What it Means For You Public Health Network Cymru: Creating Connections Behaviour Change and Addictive Behaviours Well-being of Future Generations: Public Health Opportunities Improving Health Within The LGBT Community

The Network has also assisted in organising and supporting three ‘Research in Wales’ conferences in conjunction with the Public Health Wales Research and Development team and the Network continues to jointly facilitate the annual Julian Tudor Hart lecture which has included speakers such as Professor Sir Michael Marmot in recent years. 1.3 Purpose of this evaluation The previous networks were subject to an initial evaluation in 2008 and a subsequent evaluation in 2013, however since the inception of Public Health Network Cymru no formal evaluation has taken place. This document will build on the previous evaluations and is the first evaluation of the Network in its current guise and will provide an insight into the performance of the Network to date. This evaluation sets out to ask a number of key questions in relation to the main activities of the Network. These are; • • • •

2.0 Methods The data required to inform this evaluation was collected via three primary approaches; 2.1 Member Questionnaire A questionnaire was designed and developed on SurveyMonkey with input from both the Network and the Research and Development teams within the Policy, Research and International Development Directorate (PRIDD) of Public Health Wales. This questionnaire was circulated to the email addresses provided by all members inviting them to give their views on the Network. The Member Questionnaire was designed to gather information on member demographics and also for their views on activities such as; • • • •

The Network Website The monthly E-bulletin Events including conferences, seminars and road shows Social Media Channels

The questionnaire which consisted of 52 questions (mixture of multiple choice and open ended) was launched on 1 August 2017 and remained live for a period of eight weeks (closing on 1 October 2017), between these dates at least three reminder emails were circulated to encourage participation. Members were offered an incentive of a prize draw for two £25 Amazon vouchers for completing the questionnaire. 2.2 Stakeholder Perspectives As part of the strategic oversight of the Network, there is a Network Advisory Group which represents the Network’s members. The Advisory Group has an overall remit to advise and support the Network to achieve its aims and objectives and to support Public Health Wales and its partners and stakeholders in the effective delivery of the 'Wellbeing of Future Generations (Wales) Act 2015' and other relevant policies and plans. A second set of more focussed questions was developed to gather the views of stakeholders (which included advisory group members). These questions were concentrated on the views of the individuals in relation to how well the network are meeting the key themes of the aims and objectives. The views of stakeholders (representatives of organisations who have worked directly and indirectly with the network) were sought and collated via three methods; i) Attendees at the August 2017 advisory group meeting were asked to give their feedback towards the key themes of the aims and objectives using a world café style format. ii) Any of the advisory group who were not in attendance at the August meeting were emailed copies of the questions to complete and return. iii) A list of stakeholders from all sectors not currently represented was compiled by the Network team and these stakeholders were emailed and telephone interviews conducted in a semi-structured manner arranged for a convenient date and time. In total, 14 stakeholders provided their views during this evaluation process, details of the respective organisations and/or sectors represented are documented in Table 1.

Table 1. Details of organisations/sectors represented by stakeholders that participated in the world cafĂŠ or interviews Organisations/Sectors Represented by Stakeholders Natural Resources Wales Local Public Health Team (Powys) Mind Cymru Chartered Society of Physiotherapy National Centre for Population Health and Wellbeing Research Wales Health Impact Assessment Support Unit Research & Development Community Website British Heart Foundation Cymru Terrence Higgins Trust Swansea University (Academia) The Wellbeing Planner Sport Wales Public Health Wales (Health Improvement) Welsh Local Government Association/Local Authorities 2.3 Use of Existing Event Feedback The Network routinely gathers and analyses information on delegate responses from each of their events. In the closing session of events, all attendees are provided with an evaluation form which they are asked to complete. The completion rate of event evaluations by attendees is usually around half (50%). The forms ask a number of questions ranging from a number of topics including; The organisation of the day The quality of the speakers How useful members found the event Any specific learning that will be taken away and used in practice Suggestions for future event topics All of the specific event evaluations are available in the resources section of the Network website, however an overview of these findings will be included to complement the member questionnaire views on events.

3.0 Results 3.1 Demographics of Questionnaire Respondents In total, 125 members completed at least part of the questionnaire, which represents a response rate of just over 10% of all invited members. The main characteristics of the members who completed the questionnaire are detailed in Table 2. The majority of questionnaire respondents had been a member of the Network for at least 6 months (n=114, 91.2%) and over one third were currently employed in a public health related role. One observation of note is that over half (n=64, 52.0%) of the member respondents indicated that they not a member of one of the previous networks. Table 2. Characteristics of member respondents to the questionnaire Length of Network Membership

(n=125, %)

More than 12 months

74 (59.2)

6-12 months

40 (32.0)

Less than 6 months

11 (8.8)

Member of Previous Network

(n=123, %)*

Physical Activity and Nutrition Network Wales

33 (26.8)

All Wales Sexual Health Network

26 (21.1)

All Wales Mental Health Promotion Network

14 (11.4)

Not a member of a previous network

64 (52.0)

Current Occupation (Field)

(n=121, %)

Public Health

42 (34.7)

Other

31 (25.6)

Allied Health Professions

14 (11.6)

Research

8 (6.6)

Local Government

6 (5.0)

Education

6 (5.0)

Youth worker

5 (4.1)

Community worker

4 (3.3)

Policy maker

3 (2.5)

Medicine

2 (1.6)

* Some members belonged to more than one network. 3.2 Member Opinions and Behaviours The member’s opinions and behaviours in this section are all collected from the Member questionnaire. One of the first questions of interest was with regards to where members first heard about the Network. Of the respondents, 37 (30.6%) first heard about the Network at an event, whilst a further 35 (28.9%) were made aware of the Network by a colleague. Accessing the website was the first time 22 respondents (18.2%) were made aware of the Network and 18 (14.9%) were introduced to the Network through promotional literature. When questioned as to the reasons individuals enrolled as a member of the Network revealed similar responses to the previous evaluation undertaken in 2013 (Figure 1). The most popular reason for being a Network member amongst respondents was ‘To get regular information’ (n=109, 90.1%), a substantial proportion indicated that they were members for collaboration purposes (n=58, 47.9%) or because of the ease of joining the Network (n=46, 38.0%). Over one in six respondents stated they 10 were members of the Network as they felt it was beneficial for their career prospects (n=18, 14.9%).

100 90 80

2013

2017

60 50 40 30 20 10 0

To get regular information

To make links with other people in my

Easy to join and no big commitment

Good for career

Figure 1. Reasons for being a Network member The most popular response for reasons being a member was to get regular information, and when questioned which of the services provided by the Network were used by members for this purpose; over three quarters of respondents stated that the information contained within the Ebulletin(s) (n=85, 78.7%) and Website (n=84, 77.8%) helped to inform their professional practice (Figure 2). The information contained within presentations at conferences (n=49, 45.4%) and/or seminars (n=44, 40.7%) informed the professional practice of over two fifths of respondents, whilst one in five respondents (n=23, 21.3%) indicated that the learning from road shows informed their professional practice*. *However, as discussed in a later section of this evaluation (Section 3.5), not all respondents to the questionnaire had attended a conference, seminar or road show event.

Ebulletin

78.7

Website

77.8 45.4

Conferences

40.7

Seminars Road Shows

21.3 Proportion of respondents (n=108, %)

Figure 2. Network services accessed by members to inform professional practice Another important consideration with regards to the information provided by the Network is the wider sharing and contribution to this information. Members were asked to describe how they currently identify as a Network member and how they would like to identify as a Network member from one of three options (Figure 3).

The three options were; 1) Passive Recipient (Just take and make use of information for myself) 2) Active Recipient (Seek out, use and share Network information) 3) Active Contributor (Provide information to the Network e.g. news items, articles, practice examples). Currently, only one in ten respondents (n=12, 9.9%) identify themselves as an ‘active contributor’ and the remainder of respondents equally identify themselves as either a ‘passive recipient’ or ‘active recipient’. Somewhat encouragingly, a substantial proportion of respondents (n=17, 30.9%) would like (or desire) to move away from this ‘passive recipient’ category into one of the ‘active’ categories.

}

Portion of Respondents (n=121, %)

100 90 80

Current role

Desired role

50 40 30 20 10 0

Passive Recipient

Active Recipient

Active Contributor

Figure 3 Current identified and desired self-determined statuses of Network members (* denotes a significant difference between current and desired roles.) To provide a platform to allow members to become more ‘active’ within the Network, there was a question that asked which of four online platforms the members would rank in order of preferred method to engage with their peers (Figure 4). The four choices were; (i) Twitter, (ii) Facebook, (iii) LinkedIn or (iv) Forums. The most preferred method for engagement was using either the forums on the Network website or through the Network Facebook page. Respondents ranked LinkedIn lowest in terms of ‘most preferred’ and highest as the ‘least preferred’ option, clearly demonstrating a reluctance to engage via this method.

90 80 70 60 50 40

Most Preferred

Least Preferred

20 10 0

Forums

Facebook

Twitter

Figure 4. The most and least preferred options of engaging with peers within the Network 3.3 Perspectives on Website (Members and Stakeholders) The first question of interest to members was to find out how often they access and use the Network website. Over a third of respondents (n=41, 38.0%) access the website at least once a month, whilst a further third of respondents indicated that they access the website every few months (n=35, 32.4%) and a quarter of all respondents access the website only once or twice a year (n=26, 24.1%). Although, one in twenty respondents (n=6, 5.6%) stated that they had never accessed the website. A series of similar questions to those posed in 2013 were again asked in this evaluation (2017) to gauge the member opinions on the new Network website (Table 2). The questions were focussed around the user friendliness of the website and also about the information contained within the various pages of the website. Compared to 2013, members were less likely to describe the content as ‘always’ relevant, and although the responses for reliability were mainly positive, more members in 2013 referred to content as ‘very reliable’. Some of the areas of the website that could potentially be improved upon [in the opinion of members] were the visual appeal, navigation and how ‘up-to-date’ content is, more than one in three respondents described these as ‘moderately’ appealing, easy (to navigate) and up-to-date, respectively. Some suggestions by members to improve these areas included a “less busy” or “cluttered” home page, several members commented that the scrolling/carousel moves too quickly, whilst other members remarked that text layered over images is sometimes difficult to read. Some members also asked for “news” to be a more prominent feature on the home page. To help improve navigation, suggestions included an easier search function or the addition of an A-Z list of topics.

Table 3. Member views on website usability and content Website Content

(Appealing) Extremely Very Moderately Slightly Not at all

Visual Appeal 2013 2017 n=114 n=101 6 (5.3) 7 (6.9) 43 (37.7) 42 (41.6) 55 (48.2) 43 (42.6) 8 (7.0) 7 (6.9) 2 (1.8) 2 (2.0)

(Relevant) Always Often Sometimes Slightly Never

Relevance 2013 2017 n=111 n=100 41 (36.9) 19 (19.0) 42 (37.9) 51 (51.0) 26 (23.4) 27 (27.0) 1 (0.9) 3 (3.0) 0 (0.0) 0 (0.0)

(Clear) Extremely Very Moderately Slightly Not at all

Clarity 2013 2017 n=108 n=100 19 (17.6) 16 (16.0) 60 (55.6) 56 (56.0) 28 (25.9) 25 (25.0) 1 (0.9) 3 (3.0) 0 (0.0) 0 (0.0)

(Easy) Extremely Very Moderately Slightly Not at all

Navigation 2013 2017 n=111 n=101 15 (13.5) 11 (10.9) 56 (50.5) 48 (47.5) 36 (32.4) 35 (34.7) 2 (1.8) 4 (3.9) 2 (1.8) 3 (3.0)

Extremely Very Moderately Not always Not at all

Up-to-date 2013 2017 n=110 n=101 15 (13.7) 9 (8.9) 70 (63.6) 54 (53.4) 24 (21.8) 34 (33.7) 2 (1.8) 3 (3.0) 0 (0.0) 1 (1.0)

% Change +1.6 +3.9 -5.6 -0.1 +0.2

% Change -17.9* +13.1 +3.6 +2.1 --

% Change -1.6 +0.4 -0.9 +2.1 --

Reliability 2013 2017 n=109 n=99 Very reliable 56 (51.4) 34 (34.3) Reliable 53 (48.6) 64 (64.7) Not reliable 0 (0.0) 1 (1.0)

% Change -2.6 -3.0 +2.3 +2.1 +1.2

% Change -4.8 -10.2 +11.9 +1.2 +1.0

% Change -17.1* +16.1* +1.0

Data represented as number of respondents and percentage of respondents in parentheses, * denotes a significant difference between 2013 and 2017 responses. However, it should be noted, that half of the respondents to the 2017 questionnaire were not members in 2013 when the previous evaluation was undertaken and direct comparisons between websites should be interpreted with caution. The advisory group and stakeholder interviews included a question related to the new website in relation to the previous separate websites. Five of the stakeholders were members of a previous network and could therefore compare like-for-like the new design to the old design. The views of these stakeholders were overwhelmingly positive and praised how modern and user friendly the new website was compared to the previous websites. Besides information on specific topics, the Network website also contains a number of different features (Figure 5). The questionnaire revealed that most popular feature [as accessed by members] was the ‘Events’ page, where eight out of ten respondents (n=57, 80.3%) stated they had used this feature. Around three in ten respondents had accessed the ‘Past events’ (n=22, 31.0%) or ‘Shared practice database’ (n=21, 29.6%) features. Despite the forums being the most preferred method of engaging with peers (Figure 4), less than one in four respondents (n=17, 23.9%) had actually visited

the pages on the website. Website analytic data also confirms that for the period between July and October 2017 these member responses, especially for accessing information on ‘Events’ are representative of all member website usage (Appendix A1).

80.28

Events

30.99

Past Events

29.58

Shared practice database

26.76

Consultations

23.94

Forums Podcasts

7.04 Proportion of respondents (n=71, %)

Figure 5. Website features accessed by members The Network website now also includes new features such as ‘Your Area’ which was designed to give members a more tailored experience of the Network website. Of the respondents, just over one third (n=34, 36.6%) had used the ‘Your Area’ feature, however there was some useful feedback provided to improve the ‘Your Area’ feature. The main theme of the feedback was focussed on improving the local context from both a local public health team and available services perspectives. The website also provides members the facility to include and share information for wider dissemination, only one quarter of respondents had used this feature. This feature could help more of the members to become ‘active contributors’, but the facility needs to be better promoted as 18 respondents were not aware this was possible. Another important consideration in relation to the information contained within the Network website is the impact of this information on the day-to-day roles of the members (Figure 6). The information on the website predominantly increased knowledge, either the members own knowledge (n=84, 83.2%) or increasing the knowledge of colleagues/peers through the sharing of information (n=69, 68.3%). The Network website has also enabled members to make links with other practitioners (n=35, 34.7%), although this contact has not necessarily led to further collaborations (n=19, 18.8%). Generally, the impact of the Network website on the day-to-day role of members in 2017 is comparable to 2013, however there is a tendency for less members to state that they feel that the information has ‘improved the way that you deliver services’ in 2017 (n=24, 23.8%) compared to 2013 (n=38, 35.2%).

Proportion of respondents (n=101, %)

100 90 80

2013

2017

60 50 40 30 20 10 0

Increased own knowledge

Enabled sharing of useful information

Enabled links with other practitioners

Changed the way interpret and apply health policy

Improved delivery of services

Led to further collaborations

Figure 6. Impact of the Network website information on member day-to-day roles 3.4 Perspectives on Ebulletin (Members and Stakeholders) The vast majority of respondents (n=92, 97.9%) recalled that they received the monthly Ebulletin and of these over eight in ten (n=76, 80.8%) indicated that they read the Ebulletin most of the time. Only a small proportion (n=4, 4.3%) of respondents indicated that they ‘hardly ever’ read the Ebulletin. Half of the respondents (n=47, 51.1%) forward the Ebulletin to their colleagues for interest and wider readership, whilst only a very small proportion of members (n=3, 3.3%) print out the Ebulletin to display in their workplace. The Ebulletin typically consists of five sections; 1) Spotlight (Monthly themed feature) 2) The Grapevine (News on members’ activities) 3) On the Spot (Focus on Network member) 4) News round-up (By topics) 5) What’s going on? (Events list for the month) Members were asked which of these five Ebulletin sections that they found the most useful and/ or interesting (Figure 7). Similar to the responses from the Website question (Figure 5), the most useful section was again related to events with three in four members (n=67, 75.3%) indicating that this section of the Ebulletin was one which they found useful. Other sections of the Ebulletin that proved useful and/or informative amongst members were the Spotlight and News round-up features which were chosen as such by six (n=54, 60.7%) and seven (n=63, 70.8%) in ten respondents, respectively. When asked if there was anything that members would like to see in the Ebulletins, the majority of members were happy with the current content. There were some suggestions for additional content that could mostly be themed as either research (information on grants, and the latest research and evidence) or public involvement (i.e. information on young people by young people or the views of the public on current public health issues).

75.28

Whats going on in

70.79

News Round-up Spotlight

60.67 37.08

The Grapevine

25.84

On the Spot

Proportion of respondents (n=89, %) Figure 7. Sections of the Ebulletin most useful and/or interesting amongst members Each month prior to the publication of a specific Ebulletin, members are emailed and actively encouraged to contribute to the upcoming edition if they would like to. Of the respondents, 16.3% (n=15) had previously submitted to the Ebulletin; whilst a significant proportion had not previously contributed, over half of the respondents (n=48, 52.2%) indicated that they would be interested in submitting content for a future issue. Table 4. Member perspectives on the presentation and content of the monthly Ebulletins Ebulletin Content Layout

Excellent Good Poor Very Poor

2013 n=99 27 (27.3) 69 (69.7) 3 (3.0) 0 (0.0)

2017 n=93 27 (29.0) 59 (63.5) 7 (7.5) 0 (0.0)

Excellent Good Poor Very Poor

Relevance 2013 2017 n=96 n=92 29 (30.2) 10 (10.9) 63 (65.6) 67 (72.8) 3 (3.1) 14 (15.2) 1 (1.1) 1 (1.1)

% Change +1.7 -6.2 +4.5 --

Excellent Good Poor Very Poor

Usefulness 2013 2017 n=98 n=93 36 (36.7) 22 (23.7) 60 (61.2) 68 (73.1) 2 (2.1) 2 (2.1) 0 (0.0) 1 (1.1)

% Change -13.0* +11.9 -+1.1

Clarity % Change -19.3* +7.2 +12.1* --

Excellent Good Poor Very Poor

2013 n=98 42 (42.8) 53 (54.1) 3 (3.1) 0 (0.0)

2017 n=92 19 (20.6) 69 (75.0) 4 (4.4) 0 (0.0)

% Change -22.2* +20.9* +1.3 --

Data represented as number of respondents and percentage of respondents in parentheses, * denotes a significant difference between 2013 and 2017 responses.

Members have already stated that along with the Website, the information contained within the Ebulletin helps to inform professional practice. It is therefore essential that the content within the Ebulletin is useful, relevant, clear and displayed in a format that is easy to read and understand. Members were quizzed on these four elements and how well the Ebulletin meets these factors and also compares these findings to the previous evaluation (Table 3). Overall, there is a positive reception [from members] on the Ebulletin but as with the Website there are similar observations with regards to relevance of the content. Members are now less likely to describe the relevance of the Ebulletin content as ‘Excellent’ and more likely to state the relevance as ‘Poor’, albeit in small numbers. Although the Usefulness and Clarity of the monthly Ebulletins are well received, members are more likely to describe these factors as ‘Good’ rather than ‘Excellent’ compared to the newsletters that were circulated via the three previous networks. 3.5 Events Of the questionnaire respondents, 62 individuals indicated that they had attended a Network event in the last three years. Of these half were made aware of the event they attended through an email (n=31, 50.0%) and over four in ten (n=26, 41.9%) recalled hearing about the event in an Ebulletin. Around three in ten were made aware of the event through a colleague (n=20, 32.3%) or the Network website (n=19, 30.7%). Overall, the events appear to be well received by members and substantial proportions of respondents described events as informative (n=43, 69.4%), useful (n=38, 61.3%) and/or engaging (n=34, 54.8%). From the evaluation forms and associated ‘Wordles1’ that are produced, attendees also found the events to be a positive experience with ‘interesting’, ‘informative’ and ‘worthwhile’ all popular inclusions in the produced ‘Wordles’ (see Appendices A2 and A3 for examples). In addition, one consistent observation from the evaluation forms that were completed at the events, the majority of attendees indicated that they were not currently member of the Network. In subsequent weeks following an event, there has been numerous occasions where the network observes a rise in new members; however, the events do appear to be a missed opportunity to enrol individuals as new members. An important consideration with regards to events are the reasons why members do not or are unable to attend. Members who not attended an event in the past three years were asked the reasons that prevented their attendance (Figure 8). The most prominent barrier to preventing members from attending events was that were too busy (n=24, 37.5%), and a quarter of respondents (n=16, 25.0%) stated that they were simply unaware of events. Other reasons (n=17, 26.6%) that were cited by members as reasons for not attending events included events not being directly relevant to current job role, and being a relatively new member and not yet having the opportunity to attend. Location and diary clashes have also acted as barriers to members attending events. The Network has recently introduced live streaming to events and although many of the respondents had not previously used this facility, almost two thirds (n=35, 62.5%) would be interesting in ‘joining’ an event through live streaming in the future.

Event attendees are asked to provide one word to summarise how they felt about the event. These words are then inputted into Wordle (www.wordle.net) which generates word diagrams that give greater prominence to words that appear more frequently in the text. 1

Too busy

37.5

Other

26.6

Not Aware

Not Allowed

9.4

Travel Costs

7.8

Not Interested

4.7 Proportion of respondents (n=64, %)

Figure 8. Barriers and reasons for members not attending an event To help address the issue of location being a barrier to attend events, the Network does undertake an annual ‘roadshow’ series which travels to a number of locations throughout Wales. Of the 61 respondents who indicated that they have attended an event in the past three years, less than half (n=26, 42.6%) had attended a ‘roadshow’. When questioned as to what members would like the ‘roadshow’ events to encompass the most popular response was to provide information on local [to them] initiatives. Members felt that local information could also provide a platform for better co-production in the area. Information on national initiatives and training opportunities were also popular options with the members (Figure 9).

Local Initiatives

80.8

National Initiatives

69.2

Training/Formal CPD

65.4

Information about the network Other

50 11.5 Proportion of respondents (n=26, %)

Figure 9. Member choices for inclusion at future ‘roadshow’ series 3.6 Overall Performance The final question of interest on the wider influences of the Network was to gauge the opinions of members on the impact of a number of key activities (Table 5). Members were asked to consider their level of agreement towards a number of statements that were based on the Network aims and objectives. Similar opinions were also gathered in the previous Network evaluation (2013). On the whole, the responses towards the impact of the Network activities were positive, however compared to the previous evaluation there was a tendency for respondents to ‘agree’ with statements rather than ‘strongly agree’ with the statements. Two statements that were somewhat different to the responses in 2013 concerned how ‘up-to-date’ members felt on their topic area and the opportunity to collaborate. Members were more likely to ‘disagree’ that the Network kept them ‘up-to-date’ and therefore significantly less likely to ‘strongly agree’ with this same statement. With regards to collaboration opportunities, members were less likely to ‘strongly agree’ or ‘agree’ and significantly more likely to ‘disagree’ with this statement.

In addition, the current evaluation also asked members to consider the following statement ‘The Network helps me to understand the evidence that informs my practice’. The majority of respondents had a positive reflection to this statement, where 58.7% (n=64) of respondents agreed and 11% (n=12) strongly agreed. A small proportion (2.8%, n=3) ‘strongly disagreed’ with the statement and the remaining respondents (27.5%, n=30) ‘disagreed’ with the statement. Table 5. The views and opinions of members on some of the Network’s key activities Member Perceptions on Network Activities Strongly Agree

Agree

2013

2017

% Change

2013

2017

% Change

2013

2017

The Network keeps me up-to-date on developments within my topic area

32 (30.2)

15 (13.8)

-16.4*

69 (65.1)

73 (67.0)

+1.9

5 (4.7)

20 (18.3)

The work of the Network helps me to help the public or my clients

19 (18.1)

9 (8.3)

-9.8*

74 (70.5)

78 (71.6)

+1.1

12 (11.4)

20 (18.3)

35 (33.3)

26 (23.9)

-9.4

65 (61.9)

77 (70.6)

+8.7

4 (3.8)

5 (4.6)

The work of the Network has helped me to collaborate with other people

16 (15.4)

10 (9.2)

-6.2

65 (62.5)

53 (48.6)

-13.9*

20 (19.2)

43 (39.4)

The Network provides a good platform for sharing practice and promoting health

24 (23.5)

25 (23.0)

-0.5

62 (60.8)

75 (68.8)

15 (14.7)

7 (6.4)

The work of the Network helps me to apply policy to my day-to-day work

16 (15.5)

-5.4

61 (59.2)

55 (50.4)

-8.8

25 (24.3)

39 (35.8)

The work of the Network has no effect on how I do my job

7 (6.7)

-1.2

18 (17.1)

31 (28.4)

+11.3

55 (52.4)

51 (46.8)

The Network helps to raise awareness of current issues and trends

11 (10.1)

6 (5.5)

Strongly Disagree

Disagree

+8.0

% Change

2013

2017

% Change

0 (0.0)

1 (0.9)

+0.9

+6.9

0 (0.0)

2 (1.8)

+1.8

+0.8

1 (1.0)

1 (0.9)

-0.1

+20.2*

3 (2.9)

3 (2.8)

-0.1

-8.3*

1 (1.0)

2 (1.8)

+0.8

+11.5

1 (1.0)

4 (3.7)

+2.7

-5.6

23 (23.8)

21 (19.3)

-4.5

+13.6*

4.0 Key Questions (Discussion) 4.1 Performance against Aims and Objectives Collating the views of the stakeholders (including advisory group members) who were interviewed, alongside the member questionnaire responses it is important to consider how well the Network is currently performing against their own aims and objectives. 4.1.1 Educate and inform the wider public health workforce on developments related to public health practice in Wales; The views of the stakeholders were positive overall and most, if not all agreed that the Network currently meets this objective. In particular, the news section of the website and the monthly themed Ebulletin helped to provide information to inform the wider public health workforce. These stakeholder views are complemented by the member views (Figure 2) which indicate that information to inform professional practice is mostly gathered from the Ebulletin and website. However, whether this information translates to impact on professional practice is questionable, given that 33 respondents disagreed that they understood how this evidence informs their own practice (page 17). Although, not discussed in great detail within this evaluation (one question on the features accessed by members on the Website, Figure 5), the Network has recently launched a series of podcasts as a new method to distil and inform members of new developments. This had not gone unnoticed by some stakeholders who were impressed with this new approach. Stakeholders also raised the question as to whether the Network needed to be broader than public health practice. For example, should members be informed of the Well-being of Future Generations Act and the five ways of working? To answer this question quickly, the Network has previously delivered an event on the Well-being of Future Generations Act and in the upcoming months, the Network is embarking on a roadshow series in partnership with the (Public Health Wales) Health and Sustainability Hub. There is also now a dedicated â&#x20AC;&#x2DC;Community of Practiceâ&#x20AC;&#x2122; area of the website for the Well-being of Future Generations Act that is signposted on the front page. 4.1.2 Highlight and showcase examples of good practice in Public Health delivered throughout Wales; The views from stakeholders towards this aim were mixed, a good proportion of stakeholders felt that the Network was achieving this aim but another significant amount also felt that this was an area the Network could improve. Stakeholders in general did make reference to the shared practice database that is on the website and thought that more could be done to raise the awareness of this resource. This suggestion is also justified by the member questionnaire where less than a third of respondents had accessed the shared practice database (Figure 2). Stakeholders also commented that previously there were awards for good practice and it might be good to re-introduce this, to both encourage submissions

4.1.3 Encourage interactivity and the exchange of knowledge and ideas amongst the public health workforce in Wales; Primarily interactivity and the exchange of knowledge and ideas happens during events, but although the Network provides the platform at events it relies on the engagement of members to exchange these ideas. Stakeholders suggested the forums were a good option to exchange knowledge and ideas outside of events. Whilst, acknowledging that the forums were currently underutilised, stakeholders suggested using social media engagement more to encourage the exchange of knowledge and ideas. Deliberate tactics using the social media channels asking for involvement could be an alternative strategy to use in the future. There is desire from members to become more ‘actively’ involved and this will be discussed in greater detail in Section 4.3. 4.1.4 Facilitate opportunities for debate and discussion about contemporary public health issues amongst the various stakeholders within Wales and beyond; This aim and objective is probably the one area for the Network that has the most room for improvement, but, also encouragingly the area that stakeholders and to some extent members also have the most useful suggestions for improvements. There are two clear and obvious opportunities for debate and discussions currently provided by the Network and these are the events arranged by the Network on contemporary topics and also the forum facility on the Network website. Discussions at events such as conferences and seminars provide the platform for discussions through the facilitated Q&A sessions but whether these discussions are continued following the event is unclear. The forum on the website also provides an opportunity for discussion and debate but at present this resource is underutilised and needs to be revisited. A new approach may also promote debates and discussions and stakeholders suggested a number of ideas to do this, but also acknowledged the requirement for members to engage which is something that have not been done in great detail up until now. The suggestions to ‘refresh’ the opportunities for debate and discussions were; (i) webinars, (ii) snap surveys and (iii) promotion of the forums through the various Network social media channels. 4.1.5 Promote opportunities for collaboration amongst stakeholders on new and emerging public health issues; The general observation from stakeholders is that the opportunity for collaboration is primarily at events such as conferences, seminars and road shows. However, stakeholders did comment that some of these events were not that well attended which would mean that the opportunity for collaborations is limited. From the questionnaire responses, collaborations is also low on the list of impacts on day-to-day roles, although enabling links with other practitioners does feature a little higher (Figure 6). Collaborations do require effort from individuals to be a success and despite the Network providing opportunities for collaboration at events, an alternative method to maintain and build on these links needs to be explored. The obvious choice is the use of the forum on the website, as discussed previously this is currently underutilised but remains a popular resource with members.

4.1.6 Inform and promote local, national and international public health related policy, practice and evidence; Overall, the stakeholders agreed that the Network does meet this aim and objective, again primarily through the information contained within the website, circulated via the monthly Ebulletin and/or presented at various events. As with most of these aims and objectives, suggestions for improvements have been proposed. With regards to national policies there could be a clear and explicit section on the website for this purpose which could contain the latest government policies for example. Some policies and practice are therefore unique to Wales and whether wider UK and international practice and evidence would be applicable to Wales was also questioned. Finally, as previously discussed, respondents to the member questionnaire suggested providing information on local initiatives at the road show events. 4.1.7 Increase awareness of the work of Public Health Wales to the wider workforce. To some extent, the stakeholders did feel that the Network met this aim, through the previously identified effective methods of the website, Ebulletin and events. The website and social media channels are globally accessible and there is evidence that these resources have been accessed by individuals in countries all over the world. However, again this aim was an area where it was thought the Network ‘could do better’ and some suggestions for improvements were helpfully provided. For example, improvements for the website included a clearer association with Public Health Wales branding and related services, for example, the Public Health Outcomes Framework (PHOF) tool from Health Intelligence which would be an excellent resource for Network members. One further suggestion was to rely less on members to share this information but engage with policy makers. Policy makers who engage and share the work of the Network with their associated partners would also enable ‘non-traditional’ partners to engage and therefore could lead to more successful collaborations in the future. 4.2 What are the member perspectives of the new website and Ebulletin? The website and Ebulletin were repeatedly reported by members as the main source to access regular information. Although these member perspectives have been reported in previous sections (Sections 3.3 and 3.4), it is important to re-emphasise and elaborate further on some of the main findings to help develop these member valued resources. 4.2.1 New website The website is one of the first contacts that potential members would have, the enrolment process takes place using the website. It is essential that one of the first impressions of a ‘new’ member is a positive one. The stakeholders interviewed did comment that compared to the old websites, this new website is a much needed improvement and praised the new ‘modern’ design.

The most accessed feature of the website is the section that contains information related to events (Figure 5), with past events and the shared practice database also sections that are frequently accessed by members. One interesting observation from the website analytics (Appendix A1) is that the topics that were accessed the most were related to ‘nutrition’, ‘mental health’, ‘sexual health’ or ‘physical activity’, all of which were current topics of the previous networks. This suggests that the topics of interest on the website are those historically linked with the Network. When considering the features accessed on the website it is impossible not to discuss the accessibility and usage of forums. Forums were selected as the ‘preferred’ method to engage with peers (Figure 4), and despite this observation, less than one in four respondents (n=17, 23.9%) had actually visited the pages on the website (Figure 5). Whilst this could be considered an engagement issue, the forums sit within the website and therefore have implications for both the website and engagement. Forty-one respondents did indicate that they would be willing to join or create a forum/community of interest and the suggestions for such topics included working age (employee) health and wellbeing, childhood obesity, housing and health, health policy, and substance misuse. These findings again suggest that members are willing to engage and user discussions are an important first step to help facilitate this desire of the members. Contained within the member responses were a number of grievances that they had with the current homepage of the Network. These grievances have been previously discussed but the following ‘changes’ should be considered; (i) Making the home page appear less busy, a refresh of the homepage may also help to grab the attention of members if they realise something has changed; (ii) Slowing down the carousel so that visitors to the website can digest the information; and (iii) Making both the news and shared practice features on the website more prominent so navigation can be improved. 4.2.2 Ebulletin The Ebulletin was the most frequented source of information accessed by members who responded to the questionnaire (Figure 2). In addition, the vast majority of respondents to the specific question also recalled receiving a copy of the Ebulletin each month (n=92, 97.9%). Therefore, it could be considered that at present, the Ebulletin is the feature of the Network which has the greatest potential reach. There was also a considerable interest from questionnaire respondents who had not yet contributed to an Ebulletin to submit content to a future Ebulletin issue (n=48, 52.2%). The chosen topic of the Ebulletin could also influence whether a member was to submit content to an Ebulletin on the basis that their contribution would fit the remit of the upcoming issue. One option to encourage submission and facilitate contributions could be an Ebulletin ‘Special Member Editions’. This would provide the opportunity for those members who want to submit something to do that and might also encourage new collaborations, especially considering the potential reach and readership of the Ebulletin. As with the website, the section of the Ebulletin which generated the most interest was ‘What’s going on?’ which features information about upcoming events. These observations demonstrate that the content that is most valued by members appears to be information which provides details about upcoming events.

The Ebulletin remains valued by members, however compared to 2013, the clarity of the Ebulletin has slipped from an ‘Excellent’ standard to a ‘Good’ standard. This may appear trivial, but also demonstrates the high standards expected of the Ebulletin and Network outputs. There may be a number of reasons why members opinions have changed towards the clarity of the Ebulletin and these were not captured by the questionnaire. A supplemental piece of work from this evaluation could be to further explore this concern and help to develop the Ebulletin. One consistent comment that appeared through the ‘free text’ element of the questionnaire and also highlighted by Stakeholders was to do with ISSUU and the associated access (ISSUU is blocked on some NHS servers) and readability of the Ebulletin. When distributing the Ebulletin, the ‘high quality’ version is accessed via ISSUU, and the ‘low quality’ version is a PDF version of the Ebulletin which would be welcomed by some members (e.g. to print out and read). At present, there is no explicit reference that the ‘low quality’ version of the Ebulletin is a PDF, a simple sentence when distributing the monthly Ebulletin would highlight this to Network members and perhaps further increase readership. 4.3 What does the Network need to do differently to increase engagement? The issue of member engagement is certainly the most obvious area that needs to be addressed from this evaluation. Promisingly, from the member respondents there appears to be an appetite to become more engaged with the Network. This is evident in the numbers who have expressed that they would like to transition from a ‘passive’ member to a more ‘active’ member (Figure 3). Some examples of how to facilitate this have already been discussed previously in relation to opportunities with the forum (website) and Ebulletin. Events, especially conference and seminars are also main sources of information for members (Figure 2), and are also a key facilitator in member collaboration as made apparent by Stakeholder interviews. One of the most interesting findings from this evaluation process was that from the previous event feedback, at least half of delegates who attended (and completed the feedback forms) were not currently a member of Public Health Network Cymru. The membership numbers do increase in the weeks that follow an event, but whether that includes all those potential ‘new’ members remains unclear. Therefore, the events do appear to be a missed opportunity to enrol individuals as new members and further grow and extend the reach of the Network. There were a number of barriers to members not being able to attend an event of any type in the past three years (Figure 8). The main barrier was that members reported that they were ‘too busy’ with some Stakeholders commenting in their interviews that their availability can be filled weeks and sometimes months in advance. One sensible option to reduce these diary clashes could be to circulate ‘Save the Date’ fliers a couple of months in advance of the event with the assurance of further details to follow in due course. This method could potentially allow more and a greater diversity of members to attend these events which in turn, will allow for a better exchange of ideas. One of the key ‘other’ barriers to members attending events alongside members being ‘too busy’ was cost/budget restraints. The possibility of Network events receiving some formal CPD accreditation may encourage more members to attend and also help to persuade employers (line managers etc.) to release their staff who are members to attend. The Network now has the capability to livestream events (usually on Twitter) so that members unable to attend in person can watch the presentations and also contribute to the conversations/ Q&A sessions. Many of the questionnaire respondents indicated that they had not previously used this ‘livestreaming’ facility, however most indicated that they would be interesting in ‘joining’ an event through live streaming in the future.

Another feature that has been recently introduced is podcasts which would help to provide a snapshot of events and contain interviews with some of the key speakers. From the website usage analytics (Appendix A1), the podcasts page is one of the most accessed by members which suggests a welcome inclusion to the suite of services offered by the Network. Both online streaming and podcasts help to contribute to an increased social media presence by the Network. Social media is something that is continually progressing and it is paramount that the Networkâ&#x20AC;&#x2122;s presence on Social Media also continues to evolve to meet this developing landscape. An increased social media presence will also help to â&#x20AC;&#x153;facilitate opportunities for debate and discussion about contemporary public health issues amongst the various stakeholders within Wales and beyondâ&#x20AC;? (Section 4.1.4). 4.4 Performance compared to previous evaluation (2013) Considering the changes to the Network and also the characteristics of the member respondents to this questionnaire (over half indicating that they were not a member of a previous network), like for like comparisons between this evaluation and the one in 2013 are difficult and any findings should be interpreted with caution. However, it is important to consider an overview of responses from this evaluation and the previous one in 2013. Most of the responses from the members in this evaluation are comparable with regards to member behaviours (e.g. Figure 1), and overall impact (e.g. Figure 6), for which the Network should be applauded. Although, there is a tendency that the performance of the Network to specific content and timeliness of this content have slipped compared to 2013 (e.g. Table 3, and Table 5). Timeliness and relevance (of content) could be explained by an increase in topics that are now covered by Public Health Network Cymru and ensuring that everything posted on the website, included in Ebulletins or presented at events is relevant to every member is a near impossible task. Nevertheless, some of the specific issues already discussed with regards to clarity and reliability in relation to the website and Ebulletin (Sections 4.2.1 and 4.2.2), and collaboration/engagement (Section 4.3) can be improved upon.

5.0 Conclusions 5.1 Summary of Main findings Overall, the work is the Network is positively received by both the members and the stakeholders. It is important to appreciate that the Network has upwards of 1000 members and these numbers increase each month. Therefore, in Wales especially the potential of the Network cannot be underestimated. Most of the aims and objectives of the Network are being met to some extent from collating the questionnaire responses by members and also stakeholder interviews. However, also quite clearly there is some room for improvement in some areas such as facilitating debates and discussions which are actively engaged with by members, helping to provide future collaborations and if possible, raising the awareness of the work of Public Health Wales to ‘non-traditional’ partners. Engagement is an issue with the network and this is clear from the current underutilisation of the forum resources which was requested by members prior to the construction of the new website. Engagement, or lack of is also evident from the number of respondents to the evaluation questionnaire and also to event evaluations. Somewhat promisingly there does appear to be an appetite for engagement through more members becoming ‘active’ (recipients/contributors) or by submitting articles to future Ebulletins. Further developing a social media presence through livestreaming events and circulating podcasts should also help to promote engagement amongst members. Both the Ebulletin and website are the two main sources where members access information to help inform their professional practice. The contributions of the Ebulletin and website have not gone unnoticed by stakeholders who remarked how well these resources help to share information about local and national developments and best practice. Information on upcoming events appear to be of greatest interest to members, and these respective sections of both the Ebulletin and website are the most frequented by members. Events are also a focal point to encourage collaboration and engagement between Network members. The main barrier to attending the events of the Network, were that members were ‘Too Busy’ to attend. One option to help alleviate this barrier could be to circulate ‘Save the Date’ emails to members with a date and location with further details to come. Finally, it appears that transitioning from three topic specific networks to one universal network that encompasses all public health related information may account for some of the different member respondents from 2013. The drop in proportions who indicated ‘excellent’ for timeliness, relevance and usefulness could all be explained by how the Network may not be so specific to their previous interests. More topics also requires an additional workload to continually upload resources and update the website. 5.2 Limitations This evaluation has attempted to be as comprehensive as possible and provide an up-to-date insight into the work and reception of the Network, however, there are a number of limitations that should be acknowledged.

The first limitation is the (small) number of member respondents. Despite the questionnaire being ‘live’ for two months only a small percentage (~10%) of members actually responded and it is uncertain whether these questionnaire responses are reflective of the views of all Network members. Over half of respondents to the questionnaire were not a previous member of a network so like for like comparisons between this current evaluation and 2013 should be treated with caution. Even though all stakeholders were invited (on numerous occasions) to participate in a short questionnaire not all stakeholders accepted or responded to the invitation for interview. The most notable absentee from the stakeholder list is a representative from Welsh Government. 5.3 Recommendations The work of the Network is currently well received by members and stakeholders, although there are some suggestions for improvement. Therefore, from undertaking this evaluation there are a number of recommendations to be considered to help improve further the work being undertaken by the Network and help the Network to better achieve their own aims and objectives. 1) The Network should continue to grow and maximise opportunities to ‘recruit’ new members such as at events. Not only is new growing numbers of members important but also considering whether new Stakeholders need to be recruited in the current political limelight. A re-established link with a representative from Welsh Government should be attempted and links with a representative leading on the Well-being of Future Generations (Wales) Act and within Cymru Well Wales should be considered. 2) There is an obvious desire for members to themselves become more ‘active’ and this is something that the Network should capitalise on. The forum needs to be addressed (individuals not engaging but list the use of a forum highly in terms of getting good quality information and collaboration). One possibility could be to consult with members and ask why they (forums) were not used; and secondly ask members how they would like to be able to open up discussions. Stakeholders have already suggested alternative methods for engagement and it is important to gauge whether these options are reciprocated by members, especially given many of the Network aims and objectives are reliant on member engagement. 3) Both the website and the Ebulletin are valued services by members and also currently have by far the greatest reach to members. These services should continue to be produced and developed where possible, it should also be acknowledged that the information on events is considered of most value to members, and this information should be easily accessible. 4) The events either promoted by or delivered by the Network garner the greatest level of interest from members. The delivery of events should continue with conferences and seminars promoting national programmes and if possible the road show series should showcase examples of local initiatives. This would promote local work and also help to stimulate local collaborations and help to encourage member engagement and exchange of ideas. 5) Livestreaming and podcasts are also important services to continue developing. Both of these options help to extend the reach and learning of events to those members who were unable to attend on the day. As new options for interaction and extending reach are introduced, new options should be offered to members and these new options trialled and evaluated.

6) The possibility of Network events receiving some formal CPD accreditation should be explored. Similar events currently provide CPD accreditation which is compulsory for some employment positions, in addition to providing a benefit for members, CPD accreditation may also encourage attendance. 7) The final recommendation of this evaluation is to consider stopping activities/features/sections that do not appear to be valued highly. For example, the â&#x20AC;&#x2DC;On the Spotâ&#x20AC;&#x2122; section of the Ebulletin was found to be least useful by members (Figure 7), discontinuing this section of the Ebulletin would then allow more focus to be given on the activities highly valued by members and in turn increase the impact of the Network.

Appendices Appendix A1. Public Health Network Cymru Website Analytics (Jul-Oct 2017)

Appendix A2. Example Wordle (Behaviour Change and Addictive Behaviours)

Appendix A3. Example Wordle (Adverse Childhood Experiences: What it means for you)