/AtDSLSMandateResearchReportBinder by Valencia College

VALENCIA COMMUNITY COLLEGE

SLS1122 Mandate for 3‐Prep Students Research Report Achieving the Dream Data Team March 2009

2 SLS Mandate Research Report

SLS1122 Mandate for 3‐Prep Students – Research Report Introduction: SLS 1122, Student Success, has been taught at Valencia Community College since the late 1980’s as a 3‐credit elective course. During most of that time, students were encouraged to take the course through new student orientation and advising recommendations. Through these methods, approximately 30% of the First Time in College (FTIC) students enrolled in the course. Institutional data since the mid 1990’s has suggested a correlation between students who took SLS 1122 and increased persistence rates as measured by enrollment from Fall term to Spring term and from Fall term to the following Fall term. In developing the proposal for the Achieving the Dream initiative, a collaborative process involving faculty, staff and administrators studied promising practices within Valencia and at other colleges that were correlated with increased success rates for FTIC students. The idea was to identify a few practices that when expanded to more students would lead to increased student success. These strategies would be targeted particularly at underprepared students and students of color, whose completion rates are lower than other students. Because research generally supports the benefits of student success courses it was determined that an expansion of SLS1122 would be one of the three strategies implemented through the Achieving the Dream (AtD) Grant Initiative. The first phase of the proposed expansion was the implementation of the SLS1122 mandate for 3‐prep students which required any student testing into prep levels in all three discipline areas, with the exception of EAP students, to take Student Success (SLS1122). In preparation for the expansion of the SLS mandate to include 2‐ prep students, the AtD Data Team facilitated data discussions with targeted groups (Academic Deans, Student Affairs Professionals, SLS Faculty, Statistics Faculty, and Prep. Faculty) in the Fall of 2007 (a breakdown and analysis of these discussions can be found at the end of this report). In December of 2007 the facilitators of the data discussions came to the following conclusions: 1) SLS1122 has a proven track record at Valencia and should be supported as tool for improved student success. Valencia should continue to look for ways to employ both the course, and its content, to help students succeed without impacting the content of the course.

3 SLS Mandate Research Report

2) The data from the impact of mandating SLS1122 on 3‐prep students are currently inconclusive. A reasonable evaluation of the impact of the 3‐prep mandate on student success will require more time and the collection of more and different data. 3) The inconclusive nature of the data that we currently have on the impact of the 3‐ prep mandate does not support the expansion of the SLS1122 mandate to include 2‐ prep students. Based on these findings it was recommended and approved in January 2008 that expansion of the SLS mandate to students testing into two preparatory areas be postponed until Fall 2009 or later and that further study be conducted within specified guidelines in order to better inform the decision. A research proposal was developed by the AtD Data Team and approved by the College Learning Council in Spring 2008. The following is a report on the proposed research to determine if the implementation of the SLS1122 mandate has resulted in changes in overall 3‐ prep student performance. The focus of this research was the SLS1122 mandate for 3‐prep students and not the SLS1122 course. For this reason, our report does not include any assessment of the Student Success course itself or a discussion of its benefits. The Research Hypothesis: Requiring all 3‐prep students to take SLS1122 will increase the overall performance of 3‐prep students. Research Question: Have 3‐prep students shown improvement beyond what they would have been expected to achieve without the SLS 1122 mandate? Confounding Time‐Related Variables: To answer the research question, it was decided to compare the before and after effects of student performance over time. There are, however, many confounding variables operating on the school environment that could cause success measures associated with 3‐prep students to

4 SLS Mandate Research Report

change over time. Confounding variables that could lead to natural fluctuations in performance from year to year and semester to semester include: • • • • •

Local economic conditions that would encourage or discourage community college enrollment, persistence, and retention. Admission standards, marketing strategies, and the general competitiveness of other colleges in our service area could affect the mix and caliber of students who attend Valencia. Our own marketing strategies and evolving mix of programs offered could also affect the mix of students who chose to attend Valencia. Improvements in our college and our community that lead to a better learning environment for the students enrolled at Valencia. The effects of other AtD initiatives at Valencia that affect the performance of those students being studied simultaneously as part of the SLS1122 research.

Research Limitation: Because SLS1122 is presumed to be a benefit to those students who take it, our College required all 3‐prep students to enroll in the course. Therefore, a “controlled experiment” was not possible, and the AtD Data Team elected to study the potential benefits of the student success mandate by selectively choosing a “best comparison” group to the 3‐prep treatment group. Research Groups:

5 SLS Mandate Research Report

Methodology: In spite of population differences between the treatment and comparison group, the effect of time‐related confounding variables should be similar. Our quantitative research data compares the performance time trends before and after the mandate for both groups. Research Validation: The comparison group is understood to be an imperfect control group. It is possible that the time‐related confounding variables could impact the various mandate‐level populations differently. To examine this possibility, one could study the time trends on diverse potential comparison groups to consider how similar these populations track with each other over time, especially before and after the 3‐prep mandate. If the fluctuations in performance over time track similarly among all groups over time, it should strengthen our confidence that the comparison of the treatment and comparison group is valid. Meaningful Improvement For the purposes of data supported administrative decisions concerning the continuation or expansion of student success oriented programs, the determination of “meaningful improvement” would require a balance of all or most of the following: •

Statistically significant improvement in target quantitative measures.

•

Significant improvement relative to a comparative group.

•

Economic efficiency in relationship to the difficulty of improving the success of students.

•

Reflection on the human impact in terms of the goals of the initiative and the mission of the institution.

•

A consideration of faculty / staff perception of benefit versus cost.

•

A consideration of student perception of benefit versus cost.

In consideration of this, we analyzed the following measures:

6 SLS Mandate Research Report

Measures: Trend Analysis Our trend analysis of the research question compares the two research groups in the Fall terms, starting in Fall 2003 and continuing through Fall 2008. This comparison includes three terms before the 3‐prep mandate was implemented (Fall 2003, 2004 and 2005) and three terms after the 3‐prep mandate was implemented (Fall 2006, Fall 2007 and Fall 2008). Our trend analysis considers three measures: Fall to Spring persistence, Fall to Fall persistence, and percentage of college prep courses that were successfully completed (with a grade of A, B, or C) in the specified Fall term. The trend analysis was validated by determining the statistical significance of the change in the measures within each research group. These measures have been broken down by ethnic group in order to track the success gaps that are part of the Achieving the Dream initiative (African‐American, Caucasian, and Hispanic). Persistence is a meaningful measure because it is a correlate of student success in that students cannot make progress towards degree completion if they do not continue enrollment. It is a measure of momentum towards degree completion. Successful completion of college prep courses is a meaningful measure because it is a correlate of mastery of course outcomes and leads to sequenced completion of subsequent courses in the curriculum. Research indicates that completion of the college prep course sequence is an important progression point towards degree completion. The focus of our analysis is the comparison of trend lines before and after the experimental treatment (implementation of the mandate). The table below is a summary of the findings from the trend analysis – a complete breakdown of the data associated with the trend analysis of the effect associated with the SLS Mandate can be found at the end of this document.

7 SLS Mandate Research Report

Trend Analysis – General Conclusions Trend analysis data associated with the SLS Mandate does not paint a clear picture. The mandate does not appear to be doing any harm to students generally and AtD target populations appear to be positively impacted to different degrees. On its own, the trend analysis data does not justify the expanded use of the mandate as a strategy.

•

The data indicate that there is a positive effective associated with the SLS Mandate on Fall‐to‐Spring persistence. The effect appears to be positive for all ethnicities combined and especially for African‐American and Hispanic students (the target populations of the Achieving the Dream Initiative).

•

The SLS Mandate does not currently appear to be associated with any increase in Fall to Fall persistence with exception of Hispanic students – persistence rates for Caucasian and African American student are trending downward. It should be noted that this data only represent two points in time – a more complete picture will emerge with the addition of Fall 2009 data.

•

The SLS Mandate appears to be associated with a positive effect on the completion of Prep courses for all groups except Caucasians. An independent T‐ test suggests that this effect is only significant for Hispanic students.

•

The SLS Mandate, like other strategies, appears to have greater effect on some populations than others. The trend analysis data suggest a greater effect on African American and Hispanic students than Caucasian students and a greater effect on Hispanic students than African American Students. 7

8 SLS Mandate Research Report

Student Focus Groups Student focus groups were conducted on the research question: How do students feel about being required to take SLS1122? This research question was proposed by deans, faculty and staff during the Data Team facilitated discussions in Fall 2007. The attached Discussion Guide for the Student Focus Groups was developed in consultation with Mr. Thor Falk, who owns and operates Falk and Associates, a market research company with a specialization in qualitative research. The Discussion Guide was structured to establish a comfortable atmosphere among the group and then follow a sequenced protocol that would identify when the students became aware of the SLS1122 requirement and what they thought and felt about the requirement. Students mandated into SLS1122 starting in Fall 2006 were invited to participate in the focus groups through a protocol that did not identify the specific topic of the discussion. Separate groups were formed for students who successfully completed SLS1122 and those who did not. Following focus group methodology, focus groups were repeated until the themes revealed in the groups became redundant indicating a saturation point had been reached. Six 90‐minute discussions were held with 3‐prep students who enrolled in a Student Success course between Fall 2006 and Fall 2008. Students had either enrolled in the Student Success course alone, or were enrolled in sections that were part of a Learning Community (LinC). The sessions were held in the Spring and Fall of 2008. The six groups consisted of a total of 38 participants with the following demographic breakdown: 26 females and 12 males; 15 of age 18‐19, 15 of age 20‐24, and 9 of age 25 and up; 22 African American, 9 Hispanic, 8 Caucasian; 16 earned A’s, 13 B’s, 1 C, 6 D’s, 1 I, and 1 W. During the recruitment process students were screened for eligibility and variation in demographics. They were informed only that the discussion would be focused on a topic related to education, that participation was voluntary with a monetary compensation of $50. Student Focus Groups ‐ A Summary of General Student Perceptions and Attitudes about the SLS mandate

Students were generally not aware of prep courses or the SLS1122 requirements prior to choosing Valencia. If they did have prior knowledge, it was not a deterrent for them in making the decision to attend the college.

9 SLS Mandate Research Report

When discussing requirements, most students tended to focus on preparatory course requirements rather than the SLS1122 requirement. However, in the Fall focus groups, students were more familiar with the SLS1122 requirement either mentioning it alone or at the same time as preparatory requirements.

Students were able to resolve negative feelings associated with the requirement to take SLS1122 and move on to stay and take the course.

When asked if everyone should take the course: o Among students that did not successfully complete the course, although they felt the course was beneficial most felt that students should not be required to take it. o Among students that did successfully complete the course or were currently enrolled, the overall feeling was that it was a very beneficial course and everyone should be made to take the course.

Student Focus Groups – General Findings It does not appear that SLS interferes with students’ decision to come to Valencia or to continue on at the college. In their advice to other students, the students stressed that a positive attitude about the course was an important factor. They felt that opposition to the course would be minimized by better communicating the benefits of the course or by “selling” it at various decision points and in language more familiar to incoming students.

Neither the SLS1122 course nor the perceptions prior to taking it were roadblocks to attending the college.

The SLS1122 course was not a roadblock to their success or to continuing on at Valencia.

Consideration of the course and/or the requirement did not come until orientation.

Students who had a better explanation of the course and who had it “sold” to them felt better about having to take it (i.e. the Road Map to Success Scholarship Award was not mentioned prior to taking the course but students felt that would have been a nice incentive).

Most of the students would recommend the course to other students.

10 SLS Mandate Research Report

Those students that were not happy about the requirement felt they already possessed the skills covered in the course.

Students had similar feedback about the content of the course regardless of their success or failure in the course.

A full summary of the Focus Group findings are attached. Cost Efficiency Analysis During 2008, we participated in a Lumina sponsored study called “Making Opportunity Affordable” (MOA) along with 12 other colleges. The purpose of the study was to develop a planning model that allows colleges to analyze and project the cost and benefit of implementation of new student success oriented strategies. Based on the increase in year‐to‐ year persistence of students who enroll in the Student Success course (SLS1122), the resource analysis indicated that an additional 277 students were retained (2006‐07) and based on the resource costs of the Student Success program, this yielded an additional estimated $185,501 after total program costs. Valencia Cost Data on the SLS Mandate and the MOA report are attached. Return on investment analysis on student success oriented strategies in higher education is a particularly challenging endeavor. The MOA project raised more questions than it answered and a second round of national study and analysis is being planned. For Valencia’s purposes the collection and internal analysis of cost data associated with the SLS1122 pre and post mandate was more helpful than the actual MOA report. The facilitated data discussions in the Fall of 2007 raised questions about the financial strain that the expansion of SLS Mandate to include 2‐prep students would place on the college. The completion of the cost template provided by MOA (attached) revealed that the expansion SLS Mandate would not, in itself, be a financial strain on the institution because much of the cost is absorbed by student tuition and the institution benefits from economies of scale. Faculty and Staff Discussions An abbreviated second round of facilitated discussions based on SLS Mandate trend analysis data were conducted in Spring 2009 with various groups (the Learning Council, Academic Affairs Leadership Team, IAC, and AtD Campus Meetings). Reactions to the data were

11 SLS Mandate Research Report

generally positive and translated into support for the continued use and study of the SLS Mandate in its current form. Even though the Fall to Fall persistence data is still incomplete there was a great deal of speculation concerning the meaning and solution to the perceived problem. The majority of the speculation was focused on need to develop strategies focused on student needs during the Spring and Summer terms after the Fall intervention (the SLS mandate) ended. These comments and the desire for a coordinated and sustained focus on the new student run parallel to recommendations coming from the Foundations of Excellence Process. Concern continues to be expressed about the potential long term impact of demographic changes to SLS1122 brought on by the SLS Mandate – a concern that SLS122 will come to be seen as a preparatory course and not college‐level work. The graph below reveals changes in the enrollment mix since the implementation of the SLS Mandate. The current constraints on enrollment growth may exacerbate this trend.

Summary and Recommendations: The research proposal on which this report is based argued that for the purposes of data supported administrative decisions concerning the continuation or expansion of student success oriented programs, the determination of “meaningful improvement” would require a balance of all or most of the following:

12 SLS Mandate Research Report

•

Statistically significant improvement in target quantitative measures.

•

Significant improvement relative to a comparative group.

•

Economic efficiency in relationship to difficulty of improving the success of students.

•

Reflection on the human impact in terms of the goals of the initiative and the mission of the institution.

•

A consideration of faculty / staff perception of benefit versus cost.

•

A consideration of student perception of benefit versus cost.

The intersection of the six considerations above will tend to obscure the conversation rather than clarify given the fact the various sources of data do not provide an unambiguous way forward as it relates to SLS mandate. Support for the course, SLS1122, remains strong and has been clearly articulated by participants at every stage of the data analysis process. Support for the course and its curriculum can be seen in its prominence in three conversations currently underway: the Foundations of Excellence conversation, planning for the Gates grant and the General Education discussion. Because of restrictions on enrollment growth and other concerns there appears to be little support for the expansion of the SLS mandate to include 2‐prep students at this time. There is, however, general consensus that the SLS Mandate is not doing harm to 3‐prep students or to Valencia as an institution and need not be eliminated at this time.

•

In relation to a comparison group, there is statistically significant improvement in Fall to Spring persistence for 3‐prep students in targeted populations (Hispanics and African Americans).

•

In relations to a comparison group, there is a statistically significant improvement in prep course completion for Hispanic students.

•

Other than some concerns about freedom of choice, 3‐prep students in focus group discussions report that they were not deterred from registering or remaining at Valencia as a result of the mandate. Among 3‐prep students that successfully completed the course or were currently enrolled, the overall feeling was that SLS1122 was a very beneficial course and everyone should be made to take the course.

•

The Cost Efficiency Analysis has revealed that the use of SLS1122 as a student success oriented strategy makes good financial sense during tight economic conditions. 12

13 SLS Mandate Research Report

Because student tuition is able to absorb much of the cost for this strategy we are able to extend the reach of money allocated for student success oriented strategies. The nature and understanding of the human impact associated with the SLS mandate remains the most contentious issue surrounding this and other strategies. More work needs to be done in order to operationalize this variable within the data processing activities at the college because this discussion has the greatest potential to connect our daily activities to our mission as an institution.

14 SLS Mandate Research Report

Attached Documents ¾ Results of Facilitated Discussions – Fall 2007 ¾ Summary of SLS – Prep‐Mandate Data Discussions ¾ SLS Mandate Research Plan – Spring 2008 ¾ SLS Mandate Research Report – Trend Analysis ¾ MOA – Valencia Cost data on SLS1122 Pre Mandate ¾ MOA – Valencia Cost data on SLS1122 Mandate ¾ MOA Analysis of Valencia ROI for the SLS1122 Mandate ¾ Valencia SLS Mandate Focus Group Discussion Guide ¾ Valencia SLS Mandate Focus Group Exec Summary

AtD Data Team Focus Group results Discussing possible SLS 1122 Mandate for 2-Prep students Deans 10/18/07, 9-11 A.M. East Campus 5-112; Facilitated by Kurt Ewen and Roberta Brown 100% Pct of Total: 15% 43% 7% 68 Comment Count: 10 29 5

Comments

1 2 3 4 5

6 7

8 9

10 11 12 13

Comments made during presentation Suggested that the prep faculty recommend students for SLS rather than mandate. Confusion about the intersectionof groups of students at the first "Likelihood to Earn a Credential" slide Many courses have the prefix SLS. "SLS" should be changed to "Student Success" Success rate of technical certificate students who never take CPT might be helpful. Perhaps other courses may be as useful to 2-prep/3-prep students as SLS 1122. What does "1 or more credit hour" mean on "Students with Remediation Slide"? Would our 3 credit rather than just a 1 credit hour class make a difference. Is the math 1 credit prep course included in this data? Fall 2005 we had already begun using the Banner system and so students did not necessarily meet with advisors ouside of orientation. 37% may be considered a high number. How does re-enrollment compare overall? Clarify if the students took (were required to take) SLS 1122 in the first semester on persistence of 3 prep students slide. If students didn't return in the fall, did they return in the spring? What other courses had these students completed before and after SLS mandate? Fall to fall 2-prep comparison shows a big change.

Written Comments of Deans 14 Students who take SLS persist at a higher rate. It is not clear whether the mandate helps in persistence 15 rates. SLS group scores higher in general and retention looks 16 better

Connections

Need to Know

Items of Promise

38% 26

9% 6

Items of Concern

Who else

40% 27

1% 1

4% 3 Questions/ Suggestions for Possible General Presentation Action Comments

X X X X

X X

X X X X

X X

Comments 17 18 19 20

Something good is happening connected to Student Success course(s) Very little is seen. Students who choose to take Student Success seem more successful (whether 2-prep/3-prep) Little impact is seen with students who are mandated

Connections

Need to Know

Items of Promise

X X

Items of Concern

Who else

Questions/ Suggestions for Possible General Presentation Action Comments

Spoken Comments of Deans 21 It is important to get students to succeed in something. Individualized planning for students should be sought for 22 (more advising). 23 Something more than SLS is needed.

24 SLS is great for some students and not as great for others. SLS is 3-part: study skills, planning, life skills. Which part/parts help one student/group of students over 25 another?

26 What about specializing the course? Can this be done? Is there a need for a mandate at all? Should the mandate 27 be done away with? 28 There is not much difference seen in the data. What is the cost analysis for maintaining the 3-prep 29 mandate? What were the original goals of the 3-prep mandate and 30 have we reached them? 31 Do we eliminate the mandate? SLS faculty were originally asked if there should be a 32 mandate and they had said "no." Do higher level students look at the mandate as negative? 33 Do they avoid the class based on this view? 34 Should SLS 1122 be an all or none course? Is the data unconvincing or is there just overriding 35 concern? 36 Why was this data chosen over other possible data? Would like to know about the structure of the course and 37 it's impact. 38 Something good is happening in SLS.

X X X

X X X X

X X

X X X X

X X

Comments 39 40 41

43 44 45

46 47 48

Connections

What is the cost analysis for having a 2-prep mandate and is it good enough to issue the mandate? SLS and the mandate are distinct from one another. Left as 3-prep, will SLS1122 become just another prep course? Who wouldn't want students who wish to be in the course. How do we create that desire and motivation in students towards taking SLS 1122? What are students' feelings towards the mandate? Are they resentful of the mandate? Are they resentful of the prep courses? Include Dean of Students in discussion Ask students what made SLS worthwhile. The student success model should be finding individual strengths and directing them to things that they feel good about doing. Ask students what it was in SLS that helped them. Perhaps we should look at different models and reconfigure SLS.

How do we market SLS to the students? How do we 49 inform them of the benefits they will receive taking SLS? We need to look for faculty who want to be involved and 50 have them develop a marketing scheme. 51 We need to rid Student Success of the stigma. There seems to be a shortage of chances for success for 52 students. We need a course that students can succeed in and gain 53 confidence from. What are the logistics of creating courses that students 54 can be successful in? What else can we offer to students in prep courses 55 besides SLS1122? Advisors and deans should report on the student 56 perception of SLS 1122. 57 Those two questions shouldn't be the main questions. 58 Cost benefit analysis is necessary to make this decision. This is just a first step, we need more than just these 59 numbers.

Need to Know

Items of Promise

Items of Concern

X X

Who else

X X X

X X

Questions/ Suggestions for Possible General Presentation Action Comments

X X X

X X

X X X

X X

X X X X

Comments 60 61

62 63 64 65 66

When do SLS 1122 students choose and not choose (are mandated) to take the course? Are there enough rooms? Is there enough faculty and time to implement a 2-prep mandate? The better question is: Is the cost worth the improvement or would it be better to put the money into something else (something better)? Is there something better we could do with these resources? This doesn't seem possible with present resources and structure at the college. What about having a 1 credit hour course? Do we know the SLS scale-based information about the mandate areas that are going on? Is there something out there that is already working? There were some proposals for modules that would substitute for SLS 1122, but they didn't have enough support. Maybe these need to be looked at again. There was a trial prep-math (5-week) program that occurred, but there was not much interest for this. Perhaps with enough interest this could be an option.

Connections

Need to Know

Items of Promise

Items of Concern

Who else

Questions/ Suggestions for Possible General Presentation Action Comments

X X

X X X

X X

AtD Data Team Focus Group results Discussing possible SLS 1122 Mandate for 2-Prep students Student Affairs Group, 10/19/07, 1:30-3:00 p.m., CJI Auditorium, Facilitated by Roberta Brown 100% Pct of Total: 12% 32% 50 Comment Count: 6 16

Comments Comments made during presentation 69 Concerned about student motivation and course format. Needed clarification that Florida System meant 70 Community Colleges There is a significant difference between the Florida 71 System and Valencia. 72 Clarification of 67% 3-prep enrolled in SLS Fall 2006 Fall to Fall - seems low, interested in knowing the 73 reason Written Comments of Student Affairs 74 Students who take SLS are more successful. Students who need more prep courses are less likely to 75 persist. What happened to the "non-successful" students, or 76 those that did not persist? How is the 2-prep mandate working at other 77 institutions?

78 79

80 81

82 83 84

85 86

CCRC was not comparing apples to apples. How do we determine the similarities and differences of students and success rates that has affected the data? (i.e. Some schools have 1 credit hour courses) What are the CPT/ACT/SAT cut-off scores of different Community Colleges? How do they vary? The new student orientation is being re-designed so as to separate prep students from non-prep students in the near future. This will help prep students in that their concerns will be addressed. Students may not fully understand what the SLS course is about. The SLS 1122 course does not fit into any degree except as and elective and some students do not have room for electives in their pre-major (or they may view it this way). "Have to take" has a negative impact mentally versus "I am taking it because I want to." We do not believe SLS 1122 should be offered as a hybrid, online, or FLEX start course. Would like to know about students that did not persist (who were not "successful"), particularly those who may have moved away and completed their programs in another state. Would like SLS faculty feedback

10% 5

54% 27

8% 4

14% 7

2% 1

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation

X X X

X X

X X X

X X

134%

Comments Can we track students when they leave VCC (perhaps by asking them plainly if they would allow Valencia to 87 track them)? 88 We need to get parents more involved.

89 90 91 92 93

Spoken Comments of Student Affairs Concerned about motivation of students Concerned about the SLS 1122 course format Success is associated with Student Success How many colleges require SLS? Concern that schools measure "success" differently

94 What about EAP students who don't take SLS 1122?

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation

X X

X X X

X X X X

We need a college level course that students can succeed in that will give student's the confidence and 95 motivation they need to succeed in further course work. Students don't have a clear idea of what SLS 1122 is 96 from orientation and what it is about. 97 Students don't see SLS as needed.

X X X

Students consider SLS to be an extra course outside of 98 their degree program and view it as an extra cost. Student's have not made a connection between SLS 99 and their prospective major or area of study. The mandate changes students' attitude towards SLS 100 from "wanting" to be there to "having" to be there. Is the ratio of courses successfully completed data 101 significant? Is the data skewed by students that drop for personal 102 reasons? What happens to other people who don't "persist"? 103 How do we know if the complete the credential? There seems to be a presumption that "failures" never complete their credential and this may not be a true picture as students may transfer to out of state 104 institutions. How is the 2-prep mandate working at other 105 institutions? Bring the "unsuccessful" students to the conversation to find out if they really are not seeking or have not 106 received their credentials? 107 What is the SLS faculty perspective? What is the breakdown of other colleges SLS course 108 content and credit hours? Are we getting students in FLEX start courses that 109 really need a full-term course? Concerned that prep courses are not enough 110 preparation.

X X

X X X

X X

X X X X

Comments We are redesigning new student orientation such that prep students will be addressed separately from other students in order to offer more support and encouragement to them when they enter the college. 111 More emphasis will be placed on Student Success. Concerned that "prep" or "remedial" label has negative impact on students and their outlook on SLS and prep 112 courses. It has been proven and confirmed that SLS helps 113 promote student success.

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation

X X

The data might be skewed because (in 1999) before the 114 mandate the student perspective of SLS is different.

What about creating another type of SLS course that 115 was only 1 credit that was required for all students? 116 Some other intervention is needed. 117 Do we know if 2-prep and 3-prep needs are the same? CPT/SAT/ACT cutoff scores for prep courses is a major concern as to the number of students mandated into 118 prep courses.

X X X

AtD Data Team Focus Group results Discussing possible SLS 1122 Mandate for 2-Prep students SLS Faculty Group, 10/24/07, 5:30-7:30 p.m., West Campus HSB 211, Facilitated by Lisa Armour 100% Pct of Total: 11% 48% 7% 54 Comment Count: 6 26 4

Comments Comments made during presentation Why do they not include a measure to see about 119 out of state transfer students? Sould the data be skewed by student responses 120 (such as an irregular response?) 121 What about non-traditional students?

127

128 129 130

131 132

What role does instructor effectiveness play? (The percentage of adjuncts teaching SLS 1122 133 was high when the mandate went into effect.) Is there a difference between traditional-age and 134 non-traditional-age students?

4% 2

13% 7

4% 2

6% 3

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation

X X

122 Why are only 67% of 3-prep enrolled Fall 2006? 123 Why are we only looking at Fall? How many mandated students would have chosen to take SLS on their own regardless of the 124 mandate? What about "confounding variables" or cycles/trends that would have changed the Fall to 125 Fall persistence? We're only comparing 2 years here. Shouldn't we wait and get more years worth of data to make 126 sure one year isn't a fluke year? Written Comments of SLS Faculty Comparison between 2 and 3 prep persistence between Fall and Spring does not appear to be significantly different for non-SLS students. Persistence is similar for non-SLS students regardless of 2 or 3 prep status, however, there is a difference between those who take SLS and those who do not. Successful completion rates in Fall 2005 were the same for 2-prep students. Taking SLS 1122 does not negatively impact students. Which preps are 2 prep students taking? (math and reading, reading and writing, math and writing?) What affect will mandates have on morale/attitudes of 2-prep students?

50% 27

X X

X X X

X X

143%

Comments Is two years worth of data enough to draw 135 conclusions regarding the mandate?

136

137

138 139

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation

How many total credit hours are the 3 prep mandate students taking and what type of classes are they taking (weight lifting or government?)? How many total credit hours are the 2 prep mandate students taking and what type of classes are they taking? What are the percentages of students employed full time and part time that are taking SLS 1122 as part of 3-prep mandate? What preps were the students in?

X X

140 SLS 1122 does not negatively impact students. Concerned about the quality of instruction in SLS 141 (full-time faculty vs. others) Concerned about the number of adjuncts teaching SLS (particularly because of training, and 142 investment issues) 3-prep Fall to Fall persistence rates seem low for non-SLS and the dropping rate for SLS from 2006 143 to 2007 is concerning. At what point does SLS just become another prep 144 course? (dilution of curriculum, etc.) We are expecting college-level work of prep 145 students in SLS 1122.

X X

X X X

Concerned about the reaction and/or perception 146 of students towards mandating SLS. 147 Concerned about the student/teacher ratio. We need two levels of SLS; one for 3 prep and maybe 2 prep and another for 1 prep or no-prep 148 students. Spoken Comments of SLS Faculty The comparison between 2-prep and 3-prep 149 doesn't appear to show a big difference. 150 SLS has an impact on success. Which prep courses are the students mandated 151 in? This could make a big impact on success. 152 SLS does not negatively affect success rate. What kind of effect does the mandate have on 153 student morale? 154 Is two years long enough? How many credit hours are they taking and which 155 courses?

X X

Comments We would like to see the socioeconomic and other 156 factors of students taken into account. We should know how much of students' attention 157 is given to studies compared to other areas of life. SLS does no harm, there is some success 158 happening. What grades did the student get? Were there more A's, B's, or C's in SLS compared to other courses?--Would like to see the grade distribution 159 of successful students. 160 What is the students GPA when they leave VCC? Are adjuncts getting the training they need to 161 provide quality instruction? What is the dropping rate of students in SLS 162 courses? At what point does SLS 1122 become another 163 prep course? Concerned about faculty status of those teaching 164 SLS 1122. Should there be a prep-level SLS and a separate 165 transfer-level SLS? Want to have the thoughts from the focus groups 166 pooled. Would like to have distinction between prep 167 students and non-prep students. Experienced instructors may be making proper adjustments, but unprepared instructors are of 168 concern when dealing with prep students. Concerned about student/teacher ratios. Perhaps there should be a lower cap on courses made up 169 of prep students. Would like to know from students: What was your perception of the mandate? What was your attitude towards taking SLS (both students from before and after mandate)? Which part of the SLS curriculum was most enriching for you? What part of SLS did you not like? What 170 specifically do the students think about SLS 1122. How many students couldn't get in to SLS 1122 171 because of full courses? We should explore a college level SLS and a 172 separate prep level SLS course. What happens after AtD grant is over? Will input 173 from focus groups still be sought in this manner?

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation X

X X X

X X X X X

X X

AtD Data Team Focus Group results Discussing possible SLS 1122 Mandate for 2-Prep students Statistics Faculty, 11/2/07, EC 8-131 11:55 a.m. - 12:50 p.m. 100% Pct of Total: 43 Comment Count:

174

175 176 177 178

179

180 181 182 183 184 185 186

187 188 189 190

191 192

193

Comments Comments made during presentation Was there any plan and what was the plan as to what data should be collected? We shouldn't look at the data from hindsight. Instead, plan out what data will be collected before the process begins. Don't collect data that doesn't have an inteded use attached to it. There is not a lot of data here. Another question: Are 2 prep and 3 prep students similar enough to compar them? The CCRC data may be out of date. Regression cannot prove "effect." Please remove "effect" from the slides and replace with "relation" or some other term. Correlation does not show causation. "5th year" seems to be strange information. Were other students removed? Are we looking at those who earned the credential? Why mandate? Nothing shows definitively that Student Success is the reason for success. We need to take only mandated students and compare to other students (choosing to take or) taking SLS 1122. What are the 3 prep/2 prep pass rates? "planned to take" SLS students do just as well as those who already have according to CCSSE data. Since CCSSE is self-reported, do the students indicate using services due to trained response in SLS? What does the student in SLS "currently" choose: "done", "planned" or "no"? The highlighted bar on persistence slides makes the assumption that all the students counted in the bar were affected by the mandate though it is not necessarily the truth. Does the persistence fluxuate from year to year? We would like to see several years. There is a lack of clarity in who tool SLS and who was mandated into SLS. 1st persistence slide: Re-title: "What was happening to prep students in general" and remove the highlight. Because there are so many other variables, we need more years to see patterns in the data. It's too soon to know the patterns relative to the mandate. On all slides: bottom axis should start at zero. The green bar (2006 3 prep fall to spring) represents the students who slipped by the mandate and chose not to take SLS 1122.

14% 6

44% 19

0% 0

51% 22

5% 2

21% 9

33% 14

19% 8

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation

X X X

X X

X X X X

X X

186%

Comments We need to wait to look at this data until we have a semester's data where nearly 100% of 3 prep students were affected by the mandate. Then we need to compare that 194 data to years other than 2006. The standard deviation of 1.5 to 2 % is not significant in slide 3 prep fall-spring. The side effects might be worth looking at 195 but there is not enough data. What was the culture or what were the students told when 196 they were enrolled in SLS 1122 or chose not to take it? 197 Is AtD having a Hawthorne effect on this data? 198 2 prep is more persistent than 3 prep in general. Is putting students into prep courses (courses in which they struggle) the right thing to do right when they get to college. 199 Is is having a negative affect on persistence? 3 prep impact is more significant thatn 2/1 prep students because we can almost presuppose they would not be as 200 persistent as other students. Perhaps we should focus our efforts towards 2/1 prep students, knowing that they are more likely to succeed than 3 prep students. Perhaps changing the mandate to be upon 2 201 prep rather than 3 prep. 202 We are speculating because there is not enough data.

203 204 205 206 207 208 209

210 211

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation

X X

Are we losing the strength of SLS 1122 due to lower mix of higher level students with prep students? There is less peer to peer tutoring occuring in this more homogenious group. Last slide: 55% to 58% is significant, but is it practically significant? Would like to see last slide data with SLS course removed to see if there is success in other courses. Interested in English prep student grades in SLS and not in SLS compared to others. Intentions are not clear. There is not enough evidence or information to answer the questions. What have you looked at that addresses motivation of higher level vs. prep students? Would like to see observations of mandates of other colleges. What are the successful factors that other colleges have with their mandates? Want to look at trends of and factors of motivation.

Would like to see initial motivation of students with a questionaire that perhaps they might take alongside of CPT upon entry to college and compare results. We need to ask 212 students more questions and gather more data. CCRC switches from looking at motivation to preparation and 213 left out motivation factors. 214 Motivation is a key issue.

X X

Comments Statistics group should be invited to conversations at the 215 planning stages as well. 216 We cannot answer these questions with the data we have.

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation X

AtD Data Team Focus Group results Discussing possible SLS 1122 Mandate for 2-Prep students Prep and other Faculty, 11/27/07, EC 7-112 1:00 - 2:00 p.m. 100% Pct of Total: 27 Comment Count:

217 218 219 220

Comments Comments made during presentation In the CCRC Brief did they explain why students in remedial courses were less likely to earn a credential? What was the length and time period of the CCRC study? You can see that SLS 1122 impacts success. There is a huge difference in 3 prep and 2 prep students.

26% 7

44% 12

4% 1

33% 9

X X

224 225

226 227

228 229

230

231 232

233 234

11% 3

4% 1

X X

Comments made as 6 questions were asked

223

7% 2

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation

Is there anyway that we can help 3 prep students become 2 prep students? Is there some way we can help them perform 221 better on the CPT? When can students retake the CPT? Due to financial reasons, students may be upset by 222 extending the mandate SLS 1122.

It seems like Valencia is better in comparison to the Florida System when looking at CCRC. We have more likeliness to succeed/earn a credential with SLS. We may need SLS. How does our SLS course differ in format from other institutions and what impact does that play on the data? Do we have access to that information? SLS has had a positive impact, but we'd like to see another year's worth of data. How might increased enrollment be influencing the data? More years might help. Paralelling the data to increased enrollment data may show why it appears to improve. Is one year's data meaningful? There were a higher percentage of students who did not enroll in SLS as well as a higher percentage of those who did enroll in SLS. We can't contribute all of the impact to SLS. We need more data. We suspect that SLS is and will improve success rates. Have we looked at completion of 2-prep students and compared the completion of students in SLS to students not in SLS? This might clarify the impact of SLS. In SLS, enrollment for the next term is likely to be an assignment for students. This may mean that we need to look at more than persistence. What is the effect of the SLS course format? Compare online, FLEX, late start, and other formats The amount of SLS full-time faculty is a concern. It would be best to have less adjuncts teaching SLS and more full-time faculty teaching SLS. What are the requirements and qualifications for teaching SLS 1122?

7% 2

X X

137%

Comments Would like to know what deans are thinking as well as SLS 235 instructors ideas. 236 What are the success rates for SLS? Would like to look more into successful completion rather 237 than enrollment. Summarizing Comments 238 SLS has had a positive impact. 239 There is not a lot of data on anything but persistence. We would like to know if students who were mandated into 240 SLS would have taken SLS if there was no mandate. 241 We cannot answer to what the effect of the mandate is. We can only give our opinion, which is the mandate is a great idea and there is great possibility that it will aid in success. 242 There just is not enough evidence to support this yet. Would like to know what students thought about SLS being a 3 credit course. Is it necessary to have SLS as a 3 credit 243 course?

Questions/ General Suggestions for Connections Need to Know Items of Promise Items of Concern Who else Possible Action Comments Presentation X X X

X X

AtD Data Team Focus Group results Discussing possible SLS 1122 Mandate for 2-Prep students All Focus Groups Combined Responses 100% Pct of Total: 242 Comment Count:

1 Deans 2 Deans 3 Deans 4 Deans 5 Deans

6 Deans 7 Deans

8 Deans 9 Deans

10 Deans 11 Deans 12 Deans 13 Deans 14 Deans 15 Deans 16 Deans 17 Deans 18 Deans 19 Deans 20 Deans 21 Deans 22 Deans 23 Deans 24 Deans

25 Deans

All Comments- written and spoken Suggested that the prep faculty recommend students for SLS rather than mandate. Confusion about the intersectionof groups of students at the first "Likelihood to Earn a Credential" slide Many courses have the prefix SLS. "SLS" should be changed to "Student Success" Success rate of technical certificate students who never take CPT might be helpful. Perhaps other courses may be as useful to 2-prep/3-prep students as SLS 1122. What does "1 or more credit hour" mean on "Students with Remediation Slide"? Would our 3 credit rather than just a 1 credit hour class make a difference. Is the math 1 credit prep course included in this data? Fall 2005 we had already begun using the Banner system and so students did not necessarily meet with advisors ouside of orientation. 37% may be considered a high number. How does re-enrollment compare overall? Clarify if the students took (were required to take) SLS 1122 in the first semester on persistence of 3 prep students slide. If students didn't return in the fall, did they return in the spring? What other courses had these students completed before and after SLS mandate? Fall to fall 2-prep comparison shows a big change. Students who take SLS persist at a higher rate. It is not clear whether the mandate helps in persistence rates. SLS group scores higher in general and retention looks better Something good is happening connected to Student Success course(s) Very little is seen. Students who choose to take Student Success seem more successful (whether 2-prep/3-prep) Little impact is seen with students who are mandated It is important to get students to succeed in something. Individualized planning for students should be sought for (more advising). Something more than SLS is needed. SLS is great for some students and not as great for others. SLS is 3-part: study skills, planning, life skills. Which part/parts help one student/group of students over another?

3/31/2009

14% 35

42% 102

6% 15

46% 111

Connections

Need to Know

Items of Promise

Items of Concern

7% 16

21% 52

9% 22

7% 17

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X X X

X X

X X X X X

X X

X X X

Page 16 of 28

Results of Focus Groups with Counts

All Comments- written and spoken 26 Deans 27 Deans 28 Deans 29 Deans 30 Deans 31 Deans 32 Deans 33 Deans 34 Deans 35 Deans 36 Deans 37 Deans 38 Deans 39 Deans 40 Deans 41 Deans

42 Deans

43 Deans 44 Deans 45 Deans

46 Deans 47 Deans 48 Deans

49 Deans 50 Deans 51 Deans 52 Deans 53 Deans 54 Deans

What about specializing the course? Can this be done? Is there a need for a mandate at all? Should the mandate be done away with? There is not much difference seen in the data. What is the cost analysis for maintaining the 3-prep mandate? What were the original goals of the 3-prep mandate and have we reached them? Do we eliminate the mandate? SLS faculty were originally asked if there should be a mandate and they had said "no." Do higher level students look at the mandate as negative? Do they avoid the class based on this view? Should SLS 1122 be an all or none course? Is the data unconvincing or is there just overriding concern? Why was this data chosen over other possible data? Would like to know about the structure of the course and it's impact. Something good is happening in SLS. What is the cost analysis for having a 2-prep mandate and is it good enough to issue the mandate? SLS and the mandate are distinct from one another. Left as 3-prep, will SLS1122 become just another prep course? Who wouldn't want students who wish to be in the course. How do we create that desire and motivation in students towards taking SLS 1122? What are students' feelings towards the mandate? Are they resentful of the mandate? Are they resentful of the prep courses? Include Dean of Students in discussion Ask students what made SLS worthwhile. The student success model should be finding individual strengths and directing them to things that they feel good about doing. Ask students what it was in SLS that helped them. Perhaps we should look at different models and reconfigure SLS. How do we market SLS to the students? How do we inform them of the benefits they will receive taking SLS? We need to look for faculty who want to be involved and have them develop a marketing scheme. We need to rid Student Success of the stigma. There seems to be a shortage of chances for success for students. We need a course that students can succeed in and gain confidence from. What are the logistics of creating courses that students can be successful in?

3/31/2009

Connections

Need to Know

Items of Promise

Items of Concern

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X X X

X X

X X X X

X X

X X X

X X

X X X

X X

X X X

Page 17 of 28

Results of Focus Groups with Counts

All Comments- written and spoken 55 Deans 56 Deans 57 Deans 58 Deans 59 Deans 60 Deans 61 Deans

62 Deans 63 Deans 64 Deans 65 Deans 66 Deans

67 Deans

68 Deans

What else can we offer to students in prep courses besides SLS1122? Advisors and deans should report on the student perception of SLS 1122. Those two questions shouldn't be the main questions. Cost benefit analysis is necessary to make this decision. This is just a first step, we need more than just these numbers. When do SLS 1122 students choose and not choose (are mandated) to take the course? Are there enough rooms? Is there enough faculty and time to implement a 2-prep mandate? The better question is: Is the cost worth the improvement or would it be better to put the money into something else (something better)? Is there something better we could do with these resources? This doesn't seem possible with present resources and structure at the college. What about having a 1 credit hour course? Do we know the SLS scale-based information about the mandate areas that are going on? Is there something out there that is already working? There were some proposals for modules that would substitute for SLS 1122, but they didn't have enough support. Maybe these need to be looked at again. There was a trial prep-math (5-week) program that occurred, but there was not much interest for this. Perhaps with enough interest this could be an option.

3/31/2009

Connections

Need to Know X

Items of Promise

Items of Concern

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X

X X X X

X X

X X X

X X

Page 18 of 28

Results of Focus Groups with Counts

All Comments- written and spoken 69 Stu Aff 70 Stu Aff 71 Stu Aff 72 Stu Aff 73 Stu Aff 74 Stu Aff 75 Stu Aff 76 Stu Aff 77 Stu Aff

78 Stu Aff 79 Stu Aff

80 Stu Aff 81 Stu Aff

82 Stu Aff 83 Stu Aff 84 Stu Aff

85 Stu Aff 86 Stu Aff

87 88 89 90 91 92 93 94

Stu Aff Stu Aff Stu Aff Stu Aff Stu Aff Stu Aff Stu Aff Stu Aff

95 Stu Aff

Concerned about student motivation and course format. Needed clarification that Florida System meant Community Colleges There is a significant difference between the Florida System and Valencia. Clarification of 67% 3-prep enrolled in SLS Fall 2006 Fall to Fall - seems low, interested in knowing the reason Students who take SLS are more successful. Students who need more prep courses are less likely to persist. What happened to the "non-successful" students, or those that did not persist?

Connections

3/31/2009

Items of Promise

Items of Concern

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X X

X X

How is the 2-prep mandate working at other institutions? CCRC was not comparing apples to apples. How do we determine the similarities and differences of students and success rates that has affected the data? (i.e. Some schools have 1 credit hour courses) What are the CPT/ACT/SAT cut-off scores of different Community Colleges? How do they vary? The new student orientation is being re-designed so as to separate prep students from non-prep students in the near future. This will help prep students in that their concerns will be addressed. Students may not fully understand what the SLS course is about. The SLS 1122 course does not fit into any degree except as and elective and some students do not have room for electives in their pre-major (or they may view it this way). "Have to take" has a negative impact mentally versus "I am taking it because I want to." We do not believe SLS 1122 should be offered as a hybrid, online, or FLEX start course. Would like to know about students that did not persist (who were not "successful"), particularly those who may have moved away and completed their programs in another state. Would like SLS faculty feedback Can we track students when they leave VCC (perhaps by asking them plainly if they would allow Valencia to track them)? We need to get parents more involved. Concerned about motivation of students Concerned about the SLS 1122 course format Success is associated with Student Success How many colleges require SLS? Concern that schools measure "success" differently What about EAP students who don't take SLS 1122? We need a college level course that students can succeed in that will give student's the confidence and motivation they need to succeed in further course work.

Need to Know

X X

X X X

X X

X X X X X

X X X

X X

X Page 19 of 28

Results of Focus Groups with Counts

All Comments- written and spoken

103 Stu Aff

Students don't have a clear idea of what SLS 1122 is from orientation and what it is about. Students don't see SLS as needed. Students consider SLS to be an extra course outside of their degree program and view it as an extra cost. Student's have not made a connection between SLS and their prospective major or area of study. The mandate changes students' attitude towards SLS from "wanting" to be there to "having" to be there. Is the ratio of courses successfully completed data significant? Is the data skewed by students that drop for personal reasons? What happens to other people who don't "persist"? How do we know if the complete the credential?

104 Stu Aff

There seems to be a presumption that "failures" never complete their credential and this may not be a true picture as students may transfer to out of state institutions.

96 Stu Aff 97 Stu Aff 98 Stu Aff 99 Stu Aff 100 Stu Aff 101 Stu Aff 102 Stu Aff

105 Stu Aff

106 Stu Aff 107 Stu Aff 108 Stu Aff 109 Stu Aff 110 Stu Aff

111 Stu Aff

112 Stu Aff 113 Stu Aff 114 Stu Aff 115 Stu Aff 116 Stu Aff 117 Stu Aff

118 Stu Aff

Connections

3/31/2009

Items of Promise

Items of Concern

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X X X X X X

How is the 2-prep mandate working at other institutions? Bring the "unsuccessful" students to the conversation to find out if they really are not seeking or have not received their credentials? What is the SLS faculty perspective? What is the breakdown of other colleges SLS course content and credit hours? Are we getting students in FLEX start courses that really need a full-term course? Concerned that prep courses are not enough preparation. We are redesigning new student orientation such that prep students will be addressed separately from other students in order to offer more support and encouragement to them when they enter the college. More emphasis will be placed on Student Success. Concerned that "prep" or "remedial" label has negative impact on students and their outlook on SLS and prep courses. It has been proven and confirmed that SLS helps promote student success. The data might be skewed because (in 1999) before the mandate the student perspective of SLS is different. What about creating another type of SLS course that was only 1 credit that was required for all students? Some other intervention is needed. Do we know if 2-prep and 3-prep needs are the same? CPT/SAT/ACT cutoff scores for prep courses is a major concern as to the number of students mandated into prep courses.

Need to Know

X X

X X X X

X X

X X X X X

Page 20 of 28

Results of Focus Groups with Counts

All Comments- written and spoken 119 SLS Fac 120 121 122 123

SLS Fac SLS Fac SLS Fac SLS Fac

124 SLS Fac 125 SLS Fac

126 SLS Fac

127 SLS Fac

128 SLS Fac 129 SLS Fac 130 SLS Fac 131 SLS Fac 132 SLS Fac

133 SLS Fac 134 SLS Fac 135 SLS Fac

136 SLS Fac

137 SLS Fac

138 SLS Fac 139 SLS Fac 140 SLS Fac 141 SLS Fac

142 SLS Fac

143 SLS Fac

Why do they not include a measure to see about out of state transfer students? Sould the data be skewed by student responses (such as an irregular response?) What about non-traditional students? Why are only 67% of 3-prep enrolled Fall 2006? Why are we only looking at Fall? How many mandated students would have chosen to take SLS on their own regardless of the mandate? What about "confounding variables" or cycles/trends that would have changed the Fall to Fall persistence? We're only comparing 2 years here. Shouldn't we wait and get more years worth of data to make sure one year isn't a fluke year? Comparison between 2 and 3 prep persistence between Fall and Spring does not appear to be significantly different for non-SLS students. Persistence is similar for non-SLS students regardless of 2 or 3 prep status, however, there is a difference between those who take SLS and those who do not. Successful completion rates in Fall 2005 were the same for 2-prep students. Taking SLS 1122 does not negatively impact students. Which preps are 2 prep students taking? (math and reading, reading and writing, math and writing?) What affect will mandates have on morale/attitudes of 2prep students? What role does instructor effectiveness play? (The percentage of adjuncts teaching SLS 1122 was high when the mandate went into effect.) Is there a difference between traditional-age and nontraditional-age students? Is two years worth of data enough to draw conclusions regarding the mandate? How many total credit hours are the 3 prep mandate students taking and what type of classes are they taking (weight lifting or government?)? How many total credit hours are the 2 prep mandate students taking and what type of classes are they taking? What are the percentages of students employed full time and part time that are taking SLS 1122 as part of 3-prep mandate? What preps were the students in? SLS 1122 does not negatively impact students. Concerned about the quality of instruction in SLS (full-time faculty vs. others)

Connections

Need to Know

Items of Concern

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X

X X X X

X X

X X X X X

X X

X X X

Concerned about the number of adjuncts teaching SLS (particularly because of training, and investment issues) 3-prep Fall to Fall persistence rates seem low for non-SLS and the dropping rate for SLS from 2006 to 2007 is concerning.

3/31/2009

Items of Promise

X Page 21 of 28

Results of Focus Groups with Counts

All Comments- written and spoken 144 SLS Fac 145 SLS Fac 146 SLS Fac 147 SLS Fac 148 SLS Fac 149 SLS Fac 150 SLS Fac 151 SLS Fac 152 SLS Fac 153 SLS Fac 154 SLS Fac 155 SLS Fac 156 SLS Fac 157 SLS Fac 158 SLS Fac

159 SLS Fac 160 SLS Fac 161 SLS Fac 162 SLS Fac 163 SLS Fac 164 SLS Fac 165 SLS Fac 166 SLS Fac 167 SLS Fac

168 SLS Fac

169 SLS Fac

At what point does SLS just become another prep course? (dilution of curriculum, etc.) We are expecting college-level work of prep students in SLS 1122. Concerned about the reaction and/or perception of students towards mandating SLS. Concerned about the student/teacher ratio. We need two levels of SLS; one for 3 prep and maybe 2 prep and another for 1 prep or no-prep students. The comparison between 2-prep and 3-prep doesn't appear to show a big difference. SLS has an impact on success. Which prep courses are the students mandated in? This could make a big impact on success. SLS does not negatively affect success rate. What kind of effect does the mandate have on student morale? Is two years long enough? How many credit hours are they taking and which courses? We would like to see the socioeconomic and other factors of students taken into account. We should know how much of students' attention is given to studies compared to other areas of life. SLS does no harm, there is some success happening. What grades did the student get? Were there more A's, B's, or C's in SLS compared to other courses?--Would like to see the grade distribution of successful students. What is the students GPA when they leave VCC? Are adjuncts getting the training they need to provide quality instruction? What is the dropping rate of students in SLS courses? At what point does SLS 1122 become another prep course? Concerned about faculty status of those teaching SLS 1122. Should there be a prep-level SLS and a separate transferlevel SLS?

Connections

Need to Know

Items of Concern

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X X X X X X

X X

X X X X

X X X

Want to have the thoughts from the focus groups pooled. Would like to have distinction between prep students and non-prep students. Experienced instructors may be making proper adjustments, but unprepared instructors are of concern when dealing with prep students. Concerned about student/teacher ratios. Perhaps there should be a lower cap on courses made up of prep students.

3/31/2009

Items of Promise

X X

Page 22 of 28

Results of Focus Groups with Counts

All Comments- written and spoken

170 SLS Fac 171 SLS Fac 172 SLS Fac 173 SLS Fac

Would like to know from students: What was your perception of the mandate? What was your attitude towards taking SLS (both students from before and after mandate)? Which part of the SLS curriculum was most enriching for you? What part of SLS did you not like? What specifically do the students think about SLS 1122. How many students couldn't get in to SLS 1122 because of full courses? We should explore a college level SLS and a separate prep level SLS course. What happens after AtD grant is over? Will input from focus groups still be sought in this manner?

3/31/2009

Connections

Need to Know

Items of Promise

Items of Concern

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X X

Page 23 of 28

Results of Focus Groups with Counts

All Comments- written and spoken 174 Stat Fac

175 Stat Fac 176 Stat Fac 177 Stat Fac 178 Stat Fac

179 Stat Fac

180 Stat Fac 181 Stat Fac

182 Stat Fac 183 Stat Fac 184 Stat Fac 185 Stat Fac 186 Stat Fac

187 Stat Fac 188 Stat Fac 189 Stat Fac 190 Stat Fac

191 Stat Fac 192 Stat Fac

193 Stat Fac

194 Stat Fac

195 Stat Fac

Connections

Was there any plan and what was the plan as to what data should be collected? We shouldn't look at the data from hindsight. Instead, plan out what data will be collected before the process begins. Don't collect data that doesn't have an inteded use attached to it. There is not a lot of data here. Another question: Are 2 prep and 3 prep students similar enough to compare them? The CCRC data may be out of date.

3/31/2009

Items of Promise

Items of Concern

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X

Regression cannot prove "effect." Please remove "effect" from the slides and replace with "relation" or some other term. Correlation does not show causation. "5th year" seems to be strange information. Were other students removed? Are we looking at those who earned the credential? Why mandate? Nothing shows definitively that Student Success is the reason for success. We need to take only mandated students and compare to other students (choosing to take or) taking SLS 1122. What are the 3 prep/2 prep pass rates? "planned to take" SLS students do just as well as those who already have according to CCSSE data. Since CCSSE is self-reported, do the students indicate using services due to trained response in SLS? What does the student in SLS "currently" choose: "done", "planned" or "no"? The highlighted bar on persistence slides makes the assumption that all the students counted in the bar were affected by the mandate though it is not necessarily the truth. Does the persistence fluxuate from year to year? We would like to see several years. There is a lack of clarity in who took SLS and who was mandated into SLS. 1st persistence slide: Re-title: "What was happening to prep students in general" and remove the highlight. Because there are so many other variables, we need more years to see patterns in the data. It's too soon to know the patterns relative to the mandate. On all slides: bottom axis should start at zero. The green bar (2006 3 prep fall to spring) represents the students who slipped by the mandate and chose not to take SLS 1122. We need to wait to look at this data until we have a semester's data where nearly 100% of 3 prep students were affected by the mandate. Then we need to compare that data to years other than 2006. The standard deviation of 1.5 to 2 % is not significant in slide 3 prep fall-spring. The side effects might be worth looking at but there is not enough data.

Need to Know

X X

X Page 24 of 28

X Results of Focus Groups with Counts

All Comments- written and spoken

196 Stat Fac 197 Stat Fac 198 Stat Fac

199 Stat Fac

200 Stat Fac

201 Stat Fac 202 Stat Fac

203 Stat Fac 204 Stat Fac 205 Stat Fac 206 Stat Fac 207 Stat Fac 208 Stat Fac 209 Stat Fac

210 Stat Fac 211 Stat Fac

What was the culture or what were the students told when they were enrolled in SLS 1122 or chose not to take it? Is AtD having a Hawthorne effect on this data? 2 prep is more persistent than 3 prep in general. Is putting students into prep courses (courses in which they struggle) the right thing to do right when they get to college. Is is having a negative affect on persistence? 3 prep impact is more significant thatn 2/1 prep students because we can almost presuppose they would not be as persistent as other students. Perhaps we should focus our efforts towards 2/1 prep students, knowing that they are more likely to succeed than 3 prep students. Perhaps changing the mandate to be upon 2 prep rather than 3 prep. We are speculating because there is not enough data. Are we losing the strength of SLS 1122 due to lower mix of higher level students with prep students? There is less peer to peer tutoring occuring in this more homogenious group. Last slide: 55% to 58% is significant, but is it practically significant? Would like to see last slide data with SLS course removed to see if there is success in other courses. Interested in English prep student grades in SLS and not in SLS compared to others. Intentions are not clear. There is not enough evidence or information to answer the questions. What have you looked at that addresses motivation of higher level vs. prep students? Would like to see observations of mandates of other colleges. What are the successful factors that other colleges have with their mandates? Want to look at trends of and factors of motivation.

215 Stat Fac

Would like to see initial motivation of students with a questionaire that perhaps they might take alongside of CPT upon entry to college and compare results. We need to ask students more questions and gather more data. CCRC switches from looking at motivation to preparation and left out motivation factors. Motivation is a key issue. Statistics group should be invited to conversations at the planning stages as well.

216 Stat Fac

We cannot answer these questions with the data we have.

212 Stat Fac 213 Stat Fac 214 Stat Fac

3/31/2009

Connections

Need to Know

X X

Items of Promise

Items of Concern

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X

Page 25 of 28

X X X

Results of Focus Groups with Counts

All Comments- written and spoken 217 Prep Fac

In the CCRC Brief did they explain why students in remedial courses were less likely to earn a credential?

218 Prep Fac 219 Prep Fac

What was the length and time period of the CCRC study? You can see that SLS 1122 impacts success.

220 Prep Fac

There is a huge difference in 3 prep and 2 prep students. Is there anyway that we can help 3 prep students become 2 prep students? Is there some way we can help them perform better on the CPT? When can students retake the CPT? Due to financial reasons, students may be upset by extending the mandate SLS 1122. It seems like Valencia is better in comparison to the Florida System when looking at CCRC. We have more likeliness to succeed/earn a credential with SLS. We may need SLS. How does our SLS course differ in format from other institutions and what impact does that play on the data? Do we have access to that information? SLS has had a positive impact, but we'd like to see another year's worth of data.

221 Prep Fac 222 Prep Fac

223 Prep Fac

224 Prep Fac 225 Prep Fac

Connections

234 Prep Fac 235 Prep Fac 236 Prep Fac 237 Prep Fac 238 Prep Fac

3/31/2009

X X

233 Prep Fac

Questions/ Suggestions for Presentation

We suspect that SLS is and will improve success rates.

232 Prep Fac

General Comments

229 Prep Fac

231 Prep Fac

Who else

Possible Action

230 Prep Fac

Items of Concern

228 Prep Fac

Have we looked at completion of 2-prep students and compared the completion of students in SLS to students not in SLS? This might clarify the impact of SLS. In SLS, enrollment for the next term is likely to be an assignment for students. This may mean that we need to look at more than persistence. What is the effect of the SLS course format? Compare online, FLEX, late start, and other formats The amount of SLS full-time faculty is a concern. It would be best to have less adjuncts teaching SLS and more fulltime faculty teaching SLS. What are the requirements and qualifications for teaching SLS 1122? Would like to know what deans are thinking as well as SLS instructors ideas. What are the success rates for SLS? Would like to look more into successful completion rather than enrollment. SLS has had a positive impact.

Items of Promise

How might increased enrollment be influencing the data? More years might help. Paralelling the data to increased enrollment data may show why it appears to improve. Is one year's data meaningful? There were a higher percentage of students who did not enroll in SLS as well as a higher percentage of those who did enroll in SLS. We can't contribute all of the impact to SLS. We need more data.

226 Prep Fac 227 Prep Fac

Need to Know

X X

X X X Page 26 of 28

Results of Focus Groups with Counts

All Comments- written and spoken 239 Prep Fac

240 Prep Fac 241 Prep Fac

242 Prep Fac

243 Prep Fac

Connections

Need to Know

There is not a lot of data on anything but persistence. We would like to know if students who were mandated into SLS would have taken SLS if there was no mandate. We cannot answer to what the effect of the mandate is. We can only give our opinion, which is the mandate is a great idea and there is great possibility that it will aid in success. There just is not enough evidence to support this yet. Would like to know what students thought about SLS being a 3 credit course. Is it necessary to have SLS as a 3 credit course?

3/31/2009

Items of Promise

Items of Concern

Who else

Possible Action

General Comments

Questions/ Suggestions for Presentation

X X

Page 27 of 28

Results of Focus Groups with Counts

AtD Data Team Discussion Group Results Consideration of Extending SLS 1122 Mandate to 2-Prep students Mix of Comments by Group Percent of Total Comments (within each group) All Comments

Connections Need to Know

Deans Student Affairs SLS Faculty Prep & Other Statistics Faculty All Groups

15% 12% 11% 26% 14% 14%

43% 32% 48% 44% 44% 42%

Items of Promise

Items of Concern

7% 10% 7% 4% 0% 6%

38% 54% 50% 33% 51% 46%

Who else

Possible Action

General Comments

9% 8% 4% 7% 5% 7%

40% 14% 13% 7% 21% 21%

1% 2% 4% 11% 33% 9%

Questions for Total (includes Presentation duplicates) 4% 2% 6% 4% 19% 7%

157% 134% 143% 137% 186% 153%

Mix of Comments by Group

60%

50%

46%

% Comments within Group

42%

40%

30% 21%

20% 14% 9%

10%

0% Connections

Need to Know

Items of Promise

Items of Concern

Who else

Possible Action

General Comments

Questions for Presentation

Discussion Group Deans 3/31/2009

Student Affairs

SLS Faculty

Prep & Other

Valencia Community College

Statistics Faculty

All Groups

Results of Focus Groups with Counts Summary

Summary of SLS / PrepMandate Data Discussions Fall 2007 Introduction As a result of general discomfort with an attempt last Spring to present initial results from the implementation of AtD strategies, the composition and work of the AtD Data Team was redesigned in preparation for college‐wide discussions concerning the expansion of the SLS mandate for prep students. The data processing model employed by the data team is a new initiative based on the premise that the meaning of assessment data is ambiguous. This ambiguity requires structured reflection and dialogue by people at varying levels of engagement with the data in order for the results to become meaningful / actionable information. The groups selected to participate in the data discussions include the following: Academic Deans, SLS faculty, Prep faculty, Statistics faculty, Counselors and advisors. Each group was presented with the same data in the form of a power point presentation (see attached) and were asked to consider 2 overarching questions – “What is the impact of SLS1122 on student success?” and “What was the effect of the 3‐prep mandate on 3‐ prep student success?” To guide their discussion participants were given worksheets that contained an additional 6 questions which include the following: What connections/patterns do you see in the data?; What else do you need to know?; What do you see as items of promise?; What do you see as items of concern?; Who needs to be brought into the conversation to help clarify the data? Based on what we know, what needs our attention? Notes from the 2 hour data discussions were taken and the worksheets were collected. The note‐taker’s notes and the participant comments were coded and tabulated and serve and the basis of the information presented here. The breakdown of this paper This paper will consider the comments of the participants by way of a three –fold division that emerged from within the discussions themselves. The first section reports on the perceived value of SLS1122 in its current form. The second, the groups’ collective evaluation of the success of the SLS1122 mandate on 3‐prep students and whether this mandate should continue. The third, whether or not the mandate for SLS1122 should be expand to include 2‐prep students. Perceived value of SLS1122 in its current form Participants had a generally favorable view of SLS1122 and believe that Valencia should continue to support this course as a general strategy to improve student success. Suggested areas for further study focused on questions about the specific aspects of SLS1122 that promote success within the different demographic groups that Valencia serves. Additional questions concerned the current structure of SLS1122 and whether restructuring or an expansion of offerings might be beneficial.

SLS Mandate for 3‐prep students While some questioned the logic of a mandate and suggested that the SLS mandate for 3‐prep students should be abandoned now, the majority of participants were more circumspect. Overwhelmingly, participants concluded that a reasonable evaluation of the impact of the 3‐prep mandate on student success will require more time and the collection of more and different data. Additionally, more time will allow for greater clarification of the research questions guiding this work and potential for consensus on parameters for determining the success of the strategy itself. General concern was expressed about the potential for long‐term damage to both the image and content of SLS1122. The unanticipated, long‐term impact of the mandate is that it may brand SLS1122 as a prep course and, thereby, discourage enrollment of non‐prep students. SLS Mandate for 2‐prep students Given the inconclusive nature of the data that we currently have on the impact of the 3‐prep mandate, data discussion participants did not support the expansion of the SLS1122 mandate to include 2‐prep students. Furthermore, participants thought that the logic behind the expansion of the mandate should be studied. It was unclear to a number of participants whether or not 1, 2 and 3 prep students should be viewed as sufficiently similar so as to be potentially impacted by similar strategies. General conclusions by data discussion facilitators The facilitators of the data discussions consider the following conclusions to be a reasonable reflection of the thinking of the participants: 1) SLS1122 has a proven track record at Valencia and should be supported as tool for improved student success. Valencia should continue to look for ways to employ both the course, and its content, to help students succeed without impacting the content of the course. 2) The data from the impact of mandating SLS1122 on 3‐prep students are currently inconclusive. A reasonable evaluation of the impact of the 3‐prep mandate on student success will require more time and the collection of more and different data. 3) The inconclusive nature of the data that we currently have on the impact of the 3‐prep mandate does not support the expansion of the SLS1122 mandate to include 2‐prep students.

SLS1122 Mandate for 3‐Prep Students Research Plan – Achieving the Dream Data Team Spring 2008 Introduction: Because research generally supports the benefits of student success courses it was determined that an expansion of the Student Success course, SLS1122, would be one of the three strategies implemented through the Achieving the Dream (AtD) Grant Initiative. The first phase of the proposed expansion was the implementation of the SLS1122 mandate for 3‐prep students which required any student testing into prep levels in all three discipline areas, with the exception of EAP students, to take Student Success (SLS1122). The purpose of the research proposed here is to determine if the implementation of the SLS1122 mandate has resulted in changes in overall 3‐prep student performance. The focus of this research is the SLS1122 mandate and not the SLS1122 course. For this reason, our research plan does not include any assessment of the Student Success course itself or a discussion of its benefits. Note: The SLS1122 mandate for 3‐prep students hereafter will be referred to as the SLS mandate. Hypothesis: Requiring all 3‐prep students to take SLS1122 will increase the overall performance of 3‐prep students. Research Question: Have 3‐prep students shown improvement beyond what they would have been expected to achieve without the SLS 1122 mandate? Confounding Time‐Related Variables: To answer the research question, one must compare the before and after effects of student performance over time. There are, however, many confounding variables operating on the school environment that could cause success measures associated with 3‐prep students to change over time. Confounding variables that could lead to natural fluctuations in performance from year to year and semester to semester include: •

Local economic conditions that would encourage or discourage community college enrollment, persistence, and retention.

• • • •

Admission standards, marketing strategies, and the general competitiveness of other colleges in our service area could affect the mix and caliber of students who attend Valencia. Our own marketing strategies and evolving mix of programs offered could also affect the mix of students who chose to attend Valencia. Improvements in our college and our community that lead to a better learning environment for the students enrolled at Valencia. The effects of other AtD initiatives at Valencia that affect the performance of those students being studied simultaneously as part of the SLS1122 research.

1. Treatment Group: 3‐prep students. These are those students who tested into all three discipline areas of college preparatory courses (reading, writing, and math) except for those students classified as EAP students. 2. Comparison Group: 1 and 2‐prep students combined. These are those students who tested into 1 or 2 discipline areas of college preparatory courses. These students are not required to take SLS 1122, but may have chosen to take SLS1122 anyway. Note: EAP and MAT1033 courses are not included in the prep definition here. Methodology: In spite of population differences between the treatment and comparison group, the effect of time‐related confounding variables should be similar. Our research will compare the performance time trends before and after the mandate for both groups. Research Validation: The comparison group is understood to be an imperfect control group. It is possible that the time‐related confounding variables could impact the various mandate‐level populations differently. To examine this possibility, one could study the time trends on diverse potential comparison groups to consider how similar these populations track with each other over time, especially before and after the 3‐prep mandate. If the fluctuations in performance over time

track similarly among all groups over time, it strengthens our confidence that the comparison of the treatment and comparison group is valid. Meaningful Improvement For the purposes of data supported administrative decisions concerning the continuation or expansion of student success oriented programs, the determination of “meaningful improvement” would require a balance of all or most of the following: •

Statistically significant improvement in target quantitative measures.

•

Significant improvement relative to a comparative group.

•

Economic efficiency in relationship to difficulty of improving the success of students.

•

Reflection on the human impact in terms of the goals of the initiative and the mission of the institution.

•

A consideration of faculty / staff perception of benefit versus cost.

•

A consideration of student perception of benefit versus cost.

In consideration of this, we propose to use the following measures: Measures: Trend Analysis We propose to quantify the analysis of the research question by creating a trend analysis that compares the two research groups in the Fall terms, starting in Fall 2003 and continuing through Fall 2008. This comparison includes three terms before the 3‐prep mandate was implemented (Fall 2003, 2004 and 2005) and three terms after the 3‐prep mandate was implemented (Fall 2006, Fall 2007 and Fall 2008). We propose three measures for this trend analysis: Fall to Spring persistence, Fall to Fall persistence, and percentage of college prep courses that were successfully completed (with a grade of A, B, or C) in the specified Fall term. The trend analysis will be validated by determining the statistical significance of the change in the measures within each research group. These measures will also be broken down by ethnic group in order to track the success gaps that are part of the Achieving the Dream initiative (African‐American, Caucasian, and Hispanic).

Persistence is a meaningful measure because it is a correlate of student success in that students cannot make progress towards degree completion if they do not continue enrollment. It is a measure of momentum towards degree completion. Successful completion of college prep courses is a meaningful measure because it is a correlate of mastery of course outcomes and leads to sequenced completion of subsequent courses in the curriculum. Research indicates that completion of the college prep course sequence is an important progression point towards degree completion. Note: We will not be able to have a Fall to Fall persistence measure for the Fall 2008 term groups because sufficient time will not have passed by January 2009 to compute this measure. The focus of this analysis is the comparison of trend lines before and after the experimental treatment (implementation of the mandate). Theoretically, by graphing and statistically comparing the before and after trends over time for the treatment group (3‐prep students) and the comparison group (1‐ and 2‐prep students combined), any effect associated with the treatment should show up as a favorable shift in the trend lines for the treatment group beginning with the implementation of the SLS mandate (Fall 2006). For example: Hypothetical Example: Treatment Appears to Have Had an Effect 75

Comparison Group

Performance Measure

= Effect Associated with Treatment 70

Treatment Group Treatment Group New Trend

Comparison Group Original Trend Treatment Group Original Trend

2000

2001

2002

2003

2004

2005

Time

2006

2007

2008

2009

Student Focus Groups We propose to conduct student focus groups on the research question: How do students feel about being required to take SLS1122? This research question was proposed through the Achieving the Dream Data Team facilitated discussions that were held in Fall 2007 with selected groups of deans, faculty and staff. The Discussion Guide for the Student Focus Groups is being developed in consultation with Mr. Thor Falk, who owns and operates Falk Associates, a market research company with a specialization in qualitative research. The Discussion Guide will be structured to establish a comfortable atmosphere among the group and then follow a sequenced protocol that will identify when the students became aware of the SLS1122 requirement and what they thought and felt about the requirement. Students who were required to take SLS1122 based on the SLS mandate since Fall 2006 will be invited to participate in the focus groups through a protocol that does not identify the specific topic of the discussion. They will be compensated for their participation. Separate groups will be formed for students who successfully completed SLS1122 and those who did not. Following focus group methodology, the number of focus groups will be repeated until the themes revealed in the groups become redundant indicating a saturation point has been reached. Cost Efficiency Analysis During 2008, we are participating in a Lumina sponsored study called “Making Opportunity Affordable” along with 12 other colleges. The purpose of the study is to develop a planning model that allows colleges to analyze and project the cost and benefit of implementation of new strategies. We have selected the SLS mandate as one subject of our study. Faculty and Staff Discussions We propose to conduct a second round of facilitated discussions in Fall 2008 with faculty, deans and staff based on: trend analysis data, benefit and cost analysis, and an analysis of the “human impact” of the SLS mandate. By considering the “human impact” of the SLS mandate, we mean to expand our focus beyond the quantitative measures described earlier to include a consideration of the following: •

The number of students who persisted that likely would not have done so

•

The characteristics of these students in terms of the gaps we are trying to close through Achieving the Dream

•

The projected benefits in terms of additional credits completed by these students.

4/1/2009

SLS Mandate Research Report T dA l i Trend Analysis Achieving the Dream

4/1/2009

3 Terms Before vs. 3 Terms After the SLS 3‐Prep Mandate

RESEARCH METHODOLOGY RESEARCH METHODOLOGY

4/1/2009

90%

Comparison of Fall to Spring Average Persistence p Before and After 3‐Prep SLS Mandate All Ethnicities Combined 3‐Prep Actual

85%

80%

75%

SLS Mandate Research Methodology: The research question is whether or not there is a significant improvement after the g p mandate that can be associated with the mandate and not background effects such as economic conditions or various college improvements coincident with the mandate.

70%

65%

60% Fa03‐Sp04

Fa04‐Sp05

Fa05‐Sp06

Fa06‐Sp07

Fa07‐Sp08

Fa08‐Sp09

4/1/2009

90%

Comparison of Fall to Spring Average Persistence p Before and After 3‐Prep SLS Mandate 3‐Prep Trend All Ethnicities Combined 3‐Prep Actual

85%

80%

75%

SLS Mandate Research Methodology: 3‐Prep SLS persistence rates are averaged over the 3 years before and the 3 y yyears after the SLS mandate.

70%

65%

60% Fa03‐Sp04

Fa04‐Sp05

Fa05‐Sp06

Fa06‐Sp07

Fa07‐Sp08

Fa08‐Sp09

4/1/2009

90%

Comparison of Fall to Spring Average Persistence p Before and After 3‐Prep SLS Mandate 1‐2 Prep Trend All Ethnicities Combined 1‐2 Prep Actual 3‐Prep Trend 3‐Prep Actual

85%

80%

75% SLS Mandate Research Methodology: Similarly, the SLS persistence rates are averaged for 1‐2 Prep Students over the 3 years before and after the SLS mandate.

70%

The 1‐2 Prep Students are the best comparison group to the 3‐Prep Students and presumably have been similarly affected by economic changes and other college improvements changes and other college improvements.

65%

60% Fa03‐Sp04

Fa04‐Sp05

Fa05‐Sp06

Fa06‐Sp07

Fa07‐Sp08

Fa08‐Sp09

4/1/2009

90%

Comparison of Fall to Spring Average Persistence p Before and After 3‐Prep SLS Mandate 1‐2 Prep Trend All Ethnicities Combined 1‐2 Prep Actual 3‐Prep Trend 3‐Prep Expected Trend 3‐Prep Actual

85%

80%

75%

SLS Mandate Research Methodology: Applying the before and after changes in the 1‐2 Prep comparision group to the 3‐Prep study group, the dotted line shows the level of persistence that would be expected for the 3‐Prep population if there had been no 3‐Prep SLS mandate.

70%

65%

60% Fa03‐Sp04

Fa04‐Sp05

Fa05‐Sp06

Fa06‐Sp07

Fa07‐Sp08

Fa08‐Sp09

4/1/2009

90%

85%

80% + 1.8 % better than expected

75%

70%

SLS Mandate Research Methodology: The difference between these lines is the change in persistence rates that can be change in persistence rates that can be associated with the timing of the SLS mandate.

65%

60% Fa03‐Sp04

Fa04‐Sp05

Fa05‐Sp06

Fa06‐Sp07

Fa07‐Sp08

Fa08‐Sp09

4/1/2009

3 Terms Before vs. 3 Terms After the SLS 3‐Prep Mandate

FALL TO SPRING PERSISTENCE FALL TO SPRING PERSISTENCE

4/1/2009

3 Terms Before vs. 3 Terms After the SLS 3‐Prep Mandate

FALL TO FALL PERSISTENCE FALL TO FALL PERSISTENCE

4/1/2009

3 Terms Before vs. 3 Terms After the SLS 3‐Prep Mandate

PREP COMPLETION SUCCESS RATES PREP COMPLETION SUCCESS RATES

4/1/2009

3 Terms Before vs. 3 Terms After the SLS 3‐Prep Mandate

SUMMARY OF RESULTS PATTERNS SUMMARY OF RESULTS PATTERNS BY ETHNICITY

4/1/2009

Innovation Portfolio Spending Template - SLS1122 Pre-Mandate (2005-2006) Spending Categories

Definition of Terms: Description

Compensated Personnel

All personnel who receive some for of compensation for Comments if any work in the program - Annualized FTE

Faculty Other Academic Personnel

1 FTE = 10 sections taught (2 terms on a 4month contract)

Academic Advisor

1 FTE = 1820 work hours - Based on a 261day work year (7 hours per day)

Career Advisor

1 FTE = 1820 work hours - Based on a 261day work year (7 hours per day)

Administrative Personnel Director of Student Success Assistant VP for Academic Affairs Administrative Assistant - Student Success

Atlas Manager

1 FTE = 1820 work hours - Based on a 261day work year (7 hours per day)

Student Development Coordinator

1 FTE = 1820 work hours - Based on a 261day work year (7 hours per day)

Sub-Total: Compensated Personnel

Total Annual Expenditures

149 sections taught primarily by adjunct instructors (74 adjuncts in 2005-2006). Costs are calculated based on the pay rate for a 4-month contract plus benefits (4-month contracts are full-time temporary positions renewed on a term by term basis) - $20,273.20 per term with benefits (40,546.40 per FTE). Actual amount paid for SLS faculty in 200514.90 2006 = $307,426.93 (hourly rate plus benefits = $19.89) x 2 hours x # of SLS sections (149 0.16 Sections) (hourly rate plus benefits = $17.63) x 2 hours x # of SLS sections each 0.16 term (149 Sections)

$ / Year (From all revenue sources)

$604,141.36

$5,927.22

$5,253.74

1.00 Salary plus benefits 20% of the AVP's time. Mid-range 0.20 salary for an AVP $106,150

$85,557.75

1.00 Salary plus benefits (hourly rate plus benefits = $26.26) x 1 hour x # of SLS sections (149 0.08 sections) (Hourly rate plus benefits = $24.33) x 1 hour x # of SLS sections (149 0.08 Sections)

$48,293.19

17.58

$21,230.00

$3,912.74

$3,625.17

$777,941.17

Innovation Portfolio Spending Template - SLS1122 Pre-Mandate (2005-2006) Spending Categories

Definition of Terms: Description

Non-Compensated Personnel

Volunteers donating their time to work in the program, including parent volunteers, student work-study mentors; others - Annualized FTE

Sub-Total: Non- Compensated Personnel Total: Personnel

Total Annual Expenditures

Description of Supplies, Equipment and Expenses SLS Office Supplies, Equipment and Expenses SLS Office Computers / printers SLS Office Supplies Printing Web Development Travel In-District Out-of-State (Conferences / Training)

Sub-Total: Supplies Equipment and Instructional Materials

Student Stipends, Special Scholarships, or program-related awards paid directly to students Start-Right Convocation Speaker Fees (Fall and Spring Terms) Food Costs (Fall and Spring Terms) 8 Professional staff x Average of $35.00/hour x 5 hours x 2 events Road Map to Success Budgeted 80 students per term @ $500 each x 2 terms (actually paid $79,500 to 159 students) Advising/Counseling staff - Average pay of $31.70/hour with benefits x 5 hours x 160 students Sub-Total: Student Stipends

$0.00

0.00

Record Direct Charges to the program $2,000.00 $8,600.00 $4,383.00 $500.00 $1,000.00 $1,530.00

$18,013.00

$7,000.00 $6,000.00 $2,800.00

$80,000.00 $25,360.00 $121,160.00

Innovation Portfolio Spending Template - SLS1122 Pre-Mandate (2005-2006) Spending Categories Definition of Terms: Description Training, Professional Development Outside Speaker for Professional Development Faculty Training

Total Annual Expenditures $1,000.00

Step-by-Step Faculty Credentialing (4 hours per term for prep and presentation ). Training is done by the SLS Director and full time (4-month)SLS faculty so costs are not duplicated here.

$0.00

Start of Term Training (4 hours per term for prep and presentation) Training is done by the SLS Director and full time (4-month) SLS faculty so cost are not duplicated here

$0.00

Depart training sessions x 3 per term x (hours per term for prep and presentation ). Training is done by the SLS Director and full time (4-month) SLS faculty so costs are not duplicated here.

$0.00

Sub-Total: Training, Professional Development

$1,000.00

Cost of Space Sub-Total: Cost of Space Total Expenditures Student Expenses Tuition 66.11 per credit hour x 3 credit hours x the number of students enrolled (3512 Students) Supplies Purchased by Students Textbook - $54.80 x the number of students enrolled (3512 Students) Life Skills Notebook - $3.00 x the number of students enrolled (3512 Students) Learning Styles Inventory - $10.83 x the number of students enrolled (3512 Students)

Total Student Expenses

$0.00 $918,114.17

$696,534.96 $192,457.60 $10,536.00 $38,034.96

$937,563.52

Investing in Student Success Pilot Study:   A Resource Analysis of First Year Programs    Valencia Community College: Student Success course (SLS 1122)  This report summarizes information on program goals and outcomes of the Student Success course (SLS  1122)—including evaluation data—submitted by Valencia Community College to the Investing in  Student Success (ISS) team for the purposes of this pilot.   Valencia Community College is a public, two‐year institution and part of the Florida Community College  System. Valencia serves 30,000 students, the majority of whom are 24 years old or under; 38 percent of  the population identifies as Black or Hispanic. In 2006, Valencia awarded 38 completions (degrees,  certificates, and formal awards) for every 100 FTE students enrolled compared to the public two‐year  sector average of 24 completions awarded for every 100 FTE students enrolled.1  All Valencia students who test into developmental courses (called preparatory levels) in all three of the  academic areas offering developmental courses (reading, writing, and math) are required to enroll in the  Student Success course (SLS 1122). Additionally, students who do not test into a preparatory level—or  test into a preparatory level for one or two of the academic disciplines—can also enroll in the SLS 1122  course. The college plans to expand the SLS 1122 course and will link this course with targeted gateway  courses.   Summary of Evaluation Data  The evaluation and financial data submitted for the ISS project reported on the 2006‐07 academic year.  Evaluation data compared the retention rate of students who participated in the SLS course versus  those who did not, within each preparatory level category (e.g., needing no preparatory course, one  preparatory course, two preparatory courses, or three preparatory courses).   Valencia documented the following student outcomes:  •

Across all preparatory levels, there was a higher one‐year retention rate (fall 2006 to fall 2007)  for first‐time‐in‐college (FTIC) students who enrolled in the SLS course than those FTIC students  who did not enroll in the SLS course.  o For students who placed out of a preparatory level, there was a 6‐percentage‐point  difference (76 versus 70 percent).  o For students who tested into a preparatory level for one academic area, there was an  11‐percentage‐point difference (71 versus 60 percent).  o For students who tested into a preparatory level for two academic areas, there was a 5‐ percentage‐point difference (59 versus 54 percent).

Delta Cost Project IPEDS Database.  1

For students who tested into a preparatory level for three academic areas, there was a  13‐percentage‐point difference (49 versus 36 percent).    Note: This is not the same retention data that will be used for the resource analysis below. The  resource analysis will be calculated based on the one‐year retention rate (2006‐07) for all FTIC  students who enrolled in the SLS 1122 course—whether taken in the fall or spring semester.   o

Revenue and Expenditure Data   • • • •

The Student Success Office, a part of the Academic Affairs department, manages the SLS 1122  course.  The Student Success Office is supported with General Education revenues comprised primarily  of state appropriations and student tuitions and fees.  Direct costs of the program were $793,303—or $182 per student, per year.  The vast majority of direct program costs (88 percent) go to personnel. The breakdown of costs  by personnel type shows that approximately 68 percent goes to faculty, 26 percent to  administration, and 6 percent to other academic personnel. The faculty for SLS 1122 is primarily  made up of adjunct instructors; if full‐time contract instructors taught the SLS 1122 course, the  direct program costs would increase to $251 per student.

In order to put the program spending into context, we compare the SLS 1122 expenses per student to  Valencia’s overall education and related expenses (total educational expenses) per FTE student.  Institution‐wide average education and related expenses include all spending for instruction and student  services, as well as indirect expenses for academic and institutional support, and operations and  maintenance. To provide comparable numbers at the program‐level, we have estimated the program’s  indirect costs based on the share of education and related expenses coming from indirect costs at the  institution‐level, as shown in the table below.    Institutional and Program Expenditures per Student  SLS 1122     Valencia   $182  Direct Educational Expenses  $3,578 (52.5%)  $165  Indirect Expenses  $3,231 (47.5%)  $347  Total Educational Expenses  $6,810 (100%)  Source: Delta Cost Project IPEDS Database and Valencia Student Success Office    • The estimated total SLS 1122 2006‐07 program expenses per student are 5 percent of the  institution’s education and related expenses per FTE student from 2005‐06.    Resource Analysis  The resource analysis for the SLS 1122 program is confined to the revenue/expenditure results  associated with the increased retention that may be linked to participation in the program. According to  enrollment data:  •

4,366 students enrolled in the SLS 1122 course in the 2006‐07 academic year; 2,342 of these  were degree‐seeking, first time in college (FTIC) students. 3,049 FTIC students did not enroll in  the SLS 1122 course.

For the purpose of this analysis, 277 additional FTIC students retained across all preparatory levels will  be linked to participation in the SLS 1122 course. This number is the estimated total from comparing the  retained number of students enrolled in SLS 1122 courses (1,386 students) to the predicted number if  these students were retained at the same rate as those not enrolled in SLS 1122 courses (1,109  students) as shown in the table below.

Number of  Actual  participating  One‐year  Number of  students estimated  retention rate  to be retained if they  Participating  for students  were retained at the  Students  Retained  not  rate of non‐ participating in  SLS 1122  (One‐year  participating  Cohort  Semester  Enrollment  retention rate)  the program  students  Fall  345  263 (76%)  70%  242  No Prep  Spring  37  29 (78%)  53%  20  Fall  383  273 (71%)  60%  230  One Prep  Spring  95  56 (59%)  49%  47  Fall  386  228 (59%)  54%  208  Two Prep  Spring  112  58 (52%)  42%  47  Fall  687  339 (49%)  36%  247  Three  Prep  Spring  297  140 (47%)  23%  68  Total     2342  1386  ‐‐  1109  Source: Valencia Community College, Student Success Office

Additional  number of  students retained  that may be  associated with  participation in  the program  21  9  43  9  20  11  92  72  277

If the increase in retention could be attributed to participation in the SLS 1122 course, the following  would be true:  •

•

At $6,139 per FTE student in revenues from a combination of state and local appropriations and  net tuition revenue (based on 2005‐06 data), the 277 retained students would yield $1,700,503  in additional revenues for the institution.    The SLS 1122 program is estimated to net $185,501 after total program costs based on the  increased retention of FTIC students alone. This number is likely to be an underestimate of the  cost‐effectiveness of the program as nearly half of the students enrolled in SLS 1122 are not  included in the retention data because they are not FTIC students.

Delta Cost Project IPEDS Database.  1

Revenue and Expenditure Data   • • • •

Additional  number of  students retained  that may be  associated with  participation in  the program  21  9  43  9  20  11  92  72  277

If the increase in retention could be attributed to participation in the SLS 1122 course, the following  would be true:  •

•

Valencia SLS1122 Mandate Discussion Guide

Valencia Community College SLS 1122 Mandate for 3‐Prep Students Discussion Guide **Note: For moderator’s information only – Research Question: How do students feel about being required to take SLS 1122? Introduction: •

•

Describe o Role as moderator – need for frank and honest feedback, information confidential and anonymous, etc. o Aspects of the process and facility (set‐up, people involved, etc.) o Rules of conduct Everybody talks and participates Try not to talk at the same time Keep in mind that I am independent as the moderator I have to follow this guide o Respondent Introduction (name, age, and year in school) We’re here to find out about what influenced you in your decision to go to college and your experiences here at Valencia.

Establishing Context: •

•

Their Story o When did you start at Valencia? o Before beginning college, how long had you thought about going to college? o What motivated you to attend college? o What schools did you consider? o What were your criteria or what were you looking for in a college? o What did you discover that was good or bad about the schools? o What reservations, if any, did you have about choosing Valencia or other colleges? o What ultimately led you to choose Valencia? o How prepared did you feel to attend college? Experience with Requirements o What impact, if any, did the requirements of each school you were considering have on your decision to attend school? o Once you decided to come to Valencia, what issues came up or what requirements did you discover? o What were you expecting when considering the required courses? What requirements, if any, seemed unfair to you? (Probe why) What requirements seemed fair to you? (Probe why)

1 Achieving the Dream Data Team Revised 4/9/08

Valencia SLS1122 Mandate Discussion Guide

•

Experience with Orientation: Think back to Orientation. You may not remember everything. o What, if anything, did you feel pleased to learn about from orientation? (Probe why) o What, if anything, did you feel not so pleased about from orientation? (Probe why) o At orientation, you may have been told about courses you would have to take, what do you recall about these?

(If issues around autonomy and guidance arise, perhaps use continuum on the board to measure where students feel they lie between being told what to do and being able to decide for themselves) Objectives: •

Experience with the SLS 1122 Mandate

(If SLS 1122 has not come up): Now, none of you mentioned the Student Success course. OR (If SLS 1122 has previously come up): I’d like to go back to the Student Success course you mentioned. o o o

By a show of hands, how many of you were informed about this course at Orientation? How many of you were informed that it was a requirement? If not at orientation, when did you first learn about the requirement?

(If some students were not aware of SLS 1122 being a requirement): What impact would this requirement have had on you, had you known about it before hand? o o

What were your first reactions to hearing about this? What concerns, if any, did you have when learning about this requirement?

(If specific concerns or ideas come up): That was important to you because . . . (or) That was a drawback because . . . (Clarify what they knew about the requirement to take SLS 1122. Some students may be referring to a requirement to take prep or other requirements they may have been given): What was your understanding of the requirement? What was being asked of you? What did it mean you were required to do? (Diagnose where they were right and where they were wrong in what they already knew and felt). Experience with aspects of the SLS 1122 course o Do you recall hearing about the RoadMap to Success Award or scholarship? (If no): Well, what if you had? (If yes): Well, what impact did it have? o

•

2 Achieving the Dream Data Team Revised 4/9/08

Valencia SLS1122 Mandate Discussion Guide

Do you recall being told that you would earn college‐level credit for the Student Success course? (If no): Well, what if you had? (If yes): Well, what impact did it have? o What benefits of Student Success were communicated to you? What benefits, if any, were appealing to you when you heard about them? (Probe why) o Without being too specific about professors or the details of the course, what benefits do you see now from taking the course? o Again, without being too specific about professors or the details of the course, what drawbacks, if any, were there in taking the Student Success course? Reflection o What do you think of the requirement to take Student Success now that you’ve taken the course? o

•

Wrap Up: (Ask the students to take a few minutes, without you interfering, to write down their thoughts to this question): • What would you advise a student who’s considering Valencia and facing these same requirements? Maybe write down the 3 most important pieces of advice you’d give them. (After sufficient time to write their answers, inquire what they wrote and probe): • Why would you tell them that?

3 Achieving the Dream Data Team Revised 4/9/08

Executive Summary Valencia SLS1122 Mandate for 3-Prep Students: Qualitative Discussions with Students of Valencia Community College Orlando, FL December, 2008 Prepared by: Achieving the Dream Data Team

INTRODUCTION Valencia Community College is evaluating student reactions about the requirement to take Student Success (SLS1122) for students testing into preparatory levels in all three discipline areas (3-prep). Qualitative research was conducted to explore student perceptions since implementation of the SLS1122 mandate in order to determine how students feel about being required to take the course and whether the requirement has been a barrier to student choice to attend the college. Six 90-minute discussions were held with 3-prep students who enrolled in a Student Success course between Fall 2006 and Fall 2008. Students had either enrolled in the Student Success course alone, or were enrolled in sections that were part of a Learning Community (LinC). The sessions were held in the Spring and Fall of 2008. The six groups consisted of a total of 38 participants with the following demographic breakdown: 26 females and 12 males; 15 of age 18-19, 15 of age 20-24, and 9 of age 25 and up; 22 African American, 9 Hispanic, 8 Caucasian; 16 earned A’s, 13 B’s, 1 C, 6 D’s, 1 I, and 1 W. During the recruitment process students were screened for eligibility and variation in demographics. They were informed only that the discussion would be focused on a topic related to education, that participation was voluntary with a monetary compensation of $50. Representatives of the Data Team recruited the students from contact data provided by the Office of Institutional Research. Students were grouped according to grade in the SLS1122 course: Passed (A, B, or C), grade of D, Failed or Dropped (F, I, W), and currently enrolled to ensure a mixture of participants of each type. Contact was made via email and telephone. Nicholas Bekas, Julie Phelps, Kurt Ewen, Joyce Romano, and Roberta Brown conducted the sessions with Mary Allen, Jeff Cornett, Megan Evans and Jenelle Conner serving as note takers. The following summary was prepared using information provided by the moderators and note takers.

Valencia Community College SLS1122 Mandate: 3-Prep

Caveat-The findings presented here are offered on the basis of responses heard from a limited number of students. Qualitative research is directional in nature, intended to uncover and explore issues but not measure their prevalence in the population. Therefore, the findings here are not intended to be projected to the population. Rather, these insights should be considered carefully and in context with other strategic information. KEY INSIGHTS Characteristics of Participants

Students were very willing to share their experiences and feelings during the focus groups, whether positive or negative.

Students were frank, honest, and well-spoken.

Several students either stayed afterward or emailed to express their appreciation for the opportunity to participate in the discussion.

Variety in length of attendance at Valencia. Some students were in their first semester, while others have completed one to three semesters. Some came to Valencia because of location or cost, some because of reputation, and others because friends or family suggested.

Overall Perceptions and Attitudes about the requirement

Overall, students were not aware of prep courses or the SLS1122 requirements prior to choosing Valencia. If they did have prior knowledge, it was not a deterrent for them in making the decision to attend the college.

Here is some of the feedback given as students recalled hearing about having to take the SLS1122 course: o some recalled being handed a piece of paper with required courses listed o some felt singled out o some recall having it “sold” to them and felt better after hearing more about the course o some felt that the course would be useful for them and were indifferent or happy to be taking the course

Achieving the Dream, Data Team

Valencia Community College SLS1122 Mandate: 3-Prep

o some felt that they already possessed the skills covered in the course and therefore did not need to take it o concerned about taking a class that was not part of their major requirements o concerned about having to pay for a course that they did not choose to take o did not like that they were required to take the course while other students were not required to take SLS 1122

Although, there may have been more negative feelings than positive towards the course or the requirement to take the course, students were able to resolve those feelings by the end of the day and move on to stay and take the course.

After taking the course, upon reflection most feedback was positive about the course.

KEY IMPLICATIONS

Neither the SLS1122 course nor the perceptions prior to taking it were roadblocks to attending the college.

The SLS1122 course was not a roadblock to their success or to continuing on at Valencia.

Consideration of the course and/or the requirement did not come until orientation.

Most of the students would recommend the course to other students.

Those students that were not happy about the requirement felt they already possessed the skills covered in the course.

Students had similar feedback about the content of the course regardless of their success or failure in the course.

Achieving the Dream, Data Team

Valencia Community College SLS1122 Mandate: 3-Prep

Findings It does not appear that SLS interferes with studentsâ&#x20AC;&#x2122; decision to come to Valencia or to continue on at the college. In advice to other students, the students stressed that a positive attitude about the course was an important factor. They felt that opposition to the course would be minimized by better communicating the benefits of the course or by â&#x20AC;&#x153;sellingâ&#x20AC;? it at various decision points and in language more familiar to incoming students.

Achieving the Dream, Data Team