“I never guess. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.”—Sir Arthur Conan Doyle
Making the Grade Evaluating Your School Library Media Program: Information IS Power! e do a great deal of talking about information and the power of information, especially since Information Power: Building Partnerships for Learning (AASL, AECT) was published in 1999. We know that this is the information age, and we understand our role in making information accessible—physically and intellectually—to the school community.
W
As information professionals we know the value of information to teaching and learning, but do we really have the information we need to evaluate and improve our school library media programs? • Do we have some baseline information to evaluate our efforts to move our programs in a positive direction? • Do we know what information we need to demonstrate that goals have been accomplished and student achievement has been positively impacted? • Do we know what to do with the information we have and the information we can get? • Do we use information about our programs effectively?
Donna Baumbach, Ed.D. Professor Teaching and Learning Principles College of Education University of Central Florida baumbach@mail.ucf.edu Making the Grade is an ongoing column by Dr. Donna Baumbach, Professor of Education at the University of Central Florida and Director of the Florida SUNLINK Project, focusing on SUNLINK, technology, and the status of school library media centers in Florida and how they contribute to student achievement.
Why, you may ask, would we need information about our programs? Who knows them better than we do? Well, that’s at least partly true. We all have a good idea about the strengths and weaknesses of our collections, our facilities, our programs. That is a good place to start. What information do you need to move forward? What information do you need to put some data behind those perceived strengths or to really get a handle on what needs improvement? Some data already exists, and you know where to find it. You can find collection size, age of collection, budget amounts and sources, FCAT scores and more. That is good solid, useful data. If you need to collect information, you can observe or you can ask. Observation is a bit tricky; we are human, and we can sometimes be subjective in our observations. But if you suspect something needs attention, stand back and watch. Do students fail to use quotation marks when searching for a specific phrase? Do they have trouble creating citations for their reports? Do they consistently use the same reference tools when looking for specific information, even though better tools are available? Make a few notes about what you see, how often you see it, and what might be done about it.
| Florida Media Quarterly | Spring 2007 | Page 15 |
Making the Grade continued from 15
If you need to ask people to get the data you need, you can interview them or you can ask them to complete a written survey or questionnaire. One of the tools for continuous improvement in business is the Deming Cycle. Dr. William Edwards Deming was a statistician known for improving both the production and quality of goods in the United States and Japan. The Deming Cycle for Continuous Improvement uses the PDSA approach and can be applied to surveys. 1) The Cycle begins with PLAN. Without adequate planning, your survey will be ineffective. 2) The next step in the cycle is DO. This encompasses the actual design of the survey, selection of the sample, and collection of the data. 3) The next phase of the Cycle is the STUDY phase. This is the point where you analyze the data you have collected, draw conclusions, and report your findings. 4) The last phase of the Cycle is ACT. Here is where you determine the appropriate action steps to be taken in the light of the data. Once the actions steps are taken, the Cycle leads you back to PLAN another survey to evaluate the changes made.
PLAN Before you begin, identify the purpose for the survey. What is your objective? What kinds of information will you need? Identify the best people to give you the answers you need. You should consider asking students, faculty, administrators, parents and/or community members, depending upon the purpose of the questions and the information and perspective needed. Determine the timeframe for administering the survey and how it will be administered. Also determine how you will get the word out to the appropriate audience that you need their input. Know in advance how you will analyze, report and use the data.
DO Survey construction is not easy, but it helps if you remember some simple tips: • Keep the purpose of the survey in mind. Every question you ask should support the purpose. • Ask the people who will have the information. • Keep the questions — and possible answers — simple. Don’t ask long complex questions or try to ask about two things in one question, for example, speed and accuracy or quality and quantity. Make those separate questions. • Keep it short. • Make it easy to complete. • Do a trial run with a sample of the intended audience to identify problems with wording or instructions. • Group your questions in sections and put them in a logical order. • Avoid words like always, never, invariably, etc.
Figure 1. Deming’s PDSA Cycle
• Begin with a short description of the purpose of the survey and end with a thank you.
| Florida Media Quarterly | Spring 2007 | Page 16 |
Blogs continued from 16
There are a number of free online tools to make data collection easier. If you want to generate a paper survey, you can use any word processing or page layout program. Microsoft Word, for example, has some forms tools to help you format your survey and make it easy to read and complete. There are also some templates available to help you: • Microsoft Templates (survey templates for Windows only) http://office.microsoft.com/enus/templates/CT101436331033.aspx • Microsoft Templates (Quizzes and test templates for Windows) http://office.microsoft.com/enus/templates/CT101435311033.aspx You might ask for input by email. The poll builders below can also be distributed by email (at least the URL for the poll could be emailed): • Outlook: Voting by Email – A tutorial from Kent School District http://www.kent.k12.wa.us/KSD/IT/TSC/prof_dev/ tutorials.html In addition, this site from the Washington State Library has a number of charts, matrices, samples and worksheets that can be downloaded and changed to meet individual library needs. Several might be adapted or used to generate data for library media center use. http://www.k12library.info/toolkit/chartsforms.html Since many library media center users (and non-users) are online these days, you might want to try collecting data using an online survey. This allows participants to enter the data at their own leisure from the classroom, home, the public library, or your library media center. You can direct prospective survey participants to your questionnaire by creating a link to your survey from an existing web site (or sites), by emailing the link to potential survey participants, or by publishing the link in paper fliers, post cards, newspaper
advertisements, etc. Here are some sample tools for you to consider: • Survey Builder, http://chnm.gmu.edu/tools/surveys/index.php, allows you to easily create and manage online surveys. You do not need to know how to build a Web page that has forms, set up a database to store entries, or do any of the other technical tasks that are normally required to produce interactivity on the Internet. Survey Builder does all of that for you. After contributors fill out your questionnaire, you can choose to post their responses on a Web page devoted specifically to your survey. Survey Builder also allows you to view and organize your surveys, and even exports the survey responses into a database worksheet, like Excel, for tabulation and analysis. The survey is hosted at the Center for History and New Media at George Mason University. • Survey Scholar, http://www.surveyscholar.com/index.html, is designed especially for education. It is free for students and not too expensive for educators, about $20 a month for up to 2000 responses and a penny more for each one after that. There are a few templates available, and you can generate email invitations to participate in the survey. • Quia, http://www.quia.com/, offers a free 30-day trial and is $49 for a full year subscription. Quia lets you easily author surveys and a variety of games, activities, quizzes and webpages. It also allows you to use what
Figure 2. Survey Rating Grid Generated in Quia | Florida Media Quarterly | Spring 2007 | Page 17 |
Blogs continued from 17
other people have created, so you might find a useful survey you can modify to meet your needs. • Zoomerang, http://info.zoomerang.com/, is free online survey tool; although a pro version is available that has much more power and reporting capabilities. Zoomerang offers templates, has a “survey coach” to help you construct your survey, offer skip logic (if you answer “yes” to one question, for example, it may automatically skip a follow-up question that you would see if you answered “no”) and lets you generate a URL to send to participants or import email addresses to ask for participation. There are a number of tools offered to help you analyze the data you receive. SurveyMonkey, http://www.surveymonkey.com/, offers a basic subscription for free and more features for $20 a month. Other Features are similar to Zoomerang. You might also be able to modify some quiz generators to gather the data you need: • Discovery Quiz Center http://school.discovery.com/quizcenter/quizcenter.html • QuizStar http://quizstar.4teachers.org/ For short poll-like data collection and instant feedback, try: • PollBuilder, http://chnm.gmu.edu/tools/polls/, lets you create a poll quickly and let participants choose from one of up to five answers. PollBuilder generates the code to imbed the poll on your own webpage.
Figure 3. Form for creating a poll using PollBuilder Figure 4. Poll resulting from input on PollBuilder
• SnapPoll, http://www.snappoll.com/, will host your poll or generate the code to embed it on your webpage. You can choose your own colors and layout and prevent people from voting more than once. The Freesite (www.thefreesite.com) has a number of free poll generators similar to these.
Figure 5. Poll results generated by SnapPoll
| Florida Media Quarterly | Spring 2007 | Page 18 |
Blogs continued from 18
STUDY Deal with the data. Use appropriate statistics: Averages, percentages, ratios, frequencies, totals, and correlations. If you need assistance, ask for it. There are people in every district who know how to deal with data, and analyzing the data and drawing appropriate conclusions are critical to your success. Use Microsoft Excel® or other spreadsheet software to record, compute, analyze and graph your data; that will help you — and others — make sense of it. Create tables to show your data in an easy to understand format. The National Center for Educational Statistics offers a graph generator for kids that have the power to create five different kinds of graphs that might be useful in your analysis and reports at CreateAGraph (http://nces.ed.gov/nceskids/ createagraph/). If you’re looking for other ways to make your data more visual, try the Period Table of Visualization (http://www.visual-literacy.org/periodic_table/ periodic_table.html). Be sure to make your findings available to participants and let them know what actions will be taken so they know their opinions and time are valued. Post the results on a website or bulletin board, give a presentation, or create a brochure. Write articles for the school newspaper or parent newsletter. Then, use the data to
ACT
What does this data mean? Students and parents generally feel online resources are adequate; teachers do not. What are the possible causes for this discrepancy? Possible solutions? How can we improve all perceptions? How can we improve our online resources? What follow-up questions should be asked?
Figures 6. Table and graph generated by a spreadsheet (Excel®)
Data is meaningless and the survey process is an exercise in futility if you don’t do something with it. Remember the purpose for the survey in the first place! You had an objective, and you collected data to meet that objective. Collecting data is important, but how you use it is more important. Often you’ll find that the answers to your questions will generate more questions. That’s what makes the cycle go on and on. Meet with people who can help you set and meet goals: your administrators, teachers, parents, students; and share what you have learned. Data can give you a basis for
making better decisions about your program, your collection, your website, your teaching, your services, your public relations, your technology resources, your advocacy efforts. Involving others in data collection, analysis and reporting will help them understand how and why you made your decisions. Data and data analysis can help you reflect, celebrate success, and plan for improvement. Create a plan or rubric and chart your progress. How do you know you’re making progress? Plan, do, study, act! Information is power, indeed!
| Florida Media Quarterly | Spring 2007 | Page 19 |
Blogs continued from 19
“Data is not information; information is not knowledge; knowledge is not wisdom.”—Cliff Stoll and Gary Schubert Recommended Resources: • Achieving Exemplary School Libraries http://ed.sc.gov/agency/offices/tech/ms/lms/ LMCManagement.html Sample survey forms for Administrators, Students and Faculty • Collecting Data: Templates and Resources for the School Library Media Specialist http://www.nobl.k12.in.us/media/NorthMedia/lms/data/ index.htm Sample surveys and templates and some good advice on how to use and share the data and findings. • Expectations for Collaboration, Collections, and Connections to Enhance Learning: A Program Evaluation Rubric (ExC3EL) • ExC3EL Scoring Rubric • ExC3EL Program Improvement Plan Available from the Florida Department of Education, Office of School Library Media Services (http://www.firn.edu/doe/bii/itlm/libmedia/index.html)
• K-12 Annual Report Templates and Examples http://www.k12library.info/toolkit/annualreport.html Templates for developing an annual report for your school library media center and two examples of reports on the web • SWOT: Strengths, Weaknesses, Opportunities, Threats http://en.wikipedia.org/wiki/SWOT_analysis Describes a strategic planning tool frequently used in business but appropriate to the school library media program as well • What Gets Measured Gets Done: A School Library Media and Technology Program Self-Study Workbook http://www.doug-johnson.com/wgm/wgm.pdf Organizing for evaluating your program, gathering and analyzing data, writing the report and sharing the findings.
• Indiana Learns: Data Driven Practices http://www.indianalearns.org/datadriven.asp “Library media specialists and technology specialists are presented with an overall scheme of collecting evidence upon which the program can be measured and used to continuously monitor the impact they are having.”
| Florida Media Quarterly | Spring 2007 | Page 20 |