Connection 2013 Issue 3 Technology That Works
People Who Care
Myths of Item Writing Jennifer Reed Marketing Manager
H The most important predictor for a successful item writer is their willingness to serve and their love of the profession.
igh quality items (i.e., test questions) are the cornerstone of any professionally sound, legally defensible credentialing examination. Many things go into creating these items: matching the item to test specifications, avoiding bias, ensuring only one best answer, remaining relevant to practice, etc. For many organizations, an item writing workshop will produce hundreds of draft items, and all of them will not make their way to a live test form. Item writers are typically experts in their field, but they can’t be expected to instinctively understand the nuances of item writing. In fact, some item writers initially have misconceptions about the process or what makes a good item. After more than 30 years working with item writers across a wide range of professions, we have encountered our share of item writing myths.
All Items Should be Difficult Some item writers feel it is their job to stump the candidate – an item is only good if candidates can’t get it correct. Wrong! If every item on the test is too difficult, the test may not appropriately differentiate the knowledge needed to identify competence. While the majority of the items should be considered to be moderately difficult, a range of difficulties (from fairly easy to fairly
difficult) makes the best test. The myth that “we need to include a few items that everyone can get right” is also detrimental to measurement precision. Sometimes item writers have trouble judging the level of difficulty of their items, so when writing items, it is best to not worry too much about the difficulty. Focus on appropriate content. Difficulty will be determined later through pretesting, and the items that are appropriately difficult and differentiate those who have the knowledge from those who do not will be available to compute candidates’ scores.
The More Detailed the Better “Let’s throw in all kinds of detail and see if the candidate can sift through what is important and what isn’t. After all, that is real life.” The reality is candidates only have a set amount of time to take an examination and it is typically most helpful if they can focus on the important issues when responding to items. Item writers should always focus on the question: “What are we trying to test?” We usually aren’t trying to test the candidates’ ability to read and sift through unimportant details; the tasks should be relevant to practice. So, providing enough detail to make an item realistic and ensure the distractors are plausible continued on page 2
New Business
2
www.goAMP.com
A Day in the Life of a Meeting Planner
3
On the Road
4
Stay Connected
4
Published by Applied Measurement Professionals, Inc. © 2013
““Myths of Item Writing” continued from page 1 is important. Drowning the candidate in irrelevant details isn’t recommended. However, if given the choice of too much or too little information from the item writer, the examination committee will normally prefer too much. If too little information is provided they sometimes can’t discern the intended purpose of the item.
Only Seasoned Professionals Can Write Good Items Some organizations want only individuals with years of experience to serve as item writers, but it is important to remember they aren’t the only ones who can write great items. New practitioners often have a clear view of entry-level practice and a fresh enthusiasm, which can be beneficial to item writing. Similarly, not all seasoned professionals are good item writers. Sometimes they may be too immersed in the unique
New Business:
The National Committee for Quality Assurance is a not-for-profit organization dedicated to improving health care quality. Since its inception in 1990, NCQA has been a central figure in driving improvement throughout the health care system, helping to elevate the issue of health care quality to the top of the national agenda. Towards this goal, NCQA develops quality standards and performance measures for a
aspects of their own practice, or have preconceived notions of the examination or other issues from the past. The most important predictor for a successful item writer is their willingness to serve and their love of the profession. Diversity among item writers is always valued.
My Item Must be Perfect from the Start Item-writer’s block is a real thing. It mainly stems from the pressure of creating and submitting only perfect, awesome items. We know that items become perfect only after thorough consideration and discussion by the examination committee. Much like with any other type of writing, the best approach is just to start writing and see what comes of it, following the guidelines provided in an item writing guide or workshop. Some items won’t turn into anything useable. Some items will never be presented to a candidate. Those that are, will be edited thoroughly, by both a test development specialist and an examination committee. The most helpful thing item writers can do is provide the committee with a variety of solid items labeled to the appropriate content category. Items are the raw materials — many will be winners, some won’t.
If I Don’t Have a Reference, I Shouldn’t Write It This isn’t always true. For a quality item that covers important practice-related tasks, a strong consensus among subject-matter experts may be enough to adequately support the item. The individual responsible for item banking will keep good records of the approval of the item for use. Stable statistics during pretesting or administration cycles will back that up. However, some programs require a reference establishing evidencebased practice, and in those situations a reference (or even multiple references) may be required. For these programs, including items in the bank that have not been properly referenced will only slow down the item review and approval processes. Knowing the requirements of the program will help lead item writers in this situation.
Items Should Teach the Candidate Candidates sitting for a certification or licensing examination have already met the educational prerequisites and have been learning the content. At this point we are testing the candidate – not teaching them. Items should be concise, require responses continued on page 4
broad range of health care entities. NCQA has selected AMP to provide consultation on development of test specifications, as well as examination development and computer-based test (CBT) administration services for the Patient-Centered Medical Home (PCMH) Content Expert Certification program. To learn more about NCQA, please visit http://www.ncqa.org.
www.goAMP.com
A Day in the Life of a Meeting Planner Chances are, you have attended several meetings and/or conferences during your professional career. From roundtables and workshops to lectures and symposiums, these events provide a perfect platform to get updated on research, trends, and other hot topics within your profession. And, perhaps just as important, they provide a fun environment to bring together colleagues and meet new people that share a common interest. Since your time is valuable, it is important that these meetings are run efficiently and that you feel like you are getting information that you can take away and apply in your job. If you have never been on the planning end of one of these meetings, you may not appreciate just how much preparation and organization goes into selecting a location, arranging hotel and meeting rooms, securing audio/visual equipment, planning educational sessions, booking speakers, and the list goes on and on. This is why AMP’s association management clients put their trust in our experienced meeting planners, to handle the details before, during and following their events. We recently asked one our meeting planners to journal a work day for us to give us a glimpse into the frenzy of preparation.
8:00 am My first day back after a successful 3-day annual conference we
planned for a healthcare association. While I am able to keep up with urgent messages while attending the conference, I am still greeted by a full inbox of emails to weed through. 8:45 am I receive a concerning email this morning from an attendee of an upcoming symposium. She was unable to book a hotel room, being told it was “sold out”. Knowing it wasn’t, I immediately call the host hotel and discovered the room type requested was not available leading to a miscommunication. The hotel agreed to work with the attendee and I follow up to let her know she can now book her reservation. 9:35 am I schedule an email blast to exhibitors for a conference next month, confirming their booth location and other important deadlines and information. 10:00 am Off to a conference room to have a web meeting with a Program Planning Committee. We hope to get a theme selected for their next educational conference. It is in Las Vegas, so I’m sure we can come up with something different and fun. 11:00 am I make another round of edits to a client’s Exhibitor Prospectus. This is not their typical meeting program book–they have decided to deploy a mobile device conference app and print a smaller exhibit guide which impacts sponsorship opportunities and other event details. 11:40 am I review a hotel conference room layout for an upcoming client board meeting and create a specification sheet for the hotel. 12:00 pm A few coworkers stop by my desk and invite me to lunch. I am busy, but think this will be a good opportunity to not only socialize but discuss hotel options for an upcoming client event in St. Louis. 1:00 pm Following a phone call from our accounting department, I review a catering invoice from a recent client meeting. There were some questionable charges on the invoices, so I call the restaurant and get the invoice adjusted. 1:55 pm Time for our weekly planner meeting. All meeting planners on staff gather to touch base on their upcoming events. This is a great time to
3:00
3:25
4:20 4:30 5:00
trade information about events and properties and share new ideas for our clients’ meetings. pm I come back from my meeting to find an email with an artwork approval from a Board president. I look it over one more time and submit the artwork to a promotional products vendor that is supplying the tote bags for the annual meeting. pm A hotel contract has arrived in today’s mail. I immediately open it and review it for one of our nursing clients. The hotel agreed to use the standard AMP hotel contract template but did request a few minor revisions to the language. The revisions are reasonable, so I sign off on the changes and get it mailed back to the hotel. pm Being a good team member is important in our management services division. This would explain why three professionals are crawling on the floor looking for a coworker’s lost contact lens. Found it! pm An exhibitor for an upcoming financial conference calls to request the attendee mailing list. I direct her to the form in the exhibitor packet and instruct her to email it to the client’s data administrator. pm There are times when I can head out at 5:00, but not after being out of the office at a meeting. Just too much email to reply to. For the next 45 minutes, I will have uninterrupted time to clear out my inbox and then I will call it a day.
the
““Myths of Item Writing” continued from page 2
ROA D
ABC Annual Conference 01/14/2014 - 01/17/2014 Charleston, SC AMC Institute Annual Meeting 02/26/2014 - 02/28/2014 Tempe, AZ ATP Innovations in Testing 03/02/2014 - 03/05/2014 Scottsdale, AZ ABNS Spring Assembly Meeting 03/07/2014 - 03/08/2014 Tampa, FL ARELLO Mid-Year Meeting 04/09/2014 - 04/12/2014 San Diego, CA
We look forward to seeing you at one of these upcoming On the Road events!
relevant to current practice, and identify candidates who can apply their knowledge for competent practice. There is a temptation to put extra information in an item for teaching, but sometimes that information can disclose an answer to another item (i.e., cueing). If a candidate fails, remediation or additional learning may be necessary, but the test is not the time to provide it. Of course we want candidates to learn from the process, but the time for learning is before, not during, the test administration. An item writing workshop will help debunk these and other common item-writing myths. Helping item writers focus on the important aspects of good measurement will keep misconceptions from tainting the items produced. Nobody is perfect, even the best item writers will produce some flawed items from time to time. The goal is to produce enough to accurately measure the domain of interest.
Holiday Schedule In observance of holidays, the AMP offices, test sites and candidate call center will be closed December 24-26 and December 31 – January 1.
Stay Connected
Visit www.goAMP.com and join our mailing list to receive the eConnect newsletter or sign up for RSS feeds for news and press releases.
For more information about any of our products or services, please contact the AMP Marketing department at 913.895.4600 or visit our website at www.goAMP.com.