Test Bank for Educational Research Fundamental Principles and Methods, 8th Edition by James McMilla by ACADEMIAMILL

Chapter 1: Introduction to Research in Education Chapter 1 Learning Outcome Quizzes Learning Outcome 1.1: Know how the purpose of research, finding the signal, is influenced by noise that distorts what is reported. [Q1] Considering the signal and noise metaphor from Chapter 1, which of the following represents “signal?” 1. Error 2. Deviation 3. True information [correct] 4. Randomness [Feedback for Answer Choice 1] Recall that the noise is composed of factors that drown out, distort, or dilute the signal. Error serves to distract and is therefore considered noise. [Feedback for Answer Choice 2] Recall that the noise is composed of factors that drown out, distort, or dilute the signal. Deviations cause difficulty for researchers and are considered noise. [Feedback for Correct Answer 4] The signal is what is correct, actual, and true. True information certainly falls into this category. [Feedback for Answer Choice 4] Recall that the noise is composed of factors that drown out, distort, or dilute the signal. There is always the possibility that the results of a study are simply due to randomness, which is why study replication is important. [Q2] Considering the signal and noise metaphor from Chapter 1, which of the following represents “noise?” 1. Reality 2. Randomness [correct] 3. Actual relationship

4. Accurate understanding [Feedback for Answer Choice 1] Recall that the signal is what is correct, actual, and true. When this definition is considered, reality would be classified as “signal” and not “noise.” [Feedback for Correct Answer 2] Recall that “noise” is composed of factors that drown out, distort, or dilute the signal. Randomness is always a possible reason for the occurrence of a set of data, and therefore should be considered noise. [Feedback for Answer Choice 3] Recall that the signal is what is correct, actual, and true. When this definition is considered, actual relationships would be classified as “signal” and not “noise.” [Feedback for Answer Choice 4] Recall that the signal is what is correct, actual, and true. When this definition is considered, accurate understanding would be classified as “signal” and not “noise.” [Q3] Which of the following represents the components of an observed finding? 1. Signal plus Noise [correct] 2. Signal 3. Noise 4. Either signal or noise [Feedback for Correct Answer 1] Observed findings in educational research are always the result of signal and noise. There are always variables that will be accurate and there will always be distractors. A good researcher understands how to amplify the signal and drown out the noise. [Feedback for Answer Choice 2] Signal refers to what is correct, actual, and true, but this is not all that is included when collecting data in educational research. [Feedback for Answer Choice 3] Noise refers to factors that drown out, distort, or dilute the signal. Therefore, observed findings certainly do not consist of pure noise. 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 4] Observed findings are more complex than simply including either signal or noise. Learning Outcome 1.2: Explain why empirical research is important as an approach to generating and applying knowledge. [Q1] People use the term research to describe many processes that would not be considered research using the definition given in the text, such as writing a “research paper” from the library. A more complete definition of empirical research includes which of the following? 1. A systematic process of gathering and interpreting data [correct] 2. A disciplined process of reporting information 3. An experimental method for collecting and analyzing data 4. An organized, step-wise process for evaluating knowledge [Feedback for Correct Answer 1] This definition involves a systematic, disciplined inquiry process, which can include gathering information and facts, referencing material, collecting and analyzing data, and making inferences. [Feedback for Answer Choice 2] Research is certainly a disciplined process, but this is not the best answer choice. [Feedback for Answer Choice 3] Experimental research is important, but it is not the only approach to research. [Feedback for Answer Choice 4] Research must be organized, but research is not only about knowledge evaluation. [Q2] As sources of knowledge, experience, intuition, and tradition serve which of the following roles in educational research? 1. They have a significant role in helping to choose a research question but should play no part in analysis or conclusions. [correct] 2. They tend to be biased and do not have a major role when conducting educational research. 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. They have a limited role in aspects of educational research but are not as useful as logic and reason. 4. They have a role in educational research that must be balanced with attention to their limits and potential biases. [Feedback for Correct Answer 1] Intuition, experience, and tradition all help researchers identify important questions, develop useful research methods, collect and analyze data, and draw conclusions. Logic and reason are useful sources of knowledge as well, but it’s not appropriate to say they are more or less useful than intuition and tradition. People use all of these sources when conducting and reporting research; it’s simply important to understand and respect the limits of each. [Feedback for Answer Choice 2] Of course, there is bias involved at some level; however, this does not mean they do not have value to educational research. [Feedback for Answer Choice 3] While their role may be limited, they are certainly important to educational research. [Feedback for Answer Choice 4] In empirical research, knowledge, intuition, and tradition are important, but they do not have a role that is balanced with more empirical processes. [Q3] Which of the following terms best refers to a systematic process of gathering, interpreting, and reporting information? 1. Empirical research [correct] 2. Educational research 3. Quantitative research 4. Qualitative research [Feedback for Correct Answer 1] Empirical research involves a systematic process of gathering, interpreting, and reporting information. [Feedback for Answer Choice 2] Educational researcher refers to a systematic, disciplined inquiry applied to gathering, analyzing, and reporting information that addresses educational problems and question. 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] Quantitative research refers to a type of research that uses statistical processes and numeric approaches to understanding research solutions. [Feedback for Answer Choice 4] Qualitative research refers to a type of research that uses interviewing and other nonstatistical approaches to understanding research solutions. [Q4] Which of the following terms best describes a systematic, disciplined inquiry applied to gathering, analyzing, and reporting information that addresses educational problems and questions? 1. Empirical research 2. Educational research [correct] 3. Quantitative research 4. Qualitative research [Feedback for Answer Choice 1] Empirical research involves a systematic process of gathering, interpreting, and reporting information. However, the definition in this question is more nuanced than this. [Feedback for Correct Answer 2] Educational research refers to a systematic, disciplined inquiry applied to gathering, analyzing, and reporting information that addresses educational problems and question. [Feedback for Answer Choice 3] Quantitative research refers to a type of research that uses statistical processes and numeric approaches to understanding research solutions. [Feedback for Answer Choice 4] Qualitative research refers to a type of research that uses interviewing and other nonstatistical approaches to understanding research solutions. Learning Outcome 1.3: Know the characteristics of major research paradigms. [Q1] Which of the following terms includes philosophical assumptions, shared beliefs, and a theoretical framework about the world and knowledge that guide thinking and methods employed in empirical research? 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

1. Research outlook 2. Research method 3. Research paradigm [correct] 4. Research reporting [Feedback for Answer Choice 1] Research outlook is not the correct answer. This term may refer to how one views the trajectory of their overall body of work, but this is not correct. [Feedback for Answer Choice 2] A research method may refer to a particular way of conducting research, but this is not the correct answer. [Feedback for Correct Answer 3] A research paradigm includes philosophical assumptions, shared beliefs, and a theoretical framework about the world and knowledge that guide thinking and methods employed in empirical research. [Feedback for Answer Choice 4] Research reporting refers to the way in which one communicates the findings of their research. [Q2] To which of the following research paradigms would a researcher subscribe if she valued objectivity, measurement, and quantification while ascribing more to probability than to certainty? 1. Transformative 2. Pragmatism 3. Constructivism 4. Postpositivism [correct] [Feedback for Answer Choice 1] This paradigm ascribes to the belief that the purpose of research is to improve the lives of marginalized, disenfranchised people. [Feedback for Answer Choice 2] The pragmatic researcher focuses on adapting methods to provide the best information based on real-world situations and conditions. 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] Constructivists ascribe to the idea that knowledge is socially constructed, contextualized, and personalized. [Feedback for Correct Answer 4] Postpositivists believe more in probability than certainty, using objective methods in concert with other, more intuitive judgments about the nature of causation. It basically takes a more holistic, less reductionist view of research, but it is still very much wedded to the value of objectivity, measurement, quantification, law generation, and other characteristics of the scientific method. [Q3] To which of the following research paradigms would a researcher subscribe if she felt that the function of research was to improve the lives of marginalized people? 1. Pragmatism 2. Transformative [correct] 3. Positivism 4. Constructivism [Feedback for Answer Choice 1] The pragmatic researcher focuses on adapting methods to provide the best information based on real-world situations and conditions. [Feedback for Correct Answer 2] Researchers ascribing to this paradigm ascribe to the belief that the purpose of research is to improve the lives of marginalized, disenfranchised people. [Feedback for Answer Choice 3] Positivists ascribe strictly to the scientific method, believing that a phenomenon can be studied like the natural sciences. [Feedback for Answer Choice 4] Constructivists ascribe to the idea that knowledge is socially constructed, contextualized, and personalized. [Q4] To which of the following research paradigms would a researcher subscribe if she believed that research should be approached in such a way that methods are adapted to provide the best information based on real-world situations and conditions? 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

1. Postpositivism 2. Pragmatism [correct] 3. Constructivism 4. Transformative [Feedback for Answer Choice 1] Postpositivists tend to lend toward quantitative methods, relying on probability modeling to reach research conclusions. [Feedback for Correct Answer 2] The pragmatic researcher focuses on adapting methods to provide the best information based on real-world situations and conditions. [Feedback for Answer Choice 3] Constructivists adhere to a belief that knowledge is socially constructed and tend to rely on qualitative methods. [Feedback for Answer Choice 4] Transformative researchers believe the function of research is to improve the lives of marginalized people. Learning Outcome 1.4: Understand how the principles of disciplined inquiry (“fox-like” thinking) are important for conducting educational research. [Q1] A researcher who proceeded to design a study without conducting a review of the relevant literature on the topic would violate which of the six disciplined inquiry principles? 1. Disciplined Inquiry Principle 2 [correct] 2. Disciplined Inquiry Principle 3 3. Disciplined Inquiry Principle 4 4. Disciplined Inquiry Principle 5 [Feedback for Correct Answer 1] Discipline Inquiry Principle 2 includes linking research to relevant theory. A researcher who skipped the literature review process would not be able to complete this vital component of the research design process.

[Feedback for Answer Choice 2] Discipline Inquiry Principle 3 includes using methods that permit direct investigation of the question. [Feedback for Answer Choice 3] Discipline Inquiry Principle 4 includes providing a coherent, explicit, and evidence-based chain of reasoning. [Feedback for Answer Choice 4] Discipline Inquiry Principle 5 includes replicating and generalizing across studies. [Q2] If a researcher failed to pose questions that could be explored empirically, which of the six principled of disciplined inquiry would he violate? 1. Disciplined Inquiry Principle 1 [correct] 2. Disciplined Inquiry Principle 2 3. Disciplined Inquiry Principle 3 4. Disciplined Inquiry Principle 4 [Feedback for Correct Answer 1] The very first principle of direct inquiry is to pose significant questions that can be investigated empirically. If this step is skipped, the study is flawed from the start. [Feedback for Answer Choice 2] Disciplined Inquiry Principle 2 is focused on linking research to relevant theory. [Feedback for Answer Choice 3] Discipline Inquiry Principle 3 includes using methods that permit direct investigation of the question. [Feedback for Answer Choice 4] Discipline Inquiry Principle 4 includes providing a coherent, explicit, and evidence-based chain of reasoning. [Q3] Which of the following is a characteristic of how “foxes,” or principled researchers, think? 1. Specialized 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. Multidisciplinary [correct] 3. Stubborn 4. Overconfident [Feedback for Answer Choice 1] Thinking in specialized way is a characteristic of “hedgehog-like” researchers, or researchers who fail to realize flaws in their thinking. [Feedback for Correct Answer 2] Researchers who consider multidisciplinary approaches are “fox-like” researchers who are nimble and open-minded. [Feedback for Answer Choice 3] Being stubborn is a characteristic of “hedgehog-like” researchers, or researchers who fail to realize flaws in their thinking. [Feedback for Answer Choice 4] Being overconfident is a characteristic of “hedgehog-like” researchers, or researchers who fail to realize flaws in their thinking. [Q4] Which of the following is characteristic of a “hedgehog” researcher, or a researcher who is not aware of the flaws in his thinking? 1. Adaptable 2. Stalwart [correct] 3. Self-critical 4. Cautious [Feedback for Answer Choice 1] The fox-like researcher is adaptable to change. [Feedback for Correct Answer 2] Hedgehog-like researchers are stalwart, sticking to the same “all-in” approach. [Feedback for Answer Choice 3] Fox-like researchers are self-critical and often analyze their approach and consider new pathways. 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 4] Fox-like researchers are cautious researchers who consider that their findings have limitations. Learning Outcome 1.5: Distinguish among key characteristics of quantitative, qualitative, and mixed methods types of research. [Q1] Which of the following procedures is most aligned with a researcher using quantitative methods? 1. Conducting a large number of participant interviews 2. Focusing on the participants’ views and perceptions 3. Studying an interesting phenomenon in the context in which it naturally occurs 4. Studying a large number of participants as a detached, objective observer [correct] [Feedback for Answer Choice 1] This approach is most likely taken by a qualitative researcher. [Feedback for Answer Choice 2] This approach is most likely taken by a qualitative researcher. [Feedback for Answer Choice 3] This approach is most likely taken by a qualitative researcher. [Feedback for Correct Answer 4] The degree of interaction with participants is usually low for quantitative research, and the researcher seeks a larger number of participants. Many quantitative studies are experiments, but not all. If a study involves measurement and numerical data, it can be considered quantitative in nature. [Q2] Which of the following best describes qualitative research? 1. Research that includes generalizable conclusions 2. Research that focuses on participants and settings as they exist naturally [correct] 3. Studies using whatever procedures that provide the best answers to the research questions 4. Studies strictly focused on improving the lives of marginalized people 11 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 1] This is a characteristic common to quantitative studies. [Feedback for Correct Answer 2] Qualitative researchers do not typically control or manipulate the environment they are studying. [Feedback for Answer Choice 3] This is a characteristic of researchers following a pragmatic paradigm. [Feedback for Answer Choice 4] This is a characteristic of researchers following a transformative paradigm. [Q3] Of the following research methods, which is most likely an approach taken in a mixed methods study? 1. Conducting interviews with participants after completing an instrument to assess instructional approaches to explain the results [correct] 2. Comparing numeric test results across groups to understand the effects of a particular instructional approach 3. Exploring teachers use of formative quizzes within three second-language classrooms 4. Describing the relation between students' math attitudes and their math achievement [Feedback for Correct Answer 1] This study involves a quantitative instrument of instructional approaches followed by qualitative interviews to explain the results of the instrument. The integration of qualitative and quantitative data is a key characteristic of mixed methods research. [Feedback for Answer Choice 2] This would be typical of a quantitative study. [Feedback for Answer Choice 3] This is likely a qualitative study. [Feedback for Answer Choice 4] This approach would be common in a quantitative design. 12 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q4] Which of the following describes the main difference between experimental and nonexperimental research? 1. Investigators collect quantitative data in experimental research and qualitative data in nonexperimental research. 2. Investigators can make inferences about causal relationships in experimental research but not in nonexperimental research. 3. Investigators have control over one or more interventions in experimental research but do not have control in nonexperimental research. [correct] 4. Investigators study large groups of people in experimental research but small groups or single subjects in nonexperimental research. [Feedback for Answer Choice 1] This may be true sometimes, but this is certainly not universally true. [Feedback for Answer Choice 2] This is not an accurate statement regarding either research method. [Feedback for Correct Answer 3] Experimental research is defined by the manipulation of an intervention or control of a factor to investigate its causal relationship with some other factor. Some types of nonexperimental studies allow causal inferences, and many involve collection of quantitative data. [Feedback for Answer Choice 4] While this may sometimes be true, this is not universally true. [Q5] A study designed to improve educational practice in a specific classroom is best characterized as which of the following types of studies? 1. Applied research 2. Qualitative research 3. Action research [correct] 4. Quantitative research

[Feedback for Answer Choice 1] Applied research is aimed at solving specific problems, but this may not be in a classroom setting. [Feedback for Answer Choice 2] Qualitative research is a broad classification of any research using qualitative methods such as interviewing and participant observation. [Feedback for Correct Answer 3] In the context of education, the goal of action research is to learn something that helps an educator, policy maker, or other stakeholder improve practice immediately within one or a few specific classrooms or schools. Action research is often applied, often qualitative, and often nonexperimental, but the key feature highlighted here—improving practice in a specific classroom—points toward action research as the best answer. [Feedback for Answer Choice 4] Quantitative research is a broad classification that applies to any research using statistical processes to determine study findings.

Chapter 1 Application Exercises Chapter 1: Introduction to Research in Education Application Exercise 1.1: Qualitative or Quantitative? Learning Outcome 1.5: Distinguish among key characteristics of quantitative, qualitative, and mixed methods types of research. [Q1] Can you distinguish between qualitative and quantitative data? Rachelle J. is a middleand high-school language arts and Spanish student teacher. She conducted a research project addressing the effects of formative assessment on participants’ summative assessment scores, morale, and attitude. Read through Rachelle’s report of her research, focusing specifically on the methodology sections of the paper and answer the following questions: What is Rachelle’s first research question? Did Rachelle collect qualitative data, quantitative data, or both? [Q1 Model Response] Her first question is, Do formative assessments motivate students to study harder? To answer this question, she conducted observations and kept field notes (qualitative data). She also collected survey data, which can be qualitative or quantitative, depending on the survey—she does not say directly. Because she is focused on student perspectives, qualitative data would be appropriate here. Moreover, she says she asked students “whether or not they studied more because of the weekly quizzes” and does NOT say something like, “I asked them to agree or disagree on a scale of 1 to 5.” As you will see in a later chapter, scales of that sort are used in quantitative studies. As you become more experienced at reading research articles, you will be able to tell from her description of methodology and especially her data analysis that she used qualitative measures, even though she has “numbers” in her results. [Q2] Can you distinguish between qualitative and quantitative data? Rachelle J. is a middleand high-school language arts and Spanish student teacher. She conducted a research project addressing the effects of formative assessment on participants’ summative assessment scores, morale, and attitude. Read through Rachelle’s report of her research, focusing specifically on the methodology sections of the paper and answer the following questions: What is Rachelle’s second research question? Did Rachelle collect qualitative data, quantitative data, or both? 15 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q2 Model Response] The question was, Do formative assessments reduce test anxiety for students? To answer this question, the researcher kept field notes, recording student comments about fear of tests that occurred during her observations (qualitative measures). She also says she asked students “whether or not the weekly quizzes helped to alleviate their anxiety towards larger summative exams.” Because she is focused on student perspectives, qualitative data would be appropriate here. [Q3] Can you distinguish between qualitative and quantitative data? Rachelle J. is a middleand high-school language arts and Spanish student teacher. She conducted a research project addressing the effects of formative assessment on participants’ summative assessment scores, morale, and attitude. Read through Rachelle’s report of her research, focusing specifically on the methodology sections of the paper and answer the following questions: What is Rachelle’s third research question? Did Rachelle collect qualitative data, quantitative data, or both? [Q3 Model Response] The question was, Do formative assessments improve summative assessment scores? Only quantitative data were collected to address this question: test and quiz scores. [Q4] Can you distinguish between qualitative and quantitative data? Rachelle J. is a middleand high-school language arts and Spanish student teacher. She conducted a research project addressing the effects of formative assessment on participants’ summative assessment scores, morale, and attitude. Read through Rachelle’s report of her research, focusing specifically on the methodology sections of the paper and answer the following questions: What is Rachelle’s fourth research question? Did Rachelle collect qualitative data, quantitative data, or both? [Q4 Model Response] The question was, Does the effect of formative assessments improve class morale? The author notes she relied on the same (qualitative) measures: observing and taking field notes and having students complete the end-of-term survey. The author mentions attrition rate as well (how many students left her class), which is a quantitative measure, but it does not appear that she systematically collected or used these data other than to identify students to observe. 16 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Application Exercise 1.2: Identifying Sections of a Research Report Learning Outcome 1.5: Distinguish among key characteristics of quantitative, qualitative, and mixed methods types of research. [Q1] Consider the following: David is reading a research report and thoughtfully critiquing the study. In reviewing the survey that was used in a report on the benefits of extracurricular activities for children, David notes that several of the questions in the survey include “ands” and are actually asking two different questions. Refer to the section, Research Article Organization, to answer the following questions: Which part of the report is David most likely reading? How do you know? [Q1 Model Response] David is evaluating the method of the study, and in particular the design of the instruments (i.e., the survey), and that information is found in the method section of a report. However, it’s rare that an entire survey is included in the method section of a research report—usually that section just has examples. Copies of the complete surveys themselves are often included in an appendix. [Q2] Consider the following: In writing up the report for his recent study that examines the use of animation in learning, Dan notes all the research that supports the use of animation but does not include the studies that have indicated no benefit of animation in learning. Refer to the section, Research Article Organization, to answer the following question. In what section, while reading Dan’s report, should a careful reader notice this problem? [Q2 Model Response] A careful reader should notice while reading the review of the literature that Dan is missing key references. Note that this question specifies a careful reader. Often, when evaluating a research report, the less careful reader looks only at what is said and does not consider what is NOT said. You should, of course, strive to be a careful reader. Application Exercise 1.3: Think Like a Fox Learning Outcome 1.4: Understand how the principles of disciplined inquiry (“fox-like” thinking) are important for conducting educational research. [Q1] 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Consider the following scenario: Tyler is a student in a Ph.D. program in education. He is working to design his very first research study. He has no co-authors, but instead will pursue this study alone. He hopes to understand the ways in which students perceive classroom rules and how those rules may impact their social relationships. He has designed a study that will survey a group of students at one point in time using a Likert-type scale. He will calculate and report descriptive statistics from the surveys. Refer to Table 1.2, Characteristics of Fox and Hedgehog Researcher Thinking, to answer the following questions: What early characteristics of Tyler’s approach indicate a “hedgehog-like” approach to research? How could his approach be improved to be more “fox=like”? [Q1 Model Response] Tyler is a new researcher, so he is bound to make mistakes. However, he is making some early mistakes that should be corrected. First, it’s OK to be a single author, but Tyler could benefit from vetting his ideas with individuals in similar disciplines. For example, perhaps he could speak with a student in school counseling or behavioral psychology. His design is also “one-tracked.” He is only sampling students at one point in time and seeking to draw conclusions. He should consider ways he can obtain multiple data points and consider multiple possibilities. [Q2] Consider the following scenario: Stacey is a student in a Ph.D. program in education. She is working to design her very first research study. Stacey is collaborating with a fellow student in Allied Health to better understand the classroom implications of a cochlear implant on the social and emotional well-being of students in K–12 classrooms. Stacey is absolutely sure that cochlear implants enhance the social experiences of young children, and she is determined to prove her theory. Her design includes surveying children and families as well as following up with qualitative interviews. Refer to Table 1.2, Characteristics of Fox and Hedgehog Researcher Thinking, to answer the following question: In what ways is Stacey proceeding with “hedgehog-like” thinking and in what ways is she approaching this study with “fox-like” thinking?

[Q2 Model Response] Stacey is doing many things right and thinking like a fox. Primarily, she is collaborating with another student and she’s considering mixed methods that each inform one another. This is a solid start. However, Stacey is making a common and critical mistake 18 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

that early researchers make that places her thinking more in line with the “hedgehog” mentality. She’s setting out to “prove” her theory, thus going about this research with a closed mind. This will impact her study in a negative way. Remember that a good research is flexible and open-minded.

Chapter 1 Test Items SELF-ASSESSMENT ITEMS 1) A researcher stated that children should be taught basic math computations using drill and practice because they read a book by a person who is a well-respected mathematics teacher. Which source of knowledge did the researcher use to arrive at that conclusion? a) Observational research b) Personal experience c) Authoritative expertise d) Traditional norms 2) Which of the following is a valuable source of information characterized by the systematic collection, analysis, and interpretation of data? a) Observational research b) Personal intuition c) Authoritative expertise d) Logical argument 3) Which of the following characteristics distinguishes research from other sources of knowledge? a) Verification of knowledge claims b) Control for error and bias c) Openness to scrutiny d) All of the above 4) Which of the following terms describes a study that is based on observation and/or measurement that can be replicated by others? a) Theoretical b) Conjectural c) Hypothetical d) Empirical 5) Linking research to relevant theory allows practitioners to do which of the following? a) Generalize the results to other contexts b) Help establish principles that advance knowledge c) Use a set of propositions to explain relationships among observed phenomena 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

d) All of the above 6) Beyond a description of the process by which research can be conducted, which of the following is true about the steps in the research process? a) A review of the previous studies is not necessary if the steps in the research process are carefully followed. b) The credibility of the overall result is dependent on the credibility of each individual step. c) Each step in the process must be independent of the other steps. d) Following the steps too carefully can inhibit the generalizability of the findings. 7) Which of the following is NOT a characteristic of quantitative research? a) Use of observations b) Attempt to reach objective conclusions c) Description of multiple realities d) Predetermination of method 8) Which of the following is NOT a characteristic of qualitative research? a) Emergent design b) Use of purposeful sampling techniques c) Use of observations and interviews d) Inductive analysis of data 9) Which of the following is a unique characteristic of experimental research designs? a) The investigation of cause-and-effect relationships b) The manipulation of a factor c) The control of extraneous factors that might influence the results d) The collection of descriptive information 10) If researchers are studying the relationship between computer-assisted instruction and student achievement, what type of research design are they using? a) Correlational b) Descriptive c) Comparative d) Experimental 11) Which of the following qualitative designs has as its purpose the in-depth study of a single program, event, activity, or group? a) Phenomenology 21 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Ethnography c) Grounded theory d) Case study 12) In which type of research is the focus on the study of marginalized people with investigations of justice and equality? a) Phenomenology b) Critical study c) Grounded theory d) Analytical 13) Which of the following is a term that could be used to characterize applied research? a) Contextual b) Generalizable c) Explanatory d) Fundamental 14) Which type of research is being used when researchers use questionnaires, tests, observations, and focus groups in their study? a) Mixed methods b) Qualitative c) Quantitative d) Ethnographic 15) A teacher was concerned about the attention level of some of the students in her class. She devised a plan in which she systematically tried different seating arrangements over the period of a month and kept a daily record of student attention level. At the end of the month, she analyzed the data to determine if there was an improvement in the students’ attention level. What type of research was the teacher using? a) Action research b) Basic research c) Descriptive research d) Evaluation research 16) Which of the following is a unique aspect of evaluation research? a) Teachers act as researchers to evaluate a method of instruction. b) The results can be generalized to many different educational settings. 22 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) The goal is to make judgments about alternatives for decision making. d) Broad generalizations are made about how phenomena are related. 17) In which of the following components of a research report might you find an interpretation of the results? a) Results or findings b) Method and design c) Statement of hypothesis d) Discussion and conclusions

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Standardized social studies test scores have been very low for several years in a local school district. Dr. Roberts, the superintendent, identified the outdated curriculum as a major cause for the low scores. She must recommend to the Board of Education a new curriculum (e.g., textbook series, instructional approach, supplemental materials, etc.) for district-wide adoption. Which of the following is the knowledge base that would provide information that is not influenced by subjective interpretations? a) Experimental or quasi-experimental research studies b) Interviews with teachers in nearby districts c) Logical, reasoned arguments d) Traditional norms of similar communities 2) Which of the following is a source of information characterized by the verification of findings through replication? a) Traditional practices b) Expert authority c) Disciplined inquiry d) Logical, reasoned arguments 3) Scientific research, experience, and intuition are all important sources of knowledge. What is the characteristic of scientific research that distinguishes it from experience and intuition? a) Direct b) Effective c) Systematic d) Respected 4) Why is the predictive power of scientific inquiry important in education? a) Its purpose is to produce accurate descriptions. b) It provides fundamental knowledge about a phenomenon. c) It allows researchers to control one factor to study its effect on another factor. d) Its purpose is to provide information for decision making. 5) What is empirical educational research? a) Research in which data are collected and analyzed that can be replicated 24 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Research that allows for the influence of the researcher’s subjectivity c) Research that is based on scholarship in the humanities d) Research that examines educational theories and speculation 6) Which of the following is a critically important characteristic of the research process? a) The credibility of the overall result builds on the trustworthiness of each preceding step. b) Previous research can be disregarded based on reliable and valid research. c) Mistakes in one step of the process can be compensated by actions taken in other steps. d) Each of the steps in the process is independent of the other steps. 7) Which of the following is NOT a characteristic of quantitative educational research? a) The researcher must often study the participants in intact groups. b) The purpose may be to explain natural phenomenon to understand underlying relationships. c) The research methods focus on social interactions, narratives, and interviews. d) The purpose may be to investigate a naturally occurring phenomenon. 8) Which of the following is NOT a characteristic of qualitative research? a) Multiple methods of data collection may be used. b) The focus may be on manipulation of a factor with a single subject. c) The research design emerges over time. d) The researcher conducts the study in naturally occurring situations. 9) Which of the following characteristics most clearly differentiates experimental research designs from other types of quantitative design? a) Findings are applied from basic research to a practical problem in the classroom. b) Relationships are assessed between two or more phenomena. c) Causal relationships are investigated between variables by controlling an intervention. d) Inferences are made between two or more alternatives. 10) Which of the following types of research design are directed specifically toward making a decision about the effectiveness of a program? a) Experimental b) Evaluation c) Analytical 25 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

d) Mixed methods 11) Which type of research best describes the following study? A researcher used a mid-term social studies exam to identify three students who performed exceptionally well and three students who performed exceptionally poorly. Once identified, the students were interviewed individually to understand the nature of their preparation for the exam. a) Quantitative b) Qualitative c) Analytical d) Mixed methods 12) In a mixed methods design, how does explanatory sequential design differ from exploratory sequential design? a) The quantitative methods are followed by qualitative methods in an explanatory sequential design. b) The qualitative methods are followed by quantitative methods in an explanatory sequential design. c) Both quantitative and qualitative approaches are used about equally in explanatory sequential design. d) There is no difference between the two designs; they are just different names for the same thing. 13) Which of the following questions could be answered using experimental research? a) How do teacher aides apportion their time in the classroom? b) Was the new reading program used in our school last year effective at increasing the reading scores of the fourth-grade students? c) What is the relationship between socioeconomic status and student attitudes? d) Will the use of manipulatives enhance the retention of math concepts in fifthgrade students? 14) Researcher used achievement tests, questionnaires, and lengthy, narrative interviews in their study. Which of the following types of research design would best describe their research? a) Mixed methods b) Qualitative methods c) Quantitative methods d) Ethnographic methods 26 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

15) A principal was concerned about the high absence rate of students in her high school. She decided to implement an intervention for a full year to improve attendance. Which of the following types of research designs would best describe her research? a) Basic research b) Descriptive research c) Action research d) Evaluation research Short-Answer Questions 1) Give two reasons it is essential that consumers and researchers make informed judgments about the credibility and usefulness of research studies. 2) Why is it important to read about research studies in peer-reviewed journals rather than the popular press? Essay Question Describe a study you would be interested in conducting using either a quantitative, qualitative, or mixed methods approach. Briefly describe: •

The problem,

•

Research design, and

•

Methods you would use to collect and analyze data.

Justify the appropriateness of your selection of methods in terms of how they will provide a credible answer to the problem.

Chapter 1 Test Answer Key ANSWER KEY: Chapter 1 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. b 2. a 3. d 4. d 5. d 6. b 7. c 8. d 9. b 10. a 11. d 12. b 13. a 14. a 15. a 16. c 17. d INSTRUCTOR’S EXAM ITEMS 1. a 2. c 3. c 4. d 5. a 6. a 7. c 8. b 9. c 28 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

10. b 11. d 12. a 13. d 14. a 15. c

Chapter 2 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

This work is protected by United States copyright laws and is provided solely for the use of instructors in teaching their courses and assessing student learning. Dissemination or sale of any part of this work (including on the World Wide Web) will destroy the integrity of the work and is not permitted. The work and materials from it should never be made available to students except by instructors using the accompanying text in their classes. All recipients of this work are expected to abide by these restrictions and to honor the intended pedagogical purposes and the needs of other instructors who rely on these materials.

Content Producer: Janelle Rogers Supplement Project Manager: Chithra Rajasekaran, Straive Copyright © 2022, 2016, 2012 by Pearson Education, Inc. or its affiliates, 221 River Street, Hoboken, NJ 07030. All Rights Reserved. Manufactured in the United States of America. This publication is protected by copyright, and permission should be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise. For information regarding permissions, request forms, and the appropriate contacts within the Pearson Education Global Rights and Permissions department, please visit www.pearsoned.com/permissions/ PEARSON and ALWAYS LEARNING are exclusive trademarks owned by Pearson Education, Inc. or its affiliates, in the United States, and/or other countries. Unless otherwise indicated herein, any third-party trademarks, logos, or icons that may appear in this work are the property of their respective owners, and any references to third-party trademarks, logos, icons, or other trade dress are for demonstrative or descriptive purposes only. Such references are not intended to imply any sponsorship, endorsement, authorization, or promotion of Pearson’s products by the owners of such marks, or any relationship between the owner and Pearson Education, Inc., or its affiliates, authors, licensees, or distributors.

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 2: Ethical Issues, Principles, and Practices ........................................... 1 Learning Outcome Quizzes .................................................................... 1 Application Exercises............................................................................ 14 Test Items ............................................................................................. 19 Test Answer Key................................................................................... 25

Chapter 2: Ethical Issues, Principles, and Practices Chapter 2 Learning Outcome Quizzes Learning Outcome 2.1: Understand the nature of ethics and how ethics and ethical standards are applied to educational research. [Q1] Which of the following statements best summarizes the fundamental values underlying the ethical principles for professional practice? 1. Researchers must maintain integrity and have respect for their own work, for the work of others, and for those individuals who contribute to the research. [correct] 2. Researchers must be sensitive to different characteristics of the study participants or the populations that results reflect. 3. Researchers must consider how conducting and reporting results may affect different populations in positive as well as harmful ways. 4. Researchers must follow the code of regulations and other federal and institutional efforts to promote ethical standards. [Feedback for Correct Answer 1] Although researchers should behave in all of the ways listed, the best way to summarize the values is to emphasize integrity (honesty, trustworthiness, objectivity) and respect (for self and other). These two values underlie all codes of regulations that researchers follow. [Feedback for Answer Choice 2] Researchers should be sensitive to different characteristics of study participants, but this is not an adequate summary statement. [Feedback for Answer Choice 3] Researchers should always consider the well-being of their participants, but this is not an adequate summary statement. [Feedback for Answer Choice 4] This is certainly a must for researchers, but not an adequate summary statement. [Q2] Which of the six ethical principles for educational research suggests that researchers understand work within their areas of expertise and consult with others when needed? 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

1. Integrity 2. Responsibility 3. Professional competence [correct] 4. Justice [Feedback for Answer Choice 1] Researcher integrity refers to honesty and trustworthiness. These are important principles, but not the correct answer. [Feedback for Answer Choice 2] Researcher responsibility simply means that researchers are responsible for their work and its implications. This is an important principle, but not the correct answer. [Feedback for Correct Answer 3] Professional competence refers to the fact that researchers understand work within their areas of competence and consult with others when needed. [Feedback for Answer Choice 4] Adhering to principles of justice means researchers are sensitive to the welfare of all individuals, take into account all perspectives in making decisions, and do not allow biases to result in unjust actions. [Q3] Which of the six ethical principles for educational research suggests that researchers should focus on what is best for the larger society? 1. Justice 2. Serving the public good [correct] 3. Responsibility 4. Integrity [Feedback for Answer Choice 1] In this principle, researchers are sensitive to the welfare of all individuals, take into account all perspectives in making decisions, and do not allow biases to result in unjust actions. [Feedback for Correct Answer 2] Researchers are focused on what is good for the larger society and design and report research that results in the greatest public good. 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] In this principle, researchers accept responsibility for their work. They are sensitive to the ethical behavior of colleagues. [Feedback for Answer Choice 4] In this principle, researchers are honest and trustworthy, and promote accuracy. They do not cheat, steal, deceive, or misrepresent. [Q4] Which of the following best describes the ethical research principle that suggests that researchers work to make sure bias does not lead to unjust actions? 1. Justice [correct] 2. Integrity 3. Professional competence 4. Responsibility [Feedback for Correct Answer 1] Researchers are sensitive to the welfare of all individuals, take into account all perspectives in making decisions, and do not allow biases to result in unjust actions. [Feedback for Answer Choice 2] Researchers are honest and trustworthy and promote accuracy. They do not cheat, steal, deceive, or misrepresent. However, this is not the correct answer. [Feedback for Answer Choice 3] Researchers understand work within their areas of competence and consult with others when needed. However, this is not the correct answer. [Feedback for Answer Choice 4] Researchers accept responsibility for their work. They are sensitive to the ethical behavior of colleagues. However, this is not the correct answer. Learning Outcome 2.2: Describe how the federal government regulates and informs ethical requirements and policies. [Q1] Unregulated research projects such as the Tuskegee Study of Untreated Syphilis in the Negro Male led to the development of federal ethics codes in the United States. The 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Tuskegee study was problematic for many reasons—the greatest violation of basic ethical principles was which of the following? 1. The researchers withheld useful medication from some participants without their consent. [correct] 2. The researchers studied only impoverished Black men. 3. The researchers promised “treatment for bad blood” to some men with syphilis and also to some men without syphilis. 4. The researchers “paid” the participants with medical treatment for minor complaints, burial insurance, and free meals. [Feedback for Correct Answer 1] Well before this study ended, penicillin was known and used as an effective treatment for syphilis, but it was withheld from the men as part of a study of “untreated syphilis.” Researchers did not inform the participants of this possible treatment, and thus the participants did not actively consent to remain “untreated.” This is a gross violation of the core principles of respect for persons. It’s also true that, in this study, the risks greatly outweighed the benefits (violating the principle of beneficence) and that the participants were not selected fairly (justice). [Feedback for Answer Choice 2] In and of itself, this is not an ethical violation. [Feedback for Answer Choice 3] While an ethical violation, this is not the most significant ethical violation of this study. In fact, all men were told they would receive this treatment, but obviously did not. [Feedback for Answer Choice 4] While an ethical violation, this is not the most significant ethical violation of this study. [Q2] “The Common Rule” is another name for which of the following? 1. The National Research Act, which established the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research 2. The report titled “Ethical Principles and Guidelines for the Protection of Human Subjects of Research” 3. Part of the Code of Federal Regulations that addresses the protection of human research subjects. [correct]

4. The most recent report from the President’s Commission for the Study of Bioethical Issues [Feedback for Answer Choice 1] This answer does not represent the Common Rule. [Feedback for Answer Choice 2] This answer does not represent the Common Rule. [Feedback for Correct Answer 3] The Common Rule is a label for Subpart A of The Code of Federal Regulations (CFR) Title 45 Part 46: Protection of Human Subjects. Fifteen federal agencies/departments signed onto the Common Rule, meaning they agreed to comply with the regulations. Other agencies also follow these rules, although they were not among the original signers. [Feedback for Answer Choice 4] This answer does not represent the Common Rule. [Q3] Most ethical standards of research are focused on which of the following? 1. The findings of research studies 2. How researchers interact with and treat participants throughout the study [correct] 3. Research question development 4. Replicability of research studies [Feedback for Answer Choice 1] The findings of research studies must be reported ethically, but this is not the correct answer. [Feedback for Correct Answer 2] Most ethical standards for conducting research center on how researchers interact with and treat participants throughout the study, including before and after a study is completed. [Feedback for Answer Choice 3] Research questions guide research studies, so they must be ethical in nature, but this is not the correct answer. [Feedback for Answer Choice 4] 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Replicability is not an issue of ethics. Learning Outcome 2.3: Differentiate between respect for persons, beneficence, and justice as foundational ethical principles. [Q1] Which of the following ethical principles found in the Belmont Report is being demonstrated when a researcher adapts his or her methods to the needs of different populations? 1. Justice [correct] 2. Beneficence 3. Respect for persons 4. Integrity [Feedback for Correct Answer 1] The Belmont Report identified three core principles that should govern all research and researcher-participant interactions: respect for persons, beneficence, and justice. When a researcher ensures that the benefits and burdens of the research are equitably distributed, he or she is adhering to the principle of justice. [Feedback for Answer Choice 2] Beneficence is one of the three core principles of the Belmont Report, but this is not the correct answer. [Feedback for Answer Choice 3] Respect for persons is one of the three core principles of the Belmont Report, but this is not the correct answer. [Feedback for Answer Choice 4] Integrity is not one of the three core principles of the Belmont Report. [Q2] Which of the following represent the three core principles outlined in the Belmont Report? 1. Respect for persons, beneficence, and integrity 2. Integrity, justice, and beneficence 3. Respect for persons, beneficence, and justice [correct] 4. Beneficence, justice, and responsibility 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 1] This is not a combination of ethical principles found in the Belmont Report. [Feedback for Answer Choice 2 This is not a combination of ethical principles found in the Belmont Report. [Feedback for Correct Answer 3] The Belmont Report identified three core principles that should govern all research and researcher–participant interactions: respect for persons, beneficence, and justice. [Feedback for Answer Choice 4] This is not a combination of ethical principles found in the Belmont Report. [Q3] The National Commission recommends that once children are 7 years old, they are able to offer which of the following in order to participate in research studies? 1. Child consent 2. Child assent [correct] 3. Consent to participate 4. Waiver of consent [Feedback for Answer Choice 1] Minors are not able to consent to participate in research. [Feedback for Correct Answer 2] The National Commission recommends that child consent, known as child assent, should be required at the age of 7 and older. However, federal regulations in the United States do not include specific requirements for informed assent. [Feedback for Answer Choice 3] Minors are not able to consent to participate in research. [Feedback for Answer Choice 4] Minors are not able to waive the requirement of informed consent. [Q4]

Which of the following terms best describes the principle that researchers are obligated to protect study participants from harm and to act in ways that are in the best interest of the participants’ welfare? 1. Beneficence [correct] 2. Justice 3. Responsibility 4. Integrity [Feedback for Correct Answer 1] According to the principle of beneficence, researchers are obligated to protect study participants from harm and to act in ways that are in the best interest of the participants’ welfare. Two key guidelines or rules illustrate the principle of beneficence: (1) do not harm and (2) maximize the possible benefits and minimize the possible harms. [Feedback for Answer Choice 2] While researchers must act with justice in mind, this is not the correct answer. [Feedback for Answer Choice 3] While researchers must act responsibly, this is not the correct answer. [Feedback for Answer Choice 4] While researchers must behave with integrity, this is not the correct answer. [Q5] Which of the following study elements supports a positive risk/benefit ratio? 1. Confidentiality [correct] 2. Randomization 3. Assent 4. Consent [Feedback for Correct Answer 1] One way to support a positive risk/benefit ratio is to use confidentiality. To ensure the validity of research findings, participants need to be free to communicate their views, perceptions, or thoughts accurately and honestly. [Feedback for Answer Choice 2] While this may support the quality of a study, it does not contribute to the risk/benefit ratio. 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] Minors should be allowed to offer assent, but this does not contribute to the risk/benefit ratio. [Feedback for Answer Choice 4] Study participants must consent to participate, but this does not contribute to the risk/benefit ratio. Learning Outcome 2.4: Explain why different institutional review board procedures are essential elements for conducting human subjects research with children. [Q1] Steven, a college senior, is conducting a research project focused on children’s friendships. He plans an ethnographic study to take place over the course of 1 year. Prior to conducting his study, Steven needs to consult with his university’s IRB about which of the following? 1. His plan to minimize risk to the participants. [correct] 2. His definitions of friendship. 3. His exact research questions and hypotheses. 4. His plan for publishing his findings. [Feedback for Correct Answer 1] The Institutional Review Board is charged with weighing risks and benefits prior to the start of any study. [Feedback for Answer Choice 2] The IRB is not concerned with the nuances of Steven’s definitions. [Feedback for Answer Choice 3] The IRB is not commissioned with assessing the quality of the research study. [Feedback for Answer Choice 4] The IRB is not concerned with the location of publication of findings. [Q2] Which of the following classifications would apply to a study that very clearly has minimal risk and is freed from federal regulations, but ethical principles still apply? 1. Expedited 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. Full board 3. Exempt [correct] 4. Incomplete [Feedback for Answer Choice 1] Expedited studies require more oversight than is outlined in this question. [Feedback for Answer Choice 2 Full board reviewed studies usually involve greater than minimum risk. [Feedback for Correct Answer 3] In the exempt category, a study that very clearly has minimal risk is freed from federal regulations, but ethical principles still apply. The exempt determination must typically be made by IRBs. Exempt projects are very low risk and do not require a formal consent process. [Feedback for Answer Choice 4] This is not an IRB classification. [Q3] Which of the following classifications would include studies that have minimal risk but have greater participant involvement? 1. Exempt 2. Expedited [correct] 3. Incomplete 4. Full board [Feedback for Answer Choice 1] Exempt studies represent the lowest level of risk. [Feedback for Correct Answer 2] Studies that need an expedited review are also minimal risk but have greater participant involvement; the researchers may have more direct contact with study participants than in studies meeting exempt criteria. [Feedback for Answer Choice 3] This is not an IRB classification. [Feedback for Answer Choice 4] 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Full board reviewed studies represent the highest levels of risk. [Q4] Which of the following classifications typically involve greater than minimal risk? 1. Full board [correct] 2. Exempt 3. Expedited 4. Incomplete [Feedback for Correct Answer 1] Studies that need a full board review typically involve greater than minimal risk, but do not meet the exempt or expedited review categories. Full board studies sometimes include specific “vulnerable” populations such as prisoners, children, or pregnant women. A full board review requires that a majority of the members of the IRB committee review and discuss the study at a panel or committee meeting. [Feedback for Answer Choice 2] Exempt studies represent the lowest levels of risk. [Feedback for Answer Choice 3] Expedited studies involve more risk than others, but they are still not greater than minimal risk. [Feedback for Answer Choice 4] This is not an IRB category. Learning Outcome 2.5: Understand author ethical and legal standards for publishing. [Q1] Whether you are presenting a paper at a conference or submitting a manuscript to a research journal, you must determine whether you should be sole author or share authorship with one or more other people. Which of the following best describes the etiquette of coauthorship? 1. Individuals who have made significant intellectual contributions to the work should share in its authorship. [correct] 2. Everyone involved with data collection should share in the authorship of the final paper. 3. Reviewers should be included as coauthor of a paper if they make significant contributions to a revision. 11 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

4. Students should include their major advisors as coauthors on their dissertations. [Feedback for Correct Answer 1] This statement summarizes coauthorship etiquette effectively. Not everyone involved in the research should be an author, only those who make significant intellectual contributions. Reviewers can be noted in footnotes or endnotes, but are not authors of the work, even if they make useful suggestions for revisions. Students also often share authorship with their major advisors, and they certainly should do so if they have made significant intellectual contributions to their advisor’s work, but dissertations are typically published as sole-authored papers. [Feedback for Answer Choice 2] While research takes a team, simply helping with data collection does not warrant coauthorship. [Feedback for Answer Choice 3] Reviewers are not considered as coauthors. [Feedback for Answer Choice 4] Dissertations are typically published as sole-authored. [Q2] Dr. Caulfield calls in one of his graduate students, Elise, and shares his concern that she has committed plagiarism in her research report. Elise says, “But every time I quoted someone’s words exactly, I cited them in the body of the paper, and I have a complete bibliography included at the end of the paper.” Which of the following statements is most likely to be true? 1. Dr. Caulfield is wrong in his assumption that Elise has conducted herself unethically. 2. Dr. Caulfield is worried that Elise has a conflict of interest with regard to her research. 3. Elise failed to cite paraphrased material properly. [correct] 4. Elise is lying and has included quotes without attribution. [Feedback for Answer Choice 1] If Elise has failed to attribute credit to other people’s ideas, Dr. Caulfield is not wrong in his assumption. [Feedback for Answer Choice 2 This situation does not speak to a conflict of interest. 12 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Correct Answer 3] Assuming Elise correctly attributed direct quotations to their authors (likely for a graduate student), she has probably failed to paraphrase properly or to attribute paraphrased ideas to their authors, which shows a lack of ethical behavior and is an example of plagiarism. Whenever researchers paraphrase another author, they must give him or her credit. [Feedback for Answer Choice 4] There is no evidence that Elise would be lying. [Q3] Suppose that a testing company has produced a test preparation software. If this company provides its own research regarding the efficacy of the software in relation to the test they produce, which of the following should be suspected? 1. Fraud 2. Conflict of interest [correct] 3. Deception 4. Ethical practice [Feedback for Answer Choice 1] The testing company may be honest in their research and not fraudulent. [Feedback for Correct Answer 2] Even if they company is ethical in its research practice, they still have a conflict of interest. [Feedback for Answer Choice 3] It is possible that the testing company is not seeking to deceive anyone. [Feedback for Answer Choice 4] While the testing company may not be intentionally dishonest, this is not an ethical practice.

Chapter 2 Application Exercises Chapter 2: Ethical Issues, Principles, and Practices Application Exercise 2.1: The Importance of the IRB Learning Outcome 2.4: Explain why different institutional review board procedures are essential elements for conducting human subjects research with children. [Q1] The institutional review board (IRB) is charged with identifying potential risks to participants, often risks that are unnoticed by the researchers as they plan a study. Review the section entitled Application of Ethical Principles to Research Practice to answer the following question focused on Samuel. Samuel wants to investigate how children respond to negative feedback. In his design, children complete difficult puzzles. Some of the children are informed erroneously that they are worse than their peers at solving these problems. Others are told they perform better than their peers. Are there any red flags for the IRB? [Q1 Model Response] A board would likely be concerned because of the age of the participants and because this study involves a form of deception. The feedback to children may also harm their self-concept and cause emotional problems. [Q2] The institutional review board (IRB) is charged with identifying potential risks to participants, often risks that are unnoticed by the researchers as they plan a study. Review the section entitled Application of Ethical Principles to Research Practice to answer the following question focused on Denise. Denise studies teacher burnout in an urban high school. She proposes to study workers firsthand by posing as a new clerical employee and working for a year in a school district facing many difficult issues. She believes that this is the best way to appreciate what it is like for workers "in the trenches," and she feels she can get honest answers from teachers who might not otherwise want to talk about burnout. Are there any red flags for the IRB? [Q2 Model Response] This study involves deception and as presented does not include informed consent. These issues would cause this study to fail to move forward from the IRB process.

[Q3] The institutional review board (IRB) is charged with identifying potential risks to participants, often risks that are unnoticed by the researchers as they plan a study. Review the section entitled Application of Ethical Principles to Research Practice to answer the following question focused on Danny. Danny, a kinesiologist, is hired by a school principal to develop and conduct a study on the relationship between physical activity and academic performance, Danny proposes to limit the exercise of one group of children in his study while allowing the other group to exercise as they normally would. Are there any red flags for the IRB? [Q3 Model Response] Limiting the exercise of children could cause physical, mental, and/or emotional harm. It is the role of the IRB to ensure that participants are not harmed by research, so these issues would cause the proposed study to stall in the IRB process until necessary edits are made. Application Exercise 2.2: Increasing Student Involvement in the IEP Process Learning Outcome 2.1: Understand the nature of ethics and how ethics and ethical standards are applied to educational research. [Q1] Consider the following: Jeremiah is a doctoral student engaging in his first research project. He feels pressured for time, so he is rushing to put together his IRB application. His study focuses on the health impacts of reduced recess time in elementary schools on the physical wellbeing of elementary school students. He realizes that the school may not want to participate for fear of looking bad, so in his suggested recruitment letter he has a vague statement about the study purpose indicating that he is going to focus on the positive aspects of reduced recess, though he plans to consider both positive and negative impacts. Review Table 2.1 entitled Ethical Principles for Educational Research to answer the following questions. Which of the basic ethical principles of research has Jeremiah violated? What impact might this have on his research? [Q1 Model Response] Jeremiah has violated the ethical principle of integrity. He has not been upfront about the function of his study and has sought to deceive the school district. Jeremiah’s study would not likely make it past the IRB, but if it did, he would forever damage his relationship with his school partners and may find himself in trouble for his dishonesty. 15 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q2] Consider the following: Josina is a doctoral student engaging in her first research project. Her research area is in English language learners, but there is a special call for a journal issue in the field of special education that she would like to pursue. However, she has no special education experience. She designs a study in the field of special education but does not consult any experts in the field as she feels her interests are related. Review Table 2.1 entitled Ethical Principles for Educational Research to answer the following questions. Which of the basic ethical principles of research has Josina violated? What impact might this have on his research? [Q2 Model Response] Researchers are responsible for their dedication to professional competence. In other words, they must know their fields of study so that they make informed decisions. Josina has violated this concept by branching into a field in which she does not have experience. This will likely mean her research is not going to be at a high standard. She should have considered partnering with a trusted colleague with expertise in special education in order to produce an informed study. [Q3] Consider the following: Jane is a college professor interested in students’ beliefs about the typical instructional strategies of professors. As part of her ongoing research agenda, she requires students in her research methods course to fill out questionnaires. Review Table 2.1 entitled Ethical Principles for Educational Research to answer the following questions. Are there any red flags for the IRB? [Q3 Model Response] This study would raise concerns because it does not allow participants to decline to participate. This is an important part of ethical research. Participants are always considered voluntary and they may discontinue their participation at any time. Application Exercise 2.3: The Belmont Principles in Action Learning Outcome 2.3: Differentiate between respect for persons, beneficence, and justice as foundational ethical principles. 16 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q1] Consider the following: Elaina is a researcher who studies the impact of vitamin consumption on the health and well-being of adolescents. She has designed a study that will include a control group and an experimental group. The control group will not take a multivitamin, but the experimental group will consume a proprietary vitamin that has not been approved by any health agencies. However, she feels it will be OK and that the risks are minimal. Review Table 2.2 entitled Belmont Principles and Applications for Research, to answer the following questions. Which principle of the Belmont Report is Elaina violating? How do you know? [Q1 Model Response] Elaina is violating the principle of beneficence. While she may feel the risk is minimal, she has no way of confirming this hunch. This study could become quite dangerous for participants and put her at professional risk. [Q2] Consider the following: Demitri is a researcher studying the impacts of gentrification in an urban neighborhood. In this neighborhood, Black families have been disproportionately impacted in a negative way as the community has gentrified. Key elements include the loss of Black owned businesses in the area. Demitri interviews primarily White community members who are new to the area to understand the impact of the loss of Black-owned businesses in the community. Review Table 2.2 entitled, Belmont Principles and Applications for Research to answer the following questions. Which principle of the Belmont Report is Demitri violating? How do you know? [Q2 Model Response] Demitri is violating the principle of justice. While his interviews may yield interesting information, he is not selecting participants or working with the group that would most benefit from the research. In order to be equitable, his research must include the voices of Black community members who are experiencing an impact in the change of their community. [Q3] Consider the following: 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Todd is a college professor interested in how teacher education students develop an understanding of formal lesson planning. He has planned a study with his students where he will set up a series of assignments, each with sample test items that will help him understand their outlook on lesson planning. He then plans to manipulate their assignments based on the results to confirm his findings. Review Table 2.2 entitled Belmont Principles and Applications for Research to answer the following questions. Which principle of the Belmont Report is Todd violating? How do you know? [Q3 Model Response] Todd is violating the principle of respect for persons. He is manipulating assignments and collecting data without obtaining consent from his students. His students must be informed up front and consent. He also must not have a conflict of interest between his teaching and research.

Chapter 2 Test Items SELF-ASSESSMENT ITEMS 1) Which of the following is an ethical guideline in educational research? a) Participants only need to be informed of all aspects of a research study if they are from a vulnerable population or under the age of 18. b) Participants are obligated to stay in a study once they have signed consent forms. c) The privacy of personal information of participants must be protected. d) All of the above 2) What is the purpose of the standards and guidelines for ethics in educational research? a) To determine when research at a university should go before the institutional review board b) To provide information for sound decision making and behavior of educational researchers c) To determine legitimate use of funds for human subjects’ research d) To provide details to justify conflicts of interest under certain circumstances 3) Ethical laws for research with human subjects were established after public outrage over which of the following? a) Federal prison mismanagement and abuse b) Mistreatment of Black sharecroppers c) Misuse of research funds by the National Bioethics Council d) The advent of human stem cell research 4) One of the most important principles guiding educational research is respect for persons. To what does this refer? a) There is balance between the benefits and risks to study participants. b) Participants will be informed about all aspects of a study in order to make an informed decision about participation. c) The group that benefits from the research should be similar to the group that participates in the research. d) All educational research conducted at universities will abide by the guidelines of an institutional review board. 5) A research study consent document is required to describe which of the following? a) The purpose of the study 19 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) What the participation in the study will involve c) How much time it will take to complete the study d) All of the above 6) The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research recommends that child consent, known as child assent, be established in a research study that involves children. Which of the following statements is true about child assent? a) It is required even if parents have given permission for the child to participate. b) It is required for children starting at age 12. c) It is always required even with children as young as two years old. d) It is only required for research that can cause potential harm. 7) Using deception in a research study is discouraged, but sometimes it is necessary. What should researchers do in order to mitigate the harm that may be caused by deception? a) They should never inform the participants of the actual purpose of the study. b) They should allow the participants to ask questions and discuss concerns after the data are collected. c) They should debrief the participants before the participant has completed the study requirements. d) All of the above 8) Which of the following could be affected by the accidental disclosure of a participant’s name or personally identifiable study information? a) The validity of the study b) The self-perceptions of the participant c) The participant’s community standing d) All of the above 9) Which of the following is a characteristic of research that would be considered exempt by an institutional review board? a) The research involves normal educational practices in which no identifiers are collected. b) The participants are from a vulnerable population. c) The research will be conducted in an elementary school. d) The research involves minimal risk, but the researchers have direct contact with the participants. 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

10) When is an external review required for proposed research involving human research? a) Research that is classified as expedited and full b) Research that is only classified as full c) Research that is classified as all three types: exempt, expedited, and full d) Research that is classified as exempt and expedited 11) A researcher studied the potential benefits of a commercial reading program published by Reading, Inc., which funded the research. Which of the following standards did the researcher violate? a) Nepotism b) Plagiarism c) Deception d) Conflict of interest

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) What is the purpose of the standards and guidelines for ethics in educational research? a) To determine the correct behavior for specific research situations b) To determine who is at fault when the ethics of a researcher are questioned c) To provide guidelines for sound decision making and the behavior of educational researchers d) All of the above 2) What precipitated the establishment of ethical laws for research with human subjects? a) The cloning of Dolly, the sheep, which was seen as a precursor to human cloning b) The suffering of animals and humans in a federal research facility c) The Tuskegee study of untreated syphilis in a vulnerable population d) The National Science Foundation funding of food safety research 3) One of the most important principles guiding educational research is beneficence. To what does this principle refer? a) There needs to be a balance between the benefits and risks to study participants. b) Research should only benefit the participants and cause no potential harm. c) Participants will be informed of the benefits they will receive from participation. d) A member of the group that benefits from the research should serve on the institutional review board. 4) Which of the following should be included in a research study consent document? a) A statement that the subject may leave the study at any time b) A description of the benefits to participants in the study c) A description of the purpose of the study d) All of the above 5) The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research recommends that child consent, known as child assent, be established in a research study that involves children. Which of the following statements is true about child assent? a) It is only required if parents are not available to give permission. b) It is required at the age of 12 and older. 22 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) It is required at the age of seven and older. d) It is only required for research that can cause potential harm. 6) Which of the following is true about the ethical guidelines for the use of deception in a research study? a) Deception has the potential to cause psychological harm and is therefore never acceptable. b) If deception is used, the participants should be debriefed immediately following data collection. c) It is acceptable, and usually necessary, to use deception to disguise the purpose of the study. d) If deception is used, there should be financial compensation awarded to the participants. 7) Maintaining confidentiality is one way to provide participants of a study protection from harm. To what does confidentiality refer? a) Not requiring participants to respond to delicate questions such as questions about deviant behavior b) The use of a consent document to inform participants of the potential benefits and risks of participation in a study c) The protection of the privacy of the participants by removing identifying features from any data collected in a study d) Providing participants with the opportunity to withdraw at any time during the study 8) What are the three categories of institutional review as outlined by the institutional review boards of colleges, universities, and hospitals? a) Exploratory, explanatory, and convergent b) Exempt, expedited, and full c) Potential, minimal, and maximal d) Potential risk, minimal risk, and certain risk 9) When does proposed research involving human subjects require an external assessment to ensure that the legal and ethical requirements are met to safeguard potential research participants? a) After the study is completed b) Always c) Only if the participants are children or individuals from a vulnerable population d) Only if the researcher suspects participants may be harmed 23 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

10) Which of the following is an example of a conflict of interest as it relates to educational research? a) The National Association of Year-Round Schooling is asked to supply its mailing list to a researcher interested in doing research into the effects of year-round school. b) A researcher conducts research in the same educational setting (i.e., school) many times. c) A reviewer is asked to review a research study on a theory of which she is an expert. d) All of the above 11) When is a full review by an institutional review board required? a) Any time a study result will be published in a peer-reviewed journal b) Any time a study involves children c) Any time a study involves greater than minimal risk d) Any time a study involves the use of publicly available data Short-Answer Questions 1) Give two reasons why it is necessary to have an institutional review board conduct an external review of educational research. 2) Describe an example of a study that has a poor risk/benefit ratio. Essay Question What are some ways that confidentiality can be breached and what might be the consequences of such a breach? Describe ways a researcher can ensure that identifiable information is not accidentally disclosed.

Chapter 2 Test Answer Key ANSWER KEY: Chapter 2 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. c 2. b 3. b 4. b 5. d 6. a 7. b 8. d 9. a 10. c 11. d

INSTRUCTOR’S EXAM ITEMS 1. c 2. c 3. a 4. d 5. c 6. b 7. c 8. b 9. b 10. a 11. c

Chapter 3 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 3: Research Problems and Questions ........................................................... 1 Learning Outcome Quizzes........................................................................... 1 Application Exercises .................................................................................. 12 Test Items ................................................................................................... 17 Test Answer Key ......................................................................................... 25

Chapter 3: Research Problems and Questions Chapter 3 Learning Outcome Quizzes Learning Outcome 3.1: Know the nature and sources of general research problems. [Q1] Matt must complete a project for his research class. A challenge he has faced this year is developing appropriate strategies for working with a child with a learning disability in his classroom. He wonders what strategies other middle school teachers have found effective in working with children diagnosed with learning disabilities. The source of this research problem is derived from which of the following? 1. Personal experience [correct] 2. Library immersion 3. Theory 4. Replication [Feedback for Correct Answer 1] Matt has personal experience with this topic, which provides the main source of his idea. [Feedback for Answer Choice 2] Matt did not develop his idea from library research. [Feedback for Answer Choice 3] Matt’s idea is not one rooted in theoretical underpinnings. [Feedback for Answer Choice 4] Matt is not attempting to replicate a previous study. [Q2] Consider the following sentence from the introduction to a research paper that presents a study on kindergarten readiness: With some 500,000 children in California entering kindergarten each year (cf. the California Children & Families Commission, 2003), the concept of school readiness increasingly receives public attention and funding. In this sentence, the author is expressing which of the following? 1. The purpose for the study 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. The significance of the study 3. The context for the study [correct] 4. The research problem for the study [Feedback for Answer Choice 1] In this sentence, the author is not outlining the reason the study exists; therefore, this is not the purpose for the study. [Feedback for Answer Choice 2] In this sentence, the author has not outlined why the study is significant. [Feedback for Correct Answer 3] Context explains the background or larger body of knowledge or area being investigated. In this sentence, the author shows why this topic (kindergarten readiness) is important and of current interest. [Feedback for Answer Choice 4] The research problem is not outlined in this sentence. [Q3] Which of the following terms refers to the reason or need for a study? 1. Context 2. Significance [correct] 3. Findings 4. Rationale [Feedback for Answer Choice 1] Context explains the background or larger body of knowledge or area being investigated. [Feedback for Correct Answer 2] Significance is the reason or need for the study—an argument for why the study is important. It is addressed by showing how the study will contribute to knowledge and/or practice because of a controversy, issue, or concern. [Feedback for Answer Choice 3] The findings of a study refer to the results of a study, not the reason or need for a study. [Feedback for Answer Choice 4] 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

While this term may generally refer to purpose, this is not the best fit in the context of research focused vocabulary. [Q4] John has been reading the work of prominent behaviorists and is interested in whether or not the use of positive reinforcement in his classroom will improve the behavior of his students. Which of the following is the most likely source of John’s research problem? 1. Theory [correct] 2. Interests 3. Skepticism and intuition 4. Reproducing and replicating previous research [Feedback for Correct Answer 1] John is interested in applying behaviorist theories to his practice. This is an example of theory application as the source of a research problem. [Feedback for Answer Choice 2] While John may be interested in the outcome, this is not the best source of the research problem. [Feedback for Answer Choice 3] John is not expressing skepticism or intuition in his intended research approach. [Feedback for Answer Choice 4] John may in fact replicate other studies; however, his purpose is not to simply replicate research. He has another purpose. Learning Outcome 3.2: Distinguish among types of variables used to formulate good specific research problem statements and questions, and research hypotheses, for quantitative studies. [Q1] Which of the following is the best example of a hypothesis with an operationally defined variable? 1. There is no difference in second language proficiency at the end of the semester between the two instructional approaches. 2. There is no difference in second language proficiency, as determined by several test scores, between the two instructional approaches. 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. There is no difference in second language proficiency operations between the two instructional approaches. 4. There is no difference in second language proficiency, as determined by those reaching CEFR B1 level, between the two instructional approaches. [correct] [Feedback for Answer Choice 1] In this hypothesis, it is not clear how second language proficiency will be measured. [Feedback for Answer Choice 2] In this hypothesis, it is not clear how second language proficiency will be measured. [Feedback for Answer Choice 3] In this hypothesis, it is not clear how second language proficiency will be measured. [Feedback for Correct Answer 4] This example specifies how second language proficiency is measured. All the other answer choices are too vague. [Q2] Consider the research question, “Are there differences in apparent fatigue in third-grade learners who are either given homework or not given homework?” What is the independent variable? 1. Apparent fatigue 2. Homework status [correct] 3. Given homework 4. Not given homework [Feedback for Answer Choice 1] Keep in mind that fatigue is the variable being affected; therefore, it is the dependent variable. [Feedback for Correct Answer 2] Homework status is the independent variable; given homework or not given homework are the two levels. [Feedback for Answer Choice 3] While this is a level of the independent variable, this is not the best answer. [Feedback for Answer Choice 4] 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

While this is a level of the independent variable, this is not the best answer. [Q3] A researcher is interested in the effect of two types of instruction that they call "whole math" and "drill and practice" on second-grade mathematics achievement. She assigns class 1 to "drill and practice" and class 2 to "whole math." In this design, which of the following constitutes the dependent variable? 1. Type of instruction 2. Mathematics achievement [correct] 3. Class 1 vs. class 2 4. Whole math [Feedback for Answer Choice 1] Remember that the dependent variable is the variable being affected. Type of instruction is not being affected. [Feedback for Correct Answer 2] The researcher believes that the independent variable (type of instruction) will affect the dependent variable (mathematics achievement). [Feedback for Answer Choice 3] Remember that the dependent variable is the variable being affected. The difference between class is not being affected. [Feedback for Answer Choice 4] Remember that the dependent variable is the variable being affected. The instructional approach is not being affected by any other variables. [Q4] A variable that may explain why a certain independent variable has the effect that it does on a dependent variable is called which of the following? 1. Moderating variable 2. Codependent variable 3. Mediating variable [correct] 4. Correlating variable [Feedback for Answer Choice 1] 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Moderating variables influence the nature and strength of the relationship between independent variables. [Feedback for Answer Choice 2] This is not a typical variable descriptor. [Feedback for Correct Answer 3] Mediating variables may explain why a certain independent variable has the effect that it does on a dependent variable. In contrast, moderating variables are variables that influence the nature and strength of their relationship. [Feedback for Answer Choice 4] Correlating research variables have possible strong similarities, but this does not mean either provides explanation for the other. [Q5] Nala wants to determine whether providing awards for science fair winners increases or decreases participation by average students. She thinks that by promoting science as a competition, some children may decide not to participate. To address this question, Nala assigns schools in her district to two types, awarding or nonawarding schools, and compares the participation rate of students in her two groups. Given Nala’s study, what is one extraneous variable she has considered? 1. School location 2. School type 3. Student achievement [correct] 4. Participation rate [Feedback for Answer Choice 1] Extraneous variables are conditions, events, features, or occurrences that target a specific group, that are not part of the study that influences the individuals in the group, and that may provide an alternative explanation for the results. School location is not likely to provide any concrete explanation. [Feedback for Answer Choice 2] Extraneous variables are conditions, events, features, or occurrences that target a specific group, that are not part of the study that influences the individuals in the group, and that may provide an alternative explanation for the results. School type is not likely to provide any concrete explanation. [Feedback for Correct Answer 3] 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Extraneous variables are conditions, events, features, or occurrences that target a specific group, that are not part of the study that influences the individuals in the group, and that may provide an alternative explanation for the results. In this study, because student achievement/interest in science might be a primary reason that students choose to participate in a science fair, Nala has planned to look at “average” students. We can say that she is controlling for this extraneous variable. [Feedback for Answer Choice 4] Extraneous variables are conditions, events, features, or occurrences that target a specific group, that are not part of the study that influences the individuals in the group, and that may provide an alternative explanation for the results. The participation rate will not provide that level of explanation. Learning Outcome 3.3: Understand the nature of qualitative research problem statements and central questions. [Q1] Which of the following questions is the best studied through a qualitative approach? 1. To what extent are students, parents, and faculty supportive of the proposal to increase recess time in the elementary school? 2. Is there a relationship between physical activity levels of parents and physical activity levels of their elementary-aged children? 3. What does well-being mean to elementary physical education teachers? [correct] 4. Do students assigned to an intense physical training program show an increase in self-esteem? [Feedback for Answer Choice 1] This question could be answered using quantitative methods. [Feedback for Answer Choice 2] When considering the relationship between two or more variables, a quantitative approach is likely. [Feedback for Correct Answer 3] This research question is a good start for a qualitative study. As stated, it is unlikely to include a lot of numerical, quantifiable data, and it focuses on one central concept of well-being. [Feedback for Answer Choice 4] Tracking improvement over requires a quantitative approach. 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q2] Which of the following terms refers to an issue or process that is being investigated in a qualitative study? 1. Study focus 2. Central phenomenon [correct] 3. Hypothesis 4. Rationale [Feedback for Answer Choice 1] While this may very well be what the study focuses on, this is not the best answer choice. [Feedback for Correct Answer 2] The central phenomenon in a qualitative study refers to an issue or process that is being investigated. [Feedback for Answer Choice 3] Usually, qualitative studies do not contain hypotheses. [Feedback for Answer Choice 4] The rationale for a study may refer to the reasoning behind the qualitative researcher choosing to follow a particular research path of investigation, but this is not the best answer. [Q3] Which of the following qualitative research paradigms is likely utilized by a researcher seeking to understand the perspectives of a group of people? 1. Ethnographic 2. Phenomenological [correct] 3. Grounded theory 4. Case studies [Feedback for Answer Choice 1] Ethnographic studies often seek to understand and explore, particularly in regard to the lived experiences of people. [Feedback for Correct Answer 2] 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Phenomenological studies often seek to understand the perspectives of a group of people. [Feedback for Answer Choice 3] Grounded theorists seek to discover, establish, and confirm new theories. [Feedback for Answer Choice 4] Case studies are often “deep dives” into individual or small group cases with the intent of answering “how”-type questions. [Q4] Which of the following qualitative research paradigms is likely utilized by a researcher seeking to discover, establish, and confirm? 1. Grounded theory [correct] 2. Ethnographic 3. Phenomenological 4. Case study [Feedback for Correct Answer 1] Grounded theorists seek to discover, establish, and confirm new theories. [Feedback for Answer Choice 2] Ethnographic studies often seek to understand and explore, particularly in regard to the lived experiences of people. [Feedback for Answer Choice 3] Phenomenological studies often seek to understand the perspectives of a group of people. [Feedback for Answer Choice 4] Case studies are often “deep dives” into individual or small group cases with the intent of answering “how”-type questions. Learning Outcome 3.4: Understand the nature of mixed methods research problem statements and questions. [Q1] Josh conducted focus groups of high school seniors to better understand their feelings toward college recruitment. After learning that the group felt apprehensive about the 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

aggressiveness of social media marketing, Josh conducted a quantitative analysis that tracked the number of times a high school senior was presented with an aggressive college recruitment ad. Which of the following terms best describes his design? 1. Qualitative 2. Mixed methods–explanatory 3. Mixed methods–exploratory [correct] 4. Quantitative [Feedback for Answer Choice 1] While there is a qualitative component to Josh’s research, this is not the only approach. [Feedback for Answer Choice 2] In Josh’s work, he is not attempting to explain the phenomenon, but better understand it. [Feedback for Correct Answer 3] Josh is following an exploratory approach as he continues to uncover new information regarding the topic. [Feedback for Answer Choice 4] While there is a quantitative component to Josh’s research, this is not the only approach. [Q2] Carolyn was interested in understanding the beliefs of middle school students regarding the impact of homework on their mental well-being. She completed a series of focus groups and implemented a quantitative measure of well-being. Which of the following research paradigms best fits Carolyn’s research model? 1. Qualitative 2. Mixed methods–explanatory [correct] 3. Mixed methods–exploratory 4. Quantitative [Feedback for Answer Choice 1] While there is a qualitative component to Carolyn’s research, this is not the only approach. [Feedback for Correct Answer 2] 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Carolyn is using both qualitative and quantitative approaches to understand a phenomenon. [Feedback for Answer Choice 3] Carolyn’s methods are targeted on understanding one phenomenon, not exploring other possibilities. [Feedback for Answer Choice 4] While there is a quantitative component to Carolyn’s research, this is not the only approach. [Q3] Including a mixed methods research question in a study using mixed methods approaches strengthens which of the following? 1. Credibility [correct] 2. Findings 3. Probability for publishing 4. Validity [Feedback for Correct Answer 1] In a mixed methods study, a mixed methods question that helps to show the connection between the quantitative and qualitative methods serves to strengthen the credibility of the study. [Feedback for Answer Choice 2] The strength of the questions asked in any study will make for a stronger study, but the findings are not impacted. [Feedback for Answer Choice 3] The addition of this question should have no impact on whether or not the study will be published. [Feedback for Answer Choice 4] This will not impact the study’s validity.

Chapter 3 Application Exercises Chapter 3: Research Problems and Questions Application Exercise 3.1: Narrowing Interests for Researchable Questions Learning Outcome 3.1: Know the nature and sources of general research problems. [Q1] It is important to understand how to take a broad topic and produce a researchable question. In this activity, think about how to narrow an area of interest into an appropriately specific researchable question. Consider the topic: Full inclusion of children with disabilities in general education classrooms. Narrow down the topic to be a researchable question that could be addressed by a qualitative study. [Q1 Model Response] This statement could be narrowed down in a number of ways to develop a specific research question. Most important, the researcher needs to specify what the researcher wants to know about children with disabilities in a general education classroom. For example, the researcher might be interested in the ways in which children with disabilities interact with the other children. An appropriate, straightforward question for a qualitative study might be, “How do children with disabilities interact with their peers in the fully inclusive general education classroom?” [Q2] It is important to understand how to take a broad topic and produce a researchable question. In this activity, think about how to narrow an area of interest into an appropriately specific researchable question. Consider the topic: Full inclusion of children with disabilities in general education classrooms. Narrow down the topic to be a researchable question that could be addressed by a quantitative study. [Q2 Model Response] Perhaps you want to focus on interaction with peers. An example question for a quantitative study is, “Is there a difference in how children with disabilities interact with their peers in fully inclusive general education classrooms as compared to children with disabilities who are in part-time special education and part-time general education classes?” Note you may have a different choice, but you should have clearly specified what you are studying.

[Q3] It is important to understand how to take a broad topic and produce a researchable question. In this activity, think about how to narrow an area of interest into an appropriately specific researchable question. Consider the topic: Full inclusion of children with disabilities in general education classrooms. Narrow down the topic to be a researchable question that could be addressed by an action research study. [Q3 Model Response] A question for an action research project might be, "How can I improve the interactions between children without disabilities and the child with Down syndrome in my classroom?" Be sure that your question is something that you can answer by implementing a plan of action. Application Exercise 3.2: Identifying Variables Learning Outcome 3.2: Distinguish among types of variables used to formulate good specific research problem statements and questions, and research hypotheses, for quantitative studies. [Q1] A variable is any quality or characteristic in a research investigation that has two or more possible values. For the following research statements, identify the variables. Research statement: The purpose of the current study is to test for gender differences in the self-esteem of fourth-grade children. What is the independent variable, and what is the dependent variable? [Q1 Model Response] The independent variable is the gender of the child, which can have more than one value. The dependent variable is self-esteem; that is what the researcher will measure. Note that grade is NOT a variable in this study because only fourth-grade children are included. It is possible that grade might have been explored as a moderating variable, but in this scenario it is controlled (i.e., held constant). [Q2] A variable is any quality or characteristic in a research investigation that has two or more possible values. For the following research statements, identify the variables. Research statement: For decades, researchers have questioned the effects of pictures on learning from text. Some suggest that pictures distract the reader, whereas others indicate that pictures assist the reader in the comprehension of the text. This study 13 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

compares ninth-grade readers’ comprehension of science text with and without pictures. What is the independent variable, and what is the dependent variable? Also, suggest at least one variable that the researcher might want to explore as a possible moderating variable. [Q2 Model Response] The independent variable is type of text—with or without pictures. The dependent variable is comprehension of the text. Any number of variables could be explored as moderators—fluency, for example. Perhaps students who read more fluently would do better with just text, but those who are less fluent would benefit quite a bit from the pictures. Other possibilities might be vocabulary size, reading level, and visual imagery skills. You may have thought of others; keep in mind a moderating variable will increase or decrease the relationship between independent and dependent variables. [Q3] A variable is any quality or characteristic in a research investigation that has two or more possible values. For the following research statements, identify the variables. Research statement: Research has consistently indicated that the transition to college life is difficult. The purpose of this study is to describe the typical first-year experience for women at a small liberal arts college. What is the variable under study here? Why is there no independent variable? [Q3 Model Response] This statement reflects a descriptive (most likely qualitative) study. Qualitative topics call for exploration and often focus on one central phenomenon that is an experience, perception, event, or process that occurs for specific participants at specific sites. Nothing is manipulated by an experimenter, so there is no independent variable, and because a dependent variable “depends” on an independent variable, there is no true “dependent” variable either. The description of one’s experience is the variable under study—that’s what the researcher will measure, using qualitative methods, and the description will “vary” for each participant. Application Exercise 3.3: Research Problem Components Learning Outcome 3.1: Know the nature and sources of general research problems. [Q1] Review the section entitled Research Problem Components to answer the following question: Kasey is a nurse educator who is interested in the ongoing role of the school nurse in high-risk schools. Over the last several years, she has noticed the changing nature of the school nurse as it has evolved to include increasingly challenging roles and 14 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

responsibilities with limited resources. She is working to develop a study to better understand this issue. Which of the three components of research problems is most outlined in this excerpt about Kasey? [Q1 Model Response] This example provides the context for Kasey’s work. She’s looking to explain the background or larger body of knowledge of an area being investigated. In this instance, she has experience in the area and has noted the historical change. [Q2] Review the section entitled Research Problem Components to answer the following question: Kasey is a nurse educator who is interested in the ongoing role of the school nurse in high-risk schools. She has developed a study that will include interviewing school nurses in order to document work load and understand their increasing role in the school setting. Her work will be used to provide guidance to school districts and help shape best practices for school nurses. Which of the three components of research problems is most outlined in this excerpt about Kasey? [Q2 Model Response] In this excerpt about Kasey, we see the significance of her work. Significance is the reason or need for the study. It is an argument for why the study is important. In this case, Kasey is seeking to help districts make research-based changes. [Q3] Review the section entitled Research Problem Components to answer the following question: Kasey is a nurse educator who is interested in the ongoing role of the school nurse in high-risk schools. She has conducted a study that included interviewing school nurses in order to document work load and understand their increasing role in the school setting. She is preparing a paper for publication. In this paper, she has written, “The aim of the current investigation is to better understand the diverse role of the school nurse with implications for system level changes in school districts.” Which of the three components of research problems is most outlined in Kasey’s statement?

[Q3 Model Response] This statement is a clear example of research purpose. The purpose is an indication of a more specific goal of the study. By reading this statement, we understand clearly Kasey’s intentions for the study.

Chapter 3 Test Items SELF-ASSESSMENT ITEMS 1) Which of the following is an example of a research problem purpose statement that indicates the goal or direction of the study? a) The purpose of this study is to investigate instructional strategies. b) The purpose of this study is to research technology issues in the schools. c) The purpose of this study is to investigate the relationship between active learning instructional strategies and student achievement. d) The purpose of this study is to determine if cooperative group instruction has a positive effect on the achievement of first-grade students. 2) Which of the following is an example of a research statement that provides the context for the problem? a) This study is designed to compare the effects of word-supply and word-analysis on the oral reading errors of four students with special needs. b) With the passing of the No Child Left Behind Act in 2001, there has been an increase in national interest in assessment. c) The purpose of this research is to study the social and academic skills of students with behavioral disorders. d) This study will examine the effect of the new math curriculum on student memory and application of math concepts. 3) Which of the following is a researchable problem? a) Can war be prevented in African countries? b) Should music, art, and physical education be eliminated from the curriculum in our school? c) What is the relationship between teachers’ knowledge of classroom management techniques and their use of them? d) Is values education an effective method for providing students with much-needed moral guidance? 4) Which of the following is a legitimate source of an educational research problem? a) Constructivist learning theory b) Personal experience using active-learning strategies c) Clarifying seemingly contradictory results of previous studies on the effectiveness of using technology in classrooms d) All of the above 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

5) Which of the following likely reflects a quantitative research problem? a) How often are first-, second-, and third-grade teachers using math manipulatives as an instructional method? b) What are the experiences of a child who is consistently “left out” of important social events in school? c) This study explored the meaning of “inclusion” for three special needs students who had been placed in a regular education classroom. d) This study examined the process of changing to a standards-based curriculum. 6) Variables used in an educational research study should have which of the following characteristics? a) They are continuous. b) They have distinct levels. c) They vary. d) They can be measured in multiple ways. 7) Which of the following is a conceptual definition of reading? a) Comprehension scores on the Stanford IQ Test b) Extracting meaning from a written text c) The number of words recognized correctly in a 150-word passage d) Scores on the Stanford Diagnostic Reading Test 8) Which of the following is an example of an operational definition? a) Aptitude means having an innate or acquired capacity for something. b) Socioeconomic status has three categories: high, middle, and low. c) Print availability is measured by the number of books in the classroom. d) Motivation is measured by how much effort the students show in class. 9) Which of the following italicized variables can be described as an independent variable? a) The teacher found the phonics method of teaching reading led to an increase in reading scores. b) The researcher studied the effect of parental involvement on the academic achievement of their children. c) A recent study found that SAT scores were not good predictors of college GPA. d) The teacher rearranged the seating in his room to see if this would increase the on-task behavior of some of the students. 18 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

10) Which of the following variables could be considered a confounding variable in a research study designed to assess fourth-grade students’ levels of literacy as measured by the National Assessment of Literacy? a) Grade level b) Literacy level c) Socioeconomic status d) National Assessment of Literacy 11) Which of the following variables could be classified as continuous? a) Gender b) Vocabulary level c) Political party membership d) Skilled/unskilled workers 12) Which research question below indicates the type of research that will be used in a study? a) Should students study longer for the upcoming state proficiency tests? b) What is the best way for young children to learn to read? c) What is the relationship between first graders’ attitudes toward math and their math achievement? d) What would school be like today if World War II had not happened? 13) The following is an example of which type of research hypothesis? Fourth-grade students with greater self-confidence as measured by the Rosenberg Self-esteem Scale will demonstrate more persistence in a problem-solving activity. a) Directional b) Null c) Attitudinal d) Statistical 14) Which of the following is an example of a testable research statement? a) There is a need for high school students to be assigned more homework. b) There is a need for teacher morale to improve in this school district. c) There is a relationship between attendance and grade point average. d) There is a relationship that results from students’ lack of sleep. 15) The results of a study showed that students who were taught to use the SQ3R method scored higher on a social studies test than students who were not exposed 19 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

to this method. Which of the following describes the results in terms of the null hypothesis? a) The researcher rejected the null hypothesis. b) The researcher failed to reject the null hypothesis. c) The researcher retested the null hypothesis to confirm the original findings. d) The researcher proved the null hypothesis was true. 16) Which of the following is an example of a qualitative research question? a) What challenges do students born with fetal alcohol syndrome face? b) Do students who practice with a calculator score higher than students who do not? c) What is the relationship between length of school day and SAT scores? d) What is the effect of attendance on students’ grade point average? 17) Which of the following is NOT an important characteristic of a qualitative research problem? a) It might change as data are collected and analyzed. b) It reflects the researcher’s assumptions and desired findings. c) It is written with “how” and “why” questions to focus on describing the phenomena. d) It includes a central question as well as the participants and the site.

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Which of the following is an example of a research problem purpose statement that indicates the direction of the study? a) The purpose of this study is to research leadership styles. b) The purpose of this study is to determine if portfolios have a positive effect on student achievement. c) The purpose of this study is to investigate instructional strategies and teaching methods. d) The purpose of this study is to determine the effect of teacher education level on classroom achievement. 2) Which of the following is an example of a research statement that provides context for the issue under study? a) This study is designed to compare the effects of student effort and attitude toward math achievement. b) The number of English language learners in the nation’s elementary schools has doubled since 2001. c) The relationship between attendance rate and the size of the school population will be investigated to determine the effect. d) This study will describe and examine beginning teachers’ perceptions. 3) Which of the following is a researchable question? a) Should electronic portfolios be used to assess graduate students’ development as teachers? b) Would it be a good decision to ban competitive games from elementary schools? c) Do teachers’ assessment strategies relate to students’ knowledge of key concepts in chemistry classes? d) Is the teaching of “abstinence” a justifiable approach to sex education? 4) Which of the following statements likely reflects a qualitative research design? a) The research design determines the relationships between continuous variables. b) The research design investigates cause-and-effect relationships. c) The research design emerges as the study progresses. d) The research design describes the role of researcher as an objective observer. 5) Which of the following statements likely reflects a quantitative research method? a) Results are described in narrative form. 21 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Phenomena are examined from the participant's perspectives. c) The research design emerges as the data are collected. d) Variables of interest are manipulated. 6) Which term is used to describe a concept or characteristic that differs in the size, amount, degree, or nature from something else of the same type? a) Abstraction b) Educational concept c) Scientific construct d) Variable 7) The following are definitions of the term tantrum. Identify the conceptual definition. a) An uncontrolled outburst of anger and frustration b) The number of times the student tips his desk over c) A record of the types of actions a child exhibits when presented with a frustrating situation d) The frequency of violent acts toward other students 8) Which of the following is an example of an operational definition? a) Intelligence is defined as the subject’s score on the Wechsler Intelligence Scale. b) Attitude is defined as a settled way of thinking or feeling about someone or something. c) Classroom climate is the social, emotional, and physical aspects of the classroom. d) A developmental disability is a diverse group of chronic conditions due to mental or physical impairments. 9) Which of the following italicized variables can be described as a dependent variable? a) It was found that high school GPA was a better predictor of college performance than SAT scores. b) The data from the study showed that the timeout procedure was effective in reducing the students’ misbehavior. c) The researcher studied the effect of students’ socioeconomic status on academic achievement. d) The teacher was able to increase the number of books read by the students in her class by offering a prize to students who read four books in one month.

10) Which variable could be considered a confounding variable in the research study described below? This study was designed to measure the differences in scores on a stateadministered math achievement test between fourth-grade students who participated in an after-school reading enrichment program versus those who participated in an after-school math enrichment program? a) Grade level of the students b) Time of day of the enrichment program c) Students’ math scores in third grade d) Students’ scores on the test 11) Which of the following represents a continuous variable? a) Religion b) Grade point average c) Socioeconomic status measured as high, middle, or low d) White-collar or blue-collar workers 12) Which research question below indicates the type of research that will be used in the study? a) Can laws be written that require teachers and students to change? b) What is the effect on student achievement between students who received formative quizzes versus those who received summative quizzes? c) How do you determine the overall risk level for an operation? d) Should schools have parents with sociodemographic characteristics work in the school? 13) The following is an example of which type of research hypothesis? Ninth-grade students with greater levels of math test anxiety will score lower on a math achievement test. a) Directional b) Null c) Attitudinal d) Statistical 14) The results of a study showed that first-grade students who were instructed using an active-learning approach developed literacy skills at the same rate as students who received the traditional approach. Which of the following describes the results in terms of the null hypothesis? 23 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

a) The researcher rejected the null hypothesis. b) The researcher failed to reject the null hypothesis. c) The researcher retested the null hypothesis to confirm the original findings. d) The researcher proved the null hypothesis was true. 15) Which of the following statements is testable using a research study? a) Math test anxiety should be dealt with in high school. b) There is a relationship that results from an ineffective method of changing behavior. c) There is an effect of math test anxiety on student math performance. d) Teachers need to increase their literacy knowledge and skills. 16) Which of the following is an example of a qualitative research question? a) Do students who participate in cooperative learning groups score higher than students who do not? b) What is the effect of attendance on students’ scores on a college entrance exam? c) What is the relationship between students’ reading test scores and math test scores? d) How do math teachers describe their pedagogical decisions before and after implementation of the new math curriculum? 17) Which of the following is NOT a characteristic of a qualitative research problem? a) It is open-ended. b) It is evolving. c) It is process oriented. d) It identifies specific variables to be studied. Short-Answer Questions 1) What is the main difference between a confounding variable and an extraneous variable? 2) Why is it important to use operational definitions of variables in a study? Essay Question Describe at least three main differences between a quantitative study and a qualitative study.

Chapter 3 Test Answer Key ANSWER KEY: Chapter 3 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. d 2. b 3. c 4. d 5. a 6. c 7. b 8. c 9. a 10. c 11. b 12. c 13. a 14. c 15. a 16. a 17. b

INSTRUCTOR’S EXAM ITEMS 1. b 2. b 3. c 4. c 5. d 6. d 7. a 8. a 9. b 10. c 11. b 12. b 13. a 14. b 15. c 16. d 17. d

Chapter 4 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents

Chapter 4: Locating and Reviewing Related Literature .............................................. 1 Learning Outcome Quizzes........................................................................... 1 Application Exercises .................................................................................. 11 Test Items ................................................................................................... 16 Test Answer Key ......................................................................................... 22

Chapter 4: Locating and Reviewing Related Literature Chapter 4 Learning Outcome Quizzes Learning Outcome 4.1: Describe how the review of literature is used to establish significance; to write research questions; and to identify appropriate sampling, data collection, and procedures. [Q1] A review of literature that helps to justify or document the need for the current study is found typically in which of the following? 1. Qualitative research 2. Both qualitative and quantitative research [correct] 3. Quantitative research 4. Only published research [Feedback for Answer Choice 1] Qualitative studies certainly contain literature reviews, but these are not the only studies that contain reviews of the literature. [Feedback for Correct Answer 2] Existing research is commonly cited in both qualitative and quantitative studies as the authors explain why the current study is worthwhile. The literature review helps to refine the problem and establish a framework for the study. [Feedback for Answer Choice 3] Quantitative studies certainly contain literature reviews, but these are not the only studies that contain reviews of the literature. [Feedback for Answer Choice 4] Unpublished research also contains a literature review. [Q2] Consider the following passage from the literature review of a study focused on parent– child conversation: “Children rely on and require adult scaffolding to develop their naïve understandings, especially about the natural world. For example, in science museums, children who engaged in conversation with parents spent more time exploring exhibits as parents pointed out less salient features and helped children to generate their own evidence of 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

the phenomena (Crowley et al., 2001). Parents offered causal explanations in about one-third of these interactions (e.g., “See you can make it gallop like this . . . Because every time it goes through there [points to photoelectric switch] it pops, see?”), and these explanations further encouraged children to talk about the exhibits themselves (Crowley & Callanan, 1998). These co-constructed explanations form the basis of children’s informal science education and, not surprisingly, predict science literacy in the elementary years. “However, Crowley et al. found that only 4% of the parent explanations focused on the scientific principle behind the exhibit. This finding raises several questions. For example, did parents fail to explain because they did not know the causal principles behind the complex museum exhibits (e.g., a zoetrope) or believe the children would not understand? If complexity is the issue, then a more natural environment designed for younger children may lead to more science talk among parents and their children. Another possibility is that parents fail to offer explanations because the children had already provided them. To explore this question, we look simultaneously at the questions parents and children ask, and the explanations that parents and children give, to establish a picture of the co-constructive process of informal science education.” Which of the following best explains the purpose of this part of the literature review? 1. To develop the overall significance of the study 2. To develop specific research questions [correct] 3. To identify methodological strengths and limitations in earlier studies 4. To identify contradictory findings [Feedback for Answer Choice 1] While this may be found in a complete literature review, this is not the intent of this particular passage. [Feedback for Correct Answer 2] Although the complete literature review may do all of these things, this particular passage has a clear purpose: to help focus and develop the specific questions that are addressed in the study the researcher plans to conduct. [Feedback for Answer Choice 3] While this may be found in a complete literature review, this is not the intent of this particular passage. [Feedback for Answer Choice 4] While this may be found in a complete literature review, this is not the intent of this particular passage. 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q3] A researcher who using a literature review to make a case for why the research will fill a gap in the current research base with new and needed information is using the review to do which of the following? 1. Develop research questions 2. Review current understandings 3. Formulate hypotheses 4. Develop significance [correct] [Feedback for Answer Choice 1] It is likely that this is a process that a researcher will undergo within the literature review process, but that is not the best answer choice for this question. [Feedback for Answer Choice 2] The literature review should achieve this purpose, but this is not the best answer for this question. [Feedback for Answer Choice 3] Hypotheses may be formulated as a function of a substantial review of the literature, but this is not the best answer for this question. [Feedback for Correct Answer 4] In this instance, the researcher is making a case for the significance of their work, noting gaps in the literature that should be filled in order to improve the science. [Q4] Which of the following sections of a research report is heavily influenced by the review of literature? 1. Discussion [correct] 2. Introduction 3. Methods 4. Results [Feedback for Correct Answer 1] The discussion section allows the researcher to discuss the implications of their work. These implications are situated within the previous literature. [Feedback for Answer Choice 2] 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

The introduction comes before the review of literature. [Feedback for Answer Choice 3] There may be some influence from the literature review to the methods section, but this is not a section heavily influenced by the review of literature. [Feedback for Answer Choice 4] The results section tends to focus on the study at hand, not other studies. Learning Outcome 4.2: Understand how to conduct a review of literature, including identification of key terms, databases, and search strategies. [Q1] Ariel is conducting a literature search on policy as related to food choices in urban high schools. Which of the following represent the best search terms for Ariel to use for an electronic database search? 1. Food policy and high school [correct] 2. Food policy or high school 3. School or food policy 4. Students and urban education [Feedback for Correct Answer 1] This combination of terms will yield results related to food policy and high school, which is exactly what Ariel is hoping to discover. [Feedback for Answer Choice 2] This would yield results on either food policy or high school. This would not yield productive results for Ariel. [Feedback for Answer Choice 3] This would yield results on either food policy or school. This would not yield productive results for Ariel. While not being focused on the appropriate topics, this search would also yield thousands of results. [Feedback for Answer Choice 4] These terms would not yield results related to Ariel’s topic of interest. [Q2]

Candy is studying bullying in school, and she searches an online database to find past research. Which one is the best choice? 1. ERIC, because it is focused on education [correct] 2. PsycINFO, because bullying occurs in many areas, not just in school 3. EBSCO (Academic Search Complete), because it is the most comprehensive 4. Google Scholar, because it provides access to millions of articles [Feedback for Correct Answer 1] ERIC is the Education Resources Information Center, an education-focused database, and Candy’s topic has an educational focus. Her search should return the most relevant articles to her own research problem. [Feedback for Answer Choice 2] PsycINFO is a perfectly acceptable database, but this is not the best choice for Candy given the specificity of her topic. [Feedback for Answer Choice 3] EBSCO is a perfectly acceptable database, but this is not the best choice for Candy given the specificity of her topic. [Feedback for Answer Choice 4] Google Scholar will likely yield numerous results that are not of interest to Candy, thus complicating her search. [Q3] When evaluating potential sources for inclusion in your literature review, you should do which of the following? 1. Favor review articles over articles that report just one study 2. Favor articles from refereed journals over nonrefereed journals [correct] 3. Favor Internet sites with more hits over those with fewer hits 4. Favor articles reporting quantitative studies over those reporting qualitative ones [Feedback for Answer Choice 1] Some of your most important studies in a literature review will include seminal works in the field. [Feedback for Correct Answer 2] 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Studies published in refereed journals are seen as more scholarly and trustworthy than articles from nonrefereed or popular journals. [Feedback for Answer Choice 3] The popularity of an Internet site does not mean the site is credible. [Feedback for Answer Choice 4] Quantitative and qualitative studies are different but are equally important. [Q4] Which of the following terms refers to the process by which experts in a given field evaluate the quality of research studies prior to their acceptance for publication? 1. Journal review 2. Editorial review 3. Accuracy review 4. Peer review [correct] [Feedback for Answer Choice 1] This is not a phrase used in the review of academic manuscripts. [Feedback for Answer Choice 2] Editorial review means that a journal editor or editorial review board will review the manuscript, but this is not the correct answer. [Feedback for Answer Choice 3] This is not a phrase used in the review of academic manuscripts. [Feedback for Correct Answer 4] Peer review means that experts in the field review the manuscript and make recommendations about revisions and acceptance. [Q5] Which of the following would yield the widest range of scholarly results? 1. ERIC 2. Social Science Citation Index 3. PsycINFO 4. Google Scholar [correct] 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 1] ERIC yields results only related to the field of education. [Feedback for Answer Choice 2] The Social Science Citation Index yields results in the social sciences, but this is not the broadest. [Feedback for Answer Choice 3] PsycINFO yields results in psychology and related fields. [Feedback for Correct Answer 4] Google Scholar yields the widest range of results as it pulls in any related items from books to journal articles to websites, etc. Learning Outcome 4.3: Be able to synthesize related literature and write a review of literature. [Q1] Which of the following is usually the best way to organize a literature review? 1. Organize the reviewed studies into groups of topics or themes [correct] 2. Start with the most relevant articles and then build to more general sources 3. Discuss the studies in chronological order 4. Describe only those studies that agree with one another [Feedback for Correct Answer 1] The literature should be organized in a meaningful way that indicates you have thought about the big ideas across the studies. [Feedback for Answer Choice 2] A review written this way would lack continuity. [Feedback for Answer Choice 3] Topics do change and shift over time, but by writing the review chronologically you are likely to produce a disjointed literature review. [Feedback for Answer Choice 4] It is important to include articles with differing viewpoints and findings in a quality literature review. 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q2] Which of the following represents the primary reason to use a literature matrix? 1. Identify a collection of key search terms to use with a variety of electronic databases 2. Develop a graphic, visual presentation that shows how key studies are related 3. Organize important information about each key study identified during the literature review [correct] 4. Come up with a system for coding key information relevant to the research problem [Feedback for Answer Choice 1] This is not the function of a literature matrix. [Feedback for Answer Choice 2] This is not the function of a literature matrix. [Feedback for Correct Answer 3] A literature matrix is an organized table that contains important information about each key study. [Feedback for Answer Choice 4] This is not the function of a literature matrix. [Q3] Why is a primary source often more valuable to a researcher than a secondary source? 1. A primary source gives general information about a particular topic. 2. A primary source provides detailed information about all aspects of a study. [correct] 3. A primary source reviews relevant literature. 4. A primary source quickly summarizes the results of a number of studies. [Feedback for Answer Choice 1] This is a characteristic of a secondary source, not a primary source. [Feedback for Correct Answer 2] Secondary sources are good places to start, but primary sources provide details about the method, procedure, and implications of particular studies, as well as the original data analysis. [Feedback for Answer Choice 3] 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Reviews of the literature are considered secondary sources. [Feedback for Answer Choice 4] This is a characteristic of a secondary source, and a fine place to start, but it lacks needed specificity. [Q4] Kelly’s adviser suggested that he start with secondary sources to gain a sense of the existing literature for his research topic. Which of the following is most likely to be a useful secondary source? 1. A journal article that presents a study very similar to the one he is planning 2. The professional website for a researcher who studies Kelly’s topic 3. The Review of Research in Education [correct] 4. An article downloaded from ERIC or PsycINFO [Feedback for Answer Choice 1] While this may be useful, this is certainly not the most efficient path. An article focused on one study is a primary source. [Feedback for Answer Choice 2] Remember that websites are not peer reviewed, regardless of the credibility of the owner of the site. [Feedback for Correct Answer 3] A secondary source is one that reviews, summarizes, or discusses primary research as well as theoretical positions or experience. A downloaded article or website may or may not be a good secondary source, and an article that presents one study is a primary source. The word “Review” in this title is a good tip-off to Kelly that it’s useful as a secondary source. [Feedback for Answer Choice 4] A downloaded article may or may not be a good secondary source, so this is not the best approach. [Q5] Which of the following is true regarding the literature review in a written report of qualitative research? 1. It is extensive in the beginning of the article. 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. There is no review of the literature in a qualitative study. 3. It is integrated throughout the article. [correct] 4. It is optional in a qualitative. [Feedback for Answer Choice 1] This is typical of quantitative studies, not qualitative studies. [Feedback for Answer Choice 2] Literature reviews are vitally important to qualitative studies. [Feedback for Correct Answer 3] Unlike quantitative studies, qualitative studies often do not include extensive literature reviews at the beginning of the report. Keeping the references to existing literature minimal at the start of a research report allows the views of the participants to emerge without being constrained by the views of previous researchers. [Feedback for Answer Choice 4] A quality literature review is vital, not optional.

Chapter 4 Application Exercises Chapter 4: Locating and Reviewing Related Literature Application Exercise 4.1: Searching the ERI C Database l

Learning Outcome 4.2: Understand how to conduct a review of literature, including identification of key terms, databases, and search strategies. [Q1] It is not advisable to have a research topic that is too broad, but often learners’ initial topics are much too broad. Sometimes students have difficulty applying search terms. For practice in narrowing a search, suppose that you are interested in the effects of teaching children strategies to learn from text. Consider the following: An initial ERI C search on strategies and children yields more than 18,000 sources. You realize that you are most interested in the effects of teaching learners to use particular strategies. Go to the ERI C database using the link below and search the collection for the keywords “strategies” and “children.” How many sources were returned? Next, narrow your search to include only sources since 2016. How many sources were returned? l

Website title: ERI C l

[Q1 Model Response] In November 2020, this search returned 16,417 results. Compare your results to this number. You’ll see that the research continues to grow at a rapid rate. This is a good thing! As you narrow to 2016, you’ll see that you typically find results that are more contemporary and often more easily applied to your own research area, excluding seminal work, of course. [Q2] Often a researcher must consult multiple databases in order to adequately conduct a literature search. Conduct a search for the terms “alternative education” in both the ERI C database and the Google Scholar database. What key differences do you note? What might this tell you about conducting quality literature searches? l

[Q2 Model Response] Conducting this search in both the ERI C database and via Google Scholar will yield similar results, yet Google Scholar will yield almost 20 times more results. This is an indication of how databases design their algorithms. Google Scholar has a much broader algorithm that includes a wide array of results. This should teach you that to l

conduct a quality search, it is important to explore multiple databases and consider the strengths and weaknesses of each. [Q3] Consider the following scenario: Mike is an undergraduate student and an education major. He is in a class focused on educational technology. He’s been assigned the task of reviewing the literature focused on the use of technology in a subarea of education that interests him. Mike is interested in assistive technologies for students on the autism spectrum. If you were advising Mike, where would you advice he start? What search terms should he use? [Q3 Model Response] Mike has an interest in a very important field, but also a field with a great deal of research. If he only uses one database or does not try a combination of relevant key words, he will quickly become frustrated. It would be important to remind Mike that he should use multiple databases and compare results. As all researchers should, Mike should follow the four-step process outlined in Figure 4.2: (1) identify the topic and key terms, (2) identify databases and interface, (3) use appropriate thesaurus, (4) conduct the search. Application Exercise 4.2: Identifying Primary and Secondary Sources Learning Outcome 4.1: Describe how the review of literature is used to establish significance; to write research questions; and to identify appropriate sampling, data collection, and procedures. [Q1] Based on the title and the summary for the following article reference, identify this reference as a primary or secondary source. Bruffee, K. A. (1987). The art of collaborative learning. Change, 19, 42–47. This article reviews the literature and describes the history of collaborative learning. Collaborative learning is becoming more familiar today and is used not only in primary and secondary schools but also in colleges and universities, business management, banks, legal firms, and hospitals. Collaborative learning increases interdependence in a world that requires greater flexibility and change. Willingness to grant authority, willingness to take on and exercise authority, and a context of friendliness and good grace are the three ingredients essential to successful autonomous collaboration.

[Q1 Model Response] A primary source is a first-hand account of a study written by the person who conducted it. A secondary source is one in which a brief secondhand account is provided by someone other than the person who conducted the study. This article is likely a secondary source. The summary describes it as a review of the literature rather than as a study conducted by the author of the article. [Q2] Based on the title and the summary for the following article reference, identify this reference as a primary or secondary source. Hedin, D. (1987). Students as teachers: A tool for improving school climate and productivity. Social Policy, 17, 42–47. This article reviews the evidence for the value of tutoring and examines the effectiveness of children and adolescents as tutors to their peers or to younger children. A range of social, psychological, and academic goals for the tutor and/or tutee can be achieved with this method. Changes in ego strength or moral judgment may benefit the tutor, whereas increased academic achievement may benefit the tutee. Other goals may benefit the social climate or productivity. [Q2 Model Response] This article is likely a secondary source. The abstract suggests it is a review of the evidence, rather than a firsthand account of evidence collected by the author. [Q3] Based on the title and the summary for the following article reference, identify this reference as a primary or secondary source. Turnbull, K., & Bronicki, G. J. (1989). Children can teach children. Teaching Exceptional Children, 21, 64–65. This article describes the ways in which students may work together in inclusive settings and tests the hypothesis that serving as peer tutors can enhance inclusive practices. The results indicated that students benefit from teaching other students, and that inclusion is better realized in collaborative spaces. [Q3 Model Response] This article is likely a primary source. The abstract suggests it is a description of a study and a firsthand account of the results.

Application Exercise 4.3: Identifying Primary and Secondary Sources Learning Outcome 4.3: Be able to synthesize related literature and write a review of literature. [Q1] Consider the following scenario: Allen is a master’s degree student in education. He is taking his first class focused on research methods. His major assignment for the course is to complete a literature review of a topic of his choosing. Allen has chosen to research the effects of Response to Intervention, a tiered-model used to help identify students who may qualify for special education services, on students with classroom behavior issues. Allen knows that his literature review must achieve several purposes and he is just getting started. Review the section, Writing a Review of Literature to answer the following question: What is the first goal Allen should consider when completing his literature review? [Q1 Model Response] As you learned, Allen’s first goal is to summarize the findings of previous studies in this area. He is going to spend time learning about how other researchers have conceptualized the issue he is interested in, in this case RTI, and summarizing how they came to appropriate conclusions. This is a critical first step in the process. [Q2] Consider the following scenario: Allen is a master’s degree student in education. He is taking his first class focused on research methods. His major assignment for the course is to complete a literature review of a topic of his choosing. Allen has chosen to research the effects of Response to Intervention, a tiered-model used to help identify students who may qualify for special education services, on students with classroom behavior issues. Allen knows that his literature review must achieve several purposes and he is just getting started. He has worked already to summarize previous studies. Review the section Writing a Review of Literature to answer the following question: Since Allen has summarized previous studies, what should he consider next when completing his literature review? [Q2 Model Response] Since Allen has already summarized previous studies, it is now time for him to analyze the quality of the findings of those studies. While summary is important, it would be 14 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

irresponsible to present all studies as credible without vetting them based on standards of good research. [Q3] Consider the following scenario: Allen is a master’s degree student in education. He is taking his first class focused on research methods. His major assignment for the course is to complete a literature review of a topic of his choosing. Allen has chosen to research the effects of Response to Intervention, a tiered-model used to help identify students who may qualify for special education services, on students with classroom behavior issues. Allen knows that his literature review must achieve several purposes and he is just getting started. He has worked already to summarize previous studies as well as analyzed the quality of the findings of these studies. Review the section Writing a Review of Literature to answer the following question: Since Allen has summarized previous studies and analyzed the quality of those studies, what should he consider next when completing his literature review? [Q3 Model Response] Allen has done the first two critical steps: He’s summarized previous studies and analyzed their findings. It is now time for the final step. Allen should relate findings and previous methods to his study. It’s time for him to dig in and determine how his study will relate methodologically, how he may improve on previous work, or how he may explore uncharted concepts.

Chapter 4 Test Items SELF-ASSESSMENT ITEMS 1) Which of the following is NOT a recommended step in conducting a literature review? a) Identify the key words related to the problem. b) List all the publications that might have articles on the selected problem. c) Check secondary sources for reviews of research on the problem. d) Search the indexes for primary sources. 2) Why are primary sources an important source of information to a researcher? a) They provide the principal information about a topic. b) They provide information from the primary stages of development. c) They provide original articles in which researchers communicate directly. d) They provide a synthesis of the primary research on a topic. 3) A researcher wanted to do a literature review on teaching calculator skills to students with special needs. Which of the following is a possible secondary source the researcher might use to find information on the topic? a) Journal of Special Needs Education b) Handbook of Special Education: Research and Practice c) Journal of Special Education d) Teaching Exceptional Children 4) Which of the following describes what a researcher is doing when the results of many studies are statistically combined? a) An analysis of data b) A literature review c) A meta-analysis d) A research summary 5) In order to locate literature related to a research problem, a researcher needs to identify the key terms related to the problem. Which of the following is often used to find appropriate words and phrases to conduct a literacy search? a) Educator’s Handbook: A Research Perspective b) Encyclopedia of Educational Research c) Resources in Education 16 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

d) ERI C Thesaurus l

6) A researcher identified middle school and student achievement as two key terms for his literature search. Which of the following would produce the fewest references in a computerized ERI C search? l

a) Middle schools b) Student achievement c) Middle schools or student achievement d) Middle schools and student achievement 7) What is the advantage of using refereed journal articles rather than nonrefereed journal articles when conducting a literature search? a) Refereed journals summarize results from previous research studies, and nonrefereed journals do not. b) The articles in a refereed journal have been subjected to external review and nonrefereed journal articles have not. c) Refereed journals provide original articles of related studies and nonrefereed journals do not. d) The articles in a refereed journal review and discuss both primary and secondary research and nonrefereed journals do not. 8) Which of the following is a major disadvantage of using the Internet for educational research? a) Quality information is not usually available. b) There is no standard vocabulary that facilitates a search. c) The Internet provides niche information. d) Articles available on the Internet are never from peer-reviewed journals. 9) Which of the following is likely to have the most current statistics and governmental information? a) Internet b) ERI C l

c) PsycINFO d) Review of Research in Education 10) Which of the following is commonly used to find links to educational research news, grant opportunities, statistics, publications, and other federally supported centers? a) edweek.com b) ies.ed.gov 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) nwreel.org d) uno.edu 11) What is one advantage of an Internet subject directory over an Internet search engine? a) Content has been reviewed and categorized by a person. b) Content has been reviewed electronically so there is no bias. c) Multiple search engines are used. d) The number of websites is unlimited. 12) Which of the following reflects the purpose of a literature matrix? a) It can help a researcher locate primary and secondary sources of literature. b) It provides a computer-generated environment for controlling the quality of research literature. c) It reviews and discusses primary research as well as theoretical ideas. d) It provides a way to summarize important information on a research topic. 13) Which of the following is most likely found in a literature review? a) Extensive quotations from experts in the field b) A complete review of all studies related to the topic c) A synthesis of the results from previous studies on the topic d) A description of the methods and instruments used in the study

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Which of the following reflects the general purpose of a literature review? a) To learn all there is to know about a topic b) To relate previous research to the current problem c) To isolate specific research methods d) To describe the differences between primary research and secondary research on a topic 2) What do primary sources of information provide to the researcher? a) Original articles in which researchers communicate directly b) An analysis of the primary information about a topic c) Information from the primary stages of development d) A synthesis of the primary research on a topic 3) Which of the following is a benefit of doing a literature review? a) It can help the researcher establish the conceptual framework. b) It can help the researcher develop specific research questions and hypotheses. c) It can help the researcher to determine participant selection procedures that should be avoided. d) It can provide the researcher with long quotations from experts to use in the manuscript. 4) Which of the following is analogous to reading a movie review? a) Reading a primary source b) Conducting a literature review c) Reading a secondary source d) Conducting an ERI C search l

5) Which of the following is likely to contain primary sources of literature? a) Encyclopedia of Educational Research b) Handbook of Research on Teaching c) Scholarly books d) Elementary School Journal 6) Which of the following best describes a meta-analysis? a) It is a lengthy literature review. 19 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) It allows researchers to conduct analyses on multiple variables. c) It is a review that uses statistical methods to synthesize previous research. d) It is a research summary of all the research on a topic including those that were poorly conducted. 7) What is the difference between a refereed journal and a nonrefereed journal? a) Secondary sources are not reviewed in a refereed journal whereas they are in a nonrefereed journal. b) A nonrefereed journal uses a blind review process whereas a refereed journal does not. c) A refereed journal uses external reviewers whereas a nonrefereed journal does not. d) A nonrefereed journal summarizes and analyzes results of previous studies whereas a refereed journal only publishes original articles or reports. 8) What does the ERI C Thesaurus provide? l

a) A description of research articles in educational journals b) A controlled vocabulary of key terms called descriptors c) A list of research topics d) A rating system of journal quality 9) A researcher identified drug testing and high school as two key terms. Which of the following phrases would result in the most refined literature search? a) Drug testing b) High schools c) Drug testing and high schools d) Drug testing or high schools 10) Which of the following offers the least control of the quality of information? a) Internet b) ERI C l

c) PsycINFO d) Education Index 11) In which of the following search tools has the content of Internet resources been reviewed and organized by teams of editors who have knowledge of specific disciplines? a) edweek.com b) Subject directory 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) Metasearch engine d) Search engine 12) Which of the following should researchers NOT do when reviewing studies related to their research problem? a) Summarize results of previous studies b) Comment on methodological weaknesses and limitations in previous studies c) Cite actual findings from other studies d) Assume authors are biased if citing their own work 13) What is the advantage of using a metasearch engine? a) A metasearch engine allows for complex search terms. b) A search is submitted to multiple search engines simultaneously. c) A metasearch engine provides access to all of the major search engines. d) Sponsored websites are given priority and are normally displayed first. Short-Answer Questions 1) What is the difference between a primary and a secondary source? 2) What is the purpose of a literature map? Essay Question Describe at least five reasons for conducting a review of the literature before embarking on a research study.

Chapter 4 Test Answer Key ANSWER KEY: Chapter 4 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. b 2. c 3. b 4. c 5. d 6. d 7. b 8. b 9. a 10. b 11. a 12. d 13. c

INSTRUCTOR’S EXAM ITEMS 1. b 2. a 3. d 4. c 5. d 6. c 7. c 8. b 9. c 10. a 11. b 12. d 13. b

Chapter 5 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 5: Participants and Sampling for Quantitative Designs ............................... 1 Learning Outcome Quizzes........................................................................... 1 Application Exercises .................................................................................. 12 Test Items ................................................................................................... 16 Test Answer Key ......................................................................................... 22

Chapter 5: Participants and Sampling for Quantitative Designs Chapter 5 Learning Outcome Quizzes Learning Outcome 5.1: Describe the differences between samples and populations, and how sample statistics are used to approximate what is true for the population. [Q1] Sharon is interested in the topics that art teachers choose to teach in U.S. middle schools. What would be the population for her study? 1. The art teachers who belong to one professional organization 2. The art teachers who respond to her survey 3. All art teachers in the United States [correct] 4. The art teachers in her district [Feedback for Answer Choice 1] This would be too narrow. [Feedback for Answer Choice 2] This would be her sample. [Feedback for Correct Answer 3] The population refers to the group of interest to the researcher. [Feedback for Answer Choice 4] This would be too narrow. [Q2] Greg is testing a new science curriculum in his high school. He has 150 students try the new problem-based learning tasks and 150 students use the usual curriculum. All students complete a pretest and posttest about science concepts. What is the sample in this study? 1. All science students at this high school 2. 300 high school students [correct] 3. 150 students 4. All high school students who take science 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 1] Students who do not participate could not be included in the sample. [Feedback for Correct Answer 2] Greg had a total of 300 students who were selected and participated in the study. [Feedback for Answer Choice 3] Remember that the sample would include all participants in the study. [Feedback for Answer Choice 4] This would be much too large and unrealistic. [Q3] Qualeya has conducted a study of 238 seventh-graders in a local high school. She has surveyed their use of smartphones in class. In her study, the 238 seventh-graders constitute which of the following? 1. Sample, because Qualeya has selected a subset of the larger seventh-grade population [correct] 2. Sample, because all of Qualeya’s participants are confined to one specific school 3. Population, because Qualeya has surveyed the entire seventh grade in a school 4. Population, because there are greater than 100 [Feedback for Correct Answer 1] These 238 seventh-graders constitute Qualeya’s sample. They are a subset of a larger group of the overall population of seventh-graders. [Feedback for Answer Choice 2] It is not a requirement that a sample be confined to one physical location. [Feedback for Answer Choice 3] While Qualeya has surveyed a whole school of seventh-graders, these are not the only seventh-graders to be surveyed. Qualeya could easily expand to other schools. [Feedback for Answer Choice 4] There’s no specific number that constitutes a particular sample or population. [Q4] 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Cristina is interested in conducting a study on the impact of positive reinforcement on the homework habits of third-grade students. She considers sending a survey to the parents of all third-grade students in the United States in order to explore parental perception of positive reinforcement and homework completion. For practical purposes she decided to select a random sample of parents across a random sampling of states. In this scenario, what does the total population of the parents of third-grade students represent? 1. Population, because it represents the entirety of a group [correct] 2. Population, because it is likely well over 500 individuals 3. Sample, because it represents the larger group from which the subset was derived 4. Sample, because this group could stand alone for a potential study [Feedback for Correct Answer 1] The entirety of third-grade parents in the United States represents a population. Cristina’s smaller subset constitutes her sample. [Feedback for Answer Choice 2] Population and samples are not defined by total enrollment. [Feedback for Answer Choice 3] This is not an accurate description of sample. [Feedback for Answer Choice 4] While it is possible that this group could technically stand alone, that is highly unlikely, and in this instance, not the right answer. Learning Outcome 5.2: Contrast random with nonrandom sampling procedures. [Q1] Which of the following is an example of simple random sampling? 1. Using random numbers to assign individual teachers to experimental groups 2. Dividing teachers into groups of males and females and using random numbers to select teachers from each 3. Randomly selecting five schools in the state and selecting all teachers within the schools 4. Using random numbers to select teachers from a list of all elementary teachers in a district [correct] [Feedback for Answer Choice 1] 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This is not simple random sampling. [Feedback for Answer Choice 2] This is not simple random sampling. [Feedback for Answer Choice 3] This is not simple random sampling. [Feedback for Correct Answer 4] In this example, each teacher has an equal chance of being selected. [Q2] Edmund has compiled a list of 1348 students in his high school. He has selected a sample of 42 students by choosing every 14th student on this list. Which sampling technique is he using? 1. Stratified random sampling 2. Systematic sampling [correct] 3. Cluster sampling 4. Nonproportional stratified sampling [Feedback for Answer Choice 1] In stratified sampling, elements are selected randomly from strata that divide the population. [Feedback for Correct Answer 2] Selecting every nth individual from a list is a classic example of systematic sampling. [Feedback for Answer Choice 3] In cluster sampling, equal groups are identified and selected randomly, and participants in each group selected are used as the sample. [Feedback for Answer Choice 4] In stratified sampling, elements are selected randomly from strata that divide the population. [Q3] You want to do a study to examine the effects of socioeconomic status on preschool enrollment. You can randomly sample the parents of all the children at the local preschools, but you are concerned about the disproportionate number of families with 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

higher socioeconomic status. To address this concern, you should use which of the following? 1. Stratified sampling 2. Cluster sampling 3. Convenience sampling 4. Systematic sampling [correct] [Feedback for Answer Choice 1] In stratified sampling, elements are selected randomly from strata that divide the population. [Feedback for Answer Choice 2] In cluster sampling, equal groups are identified and selected randomly, and participants in each group selected are used as the sample. [Feedback for Answer Choice 3] In convenience sampling, participants with the needed characteristics are selected on the basis of availability. [Feedback for Correct Answer 4] Selecting every nth individual from a list is a classic example of systematic sampling. [Q4] Which of the following is an example of simple random sampling? 1. Using random numbers to select teachers from a list of all elementary teachers in a district [correct] 2. Using a numbering system to assign individual teachers to experimental groups 3. Dividing teachers into groups of males and females and using random numbers to select teachers from each 4. Randomly selecting five schools in the state and selecting all teachers within the schools [Feedback for Correct Answer 1] In this example, each teacher has an equal chance of being selected, which is the hallmark of simple random sampling. [Feedback for Answer Choice 2] This is typical of systemic sampling. 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] This describes a stratified random sample. [Feedback for Answer Choice 4] This is an example of cluster sampling. Learning Outcome 5.3: Distinguish among types of nonrandom sampling procedures and possible sources of bias. [Q1] Which of the following is an example of purposeful qualitative sampling for a study about teachers' experiences with students diagnosed with autism? 1. Asking teachers of children with autism to suggest other teachers to contact [correct] 2. Randomly selecting teachers from a district list of teachers assigned to teach children with special needs 3. Contacting all teachers of science in the district 4. Selecting parents who belong to an autism advocacy group [Feedback for Correct Answer 1] Asking teachers of children with autism to suggest other teachers to contact is an example of purposefully selecting a sample on the basis of the participant’s ability to help the researcher understand the phenomenon of interest. In particular, this example represents snowball sampling. [Feedback for Answer Choice 2] This constitutes a convenience sample. [Feedback for Answer Choice 3] This constitutes a convenience sample. [Feedback for Answer Choice 4] This answer choice does not include teachers, the target population of the study. [Q2] Dr. Wise has chosen to study four classes of sixth-grade students in a local school because he has an excellent relationship with the principal and teachers in that school. To select his sample, he is likely to use which of the following? 1. Quota sampling 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. Stratified sampling 3. Convenience sampling [correct] 4. Cluster sampling [Feedback for Answer Choice 1] Quota sampling will not allow Dr. Wise to take advantage of the connections he has established. [Feedback for Answer Choice 2] Stratified sampling will not allow Dr. Wise to take advantage of the connections he has established. [Feedback for Correct Answer 3] Given the relationship Dr. Wise has with the principal and teachers, he is likely to get full cooperation for his study. This is one of the reasons why researchers use convenience sampling. [Feedback for Answer Choice 4] Cluster sampling will not allow Dr. Wise to take advantage of the connections he has established. [Q3] Dr. Clark is conducting a qualitative study to understand the role of physical education teachers in helping children become more active. To sample, she starts with two teachers she knows and asks them to suggest other physical education teachers who would be good to include in the study. She is using which of the following? 1. Homogenous sampling 2. Snowball sampling [correct] 3. Intensity sampling 4. Purposive sampling [Feedback for Answer Choice 1] The nature of this recruitment technique does not reflect homogenous sampling. [Feedback for Correct Answer 2] This researcher is using participants to help identify other participants. [Feedback for Answer Choice 3] 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This does not describe intensity sampling. [Feedback for Answer Choice 4] This does not describe purposive sampling. [Q4] For her qualitative study, Carol wants to use criterion sampling to select participants for her qualitative study about high-achieving students’ decision process for selecting a college. Which of the following participants will she likely use? 1. High school students whose average grade point average is above 3.5 or below 1.5 2. High school seniors who scored at least 1400 on the SAT [correct] 3. Seniors at one magnet school planning to go to the state college 4. High school seniors who are going to college and are identified by individuals already participating in the study [Feedback for Answer Choice 1] This sample would not meet the criteria of participants being high-achieving students. [Feedback for Correct Answer 2] The example is criterion sampling because participants all meet a set standard (obtained a certain test score). [Feedback for Answer Choice 3] This sample would not necessarily indicate that all of these students are high performers. [Feedback for Answer Choice 4] This would describe a snowball sampling procedure. Learning Outcome 5.4: Explain how factors related to sampling can influence, even bias, findings; know and apply the criteria for evaluating sample and participant sections of research articles and reports. [Q1] Jenn has conducted a study focused on the perspectives of teachers on differentiated instruction. She has selected her sample from a Facebook group of teachers who share differentiated instruction ideas. Which of the following is likely a problem with her study? 1. Sample bias [correct] 2. Sampling technique 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. Faulty research focus 4. Lack of rigor [Feedback for Correct Answer 1] Jenn has likely selected a sample that will have only one viewpoint, a pro differentiated instruction viewpoint. Therefore, her study will suffer from a sample bias. [Feedback for Answer Choice 2] There is no issue with the way Jenn has sampled. [Feedback for Answer Choice 3] Jenn’s research focus is a good one. [Feedback for Answer Choice 4] While we don’t know much about Jenn’s study, we have no reason to believe it is not rigorous. [Q2] Howard conducted a survey of teachers and learned about their preferences for virtual instruction versus face to face instruction. In his manuscript he wrote the following, “The survey was sent to a total of 483 teachers and received an overwhelming response.” Which error has Howard made? 1. Howard has failed to clearly report his response rate. [correct] 2. Howard should not have used a survey for this question. 3. Howard sent the survey to too many teachers. 4. Howard sent the survey to too few teachers. [Feedback for Correct Answer 1] When conducting survey research, it is imperative that the response rate is clearly outlined. [Feedback for Answer Choice 2] A survey is an excellent tool to answer this research question. [Feedback for Answer Choice 3] Howard’s sample is not too large. [Feedback for Answer Choice 4] 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Howard’s sample is not too small. [Q3] Justin conducted a survey focused on teachers’ perspectives on summer learning loss. In his manuscript, he wrote, “Teachers were surveyed and asked to rate several learning loss issues on a scale of 1 to 5.” Which error has Justin made? 1. Justin has not made any errors. 2. Justin should not have used a survey for this topic. 3. Justin has used improper measurement metrics. 4. Justin has failed to clearly define his sample. [correct] [Feedback for Answer Choice 1] Justin has made an error in his reporting. [Feedback for Answer Choice 2] A survey is an excellent tool to better understand this issue. [Feedback for Answer Choice 3] While we don’t know much about Justin’s survey, no problematic components are outlined here. [Feedback for Correct Answer 4] When reporting research, it is imperative that the sample is clearly defined. Justin is much too vague. [Q4] Mr. Rogan has conducted a survey intent on outlining the perspectives of special education teachers in relation to the amount of paperwork they must do after hours. He surveyed the four special educators in his school. He is now preparing a manuscript outlining the perspectives of South Carolina SPED teachers on special education paperwork. What mistake has Mr. Rogan made? 1. Mr. Rogan’s sample size is too small. [correct] 2. Mr. Rogan has not clarified who he surveyed. 3. This topic is not one of interest to readers. 4. Mr. Rogan’s topic is too controversial. [Feedback for Correct Answer 1] 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

While Mr. Rogan is seeking to describe the perspectives of a large group, he has a very limited population and cannot make those conclusions. [Feedback for Answer Choice 2] Mr. Rogan was clear on who he surveyed. [Feedback for Answer Choice 3] This topic is likely of great interest, regardless, there is nothing wrong with his topic. [Feedback for Answer Choice 4] This topic is not controversial, but even if it was, this is not an issue.

Chapter 5 Application Exercises Chapter 5: Participants and Sampling for Quantitative Designs Application Exercise 5.1: Distinguishing Sampling Techniques Learning Outcome 5.2: Contrast random with nonrandom sampling procedures. [Q1] For the following situation, identify the procedure used for selecting the sample. Explain how you came to your decision. There are 100 kindergarten classes in the school system, with an average of 20 children per class. You want a sample of 200 children. You give each class a number from 001 to 100, then use a table of random numbers to select 10 classes. You then give each student in each class a number (from 1 to the total number in that class) and use the table of random numbers to select 10 children from each class. [Q1 Model Response] This is cluster sampling, which involves the random selection of naturally occurring groups or units (clusters) and then individuals from the chosen groups are used for the study. Each class comprises a logical cluster. [Q2] For the following situation, identify the procedure used for selecting the sample. Explain how you came to your decision. There are 220 attorneys in the local bar association, and you want a sample of 40 participants. You number each of the attorneys, and then use a table of random numbers to select a sample from the population of 220 attorneys. [Q2 Model Response] This is random sampling, also known as probability sampling. All of the members of the bar association are known, and each attorney has an equal chance of being selected. [Q3] For the following situation, identify the procedure used for selecting the sample. Explain how you came to your decision. There are 2400 registered voters in the county, and there are approximately twice as many Republicans as Democrats. You want a sample of voters, and you want to ensure proportional representation. Your desired sample size is 450. You use a table of random numbers to select 300 of the Republican voters and then use the table to select 150 of the Democratic voters. 12 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q3 Model Response] This is proportional stratified sampling, which is appropriate when various strata of interest are different in size. The population here is 2400 registered voters. You want 450 in your sample, or 18.75% of the population. Given the population, 2/3 of your sample should be Republicans (2/3 of 450 is 300) and 1/3 should be Democrats (1/3 of 450 is 150). You would give each of the Republican voters in the population a number (from 1 to 1600) and then use the random number table to select 300. You’d then give each of the remaining 800 voters (Democrats) a number from 1 to 800 and randomly select 150. Application Exercise 5.2: Distinguishing Sampling Techniques Learning Outcome 5.1: Describe the differences between samples and populations, and how sample statistics are used to approximate what is true for the population. [Q1] Consider the following scenario: Dr. Marcum is a researcher interested in studying the impacts of Positive Behavioral Interventions and Supports on the morale of high school seniors. He has selected two homeroom classes at a high school located in close proximity to his university and will survey those students. In this scenario, is Dr. Marcum working with a sample or a population? Justify your answer. [Q1 Model Response] In this scenario, Dr. Marcum is working with a sample. Recall that a sample is a subset of a larger population. Dr. Marcum is only working with two groups of students in one local high school; therefore, this is a sample. [Q2] Consider the following scenario: Dr. Jarod is interested in studying how courses in diversity and equity influence the teaching practices of beginning teachers. Although there will be many variables that will hinder this, Dr. Jarod would like to get a survey to all teachers in the United States who are in their first 3 years of teaching. If this were possible, would this be considered a sample or a population? Justify your answer. [Q2 Model Response] If this were possible, this would be a population. Dr. Jarod hopes to generalize her findings to all beginning teachers in the United States. However, this is not going to be a 13 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

realistic study. Instead, Dr. Jarod will have to focus on a smaller sample of beginning teachers and hope to diversify enough to reach some level of generalization. Application Exercise 5.3: Strengths and Weaknesses of Sampling Techniques Learning Outcome 5.4: Explain how factors related to sampling can influence, even bias, findings; know and apply the criteria for evaluating sample and participant sections of research articles and reports. [Q1] Consider the following: Steven is an undergraduate researcher interested in conducting research focused on social and emotional learning programs in middle schools. However, as an undergraduate researcher, he does not have much time or access to research populations. He does, however, have connections at a local middle school who has invited him to conduct his study. Steven is okay with a convenience sample, given his situation. Review Table 5.3 entitled Strengths and Weaknesses of Quantitative Sampling Methods to answer the following question. What are the advantages and disadvantages to Steven’s sampling approach? [Q1 Model Response] There are several advantages to Steven’s approach. First, this is a low-cost approach, which is great for an undergraduate! Second, this process is less time consuming, is easy to administer, usually ensures high participation rates, and still has the potential for generalization to similar samples. However, there are several drawbacks. In reality, it will be hard to generalize his findings, his sample is not representative of an individual population, and his results depend on unique characteristics of his particular sample. [Q2] Consider the following: Darla is a doctoral student and researcher interested in conducting research focused on the role of the school nurse in low-income communities. She has identified a sample of 58 school nurses in low-income districts in the state of South Carolina. She plans to survey this purposeful sample to learn more about the ways in which their roles as school nurses are actualized. Review Table 5.3 entitled Strengths and Weaknesses of Quantitative Sampling Methods to answer the following question. What are the advantages and disadvantages to Darla’s sampling approach?

[Q2 Model Response] There are several advantages to Darla’s approach. First, Darla’s approach is convenient, and she is likely to receive the information she is looking for since she has so tightly controlled the parameters of study participation. However, her convenience comes at a price. Her sample is nonrandom and may not be easily generalized. [Q3] Consider the following: Dr. Bayne is a researcher interested in conducting research focused on the impact of boating safety courses on the behaviors of new boat owners. He was able to obtain a list of all boat safety course completers who were also first-time boat owners in Charleston, South Carolina, in 2020. Of the total population of 100, he sent a survey to a random sample of 20 of those new boat owners. Review Table 5.3 entitled Strengths and Weaknesses of Quantitative Sampling Methods to answer the following question. What are the advantages and disadvantages to Dr. Bayne’s sampling approach? [Q3 Model Response] There are several advantages to Dr. Bayne’s approach. First, his approach is likely quite representative of the population of interest. Furthermore, his data will be easy to analyze and interpret. Potential weaknesses include the task of numbering each participant, though with a population of 100 this was a reasonable task. The largest drawback is the increased sampling error of simple random sampling, which could be diminished by a stratified sampling approach.

Chapter 5 Test Items SELF-ASSESSMENT ITEMS 1) Which of the following represents the largest group? a) Target population b) Sampling frame c) Sample d) Participants 2) A researcher randomly selected 100 junior high students from a large school district to take part in a study of the effects of television violence on the social behavior of children age 11–13 years. The students selected to be in the study are considered which of the following? a) The population b) The sampling frame c) The sample d) The target population 3) A researcher is doing a study involving college freshmen at a large state university. Which of the following sampling techniques could she use in order to select a sample of freshmen that adequately represents the entire population? a) Convenience sampling b) Random sampling c) Nonprobability sampling d) Purposeful sampling 4) Why is it important to have a complete description of the sample when publishing results? a) It allows the sample to be stratified using a nonrandom sampling technique. b) It describes the participants so they can be identified. c) It informs the reader to whom the results can be generalized. d) It allows the researcher to select informative participants. 5) A researcher is conducting a study on students at his university. Which sampling procedure should he use to ensure every member of the sampling frame has an equal and independent chance of being selected? a) Cluster sampling b) Simple random sampling 16 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) Stratified sampling d) Nonrandom sampling 6) A research team from a local university conducted a study involving five high schools. The total population was about 6,000 students, so the principal investigator decided to use the systematic sampling procedure to select the sample for the study. Which of the following is NOT a step in this procedure? a) Determine the sample size b) Determine the sampling interval c) Stratify the group according to ability d) Randomly select a name from the list then every nth name 7) Which sampling procedure was used in the study described below? The participants in this study were 12 special education students in the school whose fall reading level was at least three years below the expected level. Of the 12 students, 5 were boys and 7 were girls. Their chronological ages ranged from 13.7 to 15.9 years. a) Cluster sampling b) Simple random sampling c) Systematic random sampling d) Purposeful sampling 8) What is the major concern with the use of a nonrandom sampling procedure rather than a random sampling procedure? a) The sample size will not be large enough. b) The participants will lack motivation. c) The results will not be as generalizable. d) The participants will not have the knowledge they need. 9) What is likely to be true about participants in a study who were selected using the purposive sampling procedure? a) They were selected because they were accessible and readily available to the researcher. b) They were selected on a random basis, so they adequately represent the population. c) They were selected because they were knowledgeable about the topic being investigated. d) They were selected because they know little about the area of investigation. 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

10) Which type of sampling is being used when a qualitative researcher interviews three students who are national merit finalists? a) Typical case sampling b) Maximum variation sampling c) Stratified random sampling d) Extreme case sampling 11) Which of the following is the qualitative sampling procedure that provides a wide range of perspectives on an issue? a) Critical case sampling b) Snowball sampling c) Opportunistic sampling d) Maximum variation sampling 12) A qualitative researcher wanted to interview parents in the community who attended the Welcome Back coffee on the first day of school. There was no list of names to indicate who attended the event. Which of the following sampling procedures should she have used? a) Cluster sampling b) Snowball sampling c) Proportional sampling d) Probability sampling 13) Based on data from questionnaires that were returned by one-fourth of the teachers in the county, a researcher concluded that 90% of teachers did not favor incorporating computers into the classroom curriculum. Is the conclusion valid? a) Yes, because the questionnaires were sent to all teachers. b) No, because the questionnaire is unreliable as a method of data collection. c) Yes, because 90% of the teachers is a large percentage of the teachers in the county. d) No, because the results were drawn from a potentially biased sample.

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Ms. Manley did a small experiment with first graders in her school. she wanted to be able to generalize her results to all first graders at the school, so she randomly selected 30 students. Which of the following terms is used to refer to the 30 students selected for the study? a) The sampling frame b) The target population c) The sample d) The population 2) A researcher divided the target population into homogenous subgroups then randomly selected 25 students from each group. Which sampling procedure did the researcher use? a) Simple random sampling b) Stratified random sampling c) Cluster random sampling d) Maximum variation sampling 3) Which type of sampling would likely result in the most representative sample? a) Simple random sampling b) Cluster sampling c) Purposeful sampling d) Maximum variation sampling 4) A researcher for the local newspaper conducted a survey in her town. She decided to use the systematic sampling procedure to select a sample of the adult population to participate in her survey. She selected the first name on the list and then every 250th name. Did she use the procedure correctly? Why or why not? a) No, because she did not begin with a randomly selected number. b) Yes, because she selected from a list of adults in her town. c) No, because she excluded people who were unemployed. d) Yes, because she selected every nth person in the population. 5) A political scientist wanted to compare the voting patterns of minority groups in large cities. Which sampling procedure selects the number of people from each minority group based on the percentage of people of that group in the population? a) Cluster sampling 19 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Proportional stratified sampling c) Extreme case sampling d) Simple random sampling 6) There are fewer teachers certified in content areas like art, foreign languages, and health when compared to content areas like math, science, or social studies. Which sampling method will ensure the two groups are equally represented in a statewide survey of teachers’ opinions? a) Random sampling b) Disproportional stratified sampling c) Cluster sampling d) Purposeful sampling 7) Which of the following is true of cluster sampling? a) It is the most representative of a population. b) It allows subgroups to be equally represented. c) It allows the researcher to randomly assign intact groups. d) It is the type of random sampling procedure most commonly used. 8) The principal of a school wanted to measure the level of parent satisfaction regarding a new policy. She created a survey to hand out to parents attending Backto-School Night. Which of the following sampling procedures did she use to select the parents to complete her survey? a) Purposeful sampling b) Convenience sampling c) Cluster sampling d) Typical case sampling 9) Forty high school seniors were selected to participate in a study. All had been rated by their art teachers as having exceptional artistic ability and all had won art awards. Of the total, 25 were girls and 15 were boys. Which sampling procedure was used in this study? a) Convenience sampling b) Extreme case sampling c) Stratified sampling d) Snowball sampling 10) Which of the following is the type of purposeful sampling that uses networking? a) Critical case sampling 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Maximum variation sampling c) Snowball sampling d) Typical case sampling 11) A qualitative researcher selected three primary-grade teachers based on researcherspecified characteristics from each elementary school in a medium-sized city. Which type of sampling procedure did she likely use? a) Critical case sampling b) Criterion sampling c) Opportunistic sampling d) Cluster sampling 12) A researcher mailed surveys to the first-, second-, and third-grade teachers in a large school district, of which 15% were returned. Based on the results, the researcher concluded that 80% of primary teachers were opposed to requiring statewide achievement tests at their grade level. Is the researcher’s conclusion correct? a) No, because the questionnaires are an unreliable and invalid method of measurement. b) Yes, because the questionnaire was sent to all of the teachers in the district from all three primary grades. c) No, because the results were drawn from a potentially biased sample. d) Yes, because all teachers had an equal probability of responding to the questionnaire. Short-Answer Questions 1) Why should a researcher be wary of results of an analysis using a convenience sample? 2) What are two disadvantages to using simple random sampling to select participants for a study? Essay Question Describe each of the following types of bias and, for each type, explain how the bias might affect the results of a study: • Participant motivation • Sample size • Sampling bias • Volunteer sample 21 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Chapter 5 Test Answer Key ANSWER KEY: Chapter 5 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. c 2. c 3. d 4. a 5. a 6. c 7. b 8. c 9. c 10. b 11. a 12. a 13. c 14. c 15. d 16. c 17. c

INSTRUCTOR’S EXAM ITEMS 1. c 2. b 3. a 4. a 5. b 6. b 7. c 8. b 9. b 10. c 11. b 12. c

Chapter 6 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 6: Foundations of Educational Measurement ............................................... 1 Learning Outcome Quizzes........................................................................... 1 Application Exercises .................................................................................. 12 Test Items ................................................................................................... 19 Test Answer Key ......................................................................................... 27

Chapter 6: Foundations of Educational Measurement Chapter 6 Learning Outcome Quizzes Learning Outcome 6.1: Describe the nature of measurement and how measures are used in quantitative research. [Q1] The scale of measurement that involves rank-ordering data is known as which of the following? 1. Ordinal scale [correct] 2. Nominal scale 3. Interval scale 4. Ratio scale [Feedback for Correct Answer 1] Ordinal scales involve rank ordering the data. Group 1, for example, may indicate the highest level of achievement. [Feedback for Answer Choice 2] A nominal scale is one in which there are mutually exclusive categories, without any order or relationship implied. [Feedback for Answer Choice 3] An interval scale is ordinal and has equal intervals between categories or scores. [Feedback for Answer Choice 4] A ratio scale is one in which ratios can be used in comparing and interpreting the scores. [Q2] A scale of measurement consisting of equal intervals and has a true zero point is known as which of the following? 1. Interval scale 2. Ordinal scale 3. Ratio scale [correct] 4. Nominal scale [Feedback for Answer Choice 1] 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

An interval scale is ordinal and has equal intervals between categories or scores. [Feedback for Answer Choice 2] Ordinal scales involve rank ordering the data. Group 1, for example, may indicate the highest level of achievement. [Feedback for Correct Answer 3] A ratio scale of measurement has equal intervals between values and a true zero point. [Feedback for Answer Choice 4] A nominal scale is one in which there are mutually exclusive categories, without any order or relationship implied. [Q3] Which of the following terms refers to one in which there are mutually exclusive categories, without any order or relationship implied? 1. Nominal scale [correct] 2. Interval scale 3. Ordinal scale 4. Ratio scale [Feedback for Correct Answer 1] A nominal scale is one in which there are mutually exclusive categories, without any order or relationship implied. [Feedback for Answer Choice 2] An interval scale is ordinal and has equal intervals between categories or scores. [Feedback for Answer Choice 3] Ordinal scales involve rank ordering the data. Group 1, for example, may indicate the highest level of achievement. [Feedback for Answer Choice 4] A ratio scale of measurement has equal intervals between values and a true zero point. [Q4] Which of the following terms refers to a scale of measurement that is rank ordered and has equal intervals between categories or scores? 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

1. Interval scale [correct] 2. Nominal scale 3. Ordinal scale 4. Ratio scale [Feedback for Correct Answer 1] An interval scale is ordinal and has equal intervals between categories or scores. The characteristic of equal intervals allows us to compare directly one score to another in terms of the amount of difference. [Feedback for Answer Choice 2] A nominal scale is one in which there are mutually exclusive categories, without any order or relationship implied. [Feedback for Answer Choice 3] Ordinal scales involve rank ordering the data. Group 1, for example, may indicate the highest level of achievement. [Feedback for Answer Choice 4] A ratio scale of measurement has equal intervals between values and a true zero point. Learning Outcome 6.2: Understand fundamental descriptive statistics that are used in quantitative research, including frequency distributions, measures of central tendency and variability, and correlation. [Q1] A researcher’s choice of statistical procedures depends to some degree on the extent to which they reflect a normal distribution. Arlene has graphed her data on a normal distribution scale, and has found that the peak is not in the center of the curve but rather lies to the left of the midpoint. The distribution of Arlene’s data is which of the following? 1. A normal curve 2. Platykurtic 3. Positively skewed [correct] 4. Negatively skewed [Feedback for Answer Choice 1] This is not indicative of a theoretically normal curve. [Feedback for Answer Choice 2] 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Platykurtic refers to a measure of kurtosis and therefore does not apply to this situation. [Feedback for Correct Answer 3] When a lot of data lies to the left of the midpoint, the graph is positively skewed, whereas when a lot of data lies to the right of the midpoint, the graph is negatively skewed. When a graph of data is unusually pointy or flat, the graph shows kurtosis. When a graph of the data shows any of these sorts of characteristics, we would not describe it as a normal curve. [Feedback for Answer Choice 4] Negatively skewed distributions do not have curves to the left of the midpoint. [Q2] Becky asks the teachers at her school to identify their favorite snack for in-service workshops from a list of 10 choices. The most popular response was "cheese and crackers." What statistic did Becky report? 1. Mean 2. Mode [correct] 3. Median 4. Range [Feedback for Answer Choice 1] The mean is the average of a group of data points. [Feedback for Correct Answer 2] The mode is the number or score that occurs most frequently in a distribution of data. [Feedback for Answer Choice 3] The median refers to the midpoint of a group of data. [Feedback for Answer Choice 4] The range refers to the spread of the data, listed from least to greatest. [Q3] Kelly tested her students' achievement in French class. The test had a maximum score of 100. The lowest score was a 45 and the highest score achieved was a 95. Which of the following is true for Kelly’s data? 1. The data have a variance of 25. 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. The data have a range of 50. [correct] 3. The data have a median of 70. 4. The data have a standard deviation of 4. [Feedback for Answer Choice 1] We do not have the necessary information to calculate the variance. [Feedback for Correct Answer 2] The range is simply the difference between the lowest score (45 in this case) and highest score (95 in this case) in a dataset. [Feedback for Answer Choice 3] We do not have enough information to find the median. [Feedback for Answer Choice 4] We do not have enough data to calculate the standard deviation of this dataset. [Q4] What percentage of the normal distribution falls between −1 SD and +1 SD? 1. 99% 2. 95% 3. 34% 4. 68% [correct] [Feedback for Answer Choice 1] This is not the correct percentage. [Feedback for Answer Choice 2] This is not the correct percentage. [Feedback for Answer Choice 3] This is not the correct percentage. [Feedback for Correct Answer 4] In a normal distribution, 34% of the scores are one standard deviation above the mean, and 34% of the scores are one standard deviation below the mean. [Q5] 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Which correlation coefficient indicates the strongest relationship? 1. −.80 [correct] 2. −.20 3. +.50 4. +.75 [Feedback for Correct Answer 1] The strength of the correlation is related to its numeric value, regardless of sign. Correlations close to 1.0 or −1.0 are strongest. [Feedback for Answer Choice 2] This is not the correct answer. [Feedback for Answer Choice 3] This is not the correct answer. [Feedback for Answer Choice 4] This is not the correct answer. Learning Outcome 6.3: Explain the meaning of measurement validity, sources of evidence to support validity arguments, and the effect of validity on research. [Q1] Which of the following terms best describes what is indicated by measures of validity? 1. The spread of scores [correct] 2. The average score 3. The central tendency of scores 4. The relationship between scores [Feedback for Correct Answer 1] The variability measures are indices of dispersion, indicating how spread out a group of scores is. [Feedback for Answer Choice 2] The mean is the statistic used to report the average score. The mean is not a measure of validity. [Feedback for Answer Choice 3] 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Measures of central tendency help to describe data, but not in regard to validity. [Feedback for Answer Choice 4] Relationships are measured through correlation, not validity. [Q2] In planning a research study, Sherry is questioning whether a particular instrument predicts future performance very well. Thus, she is concerned with which of the following? 1. Concurrent criterion-related evidence 2. Evidence based on convergent relationships 3. Evidence based on internal structure 4. Predictive criterion-related evidence [correct] [Feedback for Answer Choice 1] Concurrent criterion-related evidence is established by correlating two measures of the same trait that are given to the same individuals at about the same time. [Feedback for Answer Choice 2] When scores from one instrument correlate highly with scores from another measure of the same trait, we have what is called convergent evidence. However, that is not what is occurring in this example. [Feedback for Answer Choice 3] Evidence based on internal structure is provided when the relationships between items and parts of the instrument are empirically consistent with the theory or intended use of the scores. [Feedback for Correct Answer 4] Predictive criterion-related evidence pertains to how well an earlier measure can predict the criterion behavior or performance on a later measure. [Q3] Sandia is using a measurement instrument for her study—a questionnaire—that she expects will provide valid evidence of her participants’ level of depression. What type of evidence is this? 1. Evidence based on content [correct] 2. Evidence based on internal structure 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. Evidence based on discriminate relationships 4. Evidence based on predictive relationships [Feedback for Correct Answer 1] Evidence based on content is evidence indicating the extent to which the sample of items or questions in the instrument is representative of some domain, concept, or hypothetical construct—in this case, depression. [Feedback for Answer Choice 2] Evidence based on internal structure is provided when the relationships between items and parts of the instrument are empirically consistent with the theory or intended use of the scores. [Feedback for Answer Choice 3] In Sandia’s study, we are provided no evidence regarding discriminate relationships. [Feedback for Answer Choice 4] Sandia’s study does not include predictive measures. [Q4] Which of the following terms refers to an overall evaluation of the extent to which theory and empirical evidence support interpretations that are implied in given uses of the scores? 1. Reliability 2. Validity [correct] 3. Measures of central tendency 4. Measures of dispersion [Feedback for Answer Choice 1] Reliability refers to the extent to which participant, rater, and observer scores are free from error. [Feedback for Correct Answer 2] Validity refers to an overall evaluation of the extent to which theory and empirical evidence support interpretations that are implied in given uses of the scores. [Feedback for Answer Choice 3] Measures of central tendency help to describe data, but this is not the correct answer. 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 4] These measures will help you describe the spread of a dataset, but this is not the correct answer. Learning Outcome 6.4: Explain the meaning of measurement reliability, sources of evidence to estimate reliability, and the effect of reliability on research. [Q1] Which of the following terms best characterizes the most appropriate way to evaluate the reliability of observational data? 1. Interrater reliability [correct] 2. Stability reliability 3. Equivalence 4. Internal consistency [Feedback for Correct Answer 1] With observations, it is important to establish that different observers collect and score data in the same way. [Feedback for Answer Choice 2] A stability estimate of reliability is obtained by administering one measure to one group of individuals, waiting a specified period of time, and then readministering the same instrument to the same group. [Feedback for Answer Choice 3] A measure of equivalence, or what is sometimes called alternate-forms reliability, is obtained by administering two forms of the same measure to one group of individuals and then correlating the scores from the two forms. [Feedback for Answer Choice 4] Internal consistency, easily the most widely used estimate of reliability, indicates the degree to which individuals’ answers to items measuring the same trait are consistent. [Q2] Which of the following terms refers to the extent to which participant, rater, and observer scores are free from error? 1. Validity 2. Reliability [correct] 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. Measures of central tendency 4. Accuracy [Feedback for Answer Choice 1] Validity, in the context of measurement, is an overall evaluation of the extent to which theory and empirical evidence support interpretations that are implied in given uses of the scores. [Feedback for Correct Answer 2] Measurement reliability, or, more typically, simply reliability (or reliability/precision), is the extent to which participant, rater, and observer scores are free from error. [Feedback for Answer Choice 3] Measures of central tendency help to describe a set of data, but this is not the correct answer. [Feedback for Answer Choice 4] While this term seems correct in everyday language, this is not the best term for this question. [Q3] Which of the following is obtained by administering two forms of the same measure to one group of individuals and then correlating the scores from the two forms? 1. Validity 2. Equivalency [correct] 3. Reliability 4. Correlation [Feedback for Answer Choice 1] Validity, in the context of measurement, is an overall evaluation of the extent to which theory and empirical evidence support interpretations that are implied in given uses of the scores. [Feedback for Correct Answer 2] A measure of equivalence, or what is sometimes called alternate-forms reliability, is obtained by administering two forms of the same measure to one group of individuals and then correlating the scores from the two forms. [Feedback for Answer Choice 3] 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Measurement reliability, or, more typically, simply reliability (or reliability/precision), is the extent to which participant, rater, and observer scores are free from error. This answer choice is much too broad. [Feedback for Answer Choice 4] Correlation may refer to how similar two variables are, but this is not the best answer choice for this question. [Q4] Which of the following forms of reliability is being established when the same group of people is administered the same measure over a period of time? 1. Agreement 2. Equivalence 3. Internal consistency 4. Stability [correct] [Feedback for Answer Choice 1] Agreement refers to the extent to which different raters’ or observers’ answers agree. [Feedback for Answer Choice 2] Equivalence is established when two different forms of the same measure are given to the same group of individuals at about the same time. [Feedback for Answer Choice 3] With internal consistency, a single form of the measure is given to the individuals once. [Feedback for Correct Answer 4] To determine stability, the same measure is given to the same group of individuals over time.

Chapter 6 Application Exercises Chapter 6: Foundations of Educational Measurement Application Exercise 6.1: Understanding Measurement Learning Outcome 6.2: Understand fundamental descriptive statistics that are used in quantitative research, including frequency distributions, measures of central tendency and variability, and correlation. [Q1] Rachelle conducted an educational research study, part of which was to address the problem of test anxiety. In particular, she asked whether giving regular mini-quizzes during the semester helped to reduce test anxiety for students. She assessed test anxiety three times—before the first chapter test, right after that test, and then before the final exam—by asking students, “How would you rate your test anxiety?” Consider her graph of the results:

Test Anxiety

Percent of Students

35 30 25

Test anxiety, before first test

Test anxiety, after first test

Test anxiety, now

10 5 0 No anxiety

Mild anxiety

Some anxiety

High anxiety

Very high anxiety

Student Anxiety

What type of measurement scale did Rachelle use to measure students’ test anxiety? [Q1 Model Response] The scale is represented here on the X axis of the graph, from No anxiety to Very high anxiety. This is an ordinal scale. Each category can be compared to the others in terms of less than or greater than, but the degree of difference between the categories is not specified. Even if she numbered the scale and told students to report “on a scale of 1 to 12 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

5,” we cannot tell if the difference between No anxiety and Mild anxiety is of the same magnitude as the difference between High anxiety and Very high anxiety. [Q2] Rachelle conducted an educational research study, part of which was to address the problem of test anxiety. In particular, she asked whether giving regular mini-quizzes during the semester helped to reduce test anxiety for students. She assessed test anxiety three times—before the first chapter test, right after that test, and then before the final exam—by asking students, “How would you rate your test anxiety?” Consider her graph of the results:

Test Anxiety

Percent of Students

35 30 25

Test anxiety, before first test

Test anxiety, after first test

Test anxiety, now

10 5 0 No anxiety

Mild anxiety

Some anxiety

High anxiety

Very high anxiety

Student Anxiety

What kind of graph is this? [Q2 Model Response] It is a frequency graph—it shows the percentage of students who gave each possible response to the question, “How would you rate your test anxiety?” The type of frequency graph is known as a histogram—a two-dimensional graph that uses vertical columns to show the frequency of each score. [Q3] Rachelle conducted an educational research study, part of which was to address the problem of test anxiety. In particular, she asked whether giving regular mini-quizzes during the semester helped to reduce test anxiety for students. She assessed test anxiety three times—before the first chapter test, right after that test, and then before 13 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

the final exam—by asking students, “How would you rate your test anxiety?” Consider her graph of the results:

Test Anxiety

Percent of Students

35 30 25

Test anxiety, before first test

Test anxiety, after first test

Test anxiety, now

10 5 0 No anxiety

Mild anxiety

Some anxiety

High anxiety

Very high anxiety

Student Anxiety

Given this frequency distribution, is it possible to determine the “median score for student test anxiety at the end of the term”? If it’s possible, explain how you would find the score(s); if it’s not possible, explain why not. [Q3 Model Response] It’s possible. The median is the middle score of the distribution, the midpoint that divides a rank-ordered distribution into halves containing an equal number of scores. This scale is rank-ordered (from No anxiety to Very high anxiety), so you could find the median score—the score at which 50% are above and 50% are below. If you look at the yellow bars, you’ll see that more than 25% of the students reported no anxiety, and more than 30% reported mild anxiety. Thus, the 50% midpoint, or the median, is at Mild anxiety. Application Exercise 6.2: Calculating Descriptive Statistics Learning Outcome 6.2: Understand fundamental descriptive statistics that are used in quantitative research, including frequency distributions, measures of central tendency and variability, and correlation. [Q1] The following dataset represents test scores, out of 100 points, for 10 students. Score 14 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

85 75 62 90 78 83 95 71 68 71 Calculate the mean for this dataset. [Q1 Model Response] The mean is 77.8. The mean is calculated as X = å X / n For this formula, sum the scores and divide the sum by the number of scores.

(85 + 75 + 62 + 90 + 78 + 83 + 95 + 71+ 68 + 71) / 10 = 77.8 [Q2] The following dataset represents test scores, out of 100 points, for 10 students. Score 85 75 62 90 78 83 95 15 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

71 68 71 Calculate the median and mode for this dataset. [Q2 Model Response] The median is 76.5. This value represents the midpoint by which half of the values are higher and half are lower. If we sort the values, we can find the middle score. In this dataset, the median is the halfway point between the two middle scores (i.e., halfway point between 75 and 78). The mode is 71. The mode is the most frequently occurring score. [Q3] The following dataset represents test scores, out of 100 points, for 10 students who were randomly selected from a larger sample of 100 students. Score 85 75 62 90 78 83 95 71 68 71 Calculate the standard deviation for this dataset. [Q3 Model Response] The standard deviation is 10.37. 16 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This is calculated by subtracting the mean from each individual data point, summing those values, squaring the result, dividing by n − 1 in a sample or n in a population, and then taking the square root of that value. 2

(x - x) We are working with a sample, so the formula looks like this: s = å n -1

Before taking the square root, that value is simply the variance. The variance is the standard deviation squared, so the standard deviation is the square root of the variance. Your variance in this instance is 107.73 The standard deviation is the square root of the variance. The square root of 107.73 is 10.37. Note: if you use a computer program such as SPSS, R, or Excel to calculate the variance or standard deviation, your answer may vary based on the program. Some programs default to using n − 1 in the denominator, as is appropriate for a sample. When calculating the standard deviation of a population, n is used instead of n − 1. Application Exercise 6.3: Types of Reliability Learning Outcome 6.4: Explain the meaning of measurement reliability, sources of evidence to estimate reliability, and the effect of reliability on research. [Q1] Consider the following: Jane is a researcher interested in student engagement in science class. She has developed a measure that considers the engagement of seventh-grade science students following science lectures. She has given this measure to students during the first week of the semester and again at midterm. Review Table 6.8 entitled, Types of Reliability to answer the following question. Which type of reliability is explained in this example? Explain your answer. [Q1 Model Response] Jane is testing for stability. This is evident by the fact that she has given the same measure to the same group of individuals over time. [Q2] Consider the following: Jane is a researcher interested in student engagement in science class. She has developed a measure that considers the engagement of seventh-grade science 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

students following science lectures. She has given one form of this measure to a group of students on Monday and another form to the same group of students on Wednesday. Review Table 6.8 entitled Types of Reliability to answer the following question. Which type of reliability is explained in this example? Explain your answer. [Q2 Model Response] Jane is testing for equivalence. This is evident by the fact that she has given the two different forms of the same measure to the same students at about the same time. [Q3] Consider the following: Jane is a researcher interested in student engagement in science class. She has developed a measure that considers the engagement of seventh-grade science students following science lectures. She has given the measure to a group of students and calculated a Cronbach’s Alpha of .92 for the overall measure. Review Table 6.8 entitled Types of Reliability to answer the following question. Which type of reliability is explained in this example? Explain your answer. [Q3 Model Response] This process is describing an internal consistency approach. Jane has given the measure to one group, then considered how similar or different the answers were around the construct of student engagement. Her results indicated a high level of consistency among responses.

Chapter 6 Test Items SELF-ASSESSMENT ITEMS 1) Which of the following is a true statement about measures as they relate to educational research? a) Measures are not used by educational researchers because they lack validity. b) Measures refer only to the instruments that are used by researchers. c) Measures are items, techniques, or instruments used for measurement. d) Measures can only be instruments that provide objective, quantifiable data. 2) Which of the following is the type of measurement scale in which each category can be compared to others in terms of less than or greater than, but there is no indication of the magnitude of the difference? a) Nominal b) Interval c) Ordinal d) Ratio 3) An achievement test score represents which type of measurement scale? a) Nominal b) Ordinal c) Interval d) Ratio 4) Which of the following is NOT a characteristic of frequency distributions? a) They are used to explain isolated scores. b) They group data according to the number of times each score occurred. c) They indicate whether the distribution is approximately normal or skewed. d) They organize ungrouped data into grouped data. 5) The scores on a math test are normally distributed. If the mean is 50 and the standard deviation is 5, which is the range of 98% of the scores? a) 40–60 b) 45–55 c) 35–65 d) 25–75 19 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

6) Which of the following is used to describe a distribution with many high scores and very few low scores? a) Normal b) Positively skewed c) Negatively skewed d) Bimodal 7) Which of the following measures of central tendency can be influenced by extreme scores? a) Median b) Mean c) Mode d) Standard Deviation 8) Mrs. Rivera wanted to compare her students’ math scores on two tests, each of which had 40 items. She found that for Test A, the mean was 20 and the standard deviation was 4, and for Test B, the mean was also 20, but the standard deviation was 6. Which of the following statements is true? a) The distribution of scores for Test B is positively skewed. b) The scores for both tests are normally distributed. c) There is more variation in the distribution of the scores for Test B than in Test A. d) The standard error is the same for both tests because they both tests have 40 items. 9) Approximately what percentage of scores falls between ±1 standard deviation of the mean? a) 33% b) 50% c) 68% d) 97% 10) Which of the following describes the likely relationship between height and weight? a) High and negative b) High and positive c) Low and negative d) Low and positive 11) Which of the following correlation coefficients indicates the strongest relationship? 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

a) −.81 b) −.14 c) +.37 d) +.78 12) Which of the following is the term used for the degree of consistency with which a reading test measures reading ability? a) Reliability b) Measurability c) Predictability d) Validity 13) A research report stated that a sample of 50 students was given two forms of a test. Form A was administered during the first week of the new school year and Form B was administered three weeks later. The reliability coefficient between the two forms of the test was found to be .84. What estimate of reliability was used in this study? a) Interclass correlation b) Internal consistency c) Equivalence and stability d) Agreement estimate 14) Which of the following is NOT true of an internal consistency estimate of reliability? a) A reliability coefficient is obtained by dividing a test into equal halves. b) It is appropriate for surveys in which participants respond to items by giving a rating on a particular scale. c) It is typically measured by giving two different forms of a test at about the same time. d) It is used with tests that measure a single trait. 15) Which of the following statements about the relationship between validity and reliability is true? a) Reliability is more important that validity. b) If a test is valid, it can’t also be reliable. c) A test can be valid but not necessarily reliable. d) A test can be reliable but not necessarily valid. 16) A researcher is using an instrument to measure student achievement in math. There are 70 items on the test, 50 multiple-choice and 20 true-false. Which of the following reliability coefficients should be used? 21 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

a) Coefficient alpha b) ASPeCT-DD c) KR-20 d) Cronbach’s alpha 17) Two tests were given to students in the sixth grade. Test A contained 25 items and was given to a group of low-achieving sixth graders. Test B contained 50 items and was given to the other sixth-grade students in the school who were of mixed ability. Which test would most likely have a higher reliability coefficient and why? a) Test A, because the test was shorter and was administered to a homogeneous group of students. b) Test A, because it was given before Test B and might have influenced the scores on Test B. c) Test B, because the test was longer and was administered to a heterogeneous group of students. d) Test B, because the test was taken by a group of students who would perform better on the test than those who took Test A.

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Which of the following is an example of a variable that is measured on the ratio measurement scale? a) Classification according to race b) A test score c) Social class d) Grade level in school 2) Which of the following is an example of a variable that is measured on the nominal measurement scale? a) Classification according to race b) Grade level in school c) Percentage of students receiving Free or Reduced lunch d) A yardstick 3) Which scale of measurement is characterized by values that are ordered and evenly spaced, but have no true zero (e.g., temperature)? a) Nominal b) Ordinal c) Interval d) Ratio 4) What is one purpose of descriptive statistics? a) To organize and summarize data b) To assess students c) To measure performance d) To evaluate observations 5) Which of the following statements is NOT a characteristic of frequency distributions? a) They enable identification of clusters of scores. b) They identify isolated scores. c) They indicate the shape of a distribution. d) The scores are ranked according to their frequencies. 23 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

6) A teacher worked with her class in such an effective manner that all but a few of the students mastered the material at a very high level. Which of the following is likely to be true about the distribution of the exam scores? a) They are positively skewed. b) They are negatively skewed. c) They are normally distributed. d) They are positively normal. 7) Which of the following is a major concern related to using the mean as a measure of central tendency? a) It cannot be used as a statistic in most statistical analyses. b) It is susceptible to extreme scores. c) It has weak statistical properties. d) It is commonly used to represent the average of a set of scores. 8) The scores on a math test are normally distributed. If the mean is 72 and the standard deviation is 6, what is the range of 98% of the scores? a) 54–84 b) 54–90 c) 60–84 d) 66–84 9) Which of the following is NOT a true statement about 100 scores from a normal distribution? a) The mean, median, and mode are the same score. b) It characterizes many naturally occurring phenomena. c) Most of the scores are clustered in the middle of the distribution. d) About 98% of the scores are ± 1 standard deviation from the mean. 10) Which of the following is a measure of variability in a distribution of scores? a) Mean b) Standard deviation c) Bivariate correlation d) Coefficient alpha

11) A researcher found a strong negative correlation between artistic ability and reading scores for a group of first graders. Which of the following is true about the relationship? a) Children who had higher reading scores had less artistic ability. b) Children who had lower reading scores had less artistic ability. c) Children who had more artistic ability read more than the students with less artistic ability. d) Children who had more artistic ability had higher reading scores. 12) Which of the following correlation coefficients indicates the weakest relationship? a) +.01 b) −.94 c) +.73 d) −.28 13) Which of the following characterizes a test in which individuals who take the test obtain about the same score each time it is taken? a) It is free of error. b) It is reliable. c) It is valid. d) It is randomized. 14) In order to validate an instrument, a team of experts examined the items and judged how representative they were. Which type of validity evidence does this describe? a) Content b) Construct c) Reliability d) Predictive 15) Which of the following is an advantage of internal consistency estimates of reliability? a) They require only one administration of the test. b) They provide a good measure of stability over time. c) They provide a higher estimate than other reliability coefficients. d) They require only one observer of the behavior. 25 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

16) A researcher used a scale to measure students’ attitudes toward high school drug testing programs. Students respond to each item using a five (5) point Likert scale (i.e., strongly agree, agree, neutral, disagree, strongly disagree). Which reliability coefficient should be reported? a) Test-retest b) LS-5 c) KR-20 d) Cronbach’s alpha 17) Two tests were given to students in the eighth grade. Test A contained 75 items and was given to eighth-grade students who were of mixed abilities. Test B contained 20 items and was given to eighth-grade honors students. Which test is likely to have a higher reliability coefficient and why? a) Test A, because the test was longer and was administered to a heterogeneous group of participants. b) Test A, because it was given before Test B and might have influenced the scores on Test B. c) Test B, because the test was short and reliability would not have been affected by student fatigue. d) Test B, because the test was taken by a group of students who would perform well on the test. Short Answer Questions: 1) Why are descriptive statistics fundamental to valid quantitative research? 2) Give three examples of noise as it relates to measurement reliability in educational research? Essay Question: What does it mean when a researcher says an instrument is valid and reliable and why is it important that researchers report evidence of validity and reliability of the instruments used in their research? What is one way an instrument can be validated? What is one way the reliability of an instrument can be checked?

Chapter 6 Test Answer Key ANSWER KEY: Chapter 6 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. c 2. c 3. d 4. a 5. a 6. c 7. b 8. c 9. c 10. b 11. a 12. a 13. c 14. c 15. d 16. c 17. c

INSTRUCTOR’S EXAM ITEMS 1. b 2. a 3. c 4. a 5. d 6. b 7. b 8. c 9. d 10. b 11. a 12. a 13. b 14. b 15. a 16. d 17. a

Chapter 7 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents

Chapter 7: Quantitative Data Collection Techniques ................................................. 1 Learning Outcome Quizzes........................................................................... 1 Application Exercises .................................................................................. 12 Test Items ................................................................................................... 17 Test Answer Key ......................................................................................... 24

Chapter 7: Quantitative Data Collection Techniques Chapter 7 Learning Outcome Quizzes Learning Outcome 7.1: Understand how measurement sensitivity is affected by validity, reliability, and variability of observed scores, and how sensitivity impacts quantitative research. [Q1] Which of the following descriptors is used to describe a measure that can accurately identify and discriminate differences among individuals? 1. Highly sensitive [correct] 2. Valid 3. Reliable 4. Standardized [Feedback for Correct Answer 1] Sensitivity is the ability or capacity of a measure to identify and discriminate differences. Measures that are insensitive are much less likely to show relationships and differences or even to provide an adequate, accurate simple description. [Feedback for Answer Choice 2] Validity is an important aspect of a measurement’s characteristics, but validity does not refer to the identification of differences. [Feedback for Answer Choice 3] Reliability is an important aspect of a measurement’s characteristics, but reliability does not refer to the identification of differences. [Feedback for Answer Choice 4] Standardized assessments allow for the comparison of results across test-takers, but this is not the best answer. [Q2] Which of the following terms describes the ability of an assessment to measure what it is intended to measure? 1. Reliability 2. Validity [correct] 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. Sensitivity 4. Standardization [Feedback for Answer Choice 1] Reliability is important in an assessment, but this refers to the consistency of an assessment. [Feedback for Correct Answer 2] Measures deemed to be valid measure what they are intended to measure with accuracy. [Feedback for Answer Choice 3] Sensitivity is the ability or capacity of a measure to identify and discriminate differences. [Feedback for Answer Choice 4] Standardized assessments allow for the comparison of results across test-takers, but this is not the best answer. [Q3] Which of the following terms is most closely related to error in measurement? 1. Validity 2. Standardization 3. Sensitivity 4. Reliability [correct] [Feedback for Answer Choice 1] Measures deemed to be valid measure what they are intended to measure with accuracy, but this is not the best answer. [Feedback for Answer Choice 2] Standardized assessments allow for the comparison of results across test-takers, but this is not the best answer. [Feedback for Answer Choice 3] Sensitivity is the ability or capacity of a measure to identify and discriminate differences. [Feedback for Correct Answer 4]

Reliable assessments produce consistent results. If scores are influenced by error, then that means that less of the score is directly measuring the trait you have targeted. [Q4] Which of the following terms best describes the distribution of scores of an assessment? 1. Range [correct] 2. Mean 3. Median 4. Mode [Feedback for Correct Answer 1] You need a measure that will provide a good distribution of scores to show relationships and differences. If all your participants get the same score on the dependent variable, your study is doomed. This spread is referred to as the range of scores. [Feedback for Answer Choice 2] The mean of a group of data refers to the average. [Feedback for Answer Choice 3] The median of a group of data refers to the midpoint. [Feedback for Answer Choice 4] The mode of a group of data refers to the most frequent score in the dataset. Learning Outcome 7.2: Describe the characteristics of standardized tests, how they differ from locally developed tests, and how they are used in research. Know how to interpret scores reported from standardized tests. [Q1] Laura has received a copy of her son's standardized math test scores. The teacher explained to Laura that her son scored at the 85th percentile, meaning that he scored better than 85% of students who have taken the test. This information is best considered which of the following? 1. A norm-referenced interpretation [correct] 2. A criterion-referenced interpretation 3. A sensitive interpretation 4. A standards-based interpretation. 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Correct Answer 1] When a test score is interpreted relative to the performance of other students taking the same test, it is considered norm-referenced. [Feedback for Answer Choice 2] In criterion-referenced assessments, the score is interpreted by comparison with a standard or criterion, rather than with the scores of others. [Feedback for Answer Choice 3] This interpretation does not involve a high level of sensitivity. [Feedback for Answer Choice 4] In standards-based interpretations, students are typically judged to be “proficient” or “not proficient” on the basis of their scores on the tests as related to predetermined standards. [Q2] Becky has received a copy of her child's standardized math test scores. Even though her child "passed" the exam, the teacher explains that the report indicates a weakness in multiplication and division. This interpretation is best considered which of the following? 1. A norm-referenced interpretation 2. A criterion-referenced interpretation 3. A standards-based interpretation [correct] 4. A qualitative interpretation [Feedback for Answer Choice 1] When a test score is interpreted relative to the performance of other students taking the same test, it is considered norm-referenced. Becky’s child is not being compared to other students. [Feedback for Answer Choice 2] In criterion-referenced assessments, the score is interpreted by comparison with a standard or criterion, rather than with the scores of others. [Feedback for Correct Answer 3] The fact that Becky’s child showed a “weakness” suggests there was a standard that children’s scores were compared against. When a test score is interpreted relative to a 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

predetermined, external standard, this is considered a standards-based (also known as criterion-referenced) interpretation. [Feedback for Answer Choice 4] Becky’s child’s performance is being reported quantitatively in this scenario. [Q3] The difference between a standards-based test and a standardized test is best expressed as which of the following? 1. A standardized test requires norm-referenced interpretation, whereas a standardsbased test requires criterion-based interpretation. 2. A standardized test has uniform procedures for administration and scoring, whereas a standards-based test assesses student achievement of specific learning standards. [correct] 3. A standardized test is a national test, whereas a standards-based test is developed specifically for use in a school or curriculum. 4. No difference at all—these are two terms for the same thing. [Feedback for Answer Choice 1] This is not true for either assessment type. [Feedback for Correct Answer 2] These are the basic definitions of these two terms. [Feedback for Answer Choice 3] While this may seem to be the common approach, this is not a universal truth. [Feedback for Answer Choice 4] There are differences between these two types of tests. [Q4] Recently, many lawmakers have been advocating for standards-based assessments to be administered to all children at varying levels in their schooling. These types of assessments are designed to be which of the following? 1. Standardized achievement tests [correct] 2. Standardized aptitude tests 3. Standardized ability tests 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

4. Standardized diagnostic tests [Feedback for Correct Answer 1] An achievement test measures present knowledge and skills of a sample of relevant content, not aptitude or ability. It may—but need not—be used diagnostically. [Feedback for Answer Choice 2] An aptitude test would not be the best measure for determining scholastic progress in a given grade level. [Feedback for Answer Choice 3] An ability test would not be the best measure for determining scholastic progress in a given grade level. [Feedback for Answer Choice 4] A diagnostic test would not be the best measure for determining scholastic progress in a given grade level. Learning Outcome 7.3: Distinguish among different types of questionnaires and item formats. Know how to construct self-report questionnaires to provide valid and reliable information. [Q1] Marci creates a questionnaire to assess students’ study habits. The directions state, “On a scale of 1 to 5, indicate how often you participate in each of the following activities.” What kind of scale is Marci trying to use? 1. Likert-type [correct] 2. True Likert 3. Semantic differential 4. There is not enough information given to determine the scale [Feedback for Correct Answer 1] Likert-type scales are frequently used when researchers want students to indicate “how often” or “how many” or other responses (whereas true Likert scales are used to indicate just agreement/disagreement). [Feedback for Answer Choice 2] In a true Likert scale, the statement includes a value or positive or negative direction, and the respondent indicates agreement or disagreement with the statement. 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] A semantic differential scale has adjective pairs that provide a series of scales. [Feedback for Answer Choice 4] There is enough information available to appropriately answer the question. [Q2] Lorrie is conducting a study that examines the attitudes toward using marijuana for reducing stress. She creates a questionnaire with 30 items designed to assess stress level, general views about drugs, and individual drug use. Participants are asked to respond on a 5-point Likert scale (strongly disagree to strongly agree). The first item reads, “I am a regular marijuana user.” What’s the biggest problem with this? 1. It’s not phrased as a question, so this statement should be rewritten. 2. It’s biased toward people who are marijuana users, so this question should be omitted. 3. Being a marijuana user is not socially desirable so this question should not be first. 4. The language is too abstract, so this question should be rephrased. [correct] [Feedback for Answer Choice 1] It is not a problem that this is not phrased as a question. [Feedback for Answer Choice 2] This question does not show bias toward marijuana users. [Feedback for Answer Choice 3] This is not an accurate statement. [Feedback for Correct Answer 4] The word “regular” in this item requires a judgment by the person completing the questionnaire—how often is “regular” usage? That subjectivity compromises the validity of the instrument. [Q3] Janelle has written the following survey item: “Please indicate your comfortability with teaching students with behavioral challenges in the classroom: Very comfortable, Comfortable, Somewhat comfortable, Not comfortable.” Which of the following terms best describes this item? 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

1. Selected response [correct] 2. Constructed response 3. Standardized response 4. User response [Feedback for Correct Answer 1] Because the participant completing the survey is selecting a response and not adding open-ended information, this is a selected response item. [Feedback for Answer Choice 2] In this item, participants are not providing open-ended information. [Feedback for Answer Choice 3] This response does not indicate a standardized response. [Feedback for Answer Choice 4] This is not a term used to describe item responses. [Q4] Janice has written the following survey question: “Please describe your understanding of the Individuals with Disabilities Education Act.” Which of the following terms best describes this item? 1. Constructed response [correct] 2. Selected response 3. Standardized response 4. User response [Feedback for Correct Answer 1] This is a constructed response item. Participants are adding their own narrative and not selecting from pre-determined choices. [Feedback for Answer Choice 2] Participants are not selecting pre-determined answer choices. [Feedback for Answer Choice 3] This is not a term used to describe item responses. [Feedback for Answer Choice 4] 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This is not a term used to describe item responses. Learning Outcome 7.4: Distinguish among different forms of interviewing and observation techniques used to collect quantitative data and know how to address limitations of each approach. [Q1] Which of the following interview questions is a good example of a semistructured question? 1. What are some of the things on your “bucket list” to do before you turn 40? [correct] 2. What would you like to tell me about your childhood? 3. Have you ever had an out-of-body experience? 4. Why do you feel that way? [Feedback for Correct Answer 1] This question is specific in intent—focused on the “bucket” list before age 40—yet openended in that participants may have all sorts of different answers. Semistructured questions are best when they are reasonably objective but allow for probing, follow-up, and clarification. [Feedback for Answer Choice 2] This is an open-ended question. [Feedback for Answer Choice 3] This question only allows for a yes or no answer. [Feedback for Answer Choice 4] This is an open-ended question. [Q2] A teacher considers his general impression of his students when he scores a performance assessment. In this setting, he is similar to a researcher acting as an observer. Which of the following is most likely to influence his students’ scores? 1. Halo effect [correct] 2. Response set 3. Faking by the student 4. Knowledge of demand characteristics [Feedback for Correct Answer 1] 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

The halo effect occurs when an observer—in this case the teacher—allows an initial impression about a person or group to influence subsequent observations. [Feedback for Answer Choice 2] Response set is the tendency of the participant to respond in the same way, regardless of the content of the items. [Feedback for Answer Choice 3] Faking occurs when participants give deliberately inaccurate indications of their attitudes, personality, or interests. [Feedback for Answer Choice 4] This is not the threat occurring in this situation. [Q3] Jared is an eighth-grade student who struggles in math class. On a recent assessment, his teacher realized he selected answer choice “A” for each item. Which of the following has impacted this student’s scores? 1. Halo effect 2. Response set [correct] 3. Faking by the student 4. Open ended responding [Feedback for Answer Choice 1] The halo effect occurs when an observer allows an initial impression about a person or group to influence subsequent observations. [Feedback for Correct Answer 2] Response set is the tendency of the participant to respond in the same way, regardless of the content of the items. [Feedback for Answer Choice 3] This student is not faking, based on the information we have. [Feedback for Answer Choice 4] This is not an open-ended response. [Q4] 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Which of the following refers to the tendency to respond to the items in a way that is socially acceptable or desirable, regardless of the true or real attitudes or beliefs of the individual? 1. Social desirability [correct] 2. Halo effect 3. Faking 4. Teacher pleasing [Feedback for Correct Answer 1] Social desirability refers the tendency to respond to the items in a way that is socially acceptable or desirable, regardless of the true or real attitudes or beliefs of the individual. [Feedback for Answer Choice 2] The halo effect occurs when an observer allows an initial impression about a person or group to influence subsequent observations. [Feedback for Answer Choice 3] This is not an example of faking. [Feedback for Answer Choice 4] This is not a phrase used to describe threats to assessments.

Chapter 7 Application Exercises Chapter 7: Quantitative Data Collection Techniques Application Exercise 7.1: Evaluating Measures Learning Outcome 7.3: Distinguish among different types of questionnaires and item formats. Know how to construct self-report questionnaires to provide valid and reliable information. [Q1] Consider the following: A local aeronautics firm offers a summer internship program for college students majoring in astrophysics, engineering, and other natural sciences. He wants to know whether the summer program leads to student growth, both personal and professional. The program director decides to gather data from the interns at the start of the summer and then again at the end of the summer. The program director identifies a number of questions that he’d like to ask, including whether students feel they are good scientists, whether this opportunity helped them to learn, and what their personal lives are like, outside the lab. Given his general topics, do you recommend that he conduct face-to-face interviews or should he give out questionnaires? Explain the pros and cons of the choice you make. [Q1 Model Response] In general, face-to-face interviews have the distinct advantage of enabling a researcher to establish rapport with potential participants and therefore gain their cooperation, and they allow the interviewer some flexibility with follow-up questions. However, because the program director is effectively the students’ boss, the participants may feel uneasy speaking directly to him or even coerced to participate. If he chooses to use an interview, he should consider having someone else conduct it. In this case, (anonymous) questionnaires might be more appropriate, although questionnaires have a lower return rate—he cannot guarantee all the participants will choose to respond on both occasions. [Q2] Consider the following: A local aeronautics firm offers a summer internship program for college students majoring in astrophysics, engineering, and other natural sciences. He wants to know whether the summer program leads to student growth, both personal and professional. The program director decides to gather data from the interns at the start of the summer and then again at the end of the summer. 12 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

The program director would like students to complete a rating scale as part of his interview or questionnaire. Among the questions he is considering are the following: Please rate the following statements about personal growth on a scale from 1 (strongly disagree) to 7 (strongly agree). a. I usually have the patience to deal with the frustration that arises when experiments fail. b. I have reduced the number of glasses of wine I drink each week. c. I am sure that my research area is something I want to continue to pursue during graduate study. d. I have some doubts about whether my scientific skills are good enough. Review each of his questions for clarity and potential bias. [Q2 Model Response] Item a: The word "usually" is ambiguous and does not have a precise meaning. Item b: There are unwarranted assumptions in this question: there is an assumption that the respondent drinks wine, and there is an assumption that drinking wine has a negative effect on personal growth. Item c: There is an unwarranted assumption that the respondent is going to graduate school. Item d: The term "good enough" is ambiguous and lacks a point of reference. [Q3] Consider the following: A local aeronautics firm offers a summer internship program for college students majoring in astrophysics, engineering, and other natural sciences. He wants to know whether the summer program leads to student growth, both personal and professional. The program director decides to gather data from the interns at the start of the summer and then again at the end of the summer. Upon reflection, the program director begins to wonder whether his measure is really sensitive enough to identify whether the interns changed in significant ways over the course of the summer. For example, he reconsiders the item, “My knowledge of aeronautics is better than that of my peers,” wondering whether it’s sensitive enough to measure the interns’ confidence in their scientific knowledge base. What three key questions should he ask to assess the sensitivity of his measure? [Q3 Model Response] 13 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

When considering whether a measure will be sufficiently sensitive, researchers should focus on validity, reliability, and the range of obtained scores. To assess validity, he asks, “Is the construct (confidence in one’s knowledge base) being measured assessed directly and completely?” To assess reliability, he asks, “Are students’ scores on this measure influenced by error?” To assess the range of observed scores, he asks, ”Will all the participants in our summer program give exactly the same answer?” Application Exercise 7.2: Understanding Standardized SAT-Math Scores: Z-Score Interpretation Learning Outcome 7.2: Describe the characteristics of standardized tests, how they differ from locally developed tests, and how they are used in research. Know how to interpret scores reported from standardized tests. [Q1] In 2014, the SAT combined score (three tests: critical reading, math, and writing) had a standard score mean of about 1500 and a standard deviation equal to about 300. Each individual test had a mean of about 500 and standard deviation of about 100. Consider these numbers as you answer the following questions. What is the percentile rank of a student whose z score on the SAT-M (math subtest) is +2? What is the score that is reported to the student and to the colleges to which they apply? [Q1 Model Response] A z score is a measure of how far a score is from the mean in units of standard deviation. A z score of +2 is equivalent to a percentile score of approximately 98%. This student received a 700 on the math portion of the SAT = 500 (mean) + 2 × 100 (standard deviation). [Q2] In 2014, the SAT combined score (three tests: critical reading, math, and writing) had a standard score mean of about 1500 and a standard deviation equal to about 300. Each individual test had a mean of about 500 and standard deviation of about 100. Consider these numbers as you answer the following questions. What would be the score of a student whose z score on the SAT-M is −1? What percentage of students scored better than this student? [Q2 Model Response] This student received a 400 on the math portion of the SAT —mean (500) minus one standard deviation (100). 84% of the students taking the SAT scored better (see the normal curve). 14 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q3] In 2014, the SAT combined score (three tests: critical reading, math, and writing) had a standard score mean of about 1500 and a standard deviation equal to about 300. Each individual test had a mean of about 500 and standard deviation of about 100. Consider these numbers as you answer the following questions. What would be the z score of a student whose percentile rank on the SAT-M is 0.1? [Q3 Model Response] −3.0. This student’s math score is 200. Application Exercise 7.3: Types of Quantitative Interview Questions Learning Outcome 7.4: Distinguish among different forms of interviewing and observation techniques used to collect quantitative data and know how to address limitations of each approach. [Q1] Review the section entitled Types of Interview Questions. Then determine whether the following sample interview question is structured, semistructured, or unstructured. Justify your answer: “When considering your preferred modalities of instruction, which do you most commonly utilize in the classroom?” [Q1 Model Response] This is an example of a semistructured question. First, it is open ended in nature. Second, the respondent could respond in any number of ways as there are no sample approaches listed in the question. However, the interviewer has a general goal in mind, in this case, understanding instructional choices. Follow-up questions are easily derived from this queston. [Q2] Review the section entitled Types of Interview Questions. Then determine wjetjer the following sample interview question is structured, semistructured, or unstructured. Justify your answer: “From the following, which is your preferred modality of instruction: Lecture, Group Work, or Project-Based Learning?” [Q2 Model Response] This is an example of a structured question. The participant is given choices from which to choose. Essentially, this is a selected response question. 15 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q3] Review the section entitled Types of Interview Questions. Then determine whether the following sample interview question is structured, semistructured, or unstructured. Justify your answer: “What are your thoughts on differentiated instruction?” [Q3 Model Response] This is an example of an unstructured question. It is not likely that a quantitative researcher will employ such techniques in her or his study. This is a much more common approach to qualitative research. This question has no specified structure and while the general topic is outlined, respondents could respond in a variety of ways.

Chapter 7 Test Items SELF-ASSESSMENT ITEMS 1) The sensitivity of an instrument is determined by which of the following? a) How much variability is in the instrument b) How well the researcher considers the participant perspective c) How well managed the research study is d) How well the items and instructions are worded so they do not offend 2) Students in twelfth grade were given a type of instrument to determine their career preferences. Which of the following is a term for the type of instrument that could have been used? a) Achievement test b) Norm-referenced test c) Noncognitive questionnaire d) Observational instrument 3) Which of the following statements is true about a norm-referenced test? a) Individuals’ performance are compared to that of their peers b) Scores are typically higher than normal c) Results are reported as pass/fail d) The acquisition of specific skills is measured 4) In one of the local schools, it was found that the students were scoring above grade level in all areas except mathematics, where they scored well below average. What type of test should be used to pinpoint skill deficiencies? a) A criterion-referenced test b) A questionnaire c) A norm-referenced battery d) A standardized aptitude test 5) The purpose of an aptitude test is to measure which of the following? a) Knowledge b) Potential to learn c) Affective traits d) Behavior 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

6) A student received a score of 83% on a science test. His z-score for the test was +1.5. Which of the following is a correct interpretation of the student’s score? a) The score was higher relative to his classmates. b) The score was lower relative to his classmates. c) The score was average relative to his classmates. d) Not enough information is provided to determine the relationship between his score and the scores of his classmates. 7) A teacher wanted to compare the results from two different achievement tests with different numbers of items that had been administered to her class. Which of the following is the type of score she should use? a) Grade equivalent score b) Median c) Raw score d) Standard score 8) Which scale is being used when students are asked to respond on a five-point scale ranging from strongly agree to strongly disagree? a) Semantic differential b) Likert c) Checklist d) Ranked items 9) Which of the following is an example of a response set? a) A participant selects a set of responses that vary. b) A participant selects a set of responses that reflect their real attitude. c) A participant checks the same response option for each item on the test. d) A participant skips a set of responses rather than give an inaccurate response. 10) A researcher observed a student in a fourth-grade class. The number of times the student raised his hand, spoke to students around him, and left his seat were recorded. Which of the following is the term used to refer to this type of observation? a) Attitudinal observations b) High-inference observations c) Low-inference observations d) One-dimensional observations 11) Which of the following is used to describe what occurs when an observer allows an initial impression to influence subsequent observations? 18 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

a) Response set b) Contamination c) Demand characteristics d) Halo effect 12) Which of the following is a clear advantage of an interview over all other data collection methods? a) The researcher is able to measure consistently. b) The researcher is able to understand the breadth of knowledge a student has mastered. c) The researcher is able to understand the depth of a student’s responses. d) The researcher is able to influence what is heard or recorded. 13) A researcher prepared some interview questions for a study that was planned to be conducted with college students. Some, but not all, of the questions were openended, although the purpose for asking them was quite specific. Which term best describes this type of question? a) Structured b) Semi-structured c) Unstructured d) Biased 14) Which of the following is a source of information that includes an external review of a test being considered for use in a study? a) Mental Measurement Yearbooks b) The test developer c) Journal articles with validity studies of the instrument d) Department of Assessment Index to Tests

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Scores were reported on students’ attitudes toward a particular class in a local high school. Which of the following describes this measure? a) Aptitude test b) Self-report measure c) Observational measure d) Unstructured interview 2) Ms. Carter, the special education teacher, has to develop individualized educational programs for all the students in her class. What type of tests should she use to identify what the students do and do not know in each area? a) Criterion-referenced b) Norm-referenced c) Questionnaires d) Observations 3) Which of the following tests should a school district use if it wants to evaluate the extent to which fifth graders are meeting expectations based on a national sample in a number of content areas? a) Criterion-referenced achievement test b) Noncognitive test c) Standardized aptitude test d) Norm-referenced achievement test 4) Which of the following is a common use of standardized aptitude tests? a) To assess students’ progress by comparing test scores with a national standard b) To determine how well students might perform in the future c) To measure the efficacy of instructional programs d) To pinpoint strengths and weaknesses in specific content areas 5) Which of the following describes a test used to discriminate between individuals? a) Criterion-referenced test b) Formative test c) Pilot test d) Norm-referenced test 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

6) On a standardized achievement test, a fifth-grade student obtained a grade equivalent of 8.0. Which of the following is a correct interpretation of the student’s score? a) The student achieved a score that was the same as the median score for all beginning eighth graders in the norm group. b) The student had mastered the material covered in the eighth grade. c) The student should be placed in a class with beginning eighth graders the following school year. d) The student was able to respond to the test items in the same way as an average eighth grader. 7) A student received a score of 70% on a science test. His z-score for the test was +1.5. Which of the following is a correct interpretation of the student’s score? a) The score was higher relative to his classmates. b) Not enough information is provided to determine the relationship between his score and the scores of his classmates. c) The score was lower relative to his classmates. d) The score was average relative to his classmates. 8) A student completed a questionnaire that had the following response options: outstanding, excellent, good, fair, poor. Which of the following is used to describe the type of scale used on the questionnaire? a) Semantic differential b) Likert-type c) Checklist d) Ranked items 9) Which of the following is used to describe a scale in which the participant checks a point on a continuum between two adjectives? a) Interest inventory b) Rating scale c) Semantic differential scale d) Likert scale 10) Which of the following is used to describe the source of error that occurs when a participant checks the same option for each item on an attitude scale? a) Response set b) Contamination 21 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) Halo effect d) Low inference 11) A researcher observed three students on the playground. The number of times each student talked to another student, played a game with other students, and played alone was recorded. Which of the following terms is used to describe this type of observation? a) One-dimensional observations b) Attitudinal observations c) Low-inference observations d) High-inference observations 12) A sixth-grade teacher completed a behavior rating scale for one of the students in his class. The teacher was influenced by the student’s performance in academic subjects and therefore rated the student as low in intelligence, perseverance, honesty, friendliness, and other aspects of personality. The results may be inaccurate due to which of the following observer effects? a) Contamination b) Halo effect c) Semantic differential d) Unreliable responses 13) Which of the following is a possible disadvantage of using an interview rather than a self-report questionnaire? a) Less information is gathered in an interview due to time constraints. b) The subjectivity of the interview may lead to a biased interpretation. c) The interviewer can motivate the participant to respond. d) The self-report questionnaire usually has a higher response rate. 14) Which of the following is a source of information that provides evaluative reviews of the quality of an instrument? a) ETS Test Collection b) Tests in Print c) Mental Measurement Yearbooks d) Index of Tests Used in Educational Dissertations Short-Answer Questions 1) Why is sensitivity an essential quality for instruments used for educational research? 22 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2) Explain how extraneous factors on a general measure of intelligence can lessen sensitivity if used to measure the relationship between intelligence and math instructional methods. Essay Question Describe the differences between norm-referenced and criterion-referenced tests including the purposes for each type. Give an example where each type of test could be used.

Chapter 7 Test Answer Key ANSWER KEY: Chapter 7 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. a 2. c 3. a 4. a 5. b 6. a 7. d 8. b 9. c 10. c 11. d 12. c 13. b 14. a

INSTRUCTOR’S EXAM ITEMS 1. b 2. a 3. d 4. b 5. d 6. a 7. a 8. b 9. c 10. a 11. c 12. b 13. b 14. c

Chapter 8 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

Content Producer: Janelle Rogers Supplement Project Manager: Chithra Rajasekaran, Straive

Copyright © 2022, 2016, 2012 by Pearson Education, Inc. or its affiliates, 221 River Street, Hoboken, NJ 07030. All Rights Reserved. Manufactured in the United States of America. This publication is protected by copyright, and permission should be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise. For information regarding permissions, request forms, and the appropriate contacts within the Pearson Education Global Rights and Permissions department, please visit www.pearsoned.com/permissions/

PEARSON and ALWAYS LEARNING are exclusive trademarks owned by Pearson Education, Inc. or its affiliates, in the United States, and/or other countries.

Unless otherwise indicated herein, any third-party trademarks, logos, or icons that may appear in this work are the property of their respective owners, and any references to third-party trademarks, logos, icons, or other trade dress are for demonstrative or descriptive purposes only. Such references are not intended to imply any sponsorship, endorsement, authorization, or promotion of Pearson’s products by the owners of such marks, or any relationship between the owner and Pearson Education, Inc., or its affiliates, authors, licensees, or distributors.

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 8: Nonexperimental Quantitative Research Designs ................................... 1 Learning Outcome Quizzes........................................................................... 1 Application Exercises .................................................................................. 14 Test Items ................................................................................................... 20 Test Answer Key ......................................................................................... 26

Chapter 8: Nonexperimental Quantitative Research Designs Chapter 8 Learning Outcome Quizzes Learning Outcome 8.1: Identify and explain the importance of two key principles of effective quantitative research designs. [Q1] Which characteristic distinguishes nonexperimental research from experimental research? 1. The comparison of groups 2. The manipulation of the conditions [correct] 3. The measure of a dependent variable 4. Emphasis on the relationship between variables [Feedback for Answer Choice 1] Groups are compared in both nonexperimental and experimental designs. [Feedback for Correct Answer 2] Nonexperimental designs are used to study existing groups when manipulation by the researcher is not possible or ethical. Experimental designs require that the researcher manipulate conditions experienced by the participants. [Feedback for Answer Choice 3] Dependent variables are measured in both experimental and nonexperimental designs. [Feedback for Answer Choice 4] Variables are compared in regard to their relationship to one another in both nonexperimental and experimental designs. [Q2] Which of the following terms refers to the plan, structure, and procedures of the study to collect data, and in the case of experiments, to implement interventions? 1. Research plan 2. Research design [correct] 3. Research report 4. Research analysis [Feedback for Answer Choice 1] 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

While researchers certainly make plans for how their work will be methodically conducted, this is not the best answer. [Feedback for Correct Answer 2] Research design refers to the plan, structure, and procedures of the study to collect data, and in the case of experiments, to implement interventions. [Feedback for Answer Choice 3] While researchers do write research reports, this is not the appropriate answer choice. [Feedback for Answer Choice 4] Analysis is a vital component of quality research of all designs, but this is not the correct answer. [Q3] Which of the following serves as the best rationale for why a good research design is important? 1. A good design provides the probability that the research will be easy to conduct. 2. A good design provides the best prospect of finding what is true about what is studied. [correct] 3. A good design provides the best prospect of getting the results you want. 4. A good design allows the researcher to prove causality. [Feedback for Answer Choice 1] There’s no guarantee that any research will be easy. To the contrary, research is a challenging process. [Feedback for Correct Answer 2] A good design provides the best prospect of finding what is true about what is studied. It allows the researcher to focus in on the signal and reduce the noise. [Feedback for Answer Choice 3] A researcher should never chase desired results, but rather ethically report findings based on ethically collected data and analysis. [Feedback for Answer Choice 4] A good researcher understands that not all designs allow for the suggestion of causality. [Q4] 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Which of the following nonexperimental research designs would be best used to explore how two variables are related? 1. Causal–comparative 2. Descriptive 3. Comparative 4. Correlational [correct] [Feedback for Answer Choice 1] In causal–comparative research, explanations are offered by comparing groups that receive naturally occurring interventions. [Feedback for Answer Choice 2] The purpose of descriptive work is to provide a description of the what and how much of a phenomenon. [Feedback for Answer Choice 3] The purpose of comparative research is to compare dependent variable values of two or more levels of an independent variable [Feedback for Correct Answer 4] The purpose of correlational research is to show how two variables are related, using a correlation coefficient. Learning Outcome 8.2: Understand the essential characteristics and limitations of using comparative designs to investigate relationships. [Q1] Which of the following represents a frequent reason to conduct a descriptive study? 1. Identify and define variables for future study [correct] 2. Search for predictive relationships 3. Identify associations among variables 4. Test for differences between groups [Feedback for Correct Answer 1] A descriptive study simply examines what or how much something exists and is particularly valuable when an area is first investigated. Good descriptions help guide the conceptual framework and operational definitions for future research. [Feedback for Answer Choice 2] 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Descriptive studies do not seek to determine predictive relationships. [Feedback for Answer Choice 3] Descriptive studies do not suggest any type of statistical association among variables. [Feedback for Answer Choice 4] Descriptive studies may compare groups with simple descriptive statistics, but they cannot determine statistically significant difference between groups. [Q2] Which of the following terms best describes the situation when one variable varies systematically with another variable? 1. Statistical significance 2. Relationship [correct] 3. Probability 4. Causality [Feedback for Answer Choice 1] Statistical significance is a phrase used in describing probability; therefore, this is not the best answer. [Feedback for Correct Answer 2] A relationship is found when one variable varies systematically with another variable. [Feedback for Answer Choice 3] Probability is related to hypothesis testing, not variable similarity in this sense. [Feedback for Answer Choice 4] Causality would refer to how one variable “causes” fluctuation in another. This is not the correct answer. [Q3] Which of the following research designs is used to investigate the relationship of one variable to another, by simply examining whether the value of the dependent variable(s) in one non-intervened group is the same as or different from the value of the dependent variable(s) of other non-intervened groups? 1. Correlational designs 2. Causal designs 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. Comparative designs [correct] 4. Descriptive designs [Feedback for Answer Choice 1] Correlational designs explore relationships among variables, but this is not the best answer. [Feedback for Answer Choice 2] Finding causality is not the purpose of most research designs, and this is not the process described here. [Feedback for Correct Answer 3] Comparative designs are used to investigate the relationship of one variable to another, by simply examining whether the value of the dependent variable(s) in one nonintervened group is the same as or different from the value of the dependent variable(s) of other non-intervened groups. In other words, comparative designs check to see whether there are significant differences between focus variables among groups. [Feedback for Answer Choice 4] Descriptive designs don’t explore relationships; they simply describe. [Q4] Which of the following best represents a research question for a descriptive study? 1. What types of breakfast foods are common among teenagers? [correct] 2. How does breakfast impact scholastic performance? 3. What is the relationship between eating breakfast and feeling motivated in early morning classes? 4. How does eating breakfast influence a person’s body weight? [Feedback for Correct Answer 1] Descriptive questions aren’t questions that seek to compare or explain. They simply describe. This question is an example of a question that would be explored through descriptive processes. [Feedback for Answer Choice 2] In this question, the researcher is trying to understand more than just a description. [Feedback for Answer Choice 3] 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Here the researcher wants to understand a relationship among variables. This goes beyond descriptive processes. [Feedback for Answer Choice 4] Here the researcher wants to understand a relationship among variables. This goes beyond descriptive processes. Learning Outcome 8.3: Understand the essential characteristics and limitations of using correlational and predictive designs to investigate relationships. [Q1] Of the following, which best illustrates a good question for a correlational study? 1. Does parental involvement predict students' grades? [correct] 2. Are parents who receive an invitation from their child's teacher more involved than parents who do not? 3. What are parents' attitudes about being involved in their children's school? 4. How does the “Come See Us!” program encourage parents to get involved at Park Middle School? [Feedback for Correct Answer 1] Correlational studies involve collecting data to determine whether, and to what degree, a relationship exists between two or more quantifiable variables. [Feedback for Answer Choice 2] This question could simply be answered with a descriptive design. [Feedback for Answer Choice 3] This question could simply be answered with a descriptive design. [Feedback for Answer Choice 4] This question could simply be answered with a descriptive design. [Q2] When should a researcher choose a multiple regression design for their study? 1. When the researcher wants to control only one of several possible independent variables 2. When the purpose is to investigate causal relationships when an experimental study is not possible 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. When the purpose is to generalize to a population 4. When two or more independent variables are used to predict values of a continuous dependent variable [correct] [Feedback for Answer Choice 1] This is not a situation where multiple regression is the best choice. [Feedback for Answer Choice 2] Multiple regression does not infer causal relationships. [Feedback for Answer Choice 3] The results of a regression study are not necessarily generalizable to an entire population. [Feedback for Correct Answer 4] Multiple regression is also known as multiple correlation. An equation is derived to show how the independent variables, together, associate with the dependent variable. [Q3] In which of the following designs is a regression analysis used to show how one variable can predict the dependent variable that occurs at a later time? 1. Probability study design 2. Prediction study design [correct] 3. Multiple regression design 4. Correlational research design [Feedback for Answer Choice 1] Probability models are useful, but this is not the best phrase to describe this type of study design. [Feedback for Correct Answer 2] With a prediction study design, a regression analysis is used to show how one variable can predict the dependent variable that occurs at a later time. [Feedback for Answer Choice 3] While this may include multiple regression, multiple regression alone is not a broad enough descriptor. [Feedback for Answer Choice 4] 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Correlational studies indicate relationships between variables. [Q4] John conducted a correlational study that found that percentage of on-time homework completion and final grade were correlated at a rate of .50. How much of the variance can be explained using this correlation coefficient? 1. 25% [correct] 2. 50% 3. 75% 4. 100% [Feedback for Correct Answer 1] By squaring .5, we get .25, which indicates that 25% of the variance can be explained with 75% remaining unexplained. [Feedback for Answer Choice 2] This is not the correct statistic. [Feedback for Answer Choice 3] This is not the correct statistic. [Feedback for Answer Choice 4] This is not the correct statistic. Learning Outcome 8.4: Know the characteristics and limitations of causal– comparative and ex post facto studies. [Q1] Of the following, which research question is best addressed by causal–comparative research? 1. Do student smoking rates differ between rural, suburban, and urban high schools? [correct] 2. What are the characteristics of high school student smokers? 3. Is there a relation between smoking rates and depression scores for high school students? 4. Does participating in a special anti-smoking curriculum in health class lead to more negative attitudes about smoking than participating in the usual curriculum? [Feedback for Correct Answer 1] 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

In this example, the three comparison groups are defined by school setting, and the dependent variable is student smoking rate. [Feedback for Answer Choice 2] This is a question best answered through descriptive means. [Feedback for Answer Choice 3] This question is best suited for correlational designs. [Feedback for Answer Choice 4] This question is not comparing groups, but rather the impact of an anti-smoking curriculum. [Q2] A researcher identifies events that have already occurred or conditions that are already present and then collects data to investigate a possible relationship between these factors and subsequent characteristics or behaviors. This type of research design is known as which of the following? 1. True experimental design 2. Quasi-experimental design 3. Ex-post facto design [correct] 4. Pre-experimental design [Feedback for Answer Choice 1] This could not be an experimental design as the events have already occurred. [Feedback for Answer Choice 2] This could not be a quasi-experimental design as the events have already occurred. [Feedback for Correct Answer 3] Ex post facto means “after the fact.” These designs allow a researcher to investigate the extent to which existing events or conditions affect the dependent variable of interest, without manipulating any independent variable. [Feedback for Answer Choice 4] This is not any form of an experimental design as no variables are manipulated. [Q3] What is the weakest component of a causal–comparative study? 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

1. Inability to describe a population 2. Inability to obtain a correlation coefficient 3. Lack of manipulation of the dependent variable 4. Lack of manipulation of the independent variable [correct] [Feedback for Answer Choice 1] This is not an inherent weakness of a causal–comparative study. [Feedback for Answer Choice 2] Causal–comparative studies seek to suggest causal relationships among variables, not to explore correlation. [Feedback for Answer Choice 3] Dependent variables are not manipulated. [Feedback for Correct Answer 4] The inability to manipulate the independent variable is a major weakness of a causal– comparative design. [Q4] Jenna conducted a correlational study and found that when an individual’s total hours of sleep in a week increased, markers of stress decreased at a rate suggesting a significant correlation. Which of the following could represent her correlation coefficient? 1. −.83 [correct] 2. .83 3. −.24 4. .24 [Feedback for Correct Answer 1] Because the correlation is strong, we anticipate a number of around .7 or greater. Because the relationship is inverse, the coefficient is negative. [Feedback for Answer Choice 2] This indicates a strong relationship, but not an inverse relationship. [Feedback for Answer Choice 3] This would not indicate a strong relationship. 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 4] This would not indicate a strong or inverse relationship. Learning Outcome 8.5: Identify the steps taken to design and employ an effective paper or electronic survey. [Q1] A survey researcher would be most likely to plan to draw conclusions about which of the following? 1. All parents in a community [correct] 2. The sample of parents who complete the survey 3. All parents in the world 4. Different groups of parents within the sample. [Feedback for Correct Answer 1] Survey researchers aim to generalize their results to a population and make decisions about who to study so they can make these claims. It’s unlikely, however, that researchers could select a sample appropriate to generalize the results to the population of all parents in the world. [Feedback for Answer Choice 2] While the survey researcher could indeed describe this sample, this would not be the main goal. [Feedback for Answer Choice 3] This is much too broad of a sample. [Feedback for Answer Choice 4] A survey researcher is not likely trying to determine group differences through a survey design. [Q2] Which type of research is most likely to address the question, “What are the characteristics of today's middle school students?” 1. Ex post facto research 2. Survey research [correct] 3. Causal–comparative research 11 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

4. Correlational research [Feedback for Answer Choice 1] Ex post facto research involves the analysis of data “after the fact.” [Feedback for Correct Answer 2] Survey research focuses on describing trends of a group. [Feedback for Answer Choice 3] There are no inferences of causation in this research question. [Feedback for Answer Choice 4] This question is not seeking to understand the relationship between two variables. [Q3] What characteristic distinguishes a longitudinal study from other types of survey research? 1. The survey is administered at two or more points in time. [correct] 2. The survey includes all members of the population. 3. The survey includes one sample from the population. 4. The survey is administered to multiple samples from one population. [Feedback for Correct Answer 1] A longitudinal study gathers data from samples of individuals at different points of time. [Feedback for Answer Choice 2] Longitudinal studies make generalizations using samples. [Feedback for Answer Choice 3] While this may be true, this is not always true. [Feedback for Answer Choice 4] This is not characteristic of longitudinal research. [Q4] In which of the following types of surveys is information collected from one or more samples or populations at one time? 1. Longitudinal 12 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. Descriptive 3. Cross-sectional [correct] 4. Inferential [Feedback for Answer Choice 1] Longitudinal surveys involve surveying the same sample multiple times across time. [Feedback for Answer Choice 2] This is not a descriptor used to primarily describe surveys. [Feedback for Correct Answer 3] In a cross-sectional survey, information is collected from one or more samples or populations at one time [Feedback for Answer Choice 4] This is not a descriptor used to primarily describe surveys.

Chapter 8 Application Exercises Chapter 8: Nonexperimental Quantitative Research Designs Application Exercise 8.1: Understanding Correlations Learning Outcome 8.3: Understand the essential characteristics and limitations of using correlational and predictive designs to investigate relationships. [Q1] When considering correlational research as a potential method, it is important to understand the statistics underpinning correlational work. Use the table shown below to answer the following question. Blank

Ruth Carol Peggy Jane Roy Jack Betty Jerry Dennis Carl

Self-Esteem Score 27 22 26 20 22 25 27 23 20 21

Final Exam Score 88 92 94 83 67 79 86 93 87 82

Motivation Score 8 10 9 8 6 8 9 9 7 7

Test Anxiety 5 4 4 5 9 3 4 3 2 6

The correlation between self-esteem and final exam is r = .24. What conclusion can you draw? [Q1 Model Response] A correlation coefficient of just .24 indicates a weak positive relationship between the variables. This would not lead to a conclusion that these two variables have a close relationship with one another. [Q2] When considering correlational research as a potential method, it is important to understand the statistics underpinning correlational work. Use the table shown below to answer the following question.

Blank

Ruth Carol Peggy Jane Roy Jack Betty Jerry Dennis Carl

Self-Esteem Score 27 22 26 20 22 25 27 23 20 21

Final Exam Score 88 92 94 83 67 79 86 93 87 82

Motivation Score 8 10 9 8 6 8 9 9 7 7

Test Anxiety 5 4 4 5 9 3 4 3 2 6

The correlation between final exam and motivation is r = .80. What conclusion can you draw? [Q2 Model Response] These two variables have a strong positive relationship. Students with high motivation also have high exam scores. A correlation coefficient of .8 leads the researcher to believe that there is a strong connection between these two variables. [Q3] When considering correlational research as a potential method, it is important to understand the statistics underpinning correlational work. Use the table shown below to answer the following question. Blank

Ruth Carol Peggy Jane Roy Jack Betty Jerry Dennis Carl

Self-Esteem Score 27 22 26 20 22 25 27 23 20 21

Final Exam Score 88 92 94 83 67 79 86 93 87 82

Motivation Score 8 10 9 8 6 8 9 9 7 7

Test Anxiety 5 4 4 5 9 3 4 3 2 6

These two variables have a strong negative relationship. Students with more anxiety have lower exam scores. A correlation coefficient of −.72 is strong and leads the researcher to understand that these variables are closely connected in an inverse relationship. Application Exercise 8.2: Causal Comparative Designs Learning Outcome 8.4: Know the characteristics and limitations of causal– comparative and ex post facto studies. [Q1] Apply the knowledge you have gained from reading Chapter 8, particularly the section entitled Causal–Comparative Designs in order to answer the following. Consider the research question “Do students who enrolled in first-year seminars at a university have different university GPAs and different levels of persistence than do students who did not enroll in first-year seminars?” What are the considerations when selecting participants? [Q1 Model Response] The researcher in this study is interested in two existing groups: students who did enroll and students who did not enroll in first-year seminars at a university. The researcher needs to carefully define these groups (for example, perhaps not including students who attended part of a seminar but not the full sessions). Ideally the participants should be randomly selected to participate, which would require obtaining a list of all first-year students who did attend the seminar and those who did not. The researcher would want to randomly sample from these two existing populations and strive to have groups that are homogenous on all relevant variables except the variable of interest. [Q2] Apply the knowledge you have gained from reading Chapter 8, particularly the section entitled Causal Comparative Designs in order to answer the following. Consider the research question “Do students who enrolled in first-year seminars at a university have different university GPAs and different levels of persistence than do students who did not enroll in first-year seminars?” Consider the extraneous variable “academic major.” Describe how you could control this extraneous variable comparing homogeneous groups. [Q2 Model Response]

To control for academic major through comparing homogenous groups, you could select students from only one major (e.g., biology) and examine the differences between students with or without a seminar.

[Q3] Apply the knowledge you have gained from reading Chapter 8, particularly the section entitled Causal Comparative Designs in order to answer the following. Consider the research question “Do students who enrolled in first-year seminars at a university have different university GPAs and different levels of persistence than do students who did not enroll in first-year seminars?” Consider the extraneous variable “campus living arrangement.” Describe how you could control this extraneous variable through matching. [Q3 Model Response] To use matching to control for campus living arrangement, which may be an extraneous variable especially for persistence, you could match pairs of individuals based on living arrangement (e.g., in dorm, in apartment, in Greek house). The resulting seminar and the no-seminar groups would have the same number of students from each living arrangement. Application Exercise 8.3: Survey Research Learning Outcome 8.5: Identify the steps taken to design and employ an effective paper or electronic survey. [Q1] The U.S. Department of Education has within it an institute known as the National Center for Education Statistics (NCES), which collects, analyzes, and disseminates information about educational issues in America. Within NCES is the Secondary Longitudinal Studies Program, which has studied American high school students for more than 40 years. Read the basic information about the study, linked here and then answer the questions. If for some reason the link is broken, type “NCES Secondary School Longitudinal Survey” into a search engine. One of the specific studies conducted by this institution is known as the High School Longitudinal Study of 2009. This survey study had three primary research questions: •

•

What majors and careers do students decide to pursue when, why, and how?

•

How do students choose science, technology, engineering, and math (STEM) courses, majors, and careers?

In 2009, more than 23,000 ninth-graders across the country completed the survey. The students were enrolled in 944 schools (both public and private), and they were randomly selected from those schools. Explain why the sample is so large, with reference to the population that the researchers want their results to represent. [Q1 Model Response] The NCES reports, “The HSLS population consists of all ninth-graders across the U.S. enrolled in regular public, Catholic, and other private schools that include 9th and 11th grades.” [The reason they specified 11th grade was that their plan for data collection included a follow-up with the students once they were in the 11th grade.] Because the population is large and diverse, the researchers needed a large sample to ensure that it was as representative as possible within their timeframe and budget. They also needed to account for attrition (i.e., over the course of the study, some participants would likely disappear, die, or choose not to participate further). [Q2] The U.S. Department of Education has within it an institute known as the National Center for Education Statistics (NCES), which collects, analyzes, and disseminates information about educational issues in America. Within NCES is the Secondary Longitudinal Studies Program, which has studied American high school students for more than 40 years. Read the basic information about the study, linked here, and then answer the questions. If for some reason the link is broken, type “NCES Secondary School Longitudinal Survey” into a search engine. Why did the researchers collect more data in 2012 and 2013, specifically? Consider the research questions in your explanation. [Q2 Model Response] This is intended to be a longitudinal survey study so it will have multiple phases of data collection. In general, studies that focus on “students' trajectories” are longitudinal studies with panel samples, which include data from the same people over time. The specific years are determined based on the research questions. Because the researchers are interested in the transition out of high school (to postsecondary education, the workforce, or something else), they need to collect data at those times— 18 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

in 2012 most of the students would be seniors, and in 2013 many should have graduated from high school. [Q3] The U.S. Department of Education has within it an institute known as the National Center for Education Statistics (NCES), which collects, analyzes, and disseminates information about educational issues in America. Within NCES is the Secondary Longitudinal Studies Program, which has studied American high school students for more than 40 years. Read the basic information about the study, linked here, and then answer the questions. If for some reason the link is broken, type “NCES Secondary School Longitudinal Survey” into a search engine. The initial phase of this survey (2009) was most likely NOT an Internet-based survey. Why would an Internet-based survey have been inappropriate in this case? Consider sampling as well as ethical reasons. [Q3 Model Response] First, the students were randomly selected to participate from target schools all across the country. Some of them may not have had access to the Internet in a place that would allow them to complete the survey in a confidential way, and some may not know how to use the Internet at all. Second, because the participants were in the 9th grade, the researchers would need to seek permission from their guardians, so they could not simply send a questionnaire to a list of e-mail addresses and request that it be returned. Third, because the students are followed longitudinally, the researchers need a consistent way to identify each participant so they can match their 2009 responses with their later responses. At present this is difficult to do via the Internet in a way that ensures confidentiality. You may have thought of other reasons as well.

Chapter 8 Test Items SELF-ASSESSMENT ITEMS 1) Which of the following is a characteristic of a descriptive design? a) It predicts subsequent events from something happening at the moment. b) It provides causal conclusions using descriptors. c) It examines the effect of an independent variable on a dependent variable. d) It provides information about existing phenomena. 2) Results of a descriptive study on the sharing behavior of first-grade students showed that children with siblings displayed sharing behavior more often than children who did not have brothers or sisters. The researcher concluded that first-graders were more likely to learn to share if there were other children in the family. Which of the following did the researcher violate by reaching his conclusion? a) Descriptive studies can describe participants or events, not behaviors. b) Descriptive studies do not assume causal relationships between variables. c) The researcher did not identify his criteria for sharing behavior. d) The researcher only used first-graders as participants in his study. 3) Which of the following statements is NOT TRUE of relationship studies? a) Relationship studies enable us to establish a causal relationship between two variables. b) In relationship studies, groups of students can be compared. c) Relationship studies enable us to make predictions. d) Relationship studies investigate whether one variable varies systematically with another variable. 4) Which of the following statements best describes a multiple correlation study? a) The results provide more than one correlation coefficient to indicate the relationship between two variables. b) One set of data is collected for participants using more than one method. c) Causation can usually be inferred from the results because more than one variable is used. d) More than one predictor variables are used to predict a criterion variable. 5) Which of the following criteria is NOT used to judge the quality of a correlational study? a) The researcher used highly reliable measures. 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) The researcher explained what brought about a high correlation between two variables. c) The researcher used a high correlation to infer causation. d) The researcher selected an appropriate sample for the study. 6) Which of the following is NOT a characterization of ex-post-facto research? a) Group differences on the dependent variable cannot be compared because of the difference in the independent variable under investigation. b) One group is compared with another group based on one or more pre-existing conditions. c) The participants in a group are selected to be similar to the participants in another group except on the independent variable under investigation. d) Causation can be inferred even though the intervention cannot be controlled. 7) A study was conducted to investigate reading achievement of two large groups of randomly-selected second-grade students. The researcher found that children who came from homes where books were valued and where parents started reading to them from an early age were more likely to have higher reading scores than children who came from homes where reading was not considered important. Which of the following terms is used to describe this research design? a) A pretest-posttest study b) An ex-post-facto study c) An experimental study d) A longitudinal study 8) Which of the following is NOT a reason for the popularity of surveys? a) They are efficient methods for collecting data. b) They are versatile. c) Response rates are typically high. d) The results are usually generalizable. 9) A researcher studied the voting plans of a sample of voters during a gubernatorial election. At the beginning of the election campaign, half the sample supported the Republican candidate and the other half supported the Democratic candidate. The researcher measured the sample four times during the election campaign to determine if the voters had changed their opinions. At the end of the study, 20% of the sample supported the Republican candidate and 80% supported the Democratic candidate. Which of the following describes this research design? a) Cross-sectional survey research b) Causal-comparative research 21 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) Ex-post-facto research d) Longitudinal survey research 10) A researcher conducted a study examining the extent to which graduate students’ grades in an introductory research class can be predicted from their GRE test scores and their undergraduate GPA. Her sample consisted of 10 students from a class who volunteered for the study. Which of the following is one cause for concern about this study? a) The independent variables used b) The setting c) The sample size d) Reliability issues with the GRE test 11) A researcher is interested in studying the impact of attending preschool on firstgrade behavior. One approach is to compare first graders who attended preschool with first graders who did not. Which of the following terms refers to this type of research design? a) A predictive study b) A causal-comparative study c) A longitudinal survey d) A descriptive study 12) A correlational study examined the relationship between a large group of students’ overall achievement and their level of self-confidence. The results were reported as statistically significant (r = .17, p. = .01). With what should you be concerned? a) The theoretical importance of this study b) The lack of statistical significance of the results c) The lack of practical significance of the results d) The sample size

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Which of the following is NOT a characteristic of a causal-comparative design? a) The participants in one group might be matched with participants who are similar in another group. b) The researcher cannot regulate the independent variable. c) The researcher has direct control over the intervention. d) They suggest causal relationships by comparing groups. 2) Results of a descriptive study on the hyperactive behavior of fifth-grade students found that hyperactivity increased on those days when the students consumed a lot of candy. The researcher concluded that hyperactive behavior in 11- and 12-yearolds could be decreased by reducing or eliminating candy from the students’ diet. Which criterion for descriptive studies was violated in this example? a) The researcher should provide detailed descriptions of the subjects used in the study. b) Descriptive studies cannot be used to investigate behavior. c) The sample in the study was restricted to fifth graders. d) Descriptive studies do not assume causal relationships between variables. 3) Which of the following statements is NOT TRUE of relationship studies? a) Relationship studies can examine the differences between groups on a particular variable. b) Relationship studies investigate the effect of one variable on a second variable. c) Relationship studies identify causal relationships. d) Relationship studies are useful for making predictions. 4) A researcher investigated the relationship between SAT scores and college grades by randomly selecting fifty students from five colleges that have high SAT scores as one of their selection criteria. The results showed little relationship between SAT scores and college grades. Which of the following is a reason why the researcher may have failed to show a relationship? a) A homogeneous group of participants was selected for the study. b) His sample consisted of students who were randomly sampled. c) The SAT is not a reliable test. d) There is no relationship between SAT scores and college grades.

5) Which of the following is NOT a criterion used to evaluate causal-comparative research? a) It should only be used if it is not possible or feasible to conduct a true experiment. b) Extraneous variables that may affect the results should be considered by the researchers. c) The differences between groups should be controlled except on the independent variable of interest. d) The researchers should firmly establish causality. 6) A researcher studied the physical and academic development of premature babies. After the first observation at birth, he studied them every five years until age 20. This is an example of which type of research design? a) Cross-sectional study b) Longitudinal study c) Comparative study d) Ex-post-facto study 7) Which of the following is a true statement about cross-sectional survey research? a) A panel sample can be used to study the same individuals each time the data are collected. b) Observed changes over time can be due to other differences between the age groups. c) Results are based on the collection of data at one time. d) Results describe a causal relationship between two phenomena. 8) A correlational study examined the relationship between students’ prior achievement and their level of self-regulated learning. The results were reported as statistically significant (r = .13, p. = .03). With what should you be concerned? a) The theoretical importance of this study b) The sample size c) The data analysis d) The practical significance of the results 9) Results of a descriptive study on the cooperative behavior of sixth-grade students showed that children with opposite sex siblings displayed cooperativeness more often than children who did not have siblings of the opposite sex. The researcher concluded that 12- and 13-year-olds were more likely to learn to cooperate if there were opposite sex children in the family. Which criterion for descriptive studies was violated in this example? 24 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

a) Descriptive studies can describe participants or events, not behaviors. b) The researcher did not identify his criteria for cooperative behavior. c) Descriptive studies do not assume causal relationships between variables. d) The researcher only used fifth graders as participants in his study. 10) Which of the following is NOT a characterization of ex-post-facto research? a) One group is compared with another group based on a pre-existing condition. b) Causation can be inferred even though the intervention cannot be controlled. c) Group differences on the dependent variable cannot be compared because of the difference in the independent variable under investigation. d) The participants in a group are selected to be similar to the participants in another group except on the independent variable under investigation. 11) In a study on the math achievement of two groups of sixth-grade students, it was found that children who had siblings had higher math achievement than children who did not have siblings. This is an example of which type of research design? a) A cross-sectional survey b) Experimental research c) Ex-post-facto research d) A descriptive study 12) Which of the following is NOT a reason for the popularity of surveys? a) Response rates are typically high. b) They can address a wide range of problems. c) The results are usually generalizable. d) Surveys are usually less costly than interviews. Short Answer Questions: 1) How could you minimize nonresponse in a study that uses a survey to collect data? 2) Provide an example in which two variables are highly correlated, but the high correlation does not imply causation. Essay Question: Explain the difference between what is meant by practical significance and statistical significance. Provide an example of a statistically significant result that could be considered not of practical significance and vice versa.

Chapter 8 Test Answer Key ANSWER KEY: Chapter 8 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. d 2. b 3. a 4. d 5. c 6. a 7. b 8. c 9. d 10. c 11. b 12. c

INSTRUCTOR’S EXAM ITEMS 1. c 2. d 3. c 4. a 5. d 6. b 7. c 8. d 9. c 10. c 11. c 12. a

Chapter 9 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents

Chapter 9: Experimental Research Designs .................................................................. 1 Learning Outcome Quizzes ............................................................................. 1 Application Exercises .................................................................................... 15 Test Items ..................................................................................................... 21 Test Answer Key ........................................................................................... 28

Chapter 9: Experimental Research Designs Chapter 9 Learning Outcome Quizzes Learning Outcome 9.1: Explain why researchers need to MAXimize differences between interventions (reasonably), MINimize error, and CONtrol extraneous and confounding variables in designing good experimental research. [Q1] In experimental research, the researcher manipulates which of the following? 1. Some type of intervention [correct] 2. At least one dependent variable 3. All variables 4. At least one outcome [Feedback for Correct Answer 1] This characteristic differentiates experimental research from other quantitative research. Note that dependent variables (outcomes) are measured; they are not manipulated. [Feedback for Answer Choice 2] Dependent variables are outcome variables and thus not manipulated. [Feedback for Answer Choice 3] Not all variables are manipulated. Consider the role of a control variable. [Feedback for Answer Choice 4] Outcomes are measured, but not manipulated. [Q2] Which of the following research questions is best addressed by experimental research? 1. What are the writing behaviors of graduate students? 2. Is there a relationship between GRE Verbal scores and writing performance for graduate students? \

3. Does peer tutoring lead to better writing performance than online tutorials? [correct] 4. Do students who attend private schools write better than students who attend public schools? [Feedback for Answer Choice 1] 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This is a question that does not require an experiment, as only a descriptive study is needed. [Feedback for Answer Choice 2] Experimental studies are not designed to simply explore relationships. [Feedback for Correct Answer 3] The main goal of an experiment is to establish a cause and effect—peer tutoring causes changes in writing performance that are different than changes caused by online tutorials. [Feedback for Answer Choice 4] This question only requires a comparison of groups, not the manipulation of an independent variable. [Q3] What is the main goal of an experimental study? 1. Explore correlations 2. Establish cause and effect [correct] 3. Describe an understudied population 4. Compare previously existing data [Feedback for Answer Choice 1] Correlational research is designed to explore correlations. [Feedback for Correct Answer 2] The main goal of an experiment is to establish a cause and effect signal—to be able to say that the dependent variable was or was not clearly affected only by the intervention. [Feedback for Answer Choice 3] This would be appropriate for descriptive research, but not experimental. [Feedback for Answer Choice 4] Experiments are active and require variable manipulation. [Q4] In a well-designed experimental study that uses two or more groups, which of the following is vitally important? 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

1. The groups are not fundamentally different on any determinant variables [correct] 2. The groups are different in size 3. The groups have no variation whatsoever 4. The groups are fundamentally different from one another [Feedback for Correct Answer 1] It is important that groups are not fundamentally different on any key variables of the study’s focus. Statistical equivalence is important. [Feedback for Answer Choice 2] The groups should be as close in size as possible. [Feedback for Answer Choice 3] The groups may have slight variations, meaning it’s unlikely their scores on inclusion variables of focus will be exact, but they should be close. [Feedback for Answer Choice 4] The groups selected for a study should have strong commonalities. Learning Outcome 9.2: Distinguish among the threats to internal and external validity of experimental designs. [Q1] Which of the following could be a threat to the internal validity of an experimental study? 1. Differences in socioeconomic status [correct] 2. Manipulation of the independent variable 3. Too much control of the conditions of the study 4. Results that do not generalize outside of the sample [Feedback for Correct Answer 1] Internal validity is threatened by significant extraneous variables that have a confounding influence on the conditions of the experiment. For example, socioeconomic status is a powerful predictor of academic achievement and so the researcher may need to control for this when testing an educational intervention. [Feedback for Answer Choice 2] Manipulating the independent variable is the common practice in experimental research and is not problematic. 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] Controlling the conditions of the study only enhances the study. [Feedback for Answer Choice 4] Issues with external validity decrease generalizability. [Q2] In Katia’s remedial mathematics study, she collected pretest data from a group of participants. She divided the participants into three groups. One group received no treatment, one group received instruction by a teacher-delivered intervention, and one group received peer tutoring on the same problems. After the intervention she tested their math skills. With which validity threat should Katia be most concerned? 1. Maturation 2. Instrumentation 3. Pretesting [correct] 4. Participant attrition [Feedback for Answer Choice 1] Maturation occurs when participants mature or naturally change during a study. [Feedback for Answer Choice 2] In instrumentation, differences in results due to unreliability or changes in instruments, raters, or observers. [Feedback for Correct Answer 3] When participants are given a pretest, it may influence their responses on the posttest, regardless of the intervention. [Feedback for Answer Choice 4] Participant attrition refers to a loss of participants in a study. [Q3] In an experiment comparing the essay writing abilities of two groups of 12th-graders, two sets of scorers (one set for each group) were selected to evaluate the essays. What is the greatest threat to internal validity in this experiment? 1. Instrumentation [correct] 2. Intervention replication 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. Participant attrition 4. Statistical regression [Feedback for Correct Answer 1] In instrumentation, differences in results are due to unreliability or changes in instruments, raters, or observers. [Feedback for Answer Choice 2] There’s no problem here with the ability to replicate the intervention. [Feedback for Answer Choice 3] In this scenario, losing participants is not a noted issue. [Feedback for Answer Choice 4] Statistical regression (sometimes called regression toward the mean, regression effect, or regression artifact) refers to the tendency of groups of participants who score extremely high or low on a pretest to score closer to the mean on the posttest, regardless of the effect of the intervention. [Q4] Which of the following refers to the tendency of groups of participants who score extremely high or low on a pretest to score closer to the mean on the posttest, regardless of the effect of the intervention? 1. Participant attrition 2. Instrumentation 3. Maturation 4. Statistical regression [correct] [Feedback for Answer Choice 1] Participant attrition refers to a loss of study participants. [Feedback for Answer Choice 2] Instrumentation refers to differences in results due to unreliability or changes in instruments, raters, or observers. [Feedback for Answer Choice 3] Maturation simply refers to the reality, especially when working with children, that participants may mature and change during the study phase. 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Correct Answer 4] Statistical regression (sometimes called regression toward the mean, regression effect, or regression artifact) refers to the tendency of groups of participants who score extremely high or low on a pretest to score closer to the mean on the posttest, regardless of the effect of the intervention. [Q5] When designing a study, Peggy wants to ensure that her results can be generalized to other settings. What is she most concerned with? 1. External validity [correct] 2. Internal validity 3. Diffusion of intervention 4. Statistical regression [Feedback for Correct Answer 1] External validity is the degree to which study results are generalizable, or applicable, to groups and environments outside the experimental setting. [Feedback for Answer Choice 2] Internal validity refers to the extent to which the intervention, and not extraneous or confounding variables (noise), produced the observed effect. [Feedback for Answer Choice 3] In diffusion of intervention, intervention effects influence control or comparison groups. [Feedback for Answer Choice 4] Statistical regression (sometimes called regression toward the mean, regression effect, or regression artifact) refers to the tendency of groups of participants who score extremely high or low on a pretest to score closer to the mean on the posttest, regardless of the effect of the intervention. Learning Outcome 9.3: Identify and apply different types of experimental designs. [Q1] A researcher wants to study the effects of a new math curriculum and plans to assign certain math classes to try it for one semester while having the other classes use the usual curriculum. She plans to compare scores on the students’ final exams. The researcher is most likely to design a study using which of the following? 1. Nonequivalent-Groups Posttest-Only Design [correct] 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. Randomized-to-Groups Posttest-Only Design 3. Nonequivalent-Groups Pretest–Posttest Design 4. Single-Group Posttest-Only Design [Feedback for Correct Answer 1] The researcher has more than one group (different math classes). She does not appear to assign students to the classes, but rather seems to plan to identify alreadyestablished math classes, and she mentions only using the final exam as a measure. [Feedback for Answer Choice 2] This researcher does not appear to have a randomized sample. [Feedback for Answer Choice 3] There is no pretest mentioned in this design. [Feedback for Answer Choice 4] This researcher is working with more than one group. [Q2] Markel has conducted a study on the impact of soothing music on the sleeping habits of 5-year-olds. He has designed a study that first randomized participants into treatment and control groups. After implementing the intervention for a period of 8 weeks, he implemented a posttest. What type of study is Markel conducting? 1. Nonequivalent-Groups Posttest-Only Design 2. Randomized-to-Groups Posttest-Only Design [correct] 3. Nonequivalent-Groups Pretest–Posttest Design 4. Single-Group Posttest-Only Design [Feedback for Answer Choice 1] Markel does not have nonequivalent groups in this scenario. [Feedback for Correct Answer 2] Markel is working with randomized groups and he did not administer any type of pretest measure. [Feedback for Answer Choice 3] Markel has not administered any type of pretest. [Feedback for Answer Choice 4] 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Markel is working with multiple groups. [Q3] Ka’neshia wants to conduct a study using three existing classes at a local high school. She’s interested in how homework impacts motivation. She plans to administer a pretest, implement a homework intervention to the experimental groups, and finally administer a posttest. Which of the following is Ka’neshia most likely utilizing? 1. Nonequivalent-Groups Posttest-Only Design 2. Randomized-to-Groups Posttest-Only Design 3. Nonequivalent-Groups Pretest–Posttest Design [correct] 4. Single-Group Posttest-Only Design [Feedback for Answer Choice 1] Ka’neshia has used a pretest and does not have nonequivalent groups. [Feedback for Answer Choice 2] The groups in Ka’neshia’s study are already established and thus not random. [Feedback for Correct Answer 3] Ka’neshia’s groups are already intact, and thus not random, but she is using a pre- and posttest design. [Feedback for Answer Choice 4] Ka’neshia is working with more than one group and administering a pretest. [Q4] Which of the following is considered the weakest experimental designs? 1. Nonequivalent-Groups Posttest-Only Design 2. Randomized-to-Groups Posttest-Only Design 3. Nonequivalent-Groups Pretest–Posttest Design 4. Single-Group Posttest-Only Design [correct] [Feedback for Answer Choice 1] This is not the weakest experimental design. [Feedback for Answer Choice 2] This is not the weakest experimental design. 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] This is not the weakest experimental design. [Feedback for Correct Answer 4] This is the weakest experimental design because there is no other group for comparison as well as no pretest to establish any type of baseline. Learning Outcome 9.4: Understand criteria for evaluating experimental studies. [Q1] Which of the following is strong when evidence is presented that the enacted intervention is consistent with the intended intervention? 1. Intervention fidelity [correct] 2. Intervention validity 3. Intervention reliability 4. Intervention design [Feedback for Correct Answer 1] Intervention fidelity is strong when evidence is presented that the enacted intervention is consistent with the intended intervention. [Feedback for Answer Choice 2] This is not an issue of validity. [Feedback for Answer Choice 3] This is not an issue of reliability. [Feedback for Answer Choice 4] While design in general could have an impact, this is not the best term to use in this instance. [Q2] Which of the following is true regarding the evaluation of experimental research? 1. There should be direct control of the intervention. [correct] 2. There should be control of the intervention during the pretest phase only. 3. There should be control of dependent variable. 4. There should be no control of the intervention. 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Correct Answer 1] An essential feature of an experiment is that the researcher controls the intervention(s)—the nature and duration of what the participants experience. If it is not clear that such control has occurred, the ability to make causal interpretations is limited. [Feedback for Answer Choice 2] Nothing is manipulated in the pretest phase, just measured. [Feedback for Answer Choice 3] Dependent variables are outcome variables and not controlled. [Feedback for Answer Choice 4] There should be control of an intervention in an experimental study. [Q3] Which of the following variables should be controlled tightly in an experimental study in order to ensure threat reduction? 1. Dependent variables 2. Extraneous variables [correct] 3. Independent variables 4. Mediating variables [Feedback for Answer Choice 1] Dependent variables are outcome variables and thus not controlled. [Feedback for Correct Answer 2] The researcher should indicate how specific aspects of the design control possible extraneous and confounding variables. Obvious threats, such as selection in the nonequivalent-groups designs, need to be addressed. If obvious threats are not controlled by the design, the researcher should present a rationale for why a particular threat is not a plausible alternative explanation for the results. [Feedback for Answer Choice 3] The controlling of independent variables is not a tactic to reduce threat. [Feedback for Answer Choice 4] Controlling mediating variables does not impact the inherent threats to a study. [Q4] 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

In an experimental study, how many times should the intervention be replicated? 1. At least twice 2. The intervention is implemented only once 3. The intervention should not be replicated 4. As many times as there are participants in the study [correct] [Feedback for Answer Choice 1] This is not the correct answer. [Feedback for Answer Choice 2] Interventions are often replicated more than once. [Feedback for Answer Choice 3] Interventions should be replicated. [Feedback for Correct Answer 4] In reading an experimental study, you should look for the number of times the intervention is replicated, which should be the same as the number of participants in the study. Learning Outcome 9.5: Know characteristics and limitations of single-subject designs. [Q1] Which of the following questions is best studied with a single-subject research design? 1. Does singing a song at the start of day care reduce Emily’s misbehavior when her parent leaves? [correct] 2. Is a person’s political party affiliation related to their intelligence? 3. Does Mrs. Abelson’s class show improvement in creative writing over the course of the semester? 4. What types of ice cream does Akua prefer? [Feedback for Correct Answer 1] Single-subject designs, involve one or just a few participants and are used to study the influence of an intervention. Singing a song is the intervention in this case. [Feedback for Answer Choice 2] This study would include a significantly sized sample. 11 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] This study is focused on an entire class, which will inevitably contain multiple participants. [Feedback for Answer Choice 4] This is not a study, but a simple question. [Q2] Kelly wants to study the effect of writing daily goals on homework completion for a middle-school student who has been diagnosed with ADHD. She decides that she will first apply the strategy when he is in math, then in English, and finally in science. What design is Kelly planning to use? 1. An alternating treatments design 2. A multiple-baseline approach [correct] 3. A quasi-experimental design 4. An A – B – A withdrawal design [Feedback for Answer Choice 1] Kelly’s study does not mention any alternating of treatments. [Feedback for Correct Answer 2] In a multiple baseline design, the researcher studies multiple behaviors, participants, or settings (i.e., multiple classes) for the intervention. In this scenario, the researcher is doing a single-subject multiple-baseline design, with multiple settings (three different subjects). [Feedback for Answer Choice 3] There are not quasi-experimental components to Kelly’s study. [Feedback for Answer Choice 4] Kelly’s study does not involve giving, then removing, an intervention. [Q3] Which of the following terms refers to a period of time in which the target behavior (dependent variable) is observed and recorded as it occurs prior to the intervention? 1. Control 2. Intervention 12 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. Withdrawal 4. Baseline [correct] [Feedback for Answer Choice 1] Control is not a period of time, but rather an action. This is not the correct answer. [Feedback for Answer Choice 2] This does not refer to an intervention. [Feedback for Answer Choice 3] In order for a withdrawal period to occur, an intervention would already have to be implemented. [Feedback for Correct Answer 4] Baseline refers to a period of time in which the target behavior (dependent variable) is observed and recorded as it occurs prior to the intervention [Q4] Which of the following indicates a single-subject study where a baseline is established, followed by an intervention period, followed by a return to baseline? 1. A – B – A [correct] 2. B – A – B 3. A – B – A – B 4. A – B – C [Feedback for Correct Answer 1] In an A–B–A withdrawal design, the intervention is introduced after a number of observations of the baseline behavior, and is stopped to return to the same condition that was present during the original baseline measurement. The design allows a strong causal inference if the pattern of behavior changes with the addition and withdrawal of the intervention. [Feedback for Answer Choice 2] This is not a model of single-subject design. [Feedback for Answer Choice 3] This would indicate two intervention periods. [Feedback for Answer Choice 4] 13 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This would indicate an extension of the research questions and is not the correct answer.

Chapter 9 Application Exercises Chapter 9: Experimental Research Designs Application Exercise 9.1: Threats to Validity of an Experiment Learning Outcome 9.2: Distinguish among the threats to internal and external validity of experimental designs. [Q1] Consider the following: Kaitlyn is an undergraduate student interested in studying peer tutoring programs for a class project. She hypothesizes that children who are tutored by older children will be more cooperative and will learn more than children who are older than their tutors. She designs an experimental study in which one group of low-achieving seventh-grade children will be tutored by ninth-graders, and one group will be tutored by high-achieving sixth-graders. Given her plan, think about how she can ensure high validity in her study. Kaitlyn knows that good experimental research requires both random selection and random assignment. Explain the difference and describe what Kaitlyn must do so that she meets both requirements. [Q1 Model Response] Random selection is the process by which a sample of participants is selected from the population. Kaitlyn’s target population seems to be “children” (she is not very specific, unfortunately); her accessible population is probably low-achieving seventh-graders in her local school district, but she also probably has limited resources for a class project and may focus on just one school. To select a random sample, she should get a list of all seventh-graders, work with school administrators and/or teachers to identify those who are considered low achievers, and then use a table of random numbers to select a sample of the appropriate size, as described in Chapter 5. If she decides to stratify based on sex, she should randomly select from the population of girls and from the population of boys. Random assignment is the process by which the researcher determines which of the selected participants get each treatment. Kaitlyn may decide to flip a coin, roll a die, draw cards, or use a random number table to assign participants to the younger-tutor or older-tutor group. [Q2] Consider the following: Kaitlyn is an undergraduate student interested in studying peer tutoring programs for a class project. She hypothesizes that children who are tutored by older children will be 15 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

more cooperative and will learn more than children who are older than their tutors. She designs an experimental study in which one group of low-achieving seventh-grade children will be tutored by ninth-graders, and one group will be tutored by high-achieving sixth-graders. Given her plan, think about how she can ensure high validity in her study. What are some variables Kaitlyn will need to control in her study? [Q2 Model Response] A number of responses are possible here. Most important, Kaitlyn should ensure that the tutoring is equivalent for both groups—that may not be easy, given that the tutors in one group are significantly older than the tutors in the other group. She should check to make sure the materials and environments are similar; perhaps she can produce a script or protocol for the tutors to follow. Other things for her to consider include whether the participants in each group are about equal in terms of achievement and cooperation (both of which need to be operationally defined) before the tutoring begins, the amount of time spent in tutoring, and whether students engage in outside activities that interact with the tutoring (e.g., does one group study more), among others. [Q3] Consider the following: Kaitlyn is an undergraduate student interested in studying peer tutoring programs for a class project. She hypothesizes that children who are tutored by older children will be more cooperative and will learn more than children who are older than their tutors. She designs an experimental study in which one group of low-achieving seventh-grade children will be tutored by ninth-graders, and one group will be tutored by high-achieving sixth-graders. Given her plan, think about how she can ensure high validity in her study. Kaitlyn plans to give all children a pretest and a posttest to assess their content knowledge, and she plans to have teachers evaluate cooperativeness both before and after the tutoring program, which will last 4 weeks. Which of the major threats to validity should Kaitlyn be concerned about, and why? [Q3 Model Response] •

Pretesting could be a threat—doing poorly on a pretest on content knowledge could subtly encourage students to study, for example.

•

Instrumentation could also be a threat—Kaitlyn will need to operationalize cooperativeness, for example, and will need to train her teacher–evaluators so they are reliable in their assessment of students’ cooperativeness.

•

Because she is working with low-achieving children, statistical regression could be a threat, depending on her operational definition of low achievement; to reduce this 16 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

threat, she should define achievement based on multiple measures rather than only one. •

Participant attrition could be a problem; students may drop out of the tutoring program (especially if they are uncomfortable being tutored by younger children).

•

Diffusion of intervention could also be a threat, for example, if some students are upset at being tutored by younger children.

•

Experimenter effects are a threat, especially in the group with younger tutors, who may make the students feel uncomfortable or not very smart.

•

Participant effects may be a threat, especially if students feel that they have something to prove after having been labeled “low achievers” (if they are aware of that classification).

Because Kaitlyn’s program is short (i.e., only a few weeks), history and maturation are not likely to be major threats. Differential selection is not likely to be a problem if she uses random selection and assignment in her design, and her pretest measure should help her determine if it is a problem. Application Exercise 9.2: Identifying Experimental Designs Learning Outcome 9.3: Identify and apply different types of experimental designs. [Q1] Review the description of the following study and identify the research design that the researchers used. Richardson and Brouillette (2013) were interested in whether students in American History classes would benefit from having historic and ethnic music included in the curriculum. For their study, they identified a school and divided the eighth-graders into seven different social studies classes. In three of the classes, the usual curriculum was used, which emphasized text-based readings and class discussion. In the other four, the teachers used a program known as Mapping the Beat—a social-studies curriculum that integrates art and music, geography, and history. Students’ geographical knowledge and their attitudes toward geography were measured at the start and at the end of the 9-week term. What type of research design is this? [Q1 Model Response] This design is best described as a nonequivalent groups pretest–posttest design with multiple dependent variables. The classes are intact (but nonequivalent) groups, even though most students were randomly assigned to a particular class period. Geographical knowledge is one dependent variable; attitudes toward geography is another. You can learn more about the study by searching for it on ERIC. 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q2] Review the description of the following study and identify the research design that the researchers used. Filippatou and Kaldi (2010) were interested in whether problem-based learning affects academic performance, self-efficacy, and other beliefs and behaviors of students with learning difficulties. They conducted a mixed methods study. For the quantitative phase, they identified 24 students, enrolled in one of the six inclusive classes at the school, who were diagnosed as having a learning difficulty. Each target student took a series of pretests, which included tests of their basic knowledge of sea animals and a survey to assess their attitudes toward learning. They then participated in an 8-week program in which they learned about sea animals, after which they completed the same tests and surveys. What design did the researchers use for the quantitative phase of this study? [Q2 Model Response] This is a pre-experimental design, in particular a single-group pretest–posttest design. Although the students were enrolled in six different classes, class was not a variable in the study—the students all received the same treatment (the program on animals) and the same pretest and posttest measures. You can learn more about the study by searching for it on ERIC. [Q3] Review the description of the following study and identify the research design that the researchers used. Kale and Whitehouse (2012) were interested in the effects of video technology on the problem-solving skills of preservice teachers. They recruited 44 students in elementary education programs and 36 students in secondary education programs to participate. They then divided the participants into one of three conditions. In the “Entire” condition, the participants saw a video case that showed a classroom teacher in the classroom, answered some questions, and then saw another video including the teacher’s reflections, and answered more questions. In the “Segmented” condition, the participants saw the classroom video, split into two different sections, and then saw the entire reflection video, with questions after each section. In the “Sequenced” condition, the participants saw one segment of the class video, then one segment of the reflection video, then the second section of the class video, and then the second section of the reflection video, with questions after each section. Answers were coded to reflect various stages of problem solving. What design did the researchers use?

[Q3 Model Response] This is a factorial design, with two factors: grade level and video presentation. This is sometimes written as a 2 (grade level: elementary, secondary) × 3 (video presentation: Entire, Segmented, Sequenced) factorial design. You can learn more about the study by searching for it on ERIC. Application Exercise 9.3: Single-Subject Research Learning Outcome 9.5: Know characteristics and limitations of single-subject designs. [Q1] Consider the following scenario: Jessica is in your life-skills class and needs the content you teach, but she refuses to attend class regularly, and when she does attend, she does not participate in class activities. You would like to put an intervention in place to increase her participation. You ask the school psychologist to come by class and observe Jessica's current level of participation each day as you implement an A – B – A A withdrawal design. Given this scenario, how might you employ an A – B – A withdrawal design? [Q1 Model Response] In an A – B – A withdrawal design, you would first collect baseline data, then introduce the treatment (in this case, reinforcement for participation), and then return to baseline to evaluate the impact of the intervention. [Q2] Consider the following scenario: Jessica is in your life-skills class and needs the content you teach, but she refuses to attend class regularly, and when she does attend, she does not participate in class activities. You would like to put an intervention in place to increase her participation. You ask the school psychologist to come by class and observe Jessica's current level of participation each day as you implement an A – B – A withdrawal design. What role will statistics play in the analysis of your data? [Q2 Model Response] In most typical single-subject designs, the researcher will not utilize inferential statistics because they have only one participant and are not trying to draw statistical inferences. [Q3] Consider the following scenario: 19 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Jessica is in your life-skills class and needs the content you teach, but she refuses to attend class regularly, and when she does attend, she does not participate in class activities. You would like to put an intervention in place to increase her participation. You ask the school psychologist to come by class and observe Jessica's current level of participation each day as you implement an A – B – A withdrawal design. What is one major limitation to Jessica’s approach? [Q3 Model Response] One limitation of the A – B – A design is the difficulty in interpreting a positive change that is not altered during the second baseline. In this situation, the intervention may be so strong that its effect lasts a long time, or something else may have occurred with the intervention that affected the behavior and did not stop when the intervention did.

Chapter 9 Test Items SELF-ASSESSMENT ITEMS 1) In experimental research, which of the following variables are controlled? a) Only extraneous variables b) Only independent variables c) Independent and dependent variables d) Independent and extraneous variables 2) An experimental study has strong internal validity if which of the following is true? a) The independent variable alone can account for the results. b) The results of the experiment are generalizable. c) A factor is present in different amounts or intensity across all levels of the independent variable. d) Something other than the intervention is responsible for the effect. 3) Nonrandom assignment to treatments is what type of threat to internal validity? a) Experimenter effects b) Selection c) Maturation d) Diffusion of treatment 4) Natural changes in participants during the course of the experiment are what type of threat to internal validity? a) Participant attrition b) Pretesting c) Maturation d) Statistical regression 5) In an experiment comparing the story-writing abilities of two groups of fourth graders, two sets of scorers (one set for each group) were selected to evaluate the stories. What threat to internal validity has occurred? a) Diffusion of treatment b) Instrumentation c) Selection d) Treatment replication 6) A study has strong external validity if which of the following is true? 21 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

a) The findings can be generalized to other participants and settings. b) The intervention produced the desired effect. c) The researcher provides evidence that extraneous variables were controlled. d) The findings support a theory. 7) Mr. Ramsey had a serious concern about a study that concluded that cooperative group instruction was effective for all secondary students even though there were very few female students in the experimental group and a majority of the participants were female in the control group. Which of the following terms describes Mr. Ramsey’s concern? a) Participant attrition b) Selection c) Measurement effects d) Operational definitions 8) A teacher wanted to evaluate the efficacy of a new technique for teaching comprehension skills to her class of students with learning disabilities. She administered a comprehension test before introducing the new technique. At the end of the semester, she administered the test again and compared the results from the two tests. This study is an example of which of the following? a) Factorial experimental design b) Nonequivalent-groups pretest–posttest design c) Randomized-groups pretest–posttest design d) Single-group pretest-posttest design 9) Which of the following is a true statement about the single-group pretest–posttest design? a) There are more potential threats to internal validity as the time between pretest and posttest increases. b) The design is the weakest experimental design because it only examines a single group. c) The design is strengthened if there are no similarities between the pretest and posttest. d) The design cannot be used for studies in which participant characteristics unique to the individual influence results. 10) Which of the following is a serious threat to the internal validity of the nonequivalentgroups posttest-only design? a) Without a pretest, it is difficult to control for selection differences. 22 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Treatment group results cannot be compared to control group results. c) The lack of a control group makes it difficult to draw conclusions. d) Random selection of participants to each group can result in equivalent groups. 11) To assess the efficacy of the new math curriculum, Ms. Jackson conducted a small experiment using two of her four sixth-grade math classes. She gave both classes a pretest, then taught Class A using the new curriculum and continued using the old curriculum with Class B. At the end of three months, she administered a posttest to both classes. Which design did Ms. Jackson use in her experiment? a) Factorial experimental b) Multiple baseline c) Nonequivalent-groups pretest–posttest d) Randomized-groups pretest–posttest 12) Which of the following is NOT a threat to internal validity in the randomized-groups posttest-only and the randomized-groups pretest–posttest designs? a) Diffusion of treatment b) History c) Selection d) Treatment replication 13) Dr. Garcia used a factorial design to investigate the effects of using supplemental PowerPoint presentations on males and females regarding their mastery of course objectives. Which of the following effects can be examined? a) Main effect for instructional strategy b) Main effect for gender c) Interaction effect for instructional strategy and gender d) All of the above 14) Which of the following is NOT a potential threat to external validity? a) Nature and type of measure b) Characteristics of participants c) Characteristics of the setting d) Random assignment of participants to treatments

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) In experimental research, the researcher tries to keep which variable constant? a) The dependent variable b) The contextual variable c) The extraneous variable d) The independent variable 2) Which of the following is a criterion for determining the extent to which experimental research has strong external validity? a) A factor is present in different amounts and/or intensity across the levels of the independent variable. b) The results can be generalized to other individuals, measures, procedures, or settings. c) The manipulation of the experimental variable alone can account for the results. d) The instrument used in the study was valid. 3) Ninth graders in a study changed their behavior simply because they were aware they were participants in a research study. This is an example of which type of threat to internal validity? a) Statistical regression b) Treatment replication c) Participant effects d) Instrumentation 4) An achievement test was used to collect data on a group of fourth-grade students. If a fire drill occurred in the middle of the test, this would be an example of which type of threat to internal validity? a) Statistical regression b) History c) Participant effects d) Instrumentation 5) A study compared the results from an experiment. Participants in the treatment group were selected from a private school with strong parental support and participants in the control group were selected from a public school without strong parental support. This is an example of which type of threat to internal validity? a) Participant effects 24 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Instrumentation c) Selection d) Pretesting effect 6) Eighth graders in a control group were resentful of the treatment offered to their classmates in the intervention group. This is an example of which type of threat to internal validity? a) Diffusion of intervention b) Instrumentation c) Statistical regression d) History 7) A researcher studied the comparative efficacy of two methods for teaching the metric system to sixth graders. The researcher assigned Method A to the honors math class and Method B to the regular math class. What threat to internal validity has occurred? a) Diffusion of treatment b) Instrumentation c) Selection d) History 8) The extent to which the intervention, and not extraneous or confounding variables, produced the observed effect in an experimental study is related to what aspect of a study? a) Content validity b) External validity c) Internal validity d) Experimental validity 9) Which of the following is a potential threat to external validity? a) Changes in behavior generated by participation in a study b) Unplanned events that occur during the study c) Intervention effects influencing the control group d) Characteristics of the setting in which the information is collected 10) What is the major weakness of the single-group posttest-only design? a) It does not control for extraneous variables. b) It is difficult to determine if the treatment or something else is responsible for the change in the participants’ behavior. 25 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) Results from a study using this design are not generalizable. d) The internal validity is threatened by selection because of the unique characteristics of the participants. 11) Which of the following is a true statement about single-group pretest–posttest design? a) The design is the strongest experimental design because it only examines a single group. b) The design is strengthened if there are no similarities between the pretest and posttest. c) The design will have more potential threats to internal validity as the time between pretest and posttest increases. d) The design becomes more valid as the experimental situation becomes less controlled. 12) What is a major internal validity concern in a nonequivalent-groups pretest–posttest design? a) Generalizability b) Selection c) Statistical regression d) Randomization 13) Which of the following is NOT a characteristic of a true experimental design? a) Stronger internal validity b) Stronger external validity c) Random assignment d) Control over extraneous variables 14) Factorial experimental studies are designed to enable researchers to do which of the following? a) Control all extraneous variables b) Randomly assign intact groups to the treatment and control groups c) Determine if the effects are consistent across participant characteristics d) Study the relationship between two or more dependent variables 15) Which of the following criteria is NOT used to evaluate a single-subject research study? a) Detailed descriptions of participants and settings are provided by the researcher. b) A single, standardized intervention is used. 26 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) Statistically significant results are also practically significant. d) The results are generalizable. Short-Answer Questions 1) What is intervention fidelity? Describe two ways it can be evaluated. 2) What is the primary purpose of experimental research? Essay Question Describe the main differences between experimental and nonexperimental research and provide an example of each type.

Chapter 9 Test Answer Key ANSWER KEY: Chapter 9 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. d 2. a 3. b 4. c 5. b 6. a 7. b 8. d 9. a 10. a 11. c 12. c 13. d 14. d

INSTRUCTOR’S EXAM ITEMS 1. d 2. b 3. c 4. b 5. c 6. a 7. c 8. c 9. d 10. b 11. c 12. b 13. b 14. c 15. d

Chapter 10 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 10: Understanding Statistical Inferences ...................................................... 1 Learning Outcome Quizzes ........................................................................ 1 Application Exercises ................................................................................ 13 Test Items ................................................................................................. 19 Test Answer Key ....................................................................................... 27

Chapter 10: Understanding Statistical Inferences Chapter 10 Learning Outcome Quizzes Learning Outcome 10.1: Describe the nature of estimation based on sampling and measurement error. [Q1] Inferential statistics primarily help researchers do which of the following? 1. Prove the research hypothesis 2. Determine the extent to which they can generalize to a population based on a sample [correct] 3. Calculate the descriptive information from a sample 4. Provide indisputable evidence for suggesting scientific theories [Feedback for Answer Choice 1] Inferential statistics never set out to “prove,” but rather “suggest” based on probability modeling. [Feedback for Correct Answer 2] Inferential statistics help researchers to know whether they can generalize to a population of individuals based on information obtained from a limited number of research participants based on likelihood. [Feedback for Answer Choice 3] This would be the role of descriptive statistics. [Feedback for Answer Choice 4] Inferential statistics operate on the concept of probability. They are never “indisputable.” [Q2] Inferential statistics are used to make inferences on which of the following? 1. A sample 2. A population [correct] 3. A single participant 4. A subset of study participants [Feedback for Answer Choice 1] 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Inferential statistics are not used to draw inferences about a particular sample. [Feedback for Correct Answer 2] Inferential statistics allow researchers to conduct sampling procedures and then draw inferences about larger populations. [Feedback for Answer Choice 3] Inferential statistics do not focus on any single participants. [Feedback for Answer Choice 4] Inferential statistics do not focus on any subset of participants. [Q3] Which of the following may contribute to uncertainty in the findings of inferential studies? 1. Mathematical errors 2. Sampling and measurement error [correct] 3. Poorly formed research questions 4. Exclusion of qualitative questioning in quantitative studies [Feedback for Answer Choice 1] Math errors certainly happen, but this is rare. Quantitative researchers use sophisticated statistical analysis software, and their work is checked many, many times. [Feedback for Correct Answer 2] Sampling and measurement error do happen, and they should always be considered when conducting inferential studies. [Feedback for Answer Choice 3] While it is true that poor questions lead to poor data, it’s unlikely that peer-reviewed studies make it through the peer-review process if the questions are poorly formed. [Feedback for Answer Choice 4] Sometimes mixed methods approaches are better, but other times, a well-designed quantitative study is more than sufficient. This, as always, depends on the research question. [Q4] Which of the following terms best describes the function of inferential statistics? 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

1. Conclude 2. Determine 3. Prove 4. Estimate [correct] [Feedback for Answer Choice 1] Remember that inferential statistics function on probability modeling, so drawing unquestionable conclusions is unlikely. [Feedback for Answer Choice 2] Determining implies finality. Inferential statistics are based on probability. [Feedback for Answer Choice 3] Always remember that the goal of inferential statistics is never to prove a particular finding. [Feedback for Correct Answer 4] Inferential statistics are essentially estimation models based on probability principles. Learning Outcome 10.2: Understand the use of the null hypothesis and level of significance with inferential statistics. [Q1] Which of the following is an example of a null hypothesis? 1. Teachers receiving holistic stress reduction training will have similar stress scores following training as those not receiving such training. [correct] 2. Teachers receiving holistic stress reduction training will be less stressed than teachers who do not receive such training. 3. Teachers receiving holistic stress reduction training will score significantly lower on a stress test than teachers who do not receive such training. 4. Teachers receiving holistic stress reduction training and teachers who do not receive such training will not differ. [Feedback for Correct Answer 1] This statement represents a null hypothesis because it specifies no difference between the specified groups on the outcome variable. [Feedback for Answer Choice 2] 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This answer choice indicates significant differences between groups; therefore, this is not an appropriate null hypothesis. [Feedback for Answer Choice 3] This answer choice indicates significant differences between groups; therefore, this is not an appropriate null hypothesis. [Feedback for Answer Choice 4] While this answer choice may seem correct, remember that groups can have slight variation and still not be significantly different from a statistical standpoint. [Q2] Which of the following represents the best definition of hypothesis testing? 1. The process of proving the research hypothesis 2. The process of accepting the null hypothesis 3. The process of decision making in which researchers evaluate the results of a study against their original expectations [correct] 4. The process of determining whether to reject the research hypothesis in favor of the null hypothesis [Feedback for Answer Choice 1] Researchers do not seek to prove findings. Remember that hypothesis testing is based on probability modeling. [Feedback for Answer Choice 2] While the null hypothesis is a vital component of a research study, it is the rejection of the null hypothesis that would indicate group differences. [Feedback for Correct Answer 3] Hypothesis testing is this process of decision making in which researchers evaluate the results of a study against their original expectations. [Feedback for Answer Choice 4] This is a part of the process, but not the definition of hypothesis testing. [Q3] In relation to hypothesis testing, alpha, symbolized as α, is best defined as which of the following? 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

1. A criterion to determine whether to reject or fail to reject the null hypothesis [correct] 2. A criterion to determine whether to reject or fail to reject the research hypothesis 3. A criterion preselected to be .05 4. A criterion to determine the researcher’s level of confidence in the results [Feedback for Correct Answer 1] Alpha represents the probability level that serves as a criterion to determine whether to reject or fail to reject the null hypothesis (remember, we never accept the null hypothesis). [Feedback for Answer Choice 2] It is not the research hypothesis that is rejected of failed to be rejected. [Feedback for Answer Choice 3] Alpha is often set at .05, but certainly not always. [Feedback for Answer Choice 4] This would be called a confidence internal. [Q4] Ms. Landry has set a very conservative alpha level of .001 for her analysis. She is likely concerned about which of the following? 1. Type 1 Error [correct] 2. Confidence intervals 3. Standard error 4. Effect size [Feedback for Correct Answer 1] Setting alpha to a small number reduces the chance of rejecting the null hypothesis and protects against a Type 1 error. However, it raises the chances of a Type 2 error. [Feedback for Answer Choice 2] Confidence intervals would be impacted by changes in alpha, but these were not Ms. Landry’s primary concern if she’s setting such a conservative value for alpha. [Feedback for Answer Choice 3] The standard error would not be impacted by Ms. Landry’s choice of alpha. 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 4] Effect size is not impacted by alpha. [Q5] Kathy gathered data about participants' level of self-efficacy. She conducted a hypothesis test to determine whether there is a difference between boys and girls, using a level of significance of .05. When she computed the statistic, she obtained a p value of .04. Which of the following statements best reflects her interpretation of the findings? 1. She rejects the null hypothesis and concludes there is a significant difference. [correct] 2. She accepts the null hypothesis and concludes there is no difference. 3. She fails to reject the null hypothesis and concludes there is a significant difference. 4. She fails to reject the null hypothesis and concludes there is no difference. [Feedback for Correct Answer 1] Because she found that p is less than her alpha, she is able to reject the null hypothesis. Because the null hypothesis is “no difference,” when she rejects it, she concludes that it is statistically probable that boys and girls have different levels of selfefficacy. [Feedback for Answer Choice 2] Keep in mind that null hypotheses are never “accepted.” [Feedback for Answer Choice 3] Failing to reject the null hypothesis would assume that there is no statistical difference. [Feedback for Answer Choice 4] Remember that in order to fail to reject a null hypothesis, the p-value must be greater than alpha. Learning Outcome 10.3: Explain why effect size and other indicators of practical significance are essential in communicating the results of studies. [Q1] In an experiment, an effect size of .01 indicates which of the following? 1. On average, the experimental and control groups performed about the same. [correct] 2. There is no correlation between the experimental group and the control group. 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. On average, the control group performed better than the experimental group. 4. The data had very little variability. [Feedback for Correct Answer 1] Effect size is a numerical way of expressing the strength or magnitude of a reported relation but does not indicate causality. [Feedback for Answer Choice 2] Effect sizes are not indications of correlation. [Feedback for Answer Choice 3] Effect size is not used to compare groups. [Feedback for Answer Choice 4] Effect sizes are not indicators of variation in a dataset. [Q2] A 95% confidence interval would indicate that the population mean falls within how many standard deviations of the sample mean? 1. 0 2. 1 3. 2 [correct] 4. 3 [Feedback for Answer Choice 1] This is not the correct number of standard deviations. [Feedback for Answer Choice 2] This is not the correct number of standard deviations. [Feedback for Correct Answer 3] Confidence intervals use the standard error of the mean and assume a normal distribution. You may recall the 68–95–99.7 rule. This rule indicates that 95% of scores in a normal distribution fall within + and − 2 standard deviations. [Feedback for Answer Choice 4] This is not the correct number of standard deviations. [Q3] 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Which of the following terms refers to a way of quantifying the degree of difference between two groups? 1. Significant difference 2. Effect size [correct] 3. Confidence interval 4. P-value [Feedback for Answer Choice 1] Significant differences between groups are calculated using p-values, but these values simply help a researcher understand whether a difference exists. [Feedback for Correct Answer 2] The term effect size refers to a way of quantifying the degree of difference between two groups. [Feedback for Answer Choice 3] A confidence interval provides a range of values in which the population or “real” trait value lies, within the parameters of specific probabilities. [Feedback for Answer Choice 4] The p-value is simply the probability of the data, given the null hypothesis is true. [Q4] Given the following data, calculate Cohen’s d. Mean 1 = 90 Mean 2 = 80 Standard Deviation (pooled) = 6 1. 1.67 [correct] 2. 1 3. .67 4. .2 [Feedback for Correct Answer 1] The formula for Cohen’s d is: d =

X1 - X 2 SD 8

When using these provided values, a value of 1.67 is calculated. [Feedback for Answer Choice 2] This is not the correct calculation. [Feedback for Answer Choice 3] This is not the correct calculation. [Feedback for Answer Choice 4] This is not the correct calculation. Learning Outcome 10.4: Know how to use specific inferential tests to analyze data, including t-tests, analysis of variance, chi-square, and multivariate statistics. [Q1] Tyler randomly assigns students into one of two reading programs and wants to compare the two groups on their test scores at the conclusion of the program. What type of t-test is appropriate? 1. The one-sample t-test 2. The matched t-test 3. The dependent-samples t-test 4. The independent samples t-test [correct] [Feedback for Answer Choice 1] Tyler has more than one sample in his study. [Feedback for Answer Choice 2] The participants in this study are not repeating measures in order to be paired. [Feedback for Answer Choice 3] Tyler’s study involves two standalone groups; therefore, the dependent samples t-test is not appropriate. [Feedback for Correct Answer 4] The t-test for independent samples is used in designs in which there are different individuals in each group. Students are in only one group. Tyler is testing the null hypothesis that scores of two groups are the same. [Q2] 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Which of the following statistical tests would be used to test whether there is a difference in participation in school sports (those who participate and those who do not) for boys and girls? 1. Chi-square [correct] 2. Independent-samples t-test 3. Simple analysis of variance 4. Pearson product-moment correlation coefficient [Feedback for Correct Answer 1] Chi-square ( c 2 ) is the statistical test to compare frequencies occurring in different categories or groups. [Feedback for Answer Choice 2] A t-test is used to determine differences between groups, but this question is not asking about group differences. [Feedback for Answer Choice 3] ANOVA is used to determine differences between groups, but this question is not asking about group differences. [Feedback for Answer Choice 4] The research question is not asking about ways the groups are similar, so a correlation coefficient is not helpful. [Q3] Cheryl has the mean math achievement test scores for all fifth-grade students in her school. She would like to find out how scores differ between students who receive special education services, students identified as gifted, and typical students who do not qualify for any special services. What sort of test should she run? 1. Analysis of covariance (ANCOVA) 2. Simple analysis of variance (ANOVA) [correct] 3. Chi-square test of independence 4. Factorial analysis of variance [Feedback for Answer Choice 1] ANCOVA is used in experimental studies in which participant differences between groups prior to an intervention can be used to statistically adjust the dependent variable 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

results so the analysis is more powerful and accurate. This is not necessary in this study. [Feedback for Correct Answer 2] Cheryl is comparing the scores from three groups of students, but she has only one independent variable (classification of student by type of service) and only one dependent variable (scores on the math achievement test). [Feedback for Answer Choice 3] Chi-square is useful in studies where the null hypothesis is that there is no difference between an observed number and an expected number of responses or cases that fall in each category. This does not match the situation in this question. [Feedback for Answer Choice 4] Factorial designs have more than one independent variable and allow the investigation of interactions among the independent variables. This is not necessary for this study. [Q4] Jeremiah is interested in comparing three groups on pre- and posttest scores related to anxiety. However, he is not able to obtain a random sample, and one group is beginning at a lower baseline than the other. He would like to control for these differences by adjusting posttest scores statistically, based on the initial group differences. Which statistical test should he use? 1. T-test 2. ANOVA 3. ANCOVA [correct] 4. Chi-square [Feedback for Answer Choice 1] T-tests only include two groups and do not account for initial group differences. [Feedback for Answer Choice 2] ANOVA does not account for initial group differences. [Feedback for Correct Answer 3] Jeremiah has to deal with covariates in this situation. ANCOVA will allow him to account for the initial group differences, thus increasing the power of his findings. [Feedback for Answer Choice 4] 11 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Chi-square is useful in situations of frequency. It is not appropriate for this research question.

Chapter 10 Application Exercises Chapter 10: Understanding Statistical Inferences Application Exercise 10.1: The Nation’s Report Card Learning Outcome 10.2: Understand the use of the null hypothesis and level of significance with inferential statistics. [Q1] The U.S. Department of Education has within it an institute known as the National Center for Education Statistics (NCES), which collects, analyzes, and disseminates information about student learning. The National Assessment of Educational Progress (NAEP) is one ongoing study of what students in America know, and the results are published as part of the Nation’s Report Card. This activity allows you to review some of the results of this study and reflect on the statistical techniques that were used to determine them. The Nation’s Report Card for 2019 in the area of reading focused on one research question: Are the nation’s 12th grader students making progress in reading? To answer this question, they analyzed recent student scores on standardized reading tests and compared scores from 2019 to scores from 1992, 1998, 2009, and 2017. This is a cross-sectional study. State the null hypotheses for these research questions. [Q1 Model Response] The null hypothesis is that students’ scores on the 2019 reading tests will not differ from scores in 1992, 1998, 2009, or 2017. [Q2] The U.S. Department of Education has within it an institute known as the National Center for Education Statistics (NCES), which collects, analyzes, and disseminates information about student learning. The National Assessment of Educational Progress (NAEP) is one ongoing study of what students in America know, and the results are published as part of the Nation’s Report Card. This activity allows you to review some of the results of this study and reflect on the statistical techniques that were used to determine them. The Nation’s Report Card for 2019 in the area of reading focused on one research question — Are the nation’s 12th graders making progress in reading? To answer this question, they analyzed recent student scores on standardized reading tests and 13 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

compared scores from 2019 to scores from 1992, 1998, 2009, and 2017. This is a cross-sectional study. Identify the independent and dependent variables in this study based on this information. [Q2 Model Response] The independent variable is test year, and it has 5 levels (1992, 1998, 2009, 2017, and 2019). The dependent variable is reading test scores. In this study, the reading scores were not compared to math scores, so content discipline (reading or math) is not a variable. [Q3] The U.S. Department of Education has within it an institute known as the National Center for Education Statistics (NCES), which collects, analyzes, and disseminates information about student learning. The National Assessment of Educational Progress (NAEP) is one ongoing study of what students in America know, and the results are published as part of the Nation’s Report Card. This activity allows you to review some of the results of this study and reflect on the statistical techniques that were used to determine them. For this exercise, we will use the data sets available from the 2013 report card as they are reported comprehensively. Reviewing the Nation’s Report Card from 2013, you can learn that possible scores on the math test ranged from 0 to 300. In 2013, males scored an average of 155, whereas females scored an average of 152. Standard errors for both groups were 0.06. Assume a null hypothesis that the scores of the male students do not differ from the scores of the female students. When the proper statistic is computed, we find p = .0053. First, state which statistic a researcher would need to use to test this hypothesis — ttest, analysis of variance, or chi-square. Then, explain what p = .0053 means. [Q3 Model Response] The researchers must have used an independent samples t-test for this analysis. The independent-samples t-test is used in designs in which (1) there is one dependent variable (i.e., math test score); (2) the scores of two groups are being compared (i.e., males vs. females); and (3) there are different individuals in each group—in this case, the males were in one group and the females were in another group. In statistical terms, p = .0053 indicates that there is a 53/10,000 chance that the difference between the males’ and the females’ scores (3 points) is simply due to 14 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

chance, and that is very small—less than .05, and so “significantly different.” In practical terms, we can say that the male students scored “significantly” higher than the female students on the 2013 math test. Note that doesn’t mean “a lot higher”—only that the difference is statistically reliable. A quick review of the nature of inferential statistics and hypothesis testing: By “due to chance,” researchers mean (1) we do not know the population score—that is, we don’t know the score for every single 12th grader in the nation; (2) we took a sample from that population, but it’s just a sample and so likely has some error; and (3) we used statistics to estimate the likelihood that this particular sample can be used to infer something “real” about the population. In this case, the probability is very low that the scores of this sample are very different from the whole population (only 53 times in 10,000 would we find a 3-point difference, due simply to chance), and so we can feel confident that we have identified a “real” difference between male and female students. Application Exercise 10.2: Making Decisions with Inferential Statistics Learning Outcome 10.2: Understand the use of the null hypothesis and level of significance with inferential statistics. [Q1] Read the following scenario pertaining to hypothesis testing and answer the following question. Lewis is conducting a study. His research question is, “Is there a relation between time spent in the SAT preparation program and increases in second administrations of the S AT?” The research hypothesis is, “Higher amount of time spent in the SAT preparation program is related to gains in scores in a second administration of the SAT.” What is the null hypothesis? [Q1 Model Response] The null hypothesis is, “Time spent in the SAT preparation program is not related to gains in scores in a second administration of the SAT.” For this study to have significant findings, the goal would be to “reject” this null hypothesis and therefore suggest a relationship. [Q2] Read the following scenario pertaining to hypothesis testing and answer the following question. Lewis is conducting a study. His research question is, “Is there a relation between time spent in the SAT preparation program and increases in second administrations of the S AT?” The research hypothesis is, “Higher amount of time spent in the SAT preparation program is related to gains in scores in a second administration of the SAT.” Lewis 15 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

measures the hours each student in his sample spends on SAT prep. He then records the gains in scores for each student after they take the SAT a second time. He finds a positive correlation with a probability value p = .02. What decision does Lewis make? [Q2 Model Response] Lewis decides to reject the null hypothesis because p = .02 is less than the significance level of α = .05. If Lewis chose to go with a conservative significance value such as α = .01, then the decision would have been different. [Q3] Read the following scenario pertaining to hypothesis testing and answer the following question. Lewis is conducting a study. His research question is, “Is there a relation between time spent in the SAT preparation program and increases in second administrations of the S AT?” The research hypothesis is, “Higher amount of time spent in the SAT preparation program is related to gains in scores in a second administration of the SAT.” Assuming an alpha level of .05, Lewis found a positive correlation with a probability value p = .02. Explain what that means, as if speaking to someone who has not taken statistics (i.e., does not know what correlation, significant, p, or .05 means).

[Q3 Model Response] A positive correlation means that the more time a student spends in SAT prep class, the greater the difference between the students’ SAT scores from the first to the second administration. A significant correlation is one that is likely to be true of the entire population from which this sample is drawn; it’s not likely just an artifact of the sample that was selected for the study. The value p refers to probability, and in particular the probability that, if there really IS no relation between time spent in SAT prep and gain scores, Lewis would have found a relation in his study simply due to chance. The .05 is the level of risk Lewis is willing to take—if there really IS no relation in the population at large (i.e., SAT takers), Lewis could have found one in his sample fewer than five times out of 100, just by chance. That is not very many times (i.e., the probability is low), so Lewis can feel confident about his results. Application Exercise 10.3: Choosing Statistical Tests Learning Outcome 10.4: Know how to use specific inferential tests to analyze data, including t-tests, analysis of variance, chi-square, and multivariate statistics. [Q1] 16 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Read the following research description and decide the correct inferential statistic for the scenario. Four randomly formed groups of n = 20 each were administered a standardized reading achievement test as a posttest. The researcher wants to know whether the performance of the three experimental groups is significantly greater than the performance of the control group. Which test of statistical inference is most appropriate? [Q1 Model Response] This scenario would call for the researcher to use a simple ANOVA. The study has four groups (i.e., levels of one independent variable) and one dependent variable. If the AN OVA shows a significant difference among groups, the researcher would also likely use multiple comparison tests to explore the differences further. [Q2] Read the following research description and decide the correct inferential statistic for the scenario. Two randomly formed groups, one experimental and one control, have been administered a pretest to measure comprehension of scientific principles. The researcher wants to compare the posttest scores and uses the pretest scores to increase the power of the statistical test. Which test of statistical inference is most appropriate? [Q2 Model Response] This scenario would call for the researcher to use ANCOVA. The researcher used a covariate (i.e., pretest scores) to increase the power of the test; use of a covariate reduces within-group (error) variance. [Q3] Read the following research description and decide the correct inferential statistic for the scenario. A group of randomly selected high-school teachers (n = 100), middle-school teachers (n = 100), and elementary-school teachers (n = 100) were asked, “Would you recommend a career in teaching to your students?” The teachers responded as “yes” or “no.” Which test of statistical inference is most appropriate? [Q3 Model Response] 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This scenario would call for the researcher to use chi-square. The researcher would compute the number of teachers at each grade level who responded yes and the number who responded no, creating a 2 × 3 contingency table, which can be analyzed with chi-square.

Chapter 10 Test Items SELF-ASSESSMENT ITEMS 1) The results of a study examining the effectiveness of two instructional strategies indicated a statistically significant difference between the means of the two treatment groups on an achievement test. Which of the following is a reasonable conclusion? a) We can be 95% confident one instructional strategy is better than the other. b) The difference between the two instructional strategies is meaningful. c) It is likely one instructional strategy is more effective. d) Both instructional strategies are equally effective. 2) Which of the following describes an error that is likely to occur when a study involves 15 of the more than 3,000 students in a school? a) Measurement error b) Sampling error c) Type I error d) Type II error 3) Which of the following statements regarding a researcher’s use of inferential statistics is true? a) They allow researchers to find the true population parameter. b) Descriptive statistics from a sample are used to estimate the characteristics of the population. c) A random sample provides a perfect estimate of the population values. d) It is best to measure every member of a population. 4) Suppose a researcher must obtain a significance level of p = .01 to reject the null hypothesis. He finds that his results are significant at the p = .05 level. The researcher should do which of the following? a) Fail to reject the null hypothesis because .05 is greater than .01. b) Fail to reject the null hypothesis because there is not much difference between .05 and .01. c) Reject the null hypothesis because .05 indicates sampling error. d) Reject the null hypothesis because .05 is greater than .01. 5) What is the main purpose of inferential testing? a) It can allow for a comparison of an observed statistic to a hypothesized population parameter. 19 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) It eliminates the possibility of making a Type I error when rejecting the null hypothesis. c) It determines whether differences between two groups are meaningful. d) It allows the researcher to determine the true population parameter. 6) A researcher is investigating the possible relationship between age and college education among adult Americans. If the null hypothesis were true, which of the following can be inferred? a) Many older adults do not have a college education. b) There is a negative relationship between age and college education. c) There is a positive relationship between age and college education. d) There is no relationship between age and college education. 7) What does a significance level of p = .01 indicate? a) The probability is 1 in 100 that the null hypothesis is true. b) The probability is 1 in 100 that the observed effect is due to chance. c) There is a 1% probability that the difference is due to the experimental treatment. d) There is a 1% probability that the null hypothesis will be rejected. 8) Which of the following is an example of a Type I error? a) Failing to reject the null hypothesis in favor of the research hypothesis b) Failing to reject the null hypothesis when it is false c) Rejecting the null hypothesis when it is true d) Rejecting the null hypothesis when there is a large p value 9) An ANOVA was used to analyze some data at alpha = .05, and the results were reported as F = 17.93, p = .001. How should the significance be interpreted? a) Statistically significant b) Statistically not significant c) Practically significant d) Statistically insignificant 10) Nonparametric statistical procedures are recommended for which type of measurement scale? a) Nominal b) Interval c) Ratio d) Parametric 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

11) Which of the following statements is NOT true about confidence intervals? a) They are sometimes used to compare two group means to show how much overlap there is in the estimated means. b) They provide a range of values in which the true population parameter might lie. c) They can be calculated using the standard error of the mean. d) They use a p value to determine the level of confidence in a statistical result. 12) A study comparing the effects of two treatments used two groups of students that yielded two means—one for each group. Which statistical test should the researcher use to compare these two means? a) Chi-square test of independence b) Dependent samples t-test c) Independent samples t-test d) Two-way analysis of variance 13) A researcher is analyzing the data from a study in which there were five levels of the independent variable. Which statistical procedure should be used to test the null hypothesis that there is no difference between the means of the five groups? a) Chi-square test of independence b) Multiple t-tests c) One-way analysis of variance or simple ANOVA d) Five-way analysis of variance or factorial ANOVA 14) How does factorial analysis of variance differ from simple analysis of variance? a) It is used to compare the variance in two group means. b) It compares two or more groups on two independent variables. c) It compares two or more groups on one independent variable. d) It tests only one null hypothesis. 15) Which of the following statistical procedures uses pretest scores as an adjustment so that a more accurate comparison of posttest scores can be made? a) Analysis of covariance b) Two-way analysis of variance or factorial ANOVA c) Multiple-comparison procedure d) One-way analysis of variance or simple ANOVA 16) A researcher was interested in the relationship between SES and college attendance. She randomly selected 200 participants from low-, middle-, and high-SE S backgrounds and divided them into two groups according to whether they did or 21 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

did not attend college. She constructed a table of expected counts and compared the expected with the observed counts. Which statistical procedure did the researcher use? a) 2 × 3 ANOVA b) Chi-square test of independence c) Multiple t-tests d) One-way analysis of variance or simple ANOVA 17) What is recommended when using inferential statistics? a) Work with small samples b) Set the level of significance at p = .25 c) Disregard descriptive statistics so as not to allow biased results d) Report both statistical significance and practical significance 18) An independent sample t-test was used to compare two independent means at alpha = .05. The results were reported as t = 1.79, p = .201. How should these results be interpreted? a) The difference is statistically significant, but perhaps not practically significant. b) The difference is not statistically significant. c) The difference is probably practically significant. d) The difference is statistically significant and practically significant. 19) When might a researcher use a factorial design? a) To test for an interaction of two dependent variables b) To test for an interaction of two independent variables c) If the sample size is too small to permit conducting two-way ANOVA d) To determine both practical significance and statistical significance 20) A chi-square test of independence was used to analyze some count data at alpha = .05, and the results were reported as c 2 = 11.79, p = .025. Which of the following is the correct interpretation of the results? a) The difference is statistically significant. b) The difference is not statistically significant. c) The difference is marginally significant. d) The difference is statistically significant but not practically significant.

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Which of the following is a true statement about the null hypothesis? a) It is a procedure that can be used to make a decision about whether or not there is a difference between two means. b) It provides information the researcher needs to determine the appropriate alpha level for most statistical tests. c) It indicates the level of sampling and measurement error in the analysis. d) It provides descriptive statistics that nullify the results. 2) Which of the following is a true statement about an effect size measure? a) It provides an absolute rule for determining statistical significance. b) It determines the acceptable level of statistical significance. c) It is a range of values in which the true population parameter must lie. d) It is a way of quantifying the magnitude of an observed difference. 3) What is the main purpose of inferential statistical tests? a) They can be used to determine practical significance. b) They account statistically for the tendency of results from repeated studies to vary. c) They allow the researcher to organize and summarize sample data. d) They provide information about the level of precision of the statistics based on observed data. 4) A researcher compared two methods for teaching inferential statistics to college students. Group 1 was taught with Method A and Group 2 with Method B. A comparison of posttest scores indicated that the results were significantly different (p = .05) in favor of Method A. Which of the following is the correct interpretation of the results? a) Method A is 5% better than Method B. b) There is a 5% probability that the difference is due to chance. c) There is a probability of 5 in 100 that Method A is better than Method B. d) There is a 95% probability that the methods are not significantly different. 5) Suppose a researcher must obtain a significance level of p = .01 to reject the null hypothesis. After analyzing his data, he finds that his results are significant at a p = .001 level. Which of the following should the researcher conclude? a) Fail to reject the null hypothesis because .01 is greater than .001. 23 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Fail to reject the null hypothesis because there is not much difference between .01 and .001. c) Reject the null hypothesis because .001 indicates measurement error. d) Reject the null hypothesis because .001 is less than .01. 6) Which of the following decisions accurately describes a typical significant result? a) Accept the null hypothesis b) Reject the null hypothesis c) Fail to reject the hypothesis d) Reject the alternative hypothesis 7) Which of the following is true about a researcher who made a Type II error? a) The researcher failed to reject the null hypothesis when it is false. b) The researcher failed to reject the null hypothesis when there is a large probability level. c) The researcher rejected the null hypothesis when it is true. d) The researcher rejected the null hypothesis when the results are statistically significant. 8) Is it possible to have statistical significance and not practical significance? a) Yes, if the null is rejected but the effect size is large. b) No, because the results cannot be practically significant if they are not statistically significant. c) Yes, because statistical significance does not mean the difference is meaningful. d) No, because practical significance must be established in order to determine statistical significance. 9) Mr. Wilkins reported the results of an ANCOVA as F = 28.73, p = .02, with an effect size of .83. Which statement about these results is true? a) The results are statistically significant and likely to be practically significant. b) There is statistical but not practical significance. c) There is practical but not statistical significance. d) There is neither statistical nor practical significance. 10) Which of the following set of assumptions needs to be met when using parametric statistics? a) Measures of equal interval, normally distributed population, equal variances of each group b) Measures of equal interval, normally distributed population, large sample size 24 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) Rank-ordered measures, normally distributed population, equal variances of each group d) Rank-ordered measures, unequal variances between groups, randomly selected sample 11) In a study investigating the effects of a social skills training program, pretest and posttest scores were obtained from 30 students diagnosed as behaviorally handicapped. Which statistical test should the researcher use to test the null hypothesis that there is no difference between the pretest and posttest scores? a) Chi-square test of independence b) Independent samples t-test c) Dependent t-test d) Two-way analysis of variance 12) Which of the following is true of an analysis of covariance? a) It can only be used in studies with a factorial design. b) It allows the researcher to analyze two or more dependent variables. c) It adjusts for pretest differences between groups. d) It establishes relationships between two dependent variables. 13) Multivariate statistical procedures allow a researcher to do which of the following? a) Adjust for differences between scores when two or more groups are used in the study. b) Analyze a number of independent variables independently. c) Analyze each dependent variable separately. d) Analyze all the related dependent variables in a single procedure. 14) Which of the following is the correct statistical procedure to use to test the null hypothesis of no relationship between two nominal variables? a) Chi-square test of independence b) MANCOVA c) Hotelling’s T d) Factorial analysis of variance or two-way ANOVA 15) A researcher was interested in the relationship between years of teaching and attitude toward inclusion. He randomly selected 500 teachers with varying years of service ranging from 1 to 40 years and divided them into two groups according to whether they favored or did not favor inclusion. He constructed a table of expected counts and compared the expected counts with the observed counts. Which statistical procedure did the researcher use to test the null hypothesis? 25 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

a) 2 × 3 ANOVA b) Chi-square test of independence c) Multiple t-tests d) One-way analysis of variance or simple ANOVA 16) An ANOVA was used to compare means of four groups, and the results were reported as F = 1.93, p = .151. How should these results be interpreted? a) The group means are statistically different. b) Differences in the group means could just be due to chance. c) Differences between the groups are practically significant. d) The magnitude of the difference between the group means is large. 17) What does a 95% confidence interval indicate? a) The true population value is likely to lie within the upper and lower values. b) The researcher has not committed a Type I error. c) The researcher can have 95% confidence the conclusion is correct. d) There is a 1 in 5 chance the researcher has committed a Type II error. Short-Answer Questions 1) Which is worse, committing a Type I error or committing a Type II error? 2) Why should a researcher avoid committing a Type II error? 3) Describe an example in which it would be appropriate to conduct a nonparametric test such as a chi-square test of independence. Essay Question What information do inferential statistics provide that descriptive statistics do not? Give three reasons it is necessary to also consider and report descriptive statistics when reporting the results from an inferential study.

Chapter 10 Test Answer Key ANSWER KEY: Chapter 10 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. c 2. b 3. b 4. a 5. a 6. d 7. b 8. c 9. a 10. a 11. d 12. c 13. c 14. b 15. a 16. b 17. d 18. b 19. b 20. a

INSTRUCTOR’S EXAM ITEMS 1. a 2. d 3. d 4. b 5. d 6. b 7. a 8. c 9. a 10. a 11. c 12. c 13. d 14. a 15. b 16. b 17. a

Chapter 11 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 11: Qualitative Research Designs .................................................................. 1 Learning Outcome Quizzes ........................................................................ 1 Application Exercises ................................................................................ 11 Test Items ................................................................................................. 16 Test Answer Key ....................................................................................... 22

Chapter 11: Qualitative Research Designs Chapter 11 Learning Outcome Quizzes Learning Outcome 11.1: Describe the nature and characteristics of qualitative research. [Q1] Which of the following is a term more closely associated with quantitative research than qualitative research? 1. Analysis of variance [correct] 2. Field research 3. Naturalistic 4. Participant observation [Feedback for Correct Answer 1] Analysis of variance is a statistical method used in quantitative research. It is not associated with qualitative methods. [Feedback for Answer Choice 2] Field research is a term that is often synonymous with qualitative research. [Feedback for Answer Choice 3] Because qualitative research often occurs in “natural” settings, or settings where no variables are manipulated, this term is often associated with qualitative methods. [Feedback for Answer Choice 4] Participant observations are a key component of qualitative research. [Q2] Which of the following best describes the role of data interpretation in qualitative research? 1. Classify the data into meaningful categories and identify themes 2. Make sense of and find meaning in the data [correct] 3. Develop comprehensive descriptions to convey the rich complexity of the research 4. Struggle with the nuances, caveats, and subtleties in the data [Feedback for Answer Choice 1] 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This is certainly a step in the qualitative research process, but this must be done before data interpretation can take place. [Feedback for Correct Answer 2] The qualitative researcher wants to be open to new ways of understanding. As the researcher works with the data, progressively more specific findings and deeper understandings and connections are generated. [Feedback for Answer Choice 3] While this is a component of the data analysis process, this is not the main function of data interpretation. [Feedback for Answer Choice 4] While this will certainly occur in qualitative research, this is not the main function of data interpretation. [Q3] Which of the following represents a rationale for why qualitative researchers employ emergent research designs? 1. They have a general plan for conducting the research that they want to follow. 2. They believe that that participants actively construct their own reality. 3. They want to remain open and flexible during the data collection process. [correct] 4. They want to look for the process through which behavior occurs, not just the outcomes or products. [Feedback for Answer Choice 1] Qualitative researchers usually do have a plan to follow, but this is not what makes their work “emergent.” [Feedback for Answer Choice 2] Some qualitative researchers ascribe to this belief, but not all. [Feedback for Correct Answer 3] As researchers learn about the setting, people, and other sources of information, they discover what needs to be done to fully describe and understand the phenomena being studied. The design and the methods may change (i.e., emerge) as more is learned. [Feedback for Answer Choice 4]

This may be of interest to some qualitative researchers, but this is not what makes their work “emergent.” [Q4] When does data analysis occur in qualitative research? 1. During and after data collection [correct] 2. Only after data collection 3. When the researcher collects sufficient data 4. At one point in the research process [Feedback for Correct Answer 1] Data analysis in qualitative research can occur during data collection, which differs from quantitative research. [Feedback for Answer Choice 2] This is typically true of quantitative research, but not qualitative. [Feedback for Answer Choice 3] Actually, while this seems logical, this is incorrect. [Feedback for Answer Choice 4] Qualitative research is much too complex to only occur at one point in the research process. [Q5] Which of the following terms best describes the data analysis process in qualitative research? 1. Summarizing [correct] 2. Interpreting 3. Quantifying 4. Discovering [Feedback for Correct Answer 1] Data analysis in qualitative research involves summarizing the data in a dependable and accurate manner. [Feedback for Answer Choice 2] 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Data interpretation differs from data analysis. Data interpretation occurs after analysis and is an attempt to answer the “So, what?” question. [Feedback for Answer Choice 3] Some qualitative researchers may create quantitative codes to assist with data analysis, but this is not always the case. [Feedback for Answer Choice 4] Once you’ve reached the data analysis phase, you already have a very clear picture of what your data are and how your data will be analyzed. Learning Outcome 11.2: Know how threats to validity/trustworthiness can compromise the integrity, credibility, and value of qualitative findings. [Q1] Which of the following threats is best described as an insufficient description of the context and lack of consideration of contextual factors in the findings? 1. Inauthenticity 2. Researcher bias 3. Inadequate participant perspectives 4. Context insensitivity [correct] [Feedback for Answer Choice 1] This term refers to a characteristic that suggests that the findings are not genuine, actual, or real. [Feedback for Answer Choice 2] Researcher bias occurs when the researcher allows assumptions and predispositions to affect what data are gathered and how the data are interpreted. [Feedback for Answer Choice 3] This occurs when there is scant consideration of actual participant voices, language, and meanings in understanding the phenomena. [Feedback for Correct Answer 4] Context insensitivity is characterized by insufficient description of the context and lack of consideration of contextual factors in the findings. [Q2] 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Which of the following threats is characterized by participants providing less-thanauthentic responses because they are in a research situation? 1. Confirmability 2. Reactivity [correct] 3. Inauthenticity 4. Inadequate transparency [Feedback for Answer Choice 1] Confirmability is threatened when findings are not verified in some way, typically by others. [Feedback for Correct Answer 2] Reactivity occurs when participants provide less than authentic responses because they are in a research situation. [Feedback for Answer Choice 3] This is a characteristic that suggests that the findings are not genuine, actual, or real. [Feedback for Answer Choice 4] This occurs when there is a lack of information about how a study was conducted, how decision making occurred, and how interpretations were made. [Q3] Which of the following terms indicates a situation where the nature of data collection has skewed the results? 1. Inauthenticity 2. Instrumentation [correct] 3. Reactivity 4. Researcher bias [Feedback for Answer Choice 1] Inauthenticity is a characteristic that suggests that the findings are not genuine, actual, or real. [Feedback for Correct Answer 2] Instrumentation is a threat when the nature of data collection skews results. [Feedback for Answer Choice 3] 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Reactivity is a threat when participants provide less-than-authentic responses because they are in a research situation. [Feedback for Answer Choice 4] Researcher bias occurs when the researcher—often the same person who gathers information—allows assumptions and predispositions to affect what data are gathered and how the data are interpreted. [Q4] Jessica is a qualitative researcher who just completed a study that included numerous interviews with public school teachers. She summarized her findings based on her own interpretations and without allowing others to review the data. Which of the following is threatening the trustworthiness of her study? 1. Confirmability [correct] 2. Instrumentation 3. Inauthenticity 4. Inadequate transparency [Feedback for Correct Answer 1] Confirmability is threatened when findings are not verified in some way, typically by others. [Feedback for Answer Choice 2] This is a threat when the nature of the data skews the results. [Feedback for Answer Choice 3] This is a characteristic that suggests that the findings are not genuine, actual, or real. [Feedback for Answer Choice 4] This occurs when there is a lack of information about how a study was conducted, how decision making occurred, and how interpretations were made. [Q5] Joanna is an assistant principal in an elementary school. She has conducted a qualitative study that included interviewing teachers in her school about how the enforcement of school policies impacts teacher motivation. Which of the following major threats to trustworthiness is likely? 1. Inadequate transparency 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. Instrumentation 3. Sampling 4. Reactivity [correct] [Feedback for Answer Choice 1] This occurs when there is a lack of information about how a study was conducted, how decision making occurred, and how interpretations were made. [Feedback for Answer Choice 2] This occurs when the nature of data collection skews results. [Feedback for Answer Choice 3] Sampling is a threat when the nature of the individuals in the study could affect the findings. [Feedback for Correct Answer 4] This occurs when participants provide less-than-authentic responses because they are in a research situation. In this situation, these teachers are likely influenced by the fact that they work closely with Joanna. Learning Outcome 11.3: Differentiate six major approaches to conducting qualitative research. [Q1] Ethnographic research is best distinguished from other qualitative approaches by its focus on which of the following? 1. The cultural patterns and perspectives of participants [correct] 2. Discerning patterns of human behavior 3. Intensive participant observation 4. Understanding the problems of individuals [Feedback for Correct Answer 1] Ethnographic research is the study of the cultural patterns and perspectives of participants in their natural settings. [Feedback for Answer Choice 2] Ethnographic research includes understanding patterns, but this is not the best answer. [Feedback for Answer Choice 3] 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Ethnographers are engaged in deeper levels than mere observation. [Feedback for Answer Choice 4] This answer choice is much too broad. [Q2] Which of the following research topics is best suited for case study research methods? 1. The marginalization of women in industrial arts classes 2. The attitudes of teachers about the process of hiring new administrators 3. The impact of teacher technology integration training on student science test scores 4. The process of selecting a principal in South High School [correct] [Feedback for Answer Choice 1] This study is much too broad for a case study design. [Feedback for Answer Choice 2] This study will require a wider range of data collection than typical in a case study design. [Feedback for Answer Choice 3] This study would likely be best suited for quantitative research methods. [Feedback for Correct Answer 4] This topic requires an in-depth understanding of a bounded system (this one process at one school for a period of time). [Q3] Which of the following situations would make a grounded theory design an appropriate choice? 1. The research question is focused on studying a culture 2. The researcher is looking to generate a theory to explain a process [correct] 3. The researcher is looking to use an existing theory to explain a process 4. The researcher hopes to do an in-depth study of one individual [Feedback for Answer Choice 1] This is descriptive of an ethnographic research study. [Feedback for Correct Answer 2] 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

A grounded theory study is characterized by a unique purpose: to discover or generate theory from real-world data that explain central phenomena. [Feedback for Answer Choice 3] This is a deductive approach to research, and grounded theory is an inductive process. [Feedback for Answer Choice 4] This would be more typical of a case study design. [Q4] Which of the following would most likely be explored with narrative inquiry? 1. The experience of Mike Kesterke, an elementary school principal [correct] 2. The social relationships of children in multicultural classrooms 3. The experiences of first-generation college students at a prestigious university 4. Gender differences in mathematics achievement of rural middle school students [Feedback for Correct Answer 1] The goal of narrative inquiry is to use individuals’ actual lived stories to provide a deep and rich understanding of a specific phenomenon. It is most commonly used with one or only a few individuals. [Feedback for Answer Choice 2] This would involve a larger sample size than is typical for narrative inquiry. [Feedback for Answer Choice 3] It is likely that this would involve interviewing a number of participants and thus would not be best suited for narrative inquiry. [Feedback for Answer Choice 4] This would most likely be explored using quantitative methods. [Q5] Dr. Cordova is interested in studying the role of power and privilege, as it relates to race, in the exclusion of indigenous people from institutions of higher education. Which of the following likely describes her study? 1. Critical study [correct] 2. Narrative inquiry 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. Ethnography 4. Grounded theory [Feedback for Correct Answer 1] Critical studies are distinguished by a researcher role as advocate to respond to the themes and issues of marginalized individuals. [Feedback for Answer Choice 2] Narrative studies typically include one, or perhaps a few, participants and are deeply focused on personal narrative. This is not likely the design for Dr. Cordova’s study. [Feedback for Answer Choice 3] An ethnographic qualitative study (or ethnography—a term used interchangeably with ethnographic in this text) is an in-depth, comprehensive description and interpretation of cultural patterns and meanings within a culture or social group or system. [Feedback for Answer Choice 4] A grounded theory study is characterized by a unique purpose: to discover or generate theory from real-world data that explain central phenomena.

Chapter 11 Application Exercises Chapter 11: Qualitative Research Designs Application Exercise 11.1: Characteristics of Qualitative Research Learning Outcome 11.1: Describe the nature and characteristics of qualitative research. [Q1] Review Table 11.1 entitled Key Characteristics of Qualitative Research to answer the following question: Jed is a qualitative researcher interested in better understanding the decision making of elementary school teachers regarding the factors that influence their selection of a literacy curriculum. To do this, Jed interviews a range of elementary school teachers. Which characteristic of qualitative research is Jed adhering to in this example? [Q1 Model Response] In this example, Jed is engaged in direct data collection, a characteristic of qualitative research. He is going directly to the source for his data, thus allowing for more trustworthy responses. [Q2] Review Table 11.1 entitled Key Characteristics of Qualitative Research to answer the following question: Annaya is a qualitative researcher interested in learning more about the ways in which teachers seek support for stress and trauma experienced in the classroom. She has conducted numerous interviews, coded and analyzed those issues, and concluded that teachers rarely seek formal support for the trauma they experience. Which characteristic of qualitative research is Annaya adhering to in her data analysis process? [Q2 Model Response] Annaya’s analysis process consisted of determining generalizations induced from synthesizing gathered information. This is known as inductive-data analysis. This type of data analysis is common in qualitative research in the social sciences. [Q3] Review Table 11.1 entitled Key Characteristics of Qualitative Research to answer the following question: 11 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Carlos is a qualitative researcher who is conducting a series of focus groups to better understand how teachers feel about participating in extracurricular activities. As he conducts the groups, he realizes that teachers feel strongly about feeling pressured to participate in extracurriculars in order to maintain their jobs. With this new knowledge, Carlos adapts his research design to also include interviews and open-ended surveys so that participants may feel more comfortable in this setting than they may in a focus group of peers. Which characteristic of qualitative research is Carlos adhering to in his data analysis process? [Q3 Model Response] This is a classic example of the emergent research design nature of qualitative research. Carlos chose to evolve and change his design as the study takes place, because this helped him better answer his questions and understand his research population. Application Exercise 11.2: Threats to Validity/Trustworthiness Learning Outcome 11.2: Know how threats to validity/trustworthiness can compromise the integrity, credibility, and value of qualitative findings. [Q1] Review Table 11.2 entitled Threats to the Validity/Trustworthiness of Qualitative Research to answer the following question: Josina is a qualitative researcher studying the impacts of community relationships as a function of immigration reform on a local community of migrant workers. However, she failed to include a complete history of community actions related to immigration policy changes. Which threat to validity/trustworthiness is present due to Josina’s lack of historical data collection? [Q1 Model Response] Josina’s study is threatened by context insensitivity. She has an insufficient description of the context and therefore will have a lack of consideration of contextual factors in her findings. [Q2] Review Table 11.2 entitled Threats to the Validity/Trustworthiness of Qualitative Research to answer the following question:

Markel is a qualitative researcher studying the impacts of block scheduling on the academic outcomes of high school students. Markel has long been a proponent of block scheduling as he was one of the first teachers to convince his administration to bring the model to his former school. He is currently interviewing many of his former colleagues to better understand the impacts of block scheduling. Which threat to validity/trustworthiness is present in Markel’s study? [Q2 Model Response] Markel’s study is threatened by researcher bias. He has allowed assumptions and predispositions to affect the data that are gathered and how the data are interpreted. He has not sought to interview individuals with whom he does not already have a working relationship. [Q3] Review Table 11.2 entitled Threats to the Validity/Trustworthiness of Qualitative Research to answer the following question: Frank is a qualitative researcher studying the lived experiences of first-generation college students as they enter college. He interviews a sample of first-generation college students but does not outline his sampling inclusion criteria. Which threat to validity/trustworthiness is present in Frank’s study? [Q3 Model Response] Frank’s study suffers from inadequate transparency. There is a lack of information about how his study was conducted. As a result, his results cannot be fully trusted. While his sampling may have been appropriate, it is his responsibility to reveal sampling methods to readers. Application Exercise 11.3: Identifying Qualitative Research Approaches Learning Outcome 11.3: Differentiate six major approaches to conducting qualitative research. [Q1] Consider the following: Koonce (2012) was interested in the ways in which race and gender affect schooling, and she had a particular interest in studying how African American teen girls talked with their teachers. She engaged two adolescent African American females in dialogue about their experiences with their teachers when they “talk with an attitude,” encouraging them to tell their own stories. Each student participated in one in-depth and one follow-up interview within the same month. 13 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

What approach to research did Koonce use? [Q1 Model Response] This is a phenomenological study. The purpose of phenomenology, is to describe, clarify, and interpret lived experiences of participants in order to understand the “essence” of the experience as consciously perceived by and described by the participants. You can find the full report of this study, titled “ ‘Oh, Those Loud Black Girls!’: A Phenomenological Study of Black Girls Talking with an Attitude,” if you search ERIC. The complete reference is: Koonce, J. B. (2012). “Oh, those loud Black girls!”: A phenomenological study of Black girls talking with an attitude. Journal of Language and Literacy Education [Online], 8(2), 26–46. [Q2] Consider the following: Mora (2011) was interested in whether middle school students were bored in class while engaging in activities designed to prepare them for high-stakes tests. He studied a group of 30 urban students, all of whom were Latino/a, for 2.5 years, while they were enrolled in grades 6 through 8. He observed in the classroom and conducted conversational interviews. What approach to research did Mora use? [Q2 Model Response] This is an ethnographic study. An ethnography is an in-depth, comprehensive description and interpretation of cultural patterns and meanings within a culture or social group or system. In this case, the system was an urban middle school. You can find the full report of this study, titled " ‘School Is So Boring’: High-Stakes Testing and Boredom at an Urban Middle School,” if you search ERIC. The complete reference is: Mora, R. (2011). “School is so boring": High-stakes testing and boredom at an urban middle school. PennGSE Perspectives on Urban Education, 9(1). [Q3] Consider the following: McKenna and Millen (2013) were interested in the ways in which parents engage in their children’s education. They focused on two constructs: parent voice (ideas and 14 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

opinions) and parent presence (behaviors related to those ideas and opinions). The researchers held two 2-hour-long focus groups and conducted a lengthy interview with one participant. They also asked all participants in the focus groups and interviews to write hypothetical letters to a “teacher,” in which they discussed their views about key topics. From their data, they constructed two models—one for parent voice and one for parent presence—that together help to explain parent engagement. What approach to research did McKenna and Millen use? [Q3 Model Response] This is a grounded theory study. A grounded theory study is characterized not so much by its data collection as by its unique purpose—to discover or generate theory from realworld data that explain central phenomena. You can find the full report of this study, titled, “Look! Listen! Learn! Parent Narratives and Grounded Theory Models of Parent Voice, Presence, and Engagement in K-12 Education,” if you search ERIC. The complete reference is: McKenna, M. K., & Millen, J. (2013). Look! Listen! Learn! Parent narratives and grounded theory models of parent voice, presence, and engagement in K-12 education. School Community Journal, 23(1), 9–48.

Chapter 11 Test Items SELF-ASSESSMENT ITEMS 1) Which of the following is a characteristic of qualitative research? a) Multiple realities require an in-depth understanding of participants’ perspectives. b) The context in which phenomena occur is critically important to understanding that phenomena. c) It is acknowledged that all researchers have biases that limit the way in which they see and interpret phenomena. d) All of the above 2) Which of the following is true of qualitative research? a) Narrative descriptions are used by the investigator. b) The context in which behavior occurs is of secondary importance. c) The researcher avoids direct contact with the participants of the study so as not to bias the results. d) The researcher collects data to prove a theory. 3) What purpose does an emergent research design serve as used by qualitative researchers? a) The research problem is studied extensively before the study begins so there are no changes once the design has emerged. b) The concrete, sequential nature of an emergent design allows for systematic discovery. c) An emergent design allows for flexibility, so the researcher doesn’t begin the study with preconceived ideas. d) An emergent design is the most efficient way to collect data from participants. 4) A researcher studied the use of electronic, interactive whiteboards at a school by embedding herself in the school and describing their use in-depth. This is an example of which type of study? a) Ethnography b) Case study c) Grounded theory d) Critical studies 5) Which qualitative research design describes and interprets the everyday life experiences of participants to understand their experiences as they perceive them? a) Ethnography 16 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Case study c) Phenomenology d) Grounded theory 6) A researcher who focuses on unequal treatment to a marginalized group is involved in which type of qualitative research? a) Phenomenology b) Critical studies c) Narrative inquiry d) Ethnography 7) Which of the following threats to validity is typified by a researcher missing instances of a behavior because he has created an observation tool with a narrow definition of the behavior? a) Context insensitivity b) Researcher bias c) Instrumentation d) Sampling 8) A researcher interviewed multiple key stakeholders but reported only her own interpretations of their comments. This is an example of which qualitative threat to validity? a) Confirmability b) Instrumentation c) Context insensitivity d) Sampling 9) A researcher investigated the drug culture in a large urban high school but did not include in the examination the history of the drug use in the school. This is an example of which qualitative threat to validity? a) Confirmability b) Inauthenticity c) Context insensitivity d) Researcher bias 10) In a research study on sexual abuse on a large college campus, there were no interviews with sexual abuse victims or persons accused of sexual abuse. This is an example of which qualitative threat to validity? a) Triangulation 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Instrumentation c) Inauthenticity d) Inadequate transparency

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Which of the following is a characteristic of a qualitative research study? a) The influence of context is less important than understanding participants’ thoughts and feelings. b) Researcher bias is not a concern because it is expected due to the researcher’s direct involvement with participants. c) Multiple realities are acknowledged and therefore there is a focus on participants’ perspectives. d) Structured interviews with predetermined questions are used to gain a deep understanding of participants’ thoughts and feelings. 2) Which of the following is true of qualitative research? a) Bias that could be attributable to researcher participation is avoided by having neutral observers. b) Data collection is guided by hypotheses that are formulated by the researcher. c) The interaction between the individual and the environment is considered important to understanding behavior. d) Quantitative measuring techniques are frequently used to complement narrative descriptions. 3) Which of the following typifies qualitative researchers? a) They are selective about their observations to avoid overloading the knowledge base with one participant’s behavior. b) Their observations include every detail of behavior, setting, and thinking in order to capture them completely. c) They formulate hypotheses in much the same way as quantitative researchers. d) Their conclusions are developed from the top down to avoid researcher bias. 4) Why is an emergent research design used by qualitative researchers? a) They know in advance what needs to be done to understand the research problem. b) An emergent design allows them to collect data more efficiently. c) They value the precision of a structured, predetermined emergent design because it is conceptual. d) An emergent design allows for flexibility during the data collection process.

5) Which qualitative research design has as its purpose an understanding of the shared beliefs and behaviors of a specific culture or social group? a) Ethnography b) Case study c) Phenomenology d) Grounded theory 6) Which qualitative research design has as its purpose an in-depth analysis of one or more events, settings, programs, groups, or other bounded systems? a) Ethnography b) Case study c) Phenomenology d) Grounded theory 7) Which qualitative research design attempts to understand the essence of the participants’ experience through extended interviews? a) Ethnography b) Case study c) Phenomenology d) Critical studies 8) Which qualitative research design is distinguished by the researcher acting as an advocate for marginalized members of society? a) Ethnography b) Critical studies c) Phenomenology d) Narrative inquiry 9) A researcher examined racial disharmony in a large urban high school but failed to consider events in the school that might have led to the disharmony. This is an example of which threat to internal validity? a) Confirmability b) Inauthenticity c) Context insensitivity d) Researcher bias 10) A school district was considering a change to year-round schooling. The researcher assigned to determine public sentiment on the issue opposed year-round schooling. He conducted interviews with teachers, students, and parents, but the only 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

questions in the interview about year-round schooling focused on the problems it would create. This is an example of which qualitative threat to validity? a) Researcher bias b) Context insensitivity c) Sampling d) Inauthenticity Short-Answer Questions 1) What is meant by inductive data analysis as it relates to qualitative research? 2) A key characteristic of qualitative research is socially constructed meaning. What is this and how does this concept contrast with key characteristics in a quantitative research design? Essay Question Identify the key characteristics of three types of qualitative research design and give an original example of each.

Chapter 11 Test Answer Key ANSWER KEY: Chapter 11 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. d 2. a 3. c 4. b 5. c 6. b 7. c 8. a 9. c 10. c

INSTRUCTOR’S EXAM ITEMS 1. c 2. c 3. b 4. d 5. a 6. b 7. c 8. b 9. c 10. a

Chapter 12 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 12: Qualitative Data Collection ....................................................................... 1 Learning Outcome Quizzes ........................................................................ 1 Application Exercises ................................................................................ 12 Test Items ................................................................................................. 17 Test Answer Key ....................................................................................... 22

Chapter 12: Qualitative Data Collection Chapter 12 Learning Outcome Quizzes Learning Outcome 12.1: Identify essential steps to take to gather qualitative data. [Q1] Compared to being an insider or partial participant, which of the following is one advantage of being a complete outsider? 1. The researcher is less intrusive than a participant. (correct) 2. The researcher does not need to obtain permission from participants. 3. The researcher does not need to establish rapport. 4. The researcher is more flexible to collect different types of data. [Feedback for Correct Answer 1] An outsider is less intrusive and less likely to become emotionally involved than a participant. However, an outsider may have less access to the participants to collect the necessary data. [Feedback for Answer Choice 2] Researchers must always obtain permission from participants. [Feedback for Answer Choice 3] Researchers should always work to establish rapport in qualitative research. [Feedback for Answer Choice 4] This is not an accurate statement. Flexibility of the researcher will vary between studies. [Q2] Which of the following terms can be defined as a technique for bracketing possible perceptions to establish reflexivity? 1. Interviewing 2. Transcribing 3. Coding 4. Memoing [correct] [Feedback for Answer Choice 1] 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Interviewing is a common way to collect qualitative data, but this is not the definition of interviewing. [Feedback for Answer Choice 2] Transcribing is the process of taking audio recordings of interviews and producing a manuscript. [Feedback for Answer Choice 3] Coding is the process of sorting data in an attempt to find themes in qualitative data. [Feedback for Correct Answer 4] Memoing is a technique for bracketing possible perceptions to establish reflexivity. This begins with the selection of a research topic. [Q3] Researchers who have collected minutes from faculty meetings from various schools and are analyzing them to look for themes in teacher autonomy would be considered which of the following? 1. Complete insiders 2. Complete outsiders [correct] 3. Partial insiders 4. Partial outsiders [Feedback for Answer Choice 1] Complete insiders have an established role in the settings from which they gather data. These researchers are not connected in any significant ways to their research sites. [Feedback for Correct Answer 2] These researchers would be considered complete outsiders. They have no connection to their research sites on any level. [Feedback for Answer Choice 3] This is not a category used to describe the role of qualitative researchers. [Feedback for Answer Choice 4] This is not a category used to describe the role of qualitative researchers. [Q4] 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Bethany is a teacher in an alternative middle and high school. She is interested in studying the mental well-being of her colleagues as they deal with student trauma in the classroom. Which of the following best describes Bethany’s role? 1. Complete outsider 2. Complete insider [correct] 3. Partial insider 4. Partial outsider [Feedback for Answer Choice 1] Complete outsiders are totally detached from the naturally occurring behavior and activities of the participants. This is certainly not Bethany. [Feedback for Correct Answer 2] Bethany is strongly connected to the participants in her study. She has an established role in the setting. [Feedback for Answer Choice 3] This is not a category used to describe the role of qualitative researchers. [Feedback for Answer Choice 4] This is not a category used to describe the role of qualitative researchers. Learning Outcome 12.2: Know how to collect observational data. [Q1] Karen is interested in understanding how first-year teachers experience work-related stress. She has taken a job as a first-year teacher, analyzing her own stress, while also interviewing and observing other first-year teachers. Which of the following best describes Karen’s role? 1. Complete participant 2. Passive participant 3. Detached researcher 4. Participant observer [correct] [Feedback for Answer Choice 1] If the researcher is a member of the group and not known as a researcher, the role would be known as a complete participant. [Feedback for Answer Choice 2] 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Karen is not passive in her actions. [Feedback for Answer Choice 3] This is not a category used to describe qualitative researchers. [Feedback for Correct Answer 4] If the role of the researcher doing the observation is as a genuine participant in the activity being studied, yet identified to others in the setting as the researcher, the individual is called a participant observer. [Q2] Kalvin is interested in researching the classroom management approaches of teachers with 10 or more years of teaching experiences. He has conducted interviews and collected video samples of teaching practices. Which of the following best describes Kalvin’s level of participation? 1. Moderate participation 2. Passive participation [correct] 3. Active participation 4. Complete participation [Feedback for Answer Choice 1] Moderate participation entails being identified as a researcher but not taking on the role of the participants. Often called “observer participants,” these researchers are connected to the research site or sample, but identified as the researcher. Kalvin is not connected to the research site in any way other than as a researcher. [Feedback for Correct Answer 2] Passive participation means observing without becoming a part of the process in any way. This is certainly Kalvin’s approach. [Feedback for Answer Choice 3] An active participant participates as a member of the group but is known as a researcher. Kalvin is not a member of the group he is studying. [Feedback for Answer Choice 4] A complete participant participates as a member of the group and is not known as a researcher. This does not describe Kalvin’s role. [Q3] 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Which of the following roles of a qualitative researcher has the highest probability for altering what occurs in the study? 1. Passive participation 2. Moderate participation 3. Compete participation [correct] 4. Active participation [Feedback for Answer Choice 1] Passive participants are the least involved with the study sample and pose the least amount of risk to influencing study outcomes. [Feedback for Answer Choice 2] Moderate participants may impact the study, but they are not the group with the highest probability for doing so. [Feedback for Correct Answer 3] A complete participant participates as a member of the group and is not known as a researcher. Because of this, the complete participant researcher has a great deal of influence in the study. [Feedback for Answer Choice 4] Active participants are active members of the group being studied, but other participants know that the active participant is a researcher. [Q4] Which of the following describes detailed written descriptions of what was observed, as well as the researcher’s interpretations? 1. Field notes [correct] 2. Interviews 3. Transcripts 4. Detailed codes [Feedback for Correct Answer 1] Field notes are detailed written descriptions of what was observed, as well as the researcher’s interpretations. [Feedback for Answer Choice 2] Interviews refers to asking questions to participants and recording their responses. 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] Transcripts are simply text versions of audio recorded interviews. [Feedback for Answer Choice 4] Codes are used to sort data, but they are not detailed written descriptions. Learning Outcome 12.3: Describe how interview data are collected. [Q1] Marci is studying the strategies teachers use to manage the inclusion of learners with special needs within the general education classroom. While observing a seventh-grade history class, she noted that the teacher employed a cooperative learning strategy that she had not seen before. At the end of class, she asked him about the strategy and how effectively it works. Marci is conducting which of the following? 1. An unstructured interview [correct] 2. A key informant interview 3. An interview protocol 4. A conversation, not technically an interview [Feedback for Correct Answer 1] The unstructured interview allows the qualitative researcher to inquire into something that has presented itself as an opportunity to learn about something in the research setting. This particular unstructured interview could be considered an informal conversational interview. [Feedback for Answer Choice 2] This is a technique exclusive to ethnographic research. It is based on the assumption that in-depth interviews with a few “key” participants—individuals who are particularly knowledgeable and articulate—will provide insights and understandings about the problem. [Feedback for Answer Choice 3] An interview protocol is simply the structure of the questions that the research will use in the interview. [Feedback for Answer Choice 4] Marci is engaged in more than simple conversation. [Q2] 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Which of the following is most important when conducting focus group interviews? 1. Include small but heterogeneous groups to collect richer data 2. Leave the process open-ended without much researcher interference 3. Have a skilled moderator who has a topic guide [correct] 4. Take very careful field notes during the interview [Feedback for Answer Choice 1] Diverse groups produce rich data. [Feedback for Answer Choice 2] Focus groups require high levels of organization from the researcher. [Feedback for Correct Answer 3] To be effective, focus groups need to be conducted by someone with skill in interviewing and group dynamics. This person should have a basic plan for the focus group but probably will not be able to take field notes while participating. Additionally, the group should be homogeneous with respect to important participant characteristics. [Feedback for Answer Choice 4] Focus groups should be differentiated from interviews, as interviews are typically oneon-one in nature. [Q3] In which of the following types of qualitative interviews do the question pattern emerge from the immediate context and are asked in the natural course of events spontaneously? 1. Semistructured interview 2. Informal conversational interview [correct] 3. Reflective interview 4. Predetermined interview [Feedback for Answer Choice 1] With the semistructured interview, topics and some possible questions are selected in advance, but the researcher decides the sequence and wording of the questions during the interview and may use pre-established prompts and probes. [Feedback for Correct Answer 2] 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

In an informal conversational interview, the questions emerge from the immediate context and are asked in the natural course of events spontaneously. [Feedback for Answer Choice 3] This is not a particular type of qualitative interview approach. [Feedback for Answer Choice 4] This is not a particular type of qualitative interview approach. [Q4] Which of the following is considered a dichotomous question? 1. Did the teachers’ views of differentiated instruction change after the professional development session? [correct] 2. In what ways did the teachers’ views of differentiated instruction change after the professional development session? 3. How did the teachers’ views of differentiated instruction change after the professional development session? 4. What did the teachers learn from the differentiated instruction professional development session? [Feedback for Correct Answer 1] Dichotomous questions leave the participant with the ability to essentially answer in one of two ways. In this instance, the answer is either yes or no. Dichotomous questions typically are not good qualitative interview questions unless extensive follow-up questions are incorporated. [Feedback for Answer Choice 2] Because there are numerous ways this question can be answered, this is an openended question. [Feedback for Answer Choice 3] Because there are numerous ways this question can be answered, this is an openended question. [Feedback for Answer Choice 4] Because there are numerous ways this question can be answered, this is an openended question. Learning Outcome 12.4: Distinguish among document, artifact, and visual material data collection techniques. 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q1] Which of the following categories best describes a high school yearbook? 1. Artifact 2. Visual material 3. Document [correct] 4. Interview material [Feedback for Answer Choice 1] Artifacts are not typically produced as formal written records, like a yearbook, so this is not the best answer. [Feedback for Answer Choice 2] While visual material may be evident within a yearbook, a yearbook is produced with intent as a formal written record. [Feedback for Correct Answer 3] A yearbook is considered a document as it serves as a formal written record. [Feedback for Answer Choice 4] Interview materials may refer to transcribed audio recordings, but this is not the correct answer. [Q2] Which of the following categories would best describe a data source that consists of comments written in a student file? 1. Artifact [correct] 2. Visual material 3. Document 4. Interview material [Feedback for Correct Answer 1] Artifacts are not typically produced as formal written records, but they consist of archival sources that help to tell a story. Notes in a file would be considered artifactual data. [Feedback for Answer Choice 2] Visual materials usually refers to photographs, video, drawings, etc. Simple notes would not constitute visual material. 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] Documents are written records, but they are usually considered to be formally written records. Notes would not be considered formal documents. [Feedback for Answer Choice 4] Interview materials may refer to transcribed audio recordings, but this is not the correct answer. [Q3] John is interested in analyzing the Facebook profiles of incoming college freshmen to determine what themes existed in the selection of particular colleges. Which of these categories best describe John’s data? 1. Artifact 2. Visual material [correct] 3. Document 4. Interview material [Feedback for Answer Choice 1] Artifacts are not typically produced as formal written records, but they consist of archival sources that help to tell a story. This is not the type of data that John is collecting. [Feedback for Correct Answer 2] Visual materials are photographs, videotapes, drawings, paintings, movies, documentary films, graphics, social media, or other visual representations that go beyond a narrative description to provide important understandings and insights. [Feedback for Answer Choice 3] Documents are written records, but they are usually considered to be formally written records. John is not considering written records. [Feedback for Answer Choice 4] Interview materials may refer to transcribed audio recordings, but this is not the correct answer. [Q4] Which of the following is a major advantage of documents, artifacts, and visual materials as compared to interviews? 1. Is good backup if observations are not possible or are impractical 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. Provides alternative sources for triangulation [correct] 3. Facilitates verbatim transcriptions as raw data 4. Allows researcher to control the conversation and obtain the information needed. [Feedback for Answer Choice 1] This is an attribute of interviews. [Feedback for Correct Answer 2] Documents, artifacts, and visual aids are often quite helpful in triangulating interview and observation data. This is a major advantage of these sources of data. [Feedback for Answer Choice 3] This is an attribute of interviews. [Feedback for Answer Choice 4] This is an attribute of interviews.

Chapter 12 Application Exercises Chapter 12: Qualitative Data Collection Application Exercise 12.1: Collecting Observational Data Learning Outcome 12.2: Know how to collect observational data. [Q1] Cory is a graduate student researcher who wants to study the literacy practices of the parents of preschoolers in his community. Consider Cory’s situation as you answer the following question. What procedures should Cory use for developing questions to conduct structured interviews in his study of parent literacy practices? [Q1 Model Response] He will develop a set of questions ahead of time for use with all participants in the study. Cory should include a mix of open-ended and closed questions for his interview protocol. Next, it will be important to pilot test the interview questions with individuals similar to his participants. Based on the feedback from the pilot testing, he can revise the questions. [Q2] Cory is a graduate student researcher who wants to study the literacy practices of the parents of preschoolers in his community. Consider Cory’s situation as you answer the following question. Cory decides to conduct observations as a nonparticipant observer. How might he conduct the observations for the study of the literacy practices of the parents of preschoolers? [Q2 Model Response] As a nonparticipant observer, Cory might receive permission to sit out of the way as parents read to their children. The benefit of this approach is that he would be able to observe their behaviors and actions as they occur in their natural setting, without intruding in the literacy practices or becoming emotionally involved. It will be important to take field notes to document the observations. A simple observational protocol will assist in recording the observation sessions. [Q3] Consider the following: 12 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Cory is a graduate student researcher who wants to study the literacy practices of the parents of preschoolers in his community. Consider Cory’s situation as you answer the following question. What role could memoing play in Cory’s organizational process? [Q3 Model Response] Memoing is a technique for bracketing possible perceptions to establish reflexivity. This begins with the selection of a research topic. Cory has selected his research topic, so he is off to a good start. Now, though, he could use memoing to work through any potential bias he may possess, critique his own methods, and generally work to produce a more credible study. Application Exercise 12.2: Collecting Observational Data Learning Outcome 12.3: Describe how interview data are collected. [Q1] One of the most challenging tasks for a new qualitative researcher is to write quality open-ended questions as opposed to dichotomous interview questions. Review Table 12.4 entitled Dichotomous Questions Revised to be Open-Ended in order to revise the following sample question used in a study focused on teachers’ summer experiences and presented in dichotomous form, to a quality, open-ended question. Do teachers face planning challenges over the summer months? After revising the question, discuss ways the new open-ended question allows for improved data. [Q1 Model Response] Instead of this dichotomous question, instead consider, “What planning challenges do you typically face during the summer months?” This way, teachers are able to discuss any challenges they may face, allowing the researcher to develop follow-up questions and gain more knowledge. Even if a teacher responds indicating they do not face challenges, the researcher is able to follow up and dive deeper into this answer. [Q2] One of the most challenging tasks for a new qualitative researcher is to write quality open-ended questions as opposed to dichotomous interview questions. Review Table 12.4 entitled Dichotomous Questions Revised to be Open-Ended in order to revise the following sample question used in a study focused on the impacts of a professional development session and presented in dichotomous form, to a quality, open-ended question. 13 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Did the professional development focused on differentiated instruction improve your teaching? After revising the question, discuss ways the new open-ended question allows for improved data. [Q2 Model Response] Instead of this dichotomous question which will yield limited information, instead ask, “How has the professional development focused on differentiated instruction impacted your teaching?” These simple changes transform the question from a quick yes or no response to a question that allows for deep discussion. [Q3] One of the most challenging tasks for a new qualitative researcher is to write quality open-ended questions as opposed to dichotomous interview questions. Review Table 12.4 entitled Dichotomous Questions Revised to be Open-Ended in order to revise the following sample question used in a study focused on teacher involvement in community activities and presented in dichotomous form, to a quality, open-ended question. Do you frequently participate in activities in your community? After revising the question, discuss ways the new open-ended question allows for improved data. [Q3 Model Response] This question, as presented, is only going to yield a small amount of information. Knowing whether or not a teacher is involved in the local community is important, but it would be more important to better understand how. Therefore, consider asking, “In what ways do you participate in local community events?” This allows the person being interviewed to expand in a number of ways, even if the response is that they do not often participate. Application Exercise 12.3: Using Peer Helpers Learning Outcome 12.4: Distinguish among document, artifact, and visual material data collection techniques. [Q1] Review the section entitled Documents, Artifacts, and Visual Materials to answer the following question:

Janelle is conducting a qualitative study on the ways in which teacher professional dress has evolved over time. Before interviewing participants, she is collecting data using school yearbooks and memos regarding teacher professional dress. Is Janelle reviewing documents, artifacts, or visual materials? Explain your answer. [Q1 Model Response] Janelle is reviewing documents in this example. Documents are written records. They can be virtually anything written or printed, such as yearbooks, e-mails, websites, school budgets, dropout rates, committee minutes, memos, letters, newspapers, diaries, test scores, and books. In this case, the yearbooks and memos Janelle is reviewing would be considered documents. [Q2] Review the section entitled Documents, Artifacts, and Visual Materials to answer the following question: Manuel is conducting a qualitative study on the racial integration of high school sports in a local community. Before interviewing participants, he is reviewing several sources of information, including statistical records, sports team photographs, and film of local sports competitions. Is Manuel reviewing documents, artifacts, or visual materials? Explain your answer. [Q2 Model Response] In this instance, Manuel is reviewing artifacts. Artifacts are archival sources that are different from documents. These would include comments in student files; record of testing results; statistical data; objects such as athletic letters, trophies, posters, and awarded plaques; bulletin boards; photographs and videos; art objects; film; physical trace evidence (e.g., wearing on the floor); e-mails; ritual objects; and sounds, smells, and tastes. [Q3] Review the section entitled Documents, Artifacts, and Visual Materials to answer the following question: Julio is conducting a qualitative study on the ways in which teacher social media use impacts career advancement. Before interviewing participants, he has navigated the Facebook and Instagram profiles of a number of individuals who include “teacher” in their biographical description. His goal is to gain insight into how teachers may engage in social media practices as a function of their profession. Is Julio reviewing documents, artifacts, or visual materials? Explain your answer. 15 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q3 Model Response] Julio is reviewing visual material in this study. Visual materials are photographs, videotapes, drawings, paintings, movies, documentary films, graphics, social media, or other visual representations that go beyond a narrative description to provide important understandings and insights. Visual material includes what can be documented in such platforms as YouTube, Instagram, Facebook, and Twitter, as well as logs, bulletin boards, online communities, and websites. (A new branch of ethnography that uses social media is called netnography.)

Chapter 12 Test Items SELF-ASSESSMENT ITEMS 1) Which of the following data collection methods would you likely find in a qualitative study? a) Interviews and focus groups b) Observations c) Document analysis d) All of the above 2) Which of the following is NOT true about interviews in qualitative research designs? a) Preestablished prompts may be used to clarify or elaborate participants’ responses. b) In an unstructured interview, the researcher begins the interview with a list of prespecified questions with precise wording. c) When selecting participants, qualitative researchers consider a participant’s willingness and ability to communicate. d) Qualitative researchers do not ask questions that can be answered dichotomously. 3) Sometimes a qualitative researcher holds interviews with participants who are information-rich in order to gain the information needed. This technique is referred to as which of the following? a) Ethnographic discussion b) Focus group interview c) Group discussion d) Key informant interview 4) Which of the following is a process researchers use to remain reflective throughout the process of conducting qualitative research? a) Observation b) Coding c) Memoing d) Transcribing 5) Which of the following is NOT a role researchers typically take when conducting qualitative research? a) Document and/or artifact reviewer 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Interviewer c) Participator d) Observer 6) A researcher who has an established role within the setting they are conducting research would be considered a a) Complete outsider b) Complete insider c) Partial participant d) Partial researcher 7) What are the two types of information field notes usually include? a) Descriptive and reflective b) Reflective and speculative c) Descriptive and detailed d) Reflective and affective 8) When a researcher asks an interviewee to clarify or elaborate on something during an interview, the researcher is using which of the following interview techniques? a) Informal conversation b) Memoing c) Structured interviewing d) Probing

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Which of the following is NOT a true statement about data collection in a qualitative research study? a) The raw data collected in an observational study can be detailed, written descriptions. b) In phenomenological studies, the researcher will typically use a checklist to record observations of behavior. c) In some qualitative research designs, the investigator becomes a member of the group being studied. d) It is common for researchers to change their role as data are collected. 2) Which of the following is NOT an example of qualitative data collection? a) Conducting one-on-one interviews b) Observing children’s behavior during high-stakes testing c) Collecting survey data from principals in a school district d) Gathering copies of honor code policies from each school in a district 3) Sometimes qualitative researchers unexpectedly decide they need to collect additional data based on their initial findings. This process is referred to as which of the following? a) Recursive data collection b) Ad hoc data collection c) A priori data collection d) Preclusive data collection 4) Which of the following is NOT true about focus groups? a) They usually involve between 8 and 12 people. b) They are used to promote interaction among individuals. c) They typically last about one to two hours. d) The information collected is usually less useful than that which is obtained from interviews. 5) A researcher observes children’s social interactions during recess for one week at a private elementary school. After each class completes their recess time, the researcher quickly takes notes about what has been just observed, noting why the observations were important to record and any underlying assumptions made about the observations. This researcher is engaging in which of the following? 19 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

a) External audit b) Probing c) Peer debriefing d) Memoing 6) A researcher who is totally detached from the environment and behavior of the participants who are being studied is an example of which type of observer? a) Passive participant b) Complete participant c) Complete observer d) Participant observer 7) Which of the following is NOT true about interviews in qualitative research designs? a) Individuals who are particularly knowledgeable and articulate are often selected. b) One type of unstructured interview is an informal conversation. c) A set of questions or prompts is determined before the interview begins. d) Audio recordings are rarely used. 8) Which of the following is an example of a descriptive field note? a) The students appeared angry as they entered the classroom. b) The teacher kept looking out the window as she spoke. c) The teachers seemed agitated by the principal’s presence. d) I suspect the students were motivated by the upcoming field trip. 9) Which of the following is NOT a step in the process of designing qualitative interviews? a) Identify participants b) Pilot-test the protocol c) Identify the codes d) Select the setting 10) Which of the following is NOT a weakness of conducting a document, artifact, or visual analysis? a) They do not allow probing for additional information b) It is relatively inexpensive to conduct c) You may gather incomplete information d) The data are not naturally occurring behaviors 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Short-Answer Questions 1) Why is data in qualitative research collected in recursive steps? 2) What are the differences between a structured, semi-structured, and unstructured interview? Essay Question Identify an issue that could be investigated by a qualitative study and write a corresponding qualitative research question. Describe which qualitative research design you would use and how you would go about answering the research question. Justify your choices by providing specific details.

Chapter 12 Test Answer Key ANSWER KEY: Chapter 12 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. d 2. b 3. d 4. c 5. c 6. b 7. a 8. d

INSTRUCTOR’S EXAM ITEMS 1. b 2. c 3. a 4. d 5. d 6. c 7. d 8. b 9. c 10. b

Chapter 13 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 13: Qualitative Data Analysis, Validity/Trustworthiness, and Reporting ... 1 Learning Outcome Quizzes ........................................................................ 1 Application Exercises ................................................................................ 12 Test Items ................................................................................................. 17 Test Answer Key ....................................................................................... 22

Chapter 13: Qualitative Data Analysis, Validity/Trustworthiness, and Reporting Chapter 13 Learning Outcome Quizzes Learning Outcome 13.1: Be able to code qualitative data. [Q1] Which of the following types of data contain information provided by the participants, in their own words? 1. Emic [correct] 2. Etic 3. Codes 4. Themes [Feedback for Correct Answer 1] Emic data contain information provided by the participants, in their own words. [Feedback for Answer Choice 2] Etic data are representations of participant interview data by the researcher. [Feedback for Answer Choice 3] Codes refer to identifiers created to group like subject matter in an interview transcript. [Feedback for Answer Choice 4] Themes, or categories, are produced to describe common topics prevalent in a dataset. [Q2] Which of the following types of data are best described as the researcher’s representations of the participant’s specific language? 1. Codes 2. Themes 3. Etic [correct] 4. Emic [Feedback for Answer Choice 1] Codes refer to identifiers created to group like subject matter in an interview transcript. 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 2] Themes, or categories, are produced to describe common topics prevalent in a dataset. [Feedback for Correct Answer 3] Etic data are representations of participant interview data by the researcher. [Feedback for Answer Choice 4] Emic data contain information provided by the participants, in their own words. [Q3] Which of the following terms best describes labels that are used to provide a common way of describing text? 1. Themes 2. Emic 3. Etic 4. Codes [correct] [Feedback for Answer Choice 1] Themes, or categories, are produced to describe common topics prevalent in a dataset. [Feedback for Answer Choice 2] Emic data contain information provided by the participants, in their own words. [Feedback for Answer Choice 3] Etic data are representations of participant interview data by the researcher. [Feedback for Correct Answer 4] Codes are labels that are used to provide a common way of describing text. [Q4] Though the number varies by study, which of the following ranges represents a common number of codes in a qualitative study? 1. 30 to 50 [correct] 2. 5 to 10 3. 10 to 15 4. 50 to 70 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Correct Answer 1] While the number of codes used will vary from one study to another, typically 30 to 50 codes will be used. Some may be major codes, which tend to be broad, whereas others may be subcodes, which are divisions among the major codes. [Feedback for Answer Choice 2] This is not the correct range. [Feedback for Answer Choice 3] This is not the correct range. [Feedback for Answer Choice 4] This is not the correct range. Learning Outcome 13.2: Know how to analyze and interpret qualitative data, to identify patterns and to develop themes. [Q1] Which of the following terms is defined as being formed from coded data as a more general and abstract idea that represents the meaning of similarly coded information? 1. Findings 2. Groups 3. Codes 4. Categories [correct] [Feedback for Answer Choice 1] Ultimately, every researcher wants to arrive at the findings stage, but this is not the term that answers this question. [Feedback for Answer Choice 2] While the term groups may generically refer to a sorting of individuals or items into an organizational structure based on likeness, this is not the term that is used in qualitative research. [Feedback for Answer Choice 3] Codes are simply categories that are used to describe a piece of qualitative data. Qualitative studies usually contain between 30 and 50 codes. [Feedback for Correct Answer 4] 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

A category is formed from coded data as a more general and abstract idea that represents the meaning of similarly coded information. There are both major categories and subcategories in most studies. [Q2] The repeated application of a category to fit codes and data segments is known as which type of process? 1. Repetitive 2. Goodness of fit 3. Recursive [correct] 4. Analytic [Feedback for Answer Choice 1] While the process does involve repetition, this is not the best term for this question. [Feedback for Answer Choice 2] It is true that the qualitative researcher is working to “fit” together codes and themes, but this is not a phrase used in this type of research. [Feedback for Correct Answer 3] The repeated application of a category to fit codes and data segments is known as a recursive process. This process occurs when researchers return to the data to develop codes. [Feedback for Answer Choice 4] While this is a form of data analysis, this is not the best term for this question. [Q3] The process by which the researcher is continually searching for both supporting and contrary evidence about the meaning of a category is known as which of the following? 1. Constant comparative [correct] 2. Constant reflection 3. Consistent comparison 4. Consistent reflection [Feedback for Correct Answer 1] In constant comparative processes, the researcher is continually searching for both supporting and contrary evidence about the meaning of the category. This reinforces 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

the idea that qualitative research is an ongoing, active, and often cyclical process of analyzing and re-analyzing data. [Feedback for Answer Choice 2] In action, the qualitative researcher is engaged in a constant form of reflection. However, this is not a phrase used in these circumstances. [Feedback for Answer Choice 3] Consistent comparison is not a phrase used to describe this qualitative research process. [Feedback for Answer Choice 4] Consistent reflection is not a phrase used to describe this qualitative research process. [Q4] Which of the following represents a common software for analyzing qualitative data? 1. MAXQDA [correct] 2. R 3. SPSS 4. SAS [Feedback for Correct Answer 1] MAXQDA is a prominent software used to analyze qualitative data. [Feedback for Answer Choice 2] R is an open-sourced program used in statistical analysis. [Feedback for Answer Choice 3] SPSS is a statistical analysis software. [Feedback for Answer Choice 4] SAS is a statistical analysis software. Learning Outcome 13.3: Understand the processes of establishing validity/trustworthiness and transferability for qualitative studies. [Q1]

Which of the following strategies for establishing trustworthiness is completed by asking a colleague or another person to review the study for credibility and determine whether the results seem to follow from the data? 1. Peer debriefing [correct] 2. Member checking 3. Triangulation 4. Negative case analysis [Feedback for Correct Answer 1] Peer debriefing is completed by asking a colleague or another person to review the study for credibility and determine whether the results seem to follow from the data. [Feedback for Answer Choice 2] Member checking takes place when interview participants are asked to review transcripts and themes to check to make sure the researcher has interpreted the participant correctly. [Feedback for Answer Choice 3] Triangulation is a technique that seeks convergence of findings, cross validation, among different sources and methods of data collection. However, it is focused on data sources, not analysis. [Feedback for Answer Choice 4] Negative case analysis includes actively looking for findings that present discrepant information is needed to reflect the reality that not all data will provide the same information, and to change results when justified to establish confirmability. [Q2] Zackary has completed a series of teacher interviews focused on their experiences with professional development sessions offered by school districts. He has also collected data that include a formal list of professional development opportunities offered by the districts as well as lesson plans that the teachers indicate were impacted by the professional development process. Which of the following terms best describes Zackary’s approach to collecting multiple forms of data? 1. Member checking 2. Trustworthiness 3. Triangulation [correct] 4. Transferability 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 1] Member checking is completed when the researcher asks the participants to review transcripts, interpretations, and conclusions to confirm accuracy and authenticity. Zackary is not engaged in member checking by collecting multiple forms of data. [Feedback for Answer Choice 2] Validity/trustworthiness addresses whether the data collection strategies and analyses warrant the findings and claims. This does not describe Zackary’s process. [Feedback for Correct Answer 3] Triangulation addresses confirmability. It is a technique that seeks convergence of findings, cross validation, among different sources and methods of data collection. That is, data are collected from different individuals at different times or in different places, or several sources of data are used to see whether the results are consistent. [Feedback for Answer Choice 4] Transferability refers to the degree to which a study can be “transferred” from one situation to another, similar to generalizability in quantitative research. [Q3] Lawrence has completed a study that involved semistructured interviews of principals related to their views on standards-based instruction. After data analysis, Lawrence sent copies of his transcripts and findings to the principals and asked them to check for accuracy and interpretation. Which of the following terms best describes this process? 1. Member checking [correct] 2. Triangulation 3. Confirmation 4. Thematic analysis [Feedback for Correct Answer 1] Member checking is completed when the researcher asks the participants to review transcripts, interpretations, and conclusions to confirm accuracy and authenticity. This could be accomplished by having participants review interviewer or observer conclusions about what was said or done if there is no recorded transcript. [Feedback for Answer Choice 2] Triangulation is a technique that seeks convergence of findings, cross validation, among different sources and methods of data collection. That is, data are collected from 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

different individuals at different times or in different places, or several sources of data are used to see whether the results are consistent. [Feedback for Answer Choice 3] While in a sense, Lawrence is doing “confirmatory work,” this is not the most accurate response. [Feedback for Answer Choice 4] Thematic analysis is conducted once codes are in place and the researcher determines how those codes are connected thematically. This is not the process being described. [Q4] Which of the following refers to the appropriateness of applying the results of qualitative research to other contexts and settings? 1. Transferability [correct] 2. Generalizability 3. Validity 4. Reliability [Feedback for Correct Answer 1] Transferability refers to the appropriateness of applying the results to other contexts and settings. This term is used in relation to qualitative research. [Feedback for Answer Choice 2] While this term means something similar, generalizability is a concept used in quantitative research. [Feedback for Answer Choice 3] Validity/trustworthiness addresses whether the data collection strategies and analyses warrant the findings and claims. [Feedback for Answer Choice 4] Reliability is a term used in quantitative research, usually to refer to the accuracy of instruments or measures. Learning Outcome 13.4: Differentiate among different strategies for reporting qualitative findings, including how to establish an effective balance between participant quotes and narrative. [Q1] 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

When writing a qualitative research report, which of the following best describes the quantity of quotes that should be used? 1. Quotes should be used from multiple participants and in such a way that all themes are represented. [correct] 2. All of the quotes from an interview should be used in order for data to be accurately presented. 3. Quotes are never necessary in qualitative research. 4. Quotes should be used sparingly, so as not to burden the reader. [Feedback for Correct Answer 1] Quotes add quality and context to a research report. All themes should be represented, and multiple participants are included in a well-written qualitative research report. [Feedback for Answer Choice 2] This would be an inefficient way to produce a report. The report would be overly burdened by quotes and difficult for the reader to consume. [Feedback for Answer Choice 3] Quotes are a vital component of qualitative research. They add context and provide evidence. [Feedback for Answer Choice 4] While it is true that the reader should not be overburdened with quotes, it is not necessary to consider them sparingly. In this approach, the reporter may leave out critical quotes that provide important context. [Q2] Consider the following excerpt from a research report: Teachers discussed how block scheduling allowed them to better focus their lessons. They felt that the extended amount of time afforded to them in a block allowed them to have a proper introduction, lesson, and conclusion. To the contrary, teachers felt rushed in the shorter, year-long class schedule. Which of the following best describes this passage? 1. The researcher has used too many quotes. 2. The researcher has over-relied on narrative and not included enough quotes. [correct] 3. The researcher’s report does not need quotes. 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

4. Quotations would not add any additional information to this researcher’s report. [Feedback for Answer Choice 1] This researcher has not used any quotes. [Feedback for Correct Answer 2] While this narrative is perfectly fine, it is incomplete. By including quotes from teachers, this narrative could be strengthened. [Feedback for Answer Choice 3] While this narrative is OK, it is not convincing as a standalone section and could be improved. [Feedback for Answer Choice 4] To the contrary, this research report could be enhanced by allowing readers a chance to better understand the words of the teachers that led to this narrative. [Q3] Consider the following excerpt from a research report: Teachers were primarily pleased with block scheduling. For example: I believe block scheduling allows me to focus my teaching. Another stated: Before block scheduling, I never felt like I had enough time. Now I do! Still another said: I wish block scheduling was around when I was in high school! Which of the following best describes this passage? 1. The researcher has used too many quotes in relation to narrative. [correct] 2. The researcher has over-relied on narrative and not included enough quotes. 3. The researcher’s report does not need quotes. 4. Further narrative would not add any additional information to this researcher’s report. [Feedback for Correct Answer 1] This researcher has offered very minimal narrative. While this researcher has used many quotes, the interpretation is largely left to the reader. The comparison of quotes to narrative is not proportional. 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 2] This researcher has hardly used any narrative. [Feedback for Answer Choice 3] Quotes are necessary and help to support qualitative reporting. [Feedback for Answer Choice 4] Further narrative would better situate the provided quotations in context. The researcher should consider enhancing the narrative component of this report. [Q4] When reporting findings in qualitative research, which of the following is the best way to handle participant identities? 1. It is important to change all of the details of your participants, not just their identities. 2. Pseudonyms should never be used. 3. Participant identities should be revealed in order to enhance trustworthiness. 4. Participant identities should be made anonymous. [correct] [Feedback for Answer Choice 1] In this situation, key elements of the study may be changed, thus yielding the results inaccurate. [Feedback for Answer Choice 2] Pseudonyms are often used in qualitative research. [Feedback for Answer Choice 3] If participants believe their identities will be revealed, they may be less likely to provide honest data. [Feedback for Correct Answer 4] It is important to keep participant identities confidential. This means using pseudonyms and coding systems that protect the identity of participants.

Chapter 13 Application Exercises Chapter 13: Qualitative Data Analysis, Validity/Trustworthiness, and Reporting Application Exercise 13.1: Organizing and Coding Qualitative Data Learning Outcome 13.1: Be able to code qualitative data. [Q1] Consider the following: Grant is a qualitative researcher conducting research focused on the ways in which math teachers approach the teaching of number sense to students with mathematicsbased disabilities. Grant conducts a series of semistructured interviews with 10 participants. Grant took notes while each participant spoke and then went back and added his own narrative to help summarize the conversation. What key mistake in organizing data did Grant make? How could his process be improved? [Q1 Model Response] Interviews are a common source of data collection in qualitative research. It is good that Grant took notes, and it is good that he added his own narrative. However, recall that it is critical to capture the conversation as close to verbatim as possible in order to ensure accurate and complete data for analysis. Since Grant did not record the conversations, his coding process will be severely diminished, and his findings lose trustworthiness. Adding a simple audio recording would allow Grant to more thoroughly and appropriately analyze his data. [Q2] Consider the following: Rich and Diane are both qualitative researchers and they are partnering on a study focused on the experiences of first-generation college students as they attend a local community college. They have conducted 20 interviews with participants and have just received the transcribed interviews. Rich argues that they should remove “filler” words such as “um” from the transcript so that the coding process is easier. Diane disagrees and feels that this will change the structure of the language in the interviews. Which of these researchers is correct? Explain your answer. [Q2 Model Response] This scenario outlines a common debate in the field of qualitative research. Actually, neither of these researchers are “correct.” Researchers differ about whether to include 12 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

“filler” words in a transcript (e.g., “um,” “like,” “you know”). Some argue that leaving them out compromises the integrity of the transcript, while others contend that leaving them in makes the transcript more “noisy,” reducing the signal of the participant’s perspective. Whether or not to include these words is up to the researchers and their team. [Q3] Consider the following: Jamal is a qualitative researcher who recently completed interviews for a study focused on the impacts of gentrification on community members who have lived in the community for 20 years or more. Jamal conducted 24 interviews and had the interviews transcribed by a transcription service. He first read through each interview transcript in its entirety, then developed his coding structure. He has now finished coding the transcripts, but is concerned that there is more he could do to ensure that his study was considered a rigorous qualitative study. At this point, how might Jamal improve his methodological process in order to make his study more trustworthy? [Q3 Model Response] Jamal has taken all the right steps, but his study could be improved by having another researcher code his transcripts. This way, he could establish inter-coder agreement. Often, several individuals will code the same text to determine inter-coder agreement, and if Jamal is able to recruit multiple colleagues to help, this will improve his study. Application Exercise 13.2: Qualitative Research You Can Trust Learning Outcome 13.3: Understand the processes of establishing validity/trustworthiness and transferability for qualitative studies. [Q1] Consider the following: Shavoyae is a qualitative researcher who has completed a study that included interviewing high school teachers regarding their classroom rules and procedures. His participants all worked in the same school and each participant was interviewed only once. While Shavoyae feels he gained a substantial amount of important knowledge, he is concerned about the trustworthiness of his findings. What steps can Shavoyae take to improve the trustworthiness of his study? [Q1 Model Response]

While the data Shavoyae has collected are important and valuable, he has not taken all the steps he can take to ensure a trustworthy study. The most obvious step Shavoyae could take would be to triangulate his data. He should seek data from different individuals at different times or in different places. He may also consider other sources of data outside of only interviews. All of these steps would help him with developing his coding structure and analyzing his data, thus improving trustworthiness. [Q2] Consider the following: Kasey is a qualitative researcher who has just finished a study that considers the perspectives of school nurses on their role in the COVID-19 public health crisis. She conducted 30 interviews with school nurses all over the state of South Carolina, collected data on the role of school nurses from a majority of school districts in the state, and contacted representatives from the School Nurses Association for input. She read through all of her transcripts in their entirety, established a coding structure, and coded her data before analyzing for themes. Still, Kasey was concerned about the trustworthiness of her study, so she asked her colleague, Christina, to review the study and determine whether the results she concluded seemed to follow the data. In this example, what is Christina’s role in the study? How does this enhance Kasey’s study? [Q2 Model Response] Christina is serving in a peer debriefing capacity for Kasey. Peer debriefing is completed by asking a colleague or another person to review the study for credibility and determine whether the results seem to follow from the data. Someone who is knowledgeable about the topic and qualitative analyses, but sufficiently detached to provide a fresh perspective, is preferred. That person’s own biases should be reflected in their evaluation, which gives feedback about the selection and meaning of categories, themes, patterns, and study conclusions. Peer debriefing is a common technique to address both researcher bias and confirmability. [Q3] Consider the following: Donovan is a qualitative research who is studying the impact of political television ads on the perceptions of voters. He has conducted interviews, gathered statistics, developed codes, established themes, and engaged in common member checking and peer debriefing procedures. As Donovan writes up his findings, he uses detailed, indepth, thorough, and extensive descriptions. What qualitative term would be used to describe the descriptive writing Donovan employs? How would this improve his study? 14 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q3 Model Response] Credible qualitative studies use detailed, in-depth, thorough, and extensive descriptions—which are sometimes described as “thick” and/or “rich.” That is, there is abundant use of detail. This enhances validity/trustworthiness because it indicates extensive engagement with the data and an appreciation of how all information is valuable. It enables the reader to understand the complexity and realism of the site and participants. For example, a rich, detailed description of a college student commons may be needed to understand the dynamics of students meeting there for discussions with faculty. Thick descriptions include presenting verbatim language from participants and detailed field notes. The research procedures should also be described in detail. Using thick descriptions addresses context insensitivity and inauthenticity threats to validity. Application Exercise 13.3: Reporting Qualitative Findings Learning Outcome 13.4: Differentiate among different strategies for reporting qualitative findings, including how to establish an effective balance between participant quotes and narrative. [Q1] Consider the following section from a qualitative research report: Participants discussed their preference for teachers who adhered to a constructivist style of classroom management. They discussed several examples of behaviorist principles serving as academic distractors. Additionally, they felt classes with stringent class rules and procedures hindered their relationships with their teachers, thus diminishing their schooling experiences. What has this researcher left out of her/his report that would help the reader better understand the study? [Q1 Model Response] In this example, the researcher did not include any quotes from participants. The narrative is clear, but as a reader you are left wondering what it was the participants actually said. Of course, the researcher does not have to quote all students, but a healthy selection of quotes to help explain the interpretation would improve this report. [Q2] Consider the following section from a qualitative research report: Participants discussed their preference for teachers who adhered to a constructivist style of classroom management.

One student said, “I hate it when teachers act like they are the boss of everyone. Aren’t we all supposed to work together?” Another said, “I really can’t focus when teachers keep talking about rules and procedures. I mean, we’ve been in school for how long? We know how to act.” What has this researcher left out of her/his report that would help the reader better understand the study? [Q2 Model Response] This researcher has provided, essentially, only quotes with very little accompanying narrative. Without a synthesizing narrative, little is done to elaborate on the importance of what these students are saying. This section is essentially a list of quotes, rather than any sort of empirical analysis. The reader is left wondering, “What is the nature of the connecting theme?” [Q3] Consider the following: When writing up his analysis section, Nachiket created a section that outlined both his descriptions of what was observed and recorded as well as his interpretations of the data. Is this appropriate? Explain your answer in support or opposition to Nachiket’s decision. [Q3 Model Response] In the core of a qualitative article, you will find descriptions of what was observed or recorded, along with interpretations of the data. The descriptions are the basis for the researcher’s analyses and interpretations. If these descriptions are not clearly separate from the analyses and interpretations, it is difficult for you to judge the reasonableness and relative objectiveness of the researcher’s claims (e.g., whether there was selective presentation of data or whether inductive processes seem reasonable on the basis of the data presented).

Chapter 13 Test Items SELF-ASSESSMENT ITEMS 1) Qualitative researchers actively look for findings that contradict themes, patterns, and overall results to enhance credibility. This technique is referred to as which of the following? a) Key informant interview b) Comprehensive analysis c) Typical case analysis d) Negative case analysis 2) When analyzing qualitative research, investigators do which of the following? a) Categorize the data and look for patterns, themes, and ideas b) Exclude data that do not fit a clear category c) Look for correlations between different variables d) Order the data according to a time sequence 3) Generalizability in qualitative research refers to which of the following? a) The extent to which the data were recorded accurately b) The extent to which the researchers addressed threats to internal and external validity c) The extent to which the results can be transferred to other contexts and settings d) The extent to which the person who wants to use the findings determines them generalizable 4) Which of the following will NOT help to strengthen the credibility of a qualitative research study? a) Conducting the study over a long period of time b) Actively looking for findings that present discrepant information c) Administering only one form of a measure to all groups of participants d) The researcher engaging in self-reflection of possible biases 5) Which of the following is NOT a criterion for evaluating qualitative research? a) Data collection methods should be limited to one or two to allow for triangulation. b) Descriptive data should be detailed and objective. c) The researcher should give detailed information regarding participant selection. d) The study should extend over a long period of time. 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

6) A qualitative researcher may ask participants to review their interpretations and conclusions to confirm the findings. This technique is referred to as which of the following? a) Equivalence b) Member checking c) Authentication d) Triangulation 7) Qualitative researchers often use recursive steps when analyzing data. Recursive analysis refers to which of the following? a) Repetitively transcribing data from audiotapes and videos b) Repeatedly using participants’ quotes in the write-up of the analysis c) Continually searching for both supporting and contrary evidence about the meaning of a category d) Deriving overarching models to illustrate relationships between themes

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Which of the following is NOT a true statement about data analysis in a qualitative research study? a) Computer software can be used to analyze qualitative data. b) Researchers always use a predetermined list of codes to analyze data. c) Codes can be grouped to form emerging themes. d) It is common for researchers to analyze multiple sources of data. 2) Which of the following is NOT a procedure used by qualitative researchers when analyzing data? a) Categorize the data and look for patterns, themes, and ideas b) Separate the data into workable units c) Order the data according to a time sequence d) Create codes for words or phrases that occur frequently in transcripts 3) A qualitative researcher usually uses iterative methods to identify themes, patterns, and relationships. This technique is referred to as which of the following? a) Recursive analysis b) Coding transcript analysis c) Negative case analysis d) Document and artifact analysis 4) Emic data: a) Contain information about the setting in which qualitative data is being conducted b) Are not commonly collected in qualitative research c) Are the researcher’s representations of information provided by the participant d) Contain information provided by the participants using their own words 5) Sometimes a qualitative researcher will look for findings that present discrepant information. This technique is referred to as which of the following? a) External audit b) Comprehensive analysis c) Peer debriefing d) Negative case analysis 19 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

6) Qualitative researchers often use multiple data sources to confirm the consistency of a trend or pattern. Which of the following is the term used for this technique? a) Equivalence b) Authentication c) Triangulation d) Contradiction 7) A researcher who is continually searching for both supporting and contrary evidence about the meaning of a category is engaging in: a) Consensus coding b) Content analysis c) Coding d) Constant comparison 8) The purpose of qualitative data analysis is which of the following? a) To justify the researcher’s preconceived conclusions b) To critically examine, summarize, and synthesize data c) To apply deductive logic to the analysis of data d) To provide detailed field notes and observations 9) Which of the following is NOT a criterion for evaluating the credibility of a qualitative research study? a) The researcher used multiple methods to collect data. b) Descriptions are presented so that they are clearly separate from the analyses and interpretations. c) The researcher addressed the extent to which the results generalize to other participants, settings, or procedures. d) A procedure, such as triangulation, was used by the researcher to check for consistency. 10) Which of the following will NOT help to strengthen the credibility of qualitative research? a) Conducting the study over a long period of time b) Having the researcher assume the role of a nonparticipant observer c) Collecting data from different individuals at different times or in different places to see if the results are consistent d) Having an external auditor who is unfamiliar with the project examine all aspects of the study 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Short-Answer Questions 1) Why is it important for a qualitative researcher to memo throughout a study and remain reflective? 2) How is computer software sometimes used by a qualitative researcher? Essay Question A college dean wants to conduct a qualitative study to explore reasons why graduate students continue to drop out of their programs prior to graduation. Describe a plan for data collection and analysis for the dean to consider, including suggestions for ways in which the credibility and trustworthiness of the findings can be boosted.

Chapter 13 Test Answer Key ANSWER KEY: Chapter 13 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. d 2. a 3. c 4. c 5. a 6. b 7. c

INSTRUCTOR’S EXAM ITEMS 1. b 2. c 3. a 4. d 5. d 6. c 7. d 8. b 9. c 10. b

Chapter 14 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 14: Mixed Methods Designs ........................................................................... 1 Learning Outcome Quizzes ........................................................................ 1 Application Exercises ................................................................................ 12 Test Items ................................................................................................. 17 Test Answer Key ....................................................................................... 23

Chapter 14: Mixed Methods Designs Chapter 14 Learning Outcome Quizzes Learning Outcome 14.1: Identify the major characteristics, advantages, and disadvantages of mixed methods research. [Q1] Which of the following is clearly an example of a mixed methods study? 1. A study that included focus group interviews and attitude surveys [correct] 2. A study that included participant observation and unstructured interviews 3. A study that included naturalistic observations followed by a case study 4. A study that included a survey with a rating scale and data from student achievement tests [Feedback for Correct Answer 1] The term mixed methods refers to the combination of qualitative methods (e.g., focus group interviews) with quantitative methods (e.g., attitude surveys) of study. [Feedback for Answer Choice 2] Each of these are qualitative methods, so this study would be a qualitative study. [Feedback for Answer Choice 3] Each of these are qualitative methods, so this study would be a qualitative study. [Feedback for Answer Choice 4] Each of these are quantitative methods, so this study would be a quantitative study. [Q2] Would the following question be suitable for a mixed methods study: “How does an open-book testing approach impact student scores on the end-of-course test?” 1. Yes. This study could not be approached with any one methodological approach. 2. Yes. This study has multiple variables. 3. No. This study seems best suited for qualitative methods. 4. No. This study seems best suited for quantitative methods. [correct] [Feedback for Answer Choice 1] 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This study could be conducted using just one methodological approach. [Feedback for Answer Choice 2] It is not the number of variables that indicate whether or not a study should be a mixed methods study or not. Remember to let the research question guide the method. [Feedback for Answer Choice 3] There is a quantitative component to this study, so it could not be completed with only qualitative methods. [Feedback for Correct Answer 4] This study is simply asking whether a treatment effect (open-book testing) influences student test scores. This can easily be done using only quantitative methods. [Q3] Which of the following best represents the purpose of mixed methods research? 1. To understand a phenomenon more fully than is possible using either quantitative or qualitative designs alone [correct] 2. To use systematic, structured qualitative methods to study a topic 3. To use quantitative approaches that are exploratory and emergent 4. To test hypotheses comparing the difference between quantitative and qualitative methods [Feedback for Correct Answer 1] The unique contribution of the mixed methods design comes from the more complete understanding of a phenomenon by using both quantitative and qualitative methods. [Feedback for Answer Choice 2] This would be a qualitative study, not a mixed methods study. [Feedback for Answer Choice 3] This would be a quantitative study, not a mixed methods study. [Feedback for Answer Choice 4] Since this study would involve hypothesis testing, it is actually just a quantitative study. [Q4] What is a disadvantage that is common to mixed methods research? 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

1. Mixed methods research can require extensive resources and effort. [correct] 2. Mixed methods research cannot establish cause and effect. 3. Mixed methods research is not possible for most research questions. 4. Mixed methods employs two types of data. [Feedback for Correct Answer 1] Due to the need to collect, analyze, and mix quantitative and qualitative data, this approach requires the researcher to have a wide range of skills and resources. [Feedback for Answer Choice 2] This is not a disadvantage unique to mixed methods research, as establishing causality is a difficult task. [Feedback for Answer Choice 3] Mixed methods research is actually possible with many research questions. [Feedback for Answer Choice 4] This can be an advantage of mixed methods research. The quantitative and qualitative data can work together to provide a clearer understanding of the research phenomenon. Learning Outcome 14.2: Describe the steps in conducting mixed methods studies. [Q1] Which of the following should be considered the first step in any mixed methods study? 1. Identify the purpose [correct] 2. Identify the rationale 3. Establish the feasibility of conducting a mixed methods study 4. Determine the mixed methods design [Feedback for Correct Answer 1] This is the first step of any mixed methods study. Essentially, the general purpose of the research needs to include elements that are best investigated with mixed methods. This could be a general research problem statement or question, or explanation of the issue that needs investigation. [Feedback for Answer Choice 2] This is the second step in a mixed methods study. [Feedback for Answer Choice 3] 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This is the third step in a mixed methods study. [Feedback for Answer Choice 4] This is the fourth step in a mixed methods study. [Q2] Which of the following best describe the two main types of sequential mixed methods designs? 1. Embedded and convergent 2. Convergent and exploratory 3. Exploratory and explanatory [correct] 4. Explanatory and convergent [Feedback for Answer Choice 1] This not the correct combination of terms. [Feedback for Answer Choice 2] This not the correct combination of terms. [Feedback for Correct Answer 3] The two main classifications of sequential designs are exploratory and explanatory. In exploratory designs, qualitative data are collected first. In explanatory designs, quantitative data are collected first. [Feedback for Answer Choice 4] This not the correct combination of terms. [Q3] In which of the following designs would a researcher use a quantitative survey to understand teacher practices of differentiated instruction implementation, followed by an interview process that asked teachers to explain their views on differentiation? 1. Embedded design 2. Convergent design 3. Exploratory sequential design 4. Explanatory sequential design [correct] [Feedback for Answer Choice 1] 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

In an embedded design, one data set serves as secondary support to another. The order of quantitative or qualitative as the primary dataset is irrelevant. In this case, the data are each able to stand alone and inform one another. [Feedback for Answer Choice 2] When conducting a study with a convergent design, the researcher collects and analyzes the two data sets at the same time. [Feedback for Answer Choice 3] In exploratory sequential designs, qualitative data are collected before quantitative data. [Feedback for Correct Answer 4] The explanatory sequential design is used when a researcher wants to obtain quantitative results and then qualitatively explain the results. [Q4] In which of the following designs is the sample in both the quantitative and qualitative component likely to be similar? 1. Embedded design 2. Convergent design [correct] 3. Exploratory sequential design 4. Explanatory sequential design [Feedback for Answer Choice 1] In embedded designs, the second set of data collected only serves as a support. These samples may be similar, but this is not the best answer. [Feedback for Correct Answer 2] A convergent design uses sampling that will allow a synthesis of findings from each group of participants. For this design, the samples are typically very similar so that it makes sense to combine data from each phase. [Feedback for Answer Choice 3] In exploratory sequential designs, qualitative data are collected first. This sample is likely to be much smaller than the sample for the quantitative phase. [Feedback for Answer Choice 4] In explanatory sequential designs, quantitative data are collected first. This sample is likely to be much larger than the sample for the qualitative phase. 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Learning Outcome 14.3: Classify, diagram, and know when to use different mixed methods designs. [Q1] Theresa wants to understand people's perceptions of bullying at a high school. She gathers data with a survey and collects several focus group interviews and then does an analysis in which she compares the two sets of results. Which mixed methods design did she use? 1. Convergent design [correct] 2. Embedded design 3. Exploratory sequential design 4. Explanatory sequential design [Feedback for Correct Answer 1] When conducting a study with a convergent design, the researcher collects and analyzes the two data sets at the same time. [Feedback for Answer Choice 2] In an embedded design, one data set serves as secondary support to another. This is not the case in Theresa’s situation. [Feedback for Answer Choice 3] In exploratory sequential designs, qualitative methods are used to generate information to be used in conducting the quantitative phase of the study. [Feedback for Answer Choice 4] In explanatory sequential designs, findings from quantitative methods are followed by qualitative methods to provide explanations for the quantitative findings. [Q2] Which mixed methods design begins by first studying a topic qualitatively and then using the results from that phase of the study to design the next phase? 1. Embedded design 2. Convergent design 3. Exploratory sequential design [correct] 4. Explanatory sequential design [Feedback for Answer Choice 1] 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

In an embedded design, one data set serves as secondary support to another. The order of quantitative or qualitative as the primary dataset is irrelevant. [Feedback for Answer Choice 2] When conducting a study with a convergent design, the researcher collects and analyzes the two data sets at the same time. [Feedback for Correct Answer 3] The purpose of this design is typically to use the initial qualitative phase with a few individuals to identify themes, ideas, perspectives, and beliefs for the larger-scale quantitative part of the study. [Feedback for Answer Choice 4] In this design, quantitative methods are employed before qualitative methods. [Q3] A school district conducts a parent survey and finds overwhelming dissatisfaction with the item addressing parent–teacher communication. Because the survey showed only percentages of responses on a 5-point scale, the district contracts with a research organization to conduct follow-up focus groups to identify the source of the problem. What type of design does this most closely resemble? 1. Embedded design 2. Convergent design 3. Exploratory sequential design 4. Explanatory sequential design [correct] [Feedback for Answer Choice 1] In an embedded design, one data set serves as secondary support to another. The order of quantitative or qualitative as the primary dataset is irrelevant. [Feedback for Answer Choice 2] When conducting a study with a convergent design, the researcher collects and analyzes the two data sets at the same time. [Feedback for Answer Choice 3] In exploratory sequential designs, qualitative data are collected before quantitative data. [Feedback for Correct Answer 4]

The explanatory sequential design is used when a researcher wants to obtain quantitative results and then qualitatively explain the results. [Q4] Carol is studying school safety using mixed methods. She describes her study using the following notation: QUAN → qual. Which of the following best describes her data collection and analysis? 1. She will collect the quantitative data first and weight it most heavily. [correct] 2. She will gather the data concurrently but give priority to the quantitative data. 3. She will collect quantitative data first, but the data will have equal priority. 4. She will embed the qualitative data collection phase in the quantitative phase. [Feedback for Correct Answer 1] The arrow notation indicates an explanatory sequential design, in which quantitative data are collected first. In addition, only the quantitative data are indicated with capital letters, so we can assume that she is planning to place priority on the quantitative data. [Feedback for Answer Choice 2] The notation system used does not indicate concurrent data collection. [Feedback for Answer Choice 3] She will collect quantitative data first, but the notation does not indicate an equal weighting. [Feedback for Answer Choice 4] The notation system does not indicate an embedded process. [Q5] Which question is most important to ask when evaluating an exploratory sequential mixed methods study? 1. Were the data merged properly for analysis? 2. How has the researcher explained conflicting findings that resulted from different methods? 3. Should this study have been a quantitative study, instead of a mixed methods study? 4. Was the qualitative phase conducted with sufficient detail to provide solid guidance for the quantitative component? [correct] [Feedback for Answer Choice 1] 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Data are not merged for analysis in an exploratory sequential design. [Feedback for Answer Choice 2] This is an important question, to be sure. However, this is not the most important question to ask. [Feedback for Answer Choice 3] Exploratory studies begin with qualitative data. [Feedback for Correct Answer 4] The purpose of the first phase of an exploratory-sequential mixed-methods study is to help determine the focus and type of data collection that will be used in the second, quantitative phase. Thus, for this particular design, this question is critical—if the answer is no, the second (more heavily weighted) phase of the study has low validity. Learning Outcome 14.4: Know the criteria for evaluating the quality of mixed methods research. [Q1] Sequential validity refers to which of the following? 1. Whether one phase of the study has appropriately built upon the other [correct] 2. Whether the quantitative half of the study has obtained proper validity 3. Whether the qualitative half of the study has obtained proper validity 4. Whether the study has obtained generalizability [Feedback for Correct Answer 1] In sequential designs, unique validity concerns focus on the connection of the two phases, and whether one has appropriately built on the other (sequential validity). [Feedback for Answer Choice 2] Sequential validity does not focus on only the quantitative half of the study. [Feedback for Answer Choice 3] Sequential validity does not focus on only the qualitative half of the study. [Feedback for Answer Choice 4] Generalizability is not measured by sequential validity. [Q2] 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

In a convergent study, contradictory findings that are not explicitly searched for and addressed could lead to issues with which of the following? 1. Trustworthiness 2. Generalizability 3. Bias [correct] 4. Validity [Feedback for Answer Choice 1] While the study may suffer at some level, trustworthiness would not be the primary area to suffer. [Feedback for Answer Choice 2] Generalizability refers to the ability to apply the results of the study to a similar population. Mixed methods studies can often suffer from an inability to generalize, but this is not the primary issue in this instance. [Feedback for Correct Answer 3] If contradictory findings are not explicitly searched for and addressed, bias could be introduced. [Feedback for Answer Choice 4] This would not directly impact validity of the study. [Q3] Focusing on which critical feature is the primary way to evaluate the validity and rigor of mixed methods designs? 1. The validity and reliability of the measures used in the quantitative half of the study 2. The adherence to solid interviewing principles in the qualitative half of the study 3. Considering sample sizes in both quantitative and qualitative studies 4. The intentional and substantial collection of both quantitative and qualitative data [correct] [Feedback for Answer Choice 1] This is important, of course, but not considered the primary way to evaluate mixed methods studies. [Feedback for Answer Choice 2] 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Interviewing principles should certainly be critiqued, but this is also true for qualitative studies. [Feedback for Answer Choice 3] Proper sampling is critical, but that is true for all studies. [Feedback for Correct Answer 4] What sets mixed methods studies apart from other research designs is the intentional and substantial collection of both quantitative and qualitative data. Focusing on this critical feature is the primary way to evaluate the validity and rigor of mixed methods designs. [Q4] Which of the following questions best applies to the evaluation of a mixed methods study? 1. Has the researcher matched the purposes and questions to appropriate methods? [correct] 2. Has the researcher weighted quantitative methods heavier, as required? 3. Has the researcher weighted qualitative methods heavier, as required? 4. Has the researcher collected quantitative and qualitative data for equal amounts of time? [Feedback for Correct Answer 1] This is a critical component of mixed methods research. Purposes and questions must match properly with appropriate methods. [Feedback for Answer Choice 2] There is no requirement to weight quantitative methods heavier than qualitative. [Feedback for Answer Choice 3] There is no requirement to weight qualitative methods heavier than quantitative. [Feedback for Answer Choice 4] Data collection periods may be quite different for each method. That’s typical in mixed methods research.

Chapter 14 Application Exercises Chapter 14: Mixed Methods Designs Application Exercise 14.1: Advantages and Disadvantages of Mixed Methods Research Learning Outcome 14.1: Identify the major characteristics, advantages, and disadvantages of mixed methods research. [Q1] Consider the following: George is an educational researcher who focuses on school inclusion for students with disabilities. Primarily, he is interested in the ways in which schools include students with disabilities in the general education classroom, as well as the experiences of students with disabilities in those classrooms. How could George implement a mixed methods design to explore each of his connected interests? Explain your answer. [Q1 Model Response] Mixed methods designs are useful when the results of quantitative data collection and analysis do not adequately explain results, and additional data are needed to help interpret the findings. In George’s case, he is interested in the ways in which students with disabilities are included, but those data will not help him understand how those experiences are playing out. In George’s case, a mixed methods design is a wise choice. He can use quantitative methods to survey school districts about their inclusion scheduling approaches, but then use qualitative methods to interview students, families, and teachers to better understand the nuances of those scheduling decisions. [Q2] Consider the following: George is an educational researcher who focuses on school inclusion for students with disabilities. Primarily, he is interested in the ways in which schools include students with disabilities in the general education classroom, as well as the experiences of students with disabilities in those classrooms. He has designed a mixed methods study that will allow him to use quantitative methods to survey school administrators and better understand inclusive scheduling practices, while also employing qualitative methods that include interviewing teachers, students, and families about inclusive experiences. What major advantages does George’s mixed methods design have over explicitly quantitative or qualitative studies? 12 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q2 Model Response] There are several advantages to using mixed methods designs when conducting research. The two most significant advantages are (1) the ability to provide a more thorough understanding of a research problem or issue because of the opportunity to examine multiple forms of data that are more comprehensive than data that might be collected via either quantitative or qualitative methods alone, and (2) the ability to answer complex research questions that cannot be addressed through the use of quantitative or qualitative methods alone. By conducting a mixed methods study to explore his research questions, George is reaping the benefits of both of these advantages. [Q3] Consider the following: George is an educational researcher who focuses on school inclusion for students with disabilities. By training, George is a quantitative researcher. Primarily, he is interested in the ways in which schools include students with disabilities in the general education classroom, as well as the experiences of students with disabilities in those classrooms. He has designed a mixed methods study that will allow him to use quantitative methods to survey school administrators and better understand inclusive scheduling practices, while also employing qualitative methods that include interviewing teachers, students, and families about inclusive experiences. While George’s study seems sound, what major disadvantages might George face? Explain your answer. [Q3 Model Response] While George’s study sounds like a great one, there also are some disadvantages to using mixed methods designs. First and foremost, the ability to successfully implement a mixed methods study will require a solid level of expertise and comfort with both quantitative and qualitative methods. A passing or rudimentary level of understanding of procedures and data analysis techniques of each is insufficient for conducting credible mixed methods studies. As a quantitative researcher, George may struggle with the qualitative components of his study if he does not bring a co-investigator aboard. Second, mixed methods research typically involves data collection (and subsequent analysis) that is more extensive and labor intensive, takes additional time, and often requires more resources than might be required of a study employing either quantitative or qualitative methods alone. George will need to make sure he moves quickly enough to capture the current state of the issue he’s studying. Application Exercise 14.2: The Steps to Conduct Mixed Methods Research Learning Outcome 14.2: Describe the steps in conducting mixed methods studies. 13 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q1] Consider the following: Jane is an educational researcher who is designing a mixed methods study focused on understanding how school districts determine advance placement offerings and how those offerings impact the college pathways of high school graduates. She has identified her purpose and rationale for the study and established that a mixed methods design will be feasible. Presently, she is working to develop her specific research questions. Which step in the mixed methods research design process has Jane skipped? How will this impact her study? [Q1 Model Response] Jane was off to a great start, but she failed to pause and determine the specific mixed methods design she will employ. Not all mixed methods studies are the same. She should go back and determine which specific mixed methods design will be employed. This depends on the purpose of the study and resources to complete both quantitative and qualitative phases. [Q2] Marcus is an educational researcher designing a mixed methods study to better understand the transition from homeschool to public school for students who switch schooling methods. He has designed a quantitative survey component and a qualitative interview component of his study. He has identified his purpose and rationale, established that a mixed methods study is feasible, decided on a specific design, and determined that he is ready to begin collecting data. Which step in the mixed methods research design process has Marcus skipped? How will this impact her study? [Q2 Model Response] Marcus was off to a good start, but he jumped ahead of himself and began thinking about data collection before he took his broad ideas and created more targeted research questions, which would have also led him to determine specific design elements. As such, his study is ungrounded, and he will have no platform from which to conduct data analysis. [Q3] Rodrigo is an educational researcher who just completed a mixed methods study focused on the role of the counselor in helping reduce school dropout. His study included a quantitative survey component, a series of individual interviews, and schoolbased focus groups. When designing his study Rodrigo identified a purpose and 14 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

rationale, established feasibility, determined his design and research questions, and determined specific design elements. He collected data and has just completed his data analysis. Which step in the mixed methods research process comes next for Rodrigo? How do you know he is ready for this next phase? [Q3 Model Response] Rodrigo has followed the mixed methods research process with fidelity. He has a wellplanned, well-executed study. Therefore, he is ready for the final step, which includes writing up his research report so that he can share his findings. Application Exercise 14.3: Identifying Mixed Methods Research Designs Learning Outcome 14.3: Classify, diagram, and know when to use different mixed methods designs. [Q1] Consider the following: Debbie is interested in adjustment for college freshman students coming from small rural high schools to a large university. She plans to use mixed methods research and is considering different possible purposes for her study. Identify the design she is considering in the following scenario: Debbie wants to measure the adjustment scores of freshmen from rural high schools and to identify the overall trends. She then wants to explain the trends by describing the experiences of three students with high adjustment, three with medium adjustment, and three with low adjustment. Explain what design she is thinking of using here, and how you know. [Q1 Model Response] This scenario would best fit the use of an explanatory sequential design. This design begins with the collection and analysis of quantitative data, such as to describe trends in a population with a survey instrument. Once Debbie completes the first quantitative phase, she can use the results to help design a follow-up qualitative phase. In the second qualitative phase, Debbie will select the best participants and gather her qualitative data to help explain the reasons behind the initial quantitative results. [Q2] Consider the following: Debbie is interested in adjustment for college freshman students coming from small rural high schools to a large university. She plans to use mixed methods research and is 15 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

considering different possible purposes for her study. Identify the design she is considering in the following scenario: Upon further reflection, Debbie realizes that most of the research on college adjustment to date has involved students from urban settings, so she is unsure whether existing instruments are appropriate for her rural participants. She decides to first qualitatively generate a theory that describes rural students’ adjustment, and then she can quantitatively test that theory. What design is she thinking of using? [Q2 Model Response] This scenario would best fit the use of an exploratory sequential design. This design starts with the collection and analysis of qualitative data, which is appropriate for generating a theory based on participants’ views and experiences. Once Debbie has the theory in hand, she can develop or select an appropriate instrument and then test this theory for a large number of rural students using quantitative approaches. [Q3] Consider the following: Debbie is interested in adjustment for college freshman students coming from small rural high schools to a large university. She plans to use mixed methods research and is considering different possible purposes for her study. Identify the design she is considering in the following scenario: As another option, Debbie decides that she would like to focus on getting an in-depth understanding of rural students’ adjustment during the first 2 weeks of college. She decides to gather quantitative scores on various variables hypothesized to be related to adjustment and, at the same time, to ask students to tell her a story about their adjustment during the first 2 weeks. What design is she thinking of using? [Q3 Model Response] This scenario would best fit the use of a convergent design, also known as the triangulation design. This design involves the collection of both quantitative and qualitative data during the same phase of the research study in order to develop a more complete understanding. The quantitative results and the qualitative narrative stories will each provide complementary information about the college adjustment process.

Chapter 14 Test Items SELF-ASSESSMENT ITEMS 1) A mixed methods research design can be used when the goal is to do which of the following? a) To establish that a dependent variable was clearly affected only by the intervention b) To understand both the outcome and the explanation for the outcome c) To investigate current and past status of a unit of study d) To examine what and how much of something exists 2) Which of the following is a disadvantage of a mixed methods design? a) The design incorporates the strengths of each methodological approach. b) The results from one phase are often not dramatically different from another phase so the different methods may unnecessarily duplicate each other. c) The design typically involves extensive data collection that requires more resources than might be required by a study using quantitative or qualitative methods only. d) The results are often unnecessarily complex. 3) Two large school districts are compared using state-level achievement data. The information is then investigated further with qualitative research methods. This is an example of which type of mixed methods design? a) Explanatory sequential b) Exploratory sequential c) Triangulation d) Integrative 4) Which mixed methods design collects quantitative and qualitative data concurrently to validate and expand on quantitative findings using qualitative methods? a) Explanatory sequential b) Independent c) Convergent d) Exploratory sequential 5) A researcher surveyed a large sample of college students (N = 347) to examine how students’ sense of belonging was related to achievement. Following the survey, a subsample of the students was interviewed to understand their perspectives on how 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

their sense of belonging related to their achievement. This is an example of which type of mixed methods design? a) Concrete sequential b) Exploratory sequential c) Explanatory sequential d) Relational sequential 6) A researcher collects data using observations of students, interviews with teachers, surveys with administrators, and focus groups with parents. This is an example of which type of mixed methods design? a) Explanatory sequential b) Exploratory sequential c) Triangulation or nested d) Comprehensive sequential 7) What is used in mixed methods research design to assist the reader in identifying the type of design used? a) Tables of data b) A notation system c) Excerpts d) A logic model 8) A study was conducted where the researcher used stratified random sampling to survey third-grade math teachers and at the same time interviewed a different group of third-grade math teachers. This design is an example of which of the following? a) Explanatory sequential b) Exploratory sequential c) Convergent d) Phase concurrent 9) Which mixed methods design generates qualitative information that is followed by a quantitative investigation? a) Exploratory sequential b) Explanatory sequential c) Triangulation d) Multi-trait combined 10) When should mixed methods designs be used? 18 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

a) Participant behavior can be affected by participation in the research study. b) The results of quantitative data collection and analysis do not adequately explain the findings. c) In order to answer the research question, data are needed in which participants share their unique perspectives. d) There is a lack of information about how the study was conducted and how interpretations were made.

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) A mixed methods research design is used when the goal is to do which of the following? a) To examine current and past status of a unit of study b) To clarify results from a quantitative data collection and analysis in order to explain the results c) To synthesize data inductively to generate generalizations, models, or frameworks d) To determine what and how much of something exists 2) Which of the following is a disadvantage specific to a mixed methods design? a) The processing of the amount of data that is collected can be tedious. b) The formats used for reporting results of each type are very different. c) The approaches compensate for the limitations of each method. d) The current disregard for the design among educational researchers makes results less respected. 3) Which type of mixed methods design involves using different methods to gather information from individuals or groups at different levels within a system? a) Explanatory sequential b) Exploratory sequential c) Nested d) Multi-trait combined 4) A researcher interviewed college students to understand their motivations and choices of service involvement. The researcher then developed an instrument for measuring learning outcomes related to themes identified in the interviews. This is an example of which type of mixed methods design? a) Explanatory sequential b) Triangulation or nested c) Exploratory sequential d) Convergent 5) Which type of mixed methods design would collect both Likert-type survey data and in-depth interview data concurrently to examine attitudes toward taking an introduction to educational research course? a) Convergent 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Comprehensive c) Concrete concurrent d) Priority weighted 6) Which type of mixed methods design would use qualitative methods to generate information to be used in conducting a quantitative phase of the analysis? a) Exploratory sequential b) Triangulation c) Nested d) Explanatory sequential 7) A survey was given to teachers to determine their attitudes toward the implementation of a new policy. The results from the survey were analyzed and used to select negative, extreme-case participants for follow-up interviews to understand the reasons for resistance to the policy. This is an example of which type of mixed methods design? a) Triangulation b) Explanatory sequential c) Nested design d) Exploratory sequential 8) When should mixed methods designs be used? a) Participants did not respond to all questions on a survey. b) Additional data are needed to help interpret findings. c) Data are to be collected from large samples. d) Observation data will be collected over many days, weeks, or months. 9) What is the purpose of a notation system in mixed methods research design? a) It assists the reader in identifying the type of design employed. b) It helps the researcher organize the data during the distinct phases of the study. c) It provides a system for monitoring the progression of the study. d) It allows the researcher to demonstrate level of expertise in a method. 10) Which of the following is true about mixed methods sampling? a) Participants are always selected for both the qualitative phase and the quantitative phase using purposeful sampling. b) Snowball sampling is rarely used in a mixed methods research design because it produces a sample that is too similar to provide useful information. 21 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

c) Many mixed methods studies collect data from the same pool of participants but samples can also be completely independent. d) The sampling from the quantitative phase is rarely connected with the sampling from the qualitative phase. Short-Answer Questions 1) What is one factor that determines how much weighting or priority is given to each phase in a mixed methods research design? 2) How does a mixed methods researcher establish the feasibility of conducting a mixed methods study? Essay Question Describe in detail at least three advantages and three disadvantages of using a mixed methods design.

Chapter 14 Test Answer Key ANSWER KEY: Chapter 14 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. b 2. c 3. a 4. c 5. c 6. c 7. b 8. c 9. a 10. b

INSTRUCTOR’S EXAM ITEMS 1. b 2. b 3. c 4. c 5. a 6. a 7. b 8. b 9. a 10. c

Chapter 15 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 15: Action Research ................................................................................. 1 Learning Outcome Quizzes .................................................................. 1 Application Exercises ......................................................................... 13 Test Items ........................................................................................... 24 Test Answer Key ................................................................................ 30

Chapter 15: Action Research Chapter 15 Learning Outcome Quizzes Learning Outcome 15.1: Compare the characteristics, strengths, and limitations of action research to traditional types of empirical research. [Q1] Which of the following best describes how action research in education is different than other types of educational research? 1. Action research is conducted by educators with a goal to solve a particular problem. [correct] 2. Action research is always applied and involves collaboration with community members. 3. Action research is always conducted by teachers in classrooms. 4. Action research is never generalizable beyond the population in which it’s conducted. [Feedback for Correct Answer 1] This is the basic definition of action research. It may be conducted by teachers or other educators or professionals, and although it is not designed to generalize to other populations, it often leads to findings that may be relevant to other classroom and school settings. [Feedback for Answer Choice 2] Action research can be conducted alone and without collaboration from community members. [Feedback for Answer Choice 3] Action research is often conducted by teachers in the classroom, but this is not a universal truth. [Feedback for Answer Choice 4] Many types of educational research are not generalizable, so while this is generally true, this is not unique to action research. [Q2] Which of the following situations best illustrates action research?

1. Teachers attending a conference to learn new instructional strategies 2. Principals instigating change in school policies based on national trends 3. Teachers collecting data in classrooms and implementing strategies [correct] 4. Principals comparing student performance in classes using different textbooks [Feedback for Answer Choice 1] This would simply be an instance of professional learning. [Feedback for Answer Choice 2] This is the responsibility of a principal, but if they are not actively collecting their own data to influence these changes, this is not action research. [Feedback for Correct Answer 3] The purpose of action research is to provide school practitioners with a method for solving everyday problems in schools so that they can improve both student learning and teacher effectiveness. [Feedback for Answer Choice 4] This would simply be a principal collecting data to inform their own practice. There is not a research question, and the principal is not closely connected to the data, as the teachers are the ones teaching from the various texts. [Q3] Compared to other research methods, of the following, which is generally true of action research? 1. Action research takes longer to conduct than qualitative research studies. 2. Action research is smaller in scale than experimental studies. 3. Action research is concerned with teachers generalizing their findings. 4. Action research is conducted by teachers. [correct] [Feedback for Answer Choice 1] While there are no absolutes in regard to which methods requires the most time, many forms of qualitative research are quite substantial with respect to time required. [Feedback for Answer Choice 2] While an action research study may be smaller in scale than an experimental study, this is not universally true. 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] Action researchers are interested in the impact of their research on the setting in which they are immersed, typically their classrooms. They are not focused on generalizability. [Feedback for Correct Answer 4] Usually the teachers are referred to as teacher–researchers. Examples of other practitioners that use action research might include counselors and clinicians. [Q4] The main goal of action research is to give teachers a tool for which of the following? 1. Solving everyday problems in schools [correct] 2. Measuring student outcome data 3. Creating highly generalizable research 4. Supporting professors in the research process [Feedback for Correct Answer 1] The goal of action research is for teachers to be able to analyze and solve everyday problems in schools and classrooms. [Feedback for Answer Choice 2] Action research is not typically focused on outcome data, although it could be. Still, other methods may be better suited for exploring outcome data, such as standardized testing results. [Feedback for Answer Choice 3] Action research is specific to the setting in which it is conducted and isn’t highly generalizable. [Feedback for Answer Choice 4] Part of the beauty of action research is that it is a method that teachers can follow without needing to consult other professionals—unless they choose to, of course. Learning Outcome 15.2: Be able to identify and construct appropriate action research questions. [Q1] Of the following, which is likely the best research question for an action research study?

1. Are low-SES students in our school likely to have high dropout rates? 2. How can we support low-SES students in our classrooms to help keep them in school? [correct] 3. Are children of low SES struggling on our grade-three standardized test scores? 4. How are national policies affecting the likelihood of high school completion for low-S ES children? [Feedback for Answer Choice 1] This question does not require an action research approach. It can be answered with quantitative methods. [Feedback for Correct Answer 2] This question shows a concern for solving a problem that is relevant to educators— dropout rates. Support is something the schools have within their control, and it is likely something that everyone feels passionate about. [Feedback for Answer Choice 3] This question can be easily addressed using quantitative methods. [Feedback for Answer Choice 4] This question is a larger scale study than an action research project. [Q2] Action researchers are advised to think about the problem they want to address using the ORID method, which involves four types of questions. Which of the following is an example of an interpretive question? 1. When bullying occurs, what behaviors do we see from the bully and from the victim? 2. What is the primary reason that bullying is occurring in this classroom? [correct] 3. What are my own experiences with bullying that may be affecting my response to it in the classroom? 4. What actions are appropriate to take when bullying occurs? [Feedback for Answer Choice 1] This question is only observational. It does not address the root of the problem or possible causes. [Feedback for Correct Answer 2] 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Action researchers ask interpretive questions to analyze the problem, with the goal of determining possible causes. [Feedback for Answer Choice 3] This question is personal in nature, but does not attempt to analyze a larger problem. [Feedback for Answer Choice 4] This is an important question, but not one that seeks to find the root cause of the problem at hand. [Q3] Action researchers are advised to think about the problem they want to address using the ORID method, which involves four types of questions. Which of the following is an example of a reflective question? 1. What is the primary reason that bullying is occurring in this classroom? 2. What are my own experiences with bullying that may be affecting my response to it in the classroom? [correct] 3. When bullying occurs, what behaviors do we see from the bully and from the victim? 4. What actions are appropriate to take when bullying occurs? [Feedback for Answer Choice 1] This question does not involve personal reflection from the researcher. [Feedback for Correct Answer 2] Action researchers ask reflective questions to be honest about the roots of their interest in the topic and to help control any biases they bring to the research process. [Feedback for Answer Choice 3] This question does not involve personal reflection from the researcher. [Feedback for Answer Choice 4] This question does not involve personal reflection from the researcher. [Q4] After selecting a general topic, action researchers are advised to engage in fact finding, which can include a literature review and collection of preliminary data. The action researcher can best use these preliminary data for which of the following?

1. Refining the question and planning the first action [correct] 2. Comparing to posttest data, after the first action 3. Planning the quantitative phase of the study 4. Developing a plan for data analysis [Feedback for Correct Answer 1] This phase of data collection is exploratory and generally open ended, with a goal to help refine the problem and plan for an appropriate action to address it. [Feedback for Answer Choice 2] The researcher would not be ready to compare data at this stage. [Feedback for Answer Choice 3] Perhaps the research questions won’t need quantitative methods. This would not be an appropriate use of these data. [Feedback for Answer Choice 4] This would be too early in the process to determine data analytic processes. Learning Outcome 15.3: Be able to design action research. [Q1] When an action researcher uses an experimental design, his or her primary goal is to do which of the following? 1. Select a large enough sample of students to participate in the intervention condition so that the results will be generalizable to the rest of the class 2. Follow the scientific method closely so that contextual variables are tightly controlled and strong conclusions can be drawn 3. Ensure the validity and reliability of the instruments used to measure the outcomes 4. Introduce an action that may solve the problem and measure whether the intervention had the intended effect [correct] [Feedback for Answer Choice 1] Just because an action research method has an experimental approach doesn’t mean the results are generalizable. [Feedback for Answer Choice 2]

The action researcher in both experimental and nonexperimental designs follows the scientific method. [Feedback for Answer Choice 3] Valid and reliable measures should be the only measures used in any design. [Feedback for Correct Answer 4] Experimental designs are generally selected when an action researcher wants to isolate the effect of a specific classroom intervention. This design is the best to use when any researcher wants to look at cause-and-effect relationships. [Q2] Action researchers often use nonexperimental designs, especially case studies, because of which of the following? 1. They are qualitative in nature but also are controlled, because the “case” is a single classroom. 2. They are exploratory and leave room for new ideas and unexpected findings to emerge. [correct] 3. They are the most trustworthy and authentic studies of educational settings. 4. They are the most ethical method for studying a vulnerable population, such as schoolchildren. [Feedback for Answer Choice 1] Control is not usually a term used in action research that includes case study. [Feedback for Correct Answer 2] One of the hallmarks of a case study is to collect data about the topic from multiple sources and use triangulation to enhance the depth and scope of the findings. [Feedback for Answer Choice 3] A good case study can be a high-quality study, but so are many other types of action research projects. [Feedback for Answer Choice 4] Case studies can be conducted to high ethical standards, but so can many other forms of action research. [Q3] 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Mr. Harris is preparing to conduct an action research study in his third-grade classroom. He is interested in how his class rules are impacting the social and emotional well-being of his students. He is debating whether or not to include the whole class, or just a few students. Which of the following categories best describe the focus of his deliberations? 1. Reporting 2. Data analysis 3. Data collection 4. Sampling [correct] [Feedback for Answer Choice 1] Mr. Harris is a long way from the reporting stage, which would be the last stage in his study. [Feedback for Answer Choice 2] Mr. Harris does not yet have data to analyze. [Feedback for Answer Choice 3] Soon Mr. Harris will be ready to collect data, but he has not made the necessary decisions to engage in data collection just yet. [Feedback for Correct Answer 4] Mr. Harris is engaged in deliberations around the concept of sampling, or in other word, deciding who will be in his study. [Q4] Mrs. Hale has conducted an action research study which included the collection of quantitative data. She is currently working to produce descriptive statistics such as the mean, median, and range of her data. Which of the following terms best describes the process with which Mrs. Hale is engaging? 1. Data collection 2. Data analysis [correct] 3. Sampling 4. Reporting [Feedback for Answer Choice 1] Mrs. Hale has already collected her data if she is able to compute descriptive statistics. 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Correct Answer 2] This is a description of Mrs. Hale’s data analysis phase. She has collected data and is now seeking to make sense of her findings. [Feedback for Answer Choice 3] If Mrs. Hale already has data, she has surely selected a sample. [Feedback for Answer Choice 4] Mrs. Hale is not quite ready to report if she is just calculating her descriptive statistics. Learning Outcome 15.4: Know and apply the criteria for evaluating the validity of action research. [Q1] He-Ping conducted an action research project in his classroom and was pleased to discover that not only was the problem solved, but the students in the class all felt empowered as part of the solution. They told him the project helped to change their thoughts about their experiences at school and how they can help bring about other needed changes. He-Ping was glad to know that his study was high in which of the following? 1. Catalytic validity [correct] 2. Dialogic validity 3. Process validity 4. Outcome validity [Feedback for Correct Answer 1] When a study has high catalytic validity, participants and researchers are inspired to change not only their practices, but also their ways of thinking about the realities of their school setting. He-Ping’s study also had high outcome validity (the problem was solved) and probably had high democratic validity (the perspectives of the stakeholders—the students—were addressed). [Feedback for Answer Choice 2] Dialogic validity refers to the extent to which the research has inspired conversation and learning among peer practitioners. [Feedback for Answer Choice 3]

Similar to the idea of internal validity for quantitative studies, process validity is concerned with the way in which the action research study was conducted. [Feedback for Answer Choice 4] Because action research is an approach used to resolve particular problems of practice, outcome validity is an assessment of the successful resolution of the particular problem. An action research study has outcome validity when the problem is solved. Nonetheless, there is an answer that better matches the description. [Q2] Some ethical concerns that are applicable to all research are especially critical for action researchers, because of the very nature of action research. For example, action researchers should always do which of the following? 1. Make sure that the sample size is big enough to produce meaningful effects. 2. Make sure that there is no deviation from the initial, IRB-approved plan of action for the study. 3. Make sure that, because of the close relationship between research and participants, data collection does not become coercive. [correct] 4. Make sure that every individual at the research site participates in the study so that no one feels excluded. [Feedback for Answer Choice 1] Large sample sizes are rare in action research and not necessary for most action research questions. [Feedback for Answer Choice 2] It is possible that IRB applications will require amendments as necessary changes arise. [Feedback for Correct Answer 3] Educators, for all that they may not recognize it, are in positions of tremendous power and must ensure that their students are not treated unethically during action research projects. [Feedback for Answer Choice 4] Participants always have the right to decline to participate at any point in a research study. [Q3] 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Which of the following types of validity is concerned with representation of the stakeholders in either the process of conducting the action research or as data sources? 1. Outcome validity 2. Democratic validity [correct] 3. Process validity 4. Dialogic validity [Feedback for Answer Choice 1] Because action research is an approach used to resolve particular problems of practice, outcome validity is an assessment of the successful resolution of the particular problem. An action research study has outcome validity when the problem is solved. [Feedback for Correct Answer 2] Democratic validity is concerned with representation of the stakeholders in either the process of conducting the action research or as data sources. [Feedback for Answer Choice 3] Similar to the idea of internal validity for quantitative studies, process validity is concerned with the way in which the action research study was conducted. [Feedback for Answer Choice 4] Traditional research is often assessed through a process of peer review. With action research, dialogic validity refers to the extent to which the research has inspired conversation and learning among peer practitioners. [Q4] Which of the following forms of validity is most similar to the concept of internal validity in quantitative research? 1. Process validity [correct] 2. Democratic validity 3. Dialogic validity 4. Outcome validity [Feedback for Correct Answer 1] Similar to the idea of internal validity for quantitative studies, process validity is concerned with the way in which the action research study was conducted. An action 11 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

research study that pays attention to the idea of process validity demonstrates an attention to how the cycles of action research are deliberately and thoughtfully planned in ways that are effective for answering the question guiding the study, as well as ensuring the ongoing learning of individuals within the system. [Feedback for Answer Choice 2] Democratic validity is concerned with representation of the stakeholders in either the process of conducting the action research or as data sources. [Feedback for Answer Choice 3] Traditional research is often assessed through a process of peer review. With action research, dialogic validity refers to the extent to which the research has inspired conversation and learning among peer practitioners [Feedback for Answer Choice 4] Because action research is an approach used to resolve particular problems of practice, outcome validity is an assessment of the successful resolution of the particular problem. An action research study has outcome validity when the problem is solved.

Chapter 15 Application Exercises Chapter 15: Action Research Application Exercise 15.1: Action Research Questions Learning Outcome 15.2: Be able to identify and construct appropriate action research questions. [Q1] Consider the following: Imagine your role as a first-year teacher in a high school setting. As a first-year teacher, you are trying to establish a Socratic seminar method where students read an assigned text, develop discussion questions, and take turns leading weekly discussions. However, you notice that students are struggling with leadership skills. Refer to the section entitled Individual Reflection on Your Topic and specifically the ORI D process. Complete the “O” part of this process by reflecting on objective questions you may ask to help shape an action research study focused on this topic. [Q1 Model Response] There are many different ways to create objective questions related to this topic. Several possible questions include the following: •

What specific skills do students seem to be lacking?

•

When the Socratic seminar begins, how are students speaking to one another?

•

When the problem occurs, what actions were just implemented by the teacher?

•

Are the instructions clear?

[Q2] Consider the following: Imagine your role as a first-year teacher in a high school setting. As a first-year teacher, you are trying to establish a Socratic seminar method where students read an assigned text, develop discussion questions, and take turns leading weekly discussions. However, you notice that students are struggling with leadership skills. Refer to the section entitled Individual Reflection on Your Topic and specifically the ORI D process. Complete the “R” part of this process by focusing on reflective questions you may ask to help shape an action research study focused on this topic.

[Q2 Model Response] There are many different ways to create reflective questions related to this topic. Several possible questions include the following: •

How do you feel when the Socratic seminar begins?

•

How do you feel when the Socratic seminar ends?

•

What experiences do you associate with this problem?

•

Have you experienced this problem with other assignments?

[Q3] Consider the following: Imagine your role as a first-year teacher in a high school setting. As a first-year teacher, you are trying to establish a Socratic seminar method where students read an assigned text, develop discussion questions, and take turns leading weekly discussions. However, you notice that students are struggling with leadership skills. Refer to the section entitled Individual Reflection on Your Topic and specifically the ORI D process. Complete the “I” part of this process by focusing on interpretive questions you may ask to help shape an action research study focused on this topic. [Q3 Model Response] There are many different ways to create reflective questions related to this topic. Several possible questions include the following: •

Why is this problem happening?

•

Are there other explanations for the problem I’m not considering?

•

What are the root causes of this problem?

[Q4] Consider the following: Imagine your role as a first-year teacher in a high school setting. As a first-year teacher, you are trying to establish a Socratic seminar method where students read an assigned text, develop discussion questions, and take turns leading weekly discussions. However, you notice that students are struggling with leadership skills. Refer to the section entitled Individual Reflection on Your Topic and specifically the ORI D process. Complete the “D” part of this process by focusing on decisional questions you may ask to help shape an action research study focused on this topic.

[Q4 Model Response] There are many different ways to create reflective questions related to this topic. Several possible questions include the following: •

What can I do differently?

•

How can I better prepare my students for this task?

•

What actions are appropriate?

•

Whom else could help me solve this problem?

Application Exercise 15.2: Designing Action Research Learning Outcome 15.3: Be able to design action research. [Q1] Consider the following: Greg is the principal at a 400-student elementary school. He is approached by a group of parents who want him to consider implementing a full-day kindergarten program. Of course, there are a variety of factors to consider—staffing, population growth, demand for the program, equity between full-day and half-day programs, classroom space, money, and busing, among others. With an area of focus in mind, Greg’s first step is reflection and planning. Explain what this process will involve, focusing on three ways he can gain insight into the problem. [Q1 Model Response] Reflection involves gaining insight into the area of focus through self-reflection, descriptive activities, and explanatory activities. Self-reflection, for Greg, may involve consideration of why the school does not currently have full-day kindergarten and whether he believes it is a good educational program. He may reflect on reading he has done on the topic. Descriptive activities may involve a review of the current kindergarten program at his school. Greg may ask himself questions about whether students are lacking certain skills that could be taught during a longer school day or whether teachers feel pressed for time to cover relevant material or to allow more playtime, about whether parents are often late to pick up children because they are at work, and so on. Explanatory activities will focus on why the patterns he sees in the descriptive data may occur. For example, if he finds that teachers are pressed for time, he may consider whether the issue is with the curriculum (too broad) or with the length of the school day

(too short). At this time, he should develop a hypothesis or question to guide the rest of his research. [Q2] Consider the following: Greg is the principal at a 400-student elementary school. He is approached by a group of parents who want him to consider implementing a full-day kindergarten program. Of course, there are a variety of factors to consider—staffing, population growth, demand for the program, equity between full-day and half-day programs, classroom space, money, and busing, among others. Identify one quantitative data collection procedure that Greg could choose for his study. How will these data inform the action Greg takes? [Q2 Model Response] Many choices are possible, but Greg might conduct a brief survey. For example, a survey of teachers will help him identify staffing needs and interests (e.g., do current teachers want more classroom time, or would he need to hire additional teachers). A survey of parents will help him identify the need in the community. These surveys will provide information about a large number of teachers and parents. He might also examine existing data about enrollment growth and class sizes. [Q3] Consider the following: Greg is the principal at a 400-student elementary school. He is approached by a group of parents who want him to consider implementing a full-day kindergarten program. Of course, there are a variety of factors to consider—staffing, population growth, demand for the program, equity between full-day and half-day programs, classroom space, money, and busing, among others. Identify one qualitative data collection procedure that Greg could choose for his study. How will these data inform the action Greg takes? [Q3 Model Response] Many choices are possible, but Greg will most likely conduct interviews. For example, interviews with teachers will help him identify staffing needs and interests (e.g., do current teachers want more classroom time, or would he need to hire additional teachers). Interviews with parents will help him identify the need in the community. These interviews will provide more in-depth ideas than can be gathered only with a survey.

Application Exercise 15.3: An Action Research Project Learning Outcome 15.4: Know and apply the criteria for evaluating the quality of mixed methods research. [Q1] This activity is based on an article by DuBois, Erickson, and Jacobs (2007), which is available at no cost through the Education Resources Information Center. The resulting record will contain a direct link to the article, which you should read carefully. The full reference for the article is below. DuBois, K., Erickson, K., & Jacobs, M. (2007). Improving spelling of high frequency words for transfer in written work (Masters’ thesis, Saint Xavier University & Pearson Achievement Solutions). This activity provides an opportunity for you to think critically about how teacher– researchers plan, conduct, and present the findings of an action research project. As you read through the article and answer the question, keep in mind the following ideas: -

Effective action research projects are designed to address a problem identified by a specific teacher–researcher. They should have local relevance—the problem should be one that the teacher–researcher faces in their classroom or school. It may or may not have more global relevance, depending on the extent to which the problem and the results are applicable to other institutions.

The method to study the problem should be valid. A method is valid if it measures the problem or concept it’s intended to measure. For example, scores on an IQ test may be a valid measure of intelligence (based on some definitions of intelligence) but would not be a valid measure of school achievement.

Action research requires action. The researchers should identify a problem and take some action to address it. The action is typically, but need not be, some type of classroom intervention. Additionally, action research is cyclical—the outcome of one action almost invariably leads to future action. Strong action research reports end with new recommendations for future action.

Action research requires research, or in other words evaluation of the action. The researchers should measure the outcome, summarize and analyze the data appropriately, and make some inferences about the effectiveness of the action. The inferences should be clearly tied to the problem to be addressed and should be supported by the data collected in the study.

State the problem addressed in this research project. Is it one that needed to be addressed via systematic research? Consider the question from a local/personal perspective as well as from a more general (i.e., national) perspective, and explain your reasoning. 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q1 Model Response] Review the sections of the paper entitled Immediate Problem Context and National Context of the Problem, along with the section entitled Problem Evidence. The problem is that students show poor spelling on written work. The authors state that they have observed this problem (personal perspective; p. 1, p. 11), and thus it’s one that should be addressed. Their literature review beginning on p. 12 provides evidence that the problem is one of national concern, although action research need not be focused on a problem with broad appeal. [Q2] This activity is based on an article by DuBois, Erickson, and Jacobs (2007), which is available at no cost through the Education Resources Information Center. The resulting record will contain a direct link to the article, which you should read carefully. The full reference for the article is below. DuBois, K., Erickson, K., & Jacobs, M. (2007). Improving spelling of high frequency words for transfer in written work (Masters’ thesis, Saint Xavier University & Pearson Achievement Solutions). This activity provides an opportunity for you to think critically about how teacher– researchers plan, conduct, and present the findings of an action research project. As you read through the article and answer the question, keep in mind the following ideas: -

What methods did the researchers use to document the problem in their schools? Did they use valid measures? Explain your reasoning. [Q2 Model Response] The researchers used three measures: a test of spelling for students in K, grade 1, and grade 3; a survey given to teachers in the two target schools; and a student selfassessment checklist (for the two older grades). The authors provide no information on validity; you must make this judgment yourself by analyzing the measures. The spelling test is a valid measure of whether students can spell select high-frequency words but is not a valid measure of the problem raised by the authors—students’ ability to transfer spelling skills to written work. However, the researchers probably enhanced validity somewhat by focusing on high-frequency words, which are most likely to appear in student written work. It is hard to assess this further without seeing the lists of words. The self-assessment checklist (Appendix G) does not measure students’ spelling skills at all and thus is not a valid measure of the research question, but it is probably a valid measure of students’ beliefs about their own spelling skills. The authors realized it is not a valid measure for the youngest group, who lack the metacognitive skill needed to reflect on their strategies and beliefs and who may not yet know much about punctuation, dictionaries, and capital letters. It is probably not valid for the first-graders either for the same reasons. The teacher survey helps to identify whether other teachers perceive the problem, and it appears to be a valid measure for that. Questions 8–12 in particular focus on the question of transfer of skill to written work. It appears to be a reliable measure with unbiased questions; if a teacher filled it out twice, they would likely give the same answer on each question. [Q3] This activity is based on an article by DuBois, Erickson, and Jacobs (2007), which is available at no cost through the Education Resources Information Center. The resulting record will contain a direct link to the article, which you should read carefully. The full reference for the article is below. DuBois, K., Erickson, K., & Jacobs, M. (2007). Improving spelling of high frequency words for transfer in written work (Masters’ thesis, Saint Xavier University & Pearson Achievement Solutions). This activity provides an opportunity for you to think critically about how teacher– researchers plan, conduct, and present the findings of an action research project. As you read through the article and answer the question, keep in mind the following ideas:

Did the researchers plan appropriate action to address the problem? Consider the method of intervention they used as well as the time frame of the project, and provide the reasoning behind your response. [Q3 Model Response] The action seems appropriate to encourage students to learn spelling words so well that they will transfer to written work. The 16-week spelling curriculum developed by the researchers introduced several new words each week, taught using various methods that allowed students to practice using different modalities (e.g., song, shaving cream) and activities (e.g., sorting, quizzes, self-editing). The project lasted one full semester, which is a good length—the students don’t need to adjust to new methods during the term, and the researchers have a lot of time to assess the success of the project and make adjustments as necessary. With so many different kinds of spelling tasks, the researchers won’t be able to identify whether any one is more or less effective than the others, but they do not seem interested in that question. [Q4] This activity is based on an article by DuBois, Erickson, and Jacobs (2007), which is available at no cost through the Education Resources Information Center. The resulting record will contain a direct link to the article, which you should read carefully. The full reference for the article is below. 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

DuBois, K., Erickson, K., & Jacobs, M. (2007). Improving spelling of high frequency words for transfer in written work (Masters’ thesis, Saint Xavier University & Pearson Achievement Solutions). This activity provides an opportunity for you to think critically about how teacher– researchers plan, conduct, and present the findings of an action research project. As you read through the article and answer the question, keep in mind the following ideas: -

The researchers conclude that their intervention was successful. What evidence do they present in support of this conclusion? Is it enough evidence to persuade you that they’ve solved their problem? [Q4 Model Response] The authors show that, over the course of the term, the students learned to spell many of the words that they couldn’t spell on the pretest (see “Table” 3—note this is actually a figure) and had moved to a higher developmental level on the Gentry test (see “Table” 4). Recall, however, that their main goal was to enhance spelling in written work. The researchers note that they “noticed that all of the students made improvements” (p. 45), but they do not present any evidence to support this claim. Summaries of the percentages of misspelled words in written work, perhaps accompanied by samples, would provide better evidence to persuade a reader that the conclusion is justified.

[Q5] This activity is based on an article by DuBois, Erickson, and Jacobs (2007), which is available at no cost through the Education Resources Information Center. The resulting record will contain a direct link to the article, which you should read carefully. The full reference for the article is below. DuBois, K., Erickson, K., & Jacobs, M. (2007). Improving spelling of high frequency words for transfer in written work (Masters’ thesis, Saint Xavier University & Pearson Achievement Solutions). This activity provides an opportunity for you to think critically about how teacher– researchers plan, conduct, and present the findings of an action research project. As you read through the article and answer the question, keep in mind the following ideas: -

What new actions might these researchers take in the future, as a result of their findings from this project? Review their reflections, but also think critically about what other interventions and/or research topics arise from these findings, both in the area of spelling and in other content areas.

[Q5 Model Response] The researchers plan to take action based on three main lessons they learned: make activities fun, encourage self-correction and ownership, and stress spelling highfrequency words. In addition to implementing this curriculum again in their own classes, they plan to encourage their colleagues to implement spelling lessons based on these strategies. They also note other changes in their attitudes and practices, such as to be more flexible, use more cooperative learning groups and authentic assessment, and hold students more accountable for their learning. Your recommendations for actions will vary, depending on your own interests. Some examples: The researchers could further study whether one or more of their instructional strategies works best, or they could study whether similar multisensory approaches work well in other content areas (e.g., math). They could look at whether the strategies work better for older children than for younger children. They could directly measure enjoyment, or amount of time teachers spend preparing for the lessons, or any number of other topics.

Chapter 15 Test Items SELF-ASSESSMENT ITEMS 1) A study used a standardized achievement test to identify fourth-grade students across the state of Virginia at risk for experiencing difficulty mastering fifth-grade math concepts. After selecting the students, the researchers conducted interviews with parents, teachers, and students to examine, in depth, the difficulties these students faced. This is an example of which type of research design? a) Action research b) Narrative research c) Mixed methods research d) Ethnographic research 2) Which of the following is NOT one of the purposes of action research? a) It is intended to be generalizable to a larger sample. b) It provides teacher researchers with a method for solving everyday problems in their own school. c) It develops the professional dispositions of teachers. d) It develops the reflective teaching qualities in teachers. 3) Which of the following is NOT true about action research design? a) It is typically a brief, static bounded study that is designed to solve an immediate and pressing problem. b) The process includes a step in which action is taken, such as a teacher changing a method of instruction to improve student learning. c) It aspires to local relevance, not generalizability. d) It can be illustrated with a spiral where problem-posing leads to knowledge development. 4) Which of the following is NOT true about data collection in an action research study? a) The researcher typically collects data from multiple sources. b) The data collection methods can be both quantitative and qualitative in nature. c) Instruments used can include tests, quizzes, papers, projects, and questionnaires. d) The data should not be cleaned to remove any outliers or obviously inaccurate numbers to prevent researcher bias.

5) One type of validity an action researcher attempts to achieve is concerned with the way in which the action research study was conducted. What is this type of validity called? a) Democratic validity b) Process validity c) Method validity d) Development validity 6) After a research topic has been identified, the researcher might reflect using the ORI D method of questioning. What does the letter O stand for in this acronym? a) Observational questions b) Objective questions c) Outcomes-based questions d) Organizational questions 7) In school-based action research, who might determine the research question and carry out the study? a) The principal of the school b) A graduate student familiar with the school c) A trained researcher who has worked in the school before d) All of the above 8) Which of the following is typical of an action research literature review? a) It is similar in length and style to a literature review used in traditional research. b) It is typically shorter in length than a literature review in traditional research. c) It is unnecessary in published reports of action research. d) It is different in style and substance from a literature review in a traditional research study. 9) Which of the following is NOT a criterion for evaluating action research? a) Multiple data collection methods were used. b) The researcher stated several directional research hypotheses. c) There is evidence of external monitoring or objective feedback. d) The motivation of the researcher is clear. 10) Which of the following is a potential benefit of conducting an action research study?

a) Many schools use action research as an enhanced model of professional development. b) Change-oriented policies that derive from action research are often perceived by practitioners as separate from their experiences. c) It helps to widen the gap between educational research and teaching practice. d) Teachers act as subjects or participants rather than their usual role as agents of change.

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Principal McIntyre was aware that teachers in her school were concerned about lack of discipline and spoke to several members of the faculty to get their ideas for a solution to the problem. She took one of the teacher’s suggestions and devised a strategy to reduce the problem. Which of the following guidelines of action research did Principal McIntyre fail to follow when dealing with this problem? a) Identify an area of focus b) Collect data c) Analyze and interpret the data d) Develop an action plan 2) Which of the following is NOT a criterion used to judge selection of a research topic or problem in an action research study? a) The focus is something the teacher would like to change or improve. b) The topic should be selected because there is a real need for change. c) The focus is something within the teacher’s locus of control, such as teaching or learning. d) The action researcher is dispassionate about the topic in order to remain objective. 3) Which of the following is NOT true about action research design? a) It involves a cyclical process of inquiry by practitioners that includes action and evaluation. b) Principals, counselors, school nurses, and security staff can engage in the action research process. c) It aspires to local relevance and generalizability. d) It involves posing questions, using theories, collecting and analyzing data, and coming to conclusions. 4) In school-based action research, which of the following people would NOT determine an action research question and carry out the study? a) A kindergarten teacher b) A graduate student familiar with the school c) A school counselor d) The art teacher

5) Which of the following is true about the collection of data in an action research study? a) The researcher typically only collects information from one source. b) The data collection methods are more similar to those used in qualitative research rather than quantitative research. c) It’s important to collect lots of data in order for the results to be consistent and generalizable. d) The researcher typically uses locally developed instruments rather than off-theshelf measures. 6) One type of validity an action researcher attempts to achieve is an assessment of the successful resolution of the particular problem addressed by the study. This type of validity is referred to as which of the following? a) Effect validity b) Democratic validity c) Catalytic validity d) Outcome validity 7) After a research topic has been identified, it is helpful for the researcher to reflect using the ORID method of questioning. What does the letter D stand for in this acronym? a) Directional questions b) Decisional questions c) Divisional questions d) Difficult questions 8) Which of the following is true about a literature review in published reports of an action research study? a) There is no difference between a literature review used in traditional research and one used in action research. b) They are usually shorter in length than the typical literature review in traditional research. c) Most published reports of action research do not include a literature review. d) They tend to use only primary sources as references. 9) Which of the following is NOT a potential benefit of conducting an action research study? a) It can be used as an enhanced model of professional development. 28 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) It can contribute to the development of knowledge in the educational research community. c) It helps practitioners increase their level of engagement with traditional literature. d) Teachers become actionable participants rather than agents of change. 10) Which of the following is a criterion for evaluating action research? a) The researcher stated several directional research hypotheses. b) The process was emergent and cycled between research, action, and reflection. c) The researcher should have control over the intervention. d) Reliable measures should be used. Short-Answer Questions 1) What are some of the main ways the knowledge generated from action research is disseminated? 2) Give two reasons a descriptive summary, rather than an inferential analysis, is the preferred method for presenting the results from an action research study. Essay Question Discuss the potential for harm to participants in action research as it relates to ethics and human subjects protection. What are some ways an action researcher can ensure the designs are ethical? What needs to happen if the results of the study will be disseminated?

Chapter 15 Test Answer Key ANSWER KEY: Chapter 15 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. c 2. a 3. a 4. d 5. b 6. b 7. a 8. b 9. b 10. a

INSTRUCTOR’S EXAM ITEMS 1. c 2. d 3. c 4. b 5. d 6. d 7. b 8. b 9. d 10. a

Chapter 16 Test Item File Sarah A. Marrs

Educational Research: Fundamentals, Principles, and Methods Eighth Edition James H. McMillan

ISBN-10:

0-13-576987-6

ISBN-13: 978-0-13-576987-4

Table of Contents Chapter 16: Writing Research Proposals, Reports, and Articles .............................. 1 Learning Outcome Quizzes ........................................................................ 1 Application Exercises ................................................................................ 12 Test Items ................................................................................................. 17 Test Answer Key ....................................................................................... 22

Chapter 16: Writing Research Proposals, Reports, and Articles Chapter 16 Learning Outcome Quizzes Learning Outcome 16.1: Identify key elements of effective scholarly/academic writing. [Q1] Which of the following terms means to write to others in a way that systematically, clearly, and objectively presents information, and adds interpretation, analysis, limitations, and conclusions? 1. Scholarly voice [correct] 2. Academic expression 3. Inclusive language 4. Referencing [Feedback for Correct Answer 1] Scholarly voice means to write to others in a way that systematically, clearly, and objectively presents information, and adds interpretation, analysis, limitations, and conclusions. [Feedback for Answer Choice 2] While academics do express themselves in their writing, this is not the phrase best used to answer this question. [Feedback for Answer Choice 3] Inclusive language is language that does not exclude based on issues of race, class, gender, or ability. [Feedback for Answer Choice 4] Referencing refers to the act of citing previous scholars when making academic arguments. [Q2] Which of the following should be avoided in scholarly writing? 1. Simple sentences 2. Active voice 3. Casual language [correct] 1 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

4. Inclusive language [Feedback for Answer Choice 1] Actually, simple sentences that are clear and communicate efficiently are welcomed in scholarly writing. [Feedback for Answer Choice 2] Active voice is an appropriate voice for scholarly writing. [Feedback for Correct Answer 3] Typically, it is important to avoid casual language such as metaphors and jargon in scholarly writing. [Feedback for Answer Choice 4] Inclusive language should be used in all forms of writing, including scholarly writing. [Q3] Language appropriate for scholarly writing that makes sure not to discriminate based on age, race, gender, or ability status is known as which of the following? 1. Scholarly tone 2. Scholarly language 3. Scholarly voice 4. Inclusive language [correct] [Feedback for Answer Choice 1] Scholarly tone refers to a professional tone that should be found in scholarly writing, but this is not the best answer. [Feedback for Answer Choice 2] This phrase may imply language that is typical of research reporting, but this is not the best answer. [Feedback for Answer Choice 3] Scholarly voice means to write to others in a way that systematically, clearly, and objectively presents information, and adds interpretation, analysis, limitations, and conclusions. It is important, but not the specific answer sought here. [Feedback for Correct Answer 4] 2 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

It is important to use language that avoids inaccurate attribution to groups of individuals, offensiveness, and biased assumptions of persons of specific characteristics or groupings, such as age, gender, racial or ethnic identify, sexual orientation, religion, disability, or other characteristics. The goal is to write in a way that is affirmative and inclusive. [Q4] Consider the following sentence: According to Waldon (2020), “gifted children often contain significantly advanced problem-solving skills” (p. 32). Which of the following does this sentence contain? 1. In-text citation [correct] 2. Reference list 3. Footnote 4. Scholarly reference [Feedback for Correct Answer 1] In this sentence, an in-text citation, quoting Waldon (2020), is present. [Feedback for Answer Choice 2] The reference list stands alone at the end of a research report. [Feedback for Answer Choice 3] Footnotes are found at the bottom of the page and contain additional notation and information. [Feedback for Answer Choice 4] While this is a scholarly sentence, this is not a phrase used to describe the identification of previous authors. Learning Outcome 16.2: Know the sections and subsections used to format quantitative, qualitative, and mixed methods studies. [Q1] In which of the following sections of a research report would you find a brief description of the study? 1. Introduction 2. Abstract [correct] 3 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

3. Methods 4. Findings [Feedback for Answer Choice 1] The introduction section typically outlines the study problem and its context, but it does not summarize the entire study. [Feedback for Correct Answer 2] The abstract is found at the beginning of the research report. It provides the reader an overview of the study and offers a quick summary. [Feedback for Answer Choice 3] The methods section of a research report outlines the methodological components of a study. This is where you will learn whether the study was a quantitative or qualitative study, for example, as well as how the sample was derived. [Feedback for Answer Choice 4] The findings section simply provides an overview of the study’s results. [Q2] In which of the followings parts of a research report would the reader find a few paragraphs that establish the context for the study? 1. Abstract 2. Introduction [correct] 3. Methods 4. Findings [Feedback for Answer Choice 1] The abstract is found at the beginning of the research report. It provides the reader an overview of the study and offers a quick summary. [Feedback for Correct Answer 2] The introduction section typically outlines the study problem and its context, but it does not summarize the entire study. [Feedback for Answer Choice 3] The methods section of a research report outlines the methodological components of a study. This is where you will learn whether the study was a quantitative or qualitative study, for example, as well as how the sample was derived. 4 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 4] The findings section simply provides an overview of the study’s results. [Q3] Consider the following sentence: “Stratified random sampling techniques were employed in order to produce a normally distributed sample.” Which section of the research report would most likely contain this sentence? 1. Abstract 2. Methods [correct] 3. Introduction 4. Literature review [Feedback for Answer Choice 1] The abstract is found at the beginning of the research report. It provides the reader an overview of the study and offers a quick summary. [Feedback for Correct Answer 2] This sentence is describing the sampling methods of the researcher. This would naturally be found in the methods section. [Feedback for Answer Choice 3] The introduction simply sets the context for this study. It is not likely that the introduction would contain any detailed information about sampling techniques. [Feedback for Answer Choice 4] The literature review is the section devoted to describing previous studies that are related to the study being reported. Sampling techniques unique to a current study would not be found in this section. [Q4] Which of the following finding sections of a research report would likely contain a description of data coding techniques used by the researcher? 1. Qualitative [correct] 2. Quantitative 3. Action research 5 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

4. Descriptive studies [Feedback for Correct Answer 1] Qualitative methods require coding in order to develop themes and report findings. [Feedback for Answer Choice 2] Quantitative research reports will not likely contain any talk of coding of data. [Feedback for Answer Choice 3] Action research may include the coding of data, but this is not the most probable answer. [Feedback for Answer Choice 4] Descriptive studies are typically quantitative in nature and use numbers, not codes or themes, to present data. Learning Outcome 16.3: Apply the guidelines for writing effective discussion/conclusion sections. [Q1] “We found clear developmental changes, with children’s causal questions outnumbering explanations at 3 years but explanations outnumbering questions by age 5.” If found in the discussion of a research report, this sentence is considered which of the following: 1. An interpretation of the data based on the hypotheses 2. A restatement of the hypothesis without interpretation 3. An interpretation based on measurement of the variables 4. An overall summary of the findings [correct] [Feedback for Answer Choice 1] This sentence is not providing an analysis of the meaning of data. [Feedback for Answer Choice 2] This statement is not a re-listing of the hypothesis. [Feedback for Answer Choice 3] This statement is providing no discussion of how variables are measured. [Feedback for Correct Answer 4] 6 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

This sentence states what the results were, but it does not explain why the results were obtained and what the results mean. [Q2] “These cross-sectional differences suggest that inexperienced and more experienced word learners use dissimilar strategies when learning new words. However, these results may show more than just strategies for learning new words but also a pronounced change in the nature of verb semantics, and in particular declining reliance on an object as part of verb meaning more generally.” This excerpt from the discussion of a research report of quantitative study is best considered part of which of the following? 1. An interpretation based on theory 2. An interpretation related to the problem [correct] 3. An interpretation based on participant characteristics 4. An interpretation related to measurement of variables [Feedback for Answer Choice 1] This statement is not referring back to any theoretical grounding. [Feedback for Correct Answer 2] From the excerpt, we can infer that the problem under study was how inexperienced (younger) and experienced (older) people learn new words. The author addresses what the differences in the data may mean—what they tell us about how people learn new words. [Feedback for Answer Choice 3] Participant characteristics are not outlined in this statement. [Feedback for Answer Choice 4] The results of variable measurement is not outlined in this excerpt. [Q3] “This result extends Mendelsohn et al.'s (1984) findings to a younger age group, while correcting for the methodological limitations of their design.” This excerpt from the discussion of a research report of quantitative study is best considered part of which of the following? 1. An interpretation based on participant characteristics 7 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2. An interpretation related to methodology 3. An interpretation related to the nature of the intervention 4. An interpretation based on previous research [correct] [Feedback for Answer Choice 1] Participant characteristics are not outlined in this excerpt. [Feedback for Answer Choice 2] Methodology is not discussed in this excerpt. [Feedback for Answer Choice 3] The nature of the intervention is not discussed in this excerpt. [Feedback for Correct Answer 4] This excerpt focuses on how the current results relate to the results of another study. Although the author mentions methodological limitations, the focus there is on the limitations of that earlier study, not of the study conducted and reported for this paper in particular, and so this is not considered interpretation related to methodology. [Q4] “Although the case study described here included a small sample of students, we are pleased to have captured the students’ interest in a traditionally difficult course and heartened to see preliminary evidence that this new technology is enhancing student learning.” This excerpt from a research report is best considered part of the discussion of limitations based on which of the following? 1. Contextual characteristics 2. Participant characteristics [correct] 3. The measures 4. The intervention [Feedback for Answer Choice 1] The context of the study is not presented as a limitation in this excerpt. [Feedback for Correct Answer 2] The author suggests the small sample is the limitation. The author also mentions context—the students were studied in “a traditionally difficult course”—but she does not appear to consider that a limitation in her discussion; rather, she seems to see it as a 8 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

benefit (e.g., if students like and learn from new technology in a difficult course, they may also do so in a somewhat less difficult course). [Feedback for Answer Choice 3] The measures are not presented as limitations in this excerpt. [Feedback for Answer Choice 4] The intervention is not presented as limitations in this excerpt. Learning Outcome 16.4: Describe the steps used to write research proposals, reports, and manuscripts to submit to journals, and the importance of audience in determining what is presented. [Q1] Research reports written for which of the following groups should avoid technical language; focus on purpose, general findings, and conclusions; include an executive summary; and suggest implications for practice? 1. Lay public [correct] 2. Policy makers 3. Peer professionals 4. Practitioners [Feedback for Correct Answer 1] Research reports written for the lay public should avoid technical language; focus on purpose, general findings, and conclusions; include an executive summary; and suggest implications for practice. It is important to remember your audience as you write so that you are able to communicate clearly. [Feedback for Answer Choice 2] Policy makers may not be versed in specific technical language, but you will need to provide more summarized findings, such as an executive summary due to limited time. [Feedback for Answer Choice 3] Peer professionals will have the ability to engage in the scholarly language of a research report. [Feedback for Answer Choice 4] Practitioners may not always be prepared to grapple with more technical language, and implications should center on application. 9 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q2] When preparing a research report, which of the following groups is likely best positioned to engage in a research report that contains a high level of technical language? 1. Lay public 2. Policy makers 3. Peer professionals [correct] 4. Practitioners [Feedback for Answer Choice 1] The lay public is not likely prepared to engage with professional jargon. [Feedback for Answer Choice 2] Policy makers may be interested in how a study can impact policy but may not have the specific training in a particular field to engage with field-specific language. [Feedback for Correct Answer 3] Peer professionals likely have the same type of professional training as the author of a research report. When writing for this audience, feel free to use more nuanced language specific to the field. [Feedback for Answer Choice 4] Practitioners are often skilled at the application process, but not necessarily versed in nuanced professional language. [Q3] Which of the following represents the first step of the writing process? 1. Planning 2. Determining audience [correct] 3. Drafting 4. Revising [Feedback for Answer Choice 1] Planning is the second step in the research writing process. [Feedback for Correct Answer 2] It is critical to consider the audience to whom you are communicating your work. This requires identifying those who will hear and/or read your work. 10 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Feedback for Answer Choice 3] While you may feel like writing a draft is a good start, there are other steps you should consider first. [Feedback for Answer Choice 4] Revising cannot be completed until a draft has been developed. [Q4] Byron is busy reviewing his manuscript on the impact of play on the social and emotional well-being of young children. He is excited about his findings, but he is finding areas where he feels his research is not clearly communicated. In which stage of the writing process is Byron engaged? 1. Revising [correct] 2. Planning 3. Drafting 4. Proofreading [Feedback for Correct Answer 1] Byron is engaged in revising his manuscript. He has found an area that needs improvement, and he is working hard to communicate clearly. [Feedback for Answer Choice 2] If Byron is already working to find better ways to communicate, he is past the planning process. [Feedback for Answer Choice 3] If Byron is already reviewing previously written material, he is past the drafting stage. [Feedback for Answer Choice 4] Proofreading occurs at the end of the research writing process, before the final copy is submitted. The proofreading process is not the process where major changes are made to the research report.

Chapter 16 Application Exercises Chapter 16: Writing Research Proposals, Reports, and Articles Application Exercise 16.1: Analysis vs. Interpretation Learning Outcome 16.1: Identify key elements of effective scholarly/academic writing. [Q1] Consider the following: Marcus is a high school teacher conducting a study of teacher satisfaction in his school, following a change in which two schools merged. After work one day, his wife Ellen asks what he did that day, and he replies, “I spent the whole day reading through and organizing the data I’ve collected in the last month.” Did Marcus spend the day analyzing, interpreting, or neither one? [Q1 Model Response] Marcus was analyzing—he’s organizing, reading, and identifying themes in his data. [Q2] Consider the following: Marcus is a high school teacher conducting a study of teacher satisfaction in his school, following a change in which two schools merged. After work one day, his wife Ellen asks what he did that day, and he replies, “I spent the whole day reading through and organizing the data I’ve collected in the last month.” Marcus’s wife Ellen is also an educational researcher. She has been working on an ethnographic study of children’s aggressive behaviors on the playground. She tells Marcus, “Today I spent most of the day trying to figure out what’s most important in my own data.” Did Ellen spend the day analyzing, interpreting, or neither one? [Q2 Model Response] Ellen was interpreting—looking for meaning in her data. It sounds as if she was interpreting based on her data analysis. [Q3] Consider the following:

Marcus is a high school teacher conducting a study of teacher satisfaction in his school, following a change in which two schools merged. After work one day, his wife Ellen asks what he did that day, and he replies, “I spent the whole day reading through and organizing the data I’ve collected in the last month.” Marcus’s wife Ellen is also an educational researcher. She has been working on an ethnographic study of children’s aggressive behaviors on the playground. She tells Marcus, “Today I spent most of the day trying to figure out what’s most important in my own data.” A few days later, Marcus tells Ellen, “I have the feeling that tension among faculty members has been growing for some time. I think the transition would have been smoother if key faculty members had been informed and involved in the process.” Was Marcus analyzing, interpreting, or neither one? [Q3 Model Response] Marcus was interpreting—offering an answer to the “what can be learned” question. It sounds as if his interpretation is related to the problem (tension) or his hypothesis (it is not stated in the scenario, but we can infer that Marcus hypothesized that the transition led to tension). Application Exercise 16.2: Report Formatting Learning Outcome 16.2: Know the sections and subsections used to format quantitative, qualitative, and mixed methods studies. [Q1] Consider the following: Stacey has prepared a research report focused on her quantitative study of the impact of targeted tutoring on the reading levels of fifth-graders with identified learning disabilities. In one section of Stacey’s report, the entire study is overviewed briefly, including a short synthesis of major findings. Which section of the research report achieves this goal? How do you know? [Q1 Model Response] This describes the abstract of Stacey’s study. Most research reports and articles will include an abstract (proposals often do not have an abstract). The abstract is a short summary of the main elements of the manuscript (100–150 words). It typically includes the problem that was investigated and a brief description of participants, research design (measures, procedures), findings, and/or conclusions. [Q2] 13 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

Consider the following: Stacey has prepared a research report focused on her quantitative study of the impact of targeted tutoring on the reading levels of fifth-graders with identified learning disabilities. In one section of Stacey’s report, she outlines and maps all prior literature related to her study. She provides an overview of any similar studies as well as studies that focus on tutoring and reading goals. Which section of the research report achieves this goal? How do you know? [Q2 Model Response] This describes Stacey’s literature review. Essentially, the review of literature places the current or proposed study within existing research, showing the relation to previous investigations and theories. It is best to indicate how the research differs from what has been done. This is accomplished with a brief summary of previous research, a critique of the studies, and an indication of how the proposed or conducted research is informed by or fits within the existing research. [Q3] Consider the following: Stacey has prepared a research report focused on her quantitative study of the impact of targeted tutoring on the reading levels of fifth-graders with identified learning disabilities. In one section of Stacey’s report, she outlines the data collection processes she employed as well as the analytic processes used to conclude findings. Which section of the research report achieves this goal? How do you know? [Q3 Model Response] This information would be found in the methods section of Stacey’s report. The purpose of the method part of the document is to describe in detail the design of the study, how participants were selected (or, in the case of a proposal, will be selected), the nature of the measures and procedures for gathering data, and, for experimental studies, the nature of the intervention. A complete description is needed to allow others to evaluate the credibility of the investigation and, if desired, to replicate the study. Beyond these general purposes, though, writing the method section can vary greatly by whether the study is quantitative, qualitative, or mixed methods.

Application Exercise 16.3: The Right Document for the Right Audience Learning Outcome 16.4: Describe the steps used to write research proposals, reports, and manuscripts to submit to journals, and the importance of audience in determining what is presented. 14 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

[Q1] Review Table 16.4 entitled Writing Implications Based on Nature of Document and Primary Audience to answer the following: Larry is a doctoral candidate who is preparing to conduct a research study focused on the experiences of undergraduates over the age of 35. His study will be a qualitative study that involves interviewing undergrads at his current institution. He has written up his plan for research. It includes technical language, assumes knowledge of research terms, and was guided by prior research in the field. What type of document is Larry most likely preparing? How do you know? [Q1 Model Response] Larry is most likely preparing a research proposal. This is evident in his use of technical language, the assumption of knowledge of research terms and procedures, and the fact that he has prepared his proposal based on other successful proposals. [Q2] Review Table 16.4 entitled Writing Implications Based on Nature of Document and Primary Audience to answer the following: Larry is a doctoral candidate who just completed a research study focused on the experiences of undergraduates over the age of 35. His study was qualitative study that involved interviewing undergrads at his current institution. He has written up the results of his study for publication. This particular draft does not use overly technical language or jargon and instead is focused on informing his local campus about the experiences of the students in his study. In one section, he even suggested implications for how professors could change their class policies to be more accommodating to nontraditional students. What type of document is Larry most likely preparing? How do you know? [Q2 Model Response] In this instance, Larry has prepared a research report that is primarily targeted for the lay person. He has communicated his work in such a way that stakeholders can use it to make better educational decisions, but he has also just generally raised awareness for the issue at hand. [Q3] Review Table 16.4 entitled Writing Implications Based on Nature of Document and Primary Audience to answer the following:

Larry is a doctoral candidate who just completed a research study focused on the experiences of undergraduates over the age of 35. His study was qualitative study that involved interviewing undergrads at his current institution. He has written the results of his study for publication. This particular version uses appropriate technical language and assumes knowledge of research procedures. His goal is for this publication to add to the scholarly literature in the field. What type of document is Larry most likely preparing? How do you know? [Q3 Model Response] In this instance, Larry has prepared a research article. He’s used professional language and jargon, assumed an understanding of research, and written in a way that attempts to add to the science in the field in a professional format. He will submit this article to professional journals in the field and undergo the peer review process.

Chapter 16 Test Items SELF-ASSESSMENT ITEMS 1) In research reports and articles, which of the following is true about the conclusions? a) They are always written in technical language. b) They are always included as a separate section at the end of the report. c) They always include an interpretation of the results in relation to a theory. d) They always summarize the major findings. 2) Dr. Patton explained the meaning of the statistical analysis of her data in a few paragraphs. One is likely to find these paragraphs in which part of the Discussion section? a) Interpretation b) Conclusion c) Recommendations d) Implications 3) The statistical analysis of Dr. Diaz’s data indicated a significant difference in posttest scores favoring students who used laptops in their fifth-grade language arts classes. She argued convincingly that the results supported her research hypothesis and that few, if any, issues related to her methodology were problematic. Her results were also consistent with prior research. Which conclusion is most reasonable given this information? a) Laptops are effective tools for all fifth-grade students. b) Laptops should be used for all middle school subjects. c) Laptops can be an effective learning tool for some fifth-grade language arts students. d) Use of laptops will help fifth-grade students learn. 4) Mrs. Hayden conducted a nonexperimental study at her school examining the relationship between teachers’ assessment practices and their students’ achievement. She designed an observational instrument based on operational definitions of assessment practices that were well documented in the literature to measure teachers’ assessment practices. She used a well-validated and reliable achievement test as an indicator of student academic success. Her analysis found a significant positive relationship between teachers’ use of alternative assessments and students’ scores on the achievement test. Mrs. Hayden concluded that teachers should use alternative assessments on a consistent basis. On the basis of which limitation would you question her conclusions? a) Her use of a nonexperimental design to draw causal conclusions 17 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

b) Her insider role of teacher as researcher c) The fact that she used an achievement test to measure students’ academic success d) Her use of operational definitions rather than conceptual definitions to develop her observational instrument 5) Which of the following is NOT true about the conclusion in a Discussion section of a research report or article? a) The researcher may state whether the findings supported the research hypotheses. b) The conclusion may include statement limitations related to participant characteristics. c) The conclusion often refers back to the research question, hypotheses, or purpose of the study. d) Conclusions typically contain a restatement of the results using statistical language. 6) Which of the following is an indication a study has strong ecological external validity? a) The relationship between the participants and the social environment is complex. b) Contextual characteristics of the research study setting allow for the results to be generalized to different settings. c) The details of how the participants interacted with their physical environment indicate consistency. d) There are historical and external influences that determine the extent to which the results are valid or invalid. 7) Which of the following is NOT true about the Discussion section of a research report or article? a) Sometimes the results are interpreted in regard to whether they support or contradict a theory. b) Sometimes the researcher indicates a professional opinion about how well the data answered the research question. c) Sometimes unexpected findings might be presented as surprising. d) Sometimes they include a detailed description of the instruments used in the analysis.

INSTRUCTOR’S EXAM ITEMS Multiple-Choice Questions 1) Which of the following is NOT the purpose of the Discussion section of a research report or article? a) To present an integration of the findings with the research problem b) To present an explanation of how the results were obtained and what the results were c) To present a summary of the findings and an evaluation of the methodology d) To present a synthesis of the study reflecting the professional judgment of the researcher 2) Which part of the Discussion section would contain recommendations for future study? a) Interpretation b) Conclusion c) Recommendations d) Methodology 3) A researcher spent several paragraphs in his report explaining the reasons why the data collection process became so complicated. He also discussed how he thought this affected the way in which he viewed the results. This is information you would likely find in which of the following sections of the report? a) Introduction b) Interpretation c) Results d) Implications 4) A researcher conducted an experimental study examining the impact of a new curriculum on ninth graders’ achievement as measured by the state proficiency exam. The participants, who were enrolled in a large urban school district, were placed in treatment and control groups by classroom. The data collected indicated a significant difference in posttest scores favoring the students who experienced the new curriculum. The researcher concluded that the results supported her research hypothesis. Which conclusion is most reasonable given this information? a) The new curriculum would be effective for all ninth-grade students. b) The new curriculum is a much better curriculum than the old curriculum. c) The new curriculum is likely a more effective method than the traditional method with ninth graders from a large urban district. 19 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

d) The new curriculum is probably not effective with most ninth graders since the design used intact groups. 5) Mr. Lopez conducted a nonexperimental study at his school examining the relationship between teachers’ classroom behavior practices and their students’ achievement. He used a frequently used, well-validated, Likert-type survey instrument to measure teachers’ classroom behavior practices and a well-validated and reliable achievement test as an indicator of their students’ academic success. His analysis found a significant negative relationship between teachers’ use of negative reinforcement and students’ scores on the achievement test. Mr. Lopez concluded that negative reinforcement resulted in lower achievement. Which limitation might cause you to question his conclusions? a) He used a nonexperimental design to reach causal conclusions. b) He was a teacher in the school and was probably biased. c) He used an achievement test to measure students’ academic success. d) He used a Likert-type survey to measure teachers’ classroom behavior practices. 6) The Conclusion section of a research report or article would NOT include information about which of the following? a) Summary statements related to the research problem b) An interpretation of the findings using nontechnical language c) Answers to questions, hypotheses, or purposes of the research d) Operational definitions of the variables 7) A sailing camp director read an article about research into the positive effects of creating a competitive environment for students learning to sail. The participants in the study were older and had different racial backgrounds than the students at his camp. The camp director decided to focus on teaching his students to enjoy the sport for itself and not encourage competition with other student sailors. His decision to disregard the results from the research study reflects a concern for which limitation? a) Ecological external validity b) Population external validity c) Population internal validity d) Ecological internal validity Short-Answer Questions 1) Describe one important characteristic that all Discussion sections should include and tell why it should be included. 20 Copyright © 2022 Pearson Education, Inc. All Rights Reserved.

2) Why should researchers be careful interpreting results that fail to show statistical significance? 3) Describe the difference between quantitative and qualitative Results and Discussion sections. 4) Explain why it is important for authors to consider their audience prior to disseminating their research. Essay Question Describe in detail at least three limitations that should be considered when reading results from a research study. Why should researchers incorporate previous studies when explaining their results? Why should consumers of research be cautious when reading educational research articles and reports?

Chapter 16 Test Answer Key ANSWER KEY: Chapter 16 Test (Multiple Choice) SELF-ASSESSMENT ITEMS 1. d 2. a 3. c 4. a 5. d 6. b 7. d

INSTRUCTOR’S EXAM ITEMS 1. b 2. c 3. b 4. c 5. a 6. d 7. b