Responding to Reviewers’ Comments Dr Lukman Thalib Faculty of Medicine Kuwait University
02/04/13
Why papers get rejected?
2 Saudi
Painful Review Process!! “Peer review and editing lead to better reports of research results� Evidence: Comparing the quality of manuscript before and after review (JAMA;1994 and Lancet 1996)
02/04/13
Saudi Heart Association Meeting
02/04/13
Why papers get rejected?
4 Saudi
Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that, there is less than a 50% chance that the results of any randomly chosen scientific paper are true.
02/04/13
Saudi Heart Association Meeting
5
Are the reviews getting serious?
02/04/13
Why papers get rejected?
6 Saudi
Many Flavours 1. 2. 3. 4. 5. 6.
Accepted without any modifications Accepted with minor modifications Accepted with major modifications Provisional rejection Outright rejection In-house rejection without review
Outright or in-house rejection • Can it be a blessing? • Expect a rejection all the time is a better attitude • May not reflect badly on the manuscript • You may be a victim • Your paper may have some issues
Reputed rejection letter from a Chinese Journal We have read your manuscript with boundless delight. If we were to publish your paper, it would be impossible for us to publish any work of a lower standard. And, as it is unthinkable that, in the next one thousand years, we shall see its equal, we are, to our regret, compelled to return your divine composition, and to beg you a thousand times to overlook our short sight and timidity.
E-mail message from an European author upon receiving a rejection letter: Dr. Szklo, What could I expect from an american (sic) editor? I will no longer buy american (sic) products.
Appeal? • Outright rejection? • Submitting elsewhere?? • Journals are also learning that if reviewed and rejected there is always a chance to argue – That is why in-house rejection came about!!
Major modification
Important considerations in responding to the reviewers
Immediate response • Take a break – not a long one!! • Get Mad. Then Get Over It. • Reviews should typically make papers better!! • If the reviewer is wrong, does not mean you are right either!!!
Crafting a response • Consider what the decision letter says • Gather your thoughts • Choose Your Battles Wisely! • If you don’t understand the comments fully seek help in fully grasping them – Can consult a statistician if it is methods – Email the editor for clarify if needed,
Complete response • Address every single comment even if you don’t want to change (Reviewer + Editor) • Try to make all changes suggested unless the changes are inaccurate • Get someone to check your responses and changes • When you disagree provide clear evidence and provide your argument carefully. – “ I prefer this way” is not an argument!!
Conflicting reviews • Do not pit one reviewer against another • If the reviewers conflict choose the most valid comment and explain your decision in the letter carefully • If two reviewers say the same comment take that comment very seriously
Changes methods and contents • Re-evaluate and clarify your text • Check if it is feasible to change and defend yourself if not feasible – This study should have been an RCT??
• Asking for re-analyes is common and most of the time doable as you don’t need to collect new data • If complete re-write is requested – Evaluate the benefit or submit elsewhere
Changes to format, grammar, figures, references • This is generally acceptable • Follow the journal guidelines • Asking to cut the text is common and usually it makes the paper better • Rarely asked to split the paper • Cut from one and addition from other reviewer is not unusual
Tactics • Respond politely and thoughtfuly • Respond completely and clearly • Detailed response is good but no need for un-necessary details, although there is no word limit
Response • • • •
Give point-by-point responses Provide well-reasoned arguments Pay attention to detail Watch your tone!!!
Responding to Reviewers’ Comments •
In your cover letter, answer each critique/comment made by reviewers.
•
Highlight in the revised paper the changes made as a result of the reviewers’ comments.
•
Be courteous. A good idea is to start your response with, “I thank the reviewer for this thoughtful suggestion…” (that is, if you think it is thoughtful…)
•
Try to consider carefully each suggestion or comment.
•
If you disagree with a request for a change, but it is easy to implement it and it does not affect the science…do it!
•
If you disagree entirely with a criticism and believe it is scientifically
Example “Study subjects ranged in age from 42 to 75 years; 27% were 42 to 55 years and 41% were between 55 to 65 years”. Reviewer A wrote “The description of the age distribution of study subjects were unclear. Were 55 years were in the first group or second group? And the two groups add up to only 68%”
Response The reviewer was concerned about the lack of clarity in our description of the age distribution of study subjects in the first paragraph of the “Results” section. The reviewer is correct, and we appreciate the chance to make ourselves clearer. We have revised the paper as follows: “Twenty-seven precent of our study subjects were young than 55 years, 41% were 55 to 65 years and 32% were 66 to 75 years”
Why papers get rejected?
Scientific quality of the work • Good science and higher level of evidence generation always prevail! • If there is a strong scientific basis and novelty then even if badly written can survive eventually • Worst is the poor quality study, written poorly. Reviews like to use their knife liberally on these types of studies!!
List of complains?? • • • • • • •
Statistical power issues Methodology / data modelling issues Measurement issues Presentation issues Study design issues Language, grammar and syntax Length of the sections etc
Reviewer issues (Annals of Med) Methodology Reviewer • 33% Results presentn • 17% - methods Statistical reviewer • 28% Results presentn • 16% - statistics
02/04/13
Regular Reviewer • 45% non statistical comments • 18% presentation of results • Review on method issues were often contradictory to stat/method reviewer!!
Why papers get rejected?
28 Saudi
% papers rejected for wrong statistics • % vary with varying journals • Anywhere between 35% to 80% of the papers are rejected because of wrong statistics » Study design » Data analysis » Results presentation » Documentation » Interpretation /inference/transparency 02/04/13
Saudi Heart Association Meeting
29
New England Journal of Medicine A comparison of 1978 -1979 and 1990 revealed dramatic changes in the use of statistics. In particular there was now much greater use of complex methods such as Logistic regression and Proportional hazards regression 02/04/13
Saudi Heart Association Meeting
30
The Lancet • Only 54% of papers were deemed acceptable or acceptable after revision. • Methods and of results were found inadequate in about 50%of the papers • 25% with major errors in inference connected to analysis or design. Conclusions drawn not justified by study design and statistical analysis. 02/04/13
Saudi Heart Association Meeting
31
Peer review of statistics in medical research Journal reviewers are even more baffled by sample size issues than grant proposal reviewers Prof Mary Fox Peer review of statistics in medical research BMJ 2002;325:491 ( 31 August )
02/04/13
Saudi Heart Association Meeting
32
02/04/13
Why papers get rejected?
33 Saudi
02/04/13
Why papers get rejected?
34 Saudi
Other Check Lists • STRAD and QUORAM are some other check lists
02/04/13
Saudi Heart Association Meeting
35
Myth of P value?
02/04/13
Saudi Heart Association Meeting
36
The earth is round (P <0.05) (Cohen, 1994)
02/04/13
Saudi Heart Association Meeting
37
Mistaken idea!! â&#x20AC;˘ The mistaken idea that a single number can capture both the long-run outcomes of an experiment. â&#x20AC;˘ There is little appreciation in the medical community that the P value based methodology is an amalgam of incompatible elements, whose utility for scientific inference has been the subject of intense debate among statisticians for almost 70 years. 02/04/13
Saudi Heart Association Meeting
38
The P value Fallacy â&#x20AC;˘ Biological understanding and previous research play little formal role in the interpretation of quantitative results. â&#x20AC;˘ Most discussions rely on P values and ultimately can affect the reliability of the conclusions. 02/04/13
Saudi Heart Association Meeting
39
Statistics is not about calculating pvalues Statistics is about rational interpretation of incomplete and imperfect data.
02/04/13
Saudi Heart Association Meeting
40
02/04/13
Why papers get rejected?
41 Saudi
â&#x20AC;&#x153;Absence of evidence is not evidence of absence"
02/04/13
Why papers get rejected?
42 Saudi
Significant findings
â&#x20AC;˘ Large data always lead to significant findings
02/04/13
Why papers get rejected?
43 Saudi
Example â&#x20AC;˘ If you find a correlation between X and Y with a P<0.0001, what would you conclude?
02/04/13
Why papers get rejected?
44 Saudi
Effect size â&#x20AC;˘ The correlation coefficient was only 0.002 â&#x20AC;˘ Clinically significant???
02/04/13
Why papers get rejected?
45 Saudi
Clinical significance â&#x20AC;˘ Clinical and statistical significance are two completely different things. â&#x20AC;˘ Present clinically interpretable effect estimates and their precision (confidence intervals or p-values).
02/04/13
Why papers get rejected?
46 Saudi
Power, effect size and measurements 1. Under power studies 2. Small effects (for example, a drug that works for only 10% of patients), 3. Protocol and endpoints are poorly defined, allowing researchers to massage their conclusions 4. Measurement issues 02/04/13
Saudi Heart Association Meeting
47
02/04/13
Why papers get rejected?
48 Saudi
CONSORT on Power • Should report on all RCT, how the sample size was determined, including details of a prior power calculation. • There is little merit in calculating the statistical power once the results of the trial are known; the power is then appropriately indicated by confidence intervals. • CONSORT agree that failing to reach the planned sample size is not a reason to reject a paper. But power calculations are still of importance to readers.
02/04/13
Why papers get rejected?
49 Saudi
Study design • • • •
Primary outcome / aim not well defined Sample size, power, effect size Use / report on randomisation, blinding etc Inappropriate control groups – huge problem in case-control studies • Use of control groups – comparability of the groups not checked 02/04/13
Why papers get rejected?
50 Saudi
Data analysis • • • • • • •
Use of wrong statistical tests Ignore test assumptions No corrections Inflated type I errors Failure to adjust for confounders Wrong statistical models – very common Independence issues in models
02/04/13
Why papers get rejected?
51 Saudi
Independence â&#x20AC;˘ A patients multiple observations are dependent and should not be analysed using methods requiring independence, e.g. Multiple regression models, chi-squared test, Student's t-test, MannWhitney U-test, ANOVA. â&#x20AC;˘ Unless the statistical methods can deal with dependent data, analyse patients, not knees, hips, shoulders. â&#x20AC;˘ Using methods for independent samples on paired or repeated data. 02/04/13
Why papers get rejected?
52 Saudi
Adjusting for confounders Observational studies usually require adjustment for known and suspected confounding factors.
02/04/13
Why papers get rejected?
53 Saudi
Adjusting for confounders Multiple regression Stratification Randomisation Matching
02/04/13
Why papers get rejected?
54 Saudi
Common issues • Using parametric methods blindly. • Over using hypothesis tests (P values) in preference to confidence intervals. • One-tailed tests are very rarely appropriate. • Failing to analyse clinical trials by intention-to-treat. 02/04/13
Why papers get rejected?
55 Saudi
Multiple Testing • Conclusions should only be drawn from appropriate analyses of a small number of clear, pre-defined hypotheses. • Results from post-hoc subgroup should be treated as speculative. • If many such tests have been carried out adjustment for multiple testing should be considered 02/04/13
Why papers get rejected?
56 Saudi
Testing at multiple time points â&#x20AC;˘ Comparing groups at multiple time points should be avoided â&#x20AC;&#x201C; a summary statistics approach or more complex statistical methods should be used instead
02/04/13
Why papers get rejected?
57 Saudi
Interactions â&#x20AC;˘ When interaction is present cannot interpret the main effects â&#x20AC;˘ Always look for interactions particularly if the theory suggest so
02/04/13
Why papers get rejected?
58 Saudi
Clustered or nested data • Analysis ignoring the nested structure • Multi level models or GEE models • Generalized Mixed models. 02/04/13
Why papers get rejected?
59 Saudi
Modelling issues • • • • •
Correct model MV model Adjustment Interpretation Stepwise method
Conclusion • Publication is fun and mandatory for promotion • Review process is meant to improve the quality of publication, but like in any review process there are flaws and human errors. • Deal with the process in the best possible manner and you will succeed.