The problem with unadjusted multiple and sequential statistical testing

PROBLEM WITH UNADJUSTED MULTIPLE AND SEQUENTIAL STATISTICAL TESTING Tags: Statswork | Statistical Analysis | Statistical Method | Sample Size Significance | Sequential Analysis | Data Collection | Interim Analysis Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Copyright © 2019 Statswork. All rights reserved

- Large number of statistical tests are performed, then there will be a chance of increased false

Bonferroni correction

positive rates or there will be the problem of multiple testing for the sample considered.

- Bonferroni correction will be carried out to deal with the multiple testing problems without making any adjustments.

- This Bonferroni correction have serious drawback.

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Copyright Š 2019 Statswork. All rights reserved

DRAWBACKS OF BONFERRONI CORRECTION

We perform multiple independent tests, then the probability or chance of getting atleast one false positive is calculated as 1-(1-0.05)^n.

Suppose if n=10, then the probability will be 40.14 %, which is very high.

In such situations, the use of Bonferroni correction is not appropriate.

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Copyright Š 2019 Statswork. All rights reserved

Sequential testing problem is an alternative to cope up with the multiple testing problems.

gnitset laitneuqeS

Sequential testing means the researchers collect the data until we reach the fixed threshold.

It takes more effort, time and it's expensive in practice.

One can check the decreasing p-value when the samples are tested sequentially.

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Copyright Š 2019 Statswork. All rights reserved

Uncorrected Multiple Testing Procedure

This procedure seems comfortable, it will have an impact on the estimated values.

When we do sampling sequentially, researchers often face an effect of over estimates.

01

Uncorrected multiple testing procedure, one would impose the stopping rule, say, stop the process once the false positive rate reaches 25%.

03

In the same way, sequential testing problem have a serious drawback.

02

04 05

Effect size is also result in bias nature.

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Copyright Š 2019 Statswork. All rights reserved

If, we impose any stopping rule also it will exceed the limit and gives a false discovery rate.

This kind of testing affects the estimated values apart from the probability values.

Sequential sampling, distance between both group means will increase or decrease and if one wish to continue the process of sampling till both groups yields significant results.

Hence, the sequential testing is biased in significance and also in effect size.

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

gnitseT elpitluM dna laitneuqeS fo melborP

Sample size significance for the simulated 10000 sequential strategies.

From the graph, it is noted that the sequential testing (blue curve) is less severe than the uncorrelated multiple testing (red curve).

Copyright Š 2019 Statswork. All rights reserved

Problem of Unadjusted Sequential Testing

Concept of sequential testing is actually a great idea only if we make necessary corrections to make the sample to be larger in size.

If we sample the data sequentially in smaller bits and achieve the fixed limit means we actually increasing the sample size to attain our goal.

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Copyright Š 2019 Statswork. All rights reserved

Group sequential analysis or interim analysis the researcher have to make an priori specifications about the data.

For instance, one should make the prior decision that the samples should be taken as 50 samples in first level, 100 in second level, etc., and stops when the desired result is obtained.

Main advantage of this technique is that one can stop the data collection

Group Sequential Analysis

when the desired level is obtained.

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Copyright Š 2019 Statswork. All rights reserved

FULL SEQUENTIAL TECHNIQUE

Full sequential technique, there is no prior arrangements is needed.

In early 1940s, Walds used this technique in computing the cumulative log-likelihood ratio for each observation collected and stops the process when a pre-defined threshold is achieved.

This is something like the case in interim analysis.

However, the full sequential technique is not practical.

Suppose if a researcher wants to analyse the sample of 20 group therapy participants, then this may not be appropriate but the group sequential analysis will serves a purpose.

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Copyright Š 2019 Statswork. All rights reserved

CONCLUSION S.No

1.

Method Non-sequential analysis

Description

Sample size needed

It collects a single sample and perform the analysis in later stage. It's an straight forward method but has an disadvantage is that one might collect more data than necessary.

2.

Group sequential analysis

It is also called an interim analysis which make use of a priori decisions for the analysis and stops when significance is reached.

3.

Full sequential analysis

Unlike the above case, it does not requires a prior specifications. It computes a statistical analysis based on the sample once the observation is recorded and stops data collection when it lies outside the specified limit.

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Large

Moderate

Low

Copyright Š 2019 Statswork. All rights reserved

Statswork Lab @ Statswork.com www.statswork.com

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Copyright Š 2019 Statswork. All rights reserved

PHONE NUMBER UK

: +44-1143520021

INDIA

: +91-4448137070

EMAIL ADDRESS info@statswork.com

GET IN TOUCH WITH US

Freelancer Consultant Guest Blog Editor CONTACT hr@workfoster.com

Research Planing | Data Collection | Semantic Annotation | Consumer & Retail Analytics | Econometrics

Copyright © 2019 Statswork. All rights reserved