In 2006, Gelman and Stern published a paper that critiqued the assertion of effect heterogeneity on the basis of p < 0.05 in one stratum and p > 0.05 in another stratum. Although the paper has been cited over 200 times, the practice remains depressingly common. When I stumble across new examples in the literature, I will post them here. I'll also post actual heterogeneity tests as described in Kaufman & MacLehose 2013, using Stata programs for either ratio effect measures or additive effect measures.
1]
Koyama A, Houston DK, Simonsick EM, Lee JS, Ayonayon HN, Shahar DR, Rosano C, Satterfield S, Yaffe K
Association Between the Mediterranean Diet and Cognitive Decline in a Biracial Population. [PDF]
J Gerontol A Biol Sci Med Sci. 2014 Jul 3. pii: glu097. [Epub ahead of print] PMID:24994847
Association Between Mediterranean diet (MedDiet) Score and Trajectories of Modified Mini Mental State Examination scores (3MS) in 1,452 whites and 874 blacks (Table 3) (adjusted for age, sex, education, body mass index, current smoking, physical activity, depression, diabetes, total energy intake, SES)
High vs Lower Scores (points/y) stratum-specific p Value
Whites 0.09 (−0.03, 0.21) 0.14
Blacks 0.22 (0.05, 0.39) 0.01
Actual heterogeneity:
Pooled Difference = 0.13
Cochran's Q = 1.50 p-value = 0.22
Per 5 pt Increase stratum-specific p Value
Whites 0.02 (−0.02, 0.07) 0.39
Blacks 0.08 (0.01, 0.15) 0.02
Actual heterogeneity:
Pooled Difference = 0.04
Cochran's Q = 1.87 p-value = 0.17
Author interpretation: "In line with the collective evidence, our study is the first to show a possible race-specific association between the Mediterranean diet and cognitive decline."
Press release: "Our study is the first to show a possible race-specific association between the Mediterranean diet and cognitive decline."
Koyama A, Houston DK, Simonsick EM, Lee JS, Ayonayon HN, Shahar DR, Rosano C, Satterfield S, Yaffe K
Association Between the Mediterranean Diet and Cognitive Decline in a Biracial Population. [PDF]
J Gerontol A Biol Sci Med Sci. 2014 Jul 3. pii: glu097. [Epub ahead of print] PMID:24994847
Association Between Mediterranean diet (MedDiet) Score and Trajectories of Modified Mini Mental State Examination scores (3MS) in 1,452 whites and 874 blacks (Table 3) (adjusted for age, sex, education, body mass index, current smoking, physical activity, depression, diabetes, total energy intake, SES)
High vs Lower Scores (points/y) stratum-specific p Value
Whites 0.09 (−0.03, 0.21) 0.14
Blacks 0.22 (0.05, 0.39) 0.01
Actual heterogeneity:
Pooled Difference = 0.13
Cochran's Q = 1.50 p-value = 0.22
Per 5 pt Increase stratum-specific p Value
Whites 0.02 (−0.02, 0.07) 0.39
Blacks 0.08 (0.01, 0.15) 0.02
Actual heterogeneity:
Pooled Difference = 0.04
Cochran's Q = 1.87 p-value = 0.17
Author interpretation: "In line with the collective evidence, our study is the first to show a possible race-specific association between the Mediterranean diet and cognitive decline."
Press release: "Our study is the first to show a possible race-specific association between the Mediterranean diet and cognitive decline."
2]
Coogan PF, Wise LA, O'Connor GT, Brown TA, Palmer JR, Rosenberg L.
Abuse during childhood and adolescence and risk of adult-onset asthma in African American women. [PDF]
J Allergy Clin Immunol. 2013 Apr;131(4):1058-63. PMID: 23219171
Author assertion: "The IRR was higher for childhood physical abuse (IRR, 1.29; 95% CI, 1.07-1.49) than for childhood sexual abuse (IRR, 1.15; 95% CI, 0.88-1.49)....The association between childhood abuse and asthma incidence was stronger in older compared with younger women." [abstract]
"The IRR was higher for childhood physical abuse than for childhood sexual abuse"
Childhood physical abuse compared to no abuse: IRR = 1.29 (1.07, 1.49)
Childhood sexual abuse compared to no abuse: IRR = 1.15 (0.88, 1.49)
Actual heterogeneity:
Pooled IRR = 1.26
Cochran's Q = 0.58 p-value = 0.45
"The association between childhood abuse and asthma incidence was stronger in older
compared with younger women." (from Table 4)
Physical abuse and sexual abuse in childhood compared to no abuse among women 40 years
old and older: IRR = 1.35 (0.99-1.86)
Physical abuse and sexual abuse in childhood compared to no abuse among women less than
40 years old: IRR = 1.13 (0.84-1.53)
Actual heterogeneity:
Pooled IRR= 1.23
Cochran's Q = 0.62 p-value = 0.43
Author interpretation: "Key messages (p. 1063): The association was stronger for childhood physical than
sexual abuse and was stronger in older compared with younger women."
3]
Christelle M. Clary, Yuddy Ramos, Martine Shareck, Yan Kestens. Should we use absolute or relative measures when assessing foodscape exposure in relation to fruit and vegetable intake? Evidence from a wide-scale Canadian study. Preventive Medicine, 2015; 71: 83 DOI: 10.1016/j.ypmed.2014.11.023 [PDF] PMID: 25481095
Press release headline: "Men's diets are related to local offerings, unlike women's".
Author assertion: "We found that, for men only, intake of fruit and vegetables was positively associated with the proportion of healthy food outlets around home," explained Christelle M. Clary of the university's department of social and preventative medicine, who led the study. Why men and not women? "This may be because women, who are in general more nutritionally knowledgeable, may engage in different food shopping strategies than men,
and rely on other aspects of the food environment than the proportion of food stores locally available," suggested Professor Yan Kestens, who directed the study.
But here are the actual estimates from Table 3:
Men in Toronto/Montreal (n = 13,268): β = 0.012 (95% CI: 0.006; 0.018), p < 0.001
Men in Calgary/Ottawa/Vancouver (n = 8894): β = 0.008 (95% CI: 0.004, 0.012), p < 0.001
Women combined across all sites (n = 27,241): β = 0.004 (95% CI: 0.000, 0.008), p = 0.051
The heterogeneity p-value for the two groups of men is Cochran's Q = 1.18 p-value = 0.2770, so why are these groups divided?
The heterogeneity p-value for the women compared to the Calgary/Ottawa/Vancouver men is Cochran's Q = 1.92 p-value = 0.1658, so why are these treated as categorically different?
Coogan PF, Wise LA, O'Connor GT, Brown TA, Palmer JR, Rosenberg L.
Abuse during childhood and adolescence and risk of adult-onset asthma in African American women. [PDF]
J Allergy Clin Immunol. 2013 Apr;131(4):1058-63. PMID: 23219171
Author assertion: "The IRR was higher for childhood physical abuse (IRR, 1.29; 95% CI, 1.07-1.49) than for childhood sexual abuse (IRR, 1.15; 95% CI, 0.88-1.49)....The association between childhood abuse and asthma incidence was stronger in older compared with younger women." [abstract]
"The IRR was higher for childhood physical abuse than for childhood sexual abuse"
Childhood physical abuse compared to no abuse: IRR = 1.29 (1.07, 1.49)
Childhood sexual abuse compared to no abuse: IRR = 1.15 (0.88, 1.49)
Actual heterogeneity:
Pooled IRR = 1.26
Cochran's Q = 0.58 p-value = 0.45
"The association between childhood abuse and asthma incidence was stronger in older
compared with younger women." (from Table 4)
Physical abuse and sexual abuse in childhood compared to no abuse among women 40 years
old and older: IRR = 1.35 (0.99-1.86)
Physical abuse and sexual abuse in childhood compared to no abuse among women less than
40 years old: IRR = 1.13 (0.84-1.53)
Actual heterogeneity:
Pooled IRR= 1.23
Cochran's Q = 0.62 p-value = 0.43
Author interpretation: "Key messages (p. 1063): The association was stronger for childhood physical than
sexual abuse and was stronger in older compared with younger women."
3]
Christelle M. Clary, Yuddy Ramos, Martine Shareck, Yan Kestens. Should we use absolute or relative measures when assessing foodscape exposure in relation to fruit and vegetable intake? Evidence from a wide-scale Canadian study. Preventive Medicine, 2015; 71: 83 DOI: 10.1016/j.ypmed.2014.11.023 [PDF] PMID: 25481095
Press release headline: "Men's diets are related to local offerings, unlike women's".
Author assertion: "We found that, for men only, intake of fruit and vegetables was positively associated with the proportion of healthy food outlets around home," explained Christelle M. Clary of the university's department of social and preventative medicine, who led the study. Why men and not women? "This may be because women, who are in general more nutritionally knowledgeable, may engage in different food shopping strategies than men,
and rely on other aspects of the food environment than the proportion of food stores locally available," suggested Professor Yan Kestens, who directed the study.
But here are the actual estimates from Table 3:
Men in Toronto/Montreal (n = 13,268): β = 0.012 (95% CI: 0.006; 0.018), p < 0.001
Men in Calgary/Ottawa/Vancouver (n = 8894): β = 0.008 (95% CI: 0.004, 0.012), p < 0.001
Women combined across all sites (n = 27,241): β = 0.004 (95% CI: 0.000, 0.008), p = 0.051
The heterogeneity p-value for the two groups of men is Cochran's Q = 1.18 p-value = 0.2770, so why are these groups divided?
The heterogeneity p-value for the women compared to the Calgary/Ottawa/Vancouver men is Cochran's Q = 1.92 p-value = 0.1658, so why are these treated as categorically different?
4]
Stijnen P, Tuand K, Varga TV, Franks PW, Aertgeerts B, Creemers JW. The association of common variants in PCSK1 with obesity: a HuGE review and meta-analysis. Am J Epidemiol. 2014 Dec 1;180(11): 1051-65 [PDF] PMID: 25355447
Author assertions:
"This systematic Human Genome Epidemiology (HuGE) review showed convincingly that the SNPs rs6232, rs6234, and rs6235 in PCSK1 are associated with obesity in Caucasians." [abstract]
"We found an association with obesity in the Caucasian population (OR = 1.17, 95% CI: 1.09, 1.27) but not in the Hispanic population (OR = 1.50, 95% CI: 0.65, 3.48) (Table 6)." (p. 1061)
"In conclusion, the current meta-analysis showed a mild but significant contribution of the PCSK1 SNPs rs6232 and rs6234–rs6235 to the genetic predisposition to obesity in an age- and ethnicity-dependent fashion." (p. 1063)
But the authors themselves provide the contradictory heterogeneity test which shows no evidence that the groups should be interpreted as having different associations between SNPs and obesity:
"There were no significant differences in the association between the 3 ethnicity subgroups (P = 0.43)." (p. 1061)
Stijnen P, Tuand K, Varga TV, Franks PW, Aertgeerts B, Creemers JW. The association of common variants in PCSK1 with obesity: a HuGE review and meta-analysis. Am J Epidemiol. 2014 Dec 1;180(11): 1051-65 [PDF] PMID: 25355447
Author assertions:
"This systematic Human Genome Epidemiology (HuGE) review showed convincingly that the SNPs rs6232, rs6234, and rs6235 in PCSK1 are associated with obesity in Caucasians." [abstract]
"We found an association with obesity in the Caucasian population (OR = 1.17, 95% CI: 1.09, 1.27) but not in the Hispanic population (OR = 1.50, 95% CI: 0.65, 3.48) (Table 6)." (p. 1061)
"In conclusion, the current meta-analysis showed a mild but significant contribution of the PCSK1 SNPs rs6232 and rs6234–rs6235 to the genetic predisposition to obesity in an age- and ethnicity-dependent fashion." (p. 1063)
But the authors themselves provide the contradictory heterogeneity test which shows no evidence that the groups should be interpreted as having different associations between SNPs and obesity:
"There were no significant differences in the association between the 3 ethnicity subgroups (P = 0.43)." (p. 1061)
5]
Manichaikul A, Wang XQ, Musani SK, Herrington DM, Post WS, Wilson JG, Rich SS, Rodriguez A. Association of the Lipoprotein Receptor SCARB1 Common Missense Variant rs4238001 with Incident Coronary Heart Disease. PLoS One. 2015 May 20;10(5):e0125497. doi: 10.1371/journal.pone.0125497. PMID:25993026 [PDF]
Press release headline: " Common mutation linked to heart disease"
Author assertions:
“The evidence of association was strongest in African Americans”
“This… may serve to address the higher rate of cardiovascular mortality in minority populations, especially for premature cardiovascular death (<65 years of age) in African Americans.”
But here are the actual estimates from Table 2:
Combined Cohort White (n = 11,957): HR = 1.126 (95% CI: 0.97, 1.307), p = 0.119
Combined Cohort Black (n = 5,962): HR = 1.485 (95% CI: 1.068, 2.063), p = 0.019
So the authors are excited because p > 0.05 in whites and p < 0.05 in blacks.
But the heterogeneity p-value for the two groups Cochran's Q = 2.26, p-value = 0.1329.
The sex-specific estimates lower in the table are even more overlapping.
Manichaikul A, Wang XQ, Musani SK, Herrington DM, Post WS, Wilson JG, Rich SS, Rodriguez A. Association of the Lipoprotein Receptor SCARB1 Common Missense Variant rs4238001 with Incident Coronary Heart Disease. PLoS One. 2015 May 20;10(5):e0125497. doi: 10.1371/journal.pone.0125497. PMID:25993026 [PDF]
Press release headline: " Common mutation linked to heart disease"
Author assertions:
“The evidence of association was strongest in African Americans”
“This… may serve to address the higher rate of cardiovascular mortality in minority populations, especially for premature cardiovascular death (<65 years of age) in African Americans.”
But here are the actual estimates from Table 2:
Combined Cohort White (n = 11,957): HR = 1.126 (95% CI: 0.97, 1.307), p = 0.119
Combined Cohort Black (n = 5,962): HR = 1.485 (95% CI: 1.068, 2.063), p = 0.019
So the authors are excited because p > 0.05 in whites and p < 0.05 in blacks.
But the heterogeneity p-value for the two groups Cochran's Q = 2.26, p-value = 0.1329.
The sex-specific estimates lower in the table are even more overlapping.