Strong Control of the Family-Wise Error Rate in Observational Studies That Discover Effect Modification by Exploratory Methods

An effect modifier is a pretreatment covariate such that the magnitude of the treatment effect or its stability changes with the level of the covariate. Generally, other things being equal, larger treatment effects and less heterogeneous treatment effects are less sensitive to unmeasured biases in observational studies. It is known that when there is effect modification, an overall test that ignores an effect modifier may report greater sensitivity to unmeasured bias than a test that combines results at different levels of the effect modifier. This known combined test reports that there is evidence of an effect somewhere that is insensitive to bias of a certain magnitude, but it does not draw inferences about affected subgroups. If there is effect modification, one would like to identify specific subgroups for which there is evidence of effect that is insensitive to small or moderate biases. In the current paper, we propose an exploratory method for discovering effect modification combined with a confirmatory method of simultaneous inference that strongly controls the family-wise error rate in a sensitivity analysis, despite the fact that the groups being compared are defined empirically. Groups of treatment-control matched pairs are identified using a special version of CART. A new form of matching, strength k matching, permits CART to search through many covariates for effect modifiers, yet no pairs are lost providing CART settles on a tree that uses at most k covariates. In a strength k match, we can build the CART tree using more than k variables, let CART decide which k or fewer variables are the best candidates as effect modifiers, and know that all individuals can be matched exactly for the variables CART selects. We apply the method to study the effects of the powerful earthquake that struck Chile in 2010.