Environmental filtering, where the environment selects against certain species, is thought to be a major mechanism structuring communities. However, recent criticisms cast doubt on our ability to accurately infer filtering because competition can give rise to patterns identical to those caused by environmental filtering. While experiments can distinguish mechanisms, observational patterns are especially problematic. The environment determines community composition not only directly via survival, but also by influencing competition. If species population growth rates covary with environmental gradients, then outcomes of competitive exclusion will also vary with the environment. Here, we argue that observational studies remain valuable, but inferences about the importance of the environment cannot rely on compositional data alone, and that species abundances, population growth, or traits must be correlated with the environment.