Protected areas currently cover about 15% of the global land area, and constitute one of the main tools in biodiversity conservation. Quantifying their effectiveness at protecting species from local decline or extinction involves comparing protected with counterfactual unprotected sites representing "what would have happened to protected sites had they not been protected". Most studies are based on pairwise comparisons, using neighbour sites to protected areas as counterfactuals, but this choice is often subjective and may be prone to biases. An alternative is to use large-scale biodiversity monitoring datasets, whereby the effect of protected areas is analysed statistically by controlling for landscape differences between protected and unprotected sites, allowing a more targeted and clearly defined measure of the protected areas effect. Here we use the North American Breeding Bird Survey dataset as a case study to investigate the effectiveness of protected areas at conserving bird assemblages. We analysed the effect of protected areas on species richness, on assemblage-level abundance, and on the abundance of individual species by modelling how these metrics relate to the proportion of each site that is protected, while controlling for local habitat, altitude, productivity and for spatial autocorrelation. At the assemblage level, we found almost no relationship between protection and species richness or overall abundance. At the species level, we found that forest species are present in significantly higher abundances within protected forest sites, compared with unprotected forests, with the opposite effect for species that favour open habitats. Hence, even though protected forest assemblages are not richer than those of unprotected forests, they are more typical of this habitat. We also found some evidence that species that avoid human activities tend to be favoured by protection, but found no such effect for regionally declining species. Our results highlight the complexity of assessing protected areas effecti veness, and the necessity of clearly defining the metrics of effectiveness and the controls used in such assessments.