Background: The Maternity Care Classification System is a novel system developed in Australia to classify models of maternity care based on their characteristics. It will enable large-scale evaluations of maternal and perinatal outcomes under different models of care independently of the model’s name.

Aim: To assess the accuracy, repeatability and reproducibility of the Maternity Care Classification System.

Method: All 70 public maternity services in New South Wales, Australia, were invited to classify three randomly allocated model case-studies using a web-based survey tool and repeat their classifications 4–6 weeks later. Accuracy of classifications was assessed against the correct values for the case-studies; repeatability (intra-rater reliability) was analysed by percent agreement and McNemar’s test between the same participants in both surveys; and reproducibility (inter-rater reliability) was assessed by percent agreement amongst raters of the same case-study combined with Krippendorff’s alpha coefficient for a subset of characteristics.

Results: The accuracy of the Maternity Care Classification System was high with 90.8% of responses correctly classified; was repeatable, with no statistically significant change in the responses between the two survey instances (mean agreement 91.5%, p > 0.05 for all but one variable); and was reproducible with a mean percent agreement across 9 characteristics of 83.6% and moderate to substantial agreement as assessed by a Krippendorff’s alpha coefficient of 0.4–0.8.

Conclusion: The results indicate the Maternity Care Classification System is a valid system for classifying models of care in Australia, and will enable the legitimate evaluation of outcomes by different models of care.