BACKGROUND: The aim of this study was to discuss, in general, selected crucial aspects of the appropriate methodology for the development and validation of scales indicating health status, and to illustrate this empirically by within-material comparison of the accuracy of four different scale constructs based on identical raw datMETHODS: The empirical example was based on information from the parents of 99 refugee children, aged 3-15 years, from the Middle East, who participated in a structured interview on their children's mental health. Following this, they were exposed to a blinded semi-structured psychological interview. Four anxiety scales were constructed based on answers to 12 anxiety symptom questions in the structured interview: Scale 1, by cumulation of original item scores, each ranging from 0 to 3; Scale 2, by counting the number of symptoms being present; Scale 3, by counting the number of frequent or intense anxiety symptoms; Scale 4, by estimation of the multivariate probability of the child being anxious, as assessed by the psychological interview. The scales were compared for their accuracy in the identification of children assessed as anxious by psychological interview. RESULTS: The four scales correlated mutually, and each of them was significantly associated with anxiety, as assessed by psychological interview. The weighted scale, however, performed significantly better than the unweighted scales for sensitivity but not for specificity. In the present data set the overall amount of misclassification was, however, significantly less than in the unweighted scales. CONCLUSION: As expected from theory, the weighted scale was found to be superior to the unweighted scales, in identifying the anxious children of the empirical example. In the presence of a blinded criterion measurement, empirical regression-based weighting of scale items thus constitutes an accessible and valid alternative to traditional methods of health and social scaling.