Results

Table 2 shows the main characteristics of the 201 patients. The severity of the shoulder problem was greater in patients recruited by rheumatologists than in patients recruited in other settings, expressed by more frequent sleep disturbances and pain during rest, and a lower functional status (higher SDQ-score).

Table 3 presents the diagnostic classification of shoulder disorders according to both examiners. The percentage of agreement was 60%, and the overall κ was 0.45 (95% CI 0.37, 0.54). In cases of disagreement (81 patients; 40%), the examiners frequently disagreed on whether the shoulder pain should be classified as a distinct category or a mixed clinical picture (42 patients; 21%; dichotomous κ = 0.14).

Univariate analyses showed that most indicators of high severity of the complaints were associated with disagreement. Multivariate logistic regression analysis revealed that bilateral involvement, long disease duration (> 6 months), and high pain severity (mean pain score according to both examiners > 7.2) were independently associated with diagnostic disagreement (table 4). The model fitted the data well (goodness of fit test: p = 0.46) and the discrimination was acceptable (area under the ROC curve: 0.68).

Discussion

In this study, the interobserver agreement of the diagnostic classification of shoulder disorders was evaluated in a large number of consecutive patients (n=201) with varying severity and duration of complaints. Only a moderate interobserver agreement was found (percentage of agreement = 60%, κ = 0.45). Especially for patients with high pain severity, chronic shoulder complaints and bilateral involvement, it seems to be difficult to define the anatomical site of the lesion.

The patients for this study were enrolled in three different health care settings. An association between type of health care setting and diagnostic disagreement might be present when patients are more likely to be referred by general practitioners to other health care settings when shoulder disorders are difficult to diagnose. Before the study we expected that the disagreement would vary between the setting of the general practitioners (GPs) and the other two settings. However, the level of disagreement was similar for the patients recruited by the GPs (37.3% disagreement) and the clinic for rheumatology and rehabilitation (39.8%), whereas for the patients (n=33) recruited in the orthopaedic practice a higher level of diagnostic disagreement was found (54.5%). We have no clear explanation for this finding. The results of the multivariate logistic regression showed the same pattern when the patients of the orthopaedic practice were excluded from the analysis (data not shown).

In interobserver agreement studies varying experience of examiners might influence the level of reproducibility.27 In this study the examiners had a different level of clinical experience and this could have influenced the results. To lessen this influence, before the study, the examiners had already achieved theoretical consensus and the diagnostic procedure had been standardised. In routine daily practice less attention will be paid to standardisation of the physical examination. Therefore, it might be argued that, on average, the level of reproducibility in clinical practice would not reach the level in this study.

One earlier study also evaluated interobserver agreement on the basis of the diagnostic classification recommended in the guidelines of the Dutch College of General Practitioners.14 In this study, which compared the diagnosis of general practitioners with the diagnosis of physiotherapists in routine daily practice, frequent discrepancies were found (κ = 0.32). This was a disappointing result, considering the fact that the physiotherapists were not blinded for the diagnoses of the GPs.

Two other interobserver studies based on a slightly different diagnostic classification show varying results. In the study carried out by Bamji et al,15 three consultant rheumatologists agreed on the diagnosis of shoulder disorders in less than 50% of the cases involved. In contrast, high agreement (κ = 0.88) between experienced physical therapists who examined 19 patients was found by Pellecchia et al.16 However, the clinical characteristics of their study population were not presented, so it is unclear whether the study population consisted of consecutive patients or a selected group of patients. Therefore, it is difficult to establish an explanation for the high level of reproducibility found in their study.

The unsatisfactory reproducibility reported in the various studies might be explained by the fact that the diagnostic categories are insufficiently mutually exclusive. If clinical findings are not clearly attributable to one single diagnostic category, clinicians have to decide which clinical findings are most prominent to differentiate between shoulder disorders. In this study, the examiners frequently had difficulties in classifying the shoulder disorders into distinct categories, given the number of cases classified as “mixed” or “unclear clinical picture”. Based on the same diagnostic classification, Sobel and Winters28 showed that with strict application of the criteria only 3% of the cases could be distinctly classified, whereas with less stringent application of the criteria and additional tests 50% of the cases could be classified into distinct categories.

Insufficient mutual exclusiveness might also explain why patients with high pain severity, chronic complaints, and bilateral involvement represent a diagnostic challenge. These patients probably meet the diagnostic criteria for more than one category. This is understandable, because for patients with severe pain many of the test results will be positive, which makes it difficult to assess the relation between local factors and complaints. The complexity will also increase if various shoulder disorders, or a combination of a shoulder disorders with extrinsic conditions, underlie the shoulder complaints. This increased complexity might explain why it is more difficult to classify patients with bilateral involvement and chronic complaints. Moreover, for chronic complaints it has been suggested that local factors might determine the initial location of complaints, but that reasons for persistence and recurrence may be more general, such as previous episodes and psychosocial factors.29

What efforts should be made to improve the reproducibility of diagnostic classification of shoulder disorders? It has been postulated that diagnostic injections with a local anaesthetic30 or additional tests during physical examination, such as the Neer impingment test,7 ,28 are helpful in establishing a diagnosis. If those diagnostic procedures do offer a solution is unclear. Additional skills will be required to perform certain diagnostic procedures, and some procedures might result in an increase in patient discomfort. It can also be questioned whether imaging techniques, such as ultrasound and magnetic resonance imaging (MRI), are beneficial in the selection of a diagnosis. Ultrasound might be an important diagnostic procedure because it is non-invasive and the costs are low.31 Recently, a meta analysis was conducted to assess the accuracy and reliability of ultrasound for shoulder disorders.32 After evaluation of 58 studies the authors concluded that the accuracy of ultrasound was acceptable and that the reliability of ultrasound is unknown. However, often the accuracy of ultrasound was assessed for the detection of partial or complete tears of the rotator cuff. Consequently, it remains unclear what the accuracy of ultrasound will be in a population with varying severity of shoulder complaints. Although in the medical literature MRI is considered to be a useful diagnostic procedure in the evaluation of shoulder pain,33-35 this procedure is time consuming and would increase health care costs.

It has been suggested that the exact localisation of the anatomical site of the lesion is a prerequisite for effective treatment.5 ,12 ,13 In most randomised clinical trials, the main selection criterion for patients is the diagnosis, based on history taking and physical examination.5 It can be questioned whether unsatisfactory treatment outcome in some patients is because of the difficulties involved in localising the lesion. Therefore, future research should demonstrate whether additional diagnostic procedures could increase reproducibility, and thereby also improve the outcome of treatment.

A less complicated diagnostic classification system is also proposed, to reduce the complexity of diagnosing shoulder disorders.36 In our study the reproducibility was assessed for a diagnostic classification that was based on the concepts of Cyriax. Other diagnostic classifications can be based on different diagnostic criteria and have a different reproducibility. Unfortunately, there are no studies that report on the reproducibility of the various clinical findings that underlie the diagnostic classification of shoulder disorders. It has, however, been shown that various diagnostic labels are applied, even when there is consensus on the clinical findings.15 More insight into the reproducibility of clinical findings and careful examination of the diagnostic criteria is obviously needed before new classification systems can be adopted.

In conclusion, distinguishing between distinct shoulder disorders on the basis of history taking and physical examination seems to be rather complicated. Especially patients with high pain severity, bilateral involvement, and chronic complaints represent a diagnostic challenge. Serious doubt about the reproducibility of the diagnostic classification of shoulder disorders raises the question whether diagnosis based on history taking and physical examination is actually beneficial in the choice of treatment. Future studies should therefore determine whether additional diagnostic procedures improve diagnostic agreement. Moreover, additional research is needed to investigate the sources of diagnostic disagreement attributable to interobserver differences in clinical findings. This might be helpful in reaching further consensus on the appropriate diagnostic criteria.