In the clinical routine of chronic diseases, a new measurement has to be compared to an established criterion to see whether there is agreement on assessing disease progress. Because of the nonexistence of established criteria for rheumatoid arthritis the evaluation of grading systems for radiological assessment needs an alternative approach. Using the measurement error as a measure for repeatability and rater agreement in comparison to the progress of the grading score might be a solution to this dilemma.