Article Text

PDF

Multiple choice question quiz: a valid test for needs assessment in CME in rheumatology and for self assessment

Abstract

Drawing from experience in Switzerland, a test of 60 multiple choice questions covering the entire area of rheumatology was constructed and used at the international symposium of rheumatology, EULAR, Geneva 1998. It was introduced as a multiple choice question game, the language was English, and the level of acceptance was very high. Language posed only occasional problems. The reliability of the test for internal consistency was high (Cronbach α 0.852). One of the main target groups, the private practitioners, was the largest group of rheumatologists (nearly 50%). Evaluative statements of the participants indicate that it was a highly relevant test for rheumatology. This test is a valuable way for needs assessment in continuing medical education and for self assessment. Importantly, it has been shown that such a test can be conducted at an international level.

  • continuing medical education
  • assessment

Statistics from Altmetric.com

Quality assurance in medicine is of growing importance world wide and, although restrained by limited resources, good, well directed continuing medical education (CME) forms a significant part of this.1 2 Typical keywords are evidence based medicine, managed care, etc. The European Board of Rheumatology and The European League Against Rheumatism (EULAR) are working towards progression of this quality assurance by providing high quality educational activities, and have set standards for undergraduate education3 and specialist training in Europe.4 Standards for CME in Europe are now being developed, including a core curriculum. Quality assurance also requires methods to ensure maintenance of the quality of care delivered, and quality CME requires methods to assess educational need. These goals can be achieved by audit of outcome of clinical care, formal assessment of competence by examination, or by visitation and peer review. In some countries recertification is compulsory and this is being considered by others. There are therefore compelling reasons to develop suitable methods of evaluation that can be used for personal professional development and for reassuring others about the quality of care delivered. Switzerland has already implemented a system of quality assurance and self assessment. Developing methods of assessment that can be used throughout Europe is the challenge, and the EULAR symposium in Geneva 1998 provided an occasion to evaluate the approach used in Switzerland.

In the past few years a multiple choice question (MCQ) test on rheumatology has been conducted in Switzerland during the annual rheumatology congress which dealt with the issue of CME.5 6 The test used 60 MCQs covering all areas of rheumatology, with a level of difficulty that corresponds to the certifying examination for rheumatologists.

The results provided valuable information about the state of knowledge and the need for continuing education. Participant's acceptance of these tests, which lasted 1–2 hours, was high at each trial. About 40–60% of the congress attendants usually took part, representing approximately 30% of the society's membership. Interestingly, the test became a popular form of self evaluation for many.7

The Swiss experience and examination was developed into a quiz during the EULAR symposium 1998 in Geneva to consider the following questions:

  • What do we know about the knowledge of participants in an international congress?

  • Are the participants willing to take part in a test?

  • Is it possible to create, realise, and evaluate such a test?

  • What can we learn from the state of knowledge of the different groups—for example, participants from an academic setting or a non-teaching hospital?

  • Which subjects are answered controversially?

  • Do the participants benefit from self evaluation?

The EULAR Standing Committee for Education and Training and the EULAR 1998 Symposium Organising Committee charged this paper's authors with the development and implementation and realisation of the test that was included in the congress programme. The results and experiences are reported here.

Methods

The test covered the whole spectrum of rheumatology from anatomy to treatment as detailed in the European Board of Rheumatology core curriculum.4 It comprised 60 MCQs of three types—multiple choice with one correct answer; matching pictures to statements; or multiple choice where each statement has to be marked correct or incorrect (examples are given in the ).

English was chosen as the common language. To promote a high acceptance, the test was announced in the symposium programme as “Multiple choice question game” and prizes were offered.

Results

CONGRESS PARTICIPANTS

A total of 274 congress participants took part in the test, corresponding to about 25% of the day congress participants (the exact number was not available). Participants represented 35 nations, with 78 (28%) coming from Switzerland, the host country. Of the 274 participants 251 completed questions relating to work demographics. Of these 251, 88 (35%) were women; approximately three quarters worked as rheumatologists (table 1); Currently 13.6% were still in training and 27.6% were working in an academic setting, 27.6% were in a non-teaching hospital, 27.6% were in a private practice, and 3.6% in another field.

Table 1

Participants at multiple choice question game EULAR 98

The number of years of practical experience of the participants was as follows: 0–4 years 28%, 5–9 years 23%, 10 or more years 49%.

RELEVANCE AND VALIDITY OF THE TEST

The authors endeavoured to develop a test relevant to the daily practical work in a rheumatology clinic or in private practice. In the evaluation section, when asked: “How relevant are the questions in total for your work?”, 68.7% considered the questions as “highly relevant” (18%) or “relevant”(51%), 31.3% of the participants considered the questions as “moderate” (29%), “more or less” (2%) or “less” (0.4%) relevant. Such a response suggests that the effort to create a relevant test was successful. The single participant who considered the questions as “almost irrelevant” also scored poorly on the test.

There has been debate about the suitable language for such an examination and we were especially concerned whether the chosen language would pose a significant problem and threaten the entire project. Although only 9% of the participants were of English mother tongue, 37% of the responding participants stated they had no language problems and 36% had hardly any problems. Twenty one per cent had some problems, and 6% definitely had problems in understanding the questions. Although the group with no language problems had the best results, it should be noted that the group with language difficulties did not have the worst scores, and scored at the same level for both complex and simply worded questions.

The most commonly used individual form of CME is reading specialised literature and 108 (43%) stated that they devoted 1–2 hours a week to this activity with 14% stating that they only read rheumatological publications for up to two hours a month. Those devoting 1–2 hours a week achieved the best results in the test (table2).

Table 2

Reading time and test performance

Although the test was of the level of a certifying specialist exam, and covered the whole field of rheumatology, many questions should also be solvable by trainees or non-rheumatologists. The mean score of the best was 57.5% correctly solved questions. The rheumatologists achieved 60.4% and the non-rheumatologists 52.2% (significant difference, t test, p<0.0001), the specialists of internal medicine and physical medicine solved on average 55.5% correctly, significantly less than the rheumatologists (t test, p<0.02). When asked “How difficult was the MCQ game?”, the responses were in keeping with the scores attained (table 3) and for most people the level of difficulty of the test was appropriate. The respective means (percentage) of the different “difficulty groups” show, as expected, significant differences of the mean score achieved.

Table 3

How difficult was the test and performance?

Discussion

The application of this examination has given insight into the feasibility of developing methods of assessment that will be applicable across Europe. One potential caveat is whether the test scores were strongly influenced by the language. About half of the questions were deliberately created to be independent of language (examples are given in the appendix: questions 3–7); these consisted of technical terms and graphics or pictures. Looking at the “language independent” questions, one sees that the results are rarely influenced negatively by language problems. The reason why there was a correlation between language knowledge and test score may be related to the reading of specialty publications, which in most cases are in English. That is, those who devote the most time to their continuing education by reading regularly in English have a better knowledge of rheumatology, and of the English language.

The majority of the participants were rheumatologists who had been working for more than 10 years in their own practices. It is encouraging that the main target group—the freely practising rheumatologists—represented the largest group. A surprisingly high number of elderly colleagues—many of them having undergone a test in this specific form for the first time—spontaneously stated that they approved it. The fact of participation as well as the test scores of this group indicate a need for continuing education.8 9The trainees and practitioners with more than 10 years' experience had almost the same scores. The highest results were achieved by the participants who had 5–9 years' experience.

An important question is to know how good are the validity and reliability of the test.10 There are different statistical methods to prove the reliability of a test (for example, retest, split halves test, internal consistency test). We decided to calculate an often used coefficient of internal consistency, developed by Cronbach,11 called Cronbach α. It varies between 0 and 1 and values >0.8 indicate a very reliable test. The Cronbach α of the EULAR quiz was 0.852.

This appears to have been a valid test from the above described responses to the relevance as well as the results of the specific groups, as the tables and the example questions show. This is important because examinations for both certification and recertification are discussed and established.12-15 On the other hand, the analysis of several pertinent questions enables the identification of a need for CME in certain subjects. In conclusion, we feel that this test represents a valuable instrument for needs assessment and for self evaluation that can be applied in a large, international setting.

Acknowledgments

We gratefully acknowledge the support from Professor T Vischer, Division of Rheumatology, University of Geneva, Geneva.

Examples of the MCQ quiz (questions 1, 2, 3–7, and 8)

QUESTION 1

The bisphosphonate, alendronate, has clearly been established to be effective in randomised controlled trials in which of the following conditions?

(A)  Osteomalacia

(B)  Bony metastases

(C)  Algodystrophy

(D)  Postmenopausal osteoporosis

(E)  Hypercalcuria

The first item asks about the treatment of postmenopausal osteoporosis. It is encouraging that 93% knew the correct answer (D); the rest of the participants chose one of the other answers (1–2% each).

QUESTION 2

Allopurinol is used in the management of gout. Which statement below is correct?

(A)  Allopurinol is effective by increasing glomerular filtration of uric acid.

(B)  Allopurinol is effective by reducing the tubular reabsorption of uric acid.

(C)  Allopurinol is effective by inhibiting the conversion of xanthine to uric acid.

(D)  Allopurinol is indicated in a 37 year old man with a uric acid concentration of 0.42 mg/l and a history of one attack of gout.

(E)  Allopurinol is associated with aplastic anaemia in 1% of cases.

The second question about allopurinol was also well answered. It shows that important new discoveries are adopted in practice. (Correct answer (C) 88.3%, other answers (A) 4%, (B) 3.6%, (D) 0.4%, (E) 2.6%, no answer 1.1%).

QUESTIONS 3–7

Match the picture to the most relevant diagnosis.

Figure 1 shows (A)–(D); (E) is not illustrated.

Which picture shows

 Questions 3–7 are not dependent on language knowledge, but the answers require a certain clinical experience. The following list shows the percentage of answers given to each question; the correct answer is given in bold print.

 These questions provide a good example of the better knowledge of the rheumatologists, who apparently have more experience: Questions 5 and 6 were by 20–30% better answered by rheumatologists than by the internists.

QUESTION 8

A 64 year old man with a 15 year history of seropositive polyarticular rheumatoid arthritis is currently treated with prednisolone 5 mg daily and methotrexate 12.5 mg weekly. He is diabetic and treated with oral hypoglycaemic drugs. For the past three weeks his right knee has become progressively painful. Examination shows warm synovitis and obvious knee effusion in the right knee with stress pain. The left knee shows no synovitis. The other affected joints in his hands, wrists, elbows, and forefeet show some minor abnormalities but no synovitis.

Which of the following would you do at that clinic visit?

(A)  Admit the patient straight to the ward.

(B)  Aspirate and inject the right knee with steroid.

(C)  Take blood for C reactive protein, full blood count, erythrocyte sedimentation rate, and await results until the next clinic appointment.

(D)  Increase methotrexate to 15 mg weekly and offer the patient a steroid injection into the knee.

(E)  Undertake an x ray examination of the knee.

A difference of opinion was shown by question 8. (A) was defined as the correct answer, because an infection of the knee was the most probable cause of the knee effusion, requiring most probably hospital based treatment. This answer was chosen by 36.5%, (B) by 27%, (C) by 13.9%, (D) by 9.1%, (E) by 11.3%, and no answer was given by 2.2% of the participants. The appropriate action with such a complex problem was not so easily agreed as it might be that there are different ways across members of the rheumatology community (for example, hospital or practice based rheumatologists) or across Europe of managing this problem. It is notable that 45.5% of the specialists in rehabilitation chose the “correct” answer (A), (B) 18.2%, (C) 22.7%, and one of the remaining answers 4.5% each. The internists chose (A) 39.1%, (B) 26.1%, (C) 8.7%, (D) 17.4%, and (E) 8.7%. But not even the rheumatologists had clear preferences: (A) 37.8%, (B) 28.1%, (C) 11.9%, (D) 8.1%, (E) 11.9%, and no answer 1.2%.

Unfortunately, it was not possible to show whether there were national specific differences in answers. However, answer (A) was given by 68% of the specialists working in an academic setting and by 56% of the practice based rheumatologists.

References

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.