Article Text

PDF

Self assessment in continuous professional development: a valuable tool for individual physicians and scientific societies
  1. C Beyeler1,
  2. R Westkämper2,
  3. P M Villiger1,
  4. A Aeschlimann3
  1. 1Department of Rheumatology and Clinical Immunology/Allergology, University Hospital, 3010 Berne, Switzerland
  2. 2Institute for Medical Education, University, 3010 Berne, Switzerland
  3. 3Clinic of Rheumatology and Rehabilitation, RehaClinic, 5330 Zurzach, Switzerland
  1. Correspondence to:
    Dr C Beyeler
    christine.beyelerinsel.ch

Abstract

Objectives: To assess the appreciation of self assessment (SA) by multiple choice question (MCQ) tests during continuous professional development (CPD) meetings and to evaluate the attitude towards the option of recertification.

Methods: A questionnaire was completed at a meeting of the Swiss Societies of Rheumatology and Physical Medicine and Rehabilitation in 2003 (response rate 94.9% (223/235)).

Results: 60.9% of members found SA useful to assess rheumatological knowledge by MCQ tests; 71.3% thought it motivating to receive an anonymous feedback; 47.2% wanted an additional individualised feedback; 70.8% asked for SA to be continued every 2 years during CPD sessions; 26.3% favoured the option of recertification with identical standards to the Swiss certifying examination in rheumatology. Physicians in private practice less often chose the option of recertification than physicians employed by hospitals (OR = 2.09, 95% CI 1.03 to 4.18). No correlations between the type and duration of specialisation, personal reading time, frequency of attendance at CPD meetings, sex of the members, and the choices made were found.

Conclusions: SA by MCQ tests during CPD meetings is highly accepted by Swiss rheumatologists. This reliable, valid, and economic method of learning needs assessment enables individual physicians and scientific societies to plan educational with specific goals in mind.

  • CI, confidence interval
  • CPD, continuous professional development
  • MCQ, multiple choice question
  • OR, odds ratio
  • SA, self assessment
  • SGR, Swiss Society of Rheumatology
  • SGPMR, Swiss Society of Physical Medicine and Rehabilitation
  • continuous professional development
  • curriculum planning
  • multiple choice questions
  • recertification
  • self assessment

Statistics from Altmetric.com

Continuous professional development (CPD)1,2 is characterised by lifelong, self directed learning rather than supervised training and aims at maintaining and further developing a broad range of competencies for knowledge, skills, and attitudes. Learning is more likely to lead to a change in practice when needs assessment has been conducted.3 CPD planning should be based on a thorough and accurate assessment of needs, opportunities, prospective responses to internal and external forces, and an understanding of, and respect for, the learners’ interests.4 Tools for self assessment (SA) should be developed to help doctors identify their personal learning needs.5 There are various formal or informal and planned or opportunistic ways of identifying learning needs for individuals or groups. These include a clinician’s own experiences in direct patient care, interactions with the clinical team, formal approaches to quality management, specific activities directed at needs assessment including objective tests of knowledge and skills, peer review, and non-clinical activities.3

Since 1995 multiple choice question (MCQ) tests have been conducted during several annual scientific meetings of the Swiss Societies of Rheumatology (SGR) and Physical Medicine and Rehabilitation (SGPMR). The tests used 60 MCQs each covering all areas of rheumatology and were identical with the written part of the Swiss certifying examination in rheumatology. The questions of three types were developed according to a blueprint by a group of rheumatologists working in private practices and public hospitals. Participation was anonymous, free of charge and lasted for 1–2 hours. Feedback was given about the state of knowledge in comparison with peers for inflammatory, degenerative, extra-articular, and bone diseases as well as general issues. About 40–60% of the CPD meeting participants took part, with increasing numbers over the years, representing about 30% of the members of the societies.

Drawing from these experiences a similar quiz was presented during the international symposium of rheumatology, EULAR, Geneva 1998.6 The reliability (internal consistency) of this test was high and the validity in respect of the perceived relevance of the questions and the results of specific groups of participants was favourable.6

As time and money constraints have been increasing over the past few years the question of continuation of these tests has been raised. We therefore decided to conduct a survey about the attitudes of Swiss rheumatologists with the following aims:

  • To assess the appreciation of SA by MCQ tests

  • To evaluate the attitude towards the option of recertification

  • To analyse the results of the survey for influences by specific factors.

METHODS

At the beginning of the general meeting in January 2003 the presidents of the two societies SGR and SGPMR explained the aims of the survey and asked all members to fill in the anonymous questionnaire in the language of their choice. The questionnaires were collected by one author (CB) during the meeting with the consent of each person.

Statistical analyses included the calculations of frequencies, odds ratios (ORs), and 95% confidence intervals (CIs) using Epi Info 6 (Atlanta, Georgia, USA) and Statistical Analysis System (SAS Institute Inc, Cary, North Carolina, USA) under licence of the University of Berne. As a high proportion of cells in the frequency tables had expected counts less than 5, 2×2 tables were generated according to specifications given in the “Results” section.

RESULTS

Of 235 members attending the meeting, 223 returned the questionnaire (179 men, 41 women, 3 without specifications; response rate 94.9%). One hundred and seventy four questionnaires were completed in German, 49 in French. Of the participants of the survey, 180 were certified in rheumatology in combination with internal medicine (IM 117) or with physical medicine and rehabilitation (PMR 63), whereas 14 were still in postgraduate training for rheumatology. The remaining 29 physicians were specialised in PMR (20), IM (5), or general medicine (3), or gave no information (1). One hundred and nine had been certified for more than 10 years. One hundred and forty seven were working in private practice, 65 were employed by a hospital (34 non-teaching, 21 academic, 10 private hospital), and 11 gave no specifications.

Figure 1 shows the reading times and frequency of attendance at CPD meetings declared by the participants. Employed physicians more often read publications for >2 hours a week than physicians in private practice (OR = 3.29, 95% CI 1.56 to 6.90). Conversely, the frequency of visits to CPD meetings was similar in both groups (OR = 0.95, 95% CI 0.50 to 1.79). Physicians specialised for >10 years more often visited CPD meetings for ⩾7 times a year than colleagues with a specialisation for ⩽10 years (OR = 1.95, 95% CI 1.10 to 3.46). On the other hand, the likelihood of reading for >2 hours a week was similar in both groups (OR = 0.97, 95% CI 0.48 to 1.95). No significant differences between men and women were found.

Figure 1

 (A) Reading time; (B) frequency of attendance at CPD meetings.

Table 1 shows attitudes about concepts of CPD, SA by MCQ tests, and the option of recertification. Owing to an error in the French form, in which question number 3 was missing, the total number of answers is lower than expected for this particular question. However, subanalysis of the other five questions showed no language effect, making the introduction of a bias for question number 3 unlikely.

Table 1

 Attitudes of the members about CPD

Additional subanalyses for type of specialty, duration of specialisation, duration of reading time, and frequency of attendance at CPD meetings showed no differences. There was a trend for physicians in private practice to be more sceptical about SA and feedback by MCQ tests. However, physicians in private practice more often denied the option of a recertification with identical standards for the certifying examination (OR = 2.09, 95% CI 1.03 to 4.18). Physicians who rejected the offer of an individual, personalised feedback in addition to the anonymous participation in SA more often denied the option of recertification (OR = 7.95, 95% CI 4.26 to 24.13).

DISCUSSION

MCQ tests assess knowledge and, partly, problem solving in an objective and reliable way.7 The development of the questions is challenging and time consuming. On the other hand, evaluation of the tests and the questions is very quick if automated analysis is used. All the efforts put into the development of the compulsory MCQ tests for the written part of the certifying examinations can be used for the optional SA in CPD. We estimate extra costs of about €10 for each participant to print the question booklets and to analyse the answers. Owing to the high number of participants, MCQ testing becomes a very economical method of assessing the updating of knowledge in order to understand, diagnose, treat, and prevent diseases.

However, MCQ tests do not evaluate other competencies, including clinical reasoning and communication skills. Other, far more expensive methods, such as the objective structured clinical examinations8 or the standardised physician assessments in medical practice,9 are more suitable for assessing a wide range of competencies and skills necessary for the delivery of various professional tasks. Nevertheless, the results of MCQ tests are related to clinical competence, quality of practice, and patient outcomes.10,11

Analyses of the test results enable individual physicians and scientific societies to plan educational experiences towards a specific goal. There is no certainty that physicians who complete SA programmes will participate in education sessions that relate to their self assessed needs. But the fact that more than two thirds of our participants, including those who had completed their postgraduate training more than a decade ago, feel motivated by this kind of feedback gives rise to optimism that many of them will construct their lifelong, self directed learning around their SA results and their individual practices after completion of such tests. Moreover, organising committees of scientific sessions can put more emphasis on areas where a high proportion of participants show deficits in topics relevant to their practice. In addition, CPD planners can change their role from providing professional education and instruction to facilitating professional learning to enhance competence and performance wherever and whenever this may occur.4

The goals of recertification are to improve the care of patients, to set standards for practice of medicine, to encourage continued learning, and to reassure patients and the public that physicians remain competent throughout their areas.12 Our approach could be used as screening method13 and as basis of a credit system for monitoring and promoting excellence.14 When poor results are obtained, these MCQ tests could be followed by peer review of practice to assess performance in the work place and by tests of competence, including written and practical examinations, to assess knowledge and skills.13 Discussions of this kind have not yet been carried out within our societies.

Not surprisingly, only one quarter of the participants of this representative survey favour recertification. This illustrates the clear distinction between the formative and summative purpose of MCQ tests by our members. The majority favour feedback in order to improve their competence. In contrast, only a minority approves a final assessment, possibly because of fear of failing the examination and losing anonymity. The latter is supported by our finding that caution is more pronounced in members who reject individual, personalised feedback in addition to the anonymous participation in SA. Scepticism is less in employed physicians who are reading more according to their self declaration and work preferentially in secondary and tertiary referral centres. Similar attitudes can be expected in other European countries. Nevertheless, the implementation of voluntary periodic recertification could become standard policy in the future.15

Acknowledgments

We thank all members of the committee of the national certifying examination in rheumatology for the development of MCQ tests over the past 8 years. We thank all members of the SGR and SGPMR for their willingness to answer the questions and to add comments.

REFERENCES

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.