Introduction

Revalidation is a mechanism by which a practitioner providing the care may be seen to be up to date and fit to practise.1,2,3 The background to revalidation has been explored more fully in a previous paper.4 Monitoring quality and performance are accepted by staff and consumers as facets of quality assurance in many walks of life.5 This implies that outcomes of assessments of quality standards should be kept for review and inspection. The clinical governance process produces improvements in patient care by reflective practice using these records. Systems and processes for monitoring and improving services exist already,6 but in future an externally verified system is likely to be necessary to retain public confidence. Increasingly professional self-regulation is seen to be more than minimum levels of public protection and is expected to play a major role in the improvement of standards of patient care. Not only must health professionals strive to improve the quality of care, they must also be able to show that they are doing so.7

The aims of this pilot project were to explore whether a revalidation scheme would be acceptable to the profession and be robust enough to show that dentists who had completed the scheme were up to date and fit to practise.

Method

Practitioners were recruited from Vocational Trainers in South East Scotland. Each dentist was provided with a portfolio at an introductory course. A fuller description of methods is given in the first paper.4

An action research methodology was used to evaluate participants' perceptions of revalidation — involving the collection, organisation and interpretation of valid textual material derived from discussion. This consisted of two key components:

  • A focus group of dentists in the study. Analysis of the data gathered informed the semi-structured interview. Focus group data were returned to each of the dentists and comments sought at the subsequent interview, to validate the data. Six of the 10 dentists attended this focus group.

  • A semi-structured interview with each of the dentists explored their use of the portfolio

  • Data analysis was achieved by identifying the main themes from both the focus group and the semi-structured interviews. This was done using an editing style. These themes were informed and refined by revisiting the data from both sources as part of the immersion process.

Results

What did the participants think about revalidation and the use of a portfolio?

Revalidation was described as:

  • The need to satisfy an external body, via continuing professional development (CPD) and clinical governance that dentists remain up to date

  • A licence to practise

  • Meeting external standards in order to continue to be a dentist

  • Being able to demonstrate clinical skills of a suitable standard, taking into account the time since graduation

  • A way of ensuring dentists attend courses and maintain quality of practice

  • A strategy by which the profession avoids damage and can 'get rid of' poorly performing dentists

  • A way of involving patients

  • About dentists keeping their registration.

Revalidation was seen as better than recertification, although not all the dentists thought that this portfolio was the best way to evidence practice. There was a concern that an individual could be good at form filling but not good as a dentist. One dentist felt that the portfolio had been a bad use of taxpayers' money and that money would be better spent by peers observing dentists and then having direct referral to Dental Reference Officers (DROs).

Would this portfolio be acceptable in professional life?

Some of the group found it acceptable while others found it onerous and thought their peers would also. As a group they felt it would be acceptable to some dentists but not to others. Dentists who were Vocational Trainers would find it easy to use, while those closer to retirement might find using it more challenging. Young practitioners who had been trained using such methods 'wouldn't think twice about it'. Some of the challenge for the older dentist might be in using learning plans and significant event analysis.

The key to making it easier to use was seen as having timely reminders sent to each dentist, as well as some form of financial remuneration for all the time spent in collection of data. It was noted that an average general dental practitioner (GDP) would need some sort of incentive to complete their portfolio. The pilot group had some significant concerns about the use of the portfolio in revalidation. Much of this related to the recording of activity and the process of review:

'Revalidation (using a portfolio) doesn't prove you are a good dentist or a safe dentist, it proves you can fill in a book.'

Revalidation was seen as a way of the profession maintaining control and it was felt that this would bring practices up to a standard. However this view was not held by the entire pilot group. Reservations over the process of producing a portfolio were linked to concern over the nature of its assessment.

Who should be reviewing the portfolio?

For most of the dentists it was essential to have respected peers reviewing a portfolio. The challenge was seen as being able to make it 'even handed'. Concerns were expressed about the confidentiality of the review and lay involvement. For some, patient involvement was seen as important, while for others it was important to have someone involved in dental education. Some felt academic dentists were not close enough to the world of business practice to be able to contribute. Most of the group did not think that knowing the membership of the assessing panel would influence the way they constructed their portfolio.

Over what period should the revalidation take place?

The challenges of having a revalidation period of a year were mostly countered by suggestions that a three year or five year interval would fit better into re-certification, vocational training assessments, audit and research cycles.

Having expressed concerns about the principle of using a portfolio for revalidation, the dentists then provided valuable feedback on their experience of using this particular portfolio design.

If a portfolio system is used how can it be introduced and supported?

It was perceived that the portfolio had been enthusiastically introduced to this pilot group. One dentist had not seen the portfolio as threatening but had thought it was better not to introduce it 'cold' to other dentists. After the introduction of the portfolio most of the group felt further support was unnecessary, but all knew that support was available. For others an additional meeting about six weeks after the introductory workshop would have been valued. As the group became more familiar with the layout of the portfolio over time, and with use, they felt they used it in a more efficient way.

What were the valuable aspects of using a portfolio?

The following were seen as important for a successful portfolio:

  • Having a good layout

  • Helping to examine processes in working life and having individual strong points highlighted

  • Feeling valuable to the dentists who use it

  • Pulling together information about what the dentist does

  • Providing opportunities to be part of a learning cycle

  • Helping the individual to become aware of their career path

  • Helping to improve patient care.

As an administrative task

Some of the group felt that the use of a portfolio for revalidation was an easy process and that the layout was accessible. Only one dentist did not delegate any of this work. The key person to whom completion of the portfolio was delegated was the practice manager, although some used other staff.

On the reflective component

Several dentists liked the reflective elements of the portfolio. However, it was seen as something that is already done in practice, but not always recorded. For others there was a tension in reflection and its recording: 'It's a chore to get pieces of paper from a range of sources, but the bits that are recording the thinking are more valuable.'

On planning learning

It was felt that the portfolio had a sense of a future with it, ie a plan for the future. For two dentists the portfolio had been a way of formalising where they might want to go and what they wanted to do.

What were the difficult areas of using a portfolio?

The difficult areas in using a portfolio were:

  • Having the time

  • Having the discipline to fill things in

  • Feeling like a chore

  • Feeling under pressure to complete it

  • Gathering information, which had irritated some dentists and their staff

  • Completing patient satisfaction questionnaires, which for some gave no additional information

  • Using the reflective practice component.

On the practical use of a portfolio

The focus group did not know what another body would be able to deduce from a portfolio:

'If this (portfolio) is going to another organisation to look at how we are doing, then the people who do well are the ones who are good at filling in forms.'

Generally there was a sense that the culture of general dental practice was one of activists who do not enjoy paperwork:

'general dental practitioners (GDPs) are not people who enjoy filling in forms we are people who do things. Give us a problem and we will solve it'.

While the portfolio was new, it was felt that it did not ask anything that each individual dentist in this group was not already asking themselves. The difference was in the ways that it was recorded. There was a sense that if individuals were already doing these things in practice then the portfolio became a paper exercise.

Accessing documentation

Practice staff had key roles in accessing documentation. It was perceived there were a lot of things to collect which it was felt were time-consuming and unnecessary:

'The certificates have already been seen this year, everything is in the practice. This means it is just moving one document from one place to another place.'

Feedback and reflection

The use of patient questionnaires was new for some of the group. For others the data were not of value or they would have benefited from having feedback earlier as well as feedback about how they ranked alongside their peers.

Although for some the portfolio had provided an opportunity to reflect on performance, others felt it was nothing new.

Some people preferred a structured format, while others preferred to have something to complete in their own way. Hence flexibility in design would be valuable.

Management of documents

  • It was felt by some that significant event analysis and personal learning plans were valuable but everything else required in the current revalidation portfolio could be given to another member of staff to collect or collate.

  • It would be valuable if all documentation held centrally was made available to each individual dentist.

Layout and design

In general the layout of this portfolio was satisfactory, clear and easy to use. The only deficiency was a lack of space to add written text. Polypockets were useful to hold critical incident analysis and to put in reports and the patient satisfaction survey.

How could the portfolio work more effectively for other dentists?

  • The revalidation pilot was acceptable because dentists were paid to do it; however, it was felt they would feel aggrieved if they had to spend hours completing it without remuneration

  • There was a recognition that the portfolio could fit in with appraisal in the practice and awareness that such schemes need support

  • Quality of feedback that a dentist receives currently is not very good and not done by people who work in the system

  • The use of observation of practice, without pre-warning. DRO inspection is currently given two months' warning. It was acknowledged that while the trainers are used to being observed, the rest of the profession is not used to this

  • Increased DRO inspections. It was suggested that these might monitor performance. Concern was expressed that DRO visits will meet the patient but not see the practice running, so cannot monitor dentists' performance. It was suggested that clinical work should be assessed more regularly.

What other ways might competence be evidenced?

  • Videoing of dentist practising in their surgery

  • Peer review.

    Verification of clinical practice could be achieved through a process of reviewing clinical activity. Reviewers would need to be GDPs and respected from within the profession.

Assessors' calibration exercise

All assessors demonstrated a uniform interpretation of the concept of revalidation and agreed that the one hour set aside for the assessment was sufficient to complete it. They expressed difficulties in assessing the 'reflection on practice' section of the CPD domain. One of the assessors was unable to use the part of the assessment tool relating to personal learning plans (PLPs) and another found difficulty understanding what to do in this section. The non-dental assessor suggested redesigning the CPD section into three parts: one for CPD personal to the dentist, a second for CPD relating to the practice/teamwork and a final section for special interests and linking the whole CPD section to PLP development.

Assessors' comments on portfolio assessment

Greater clarity in identifying the individual dentist's patient lists from the practice lists was required. Complaints reported should relate to the dentist concerned and not those generic to the practice (although these might be included alongside dentist specific ones). Some of the audits presented were not on the portfolio dentist's own work or care, but were Vocational Training (VT) projects that had been supervised by them. Audits presented for revalidation purposes should be on the dentist's own work and should at least have some input from the dentist concerned. The critical incident section attracted almost universally good comments and seemed to have been well done by all the dentists. The comments on the CPD section ranged from 'Exemplary CPD record' to 'Nil none recorded'. Similarly the reflection on practice section attracted a range of comments from 'Enthusiasm for research', to 'None'. The PLP comments again ranged from 'Excellent completion' to 'no evidence of reflection'.

The overall comments give a good sense of the spread of quality of portfolio completion. These varied from an almost ideally completed one: 'Feel sense of concerned and studied practitioner, who has used clinical experiences to change practice. Patient focused in detailing 'whole team quality of care' to 'Incomplete portfolio — not enough to satisfy basic requirement'. Inbetween there was a spread from good, but less ideally documented ones, to the other portfolio which had to be entered into the remedial process. The deficiencies in the two non-revalidating portfolios were minor and were essentially due to omission of documents and failure to complete reflective elements of pro-formas. Supplementary submissions were requested and submitted quickly by the relevant dentists. In both cases these submissions were sufficient to satisfy the assessors.

Assessors' post-assessment questionnaire

The participants had provided adequate information, but extra information would have been helpful, such as a pro-forma sheet guiding the dentist through reflection upon the dental practice division (DPD) practitioner prescribing profile. All felt that some sections could benefit from restructuring. The non dental assessor suggested making the 'patient assessment' section part of a separate 'Public involvement' domain, to include for example, 'focus group' results. All commented upon lack of uniformity of approach to audit projects. A view was expressed that a tick-box approach to those verifiable CPD types that are or may become obligatory would make assessment easier. An obvious extra section or sub section would be for VT/GPT experience as a trainee and experience of the same as a trainer. The point that CPD and PLP sections should be linked was made again by all the assessors.

The general quality of evidence submitted was 'adequate'. The non-dental assessor again emphasised the limited reflection recorded on the evidence presented. The assessors all felt that there was little to be gained from being able to interview the dentist. However, they all felt that it would be useful for the dentist to have had some mentored reflection upon their evidence supporting revalidation. They suggested that this might be provided in the form of appraisal. The assessors all agreed, after discussion on the morning of the assessment day, that they felt adequately prepared to proceed. The non-dental assessor stated a specific need for help in interpreting technical DRO reports for a layperson.

Reservations expressed by the dental members included: 'Is this to be a hard or soft exercise?'

'Hard: To restrict and deny the right to work and employment has such major consequences that the revalidation decision process needs to be clearly defined. It will be necessary to check external references.

Soft: The revalidation portfolio should be the basis for an appraisal interview. In this scenario the present format is more positive. I would suggest a rearrangement so that each section prominently started with the personal reflection on the included data and evidence. Each appraisal record would, after year 1, reference previously agreed targets for personal and practice development. This would be an opportunity to show positive action before the end of the revalidation cycle. “Remedial” GDPs could work with postgraduate support and within a peer review setting.'

Discussion

Consideration has already been given to the small number of dentists involved in the pilot.4 However, this did allow use of the qualitative methods described and reported. To reproduce this on a larger scale would require significant extra research worker manpower and a consequent significant investment of resource.

Most of the difficulties in validating the use of the portfolio arose in the CPD section. This reflects the uncertainty of relating how CPD activity can be used to manage and shape practice. These difficulties were resolved in the training session before commencing the assessments. It was significant that all the assessors had had difficulties with the same sections. All had difficulties in assessing the 'reflection on practice' section of the CPD domain. This may reflect the lack of structure to this section and the uncertainty of the users — formally recording this activity is a new sphere for dentists. It should be emphasised that audits need to be undertaken by the practitioner themselves and include at least some of their own care or patients. They should also be of better quality. These issues are currently being addressed by the requirement to carry out a minimum of 15 hours of clinical audit over a three year cycle (introduced in April 2002).8 As well as this they are supported by a prior approval system with a network of regional assessment committees to assure probity and maintain standards.

If revalidation is to be a 'test' to permit continuation of practice, then clearly defined standards free of variable interpretation will be required in order to be able to defend the inevitable legal challenge when someone fails. In this case it could be argued that the work carried out on the Competencies for Dental Vocational Training and General Professional Training in Scotland8 represents only the beginning of the work required not only to create such a framework of standards, but it would also require ongoing work to maintain its validity, sensitivity and specificity.

If the data contained in the portfolio are used as the basis for an appraisal interview, the dentist could reflect upon their practice and demonstrate development to address quality issues. This could form the basis of future appraisal discussions and be included as evidence of keeping up to date.

The main areas of contention that emerged in this pilot surrounded the question of the validity of the portfolio. This is an important issue for the dental profession and provides valuable information for those involved in developing revalidation. Dentistry is a very practical profession and this may well account for the antipathy of certain of the pilot group to this portfolio as a mechanism for revalidation. Some of the portfolio's components involve recording evidence of reflection on action, which was cited among the pilot group as a preferred way of working.10

Conclusions

The pilot scheme appears to have been acceptable to the dentists, given a number of caveats:

  1. 1

    More support in the use of the portfolio

  2. 2

    A need for training, especially amongst more mature dentists who are less likely to have had experience of portfolios and formalised reflection on practice

  3. 3

    A more flexible design of portfolio, to allow for different styles of presenting evidence to support meeting the stated standards

  4. 4

    That a method of collating centrally produced data be explored, to cut down the clerical elements of the process

  5. 5

    An enhanced mechanism(s) should be put in place to demonstrate clinical competence through observation and peer review. A system of appraisal, such as that being introduced for general medical practitioners, would be supportive.

The assessors agreed that the scheme would be significantly enhanced by appraisal of the dentist, effectively triangulating the data and its interpretation. Indeed they were uncomfortable about making a 'hard' decision on recommending that a dentist should or should not revalidate, without a record of some form of appraisal or mentored review forming part of the process. It was apparent that dentists who had been through vocational training themselves were more at ease with reflective processes than those who had not.

Appraisal appears essential to a successful revalidation process, both in terms of the use of the portfolio by the dentists and its assessment by the assessors. Also, in terms of acceptability to the profession – having the opportunity to discuss, with a peer, the contents that will form the basis of the revalidation portfolio more than once per revalidation cycle. This would allow the dentist to plan and demonstrate a response to issues raised. The apparent desire for some type of mentored approach to identifying learning and training needs of general dental practitioners reflects a similar finding amongst general medical practitioners.11