As a result of the E-TRAIN project a handbook for the training of panel members for external QA procedures has been published. The handbook has been written by Dr Gillian King. This handbook gives practical guidelines on how to organize effective training sessions for panel members. All steps of the training process are covered in the handbook, from the planning phase up to the necessary evaluation of the delivered trainings. It is believed that both quality assurance agencies and higher education institutions, and also the individual experts, can profit from its reading.
The Handbook can be downloaded Handbook for the training of panel members.
Handbook for the Training of Panel Members for External Quality Assurance Procedures
Foreword The trust building potential of external quality assurance (QA) procedures depends upon the quality and professionalism of the involved external experts. In an increasingly internationalized higher education area independent, highly competent and intercultural aware experts are urgently needed in order to guarantee fair decision making and to benefit higher education institutions in all types of external QA procedures. The European Consortium for Accreditation in Higher education (ECA) has acknowledged the vital role of the expert panel members ever since its inception (Code of good practice, 2004 and Guidelines for the selection of external expert panel members, 2007). After careful review of the training practices of its members agencies, ECA in 2010 launched its EU funded “E-TRAIN” project with the goal to train and to share a community of knowledgeable and internationally experienced experts in the domain of external QA in higher education. Based on the collection of international good practices ECA successfully offered pilot trainings to experts and established a „train the trainer“-programme.
The E-TRAIN project is strictly output-oriented and offers guides to good practices, a training handbook, an electronic training portal for experts, standardised expert portfolios and a European Expert Exchange Platform which provides access to a searchable database of experts.
This handbook gives practical guidelines on how to organize effective training sessions for panel members. All steps of the training process are covered in the handbook, from the planning phase up to the necessary evaluation of the delivered trainings. It is believed that both quality assurance agencies and higher education institutions, and also the individual experts, can profit from its reading.
Today all relevant higher education stakeholder groups of the European Higher Education Area acknowledge the need for well trained, professionally acting and internationally competent expert panellists. The E-TRAIN project has powerfully contributed to the establishment of such a community of experts and is clearly strengthening the European dimension in quality assurance. Based on the success of its operations ECA will – together with its partners – continue its efforts to organise regular trainings in order to assure the required capabilities and capacities of their external experts and to promote consistency of decisions in external quality assurance.
We hope you will find this handbook interesting and enjoyable. Additional information about this project is provided on our website (http://www.ecaconsortium.net). Please feel free at any time to address your questions and comments also directly to our secretariat. Rolf Heusser, Chairman of ECA The Hague, The Netherlands.
1. Introduction This guidance is aimed at any person who will be delivering training to members of expert panels. The term ‘panel member’ is used to refer to any person who will be carrying out a panel task, such as a review or audit or accreditation activity for a quality assurance agency (QA agency). In some agencies or countries those persons are referred to as ‘experts’, ‘reviewers’, ‘auditors’, ‘review/audit secretary’, etc. This handbook uses the term ‘panel members’ for all such people.
This handbook refers to ‘training event’: this is intended to include all kinds of training sessions whether they last 3 hours or 3 days. This handbook is not concerned with training that is primarily carried out online or remotely. It is concerned with training that is primarily delivered to participants face-to-face. A face-to-face event, however brief, has uses other than just passing on knowledge or acquisition of skills. It enables the trainer (who will usually be a staff member of a quality assurance or accreditation agency) to get to know the panel members who will be carrying out work for the agency in the future. It also helps participants who may work together in the future, to meet and get to know each other a little. This will help to build a community of practice amongst the panel members, which in turn will help them to support and learn from each other. Meeting face-to-face will give the trainer an opportunity to model the values and expectations of the quality assurance agency and encourage a professional attitude amongst panel members.
The guidance for planning a training event given below should be helpful whatever the length of the training, whoever the participants are, and whatever the objectives of the training are. The general principles outlined below will still apply. The guidance is divided into 5 sections: deciding on the overall aim of training; deciding on content, session aims and learning outcomes; looking at the needs of participants; delivering the training; ensuring that the training is effective. A stage-by-stage summary of the main stages of the guidance may be found in the checklist in
Annex 1: Stage by stage checklist of main points for training.
In each section of the guidance the trainer will be encouraged to ask some simple questions. It is important that the trainer knows the answers to these questions so that the training prepared is appropriate, relevant and effective.
There is also a list of publications and websites in Annex 2: References which you might find useful if you need more information or ideas.
MORE...
6. How to ensure that training has been effective The trainer needs to ask him/herself the questions: “How do I plan the training so that it helps participants to retain the material? How do I get feedback on the training and whether it has helped participants in their panel work?”
6.1. Retention We know that much of the knowledge and skills learned on a training event is lost very quickly unless it is put into use straightaway on a day-to-day basis. We noted in section 4.1. Knowing your participants
• Make sure the design of training matches the needs that you and your organisation have identified. In this way you should be able to ensure that there will be opportunities for participants to use the knowledge and skills acquired on the training event, even though these might be some time in the future
that, for many of our participants, there may be a significant gap between training and carrying out panel work. If training is to be effective it has to be retained long enough to be taken into the workplace or real-life situation – in our case, into the panel situation. We can maximise the chances of this in various ways:
• Make sure that anyone from your organisation who will be working with your panel members also knows what the training has included, and is prepared to coach panel members when they carry out panel work to help them to put skills and knowledge into practice. It is very useful if members of your organisation can attend at least one training event
• Make sure that the training includes practical examples which reflect the actual work of the panel member. These can be descriptions of the work, case studies, or practice activities such as mock meetings. Introducing activities into the training which encourage ‘doing’, rather than just ‘listening’ or ‘speaking’, also encourages retention of the material
• If you have the time during the training event try to encourage participants to discover the principles involved in panel work for themselves. We tend to remember the knowledge that we have discovered for ourselves better than things we are simply told. You could use case-studies for problem-based learning activities, or give small groups open-ended questions or problems to discuss. Remember to share the solutions that individuals or groups discover with the whole group after the exercise
• Give participants an aide-mémoire of the process that they will be using as a panel member. This could take the form of a guide to procedure (just bullet-points or headlines) which you could refer to during the training. It can then be used by participants during their panel work to ‘jog their memory’ of the main points that they learned on training
• When you design hand-outs or other training materials for the participants think about how the participant will use them after the training and try to ensure that they will still be comprehensible and useful even after the training has finished. If the training is changed after you deliver it for the first time you will need to remember to update those who have already had the training. If it is possible, it might be a good idea to put the training materials on a part of your organisation’s website dedicated to panel members. Then you will be able to update training materials easily.
• Encourage action planning on the part of participants. It was noted above (in section 3.4.2. Reflection, summary of the day, action planning) that it is a good idea to allow time in the training event for participants to reflect on their learning and put together some action points to be started when they return home. These action points will give a trigger for the participant to keep thinking about the training event, and to carry on with their learning after the training until they have a chance to carry out some panel work. It is also useful if the new panel member is given the opportunity to reflect after the first panel work, and see whether there are any new action points to consider (see section 6.3 Continued engagement with participants
• Encourage follow-up between participants, and if you have the time, with yourself. This could be through email, online, get-togethers, support groups, etc. This will help to build up a community of panel members who can support each other (see section 6.3 Continued engagement with participants).
• If you can, use past participants as mentors. Once a panel member has been trained and has carried out some panel activity he/she may be prepared to mentor new panel members. The mentoring relationship may have many forms so you will want to put in writing what the mentor is expected to do. It may simply involve being available should the new panel member wish to email with a question. Or it could include meeting and discussing the new panel member’s work.
6.2. Evaluation It is no good encouraging your participants to retain what they have learned on the programme if what you have delivered is not relevant to the learning outcomes, or has not been delivered in a way that helps them to learn. So you need to find out how effective the training has been and, if it has not worked as well as you wanted, to change some aspects of it. This is the process of evaluation and it is a very important part of the overall training cycle (see the diagram in Annex 7: The training cycle).
There are different options for carrying out evaluation and gathering feedback.
• Spot checks during the training: you can find out how participants are doing on the training event by asking questions now and again. This method is probably most helpful during a longer training event where participants have time to settle down and start learning, and where the learning outcomes are build upon each other sequentially. You could ask general questions like ‘What have you learned so far?’ or more detailed ones about the content of the programme, like ‘Can you explain the three main sections of the Qualifications Framework?’. Participants could write down their answers quickly so that you could gather them in for analysis. Simply observing the participants will also give you an idea of whether they are understanding what is going on and are finding the training material intelligible.
• Give out a ‘Reactionnaire’ straight after the training has finished. As the name implies this kind of evaluation instrument gathers the participants’ reaction to the training. It can be short or detailed, and ask a few questions or many. You could use this kind of questionnaire to ask participants whether the aims of the training have been met. To do that you would need to make sure that you had set the aims of the training, and then decide what questions you would ask the participants to try to establish whether your aims have been met. An example is given in Annex 8: Example of a training ‘Reactionnaire’.
• Evaluation of key learning outcomes: this is a more sophisticated kind of evaluation questionnaire. It can ask all the questions in a Reactionnaire, but it concentrates on asking the participants whether they think they have been able to meet the learning outcomes of the training. This kind of evaluation demands (a) that you have set learning outcomes; and (b) that the participants know what they are.
• Group discussion: if you have some time after the training event has ended and some of your participants can stay around for an hour you could run a group discussion on how the training went. You could ask the same questions as in the Reactionnaire or learning outcome questionnaire, but having the group of people in the room means that you can drill down into their responses and get more information about why aspects of the training worked well, and why some did not. Or you could focus on the aspects of the training that you personally were less confident about. Remember though that the opinions expressed by the small group may not be completely representative of the whole group of participants.
• Delayed questionnaire: it can be very useful to give participants a questionnaire some time after the training (and ideally after they have carried out some panel work) to ask whether the training was appropriate for their panel task and whether they have been able to put the training to use. This can be in addition to a Reactionnaire and/or Evaluation of learning outcomes, depending on how much time and money you can spend on evaluation. It might make more sense to invest in detailed evaluation at the start of a training cycle, so that you can be sure that the training is meeting its aims, and then perhaps reduce evaluation, or use different evaluation instruments, later on.
• Pre-training questionnaire: the delayed questionnaire could be combined with a pre-training questionnaire which asks the participants for their perception of how well prepared they are for panel work. They can then be asked the same questions after training and/or after their first panel work to see whether they feel more confident and better prepared for their work. If they do not, then something is not working and you need to think again about the learning outcomes of the training.
These kinds of questionnaires are designed to help with your own developmental needs and with the development of the training event. They will provide useful information for you but may not be sophisticated enough for other purposes, such as demonstrating a business case, or applying for funding. If you need sophisticated or statistical information you may need to consult a professional who can devise evaluation methods for you.
If you are tempted to think that evaluation is a waste of time, remember that it will actually save you time and effort in the future. If you have chosen your learning outcomes carefully, and delivered training effectively, this means that you will produce panel members who know the process, adhere to that process, behave professionally and produce reports on time. If this is not happening the training is a waste of time and money, and your life will be harder not easier. So you need to find out whether your training is working, and if it is not, why not.
6.3. Continued engagement with participants In the last section we noted that giving participants the opportunity to keep in contact after the training was one of the ways you could increase retention of the training material and encourage participants to put the training into action when they carry out panel work. In this way participants can continue to ask questions about panel work and increase their knowledge of the processes that they will be working in. They can also swap ideas for dealing with difficult situations. They can mentor or buddy one another to develop good practice in panel activities. In this way they will build up a community of panel members who will help to improve not only their own performances but also increase the consistency with which panel activities are carried out.
The most obvious way that you can facilitate this activity is providing an online facility, like a ‚chat room’, or by enabling participants to keep in email contact with each other. Unless you have made this a condition of the training, you should make sure that all participants are happy about having personal details such as email addresses circulated to everyone else. You should also make it clear what the rules for communication are, and whether anyone will moderate the chat room or emails.
If you have the resources, offering refresher training, annual meetings, or workshops are very good ways of keeping your community of panel members together and will give you opportunities to update them on any changes to the panel processes. Of course, providing these activities is expensive and panel members may not have the time to attend, so you might choose to do this updating using a website. Another alternative is to set aside a small amount of time before a panel activity (that is, a review, accreditation event, etc.) to update and refresh panel members’ knowledge before they start a new piece of panel work.
You can also encourage new panel members to reflect on their first panel event and draw up any action points that they feel they should address. If you have the resources, it would be good for a member of your agency to go through the action points with the new panel member to ensure that they can be dealt with successfully – perhaps with a mentor, or through using information on your website.
If a member of your agency is present with the panel team in its panel work, then you might be able to give a new panel member immediate feedback on their first panel activity. If you do this, then ideally you should have some criteria against which to give feedback. Some examples might be: was written work completed before the deadline? was the panel member punctual at the panel event? did the panel member contribute to panel discussions constructively and knowledgeably? did the panel member behave in a professional way with meetings with the institution? Work out what you want your panel member to do, and how you want them to behave. Then draw up your feedback criteria to match.
In order to be most effective training should not end when the last participant leaves the training room. If you can encourage panel members to stay in contact with you and with each other you will help to produce a community of panel members who will not only be interested in keeping their knowledge up to date, but also help to improve panel processes, and provide a source of support for you and for each other.
The Handbook can be downloaded Handbook for the training of panel members.