Abstract:

Introduction:

Clinical teaching is based on a real work environment, in professional practice settings, such as health services and units, under the supervision of the preceptor. Providing medical teachers with an assessment of their teaching skills is a powerful tool for improving clinical learning for students in training. In this context, the EFFECT (Evaluation and Feedback for Effective Clinical Teaching) questionnaire was developed by Dutch researchers in 2012 for teacher evaluation, being validated based on the literature about medical teaching in the workplace and incorporates the skills of the Canadian competency-based medical curriculum.

Objective:

To translate and cross-culturally adapt into Brazilian Portuguese and to validate the EFFECT questionnaire for teacher evaluation by Medical students.

Method:

Cross-cultural adaptation with the following steps: initial translation of the English version, synthesis of translated versions, back-translation, creation of a consensual version in Brazilian Portuguese, with adaptation, review, and analysis of content validity by an expert committee, pre-test with retrospective clarification interview, and reliability analysis by factorial analysis and internal consistency test (Cronbach’s alpha coefficient).

Result:

In the translation and back-translation stages, the disagreements were related to the use of synonyms and none of the items were modified in terms of their understanding, but in terms of adaptation into the Brazilian context. The evaluation of the expert committee showed the versions maintained the semantic and idiomatic equivalences of the content. Eighty-nine students participated in the pre-test. The internal consistency of the EFFECT questionnaire in Brazilian Portuguese was excellent for all domains, with Cronbach’s alpha coefficient ranging from 0.82 to 0.94.

Conclusion:

The translated and adapted version of the EFFECT questionnaire into Brazilian Portuguese is equivalent to the original instrument and has evidence of high validity and reliability, being able to constitute a national tool to evaluate the efficiency of clinical medicine teaching.

Keywords:
Educational Assessment; Teaching; Medicine; Translation

Resumo:

Introdução:

O ensino clínico é baseado em ambiente de trabalho real, em cenários de prática profissional, como serviços e unidades de saúde, sob a supervisão do preceptor. Proporcionar aos docentes de graduação médica uma avaliação sobre suas habilidades de ensino é uma ferramenta poderosa para melhorar a aprendizagem clínica dos estudantes em formação. Nesse contexto, o questionário Evaluation and Feedback for Effective Clinical Teaching (EFFECT) foi desenvolvido por pesquisadores holandeses em 2012 para avaliação docente, sendo validado com base na literatura sobre o ensino médico em local de trabalho. Esse instrumento incorpora as competências do currículo baseado em competências canadense.

Objetivo:

Este estudo teve como objetivos traduzir, adaptar transculturalmente para português do Brasil e validar o questionário EFFECT para avaliação docente por estudantes de Medicina.

Método:

A adaptação transcultural empregou as seguintes fases: tradução inicial da versão em inglês; síntese de versões traduzidas; tradução reversa; criação de versão consensual em português do Brasil, com adaptação, revisão e análise de validade de conteúdo por comitê de especialistas; pré-teste com entrevista retrospectiva de esclarecimento e análise de confiabilidade por análise fatorial e teste de consistência interna (coeficiente alfa de Cronbach).

Resultado:

Nas etapas de tradução e tradução reversa, as discordâncias relacionaram-se ao uso de sinônimos, e nenhum dos itens foi modificado em relação ao seu entendimento, e sim na adequação para a realidade brasileira. A avaliação do comitê de especialistas demonstrou que as versões mantinham a equivalência semântica e idiomática do conteúdo. Participaram do pré-teste 89 alunos. A consistência interna do EFFECT em português do Brasil mostrou-se excelente para todos os domínios, com coeficiente alfa de Cronbach variando de 0,82 a 0,94.

Conclusão:

A versão traduzida e adaptada do questionário EFFECT em português do Brasil possui equivalência cultural com o instrumento original e evidência de alta validade e confiabilidade, podendo constituir-se em instrumento nacional de avaliação da eficiência do ensino clínico de docente de Medicina.

Palavras-chave:
Avaliação Educacional; Ensino; Medicina; Tradução

INTRODUCTION

The several changes that have occurred in the field of medical education in the last decades, towards the construction of competency-based curricula (CBC) and evaluation of results and performance¹1. Costa LB, Esteche FF, Augusto Filho RF, Bomfim ALB, Ribeiro MTAM. Competências e atividades profissionais confiáveis: novos paradigmas na elaboração de uma matriz curricular para residência em medicina de família e comunidade. Rev Bras Med Fam Comunidade. 2018;13(40):1-11. doi: 10.5712/rbmfc13(40)1632.
https://doi.org/10.5712/rbmfc13(40)1632... ^),(22. Ferreira MA. Educação médica no século XXI: o desafio da integração da tecnologia e humanidades. Gazeta Médica. 2016;3(4):156-61. doi: 10.29315/gm.v3i4.42.
https://doi.org/10.29315/gm.v3i4.42... , have led to several intervention proposals in the training of health professionals, generating a new social mission of the Higher Education Institutions (HEIs)³3. Cavalcante Neto PG, Lira GV, Miranda AS. Interesse dos estudantes pela medicina de família: estado da questão e agenda de pesquisa. Rev Bras Educ Med. 2009;33(2):198-204. doi: 10.1590/S0100-55022009000200006.
https://doi.org/10.1590/S0100-5502200900... . Competency matrices for medical education, both for undergraduate courses, as proposed by the AAMC (Association of American Medical Colleges), and for medical residency, such as CanMEDS, of the Canadian Royal College of Physicians and Surgeons, and the Milestones Project, of the North American Accreditation Council for Graduate Medical Education, constitute the training base for the majority of young medical apprentices in the Western world and are examples for medical schools worldwide⁴4. Torralba KD, Jose D, Katz JD. Competency-based medical education for the clinician-educator: the coming of Milestones version 2. Clin Rheumatol. 2020;39(6):1719-23. doi: 10.1007/s10067-020-04942-7.
https://doi.org/10.1007/s10067-020-04942... ^)-(88. Lomis K, Amiel JM, Ryan MS, Esposito K, Green M, Stagnaro-Green A, et al. Implementing an entrustable professional activities framework in undergraduate medical education: early lessons from the AAMC core entrustable professional activities for entering residency pilot. Acad Med. 2017 June;92(6):765-70. doi: 10.1097/ACM.0000000000001543.
https://doi.org/10.1097/ACM.000000000000... .

In Brazil, the National Curriculum Guidelines (NCGs) propose the centralization of the teaching of medical training in primary health care (PHC)⁹9. Demarzo MMP, Almeida RCC, Marins JJN, Trindade TG, Anderson MIP, Stein AT, et al. Diretrizes para o ensino na atenção primária à saúde na graduação em Medicina. Rev Bras Educ Med. 2012;36(1):143-8. doi: 10.1590/S0100-55022012000100020.
https://doi.org/10.1590/S0100-5502201200... ^),(1010. Falk JW. A medicina de família e comunidade e sua entidade nacional: histórico e perspectivas. Rev Bras Med Fam Comunidade. 2004;1(1):5-10. doi: 10.5712/rbmfc1(1)2.
https://doi.org/10.5712/rbmfc1(1)2... . Therefore, there is a growing interest in the search for innovative ways of teaching that involve ethical, critical, reflective and transformative training¹⁰10. Falk JW. A medicina de família e comunidade e sua entidade nacional: histórico e perspectivas. Rev Bras Med Fam Comunidade. 2004;1(1):5-10. doi: 10.5712/rbmfc1(1)2.
https://doi.org/10.5712/rbmfc1(1)2... and that attract students to PHC and public health⁸8. Lomis K, Amiel JM, Ryan MS, Esposito K, Green M, Stagnaro-Green A, et al. Implementing an entrustable professional activities framework in undergraduate medical education: early lessons from the AAMC core entrustable professional activities for entering residency pilot. Acad Med. 2017 June;92(6):765-70. doi: 10.1097/ACM.0000000000001543.
https://doi.org/10.1097/ACM.000000000000... ^),(1111. Mitre SM, Siqueira-Batista R, Girardi-de-Mendonça JM, Morais-Pinto NM, Meirelles CAB, Pinto-Porto C, et al. Metodologias ativas de ensino-aprendizagem na formação profissional em saúde: debates atuais. Ciênc Saúde Colet. 2008;13(supl 2):2133-44. doi: 10.1590/S1413-81232008000900018.
https://doi.org/10.1590/S1413-8123200800... . The creation of integrated curricula that demonstrate the early integration of students since the first years of undergraduate school with professionals and users of health services, in longitudinal clinical practices is recommended¹²12. Ford CD, Patel PG, Sierpina VS, Wolffarth MW, Rowen JL. Longitudinal continuity learning experiences and primary care career interest: outcomes from an innovative medical school curriculum. J Gen Intern Med. 2018 Oct;33(10):1817-21. doi: 10.1007/s11606-018-4600-x.
https://doi.org/10.1007/s11606-018-4600-... ^)-(1414. Silva ATC, Medeiros Junior ME, Fontão PN, Saletti Filho HC, Vital Junior PF, Bourget MMM, et al. Medicina de família do primeiro ao sexto ano da graduação médica: considerações sobre uma proposta educacional de integração curricular escola-serviço. Rev Bras Educ Med . 2017;41(2):336-45. doi: 10.1590/1981-52712015v41n2rb20160016.
https://doi.org/10.1590/1981-52712015v41... ^).

In parallel with the development of concepts related to teaching over the years, in Brazil, national systems for institutional assessment of higher education have been created and improved, including accreditation processes¹⁵15. Amaral E. Além do horizonte do licenciamento e da certificação. Interface (Botucatu). 2020;24:e190736. doi: 10.1590/interface.190736.
https://doi.org/10.1590/interface.190736... . Usually, the assessment includes aspects related to the agreement with the NCGs, among others: teaching, research, extension, social accountability, facilities, student performance and the faculty¹⁶16. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Processo de avaliação. Brasília: Inep; 2015 [access in 8 feb 2021]. Available from: Available from: http://portal.inep.gov.br/web/guest/processo-de-avaliacao .
http://portal.inep.gov.br/web/guest/proc... .

The teaching-service integration indicated by the NCGs may encounter obstacles, such as the presence of preceptors and teachers with insufficient training for teaching¹⁴14. Silva ATC, Medeiros Junior ME, Fontão PN, Saletti Filho HC, Vital Junior PF, Bourget MMM, et al. Medicina de família do primeiro ao sexto ano da graduação médica: considerações sobre uma proposta educacional de integração curricular escola-serviço. Rev Bras Educ Med . 2017;41(2):336-45. doi: 10.1590/1981-52712015v41n2rb20160016.
https://doi.org/10.1590/1981-52712015v41... . The teacher plays a leading role in curricular changes¹⁷17. Perim GL, Ively GA, Abdalla G, Aguilar-da-Silva RH, Lampert JB, Stella RCR, et al. Desenvolvimento docente e a formação de médicos. Rev Bras Educ Med . 2009;33(1):70-82. doi: 10.1590/S0100-55022009000500008.
https://doi.org/10.1590/S0100-5502200900... and, for their effective implementation, it is necessary to develop instruments to evaluate the teaching process and for the global assessment of programs and curricula¹⁸18. Zimmermann MH, Silveira RMCF, Gomes RZ. O professor e a arte de avaliar no ensino médico de uma universidade no Brasil. Rev Bras Educ Med . 2019;43(3):5-15. doi: 10.1590/1981-52712015v43n3rb20180167.
https://doi.org/10.1590/1981-52712015v43... .

Most of the research on medical education assessment focuses on the use of several instruments, based on the competence view that consists of knowledge, skills, and attitudes, in a disintegrated way¹⁹19. Dijkstra J, van der Vleuten CPM, Schuwirth LWT. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. 2010;15(3):379-93. doi: 10.1007/s10459-009-9205-z.
https://doi.org/10.1007/s10459-009-9205-... ^)-(2121. van der Vleuten CPM, Schuwirth LWT. Assessing professional competence: from methods to programmes. Med Educ. 2005 Mar;39(3):309-17. doi: 10.1111/j.1365-2929.2005.02094.x.
https://doi.org/10.1111/j.1365-2929.2005... . As for teacher evaluation, there is no standardization, and most initiatives are based on assessments made by the students, with excessively broad, subjective criteria, often answered based on the teachers’ personality and not on their skills, and subject to confounding factors²²22. Kassis K, Wallihan R, Hurtubise L, Goode S, Chase M, Mahan JD. Milestone-based tool for learner evaluation of faculty clinical teaching. MedEdPORTAL. 2017 Sep 18;13. doi: 10.15766/mep_2374-8265.10626.
https://doi.org/10.15766/mep_2374-8265.1... ^)-(2525. Urrutia-Aguilar ME, Sanchez-Mendiola M, Guevara-Guzman R, Martinez-Gonzalez A. Comprehensive assessment of teaching performance in medical education. Procedia Soc Behav Sci. 2014 Aug 24;141:252-9. doi: 10.1016/j.sbspro.2014.05.044.
https://doi.org/10.1016/j.sbspro.2014.05... .

An evaluation method for teachers requires some characteristics to be effective: to be continuous and systematic; include all actors involved in the teaching-learning process; generate information about the educational diagnosis; lead to reflections that can contribute to the improvement of the course and teacher development. It is also necessary to safeguard the anonymity of students and ensure that teachers are not present during the assessment²⁶26. Belfor JA, Sena IS, Silva DKB, Lopes BRDS, Koga Júnior M, Santos BEF. Competências pedagógicas docentes sob a percepção de alunos de medicina de universidade da Amazônia brasileira. Ciênc Saúde Colet . 2018;23(1):73-82. doi: 10.1590/1413-81232018231.21342017.
https://doi.org/10.1590/1413-81232018231... .

Most medical professors are trained in a hospital-centered model, in which most of their experiences derive more from their clinical knowledge than from specific teaching training. Therefore, teacher training is essential for the development of competency-based education, as it is undeniable that isolated clinical training does not provide adequate preparation for the development of health professionals into teachers¹⁷17. Perim GL, Ively GA, Abdalla G, Aguilar-da-Silva RH, Lampert JB, Stella RCR, et al. Desenvolvimento docente e a formação de médicos. Rev Bras Educ Med . 2009;33(1):70-82. doi: 10.1590/S0100-55022009000500008.
https://doi.org/10.1590/S0100-5502200900... .

Understanding the importance of teacher evaluation, especially clinical preceptors, and considering the scarcity of instruments for practice evaluation and the satisfactory training of medical competencies, a group of Dutch researchers developed an instrument for teacher assessment by residents called the “Evaluation and Feedback for Effective Clinical Teaching” (EFFECT Questionnaire), which was validated based on the literature on medical teaching in the workplace and incorporating the CanMEDS competencies²⁷27. Fluit C, Bolhuis S, Grol R, Ham M, Feskens R, Laan R, et al. Evaluation and Feedback for Effective Clinical teaching in postgraduate medical education: validation of an assessment instrument incorporating the CanMEDS roles. Med Teach . 2012;34(11):893-901. doi: 10.3109/0142159x.2012.699114.
https://doi.org/10.3109/0142159x.2012.69... ^),(2828. Fluit CRMG, Bolhuis S, Grol R, Laan R, Wensing M. Assessing the quality of clinical teachers: a systematic review of content and quality of questionnaires for assessing clinical teachers. J Gen Intern Med . 2010;25(12):1337-45. doi: 10.1007/s11606-010-1458-y.
https://doi.org/10.1007/s11606-010-1458-... .

For the best student performance and training of new doctors, one can realize how crucial the continuous processes of faculty and clinical preceptor training are, and the greater application of the tools that assist this process. Therefore, it is undeniable the relevance of the translation and adaptation to the Brazilian reality of an assessment instrument capable of broadly evaluating all aspects inherent to teaching practice, aiming at encouraging continuous evaluation among students, teachers, and the Educational Institution, as well as the promotion of theoretical-practical integration and teaching-society.

This article aims to present the process of translation and cross-cultural adaptation of the EFFECT questionnaire into Brazilian Portuguese, which includes testing its validity and reliability so that it can be used for teaching evaluation by students in Brazilian medical schools.

METHODS

Study design

Translation and cross-cultural adaptation that included the following phases²⁹29. Souza AC, Alexandre NMC, Guirardello EB. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saude. 2017;26(3):649-59. doi: 10.5123/S1679-49742017000300022.
https://doi.org/10.5123/S1679-4974201700... ^)-(3333. Vaižgėliene E, Padaiga Ž, Rastenytė D, Tamelis A, Petrikonis K, Kregždytė R, et al. Validation of the EFFECT questionnaire for competence-based clinical teaching in residency training in Lithuania. Medicina (Kaunas). 2017;53(3):173-8. doi: 10.1016/j.medici.2017.05.001.
https://doi.org/10.1016/j.medici.2017.05... : 1. Initial translation of the original version; 2. Synthesis of translated versions; 3. Back-translation; 4. Creation of a consensual version in Brazilian Portuguese, with adaptation and review by an expert committee; 5. Initial pre-test, with retrospective clarification interview; 6. Reliability analysis. This study was carried out at Universidade de Fortaleza (UNIFOR), Ceará, between June 2019 and April 2020. The study received approval from UNIFOR’s Research Ethics Committee (Opinion N. 3,341,518, of May 22, 2019).

The EFFECT questionnaire

The EFFECT questionnaire aims to provide clinical preceptors and tutors with an assessment of their qualities as teachers in clinical practice²⁷27. Fluit C, Bolhuis S, Grol R, Ham M, Feskens R, Laan R, et al. Evaluation and Feedback for Effective Clinical teaching in postgraduate medical education: validation of an assessment instrument incorporating the CanMEDS roles. Med Teach . 2012;34(11):893-901. doi: 10.3109/0142159x.2012.699114.
https://doi.org/10.3109/0142159x.2012.69... . The questionnaire was originally validated by its authors based on the literature about medical education in the workplace and incorporates CanMEDS competencies. It is a self-administered questionnaire, containing 58 questions, of which 55 are grouped in 7 domains (The preceptor as a role model; Task allocation; Planning of training activities; Providing feedback; Teaching skills; General characteristics of the preceptor; and Assessment). The answers to each objective question are graded on a 6-point Likert scale (in Portuguese: 1= “crítico, impossível continuar assim” (Critical, impossible to continue like this); 2= “insatisfatório, muitas melhorias necessárias” (Unsatisfactory, many improvements are required) ; 3= “médio, precisa melhorar alguns pontos” (Regular, needs to improve some points); 4= “satisfatório, pode melhorar em detalhes” (Satisfactory, some details can be improved); 5= “Bom, continue assim” (Good, keep up the good work); 6= “excelente, exemplo para os outros” (Excellent, an example to others); e NAE= not-yet able to evaluate or “ainda não é capaz de avaliar”). The “NAE” option can be chosen if a specific item has not (yet) occurred during the teaching process. The last three open questions can be used as complementary comments for the teacher. It takes eight to ten minutes to complete the questionnaire. The questionnaire does not generate a final score; instead, each teacher who is evaluated by the method receives a report with the average scores of each item of the questionnaire (minimum of 1 and maximum of 6) based on the results of the respondents, in addition to the open responses in full. The EFFECT questionnaire was originally developed in Dutch with resident physicians²⁷27. Fluit C, Bolhuis S, Grol R, Ham M, Feskens R, Laan R, et al. Evaluation and Feedback for Effective Clinical teaching in postgraduate medical education: validation of an assessment instrument incorporating the CanMEDS roles. Med Teach . 2012;34(11):893-901. doi: 10.3109/0142159x.2012.699114.
https://doi.org/10.3109/0142159x.2012.69... and was translated by the original authors into English, of which version remains unpublished. The instrument has been also validated into the Lithuanian language, in a study with residents of several medical specialties³⁴34. Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciênc Saúde Colet . 2011;16(7):3061-8. doi: 10.1590/S1413-81232011000800006.
https://doi.org/10.1590/S1413-8123201100... . Thus, as the EFFECT questionnaire showed to be adequate when used in the context of clinical teaching in different medical residencies, it was assumed in this research that it could be tested in the outpatient context of PHC during the undergraduate medical course, without mischaracterizing the translation and adaptation process. The English version was granted, and its use in this study was authorized by the creators of the instrument.

Two items were added to the questionnaire to characterize the participants: age and which year of the medical course they are attending.

Participants

The study sample comprised 89 medical students from UNIFOR, aged at least 18 years, who agreed to participate in the study and had available time and Internet access. The sample recruitment to participate in the research was carried out by public call, with wide dissemination of the research, in physical form through posters in the educational institution and virtually via the university’s own communication application. The questionnaire was sent online, along with the free and informed consent form for obtaining the answers online, using the Google Forms® platform (https://www.google.com/forms/about/).

Procedures

First stage: two certified bilingual translators, independently carried out the initial translation of the English version into Brazilian Portuguese, from June to July 2019. Only one of the translators was aware of the concepts the questionnaire intends to measure, to provide a translation that most closely resembles the instrument in the English version. There was a third translation by one of the study’s authors, fluent in English. Therefore, the translated versions T1, T2 and T3 were generated.

Second stage: synthesis by the authors of a translated version, T12.

Third stage: two bilingual translators, one from the Letrare Extension Project of Universidade Federal do Ceará (constituted by young translators from the Language course) and a North American medical student fluent in Brazilian Portuguese, independently back-translated the T12 synthesis version from Brazilian Portuguese into English, to ensure the accuracy of the translation. Thus, the versions BT1 and BT2 back-translated into English, were created. The BT1 and BT2 items were compared with the English version, and then the choice of the best version of each item was made to adjust T12. This is a validity check process to ensure that the translated version reflects the same item content as the English version. This step often improves the unclear text, highlighting gross inconsistencies or conceptual errors in the translation.

Fourth stage: a group of 6 experts (including teachers of medicine, experts in medical education, professionals familiar with the construct of interest and the translators), using the Content Validity Index (CVI)²⁹29. Souza AC, Alexandre NMC, Guirardello EB. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saude. 2017;26(3):649-59. doi: 10.5123/S1679-49742017000300022.
https://doi.org/10.5123/S1679-4974201700... ^),(3434. Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciênc Saúde Colet . 2011;16(7):3061-8. doi: 10.1590/S1413-81232011000800006.
https://doi.org/10.1590/S1413-8123201100... defined, together with the authors of this study, a consensual translated version in Brazilian Portuguese (VCPB, versão consensual em português do Brasil) adapted to the cultural characteristics of Brazil. The CVI measures the proportion or percentage of judges in agreement with certain aspects of an instrument and its items. It consists of a Likert scale with a score of 1 (non-equivalent item) to 4 (absolutely equivalent item). The items that received a score of 1 or 2 were reviewed. The acceptable level of agreement of the content among the members of the expert committee must be at least 0.80 and, preferably, higher than 0.90²⁹29. Souza AC, Alexandre NMC, Guirardello EB. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saude. 2017;26(3):649-59. doi: 10.5123/S1679-49742017000300022.
https://doi.org/10.5123/S1679-4974201700... ^),(3434. Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciênc Saúde Colet . 2011;16(7):3061-8. doi: 10.1590/S1413-81232011000800006.
https://doi.org/10.1590/S1413-8123201100... . The committee was responsible for a review of all the translation phases and evaluated whether the translated versions (T1, T2, T3, T12, VCPB), the back-translated ones (BT1 and BT2) and the English version showed semantic, idiomatic, experimental, and conceptual equivalence³⁵35. Caires IS, Goldberger BU, Colares MFA, Gale R, Grant J, Troncon LEA. Tradução, adaptação, validação e avaliação para uso no Brasil de um instrumento britânico de auxílio à escolha da especialidade médica. Rev Bras Educ Med . 2017;41(4):540-50. doi: 10.1590/1981-52712015v41n3rb20160055.
https://doi.org/10.1590/1981-52712015v41... .

Fifth stage: with the final version in Brazilian Portuguese (VFPB, versão final em português do Brasil), this version was applied in a pre-test, in November 2019, answered by 89 participants. Additionally, a question was added to the end of the questionnaire to obtain suggestions for possible changes. The minimum number of respondents recommended for a pre-test is 30 to 40 subjects³²32. Fortes CPDD, Araújo APQC. Check list para tradução e adaptação transcultural de questionários em saúde. Cad Saude Colet. 2019;27(2):202-9. doi: 10.1590/1414-462x201900020002.
https://doi.org/10.1590/1414-462x2019000... ^),(3636. Tsang S, Royse CF, Terkawi AS. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J Anaesth. 2017;11(Suppl 1):S80-90. doi: 10.4103/sja.SJA_203_17.
https://doi.org/10.4103/sja.SJA_203_17... ^),(3737. Perneger TV, Courvoisier DS, Hudelson PM, Gayet-Ageron A. Sample size for pre-tests of questionnaires. Qual Life Res. 2015;24(1):147-51. doi: 10.1007/s11136-014-0752-2.
https://doi.org/10.1007/s11136-014-0752-... .

The researchers asked the participating students to answer the questionnaire thinking about their current or last PHC preceptor (longitudinal practical care discipline, which happens throughout the UNIFOR curriculum). After the questionnaires were completed by the respondents themselves, the authors performed a retrospective clarification interview with twelve participants to check whether there were items difficult to understand, irrelevant or offensive, and to verify the understanding of each item using the verbal probing technique (cognitive interview in which the interviewer asks research questions specifically designed to extract more detailed information in addition to those usually provided by the interviewees)³⁸38. Willis GB, Artino AR. What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. J Grad Med Educ . 2013;5(3):353-6. doi: 10.4300/JGME-D-13-00154.1.
https://doi.org/10.4300/JGME-D-13-00154.... . The understanding of the items by less than 80% of the interviewees or the suggestion of modifications by more than 20% of the respondents was established as a criterion for reviewing and modifying the translated version³⁹39. Zumpano CE, Mendonça TMS, Silva CHM, Correia H, Arnold B, Pinto RMC. Adaptação transcultural e validação da escala de saúde global do PROMIS para a língua Portuguesa. Cad Saude Publica. 2017;33(1):1-14. doi: 10.1590/0102-311x00107616.
https://doi.org/10.1590/0102-311x0010761... .

Sixth stage: validation of the final version through psychometric tests of dimensional structure evaluation (exploratory factor analysis - EFA)⁴⁰40. Hair JF, Black WC, Babin BJ, Anderson RE, Black WC, Anderson RE. Multivariate data analysis. 8th ed. Stamford: Cengage Learning; 2018. p. 95-120.^),(4141. Damásio BF. Uso da análise fatorial exploratória em psicologia. Aval Psicol. 2012;11(2):213-28 [access in 17 nov 2020]. Available from: Available from: http://pepsic.bvsalud.org/pdf/avp/v11n2/v11n2a07.pdf .
http://pepsic.bvsalud.org/pdf/avp/v11n2/... and Cronbach’s alpha reliability test (internal consistency)⁴²42. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297-334. doi: 10.1007/BF02310555.
https://doi.org/10.1007/BF02310555... ^),(4343. Pasquali L. Psicometria: teoria e aplicações: a teoria clássica dos testes psicológicos. Brasília: Editora da UnB; 1997..

Data analysis

A descriptive statistical analysis was performed, with the presentation of means, standard deviations and factorial load for each of the 55 variables in the translated questionnaire⁴⁰40. Hair JF, Black WC, Babin BJ, Anderson RE, Black WC, Anderson RE. Multivariate data analysis. 8th ed. Stamford: Cengage Learning; 2018. p. 95-120.. Factor analysis (FA) is used to investigate latent relationships for a large number of variables (in this case, questionnaire items) and to determine whether the information can be summarized in a smaller set of factors. Therefore, one can assess whether a research instrument, such as a questionnaire, is measuring what it is intended to measure. The factor can be defined as a linear combination of the original variables. The factors represent the latent dimensions (also called constructs), which summarize the original set of variables, while maintaining the representativeness of the characteristics of the original variables. The analysis can be exploratory factorial (EFA) or confirmatory factorial analysis (CFA). The dimensionality found in previous studies can be accessed initially by EFA⁴⁰40. Hair JF, Black WC, Babin BJ, Anderson RE, Black WC, Anderson RE. Multivariate data analysis. 8th ed. Stamford: Cengage Learning; 2018. p. 95-120.^),(4141. Damásio BF. Uso da análise fatorial exploratória em psicologia. Aval Psicol. 2012;11(2):213-28 [access in 17 nov 2020]. Available from: Available from: http://pepsic.bvsalud.org/pdf/avp/v11n2/v11n2a07.pdf .
http://pepsic.bvsalud.org/pdf/avp/v11n2/... , a technique used in the present study. Thus, we have several items to measure each latent construct (factor), which is not directly observable. The correlation coefficient is the covariance corrected by the differences in standard deviation. Therefore, the factor analysis identifies variables that have the same underlying structure.

To verify the internal consistency of each domain and of the questions as a whole, Cronbach’s alpha coefficient was used, considering values > 0.7 as an acceptable standard⁴²42. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297-334. doi: 10.1007/BF02310555.
https://doi.org/10.1007/BF02310555... ^),(4343. Pasquali L. Psicometria: teoria e aplicações: a teoria clássica dos testes psicológicos. Brasília: Editora da UnB; 1997.. Cronbach’s alpha coefficient assesses the degree to which items in a data matrix are correlated with each other. It is the most frequently used method in cross-sectional studies. All analyses were performed using the Stata 15.1© software (StataCorp LLC, College Station, TX, USA).

RESULTS

Eighty-nine students participated in the study, divided by medical course semester: 15 from the third (16.9%), 4 from the fourth (4.5%), 16 from the fifth (18%), 12 from the sixth (13.5%), 17 from the seventh (19.1%), 12 from the eighth (13.5%) and 13 from the first semester of medical internship (14.6%) (Table 1). The mean age of the participants was 22.9 years (range 19-41, SD 4.29).

Since this article has two related but different objectives, the results will be presented in 2 separate sections. The first one will deal with the results of the process of translation and cross-cultural adaptation of the questionnaire; and the second will show the data pertinent to the description of the characteristics of variables and the internal consistency psychometric data.

Translation and cross-cultural adaptation of the EFFECT questionnaire content

During the initial translation process from the English version into Brazilian Portuguese, some disagreements can be observed between the translators (T1 and T2) regarding the two translations. The differences, however, were minimal, using synonymous words and writing forms that did not change the meaning; for instance: Translator 1 translated the phrase “treat patients respectfully” as “tratar os(as) pacientes de maneira respeitosa”, while translator 2 translated as “tratar o paciente de forma respeitosa”. The most critical items were reviewed in the following stages of analysis by the expert committee and after interviewing the pre-test respondents.

In the back-translation stage, there were no difficulties when back-translating the items. Although the BT1 and BT2 versions did not have many items that were literally equal to the English version, they maintained the semantic equivalence.

During the adaptation stage, words, expressions and verbal conjugations were changed, aiming to improve the subject’s understanding in the Brazilian context. Moreover, at this stage, possible errors in grammar, spelling, typing and formatting were observed and corrected. The level of agreement between the experts, using the Content Validity Index, varied between 1 and 0.833 for all statements and items of the questionnaire.

Throughout the statements, the words “instructor/supervisor” were translated as “preceptor/supervisor”. In item 1, originally “ask for a patient’s history”, we chose the literal translation “perguntar o histórico do(a) paciente” instead of the expressions “realizar anamnese” or “coletar história”, which are common in Brazil, to maintain better translation equivalence. The expression “have a bad news conversation” in item 11 was changed to “transmitir más notícias ao paciente”. Item 31, “how I monitor the boundaries of my clinical work”, was translated as “minha postura ética durante meu trabalho clínico” due to the strangeness of the literal translation “como eu monitoro as fronteiras do meu trabalho clinico”.

Some expressions were modified due to cultural adaptation. Item 34, “how I make my reports”, in the context of clinical practice, refers to the activity of recording patient evolution in medical records, so it was translated to “como faço o registro nos prontuários”. Item 53, which in the English version was “reviews my portfolio during the assessment”, was translated as “avalia meu portfólio durante a avaliação”. However, after responses and interviews with the respondent students in the pre-test, it was observed that the word “portfolio” was unknown for some of them, probably because it is a non-standardized evaluation method for all courses or HEIs. Therefore, a short explanatory text was added after the item. In the “avaliação” (Evaluation) statement section, it was necessary to add the words “formativa” and “somativa” to facilitate the understanding of the sentence: “Durante seu treinamento, ocorreram várias entrevistas sobre o seu progresso. Pode-se enfatizar - quando necessário - os ajustes (durante uma revisão de progresso - formativa), ou na conclusão de parte da sua formação (durante uma avaliação de rodízio ou entrevista de avaliação - somativa)”.

On multiple occasions, English verbs followed by the pronoun “me” were inverted; for instance “gives me” was translated into “me dá”, as it is the more popular idiomatic expression in the spoken Brazilian Portuguese language.

The word “feedback”, which can be translated as “retroalimentação” or “retorno”, remained in English due to its wide use in the educational context and because its use is disseminated in the Brazilian Portuguese language.

Description of the variables and analysis of construct validity and internal consistency of the questions

The study of the characteristics of the questions shows averages ranging from 4.96 (item 28 - “retorna ao feedback dado anteriormente”) to 5.81 (item 43 - “me trata respeitosamente”).

The factorial loads ranged from 0.2816 (item 3 - “realizar abordagens clínicas”) to 0.9109 (item 33 - “como explico minhas escolhas por uma abordagem particular”) (Table 2). The factorial load can be defined as the correlation of the variable (item) with the factor, indicating how much each variable contributes to the factor (questionnaire construct). If this load has a positive value, it means that the variable is positively correlated with the factor, and, if it has a negative value, this correlation is negative. They are considered significant when they exceed the absolute value of 0.30⁴⁴44. Tabachnick BG, Fidell LS. Using multivariate statistics. 3rd ed. New York: Harper Collins; 1996.. Thus, it is observed that, except for item 3, all items of the translated questionnaire were positively correlated to the analyzed domains.

A large proportion of “NAE” responses (não capaz de responder ainda) were observed for items 48 (“prepara avaliações de progresso”), 53 (“avalia meu portfólio durante a avaliação”) and 54 (“presta atenção em minha autorreflexão”), all related to evaluation methods.

The description of the data according to the domains is shown in Table 3. The domain with the lowest average was “fornecendo feedback” (average 5.19; SD 0.857) and the one with the highest average was “características gerais do preceptor/supervisor” (mean 5.57; SD 0.714). The lowest internal consistency was found for the domain “cuidando do trabalho significativo” (0.83), and this value being classified as acceptable³⁶36. Tsang S, Royse CF, Terkawi AS. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J Anaesth. 2017;11(Suppl 1):S80-90. doi: 10.4103/sja.SJA_203_17.
https://doi.org/10.4103/sja.SJA_203_17... .

DISCUSSION

Over the past few decades, profound changes have occurred in the medical education scenario, such as the introduction of new teaching and assessment techniques, as well as changes in the profile and involvement of university students⁴⁵45. Troncon LEA, Bollela VR, Borges MC, Rodrigues MLV. A formação e o desenvolvimento docente para os cursos das profissões da saúde: muito mais que o domínio de conteúdos. Medicina (Ribeirão Preto). 2014;47(3):245-8. doi: 10.11606/issn.2176-7262.v47i3p245-248.
https://doi.org/10.11606/issn.2176-7262.... . These belong to “Generation Z” and grew up immersed in this reality of technological innovations⁴⁶46. Caramori U, Mello JB, Barretto CAP, Costa RMR, Peña SS, Ramos ALC, et al. Projeto Fellows: habilidades de educação para estudantes das profissões da saúde. Rev Bras Educ Med . 2020;44(1):1-8. doi: 10.1590/1981-5271v44.1-20190233.
https://doi.org/10.1590/1981-5271v44.1-2... ^),(4747. Eckleberry-Hunt J, Lick D, Hunt R. Is medical education ready for generation Z? J Grad Med Educ . 2018;10(4):378-81. doi: 10.4300/JGME-D-18-00466.1.
https://doi.org/10.4300/JGME-D-18-00466.... . For this generation, it is necessary to use student-centered methodologies, favoring collaboration, interdisciplinarity, autonomy and critical and citizen education, supporting the acquisition of communication, leadership, research and management skills⁴⁶46. Caramori U, Mello JB, Barretto CAP, Costa RMR, Peña SS, Ramos ALC, et al. Projeto Fellows: habilidades de educação para estudantes das profissões da saúde. Rev Bras Educ Med . 2020;44(1):1-8. doi: 10.1590/1981-5271v44.1-20190233.
https://doi.org/10.1590/1981-5271v44.1-2... .

To follow these changes, it is necessary to review the teacher’s role, abandoning the concept of academic or encyclopedic professor, as a mere transmitter of knowledge, and that of the technical specialist⁴⁸48. Ferreira CC, Souza AML. Formação e prática do professor de medicina: um estudo realizado na Universidade Federal de Rondônia. Rev Bras Educ Med . 2016;40(4):635-43. doi: 10.1590/1981-52712015v40n4e01012015.
https://doi.org/10.1590/1981-52712015v40... , and to start exercising a variety of duties³⁷37. Perneger TV, Courvoisier DS, Hudelson PM, Gayet-Ageron A. Sample size for pre-tests of questionnaires. Qual Life Res. 2015;24(1):147-51. doi: 10.1007/s11136-014-0752-2.
https://doi.org/10.1007/s11136-014-0752-... . Therefore, the HEIs have emphasized the evaluation of their teachers based on their clinical competency, teaching skills, personal qualities, involvement with students, quality in providing care to patients, in addition to the ability to provide feedback and guidance throughout the teaching activities⁴⁹49. Al Ansari A, Arekat MR, Salem AH. Validating the modified system for evaluation of teaching qualities: a teaching quality assessment instrument. Adv Med Educ Pract. 2018;9:881-6. doi: 10.2147/AMEP.S181094.
https://doi.org/10.2147/AMEP.S181094... .

In this study, the translation, cross-cultural adaptation and evaluation of the validity and reliability of the EFFECT questionnaire were carried out in Brazilian Portuguese, demonstrating that the instrument showed to be valid and reliable to be used in the context of Brazilian medical education. The results show that there were no major difficulties during the stages of translation, back-translation, adaptation and validation by the experts, considering that many educational concepts and values are common to several medical schools worldwide. It is important to note that the respondents’ difficulties and doubts in relation to the questionnaire were observed and considered for the preparation of the final version.

The results of this study indicate that the Brazilian version of the EFFECT questionnaire has acceptable psychometric properties and can be used to evaluate the teachers’ clinical teaching from the perspective of undergraduate medical students. All domains showed a satisfactory reliability coefficient. It is noteworthy that the participating students indicated that some items ‘not-yet able to evaluate’ (NAE). Most of these items are due to the domain “avaliação”, even though it was answered only by those who responded affirmatively to the domain’s introductory question (“Seu preceptor/supervisor conduz revisões de progresso ou entrevistas de avaliação com você?”). This fact may be associated with a non-standardization of the evaluation processes of the students’ competencies or the fact that these are evaluated in other ways, not necessarily the one present in the questionnaire. Only 4 items of a total of 55 in the questionnaire had an average score lower than 5 (answer equivalent to “Bom, continue assim”, demonstrating a good evaluation of the faculty by the students. When the domains were analyzed, all of them obtained an average greater than 5.

In the domain “o(a) preceptor(a) como modelo”, items 6 (“aplicar resultados de pesquisas acadêmicas”) and 14 (“pede a opinião de outros sobre seu próprio desempenho”) were the only ones with an average lower than 5. As for the domain “planejamento de atividades de treinamento”, a high score (average of 5.52) was obtained when questioning the teacher’s availability when the student deemed it necessary. However, there was a decrease in the score (5.38) when asked whether the preceptor sets aside time for the student when they need it, and a greater decrease (5.15) when asked whether the teacher reserves time to supervise/advise the students. Regarding “fornecendo feedback”, two items had an average lower than 5: “me deixa pensar sobre meus pontos fortes e fracos” and “retoma feedbacks dados previamente”.

These results demonstrate how important and appreciated by the students is the fact that teachers are open to welcoming their needs and that they plan specific moments for evaluation. The feedback strategy provides the student with a self-assessment and should have an impact on their development. These are complex and dynamic moments of student-teacher interaction⁵⁰50. Ramani S, Könings KD, Ginsburg S, van der Vleuten CPM. Meaningful feedback through a sociocultural lens. Med Teach . 2019;41(12):1342-52. doi: 10.1080/0142159X.2019.1656804.
https://doi.org/10.1080/0142159X.2019.16... . An effective feedback should be assertive, respectful, descriptive, timely and specific⁵¹51. Zeferino AMB, Domingues RCL, Amaral E. Feedback como estratégia de aprendizado no ensino médico. Rev Bras Educ Med . 2007;31(2):176-9. doi: 10.1590/S0100-55022007000200009.
https://doi.org/10.1590/S0100-5502200700... . It is expected that the teacher be able to mediate, listen and dialogue, favoring the exchange of ideas and the articulation between the knowledges⁵²52. Leite VT, Vazzi PIFL, Moura MBR, Pereira LS, Caldas Neto TP, Lima EHM. Avaliação do perfil dos professores de Medicina de uma universidade do interior de Minas Gerais. Rev Bras Educ Med . 2020;44(3):e096. doi: 10.1590/1981-5271v44.3-20190330.
https://doi.org/10.1590/1981-5271v44.3-2... .

It can be observed that in the “habilidades de ensino” domain, there was a good score in relation to encouraging active participation in discussions and encouraging questions (averages of 5.37 and 5.33, respectively). In parallel, a slightly lower score was obtained in several aspects, such as the review of learning objectives (average of 5.0), discussion of ethical issues (average of 5.02) and the integration of evidence-based medicine at work (average of 5.15). This result corroborates the importance of teachers to keep up-to-date on teaching-learning strategies to ensure efficient pedagogical practices⁵³53. Almeida MTC, Batista NA. Ser docente em métodos ativos de ensino-aprendizagem na formação do médico. Rev Bras Educ Med . 2011;35(4):468-76. doi: 10.1590/S0100-55022011000400005.
https://doi.org/10.1590/S0100-5502201100... .

It is known that there is a lack of teaching professionalization among medical professors, mainly related to teaching skills, and often associated with a lack of knowledge of the curricular structure and the recently-graduated student profile desired by educational institutions. It is necessary to overcome information overload-based teaching, where the teacher is focused on the technique and the mere transmission of information, without integrating disciplines and using student assessment methods, with a sometimes punitive character. This situation can be an obstacle for doctors who do not consider teaching as their main professional activity⁵⁴54. Garcia MAA, Silva ALB. Um perfil do docente de medicina e sua participação na reestruturação curricular. Rev Bras Educ Med . 2011;35(1):58-68. doi: 10.1590/S0100-55022011000100009.
https://doi.org/10.1590/S0100-5502201100... .

University-level education can be understood in two ways: non-professional, which considers the act of teaching as repeating previous models based on everyday practical experience; and the professional one, which defends teaching as a complex act consisting of skills that can be acquired, improved and expanded through a consistent training process⁵⁵55. Costa NMSC. Formação pedagógica de professores de medicina. Rev Lat Am Enfermagem. 2010;18(1):102-8. doi: 10.1590/S0104-11692010000100016.
https://doi.org/10.1590/S0104-1169201000... . The criterion of being considered a good medical professional in a specific area of clinical practice, although common when hiring teachers for medical courses, is not enough to guarantee quality teaching.

Regarding the domain “características gerais do(a) preceptor(a)”, the item with the highest average (5.81) was related to a respectful student-teacher relationship (item 43). The lower averages were those for items 45 (“deixa claro que posso contar com ele(a)’’) with an average of 5.57, and 47 (“está aberto(a) a questões/problemas pessoais”) with an average of 5.24. Although traditionally, the actions of guiding, orienting and advising in achieving personal goals, also seeking interpersonal and psychosocial development beyond the professional environment are considered the “Mentor’s role, and not specifically the “Preceptor’s” or the “Supervisor’s”, roles that are more associated to teaching in real clinical situations⁵⁶56. Botti SHO, Rego S. Papel de preceptor, supervisor, tutor e mentor. Rev Bras Educ Med . 2008;32(3):363-73. doi: 10.1590/S0100-55022008000300011.
https://doi.org/10.1590/S0100-5502200800... , these results demonstrate that there is still room for improving the student-teacher relationship.

The difficulty in obtaining the adherence of medical teachers to curricular reforms has been recurrent and can be explained by several factors, such as the non-professionalization of the teaching function, partial dedication to schools and insecurity. A study sought to analyze how the medical teachers’ characteristics at a university in Minas Gerais had an impact on the way students evaluate teaching. The data showed that teachers with less time since graduation, less time working as a teacher and less experience but are motivated by the pleasure of teaching, are better evaluated than older and more experienced ones⁵²52. Leite VT, Vazzi PIFL, Moura MBR, Pereira LS, Caldas Neto TP, Lima EHM. Avaliação do perfil dos professores de Medicina de uma universidade do interior de Minas Gerais. Rev Bras Educ Med . 2020;44(3):e096. doi: 10.1590/1981-5271v44.3-20190330.
https://doi.org/10.1590/1981-5271v44.3-2... .

In Brazil, a cross-sectional study was carried out with 28 students on the competencies of teachers of the medical course at Universidade Federal do Amapá, using focal groups. The students emphasized the importance of providing formative feedback to teachers. For them, the evaluation helps so that the teacher identifies their deficiencies and can seek ways to solve them, in addition to increasing the bond with the students. The study also showed, in the students’ opinion, that the best way to assess teachers would be through a structured questionnaire with objective and subjective items²⁶26. Belfor JA, Sena IS, Silva DKB, Lopes BRDS, Koga Júnior M, Santos BEF. Competências pedagógicas docentes sob a percepção de alunos de medicina de universidade da Amazônia brasileira. Ciênc Saúde Colet . 2018;23(1):73-82. doi: 10.1590/1413-81232018231.21342017.
https://doi.org/10.1590/1413-81232018231... .

Any instrument designed to assess the quality of education must have good validity and reliability. Moreover, the generated results need to be useful to provide improved teaching, support and motivation for the teachers, academic advancement and promotion, curricular evolution, in addition to providing subsidies for the creation of faculty development programs⁵⁷57. Kikukawa M, Stalmeijer RE, Okubo T, Taketomi K, Emura S, Miyata Y, et al. Development of culture-sensitive clinical teacher evaluation sheet in the Japanese context. Med Teach . 2017;39(8):844-50. doi: 10.1080/0142159X.2017.1324138.
https://doi.org/10.1080/0142159X.2017.13... . These characteristics are present in the EFFECT questionnaire.

In comparison with the original study for the development of the EFFECT²⁷27. Fluit C, Bolhuis S, Grol R, Ham M, Feskens R, Laan R, et al. Evaluation and Feedback for Effective Clinical teaching in postgraduate medical education: validation of an assessment instrument incorporating the CanMEDS roles. Med Teach . 2012;34(11):893-901. doi: 10.3109/0142159x.2012.699114.
https://doi.org/10.3109/0142159x.2012.69... questionnaire and another previous study about its validation in Lithuanian³³33. Vaižgėliene E, Padaiga Ž, Rastenytė D, Tamelis A, Petrikonis K, Kregždytė R, et al. Validation of the EFFECT questionnaire for competence-based clinical teaching in residency training in Lithuania. Medicina (Kaunas). 2017;53(3):173-8. doi: 10.1016/j.medici.2017.05.001.
https://doi.org/10.1016/j.medici.2017.05... , there are some differences in our research that may have influenced the results. The present study was carried out with a smaller number of participants. Moreover, previous studies used the EFFECT questionnaire with resident physicians, while our study applied it to undergraduate medical students. Another limitation of the present study is that there was no re-test.

The EFFECT questionnaire in its Brazilian Portuguese version showed to be effective and easy to apply, generating information that can be useful for educational institutions to prepare faculty development plans. The university where this study was developed has a Professional Development Program in Education (PDPE, Programa de Desenvolvimento Profissional em Educação), coordinated in partnership with the Center Pedagogical Advisors and the Educational Technologies Center, where courses are systematically offered to all university teachers. The PDPE proposes continuing education on current and innovative topics in the context of higher education for teachers. Therefore, there is an expectation that the instrument will be adopted to generate information for the process of planning institutional actions for continuing teacher education.

CONCLUSIONS

The results of this study indicate that the Brazilian Portuguese version of the EFFECT questionnaire has acceptable psychometric properties for the evaluation of clinical teachers by students from medical courses at Brazilian Higher Education Institutions. It is noteworthy that there is no available validated instrument yet in the local literature that can measure the same teaching competencies as the EFFECT.

ACKNOWLEDGMENTS

We would like to thank Lia Fluit, associate professor at Hoofd Research in Learning and Education of the Medical Center of Radboud University, Nijmegen, Netherlands, for the collaboration and for granting permission and allowing the use of the original English version of the EFFECT questionnaire.

REFERENCES

Edited by

Publication Dates

History