Abstract

The strengthening of psychological assessment in Brazil in the twenty-first century can be understood as a result of the foundation of Psychological Test Evaluation System (SATEPSI, its Portuguese acronym) by Resolution 02/2003 of the Federal Council of Psychology. In this sense, the present study aimed to describe the impact of SATEPSI for research in the area. A survey of Brazilian researchers' publications was conducted in two periods in SciELO and BVS-Psi databases - 1993-2002 and 2003-2014 - using the terms "psychological evaluation", "psychological tests" and "construction / adaptation of psychological instruments". Overall, the results point to the positive impact of SATEPSI for the area of psychological assessment. It was observed an increasing in quantity and quality of publications, as well as a greater variety of psychological tests used in different contexts. It can be concluded that the pursued goals of the Resolution 02/2003 were mostly achieved.

Keywords:
psychological assessment; psychological tests; scientific publications

Resumo

O início do século XXI marcou o fortalecimento da Avaliação Psicológica no Brasil como resultado de ações que culminaram na implantação do Sistema de Avaliação de Testes Psicológicos (SATEPSI) por meio da Resolução nº 02/2003 do Conselho Federal de Psicologia. Nesse sentido, o presente trabalho objetivou descrever o impacto da criação do SATEPSI para a pesquisa na área. Para isso, foi realizado um levantamento das publicações de pesquisadores brasileiros em dois períodos - 1993 a 2002 e 2003 a 2014, nas bases de dados SciELO e BVS-Psi - utilizando os termos "avaliação psicológica", "testes psicológicos" e "construção/adaptação de instrumentos psicológicos". De forma geral, os resultados encontrados apontam para o impacto positivo do SATEPSI para a área de avaliação psicológica, observada no aumento da quantidade e qualidade das publicações, bem como em uma maior variedade de contextos e testes psicológicos utilizados nas pesquisas. Conclui-se que as metas almejadas com a publicação da Resolução foram, em sua maioria, alcançadas.

Palavras-chave:
avaliação psicológica; testes psicológicos; publicações científicas

Resumem

El comienzo del siglo XXI marcó el fortalecimiento de la Evaluación Psicológica en Brasil, como resultado de acciones que culminaron en la implantación del Sistema de Evaluación de Tests Psicológicos (SATEPSI) a través de la Resolución Nro.02/2003 del Consejo Federal de Psicología. En este sentido, el presente estudio tuvo como objetivo describir el impacto del SATEPSI en la investigación en el área. Fue realizado un levantamiento de las publicaciones de investigadores brasileños en dos períodos, 1993 a 2002 y 2003 a 2014, en las bases de datos SciELO y BVS-Psi, utilizando los términos "evaluación psicológica", "tests psicológicos" y "construcción / adaptación de instrumentos psicológicos". En general, los resultados señalan el impacto positivo del SATEPSI en el área de la evaluación psicológica, observado en el aumento de la cantidad y la calidad de publicaciones, así como también en la mayor variedad de contextos y tests psicológicos utilizados en las investigaciones. Las metas deseadas con la publicación de la Resolución fueron en su mayoría alcanzadas.

Palabras clave:
evaluación psicológica; tests psicológicos; publicaciones científicas

Psychological Assessment (PA) appears as an important work field for psychologists, which not only relates to the creation of evaluating instruments and techniques but, above all, the development of Psychology itself as a science, allowing psychological theories to be objectificated and operationalized (Anastasi & Urbina, 2000Anastasi, A., & Urbina, S. (2000). Fundamentos da testagem psicológica. Fundamentos da testagem psicológica. Porto Alegre: Artmed.; Cross & Alchiei, 2003; Primi, 2010Primi, R., & Nunes, C. H. S. (2010). O Satepsi: Desafios e propostas de aprimoramento. In Conselho Federal de Psicologia (Ed.), Avaliação psicológica: Diretrizes na regulamentação da profissão, 129-148. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ). It is a process aimed at understanding the psychological functioning of individuals with or without the use of psychological tests, in order to build a knowledge that can serve as basis for making decisions that benefit the individual and, if necessary, direct the beginning of treatment (Noronha & Alchieri, 2004Noronha, A. P. P., & Alchieri, J. C. (2004). Conhecimento em avaliação psicológica. Estudos de Psicologia, 21(1), 43-52. doi: 10.1590/S0103-166X2004000100004
https://doi.org/10.1590/S0103-166X200400... ; Urbina, 2007Urbina, S. (2007). Fundamentos da testagem moderna. Porto Alegre: Artmed.).

The PA has occupied a prominent position in the development of the Psychology scenario nationally, contributing significantly to its consolidation in the country, having gone through several phases during its history (Pasquali, 2001Pasquali, L. (2001). Técnicas de exame psicológico - TEP: Manual. São Paulo, SP: Casa do Psicólogo.; 2003Pasquali, L. (2003). Psicometria: Teoria dos testes na psicologia e na educação. Petrópolis, RJ: Vozes.). For example, in the initial period (before 1930), the psychological tests were mostly used by other professionals such as doctors and teachers. With the increasing development of psychology in the country in the following decades, there was a growing interest in psychological testing and research to build instruments grew rapidly (Pasquali, 2003Pasquali, L. (2003). Psicometria: Teoria dos testes na psicologia e na educação. Petrópolis, RJ: Vozes.). Although there were investments for test building, there was a lack in concern in checking scientifically whether such instruments complied with the parameters necessary for use in the national context. Many psychological instruments were therefore used for over thirty years in an abusive manner, without studies to prove their technical and scientific quality.

As a result, between the decades of 1970 and 1990 harsh criticism rose regarding the professional activities in PA, particularly the use of psychological tests, which resulted in decreased use and growth of publications in the area (Augras, 1990Augras, M. (1990). Testes psicológicos e antropologia. Arquivos Brasileiros de Psicologia, 42, 78-87. Retrieved from http://bibliotecadigital.fgv.br/ojs/index.php/abp/article/view/21770/20523
http://bibliotecadigital.fgv.br/ojs/inde... ; Collares, 1994Collares, C. A. L. (1994). O cotidiano escolar patologizado: Espaço de preconceitos e práticas cristalizadas Campinas, SP (Tese de livre docência). Faculdade de Educação da UNICAMP. Retrieved from http://www.bibliotecadigital.unicamp.br/document/?code=000110119&opt=4
http://www.bibliotecadigital.unicamp.br/... ; Mindrisz, 1994Mindrisz, R. K. (1994). A tirania do QI: O quociente de inteligência na caracterização do indivíduo deficiente mental. (Dissertação não de mestrado). PUC, São Paulo, SP.; Patto, 1997Patto, M. H. S. (1997). Para uma crítica da razão psicométrica. Psicologia USP, 8(1), 47-62. Retrieved from http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0103-65641997000100004&lng=en&tlng=pt.10.1590/S0103-65641997000100004.
http://www.scielo.br/scielo.php?script=s... ; Wanderey, 1985). (Wanderley, 1985Wanderley, W. M. (1985). Os testes psicológicos em seleção de pessoal: Análise crítica dos conceitos e procedimentos utilizados. Arquivos Brasileiros de Psicologia, 37(2), 16-31. Retrieved from http://bibliotecadigital.fgv.br/ojs/index.php/abp/article/view/19187
http://bibliotecadigital.fgv.br/ojs/inde... ), for example, questioned the use of psychological tests in professional selection process, pointing unsatisfactory validity indexes for most of them. He argued that there was often no relationship between the characteristics measured by tests and those necessary for proper work performance. He also mentions the standards placed at the time by the US government to control the use of psychological tests in selection processes in the country, such as requirement to provide predictive validity of tests for the work context and present studies that pointed the relationship between the characteristic evaluated and the one necessary for work performance. The author regarded such standards as a strategy that should be adopted in Brazil to control the indiscriminate use of psychological tests in the organizational context. In this sense, (Augras, 1990Augras, M. (1990). Testes psicológicos e antropologia. Arquivos Brasileiros de Psicologia, 42, 78-87. Retrieved from http://bibliotecadigital.fgv.br/ojs/index.php/abp/article/view/21770/20523
http://bibliotecadigital.fgv.br/ojs/inde... ), criticizes the use of tests created and standardized for different cultures, without due care for adaptation. The author cites anthropological studies conducted with minority cultural groups, in which psychological tests were used and showed inconsistent results, thus enabling the confirmation of dominant Western culture superiority ideals.

The construction and use of psychological instruments of dubious scientific quality, the lack of specialized professionals and the shortage of graduation courses qualified to teach PA contributed to the declining interest of professionals for the area, as well as to the harsh criticism carried out by psychologists and other mental health professionals and the general discrediting of the area by the population (Pasquali, 2001Pasquali, L. (2001). Técnicas de exame psicológico - TEP: Manual. São Paulo, SP: Casa do Psicólogo.; 2003Pasquali, L. (2003). Psicometria: Teoria dos testes na psicologia e na educação. Petrópolis, RJ: Vozes.). However, such criticisms served as a warning to the need for consideration on the training and professional practice in the field, working ultimately as a push toward change in this reality.

From the 1990s onwards, researchers and professionals engaged in a concerted effort for change that, according to (Noronha and Reppold, 2010Noronha, A. P., & Reppold, C. T. (2010). Considerações sobre a avaliação psicológica no Brasil. Psicologia Ciência e Profissão, 30, 192-201. doi: 10.1590/S1414-98932010000500009
https://doi.org/10.1590/S1414-9893201000... ), could be initially perceived in many held events and meetings. Other achievements emerged, such as the creation of the Brazilian Institute of Psychological Assessment (IBAP) in 1997, and the Psychological Assessment journal in 2002. The goal was to solve issues perceived in the area by proposing quality improvement on university teaching of professionals in the PA area, the production of research for the development and validation of psychological instruments in the country, among others.

Nevertheless, the turning point in this context of changes was the implementation, by the Conselho Federal de Psicologia (CFP, Brazilian Federal Council of Psychology), of the Consultative Committee on Psychological Evaluation and the Psychological Tests Evaluation System (SATEPSI), started in 2001 and consolidated through Resolution No. 02/2003 (later amended by Resolution no. 05/2012). The SATEPSI can be understood as a national certification system of the scientific and technical quality of psychological tests commercialized in the country (Anache & Corrêa, 2010Anache, A., & Corrêa, F. (2010). As políticas do conselho federal de psicologia para a avaliação psicológica. In Conselho Federal de Psicologia (Ed.), Avaliação psicológica: Diretrizes na regulamentação da profissão, 19-30. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ; Primi & Nunes, 2010Primi, R., & Nunes, C. H. S. (2010). O Satepsi: Desafios e propostas de aprimoramento. In Conselho Federal de Psicologia (Ed.), Avaliação psicológica: Diretrizes na regulamentação da profissão, 129-148. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ). This certification is given on the basis of minimum technical requirements (validity studies, accuracy, standardization and relevant theoretical basis), which were defined based on international rules (e.g., American Psychological Association, National Council on Measurement in Education) for the construction of psychological tests (Noronha & Reppold, 2010Noronha, A. P., & Reppold, C. T. (2010). Considerações sobre a avaliação psicológica no Brasil. Psicologia Ciência e Profissão, 30, 192-201. doi: 10.1590/S1414-98932010000500009
https://doi.org/10.1590/S1414-9893201000... ). The SATEPSI, as regulates the CFP Resolution No. 002/2003 through their counsel members, therefore has a duty to ensure the ethical and detailed evaluation of psychological instruments developed or adapted in Brazil relative to its graphic, theoretical, psychometric and ecological quality (Anache & Corrêa, 2010Anache, A., & Corrêa, F. (2010). As políticas do conselho federal de psicologia para a avaliação psicológica. In Conselho Federal de Psicologia (Ed.), Avaliação psicológica: Diretrizes na regulamentação da profissão, 19-30. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ; Noronha & Reppold, 2010Noronha, A. P., & Reppold, C. T. (2010). Considerações sobre a avaliação psicológica no Brasil. Psicologia Ciência e Profissão, 30, 192-201. doi: 10.1590/S1414-98932010000500009
https://doi.org/10.1590/S1414-9893201000... ; Primi & Nunes, 2010Primi, R. (2010). Avaliação psicológica no Brasil: Fundamentos, situação atual e direções para o futuro. Psicologia: Teoria e Pesquisa, 26(25anos), 25-36. Retrieved from http://www.scielo.br/pdf/ptp/v26nspe/a03v26ns.pdf
http://www.scielo.br/pdf/ptp/v26nspe/a03... ).

Initially, the work of the Consultative Committee and the implementation of SATEPSI did not please professionals in the field, since many massively used instruments were evaluated negatively and withdrawn from the market. Thus, many psychologists found themselves facing a dilemma, since they were being confronted with their own work instruments. However, after more than 10 years of its implementation, there seems to be some consensus today about the importance of SATEPSI for the technical, scientific and ethical development of the area in the country, as well as to raise awareness of psychologists themselves on their professional performance (Primi & Nunes, 2010Primi, R., & Nunes, C. H. S. (2010). O Satepsi: Desafios e propostas de aprimoramento. In Conselho Federal de Psicologia (Ed.), Avaliação psicológica: Diretrizes na regulamentação da profissão, 129-148. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ). Some authors have raised data that would support the claims of SATEPSI's importance to the growth of the area, although none of them has, in fact, compared the periods before and after its creation (Anache & Corrêa, 2010Anache, A., & Corrêa, F. (2010). As políticas do conselho federal de psicologia para a avaliação psicológica. In Conselho Federal de Psicologia (Ed.), Avaliação psicológica: Diretrizes na regulamentação da profissão, 19-30. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ; Barroso, 2010Barroso, S. M. (2010). Avaliação psicológica: Análise das publicações disponíveis na SciELO e BVS-Psi. Fractal: Revista de Psicologia, 22(1), 141-154. Retrieved from http://www.uff.br/periodicoshumanas/index.php/Fractal/article/viewArticle/174
http://www.uff.br/periodicoshumanas/inde... ; Joly, Silva, Nunes, & Souza, 2007Joly, M. C. R. A., Silva, M. C. R. D., Nunes, M. F. O., & Souza, M. S. D. (2007). Análise da produção científica em painéis dos congressos brasileiros de avaliação psicológica. Avaliação Psicológica, 6(2), 239-252. Retrieved from http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712007000200013&lng=pt&tlng=pt.
http://pepsic.bvsalud.org/scielo.php?scr... ). (Joly, Silva, Nunes and Souza, 2007Joly, M. C. R. A., Silva, M. C. R. D., Nunes, M. F. O., & Souza, M. S. D. (2007). Análise da produção científica em painéis dos congressos brasileiros de avaliação psicológica. Avaliação Psicológica, 6(2), 239-252. Retrieved from http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712007000200013&lng=pt&tlng=pt.
http://pepsic.bvsalud.org/scielo.php?scr... ) note, for example, an increase, since 2003, in the number of panels and researchers participating in national congresses of psychological assessment, though there yet seems to be a higher concentration of works coming from the Southeast region. Similarly, (Barroso, 2010Barroso, S. M. (2010). Avaliação psicológica: Análise das publicações disponíveis na SciELO e BVS-Psi. Fractal: Revista de Psicologia, 22(1), 141-154. Retrieved from http://www.uff.br/periodicoshumanas/index.php/Fractal/article/viewArticle/174
http://www.uff.br/periodicoshumanas/inde... ) also points to an increase in the publication of theoretical papers and research reports in journals of psychological assessment, with higher concentration in the Southeast and South regions.

In this sense, and considering the lack of published works about the SATEPSI in the country, the aim of this paper is to describe the impact of the creation of SATEPSI for the development of research in the psychological assessment area by comparing the scientific publications in the area in Brazil during two periods, covering the years 1993 to 2002 (before the Resolution No. 02/2003) and 2003 to 2014 (after the Resolution No. 02/2003). This study therefore aims to test the hypothesis that some of the main goals of the Consultative Committee and the SATEPSI were reached after more than ten years of its implementation. These goals would be, according to (Primi and Nunes, 2010Primi, R., & Nunes, C. H. S. (2010). O Satepsi: Desafios e propostas de aprimoramento. In Conselho Federal de Psicologia (Ed.), Avaliação psicológica: Diretrizes na regulamentação da profissão, 129-148. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ), an increase in the number of searches performed in different contexts of psychological assessment, diversification of the studied psychological tests and a raise in quality of the articles produced and published in the area.

Method

Analysis of the scientific production evolution in the area was chosen as the data collection methodology, since it enables the mapping of the main topics discussed in the knowledge area in question as well as the theoretical development gaps in it (Witter, 1999Witter, G. P. (1999). Leitura: Textos e pesquisas. Campinas: Alínea.; 2005Witter, G. P. (2005). Metaciência e psicologia. Campinas: Alínea.).

Virtual Health Library - Psychology (BVS-Psi) and the SciELO (Scientific Electronic Library Online) were consulted, as they are the main electronic databases of Brazil and Latin America. Search keywords used were "psychological evaluation" [avaliação psicológica], "psychological tests" [testes psicológicos] and "construction of psychological instruments" [construção de instrumentos psicológicos]. Once the survey was finished, abstracts were read and analyzed according to the determined inclusion/exclusion criteria. The remaining papers after this initial analysis were read and analyzed in full.

The search was limited to review articles or research reports (empirical)¹ 1 As such were considered empirical articles/research reports those involving field research with individuals, groups, institutions or document analysis research. indexed and available in full text, written in Portuguese, English or Spanish, but published by researchers from Brazilian universities. The survey covered the period from 1993 to 2002 (before the creation of SATEPSI) and 2003 to 2014 (after the creation of SATEPSI). Studies that, despite using psychological tests among the instruments for data collection did not have as their main purpose to promote thought on the psychological assessment practice or to investigate the psychometric characteristics or procedures involved in cross-cultural adaptation of psychological tests themselves were excluded. In order to describe the resulting changes from the creation of SATEPSI in psychological assessment research, the two periods were compared (before and after the system's creation) according to the following indicators: a) number of publications; b) contexts/contemplated subareas; c) number and variability of psychological tests used/investigated; and d) quality of publications. The division of articles by context took as reference the subfields of Psychology set by the Federal Council of Psychology² 2 This division can be found at http://site.cfp.org.br/index.php . In the analysis of papers found by context it was taken into account the objectives of each study, which should clearly present the context studied. Some studies set out to analyze more than one context, or could not be classified among the defined categories. All of these were classified as "Miscellaneous". The analysis of the publications' quality was performed using the Qualis system of Brazilian journals evaluation, maintained by Brazilian Higher Education Personnel Improvement Coordination (Coordenação de Aperfeiçoamento de Pessoal de Ensino Superior, CAPES). This system lists and classifies the media used for the dissemination of intellectual production of graduate programs in the strict sense (masters and doctorate), regarding the circulation scope (local, national or international) and quality (A, B, C), based on assessment area³ 3 Consultation of the Qualis rating for each journal can be performed using the address http://qualis.capes.gov.br/webqualis/principal.seam. For this study it was considered the current classifications of journals and not the classification at the time of the articles' publication. .

Results and Discussion

Having the search keywords defined, there were found 193 articles from 1993 to 2002 and 245 articles from 2003 to 2014. The initial reading of abstracts led to the exclusion of 99 articles published between 1993 and 2002 and 64 articles published between 2003 and 2014, due to most of them not discussing or producing data on the psychological assessment area or psychological tests themselves. Sixty-six articles appeared repeatedly when the keywords "psychological evaluation" and "psychological tests" were used, and were thus considered only once for analysis.

This resulted, therefore, in a total of 62 articles published between 1993 and 2002 and 179 articles published between 2003 and 2014, totaling 241 items in both periods considered for the analysis, which were investigated in greater detail.

Of the 241 articles found, seven were written in English, two of them in period I (1993-2002), and five in period II (2003-2014). Of the 62 articles published in period I, 48 refer to research reports (77.42% of total). Among the studies published in period II were 136 (75.97% of total) research reports. Studies of literature review were more frequent from the year 2001 onward. This increase in theoretical review articles from 2001 overlaps the implementation of SATEPSI and the foundation for the work of the Consultative Commission. One might think that the need for reflection on professional practice and on research conducted so far has motivated experts in psychological assessment to produce literature review articles aimed at resuming old issues and bringing new and essential aspects to strengthen psychological assessment in Brazil.

A comparison between numbers of publications on testing and psychological assessment throughout the study period, as shown in Figure 1, points to a steady growth over the years, which indicates a possible increase in interest from researchers for the quality and adequacy of instruments and theories in psychological assessment. A similar result was also found by (Anache and Corrêa, 2010Anache, A., & Corrêa, F. (2010). As políticas do conselho federal de psicologia para a avaliação psicológica. In Conselho Federal de Psicologia (Ed.), Avaliação psicológica: Diretrizes na regulamentação da profissão, 19-30. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ), who studied the progression in number of theses/dissertations published in the Psychological Assessment field between the years of 1990 and 2010. The authors show that in 1993 there were only 21 theses/dissertations, while in 2010 there were 494, an increase of 473 works in 17 years. Moreover, 2002 was the year with the highest number of publications among all surveyed. This may be a result of the creation of SATEPSI in the year 2001, which starts to question the quality of psychological tests in Brazil, and may have raised demand for this type of study. This finding can be supported by the fact that almost all articles published in 2002 refer to research reports rather than theoretical studies, demonstrating the importance of such system, as pointed (Primi and Nunes, 2010Primi, R., & Nunes, C. H. S. (2010). O Satepsi: Desafios e propostas de aprimoramento. In Conselho Federal de Psicologia (Ed.), Avaliação psicológica: Diretrizes na regulamentação da profissão, 129-148. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ), in increasing the interest of researchers to verify the validity and/or accuracy of psychological assessment instruments that were then used in Brazil.

Figure 1
Progression in the number of publications between the years of 1994 and 2014.

As regards to the regions of origin of the publications, as shown in Figure 2, in period I there is a smaller amount of publications coming from all regions of the country, with the Southeast comprising 71% of publications (n = 45). In period II, the Southeast Region maintains its prevalence of published articles (66%) and the state of São Paulo accounts for over a half of articles published in the period (56.74%). This finding is similar to that found by (Joly et al., 2007Joly, M. C. R. A., Silva, M. C. R. D., Nunes, M. F. O., & Souza, M. S. D. (2007). Análise da produção científica em painéis dos congressos brasileiros de avaliação psicológica. Avaliação Psicológica, 6(2), 239-252. Retrieved from http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712007000200013&lng=pt&tlng=pt.
http://pepsic.bvsalud.org/scielo.php?scr... ) and (Barroso, 2010Barroso, S. M. (2010). Avaliação psicológica: Análise das publicações disponíveis na SciELO e BVS-Psi. Fractal: Revista de Psicologia, 22(1), 141-154. Retrieved from http://www.uff.br/periodicoshumanas/index.php/Fractal/article/viewArticle/174
http://www.uff.br/periodicoshumanas/inde... ), who point to the Southeast and São Paulo, in particular, as center of scientific production (be it in publications or in presenting papers at national conferences) in the Psychological Assessment area during the 1977-2008 period. It is interesting to note, however, that in period II you can see a significant growth of publications of scientific articles coming from other regions of the country, especially the South, authoring 13 publications in period I and 45 articles in period II. Publications of researchers from the Northern, Northeast and Midwest regions start to become part of the national scientific research scenario.

Figure 2
Distribution of papers by region and time of publication (Period I from 1993 to 2002; Period II from 2003 to 2014).

A similar result was found by (Joly et al., 2007Joly, M. C. R. A., Silva, M. C. R. D., Nunes, M. F. O., & Souza, M. S. D. (2007). Análise da produção científica em painéis dos congressos brasileiros de avaliação psicológica. Avaliação Psicológica, 6(2), 239-252. Retrieved from http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712007000200013&lng=pt&tlng=pt.
http://pepsic.bvsalud.org/scielo.php?scr... ) when examining the origin of summaries of three National Congress of Psychological Assessment panels, between the years 2003 and 2007. This suggests a tendency toward decentralization of scientific knowledge production in psychological assessment in Brazil, since the publication of Resolution No. 02/2003 and the creation of SATEPSI. The emphasis on the need for social and cultural adjustments in norms and other parameters of the psychological tests, emphasized in this resolution, appears, therefore, to have contributed greatly to the inclusion and increased interest of researchers from historically marginalized regions in the production of national scientific knowledge.

It was also raised as a hypothesis that, with the advent of SATEPSI, a greater diversity of publications on psychological assessment in different application contexts would be found, when compared to the period prior to this Resolution. This because, especially with regard to the construction and adaptation of psychological tests, the resolution emphasizes the need to clearly present the instrument's purpose and to lift empirical evidence of validity and accuracy to the main contexts for which it was developed (CFP, 2003Conselho Federal de Psicologia (2003). Resolução nº. 02/2003. Conselho Federal de Psicologia. Brasília, DF. Retrieved from http://site.cfp.org.br/legislacao/resolucoes-do-cfp
http://site.cfp.org.br/legislacao/resolu... , 2013Conselho Federal de Psicologia (2013). Relatório do ano temático da avaliação psicológica 2011/2012. Conselho Federal de Psicologia. Brasília, DF. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/12/264tico-de-Avaliacao-Psicologica_versao-final.pdf
http://site.cfp.org.br/wp-content/upload... ).

Figure 3 shows the division of articles by context. It is noted that most of the research is associated with the clinical context/mental health and school, both in Period I and Period II, as already pointed out by (Joly et al., 2007Joly, M. C. R. A., Silva, M. C. R. D., Nunes, M. F. O., & Souza, M. S. D. (2007). Análise da produção científica em painéis dos congressos brasileiros de avaliação psicológica. Avaliação Psicológica, 6(2), 239-252. Retrieved from http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712007000200013&lng=pt&tlng=pt.
http://pepsic.bvsalud.org/scielo.php?scr... ). In period I there was a higher proportion of publications in the school context, which may represent a reflection of the historical influence of this context over psychological assessment practices (Noronha & Reppold, 2010Noronha, A. P., & Reppold, C. T. (2010). Considerações sobre a avaliação psicológica no Brasil. Psicologia Ciência e Profissão, 30, 192-201. doi: 10.1590/S1414-98932010000500009
https://doi.org/10.1590/S1414-9893201000... ; Pasquali, 2001Pasquali, L. (2001). Técnicas de exame psicológico - TEP: Manual. São Paulo, SP: Casa do Psicólogo.; 2003Pasquali, L. (2003). Psicometria: Teoria dos testes na psicologia e na educação. Petrópolis, RJ: Vozes.). In Period II, it is perceived a proportional increase in number of publications in the clinical setting and mental health, pointing to a change in the interests of researchers, who are giving more attention to the assessment of mental disorders and their impact on quality of life. It is important to note, also, the significant increase in publications in Organizational and Work Psychology and in Vocational Guidance, as well as the emergence of studies in the context of transit and forensics. These data support the hypothesis of a current trend to ward diversification of the study contexts of psychological assessment, although some still require more research (CFP, 2013Conselho Federal de Psicologia (2013). Relatório do ano temático da avaliação psicológica 2011/2012. Conselho Federal de Psicologia. Brasília, DF. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/12/264tico-de-Avaliacao-Psicologica_versao-final.pdf
http://site.cfp.org.br/wp-content/upload... ; Ferreira & Santos, 2010Ferreira, M. C., & dos Santos, A. A. A. (2010). A avaliação psicológica no contexto organizacional e do trabalho. In Conselho Federal de Psicologia (Ed.). Avaliação psicológica: Diretrizes na regulamentação da profissão. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ).

Figure 3
Percentage of articles published in periods I (1993-2002) and II (2003-2014) by context in psychological assessment.

Finally, it was worked with the hypothesis that in the second period reported (2003-2014) there would be found a greater amount of psychological tests being studied and used by researchers throughout the national territory. The starting assumption was that SATEPSI would act as an incentive to investigate and use psychological tests, as well as recognition of the effort made by researchers to develop studies up to date with new psychological tests and those already existing in Brazil. A total of 44 different instruments were mentioned in period I (1993-2002) and 108 cited in period II (2003-2014). Table 1 provides an overview in absolute numbers of the most commonly used tests, with an absolute frequency equal or higher than 04 in at least one of the periods. The other tests mentioned in a frequency of 03 or less are not included in Table 1 due to space limitations.

Overall, the most frequently cited tests, when considering both periods, were the BPR-5, Human Figure Drawing, Pfister and Rorschach, the first two being cognitive tests and the others personality ones. The standard for tests most frequently cited in articles are also the ones most often cited in panel abstracts at scientific meetings in the area, as point (Joly et al., 2007Joly, M. C. R. A., Silva, M. C. R. D., Nunes, M. F. O., & Souza, M. S. D. (2007). Análise da produção científica em painéis dos congressos brasileiros de avaliação psicológica. Avaliação Psicológica, 6(2), 239-252. Retrieved from http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712007000200013&lng=pt&tlng=pt.
http://pepsic.bvsalud.org/scielo.php?scr... ). Interestingly, the tests used in research and cited in articles and abstracts match those best known by professionals and psychology students, according to (Noronha, Beraldo and Oliveira, 2003Noronha, A. P. P., Beraldo, F. N. D. M., & Oliveira, K. L. D. (2003). Instrumentos psicológicos mais conhecidos e utilizados por estudantes e profissionais de Psicologia. Psicologia Escolar e Educacional, 7(1), 47-56. Retrieved from http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1413-85572003000100005&lng=pt&tlng=pt
http://pepsic.bvsalud.org/scielo.php?scr... ) and (Noronha et al., 2002Noronha, A. P. P., Oliveira, A. F. D., Cobêro, C., Paula, L. M. D., Cantalice, L. M., Guerra, P. B. D. C.,... & Felizatti, R. (2002). Instrumentos psicológicos mais conhecidos por estudantes do sul de Minas Gerais. Avaliação Psicológica, 1(2), 151-158. Retrieved from http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712002000200008&lng=pt&tlng=es
http://pepsic.bvsalud.org/scielo.php?scr... ). It is essential to emphasize that there was an increase in the amount of tests used in research when comparing the two periods here presented (Table 1). This result points to an interest of Brazilian researchers to study other constructs and different instruments of psychological testing. Such increase in variability of instruments studied in Brazil can serve as a facilitator for professionals and researchers in the measurement of psychological characteristics, allowing the use of different instruments to assess the same construct, which would reduce the effects of practice in cases of multiple applications.

As regarding to the quality of publications, it is observed that, in both periods, most of the publications (66% and 84%, respectively) were made in magazines currently considered of high quality (Qualis A1 and A2). The percentage of publications in Qualis B and C journals was still significant in period I (around 34%), while these were the minority in period II (about 16%). Attention should be paid to the fact that, until 1997, there was no evaluation system for publications in Brazil whatsoever. It was only in 1998 that Qualis was implemented by CAPES (Frigeri, 2012Frigeri, M. (2012). Entendendo o qualis: Um estudo sobre a avaliação dos periódicos brasileiros científicos brasileiros (Dissertação de mestrado). Universidade Estadual de Campinas. Instituto de Geociências. Retrieved from http://www.bibliotecadigital.unicamp.br/document/?code=000875213
http://www.bibliotecadigital.unicamp.br/... ; Frigeri & Monteiro, 2010Frigeri, M., & Monteiro, M. S. A. (2010). Implantação do SEER na Revista Brasileira de Inovação: Melhora da qualidade no processo editorial? Trabalho apresentado no I Encontro Nacional de Usuários do Sistema Eletrônico de Editoração de Revistas. Brasília, DF. Retrieved from http://euseer.ibict.br/index.php/iieuseer/euseer1/paper/view/59
http://euseer.ibict.br/index.php/iieusee... ), which may have led researchers to look for higher quality journals to publish their studies. Either way, this result seems to indicate a tendency for researchers to publish their work, whether of theoretical or empirical nature, in more highly rated journals.

As an additional way to assess the quality of published articles in both periods, an analysis was made of the average impact factor (IF) of each. This analysis was used since the current Qualis is not a good indicator of the quality of articles published in a journal five or ten years ago, as a journal previously rated A2 can be currently A1. To calculate the IF, the numbers of constant citations in Google scholar was divided by the number of years since publication of each article. Thus, for instance, an article written in 2001 that had had 20 quotes has an IF of 1.54, when dividing 20 citations by 13 years since publication (2014-2001). Studies published in 2014 were excluded, as they had not yet been cited. The IF average for period I was 2.65 (SD = 5.92) and for period II was 2.22 (SD-2.27). Mean comparison test (Mann-Whitney test for independent samples) performed using the SPSS statistical software, version 22, showed no significant differences between the two periods regarding IF (p = 0.688). One must consider, however, that the articles published in recent years may not have showed their real impact on national publications, as the interval between their publication and consequential citations was lower than in period II.

Conclusion

Metascience as practice to investigate the scientific production in a given area of knowledge has been seen as crucial methodological strategy to provide easy access to researchers to the large volume of information available in their area (Joly et al., 2007Joly, M. C. R. A., Silva, M. C. R. D., Nunes, M. F. O., & Souza, M. S. D. (2007). Análise da produção científica em painéis dos congressos brasileiros de avaliação psicológica. Avaliação Psicológica, 6(2), 239-252. Retrieved from http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712007000200013&lng=pt&tlng=pt.
http://pepsic.bvsalud.org/scielo.php?scr... ). Studies of scientific production allow not only to identify the subjects that have been receiving greater prominence in academia, but to also identify gaps in scientific knowledge production (Witter, 2005Witter, G. P. (2005). Metaciência e psicologia. Campinas: Alínea.). Thus, this study aimed primarily to describe the impact of SATEPSI and its consequences for research in Psychological Assessment, through a review of the scientific literature that compared two periods - before and after the creation of SATEPSI.

Overall, the results revealed a positive impact of the Consultative Committee and SATEPSI creation for the production of knowledge in the psychological assessment area. There was an increasing amount of publications as well as in their quality. There also seems to have been a change in the concentration of these publications among Brazilian regions. Production remains largely concentrated in the Southeast, followed by the South, in both periods considered. A result which is line with the observations made by (Souza Filho, Belo, and Gouveia, 2016Souza Filho, M. L. S., Belo, R., & Gouveia, V. V. (2006). Testes psicológicos: Análise da produção científica brasileira no período 2000-2004. Psicologia Ciência e Profissão, 26(3), 478-489. doi: 10.1590/S1414-98932006000300011
https://doi.org/10.1590/S1414-9893200600... ) and (Barroso, 2010Barroso, S. M. (2010). Avaliação psicológica: Análise das publicações disponíveis na SciELO e BVS-Psi. Fractal: Revista de Psicologia, 22(1), 141-154. Retrieved from http://www.uff.br/periodicoshumanas/index.php/Fractal/article/viewArticle/174
http://www.uff.br/periodicoshumanas/inde... ). The increased participation of other Brazilian regions in the psychological assessment production, especially in the Northeast after the year 2003, is, however, noteworthy. It has been found, moreover, a diversification of research contexts in psychological assessment after 2003 and a greater variety of instruments reported in them.

Together, these results confirm that, as pointed out by (Primi and Nunes, 2010Primi, R., & Nunes, C. H. S. (2010). O Satepsi: Desafios e propostas de aprimoramento. In Conselho Federal de Psicologia (Ed.), Avaliação psicológica: Diretrizes na regulamentação da profissão, 129-148. Retrieved from http://site.cfp.org.br/wp-content/uploads/2012/07/Diretrizes.pdf
http://site.cfp.org.br/wp-content/upload... ), the creation of the Consultative Committee and SATEPSI was indeed a turning point for the Brazilian psychological assessment. At least with regard to the production of knowledge in the area, there is evidence that the creation of SATEPSI seems to have enabled the development of national psychological assessment, which then sought to meet international standards of rigor in the construction, evaluation, marketing and use of psychological tests. The growing involvement of researchers from different Brazilian states and universities, as previously reported by (Hutz & Bandeira, 2003Hutz, C. S., & Bandeira, D. R. (2003). Avaliação psicológica no Brasil: Situação atual e desafios para o futuro. In O. H. Yamamoto & V. V. Gouveia (Eds.), Construindo a psicologia brasileira: Desafios da ciência e prática psicológica (pp. 621-277). São Paulo, SP: Casa do Psicólogo.), signals for the establishment of policies more committed to ethics and scientific rigor.

Despite the creation of the Consultative Committee and SATEPSI having contributed greatly to a change in the theoretical and practical concepts of work in psychological assessment, these measures have evidently not solved all the issues and flaws existing in the area. The Consultative Committee, along with Brazilian professionals and researchers, needs to promote a broad debate that focus on the matters of professional education, the establishment of more rigorous criteria for evaluating the quality of psychological tests, the improvement of discussions on human rights in the practice and research of psychological assessment and ethical aspects concerning the use of psychological instruments. It is also expected that this study raises other considerations that assist in the continuation of psychological assessment growth in Brazil, for through the joint efforts of researchers and professionals the psychological assessment field may respond more effectively to its continuously faced challenges. Although this study presents as limitations the research in only two virtual databases, filtering only the articles from indexed periods and with full text available online, it is considered that its proposed objective has been reached. Future studies on the impact of SATEPSI over the growth of national scientific research in psychological assessment should consider other important sources of information on domestic production such as theses/dissertations banks, annals of the country's major psychological assessment events, ongoing research projects in the area, lattes curriculum of researchers and working groups enrolled in the meetings of the National Association of Research and Postgraduate Studies in Psychology (ANPEPP). Moreover, it would be important to consult the researchers and professionals of the area about their perceptions about the impacts and future directions of the Consultative Committee and SATEPSI.

References

Publication Dates

History