Think aloud: using cognitive interviewing to validate the PISA assessment of student self-efficacy in mathematics

Article English OPEN
Pepper, D. ; Hodgen, J. ; Lamesoo, K. ; Kõiv, P. ; Tolboom, J. (2016)
  • Publisher: Taylor & Francis
  • Related identifiers: doi: 10.1080/1743727X.2016.1238891
  • Subject: Cognitive interviewing, think aloud, validation, assessment, PISA, international, self-efficacy, mathematics

Cognitive interviewing (CI) provides a method of systematically collecting validity evidence of response processes for questionnaire items. CI involves a range of techniques for prompting individuals to verbalise their responses to items. One such technique is concurrent verbalisation, as developed in Think Aloud Protocol (TAP). This article investigates the value of the technique for validating questionnaire items administered to young people in international surveys. To date, the literature on TAP has focused on allaying concerns about reactivity – whether response processes are affected by thinking aloud. This article investigates another concern, namely the completeness of concurrent verbalisations – the extent to which respondents verbalise their response processes.\ud An independent, exploratory validation of the PISA assessment of student self-efficacy in mathematics by a small international team of researchers using CI with concurrent verbalisation in four education systems (England, Estonia, Hong Kong, and the Netherlands) provided the basis for this investigation. The researchers found that students generally thought aloud in response to each of the items, thereby providing validity evidence of responses processes varying within and between the education systems, but that practical steps could be taken to increase the completeness of concurrent verbalisations in future validations.
  • References (28)
    28 references, page 1 of 3

    American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. 1999. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.

    American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. 2014. US: Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.

    Bandura, Albert. 1986. Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: PrenticeHall.

    Bandura, Albert. 1995. Self-efficacy in Changing Societies. Cambridge: Cambridge University Press.

    Bandura, Albert. 1997. Self-efficacy: The Exercise of Self-Control. New York: W.H. Freeman.

    Bandura, Albert. 2006. “Guide for Constructing Self-Efficacy Scales.” In Self-Efficacy Beliefs of Adolescents, edited by F. Pajares and T. Urdan, 307-337. Greenwich, CT: Information Age.

    Bandura, Albert. 2010. “Self-Efficacy.” In The Corsini Encyclopedia of Psychology, edited by Irving B. Weiner and W. Edward Craighead, 1534-1536. Hoboken, NJ: John Wiley.

    Breakspear, S. 2012. “The Policy Impact of PISA.” OECD Education Working Papers. Paris: OECD.

    Castillo-Díaz, Miguel, and José-Luis Padilla. 2013. “How Cognitive Interviewing Can Provide Validity Evidence of the Response Processes to Scale Items.” Social Indicators Research 114 (3): 963-975.

    Eklöf, Hanna. 2010. “Skill and Will: Test-taking Motivation and Assessment Quality.” Assessment in Education: Principles, Policy & Practice 17 (4): 345-356.

  • Metrics
    No metrics available
Share - Bookmark