Validity evidence based on response processes for an instrument assessing functional thinking
DOI:
https://doi.org/10.1590/S1678-4634202551287163esKeywords:
Assessment, Validity evidence, Early algebraic thinking, Functional thinking, Primary educationAbstract
Various studies from the perspective of early algebra have employed written tests to assess students’ functional thinking. However, there has been little research on the validity of the interpretations derived from these instruments. The purpose of this study was to validate the scores of an instrument designed to assess functional thinking in third-grade students by providing evidence based on response processes. A test and a retrospective cognitive interview were administered to 24 third-grade students from a school in Santiago, Chile. Responses from both the test and the interview were coded and compared to assess the agreement between the interpretations derived from each instrument. The results showed a strong and statistically significant correlation in overall performance and revealed that the item with the greatest divergence between the test and the interview involved describing the relationship between variables using natural language. In conclusion, the instrument offers a reliable means of capturing students’ functional thinking, although it may underestimate their performance when expressing functional rules using natural language. Future research could use this instrument while incorporating brief interviews for students in the numerical-functional category, in order to more accurately capture their ability to generalize in natural language without relying on their writing skills.
Downloads
References
AERA; APA; NCME. Standards for Educational & Psychological Testing. Washington, DC: American Educational Research Association, 2014.
BLACKSTOCK-BERNSTEIN, Anne; WOODBRIDGE, Amy y BAILEY, Alison. Comparing elementary students’ explanatory language across oral and written modes. The Elementary School Journal, Chicago, v.122, n. 3, p. 315-340, 2022. https://doi.org/10.1086/718077
» https://doi.org/10.1086/718077
BLANTON, Maria; BRIZUELA, Barbara; GARDINER, Angela Murphy; SAWREY, Katie; NEWMAN-OWENS, Ashley. Progression in first-grade children’s thinking about variable and variable notation in functional relationships. Educational Studies in Mathematics, v. 95, p. 181-202, 2017. https://doi.org/10.1007/s10649-016-9745-0
» https://doi.org/10.1007/s10649-016-9745-0
BLANTON, Maria; KAPUT, James. Functional thinking as a route into algebra in the elementary grades. In: CAI, Jinfa; KNUTH, Eric (ed.). Early algebraization. Berlin: Springer, 2011. p. 5-23.
BLANTON, Maria; STEPHENS, Ana; KNUTH, Eric; GARDINER, Angela Murphy; ISLER, Isil; KIM, Jee-Seon. The development of children’s algebraic thinking: The impact of a comprehensive early algebra intervention in third grade. Journal for Research in Mathematics Education, Reston, v. 46, n. 1, p. 39-87, 2015. https://doi.org/10.5951/jresematheduc.46.L0039
» https://doi.org/10.5951/jresematheduc.46.L0039
BLANTON, Maria; STROUD, Rena; STEPHENS, Ana; GARDINER, Angela Murphy; STYLIANOU, Despina; KNUTH, Eric; ISLER-BAYKAL, Isil; STRACHOTA, Susanne. Does early algebra matter? The effectiveness of an early algebra intervention in grades 3 to 5. American Educational Research Journal, Washington, DC., v. 56, n. 5, p. 1930-1972, 2019. https://doi.org/10.3102/0002831219832301
» https://doi.org/10.3102/0002831219832301
BRIZUELA, Barbara; BLANTON, Maria; SAWREY, Katharine; NEWMAN-OWENS, Ashley; GARDINER, Angela Murphy. Children’s use of variables and variable notation to represent their algebraic ideas. Mathematical Thinking and Learning, Cham, v. 17, n. 1, p. 34-63, 2015. https://doi.org/10.1080/10986065.2015.981939
» https://doi.org/10.1080/10986065.2015.981939
CAICEDO, Estefanía; ZALAZAR-JAIME, Mauricio Federico. Entrevistas cognitivas: revision, directrices de uso y aplicación en investigaciones psicológicas. Avaliação Psicológica, São Paulo, v. 17, n. 3, p. 362-370, 2018. https://doi.org/10.15689/ap.2018.1703.14883.09
» https://doi.org/10.15689/ap.2018.1703.14883.09
CANADAS, María; BLANTON, Maria; BRIZUELA, Barbara. Special issue on early algebraic thinking. Journal for the Study of Education and Development, Madrid, v. 42, n. 3, p. 469-478, 2019. https://doi.org/10.1080/02103702.2019.1638569
» https://doi.org/10.1080/02103702.2019.1638569
CASTILLO, Miguel; PADILLA, José Luís. How cognitive interviewing can provide validity evidence of the response processes to scale items. Social Indicators Research, Dordrecht, v. 114, p. 963-975, 2012. https://doi.org/10.1007/s11205-012-0184-8
» https://doi.org/10.1007/s11205-012-0184-8.
CERDA L, Jaime; VILLARROEL DEL P, Luis. Evaluación de la concordancia inter-observador en investigación pediátrica: Coeficiente de Kappa. Revista Chilena de Pediatría, Santiago de Chile, v. 79, n. 1, p. 54-58, 2008.
CHIMONI, María; PITTA-PANTAZI, Demetra; CHRISTOU, Constantinos. The impact of two different types of instructional tasks on students’ development of early algebraic thinking (El impacto de dos tipos diferentes de tareas instruccionales en el desarrollo del pensamiento algebraico temprano de los estudiantes). Journal for the Study of Education and Development, Madrid, v. 44, n. 3, p. 503-552, 2021. https://doi.org/10.1080/02103702.2020.1778280
» https://doi.org/10.1080/02103702.2020.1778280
CONRAD, Frederick G.; BLAIR, Johnny. Sources of error in cognitive interviews. The Public Opinion Quarterly, Oxford, v. 73, n. 1, p. 32-55, 2009. https://doi.org/10.1093/poq/nfp013
» https://doi.org/10.1093/poq/nfp013
FÖRSTER, Carla; ZEPEDA, Sandra; NUÑEZ, Claudio. Instrumentos para la evaluación de aprendizajes, ¿con qué evaluar? In: FÖRSTER, Carla E. (ed.). El poder de la evaluación en el aula: mejores decisiones para promover aprendizaje. Santiago de Chile: Universidad Católica de Chile, 2017. p. 177-230.
GOÑI-CERVERA, Juncal; CAÑADAS, María; POLO-BLANCO, Irene. Generalizations in students with autism spectrum disorder: an exploratory study of strategies. ZDM-Mathematics Education, Berlin, v. 54, n. 6, p. 1333-1347, 2022. https://doi.org/10.1007/s11858-022-01415-w
» https://doi.org/10.1007/s11858-022-01415-w
HUGHES, Elizabeth; RICCOMINI, Paul; LEE, Joo-Young. Investigating written expressions of mathematical reasoning for students with learning disabilities. The Journal of Mathematical Behavior, v. 58, n. 2, 2020. https://doi.org/10.1016/j.jmathb.2020.100775
» https://doi.org/10.1016/j.jmathb.2020.100775
KAPUT, James. What is algebra? What is algebraic reasoning? In: KAPUT, James; CARRAHER, Davis; BLANTON, Maria (ed.). Algebra in the early grades. New York: Taylor and Francis, 2008. p. 5-17.
KENDALL, Maurice. A new measure of rank correlation. Biometrika, Oxford, v. 30, n 1/2, p. 81-93, 1938. https://doi.org/10.2307/2332226
» https://doi.org/10.2307/2332226
KIERAN, Carolyn. The multi-dimensionality of early algebraic thinking: background, overarching dimensions, and new directions. ZDM Mathematics Education, Berlin, v. 54, n. 6, p. 1131-1150, 2022. https://doi.org/10.1007/s11858-022-01435-6
» https://doi.org/10.1007/s11858-022-01435-6
LARSSON, Anneli; GRANHAG, Pär Anders. Interviewing children with the cognitive interview: assessing the reliability of statements based on observed and imagined events. Scandinavian journal of psychology, Stockholm, v. 46, n. 1, p. 49-57. 2005. https://doi.org/10.1111/j.1467-9450.2005.00434.x
» https://doi.org/10.1111/j.1467-9450.2005.00434.x
MASON, John; STEPHENS, Max; WATSON, Anne. Appreciating mathematical structure for all. Mathematics Education Research Journal, Sydney, v. 21, n. 2, p. 10-32, 2009. https://doi.org/10.1007/BF03217543
» https://doi.org/10.1007/BF03217543
MEADOWS, Keith. Cognitive Interviewing Methodologies. Clinical Nursing Research, Thousand Oaks, v. 30, n. 4, p.375-379. 2021. https://doi.org/10.1177/10547738211014099
» https://doi.org/10.1177/10547738211014099
MOSES, Robert y COBB, Charles. Radical equations: math literacy and civil rights. Boston: Beacon Press. 2001.
MUSEUS, Samuel; PALMER, Robert; DAVIS, Ryan; MARAMBA, Dina. Special issue: racial and ethnic minority students’ success in STEM education. ASHE Higher Education Report, Hoboken, v. 36, n. 6, p. 1-140. 2013. https://doi.org/10.1353/rhe.2013.0046
» https://doi.org/10.1353/rhe.2013.0046
NCTM. National Council of Teachers of Mathematics. Principles and standards for school mathematics. Reston: NCTM. 2000.
PADILLA, José Luís; BENÍTEZ, Isabel. Validity evidence based on response processes. Psicothema, Oviedo, v. 26, n. 1, p. 136-144, 2014. https://doi.org/10.7334/psicothema2013.259
» https://doi.org/10.7334/psicothema2013.259
PADILLA, José Luís; LEIGHTON, Jacqueline. Cognitive interviewing and think aloud methods. In: ZUMBO, Bruno; HUBLEY, Anita (ed.). Understanding and investigating response processes in validation research. [S. l.]: Springer International/Springer Nature, 2017. p. 211-228. https://doi.org/10.1007/978-3-319-56129-5_12
» https://doi.org/10.1007/978-3-319-56129-5_12
PINTO, Eder; CAÑADAS, María. Generalizations of third and fifth graders within a functional approach to early algebra. Mathematics Education Research Journal, Sydney, v. 33, n. 1, p. 113-134. 2021. https://doi.org/10.1007/s13394-019-00300-2
» https://doi.org/10.1007/s13394-019-00300-2
RADFORD, Luis. The emergence of symbolic algebraic thinking in primary school. In: KIERAN, Carolyn (ed.). Teaching and learning algebraic thinking with 5- to 12-year-olds. Cham: Springer, 2018. p. 3-25. ICME-13 Monographs. https://doi.org/10.1007/978-3-319-68351-5_1
» https://doi.org/10.1007/978-3-319-68351-5_1
STEPHENS, Ana; FONGER, Nicole; STRACHOTA, Susanne; ISLER, Isil; BLANTON, Maria; KNUTH, Eric; GARDINER, Angela. A learning progression for elementary students’ functional thinking. Mathematical Thinking and Learning, Cham, v. 19. n. 3, p. 143-166, 2017. https://doi.org/10.1080/10986065.2017.1328636
» https://doi.org/10.1080/10986065.2017.1328636
STYLIANOU, Despina; STROUD, Rena; CASSIDY, Michael; KNUTH, Eric; STEPHENS, Ana; GARDINER, Angela; DEMERS, Lindsay. Putting early algebra in the hands of elementary school teachers: examining fidelity of implementation and its relation to student performance. El álgebra temprana en manos del docente de primaria: un análisis de la fidelidad de ejecución y su relación con el rendimiento de los escolares. Journal for the Study of Education and Development, Madrid, v. 42, n. 3, p. 523-569, 2019. https://doi.org/10.1080/02103702.2019.1604021
» https://doi.org/10.1080/02103702.2019.1604021
UREÑA, Jason; RAMÍREZ, Rafael; MOLINA, Marta. Representaciones de la generalización de una relación funcional y el vínculo con la mediación del entrevistador. Journal for the Study of Education and Development, Madrid, v. 42, n. 3, p. 591-614, 2019. https://doi.org/10.1080/02103702.2019.1604020
» https://doi.org/10.1080/02103702.2019.1604020
WILLIS, Gordon. Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks: Sage, 2019.
WILLSON, Stephanie; MILLER, Kristen. Data collection. In: MILLER, Kristen; WILLSON, Stephanie; CHEPP, Valeria; PADILLA, José-Luís (ed.). Cognitive interviewing methodology. Hoboken: Wiley, 2014. p. 15-33.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Educação e Pesquisa
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors assume exclusive responsibility for the concepts expressed in their articles, which do not necessarily reflect the journal’s opinion.
Permission to photocopy all or part of the material published in the journal is granted provided that the original source of publication be assigned.
