Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial

Resumen:

Texto completo

Large language models (LLMs) like ChatGPT have shown promise in generating educational content, including questions. This study evaluates the effectiveness of LLMs in classifying and generating educational-type questions. We assessed ChatGPT's performance using a dataset of 4,959 user-generated questions labeled into ten categories, employing various prompting techniques and aggregating results with a voting method to enhance robustness. Additionally, we evaluated ChatGPT's accuracy in generating type-specific questions from 100 reading sections sourced from five online textbooks, which were manually reviewed by human evaluators. We also generated questions based on learning objectives and...

Ver ficha completa

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial

Resumen:

Texto completo

This study investigates the reliability of Large Language Models (LLMs) in evaluating the quality of written lesson plans from pre-service teachers. A total of 32 lesson plans, each ranging from 60 to 100 pages, were collected during a teacher internship program for civic education pre-service teachers. Using the ChatGPT-o1 reasoning model, we compared a human expert standard with LLM coding outcomes in a two-phase explanatory sequential mixed-methods design that combined quantitative reliability testing with a qualitative follow-up analysis to interpret inter-dimensional patterns of agreement. Quantitatively, overall reliability across six qualitative components of written lessons plans (...

Ver ficha completa

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial

Resumen:

Texto completo

In large-scale international writing assessments, human raters often exhibit inconsistency, undermining reliability and validity. Large language models (LLMs) offer a potential solution, but their assessment reliability remains underexplored. This study employed generalizability theory and many-facet Rasch modeling to compare human and LLM raters across three essay genres (4315 samples).Findings reveal that human-LLM discrepancies stem from fundamental evaluation differences, with minimal divergence in key-point scoring. Humans excel in holistic scoring scenarios but struggle with complex analytical rubrics where LLMs demonstrate advantages. While LLMs perform adequately for relative...

Ver ficha completa

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial

Resumen:

Texto completo

Currently, many generative Artificial Intelligence (AI) tools are being integrated into the educational technology landscape for instructors. Our paper examines the potential and challenges of using Large Language Models (LLMs) to code student-generated content in online discussions based on intended learning outcomes and how instructors could use this to assess the intended and enacted learning design. If instructors were to rely on LLMs as a means of assessment, the reliability of these models to code the data accurately is crucial. Employing a diverse set of LLMs from the GPT family and prompting techniques on an asynchronous online discussion dataset from a blended-learning bachelor-...

Ver ficha completa

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial

Resumen:

Texto completo

Developing assessment rubrics is resource-intensive, limiting frequent formative assessment. This study introduces HARMOGEN-R (Hybrid Assessment Rubric Model Generation with Reasoning), a framework that uses reasoning-enhanced Large Language Models (LLMs) for initial rubric generation and standard models for synthesis. It supports two generation approaches, namely, structured, with predefined evaluation criteria, and free-form, with Artificial Intelligence (AI) defined criteria. Using a within-subjects design, four AI-generated rubrics and a human-created baseline were compared across 308 open-ended responses (text and code) from three formative programming assignments in an undergraduate...

Ver ficha completa

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial

Resumen:

Texto completo

The accessibility of advanced Artificial Intelligence-based tools, like ChatGPT, has made Large Language Models (LLMs) readily available to students. These LLMs can generate original written content to assist students in their academic assessments. With the rapid adoption of LLMs, exemplified by the popularity of OpenAI's ChatGPT, there is a growing need to explore their application in education. Few studies examine students' use of LLMs as learning tools. This paper focuses on the application of ChatGPT in engineering higher education through an in-depth case study. It investigates whether engineering students can generate high-quality university essays with LLMs assistance, whether...

Ver ficha completa

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial

Resumen:

Texto completo

This study presents the results of the implementation of Generative-AI Interactive Textbook, an intelligent textbook built on a large language model based on GPT-4, which is directly integrated into the subject of Electrical Engineering course and provides interactive, adaptive teaching functions. The textbook was deployed in the 2024/2025 academic year, and the data was compared with the summary results from 2021 to 2024. The sample consisted of 736 students. The results suggest that the effects associated with the introduction of GenAI-T vary depending on the type of assessment and the analytical context. In multi-year analyses, a statistically significant improvement in mid-term...

Ver ficha completa

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial

Resumen:

Texto completo

Learning outcomes are clear and concise statements that describe what students should be able to do or know at the end of a particular course. These statements are crucial in instructional planning, curriculum development, and assessment of student progress and learning. Although there is no universal guidance on how to develop learning outcomes, Bloom's taxonomy is one widely used framework that helps instructors develop outcomes that reflect different levels of thinking, from basic remembering to creative problem-solving. This study investigates the potential of generative AI, specifically GPT-4, in classifying course learning outcomes according to their respective cognitive levels within...

Ver ficha completa

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial

Resumen:

Texto completo

The importance and challenge of Algebra learning is widely recognized, with students across the U.S. facing difficulties due to the subject's complexity. While extensive research has focused on enhancing Algebra learning in K-12 education, the reusability, scalability, and effectiveness of the strategies employed (e.g., manual interventions and digital tutoring platforms) remain limited. Conversational AI (ConvAI), enabled by the advancement of large language models (LLMs), emerges as a potential tool for automatic, personalized, and effective student support. However, ethical concerns surrounding diversity, safety, sentiment, and stereotype associated with ConvAI are prominent, and...

Ver ficha completa

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial

Resumen:

Texto completo

Automated scoring (AS) has become increasingly prevalent in educational measurement. However, applying it to international reading assessments remains challenging, particularly due to the length and complexity of the required prompting, driven by the need to include lengthy reading passages and detailed scoring guides. Processing these lengthy inputs results in high computational costs and may impede the performance of large language models (LLMs). This study explored the potential of optimizing AS with prompt compression using OpenAI's LLM, GPT-4o. Our results show that prompt compression significantly reduces the length of reading passages and scoring guides while maintaining their...

Ver ficha completa