IA y evaluación

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial evaluación creación de materiales

Temas IA: IA y evaluación IA y creación de materiales modelos de lenguaje (LLM)

Resumen:

Large language models (LLMs) that power chatbots such as ChatGPT have capabilities across numerous domains. Teachers and students have been increasingly using chatbots in science, technology, engineering, and mathematics (STEM) subjects in various ways, including for assessment purposes. However, there has been a lack of systematic investigation into LLMs’ capabilities and limitations in automatically generating items for STEM subject assessments, especially given that LLMs can hallucinate and may risk promoting misconceptions and hindering conceptual understanding. To address this, we systematically investigated LLMs' conceptual understanding and quality of working in generating question...

“ChatGPT is the companion, not enemies”: EFL learners’ perceptions and experiences in using ChatGPT for feedback in writing

Mark Feng Teng (2024)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial ChatGPT expresión escrita

Temas IA: IA y enseñanza-aprendizaje de lenguas chatGPT IA y evaluación

Resumen:

The present research was to bridge a research gap in comprehensively understanding students’ perceptions and experiences in utilizing ChatGPT for their writing process in English as a foreign language (EFL) context. The participants were 45 EFL learners in Macau. The study aimed to explore the potential impact of ChatGPT on writing, as well as their perceptions and experiences of ChatGPT in generating feedback for writing. A mixed-methods approach was employed, with quantitative data collected from a questionnaire and qualitative insights drawn from interviews conducted after a semester-long writing course. The findings supported the significant positive effects of AI assistance on writing...

AI-assisted knowledge assessment techniques for adaptive learning environments

Sein Minn (2022)

Detalles Cerrar ✕

Tipo: artículo

Metodología: revisión bibliográfica

Temas: inteligencia artificial evaluación tecnología educativa

Temas IA: IA y evaluación revisión de bibliografía

Resumen:

The growth of online learning, enabled by the availability on the Internet of different forms of didactic materials such as MOOCs and Intelligent Tutoring Systems (ITS), in turn, increases the relevance of personalized instructions for students in an adaptive learning environment. There are increasing interests as well as many challenges in the application of Artificial Intelligence (AI) techniques in educational settings to provide adaptive learning content to learners. Knowledge assessment is necessary for providing an adaptive learning environment. A student model serves as a fundamental building block of knowledge assessment in an adaptive learning environment. This paper intends to...

Stretching AI's reach: Assessing an AI-driven feedback system for extended academic writing

Jim Lo, Christy Wong, Agnes Ng (2026)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial escritura académica feedback/retroalimentación

Temas IA: IA y enseñanza-aprendizaje de lenguas IA y evaluación modelos de lenguaje (LLM)

Resumen:

Advances in large language models (LLMs) enable timely and scalable writing evaluation. Previous research has shown that LLM-driven conversational systems, such as ChatGPT, can provide feedback on short essays. However, it is unclear whether AI can effectively evaluate more demanding genres. This study investigates a custom-built writing feedback system developed at a Hong Kong university that uses OpenAI's GPT-4 Turbo (0125-preview) to provide rubric-based feedback on a 1500-word academic report. Guided by a detailed, rubric-aligned prompt, the system generated 333 feedback items from 37 undergraduates, which were analysed for accuracy, tone, and inclusion of examples. The analysis showed...

How well can LLMs grade essays in Arabic?

Rayed Ghazawi, Edwin Simpson (2025)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial expresión escrita evaluación

Temas IA: IA y enseñanza-aprendizaje de lenguas IA y evaluación modelos de lenguaje (LLM)

Resumen:

This research assesses the effectiveness of state-of-the-art large language models (LLMs), including ChatGPT, Llama, Aya, Jais, and ACEGPT, in the task of Arabic automated essay scoring (AES) using the AR-AES dataset. It explores various evaluation methodologies, including zero-shot, few-shot in context learning, and fine-tuning, and examines the influence of instruction-following capabilities through the inclusion of marking guidelines within the prompts. A mixed-language prompting strategy, integrating English prompts with Arabic content, was implemented to improve model comprehension and performance. Among the models tested, ACEGPT demonstrated the strongest performance across the...

Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability

Austin Pack, Alex Barrett, Juan Escalante (2024)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial expresión escrita evaluación

Temas IA: IA y enseñanza-aprendizaje de lenguas IA y evaluación modelos de lenguaje (LLM)

Resumen:

Advancements in generative AI, such as large language models (LLMs), may serve as a potential solution to the burdensome task of essay grading often faced by language education teachers. Yet, the validity and reliability of leveraging LLMs for automatic essay scoring (AES) in language education is not well understood. To address this, we evaluated the cross-sectional and longitudinal validity and reliability of four prominent LLMs, Google's PaLM 2, Anthropic's Claude 2, and OpenAI's GPT-3.5 and GPT-4, for the AES of English language learners' writing. 119 essays taken from an English language placement test were assessed twice by each LLM, on two separate occasions, as well as by a pair of...

Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback increases secondary students’ text revision, motivation, and positive emotions

Jennifer Meyer, Thorben Jansen, Ronja Schiller (2024)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial feedback/retroalimentación expresión escrita

Temas IA: IA y enseñanza-aprendizaje de lenguas IA y evaluación modelos de lenguaje (LLM)

Resumen:

Writing proficiency is an essential skill for upper secondary students that can be enhanced through effective feedback. Creating feedback on writing tasks, however, is time-intensive and presents a challenge for educators, often resulting in students receiving insufficient or no feedback. The advent of text-generating large language models (LLMs) offers a promising solution, namely, automated evidence-based feedback generation. Yet, empirical evidence from randomized controlled studies about the effectiveness of LLM-generated feedback is missing. To address this issue, the current study compared the effectiveness of LLM-generated feedback to no feedback. A sample of N = 459 upper secondary...

A hybrid reasoning framework for artificial intelligence assessment rubric generation in human and automated contexts: Evidence from an undergraduate programming course

Pedro C. Mendonça, Filipe Quintal, Mário Figueiredo (2026)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial evaluación grandes modelos de lenguaje

Temas IA: IA y evaluación modelos de lenguaje (LLM) estudio empírico

Resumen:

Developing assessment rubrics is resource-intensive, limiting frequent formative assessment. This study introduces HARMOGEN-R (Hybrid Assessment Rubric Model Generation with Reasoning), a framework that uses reasoning-enhanced Large Language Models (LLMs) for initial rubric generation and standard models for synthesis. It supports two generation approaches, namely, structured, with predefined evaluation criteria, and free-form, with Artificial Intelligence (AI) defined criteria. Using a within-subjects design, four AI-generated rubrics and a human-created baseline were compared across 308 open-ended responses (text and code) from three formative programming assignments in an undergraduate...

Deep learning and fuzzy algorithm in improving the effectiveness of college English translation teaching

Biao Kong, Che He (2025)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial traducción enseñanza/aprendizaje de lenguas

Temas IA: IA y enseñanza-aprendizaje de lenguas IA y evaluación estudio empírico

Resumen:

With the development of globalization, college English translation teaching is faced with the challenge of dealing with complex language structure and cross-cultural content. The traditional teaching methods are inadequate in evaluating translation quality and correcting translation errors, which is difficult to meet the actual needs of students. This study combines deep learning and fuzzy algorithm to improve the effect of translation teaching. Based on the data analysis of 387 students, the BiLSTM model is used to train translation tasks, and the fuzzy inference system is used to evaluate translation quality comprehensively. The results show that this method improves students’ translation...

Machine learning based feedback on textual student answers in large courses

Jan Philip Bernius, Stephan Krusche, Bernd Bruegge (2022)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial feedback/retroalimentación evaluación

Temas IA: IA y evaluación estudio empírico

Resumen:

Many engineering disciplines require problem-solving skills, which cannot be learned by memorization alone. Open-ended textual exercises allow students to acquire these skills. Students can learn from their mistakes when instructors provide individual feedback. However, grading these exercises is often a manual, repetitive, and time-consuming activity. The number of computer science students graduating per year has steadily increased over the last decade. This rise has led to large courses that cause a heavy workload for instructors, especially if they provide individual feedback to students. This article presents CoFee, a framework to generate and suggest computer-aided feedback for...