modelos de lenguaje (LLM)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial grandes modelos de lenguaje chatbots

Temas IA: análisis de producción de IA modelos de lenguaje (LLM) estudio empírico

Resumen:

Generative Artificial Intelligence (GenAI) chatbots continue to amaze users worldwide with their rapid improvements. These tools possess vast general knowledge and can thus be used in various fields, including education. However, before rolling out these models in pedagogical applications, it is fundamental to understand whether the information provided is reliable and if any of the currently available chatbots are best suited for domain-specific tasks. The objective of this study is to thoroughly investigate these aspects in a specific domain, fermentation, with the overarching goal of providing guidelines to students and teachers to select the best GenAI assistant. To achieve this goal,...

EvalYaks: Instruction tuning datasets and LoRA fine-tuned models for automated scoring of CEFR B2 speaking assessment transcripts

Nicy Scaria, Silvester John Joseph Kennedy, Thomas Latinovich (2026)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial expresión oral evaluación

Temas IA: IA y enseñanza-aprendizaje de lenguas IA y evaluación modelos de lenguaje (LLM)

Resumen:

Relying on human experts to evaluate the Common European Framework of Reference for Languages (CEFR) speaking assessments in an e-learning environment creates scalability challenges, as it limits how quickly and widely assessments can be conducted. We aim to automate the evaluation of CEFR B2 English speaking assessments in e-learning environments from conversation transcripts. First, we evaluate the capability of leading open source and commercial Large Language Models (LLMs) to score a candidate’s performance across various criteria in the CEFR B2 speaking exam in both global and India-specific contexts. Next, we create a new expert-validated, CEFR-aligned synthetic conversational dataset...

Assessing the proficiency of large language models in automatic feedback generation: An evaluation study

Wei Dai, Yi-Shan Tsai, Jionghao Lin (2024)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial feedback/retroalimentación grandes modelos de lenguaje

Temas IA: IA y evaluación modelos de lenguaje (LLM) análisis de producción de IA

Resumen:

Assessment feedback is important to student learning. Learning analytics (LA) powered by artificial intelligence exhibits profound potential in helping instructors with the laborious provision of feedback. Inspired by the recent advancements made by Generative Pre-trained Transformer (GPT) models, we conducted a study to examine the extent to which GPT models hold the potential to advance the existing knowledge of LA-supported feedback systems towards improving the efficiency of feedback provision. Therefore, our study explored the ability of two versions of GPT models – i.e., GPT-3.5 (ChatGPT) and GPT-4 – to generate assessment feedback on students' writing assessment tasks, common in...

The promise and limits of LLMs in constructing proofs and hints for logic problems in intelligent tutoring systems

Sutapa Dey Tithi, Arun Kumar Ramesh, Clara DiMarco (2025)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial grandes modelos de lenguaje feedback/retroalimentación

Temas IA: modelos de lenguaje (LLM) análisis de producción de IA prompts

Resumen:

Intelligent tutoring systems have demonstrated effectiveness in teaching formal propositional logic proofs, but their reliance on template-based explanations limits their ability to provide personalized student feedback. While large language models (LLMs) offer promising capabilities for dynamic feedback generation, they risk producing hallucinations or pedagogically unsound explanations. We evaluated the stepwise accuracy of LLMs in constructing multi-step symbolic logic proofs, comparing six prompting techniques across four state-of-the-art LLMs on 358 propositional logic problems. Results show that DeepSeek-V3 achieved superior performance with upto 86.7 % accuracy on stepwise proof...

Augmenting assessment with AI coding of online student discourse: A question of reliability

Kamila Misiejuk, Rogers Kaliisa, Jennifer Scianna (2024)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial evaluación grandes modelos de lenguaje

Temas IA: IA y evaluación modelos de lenguaje (LLM) estudio empírico

Resumen:

Currently, many generative Artificial Intelligence (AI) tools are being integrated into the educational technology landscape for instructors. Our paper examines the potential and challenges of using Large Language Models (LLMs) to code student-generated content in online discussions based on intended learning outcomes and how instructors could use this to assess the intended and enacted learning design. If instructors were to rely on LLMs as a means of assessment, the reliability of these models to code the data accurately is crucial. Employing a diverse set of LLMs from the GPT family and prompting techniques on an asynchronous online discussion dataset from a blended-learning bachelor-...

Comparing human and LLM ordered coding of qualitative data: How coding differences cascade through temporal analysis

Kamila Misiejuk, Sonsoles López-Pernas, Eduardo Araujo Oliveira (2026)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial metodología de investigación grandes modelos de lenguaje

Temas IA: modelos de lenguaje (LLM) análisis de producción de IA estudio empírico

Resumen:

Automating the process of qualitatively coding text data from learners has been a long-standing ambition of learning analytics researchers since it represents an essential step toward delivering timely and scalable feedback. Automating this process is especially challenging in the case of ordered coding schemes —necessary for temporal analytical methods— where one text utterance can be assigned more than one qualitative code and the assignment order matters. This problem goes beyond multi-class and multi-label classification and, therefore, cannot be easily tackled using classic language models such as BERT. Recent advances in generative artificial intelligence, especially with the advent...

Evaluating the capability of large language models in characterising relational feedback: A comparative analysis of prompting strategies

Wei Dai, Yixin Cheng, Ahmad Ari Aldino (2025)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial feedback/retroalimentación grandes modelos de lenguaje

Temas IA: modelos de lenguaje (LLM) prompts IA y evaluación

Resumen:

Relational feedback is increasingly recognised for its crucial role in enhancing student-instructor relationships and promoting the assimilation of feedback. Despite its significance, no studies have tried to develop automated methods to analyse written feedback for properties of relational feedback to promote its use at scale and assist feedback providers with their relational feedback practices. This automated analysis of relational feedback can be performed as a classification task. However, traditional machine and deep learning methods for text classification typically require extensive human labelling and pose a significant challenge for educators and researchers lacking machine...

AI-enhanced interview simulation in the metaverse: Transforming professional skills training through VR and generative conversational AI

Abdullah Bin Nofal, Hassan Ali, Muhammad Hadi (2025)

Detalles Cerrar ✕

Tipo: artículo

Temas: inteligencia artificial realidad virtual grandes modelos de lenguaje

Temas IA: modelos de lenguaje (LLM) chatbots

Resumen:

Interviewing skills play a pivotal role in the job application and search, professional development to prepare for interviewing is a neglected area of research. Professional training methods are available but are often prohibitively expensive, limiting opportunities primarily to privileged individuals. To bridge this accessibility gap and democratize access to job opportunities, there is a need to develop automated interview simulation platforms. The advent of Generative AI (GenAI) technology, in particular Large Language Models (LLMs), makes this is viable proposition but progress is hindered by the absence of open-source implementations for reproducibility and comparison, as well as the...

A framework for evaluation of large language models in essay assessment: Reliability, alignment, and causal reasoning

Tongxi Liu, Luyao Ye, Wei Yan (2026)

Detalles Cerrar ✕

Tipo: artículo

Metodología: estudio empírico

Temas: inteligencia artificial expresión escrita evaluación

Temas IA: IA y enseñanza-aprendizaje de lenguas IA y evaluación modelos de lenguaje (LLM)

Resumen:

Recent advances in large language models have revitalized research on automated essay evaluation, yet critical concerns remain regarding their reliability, validity, and interpretability. This study presents a comparative analysis of five LLMs (GPT-4.1, Llama 4 Maverick, Gemini 2.5 Flash, Claude Sonnet 4, and DeepSeek R1) in the assessment of long English essays authored by non-native speakers in higher education. The analysis draws on LLM-generated scores for 60 essays to examine (a) intra-model reliability across repeated scoring runs, (b) the degree of alignment between model outputs and expert human ratings, and (c) causal feature dependencies that clarify how linguistic characteristics...

Large language models in education: a systematic review of empirical applications, benefits, and challenges

Yuhong Shi, Kun Yu, Yifei Dong (2026)

Detalles Cerrar ✕

Tipo: artículo

Metodología: revisión bibliográfica

Temas: inteligencia artificial grandes modelos de lenguaje tecnología educativa

Temas IA: modelos de lenguaje (LLM) IA y educación revisión de bibliografía

Resumen:

The rapid advancement of Large Language Models (LLMs), particularly following the release of ChatGPT in November 2022, has significantly transformed educational methodologies. This systematic review aims to synthesize empirical studies published between November 2022 and March 2025, examining the implementation and effectiveness of LLMs in educational settings. 88 empirical studies identified key applications, benefits, and challenges associated with LLM integration in education. Our findings reveal that LLMs are utilized across various educational contexts in six primary applications, with Intelligent Tutoring Systems being particularly prominent. The benefits include improved academic...