EVALUATION OF GENERATIVE ARTIFICIAL INTELLIGENCE MODELS IN TEACHING PROGRAMMING LOGIC: A COMPARATIVE ANALYSIS BETWEEN THE GEMMA AND META LLAMA MODELS

Abstract

This study aims to compare the performance of Generative Artificial Intelligence (GAI) models, specifically Gemma and Meta LLaMA, in the context of teaching programming logic. The research seeks to analyze which model presents better performance considering criteria such as response time, textual consistency, and pedagogical adequacy. To achieve this, an experimental approach was adopted, with the models being executed in a local environment using the LM Studio software, allowing access to their APIs. The evaluation metrics included BLEU, ROUGE, METEOR, and BERTScore, in addition to measuring response time in milliseconds. Furthermore, a subjective evaluation was conducted with the participation of 30 students from the Information Systems course, who analyzed the responses generated by the models based on criteria such as clarity, objectivity, and pedagogical usefulness. The results indicate that Meta LLaMA presented better performance in terms of computational efficiency and structural similarity of responses, while the Gemma model demonstrated greater semantic richness and explanatory capacity in contexts that require deeper conceptual understanding. The human evaluation corroborated these findings, showing a preference for Meta LLaMA in objective questions and for Gemma in questions that require more detailed explanations. It is concluded that the models have complementary characteristics, and their combined use is recommended to enhance the teaching-learning process in programming logic.