Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach
0
🔎
Sign in to get full access
Overview
- This paper presents an approach to automatically assessing learner competencies in intelligent tutoring systems.
- The model is based on Bayesian networks and uses logical gates with uncertainty to simplify the model and enable real-time inference.
- The approach is demonstrated through automating the assessment of an activity designed to test computational thinking skills.
Plain English Explanation
Intelligent tutoring systems aim to provide personalized learning experiences by assessing a student's knowledge and skills. Automated assessment is a key component of these systems, as it allows them to quickly and objectively evaluate a student's competencies.
The researchers in this paper propose a method for deriving a learner model directly from an assessment rubric. An assessment rubric typically describes the different levels of competence a student can demonstrate. The researchers use Bayesian networks, which are a type of statistical model, to represent these competence levels and how they relate to each other.
To make the model simpler and easier to use, the researchers employ "logical gates with uncertainty," also known as "noisy gates." These allow the model to capture the inherent uncertainty in assessing a student's skills without requiring a large number of parameters that would be difficult for experts to provide.
The researchers demonstrate how this approach can be used to automatically assess an activity designed to test a student's computational thinking skills. By starting with the assessment rubric, the model can be quickly set up, making it easier to automate the assessment of various tasks and integrate them into adaptive assessment tools and intelligent tutoring systems.
Technical Explanation
The researchers propose a Bayesian network-based approach to modeling learner competencies directly from an assessment rubric. Assessment rubrics typically describe the different levels of competence a student can demonstrate for a given task or skill.
The Bayesian network model uses logical gates with uncertainty, also known as "noisy gates," to capture the relationships between the competence levels. This reduces the number of parameters required to specify the model, making it easier for experts to elicit and enabling real-time inference in intelligent tutoring systems.
The researchers demonstrate the approach by automating the assessment of an activity designed to test computational thinking skills. They show how the model can be quickly derived from the assessment rubric, allowing for the automated assessment of various tasks and their integration into adaptive assessment tools and intelligent tutoring systems.
Critical Analysis
The researchers acknowledge that their approach relies on the availability of a well-defined assessment rubric, which may not always be the case. Additionally, the use of noisy gates, while effective in reducing the model complexity, may not capture all the nuances of human assessment.
Further research could explore ways to automatically derive assessment rubrics from learning objectives or other sources, reducing the burden on experts. Investigating more sophisticated Bayesian network structures or alternative modeling approaches could also enhance the model's ability to capture the complexities of human assessment.
Conclusion
This research presents a promising approach to automating the assessment of learner competencies in intelligent tutoring systems. By leveraging Bayesian networks and noisy gates, the model can be quickly derived from assessment rubrics, enabling the automated assessment of various tasks and their seamless integration into adaptive assessment tools. This work has the potential to significantly streamline the assessment process and improve the personalization of learning experiences in intelligent tutoring systems.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🔎
0
Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach
Francesca Mangili, Giorgia Adorni, Alberto Piatti, Claudio Bonesana, Alessandro Antonucci
Automatic assessment of learner competencies is a fundamental task in intelligent tutoring systems. An assessment rubric typically and effectively describes relevant competencies and competence levels. This paper presents an approach to deriving a learner model directly from an assessment rubric defining some (partial) ordering of competence levels. The model is based on Bayesian networks and exploits logical gates with uncertainty (often referred to as noisy gates) to reduce the number of parameters of the model, so to simplify their elicitation by experts and allow real-time inference in intelligent tutoring systems. We illustrate how the approach can be applied to automatize the human assessment of an activity developed for testing computational thinking skills. The simple elicitation of the model starting from the assessment rubric opens up the possibility of quickly automating the assessment of several tasks, making them more easily exploitable in the context of adaptive assessment tools and intelligent tutoring systems.
Read more8/6/2024
➖
0
Rubric-based Learner Modelling via Noisy Gates Bayesian Networks for Computational Thinking Skills Assessment
Giorgia Adorni, Francesca Mangili, Alberto Piatti, Claudio Bonesana, Alessandro Antonucci
In modern and personalised education, there is a growing interest in developing learners' competencies and accurately assessing them. In a previous work, we proposed a procedure for deriving a learner model for automatic skill assessment from a task-specific competence rubric, thus simplifying the implementation of automated assessment tools. The previous approach, however, suffered two main limitations: (i) the ordering between competencies defined by the assessment rubric was only indirectly modelled; (ii) supplementary skills, not under assessment but necessary for accomplishing the task, were not included in the model. In this work, we address issue (i) by introducing dummy observed nodes, strictly enforcing the skills ordering without changing the network's structure. In contrast, for point (ii), we design a network with two layers of gates, one performing disjunctive operations by noisy-OR gates and the other conjunctive operations through logical ANDs. Such changes improve the model outcomes' coherence and the modelling tool's flexibility without compromising the model's compact parametrisation, interpretability and simple experts' elicitation. We used this approach to develop a learner model for Computational Thinking (CT) skills assessment. The CT-cube skills assessment framework and the Cross Array Task (CAT) are used to exemplify it and demonstrate its feasibility.
Read more8/6/2024
🐍
0
Intelligent tutoring systems by Bayesian networks with noisy gates
Alessandro Antonucci, Francesca Mangili, Claudio Bonesana, Giorgia Adorni
Directed graphical models such as Bayesian nets are often used to implement intelligent tutoring systems able to interact in real-time with learners in a purely automatic way. When coping with such models, keeping a bound on the number of parameters might be important for multiple reasons. First, as these models are typically based on expert knowledge, a huge number of parameters to elicit might discourage practitioners from adopting them. Moreover, the number of model parameters affects the complexity of the inferences, while a fast computation of the queries is needed for real-time feedback. We advocate logical gates with uncertainty for a compact parametrization of the conditional probability tables in the underlying Bayesian net used by tutoring systems. We discuss the semantics of the model parameters to elicit and the assumptions required to apply such approach in this domain. We also derive a dedicated inference scheme to speed up computations.
Read more9/10/2024
0
RICA2: Rubric-Informed, Calibrated Assessment of Actions
Abrar Majeedi, Viswanatha Reddy Gajjala, Satya Sai Srinath Namburi GNVV, Yin Li
The ability to quantify how well an action is carried out, also known as action quality assessment (AQA), has attracted recent interest in the vision community. Unfortunately, prior methods often ignore the score rubric used by human experts and fall short of quantifying the uncertainty of the model prediction. To bridge the gap, we present RICA^2 - a deep probabilistic model that integrates score rubric and accounts for prediction uncertainty for AQA. Central to our method lies in stochastic embeddings of action steps, defined on a graph structure that encodes the score rubric. The embeddings spread probabilistic density in the latent space and allow our method to represent model uncertainty. The graph encodes the scoring criteria, based on which the quality scores can be decoded. We demonstrate that our method establishes new state of the art on public benchmarks, including FineDiving, MTL-AQA, and JIGSAWS, with superior performance in score prediction and uncertainty calibration. Our code is available at https://abrarmajeedi.github.io/rica2_aqa/
Read more8/9/2024