Dr. Gary Klein and Joseph Borders of Shadowbox LLC conducted a pilot study to determine if an operator’s mental model could be made explicit or objective. One part of the project uncovered that a correct mental model is not enough for correct diagnosis as an operator who had the knowledge failed to access it. A second part identified that the mental models were composed of four parts: (1) features of the system, (2) how the system can fail, (3) how to operate the system in context, and (4) cognitive barriers.
A mental model is a set of beliefs about how something works. Process control operators rely on mental models and cognitive skills to manage highly complex systems. An operator’s model (i.e., representation) provides the basis for how s/he interprets incoming data and diagnoses problems when they arise. Inaccurate models can be dangerous as they contribute to personnel making flawed assumptions, taking incorrect actions, and/or failing to take actions in critical situations (examples include Three Mile Island, BP Texas City). The purpose of this pilot effort is to develop a tool or systematic method that 1) accurately captures an operator’s mental model (given certain parameters), 2) identifies deficiencies and/or gaps in the model that can be corrected, and 3) determines the cause of those errors (e.g., lack of declarative knowledge, inability to connect beliefs to make inferences). An objective of this research is for trainers and facilitators to have the appropriate information they need to direct more targeted training to assist operators in developing richer mental models. To carry out this effort, MacroCognition will repurpose existing Decision Making Exercise (DMX) and ShadowBox exercises surrounding a basic chemical process, which will also provide an appropriate operational context and bound the analysis to a smaller yet generalizable system. The goal is to generate an efficient methodology that extracts the trainee’s subjective representation of the system, reveals critical deficiencies, and provides useful information for trainers to facilitate more accurate mental models.
Can an operator’s mental model be assessed to identify gaps or errors that can be corrected? If there is an error or gap, is it due to (1) lack of procedural knowledge and/or (2) an inability to connect beliefs in order to make inferences.
Modern manufacturing has led to the design of very complex systems that can have complex interactions requiring highly cognitive skills and dynamic mental models. Erroneous models have been a contributing factor in personnel taking incorrect actions or failing to take actions in critical situations. It is generally recognized that a person’s mental model improves with experience and training. Experts in a given area can generally identify who they feel has a good mental model and those whose mental model is in need of improvement.
An aphorism of quality is that something must be measurable to be controlled. If good mental models are to be created, then mental models must become measurable in some fashion. The state of person’s mental model must move from a subjective assessment to an objective representation. With that representation, there needs to be the ability to identify gaps or deficiencies in the model which can be corrected through some form of training.
MacroCognition LLC will provide the following project deliverables:
1. Written report of the project including the qualitative results of the assessment and suggested methodology for mental model elicitation, development, and representation.
2. Descriptions of the incidents, including the types of decision prompts that we create in collaboration with the process company