testing with quiztalker.py

quiztalker.py is a PythonACT-R model that both presents and speaks a multiple choice test - item by item. A this point it is pedogically useful for showing strategies for taking quizes by explaining the spoken part as a demonstration of inner speech. The teaching point being that when a student rushes in reading the question stem and does not pause before reading the answers then the answers/foils can sometimes change the meaning of the question and induce errors. The mechanism is spreading activation where the meanings of the foils spread to the meaning of the questions stem.

The quiztalker can be used with different voices in the windows SAPI interface. The quiz items can be added from a test development resource http://edutools.ca/xamtoolbox/links2tools.htm
that I developed for a Moodle project by using a little python script http://a.edutools.ca/actr/makequiztalker.py that converts text item texts into PythonACT-R models with a little cut and past of the reformated items. The demonstration example is a set of review questions from introductory psychology http://a.edutools.ca/actr/quiztalker.py

If the quiz items were created from glossary items using the xamtoolbox tools then it would be more feasible to develop a real PythonACT-R model to model the answering of the questions. This would require some access to a Latent Semantic Analysis engine to get the scaled similarities between the question stems and each of the item answers including the foils. The research by Landauer et al has found that using the semantic distances between the stem and the choices can enable reasonable quiz performance (getting a C on test created for introductory psychology material).

The quiztalker.py model provides a simple way to make the discussion of cognitive processing involved in answering multiple choice items more tangible and focused on what I think are the critical aspects: getting the meaning of the stem as a network of spreading activation and then selecting the answer that best fits with the question stem. By relating the cogntive processing involved with personal quiz performance with the xamtoolbox test development process of finding the better items based on analysis of group performance the modeling style of research can be contrasted with the correlational style of research on the same issues. The discussion can be extended into an assignment of completing an online self-test produced with http://edutools.ca/xamtoolbox/xt26test2selftest.htm where a multiple choice test is converted into a fillin the blank style quiz that encourages more active learning than traditional multiple choice testing (at least in my opinion) because the answer much be generated and simple strategies like "pick option C" do not work at all. (In one of my previous courses there was a r=.6 correlation between the number of self-tests completed and final course mark which encourages me to continue to to provide fillin self-test for students to "practice" with).

act-r teaching materials links

ACT-R teaching materials

Best Evidence Encyclopedia [B.E.E.]

Online Reaction Time Demo10

Online Reaction Time Demo10

ByrneClass.pdf (application/pdf Object)

ACT-R Frequently Asked Questions List

stewart.pdf (application/pdf Object)

cs630-l3.pdf (application/pdf Object)

cogsysIII-6.pdf (application/pdf Object)

Lecture Notes in Computer Science:

SpringerLink - Book Chapter

Reading Schedule (Fall) « CogSci 3790: Introduction to Cognitive Science

lecture1.pdf (application/pdf Object)

CogSci 7790 - Cognitive Modeling, Fall 2003

CM10135 Lecture 18 2005

icm-d2.pdf (application/pdf Object)

apcs-1.pdf (application/pdf Object)

Lectures

lecture12.pdf (application/pdf Object)

Psychology Journal Psychology Site Links Glossary News Quizzes Resources Colleges Teachers Students

reaction time

Time to Think?

http://faculty.washington.edu/chudler/java/reacttime.html

Visual Expert Human Factors: Driver Reaction TimeReaction Time Experiment Set Instructor's Page

The JavaScript Source: Games: Reaction Time - Button(Netscape only)

Research

MIT Media Lab: Affective Computing Group

TerryStewart: Cognition and Artificial Systems CGSC5001: Cognition and Artificial Systems course notes

concept maps

Concept Mapping Resource Guide

WebCT Guide - Faculty and Staff: Resources

Personal evaluation of ACT-R models

One of the ways to engage students to think more about ACT-R is to provide a way for them to see how well the theory of the mind corresponds to their personal experience. My approach to this is to start with something really basic like reaction time. Gunzelmann et al (2005) developed a well explained ACT-R model of Psychomotor Vigilance Task (PVT) and worked out the parameters to fit the their model to data from 88 hours of total sleep deprivation. With Terry Stewart's efforts this the ACT-R model was translated into PythonACT-R and so is available for students.

I have extended the model to include age and non-verbal IQ based on a brief review of the literature where both of these variables seem to operate through the speed of the production processing cycle of about 50 miliseconds. Age after 20 seems to slow the cycle and non-verbal IQ above 100 is associated with quicker reaction times and by inference faster processing cycles.

So the plan is to have students adjust the PVTdemoAGEPIQ.py model to make it specific to their individual situation and then run some simulations to see what the model predicts for situations like there own. To evaluate the prediction what is needed is a way for them to do the PVT and generate some comparable data. It is possible to use the environment part of PythonACT-R to collect some data but that requires a working ccmsuite simulation environment. As a backup to that situation (which is not possible with the standard ACT-R 6.0 simulator - it will run the task but not collect data as I understand it) I have modified the online reaction time tester by Jim Allen to be pretty close to the standard laboratory PVT equipment. This can be use to demonstrate the PVT situation with only ten trials (redemo10.html) or for actually collecting some comparison data with the 100 trial version (rtdemo100.html). Both situations: the PythonACT-R and the web page version produce a mean and standard deviation for the reaction times so comparisons can be straight forward. The difficulty of using an unstandardized version of the PVT is that at this point it may need to be calibrated but that raises the issue of experimental errors in replication which is a good discussion topic for students.

The teaching plan is that this "compare yourself to the PythonACT-R model" will provide both some interest and some challenges to students as they try to understand why the results came out the way they did. One interesting issue that I do not know how to address in a model is the consistant finding the women have slower reaction times by about 40 milliseconds and make fewer false alarms. I am hopeful that this will replicate in the class results and that it will lead to an interesting discussion of just what women are doing differently in the PVT situation (researchers have speculated that the difference is due to men an women using different "cognitive strategies" on the same task with the same instuctions -- "as fast as possible").

Using a model of reaction time as the starting point

Another article from the ACT-R collection that provides a nice ACT-R introduction to reaction-time and sleep deprivation is:

Gunzelmann, G., Gluck, K. A., Van Dongen, H. P. A., O'Conner, R. M., & Dinges, D. F. (2005). A neurobehaviorally inspired ACT-R model of sleep deprivation: Decreased performance in psychomotor vigilance. In B. G. Bara, L. Barsalou, and M. Bucciarelli (Eds.), Proceedings of the Twenty-Seventh Annual Meeting of the Cognitive Science Society, (pp. 857-862). Mahwah, NJ: Lawrence Erlbaum Associates.
[info and link to pdf] http://act-r.psy.cmu.edu/publications/pubinfo.php?id=596

For my teaching purposes this provides the bridge into the PythonACT-R models because the model in this article was translated into Python ACT-R by Terry Stewart with a little of my help.
http://brucelandon.douglas.bc.ca/2360w2007/demo/PVTdemo.py

The basic idea in building a theory is to start small and make it simple in the beginning.
Here the idea was to start with the simplest task - psychomotor vigilance task - "when the light goes on press the button as fast as you can" and work with simple reaction times as the data to model. It turns out that there is a very large literature in psychology about reaction time and its relation to many psychological variables: age, IQ, gender, personality, etc. Consequently the model of reaction-time and be a bridge from a simple PythonACT-R model with 3 productions to more complex models that explore intersting variables for students to think about.

ACT-R Teaching Materials from the ACT-R site

Some of the best ACT-R Teaching materials are on the ACT-R site and for my students the most popular one was the one page overview of the theory with diagrams at url:
http://act-r.psy.cmu.edu/about/

There are also 7 Tutorials about how the LISP ACT-R works (in html doc or pdf)
(with LISP code examples that run in ACT-R 6.0 environment)
# Unit1: Understanding Production Systems
# Unit2: Perception and Motor Actions in ACT-R
# Unit3: Attention
# Unit4: Base-level Learning and Accuracy
# Unit5: Activation and Context
# Unit6: production Utilities
# Unit7: Production Rule Learning

The best recent explanation of ACT-R theory by John Anderson that I have used in teaching is a complex article that introduced an extension of the theory with the imaginal buffer:
Anderson, J. R. (2005) Human symbol manipulation within an integrated cognitive architecture. Cognitive Science, 29(3), 313-341.
[info and link to pdf] at http://act-r.psy.cmu.edu/publications/pubinfo.php?id=580

This article was difficult for my students to read but well received. To make it more likely that they would process all of it I made a text-to-speech version on MP3 files for them and got positive feedback that this addion audio version helped.

PythonACT-R benefits for teaching

Because python is a very popular scripting language many people have contributed code solutions to many kind of problems. Today I spent some time getting one of the extras bundled with PythonACT-R working. The extra is a text-to-speech processor so that now the demonstration model on sequential memory has a talking experimenter and the participant speaks to recite what has been learned. I am hoping that all of this make a provocative in class demonstration next term.

The side benefit is that my little elizabot.py one page introduction to python programming now can have speech output as well. The responsive speaking program makes a nice introduction to the Artificial Intelligence bots and the Turing Test competition (Lobner prize). This in turn sets the stage for the discussion of the differences between AI models and ACT-R models. (code for all of the python course models will be available via my home page http://brucelandon.douglas.bc.ca

ACT-R Teaching Materials

The approach is to make working with ACT-R theory of the mind have a low enough threshold so that undergraduates can come to appreciate computer simulation of human cognition. PythonACT-R developed by Terry Stewart enables both the theorizing by scripting mental processes to be easier as well as the evaluation of models by simulations and equivalence testing.
[Terry Stewart' Lecture Notes on PythonACT-R]