Continuous Performance Tests reviewed by Ron Dumont, Anna Tamborra and Brian Stone

(published here by kind permission of Ron Dumont Ed.D., Director of the School of Psychology Program, Fairleigh Dickinson University)
Link to comparison table

Three computerized continuous performance tests were reviewed by these authors. The goal of these reviews was to compare the ease of use, computer requirements, normative data, test result, and interpretability. No attempt was made to distinguish which program might “better identify” a sample of ADHD children from a control group. Materials reviewed generally were those that a practitioner would receive when purchasing the software package. Although extensive research may already have been published on CPTs, it was our goal to review only the materials that a school psychologist would receive when purchasing the tests themselves. Admittedly, no attempt has been made to extensively research the background of the tests and their historical use. These reviews should be considered general overviews of the tests and are not meant to be comprehensive in their nature.

Test of Variables of Attention	(Link to users comments)

The Test of Variables of Attention (T.O.V.A.), is a computerized, 23-minute (11 minutes for 4-5 year olds), non-language based, fixed interval, visual performance test for use in the screening; diagnosis; and monitoring of treatment of children and adults with attention deficits. It was created by Dr. Lawrence Greenberg and is distributed by Universal Attention Disorders, Inc. as well as American Guidance Service (AGS). Cost for this test is $495. This price includes the T.O.V.A. disk, micro switch (button), the T.O.V.A. box (for keeping track of additional tests), two T.O.V.A. videos, an interpretation manual, and an installation manual. The initial cost also allows 5 interpretations. Each additional interpretation costs between $5 and $6 depending on the number you purchase. (Included in packets sent to us a year before doing this review were a number of interesting materials that probably reflect the difference in the professions of those who might use the T.O.V.A.. On one promotional page, under the heading “Benefits of T.O.V.A.”, the following were listed: Enhance revenues; Retain patient within doctor’s practice; Builds practice/builds referral base; and Reimbursable through major medical/psychological benefits. Also included were two sample letters to insurance companies demonstrating how to bill for using the T.O.V.A. for either a C.N.S. diagnosis or an Organic Brain Syndrome Diagnosis.) Could school districts or school psychologists that use the T.O.V.A. request third party reimbursement?

The manual states three clinical uses of the T.O.V.A.: 1. as a screen for students suspected of having ADHD or learning problems; 2. as a diagnostic tool as part of a multi-disciplinary assessment of children and adults who may have attention deficit; and 3. as an aid in helping to determine the dosage level and to monitor the use of medication over time.

The test itself consists of repeated exposures on the computer screen of two different squares. The squares differ in that one has a ‘hole’ near the top (target figure) while the second has a ‘hole’ near the bottom. The subject is to press the button every time the square with the hole near the top is flashed on the screen. The T.O.V.A. variables include: Errors of omission (inattention) and commission (impulsivity); response time; standard deviations, anticipatory responses, post-commission responses, and multiple responses.

The Test of Variables of Attention (T.O.V.A.) computer program (version 1.3.1) was reviewed using a PowerMac 7100/66, with 16MB of memory. The manual that accompanied the software was for version 1.2. Installation of the T.O.V.A. software itself was flawless. Simply dragging the T.O.V.A. icon to the hard drive installs the program. The problem came when trying to connect the T.O.V.A. button to the modem port. It didn’t fit. The configuration of the Mac’s serial port had evidently changed from earlier models to the Power Macs and the 8 pin plug provided for the T.O.V.A. would not fit into the serial port. Luckily a toll free number is provided for technical support. After speaking with Andrew Greenberg, we chose to attempted the “low tech” solution of using an exacto knife to do away with the plastic surrounding the pins. When this didn’t work, Andrew gladly sent, and we received in 1 day, a micro processing switch that solved the problem. Technical support for computer problems and the availability of people knowledgeable of the T.O.V.A. when questions arose was excellent throughout the reviewing process.

One strange alert box appeared before the correct micro-switch arrived. The alert suggested that we might choose to use the computer mouse button instead of the micro-switch and directs the user to push the “Use mouse” button. There was however no button to click on! This is probably a good thing, since the test measures response time in microseconds and any inaccuracies would greatly affect the interpretation of the T.O.V.A..

What appears to be an error in the computer program was discovered when we entered the age of a child as 7 years old and the computer generated the incorrect form (#6 – Age 4-5 (IF)). Once the form had been set by the computer at #6, it could not be brought back to the correct age form (#1) without creating a new test subject set-up. Examiners not aware of this form change requirement could in fact administer the wrong test to the subject. (Andrew Greenberg reported that this error would be fixed immediately.)

Another caution must be noted. It is possible for the results to differ from the child’s actual performance. We found during one administration that when the results were sent to the printer, certain scores (omission errors) were reported when they had not occurred during the test taking. This was possibly caused by a ‘powering-down’ energy saving system in the printer hardware. To avoid this problem, examiners are cautioned to be sure they are using a printer that is fully on line from start to finish and that examiners remain with, and closely observe the actual performance of each person tested.

The manual provides normative data on 1590 subjects, at 15 different ages separated by sex. Male and female norms are reported separately because, on the average, males have faster reaction times but make significantly more errors of commission (impulsive guessing). The norms clearly show that sustained attention increases with age, levels out at adulthood, and then deteriorates slightly in older adults. The norms are not stratified and little, if any, information is provided about the makeup of these children and adults. No breakdowns for socioeconomic levels, geographic regions, education levels, or race information is provided. There is no evidence in the manual that the normative sample includes (or for that matter, excludes) special education students or children on stimulant medication. Above age 20, there are very few males in the norming tables. For ages above 19 the numbers in the norming sample age groups drops considerably from an average of 168 subjects per group (age 4 to 19) to 36 subjects per group (age 20-80+). At some ages male subjects in the norm sample made no errors, hence there was no variability. Thus, actual standard scores at these points are quite artificial. (In separately published and unpublished information not sent with the test materials, the T.O.V.A. normative group appears to be created from “rolling norms”, the continual addition of people to the sample at varying stages and then recalibrating the averages. An early sample included 775 children aged 6 to 16. These children came from grades 1, 3, 5, 7 and 9 in three Minneapolis, Minnesota, suburban public schools. The children were “mainly middle to upper-middle social class and was predominantly Caucasian (99%). A second sample of 821 children and adults was later added to the original total. These new subjects came from an early education screening project; randomly selected classes in one grade- and one high-school in a rural Minnesota community; volunteer undergraduates in three Minnesota liberal arts colleges; and adults living in six adult community settings. Children in special education classes were excluded from each sampling. The total number of subjects in the norming sample is somewhat confusing. The print-outs from the T.O.V.A. states that the norming base is “of 2000 children and adults.” The manual presents data dated 7/94 that includes 1590 children and adults. These are entitled “revised norms” yet are the same as those published in a paper dated 9/92. There is no mention of the remaining 400+ subjects.) Still, the norm sample is impressive for a test not published by a major company.

The primary author of the T.O.V.A., Dr. Lawrence Greenberg, is a psychiatrist, and the concepts of reliability and validity appear to be addressed in a somewhat different fashion than is typical in our field. To it’s credit, many differential diagnoses studies are cited where the T.O.V.A. is used (alone, and in conjunction with the Connors Parent Teacher Questionnaire (CPTQ)) to discriminate between children with attention deficit disorder and normals (also children with behavior disorders/and other diagnoses). The T.O.V.A. appears to have good sensitivity and specificity in this regard, particularly when used, as the authors recommend, in conjunction with other instruments. The T.O.V.A. was best at differentiating between attention deficit and normals. Still some normals overlap with some attention deficit disorder children. (There are no studies to show the T.O.V.A. is able to differentiate an attentional disorder from a specific learning disability. One statement made in the promotional materials and restated on the video is that because the T.O.V.A. uses a task that is “non-language based” it can differentiate ADD from learning disorders. We are not sure that that statement is sufficient to prove the point. If this was in fact true, why were the special education students excluded from at least the original normative sampling? It might have been helpful to have tested children identified as having a specific learning disability and then compare those results to the normative group.)

One concurrent validity study with very few subjects looked at the overlap between the T.O.V.A. and the CPTQ. Unfortunately, the authors employed a canonical correlation with 23 subjects and approximately 10 variables (it was unclear exactly how many variables were utilized). This is far too few subjects for such a study, and is therefore uninterpretable.

The authors looked at test-retest data for 97 subjects across ages and found no significant differences between testings “except for commission errors which…improved during the first half of the test from first to second test but not for two subsequent tests.” (Manual, p. 2). Interestingly, the authors note that practice effects tend to be reverse of other tests, in that subjects tend to do worse on it, as the novelty of the stimulus wears off. Overall, the authors concept of reliability in the manual refers to what are basically “lie scales”, however, these scales appear very useful in telling if the subject is merely responding at random. Psychometric reliability data would be welcome.

More validity studies would be useful, particularly in a divergent/convergent framework (e.g., does reaction time (or any of the measures) as used in the T.O.V.A. correlate with cognitive ability; do they correlate with other observable behaviors, etc.). Correlation between the different T.O.V.A. measures would also be useful. The authors state that they assume that a child 2 standard deviations below the mean on IQ would also be 2 standard deviations below the mean on the T.O.V.A.. Actually, the lower the g-loading of a given T.O.V.A. measure, the more it would tend to regress (be closer to) the mean.

The authors do an excellent job at showing how stimulant therapy affects T.O.V.A. responses. The T.O.V.A. appears particularly useful in being used to establish a baseline, prior to stimulant medication, then used to monitor stimulant medication afterwards. The T.O.V.A. measures appear very sensitive to stimulant therapy. This finding is quite impressive and certainly bolsters the validity of the T.O.V.A..

The authors take great care to point out the T.O.V.A. is not meant to diagnose attention deficit disorders, but is a good screener, and is useful as a part of a larger battery. They advocate behavioral interventions, possibly in conjunction with stimulant medication.

The T.O.V.A. was easy to load and run. The program worked effortlessly with the minor exceptions noted above. It is purposefully boring, and probably more so for the examiner who must sit patiently through the 23 minute test. Examiners may find themselves leaving the client alone while the test continues, but this seems like a bad idea since the clients behavior during the testing may be important in the interpretation of the results.

The T.O.V.A. looks promising and would make a good tool for further research. The manuals are replete with typos, but perhaps that was a test of our vigilance. The manuals could have benefited from a historical and theoretical perspective, as well. Overall, the test would certainly benefit from more of the typical reliability and validity data, but was impressive in many areas, including differential diagnosis and sensitivity to stimulant medication. It should serve researcher’s well, and would be fun to use for those considering masters’ thesis and doctoral dissertation work in the area.

Conners’ Continuous Performance Test (CPT)

The Conners’ Continual Performance Test (CPT) is a computerized, 14-minute, visual performance task in which the subject must respond repeatedly to non target figures and then inhibit responding whenever the infrequently presented target figure appears. The test is a “useful attention and learning disorder measure for children, and is sensitive to drug treatment in hyperactive children.” The manual states that the program is most useful for children between the ages of 6 and 17. Among the many variables are: Number of Hits, Omission, Commission; Response Time. It was created by Dr. Keith Conners and is distributed by Multi-Health Systems, Inc. as well as The Psychological Corporation. Cost for this test is $495 (this is for Version 4; current Version 5 is $595 with added features). This price includes the CPT disk, and an interpretation and installation manual. The program offers unlimited administration, scoring and interpretations of the complete “Standard” paradigm. For research purposes, the computer program offers the ability to create customized paradigms with varying letters, presentation time, trials per block, etc.. It must be noted that normative data is only available for the standard paradigm. Anyone using the customized paradigm must do so with the understanding that no normative data is available for any such changes.

The “Standard” test itself requires the subject is to press the appropriate mouse button or the keyboard’s spacebar for any letter except the letter X. There are 6 blocks, with 3 sub-blocks each of 20 trials (letters presented whether target or not). For each block, the sub-blocks have different stimulus intervals. These intervals vary between blocks.

The Conners’ CPT computer program was reviewed using an IBM computer as well as a Power Mac 7100/66, running Soft Windows with 16MB of memory. Although the program was easily loaded onto the Power Mac, it could not be run under the simulated DOS. A toll free technical support number is available for anyone having difficulty with the program. The first time we called we were put on hold for 30 minutes before the technical support person came on. The next two times we were connected to technical support within a minute. All questions were answered quickly and courteously. Once properly installed on the IBM, the program ran flawlessly.

The manual provides normative data on 1190 subjects, at 8 different age groupings. This sampling is further broken down into two groups: General population (n=520) and Clinical sample (n=670). Careful reading of the manual indicates that this clinical sample was further broken down to 484 people after 130 subjects were removed for a cross validation study, 46 removed for being “outliers”, and 10 more removed because of being on medication. The 484 was finally reduced to 238 subjects comprised of ADD/ADHD and comorbid cases (including ADD/ADHD as one of the diagnoses). Male and female norms are used by the computer program but are not reported separately in the manual. The “general population” and “clinical population” consisted of 51.2% and 75.4% males respectively. No breakdown by age category is offered. (In fact, in the manual, no normative score data is given with the exception of that stated above). The norms are not stratified and little, if any, information is provided about the makeup of these children and adults. Very little information regarding socioeconomic levels, geographic regions, education levels, or race information is provided. It is noted that data for the general population came from 5 states and “Southern Ontario.”

The program provides data as both raw scores, T scores, percentiles, and descriptive classifications (e.g.., Within Average range, Mildly atypical, etc.). Reports are available on screen, as a print out, and as an ASCII file saved to disk.

The concepts of reliability and validity were not addressed thoroughly in the manual. It appears from reading the extensive annotated bibliography that some studies may have been carried out by independent researchers. However, with only the manual to rely on, we were left with many questions regarding these issues.

The major validity issue addressed in the manual looked at the ability of the CPT to discriminate between children with attention deficit disorder, “normals” (includes children with behavior disorders/and other diagnoses), and a comorbid group (children with dual diagnoses of ADHD and other disorders).

The CPT appeared to discriminate well, typically having the poorest mean score in the pure ADHD group, a somewhat better mean score in the comorbid group, and the “best” mean score in the “other” group, for the majority of variables. Unfortunately, the standard deviations were not listed, so the degree of overlap between groups on these variables is unknown. Another statistical technique, such as discriminant analysis would have been nice. Also problematic would be the existence of subtypes of ADHD within the ADHD sample. Perhaps the greatest display of validity is the letter of support issued in Russell Barkley’s newsletter that states the CPT is very much in line with current theory compared to many other instruments on the market (1993, June).

The manual seemed more concerned with history and theory than reliability and validity issues. The admissions in the manual were well appreciated, including the variability in sustained attention across times with the same subject, and the idea that, like IQ, there are many reasons for poor scores.

More research is needed on the stability of the many variables this test offers. Also needed is information regarding the independence of these variables (are they highly correlated with each other? What other measures do they correlate with?). Some of the independent research listed addressed these questions, but often the short abstracts of the studies listed were far too scanty to cull such information from.

However, kudos to the publisher for compiling the reference bibliography with abstracts (the little information contained was tantalizing and should send many buyers to their respective research libraries.)

To the author’s credit, an excellent job is done at showing how stimulant therapy affects CPT responses. The author also takes great care to point out throughout the manual that the CPT is not meant to diagnose attention deficit disorders by itself, and is useful as a part of a larger battery.

Intermediate Visual and Auditory Continuous Performance Test (IVA)

The Intermediate Visual and Auditory Continual Performance Test (CPT) is a computerized, 13-minute, visual and auditory performance task in which the subject must click the mouse only when he or she sees or hears the number 1 and not click when he or she hears or sees the number 2. The test is designed to assess two major factors: response control and Attention. In addition, the IVA provides “an objective measure of fine motor hyperactivity.” The manual states that the program is useful for persons between the ages of 5 and 90+. Among the many variables are six core quotients and 22 subscales. It was created by Drs. Joseph Sanford and Ann Turner and is distributed by BrainTrain. Cost for this test varies. A limited use kit (25 administrations) costs $598. This price includes the IVA disk (IBM 3.5 or IBM 5.25), and an interpretation and installation manual. Disks with an additional 25 tests may be purchased at a cost of $75. Users also have the option of purchasing an “Unlimited Use Version” for $1495.

The IVA CPT computer program requires an IBM computer with DOS 5.0 or later; 1 MB RAM/2 MB harddrive; a graphic monitor; serial mouse (Microsoft recommended); Creative labs Soundblaster card; Headphones or external speakers. (A toll free technical support number is available for anyone having questions with the program or the interpretation.) These requirements caused the most difficulty for these reviewers. In order to properly run the program, we had to find a computer that met each of the requirements, the most important being the Soundblaster card, the Microsoft mouse, and the headphones. Those well versed in IBM computers may feel right at home with this product, but these reviewers struggled for over an hour trying to get the mouse driver configured and the sound card and driver running. The installation of the program itself was not difficult. Step by step directions are provided in the well written manual. One hopes that the program can be re-written for a Macintosh since those computers come with voice capability and speakers built in.

The program uses normative data from 781 subjects (423 female, 358 male), at 10 different age groupings. No breakdown by age category is offered in the manual. There is no evidence that the norms are stratified and little, if any, demographic information is provided about these subjects. No information regarding socioeconomic levels, geographic regions, education levels, or race is provided. It is noted that the groups were comprised only of persons “who do not report any attention, learning, neurological or psychological problems.” The normative data file, contained on the program disk, was easily read by us using a Macintosh computer. The 10 age groups averaged approximately 42 female (range 15 at age 55+ to 75 at age 7-8) and 36 males (range 17 at age 45 – 54 to 68 at age 7-8). Age groupings are: 2 years (5 – 10), 3 years (11 – 13), 4 years (14 – 17), 7 years (18 – 24) and 10 years (25 – 55+).

The program provides data as both Quotient scores (mean 100, SD 15), percentiles. Graphs also are used to represent the results. The interpretation section of the manual is easy to read yet quite complex. The 6 quotient scores plus 22 subscales offer a large number of decisions and comparisons. The manual presents 17 pages of description and definition for each scale and 34 pages of interpretive suggestions. Included is a 21 step “procedural guidelines” for interpreting the IVA. The program offers 3 “Validity” scales used to confirm or refute the IVA results. Reports are available both on screen and as printouts. Data is stored and available on disk for retesting and comparisons.

The packet we reviewed contained 5 unpublished studies (presented at the 1995 APA conference). These studies address: normative, reliability, and validity data, differences in auditory and visual processing, and finally developmental age and sex differences on the IVA.

The extremely well-written manual was readable and informative. It addressed both reliability (stability?) and validity issues. It also reported (admitted) the less than stellar test/retest correlations across some variables (ranged from .37 to .75 for the composite variables). Even though the studies were only APA conference presentations, the authors have attempted to look at the important issues. An important question as yet unanswered by the materials included was to what extent did the auditory and visual variables correlate – how separate are they? Are they as highly correlated as the Wechsler verbal and performance scales? If so, are they subject to the same argument proposed by MacMann and Barnett who suggest that the Wechsler scales are so highly correlated as to render them similar measures of the same construct (for a review, see Kaufman, 1994).

Particularly impressive were the three validity scales, which ensure scores in ADHD ranges come from ADHD behaviors and not motor problems, fatigue, or random answering. The manual also acknowledged the relationship between IQ and sustained attention, and suggested IQ scores of 120 and above would be well-served by comparison to the next age norm table up. The authors also acknowledged the issue of subtypes of ADHD (inattentive, impulsive, “mixed,” and other). The authors addressed the issues head on, and asked the best questions. While all three tests addressed reliability and validity to some degree, the IVAs authors did the best job at asking the right questions. While all three tests are really in the beginning stages of compiling research data on reliability and validity, the IVAs authors are headed in the most compelling direction.

Test takers’ point of view

Each of the CPTs was administered to one or more of these reviewers to assess ease of use from the test takers point of view. Since the tests varied in length from 13 to 23 minutes, plus some additional time for practice testing, we found that our attention varied as the tests extended in time. Early in the testing sessions, we found ourselves being very cautious and completely focused on the screen, but as the tests continued on in time, it seemed more and more difficult to maintain our focus on the stimuli from the computer. The speed of some of the tests’ stimuli presentations was so rapid that we found ourselves almost afraid to blink. This led to our eyes becoming tired, and to a heightened sense of anxiety. (Because the tests are standardized, we assume that those included in the norming sample probably felt some similar feelings, and the normative scores adjust for such feelings.) Testing was done in a fairly sterile room, but we did not attempt to ‘sanitize’ the room completely. There were some materials on the walls and in shelves around the computer. We found that even these few things became very tempting distractions during the testing. Not only did we find ourselves easily distracted by these visual materials, we found ourselves being drawn to and distracted by sounds outside the testing room. Each manual provided instructions to the testers about how to create a positive testing environment, and we strongly reinforce the need to follow these instructions. Any extraneous material may have the potential to interfere with performance. Because the examiner needs to be present during each of these CPTs, the examiners must assure that they do not become a distraction themselves by unnecessarily moving about or making any noise. (This may become somewhat difficult, especially after sitting through a number of these admittedly ‘boring’ tests.) One final caution we learned by taking the tests was the need to give the directions exactly as they are presented in the manual. For example, one of us took a test without having the instructions read verbatim from the manual, and without any emphasis placed on the direction to do the task “As fast as you can.” Interestingly, the resulting printout recommended further assessment because of the suspicion of an attention disorder.

CPT Bottom Line

Choosing between the different tests will depend on many individual factors. The three CPTs reviewed in this issue of the Communiqué each offers something unique to the examiner and examinee. The T.O.V.A. uses a design (square), Conners’ letters, and the IVA numbers (1 and 2). The IVA is the only CPT to offer both auditory and visual procedures. If cost is a factor, the Conners’ is the least expensive while the IVA the most expensive. If the computer system is an issue, the T.O.V.A. is the only CPT that runs on a Macintosh, while all three offer versions for IBM based machines. The T.O.V.A. and the Conners’ require the least amount of “extra” hardware. Normative data for the CPTs was largest for the T.O.V.A. although none of the test provided enough demographic information about the subjects to make informed judgments about the suitability of the data. If time is a factor, the Conners’ and the IVA were the shortest tests. Ease of use was comparable for each of the tests. Support for the programs by way of toll free telephone numbers is provided by each system.

Our experience using these three programs was generally very positive. We stress however that the programs are simply one tool to be used in a multi-dimensional assessment. Each test product included clear warnings about not basing diagnosis on the single instrument or result. We whole-heartedly endorse this caution, especially given the vast differences between computers, computer systems, and the few ‘kinks’ we discovered during our limited reviews.

The editors of the Communiqué would like to thank each of the three companies for providing the programs for review.

Content on these pages is copyrighted  by Dumont/Willis © (2001) unless otherwise noted. The original website can be found at http://alpha.fdu.edu/psychology/cpt_review.htm