Skip Navigation


Schizophrenia Bulletin Advance Access originally published online on August 24, 2005
Schizophrenia Bulletin 2005 31(4):942-953; doi:10.1093/schbul/sbi045
This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
31/4/942    most recent
sbi045v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Medalia, A.
Right arrow Articles by Richardson, R.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Medalia, A.
Right arrow Articles by Richardson, R.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

© The Author 2005. Published by Oxford University Press on behalf of the Maryland Psychiatric Research Center. All rights reserved. For permissions, please email: journals.permissions@oupjournals.org.

What Predicts a Good Response to Cognitive Remediation Interventions?

Alice Medalia
Montefiore Medical Center, Klau-2, 111 East 210 Street, Bronx, NY 10467
Department of Psychiatry, Albert Einstein College of Medicine, and Montefiore Medical Center, Bronx, NY

Randall Richardson
Department of Psychiatry, Albert Einstein College of Medicine, and Montefiore Medical Center, Bronx, NY


   Abstract
Top
 Abstract
Introduction
 Theoretical Approach
 Training Method: The...
 Defining Improvement
 Analytic Approach
 Results
 Discussion
 Conclusion
 References
 
Cognitive remediation is a behavioral intervention designed to improve cognition in people who have suffered a decline in neuropsychological functioning. The application of cognitive remediation for people with schizophrenia has become increasingly popular. In this article we attempt to determine what factors moderate outcome in cognitive remediation. This issue is of primary concern, as the efficacy of treatments largely depends on the appropriate selection of individuals and treatment methodologies that are most likely to yield gains. Three databases were examined, dividing the samples into participants who showed or did not show a marked improvement from baseline to posttest. We examined patient, illness, and treatment factors that might affect treatment response. Whereas certain patient and treatment variables did differentiate "Improvers" from "Nonimprovers," illness factors were the least likely to influence treatment outcome. Treatment intensity, type of cognitive remediation program, and therapist qualifications were identified as key treatment variables. Patient levels of motivation for treatment and baseline sets of work habits were also important moderators. While the issue of treatment intensity appears to be especially salient, there is evidence that to some extent patient and treatment variables intertwine to produce a positive response to cognitive remediation.

Keywords: treatment response / treatment moderators / cognitive remediation / schizophrenia


   Introduction
Top
 Abstract
 Introduction
Theoretical Approach
 Training Method: The...
 Defining Improvement
 Analytic Approach
 Results
 Discussion
 Conclusion
 References
 
Cognitive impairment is increasingly recognized as a core symptom of schizophrenia that is evident in almost all people who carry that diagnosis.1–2 Most commonly manifested as a decline in attention, memory, and executive functions, these deficits have been shown to negatively influence multiple aspects of daily functioning. Cognitive deficits have been linked to treatment response, employment status, social relationships, living status, insight into illness, therapeutic alliance, and community functioning.3–5 The mounting evidence linking impaired cognition and functional outcome has fueled an interest in finding efficacious pharmacologic and behavioral treatments to enhance cognition. At this point, there are no drugs specifically approved by the Food and Drug Administration as cognitive enhancing; however, there is evidence that atypical antipsychotics can provide some cognitive benefit.6–7 In the realm of behavioral treatments, there are specific interventions that have been developed, and these are broadly classified as cognitive remediation or cognitive rehabilitation.

Cognitive remediation can be administered in individual, group, computerized, and noncomputerized formats and typically requires that a client attend at least 10 but typically more than 25 sessions, which involve doing exercises designed to improve cognitive skills. At the most basic level, cognitive remediation might involve simply doing computerized exercises that were developed to improve cognition for head-injured patients, but there are also some comprehensive programs designed specifically for psychiatric patients—programs that are manualized, have a theoretical rationale, and have a systematic stage approach to improving cognition.8–10 The Training Grid Outlining Best Practices for Recovery and Improved Outcomes for People With Serious Mental Illness developed by the American Psychological Association Committee for the Advancement of Professional Practice recommends cognitive remediation as a key component of treatment and refers to 8 different approaches that have been used in clinical trials.11

There have been enough published trials with cognitive remediation to support several meta-analyses of effect sizes. In general these have been positive, with effect sizes ranging from large for studies on improvement in the training exercise12 to moderate for studies of generalization from the training exercise to independent measures of neuropsychological functioning and daily functioning.13–14 Suslow et al.,15 who focus only on attention remediation, and Pilling et al.,16 who only review a few studies, have found smaller effect sizes. With considerable evidence that cognitive remediation is effective at improving not only cognitive skills but also real-world functioning, interest is shifting to an articulation of the issues that potentially affect the success of cognitive rehabilitation programs.17 One way to examine this issue is to consider the factors that differentiate those individuals who show a positive response to cognitive remediation from those who did not. Identification of the factors that are associated with an improvement in cognitive and real-world functioning can help us refine the treatments and guarantee a more successful implementation of cognitive remediation programs in the arena of treatment services.

Treatment controlled studies of the efficacy of cognitive remediation provide important information about the impact of the intervention on a group of individuals, but the group data often mask information about individual response. In any group of individuals exposed to a treatment, some may have an excellent response, while others may have no response at all. As an example, a group of 47 treatment-exposed patients in one of our clinics improved 0.5 standard deviation on memory tests. However, closer examination of the data indicates that 13 patients made no improvement, and if one looks at only the 34 responders, the improvement for that subgroup was 1 standard deviation. Clearly, the treatment worked quite well for the majority of patients but was not helpful for a minority. The question then arises: What was different about the patients and treatment experience for those who showed a treatment response? A delineation of those factors can elucidate the mechanisms of a positive treatment response. This type of investigation involves analyzing moderators of treatment efficacy.18 Moderator variables are the baseline characteristics of a sample that are unrelated to experimental condition. As such they function as independent variables that may interact with the predictor variable to influence outcome. To the degree that a baseline characteristic that is uncorrelated with the predictor variable (e.g., condition assignment) predicts the dependent variable, it is said to "moderate" the outcome.19

In this article we attempt to determine what factors contribute to a positive outcome in cognitive remediation. This issue is of primary concern for researchers and practitioners, as the efficacy of treatments largely depends on the appropriate selection of individuals and treatment methodologies that are most likely to yield gains. In order to better understand what characteristics predict a positive outcome in cognitive training, we have examined 3 databases (see table 1) where the cognitive remediation participants could be classified according to the presence or absence of cognitive improvement. We also draw upon the findings of several outcome studies and meta-analytic reviews to supplement our data and further elucidate the factors that bring about the greatest cognitive improvement in patients.


View this table:
[in this window]
[in a new window]
  		Table 1. Database Information

 

   Theoretical Approach
Top
 Abstract
 Introduction
 Theoretical Approach
Training Method: The...
 Defining Improvement
 Analytic Approach
 Results
 Discussion
 Conclusion
 References
 
There are inevitably numerous possible reasons why a client may or may not respond well to a treatment. The range of factors that may contribute to success or failure in treatment spans the gamut of biological, social, environmental, and psychological characteristics that constitute an individual's makeup. These may include genetic factors, social environment, prior learning experiences, illness exacerbations, and so on. The factors we chose to examine were largely determined by their relevance to the form of behavioral treatment we provide, the social/rehabilitation settings in which we apply our methods, and their measurability according to available metrics. Consequently we conceptualized these many reasons as falling into 3 broad categories: patient characteristics, illness characteristics, and treatment characteristics. These 3 areas may be understood as lenses through which we may view a patient's progress in treatment. Each allows for a different set of interventions and has different implications for further research on the efficacy of cognitive remediation.

Within each of the categories—patient, illness, and treatment characteristics—we looked at several factors based on the information available in the various databases. Since the investigation was de facto limited by the information documented in the databases, this should not be viewed as a comprehensive analysis of all factors that might contribute to a positive treatment response but, rather, as a first effort to look at some of the factors that might be relevant.


   Training Method: The Neuropsychological Educational Approach to Remediation Model
Top
 Abstract
 Introduction
 Theoretical Approach
 Training Method: The...
Defining Improvement
 Analytic Approach
 Results
 Discussion
 Conclusion
 References
 
In each of the 3 databases used to identify the factors that differentiate treatment responders from nonresponders, patients were trained using the Neuropsychological Educational Approach to Remediation (NEAR) or elements of it. NEAR is a manualized, comprehensive treatment program developed by Medalia and colleagues.10 This technique of primarily computer-based cognitive training emphasizes the educational and cognitive aspect of remediation, encouraging participants to reflect on their unique learning styles while taking an active role in their training experience. Using commercially available educational software that was selected for its merits in stimulating various neuropsychological functions, learning tasks are designed to be engaging, enjoyable, and intrinsically motivating. These programs feature a diverse range of sensory stimuli, provide immediate feedback, and may be easily graded in difficulty by the participant. This contributes to a learning environment that fosters independence, self-efficacy, and persistence on learning tasks. In this way NEAR is distinct from cognitive remediation models that utilize external reinforcements to promote task engagement.

The NEAR model was developed specifically for use with psychiatric patients, whose illnesses require a different treatment orientation from those who are head injured. Head-injured patients experience a sudden, precipitous loss of functioning and are often highly motivated to recover their premorbid potential. Remediation programs developed for head-injured patients inadvertently benefit from the intrinsic motivation of these patients, who are more willing to tolerate repetitive drill exercises. In contrast, schizophrenic patients, whose cognitive impairments typically emerge long before the onset of psychotic symptoms, experience lapses in their psychological and intellectual development.20–21 Repeated experiences of failure in learning situations during adolescence—a critical phase of cognitive development—has often hardened patients against learning experiences. This response, combined with the motivational problems caused by the disease itself, sets the stage for a disengaged attitude toward learning and cognitive activity. To counteract such disengagement, NEAR tasks are designed to be stimulating, dynamic, and enjoyable so as to elicit intrinsic motivation.

Although NEAR incorporates some exercises that focus on more discrete cognitive skills, NEAR exercises favor a top-down approach to remediation. As opposed to bottom-up approaches that typically use drill and practice to focus on isolated neuropsychological skills, NEAR training tasks incorporate several skills at once and are presented in a contextualized format. This approach not only elicits several types of responses in concert but also allows for more flexibility in designing tasks that are engaging and intrinsically rewarding. Such tasks are known to foster enhanced learning.22

NEAR has been implemented around the world in experimental and clinical settings. The range of treatment settings has included research labs, acute and chronic inpatient psychiatric units, continuing day treatment and intensive psychiatric rehabilitation programs, forensic settings, and generic outpatient programs. It has also been adapted for use in individual training and in groups of up to 10. Because participant autonomy is an important aspect of NEAR, trainers are able to work effectively with several participants in a single 1-hour session. In terms of effectiveness, clinical trials involving the NEAR model have shown that clients improve performance not only on neuropsychological tests but also on proxy measures of functional outcome and real-world functioning.23–25


   Defining Improvement
Top
 Abstract
 Introduction
 Theoretical Approach
 Training Method: The...
 Defining Improvement
Analytic Approach
 Results
 Discussion
 Conclusion
 References
 
Two critical methodological issues that need to be addressed in any empirical analysis of clinical improvement are (1) what kind of improvement is considered relevant and (2) how much improvement is enough to show that a treatment is efficacious. The first question brings into focus the quality of variables one may use to measure improvement, for example, the construct validity of a measurement device and the degree to which it predicts performance in other relevant domains. The second relates to the quantity of improvement that may be satisfactorily assumed to be sufficient, that is: Is any improvement acceptable, or are there criteria for determining what is clinically or statistically meaningful?

Neuropsychological measures of attention, concentration, or executive functioning may be valid as measures of cognitive functioning; however, they rarely involve tasks that are undertaken in everyday life. In actual clinical settings, improvement related to cognitive remediation may show itself in many forms. For example, a patient who is more cognitively intact may utilize available treatment resources more effectively, may remain more compliant with medications and consequently experience greater symptom reduction, may be able to garner better social and familial supports, or may experience him- or herself as more intellectually competent, leading to enhanced self-esteem and better self-care.

Proximal Versus Distal Measures
If we were to place outcome measures along a continuum of conceptual proximity to the treatment context, on the one extreme we would find dependent measures that closely resemble the training task, and on the other we would see measures that are virtually unrelated to the task. More proximal measures have the advantage of allowing tighter experimental control and better utility in establishing cause and effect relations. More distal measures might offer greater generalizability of findings to real-world settings.

Kurtz et al. summarized research examining the training of schizophrenic patients on the Wisconsin Card Sorting Test, and found that the mean effect size of treatment was quite large (d = 0.96).12 Because these studies all measured outcome as performance on the training task, they demonstrate only that learning is possible for schizophrenia patients. These findings prepared the ground for future developments, particularly in the face of assumptions that such learning is impossible.12 Nonetheless, the close proximity of the outcome measure used did not afford wide generalizability of these effects to other aspects of social or occupational functioning. Research designs employing dependent measures that are distal to the training task are able to distinguish whether changes in performance are due to real cognitive improvement or mere expertise on the training task. Real-world variables such as treatment compliance, social engagement, occupational advancement, work-related skills, or social problem solving have greater ecological validity and better address the functional goals of the patient.

More distal assessments are also congruent with the therapeutic spirit of cognitive remediation, which intends to improve the quality of a patient's life and community functioning. At the same time, very distal measures of treatment outcome, such as occupational advancement or independent living, risk compromised reliability, as any number of intervening variables may influence them in a way that is difficult to control. For example, if we attempt to examine the effect of cognitive remediation on success in finding employment, we would also need to account for the influence of socioeconomic status, educational background, prior employment, social skills, and available employment opportunities. Attempting to control so many variables can result in unwieldy research designs and data analyses that are difficult to interpret effectively. Researchers using a combination of proximate and distal measures will be better able to distinguish whether cognitive improvement has a significant impact on real-world functioning.

In the analyses conducted for this study, a positive response to cognitive remediation was determined with outcome measures that fall at various points along the proximal–distal continuum. Improvement on the training task was rejected as too proximal, and work or school outcomes were deemed too distal. Instead, as depicted in table 1, a combination of neuropsychological tests and proxy measures of functional outcome was used. As such, these outcome measures provide a mix of ecologically and neuropsychologically valid assessments of treatment-related cognitive change.

Meaningful Improvement
Several standards have been used to determine the magnitude of improvement that is clinically meaningful. Most researchers utilize a criterion of statistical significance (typically p = .05) as a clear indication that improvement from pretest to posttest is empirically valid. Many then compare the treatment group's change over time to that made by a control group to determine whether a treatment has been effective.

Some researchers insist that there is a normative dimension to clinically meaningful improvement. For example, Kendall and Grove argue that even though an individual may improve his or her performance to a degree that is statistically significant, this improvement is only clinically meaningful if it falls within normal limits.26 These authors define clinically significant improvement as occurring when "the once troubled and disordered clients are now, after treatment, not distinguishable from a meaningful and representative non-disturbed reference group."26(p148) To this end, some researchers recommend utilizing a cutoff score that falls 1 standard deviation from the normative mean. Those patients who make a significant change from baseline and fall within normal range are regarded as clinically significant improvers. When working with severely impaired populations, however, this presents a potentially prohibitive criterion for success. Given the developmental obstacles associated with schizophrenia and other major mental illnesses, applying normative expectations may be unrealistic and carries a risk of disqualifying important improvements that patients might make.

Nonetheless, a reliance on significance tests alone as a means of determining improvement is potentially problematic, as this computation does not take into account task-specific practice effects. This issue is particularly relevant in studies in which the dependent measure closely mirrors the training task. If one's treatment model assumes that a patient is able to learn, then one's method of evaluation must also make this assumption. As such, the variance in performance that may be attributed to practice effects, or other psychometric properties of a measure, should be taken into account when evaluating improvement. This method of assessing improvement ensures not only that a change is significant but that it is also reliable. For the purpose of this article, which has as its goal evaluating the efficacy of a treatment technique, we used an index of reliable change because it accounts for the magnitude of treatment effect and the psychometric instabilities of a measure. This method is adequately stringent yet sufficiently inclusive to ensure a meaningful interpretation of results.27


   Analytic Approach
Top
 Abstract
 Introduction
 Theoretical Approach
 Training Method: The...
 Defining Improvement
 Analytic Approach
Results
 Discussion
 Conclusion
 References
 
In order to better understand what characteristics predict a positive outcome in cognitive training, we analyzed data from 3 databases comprising a total of 117 patients (see table 1). Each treatment sample was divided into participants who showed or did not show a reliable improvement from baseline to posttest on at least 1 dependent measure. We also reviewed several meta-analyses and outcome studies to draw attention to what aspects of cognitive remediation have been shown to bring about the greatest improvement in patients.

For each data set, reliable change indexes were calculated by dividing change scores on each dependent measure by its standard error of measurement. Wherever possible, the standard error of measurement was calculated using test-retest reliability coefficients from control subjects in the same study. Where no control condition was employed, reliability coefficients from published norms that matched the general demographics of the experimental sample were used (for a comprehensive discussion of the reliable change index, see 27). The reliability coefficients of the dependent measures in these 3 data sets ranged from moderate to high, thereby giving sufficient power to detect change with the reliable change index. Individuals whose change scores yielded an index value of 1.96 or higher (95% confidence level) were identified as having made a reliable improvement on that measure from pretest to posttest. This method applied such a stringent criterion for improvement that some participants who did in fact make a statistically significant change from pretest to posttest were not included in the set of "reliable improvers." Nonetheless, as table 1 details, 49.5% of those receiving cognitive remediation showed a reliable improvement in at least 1 domain of cognitive ability. This group was then called "Improvers," as contrasted to the group of individuals who did not show improvement with the reliable change index, which was called "Nonimprovers."


   Results
Top
 Abstract
 Introduction
 Theoretical Approach
 Training Method: The...
 Defining Improvement
 Analytic Approach
 Results
Discussion
 Conclusion
 References
 
Within each of the 3 data sets, comparisons were made between Improvers and Nonimprovers on several areas of functioning, in order to better elucidate the distinctions between those individuals who make substantial gains in treatment and those for whom such gains are less likely. For variables featuring a distribution of interval scores approximating normality, analyses of variance were conducted; otherwise nonparametric tests were used.

Patient Factors
The factors related to the patient that were analyzed include demographic characteristics, baseline cognitive functioning, motivation for attending cognitive remediation, and work style in the Learning Center.

Demographics. There were no statistically significant differences between Improvers and Nonimprovers in terms of gender, age, ethnicity, socioeconomic status, or years of education in the any of the 3 studies. Since raw scores for several of our dependent measures were converted to standard scores using age-corrected norms, age-based differences in Improver status might have been masked. Correlations of age with raw scores on these measures yielded no significant effects, however. There was no evidence that demographic factors informed whether a patient was in the Improver or Nonimprover group.

Baseline Impairment. The degree to which a person is cognitively impaired when he or she starts cognitive remediation may have several effects on outcome in treatment. A more impaired person will have greater difficulty learning a task and thereby spend less time actually practicing the needed skill. Greater impairment may be associated with a higher level of frustration, resulting in lowered motivation for training tasks and lowered self-esteem. Whereas compensatory strategies may be employed for focal impairments, the more pervasive impairments found among lower-functioning schizophrenic patients may make learning alternative strategies difficult.

We found no evidence that baseline scores on the neuropsychological tests of processing speed, working memory, or immediate story recall differed between the Improver and Nonimprover groups in Study 3 or that baseline scores on problem solving or memory differed between Improvers and Nonimprovers in Study 2. However, in Study 3, baseline Delayed Story Memory raw scores differed between Improvers (X = 7.2, SD = 2.3) and Nonimprovers (X = 9.4, SD = 2.77; F [1,24] = 4.24, p < .05).

Baseline performance on 2 of the 3 ecologically oriented measures of cognition proved to differentiate Improvers from Nonimprovers. Improvers in Study 1 showed superior baseline performance on the Minnesota Clerical Test (MCT; F [1,53] = 11.18, p < .002), although their scores were still impaired at baseline relative to the normative sample of clerical workers. The MCT is a timed, 200-item task on which participants compare 2 strings of 5 numbers or letters and indicate which strings are identical.28 The average standard score at baseline for the Improvers was below 91% of clerical workers (X = 80, SD = 10.9), whereas for the Nonimprovers the mean score was below 97% of clerical workers (X = 71, SD = 8.08).

In Study 2, Improvers were rated by clinicians as less impaired on the Positive and Negative Syndrome Scales (PANSS) Cognitive Factor, which is a clinician-rated measure of disorientation, poor attention, difficulty in abstract thinking, and conceptual disorganization (Z [2,35] = –3.21, p = .001). Mean baseline score for the Improver group was 12 (SD = 2.5), whereas for the Nonimprover group the mean baseline PANSS score was 15.5 (SD = 3).

Motivation. Motivation has been found to be a crucial factor in the treatment of substance abuse, chronic disease management, and psychotherapy.29 There is also evidence that it is important in cognitive remediation.30 The type of motivation that is most associated with therapeutic responsiveness is called intrinsic motivation: the motivation to do something because of an appreciation of its inherent value. Clients who go to therapy because of extrinsic motivation (e.g., court order) do not benefit as much as those who attend because they feel that therapy will help them.29 The literature on learning features the same dichotomy in response—students learn more, persist longer at learning tasks, are more creative, are more engaged, and have greater self-esteem when they are intrinsically motivated (learning for learning's sake), as opposed to extrinsically motivated by such rewards as grades or parent enticements.31–33 Inasmuch as cognitive remediation is both a learning and a therapeutic activity, it is reasonable to question whether intrinsic motivation is also a pertinent factor in treatment response.

The measurement of intrinsic motivation among nonpsychotic individuals typically either relies on self-report scales that assess attitudes about treatment or is determined by measuring actual attendance to sessions, where attendance is completely voluntary. Since self-report measures of motivation have not been shown to be reliable in psychiatric populations, we used attendance as the independent variable. In a context where attendance is entirely voluntary, if a patient shows up to sessions, it may be taken as a statement of his or her motivation for treatment.30 In 1 of our study settings (Study 1) attendance was voluntary and was tracked; as such we used these data to examine the issue of motivation. The dependent variable was the number of days it took each patient to complete 26 sessions. Given that clients were expected to arrive for sessions twice a week, the minimum amount of time in which the training could be completed was 13 weeks or 3.25 months. Permitting additional days for holidays, inclement weather, and temporary illness, an optimal period for completion of training would be about 3.5 months (100 days).

We found that Improvers had far better attendance than Nonimprovers. The Improvers completed their training period in 109 days on average (3.6 months, SD = 37 days), whereas the Nonimprovers took an average of 149 days (4.9 months, SD = 48 days) to complete training. This difference was highly significant (F [1, 52] = 10.3, p = .002). Inasmuch as voluntary attendance is a measure of intrinsic motivation for treatment, this indicates that motivation is an important patient characteristic when aiming for a positive treatment outcome.

Work Style. The manner in which an individual approaches a work-related task is highly informative of his or her level of organization and commitment to success. These factors in turn may influence improvement in training. We were therefore interested to know whether Improvers had superior work habits to Nonimprovers at the outset of treatment.

Work-related behaviors were measured in Study 1 using the Work Behavior Inventory (WBI), which was developed by Bryson et al. for the purpose of evaluating and predicting the occupational functioning of people with mental illnesses.34 For this study, the WBI was modified so that questions pertained to behavior in the Learning Center, where clients received cognitive remediation. The WBI subscales contain between 5 and 7 items and are rated on a Likert scale from 1 (persistent problem area) to 5 (frequent area of strength). Cognitive remediation trainers were asked to score this measure in the third and twenty-sixth session of training.

Analyses of variance show that individuals who improved in training received significantly higher scores on each of the 5 subscales prior to training. In other words, these patients were seen by their clinicians as having better social skills (F [2,54] = 4.17, p = .04), being more cooperative (F [2,54] = 6.64, p = .01), demonstrating superior work habits when doing the cognitive remediation exercises (F [2,54] = 6.22, p = .01), producing work of higher quality in the Learning Center (F [2,54] = 7.83, p = .007), and being more impressive in their personal presentation (F [2,54] = 7.91, p = .007).

Illness Factors
As cognitive remediation becomes more widely implemented, it will be important to identify the patient populations and treatment settings for which it is best indicated. It is unknown, for example, if patients with particular diagnoses gain more from cognitive remediation than others. Most studies of cognitive remediation for psychiatric populations have focused on schizophrenia patients who are known to be cognitively impaired, but there is ample evidence that people with affective disorders and substance abuse also have cognitive impairments.35–36 It is also not yet clear whether acutely ill psychiatric patients respond differently than stabilized outpatients or chronically decompensated inpatients. These illness factors are the focus of the following section, in which we examine whether illness factors such as diagnosis, symptom profile, and acuity moderate a positive response to cognitive remediation.

Diagnosis. In studies 1 and 3, patients with both schizophrenia and affective spectrum diagnoses participated. There is no evidence that diagnosis differentiated Improvers from Nonimprovers in these 2 samples. In Study 2, neither a diagnosis of schizophrenia versus schizoaffective disorder nor the presence of comorbid substance abuse differentiated the Improver and Nonimprover groups.

Symptom Profile. In Study 2, no significant correlations were found between 8 PANSS measures (General, Positive, and Negative Scales and 5 factors) and outcome on cognitive tests. Furthermore, PANSS scores on the 8 measures did not significantly differ between Improvers and Nonimprovers. However, we found some evidence that age of onset and treatment refractoriness affect response to cognitive remediation, at least in this severely ill inpatient sample. After removing outliers from the sample whose hospitalizations were in excess of 3 standard deviations from the sample mean (4 subjects, average hospitalizations of 16 years), we found that those individuals who had shorter hospital stays were more likely to be Improvers (F [1,30] = 11.34, p = .002) than those who were hospitalized longer. This finding should be interpreted cautiously given the fact that the average length of stay for the Improvers in this state hospital sample (8.5 months, SD = 5.4) was in itself quite long. Nonetheless, the implication for Nonimprovers, whose average length of hospitalization was 1.9 years (SD = 1.4), is that the most impaired patients are unlikely to improve with a short course of cognitive remediation.

We also saw a trend in which patients who had their first hospitalization later in life were more likely to benefit from cognitive remediation (F [1,34] = 3.51, p = .07). The average age of first hospitalization for Improvers was 22 years (SD = 4.1), whereas for Nonimprovers it was 19 years (SD = 3.9).

Acuity. Information about the impact of acuity comes indirectly from treatment controlled studies of cognitive remediation efficacy. NEAR was found to be effective with patients in an acute-care inpatient sample where the average length of stay was 14 days.37 Studies with chronic inpatients and stabilized outpatients have also yielded positive results.14 While none of these data directly compares level of acuity in Improvers versus Nonimprovers, the sum of evidence is that cognitive remediation can be applied in many treatment settings. Some support for this also comes from the finding that across our 3 samples, the percentage of Improvers versus Nonimprovers was roughly equal for inpatients (50%) and outpatients (54%).

Treatment Factors

Type of Medication. All of the patients recruited for these studies were in psychiatric treatment where their medications were prescribed by the treating physician. Although medication use was tracked, there was no attempt made to control the types of medications they were given. Additionally, the participants were prescribed a vastly heterogeneous set of medication regimens, including up to 5 different medications in some cases. It was therefore not possible to determine which medication regimen contributed more to cognitive improvement. Most studies of treatment efficacy have not controlled for this variable, but 1 post hoc analysis of tracked data about medication use suggests that patients taking atypicals were more likely to improve in cognitive remediation than those on conventional antipsychotics.38

Type of Cognitive Remediation Program. The studies discussed here all utilized the same training method; as such no direct comparisons could be made regarding type of training using this data. Nonetheless, other research comparing effect sizes between studies employing different methods has found that strategy learning has a slight though not significant benefit over drill-and-practice approaches.13 Supporting this finding, Kurtz et al. report that strategy training that encourages schizophrenic participants to verbally elaborate on their encoding strategies can return memory functioning to a level comparable to that of normal controls.12 Some remediation programs emphasize computer-based exercises, while others use verbal groups and/or non-computer-based learning activities. In their meta-analysis of outcome studies Suslow et al.15 and Twamley et al.14 report that effect sizes for computer-assisted remediation are generally higher than for paper-and-pencil techniques.

Treatment Intensity. Related to the issue of attendance frequency above is the question of treatment intensity, as a patient attending sessions regularly also has a more intensive exposure to treatment than one who attends occasionally. To examine whether there was a difference in degree of improvement between individuals who took longer to complete training and those who completed training closer to schedule, we split the sample at the median (128 days/4.5 months) and compared change in standard scores between those taking less than 128 days and those taking more than 128 days to complete training (Study 1). As expected, patients taking longer than the median time to complete training benefited significantly less (X = 5.1, SD = 14.4) than those completing training in a shorter period of time (X = 14.8, SD = 10.1; F [1, 52] = 10.3, p = .002). As this study utilized a test-retest design without a control group, treatment effect sizes for these 2 groups were computed using a variant of Cohen's d that was modified for a test-retest experimental design by substituting the baseline group mean and pooled baseline standard deviation for control group scores.39 The effect size for patients experiencing a high-intensity treatment (completing training in less than 128 days) was quite large (d = 1.46), whereas the effect size for the group of patients experiencing a lower-intensity treatment (more than 128 days) was small (d = 0.26). (See figure 1.) What this remarkable difference in effect sizes underscores is the important role that treatment intensity has in the gains patients make in cognitive remediation.



View larger version (23K):
[in this window]
[in a new window]
  		Fig. 1. Cognitive Remediation Effect Size as a Function of Treatment Intensity

 

Therapist Training. The level of training of the therapist is an important aspect of the treatment experience, as a more highly trained therapist may be sensitive to subtler aspects of cognitive deficits and may be able to direct a personalized treatment more effectively. Our Study 1 outpatient sample contained data from patients enrolled at several satellite clinics that featured a wide diversity of treatment providers, including social workers, rehabilitation technicians, and postdoctoral research fellows. Thirty of the 55 patients were trained by a Ph.D.-level clinician, whereas 25 were trained by technicians of varying levels (Associate's degree to Master of Social Work).

We found that patients trained by a Ph.D.-level clinician were significantly more likely to be in the Improver group (Z = 2.7, p = .006). Indeed, of the 22 patients making reliable improvements in this study, 17 (77%) were trained by a Ph.D.-level clinician. The magnitude of improvement between these 2 groups was widely divergent: d = 1.41 for patients trained by Ph.D. clinicians (X = 12.3, SD = 13.1) and d = 0.32 for those trained by non-Ph.D. clinicians (X = 6.2, SD = 12.7).

Not only was degree of improvement enhanced by the clinician's level of education, but also attendance (and consequently intensity of training) increased. Those patients being trained by a Ph.D. clinician completed training in significantly less time (mean = 121 days, SD = 38) than those being trained by other clinicians (mean = 147 days, SD = 55; F [1,52] = 3.98, p = .05).

Interaction Between Treatment and Patient Factors
The analyses discussed above indicate that patients who had better concentration and work habits, and who attended sessions regularly, were more likely to benefit from cognitive remediation. Conceivably, these factors interact to moderate a positive response. To wit, patients who have better work habits may be better able to keep their appointments, pay attention better in sessions, and consequently make greater improvements than those with poorer work habits. To test this hypothesis, we computed 2 analyses. First, a multiple regression analysis examining the relationship between each of the WBI baseline subscale totals and time to complete training was not significant (Study 1). Second, we computed an analysis of covariance examining the effect of treatment intensity on post-treatment MCT scores, while entering overall WBI and MCT pretest scores as covariates. This test of mediation assumed that if work behavior and prior cognitive ability accounted for a large enough proportion of the variance in improvement, then the effect of treatment intensity on improvement should be reduced to nonsignificance.18 Although the p value for this analysis increased somewhat, the effect of treatment intensity on improvement remained significant notwithstanding (F [3,49] = 4.88, p = .03). This indicates that treatment intensity had a singularly predictive relationship to outcome that was not significantly mediated by other psychosocial or cognitive factors.


   Discussion
Top
 Abstract
 Introduction
 Theoretical Approach
 Training Method: The...
 Defining Improvement
 Analytic Approach
 Results
 Discussion
Conclusion
 References
 
The increasing recognition that cognitive dysfunction contributes significantly to functional outcome in schizophrenia has fueled an interest in developing pharmacologic and behavioral treatments to improve cognitive functioning. There are now sufficiently numerous studies on the efficacy of cognitive remediation to allow a synthesis of the research findings with meta-analytic studies.12–15 Taken together, the meta-analytic studies suggest that cognitive remediation has moderate effect sizes on cognition and functional outcome. The task ahead is to identify ways to improve the treatment so that it will be most effective. To this end, we have looked at the factors that are associated with a positive treatment response, using our available databases as well as results from other published studies. An understanding of the variables that significantly contribute to treatment success can be used to tailor treatments so that they are more effective.

Synthesis of the Findings
We looked at patient, illness, and treatment factors that might influence treatment response. Of these, it emerged that illness factors were the least likely to influence treatment outcome. There was no evidence that diagnosis, acuity, or most aspects of symptom profile were significant variables that differentiated patients' responses to cognitive remediation. On the other hand, certain treatment and patient variables did differentiate Improvers from Nonimprovers. Treatment intensity, type of cognitive remediation program, and therapist qualifications were identified as key treatment variables to consider. We also found that a patient's level of motivation for treatment and his or her baseline set of work habits were important characteristics. (See table 2.)


View this table:
[in this window]
[in a new window]
  		Table 2. Factors Influencing a Positive Response to Cognitive Remediation

 
In terms of patient variables, these results indicate that in an outpatient sample, when rate of voluntary attendance is used as a measure of intrinsic motivation for treatment, there is a much higher rate of motivation in people who show a positive treatment response. The finding that motivation for treatment is an important patient variable is in keeping with the literature on treatment outcomes in medical and substance abuse settings.29 It is also consistent with the extensive educational literature on the importance of intrinsic motivation for learning.31 Intrinsic motivation is consistently associated with better treatment and educational outcomes, and our results indicate that this holds true for cognitive remediation as well. To use voluntary attendance as a measure of intrinsic motivation may be considered controversial, since we did not track variables like illness or psychosocial stress that could conceivably affect attendance. However, all the clients had the same weather, car fare, and treatment parameters, and the difference in attendance rates was highly significant. Still, the results should be viewed as preliminary and taken as a call for further investigations with better measures. Also, future research needs to disentangle the relationship between motivation and improvement. It is possible that clients who show improvement on outcome measures are making gains on the cognitive training tasks and become more motivated to attend sessions because they sense their improvement. Thus there may be an interaction between competency on the training tasks and motivation for training.

The other patient variable that differentiated Improvers from Nonimprovers was baseline work habits. We found that clinicians rated the Improvers as being at a significantly higher level on baseline indexes of social skills, cooperativeness, work habits, quality of work, and personal presentation. These ratings were done by the cognitive remediation clinicians after watching the clients in the first 3 NEAR sessions. Since NEAR emphasizes independent work, the clinicians had an opportunity to rate clients on how they approached the tasks of choosing and performing a remediation activity, relating to the clinician and other members of the group, and presenting themselves. When clients first entered the NEAR program there was wide variability in their ability to present themselves at sessions in a timely manner, appropriately groomed, and socially related, and there was a range of abilities to stay on task, ask for help when they needed it, and work independently. These are the skills that clinicians found to be more developed at baseline in the Improver group. These results suggest that, not surprisingly, work habits are important for learning to occur.

Although one might expect that people with better work habits would be more likely to recognize the value of consistent attendance and, therefore, go to sessions more regularly, we did not find this to be the case. Instead, it appears that the client work behaviors were largely independent from factors pertaining to attendance at sessions.

To a small extent, the baseline level of patient cognitive skill was also relevant to treatment outcome. The majority of analyses of the baseline cognitive data did not reveal them to be important, but there is a cluster of results that indicate that for the most cognitively disorganized, chronic inpatients, especially those hospitalized for more than 2 years and with an earlier onset of illness, a 10-session course of treatment is not sufficient. Also, we found that baseline scores on a vocational test of attention and processing speed differed in an outpatient sample, though it should be noted that the baseline score of the Improver group was still below 91% of the normal population. These clients had considerable room for improvement; the fact that they did improve after 26 sessions while the even lower-scoring clients did not may speak to the inadequacy of the length of the treatment trial for the most impaired.

Treatment variables emerged as a very important factor in this study, as they have in other studies. Previous research has indicated that differences in the actual nature of the remediation exercises can contribute to variable effect sizes.12–15 We found that, in addition, it is important to consider treatment intensity and clinician qualifications. Treatment intensity refers to the spacing of sessions, given a defined treatment package. The clinician qualifications that we considered referred to training degree.

The finding that treatment intensity differentiated Improvers from Nonimprovers is consistent with general life experience. Anyone embarking on a learning experience, whether to learn a physical, artistic, academic, or occupational skill, is unlikely to make gains if he or she does not regularly devote time to practice and lessons. The questions are: What is enough practice? And how many lessons are needed? In the case of the NEAR cognitive remediation program, we found that twice-a-week sessions were necessary to make gains. The treatment effect size for clients who attended regularly was quite large, whereas it was negligible for those with low treatment intensity. Interestingly, we did not find that client baseline work habits were related to attendance, but we did observe a significant interaction between therapist level of training and treatment intensity.

The level of training of a therapist proved to affect not only treatment outcome but also treatment adherence. Clients trained by doctoral-level staff were more likely to make cognitive gains and to come regularly to sessions than were clients of staff with less formal training in mental health. To understand this, it is important to appreciate that the doctoral-level trainers employed in these studies differed from nondoctoral staff in ways other than mere educational level. As they were postdoctoral research fellows, their involvement in the treatment setting was largely voluntary, and they were themselves intrinsically motivated to learn about the effects of cognitive remediation. This was not always true for the rehabilitation technicians and social workers who also delivered training. In many cases these trainers were unsure of the putative benefits of cognitive remediation and felt that their role as trainers was not entirely voluntary. It is possible that a clinician's attitude to treatment may have been transferred onto the patients being treated. Therefore it is our view that clinician variables may reflect not only the clinician's level of sophistication in identifying and addressing cognitive deficits but also the clinician's own sense of intrinsic motivation and commitment to the training task.

Implications for Clinical Practice
With the caveat that our findings represent a limited analysis of these issues and that much more work needs to be done to refine the formula for treatment success, we propose that there are some important messages for clinicians who do cognitive remediation. First, given that treatment intensity appears to be a salient factor, every effort should be made both to provide treatment regularly and to ensure that clients go to sessions. This may not be easy to accomplish in a treatment environment where fiscal pressures curtail treatment intensity, and client and administrative expectation may be that irregular attendance is better than no attendance at all. In fact, our research suggests that irregular attendance is not much better than no attendance. If clients do not improve unless a threshold of treatment intensity is met, it is a waste of mental health resources to continue offering the treatment to a client who does not come regularly. This does not mean that irregular attenders should be permanently excluded from cognitive remediation. Rather, they need to be engaged, and if that fails, they should subsequently be given opportunities to reapply for the treatment once they can make the commitment to attend regularly.

The issue of how to engage a client in cognitive remediation is beyond the scope of this article, but the implication of this study is that a positive response to the treatment is more likely to occur if clinicians offer a treatment that is engaging. Clients are more likely to be motivated for treatment if the cognitive remediation methodology reflects an understanding of how to enhance intrinsic motivation. The reader is referred to the works of Silverstein and Wilkniss17 and Medalia et al.10 to understand how extrinsic and intrinsic motivators can combine to enhance learning in schizophrenia.

Of less concern for clinicians who enroll clients in cognitive remediation groups are demographic or diagnostic factors. Instead, clinicians may want to focus on severity of illness and consider that the more severely impaired clients are not likely to respond to a short course of treatment. The importance of clinician qualifications has clear practice implications. Much more work needs to be done to understand why clients improved more with doctoral-level clinicians, but the immediate implication is that training matters. In a fiscally strained health care environment, it is important for administrators to appreciate that using poorly trained staff may be "penny wise and pound foolish."

Implications for Research
Given that our methodology involved a retrospective analysis of data obtained in a different context, there is a limit to the conclusions that can be drawn. However, if this study is viewed as a first step toward identifying some of the factors that predict a treatment response, several implications for future research become evident.

First, the issue of treatment intensity requires explication. Two sessions a week appear to be helpful; would 3 sessions be more helpful? Added to this is the question of treatment threshold. What is the optimal number of sessions? Does that vary for different types of patients? What are the factors that are likely to improve attendance? We did not find that baseline work habits were linked to treatment adherence, but clinician training was. The data on clinician credentials raises many questions about who is best suited to provide this treatment.

Related to the issue of treatment intensity is motivation for treatment. Typically, motivation in schizophrenia is addressed as an aspect of negative symptoms or in the context of programs that use external reinforcers to modify behavior. While both of these aspects of motivation are extremely important, they represent only part of a picture—a picture that will be enhanced by also considering intrinsic motivation. Intrinsic motivation is consistently associated with better treatment and educational outcomes in nonpsychiatric populations. Our research provides some evidence that the same finding is true in a psychiatric population; however, more rigorous study of the issue is necessary. This will require the development of measures that are valid and reliable in a schizophrenia population. Also, the relationship between competency on training tasks and motivation for treatment needs to be clarified.

Another consideration is the interaction of cognitive remediation with other treatments, especially pharmacologic ones. As pharmacologic and behavioral treatments for cognition evolve, it will be important to delineate the unique contribution of each of these approaches. Controlling for the type of antipsychotic treatment will facilitate that endeavor, and once the relative strengths of each approach are identified, we will be in a better position to evaluate if there are synergistic effects of combining treatment.


   Conclusion
Top
 Abstract
 Introduction
 Theoretical Approach
 Training Method: The...
 Defining Improvement
 Analytic Approach
 Results
 Discussion
 Conclusion
References
 
This research indicates that a positive response to cognitive remediation is most likely to be obtained if the client has some work habits in place before starting treatment; is motivated to attend treatment; participates regularly in a program that uses computer-based, strategy-oriented exercises; and works with a clinician who by virtue of training or interest is more sophisticated about treatment. While the issue of treatment intensity appears to be especially salient, whereby clients only show benefit in the context of regular attendance, there is evidence that to some extent patient and treatment variables intertwine to produce a positive response to cognitive remediation.


  Footnotes
 
To whom correspondence should be addressed; tel: 718-920-7311, e-mail: amedalia{at}aol.com.


   Acknowledgments
 
This research was partially supported by the Kessel Fellowship and Pibly Fund.


   References
Top
 Abstract
 Introduction
 Theoretical Approach
 Training Method: The...
 Defining Improvement
 Analytic Approach
 Results
 Discussion
 Conclusion
 References
 

  1. Heaton, RK, Gladsjo, JA, Palmer, B, Kuck, J, Marcotte, TD, Jeste, DV. The stability and course of neuropsychological deficits in schizophrenia. Arch Gen Psychiat 2001; 58:24–32.[Abstract/Free Full Text]

  2. Heinrichs, RW. The primacy of cognition in schizophrenia. Am Psychol 2005; 60:229–242.[CrossRef][Medline]

  3. Green, MF. What are the functional consequences of neurocognitive deficits in schizophrenia? Am J Psychiat 1996; 153:321–330.[Abstract/Free Full Text]

  4. Harvey, PD, Howanitz, E, Parrella, M, et al. Symptoms, cognitive functioning, and adaptive skills in geriatric patients with lifelong schizophrenia: a comparison across treatment sites. Am J Psychiat 1998; 155:1080–1086.[Abstract/Free Full Text]

  5. Olfson, M, Mechanic, D, Hansell, S, et al. Predicting medication noncompliance after hospital discharge among patients with schizophrenia. Psychiatr Serv 2000; 51:216–222.[Abstract/Free Full Text]

  6. Harvey, PD and Keefe, RSE. Studies of cognitive change in patients with schizophrenia following novel antipsychotic treatment. Am J Psychiat 2001; 158:176–184.[Abstract/Free Full Text]

  7. Velligan, DI and Miller, AL. Cognitive dysfunction in schizophrenia and its importance to outcome: the place of atypical antipsychotics in treatment. J Clin Psychiat 1999; 60:25–28.

  8. Brenner, HD, Hodel, B, Roder, V, Corrigan, PW. Treatment of cognitive dysfunction and behavioral deficits in schizophrenia: integrated psychological therapy. Schizophrenia Bull 1992; 18:21–26.

  9. Hogarty, GE, Flesher, S, Ulrich, R, et al. Cognitive enhancement therapy for schizophrenia: effects of a 2-year randomized trial on cognition and behavior. Arch Gen Psychiat 2004; 61:866–876.[Abstract/Free Full Text]

  10. Medalia, A, Revheim, N, Herlands, T. Remediation of Cognitive Deficits in Psychiatric Outpatients: A Clinician's Manual New York: Montefiore Medical Center Press 2002.

  11. APA/CAPP Task Force on Serious Mental Illness and Severe Emotional Disturbance. Training Grid Outlining Best Practices for Recovery and Improved Outcomes for People With Serious Mental Illness. Available at www.apa.org/practice/smi_grid2.pdf, accessed May 8, 2005.

  12. Kurtz, MM, Moberg, PJ, Gur, RC, Gur, RE. Approaches to cognitive remediation of neuropsychological deficits in schizophrenia: a review and meta-analysis. Neuropsychol Rev 2001; 11:197–210.[CrossRef][ISI][Medline]

  13. Krabbendam, L and Aleman, A. Cognitive rehabilitation in schizophrenia: a quantitative analysis of controlled studies. Psychopharmacology 2003; 169:376–382.[CrossRef][Medline]

  14. Twamley, EW, Dilip, VJ, Bellack, AS. A review of cognitive training in schizophrenia. Schizophrenia Bull 2003; 29:359–382.

  15. Suslow, T, Schonauer, K, Arolt, V. Attention training in the cognitive rehabilitation of schizophrenic patients: a review of efficacy studies. Acta Psychiat Scand 2001; 103:15–23.[CrossRef][ISI][Medline]

  16. Pilling, S, Bebbington, P, Kuipers, E, et al. Psychological treatments in schizophrenia II. meta-analyses of randomized controlled trials of social skills training and cognitive remediation. Psychol Med 2002; 32:783–791.[CrossRef][ISI][Medline]

  17. Silverstein, SM and Wilkniss, SM. At issue: the future of cognitive rehabilitation of schizophrenia. Schizophrenia Bull 2004; 30:679–692.

  18. Baron, RM and Kenny, DA. The moderator–mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol 1986; 51:1173–1182.[CrossRef][ISI][Medline]

  19. Kraemer, HC, Wilson, TG, Fairburn, CG, Agras, SW. Mediators and moderators of treatment effects in randomized clinical trials. Arch Gen Psychiat 2002; 59:877–884.[Abstract/Free Full Text]

  20. Cornblatt, B, Obuchowski, M, Roberts, S, Pollack, S, Erlenmeyer-Kimling, L. Cognitive and behavioral precursors of schizophrenia. Dev Psychopathol 1999; 11:487–508.[CrossRef][ISI][Medline]

  21. Erlenmeyer-Kimling, L, Rock, D, Roberts, SA, et al. Attention, memory, and motor skills as childhood predictors of schizophrenia-related psychoses: the New York High-Risk Project. Am J Psychiat 2000; 157:1416–1422.[Abstract/Free Full Text]

  22. Terrell, S and Rendulic, P. Using computer-managed instructional software to increase motivation and achievement in elementary school children. J Res Comput Educ 1996; 26:403–414.

  23. Medalia, A, Aluma, M, Tryon, W, Merriam, AE. Effectiveness of attention training in schizophrenia. Schizophrenia Bull 1998; 24:147–152.

  24. Medalia, A, Revheim, N, Casey, M. The remediation of problem-solving deficits in schizophrenia. Schizophrenia Bull 2001; 27:259–267.

  25. Medalia, A, Herlands, T, Baginsky, C. Cognitive remediation in the supportive housing setting. Psychiatr Serv 2003; 54:1219–1220.[Abstract/Free Full Text]

  26. Kendall, PC and Grove, WM. Normative comparisons in therapy outcome. Behav Assess 1988; 10:147–158.

  27. Evans, C, Margison, F, Barkham, M. The contribution of reliable and clinically significant change methods to evidence-based mental health. Evidence-Based Ment Hlt 1998; 1:70–72.

  28. Andrew, DM, Paterson, DG, Longstaff, HP. Minnesota Clerical Test Manual: Second Edition San Antonio: Psychological Corporation 1979.

  29. Sheldon, KM, Williams, G, Joiner, T. Self-Determination Theory in the Clinic: Motivating Physical and Mental Health New Haven: Yale University Press 2003.

  30. Choi, J and Medalia, A. Factors associated with a positive response to cognitive remediation in a community psychiatric sample: a brief report. Psychiatr Serv 2005; 56:602–604.[Abstract/Free Full Text]

  31. Eccles, JS and Wigfield, A. Motivational beliefs, values and goals. Annu Rev Psychol 2002; 53:109–132.[CrossRef][ISI][Medline]

  32. Cordova, D and Lepper, MR. Intrinsic motivation and the process of learning: beneficial effects of contextualization, personalization, and choice. J Educ Psychol 1996; 88:715–730.[CrossRef]

  33. Ryan, RM and Deci, EL. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol 2000; 55:68–78.[CrossRef][Medline]

  34. Bryson, G, Bell, M, Lysaker, P. The Work Behavior Inventory: a scale for assessing the work behavior for people with severe mental illness. Psychiatr Rehabil J 1997; 2:45–55.

  35. Bates, ME, Bowden, SC, Barry, D. Neurocognitive impairment associated with alcohol use disorders: implications for treatment. Exp Clin Psychopharm 2002; 10:193–212.[CrossRef][ISI][Medline]

  36. Fleming, SK, Blasey, C, Schatzberg, AF. Neuropsychological correlates of psychotic features in major depressive disorders: a review and meta-analysis. J Psychiat Res 2004; 38:27–35.[CrossRef][ISI][Medline]

  37. Medalia, A, Dorn, H, Watras-Gans, S. Treating problem-solving deficits on an acute-care psychiatric inpatient unit. Psychiat Res 2000; 97:79–88.[CrossRef][ISI][Medline]

  38. Wykes, T, Reeder, C, Corner, J, Williams, C, Eyeritt, B. The effects of neurocognitive remediation on executive processing in patients with schizophrenia. Schizophrenia Bull 1999; 25:291–307.

  39. Dunlop, WP, Cortina, JM, Vaslow, JB, Burke, MJ. Meta-analysis of experiments with matched groups or repeated measures designs. Psychol Methods 1996; 1:170–177.[CrossRef][ISI]


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?


This article has been cited by other articles:


Home page
 Journal of Special EducationHome page