Schizophrenia Bulletin Advance Access originally published online on January 31, 2005
Schizophrenia Bulletin 2005 31(1):73-83; doi:10.1093/schbul/sbi008
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The Micro-Module Learning Tests: Work-Sample Assessments of Responsiveness to Skills Training
Associate Professor of Psychiatry, Center for Cognitive Medicine, University of Illinois at Chicago, Chicago, IL
Adjunct Professor of Psychology, University of California at Los Angeles Neuropsychiatric Institute, Department of Psychiatry, Los Angeles, CA
Project Coordinator, University of Illinois at Chicago
Send reprint requests to Dr. C.J. Wallace, Psychiatric Rehabilitation Consultants, P.O. Box 2867, Camarillo, CA 93011–6022; e-mail: cwall886{at}concentric.net.
 |
Abstract |
|---|
 Top Abstract Rationale for Development of...Administration and Scoring of...Development of the MMLTFour Studies of the...DiscussionReferences |
|---|
While outcome of psychiatric rehabilitation has been successfully predicted by cognitive tests, efforts to design a measure to assess responsiveness to rehabilitation have been lacking. In this report, we describe the rationale for and development of a face-valid measure of responsivity to the three core components of skills training: responsiveness to verbal instruction, ability to learn from viewing the behavior of a model, and ability to demonstrate skills observed during a subsequent role-play. Seven alternate forms of the new measure, called the Micro-Module Learning Test (MMLT), demonstrated adequate internal consistency and alternate-form reliability. We also present results from four studies in which the MMLT was used to collect normative data as well as data on relationships with symptoms, cognitive tests, and treatment outcome. Results indicate that the MMLT is associated with cognitive factors found to predict treatment outcome in prior studies (e.g., verbal memory and fluency), as well as lesser investigated functions such as theory of mind ability. In addition, MMLT scores were correlated negatively with psychotic disorganization and positively with performance during a full-length skills training group. The MMLT appears to be a reliable and valid measure for rapidly assessing responsiveness to skills training procedures.
Keywords: Schizophrenia / rehabilitation / assessment / skills training / cognition / disorganization / outcome
Although the results of numerous studies indicate that psychiatric rehabilitation helps individuals with schizophrenia improve their social and community functioning (Silverstein 2000
; Wallace et al. 2000), the results also indicate that many individuals fail to realize the maximum benefit from their rehabilitation services. Assuming the services have been properly designed and implemented, an explanation for these suboptimal outcomes may be individuals' lack of "rehabilitation readiness." Anthony and colleagues (Anthony et al. 1983; Cohen and Mynks 1993) have defined readiness as individuals' ability to take an active role in their rehabilitation plans and articulate goals in critical life functions such as work, education, recreation, interpersonal relationships, and parenting. This definition emphasizes individuals' motivation, demonstrated in their commitment to be actively engaged in setting the goals and directions for their own services in collaboration with their service providers.
Alternately, rehabilitation readiness may be defined in terms of individuals' possessing the minimum cognitive functioning needed to benefit from their rehabilitation services. The results of numerous studies conducted during the past decade have consistently identified several cognitive functions that mediate the outcomes of rehabilitation services (Green 1996; Silverstein et al. 1998b; Green et al. 2000). To date, however, these results have not been used to develop a measure to assess individuals' "cognitive readiness" for rehabilitation. Such a measure could help service providers distinguish individuals who would likely benefit from rehabilitation services from those who need preparatory interventions such as cognitive training. Furthermore, such a measure could be administered to assess the moderator effects of variables such as amounts and types of medications, symptom patterns, and demographic characteristics on responsiveness to rehabilitation interventions.
This article describes the development of such a measure, the Micro-Module Learning Test (MMLT). The MMLT is specifically designed to assess individuals' ability to respond to one form of psychiatric rehabilitation intervention, skills training. After providing the rationale for the development of the MMLT, we describe the steps involved in its development, then present data from four studies that demonstrate the MMLT's concurrent and predictive validity.
|
Rationale for Development of the MMLT |
|---|
|
TopAbstractRationale for Development of...Administration and Scoring of...Development of the MMLTFour Studies of the...DiscussionReferences |
|---|
For Which Intervention Should Readiness Be Assessed?.
The term psychiatric rehabilitation encompasses an array of diverse interventions intended to improve individuals' social and community functioning. Despite the interventions' common focus on improved functioning, their diversity ensures that no single measure will fully assess individuals' ability to benefit from psychiatric rehabilitation. However, there is one intervention, skills training, that not only fits particularly well the theory and philosophy of psychiatric rehabilitation but also can be easily integrated into other psychiatric rehabilitation interventions and form a "core" component that links these diverse interventions. Furthermore, it is widely conducted (Wallace et al. 2000); it can be "manualized" so that the fidelity with which it is conducted can be systematically monitored and corrected (e.g., University of California, Los Angeles [UCLA], Social and Independent Living Skills Modules, Liberman and Corrigan 1993); its instructional value has been repeatedly demonstrated in several studies (see reviews by Heinssen et al. 2000; Silverstein 2000; Wallace et al. 2000); and generalization of the skills it trains has been enhanced by several recently developed techniques (Wallace et al. 2000) that can increase the effectiveness of the training. Hence, skills training was chosen as the intervention for assessing individuals' cognitive rehabilitation readiness.
What Is Known About the Effects of Cognitive Functioning on Skills Training?.
Schizophrenia is typically characterized by deficits in cognitive functions such as visual information processing (Green 1998; Knight and Silverstein 1998); attention (Nuechterlein 1991); working memory (Park and Holzman 1992; Docherty et al. 1996); short-term memory (Calev et al. 1987); executive functioning (Goldberg et al. 1987); context processing (Cohen and Servan-Schreiber 1992; Silverstein et al. 1996; Cohen et al. 1999; Phillips and Silverstein 2003); and social perception and cognition (Corrigan and Penn 2001). While no single profile of deficits characterizes all individuals, almost all are deficient in at least one area of cognitive functioning (Morice and Delahunty 1996; Palmer et al. 1997).
There is substantial evidence that deficits in these cognitive functions are "rate-limiting" factors in skills training and rehabilitation outcomes. Bowen et al. (1994), Corrigan et al. (1994), Kern et al. (1992), Mueser et al. (1991), and Silverstein et al. (1998a, 1998b) have all reported that deficits in sustained attention and verbal memory are associated with reduced skills acquisition. These deficits in sustained attention and verbal memory are also associated with poorer vocational outcomes and reduced social problem solving (reviewed in Green 1996). Moreover, the relationships between cognitive deficits and outcomes are relatively independent of symptoms, and cognitive functioning is a better predictor of community functioning and outcome than positive or negative symptoms (e.g., Mueser et al. 1991; Green 1998). All of these data indicate that impairments in attention and memory are critical components of cognitive readiness for rehabilitation, and they indicate that the assessment of readiness would be fruitful if conducted within a skills training format.
Despite the consistency of findings relating impaired cognition to poorer skills training outcomes, a significant limitation of past studies has been the similarity of the methods used to assess the independent and dependent variables. For example, the measure of the outcome of training in several studies that found verbal memory to be a significant predictor (e.g., Kern et al. 1992; Bowen et al. 1994; Silverstein et al. 1998a, b) was a measure of how much information the individuals could recall from that presented in the module, with little or no assessment of enacting new skills. It is possible that the prior studies overestimated the importance of specific cognitive abilities for rehabilitation readiness by not incorporating assessments of the full range of behaviors that define responsiveness to skills training.
Hence, the MMLT was developed as an assessment of individuals' performance in brief "work samples" of social and independent living skills training. These work samples were specifically designed to include the three core components of skills training: verbal instruction, modeling, and role-play. Thus, they measured individuals' cognitive abilities to encode, process, and enact information presented to them. Furthermore, although traditional neuropsychological measures have been associated with performance in skills training as detailed above, the work-sample format of the MMLT, by using material from actual skills training packages (see below), enhanced the ecological validity and applicability of its results compared with those of general cognitive measures. In addition to rehabilitation readiness (i.e., serving as a predictor of outcome), the MMLT was also intended as an analogue measure of skills training performance, with which the effects of other variables (e.g., type and dose of psychoactive medication, cognitive rehabilitation, diagnosis) on outcome could be ascertained.
The remainder of this article describes the administration and development of the MMLT and then presents the results from four studies of its concurrent and predictive validity.
|
Administration and Scoring of the MMLT |
|---|
|
TopAbstractRationale for Development of...Administration and Scoring of...Development of the MMLTFour Studies of the...DiscussionReferences |
|---|
Because the MMLT presents information to be learned and retained, repeatedly assessing individuals by presenting the same information would be completely confounded by the carryover effects from previous administrations. Hence, seven versions of the MMLT were developed to provide the option of repeated assessments. Each version requires from 35 to 50 minutes to administer. The administration and scoring procedures are as follows.
Comprehend Verbal Instruction. Each version begins with the assessor reading two paragraphs from one of the UCLA Social and Independent Living Skills Modules (described below) and asking from three to four questions per paragraph to assess the respondent's comprehension of the material. The paragraphs, questions, and criteria for scoring the answers are printed on a scoring sheet; if the answer is correct, the assessor asks the next question, and if the answer is incorrect, the assessor reads a simplified version of the material that includes fewer nonessential details. The simplified version is printed on the scoring sheet; once it has been read, the assessor repeats the question. If the answer is correct, the assessor asks the next question, but if the answer is again incorrect, the assessor reads an even simpler version at a slower pace, "highlighting" the correct answer with his or her vocal inflections. This even simpler version is printed on the scoring sheet; once it has been read, the assessor repeats the question again, and, regardless of the answer, moves on to the next question.
Thus, each question presents three "opportunities" for the respondent to master the task of comprehending the verbal instruction. A score of 3 is given if the respondent's first answer is correct; a score of 2 if the respondent's answer is correct after the first reading of the simplified material; a score of 1 if the respondent's answer is correct only after the reading of the even simpler material; and a score of 0 if the respondent's answer is incorrect throughout. The scores for each question are summed to obtain totals for each of the two paragraphs.
Comprehend Visual Demonstration (Modeling). A similar procedure is used to assess the respondent's comprehension of the visual demonstrations of the skills to be acquired. For each version of the MMLT, the assessor plays a videotape that demonstrates one of the skills taught in the same skills training module from which the paragraphs were drawn. The videotape has periodic stops that cue the assessor to ask questions to assess the respondent's comprehension. Each version's videotape has six stops, with one question per stop. The questions and the criteria for scoring the answers are printed on the scoring sheet. If the answer is correct, the assessor continues the tape and plays it until the next stop; if the answer is incorrect, the assessor plays a subsequent, shortened version of the videotape segment that focuses on the scenes that portray the correct answer. The question is repeated, and a correct answer results in continuing the tape to the next stop. An incorrect answer results in playing an even shorter version of the tape segment that pauses at the scene on the tape that portrays the correct answer. The assessor verbally highlights the correct answer on the paused tape, and then repeats the question. Regardless of the answer, the assessor continues the tape and moves to the next stop. Scoring of the answers is handled as it is for the verbal comprehension questions.
Role-Play. After the videotape has been viewed, the respondent is asked to practice the skills that have just been demonstrated. The criterion behaviors, five to six for each version of the MMLT, are printed on the scoring sheet, and a respondent's complete and correct performance results in the end of the assessment. If the respondent's practice is incomplete or incorrect, the assessor provides verbal feedback that details the missing/incorrect behaviors. The respondent is asked to practice again, and another incomplete or incorrect role-play results in lengthier, more detailed, and focused feedback. If the respondent's practice is again incomplete or incorrect, the tape is replayed, and the respondent is asked to practice for a third time. The assessment ends with this practice. Each practice is scored for the presence/absence of each criterion behavior, and a total is calculated across all practices. A respondent whose practice is complete and correct on the first or second attempt is given a perfect score for each of the unneeded attempts.
It is important to note that the MMLT assesses not only what people know but also how well they can learn and benefit from feedback. Thus, the MMLT can be viewed as a method for dynamic assessment (Budoff 1987) as opposed to a static method, such as the methods used for most cognitive and intellectual testing (which assess performance without a learning context). Because psychiatric rehabilitation is, by its very nature, learning based, dynamic assessment methods are particularly well suited for the testing of rehabilitation readiness and responsiveness.
|
Development of the MMLT |
|---|
|
TopAbstractRationale for Development of...Administration and Scoring of...Development of the MMLTFour Studies of the...DiscussionReferences |
|---|
Development of the seven versions of the MMLT began with selection of their content. The content had to be interesting enough to hold respondents' attention, or the results would reflect their lack of interest rather than their cognitive functioning. The content also had to be relevant to respondents' general fund of information, or the results might reflect their limited knowledge rather than their cognitive functioning. A potential source of appropriate content was the UCLA Social and Independent Living Skills Modules, which had been administered in several studies (e.g., Wallace et al. 1992
, 1999; Kopelowicz et al. 1998; Liberman et al. 1998). Comments from participants in these studies indicated that the information and skills presented in the modules were familiar, interesting, and relevant to their concerns about improving their community functioning. Furthermore, the studies' results indicated that participants significantly and substantially improved their knowledge of the information and performance of the skills presented in the modules and retained their knowledge and performance over extended followup periods. Thus, the modules seemed to be a source of content suitable for inclusion in the versions of the MMLT. The modules' manuals, videotapes, and workbooks were then reviewed to identify material that fit the limitations imposed by the MMLT's format—relatively short paragraphs, brief but complete segments that could be excerpted from the modules' videotapes, and rapidly enacted and easily scored role-plays. Three versions of the MMLT were constructed from the content of the Basic Conversation Skills Module, two from the Recreation for Leisure Module, and one each from the Medication Management and Symptom Management Modules.
The seven versions were then administered to 15 treatment-refractory individuals diagnosed with schizophrenia. All of the administrations were recorded on videotape, and a randomly selected 20 of the 105 (7 x 15) were viewed and scored by a second rater. Totals were calculated for each administration's verbal comprehension, visual demonstration, and role-play tasks, and correlations were calculated between the two raters' totals for the 20 administrations. The correlations were 0.92 for the verbal task, 0.91 for the visual task, and 0.88 for the role-play.
Coefficient alphas were also calculated for each version's three tasks and overall total. The results ranged from 0.75 to 0.86 for the seven versions' verbal instruction task, 0.77 to 0.89 for the visual comprehension task, 0.68 to 0.87 for the role-play, and 0.79 to 0.92 for the overall total. Thus, the MMLT demonstrated strong internal consistency, despite the limitations of the sample in terms of size and homogeneous level of patient disability. Test-retest correlations over 3- to 6-month intervals were calculated for 8 of the 15 individuals who had earlier participated in three versions of the MMLT (a different three for each participant). The test-retest correlations were 0.58 for the verbal comprehension task, 0.72 for the visual comprehension (modeling) task, 0.68 for the role-play, and 0.73 for the overall total. Scores of six of the eight subjects were higher for the second testing than the first, suggesting that the six may have learned and retained the information given in the first administration, even though the two administrations were 3 to 6 months apart.
Several laboratory-based measures of cognitive functioning were also administered: the UCLA Degraded Stimulus Continuous Performance Test (DSCPT; Nuechterlein et al. 1992), the Digit Span Distractibility Test (DSDT; Oltmanns and Neale 1975), the Span of Apprehension Test (Asarnow et al. 1991), the Sustained Attention Test (SAT; Silverstein et al. 1998a), the Wisconsin Card Sorting Test (WCST; Berg 1948, Heaton 1981), the California Verbal Learning Test (CVLT; Delis et al. 1987), and the Controlled Oral Word Association Test (COWAT; Spreen and Benton 1977). The total scores from the seven MMLTs were combined into one overall total and then correlated with the cognitive measures. Given the few subjects and the large number of measures, the correlations were evaluated with an alpha of 0.01. This provided some protection for the inflation of alpha balanced with avoiding type II errors. The results indicated that the overall total was significantly and substantially correlated with the total score on the COWAT (0.62), the CVLT total across its five trials (0.59), and the DSDT distraction total correct (0.65). This suggests that performance on the MMLT is associated with general verbal fluency, attention, and memory, which seems reasonable given the MMLT's demands on processing verbal information (e.g., the videotaped demonstrations include verbal annotations), responding verbally, and categorizing/storing information for later processing and responding.
|
Four Studies of the Concurrent and Predictive Validity of the MMLTs |
|---|
|
TopAbstractRationale for Development of...Administration and Scoring of...Development of the MMLTFour Studies of the...DiscussionReferences |
|---|
Study 1.
This study extended the normative data generated with the small sample of participants in the MMLT development to a larger and more heterogeneous sample of individuals. The study also obtained data about the correlations of the MMLT with several cognitive measures and investigated the predictive validity of the MMLT—that is, the correlation between scores on the MMLT and performance in a skills training program. Details of this study have been published as part of a larger study of predicting psychiatric rehabilitation outcomes (Silverstein et al. 1998b), and only the key results will be summarized here.
A total of 26 individuals, 15 male and 11 female, participated in the study. All were diagnosed with chronic schizophrenia, had been continuously hospitalized for at least 3 years, were on stable medication regimens, and were residing on an intensive behavioral rehabilitation inpatient unit. All completed the Recreation for Leisure–I version of the MMLT, as well as a test of simple reaction time (Silverstein et al. 1998b), the DSCPT, the SAT, the Rey Auditory Verbal Learning Test (RAVLT; Rey 1964), the COWAT, the Hinting Task (a measure of theory of mind, Corcoran et al. 1995), and the WCST. After completing the measures, individuals participated in the UCLA Basic Conversation Skills Module. The module was conducted for three sessions per week with groups of six participants for approximately 6 months. At the end of the group, participants' skills were assessed using the Comprehensive Module Test for the Basic Conversation Skills Module (Wallace et al. 1992).
Because the correlations between the MMLT's three tasks and the other variables were similar across the three tasks (see Silverstein et al. 1998b), only the correlations of the MMLT total are presented. The relationships between the MMLT and the cognitive measures were similar to those found in the MMLT development: significant correlations between the MMLT and verbal fluency (COWAT; r = 0.66, p = 0.01), verbal memory (Trial 5 of the RAVLT, r = 0.60, p = 0.01; Trial 1 of the RAVLT, r = 0.44, p < 0.10), WCST errors (r = 0.48, p = 0.05), and the Hinting Task (r = 0.77, p = 0.001). The MMLT was not correlated with simple reaction time, sensitivity and response bias indexes from the DSCPT, or accuracy and error rate data from the SAT. These results not only confirm past findings on the role of verbal short-term memory in skills acquisition but also indicate that verbal fluency and inferential reasoning, two processes known to be impaired in schizophrenia but rarely investigated in the context of skills training, are related to responsiveness to skills training procedures. Thus, responsiveness to skills training seems to be associated with higher level cognitive functions, including memory, reasoning, and fluency. This is consistent with models of skills training (Wallace and Boone 1984) that posit the need to address three overall processes: encoding/receiving (short-term memory), processing (reasoning), and sending/communicating (fluency).
Additionally, the correlation between the MMLT and the Comprehensive Module Test was significant (r = 0.42, p = 0.05), suggesting that the MMLT is predictive of the outcomes of skills training over a relatively long interval between prediction and outcome. Furthermore, the predictive value of the MMLT in this study was not an artifact of the use of the same content in the MMLT and the module. Rather, the predictive value of the MMLT was demonstrated within the context of the training methods per se, not a confounded combination of training methods and content.
Study 2. The purposes of this study were to gather additional information about the relationships between the MMLT and various cognitive measures in a sample of individuals with recent onset of a schizophrenia spectrum disorder and to compare this sample's MMLT results with those of a non–mentally ill sample. This study was conducted in conjunction with an ongoing longitudinal project at UCLA that investigated the influence of etiologic and treatment factors on the course of recent-onset schizophrenia. The longitudinal project was in progress when this study was conducted.
Participants.
Participants were enrolled in the longitudinal study during their inpatient treatment based on a DSM–IV (American Psychiatric Association 1994) diagnosis of schizophrenia or schizoaffective disorder with no evidence of an organic disorder and no current dependence on alcohol or illicit drugs. Recruitment was completed within 2 years of the onset of their disorder, and there were no contraindications for maintenance treatment with risperidone. Once enrolled, participants received comprehensive clinical services, including social and independent living skills training, case management, family education, and administration of risperidone.
Normative data for this study were obtained from non–mentally ill individuals who had been recruited from the staff of various local treatment facilities based on no evidence of a DSM–IV diagnosis of a major mental or organic disorder, no evidence of a first degree relative's having a major mental illness, and no current dependence on alcohol or illicit drugs. The criteria were assessed by administration of screening questions adapted from the Structured Clinical Interview for DSM–IV Axis I Disorder (SCID I/P Version 2.0; First et al. 1995).
Methods.
Three measures were administered to both the mentally ill and the non–mentally ill participants: the Medication Self-Management and Symptom Self-Management versions of the MMLT, and the CVLT. In addition, the mentally ill participants were periodically assessed with several measures, including the DSCPT, the Span of Apprehension Test, and the UCLA expanded Brief Psychiatric Rating Scale (BPRS; Lukoff et al. 1986). These measures were not repeated for this study; rather, the results of the administration closest in time to the administration of the two versions of the MMLT were used for the analyses.
Results. The 39 mentally ill and 25 non–mentally ill participants were similar in marital status (85% single and 72% single, respectively) and ethnicity (56% Anglo, 19% Latino, 13% African-American, and 12% Asian; 64% Anglo, 16% Latino, 16% African-American, and 4% Asian, respectively). Significantly more of the non–mentally ill participants were male (64% vs. 24%) and employed (100% vs. 34%), and their average age and years of education were somewhat but not significantly lower (28.52 vs. 30.38 years old; 13.54 vs. 14.48 years of education). For the entire sample of 64 participants, education was significantly correlated with the total score across both MMLTs (r = 0.500, df = 64, p < 0.001), and education was used as a covariate to increase the power of the between-group analyses.
The differences between the mentally ill and non–mentally ill participants on the MMLT were analyzed with one-way analyses of covariance. The results indicated that the two groups were significantly different on each of the three tasks for each version of the MMLT (smallest F = 4.12, df = 1, 62, p = 0.047) and for each version's total score (Medication MMLT, F = 14.705, df = 1, 62, p < 0.001; Symptom MMLT, F = 4.02, df = 1, 62, p = 0.048).
Additionally, two CVLT measures—total recalled over the five trials and semantic clustering—were separately analyzed to determine the significance of the differences between the groups. Neither age nor education was significantly correlated with either CVLT measure, and so the analyses were conducted with one-way analyses of variance (ANOVAs). As expected, the non–mentally ill participants performed better on both measures (smaller F = 9.49, df = 1, 62, p < 0.001).
Because the mentally ill participants in this study were less chronically ill than those who participated in the MMLT development, the MMLT results were analyzed to determine the coefficient alphas for each task and overall total in each version. The results were quite similar to those found in the initial development of the MMLT; the alphas were 0.76, 0.75, 0.72, and 0.82 for the Medication version, and 0.69, 0.56, 0.72, and 0.76 for the Symptom version.
Correlations were calculated between each version of the MMLT and each of four factors of the BPRS (thought disorder, anxiety/depression, hostility/suspiciousness, negative symptoms) as well as the total of all 24 items. It was anticipated that the MMLT would not be correlated with symptoms, much as had been reported for the cognitive measures such as the DSCPT. Indeed, all of the correlations between the MMLT and symptoms were not significant (largest r = –0.162). However, participants' symptoms were well controlled (average item score of 1.67 on the 1–7 BPRS scale), and this restricted variance did not allow an adequate test of the relationships between symptom severity and MMLT performance.
Correlations were also calculated between the cognitive measures and the combined total of the two versions of the MMLT. Because there were a large number of summary scores that could have been calculated from the three cognitive measures, conducting all possible correlations would have resulted in an unacceptably high risk of type I error. To maintain alpha at an acceptable level, correlations were conducted for only indexes that were considered to be the key ones for each cognitive test. These summary scales included two from the DSCPT (false alarm rate and A prime), two from the Span of Apprehension Test (number correct for array sizes 1 and 12), and two from the CVLT (number correct across all five trials and the semantic clustering ratio).
The two DSCPT indexes (false alarm rate and A prime) were positively skewed (i.e., absolute value of the ratio of skewness to standard error of skewness >2), and so these scores were transformed to the log (base 10) of the original scores. Similar to the results found in the MMLT development, neither of the DSCPT measures was correlated significantly with the MMLT. In contrast, both of the Span of Apprehension Test scores and one of the two CVLT scores, number correct across all five trials, were significantly correlated with the MMLT total (r = 0.537 for Span array size 1, p = 0.01; r = 0.457 for array size 12, p < 0.05; r = 0.427 for CVLT total correct, p = 0.05).
Study 3. The primary purpose of this study was to determine the contrasted groups validity of the MMLT. Three groups of individuals participated: those diagnosed with schizophrenia or schizoaffective disorder who were long-term residents of a state hospital, those similarly diagnosed who were being discharged from a brief inpatient stay to the community where they had lived successfully for years, and those similarly diagnosed who were stable outpatients and had not received inpatient treatment during at least the previous 6 months. A secondary purpose was to gather additional information about the relationships between the MMLT, the BPRS, and several cognitive measures.
Participants.
A total of 34 individuals, 25 males and 9 females, mean age of 38.43 years (standard deviation [SD] of 9.44), participated in the study. All were enrolled in various treatment programs conducted at the Westchester Division of New York Presbyterian Hospital. Eleven were enrolled in the Second Chance Program (Silverstein et al. 2002), an intensive behavioral rehabilitation inpatient program for treatment-refractory state hospital patients; 9 were receiving treatment in a brief-stay inpatient program (average length of stay of 20 days); and 14 were stable outpatients. All met DSM–IV criteria for either schizophrenia or schizoaffective disorder, based on a several-step review of participants' records. Admitting diagnoses were confirmed or modified by the programs' chief psychiatrists, who were also members of the research group; these diagnoses were reviewed by members of the research group who did not work in the programs of patients whose records they reviewed; and these diagnoses were reviewed by the first author, who was the program director for the hospital division that included all of the programs from which the participants were recruited. Participants were considered eligible for this study only if all three steps produced the same diagnoses. All of the short-stay inpatients and the outpatients had been treated at the hospital for at least the previous 3 years, and most had participated in prior studies in which the SCID I/P Version 2.0 (First et al. 1995) had been administered by this research group. Discrepancies between past and current diagnoses were resolved by a consensus among the members of the research group after each had further reviewed the clinical records. All participants were receiving antipsychotic medication, mostly clozapine or olanzapine, combined with secondary antipsychotic or other (e.g., mood stabilizer, anxiolytic) medications.
Methods.
In addition to the Recreation for Leisure Module–1 version of the MMLT, participants were administered the COWAT; the Hayling Sentence Completion Test, Parts 1 and 2 (Burgess and Shallice 1996a, 1996b); the Brixton Spatial Anticipation Test (Burgess and Shallice 1996a, 1996b); the Hinting Task; the BPRS; and the Shipley Institute of Living Scale vocabulary subscale (Zachary 1991). The Hayling Test measures the time required for respondents to generate appropriate (Part 1) and inappropriate (Part 2) words to end sentences. The Brixton Test measures nonlinguistic context processing in the form of visual pattern detection and anticipation of next-in-sequence.
Results. The MMLT total and the scores per task were not significantly correlated with gender, age, number of prior hospitalizations, age at first diagnosis, age at first hospitalization (all p > 0.8), or years of education (p > 0.20). The differences among the three groups on the MMLT total and scores per task were separately analyzed with one-way ANOVAs. The main effect of group was significant for the MMLT total (F = 3.80, df = 2, 31, p < 0.05); the results of a Fisher's Least Significant Difference (LSD) post hoc analysis indicated that the long-stay individuals performed significantly worse (mean = 56.36, SD = 18.91) than the other two groups, with nonsignificant differences between them (short-stay inpatient mean = 76.67, SD = 20.01; outpatient mean = 75.43, SD = 19.31). The same pattern of results was found for the role-play task (F = 4.36, df = 2, 31, p < 0.05); a post hoc LSD analysis indicated that the long-stay individuals performed significantly worse (mean = 15.73, SD = 7.21) than the other two groups, who were not significantly different from one another (short-stay inpatient mean = 23.55, SD = 7.67; outpatient mean = 23.86, SD = 7.38). For the verbal instruction task, the results of the ANOVA were not significant (p = 0.15), but the order of the three groups' means was the same (long-stay individuals' mean = 13.00, short stay individuals' mean = 16.67, and outpatients' mean = 16.57). For the visual demonstration task, there was a trend toward a significant main effect (F = 3.24, df = 2, 31, p = 0.053); a post hoc LSD analysis again indicated that the long-stay individuals (mean = 27.64, SD = 8.12) performed significantly worse than the other two groups, who were not significantly different from one another (short-stay inpatient mean = 36.44, SD = 8.43; outpatient mean = 35.00, SD = 9.06).
Examination of the distributions of the cognitive test scores indicated that scores on the Hayling Test, Part 1, were positively skewed, while scores on the Hinting Task were negatively skewed (i.e., absolute values of the ratio of skewness to standard error of skewness > 2). Therefore, in the analyses reported below, the raw Hayling Test, Part 1, scores were replaced by log-transformed values, and the raw Hinting Task scores were replaced by square root–transformed values.
Similar to results presented previously, the MMLT was significantly correlated with scores on the COWAT (r = 0.49, p < 0.005) and the Hinting Task (r = 0.65, p < 0.001). Additionally, the MMLT was correlated with the number of errors on the Brixton task (r = –0.65, p < 0.001) and scores on Part 1 of the Hayling Test (r = –0.43, p < 0.05) but not Part 2 (p > 0.23). There was also a significant correlation with the Shipley Institute of Living Scale vocabulary subscale–derived IQ estimate (r = 0.59, p < 0.001), suggesting that the MMLT involves a verbal learning component that, like similar tasks, is associated with general measures of IQ.
The BPRS ratings were combined into six factors (psychotic disorganization, emotional blunting, paranoia, anxiety/depression, hallucinations/delusions, and agitation/elation; Spaulding et al. 1999) and correlated with the MMLT total. Unlike Study 2, this study found that the MMLT total was negatively correlated with all six factors, significantly so for psychotic disorganization (r = –0.61, p < 0.001), emotional blunting (r = –0.47, p < 0.01), paranoia (r = –0.42, p < 0.01), and hallucinations/delusions (r = –0.47, p = 0.005). The contrast between the two studies' results likely reflects the differences in their participants' psychopathology. The mean BPRS rating of participants in Study 2 was 1.67, a rating between not present and very mild; this study included more heterogeneous participants ranging from stable outpatients to long-stay inpatients with severe persisting symptoms.
The pattern of BPRS results does suggest that performance on the MMLT was inversely associated with the psychosis-related symptoms of cognitive dysfunction and thought disorder. In addition, the negative relationship between the MMLT and emotional blunting may indicate that individuals with negative symptoms such as flat affect and withdrawal are less responsive to external stimuli.
Study 4. The purposes of this study were to obtain information about the MMLT performance of another sample of chronically ill individuals and to further investigate the relationships of MMLT scores with symptoms and cognitive functioning.
Participants.
Participants were 33 long-stay residents of a state hospital's behavioral rehabilitation unit at the Lincoln Regional Center (see Spaulding et al. 1999). All were diagnosed as having either schizophrenia or schizoaffective disorder based on SCID interviews administered by trained research assistants who were involved in a long-standing program of research. Of the 33 participants, 22 were male and 11 were female; their mean age was 41.90 years (SD = 10.72); their mean education was 12.1 years (SD = 1.78); and their mean length of stay during the current hospitalization was 35.92 months (SD = 49.15).
Methods.
As in Study 3, the Recreation for Leisure Module–1 MMLT, the COWAT, the Hayling Test, the Brixton Spatial Anticipation Test, the Hinting Task, and the BPRS were administered to all participants. The BPRS scores were combined into the six factors scored in Study 3 (Spaulding et al. 1999).
Results. As in Study 3, the MMLT total and the scores per task were not correlated with gender, age, number of prior hospitalizations, age at first diagnosis, age at first hospitalization (all p > 0.8), or years of education (p > 0.23). Similar to Study 3, examination of the distributions of the cognitive test scores revealed significant positive skewness for Hayling Test, Part 1, scores, and significant negative skewness for Hinting Task scores (i.e., absolute values of the ratio of skewness to standard error of skewness >2). Therefore, the raw scores for these tests were replaced by log-transformed and square root–transformed values, respectively. Similar to the results reported in Study 3, the MMLT total score was significantly correlated with scores on the Hinting Task (r = 0.42, p < 0.005) and the number of errors on the Brixton Test (r = –0.47, p < 0.005). Unlike the results of Study 3, the correlation with the COWAT did not reach significance (r = 0.22, p < 0.18), and the correlations with Parts 1 and 2 of the Hayling Test were small and nonsignificant (r < 0.10).
Also contrary to Study 3, this study found that the correlations of the MMLT total with the six factors of the BPRS were not significant (largest r = 0.19), except for the correlation with the psychotic disorganization factor (r = –0.54, p < 0.001), which was replicated. Given the homogeneity of the participants in this study compared with the heterogeneity of those in Study 3, the BPRS results likely reflect the limited variability among these participants' psychopathology, much as in Study 2.
|
Discussion |
|---|
|
TopAbstractRationale for Development of...Administration and Scoring of...Development of the MMLTFour Studies of the...DiscussionReferences |
|---|
The MMLT was developed to provide samples of the "work" participants would have to perform to acquire social and independent living skills. The primary aim was to give practitioners a reliable and valid method to assess individuals' cognitive readiness to benefit from skills training. Assuming "motivational readiness," practitioners could use the results of the MMLT to allocate resources for skills training to cognitively ready individuals and delay such allocations for those not cognitively ready.
A secondary aim was to develop the MMLT as a measure of the effects of other variables such as medications, cognitive rehabilitation, demographic characteristics, and family support. Determining the effects of these other variables on a full course of skills training would require a good deal of time and money, and performance in brief samples of the training might provide results that could be reasonably extended to the complete training but for far lower costs.
Therefore, the seven versions of the MMLT were specifically designed to implement the three major procedures of skills training (verbal instruction, encoding of visually demonstrated behaviors, and role-playing) using content extracted from widely used skills training materials, the UCLA Social and Independent Living Skills Modules. This article reported psychometric data obtained during test development, as well as data from four studies that investigated the MMLT's relationships with various cognitive functions.
The psychometric data indicated that the seven alternate forms of the MMLT have good to excellent alternate form reliability and internal consistency, and practitioners and researchers can exercise the option to have repeated assessments using the seven alternate versions. Evidence of the MMLT's construct validity was collected in each of the four studies and in the MMLT development. The correlations between the MMLT and performance on various cognitive tests (table 1) were similar to those reported for the relationships between the outcomes of a full course of skills training and the same cognitive measures, and MMLT total score correlated at the p = 0.05 level with ability to perform taught skills after 6 months of participation in a skills training group.
|
Verbal memory scores, assessed by the RAVLT in Study 1 and the CVLT in the development data and Study 2, were consistently correlated with MMLT scores. Verbal fluency, assessed by the COWAT, was significantly correlated with MMLT scores in the development data and Studies 1 and 3. Although the correlation with the COWAT did not reach statistical significance in Study 4, it was in the predicted direction (r = 0.22, p < 0.18). The correlation with the Shipley Institute of Living Scale vocabulary subscale–derived IQ estimate was significant in the one study in which it was examined, supporting the COWAT results and suggesting that the MMLT involves a significant verbal learning component.
The correlations between the theory of mind tests and the MMLT in Studies 1, 3, and 4 were significant and may indicate that the ability to infer the mental state of others is involved in learning skills. Although this fits the "encoding" and "processing" phases of the receive-process-send assumptions about the requirements for skillful social responding (Wallace and Boone 1984), a more parsimonious explanation is that the Hinting Task relies heavily on verbal memory. A respondent must remember the details of a brief story while trying to infer the meaning in the final statement.
Correlations between the MMLT and the Hayling Test were inconsistent across Studies 3 and 4, while the scores on the Brixton Test were correlated with the MMLT in both studies. This is difficult to reconcile because both tests are viewed as assessing the supervisory attentional system (Burgess and Shallice 1996a, 1996b), a cognitive model of executive functioning that has been found to be impaired in individuals with schizophrenia (Marczewski et al. 2001). While there are certainly subtle differences between the Hayling and Brixton Tests that may explain the inconsistent results of these studies (e.g., the Hayling Test involves generation or inhibition of a dominant verbal response, while the Brixton test involves recognizing patterns in sequences of visual stimuli and anticipating future patterns based on this contextual information), data are needed to determine the extent to which specific executive functions are involved in learning skills. The consistent correlations of the MMLT with the COWAT and the correlation with WCST errors in Study 1 support the hypothesis that some aspect of executive functioning is involved.
The relationships between the MMLT and symptoms were significant primarily for Study 3, with no significant findings for Study 2 and only one significant finding for Study 4. Despite these differences, a clear picture emerged. The strongest relationship in Study 3, that disorganization was inversely related to MMLT performance, was replicated in Study 4, and this was the only MMLT-symptom relationship that was found in more than one study. While disorganization was not correlated with MMLT scores in Study 2, this is likely for one or both of two reasons. First, BPRS factor scores were based on a weighted system (based on a recent factor-analytic study of the BPRS with chronic schizophrenia patients) in Studies 3 and 4 but not in Study 2. Second, and more likely (because the unweighted scoring system has demonstrated validity in other studies), patients in Study 2 were relatively unsymptomatic, while the patients in Studies 3 and 4 were highly symptomatic and in many cases considered treatment-refractory (with the majority of patients across those two studies either residing at a state hospital or having been recently transferred from a state hospital to a specialized private hospital unit for treating such patients). Therefore, there was a much greater range of symptoms among patients in Studies 3 and 4 compared with Study 2, which would enable a relationship between symptoms and test scores to be more easily detected.
Of interest, while past research has generally found cognitive deficits to be better predictors of rehabilitation outcomes than symptoms, most prior studies used a two-factor symptom model (i.e., positive and negative symptoms). Recent studies suggest, however, that disorganized symptoms are strong predictors of community functioning (Hoffman and Kupper 1997; Norman et al. 1999; Smith et al. 2002). The fact that this symptom factor was the most closely related to MMLT scores supports the idea that disorganization is a factor in response to skills acquisition, as measured by the MMLT. This hypothesis is also supported by data showing that disorganized symptoms are consistently related to disorganized cognitive activity as measured by laboratory perceptual and linguistic tasks, and that perceptual and cognitive fragmentation on such tasks is related to poorer response to psychiatric rehabilitation (Silverstein et al. 1998b; Phillips and Silverstein 2003). All of these data suggest that when symptom-outcome relationships are assessed, it is important to study patients that demonstrate a sufficiently broad range of symptoms, to assess the full range of symptoms (as opposed to simply examining positive vs. negative symptoms), and to interpret different results across studies in terms of the extent to which different symptoms were present. One important issue for future studies is the nature of the disorganization-outcome relationship across narrow and global outcomes. For example, is the manner in which cognitive disorganization interferes with the ability to process and respond to information presented in a skills training format the same as that which is involved in more global outcomes such as social and vocational functioning? Clarifying how cognitive disorganization interferes with functioning across outcome domains is important, because this can be the first step toward developing methods for disorganization reduction that can be included in psychiatric and cognitive rehabilitation interventions.
The results of Studies 2 and 3 provide evidence of the MMLT's contrasted groups validity. Study 2 indicated that individuals with no evidence of mental illness performed significantly and substantially better than individuals with schizophrenia who had a relatively recent onset of their symptoms that were well controlled at the time of the assessment. This finding suggests that the MMLT results are not just a function of the waxing and waning of individuals' symptoms. The results of Study 3 also indicate that individuals with schizophrenia who have demonstrated the ability to live successfully in the community perform better than those who have not been able to maintain community living. This suggests that the MMLT, while intended as a measure of responsiveness to skills training, is also tapping into processes related to reduced adaptability in general.
In conclusion, the preliminary data reported here on the MMLT suggest that it is a reliable and valid measure of responsiveness to the core components involved in skills training. It is also a face-valid measure that can provide clinicians with a qualitative, as well as a quantitative, sense of how patients are likely to respond to such procedures. Widespread use of the MMLT may thus allow for more rapid, data-driven decision making regarding which patients are ready to benefit from skills training interventions, thereby also assisting with the cost-effectiveness of rehabilitative treatment.
|
References |
|---|
|
TopAbstractRationale for Development of...Administration and Scoring of...Development of the MMLTFour Studies of the...DiscussionReferences |
|---|
-
American Psychiatric Association. DSM–IV: Diagnostic and Statistical Manual of Mental Disorders 4th ed. Washington, DC APA1994.
Anthony, WA, Cohen, MR, Cohen, BF. The philosophy, treatment process and principles of the psychiatric rehabilitation approach. New Directions for Mental Health Services 17: pp. 67–791983.
Asarnow, RF, Granholm, E, Sherman, T. Span of apprehension in schizophrenia. In Steinhauer, SR, Gruzelier, JH, Zubin, J (Eds.). Handbook of Schizophrenia: Neuropsychology, Psychophysiology, and Information Processing Amsterdam, The Netherlands Elsevier pp. 335–3701991.
Berg, EA. A simple objective test for measuring flexibility in thinking. Journal of General Psychology 39:15–221948.[Medline]
Bowen, L, Wallace, CJ, Glynn, SM, Nuechterlein, KH, Lutzker, JR, Kuehnel, TG. Schizophrenics' cognitive functioning and performance in interpersonal interactions and skills training procedures. Journal of Psychiatric Research 28:289–3011994.[CrossRef][Web of Science][Medline]
Budoff, M. Measures for assessing learning potential. In Lidz, CS (Ed.). Dynamic Assessment New York, NY Guilford Press pp. 173–1951987.
Burgess, PW and Shallice, T. Bizarre responses, rule detection, and frontal lobe lesions. Cortex 32:241–259 a1996.[Web of Science][Medline]
Burgess, PW and Shallice, T. Response suppression, initiation and strategy use following frontal lobe lesions. Neuropsychologia 34:263–273 b1996.[CrossRef][Web of Science][Medline]
Calev, A, Korin, Y, Kugelmass, S, Lerer, B. Performance of chronic schizophrenics on matched word and design recall tasks. Biological Psychiatry 22:699–7091987.[CrossRef][Web of Science][Medline]
Cohen, JD, Barch, DM, Carter, C, Servan-Schreiber, D. Context-processing deficits in schizophrenia: Converging evidence from three theoretically motivated cognitive tasks. Journal of Abnormal Psychology 108:120–1331999.[CrossRef][Web of Science][Medline]
Cohen, JD and Servan-Schreiber, D. Context, cortex dopamine: A connectionist approach to behavior and biology in schizophrenia. Psychological Review 99:45–771992.[CrossRef][Web of Science][Medline]
Cohen, M and Mynks, D. Compendium of Activities for Assessing and Developing Readiness for Rehabilitation Services Boston, MA Boston University Center for Psychiatric Rehabilitation1993.
Corcoran, R, Mercer, G, Frith, CD. Schizophrenia symptomatology and social inference: Investigating "theory of mind" in people with schizophrenia. Schizophrenia Research 17: pp. 5–131995.[CrossRef][Web of Science][Medline]
Corrigan, PW and Penn, DL. Social Cognition and Schizophrenia Washington, DC American Psychological Association2001.
Corrigan, PW, Wallace, CJ, Schade, ML, Green, MF. Learning medication self-management skills in schizophrenia: Relationships with cognitive deficits and psychiatric symptoms. Behavior Therapy 25: pp. 5–151994.[CrossRef]
Delis, DC, Kramer, JH, Kaplan, E, Ober, BA. The California Verbal Learning Test (Manual) San Antonio, TX Psychological Corporation1987.
Docherty, NM, Hawkins, KA, Hoffman, RE, Quinlan, D, Rakfeldt, J, Sledge, WH. Working memory, attention, and communication disturbances in schizophrenia. Journal of Abnormal Psychology 105: pp. 212–2191996.[CrossRef][Web of Science][Medline]
First, MB, Spitzer, RL, Gibbon, M, Williams, JBW. Structured Clinical Interview for DSM–IV Axis I Disorder, Patient Edition (SCID I/P), Version 2.0 New York, NY Biometrics Research Department, New York State Psychiatric Institute1995.
Goldberg, TE, Weinberger, DR, Berman, KF, Pliskin, NH, Podd, MH. Further evidence for dementia of the prefrontal type in schizophrenia? A controlled study of teaching the Wisconsin Card Sorting Test. Archives of General Psychiatry 44: pp. 1008–10141987.
Green, MF. What are the functional consequences of neurocognitive deficits in schizophrenia? American Journal of Psychiatry 153:321–3301996.
Green, MF. Schizophrenia From a Neurocognitive Perspective Boston, MA Allyn and Bacon1998.
Green, MF, Kern, RF, Braff, DL, Mintz, J. Neurocognitive deficits and functional outcome in schizophrenia: Are we measuring the right stuff? Schizophrenia Bulletin 26: pp. 119–1362000.
Heaton, RK. Wisconsin Card Sorting Test Manual Odessa, FL Psychological Assessment Resources1981.
Heinssen, RK, Liberman, RP, Kopelowicz, A. Psychosocial skills training for schizophrenia: Lessons from the laboratory. Schizophrenia Bulletin 26: pp. 21–462000.
Hoffman, H and Kupper, Z. Relationships between social competence, psychopathology and work performance and their predictive value for vocational rehabilitation of schizophrenic outpatients. Schizophrenia Research 23:69–791997.[CrossRef][Web of Science][Medline]
Kern, RS, Green, MF, Satz, P. Neuropsychological predictors of skills training for chronic psychiatric patients. Psychiatry Research 43:223–2301992.[CrossRef][Web of Science][Medline]
Knight, RA and Silverstein, SM. The role of cognitive psychology in guiding research on cognitive deficits in schizophrenia. In Lenzenweger, MF and Dworkin, RH (Eds.). Origins and Development of Schizophrenia: Advances in Experimental Psychopathology Washington, DC APA Press pp. 247–2951998.
Kopelowicz, A, Wallace, CJ, Zarate, R. Teaching psychiatric inpatients to re-enter the community: A brief method of improving the continuity of care. Psychiatric Services 49:1313–13161998.
Liberman, RP and Corrigan, PW. Designing new psychosocial treatments for schizophrenia. Psychiatry 56:238–2491993.[Web of Science][Medline]
Liberman, RP, Wallace, CJ, Blackwell, GA, Kopelowicz, A, Vaccaro, JV, Mintz, J. Skills training vs. occupational therapy for persons with persistent schizophrenia. American Journal of Psychiatry 155:1087–10911998.
Lukoff, D, Nuechterlein, KH, Ventura, J. Manual for the expanded Brief Psychiatric Rating Scale (BPRS). Schizophrenia Bulletin 12:594–6021986.
Marczewski, P, Van der Linden, M, Laroi, F. Further investigation of the supervisory attention system in schizophrenia: Planning, inhibition, and rule abstraction. Cognitive Neuropsychiatry 6:175–1922001.
Morice, R and Delahunty, A. Frontal/executive impairments in schizophrenia. Schizophrenia Bulletin 22:125–1371996.
Mueser, KT, Bellack, AS, Douglas, MS, Wade, JH. Prediction of social skill acquisition in schizophrenic and major affective disorder patients from memory and symptomatology. Psychiatry Research 37:281–2961991.[CrossRef][Web of Science][Medline]
Norman, RM, Malla, AK, Cortese, L, Cheng, S, Diaz, K, McIntosh, E, McLean, TS, Rickwood, A, Voruganti, LP. Symptoms and cognition as predictors of community functioning: A prospective analysis. American Journal of Psychiatry 156:400–4051999.
Nuechterlein, KH. Vigilance in schizophrenia and related disorders. In Steinhauer, SR, Gruzelier, JH, Zubin, J (Eds.). Handbook of Schizophrenia: Neuropsychology, Psychophysiology, and Information Processing Vol. 5 Amsterdam, The Netherlands Elsevier pp. 397–4331991.
Nuechterlein, KH, Dawson, ME, Gitlin, M, Ventura, J, Goldstein, MJ, Snyder, KS, Yee, CM, Mintz, J. Developmental processes in schizophrenic disorders: Longitudinal studies of vulnerability and stress. Schizophrenia Bulletin 18:387–4251992.
Oltmanns, TF and Neale, JM. Schizophrenic performance when distractors are present: Attentional deficit or differential task difficulty? Journal of Abnormal Psychology 84:205–2091975.[CrossRef][Web of Science][Medline]
Palmer, BW, Heaton, RH, Paulsen, JS, Kuck, J, Braff, D, Harris, MJ. Is it possible to be schizophrenic yet neuropsychologically normal? Neuropsychology 11:437–4461997.[CrossRef][Web of Science][Medline]
Park, S and Holzman, PS. Schizophrenics show spatial working memory deficits. Archives of General Psychiatry 49:975–9811992.
Phillips, WA and Silverstein, SM. Convergence of biological and psychological perspectives on cognitive coordination in schizophrenia. Behavioral and Brain Sciences 26:65–1382003.
Rey, A. L'examen clinique en psychologie. [The clinical examination in psychology.] Paris, France Presses Universitaires de France1964.
Silverstein, SM. Psychiatric rehabilitation of schizophrenia: Unresolved issues, current trends and future directions. Applied & Preventive Psychology 9: pp. 227–2482000.
Silverstein, SM, Light, GA, Palumbo, DR. The Sustained Attention Test: A measure of cognitive dysfunction. Computers in Human Behavior 14:463–475 a1998.[CrossRef]
Silverstein, SM, Matteson, S, Knight, RA. Reduced top-down influence in auditory perceptual organization in schizophrenia. Journal of Abnormal Psychology 105:663–6671996.[CrossRef][Web of Science][Medline]
Silverstein, SM, Schenkel, LS, Valone, C, Nuernberger, S. Cognitive deficits and psychiatric rehabilitation outcomes in schizophrenia. Psychiatric Quarterly 69:169–191 b1998.
Silverstein, SM, Hatashita-Wong, M, Bloch, A. A second chance for people with "treatment-refractory" psychosis. Psychiatric Services 53:4802002.
Smith, TE, Hull, JW, Huppert, JD, Silverstein, SM. Recovery from psychosis in schizophrenia and schizoaffective disorder: Symptoms and neurocognitive rate-limiters for the development of social behavior skills. Schizophrenia Research 55:229–2372002.[CrossRef][Web of Science][Medline]
Spaulding, WD, Reed, D, Sullivan, M, Richardson, C, Weiler, M. Effects of cognitive treatment in psychiatric rehabilitation. Schizophrenia Bulletin 25:657–6761999.
Spreen, O and Benton, AL. Neurosensory Center Comprehensive Examination for Aphasia (NCCEA) Revised ed. Victoria, Canada University of Victoria Neuropsychological Laboratory1977.
Wallace, CJ and Boone, SE. Cognitive factors in the social skills of schizophrenic patients: Implications for treatment. In Spaulding, WD and Cole, JK (Eds.). Nebraska Symposium on Motivation, 1983: Theories of Schizophrenia and Psychosis Lincoln, NE University of Nebraska Press pp. 283–3181984.
Wallace, CJ, Liberman, RP, Kopelowicz, A, Yaeger, D. Psychiatric rehabilitation. In Andreasen, N and Gabbard, GO (Eds.). Treatment of Psychiatric Disorders: The DSM–IV edition Washington, DC American Psychiatric Press pp. 1093–11122000.
Wallace, CJ, Liberman, RP, MacKain, SJ, Blackwell, G, Eckman, TA. Effectiveness and replicability of modules for teaching social and instrumental skills to the severely mentally ill. American Journal of Psychiatry 149:654–6581992.
Wallace, CJ, Tauber, R, Wilde, MS. Teaching fundamental workplace skills to persons with serious mental illness. Psychiatric Services 50:1147–11491999.
Zachary, RA. Shipley Institute of Living Scale Revised Manual Los Angeles, CA Western Psychological Services1991.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||