1. "Validation" in our preliminary study meant three things: (1) the correlation of survey items with the parallel "objective observer" item scored by the research team member; (2) the correlation of individual survey items with "validation factor scores" built by combining a set of coded responses made by the research team members from a variety of evidence (interviews, observations, coding of artifacts, and the "objective observer" report); and (3) the correlation of an index combining substantively related teacher survey responses with the validation factor scores described above. The indices and validation factor scores represented efforts to operationalize theoretically derived components and sub-components of the overall construct of constructivist-compatible pedagogy. Items measuring substantively similar beliefs and practices were combined in the analysis because exploratory factor analysis found no way to consistently distinguish them based on factor loadings. (In the current study, our factor analyses did consistently distinguish between belief and practice survey items. See Becker and Anderson (1998) and Becker (2000b) for more details about the validation study.
2. The median number of items in an index was 5. The range was from 3 to 10 items per index.
3. There were 20 survey items measuring meaningful tasks, 23 measuring cognitively demanding tasks, and 19 measuring social learning.
4. One survey item, "work in small groups to come up with a joint solution or approach to a problem or task," is included in both the "group-work" and "problem-solving" measures.
5. Survey items that had low communalities during initial factor analyses were excluded for subsequent analyses. We did not use factor scores in this study, but used factor analysis as a technique for decisions about item inclusion in indices. We used oblique rotation within a principal axis factoring method because we envisioned the different types of practice, while distinct, to be nonetheless related and therefore not orthogonal constructs. Belief items, when they were included, almost always loaded on separate factors from practice items, in contrast to the findings of the preliminary validation study. In general, it was during this stage that we observed that project-related items were least well correlated with the other practice items; i.e., always producing their own factor. In contrast, reflective writing and the other meaningful thinking items appeared to be very closely related. Group-work items tended to load with the meaningful thinking items, pointing to a possible contrast between projects and group work in how each relates to the level of cognitive challenge in a classroom.
6. The self-assessment item also incorporated "serious discussion" among students as well as written assessments.
7. Of course, teachers of self-contained classes have their students in class for many more hours per week (and are the primary source of their students' home assignments).
8. Many people would regard teacher-led discussion as no less likely to promote meaningful thinking than student-led discussion. This may be the case, although for the students who are presenting to their peers, the requirement to communicate with an audience would seem to make greater thoughtfulness more likely for them. Overall, in our data, frequency of student-led discussions is positively associated with other elements of cognitive challenge, such as reflective writing and problem-solving, whereas frequent teacher-led discussions is negatively correlated with those elements.
9. The Problem-Solving index, like others developed here, combines scores on component survey items (four items in this case, including one not shown in the Table), by standardizing each component item before taking the mean across items. The transformation to z-scores shown similarly maintains a constant level of variability across indices for purposes of comparing across different measures.
10. The particular Meaningful Thinking score used is the within-subject z-score, defined so that the average score for every subject is set at 0 with a standard deviation of 1.
11. Another way of presenting the same information is through correlation coefficients between Problem-Solving and Meaningful Thinking. Those are shown in Appendix Table A-4.
12. Another interpretation of the positive correlation between Meaningful Thinking scores and Problem-Solving scores (or between any pairs of measures that are based on "frequency" estimates or similar Likert scales is that the association is a function of different questionnaire response styles among teachers. That is, some teachers are more likely to give higher frequency estimates than other teachers are, regardless of the item prompt. Such response styles no doubt play a role. However, the size of these associations—most of those reported involve effect sizes above 0.5—suggest that more than response styles are at work here.
13. The fourth Problem Solving item, "working in groups to jointly solve a problem," was excluded from the combined Cognitive Challenge index because it was assigned instead to the "Group Work" component of constructivist practice.
14. Although the survey item has explicit "problem-solving" elements in it, the effect size was nearly as strong (E.S.=0.7) when only considering the Meaningful Thinking sub-scale—i.e., excluding the problem-solving items.
15. This analysis contrasts teachers who report that product-for-use and demonstration-to-outsider activities were part of their teaching practice at all versus teachers who did not do these things. The earlier descriptive analysis dichotomized these variables at "monthly or more" versus less often or not at all. However, because even monthly frequency is so rare, dichotomizing at "monthly or more" would have produced too few cases for most subjects to permit calculation of reliable effect sizes.
16. Correlation coefficients in Appendix Table A-6 produce essentially the same results.
17. One distinction comes from the exploratory factor analyses. While project activities consistently loaded on a separate factor, group work items often loaded with other items linked with cognitive challenge.
18. For the actual mean z-scores on each of the constructivist practice components for each subject and level combination shown in Table 21, see Table A-8.
19. We do not want to deny the value of traditional methods for producing learning under the right circumstances, for example when time is short and specific skills are desired. The question is more the quality of the learning that is taking place, and the meaningfulness of the goals. When they are developed and used effectively, there are instructional methods that are consistent with so-called "traditional teaching" that could engender deep thinking in students under specific circumstances, for example an inspirational lecture before an activity where important issues are raised. Are we saying a teacher should never lead the whole class in a discussion? No, just that the amount of time spent on that does take away from other activities. Judicious use of traditional methods may be entirely appropriate at times. Clearly, this survey emphasized constructivist-compatible practices and did not attempt to measure the quality of so-called traditional instruction as carefully as it did the many faces of constructivist teaching. It is possible that if we had asked enough questions, we would have found traditional-compatible practices that could promote deep thinking, or that would impact some other measures.
20. The same question can be asked about the relationship between infrequent use of traditional practices and frequency of use of projects and of group work. The correlations among the three components of what we call "Active Learning Strategies" are shown in Appendix A-4, separately for each subject-level combination. Correlations between infrequent traditional practices and use of projects are quite high, averaging nearly 0.4 for the 15 categories of teachers examined, nearly the identical average as found for the correlation between projects and group work. However, the correlations between infrequent use of traditional practices and group work are substantially smaller, averaging only 0.13 for the same categories of teachers.
21. In this case, a few of the correlation coefficients produce somewhat different results. In particular, the correlation for middle school math teachers is not as high as the effect size would indicate (r=0.2). See Appendix Table A-6.
22. The smaller association between infrequent use of transmission practices and frequent use of cognitively challenging ones (compared to the associations reported among other pairs of components) is at least partly due to the effects of questionnaire response style on measured effect sizes. Here we are finding small positive asssociations between low frequency of one type of practice and high frequency of another type of practice, whereas the other findings (high positive associations) are between high frequency of two types of practices. Thus, whereas in the other findings response style patterns resulted in measured associations that are probably higher than are true, in fact, in this case the response style effects work to diminish the size of the association. Consequently, to the extent that response style patterns are operating, the true effect sizes between infrequent traditional practices and Cognitive Challenge are larger (more positive) than the measured effect sizes.
23. As indicated earlier, factor analyses were used to decide which items to group together, but indices combining items were constructed by taking mean values of items, after standardizing each item to have similar variance.