Center for Teaching and Learning

Focused Inquiry Groups (FIGs) - Title III: Critical Thinking

Critical Thinking - Members: Scott Hildreth

Critical Thinking Assessment Pilot Project Investigating Subjective Gains in Scientific Literacy Fall 2008

Abstract

My primary goal was to look at how a Student-Learning-Outcome (SLO) rubric might be used to gauge critical thinking skills demonstrated on a particular assignment.  I also wanted to explore whether any noticeable improvement might be realized within the semester by using the assessment rubric for two similar assignments. And given that I was teaching two sections of the same class, one online and one on-campus, doing the same assignments, I also wanted to see whether there were any significant differences in the assessment results between those two delivery modes.

Application

The course chosen for investigation in the pilot was Astronomy 20: An Introduction to Astronomy: Stars, Galaxies, and the Universe, a three-unit general education science lecture class with no prerequisites, taken most often by students with little or no science background. 

The student learning outcome chosen for this project involved scientific literacy skills:

“Read, analyze, and critique a magazine or newspaper article about a current discovery in astronomy, specifically identifying parts of the process of science, including evidence of observation, past research, testable hypotheses, experimental results, data analysis, support for or against prior theory, peer review of results, and publishing.”

The assignment tied specifically to this SLO was “Astronomy in the News”; the version used in the on-campus class is available at http://online.chabotcollege.edu/shildreth/astronomy/06news.html .  The online version was identical, differing only in due-dates and mode of submission (posted online, rather than turned in on paper.)  This assignment is typically assigned in Week 12 of the semester, after students have been exposed to the overall process of science, and the pieces of the traditional “scientific method” through numerous contemporary and historical examples of both good and bad science in astronomy.  Students have the freedom to choose their article from any source(s), and one of the key criteria used in scoring their work (and evaluating their achievement in terms of critical thinking) was looking at their sources and the depth of analysis provided in the articles they chose.

The chosen critical thinking rubric was “Rubric for the Analytical Assessment of Critical Thinking across the Curriculum” from Valencia Community College, June 10, 2005. 

In an attempt to discern whether students were improving in their understanding of the process of science through the term, the same critical thinking rubric was used to assess results on a prior assignment (Week 4) about whether the full moon has an affect on human behavior, available at http://online.chabotcollege.edu/shildreth/astronomy/06moon.html .  Here, students were provided with two articles, and asked to compare and contrast them regarding their perceived scientific merit.  One was a pseudo-scientific treatment written by an astrologer, claiming the full moon had proven effects upon people, with numerous inaccuracies, false data, and no references.  The other was a scientific rebuttal of the idea, citing numerous studies and data that could be double-checked independently.

The two assignments are sufficiently different as to make comparisons between overall student success very difficult.  But I nonetheless see benefit by looking at the results of specific students over the two assignments, and seeing whether they showed noticeable differences in their ability to pick up on key elements of the process of science over the term.

Summary of Observations

1)      Using the critical thinking rubric was straightforward, and matching its five key “think indicator” elements to the student essays was not difficult. I found that I needed to comb through its terms to find matches to the expected elements from my selected assignments, but once those matches were identified, scoring was quick. 

2)      Comparing results within same class over the term using two different assignments was complicated because of student drops and missed work.  One unfortunate consequence of using Blackboard for submission of student homework is that once they are dropped or withdraw, all of their submitted work is immediately unavailable.  There were at least a dozen assignments from the 4^th week submitted in both sections that were unavailable, and another half-dozen from the 12^th week.

3)      Of the on-campus class, 24 assignments from the 4^th week were available for analysis, and 16 assignments from the 12^th week.  Some of the students originally in the class had withdrawn, and others had not submitted the first or second assignment. There were 13 pairs of assignments from students who had submitted both.  Of those 13, seven (7) showed some improvement between the early and later assessments in terms of holistic scores.  Four (4) showed a decline holistically, and two (2) showed no difference.   Those showing no difference had achieved the “accomplished” mark with the first assignment, and did not change by the second assignment. 

For the summary tables below, a numeric rubric scoring system was used to match the levels of achievement specified:

0 = No Achievement

1 = Beginning

2 = Developing

3 = Competent

4 = Accomplished

4)      Of the on-line class, 18 assignments from the 4^th week were available for analysis, and 15 assignments from the 12^th week.    There were 11 pairs of assignments from students who had submitted both.  Of those 11, eight (8) showed some improvement between early and later assessments in terms of holistic scores. Three (3) showed no difference.

Conclusions

The critical thinking rubric used in our pilot program seems to be a useful assessment option, given its comprehensive categorization for achievement levels.  It was fairly easy to adapt the rubric to existing assignments, and score student work.   And if there are multiple assignments during the term that could be assessed for critical thinking achievement, using the same rubric does allow for some comparison of student work during the term. 

The power of the rubric, and the assessment overall, will possibly be shone over multiple applications during the upcoming years.  Of great immediate benefit is the chance to ask the questions about how our students are doing in terms of acquiring and honing their critical thinking skills, and how we can get a sense of that success.  Of equal benefit is the chance to talk with colleagues across the campus, share assessable questions and strategies, and share ideas about our shared craft of teaching.