Angelo - Doing Assessment As If Learning = Matters Most

=20

Home | About = AACSB | Newsroom |=20 Contact=20 Us

=20

Member=20 Services
- Membership=20 Benefits=20
- Busin= ess=20 Services Directory=20
- Membership=20 Applications=20
- Educational=20 Institutions=20
- Corporate,=20 Government, and Non-Profit=20
- Online=20 Member Directory=20
- Business=20 Development=20
- Special=20 Lists=20
- Affinity=20 Groups=20
- Re= gional=20 Deans Associations=20
- Associ= ation=20 Counterparts=20
- GFME=20
- Management= =20 Education Links=20
- AACSB=20 Governance=20
- Newsroom=20
- Staff = Contacts=20
Accreditation
- Accreditation=20
- Accredi= ted=20 Institutions=20
- Promoti= ng=20 Accreditation=20
- The=20 Standards=20
- Accredita= tion=20 Process=20
- Proce= ss=20 Documents=20
- Accreditation=20 Guidelines=20
- Peer=20 Review Training=20
- White=20 Papers=20
- Committees= =20
- Association=20 Links=20
- Staff=20 Contacts=20
- Glossary=20
Conferences
- Conferen= ces=20
- Seminars=20
- Asia=20 Events=20
- Web= inars=20
- Bridge = Programs=20
- S= eminar=20 Host Opportunity=20
- Sponsorship=20 Opportunities=20
- = Exhibit=20 Opportunities=20
- Requests=20 for Proposals=20
- Presenter=20 Resources=20
- Conference=20 Handouts=20
- Staff=20 Contacts
Publications
- BizEd=20 Magazine=20
- = BizEd=20 Article Reprints=20
- Publications=20
- eNEWSLINE=20
- Management=20 Education at Risk=20
- Sustaining=20 Scholarship in Business Schools=20
- Ethics = Education in Business Schools=20
- T= he=20 Business School Rankings Dilemma
Data=20 & Reasearch
- Business= =20 School Profiles=20
- Academic=20 Leader Assessment=20
- D= ataDirect=20
- Reports<= /A>=20
- Surveys<= /A>=20
- Report=20 Catalog
Resource=20 Centers
- Ass= essment=20
- Business and=20 Business Schools=20
- Dean's=20 Resources=20
- Do= ctoral=20 Education=20
- Ethics=20 and Sustainability=20
- Fac= ilities=20
- Impac= t=20 of Research=20
- Ranki= ngs=20
- Value=20 of Management Education
Business School=20 Jobs

=20

A= ccreditation=20 Standards

Reading Lists =20

Doing Assessment As If Learning = Matters=20 Most
By=20 Thomas A. Angelo

From=20 the May 1999 AAHE Bulletin

After=20 nearly two decades of uphill struggle, the assessment = movement=20 has reached a promising plateau. In general, U.S. = higher=20 education has moved beyond unproductive, dualistic = debates=20 (remember "four legs good, two legs bad" from = Animal=20 Farm?) over whether assessment should focus on=20 accountability or improvement. Today, most faculty and = academic administrators have finally, if reluctantly, = come to=20 accept that dealing with both is a political and an = economic=20 inevitability. Nonetheless, most of us think = assessment should=20 be first and foremost about improving student learning = and=20 secondarily about determining accountability for the = quality=20 of learning produced. In short: Though accountability = matters,=20 learning still matters most.

This=20 realignment of opinion hasn=92t occurred by chance. = Since the=20 mid 1980s, a dedicated and widely dispersed cadre of = activists=20 and opinion leaders from higher education = associations,=20 regional and professional accrediting agencies, = disciplinary=20 societies, and campuses have urged us to use = assessment to=20 improve learning quality and productivity. In = response, tens=20 of thousands of faculty and administrators on hundreds = of=20 campuses have endured speeches, labored in workshops, = and=20 conferred at conferences on assessment. Collectively, = on=20 committees and task forces, they=92ve produced cubic = yards of=20 plans, projects, statements, and reports. Thousands = have=20 become familiar with, even expert in, assessment. = Examples of=20 clever adaptations and creative invention abound. All = this=20 effort has been expended despite the fact that = involvement in=20 assessment typically counts for little or nothing in = pay or in=20 tenure, retention, and promotion decisions. Thus, when = most=20 academics "do assessment," personal and professional = values=20 motivate them. And the strongest of those intrinsic = motivators=20 is undoubtedly the desire to improve student=20 learning.

So=20 the good news is that, over the past two decades, = we=92ve made=20 impressive progress in assessment. On the other hand, = we still=20 don=92t have much solid evidence of learning = improvement. Why=20 hasn=92t so much hard work by so many smart and = dedicated people=20 led to better outcomes?

Why=20 Hasn=92t Assessment Led to More Learning=20 Improvement?
In=20 his December 1997 AAHE Bulletin article = "Organizing for=20 Learning," Peter Ewell provides an insightful response = to this=20 question. Commenting on a range of academic change=20 initiatives, assessment included, Ewell argues that = our lack=20 of success in improving collegiate learning stems from = two=20 common flaws in our change strategies: Initiatives = have been=20 implemented without a deep understanding of what = "collegiate=20 learning" really means and which strategies are likely = to=20 promote it; and initiatives have, for the most part, = been=20 attempted piecemeal within and across institutions. = Echoing=20 Ewell, I=92ll argue that most assessment efforts have = resulted=20 in little learning improvement because they have been=20 implemented without a clear vision of what "higher" or = "deeper" learning is and without an understanding of = how=20 assessment can promote such learning. I=92ll also = propose that=20 our piecemeal attempts stem partly from a mechanistic, = additive model of assessment, which needs to be = replaced by a=20 transformative assessment-as-culture-change model if = we=92re to=20 make real progress.

Three=20 Steps Toward Transformative Assessment
The=20 assessment movement needs a new, more compelling = vision to=20 reinspire and reorient our efforts. To steal a phrase = from=20 Steve Gilbert of the TLT=20 Group (the Teaching, Learning, and = Technology=20 Affiliate of AAHE), we need a "vision worth working = toward."=20 Second, we need a different concept of assessment = itself, a=20 new mental model. And third, we need research-based = guidelines=20 for effective assessment practice that will increase = the odds=20 of achieving more productive instruction and more = effective=20 learning.

A=20 Vision Worth Working Toward: Assessment for Learning=20 Communities
Our=20 vision of and goals for assessment have led us to=20 less-than-optimal outcomes. For example, we=92ve = sometimes=20 confused means and ends, doing assessment as if the = assessment=20 process matters most, losing ourselves in the = technique and=20 method. More often we act as if winning matters most = =97 whether=20 the prize is status, higher funding, or accreditation. = Though=20 means and extrinsic ends are important, it=92s time to = put the=20 highest priority on doing assessment as if learning = matters=20 most.

Let=20 me make a comparison to the U.S. space program. NASA, = under=20 great political pressure, first focused on winning the = space=20 race with the Soviets. When NASA made gains, the = agency was=20 rewarded with accolades and bigger appropriations. = Along the=20 way, many NASA scientists and engineers became fixated = on the=20 scientific and technological aspects of the enterprise = =97 on=20 designing more ambitious, more complex, and more = costly=20 equipment and missions. The Hubble Space Telescope and = the=20 space station are outcomes of those aims. The = overarching aims=20 and vision of space exploration =97 to discover new = worlds and=20 extend our understanding of the universe =97 were = often lost in=20 the shuffle. Thus, even before the Soviets = disappeared, most=20 Americans had lost interest in the race and didn=92t = share=20 NASA=92s fascination with the technology. Now NASA is = struggling=20 to rebuild public support by focusing more attention = on=20 cultivating the intrinsic human interest in discovery = =97 our=20 collective desire "to boldly go where no one has gone=20 before."

In=20 a widely read and discussed article, Robert Barr and = John Tagg=20 developed the thesis that U.S. higher education is in = the=20 midst of a historic shift from a teaching-centered to = a=20 learning-centered paradigm ("From Teaching to = Learning,"=20 Change, Nov/Dec 1995). In this emerging = paradigm, Barr=20 and Tagg see the primary purpose of colleges and = universities=20 as producing learning rather than providing = instruction, and=20 traditional teaching as only one of many means of = learning=20 production. Drawing inspiration from Barr and Tagg, I=20 predicted in a 1997 article that one major outcome of = this=20 paradigm shift would be the transformation of colleges = and=20 universities from "teaching factories" into "learning=20 communities" (see "The Campus as Learning Community," = AAHE=20 Bulletin, May 1997). By learning communities, I = meant=20 carefully designed groupings of students and faculty = working=20 intensively and collaboratively toward shared, = significant=20 learning goals =97 often by focusing on themes that = cut across=20 several traditional disciplines.

As=20 K. Patricia Cross pointed out in ACPA=92s July/August = 1998 issue=20 of About Campus, strong support for engaging = students=20 in interactive, collaborative learning communities can = be=20 found in the research on learning outcomes, on = development,=20 and on cognition and motivation. She also noted = important=20 pragmatic reasons for creating learning communities, = among=20 them workforce training and citizenship = education. =20

In=20 my view, the learning communities ideal and many of = its best=20 current manifestations represent a vision worth = working=20 toward, not just for assessment but also for = educational=20 change efforts in general. Having the construction of = learning=20 communities as a goal is quite different from aiming = at=20 incrementally improving our present system. It=92s a = whole new=20 ball game. If we accept, at least for the moment, = creating=20 productive learning communities as an orienting = vision, then=20 our concept of assessment must also change to support = that=20 vision.

Changing=20 Our Mental Models: Assessment as Culture=20 Transformation
The=20 second reason our assessment efforts have been less = successful=20 than desired has to do with our concept of assessment = itself.=20 Some view assessment as a mechanistic, technical = process =97 a=20 collection of monitoring and problem-solving devices = that can=20 be dropped into or added onto existing academic = programs, much=20 as we might connect an antismog device to a car = engine.=20 Others, probably conditioned by program evaluation and = accreditation experiences, see assessment as a = necessary,=20 periodic bother, like a visit to the accountant at tax = time.=20 In either case, assessment is seen as something that = might=20 result in small changes, usually adding = data-collection and=20 reporting processes here and there. At this point, we = have=20 enough collective experience to realize that these = additive,=20 episodic approaches to assessment rarely work or=20 last.

To=20 improve learning and promote learning communities, we = must=20 recognize that successful assessment is not primarily = a=20 question of technical skill but rather one of human = will. To=20 return to the NASA analogy, all the advanced rocket = science in=20 the world is of little use if there is no widely = shared=20 interest in exploring the universe. Assessment may not = be=20 rocket science, but the same principle holds: = Assessment=20 techniques are of little use unless and until local = academic=20 cultures value self-examination, reflection, and = continuous=20 improvement. In general, already existing assessment=20 techniques and methods are more than sufficient to = meet the=20 challenges we face. It=92s the ends toward which, and = the ways=20 in which, we use those tools that are the = problem.

Four=20 Pillars of Transformative Assessment
I=20 don=92t believe we can construct learning communities = with our=20 students or practice transformative assessment unless = we first=20 develop what Peter Senge, in The Fifth = Discipline=20 (Doubleday, 1990), calls "personal mastery." Thus, in = order to=20 move beyond piecemeal and superficial change and = toward=20 transformation, we need to develop a learning = community-like=20 culture among the faculty and administrators involved = in=20 assessment. Four basic preconditions are key to this=20 collective personal mastery. First, we need to develop = shared=20 trust; second, shared visions and goals; and third, = shared=20 language and concepts. Fourth, we need to identify=20 research-based guidelines that can orient our = assessment=20 efforts toward the goal of creating productive = learning=20 communities. (Since all these preconditions need to be = developed at more or less the same time, their = presentation=20 order is relatively arbitrary.)

Build=20 shared trust: Begin by lowering social and = interpersonal=20 barriers to change. Most=20 of us learn little of positive or lasting value from = people we=20 don=92t trust. To form a productive learning = community, the=20 faculty involved in assessment must first come to = trust one=20 another. Let me suggest a simple first step. Before = turning to=20 the problems, tasks, and issues to be resolved, take = time to=20 highlight what participants are doing well and to = share=20 successes. Encouraging participants to share examples = of=20 successful teaching or assessment practices allows = them to=20 present their best face and demonstrates that each is = a smart=20 person with ideas to contribute. Whatever the means, = the point=20 is to start not with problems and debate but by = helping=20 participants feel respected, valued, safe, and in the = company=20 of worthy peers.

Build=20 shared motivation: Collectively identify goals worth = working=20 toward and problems worth solving =97 and consider the = likely=20 costs and benefits.=20 Since goals powerfully motivate our behavior, = developing a set=20 of shared learning/assessment goals is a logical next = step in=20 building a productive learning community once shared = trust has=20 been established. Most of us are more productive when = we=92re=20 working toward clear, personally meaningful, = reasonable goals.=20 While students and faculty members typically have = goals, they=20 rarely can articulate what these goals are, rarely = know how=20 well these goals match their peers=92 goals, and = rarely focus on=20 learning. Faculty goals tend to focus on what they = will teach,=20 rather than what students will learn; student goals = often=20 focus on "getting through."

There=20 are many techniques for assessing goals, but the key = is to=20 find learning-related goals in common. The "Teaching = Goals=20 Inventory," developed by K. Patricia Cross and me, is = a=20 (non-copyrighted) quick self-scorable questionnaire to = help=20 faculty identify their most important instructional = goals. You=20 can find the Inventory in Classroom Assessment=20 Techniques (Jossey-Bass, 1993). An even simpler = approach=20 is to ask faculty to list two or three assessment = questions=20 they would like to see answered in the coming year, or = things=20 they would like to ensure that students learn well = before=20 graduating, and then look for common goals across the = lists.=20 Whatever the shared goals, in order to be useful they = must be=20 clear, specific, linked to a timeframe, feasible, and, = most=20 important, significant.

Goals=20 are not always sufficient to motivate us to learn. = After all,=20 if the status quo is not problematic, why change? But = not all=20 problems provide useful starting points. As Ewell = noted,=20 "Maximum learning tends to occur when people are = confronted=20 with specific, identifiable problems that they want to = solve=20 and that are within their capacity to do so." In any = case,=20 it=92s critical to connect and frame problems within a = larger=20 vision of shared goals so that energies and resources = aren=92t=20 dissipated in myriad efforts that add up to little or = no=20 improvement.

Here=92s=20 a three-step thought exercise faculty and students can = use to=20 identify promising assessment problems. First, once = you have a=20 problem in mind, write down what you think the best = solution=20 would be. Second, assuming that were the solution,=20 could the group actually implement it? And = third, even=20 if it could be implemented, would the group choose to = do so?=20 If the answer to either of the latter questions is = "no," it=92s=20 probably not a problem worth taking on. =

If=20 the answers are "yes," then it=92s time for a = cost-benefit=20 analysis =97 however informal =97 of the proposed=20 solution.

I=20 recommend trying to "guesstimate" the following types = of costs=20 before committing to an assessment problem: costs in = human=20 time and effort, costs in financial resources, costs = in=20 political capital, and opportunity costs (i.e., what = other=20 important problems won=92t you be able to tackle if = you follow=20 this path?).

Build=20 a shared language: Develop a collective understanding = of new=20 concepts (mental models) needed for=20 transformation.=20 Building a shared vision for transformative change = requires=20 shared mental models and shared language for = describing and=20 manipulating those models. In other words, before we = can=20 collaborate productively we must establish what we = mean by=20 terms such as learning, community, improvement,=20 productivity, and assessment. Taking this = step will=20 allow us to make any implicit conflicts among our = mental=20 models explicit so that we can work them out. =20

One=20 simple strategy for uncovering different mental models = is to=20 ask faculty to define in writing what they mean by one = key=20 term, such as assessment. Then collect those = responses=20 and discuss them or create a concept map from them, = making=20 visually apparent the areas of agreement and = difference. You=20 may find that assessment means, variously, = standardized=20 testing, student ratings of faculty, grading, = institutional=20 research, and time wasted. Rather than arguing for one = correct=20 definition, I suggest proposing the adoption of an = additional,=20 shared working definition, much like adding another = meaning=20 after a word listed in a dictionary. This strategy = doesn=92t=20 force individuals to change their mental models, = something=20 many will resist. Rather, it asks only that they = acknowledge=20 differences between their models and the group=92s and = that they=20 use the group model when collaborating.

Build=20 shared guidelines: Develop a short list of = research-based=20 guidelines for using assessment to promote=20 learning.=20 Several lists of guidelines for effective assessment = already=20 exist, most notably AAHE=92s "Principles of Good = Practice for=20 Assessing Student Learning" (posted to AAHE=92s = website,=20 www.aahe.org). What I=92m suggesting here, however, is = that=20 individual campuses and programs can benefit from = constructing=20 their own specific lists of principles or guidelines = =97 lists=20 that can also serve as the criteria for evaluating = their own=20 assessment plans and efforts. As an example, here=92s = a list of=20 principles I=92ve developed, based on my reading of = the research=20 on improving learning and development.

10=20 Guidelines for Assessing As If Learning Matters=20 Most
If=20 learning really matters most, then our assessment = practices=20 should help students develop the skills, dispositions, = and=20 knowledge needed to:

Engage=20 actively =97 intellectually and emotionally =97 in = their=20 academic work. Set and=20 maintain realistically high, personally meaningful=20 expectations and goals. Provide,=20 receive, and make use of regular, timely, specific=20 feedback. Become=20 explicitly aware of their values, beliefs, = preconceptions,=20 and prior learning, and be willing to unlearn when=20 necessary. Work in=20 ways that recognize (and stretch) their present = learning=20 styles or preferences and levels of = development. =20 Seek and=20 find connections to and real-world applications of = what=20 they=92re learning. Understand=20 and value the criteria, standards, and methods by = which they=20 are assessed and evaluated. Work=20 regularly and productively with academic = staff. =20 Work=20 regularly and productively with other = students. =20 Invest=20 as much engaged time and high-quality effort as = possible in=20 academic work.

The=20 limits of this article do not allow me to illustrate = the=20 guidelines above, but many appropriate examples can be = found=20 in the assessment literature. But those examples, and = any list=20 of assessment guidelines, will only be useful to the = extent=20 that we, the assessment activists, first establish the = fundamentals. To achieve transformation in higher = learning, we=20 must develop shared trust, a transformative vision of = goals=20 worth working toward, and shared language and concepts = equal=20 to the challenge. If we plan and conduct our = assessment=20 projects at every step as if learning matters most =97 = and not=20 just student learning, but ours as well =97 then the = distance=20 between means and ends will be reduced and our chances = of=20 success increased.

Reprinted=20 with permission from the American Association for = Higher=20 Education