Chapter 6. Implementation

Congratulations. If you’re reading this chapter, it probably means that your faculty have developed a mission statement, listed a set of learning outcomes, and selected a tool for assessment. Now it comes to the details: who, what, when, how. Here we address the specific issues of: determining sample size, evaluating student work, identifying the resources needed, and using the results.

A. Sampling Method and Sample Size

Critical to the question of implementation is how evidence of student work will be collected, and how much evidence will be used for assessment. For small programs, these questions may be relatively easy, since it might be feasible or even necessary to assess every student in the program. For large programs, you will need to consider: how many students will have their work sampled, and from which courses or tests will evidence be extracted?

Cost and time are major issues. Be prepared to compromise on the sample size in order to attain feasibility in implementation. The more work collected for analysis, the greater is the cost of the implementation.

If the assessment is intended to look at student development, then the work of individual students has to be tracked over a period of time. That is different from, say, collecting a random sample at year 1 and another random sample at year 4. If students are selected to have their work tracked, the faculty will need to determine how the identity of those students will be safeguarded and how the students will be notified. See BSC’s Institutional Review Board.

B. Rubrics

Selection of an assessment tool (Chapter 5) does not, by itself, specify the parameters of the assessment tool. For example, use of student writing in a capstone project to assess critical thinking leaves unstated how the judgments about critical thinking will be made from this writing. What are the demonstrations of critical thinking? What are the qualities in the writing that will be considered evidence of good, bad, or indifferent critical thinking? To answer these questions, enter…rubrics.

1. What Is a Rubric?

Once an assessment tool has been settled on, specific decisions may have to be made about the criteria by which student work will be assessed, depending on the learning outcome being assessed and the tool for assessment. Choosing criteria is where rubrics come in.

A rubric is a set of criteria for assessing student work or performance. Rubrics are particularly suited to learning outcomes that are complex or not easily quantifiable, for which there are no clear “right” or “wrong” answers, or which are not evaluated with standardized tests or surveys. Assessment of writing, oral communication, critical thinking, or information literacy often requires rubrics.

Rubrics have two dimensions: they identify the various characteristics of the outcome, and they specify various levels of achievement in each characteristic. Thus, a well-designed rubric consists of 1) clear definitions of each characteristic to be assessed for a given learning outcome, and 2) clear descriptions of the different levels of achievement for each characteristic. For example, to assess writing requires a set of characteristics of writing that are being examined (e.g., logical organization) and a set of levels indicating the quality evident in those characteristics (e.g. what constitutes excellent, good, fair, or poor logical organization).

Because rubrics establish criteria, they can help make assessment more transparent, consistent, and objective. Faculty members and evaluators can use rubrics to communicate to students and each other what they see as excellent work, while students gain an understanding of what is expected and how their performance will be assessed.

Rubrics are also useful when there is more than one evaluator; rubrics can serve as standardized scoring guides that assist different evaluators to determine the quality of student work in a consistent manner.

2. Examples of Rubrics

Examples are provided of rubrics for writing, critical thinking, problem solving, information literacy, and oral communication.

C. Identification of Resources Needed

The resources required to carry out an assessment plan should be considered in total and then proposed to the administration. Include the following categories:

Cost of standardized tests, including the purchase of the tests and the cost of processing the results. Outfits such as ETS will supply both of these steps as a service, for a fee.

Cost of surveys, including the design of the survey, mailings, web-based postings, e-mail, and compilation of the results. If large numbers of students are being surveyed, a bubble form suitable for scanning is probably most appropriate; the Office of Institutional Research houses a scanner and software for processing bubble forms, but the availability of this service is limited. Commercial services such as http://createsurvey.com/, http://www.advancedsurvey.com/, and http://web-online-surveys.com/ provide online survey development, posting, and data collection, for a fee. BSC's Information Technology division can also build web-based surveys and compile the results (with skill, you can construct your own web-based forms and have the results sent to you by e-mail, see http://it.bridgew.edu/CIS/WebDev/HowToMailTo.cfm?CISLinks=).

Time and cost of collecting and evaluating student work. A rough estimate for scoring student writing is that, once faculty become adept at applying rubrics to evidence from portfolios, course-embedded assignments, or capstone courses, scoring 20 pages of student writing requires about an hour.

Training for faculty evaluators. If the assessment tool involves scoring student work or conducting interviews, faculty may need to be trained to do this work so that it is consistent among all of the evaluators. Training would involve practice scoring the same work and normalizing the outcomes.

Assessment coordinator. For large programs requiring complex assessment tools, it may be necessary to assign a faculty member the ongoing task of assessment coordinator.

D. Using the Results

Faculty will find value in the assessment effort when the assessment results begin to flow in. The normal department communication systems can be brought to bear on the dissemination process: faculty meetings, committee discussions, e-mail, a Blackboard site, etc.

Assessment results almost always indicate an apparent shortcoming in the program. That is a useful result, since it helps people think about improvement. But be prepared for people to question the basis of the data. It is a natural response to focus on the limitations of the assessment instrument or sampling method when results are negative, and sometimes this is rightfully so.

All assessment tools have shortcomings. Assessment validity is higher when there are multiple indicators giving the same message. For example, evidence of student work combined with feedback from exit interviews is more compelling than just one or the other. So if evidence points to a particular problem, be prepared to dig more deeply by enlisting another tool to elucidate the seriousness and sources of a problem.

How one frames the results is critical. Assessment results should never be used as evidence of a particular person's shortcoming. In the spirit of program assessment, one always needs to ask, “How can we do better?” Viewed this way, assessment is an opportunity to improve ourselves and do a better job for the students we serve. With continual improvement, we all succeed.

Example: Dr. Catherine Womack’s statement on the capstone course in the BSC Philosophy program

For more reading, see the North Carolina State University website on Internet Resources for Higher Education Outcomes Assessment.

Last Modified: June 5, 2008