Analyze Evidence
Analyzing evidence of student learning is one of the most important steps, and there are many ways you could do this. Here we will focus largely on rubrics - a versatile tool that can be adapted to analyze many kinds of evidence- and then make some suggestions about analysis in general.
Rubrics are most often set up as a matrix that specifies various levels of performance across one or more dimensions of student learning. The following is an example of a rubric created by UCR’s Department of Theatre, Film, and Digital Production that could be used to assess oral presentations by students.
| Criteria |
Emerging |
Developed |
Highly Developed |
|
Content of Presentation |
Presenter shows limited command of subject matter; insufficient coverage of context or background; inadequate use of details. |
Presenter shows command of subject matter; sufficient coverage of context or background but does not generate additional interest in subject matter; adequate coverage of detail and main points. |
Student shows excellent command of subject matter; provides context and background in a way that generates interest from those without specific prior knowledge; provides appropriate amount of detail for audience in time allotted. |
|
Structure/ Organization of Presentation |
Presentation starts or stops abruptly; connection between some elements of the presentation may not be clear at all; movement from topic to topic is abrupt or unpredictable. |
Presentation is clear overall; the logical relation between some elements may not be immediately clear; some transitions may create momentary confusion among the audience. |
Presentation is compelling with a clear beginning, middle and end; presentation is a cohesive whole; movement from topic to topic is marked by logical transitions. |
|
Nonverbal Delivery |
Presenter makes limited eye contact; use of notes or prompts is a distraction; presenter’s use of physical space distracts audience; use of gesture (or lack thereof) distracts from presentation. |
Presenter makes eye contact as appropriate; use of notes or prompts may briefly distract; limited use of physical space and/or expressive gestures. |
Presenter use of eye contact generates and holds attention; use of notes or prompts is subtle; presenter uses the space available in a way that focuses attention or interest; expressive gestures add clarity or interest to presentation. |
|
Verbal Delivery of Presentation |
Presenter’s voice consistently indicates nervousness; pacing is uneven throughout; volume creates difficulties understanding content; wording choices obscure meaning or create confusion. |
Presenter’s voice is calm and tempo is generally appropriate throughout; volume is appropriate throughout; wording choices are clear and consistent with general usage. |
Presenter modulates tempo and timbre to create interest; excellent pacing; volume is clear and modulated to add interest; wording choices are imaginative and memorable. |
|
Time Frame |
Presentation poorly fits time allotted, e.g., leaving either significant time at the end, running over time, or not covering all content before time expired. |
Presentation nearly completed in time allotted. e.g., left a small of time or had to rush near the end. |
Length of presentation almost perfectly matched to time allotted. |
The rubric breaks down the more complex skill of oral communication into several more specific dimensions such as content, structure/organization, time use, and verbal and nonverbal delivery. The columns provide detailed descriptions of what each level of performance looks like.
Rubrics are versatile and can be used to assess many kinds of evidence. You can create your own rubric or modify one of the many existing rubrics. For example, see the American Association of Colleges and Universities rubrics for learning outcomes such as critical thinking, written communication, and information literacy as well as many others. (There are many, many more examples of rubrics on the Internet.)
(Rubrics can also provide relatively detailed feedback to students without requiring the grader to write many comments, and if you share a rubric with students before assignments are due, they can help students better understand what to spend time on when they work on assignments.)
Rubrics can help standardize the assessment of student work across a number of individuals; for example, you might split up term papers among two colleagues and yourself and assess them with a common rubric. This can reduce the amount of time needed to conduct assessments or allow you to look at a larger sample in the same amount of time. If you use multiple people to assess student work- and you should- it is important to think about the level of agreement between the raters. A more technical term for this agreement is interrater reliability. We say interrater reliability is higher where raters give more similar scores and lower where there scores are more different. To increase interrater reliability, it is usually a good idea to have all raters look at a few of the same examples of student work, assess them, and then discuss why they gave the scores they did. This exercise, called “norming,” helps individuals calibrate their expectations or, alternatively, may reveal ambiguity in the rubric you are using. You will probably want to norm on five or so examples before splitting up the work among multiple raters.
Once you have gathered information from a large number of students, you will want to summarize the information in some way.
One of the simplest ways to organize evidence is with a table that shows how many students performed at what level along what dimensions. Imagine we analyzed evidence from thirty students using the example rubric discussed earlier and organized it into the following table.
|
|
Emerging |
Developed |
Highly Developed |
|
Content of Presentation |
5 |
5 |
20 |
|
Structure/Organization |
5 |
5 |
20 |
|
Nonverbal Delivery |
10 |
15 |
5 |
|
Verbal Delivery |
10 |
10 |
10 |
|
Time Frame |
10 |
10 |
10 |
Even this relatively simple presentation shows us students are, in general, doing fairly well with the content and structure. However, they appear to be struggling with the elements of delivery and their use of time.
There are a number of reasons why assessment works best when it is a collaborative enterprise. First, and most important, when you are assessing at the programmatic level, you are really assessing the entire curriculum and this curriculum is “owned” by the collective faculty of a program. Second, assessment at the programmatic level generally works better when you gather evidence of student learning from more than one course and select more than a few examples of student work. Recruiting the instructors in these classes to help you gather and analyze student evidence can be a way to spread out the work. Third, different people will always bring different experiences and perspectives, and involving them in the analysis of student evidence may help you identify and understand patterns and trends you might have missed.
Your department may have an undergraduate affairs committee or group that is engaged in the undergraduate curriculum, and this would be a natural group to involve in assessment at the program level.
It is useful to know what the average or typical student is capable of doing. These kinds of questions are best asked by looking at averages and medians. At the same time, it is also useful to think about variation. This can be variation from student to student on the same measure, variation on subscales or submeasures of larger outcomes, or variation across time or between classes, majors, or other groups of students. Which of these is more important - central tendency or variability - will vary in different contexts, but it is always a good idea to look at your results, at least briefly, through both sets of lenses.
You may want to think about setting benchmarks, the overall level of student performance that will signify successful learning, beforehand. You might select your benchmark in a number of ways, for example, by coming to some kind of consensus with your colleagues, looking at previous assessments, or looking to national norms or expectations from employer or graduate school entrance standards. Formalizing your expectations before analyzing the evidence may allow you to make a clearer assessment of the extent to which students are meeting your expectations. (If you do not have benchmarks in mind, it is always tempting to say that whatever level of student performance you observed is satisfactory.)
Charts, graphs, or other visual representations of assessment results may be useful to the extent they summarize data, make it easier to spot trends and patterns, and accurately represent the data. (If you are working with a Microsoft product, here is a step by step guide to creating charts and graphs.) However, if a chart or graph is difficult to read or interpret (or takes much more time to create than a written description of results), it probably does not add value.