Main page

1 Grading theory and ideals

• Cui Bono

  • Discussion: Grades are a product produced by educators; who are the consumers? I generally get a subset of the following list, and sometimes I fill in missing parts

    • Students
    • Parents
    • Course staff
    • Next course
    • Grad school admissions
    • First employer
    • Merit-based scholarships
    • smart-person clubs and awards

  • Discussion: Why does each consumer care about grades? What do they want the grades to be?

    • Student, course staff: feedback on what needs more attention
    • Next course (and grad school and some employers, to a lesser extent): assurance that prerequisite material is learned before moving on
    • Parents: assurance that students are being good students
    • Grad school, employers, scholarships, clubs, awards: a ranking of goodness
      • which is a combination of factors (discussed in the next topic within this session; I usually say here something like it’s not just intelligence/skill/knowledge/other-ta-suggest-term: most want the well-rounded educated people that a liberal arts education is supposed to create. We’ll look more into what that means shortly.)

• If we were omniscient, what would grades be?

  • What do we value?

    • Activity: ask TAs to list things that contribute to a good grade. Typically I get a list like the following:
      • Understanding of material
      • Timeliness
        • Speed (timed exams)
        • Time management (late penalties)
      • Social skills
        • Understanding what instructor meant by ambiguous statements
        • Knowing how to pose questions well
      • Luck
    • Activity: If we had a way to grade each of those separately, how much should each one contribute to the final grade?
      • Have each TA come up with a weighting
      • Discuss the answers, including that they are different (I’ve never had unanimity; I typically do not offer my opinion, instead supporting each weighting raised)
      • Discuss the following quote, which received no argument when stated:
        • Learn the skills should be synonymous with Pass the course – Arnold Pears
        • Is that 100% understanding of material or does saying learn instead of know imply other aspects too?
    • (if conversation lags, I sometimes stimulate it with) which consumers value what?
      • For many consumers, the answer is we want smart graduates not any particular skill or knowledge.
  • Courses contain multiple topics

    • Example: draw something like the following, picking topics from a course at your school and using numbers that match your school’s typical points-to-letters scale.

      student	C++	Trees	ADTs	Other DS	Grade
      bland	74%	75%	77%	74%	C
      special	100%	0%	100%	100%

      Explain: suppose, for the sake of argument, that course’s content is divided evenly into four topics, each equally important, and that we grade at a granularity where we have an individual grade for each part.

      Question: What grade should special student get? (typically I get a mix of C and F, and prompt/lead students to the following)

      • Mean
        • 75% is 75%, no matter how it is distributed
        • Means earned a C in prerequisite course does not mean has C-level knowlege of prerequisites
          • Makes teaching harder
          • Disservice to students who end up missing prereqs later
        • Fair
      • Min
        • If you don’t know one of the topics, you can’t move on to the next course
          • Implies minimum of grades per topic
        • High-stress, low-motivation
          • Any task can break your grade
          • If botch first topic, no reason to try on rest
          • Solution: mastery learning
            • i.e., course designed so that each topic can be re-evaluated
            • grade = min_topics(max_tries(topic))
            • difficult to design in cheat-resistant way
        • Granularity matters!
          • don’t want you got a 40% on quiz 18, so you fail the course
          • … meaning need different within-topic grading rules
      • Other
        • Can easily imagine other f(g₁, g₂, g₃, g₄) functions
        • Could even have different internal and external grades
          • Internal: list per-topic grades, for use in prereqs
          • External: use standard mean, for ranking students
            • Maybe also a list of strengths?

• Aside: the motivational impact of grade presentation

  I don’t usually have time for this, but the conversation is interesting when I do. I sometimes combine it with a separate session on engagement.

  I present three hypothetical grade interfaces

  • Grade-to-date: as each assignment is graded, the numerator and denominator of the grade both increase
  • Points gained: the progress-bar view: you start with zero and gain some points with each assignment
  • Points missed: the inverse progress-bar view: you start having missed nothing lose some possible maximum points with each assignment

  And we discuss the motivational impact of each

  • Grade-to-date
    • Common grade jump pattern: A, A, D, C+, B, B-, B-, B, …
    • Gives feeling of honing in on truth – and hence inevitability, or being pigeonholed
  • Points gained
    • Commonly has slow start, as early assignments have low weight
    • Gives feeling of uphill climb (a long time in the F territory), but continual progress
  • Points missed
    • In some sense, optimistic (you could still earn an A!)
    • Depressing (watching one tier after another slip into the unattainable)

  Can combine them all

  0                       D   C   B   A   100
  [====================         *    ======]
      gained                 so-far  missed

  What is the message of such a unified presentation?

  How do each of these change if we add

  • Min not mean
  • Mastery learning
  • Extra credit

2 Grading practice

• Goals

  • Fairness/consistency
    • Use the same rules for everyone
    • Prevents but my friend had the same answer and got more points!
    • Assists in deflecting blame from graders (I was just following the rubric)
    • Reduces chance of favoritism and bias impacting grades
  • Accuracy
    • Students are individuals, with individual characteristics
    • Firm rules have loopholes; look for overall understanding, not hitting all the checkboxes
  • The two goals are often at odds
    • Typically, the larger the class the more Consistency wins out over Accuracy

• Ambiguous answers

  • Suppose an answer is vague or ambiguous; you are certain they know at least 5 points worth, think it most likely they know 8 points worth, and can imagine someone knowing 10 points worth giving this answer
    • What should you give them?
      • 5: it’s the student’s job to prove they know
      • 8: it’s a noisy process, we just do our best
      • 10: it’s the staff’s job to prove they lack knowledge
    • All three philosophies are rationally defensible
    • We have faculty with all three philosophies
  • Ask your supervisor what the rule for your class is
    • (warning: many of them won’t understand the question… do your best to interpret their response)

• Rubrics

  • A rubric describes how various classes of student answers should map to student grades
    • help keep things fair
    • help keep all graders on the same track
    • are almost never complete
  • Not all classes have rubrics for all graded assignments
    • … and not all rurbics that are created are usefully unambiguous and complete
    • … and many faculty are less-than-engaged in clarifying ambiguities
    • suggestion: create/expand on rubric supplied
      • i.e., document the judgment calls you make

• Communicate judgment-calls

  • If the number of graders > 1, communicate any repeating grading ambiguity judgment calls you make to other graders
    • Generally > 1 grader → many students → fairness is supreme
    • fairness and consistency should span graders
  • Ask if there is an established medium for such communication (email account, slack channel, private post on piazza, etc)
    • if so, use it
    • if not, create one

• Avoid double-jeopardy

  • In general, grade cost should be proportional to misunderstanding size
    • Most rubrics have this built in
    • … though not always accurately (e.g., single-character typo can result in full does not compile penalty)
  • Common problem: double-jeopardy
    • penalizing for same misunderstanding repeatedly
    • often on exams (e.g., if the student mis-remembers a function name they may lose points on several questions that use that function)
    • In general, double jeopardy is not wanted
      • but some rubrics amortize the point cost, expecting it
      • and few rubrics have a way to not penalize
  • Ask supervisor what the double-jeopardy rule is
    • may vary per mistake type and per assignment… a good single-jeopardy policy is hard to create