The Emotional Work of Grading

There is quite a bit of talk these days about ungrading, but this post does not go in this direction.

Grades are clearly emotional to students, which might partially explain why they make students so anxious and, simultaneously, so obsessed with them. But this post is also not about this.

This post is about the less-told side of what grading means to the instructor and, in particular, to an instructor that has created the evaluation.

An important start point is that setting up evaluations is hard. My students seem shocked when I tell them that it takes me between one and two full days or work to write a midterm. This is one of these things that do look very different from the other side. The students are probably thinking: well, this guy has 10-15 years of education and knows it all about the topic, so this job is mostly about putting things on paper and formatting. Well, no: as instructors we have to work hard to make sure that any evaluation aligns with the learning objectives (and really tests them), that is not easy to subvert (makes cheating harder than not cheating), that it is clear an unambiguous, that it is not too difficult (because this means too much work later on, and unhappy students), that it is not too easy (since it would remove the student’s incentive to learn), that can be marked fairly (this is trickier than it sounds), and that it does not make my life miserable (i.e., that it does not take forever to mark). In other words, setting up evaluations is a complex multi-objective optimization problem that also requires lots of creativity. And this is all assuming that the midterm is in person and not take home, which adds the risk of being chatGPTized…

Then, once you think you have a good midterm/final exam etc., (or you run out of time), it’s time for the actual test. When this happens in person (i.e., in a large room) it is not unusual to see students leave the exam about 25 minutes in (for a 50 minute exam). Doubts start to creep in… was the midterm too easy? Too difficult? Did I implicitly provide answers to the questions in the class? — time passes and then you have the opposite set of doubts… why are there still so many people who seem to be in the middle of it? Was it too difficult? will I be ruining student’s life’s because of my zeal to make my questions climb up Bloom’s taxonomy little pyramid? In my experience this is usually OK, just the expression of the amazing variability of human timing, behaviour and skill (note: I recognize the issues with timed examinations and how excluding they can be, but this is a discussion for another day).

Now you got your filled exams, and it is time to grade. This is the most nerve-wracking part of the process. It goes like this:

The first one is terrible. It is like they did not understand the question, or they attended a set of lectures of a different course in a different department. It’s very difficult to keep the inner doubts at bay and remain calm and objective; no matter how many times you have read and re-read the questions, you read them again… is there a possibility that this question could reasonably be interpreted in that other strange way? Am I just a gatekeeper of a language and a way to look at the material that is personal, biased and unfair?

The second one is terrible again, but in completely different ways. Were my lectures awful? Was it impossible to get to a reasonable solution with the material, help and time that I have provided during the class? Your mind starts coming up with possible ways to grade more generously… is is fair to give some points just for writing anything in the box? What happens if everyone fails the midterm?

The third one is quite good. Sigh of relief! At least someone gets something!

The fourth one is perfect… How nice! But… did I really teach them anything? Is it possible that they did already know the stuff?

The fifth one is perfect again… was the exam too easy? Did I make it too easy to cheat? how is this possible?

And, of course, after a while you tend to get the full range of answers, with some mistakes more common than others. If you are not grading blind to the name of the student (I prefer blind, although it can be logistically harder than it seems), some answers do seem like a disappointment, almost a betrayal. And, occasionally, you can feel a bit of pride on what you have done.

I expect that the experience of the instructor may vary significantly, especially depending on personality. However, the roller-coaster that I’ve described above is still, for me, and after many years of setting up evaluations and grading, pretty much unchanged.

In summary, we try to be neutral and fair, and leave emotion aside, but grading is hard work, not only because it is tedious work that simultaneously requires a large mental load to be consistent, but because it is emotional work.

The only things that I found that soften the experience a bit for me:

  • Remind myself that students are people with circumstances. They could have a bad day, a caring duty, a friend in distress, food insecurity (help if you can!).
  • Recognize that it is not personal. A bad answer is not an insult, and most often than not, it does not reflect the attitude of the student towards you.

How does grading feel to you as a student or an instructor/grader? What are your tricks?

Student Paper on Representations and Transformations

Adam Binks (A PhD Student at the University of St Andrews) and myself (Miguel Nacenta) just published an article in the International Journal of Human Computer Studies) about our recent research.

Adam built a new tool to support people when transforming complex networks of thoughts and ideas into text prose. The tool itself (called Write Reason) and a short description of what we found when we studied essay writers using the tool is accessible through the WriteReason site.

One of the most interesting things about our findings is that there is a lot of interesting stuff going on in between the creation of different representations of the ideas. In other words, transformations between representations are much more complex, interesting and important than we thought before.

This post is also in the VIXI website.

How to talk to your supervisor about what you have read

This is a post in a series of posts regarding student-supervisor communication.

Audience: you are a PhD or MSc or project student, and you meet your supervisor regularly. In the previous meeting, your supervisor asked you to do research in some topic area, perhaps by a keyword (e.g., please do some search in the area of “provenance in User Interfaces”), or perhaps through another paper (e.g., follow the references in this other paper).

Naturally, you have done the work, and you have read (there will be another post on what this means in the future) a bunch of papers. Now you come to your supervisor, eager to demonstrate that you have done what you were expected to do, and indeed, you have found and read some interesting papers.

What NOT to do

Perhaps more important than the “do’s”, consider the things that you should avoid. Unless your supervisor has explicitly asked for any of these, you should NOT:

  • Detail the contents of a paper in general (your supervisor would generally not be interested in the full content of the paper. If they already know it what is the point on repeating pretty much everything on it? If they are not familiar with it, they probably will not be interested in the full content, but only in how it relates to your own project; your supervisor has probably very little time for this, and communication about previous work usually needs to be done in an efficient way.
  • Read a summary that you have written (if you have written something about a paper, your supervisor can probably read it much faster than you can read it aloud – if you really need them to read your summary, and they agree with this, send it in advance of your meeting).
  • Forget, loose or misplace the paper (your supervisor might want to come back to it, it is your responsibility to be able to retrieve that paper quite quickly if you need to).
  • Insist on going over all the information that you have retrieved without paying attention to your supervisor’s reception of it (your supervisor). Do not discuss things that you do not think are important.

What to do

These are all items that you might not know are important, but are often key for good communication about research.

Background and context: Not all papers are created equal or are going to have the same importance. If you only list the title (and yes, the title is important), you are missing many of the ways in which your supervisor might help you filter out the really relevant papers from those which are not. More specifically, this is information that is likely to be useful:

  • Year (of publication). This information is key. The paper might be precisely about your topic of interest, but too old to have been important. Perhaps the opposite: the paper is very old and incredibly relevant, but has been ignored. Or the paper just came up last month and your supervisor is not aware of it (and should be!). Making all these judgments is much more difficult without the date.
  • Authors. It is important for you to start recognizing the people who have work a lot in that particular subarea, and their collaborators and students. Your supervisor might use that information to remember about other important papers by the same author, or to assign more weight to that paper because they trust the author (or even know it personally). Put the authors next to the title if you are showing only a subset of the paper. Sometimes knowing which institutions the authors come from can be useful. For example, if they come from a place that is well-known for research in the area.
  • Venue. Although it is not a perfect correlation, a paper published in a reputable and competitive venue is likely to be more important, since it denotes interest by the community, and some degree of rigour. A paper in a leading venue which is ignored in a literature review is also likely to cause more trouble in the review cycle, whereas most reviewers will easily forgive (or forget) a poster in an unknown or disreputable venue.
  • Format/length. Scientific publications come in many shapes and forms. For example, a valuable piece of scientific writing can be a textbook, a chapter in a textbook, a chapter in an edited book, a survey article, a long journal article, a paper, a short paper, a poster, a position paper, or even a blog or a tweet. It does not help if your notes, summary or what you are showing or describing to your supervisor hides what kind of publication it is. Your supervisor (and yourself) can use this information to determine the importance and value of that work.

Note that giving some kind of impression of the above should take very little time. You do not have to list or read aloud the whole list of authors, but having it out there so that your supervisor can take a glance at it can be very useful.

Content: An article or piece of scientific literature typically contains much information, more than you can efficiently convey within a meeting. If you have to summarize any part of the work, because you think it is relevant, this does not mean necessarily that this is going to be a piece of text. There are often different kinds of elements that might be way more effective at summarizing a paper. For example:

  • A diagram included in the paper.
  • An image of the system/technique that describes the core of the paper (or the part that is most related to your work).
  • A short fragment of video.
  • A formula.
  • A table (with results or a taxonomy)
  • A chart showing results (e.g., a bar chart).
  • A couple of sentences describing results of conclusions (yes, this is text, and is legitimate).

Note that any of the elements above might be incredibly difficult to understand in isolation, but this is why you are having a meeting: a quick few sentences can help make any of these items really effective.

Relationship: Probably the most important reason why your supervisor wants you to read a piece of scientific literature is because it relates to your work. Very often this relationship remains implicit. You should be able to describe as succinctly and accurately as possible how what you have read relates to one or more of the following (depending on the stage of your research):

  • The fundamental issues, techniques or technologies upon which your work builds upon.
  • One or more of your research questions.
  • The specific contribution that you are hoping to make in your research.
  • What you have designed/built/proposed already.
  • The methodologies that you have applied.

By making explicit the relationship between your work and what you have read, you are also communicating a number of things that are important for the supervisory relationship and signalling your level of maturity in research. For example, your supervisor will learn to what extent you are able to relate your work to the close and fairly distant (perhaps only abstractly related) pieces of literature, and whether you really understand the contributions of existing work. This is a kind of syntopical reading or deep reading that is very important for academic work.

Please let me know if you have found this article useful, or if you have concrete examples, or possible improvements that you think I can use to improve it.

“A Delicate Agreement” gets an Alberta Digital Award!!

Congrats to Lindsay MacDonald for getting the prestigious Alberta Digital Award for her work (with us) in the digital piece A Delicate Agreement.

She received the award on Monday the 6th at the Hotel Arts in Calgary.

We are only hoping to see more awards coming… way to start a PhD Lindsay!!

Some more coverage here.

Also congrats to Jon Haber for getting his QEII scholarship! It’s nice to see students/collaborators getting prices!