TA Tools - Autoformatter
One of the worst tasks as a TA was grading. One of the worst grading tasks was grading tests. Student code frequently didn't compile (despite being allowed to use a compiler during the test). The web interface for displaying code was atrocious to type in so we let students write in an editor and paste their answer in. Of course the formatting was all kinds of funky after doing this. Finally we settled on file upload as the easiest solution. However, this came with a new problem: when grading you had to download and open the file for each student. You could download a zip file of all solutions, but with 300 students, finding the one you needed was frequently more challenging than downloading as you graded. In an attempt to alleviate these pain points, I tried to automate some of the grading process.
The plan was simple: download all student code, sort in the same order that students appeared in the gradebook, format the student code in markdown, and render out the markdown into a PDF that could be used for grading. This was surprisingly easy and worked with little tweaking (aside from those students who uploaded compiled binaries rather than source code). However another problem was discovered: we often only cared about the student solution, a 10-20 line function buried in (sometimes several) a hundred lines of previous attempts, copy and pasted backups, test cases, etc.
Never one to back down from a challenge, I set my sights on extracting just the student submission. Eventually, I got something decently reliable (after all, we knew the name and signature of the function we were looking for). In the case where it failed, we could always fall back on just displaying the entire submitted file if needed. With this, the next question was if we could integrate the concepts of the Gradescope Grader to auto-grade the student code. It turns out that you can with two caveats:
- You want to be very sure that the points that are taken off should be taken off.
- You dont want to give points that are undeserved; this will lead to emails and annoyed students in office hours
These are more or less the general concepts of unit testing anyway: no false positives and no false negatives. In the end only about one-third to one-half of the student submissions would compile, but those who did generally would pass all the test cases. For this subset of the class, grading became a breeze. I even tried to experimentally fit big-o if their code was otherwise perfect. Unfortunately, that meant that the assignment that were hand graded were generally rough: hard to discern and hard to give points to. This took a human toll on the people grading as the grading work became more intense. You win some you loose some.
An example formatted output
