Ask HN: How do I examine an undergraduate programming course in the era of GPT?
I teach a course on Python programming at a UK university. The assessments are 'take-home' programming problems a little like Advent of Code. Cheating is always a concern but now it is trivially easy to generate code that gets a very high mark (I have checked). What do HN readers suggest as an strategy moving forward? In person exams are possible but not ideal.
15 comments
I have a radical take on this. Why not just let them do what they want? Put all the emphasis on the exams.
Education for some reason has always been about bullshit busy work anyway. Homework is for people who don’t understand the material and need more practice imo and should be optional and worth 0-5% of the grade.
Have midterms finals and projects count more, and the tests would probably be monitored with proctors anyway right?
I went to a mediocre college because in high school I would get F for homework. 95-100 on midterms and finals. 5s on all AP tests and almost perfect on SAT. artificial gate keeping based on a GPA based on BS work because even if I got the material and ace the tests the best I can get is like a B.
And I did “bad” in college too where I failed homework and scored 95-100 on exams. If you’re gonna impose busy work on me I’m not doing it if I don’t feel it’s necessary for me. And I shouldn’t be punished for it. And neither should these students if they are genuine about learning. I’m there to learn by choice.
Education needs to evolve. It keeps some people down for no reason.
Education for some reason has always been about bullshit busy work anyway. Homework is for people who don’t understand the material and need more practice imo and should be optional and worth 0-5% of the grade.
Have midterms finals and projects count more, and the tests would probably be monitored with proctors anyway right?
I went to a mediocre college because in high school I would get F for homework. 95-100 on midterms and finals. 5s on all AP tests and almost perfect on SAT. artificial gate keeping based on a GPA based on BS work because even if I got the material and ace the tests the best I can get is like a B.
And I did “bad” in college too where I failed homework and scored 95-100 on exams. If you’re gonna impose busy work on me I’m not doing it if I don’t feel it’s necessary for me. And I shouldn’t be punished for it. And neither should these students if they are genuine about learning. I’m there to learn by choice.
Education needs to evolve. It keeps some people down for no reason.
Does cheaters getting a high mark have a direct negative impact on other students? (E.g. grading on a curve.) In that case, you should probably switch to a method that makes everyone's mark depend only on their own performance.
Other than that, students who don't understand the material well being able to outperform others who do may seem unfair, but in the end your course probably exists to teach students programming fundamentals that are expected as prerequisites in other courses, and the exercises serve to prepare students for the more difficult problems they'll encounter in those other courses.
If a student decides they don't need that practice and ends up stumped later when GPT can't help them anymore, that's their own bad decision to deal with. (On the other hand, if that never happens maybe the Python course wasn't all that necessary for them to begin with.)
So I recommend optimizing mostly for the students who do want to learn something; put enough effort into anti-cheating measures to keep honest people honest, but don't worry too much about cheaters beyond that.
Other than that, students who don't understand the material well being able to outperform others who do may seem unfair, but in the end your course probably exists to teach students programming fundamentals that are expected as prerequisites in other courses, and the exercises serve to prepare students for the more difficult problems they'll encounter in those other courses.
If a student decides they don't need that practice and ends up stumped later when GPT can't help them anymore, that's their own bad decision to deal with. (On the other hand, if that never happens maybe the Python course wasn't all that necessary for them to begin with.)
So I recommend optimizing mostly for the students who do want to learn something; put enough effort into anti-cheating measures to keep honest people honest, but don't worry too much about cheaters beyond that.
I pretty much agree with this. I'd even be honest with them up front. "Yes, I'm aware of ChatGPT, no there's not much I can do to stop you from using it, but that's not going to help you later."
(Also, you probably aren't getting paid well enough to bust your ass finding the cheaters.)
(Also, you probably aren't getting paid well enough to bust your ass finding the cheaters.)
At UIUC we've been moving towards testing student programming skills in our Computer-Based Testing Facility (CBTF). This is a set of computer labs that are firewalled from the internet. Students self-schedule into testing slots at the CBTF, then at their scheduled time they go to the CBTF and take a programming (or other online) test that is auto-graded on the spot. This is very secure and has little overhead for faculty. It also automatically handles disability accommodations like extended time tests and reduced distraction requirements, and if students are sick then they can automatically reschedule their own exams. We currently run about 70,000 exams per term like this and it's heavily used by our largest classes (over 1000 students in a class). More details here: http://zilles.cs.illinois.edu/papers/zilles_csedu_cbtf_2019....
For OP, we've also been using Zoom-based proctoring options for students/courses that can't use the CBTF for some reason. I'd be happy to follow up with you if you want more details about how we make this work at scale.
For OP, we've also been using Zoom-based proctoring options for students/courses that can't use the CBTF for some reason. I'd be happy to follow up with you if you want more details about how we make this work at scale.
In my undergrad CS coursework, we were graded largely on projects, not exams.
If the solution isn't short (more than 200 lines), ChatGPT and Copilot can help but cannot solve the problem completely. The more complex the project, the harder it will be for someone to generate it whole cloth with an AI, and even if someone is using an AI to help they'll still need to understand the concepts well enough in order to get good results.
If the solution isn't short (more than 200 lines), ChatGPT and Copilot can help but cannot solve the problem completely. The more complex the project, the harder it will be for someone to generate it whole cloth with an AI, and even if someone is using an AI to help they'll still need to understand the concepts well enough in order to get good results.
Grading on projects is the answer here. Not only are you designing pedagogy that allows students to see cs as a tool for their own goals, you are also directly placing them at the top of Bloom's taxonomy: origional synthesis. Which, incedentally, is the skill that chatGPT isn't able to replicate.
Synthesis is relative to skill, chatGPT is more than capable of replicating it at novice levels at the least.
Assuming there's more well-authenticated artifacts, you could use something like a causal-graph to identify where students may not have been honest in their take-home work. If a 100% comprehension level of say, nested for-loops, suddenly becomes a 30% comprehension when the student is being watched, that's cause for concern, you'd just need to have infrastructure in place to trace those relationships.
Another good option would be to ask for pseudocode, drawings, flowcharts instead of Python. I don't have access to see if Copilot can pseudocode things, but I don't think the mechanical task of literally writing code is the best way to test this sort of thing anymore or perhaps, it's a bad target/metric.
Switching the course to some obscure graphical programming language might be another good option as well.
Another good option would be to ask for pseudocode, drawings, flowcharts instead of Python. I don't have access to see if Copilot can pseudocode things, but I don't think the mechanical task of literally writing code is the best way to test this sort of thing anymore or perhaps, it's a bad target/metric.
Switching the course to some obscure graphical programming language might be another good option as well.
I am director of an online bootcamp, and we tackle the cheating problem with a custom browser-based platform that records 3 video feeds during an exam: 1. the screen capture, 2. the webcam and 3. the video feed of the students phone placed so we can see the students desk. It makes it pretty hard to cheat, but does require lots of supervisor time getting all students set up correctly and then monitoring the video feeds
Can you teach using git repositories?
Have them commit in a way that shows their understanding at each point as they incrementally improve or solve the problem.
Have them split into groups and submit MR's to each other.
Give them them ability to tag other.
Have them upload their MR to a system with CI and check that it passes tests, lint, style etc.
If the commits are broadly similar then likely they did the same process as most of the class, if the commits are much different then check their understanding and/or cheating.
Have them commit in a way that shows their understanding at each point as they incrementally improve or solve the problem.
Have them split into groups and submit MR's to each other.
Give them them ability to tag other.
Have them upload their MR to a system with CI and check that it passes tests, lint, style etc.
If the commits are broadly similar then likely they did the same process as most of the class, if the commits are much different then check their understanding and/or cheating.
I think teaching proper engineering, i.e. building whole systems where the design/architecture is what gets graded, would make you immune to GPT.
Obviously, this only makes sense if your students are CS majors. If they are not, you can always teach a light version of this in the spirit of HtDP or https://dcic-world.org.
Obviously, this only makes sense if your students are CS majors. If they are not, you can always teach a light version of this in the spirit of HtDP or https://dcic-world.org.
When I was in college, students could do any math exam in Maple and Matlab, so the questions required Maple and Matlab. The rest were derivation questions, which both didn't do.
Just raise the bar. Give them a lot of code and ask them to hack a feature into it. Chain together multiple difficult programming questions to the point that they have to use GPT to get the solution.
Just raise the bar. Give them a lot of code and ask them to hack a feature into it. Chain together multiple difficult programming questions to the point that they have to use GPT to get the solution.
Steer clear of the most classic programming problems, don't use problems considering widely known concepts (like prime numbers) but rather define your own concepts that are not as widelybused. Look for/design questions that require the student to recognize edge or special cases and handling them. Combination of these seems to usually throw ChatGPT off
Don't do exams, give them readings plus group projects and let them present them. Not sure how large the class is but this is probably not realistic when you have 10+ groups.
Perhaps you can have your students implement C-style for loops in Python?