Notes on the pedagogical design
One of the first things you realize when you try to design and run a real experiment for the first time is that even the simplest idea hides layers of complexity—if you try to do it right.
For example, let’s take the question of figuring out the best way to thaw and cook frozen chicken breasts. Chicken breasts come in a variety of portions, which clearly impacts the cooking outcome. Should you control the weight or the thickness, or both? For the cooking procedures, should you control the time or the temperature? How do you measure “best,” by a taster’s (or tasters’) preference on a Likert scale or an objective proxy, e.g., moisture content? And don’t even get me started on the question of how to assign available cooking equipment, whether to cook serially or in batches, or how to handle residual heat between sessions. There are infinite ways a design can end up having flaws, and only a few that get the job done given the resource constraints.
What this means is that this is really not the kind of project you can get right in one go—unless, of course, you already have a decent amount of experience working on similar problems for whatever reason, perhaps because you are an employee of CJ Foods and your job is to produce those manufacturer instructions for their food products. Barring that, it is almost inevitable that you will overlook details that would turn out to matter.
As most (indeed, if not all) students taking this course come with zero experience when it comes to designing and running experiments, the expectation should be that students will make a lot of mistakes—many of them critical. The trouble is that not all mistakes are equal. For an experimental design project, what may appear to be a rather trivial oversight at the design stage of the project can end up completely sinking the project. It is very hard to torture an experiment to tell a story that it was not designed to contain.
Therefore, in designing this project, I made sure to carve out two smaller assignments, in addition to the final deliverable:
- Project Proposal: Shortly after the students are asked to start working on their projects, each team gives a brief presentation (5 minutes) describing their research question and a rough sketch of their design including key components, e.g., treatments and response.
- Progress Report: At about the midpoint of the full project timeline, each team is asked to submit a report that fleshes out all operational details of their design.
- Final Report and Presentation: At the end of the course, students analyze the data from their experiment and report findings.
Within about a week after each presentation or submission, every team receives a decision (minor revision, major revision, or complete revision) along with detailed feedback on the issues they need to address before the experiment can go forward. Anything below minor revision requires corrections approved by the instructor before the project can move forward, and those teams are invited to sit down with me to make sure they understand what they are being asked to do. Students also receive grades for the two intermediate assignments, but any points lost early on are restored if the underlying issues are resolved by the final evaluation. This creates real motivation to act on the feedback.
In this initial run, I found the pedagogical design to work better than I had expected. Most teams committed at least one critical error in the earlier stages, but many self-corrected once those errors were pointed out. Importantly, many, though not all, were able to do this even though the feedback document deliberately withheld the fix in many cases. As a result, although at the initial checkpoint only one team out of seven appeared to have a chance at a successful project, by the final evaluation three teams had submitted solid designs, three others had designs with only minor errors, and only one still had critical flaws. Nearly all teams also ended with perfect marks on both intermediate assignments after restoration.