I find graphs of multiple choice question response data very useful in analysis of misconceptions, so I developed a procedure to rapidly (i.e. within 10 min) generate such graphs with freely available software and minimum user input. It does, however, rely on being able to rapidly scan large numbers of papers quickly (e.g. using the scan and send function on a photocopier).
Basically the process goes like this:
- Scan MCQ papers and manipulate such that you end up with a folder of low-resolution images.
- Use ImageJ (and plugins) to load these image files and for each question, read the lightness/darkness of each choice bubble (i.e. has it been coloured in or not).
- Peform calculations and generate graphs from this data using a spreadsheet.
You will need access to the following software:
- Spreadsheet program (Excel or equivalent)
- ImageJ (download the appropriate version and install)
- MicroArray Profile plugin for ImageJ (download and extract to the ImageJ plugins folder)
- The analysis spreadsheet: microarray profile-graphs.xlsx
- If you wish to try the rest of the process out you can download this zip file, which contains some sample MCQ scans: MCQsample.zip
1. Scan MCQ papers
I am very fortunate to have access to a photocopier that will feed and scan multiple documents rapidly and then email the resulting pdf to you.
I take about 20 MCQ sheets at a time and make sure they’re lined up with each other, then place in the document feeder (the alignment has to be pretty good; you want the same choice bubbles in the same place on each scanned page). Repeat as required, then scan. If you have a large number of papers (> 120) you may wish to do this in a couple of batches. Scanning takes me a couple of minutes for 180 papers.
2. Export pdf pages to image files
Open the pdf and in Acrobat choose File – Export – Image – JPEG (or whatever you need to do to get to a stage where each paper has been scanned to an individual image file) and save them to a blank folder.
When you need to choose JPEG settings, I use the following (96 pixels/inch is enough). You can use higher quality files if you like, but it’s not super necessary and it can hog a lot of memory when you import them into ImageJ in the next step.
If you wish to try the rest of the process out you can download this zip file, which contains some sample MCQ scans: MCQsample.zip
3. Import image files into ImageJ
Open ImageJ, and File – Import – Image Sequence and select the first image file you generated in step 2. You will be prompted to import the files, hit OK.
I usually rotate and crop images in order the facilitate placing the grid (step 5). My spreadsheet is set up for images rotated 90 degrees left, i.e. top of the page facing left). To rotate, Image – Transform – Rotate 90 degrees left. To crop, select the area you wish to crop with the rectangle tool (selected by default) then Image – Crop (or Ctrl + Shift + X). Zoom to your desired size so the image takes up most of the screen.
Select Plugins – Microarray Profile, then select the “Reset Grid” button from the MAP window that appears. Input the number of choice bubbles per question in “Rows” and the number of questions in “Columns”. In this example, there are 14 MCQ, each with 4 choice bubbles (A through D).
- If the grid of circles is much bigger than the grid of bubbles, Shift-drag a corner circle to shrink the grid to approximately the right area (in this case I shift-dragged the top right circle).
- Alt-drag a corner circle so that it aligns with the corresponding corner choice bubble in the image (in this case, I alt-dragged the bottom right circle to Q14 A).
- Shift-drag an adjacent corner circle so that it is positioned properly. Now the whole edge connecting the two corners should be lined up with the spots (in this case, I shift-dragged the top right circle; now the Q14 circles all line up).
- Shift-drag a spot in the interior of the opposite edge until that edge is lined up with the spots (in this case, I shift-dragged the circle corresponding to Q1 B).
- All of the circles should now be aligned more or less aligned with the spots. You can drag any individual circle to adjust its position independently of the other circles.
- (Optional) Scroll through the a few images to verify that the grid placement is correct for these images.
6. Read histogram and save results
Select the “Histograms” button from the MAP panel and wait patiently. When the results window pops up, select File -> Save As and save the file somewhere sensible (the default filename is Results.xls).7. Import data into spreadsheet
- Open microarray profile-graphs.xlsx. Select cell A1 of the ‘microarray‘ worksheet.
- Open Results.xls (that you exported in step 6 above). Excel will have a whinge at you that it’s not in the correct format; ignore it and select Yes.
- Select and copy all of the data in Results.xls (Ctrl + A, Ctrl + C). The top left of the sheet will look like this:
- Paste this data into the ‘microarray‘ worksheet of microarray profile-graphs.xlsx.
8. Set black/white threshold
- The data in column I (Mean) is the mean value assigned by ImageJ (where 0 = black and 255 = white). We need to pick a black/white threshold value so that coloured-in bubbles have mean values below the threshold, and blank bubbles have mean values above the threshold. Eyeball the data in the ‘microarray‘ worksheet, compare with the image files in ImageJ, and decide on a black/white threshold value (for my exam papers it’s usually somewhere between 170 and 210 depending on paper and scanning settings).
- Select the ‘summary data‘ worksheet. Adjust the black/white threshold value, if required (cell T1).
And that’s it! The results are now displayed in the ‘summary data‘ worksheet, and graphically in the ‘Graph *‘ worksheets.
Restrictions and limitations
The one major limitation is that choice bubbles coloured-in and then crossed out will be counted as coloured-in. For example, question 4 for the paper below will be counted as having both A and B coloured-in:
The only realistic way around this is to make students erase any incorrect responses. As it stands, though, this can actually provide some extra feedback on useful distractors even if the correct answer is ultimately selected.
I hope this was of use, please leave a comment if you have any suggestions for further improvement!