You have worked hard on your project. You searched the literature, learned about the methods, painstakingly designed an experiment, and have almost finished implementing the software, but… what about the logging?
Most students think that logging is easy. Just write some lines on a text file. A couple of hours on the software should do, right? I don’t think so.
Experience has shown me that logging is extremely error-prone, and that paying little attention to it results in incredible loss of valuable information and time, and that most students do not realize how important doing good logging is.
Before I go on let me qualify a little bit what kind of logs I’m talking about. I’m referring mostly to the kind of experiment that you often see in HCI or Experimental Psychology, were there are many participants, and each participant performs many trials, possibly in multiple conditions. This is usually information that is suitably recorded in a simple format like comma separated values (CSV).
Let me state then my five fundamental laws of logging:
1. Log everything you can
Disk space is cheap, your time is expensive. Probably the most common mistake here is not to record enough information in each record because you think it is redundant and a waste of space. For example, why record today’s date in each record of each trial, if they are all the same? I tend to record the same information in each trial anyway, because it is always easy to discard info, but it takes a long time to recover data from different sources (including the name of the file, the creation date that the OS stores etc.). Don’t assume that you will remember where you are storing all that information when the time for analysis comes. Things that I tend to save in each trial record: userId, all condition values for all factors, the number of the trial with respect to the condition, the number of the trial with respect to the cell, the absolute number of the trial within the experiment (and the phase) and, of course, all the dependent variables.
Perhaps the only caveat to this is that all this recording should not negatively affect the performance of your program or the accuracy of the time measurements. If performance and timing accuracy is important, good strategy is to write first to memory, and only save to disk in between trials (or when timing is not an issue).
2. Make your logs self-contained
Name your variables wisely, and always include names on top of the file. This should be quite explanatory, and most analysis programs (e.g., SPSS) will allow you to name the variables automatically from the file. Handy and convenient. A good complementary practice is to have some comments (or a separate file) that provides an explanation of how each variable is recorded, but this requires discipline to maintain, because the logger program tends to evolve. Best to keep your measures simple.
3. Debug, debug, debug
Never assume that your code is recording properly. Simple visual inspection won’t cut it. I have experienced many problems that only became visible after all the experiments were recorded. The best way to avoid problems here is not only to debug, but also to use your pilots to gather realistic data, and analyze it in the same way that you will analyze the overall results from the finished study. This is not only good for your logs, it is also helpful to avoid possible flaws in your statistics (e.g., I do not believe in a posteriori power analysis).
4. Backup, backup, backup
Don’t trust your hard-drive, don’t trust your experimental software. Within your program, save the data to drive as soon as you can (but take into account the comments in point 1). This will allow you to recover from failures in your software. It is actually kind of nice to code your experimental software so that you can restart it again at any given trial within the session. When the experiment is finished, the first experimenter action should be to verify that the data is in the right place, and perhaps making a copy (or send yourself the data to your gmail account – if your data is properly anonymized, of course).
5. Protect yourself against confusion
If something can go wrong during the experiment, it probably will. It is good practice to save the date hour and second of experimental recording in the name of the file that your program saves. This will help you prevent accidental overwrites. Similarly, try to leave as little as possible human intervention for the actual session. For example, I never trust the experimenter -often myself- to select the right name for multiple files depending on the condition. Have the software do something reasonable for you. The only thing that I often make configurable is the participant identifier, so that I can separate real trials from debugging logs.
Hopefully these might be useful to you some time. Write a message below if you agree/disagree or want to add some more advice!
Great post. Three additions:
1. Always store all raw input data (e.g. the phrase set users have to type using your new input method, the intro text you read to all participants, etc.) in addition to the logged data. Having to parse the log file for getting basic information such as “How often was each task repeated” is very annoying.
2. Automate as much of the data preprocessing and analysis and document it in an idempotent script. In five years you won’t remember the grep/sed/etc. commands you used for filtering out unwanted data. Also, never use manual copy&paste in data analysis.
(A student of mine insisted on using Excel for analyzing his quite complex user study, manually copying the results of each analysis step into a new sheet. After a dozen such copy/paste steps, the results were obviously wrong, and it was impossible to find out where the error occurred.)
3. Use version control for your data and software, and put both together in one repository. Also, commit everything after each test run. By doing this, you directly link the logged data to the version of the software which created the data. This will allow you to easily identify all logs which have been created with a buggy version of the software, or with different input parameters.
A good and much valued read – thanks very much!