What and how to log in your experimental HCI software

You have worked hard on your project. You searched the literature, learned about the methods, painstakingly designed an experiment, and have almost finished implementing the software, but… what about the logging?

Most students think that logging is easy. Just write some lines on a text file. A couple of hours on the software should do, right? I don’t think so.

Experience has shown me that logging is extremely error-prone, and that paying little attention to it results in incredible loss of valuable information and time, and that most students do not realize how important doing good logging is.

Before I go on let me qualify a little bit what kind of logs I’m talking about. I’m referring mostly to the kind of experiment that you often see in HCI or Experimental Psychology, were there are many participants, and each participant performs many trials, possibly in multiple conditions. This is usually information that is suitably recorded in a simple format like comma separated values (CSV).

Let me state then my five fundamental laws of logging:

1. Log everything you can

Disk space is cheap, your time is expensive. Probably the most common mistake here is not to record enough information in each record because you think it is redundant and a waste of space. For example, why record today’s date in each record of each trial, if they are all the same? I tend to record the same information in each trial anyway, because it is always easy to discard info, but it takes a long time to recover data from different sources (including the name of the file, the creation date that the OS stores etc.). Don’t assume that you will remember where you are storing all that information when the time for analysis comes. Things that I tend to save in each trial record: userId, all condition values for all factors, the number of the trial with respect to the condition, the number of the trial with respect to the cell, the absolute number of the trial within the experiment (and the phase) and, of course, all the dependent variables.

Perhaps the only caveat to this is that all this recording should not negatively affect the performance of your program or the accuracy of the time measurements. If performance and timing accuracy is important, good strategy is to write first to memory, and only save to disk in between trials (or when timing is not an issue).

2. Make your logs self-contained

Name your variables wisely, and always include names on top of the file. This should be quite explanatory, and most analysis programs (e.g., SPSS) will allow you to name the variables automatically from the file. Handy and convenient. A good complementary practice is to have some comments (or a separate file) that provides an explanation of how each variable is recorded, but this requires discipline to maintain, because the logger program tends to evolve. Best to keep your measures simple.

3. Debug, debug, debug

Never assume that your code is recording properly. Simple visual inspection won’t cut it. I have experienced many problems that only became visible after all the experiments were recorded. The best way to avoid problems here is not only to debug, but also to use your pilots to gather realistic data, and analyze it in the same way that you will analyze the overall results from the finished study. This is not only good for your logs, it is also helpful to avoid possible flaws in your statistics (e.g., I do not believe in a posteriori power analysis).

4. Backup, backup, backup

Don’t trust your hard-drive, don’t trust your experimental software. Within your program, save the data to drive as soon as you can (but take into account the comments in point 1). This will allow you to recover from failures in your software. It is actually kind of nice to code your experimental software so that you can restart it again at any given trial within the session. When the experiment is finished, the first experimenter action should be to verify that the data is in the right place, and perhaps making a copy (or send yourself the data to your gmail account – if your data is properly anonymized, of course).

5. Protect yourself against confusion

If something can go wrong during the experiment, it probably will. It is good practice to save the date hour and second of experimental recording in the name of the file that your program saves. This will help you prevent accidental overwrites. Similarly, try to leave as little as possible human intervention for the actual session. For example, I never trust the experimenter -often myself- to select the right name for multiple files depending on the condition. Have the software do something reasonable for you. The only thing that I often make configurable is the participant identifier, so that I can separate real trials from debugging logs.

Hopefully these might be useful to you some time. Write a message below if you agree/disagree or want to add some more advice!

Our mini central-european tour: Munich, Konstanz, Zurich

Last week, Uta and I had the chance to take a tour of three impressive labs in Germany and Switzerland. The German and Swiss hospitality cannot be overstated, but most impressive was the range of research.
curvepublic wall displayIn Munich we visited the Media Informatics and Human-Computer Interaction Group, invited by Andreas Butz. It was really nice to see finally the curve (in the picture), among lots of other excellent research, including work by Alice Thudt, a current collaborator of Uta.

collaborative search controlrooms2In Konstanz, we visited the Human-Computer Interaction group led by Prof. Harald Reiterer. The range of research and development is very broad. A particular favorite of mine is the work on zoomable multi-display environments (the ZOIL API), and a number of other interesting experiments related to large displays.

Finally, we had the chance to visit Dr. Elaine Huang and her ZPAC laboratory; we have strong links with this lab (including Helen, another iLab graduate), but there were many other strong research reasons to visit ZPAC; most related to me is the work by Gunnar Harboe, but it was great to learn too about projects on sustainability, cultural communication, and domestic ubicomp.

Naturally, I cannot make justice to everything that all these researchers do in a few lines… maybe you should just visit them too :). We really would like to thank all our hosts for wonderful and insightful visits (special thanks to Fabian, Hans-Christian, Christian, Alice and Helen for bearing with us for so long). We are looking forward to your visits!

Amazing visit to the special collections of the University of St Andrews

Yesterday was a busy day. We have been working with the University of St Andrews Library and the Museum of the University of St Andrews (MUSA) to put together some seminars (first, second) on the use of interactive technology in libraries and museums, and to get some exciting collaborations started. We were lucky to have very interesting speakers and, as a treat, we got to take a tour of the Special Collections. Daryl Green and Maia Sheridan showed us some amazing artifacts that we are hoping we can help make available to more people through interactive technologies in new and exciting ways. The photos show some of the amazing books (although they have all sorts of interesting artefacts). I’ll leave it to you to go to the library and found out which ones these are, but some of the stuff we saw has existed for more than 900 years!!

This is one of the perks of working in an institution with 600 years of history. If you like this stuff, you should not miss  the  blog of the Special Collections.

 

Umer graduating…

Today, Umer Rashid (almost Dr. Umer Rashid) came to my office to give me a copy of his PhD thesis: Cross-display Attention Switching in Mobile Interaction with Large Displays.

Great moment of pride and satisfaction for a job well done… Now it is time for Umer to leave the nest and show the world his best researcher skills. Good luck in Lincoln!

PhD Studentship on Perceptual Gaze-contingent Displays at St Andrews (with me)

I’m looking for a student to work with me in St Andrews for the next few years (fully funded). The topic is gaze-contingent displays. I’m looking for someone with a Computer Science background that has an interest in perception and visualization, and a big curiosity about human perception and what we can do to enhance it. Please, visit the studentship offer, and send me an e-mail if you are interested! (deadlines for the studentship are soon, so don’t delay!!).

Friday 6th, event with the Library (Shawna’s visit)

The title is: Interaction and Visualisation Technologies in the Library – Open Session, and the idea is to bring together the people at the university that might benefit from data and visualisation support in the library. We’ll have the privilege of listening to Shawna Sadler, who I have worked with in Calgary.

This is the main page of the event.

Mathias Frisch Defence (VIVA) in Magdeburg

This week I had the honor to attend the defence of Mathias Frisch, student of Raimund Dachselt at the Otto von Guericke University in Magdeburg (now moving to the new lab in Dresden). I was very happy to review such excellent work, (in a committee that includes Prof. Reiterer, from Konstanz) but also to take part in the traditions of doctoral graduation. I think that it is very valuable that a doctoral graduation is approached as a festivity, and loaded with symbols of what it means to be a scholar. These include the new doctor having to wear a very heavy chain (that represents the loads awaiting in his future career),

wearing a hat representing the knowledge of the candidate (physically and literally),

being taken on a pageant through the city, escorted by a troupe of people in medieval outfits (I guess to let the town folk know that there is a new doctor around),

having to drink an “unusual” concoction (in this case tomato juice + coffee + salt and pepper),

as well as later on having to go through the “real exam”, in which the supervisor has to show knowledge and commitment for the candidate in multiple odd ways.

In any case, it was a wonderful day for me, full of good HCI, fun and food. Congratulations Mathias!!