User Experiments

In BCI, there is a need to conduct user experiments, in order to gauge the performance and reliability of our systems.  Here I list some simple checklist of what you might want to know.   All of this information should be easily learned from reading papers and attending seminars:

Checklist for experimental design:

  1. What are IVs and DVs
  2. What are within- and between-subjects design and pros and cons
  3. What are the meanings of factors, levels, blocks, and trials, also learn the synonyms (e.g., IV = factors, levels = conditions, metrics = DV)
  4. Internal validity, external validity, construct validity (most non-successful papers failed in one or more of these categories)
  5. Clear idea why you choose certain research methodologies (don't make up your own method)
  6. Clear idea why you choose to do the experiment in this way (why give users feedback versus not)
  7. Why we need to order conditions, and how many ways to order conditions: fully counterbalanced, partially counterbalanced, fully randomized;  What can be randomized and what cannot
  8. How to decide sample size using power analysis
  9. Balancing quantitative and qualitative data; what are the pros and cons of each data, and it is usually best to use mixed methods

When you design your experiment, keep in mind:

  1. Conduct pilot studies: conduct several pilot studies before the actual study
  2. Avoid making your own new methodologies without good reason; always learn what are the standard methodologies and tasks
  3. Be clear yourself why you choose a methodology over others, and what are the pros and cons
  4. Avoid "exploratory study" when possible.  Most of the time, this kind of “exploratory” experiments such as doing a survey or conducting interview will not lead you to any results.  Investigate with a solid hypothesis is much more efficient than exploring aimlessly.
  5. Make sure your participants are not too tired.  Manage your experimental time wisely.  1 hour is usually the maximum amount of time per participant.
  6. Make sure you log everything (logger, webcam, notes-taking).  The purpose of logging is so that we can replicate the whole experiment and see what happens. 
  7. Make sure you have enough participants.  Learn power analysis.
  8. Make sure your experiment pass the three validity test:
    1. Internal validity:  whether what you want to find out is well-isolated from other noise effects
      1. When you are comparing two conditions, did you make sure everything else is equal except what you are manipulating?
      2. Are you sure that the variables you are manipulating have really the desired effect that you want?  Run a usability test before an actual test can help.
      3. Did you correctly order the experimental conditions?  Order effects should be always minimized.
      4. Did you assign users to different groups in a randomized way?  If not, why?
      5. Did you take care of learning effects by applying appropriate training before the experiment or applying block design?
    2. External validity: is your result generalize across people and contexts
      1. Did you choose the participants that are representative of the world?
      2. Did you choose the experimental task that is representative of the world?
      3. Is your system or experimental tools representative of the world?
      4. At last, did I over claim my results in the paper?
      5. Did you discuss any design or real-world implications when the results are not significant?
    3. Construct validity: whether you are measuring things based on what you claim
      1. Measuring happiness but uses only interview or user preference  or measuring text-entry performance with only speed but not errors
      2. Measuring typing performance but ignore that people can type while they are walking or sitting or standing
      3. Try to use mixed methods - combining quantitative and qualitative
      4. Talking about habit formation but collect data using only five days of experiment

Checklist for apparatus:

    1. Recruit participants
      • Recruit people
      • Make sure they are either paid or are provided with some gift cards (it's important to provide enough motivation to conduct the experiment)
      • Make sure you do not recruit people that have some potential biases such as your close friend (
    2. Informed consent
      • Before doing any experiment, it is ethical and it is the first thing in an experiment to let participants know about the experiment (about the possible injury or participant disclosure, etc), and give consent about participating.  Participants should allow knowing their rights, their privacy, and the amount of time they need to spend in the experiment.  If they do not want to participate anymore, they should allow to freely withdraw.   Of course, you should also avoid priming, i.e., asking some questions that may affect their performance or results.  Priming is often a confounding variable for a poorly designed experiment, so please read more about priming.  See example informed consent doc here.
    3. Inform participants before the experiment - for physiological experiments, please inform the participants prior to the experiment, what they should do, e.g., do not drink coffee, do not shower one hour before the experiment, etc.
    4. Demographic sheets
      • In the Participants section, you need to write who do you recruit, their age, their gender, any characteristics that may affect the study results (e.g., experience) and the criteria of inclusion and exclusion.  Usually, you provide this questionnaire to the participant as the second thing, after the informed consent.  See here for example.  I recommend using Google Forms for collecting such data (using iPad or desktop), which is an excellent e-tool to automatically put your data in digital form.  Also, it is environmental-friendly.
    5. Your software or tools - make sure your software logs everything, key point is that you should be able to reconstruct the whole experiment and understand what exactly happens during the experiment.  For example, time log or any measurement tool should be in milliseconds, not seconds.
    6. A checklist of your procedure - make sure you follow the procedure.  This is to avoid human errors which are often the case when there are many tiny steps.  For example, forgetting to instruct your participants not to move their head could completely destroy your whole EEG data.  Or forgetting to measure the baseline physiological data make your data uncomparable.
    7. Webcam, video camera, screen capture.  Very important to capture temporal behaviors of users.
    8. All questionnaires - use only standard questionnaires.  Please do not come up with your own questionnaire.  All standard questionnaires have been scientifically proven while your own is likely to be faulty.
    9. Make sure the setup is clean and quiet, and same across all conditions.  Don't overlook this.


    Below contains some useful links where you can learn from.

    Learn from Jakob Wobbrock -

    Learn from Mackenzie -

    Learn from Shengdon Zhao -

    An excellent reference about research methods by Anna Cox and Paul Cairns -

    For field experiment, check out this handbook -