Statistics is an essential knowledge that all researchers in the world should know.
You need to understand the followings:
- Descriptive statistics - mean, median, SD, categories of data, normal distributions, histogram
- Inferential statistics - parametric and non-parametric
- Null and alternative hypothesis
- Type-I and type-II errors
- P-value means and what it does not (statistical significance vs. practical significance)
- Assumptions of each test
- Know what "main effect" and "interaction effect" means
- Know how to report in APA style, e.g., p-value, effect size
- Know what is posthoc test, and when should you use it
- Know how to plot data correctly with 95% CI error bars, and visualization approaches
- Aware that correlation does not mean causation
- If you apply machine learning, understand the math behind, the limitations of each algorithm, instead of just using the algorithm in blackbox.
- Not significant does not mean your work is bad, it could be interesting, try to think why it happens
- Statistics is just a tool. Focus on interpreting against your hypotheses.
Example of real review you will get when you use wrong statistical methods:
"The data is ordinal, and cannot be analyzed with parametric methods. In some cases, where the sample size is large, t-test might be applied for ordinal data, but not here, where the sample size (the number of participants) is below 20.
To me, Wilcoxon signed rank (paired samples or matched pairs) test should be the correct method"
Learn from Jakob Wobbrock - https://www.coursera.org/learn/designexperiments - Highly recommended!
Excellent self-study tool for statistics provided by Jakob Wobbrock - http://depts.washington.edu/aimgroup/proj/ps4hci/
Learn basics - https://www.coursera.org/specializations/statistics - Highly recommended
Learn from Koji Yatani - http://yatani.jp/teaching/doku.php?id=hcistats:start - good for code snippets
Book written by experts - http://www.springer.com/in/book/9783319266312
Learn how to use R - http://www.amazon.co.jp/Discovering-Statistics-Using-R/dp/1446200469 - good reference for R; I highly recommended R as the usefulness of R functionalities expands beyond HCI research; For those who likes to use GUI, maybe you can try SPSS, the same author Andy Field writes the similar book for SPSS.
How to report statistics in APA style (used in SIGCHI papers) - http://my.ilstu.edu/~jhkahn/apastats.html - Please read this before you report your data; beware that there are some CHI papers that report data wrongly. Also, please report effect size (e.g., eta square) for statistical tests, which is a good practice.
Excellent work by Paul Cairns disseminating the problems of statistics in HCI papers - http://dl.acm.org/citation.cfm?id=1531321
Important principles as a statistician - http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004961