To support efforts of introducing an impact sound requirement into the National Building Code of Canada (NBCC), the National Research Council of Canada has initiated a long-term research project. In cooperation with the Hochschule Duesseldorf in Germany and Kangwon National University in Korea, several listening tests were performed to investigate the annoyance due to impact sound as it is perceived by building occupants, and how this annoyance relates to the results of standardized laboratory measurements. In this contribution, the different listening test setups will be described and compared. Tests were performed in the laboratory using loudspeakers, headphones, and also with a Virtual Reality headset. In addition, an online listening test was launched to expand the reach of such studies to the general population. The results will be summarized and discussed regarding appropriate rating methods. The goal of these studies is to gain a better understanding of the relevant factors that affect the perceived annoyance, such as the impact source type and the dominant frequency range, but also how the test environment influences the results.