Propensity score - A method of adjustment to revise internet surveys

Transition towards internet surveys

Requiring less money and time than traditional approaches such as interview surveys or mail surveys, internet surveys have been rapidly increasing its market share in recent years. A number of companies have replaced the traditional approaches which they have used for a long time with internet surveys. However, this transition often involves problems. One problem is "The results of the internet survey are completely different from those of our older surveys!" SPI offers a solution for this case.

What is "propensity score" ?

To solve the above problem, a method that applies weight on each respondent using "propensity score" has started garnering attention these days. The "propensity score" is described as a probability that a respondent is more likely to emerge in an internet survey, and it is used in accordance with the logic that if the inverse number of the probability were used as a weight on the respondent, then the bias would be corrected. For instance, if the emergence probability of sample group A is 80% in the internet survey, you have to apply 1/0.8=1.25 points. And if the probability of group B is 25%, you have to apply 1/0.25=4 points. Therefore, we put greater influence on the sample group that does not tend to emerge rather than the other group that tends to do so.

How the "propensity score" would be calculated

Generally speaking, when you calculate the propensity score, you use the logistic regression analysis which is effective for probability calculation. First of all, the data from both an internet survey and a traditional survey are combined. Next, the sample group from the internet survey would be assigned 1 (equal to 100%) as a dependent variable and the group from the traditional survey would be assigned 0 (equal to 0%). Then you select multiple items that are seemingly concerned with the emergence ratio in the internet. To regard the items as explanatory variables, the propensity score model, also known as the emergence ratio model, of the internet survey can be built. After following these steps, the emergence ratio of each respondent in the internet survey can be predicted as exemplified below:

Things to watch out for when you build a model

Since it is the model that predicts the emergence ratio in an internet survey, the selection of the explanatory valuables would be significant in making it accurate. Survey items such as sex, age, and frequency of the internet usage might be essential. In addition, in case the proportion of an item adopted as one of explanatory valuables of both internet survey and the traditional one greatly differ from each other, the weightiest points would be added. You need to be aware of this. In actual fact, we found that we could not correct the bias even when we calculated the propensity score. That is because the questionnaire did not include a question which asked about frequency of internet usage. Moreover, the age composition was totally different between the internet survey and the traditional survey in that case. We think that if the propensity score is to be used, the decision to use it during the initial phase of survey design is crucial to avoid this problem.

Is the bias actually corrected?

There are various research results connected with propensity scores, and they say if the explanatory variables are adequately selected, you can correct the bias. However, one thing to remember is that the propensity score is calculated and added as a weight not only on the respondent in internet surveys but also on the respondent in traditional surveys. Thus bringing both survey data closer to each other, this method realizes a correction of the bias of the survey data.

 

Besides the solution that we have presented in this article, SPI has been conducting a lot of consultations on marketing research. Please contact us for more information.

Author: Yusuke Saito / Senior Analyst

Please contact us with questions or for more detailed information.
spiindex@spi-consultants.net