How do I set up plausibility checks in questionnaires?

Internet surveys permit quality checks and data checks as early as during entry. Minor errors and missing information can thus be noted immediately during entry. The respondent will receive a short message indicating that they may have overlooked something. You will not have to ask follow-up questions afterwards but can correct the incorrect information immediately during entry instead. Checks for the correctness of answers are commonly referred to as “plausibility checks”.

Plausibility checks are used to ensure a certain level of data quality. Sometimes, respondents will overlook a question or commit minor errors when setting an answer.Sometimes, questions are also answered incompletely, if the respondent for example wants to get an overview of the questionnaire first or completes the questionnaire without serious interest. The information given should be checked especially if branchings at later stages in the questionnaire are based on the answers to preceding questions. Plausibility checks are advisable in the following cases:

• The answer to the question is taken up in later questions, either in a filter condition or in a display of dynamic questions.

• Assuring fill-in instructions, such as “Select the three most important properties of products xy”.

• Answers to a specific question are of particular interest for the evaluation.

• The internal consistency of the data is of particular importance for the evaluation.

• The data being queried already exist in participant administration.

Be aware of which answers are important for the routing of the survey and which data are central to the evaluation purpose. However, use plausibility checks with care. For the respondent, plausibility checks are, first and foremost, annoying and tiresome because their behavior is questioned and their progress through the survey is hindered. Too many plausibility checks will destroy the relationship of trust between you and the respondent. Also, take into consideration that respondents may not yet have formed an opinion on some questions. In such cases, you will often achieve better data quality by allowing incomplete information than by forcing the respondent to give an answer which might only be valid for the moment and not actually meant by the respondent.

Think about aspects such as the following:

• Which answers are indispensable for the routing of the survey?

• To which questions do you want to obtain answers from as many respondents as possible?

• How would you react if your answer to a question were rejected as incorrect by a survey system?

In addition to the psychological effects on the respondents, there are technical aspects as well:

• On each questionnaire page, you can create as many plausibility checks as required. But plausibility checks require a lot of server resources, i.e. they affect the performance of the questionnaire, similar to other checks and dynamic features. Therefore, it is recommended to use not more than 50 plausibility checks per page.

• For plausibility checks, no dedicated sort order is implemented. I.e. while you may arrange the checks on a specific page in a specific sequence, this order may get lost especially if copying or importing the project.

• The conditions of plausibility checks should refer only to variables which have a defined value. If condition variables are not filled at all or contain missings (e.g. because a question or answer option has been hidden or if a respondent hasn’t filled an entry field) the checks can have unexpected or even wrong results.

• Try to avoid complex plausibility check conditions or to substitute them by multiple simple plausibility checks. This way it is easier to get an overview of the variables used in the plausibility check and to avoid unexpected results.

• If necessary, the pro editor allows to define complex conditions.

• In general, it is recommended to abstain from applying both hiding conditions and plausibility checks to the same variables.

• When dealing with question types whose variables have characteristics (e. g. single response list), please note: As soon as a single answer option of the question, i.e. one characteristics of the variable, is hidden, the variable will be defined as “missing” when used in a plausibility check. Therefore, in this case, the actual value of the variable cannot be determined by means of a plausibilty check. If you cannot or do not want to work without the plausibility check, you might want to consider using a list instead of hiding conditions.

• If a plausibility check checks several variables, and only one or a few of these are affected by hiding conditions, it may make sense to deploy the option “Execute check if one or more items are blinded out?”. If this option is activated, the variables affected by the hiding condition will simply be skipped when executing the check. Mind that if all variables of the check are affected, it will not be executed, to prevent that participants get stuck on the page.

Let’s assume that you ask the participants of your survey to enter their year of birth into an open entry field. You want to ensure that they enter only reasonable values: The values should be four-digit numbers inside a reasonable range, e.g. between 1900 and 2000.

To realize this with a plausibility check, please proceed as follows:

• In the questionnaire editor, choose the page on which you want to perform the plausibility check. Click on the title of that page. The page view will open.

• Click on the Plausibility checks menu.

Open

• The overview of plausibility checks is opened. No checks have been defined yet.

• Click on the + Plausibility check button.

• Enter the title.

• Select the “Range check” check type. For detailed explanations on the check types, please see this post.​

Open

• Confirm with Proceed.

• The entry dialog is opened.

• In the “User may ignore this check” field, the “No” option should be activated. This means that the person completing the questionnaire must correct any incorrect entry. Otherwise, the next survey page will not be displayed.

• Only in projects with classic system layout: In the “Systems to use for check” field, the “Client (JavaScript) AND Server (PHP)” option should be activated.

• Do not change the setting of the field „Execute check if one or more items are hidden??“. In the current example situation, it does not matter as there are no hidden items.

• When executing JavaScript plausibility checks you may choose to let the participant neglect the plausibility check. In this case check Yes with the option offered.

• Choose which system you would like to check in the responsive layout: Client (JavaScript), Server (PHP) or Client (JavaScript) AND Server (PHP).

• In the following field, you can edit the message which is displayed if the check condition applies. Please replace the default message “An error occurred!” by “Please enter your year of birth as a four-digit number!”.

• Next, define the check condition itself: First, select the variable which you want to check.

• Then, enter the range within which the values are to be valid. In the example shown, “1900” is the minimum, “2000” the maximum.

• Click on the Save button.

• To see the pop-up which will be displayed to the respondents, open the Page preview tab and enter an erroneous value to trigger the check. 

• The new check is listed in the overview.