1KA Online Probability Panel

The 1KA Panel is an infrastructure for scientific research into the methodology of online panels, as well as for studying selected topics in social science research in the academic and public sectors, where a higher quality of survey estimates is required. In developed countries, a relatively clear distinction has emerged between probability-based panels, which are used almost exclusively in the public and academic sectors, and business, market research, and public opinion surveys, which rely almost exclusively on non-probability panels (so-called access panels).

The 1KA Panel includes adult residents of Slovenia. Where specific requirements arise, additional recruitment can be carried out (e.g., young people aged 15–17). Recruitment is conducted from the population register using probability sampling, while subsequent panel waves include individuals who have agreed to participate in online surveys conducted by the public and academic sectors (details on the sample, recruitment, and participation are available in the Methodology).

Surveying in the 1KA Panel is conducted primarily online; where additional requirements arise, data collection can be supplemented by other modes (e.g., postal or telephone surveys). The 1KA Panel infrastructure can also serve as the basis for an entirely new or independent recruitment wave, in which case the participation rate is considerably higher.

The 1KA Panel comprises 3,500 panel members. Due to its probability-based sample, relatively high participation rate, and careful implementation, it provides high-quality survey estimates.

The 1KA Panel is operated by the Centre for Social Science Informatics (CSI) of the Faculty of Social Sciences at the University of Ljubljana, which also develops the online survey tool 1KA. The Panel is funded through research project funding and client funding.

Mission of the 1KA Panel

The mission of the 1KA Panel is to establish and maintain public, non-profit infrastructure for conducting surveys of the general population, following the principles of probability sampling and the highest methodological standards.

The Panel is intended to support scientific research into the methodology of online panels, as well as survey research in the public and academic sectors.

In terms of subject matter, the 1KA Panel primarily covers the social aspects of artificial intelligence and related social science fields that fall within the European vision defined by the NextGenerationEU instrument (Make it Green, Digital, Healthy, Strong, Equal).

What the Panel Does Not Do

The Panel does not conduct commercial market research. This excludes the testing of products, brands, advertisements, and prices; measuring customer satisfaction on behalf of clients; and any form of generating sales leads. Every survey must have a research purpose in the public interest. The results should, in principle, be publicly accessible, while microdata are generally archived in the Slovenian Social Science Data Archives or an equivalent data archive.

Establishment and Development of the 1KA Panel

The Panel was established on a pilot basis in 2022 as part of the CRP project Probability-Based Online Panels in Official Statistics for Individuals and Households, which examined the potential of a probability-based online panel of the general population for the purposes of official statistics.

Further methodological research on online panels was conducted as part of the TAP project Evaluation of Probability-Based and Non-Probability Online Panels, which developed an integrated approach for comparing probability-based and non-probability online panels, as well as within the TAP project Assessing and Modelling Nonresponse Bias in Probability-Based Online Surveys.

Continuous research into the methodological aspects of online panels is also carried out within the basic research programme of the research group Internet Research.

Advantages of 1KA Panel

The 1KA Panel is based on probability sampling and has the following advantages compared to probability samples using traditional survey methods (postal, face-to-face and telephone surveys):

• Significantly lower costs.
• Faster data collection process, because recruitment (including respondent consent) has already been done.
• Faster data processing, because each completed survey is automatically entered into the database.
• Lower social desirability bias, leading to more honest answers on sensitive topics (e.g. drugs, health, sexual activity, finance, politics).
• As the panel is relatively large, it enables the selection of the target population based on the data obtained, e.g. only units of a certain age, region, income group, etc.

Compared to commercial or non-probability online panels, the main advantages of the 1KA Panel are:

• Significantly better representativeness of the sample, as the sample is (probabilistically) recruited from the entire adult population in Slovenia (1.7 million) and not just from the few percent of the Slovenian population willing to join commercial non-probability online panels.
• This is also associated with lower bias and thus higher accuracy of estimates, which stems from the probability nature of unit selection. A comprehensive comparison of probability and non-probability panels (TAP) has shown that probability panel estimates are significantly closer to the population value.
• For academic research where data are modeled, an important advantage is the significantly greater diversity of respondents. This also increases the variance that the models can explain. In non-probability panels, respondents are often very similar and most responses may be primarily at one value, e.g. 4 (on a scale of 1-5), with minimal occurrence of other responses (3, 5). In a probability panel, however, the variation is significantly greater because units are recruited from the entire population.
• In particular, probability samples – also called scientific samples (Kalton & Vehovar, 2001, p. 11) – provide a formal basis for statistical inference, so that the risks of sampling error can be calculated in the first place.
• The scientific nature of probability samples can, in the case of academic research, significantly increase the likelihood of results being published in the most reputable scientific journals.
• An important advantage is also the reduced burden on the respondent. This is because panellists receive an invitation to take part in a survey at most once a month, which is very different from commercial panels where invitations are much more frequent. This avoids the risk of panellists becoming so-called ‘professional respondents.’
• The topics covered by the surveys in the 1KA Panel are generally relevant social issues, thus avoiding the extremely broad spectrum of commercial sector content. This general relevance may discourage panellists from responding in a disengaged and routinised manner and especially discourages the over-involvement of respondents interested in a particular topic.
• All questionnaires are systematically reviewed for compliance with the 1KA Panel’s ethical and professional standards, ensuring a high level of quality.
• The participation rate of the panellists is very high (around 80% after two years) for all the reasons mentioned above, which contributes significantly to the quality of the results.

It is also worth highlighting the advantages of the 1KA platform, which the 1KA Panel uses for its surveys:

• Users can develop, create and test the questionnaire themselves in the 1KA tool, which is a very user-friendly, simple and intuitive online survey tool that has been used by more than 100.000 users in Slovenia. This way, they save time and resources while at the same time developing a hands-on relationship with their questionnaire. Of course, external methodologists or content experts can also be involved in the development of the questionnaire.
• Proper use of the 1KA tool eliminates the time-consuming pre-preparation of data, especially in terms of missing values and non-responses (e.g. Don’t know, Prefer not to answer).
• Users can (if they wish) easily weight the data themselves and play with the different options, as a user-friendly 1KA weighting application has been developed for this purpose.
• Both the 1KA platform as well as the 1KA Panel are managed by the Centre for Social Informatics, so the integration of IT support for each step of the survey process is particularly efficient.

Privacy

All research is conducted in accordance with the European General Data Protection Regulation (GDPR) and national data protection laws. The sample consists of residents of Slovenia randomly selected on the basis of the Central Population Register of the Republic of Slovenia. Participation is voluntary and participants may withdraw at any time.

In surveys, no personal data is usually collected. If this is not the case, respondents are informed accordingly. In line with the GDPR, for academic and research purposes in the public interest, personal contact data are used solely for sending invitations.

How we handle the collected data:

• All personal information is stored securely, in accordance with European and Slovenian legislation.
• Access to contact data (e.g. name and address) during the course of the study is securely restricted to the Centre for Social Informatics (CSI) at the Faculty of Social Sciences, University of Ljubljana, which is conducting the research.
• Names and contact details are reliably and permanently deleted at CSI after the completion of data collection and before any publication of results, while anonymised responses (from which it is no longer possible to identify the respondent) are securely stored for an indefinite period.
• No respondent can be identified from the published results. Sensitive answers and their combinations that could even potentially enable the identification of a respondent are deleted or protected by a special restricted-access regime.

Participants’ rights: Until the publication of the data, participants have the right to amend, supplement, or request the deletion of their answers to any question (form). After the deletion of personal data, it will no longer be possible to retrieve the responses, and therefore later changes or deletions will not be possible. In the event of a violation of rights, participants may contact the Office of the Information Commissioner of the Republic of Slovenia.

Responsible persons: The research is conducted by the Centre for Social Informatics at the Faculty of Social Sciences, University of Ljubljana, which is also the data controller. The head of the research is Prof. Dr Vasja Vehovar.

Professional Standards

The online survey process is conducted according to 12 steps and corresponding recommendations – see Callegaro et al. (2015) adhering to professional standards set by AAPOR and ESOMAR.

While explicit ethical consent is not required for including a survey module in the 1KA Panel, the 1KA Panel team reviews every questionnaire to assess whether it violates general professional or ethical standards of the University of Ljubljana (UL) or the Faculty of Social Sciences (FDV), and notifies the client accordingly.

However, those who require ethical approval due to the demands of a client, publisher, or journal, or because of the nature of the research (e.g., doctoral studies), must obtain the appropriate approval in Slovenia, just as they would elsewhere in the world where the IRB (USA) or ERB (Europe) systems are established. If you require or if approval within the framework of the FDV is sufficient, you may contact the FDV Ethics Committee in Research via email at kerfdv@fdv.uni-lj.si.

The Research Ethics Committee of the Faculty of Social Sciences:

• reviews applications for the ethical assessment of research assignments and projects conducted within the Faculty of Social Sciences that involve research work impacting human privacy or involving human subjects.
• accepts applications from professors, researchers, and research associates employed at UL FDV, as well as students (upon the recommendation of their mentor), and provides opinions on proposed research projects involving human subjects that utilize humanities and social science methodologies.

Online Survey Panels

Online survey modes have revolutionised survey research and the survey industry (see Callegaro et al., 2015). Online surveys have become the dominant survey mode, while at the same time making surveys much simpler, faster, cheaper and more accessible. Today, virtually any internet user can create an online survey (not just specialised agencies as in face-to-face or telephone surveys). As a result, the number of surveys has increased exponentially.

On the other hand, recruiting people to participate in online surveys is becoming increasingly difficult, mainly due to declining participation, rising costs (respondents need to be rewarded) and growing privacy concerns. In this context, online panels have emerged where people agree to regularly participate in various online surveys in exchange for certain rewards. This eliminates the need to recruit respondents for each individual survey.

The vast majority of online panels do not control the probability of people being included in such panels. In this case, we are talking about commercial or non-probability online panels (a.k.a. access panels), which recruit people to the panel in a variety of ad hoc ways (e.g. online, through another survey).

However, for more complex surveys in the public, non-profit and academic sectors, online probability panels have been set up in most developed countries, which select respondents probabilistically/randomly (e.g. from a population register) and therefore do not involve any volunteers. For details, see Vehovar & Čehovin, 2022.

Probability Samples

Online probability panels, such as the 1KA Panel, are based on a probability sample, where each unit in the population has a known and positive probability of being included in the sample (Kalton & Vehovar, 2021, p. 11). Probability samples typically require a list of all units; in the case of the general population, this is usually the population register or a list of all postal addresses in the country.

Probability samples are a prerequisite for generalisation or statistical inference from a sample to a population, i.e. for calculating confidence intervals and testing assumptions. If the sample is not probability-based, then generalisation to the population is, in principle, not possible, and the associated risk is unknown.

Non-probability samples (and also non-probability panels), where we do not know the probability of a unit’s inclusion, are faster and cheaper than probability samples, but give less accurate estimates. Many comparative studies have been carried out (e.g. Cornesse et al., 2020; Mercer & Lau, 2023; Dutwin & Buskirk, 2017), which have uniformly shown that estimates derived from probability panels are significantly more accurate than those from non-probability panels.