Volume 10, 2022: Issue 2

 PDFDownload the article (Free)


Pilot testing of experimental procedures to measure user's judgment errors in simulated social engineering attacks


Tommy Pollock, Nova Southeastern University, USA

Yair Levy, Nova Southeastern University, USA

Wei Li, Nova Southeastern University, USA

Ajoy Kumar, Nova Southeastern University, USA


Distracted users appear to have difficulties correctly distinguishing between legitimate and malicious emails or search engine results. Additionally, mobile phone users appear to have a more challenging time identifying malicious content due to the smaller screen size and the limited security features in mobile phone applications. Thus, the goal of this research study was to conduct a pilot test and validate a set of field experiments based on Subject Matter Experts (SMEs) feedback to assess users’ judgment when exposed to two types of simulated social engineering attacks: phishing and Potentially Malicious Search Engine Results (PMSER), based on the interaction of the environment (distracting vs. non-distracting) and type of device used (mobile vs. computer). This paper provides the results from the pilot test we conducted using recruited volunteers consisting of 10 participants out of 20 volunteers invited. Due to COVID-19 restrictions, all interactions in this pilot testing were conducted remotely. These restrictions somewhat limited our ability to control the testing environment to ensure a completely non-distractive environment during these parts of the study; however, a significant attempt was made to ensure such a non-distractive environment was genuinely adhered to during that part of the study. Our initial pilot testing results indicate that the findings were counterintuitive for the Phishing Intelligence Quotient (IQ) tests. In contrast, results of the PMSER were intuitive with improved detection on a computer compared to mobile. We conclude with a discussion on the study limitations and further research.


Social engineering, cybersecurity, judgment error in cybersecurity, phishing email mitigation, distracting environments.




Research paper


The Online Journal of Applied Knowledge Management (OJAKM), ISSN: 2325-4688


International Institute for Applied Knowledge Management (IIAKM)


3 March 2022


13 July 2022; 1 August 2022; 13 September 2022


15 September 2022

Accepting Editor:

Meir Russ