Users’ web searching and browsing behavior can be influenced by a number of factors such as features of a system as well as search education. In contrast to the long-lasting attention to the algorithm and interface dimensions of web search, the potential effects of user education on search behavior have been understudied.
In this project, I led a research team composed of quantitative researchers, developers, and undergraduate assistants to run a study to examine the effects of user education from two sources – peer advice and cognitive authority on users’ online interactions with information systems. We also tested if these behavioral effects lasted when the explicit tips were removed. This research has practical implications for the design of context-aware IR systems that help users modify their search behaviors.
We conducted a three-session lab-field-combined study to find out the influences of cognitive authority and peer advice on users’ online information interactions. The procedure is outlined as follows.
Field Session One
Thirty-one undergraduate students completed the study. They signed up online and received the instruction to start the study via email. Each participant started with a field session in which they completed two search tasks (i.e., searching for information online to finish a task) at locations of their choices. We designed a Chrome browser extension to display the tasks and to log participants’ online activities. Two types of events were collected: search events (queries issued by the participant and the search engine(s) used and browsing events (page type (SERP or content page) and clicks). They filled out a questionnaire before and after each task to report their task perceptions (e.g., task difficulty, task complexity). We also asked them to document their experience in an online diary such as reasons behind their search strategies and any barriers they might have encountered. The purpose of the first field session was to collect users’ baseline behaviors before any treatments were given.
After finishing the first field session, each participant came individually to our interaction lab located at Rutgers University to participate in a lab session. They were randomly assigned to one of three conditions described below.
In the lab, participants who were assigned to the cognitive authority group first watched a 10-min video in which a professor in information science introduced tips on how to search on Google while performing two search tasks on a screen (see the upper image). Participants finished two search tasks after watching the video.
Participants in the peer advice group watched a 10-min video in which two students were interviewed individually by an interviewer about how to search on Google (see the lower image). Students in the video performed the same two search tasks as the professor on a computer screen. Participants were able to see their faces on the bottom right corner and the screen on which they were searching and discussing their search strategies. People in both videos introduced exactly the same set of search tips using the same two tasks. After watching the video, participants finished two search tasks.
Participants in the control group did not receive any treatment, and they did not know the existence of a treatment. They directly started the two tasks. We monitored participants’ screen in anther room during the lab session. At the end of each lab session, we conducted a brief semi-structured individual interview to further inquire about their experiences. The interview was a complement to the diary and log data and helped us gain insights into participants’ experiences with each search task and their opinions about the videos.
Field Session Two
In the lab session, we measured participants’ online behaviors immediately after receiving the treatments. We were also interested in if their behavioral changes (if there were any) lasted after the explicit tips were removed. Therefore, we requested them to complete the second field session in which they finished two search tasks at any locations of their choices. The set-up was the same as the first field session except that they had received the treatment.
To measure the effects of peer advice and cognitive authority on user behavior, we extracted six behavior measurements from the raw log data that were directly associated with the search tips introduced in the treatment videos: number of search operators used, number of unique queries, number of unique query terms, number of web pages viewed, mean dwell time on search engine results pages (SERPs), and mean dwell time on web page.
Using Kruskal-Wallis tests, we examined the differences in search behavior, topic knowledge, and perceived task difficulty across groups before participants received any treatments (i.e., field session one). This was to see if their baseline behaviors were similar. We then used Kruskal-Wallis tests and Dunn’s post-hoc pairwise test to examine the between-group behavioral differences in the lab session and the second field session. Other than the main analysis, we also conducted a few other analyses that examined participants’ query successes and failures using ANOVA and multiple regression.
To hear the users’ voice, we qualitatively coded the diary and interview data, particularly the discussion related to participants’ understanding and usage of the tips introduced in the videos.
- Cognitive authority and peer advice operationalized as video-based search advice can bring both immediate and short-term effects on some aspects of users’ web searching and browsing behaviors, such as usage of search operators.
- Peer advice can be an alternative to the didactic approach of teaching in web search education.
- Search advice had broader impacts in online search tasks with an amorphous goal than in tasks with a specific goal.
- Both sources of advice had similar immediate effects, but cognitive authority was more likely to have lasting effects on search behavior.
- This study reveals the promising future of utilizing search advice from experts or peers to improve online search behaviors and outcomes.
Find out more about this research in the following publications:
Liu, J., Wang, Y., Mandal, S., & Shah, C. (2019). Exploring the immediate and short-term effects of peer advice and cognitive authority on Web search behavior. Information Processing & Management, 56(3), 1010-1025. [Journal article][HIB][HCI]
Wang, Y., Liu, J., Mandal, S., & Shah, C. (2018). Persuasion by peer or expert for Web search. In Proceedings of ACM Conference on Computer Supported Cooperative Work (CSCW). November, 2018. [Conference short paper][HIB][HCI]
Wang, Y., Liu, J., Mandal, S., & Shah, C. (2017). Search successes and failures in query segments and search tasks: A field study. In Proceedings of the Association for Information Science and Technology Annual Meeting, 54(1), 436-445. [Conference full paper][HIB][IIR]