Online Social Media Analytics Software as a Tool for Automating Data Collection: Concurrent Validity and Feasibility Study

Abstract

Background: Facebook based research is emerging and social media analytics software may be a tool that could lower the cost of research. Unfortunately, the quality of data it extracts has not be documented or validated.

Objectives: To test accessibility and efficiency of social media analytics software in a Facebook based study and to test concurrent validity of Likes extracted.

Methods: We conducted a review of accessible online social media analytics software and selected one for a case study comparing it to manual extraction procedures. Thereafter we tested concurrent validity of the social media analyzer as a method for extracting Likes from Facebook pages. The agreement in Likes extracted was tested with intraclass correlation coefficient (ICC), concordance correlation coefficient (CCC), and Bland and Altman plot.

Results: Eighteen software were found with five being completely free. The selected software was used in the completion of a case study at no cost but took a longer time to extract data compared to manual extraction procedures. Exact data points were matched in only a few pages (n=20, 33.9%) but differences between Likes extracted by the software and manual extraction was not statistically different (p=0.471). The software was found to have perfect ICC for half of the studies with the rest having “almost perfect” agreement (ICC = 0.97 and ICC = 0.98, for 3^rd and 4^th quartile, respectively). Concurrent validity was high (CCC = 0.995) with Bland and Altman plot showing only 5% of measurements outside 95% agreement level.

Conclusion: Social media analyzer software are accessible and can be used at no cost. Facebook Likes extracted through software compared to Likes manually extracted may not be exact matches but have strong agreement and validity.