The Importance of Assessing Clinical Phenomena in Mechanical Turk Research

Kimberly A. Arditte, Demet Çek, Ashley M. Shaw, Kiara R Timpano

Research output: Contribution to journalArticle

87 Citations (Scopus)

Abstract

Amazon.com's Mechanical Turk (MTurk) website provides a data collection platform with quick and inexpensive access to diverse samples. Numerous reports have lauded MTurk as capturing high-quality data with an epidemiological sample that is more representative of the U.S. population than traditional in-person convenience samples (e.g., undergraduate subject pools). This benefit, in combination with the ease and low-cost of data collection, has led to a remarkable increase in studies using MTurk to investigate phenomena across a wide range of psychological disciplines. Multiple reports have now examined the demographic characteristics of MTurk samples. One key gap remains, however, in that relatively little is known about individual differences in clinical symptoms among MTurk participants. This paper discusses the importance of assessing clinical phenomena in MTurk samples and supports its assertions through an empirical investigation of a large sample (N = 1,098) of MTurk participants. Results revealed that MTurk participants endorse clinical symptoms to a substantially greater degree than traditional nonclinical samples. This distinction was most striking for depression and social anxiety symptoms, which were endorsed at levels comparable with individuals with clinically diagnosed mood and anxiety symptoms. Participants' symptoms of physiological anxiety, hoarding, and eating pathology fell within the subclinical range. Overall, the number of individuals exceeding validated clinical cutoffs was between 3 and 19 times the estimated 12-month prevalence rates. Based on the current findings, it is argued that MTurk participants differ from the general population in meaningful ways, and researchers should consider this when referring to this sample as truly representative. (PsycINFO Database Record

Original languageEnglish (US)
JournalPsychological Assessment
DOIs
StateAccepted/In press - Aug 24 2015

Fingerprint

Mechanical Phenomena
Anxiety
Research
Individuality
Population
Eating
Research Personnel
Demography
Depression
Pathology
Psychology
Costs and Cost Analysis

Keywords

  • Mechanical Turk
  • MTurk
  • Prevalence
  • Psychopathology
  • Symptoms

ASJC Scopus subject areas

  • Psychiatry and Mental health
  • Clinical Psychology

Cite this

The Importance of Assessing Clinical Phenomena in Mechanical Turk Research. / Arditte, Kimberly A.; Çek, Demet; Shaw, Ashley M.; Timpano, Kiara R.

In: Psychological Assessment, 24.08.2015.

Research output: Contribution to journalArticle

@article{7ea11835a4e547d79ee69cba3c3fb63e,
title = "The Importance of Assessing Clinical Phenomena in Mechanical Turk Research",
abstract = "Amazon.com's Mechanical Turk (MTurk) website provides a data collection platform with quick and inexpensive access to diverse samples. Numerous reports have lauded MTurk as capturing high-quality data with an epidemiological sample that is more representative of the U.S. population than traditional in-person convenience samples (e.g., undergraduate subject pools). This benefit, in combination with the ease and low-cost of data collection, has led to a remarkable increase in studies using MTurk to investigate phenomena across a wide range of psychological disciplines. Multiple reports have now examined the demographic characteristics of MTurk samples. One key gap remains, however, in that relatively little is known about individual differences in clinical symptoms among MTurk participants. This paper discusses the importance of assessing clinical phenomena in MTurk samples and supports its assertions through an empirical investigation of a large sample (N = 1,098) of MTurk participants. Results revealed that MTurk participants endorse clinical symptoms to a substantially greater degree than traditional nonclinical samples. This distinction was most striking for depression and social anxiety symptoms, which were endorsed at levels comparable with individuals with clinically diagnosed mood and anxiety symptoms. Participants' symptoms of physiological anxiety, hoarding, and eating pathology fell within the subclinical range. Overall, the number of individuals exceeding validated clinical cutoffs was between 3 and 19 times the estimated 12-month prevalence rates. Based on the current findings, it is argued that MTurk participants differ from the general population in meaningful ways, and researchers should consider this when referring to this sample as truly representative. (PsycINFO Database Record",
keywords = "Mechanical Turk, MTurk, Prevalence, Psychopathology, Symptoms",
author = "Arditte, {Kimberly A.} and Demet {\cC}ek and Shaw, {Ashley M.} and Timpano, {Kiara R}",
year = "2015",
month = "8",
day = "24",
doi = "10.1037/pas0000217",
language = "English (US)",
journal = "Psychological Assessment",
issn = "1040-3590",
publisher = "American Psychological Association Inc.",

}

TY - JOUR

T1 - The Importance of Assessing Clinical Phenomena in Mechanical Turk Research

AU - Arditte, Kimberly A.

AU - Çek, Demet

AU - Shaw, Ashley M.

AU - Timpano, Kiara R

PY - 2015/8/24

Y1 - 2015/8/24

N2 - Amazon.com's Mechanical Turk (MTurk) website provides a data collection platform with quick and inexpensive access to diverse samples. Numerous reports have lauded MTurk as capturing high-quality data with an epidemiological sample that is more representative of the U.S. population than traditional in-person convenience samples (e.g., undergraduate subject pools). This benefit, in combination with the ease and low-cost of data collection, has led to a remarkable increase in studies using MTurk to investigate phenomena across a wide range of psychological disciplines. Multiple reports have now examined the demographic characteristics of MTurk samples. One key gap remains, however, in that relatively little is known about individual differences in clinical symptoms among MTurk participants. This paper discusses the importance of assessing clinical phenomena in MTurk samples and supports its assertions through an empirical investigation of a large sample (N = 1,098) of MTurk participants. Results revealed that MTurk participants endorse clinical symptoms to a substantially greater degree than traditional nonclinical samples. This distinction was most striking for depression and social anxiety symptoms, which were endorsed at levels comparable with individuals with clinically diagnosed mood and anxiety symptoms. Participants' symptoms of physiological anxiety, hoarding, and eating pathology fell within the subclinical range. Overall, the number of individuals exceeding validated clinical cutoffs was between 3 and 19 times the estimated 12-month prevalence rates. Based on the current findings, it is argued that MTurk participants differ from the general population in meaningful ways, and researchers should consider this when referring to this sample as truly representative. (PsycINFO Database Record

AB - Amazon.com's Mechanical Turk (MTurk) website provides a data collection platform with quick and inexpensive access to diverse samples. Numerous reports have lauded MTurk as capturing high-quality data with an epidemiological sample that is more representative of the U.S. population than traditional in-person convenience samples (e.g., undergraduate subject pools). This benefit, in combination with the ease and low-cost of data collection, has led to a remarkable increase in studies using MTurk to investigate phenomena across a wide range of psychological disciplines. Multiple reports have now examined the demographic characteristics of MTurk samples. One key gap remains, however, in that relatively little is known about individual differences in clinical symptoms among MTurk participants. This paper discusses the importance of assessing clinical phenomena in MTurk samples and supports its assertions through an empirical investigation of a large sample (N = 1,098) of MTurk participants. Results revealed that MTurk participants endorse clinical symptoms to a substantially greater degree than traditional nonclinical samples. This distinction was most striking for depression and social anxiety symptoms, which were endorsed at levels comparable with individuals with clinically diagnosed mood and anxiety symptoms. Participants' symptoms of physiological anxiety, hoarding, and eating pathology fell within the subclinical range. Overall, the number of individuals exceeding validated clinical cutoffs was between 3 and 19 times the estimated 12-month prevalence rates. Based on the current findings, it is argued that MTurk participants differ from the general population in meaningful ways, and researchers should consider this when referring to this sample as truly representative. (PsycINFO Database Record

KW - Mechanical Turk

KW - MTurk

KW - Prevalence

KW - Psychopathology

KW - Symptoms

UR - http://www.scopus.com/inward/record.url?scp=84939863402&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84939863402&partnerID=8YFLogxK

U2 - 10.1037/pas0000217

DO - 10.1037/pas0000217

M3 - Article

C2 - 26302105

AN - SCOPUS:84939863402

JO - Psychological Assessment

JF - Psychological Assessment

SN - 1040-3590

ER -