Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention

Carlos Gallo, Hilda Pantin, Juan Villamar, Guillermo Prado, Maria Tapia, Mitsunori Ogihara, Gracelyn Cruden, C. Hendricks Brown

Research output: Contribution to journalArticle

13 Scopus citations

Abstract

Careful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald and Garland, Psycholog Assess 25:146–156, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on “joining,” which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached 0.83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.

Original languageEnglish (US)
Pages (from-to)574-585
Number of pages12
JournalAdministration and Policy in Mental Health and Mental Health Services Research
Volume42
Issue number5
DOIs
StatePublished - Sep 22 2015

Keywords

  • Automated fidelity ratings
  • Computational linguistics
  • Computational methods
  • Fidelity assessment
  • Implementation research
  • Preventive intervention

ASJC Scopus subject areas

  • Phychiatric Mental Health
  • Health Policy
  • Public Health, Environmental and Occupational Health
  • Psychiatry and Mental health

Fingerprint Dive into the research topics of 'Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention'. Together they form a unique fingerprint.

Cite this