Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention

Carlos Gallo, Hilda Pantin, Juan Villamar, Guillermo J Prado, Maria Tapia, Mitsunori Ogihara, Gracelyn Cruden, C. Hendricks Brown

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

Careful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald and Garland, Psycholog Assess 25:146-156, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on "joining," which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached 0.83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.

Original languageEnglish (US)
Pages (from-to)574-585
Number of pages12
JournalAdministration and policy in mental health
Volume42
Issue number5
DOIs
StatePublished - Sep 1 2015

Fingerprint

Linguistics
Risk-Taking
Hispanic Americans
Sexual Behavior
HIV
Organizations
Costs and Cost Analysis
Research

ASJC Scopus subject areas

  • Phychiatric Mental Health
  • Health Policy
  • Public Health, Environmental and Occupational Health
  • Psychiatry and Mental health

Cite this

Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment : Experience with the Familias Unidas Preventive Intervention. / Gallo, Carlos; Pantin, Hilda; Villamar, Juan; Prado, Guillermo J; Tapia, Maria; Ogihara, Mitsunori; Cruden, Gracelyn; Brown, C. Hendricks.

In: Administration and policy in mental health, Vol. 42, No. 5, 01.09.2015, p. 574-585.

Research output: Contribution to journalArticle

@article{12817f0119094fc79f150065e585c1ea,
title = "Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention",
abstract = "Careful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald and Garland, Psycholog Assess 25:146-156, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on {"}joining,{"} which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached 0.83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.",
author = "Carlos Gallo and Hilda Pantin and Juan Villamar and Prado, {Guillermo J} and Maria Tapia and Mitsunori Ogihara and Gracelyn Cruden and Brown, {C. Hendricks}",
year = "2015",
month = "9",
day = "1",
doi = "10.1007/s10488-014-0538-4",
language = "English (US)",
volume = "42",
pages = "574--585",
journal = "Administration and Policy in Mental Health and Mental Health Services Research",
issn = "0894-587X",
publisher = "Springer New York",
number = "5",

}

TY - JOUR

T1 - Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment

T2 - Experience with the Familias Unidas Preventive Intervention

AU - Gallo, Carlos

AU - Pantin, Hilda

AU - Villamar, Juan

AU - Prado, Guillermo J

AU - Tapia, Maria

AU - Ogihara, Mitsunori

AU - Cruden, Gracelyn

AU - Brown, C. Hendricks

PY - 2015/9/1

Y1 - 2015/9/1

N2 - Careful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald and Garland, Psycholog Assess 25:146-156, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on "joining," which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached 0.83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.

AB - Careful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald and Garland, Psycholog Assess 25:146-156, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on "joining," which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached 0.83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.

UR - http://www.scopus.com/inward/record.url?scp=85016399487&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85016399487&partnerID=8YFLogxK

U2 - 10.1007/s10488-014-0538-4

DO - 10.1007/s10488-014-0538-4

M3 - Article

C2 - 24500022

AN - SCOPUS:84939563091

VL - 42

SP - 574

EP - 585

JO - Administration and Policy in Mental Health and Mental Health Services Research

JF - Administration and Policy in Mental Health and Mental Health Services Research

SN - 0894-587X

IS - 5

ER -