Electronic versus paper diaries: a pilot study of concordance and adherence in head and neck cancer patients receiving radiation therapy
© Blondin et al; licensee BioMed Central Ltd. 2010
Received: 15 September 2010
Accepted: 18 October 2010
Published: 18 October 2010
Interactive Voice Response Systems (IVRS) and other electronic data collection methods have begun to replace conventional paper diaries as a way to capture daily patient reports. However, these methods have not been compared in head and neck (H&N) cancer patients receiving radiation therapy.
15 subjects with H&N cancer were asked to complete daily IVRS calls and daily paper diaries during radiation therapy. We compared response consistency and comparative adherence across the two methods.
86.1% (1920/2231) of participants' responses were consistent between IVRS and paper diaries. 79.5% of the expected number of paper diaries were completed, compared to 66.2% of IVRS phone calls.
In this pilot study of H&N cancer patients undergoing radiation therapy, concordance was high between responses recorded by paper diaries and IVRS. Although adherence appeared to be higher for the paper diaries, it is possible that they may not have been completed at the proper time.
Electronic data collection methods such as Interactive Voice Response Systems (IVRS) are becoming increasingly popular in clinical research . IVRS allow clinical research participants to answer automated survey questions over the telephone, either by pressing the telephone keypad or by issuing a voice response, where the responses are validated and stored in a database for later retrieval and analysis.
Advantages of IVRS include: consistency of survey administration, real time or close to real time data collection and storage, automated reminder calls, and the ability to restrict the time window during which the IVRS survey is available to study participants . However, IVRS requires training and surveys may initially take longer to complete than corresponding paper diaries. Furthermore, there may be an additional cost associated with creating and maintaining an IVRS.
Due to the increasing popularity of IVRS, it has become important to know how IVRS and conventional paper diaries compare with respect to concordance of the collected data and adherence to each data collection method. Lauristen and colleagues compared completion rates of paper, IVRS, and telephone, diaries in which subjects were asked to record symptoms of gastroesophageal reflux disease twice daily. They found that the completion rate for paper diaries was significantly higher than IVRS or the telephone diary . However, there was a higher autocorrelation of consecutive entries in paper diaries, indicating possible backfilling, and as a result they concluded that although paper diaries seem to yield a higher adherence rate, electronic diaries may improve the quality of data gathered. On the other hand, Weiler et al. compared data collected using both paper and IVRS from 87 adults with allergic rhinitis and found that the data obtained by these two methods were indistinguishable from one another . Lundy and Coons demonstrated the measurement equivalence of IVRS and paper forms of a widely used instrument -the EuroQol-5D- for measuring health outcomes . Other studies have established equivalence for paper and electronic versions of instruments such as the Changes of Sexual Functioning Questionnaire (CSFQ) ; and self-report diaries in a population of drug-addicted and alcohol-dependent individuals . A study of orthopedic clinic patients who completed the Short Musculoskelatal Function Assessment (SMFA) using both paper and IVRS found no significant differences in individual subject responses between the two methods . However, the IVRS completion rate was significantly less than the paper completion rate.
After reviewing the available literature, Tennen et al proposed that paper diaries can be effective in situations where behaviors being studied are discrete, when same-day relationships are informative, if a lagged effect across days is expected, or if the end of day recall experience associated with paper diaries is desirable. On the other hand, electronic diaries are preferable when examining within-day temporal dynamics and when evaluating rapidly changing temporal phenomena. They also pointed out that the characteristics of the study population can influence the acceptability of electronic diaries, such as in older populations .
In this context, patients undergoing radiation therapy for head and neck cancer represent a unique population with significant morbidity associated not only with the underlying disease but also with the treatment (radiation therapy). There has been a large increase in clinical research in this population, addressed to treatment of the cancer and the side-effects of radiation therapy, notably mucositis [9–11]. In addition, routine clinical care of these patients also involves collection of data to monitor pain, side-effects, etc. It therefore becomes important to know how paper and electronic diaries, including IVRS, compare in this population so that future research studies and clinical data collection can be appropriately designed.
Fifteen head and neck cancer patients, participating in an IRB-approved clinical trial of an anti-inflammatory medication for radiation-induced oral mucositis, completed IVRS and paper diaries daily over a six to seven week treatment period . Written informed consent was obtained from all patients. Subjects were trained on how to complete the paper and IVRS diaries. They were instructed to complete the paper diary anytime during the day before bedtime and the IVRS telephone call before 7 pm. They were not instructed in which order to complete the diaries. Subjects who failed to complete the IVRS call by 7 pm received an automated telephone call reminding them to complete their daily IVRS diary and offered the option to connect them with the IVRS in order to complete the diary at that time. They received verbal reminders to complete the paper diaries when they came in for study visits. There was no financial compensation for completing the diaries. Onsite study visits took place every 2-3 days over this period, during which the subject was examined and the paper diaries were collected.
Concordance rates, between paper diaries and Interactive Voice Response diaries, for individual questions across all subjects
Did you take your study medication today?
Did you experience any side effects today?
Did you take any additional medication today?
What time did you take your study medication?
What time did you experience a side effect?
What was the side effect?
What time did you take the additional medication?
What is the name of the additional medication?
What dosage was the additional medication?
Why did you take the additional medication?
Both the IVRS and paper diaries allowed participants to report more than one side effect and more than one additional medication. Some of the IVRS secondary questions required a spoken response, whereas others required keypad entry. Spoken responses were recorded and later transcribed for analysis.
To evaluate the concordance of responses collected via paper and IVRS diaries, the number of consistent and inconsistent responses for every diary question was calculated using the following pre-defined criteria:.
For the 3 primary questions, which required a "yes/no" response, an identical response across the two methods was considered consistent.
For the question on the time of taking study medication, times within 30 minutes of each other were considered consistent.
For the questions on the description of side-effects and reason for taking additional medications, consistency grade was assigned by reviewers based on comparison of the verbal responses.
For the question requesting the name of the concomitant medication, the data was considered consistent if responses via both methods of data entry contained the brand or generic name of the same drug.
For the question regarding dosage of concomitant medications, exactly the same dosage was considered consistent.
Using these criteria, concordance rates for each question across all subjects, and concordance rates for each subject across all questions were calculated. In addition, adherence with diary completion, for paper as well as IVRS, was calculated using the number of completed diaries as a percentage of the total number expected for each subject.
Concordance rates, between paper diaries and Interactive Voice Response (IVR) diaries, for individual subjects
Percentage of questions to which responses were consistent between paper diaries and IVR.
Percentage of questions to which responses were inconsistent between paper diaries and IVR.
There were 98 days on which the paper diary was completed and IVRS was not. On these 98 days, the total number of concomitant pain medication doses used was 40 (including the same medications taken on multiple days and multiple medications taken on the same day). There were 115 days on which IVRS was completed but not the paper diary. On these 115 days, the total number of concomitant pain medication doses used was only 2.
We found a high level of day-to-day concordance between daily patient reports entered via IVRS and those entered via a paper record. This high level of concordance was obtained despite the fact that the specific times at which each diary was completed may have varied daily and some paper diaries may have been completed one or two days after the target day (as paper diaries were collected every 2-3 days). We caution that this level of concordance may not generalize to diary items measuring momentary emotional states or other features of daily experience that cannot be recalled reliably.
Only two diary items, "Did you take any additional medication today?" and "What dosage was the additional medication?" yielded less than 80% concordance. These are also the responses most likely to vary from day to day. It is also possible that some subjects took an additional medication after completing the IVRS call for the day, but reported the additional medication on that day's paper diary (since the paper diary was accessible to the subject till handed in at the next study visit). Furthermore, subjects who had trouble speaking due to radiation-induced oral ulcerations may have preferred to select a negative response that required pressing the phone keypad but not a subsequent voice response.
Previous studies comparing adherence to IVRS to that for paper diaries have yielded mixed results [2, 13]. In this study, we found greater adherence with the paper diaries. Several factors may explain this finding. The predominantly older head and neck cancer population may not be as adept at or willing to use an automated IVR system . In addition, most patients receiving radiation therapy for head and neck cancer develop severe mouth and/or throat ulcerations (mucositis) that can make it very painful to speak. This may have led to some reluctance to complete the IVRS calls, which in some cases required verbal responses. This explanation is supported by our finding of greater pain medication use (indicating more severe mucositis) on days on which the paper diary was completed but not the electronic diary. Another potential concern is that subjects having difficulty speaking may have preferred to answer "No" to the primary questions so as to avoid answering secondary questions, some of which required spoken responses. Thus, there may have been an unintentional "response cost" associated with a "Yes" response to the primary questions. While it is difficult to control for such behavior, if it did occur, it would argue against use of IVRS with verbal responses in this population. It is note-worthy that our use of IVRS to record verbal responses is unusual and is not a feature of many commercial IVR systems.
A limitation of this study, in addition to our small sample, was that it compared a sophisticated IVR system, including automated reminders and time stamped data entry verification, to a paper diary for which on-time data entry was not verified. Thus, there is a possibility that some paper diaries may have been backfilled a day or two later . However, our high level of day-to-day cross-method concordance, and collection of paper diaries at study visits every 2-3 days, argues against significant backfilling. It is also possible that subjects may have referred to the paper diaries while responding to the IVRS questions. However, this is unlikely since subjects were provided a separate handout with IVRS instructions and access codes, which they needed to refer to while making the IVRS calls.
In conclusion, in this pilot study of head and neck cancer patients receiving radiation therapy, we observed high concordance between paper diaries and IVRS, and lower adherence to IVRS. These findings have implications for the optimal design of clinical and research data collection methods in this complex population. Because other electronic data capture methods, such as personal digital assistants (PDAs) and IVR without verbal responses, would not have required spoken responses, direct comparisons of paper diaries to these electronic diary methods in comparably challenged patient samples could be revealing.
The authors thank Dr. Stephen J. Walsh for help with statistical analyses. This research was supported by NIH/NIDCR grant K23DE016946 awarded to Dr. Lalla, NIH/NIDCR grant T32DE007302 awarded to the UConn School of Dental Medicine and by NIH/NCRR grant M01RR006192 awarded to the UConn General Clinical Research Center.
- Abu-Hasaballah K, James A, Aseltine RH: Lessons and pitfalls of interactive voice response in medical research. Contemp Clin Trials. 2007, 28: 593-602. 10.1016/j.cct.2007.02.007.View ArticlePubMedGoogle Scholar
- Lauritsen K, Innocenti AD, Hendel L, Praest J, Lytje M, Clemmensen-Rotne K, Wiklund I: Symptom recording in a randomized clinical trial: paper diaries vs. electronic or telephone data capture. Controlled Clinical Trials. 2004, 25: 585-597. 10.1016/j.cct.2004.09.001.View ArticlePubMedGoogle Scholar
- Weiler K, Christ AM, Woodworth GG, Weiler RL, Weiler JM: Quality of patient-reported outcome data captured using paper and interactive voice response diaries in an allergic rhinitis study: is electronic data capture really better?. Ann Allergy Asthma Immunol. 2004, 92: 335-339. 10.1016/S1081-1206(10)61571-2.View ArticlePubMedGoogle Scholar
- Lundy Jason J, Coons Stephen J: Testing the Measurement equivalence of interactive voice response (IVR) and paper versions of the EQ-5D. Poster presented at ISPOR 11th Annual European Congress 8-11. 2008, NovemberGoogle Scholar
- Dunn JA, Arakawa R, Greist JH, Clayton AH: Assessing the Onset of Antidepressant-Induced Sexual Dysfunction Using Interactive Voice Response Technology. J Clin Psychiatry. 2007, 68: 525-532. 10.4088/JCP.v68n0406.View ArticlePubMedGoogle Scholar
- Alemi F, Stephens R, Parran T, LLorens S, Bhatt P, Ghadiri A, Eisenstein E: Automated monitoring of outcomes: application to treatment of drug abuse. Med Decis Making. 1994, 14: 180-7. 10.1177/0272989X9401400211.View ArticlePubMedGoogle Scholar
- Agel J, Rockwood T, Mundt J, Greist J, Swiontkowski M: Comparison of Interactive Voice Response and Written Self-Administered Patient Surveys for Clinical Research. Orthopedics. 2001, 24: 1155-1157.PubMedGoogle Scholar
- Tennen H, Affleck G, Coyne JC, Larsen RJ, Delongis A: Paper and plastic in daily diary research: Comment on Green, Rafaeli, Bolger, Shrout, and Reis. Psychol Methods. 2006, 11: 112-118. 10.1037/1082-989X.11.1.112. discussion 123-115View ArticlePubMedGoogle Scholar
- Lalla RV, Peterson DE: Treatment of Mucositis - Including New Medications. The Cancer Journal. 2006, 12: 348-354. 10.1097/00130404-200609000-00004.View ArticlePubMedGoogle Scholar
- Lalla RV, Sonis ST, Peterson DE: Management of Oral Mucositis in Patients Who Have Cancer. Dent Clin North Am. 2008, 52: 61-77. 10.1016/j.cden.2007.10.002.PubMed CentralView ArticlePubMedGoogle Scholar
- Peterson DE, Lalla RV: Oral Mucositis: the New Paradigms. Current Opinion in Oncology. Curr Opin Oncol. 2010, 22: 318-22. 10.1097/CCO.0b013e32833a9fab.View ArticlePubMedGoogle Scholar
- Lalla RV, Pilbeam CC, Walsh SJ, Sonis ST, Kefe DMK, Peterson DE: Role of the Cyclooxygenase Pathway in Chemotherapy-induced Oral Mucositis: A Pilot Study. Supportive Care in Cancer. 2010, 18: 95-103. 10.1007/s00520-009-0635-1.View ArticlePubMedGoogle Scholar
- Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR: Patient compliance with paper and electronic diaries. Controlled Clinical Trials. 2003, 24: 182-199. 10.1016/S0197-2456(02)00320-3.View ArticlePubMedGoogle Scholar
- Bernhardt JM, Strecher VJ, Bishop KR, Potts P, Madison EM, Thorp J: Handheld computer-assisted self interviews: User comfort level and preferences. American Journal of Health Behavior. 2001, 25: 557-563.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.