Development and Validation of the Coronary Heart Disease Scale Among the System of Quality of Life Instruments for Chronic Diseases QLICD-CHD (V2.0) Based on Classical Test Theory and Generalizability Theory

Introduction

Coronary heart disease (CHD) is a chronic condition characterized by myocardial dysfunction and/or organic lesions caused by narrowing of the coronary arteries and insufficient blood supply.¹ According to the latest statistics, CHD is the leading single cause of mortality and Disability Adjusted Life Year (DALY) lost worldwide. A large proportion of this burden occurs in low- and middle-income countries. It is responsible for nearly 7 million deaths and 129 million DALYs annually and represents a significant global economic burden.² In recent decades, with the improvement of people’s living standards, the accelerated pace of life and the aggravation of social pressure, the morbidity and mortality of CHD in China have shown a rapidly increasing trend.³ At present, CHD is the fastest rising cause of death among Chinese residents and is the disease with the highest rate of death and disability in China.^4,5 The pathogenesis of CHD is complex and difficult to cure, patients often need to endure long-term pain and need to be strictly supervised by doctors, patients are usually difficult to have a high quality of life(QoL), so to improve the QoL of patients with CHD has also become the primary goal of researchers and doctors.^6,7

Tools for assessing health needs and QoL are increasingly being developed to assess specific areas and levels of need within defined time frames as a means of improving the QoL and programs of care for patients, particularly in the area of chronic diseases such as cancer^8,9 and cardiovascular disease.¹⁰

Quality of life in patients with coronary artery disease is usually assessed using generic and specific scales. The main generic scales are SF-36, sickness impact profile (SIP), Nottingham health profile (NHP), the EuroQoL-5 dimensions (EQ-5D),¹¹ World Health Organization Quality of Life −100 scale (WHOQOL-100), WHOQOL-BREF, and so on. Generic scales only take into account certain symptoms of chronic diseases and are unable to comprehensively assess the impact of a specific disease on an individual, while specific scales are more sensitive and precise for a particular disease or symptom, and have a higher accuracy in the assessment of QoL. The main specific scales currently applied to CHD include: Seattle Angina Questionnaire(SAQ),¹² Myocardial Infarction Dimensional Assessment Scale(MIDAS),¹³ Quality of Life after Myocardial Infarction(QLMI),¹⁴ Angina Pectoris Quality of Life Questionnaire(APQLQ), Chronic Heart Failure Questionnaire(CHQ),¹⁵ Coronary Revascularization Outcome Questionnaire(CROQ), Cardiac Health Profile(CHP), Cardiovascular Limitations and Symptoms Profile(CLASP), etc.¹⁶

Due to the influence of differences in culture and customs on the sensitivity and specificity of QoL scales, Wan et al¹⁷ developed the first version of the QoL Instruments for Chronic Disease (QLICD) based on China’s national conditions, including the QoL Instrument for CHD (QLICD-CHD) (V1.0),¹⁷ which has been put into wide use. However, the QLICD-CHD (V1.0) has also revealed some problems during its subsequent use. With the improvement of medical care, the items in the specific module are gradually not applicable to the primary needs of patients with CHD at this stage, and some of the items show low responsiveness. Secondly, the structure of the scale needs to be adjusted, and the descriptions of the items should be streamlined as much as possible, and the number of items should be reduced as much as possible while ensuring sufficient information, so as to improve the efficiency of the survey and the response rate of the patients.

In response to the limitations of version 1.0, Professor Wan’s team began to develop version 2.0 of the QLICD in 2008 based on the modular approach, which is the combination of the general module (QLICD-GM) and specific modules.¹⁸ Up to now, 34 common chronic disease QoL scales under this system have been developed including QLICD-CHD (V2.0). The purpose of this study was to develop and validate CHD-specific scale QLICD-CHD (V2.0) based on combination of classical test theory(CTT) and generalizability theory(GT).

Methods

Patients

The study was based on inpatients with the clinical diagnosis of CHD and pathological examination diagnosis at the Affiliated Hospital of Guangdong Medical University. The inclusion and exclusion criteria were as following:

Inclusion criteria: (1) Patients who meet the latest diagnostic criteria for CHD and have a clear diagnosis; (2) Patients with primary school education or above, with good reading and comprehension skills and able to fill in the questionnaire on their own.

Exclusion criteria: (1) Patients who are critically ill, combined with other serious diseases, severe mental illnesses such as schizophrenia, mental retardation, etc.; (2) Cognitive dysfunction; (3) Illiteracy; (4) Those who refused to participate in the study or those who were less cooperative.

Development of the QLICD-CHD (V2.0)

QLICD-CHD (V2.0) was further developed on the basis of QLICD (V1.0) in strict accordance with the programmed mode.¹⁹ 2.0 version still adopts the combination of general module QLICD-GM (V2.0) and the specific module, and the development process has been supervised and discussed by the experts in the whole process. The working group forthe scale study is composed of experts and scholars from multiple disciplines, including experts in chronic diseases of the elderly, clinical experts in hypertension or CHD, healthcare professionals with rich clinical experience, public health experts, statisticians, psychologists, and scholars in multidimensional health assessment research, etc. Experts and scholars from different fields formed a nominal group and a focus group. Members of the nominal group were mainly responsible for reviewing relevant literature at home and abroad, referring to relevant mature scales at home and abroad, summarizing clinical experience, proposing items to form a pool of items, and modifying items; the focus group was mainly responsible for organizing the core work, such as proposing, discussing, screening and modifying the scale items. The new version of QoL for patients with CHD was designed to better meet China’s national conditions. The main steps are as follows: (1) setting up a scale development team, proposing scale items to form a pool of items; (2) screening and determining the items; (3) pre-test and screening of items; (4) test survey and re-screening of items; and (5) evaluation of the scale. The development process is shown in Figure 1.

Figure 1 Stages in the development and validation of QLICD-CHD (V2.0).

The final version of the QLICD-CHD (V2.0) consists of the General Module QLICD-GM and a CHD-Specific Module with 14 items. The QLICD-GM consists of 28 items on 9 facets in 3 domains, namely Physical Functioning (9 items), Psychological Functioning (11 items), and Social Functioning (8 items). The specific module includes 14 items on 3 facets of heart failure symptoms (2 items), chest and abdominal pain (7 items), and special psychological effects on life (5 items). The entire scale consists of 42 items, each of which is a five-level Likert hierarchical item.

Validation of the QLICD-CHD (V2.0)

Survey Methodology

The investigators (doctors, nurses and postgraduate medical students) were first required to explain the content and purpose of the survey to the patients, and after obtaining the patients’ consent and signing the informed consent form, the investigators sent the QLICP-CHD (V2.0) to the patients to fill out on their own. The first questionnaire was administered on the first day after admission, a retest survey using the same questionnaire was administered on days 2 after admission to assess test-retest reliability, and a third survey was administered before discharge to assess responsiveness.

Scoring Methodology

Each item is scored using the Likert format (not at all, a little bit, somewhat, quite a bit and very much), and positive items will be scored directly from 1 to 5 points, while negative items will be scored the opposite way. Domain/facet and overall scores are obtained by adding the relevant item scores, both converted linearly to standardized scores on a scale of 0–100.QLICP-CHD (V2.0) scores before and after conversion higher means better QoL for the patient.

Validity Analysis

Validity was analyzed from the perspectives of content validity, construct validity and Criterion-related validity. The assessment of construct validity in classical test theory mainly includes item-domain correlation coefficients and factor analysis, and the Criterion-related validity of a scale is a measure of the degree of correlation between the scale and other extrinsic criteria, which is mainly assessed by using the correlation coefficient between the two scales. In this study, SF-36 and QLICD-CHD (V2.0) domain scores were selected for correlation analysis comparison. The SF-36 is a validated generic instrument incorporating 8 domains and has been widely used as a QoL measure in CHD patients.^20–22 When the coefficient of the correlation domain of the two scales is greater than 0.4, it proves that the scale has good convergent validity, and when the coefficient of the correlation domain is higher than the coefficient of the non-correlation domain, it proves that the scale has good discriminant validity.²³

Confirmatory factor analysis (CFA) was performed on the scales using structural equation modeling. Comparative fit index (CFI), Tucker-Lewis index (TLI), root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR) indicators were used to assess how well the data fit the structure of the scales. CFI and TLI>0.9, RMSEA and SRMR<0.08 represent a good model fit, the scale dimensions are set reasonably well, and there is satisfactory construct validity.²⁴

Reliability Analysis

The reliability analyses were conducted on the internal consistency reliability and test-retest reliability of the scales. Cronbach’s alpha coefficient was used to determine internal consistency reliability, and Pearson’s correlation coefficient and intraclass correlation coefficient (ICC) were used to determine test-retest reliability. When the Cronbach’s alpha coefficient of the items was between 0.70 and 0.95 it indicated a good internal consistency reliability.²⁵ A test-retest reliability >0.8 is considered ideal.²⁶

Responsiveness Analysis

Responsiveness is the ability of a scale to measure changes in QoL over longitudinal time, ie, changes in a patient’s QoL following the influence of external factors.²⁷ In this study, the mean scores of each domain/facet were calculated before and after treatment (first and third assessment). Responsiveness was assessed using a paired t-test and the standardized responsiveness mean (SRM) was calculated.^28–30 An SRM of approximately 0.20 is generally considered to indicate low responsiveness, 0.50 to 0.80 is considered good responsiveness, and greater than 0.80 indicates excellent responsiveness.³¹

GT Analysis

GT uses analysis of variance to decompose errors and control the error variance in order to improve reliability, thus helping researchers to understand the relationship between scale reliability and error more clearly.³² GT consists of two processes: the G-study and the D-study. The G-study, also known as the generalizability study, has the main task of identifying, as far as possible, the various potential sources of measurement error in the study design over the full range of observations and estimating the variance components of these sources of error. The D-study, also known as the decision research, has the main task of exploring how to control and regulate the measurement error by adjusting the various relationships in the measurement process based on the G-study, whose indicators are the generalizability coefficient (Eρ²) and reliability index (Φ).³³

The results of the G-study of this scale showed that the main sources of variability in scale scores were patients and patient-item interactions. The purpose of this scale is to determine the health status of the patient, so the source of variation in the scale is more reasonable. D-study gave the generalizability coefficient and reliability index at the current number of items, and after fixing the subjects, it gave the suggested number of scale items based on the generalizability coefficient and reliability index.¹⁹

Data Analysis Software

Reliability, validity and responsiveness were calculated using SPSS 26.0, structural equation modeling was performed using Mplus, and GT analysis was performed using mGENOVA.

Results

Socio-Demographic Characteristics of the Sample

A total of 189 patients diagnosed with CHD, aged 22–88 years, with a mean age of 66.9 years, of whom 118 were male accounting for 64.2% and 71 were female accounting for 37.6%, and other specific demographic data are shown in Table 1. All patients were first filled out on admission and second filled out before discharge, which included general patient information, QLICD- CHD (V2.0) scale and the SF-36 scale. Patients filled out the questionnaire by themselves with the help of the investigator, and the average time to complete the questionnaire was 20 minutes, which ensured a good quality of questionnaire response.

Table 1 Sociodemographic Characteristics of the Participants (N=189)

Validity

Content validity

The structure of the group includes teachers with specialization in medical statistics and epidemiology, medical psychology, as well as clinicians and researchers who have been engaged in clinical work for many years. After repeated discussions among experts, QLICD-CHD (V2.0) was prepared according to a strict procedural approach. The specific module for CHD has three facets, including heart failure symptoms, chest and abdominal pain, and the impact of the specific psychological life of the disease. It is possible to adequately cover the keys that affect the QoL in patients with CHD in terms of content, thus ensuring good content validity.

Construct Validity

Item domain Pearson correlation coefficients and factor analysis were used to evaluate construct validity. Correlation analysis of the data measured at admission showed strong correlations between the items and the domains (mostly above 0.40). However, the relationship between the item and the other domains was weak (refer to Table 2 for details). For example, the correlation coefficients between SOD and GSO1-GSO8 ranged from 0.49 to 0.74 (bolded part), which was higher than the correlation coefficients between SOD and other items.

Table 2 Correlation Coefficients r Among Items and Domains of QLICD-CHD (V2.0) (n=189)

CFA was performed on the specific module. By the modification index (MI) indication, CHD13 and CHD14 had higher MI values, the reason for this result could be “Do limitations in the amount or rate of physical activity affect your life or work much?” (CHD13) and “Do you go out less or even can’t go out because of your illness?” (CHD14) are more similar and may have multicollinearity. Therefore, we made one modification to the model and obtained the good results: Chi-square χ2= 151.944 (P < 0.001), TLI =0.899, CFI =0.919, RMSEA=0.076, SRMR=0.078. The detailed results are shown in Table 3 and Figure 2.

Table 3 Structure of the Specific Module of the QLICD-CHD V2.0 Confirmed by SEM (n=189)

Figure 2 The structure of the specific module of QLICD-CHD (V2.0) by structural equation modeling.

Criterion-Related Validity

Table 4 shows Pearson’s correlation coefficients of the domain scores of the QLICD-CHD V2.0 with the SF-36. Results indicated positive correlations. The correlation coefficients between the same domains or similar domains are significantly higher than the coefficient values between unrelated domains in the same column, and most of the coefficient values are greater than 0.4, indicating that QLICD-CHD (V2.0) has better convergence validity and discriminant validity.

Table 4 Correlation Coefficients Among Domains Scores of QLICD-CHD (V2.0) and SF-36 (n=189)

Reliability

From the results, it can be seen that the scale has high internal consistency reliability overall, with Cronbach’s alpha coefficient >0.9 for all domains and Cronbach’s value between 0.82–0.84 for each side. The test-retest reliability of each domain ranged from 0.53–0.73, and the test-retest reliability of the total scale was 0.73, which basically met the requirement of test-retest reliability >0.7. The possible reason for this result was that the secondary measurement data of some patients were not collected. Table 5 shows a record of the details of Cronbach’s a and ICC.

Table 5 Reliability of the Quality of Life Instrument QLICD-CHD (V2.0) (n=189 for α, n=157 for r, ICC)

Responsiveness

Paired t-tests and the response index SRM were used to examine the change in mean scores for each domain/facet of QLICD-CHD (V2.0) before and after treatment, and the results are shown in Table 6. It can be seen that the pre- and post-treatment scores for the total scale and each domain were statistically significantly different (P<0.001), with only one facet, Independence, showing no statistically significant difference in pre- and post-scores. The SRM ranged from 0.09 to 0.51 and the domain level SRM ranged from 0.27 to 0.50.

Table 6 Responsiveness of the Quality of Life Instrument QLICD-CHD (V2.0) (n=189)

Results from GT

G-Study Results

Each variance component for patients represents the estimated “true score” variance estimated by the patient on a particular domain of the scale. The variance components for the person-item interaction were 0.95, 0.92, 0.79, and 1.27 for the PHD, PSD, SOD, and SPD domains, respectively. The variance components for the five domains were between 0.26 and 0.39, and the variance components for the items were between 0.08 and 0.32, indicating that the largest source of variation in a domain score was from the person-item interaction. This information indicates that, relatively speaking, the QoL of patients with CHD had the greatest variation in the PSD domain and the least variation in the SPD domain. The correlation coefficients between the four domains fluctuated at the level of 0.65, which is within an acceptable range. These results further confirm the appropriateness of using GT to evaluate the reliability of QLICD-CHD (V2.0) scale. The specific results are shown in Table 7.

Table 7 Estimation of Variance Components in Various Domains in the P×i -Designed G-Study(n = 189)

D-Study Results

D-Study helped us to find the optimal number of items for the scale domains. As shown in Table 8, both the probability coefficient and the reliability index of the scale reach a more desirable level when the numb er of entries for each domain reaches the target number, and the subsequent increase in the probability coefficient and reliability index slows down as the number of items increases.

Table 8 P×i - Designed D-Study Results of the Various Domains of QLICP-CHD (V2.0)

Discussions

This paper reports on the development and validation process of a scale for the measurement of QoL in patients with coronary artery disease. The development of this scale was carried out using a set of procedural, modularized approaches that combine general and disease-specific modules to ensure that the investigator has a comprehensive picture of the changes in the QoL of the patient.^17,34 Such a development model has a number of advantages.^17,23,35 First, general module applied to different diseases allow for concise comparisons of differences between the same domains of different diseases. Second, the specific modules are highly sensitive to specific diseases, increasing the response rate of patients when completing the questionnaire and thus improving the accuracy of the survey. Third, the scales developed using a combination of general and specific modules together form QLICD system, which was developed according to a rigorous process and supervised by experts. Members of the research team combined the results of the survey and analysis with the relevant clinical characteristics to modify and improve the scale items until the official scale was completed and applied. Fourth, our team provided strict training for both scale distributors and collectors, and instructed each investigator on the correct use of the scale, which ensured the accuracy of the data and facilitated investigators to select the appropriate scale from them. The validation process is also constantly innovated to make the scale rigorous and reliable. QLICD-CHD (V2.0) is a QoL scale for CHD patients only in the QLICD system, which has the above advantages. Compared with other domestic and foreign scales, it is a scale based on Chinese culture and meets Chinese conditions, and the descriptions of the items are in line with the language habits of the Chinese, so that the QoL of Chinese patients can be better reflected.

In this study, we used an integrated and comprehensive approach to validate the properties of the scale. First, we confirmed that the scale had good validity through item-domain correlation analysis, confirmatory factor analysis, and SF-36 as a criterion for calculating criterion-related validity. Cronbach’s α coefficient, test-retest reliability, and ICC were calculated to confirm that the scale had good reliability. The test-retest reliability values were within the acceptable range. In addition, the results of the paired t-test and the calculation of the SRM metrics indicated that the scale exhibited good responsiveness both overall and in the specific modules.

Validating the reliability of the scale using GT made the study more informative and accurate. The GT results of the scale showed that the main source of variation in scale scores was the interaction between patients and items, coincidentally confirming the nature of the scale itself, which is a measure of the QoL of patients. The D-study results showed that the number of items in all domains of the scale possessed high G and Ф coefficients, and the standard of G and Ф coefficients was 0.6. When both of these indicators were greater than 0.6, the scale is reliable.¹⁹ Combining the indicators, QLICD-CHD (V2.0) has good reliability, validity and responsiveness, which is in line with our expectation.

Compared with QLICD-CHD (V1.0),¹⁹ version 2.0 deleted the question on sexuality that Chinese people avoided, ie, “Have you been bothered by sexual problem caused by disease?” (V1.0 CHD14), and deleted some low response items such as “Did you feel trouble about your weight?” (V1.0 CHD11), “Did your disease make you lack of safety? ” (V1.0 CHD13). The number and content of items have been adjusted to make the scales more concise and the questions more accurately described. The Cronbach’s α coefficients of the total scale and the domains/facets are >0.8, which gives a higher reliability than the old version. With the same SF-36 as the standard, the delineation of domains in the QLICD-CHD V2.0 was clearer, and the correlation coefficients within similar domains for PSD, SOD, and SPD were larger than those of version 1.0, which proved that it had better criterion-related validity. In addition, the responsiveness of the second version of the scale was more pronounced, and the differences in before and after treatment scores were statistically significant in all domains. The first version of the scale had six facets with statistically insignificant results, whereas version 2.0 had only one facet, “Independence”, with much greater responsiveness.

This study develops and validates a new QoL scale specific to patients with CHD, which also has some limitations. Although this study can confirm that the QLICD-CHD (V2.0) has ideal reliability, validity and responsiveness, the sample may still be insufficient, and a larger sample should be further collected for validation. Meanwhile, the scale is a self-assessment scale, patients need to have a certain level of comprehension and literacy, and a large proportion of patients with CHD are elderly, so the selection of the target population may not be sufficiently comprehensive coverage, which limits the popularization and application of the scale to a certain extent. At the scale development stage, patients were not screened for psychological states such as depression or anxiety, and in future studies we are considering the simultaneous application of depression, anxiety, and other related scales to carry out research on the factors affecting the quality of life of patients with coronary artery disease.

Conclusion

QLICD-CHD (V2.0) is a QoL scale for patients with coronary heart disease based on a combination of general and specific modules in a Chinese cultural context. After rigorous development and validation, QLICD-CHD (V2.0) has good validity, reliability and responsiveness, which is better than version 1.0, and can be widely used to measure QoL in Chinese patients with coronary heart disease.