|Year : 2022 | Volume
| Issue : 2 | Page : 257-261
Predicting the 7th day efficacy of acupoint application of Chinese herbs (Xiao Zhong Zhi Tong Tie) in patients with diarrhea – A machine-learning model based on XGBoost algorithm
Song Sheng1, Rui Li2, Xing Wang1, Hong-Yang Gao1, Yan-Hong Zhang1, Feng-Qin Xu3
1 Department of Emergency, Xiyuan Hospital of China Academy of Chinese Medical Sciences, Beijing, China
2 Institute of Clinical Pharmacology, Xiyuan Hospital of China Academy of Chinese Medical Sciences, Beijing, China
3 Institute of Geriatrics, Xiyuan Hospital of China Academy of Chinese Medical Sciences, Beijing, China
|Date of Submission||14-Jun-2021|
|Date of Acceptance||12-Jul-2021|
|Date of Web Publication||16-Sep-2021|
Institute of Geriatrics, Xiyuan Hospital of China Academy of Chinese Medical Sciences, Beijing
Source of Support: None, Conflict of Interest: None
Objective: Extreme gradient boosting (XGBoost) was used to predict the 7th day efficacy of the acupoint application (AP) of Chinese herbs (Xiao Zhong Zhi Tong Tie) in patients with diarrhea. Materials and Methods: We consecutively collected medical records of patients with diarrhea nationwide on the Chun Bo Wan Xiang cloud platform from August 22 to November 5, 2020. Demographic and clinical data and the fecal properties were included in this study. We established the XGBoost model to predict the 7th day efficacy of AP in patients with diarrhea. The XGBoost model was evaluated using the area under the receiver operating characteristic (ROC) curve (AUC). We next compared the performance of XGBoost with that of artificial neural network (ANN), ANN + boosting, ANN + bagging, and support vector machine (SVM). Results: The XGBoost model provided a prediction accuracy of 84.86% (95% confidence interval = 82.74% to 86.81%) and the ROC curve analysis showed an AUC of 0.81. The top-three variables with the highest importance are age, duration of diarrhea, and region (North). Our study revealed that XGBoost was not superior to ANN, ANN + boosting, ANN + bagging, and SVM. Conclusions: The established XGBoost model for predicting the 7th day efficacy of AP in patients with diarrhea exhibited good accuracy and precision, which can be used for efficacy prediction.
Keywords: Acupoint application, diarrhea, efficacy prediction, XGBoost, Xiao Zhong Zhi Tong Tie
|How to cite this article:|
Sheng S, Li R, Wang X, Gao HY, Zhang YH, Xu FQ. Predicting the 7th day efficacy of acupoint application of Chinese herbs (Xiao Zhong Zhi Tong Tie) in patients with diarrhea – A machine-learning model based on XGBoost algorithm. World J Tradit Chin Med 2022;8:257-61
|How to cite this URL:|
Sheng S, Li R, Wang X, Gao HY, Zhang YH, Xu FQ. Predicting the 7th day efficacy of acupoint application of Chinese herbs (Xiao Zhong Zhi Tong Tie) in patients with diarrhea – A machine-learning model based on XGBoost algorithm. World J Tradit Chin Med [serial online] 2022 [cited 2022 Dec 10];8:257-61. Available from: https://www.wjtcm.net/text.asp?2022/8/2/257/326076
| Introduction|| |
There is a long history in China of treating diarrhea with the acupoint application (AP) of Chinese herbs, and previous studies have revealed that AP is safe and effective for the management of diarrhea., Based on clinical experience, there is a significant difference in the therapeutic effect between patients with diarrhea who received the same AP therapy. Therefore, recognition of the benefitting population is crucial for the improvement of the efficacy and precise treatment of diarrhea, and an accurate predictive model is needed to prejudge the efficacy according to the baseline characteristics of the patients. Unfortunately, there is no relevant literature regarding a practical efficacy prediction tool for the diarrheal recovery of patients through AP therapy. For this reason, our study applied a machine-learning model based on extreme gradient boosting (XGBoost) to establish a predictive model of the 7th day efficacy of Xiao Zhong Zhi Tong Tie (XZZTT) in patients with diarrhea and eventually provide aid to the clinical decision making. Next, we compared the established XGBoost model with four other machine-learning approaches, including artificial neural network (ANN), ANN + boosting, ANN + bagging, and support vector machine (SVM) for classification capability. We hope that our study will contribute to this vexing clinical problem and provide new evidence for a model refinement in advance.
| Materials and Methods|| |
Study population and data collection
This was a registered retrospective cohort study, which consecutively collected medical records of patients with diarrhea from hundreds of primary hospitals nationwide on the Chun Bo Wan Xiang cloud platform from August 22 to November 5, 2020. Patients with diarrhea were diagnosed based on the Chinese guidelines for the diagnosis and treatment of diarrheal disease. We excluded patients who met the following criteria: Having only one clinic visit, no use of AP, and samples with missing baseline data. Thus, 1235 patients were included in the study. A flow chart of the study population is presented in [Figure 1]. The AP used in this study was XJZTT (Country Medicine Accurate Character Number: B20020725) manufactured by YABAO Pharmaceutical Group Co., Ltd. The participants were treated once per day. During the treatment, the specific treatment regimen and time course were optional based on the patient's condition and willingness. There are no restrictions regarding the Traditional Chinese Medicine used in AP therapy. There were two types of AP treatment: Wet and medicated applications. In the wet application group, patches were wetted with physic liquor (menthol and Shui Man Qing) and applied to acupoints of the human body. In medicated applications group, herbal formulas prescribed according to TCM syndrome differentiation were ground and prepared as a paste with physic liquor (menthol and Shui Man Qing) prior to application.
Demographic and clinical data including region, age, sex, use of oral Chinese Medicine decoction, use of Western Medicine, duration of diarrhea, stool frequency, and methods of AP were recorded. We also collected stool information including loose stool, watery stool, mucous or bloody stool, acid odor stool smell, and unobvious stool smell. Protocol-specified follow-up was conducted at each outpatient visit using a standardized form to record the study information. The primary outcome was diarrhea recovery on the 7th day after the initiation of AP therapy. The standard of diarrheal recovery, i.e., the frequency and character of stool back to normal status and the disappearance of systemic symptoms, was based on the Chinese Guidelines for the Diagnosis and Treatment of Diarrheal Disease.
All subjects or their parents provided informed consent to participate in the follow-up program and upload their data for this study. Data collection was conducted by previously trained community physicians. Institute of Clinical Pharmacology of Xiyuan Hospital, China Academy of Chinese Medical Sciences, was responsible for data management and verification. This study was reviewed and approved by the ethical committee of Xiyuan Hospital of China Academy of Chinese Medical Sciences (No. 2021XL004-1). No additional informed consent was required for analysis of the anonymized data.,
The software R 4.1.0 (https://www.r-project.org, TheR Foundation) and SPSS modeler 14.1 (https://www.ibm.com/products/spss-modeler, IBM) were used for all statistical analyses. Continuous variables are shown as mean ± standard deviation, and categorical variables are expressed as number (percentage). Comparisons between groups were conducted using Student's t-test, Mann–Whitney U-test, and Chi-square test.
The contributions of the individual XGBoost predictors were measured and ranked based on gain, cover, and frequency, which are proportional to the importance of the predictors. Gain is the improvement in accuracy brought by a feature to the branches it is on. Cover measures the relative number of observations related to a feature. Frequency is a simpler way to measure the gain and counts the number of times a feature is used in all generated trees. The model was evaluated using the area under the receiver operating characteristic curve (AUC). The AUC value ranges from 0-1, and the higher the value, the better the classification capability of the model. The criteria for AUC classification were 0.90-1 (excellent), 0.80-0.90 (good), and 0.70-0.80 (fair). Meanwhile, we constructed ANN, ANN + boosting, ANN + bagging, and SVM models for predicting the 7th day efficacy of AP with the SPSS modeler. A neural network module was used to build the ANN model. We employed a multi-layer perceptron to automatically calculate the number of units, and a maximum training time of 15 min was applied as a stopping rule. We additionally added boosting and bagging options in the ANN + boosting and ANN + bagging models, respectively, on the basis of the parameter settings of the ANN. The expert mode was applied, and all parameters were set as a default in the SVM model. XGBoost, ANN, ANN + boosting, ANN + bagging, and SVM are inherently binary classifiers. For a particular classifier, various standard performances can be assessed based on the confusion matrix for comparing the model prediction performance. Therefore, the accuracy (ACC), precision (P), and F1 score were used to compare the models in terms of the classification performance.
| Results|| |
Basic characteristics of the study population
A total of 1235 participants was included in this study. Of the 1030 patients in the recovered group, 594 (57.67%) were men with a mean age of 6.28 ± 11.97 years. Of the 205 patients in the unrecovered group, 121 (59.02%) were men with a mean age of 7.63 ± 14.20 years. No significant differences were found in region, age, sex, duration of diarrhea, use of oral Chinese Medicine decoction, use of Western Medicine, stool frequency, methods of AP, loose stool, watery stool, mucous or bloody stool, and unobvious stool smell between the two groups (P > 0.05). The proportion of acid odor stool smells in the recovered group was significantly lower than that in the unrecovered group [Table 1].
|Table 1: Baseline characteristics of patients in the recovered group and unrecovered group|
Click here to view
XGBoost model building and evaluation
The parameters were tuned to the optimum performance with the train () function of the Caret package. The optimum parameters are a maximum depth of 6, min child weight of 6.87, 0.95 for the subsample, and 0.75 for the colsample bytree. The model provided a prediction accuracy of 84.86% (95% confidence interval [CI] =82.74% to 86.81%). The discriminative ability of the model was good, with an AUC of 0.81 [Figure 2]. The top-three variables with the highest importance are age, duration of diarrhea, and region (North). The variable importance of the model is shown in [Figure 3].
Comparisons of model performance
As visible in [Figure 4], the highest P is 100.00% in XGBoost. Minor differences in the ACC and F1 scores were observed among the five models. The highest ACC was found in ANN + boosting at 86.78%, and the lowest was found with the ANN at 86.17%. In addition, the highest F1 score was 92.88% (ANN + boosting), and the lowest F1 score was 92.58% (ANN).
| Discussion|| |
We developed a machine-learning model for predicting the 7th day efficacy of AP in patients with diarrhea based on the XGBoost algorithm. The XGBoost model had an AUC of 0.81 and exhibited a high accuracy, precision, and F1 score. Thus, the model can be used to identify the benefitting population of patients with diarrhea receiving AP therapy.
XGBoost, a machine-learning algorithm, is a combination of classification and regression tree and ensemble learning with gradient tree boosting. On the one hand, XGBoost optimizes control of the model complexity and postpruning processing, and uses regularization technology to reduce the model over-fitting and ensure the model robustness. On the other hand, XGBoost supports parallelization and multi-threaded calculations, greatly accelerating the speed of the model calculation. In summary, it is a machine learning algorithm with both robustness and efficiency. XGBoost has the advantages of processing multivariable interactions and nonlinear relationships and tends to provide a higher predictive accuracy. However, the actual classification capability of XGBoost is not universally identical because of the different study designs, data sources, and adjusted parameters., Our study revealed that XGBoost is not superior to the ANN, ANN + boosting, ANN + bagging, and SVM models, which could occur for the following reasons. First, the sample size of this study was limited, because only 1235 cases were included. In addition, only 13 variables were included in the modeling as predictors, and most were binary variables. The relatively small sample size and limited quantity and quality of the variables diluted the power and advantage of the XGBoost algorithm. Second, some strong predictors of diarrheal recovery, such as disease history including diabetes, hyperthyroidism, ulcerative colitis, irritable bowel syndrome, fecal examination, and blood tests were not recorded in this study. Consequently, these variables cannot be used for model building or outcome prediction. Third, this was a retrospective cohort study design. It is inevitable that the data completeness, authenticity, and homogeneity will be inferior to those obtained from prospective studies, which may have affected the reliability of the results. Consequently, the classification and prediction capabilities of XGBoost require further verification in a prospective study.
Finally, we need to consider the feasibility and convenience of the model development and application. As is well known, a traditional predictive model is based on a linear regression. It has the advantage of relatively well-understood statistical principles and a friendly development environment and can be conducted directly using a formula, nomogram, rating scale, dynamic web page calculator (R Shiny package) and other simple methods in clinical applications. By contrast, XGBoost is so complicated that it is termed a “black box,” and it is not possible to interpret the model with a single formula such as a linear regression. For this reason, the above prediction methods cannot be implemented in XGBoost. In addition, it is necessary to modify the parameters and update the model for a performance optimization through the continuous inclusion of new data. Thus, generalization of the established XGBoost model to basic-level hospitals may be inconvenient. To address this question, we need to integrate it into clinical diagnosis and treatment systems and tune the parameters for a performance optimization with the assistance of computer engineers.
| Conclusions|| |
The established XGBoost model for predicting the 7th day efficacy of AP in patients with diarrhea exhibited high accuracy and precision, and can be used for efficacy prediction.
Song Sheng completed the manuscript and conducted the data analysis. Rui Li provided patient data. Rui Li, Hong-Yang Gao and Yan-Hong Zhang provided invaluable advice on the data analysis. Xing Wang and Feng-Qin Xu reviewed the article critically and contributed to revising the language.
Financial support and sponsorship
This study was financially supported by the Fundamental Research Funds for the Central public welfare research institutes (ZZ13-024-4).
Conflicts of interest
There are no conflicts of interest.
| References|| |
Zhang YJ, Jia LR. Clinical study of acupoint application in treating acute diarrhea in infants and young children (Chin). Chin Pediatr Integr Tradit Western Med 2018;10:163-6.
Zhu ZN. Clinical observation on Fuxiegao application in adjuvant treatment of 40 cases of infantile diarrhea with insufficiency of both the spleen and the kidney type (Chin). J Pedia Tradit Chin Med 2019;15:47-9.
Fang HS. Chinese guidelines for the diagnosis and treatment of diarrheal disease (Chin). Chin J Pract Pediatr 1998;13:381-4.
Filion KB, Azoulay L, Platt RW, Dahl M, Dormuth CR, Clemens KK, et al
. A multicenter observational study of incretin-based drugs and heart failure. N Engl J Med 2016;374:1145-54.
Geleris J, Sun Y, Platt J, Zucker J, Baldwin M, Hripcsak G, et al.
Observational study of hydroxychloroquine in hospitalized patients with COVID-19. N Engl J Med 2020;382:2411-8.
Chen TQ, Guestrin C. XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD 2016. San Francisco, California, USA: 2016;785-94.
Robba C, Cardim D, Tajsic T, Pietersen J, Bulman M, Donnelly J, et al.
Ultrasound non-invasive measurement of intracranial pressure in neurointensive care: A prospective observational study. PLoS Med 2017;14:e1002356.
Paydar K, Niakan Kalhori SR, Akbarian M, Sheikhtaheri A. A clinical decision support system for prediction of pregnancy outcome in pregnant women with systemic lupus erythematosus. Int J Med Inform 2017;97:239-46.
He XW, Chen MD, Du CN, Zhao K, Yang MF, Ma QF. A novel model for predicting the outcome of intracerebral hemorrhage: Based on 1186 Patients. J Stroke Cerebrovasc Dis 2020;29:104867.
Guan X, Zhang B, Fu M, Li M, Yuan X, Zhu Y, et al.
Clinical and inflammatory features based machine learning model for fatal risk prediction of hospitalized COVID-19 patients: Results from a retrospective cohort study. Ann Med 2021;53:257-66.
Deo RC. Machine learning in medicine. Circulation 2015;132:1920-30.
Hou N, Li M, He L, Xie B, Wang L, Zhang R, et al.
Predicting 30-days mortality for MIMIC-III patients with sepsis-3: A machine learning approach using XGboost. J Transl Med 2020;18:462.
Gu HQ, Wang CJ, Li ZX, Wang YL, Wang YJ, Jiang Y. Comparison of prediction models for in-hospital stroke recurrence in patients with ischemic stroke based on logistic regression and XGBoost methods (Chin). Chin J Stroke 2020;15:587-94.
Frías Ordoñez JS, Otero Regino W. Chronic diarrhea in the diabetic. A review of the literature. Rev Gastroenterol Peru 2016;36:340-9.
Ebert EC. The thyroid and the gut. J Clin Gastroenterol 2010;44:402-6.
Ungaro R, Mehandru S, Allen PB, Peyrin-Biroulet L, Colombel JF. Ulcerative colitis. Lancet 2017;389:1756-70.
Lacy BE, Pimentel M, Brenner DM, Chey WD, Keefer LA, Long MD, et al.
ACG clinical guideline: Management of irritable bowel syndrome. Am J Gastroenterol 2021;116:17-44.
Ó Hartaigh B, Gransar H, Callister T, Shaw LJ, Schulman-Marcus J, Stuijfzand WJ, et al.
Development and validation of a simple-to-use nomogram for predicting 5-, 10-, and 15-year survival in asymptomatic adults undergoing coronary artery calcium scoring. JACC Cardiovasc Imaging 2018;11:450-8.
Stynes S, Konstantinou K, Ogollah R, Hay EM, Dunn KM. Clinical diagnostic model for sciatica developed in primary care patients with low back-related leg pain. PLoS One 2018;13:e0191852.
Chen X, Pang Z, Wang Y, Bie F, Zeng Y, Wang G, et al.
The role of surgery for atypical bronchopulmonary carcinoid tumor: Development and validation of a model based on Surveillance, Epidemiology, and End Results (SEER) database. Lung Cancer 2020;139:94-102.
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 1], [Figure 2], [Figure 3], [Figure 4]
[Table 1], [Table 1]