From chemical compounds to herbal interventions: a transfer learning-based perturbational transcriptome prediction framework for cancer drug discovery

Framework of the transfer learning-based perturbational transcriptome prediction method for herbal agents. The proposed framework adopts an encoder-decoder architecture integrated with a self-attention mechanism as the core structure and jointly incorporates perturbational transcriptomic data from chemical compounds and herbal agents to predict transcriptional responses. Compound and herbal features are extracted by a perturbation encoder, while cell line characteristics are captured by a cell line encoder. Experimental conditions, including dosage and treatment duration, are incorporated as auxiliary inputs. Predicted transcriptomic responses to herbal perturbations are generated by a perturbation decoder. Model performance is evaluated across four modules: baseline model comparison; assessment of transfer learning effects; out-of-distribution generalization analysis; and case studies. Figure created using BioRender (www.biorender.com).

Performance of the compound perturbation prediction model. (A) Loss curves during training and validation. (B) Scatter plot of predicted vs. observed transcriptomic values. Figure created using Python 3.10.14.

Performance of the herbal perturbation prediction model. (A) Loss curves during training and validation. (B) Scatter plot of predicted vs. observed transcriptomic values. Figure created using Python 3.10.14.

Performance comparison of herbal perturbational transcriptome prediction models. (A) Baseline model performance comparison. Model denotes the proposed method, LR denotes the logistic regression model, RF denotes the random forest regression model, and SVR denotes the support vector regression model. Each model was evaluated over 20 repeated experiments. (B) Transfer learning performance comparison. Transfer learning was applied using different proportions of herbal datasets, with each experiment repeated 20 times. LR, logistic regression; MSE, mean squared error; RF, random forest regression; SVR, support vector regression. Figure created using Python 3.10.14.

Out-of-distribution (OOD) generalization performance of the herbal perturbational transcriptome prediction model. (A) Performance metrics from batch-wise testing on unseen herbal agents, repeated 100 times. (B) Performance metrics from batch-wise testing on unseen cell lines, repeated 100 times. (C) Performance metrics from leave-one-herb-out testing, repeated 20 times. (D) Performance metrics from leave-one-cell-line-out testing, repeated 20 times. (E) t-SNE dimensionality reduction and clustering plots of the herbal perturbational transcriptome, annotated by herbal category, dose, and cell line category, respectively. MSE, mean squared error; OOD, out-of-distribution; t-SNE, t-distributed stochastic neighbor embedding. Figure created using Python 3.10.14.