Predicting CYP Inhibition: Lessons from the OpenADMET Challenge

Can we predict whether a molecule will inhibit a drug-metabolizing enzyme before its experimental results are revealed? OpenADMET’s CYP inhibition blind challenge tests that question in two ways: predicting inhibition strength and identifying inhibition that increases with time. [2]

Cytochrome P450 enzymes, or CYPs, help clear many medicines. Inhibiting one can change how another drug is processed, making CYP activity an important property to investigate early. [3] UniBio Intelligence is participating in both inhibition tracks. The interim evaluation gave us a chance to review our approach—and the experiments that did not improve it. [1]

Two questions, two kinds of prediction

Direct inhibition: how strongly does a compound inhibit CYP1A2, CYP2C9, CYP2D6, or CYP3A4? We predicted pIC50, a logarithmic measure of potency where a higher value means stronger inhibition. Time-dependent inhibition (TDI): does inhibition become stronger after preincubation? This track required a yes/no prediction for CYP2D6 and CYP3A4, using the challenge’s assay-based labels. [2][3]

The interim evaluation covers the full 750-compound test set; the live leaderboard evaluates only half. Each track has its own scoring metric. In both names, MA means macro-averaged: calculate a score for each enzyme, then take their simple average. Each enzyme has equal weight in the overall score. [2][12]

How to read the scores and tiers

MA-ST-RAE: macro-averaged soft-threshold relative absolute error. This measures direct-inhibition prediction error while allowing for experimental uncertainty. A prediction inside a compound’s assay-derived credible interval receives zero error; outside it, the error is the distance to the nearest interval boundary. For example, if the interval is pIC50 6.2–6.8, predictions of 6.5 and 7.0 receive errors of 0 and 0.2, respectively. [2]

The total error for each enzyme is divided by the error of a simple reference prediction: assigning every compound that enzyme’s mean measured test-set pIC50, scored against the same intervals. An ST-RAE below 1 improves on that reference; 0 means every prediction falls within its interval. The four enzyme scores are then averaged. Lower is better. [12]

MA-MCC: macro-averaged Matthews correlation coefficient. This evaluates the TDI yes/no calls using all four outcomes: correctly identified inhibitors, correctly identified negatives, false alarms, and missed inhibitors. It is useful when positives are uncommon, because calling nearly everything negative can give deceptively high percentage accuracy. MCC ranges from −1 to 1: 1 means perfect agreement, 0 means no association between predictions and labels, and −1 means completely reversed labels. The CYP2D6 and CYP3A4 MCC values are averaged. Higher is better. [2][16]

Tiers describe statistical separation. Small score differences can reflect which compounds happen to be in the test set. The organizers assessed uncertainty by repeatedly resampling test compounds (bootstrapping), then applied the Benjamini–Hochberg correction across the many model comparisons. Their stated false-discovery-rate threshold was 5%, controlling the expected proportion of false positives among differences declared significant. [1][15]

Starting from the best-scoring entry, Tier 1 includes the consecutive entries whose performance is not significantly different from that tier’s leading model. The first significantly different entry starts Tier 2 and becomes its reference; the process repeats down the table. A shared tier means the procedure did not clearly separate those entries on this test set. It does not prove that their predictions are identical or that their performance will remain equivalent on new compounds. [1][15]

How we built the predictions

Our research pipeline represented each test compound by its chemical structure: the atoms, bonds, and molecular fragments encoded in its SMILES string. We used these descriptions to learn relationships between a compound’s chemistry and its measured inhibition of each enzyme. The starting point was the organizers’ experimental data: dose–response measurements, TDI labels, and a larger single-concentration screen. [4]

• Learn from chemical neighbors. We used molecular fingerprints—compact descriptions of a molecule’s chemical fragments—to find related compounds with measured activity. RDKit supplied the fingerprints and descriptors, including properties such as size and polarity. [5]
• Combine simple and learned representations. We trained LightGBM models on fingerprints and descriptors, alongside Chemprop graph networks that learn from atoms and bonds. Multitask models learned related assay measurements together while allowing for missing labels. The aim was to combine models that capture different aspects of the chemistry. [6][7]
• Use public data with attention to the assay. ChEMBL and published CYP screening data supplied additional signals. [8][9] Assay protocols can differ between laboratories, including the substrate and incubation conditions. We therefore tested ways to use public measurements as supporting information, with separate model outputs or calibration to the challenge data, rather than simply pooling every reported potency value.
• Choose combinations by enzyme. We blended and calibrated regression predictions separately for each CYP. For TDI, we used enzyme-specific classifiers and decision thresholds. Calibration adjusts the numerical scale; changing the threshold changes which compounds receive a positive call.

We built on community methods and resources, including Jeremy’s published TDI work and public predictions. [10] Our local experiments included training versions of his approach and testing a calibrated blend with an additional model. Briford’s public Workbench work informed our data integration, graph-model ensembles, and calibration. [11]

To test performance on unfamiliar chemistry, we kept molecules with a shared chemical core, or closely related structures, together when separating training and validation data. We also compared new components with matched controls: the same held-out molecules and evaluation procedure, with the proposed change isolated as far as possible.

What did not work—and what we learned

Several plausible ideas failed to produce a reliable improvement. The examples below come from our local experiments, not leaderboard comparisons. They describe the particular models and data preparations we tested.

1. A pretrained molecular language model did not beat simpler features. We tested MoLFormer, which learns chemical representations from SMILES strings. [13] We kept the pretrained model fixed and trained a simple regression model on its outputs. Using the same regression method with fingerprints and descriptors worked better overall; the MoLFormer version improved only one of five validation folds. Pretraining alone did not make this representation a better fit for our CYP2D6 task.
2. Predicting one enzyme through another added error. We tried predicting CYP2D6 potency as a predicted CYP1A2 potency plus a learned difference between the enzymes. This seemed a way to share information across related tasks. It worsened performance in all five validation folds compared with a direct CYP2D6 model ensemble using the same total number of trees. For this dataset, the two-step prediction was less reliable than learning the endpoint directly.
3. A loss function closer to the scoring rule did not transfer reliably. We trained a graph model to tolerate predictions inside the assay’s reported uncertainty interval. A promising CYP1A2 improvement on development data reversed on a separate held-out set. Matching the evaluation formula more closely was not enough to improve predictions on new chemical groups.
4. Docking needed to pass a basic check first. Before using predicted protein–compound contacts as model inputs, we tested whether an AutoDock Vina workflow could recover ligand positions in two known CYP2D6 complexes. [14] None of four runs met our predefined 2 Å pose-error limit. We stopped before generating features for the challenge compounds. This identified a problem with our tested preparation and scoring workflow; it did not establish that docking could never help.
5. A useful data audit did not automatically improve the model. We found public records where compounds with the same connectivity but different stereochemical identities had different measured potencies. Preserving those identities in one controlled training comparison did not improve overall prediction error. The audit still mattered: it showed where a simplified chemical identity can hide meaningful experimental differences.

We also tried separating TDI negatives according to how their assay labels were assigned, and adding more publicly measured inactive compounds. Neither produced a reliable improvement in the tested comparisons. Each addition needed to demonstrate a reliable improvement over a simpler control.

The lesson we are taking forward

The difficult part was getting an apparent improvement to survive a change in chemical series or validation set. A model could fit one set of compounds better without learning a relationship that carried over to the next. Checking those transfers, and keeping track of assay differences, helped us decide which ideas deserved further work.

The failed experiments were useful decisions in their own right: they showed when to stop scaling an unreliable docking workflow, when a simpler molecular description was sufficient, and when a data-quality finding deserved attention even without an immediate score gain. That is the value of a blind challenge beyond the final result: it tests both our predictions and our assumptions about how to improve them.