Contents

• 1. Motivation
• 2. Method
• 3. Results
• 4. HPO budget ablation
• 5. Interval coverage
• 6. Target transform behaviour
• 7. Discussion
• 8. Limitations
• 9. Related work
• 10. Reproduction

1. Motivation

Evidence published by a tool's own author has a structural weakness: the author selects the dataset, the metric and the comparison. P2Predict's existing case studies are all of this kind. They are reproducible from public data, but the evaluation design is ours.

An external benchmark removes that degree of freedom. TDC defines the train and test partition, defines the metric, fixes the seed protocol, provides the scoring function, and hosts a leaderboard populated by independent submissions, several from groups working exclusively on molecular property prediction. Nothing about the evaluation is under our control except the model we submit.

The question is narrow and is not a question about chemistry. Given a tabular regression engine developed and tuned against procurement data, does it behave correctly when handed a table from a domain it has never encountered: does model selection converge sensibly, do the automatic preprocessing rules fire correctly, and do the calibrated intervals hold?

2. Method

2.1 Tasks

Three regression endpoints from the TDC ADMET group were selected before any results were observed, chosen to span roughly an order of magnitude in dataset size and to use MAE as the reported metric.

Table 1. Task dimensions. Train and valid sizes are for seed 1; TDC repartitions train_val per seed. Feature counts are the RDKit descriptors surviving the filter in 2.2.

2.2 Representation

Each molecule was converted from SMILES to the full set of RDKit 2D descriptors via Descriptors.CalcMolDescriptors, giving 217 raw columns. Descriptors are computed per molecule and independently of any other molecule, so the representation step cannot transfer information across the split boundary.

Columns were then filtered using the training partition only: those missing on more than 10% of training rows, and those with no variance on the training partition, were dropped. Remaining non-finite values were replaced by the training-set median. Between 7 and 10 columns were removed per task. No feature selection was applied beyond this filter; all surviving descriptors were offered to the model. No fingerprints, no graph or learned representation, and no descriptor engineering of any kind were used.

2.3 Protocol

The evaluation follows TDC's published procedure exactly:

1. TDC supplies a fixed scaffold-split partition of each dataset into train_val and test. The split is not chosen by us and cannot be reselected.
2. For each of five seeds, TDC partitions train_val into train and valid.
3. The model is fitted on train only. valid is used exclusively to compute split-conformal calibration residuals and never influences fitting or model selection.
4. Predictions on test are passed to group.evaluate_many(), which computes the leaderboard metric and its across-seed standard deviation.

Fitting on train alone is more conservative than necessary. Submissions are permitted to use all of train_val, so this configuration trains on approximately 12.5% less data than the protocol allows.

2.4 Configuration

P2Predict 1.1.0 was used unmodified through its public Python API. auto_train performs model selection over Ridge, Random Forest and XGBoost using HalvingRandomSearchCV. Both hyperparameter budgets were run for every task. The log-target rule was left on auto. Conformal intervals were requested at 90% nominal coverage. No task-specific configuration was applied to any run.

3. Results

Table 2. All six runs. MAE and SD are the official evaluate_many output over five seeds. R² is the mean across seeds on the test partition. Coverage is the observed range of empirical coverage of the nominal 90% interval, across seeds. Wall clock is total training time for all five seeds.

3.1 Leaderboard position

Published leaderboard entries are reproduced below with our result inserted at its rank. Ranks are as published at the time of writing.

Table 3. Selected entries from the Caco-2 (24 total) and PPBR (20 total) leaderboards, with P2Predict inserted at rank. On Caco-2 the result ties the seventh entry to three decimal places with a smaller across-seed SD (0.004 against 0.011).

Table 4. Lipophilicity leaderboard, selected entries of 21. This is the task on which the engine performs worst relative to the field, and the only one of the three where graph models occupy the top three positions.

3.2 Model selection

XGBoost was selected in 29 of 30 runs. The single exception, seed 2 of the PPBR fast sweep, selected Random Forest and produced the highest test MAE of that sweep (8.325 against a sweep mean of 8.124). Ridge was never selected on any run.

The Ridge result is confounded and should not be read as evidence about the algorithm. RDKit's Ipc descriptor reaches magnitudes on the order of 10³¹ on these datasets, and Ridge's cross-validated R² reached values on the order of −10³² accordingly. The linear path is not defended against extreme-magnitude numeric columns, so Ridge was effectively removed from contention by a scaling pathology rather than by predictive inadequacy. On P2Predict's own procurement case studies the selection is not degenerate: Ridge wins on one, Random Forest on another, XGBoost on the remaining two.

4. HPO budget ablation

Both hyperparameter search budgets were run on all three tasks. The wider search is not reliably worth its cost.

Table 5. Budget ablation. Negative Δ favours the wider search. Compute is the ratio of total wall-clock time across five seeds.

The wider search produced a material improvement on one task of three. On the smallest task it left the mean unchanged while tripling the across-seed standard deviation (0.004 to 0.012), which is consistent with the larger search space overfitting the cross-validation objective when few rows are available to cross-validate against. We note that the obvious interpretation, that a wider search pays off as data grows, is not supported: the largest task showed the smallest benefit.

5. Interval coverage

All runs requested 90% nominal coverage. Split-conformal calibration was computed on the valid partition and applied to test.

Table 6. Empirical coverage of the nominal 90% interval, against calibration set size.

Coverage converges toward nominal as the calibration set grows, and both the width and the location of the band move monotonically with it. At n=91 the empirical quantile of the residual distribution is coarse, and the resulting intervals are conservative and unstable across seeds. At n=420 they sit close to the promise and dip slightly below it.

Coverage below nominal is expected here rather than anomalous. The split-conformal guarantee holds under exchangeability between calibration and test data. A scaffold split deliberately violates that assumption: test molecules are chosen to be structurally dissimilar from training molecules, which is the point of the split. The observed behaviour, conservative where the calibration sample is small and slightly anti-conservative where distribution shift is largest relative to residual scale, is what the theory predicts under these conditions.

This has a direct consequence for the engine's intended use. Procurement datasets are typically 50 to 300 rows, placing the calibration set at 10 to 60 residuals, well inside the regime where intervals are conservative. Reported ranges in that regime should be expected to be wider than strictly necessary rather than narrower, which is the safer direction of error for the decisions they inform.

6. Target transform behaviour

P2Predict applies a log transform to the target when the training distribution is positive and right-skewed beyond a threshold, and otherwise does not. The rule ran on auto for every task and declined the transform on all three, for two distinct reasons.

Table 7. Automatic target transform decisions, as recorded in the run metadata.

Each decision is correct, and none was configured. Applying the transform on Caco-2 or Lipophilicity would have failed or produced undefined values; applying it on PPBR would have been inappropriate for a left-skewed bounded quantity. The rule is a small piece of machinery, but it is the kind that silently corrupts results when it generalises poorly, and it generalised correctly to a data domain it was not designed against.

7. Discussion

7.1 Performance against dataset size

Relative standing declines monotonically across the three tasks as training set size increases: 7th of 24 at n=637, 7th of 20 at n=1952, approximately 16th of 22 at n=2940. Lipophilicity is also the only one of the three whose leaderboard is led by graph models, three of which occupy the top positions.

The straightforward reading is that fixed descriptor representations are competitive while data is too scarce for a learned representation to pay for itself, and are outperformed once it is not. Three points do not establish a trend, and dataset size is confounded with task identity, intrinsic noise and the composition of each leaderboard. The pattern is consistent across these tasks but should be treated as an observation rather than a demonstrated relationship.

7.2 What this does and does not establish

It does not establish a new result about molecular property prediction. That descriptor-based gradient boosting is competitive with graph neural networks on ADMET regression is established in the literature [2], consistent with the broader finding that tree ensembles remain strong on tabular problems [3][4], and already visible in the leaderboards themselves, where four of the top seven Caco-2 entries are boosting or descriptor methods. Nor does it establish that P2Predict's modelling is unusual. Its algorithm set is small and standard, and a team with more compute and task-specific features would likely improve on these numbers.

What it does establish is narrower. An engine developed against procurement data, run with no domain adaptation and no per-task configuration, produced competitive results on an externally scored benchmark; its automatic preprocessing decisions were correct in a domain they were not designed for; and its uncertainty quantification behaved as theory predicts under a deliberate violation of the exchangeability assumption it relies on. Those are claims about implementation correctness and generalisation of the tooling, not about modelling novelty.

8. Limitations

1. Self-scored. TDC operates on an honour system; test labels are distributed with the data. The protocol is documented and the code is published, so each step can be checked, but no independent party verified this run.
2. Three tasks. Three regression endpoints from one benchmark group. Endpoints were fixed before results were seen, but the sample is small and all three use MAE.
3. Leaderboard entries are not directly comparable. Submissions differ in whether they train on train or all of train_val, in feature sets, and in tuning effort. The ranking places the result in context; it is not a controlled comparison.
4. Model selection evidence is confounded by the Ipc scaling pathology described in 3.2. No conclusion about the algorithm set should be drawn from these runs.
5. Descriptors only. No fingerprints were used. Several higher-ranked entries are understood to use them, so part of the gap to the top of each board is attributable to representation rather than to the estimator.
6. Coverage has no external comparison. No leaderboard entry reports calibrated intervals, so the coverage results characterise our output but cannot be benchmarked against the field.

9. Related work

1. Huang, K., Fu, T., Gao, W., Zhao, Y., Roohani, Y., Leskovec, J., Coley, C. W., Xiao, C., Sun, J., Zitnik, M. (2021). Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development. NeurIPS Datasets and Benchmarks Track. arXiv:2102.09548. The benchmark group, splits, protocol and scoring function used throughout this report.
2. Jiang, D., Wu, Z., Hsieh, C.-Y., et al. (2021). Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models. Journal of Cheminformatics 13:12. doi:10.1186/s13321-020-00479-8. The direct precedent for the primary result here: across 11 datasets, descriptor-based models outperformed graph-based models on average in both accuracy and computational cost.
3. Grinsztajn, L., Oyallon, E., Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems 35, Datasets and Benchmarks Track. arXiv:2207.08815, where the preprint carries the shorter title “on tabular data”.
4. Shwartz-Ziv, R., Armon, A. (2022). Tabular data: Deep learning is not all you need. Information Fusion 81, 84–90. arXiv:2106.03253.
5. Vovk, V., Gammerman, A., Shafer, G. Algorithmic Learning in a Random World. Springer, 1st ed. 2005, 2nd ed. 2022. The conformal prediction framework underlying the interval construction in section 5, including the exchangeability condition discussed there.
6. Norinder, U., Carlsson, L., Boyer, S., Eklund, M. (2014). Introducing Conformal Prediction in Predictive Modeling. A Transparent and Flexible Alternative to Applicability Domain Determination. Journal of Chemical Information and Modeling 54(6). doi:10.1021/ci5001168. Prior application of conformal methods to QSAR-type problems.

10. Reproduction

The harness, the pinned environment and every per-seed result file are published alongside the source. Results are stamped with the P2Predict version that produced them and stored under a version-scoped directory, so subsequent releases accumulate rather than overwrite. The benchmark data is downloaded by the harness and is publicly available at no cost.

The evaluation environment is deliberately separate from the P2Predict package environment. The benchmark tooling pins older versions of several scientific libraries, and none of it is a dependency of P2Predict itself. Setup order is documented in the folder README.