Goncalves, T.; Pulido, D.; Perrino, C. M.; Lomphithak, T.; Cleveland, M.; Dalca, A. V.; Gerstner, E.; Hipp, J.; Patel, J. B.; Rosen, B.; Sirintrapun, S. J.; Wander, S. A.; Parwani, A.; Tozbikian, G.; Niazi, M. K. K.; Cardoso, J.; Brock, J.; Zanfagnin, V.; Gazzaniga, F.; Iafrate, A. J.; Flaherty, K. T.; Sgroi, D. C.; Guttag, J. V.; Bridge, C. P.; Kim, A. E.

Precision oncology lacks scalable tools to assess, at the patient level, systems-level tumor microenvironment (TME) programs driving therapeutic resistance. To address this gap, we trained a weakly-supervised deep learning model that uses routine H&E whole-slide images (WSIs) to derive quantitative activity for therapeutically-relevant TME phenotypes, spanning immune, metabolic, and tumor cell-intrinsic programs. Using 3111 breast cancer H&E WSIs with matched bulk transcriptomics, our model accurately infers these biological states, defined by pathway enrichment scores (AUROC>0.80; PCC>0.64). Validation spanned three levels: (i) tissue-matched multiplexed immunofluorescence, showing concordance between inferred functional states and immune cell fractions (p=0.006-0.106), (ii) blinded reader assessments, confirming localization of phenotype-specific morphology (p<3 * 10^-5), and (iii) multi-institutional patient cohorts, where model-derived phenotypes stratified for clinical response (p<0.045). Unlike methods requiring resource-intensive spatial profiling data for training, our approach leverages widely-available therapeutic outcomes or bulk profiling as slide-level labels to assess functional biology. This strategy offers a scalable complement to spatial Omics for investigating therapeutic resistance across the pan-cancer landscape through using WSIs and clinical outcomes from massive legacy biobanks.