DERI Seminar with Prof Frank Hutter (ML Lab at the University of Freiburg)

When: Thursday, October 12, 2023, 11:00 AM - 12:00 PM
Where: zoom

Speaker: Prof Frank Hutter from the University of Freiburg and  Noah Hollmann

DERI/PHURI seminar presented by Prof Frank Hutter from the University of Freiburg and  Noah Hollmann

Zoom link: https://qmul-ac-uk.zoom.us/j/81148100921

Title: TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second

Abstract: Machine learning approaches have recently shown improvement over conventional modelling techniques by capturing complex interactions between patient covariates in a data-driven manner. However, modern machine learning approaches are particularly hard to apply to medical data, since (1) they are often hard to use without technical expertise,(2) complex models are harder to analyze and understand, less reliable and often fail when sample sizes are low and(3) "Tabular data" such as diagnostic codes, lifestyle factors, genomics, and other omics-type data present the most common clinical data type, however, compared to image and text processing, this data type is particularly hard to process with current machine-learning approaches.

We present TabPFN, a trained Transformer Model that can do supervised classification for small tabular datasets in less than a second, needs no hyperparameter tuning and is competitive with state-of-the-art classification methods. TabPFN is fully entailed in the weights of our network, which accepts training and test samples as a set-valued input and yields predictions for the entire test set in a single forward pass. TabPFN is a Prior-Data Fitted Network (PFN) and is trained offline once, to approximate Bayesian inference on synthetic datasets drawn from our prior. This prior incorporates ideas from causal reasoning: It entails a large space of structural causal models with a preference for simple structures.

On the 18 datasets that contain up to 1000 training data points, and up to 100 purely numerical features, we show that our method clearly outperforms boosted trees and performs on par with complex state-of-the-art AutoML systems with up to 70× speedup. This increases to a 3200× speedup when a GPU is available.