Rahim Entezari

I am a research scientist at Stability.ai, working on generative models. Before that I was PhD studet at Institute for Technical Informatics, TU Graz and Complexity Science Hub, Vienna. I try to understand how/why deep learning works.

I have successfully defended my PhD thesis with distinction. My thesis and my presentation .

Email  /  CV  /  Google Scholar  /  Twitter  /  Github

• July 2023: I earned my PhD with distinction. My thesis and my presentation .
• May 2023: I joined Stability as a research scientist.
• March 2023: Check out our new pre-print on The Role of Pre-training Data in Transfer Learning. A shorter version is now accepted at ICLR Workshop on Multimodal Representation Learning.
• Janunary 2023: Our REPAIR paper is accepted at ICLR2023. This is a follow up along permutation invariance of neural networks.
• January 2023: As part of my internship at UW, We introduce DataComp a new large scale multimodal benchmark to measure the effect of data curation strategies on downstream model performance. Stay tuned!
• January 2023: I am invited to give a talk about our recent works on permutation invariances at MIT-IBM Watson AI Lab, IBM Research for Febrauray 2nd. If you want to attend, please drop me an email.
• August 2022: One paper is accepted at MICCAI 2022 Applications of Medical AI (AMAI) workshop.
• June 2022: Two papers are accepted at ICML 2022 Hardware Aware Efficient Training and Pre-training: Perspectives, Pitfalls, and Paths Forward workshops.
• June 2022: I oraganized 2nd Efficient Machine Learning Workshop , this time in Vienna.
• February 2022: I will give a talk about our recent work on Permutation Invariance at EPFL Virtual Symposium: Loss Landscape of Neural Networks.
• Janunary 2022: One paper is accepted at ICLR2022(score:7.0, top 5%).

I'm interested in deep learning, optimization/generalization and sparsity. Much of my research is about understaning deep learning phenomena.

Despite the success of transfer learning paradigm for various downstream tasks, still a question remains: what data and method should be used for pre-training? We study the effect of the pretraining data distribution on transfer learning in the context of image classification, investigating to what extent the choice of pre-training datasets impacts the downstream task performance.

In this paper we look into the conjecture of Entezari et al. (2021) which states that if the permutation invariance of neural networks is taken into account, then there is likely no loss barrier to the linear interpolation between SGD solutions. First, we observe that neuron alignment methods alone are insufficient to establish low-barrier linear connectivity between SGD solutions due to a phenomenon we call variance collapse: interpolated deep networks suffer a collapse in the variance of their activations, causing poor performance. Next, we propose REPAIR (REnormalizing Permuted Activations for Interpolation Repair) which mitigates variance collapse by rescaling the preactivations of such interpolated networks.

There are two prevailing methods for pre-training on large datasets to learn transferable representations: 1) supervised pre-training on large but weakly-labeled datasets; 2) contrastive training on image only and on image-text pairs. While supervised pre-training learns good representations that can be transferred to a wide range of tasks, contrastively trained models such as CLIP have demonstrated unprecedented zero-shot transfer. In this work we compare the transferability of the two aforementioned methods to multiple downstream tasks.

We study the impact of different pruning techniques on the representation learned by deep neural networks trained with contrastive loss functions. Our work finds that at high sparsity levels, contrastive learning results in a higher number of misclassified examples relative to models trained with traditional cross-entropy loss.

We show that, up to a certain sparsity achieved by increasing network width and depth while keeping the network capacity fixed, sparsified networks consistently match and often outperform their initially dense versions.

we propose an iterative deep model compression technique, which keeps the number of false negatives of the compressed model close to the one of the original model at the price of increasing the number of false positives if necessary.

we present a hierarchical energy system architecture with embedded control for network control in microgrids.

We propose an end-to-end deep network for detection and fine localization of irregularities in videos (and images).

Thanks Jon!