Research / Harvard University, undergraduate thesis

Improving Neural Networks with Generalizable Performance Predictors and Generative Code Language Models

Aakash Mishra

Harvard University, undergraduate thesis · May 2023 · Advised by Stratos Idreos

NAS-Assistant workflow: a model and its data pass through a performance predictor and layer ranking to a generative code model that writes a new model

NAS-Assistant predicts how well any network will perform, ranks which layers matter, and uses a code language model to write better architectures.

Abstract

Neural Architecture Search (NAS) is a growing field with many evolving facets, from evaluation strategies and search-space criteria to architecture optimization and performance prediction. These spaces are currently disjoint and constrained by a lack of generalizability: structured search spaces restrict algorithms to specific architectures, while performance estimators are fixed to given benchmarks without the ability to conduct zero-shot evaluation.

Using advances in generative AI, we present a combination of these methods in a tool called NAS-Assistant. It consists of a new generalizable GNN-based neural architecture encoder and a clustering, attention-based regression network that predicts model performance with high accuracy and transferability. We also propose a method for evaluating the contribution of each layer of a network, combined with zero-cost NAS evaluation. Lastly, we develop a framework for using generative code language models to explore any model search space requested from NAS-Assistant. This thesis aims to demonstrate the first integrated generative AI optimizer for Neural Architecture Search.

Cite

@thesis{mishra2023generalizable,
  title = {Improving Neural Networks with Generalizable Performance Predictors and Generative Code Language Models},
  author = {Aakash Mishra},
  year = {2023},
  school = {Harvard University},
  note = {Undergraduate thesis, advised by Stratos Idreos}
}