Autocode Suggestion using GPT-2

Keywords: Transfer Learning of model, GPT-2.

Introduction

"Time to Market" is a key ingredient for the success of any product in recent times. In the current industry scenario in Software Engineering, software development with programming languages is the most critical part of project execution. Lower-level software programming takes the majority of time and impacts the quality of deliverables. Auto Code Generation or Suggestion tools can assist the software development process and improve programmer's productivity with fewer errors.

Autocode - Video

System Overview

The overall system, including model transfer learning and inferencing, is composed of several key components.

GPT-2 Architecture

GPT-2, or Generative Pre-trained Transformer 2, is a large language model from OpenAI designed for sequence prediction. It uses a decoder-only transformer architecture and is pre-trained on a massive 40GB WebText dataset, making it ideal for transfer learning on specialized datasets like C programming code.

Transfer Learning

Transfer learning adapts a pre-trained model for a new, specific task. In this project, a pre-trained GPT-2 model was fine-tuned on a C Programming Dataset. The process involves tokenizing the C code and adjusting the model's weights to better predict sequences in this new domain.