GPT-3 Reflection

The past

Before writing GPT-3 Reflection, I am trying to jot down some part of a memory left 4 years ago to set baseline. In 2016, I attended a conference of IBM Watson. There were a couple of interesting demo not less than my visit in CES 2019 with quantum computer and flying cars.

For example, personality quiz matching with celebrity types, Harry Potter sorting hair, voice command with virtual reality, control a robot using only thoughts.

IBM Waston was known as the prominent of NLP community long time ago with a television one million prize winner called Jeopardy. In 2017, it was the best candidate recommended by industry education company - Udacity for their official debut AI Nano degree program.

The present

The hot trend of this year is GPT-3 produced by Open AI, soon-to-be a product company with their fine-tuned API. The demo day from Pioneer is quite down to earth with some potential no-code use cases:

The future

TL;DR: with $5-10M and minGPT-3, you can replicate the work at home.

Discussion about Andrey Karpathy’s open source of min-GPT-3 with panelist.

The source code includes the most basic structure of GPT-3 based on previous versions. In overall perception of demos, people changes the prompt to have different answers but not the weights of model. In one scenario, different kinds of GPT-3 can be trained by anyone at home.

One of the concerns is about the complexity of training such a super large model with optimization, distributing computing, data center and data processing with ETL. Nevertheless, people can continue to fine-tuned pre-trained model with other data set. It would be a shift between a feature engineering to prompt engineering. And leads to a possibility of open Wikipedia 2.0.


I still remembered a conversation from top researchers and engineers in the conference that year. It was about a new kind of programming without code, but a block of cognitive services similar to play lego. The progress was still slow and only open to small selective community in North America.

The future is approaching very close to majority, probably it is the best highlight for an economical downturn caused by a Pandemic. Likely, we are entering to a new dawn of real AI economy emergence when people can’t go to work and travel, universal basic income, most of business going online on the internet with a global effect.

When you fail, make it a dance.

Tagged #reflection, #gpt3, #ai.