[lecture] Building a micro GPT (Blog Post)

[mmcky]

This blog post is pretty cool

https://www.towardsdeeplearning.com/andrej-karpathy-just-built-an-entire-gpt-in-243-lines-of-python-7d66cfdfa301

It builds a GPT in 243 lines of python including the autograd engine.

It learns baby names and then creates new ones.

Q: How is this different to a simple transition matrix building probabilities off of word transitions? Token size, how it builds patterns?

I think this could form the foundation of an interesting lecture using python.

[mmcky]

Here is the GIST

https://gist.github.com/karpathy/8627fe009c40f57531cb18360106ce95

the comments on the GIST are pretty fun, people are porting and having a bit of fun with this.

[mmcky]

Moving this to the lectures monorepo — under the pool model a new lecture is pitched to the shared pool, and which product publishes it is a later manifest decision.

Both links are still live, including the gist.

Nothing in the series covers transformers. A search for "transformer" across lecture-python.myst, lecture-python-intro, lecture-jax and lecture-datascience.myst returns zero.

The question in the issue body is the right spine for the lecture, and we already teach both ends of it. Asking how this differs from "a simple transition matrix building probabilities off of word transitions" is a question our own material sets up:

• finite_markov.md teaches transition matrices, which is exactly the bigram baseline being compared against
• back_prop.md, "Introduction to Artificial Neural Networks", builds a network and derives back propagation through the chain rule by hand

That makes this an unusually well-scaffolded proposal: the two prerequisites already exist, and the lecture is the bridge — begin with the Markov model the reader knows, show where it fails on longer-range structure, and introduce learned representations and attention as what buys the difference. Karpathy's implementation including its own autograd engine pairs neatly with back_prop.md, which likewise avoids leaning on a framework.

Two things worth settling early. First, whether the lecture reproduces the 243-line implementation or writes a QuantEcon version against it — the pedagogical shape may want to differ, particularly to make the Markov comparison explicit rather than implicit. Second, the licence on the gist needs checking before any code is reused; gists frequently carry none, in which case default copyright applies and a clean-room implementation is the safer route.

Moved from #5 as part of the disposition review in QuantEcon/meta#344.