Notes

I draw visuals in order to understand concepts. Sometimes friends have found these helpful, so I thought I'd make them public.

These are raw notes from when I was learning each subject (sometimes many many years ago!), so they could be wrong! If you have a question about something, feel free to ask.

How probability distributions relate to information & entropy

Why continuous distributions have infinite entropy

Cross-entropy

Conditional entropy

Independence

That moment when it clicks why KL divergence between P(X,Y) and P(X)P(Y) equals mutual information I(X,Y)! Thanks to Chris Olah for the rain/sun coat/no coat joint distribution example shown here.