the bug that taught me more about PyTorch than years of using it

26 points by knl

icefox

Translation of the introduction from machine-learning speak:

hyperparameter: your model consists of a function (in this case the neural net) and a bunch of parameters (the weights), and training is the process of searching for parameters to tune this function to give the result you want, such as letting it take a pile of pixels as input and give you the output "is this a photo of a cat? y/n". The hyperparameters are the inputs you give the training process for how to do the search.
loss: a function that describes how crappy your model is at giving the result you want

If you think "wait if we have a search function that takes a bunch of inputs, and a meta-search function that tunes the search function, can't we just make another search function that searches for good inputs to the meta-search function?" then the answer is "yes, but people have been trying for 40+ years and afaik it still kinda sucks".

sanxiyn

Re: hyperparameter search. I agree "it still kinda sucks". I found Bayesian Optimization in AlphaGo to be an interesting look into hyperparameter search in practice in a production system.

mjn

Wow, this was a great read. I did not guess the bug before I got to it! Good time of year too; "it's never a compiler* bug... until it is" makes for a good comp sci horror story setup.

* Using compiler here in a broad sense.