Carrying out VMC calculations with neural network wavefunctions comes with a lot of optimization-related challenges which can go under-reported in final publications. These include mystery NaNs, energies not converging – or worse, converging inconsistently, and poorly understood dependence on hyperparameter selection. In this talk I will offer a frank description of our experience with these issues, mainly in the context of periodic systems, and discuss the approaches we and others are taking to improve the situation.