+- MKLab (https://mklab.gr)
+-- Forum: [INDEX] (https://mklab.gr/forumdisplay.php?fid=1)
+--- Forum: ARTFICIAL INTELLIGENCE (AI) (https://mklab.gr/forumdisplay.php?fid=5)
+---- Forum: ARTICLES (https://mklab.gr/forumdisplay.php?fid=33)
+---- Thread: Neural network training makes beautiful fractals (/showthread.php?tid=2066)
Neural network training makes beautiful fractals - mklabgr - 09-22-2026
Jascha Sohl-Dickstein explores a striking connection between fractal mathematics and neural-network training. Both processes repeatedly apply a function whose behavior depends on parameters: in fractals, changing parameters can determine whether an iterative process remains bounded or diverges, while in neural networks, hyperparameters such as learning rates determine whether training converges successfully or becomes unstable. By visualizing neural-network training across different combinations of hyperparameters, Sohl-Dickstein finds that the boundary between successful and failed training forms intricate fractal patterns. Zooming into these boundaries reveals new structures at progressively smaller scales, and the phenomenon persists across different network architectures, activation functions, datasets, and training configurations.
The author argues that this result is not entirely surprising because neural-network training is itself an iterative optimization process, much like the iterations that generate Mandelbrot, Julia, or Newton fractals. This fractal structure also helps explain why meta-learning and hyperparameter optimization can be difficult: tiny changes in hyperparameters may produce dramatically different training behavior, making the resulting meta-loss landscape highly irregular and chaotic. Rather than viewing this merely as an undesirable complication, the article presents the fractal structure as a deeper mathematical feature of optimization dynamics—and, fittingly, as something that can produce surprisingly beautiful visual patterns.
ΑΡΤΙΨΛΕ