hessian-eigenthings

Iterative eigendecomposition of curvature operators (Hessian, GGN, empirical Fisher) for arbitrary PyTorch models, including HuggingFace and TransformerLens transformers. Top eigenvalues via Lanczos or power iteration, trace via Hutch++, and the spectral density via Stochastic Lanczos Quadrature, all matrix-free.

The full Hessian of a model with \(n\) parameters costs \(O(n^2)\) memory, infeasible past toy scale. Iterative methods only need Hessian-vector products, which cost \(O(n)\). That's what this library is built around.

Where to start

• Quickstart: 30-second example on a small MLP.
• Transformers quickstart: same shape, on a HuggingFace causal LM.
• Concepts: what each algorithm computes and when to pick which. The GGN vs Fisher vs Hessian page is worth reading early since they're easy to conflate.
• How-to recipes: task-oriented walkthroughs.
• API reference: every public symbol.

Install

pip install hessian-eigenthings
pip install "hessian-eigenthings[transformers,transformer-lens]"   # with helpers

Source on GitHub: https://github.com/noahgolmant/pytorch-hessian-eigenthings.