Ch 11, stuck - matrix inverses

After implementing the Ch11 scene and reflections / refractions, I noticed my app crashing. The discovery trace went as follows:

1. C# crashes during intersection sorting (world intersections with ray).
public List<Intersection> Intersect(in Ray r)
{
   List<Intersection> retVal = new List<Intersection>();
   foreach (HitObject item in objects)
   {
       List<Intersection> xs = item.IntersectionsWith(r);
       if (xs.Count != 0)
           retVal.AddRange(xs);

   }
   retVal.Sort((Intersection a, Intersection b) => MatMaths.SpaceshipOp(a.t, b.t));
   return retVal;
}
2. Discover that t values are all NaN (sometimes), therefore the comparison gives up (I think).
3. Discover that t values are already NaN during HitObjects intersections creations (Spheres, planes). Plane code below.
protected override List<Intersection> LocalIntersectionsWith(in Ray ray)
{
   if (Math.Abs(ray.direction.Y) < MatMaths.eps) // IF  || double.IsNaN(ray.direction.Y) is added, app does not crash but image still incorrect
       return new List<Intersection>();

   double t = -ray.origin.Y / ray.direction.Y;

   return new List<Intersection>() { new Intersection(this, t) };
}4. Discover that ray values (origin, direction) are all NaN (sometimes).
5. Finally discover the culprit (I think) - Matrix inversion fails and returns NaNs.
public List<Intersection> IntersectionsWith(in Ray ray)
{
   Ray r = ray.Transform(Transformation.Inversed());

   return LocalIntersectionsWith(r);
}
Now, I use a custom inversion method (augmented matrix M|I) but I don't think that's the issue (I tested using System.Numerics.Matrix4x4.Invert method and get the same results).

The issue occurs when the depth of recursion is > 0 (i.e. reflections / refractions without them the code runs okay). And lastly I attach an output image as there are artifacts.

What should I do in case of non-invertible matrices? I recall the book mentioning throwing an exception but that's not a solution.
Any tips? Where to investigate further? Repo found here