Yes, I've often wondered about this.
When I was a boy I read a couple of books with titles on the lines of "Relativity for T C Mits" (T C Mits being The Celebrated Man in the Street). These books laid out the development of Einstein's Special and General Theories. They introduced a modicum of tensor analysis geared to the understanding of a 14 year old.
One of the things I always found dubious was their glib references to massaging the equations so as to eliminate the infinities. Now to my immature mind at the time, this always smelt of cooking the books, and I naively thought that Scientists did not fudge facts to fit theories. If there were awkward infinities in the equations then surely these equations represented some phenomena which included infinite quantities.
Messing with the Maths to get rid of these infinities was surely moulding our understanding of the physical phenomena to fit some pre-conceived comfortable "comfort food" for the limited human understanding.
Now the subject has cropped up again. If the equations contain infinities, then it's because the phenomena described by the equations have infinite values somewhere hiding in them, or division by zero, which as any first year comp sci student will tell you, is forbidden by convention only, not necessarily by "truth".
Sorry, I'm a bear of very little brain.