"We're not dinosaurs, reliant on a certain climate or delicate environment..."
I recommend you look up a book/documentary called "Guns, Germs and Steel" by Jared Diamond. He's an anthropologist who spent 30 years researching why Eurasian civilisations dominated the world over other cultures, and his thesis states that geography - specifically the temperate conditions that allowed certain species of food plants (wheat, oats, barley) and animals (goats, cattle, horses) to exist - is what gave the Eurasian cultures their edge, by freeing them from the exigencies of survival and enabling specialisation. When they attempted to colonise the tropics, their traditional survival methods failed because the plants and animals upon which our civilisation depends were not adapted to survive there.
So we do need surprisingly specific conditions to survive, or at least to maintain the structures of advanced civilisation. Without our temperate-zone climate, wheat and other staple foods don't grow, and without them civilisation as we know it cannot stand.
In a post-impact Earth, survival alone becomes only a thin possibility. If we look at past extinction events - the Permian-Triassic or the Cretaceous-Tertiary, for example - we see that the climate changes that resulted from them weren't measured in decades or even millennia. The lushness of the Permian gave way to the deserts of the Triassic; likewise with the verdant Cretaceous and the barren Tertiary, both of which lasted for millions of years before the Earth recovered.
In a mass-extinction event of this magnitude, large, complex lifeforms cannot survive. Only the smallest, simplest creatures can eke out an existence, and give rise to new evolved forms over time; the dinosaurs from the Permian-Triassic, and the mammals from the Cretaceous-Tertiary. The poisonous atmosphere, the centuries of global winter darkness resulting from an impact - these effects would last far longer than our civilisation has already existed and developed, and so the chances of anything much more complex than a frog surviving them are marginal at best.
So "re-colonising Earth" is simply not a viable proposition in the aftermath of such an event. Not unless we can maintain a civilisation in underground bunkers for a few million years at a stretch, until the Earth becomes inhabitable again...