Exactly, it's balancing that against trying to justify to communities that would literally starve without it (thinking of the Inuit et al). Many societies in the past that we would consider 'undeveloped' had at least an appreciation of the impact of their impact on their environment. Some took it quite far with the Hawaiians (kanaka maoli) having a fairly advanced system of limited seasons and restocking with transgressions severely punished. Most of the societies couldn't impact whale numbers significantly even if they wanted to.
Attitudes towards other animals have evolved, Green Sea Turtles being another example here. Whilst they had been eaten in the past it's rare (and illegal) these days and most have accepted the need to put aside the practice as it is no longer necessary and the impact was too great. Numbers have started to increase which is great. Over fishing of other species has had a huge effect though, something that wasn't a problem in the pas (pre contact)t even when sourcing all the food for nearly the same population level from here (without Costco to help). The modern attitude of racing to take everything and ^&%$ tomorrow has prevailed.
Regulating global impacts is important but I think different countries require different approaches. We cannot (or rather we do not want to) force Japan to give up whaling. I think that the more we fight them, the harder we fight them, the more resolute they will be. On the one hand I think it is great that Seashepherd is out there actually doing something and whilst they do manage to limit the catch I think they also ensure that Japan will be back again next season no matter what the cost. If Japan were led to a position where they 'voluntarily' agreed to limit their activities to subsistence levels in their own territorial waters I think that would be a better situation for the Whales. Personally I can't understand why anyone would want to kill any of them but such is life, to some they are just meat.
It amazes me just how many Japanese are actually against Whaling. We get a lot of Japanese tourists here in Hawai'i and from speaking to them whilst many don't view it as barbaric, they do view it as something that belongs in the past. It maintains political support because the government uses it to keep support from a subset of hardliners in the same manner as the Republicans support gun rights (i.e. blindly without any concessions because any weakening would result in revolt).