It's a bit more complicated than that. It was Brandis in that Brandis introduced the bill that eventually passed. On that note, he demonstrably showed a lack of judgement that will come back to bite us in the future, so definitely no free pass for him.
Indirectly, the ALP is tarred with the same brush here. There are times when you could argue that positions were changed by compromise and negotiation. For example, a party may claim to be against a particular service cut/tax hike but negotiate it through in such a way that the constituency that they are concerned about is compensated for that change. Sometimes it is a pragmatic decision to take a lesser of two evils on offer. That wasn't the case here though. The alternative "do nothing" was indeed a live choice and the preferred position of a significant minority of both major parties.
More directly though, there were definitely rumblings back in 2010 and 2012.
Here is a link to a senate investigation on the matter.
So Conroy is knee deep in it, even if he himself didn't pull the trigger.