I think the controller was wrong. He should have asked you to read back the clearance, which you obviously gave, but he didnt hear. That is the catchall and i think may have more specifically alerted you to the problem. When he didnt get a read back he should have cancelled your departure. Seems to me there was time for all that.
Neither you or he should have debated the matter after departure. If he persisted you should state you will give the tower a call when back on the ground. Exactly this type of distraction in an already tense situation is unhelpful.
All that said departing with a bad radio really shouldnt be the end of the world, we are far to obssessed with the need for a radio.
