In the FIRE podcast with Ira Glasser (former executive director of ACLU) he tells an interesting story. The ACLU was trying to get some small political parties on the ballot in various states, and one party said: yes, I understand why you are trying to get us on the ballot, but why do those other parties need to be in the lawsuit? Glasser says that everyone's interest in free speech is, in the first instance, an interest in their own free speech, not that of other people.
Consider the "heckler's veto." This is when a speaker is shouted down and cannot give her speech. One way to think about it is to consider what it would mean if Martin Luther King came to your college and was shouted down by racist students. How would you feel about that? Someone might answer: "that's different. The heckler's veto is only legitimate if the speaker being heckled is a bad guy." But who gets to decide? Ultimately, it's not the students, but the administration.
Glasser makes the point that the first use of campaign finance restrictions was against some old radicals who put an ad in the New York Times to denounce Nixon's bombing of Cambodia. In other words, once you authorize censorship, then you have no way of ensuring that the "good guys" won't be censured too.
That's why free speech has to be a content-neutral principle. It does no good if it only protects the speech that someone in power happens to like at any given time.