freedom of speech and responsibilities

Freedom of speech is a bedrock principle of a free people. To suppress a book, article or speech on any subject, is to potentially suppress everyone. Certain social, political and religious subjects inspire passion, often to the exclusion of reason, and these can make freedom of speech an intellectual mine field.
You have the right to stand up in front of a group and deliver a speech on any topic you like. However, I do not have the right to shout you down and drive you from the podium, no matter how passionately I disagree. That would be my free speech negating yours. I do, however, have the right to wait my turn and then offer a rebuttal to everything you've said. That's how freedom of speech should work, and I'm fairly certain it is how our country's founders meant it to work.
Also, when we move into the stormy seas of personal attack, we test the limits of free speech. If I don't have a cogent argument against your position and thus decide to attack you as a person, what are my limits? I can certainly point out your past support of organizations that oppose my position, you prior remarks denigrating my position or your past actions that might reflect on your credibility. That, however, does not give me the right to lie about you or insinuate things about your character and behavior that is speculative. This last has become an issue within the social networking sites.
In the end, all rights come with responsibilities. Civilized debate is essential to a civilized society, and a verbal mob is no different in essence that a physical mob. Freedom is a high concept, one we must continual rise to meet.