What is so special about [free] speech?

Legal and political philosophers (e.g., Scanlon, Schauser, etc.) typically regard speech as special in the sense that conduct that causes harm should be less subject to regulation if it involves speech than if it does not. Though speech is special in legal analysis, I argue that it should not be given comparable status in moral theory. I maintain that most limitations on state authority enacted on behalf of a moral principle of freedom of speech can be retained without supposing that speech is entitled to a degree of protection not afforded to (most) other forms of conduct. My argument questions some standard assumptions made by philosophers about the relationship between moral and legal principles.