There are 4-5 million American Muslims. Of these, I can recall only five who have engaged in violence or attempted violence related to terrorist ideology (shoe bomber, Times Square bomber, Fort Hood shooter, and the two Boston Marathon dudes). That would make it about one in a million, though I might be forgetting someone. Either way, the point about numbers stands. As a practical matter, if any worthwhile proportion of these Muslims were inclined to terrorism, we'd see far more of it than we could possibly handle, which is clearly not the case.
In response to a possible counter of the above point by quoting Qur'anic verses, you can then say that people can interpret the Qur'an and life of Muhammad all they want, but unless large numbers of Muslims agree with those interpretations, what they are doing is coming up with reasons why they, personally, are not planning to convert to Islam, not making practically important statements about Islam as it exists in the world.
I'm limiting this to the United States because it is the easiest place for one answering someone with anti-Islamic instincts or opinions to get back on friendly ground, though once on this ground it becomes easier to potentially extend the analogy overseas. It is also clearly more complicated for actual Muslims, who might be called upon to defend themselves for understanding Islam the way they do, a battle which is more optional for the non-Muslim. Still, I've learned to appreciate the need for points that are hard to argue with and easy to understand.
UPDATE: Just realized I should add Anwar al-Awlaki, Adam Gadahn, and John Walker Lindh to the list of American Muslim terrorists. However, the fact we are still counting individuals is telling.