I agree with you there. The purpose of anime is to be viewed and heard, not read. Watching it in a language you do not know means that you lose some of the inflection (though you can certainly pick up some if you know the basics of a language). Also, if your series is worth much of anything and not old, there should be things moving onscreen that you will want to be watching.
That having been said, I don't have problems reading subtitles, even on some of the fastest-speaking anime. When multiple people are speaking at once it can get tricky, and sometimes I have to pause.
Now, as to language "speeds": I belive that people generally consider other languages to be faster than ones they know because their brain processes individual sounds instead of entire words. I have a friend who is half-American, half-Argentinian and grew up in both cultures; he says that all his Argentinian friends ask him why Americans talk so quickly and that Americans say the same about Spanish. Personally, I think Spanish sounds much slower to me now that I have a basic grasp of it.