Well I would say Japanese is one of the easier languages to learn as a second language even if it may not be the easiest to some. But don't get scared by the Wiki entry on Japanese grammar.
Japanese grammar is highly structured with only a very limited number of exceptions when compared to the European Languages. (Damn I have been spending over 20 years on English as a second languae and still fall into its grammatical traps every single day...) The only hurdle is maybe the vocabulary. But I still remember the amazing sense of achievement of understanding 70%+ of the words in a PS2 driving simulator game (mostly technical parts and brands in katakana) by just spending two hours to swallow the katakana list.
After that, I took a weekend-3-hour-course for two years and got my lv 1. By now I can already work in a normal Japanese office where no one can speak English at all.
No, you don't need to get to the Shakespearean-level of English to survive in UK/US or enjoy their TV shows (although your English teacher may suggest otherwise
). The same applies for Japanese. Don't get too stressed with the language as most likely you will only use it for travel and anime. Think of it as an opportunity to take a break from your dull, meaningless daily school/work and give you some kind of realistic goal. You can also make friends with people who are as interested in the Japanese culture as you by taking language classes (and many of them maybe anime fans too!).
IMO the key is to learn at your own pace for your own leisure (instead of for the JLPT, for bragging or for some sort of future income stream), find good study buddies, and enjoy the ride as well as the sense of achievement on the way. You will realize how much and how quick you have learnt when you look back after two years.
For the wiki entry on Japanese grammar, although I basically know everything it tries to convey, it still makes me go all dizzy. It only serves to scare away those who knows little about Japanese, and for those who do know (you may grab a native Japanese for a test), they will still get utterly confused by the mumbo jumbo of meta terms like indeterminate non-volitional whatever. Leave it to the linguists and grammarians. A good teacher and textbook is all you need.
My advice to those who are interested is go straight to find a native as your tutor or take a community college course. You will know whether the language is suitable for you in one month. Even if it turns out that you hate it with a passion, you will still have your katakana list in your head, that can be extremely useful for gaming or other purposes.