You could probably find more people to talk to about Lost or Heroes or any other TV show than all of anime combined. So it's hard to randomly come meet people on the street who watches anime. That said, it's still not that hard to find other people who watch anime:
http://anime.meetup.com - 125 Groups and 15,625 Members in 100 cities so far
and
http://otakubooty.com 
with 19,744 Members so far