Funny how three bishoujo games made it onto the list
Kanon - One of the most emotional game I've ever played. The story is top notch, with the anime version only composing of a mere fraction of the game. The game is also much, much sadder than the anime.
Mizuiro - OMG, the anime did not give this game justice. Hiyori's storyline was butchered in the anime. Her story in the game is so, so,
so sad. I cried so much
Air - There's a reason why this game was voted as the best game in Japan for two years in a row, and yes, it's a dating sim.
Zone of the Enders: Fist of Mars/2173 Testament - Hands down, best story for the GBA. The only other real contender is Super Robot Wars R. If you haven't played it yet, do yourself a favor and get it. Having a great storyline is one thing, but most games lack another important factor: Character Interaction. I think it was the only GBA game to ever make me cry.
Super Robot Wars Alpha - Imagine a strategy game that takes the characters/storyline of over 15 mecha series such as Getter Robo, Mazinger Z, Evangelion, Gundams, Macross, etc and combines them all together in the same universe to form and epic storyline. Best scene ever: reinacting End of Eva, and stopping it.
Xenogears - I don't think I need to explain why.