Both of you nail how others misuse the term "loli" and I agree with you. They hurl it at anything that makes them worry about liking it until it sticks (e.g Manabi Straight, Sky Girls, etc). I happen to like small, petite women (am married to one) and when I express enjoyment of a classic like, say, Ichigo Masimaro -- here comes the name-calling from parts of the anime jungle.
Careful about the "underage" definition though --- that varies widely from state to state and country to country (ranges from 14 to 21).
My best definition is that "loli" (derived from "lolita" of course) mainly refers to that post-pubescent period of around age 12-14ish. Admiring women older than that is neither pedo nor loli .... just "underage" depending on what jurisdiction you're standing in.
But admiring certain types of females is one thing --- actively courting them can be something else. My wife knows a coworker who is 40.. .married a 19yr old from his church. Normally I'd say "well.. maybe they have a lot of common interests"... except that we know this guy and he's a spoiled immature brat so its not hard to do the math
Back on track: Aye, listening to why Kanata and Soujirou connect is pretty romantic and adorable in their quirky way.