Yes, you're right. "I love you" is more commonly used to express a romantic kind of love, except with family members of course. It seems to hold much more meaning than just "I like you" so a person would typically only say it to someone who is very special to them.

英語が上手ですね。さすがにアメリカに住んでいる人。がんばってね！

You're right, we are very careful when to use 'I love you'. 'Love' is used for family members and boyfriend/girlfriend relationships. You can also say you 'love' your friends.
But be wary, when you first ask someone to be your girlfriend/boyfriend you don't say 'I love you', you say 'I like you'. Because 'love' is a very strong word and it can scare people off if you say it to them too quickly.
It's a common problem people who are just starting out in their relationship because they're not sure if it's too soon to say 'I love you' to the other person, and they get worried that if they say it to them then the other person won't say it back. There's really no set time for when you can say 'I love you' in terms of a boyfriend/girlfriend relationship, it's really determined by how close the two are and how they feel.
I hope this helped.

最近、私には、経験済み「あなたが愛してい」て「あなたが好きな」ような誤解がありました。私には、台湾からのガールフレンドがいました。私は愛していると彼女に言いました。 しかし、彼女は、私が好きであったと私に言っただけです。彼達の"愛"と"好き"が非常に異なっていることを学びました。愛ことに強い感じです。ほとんどの人々は結婚するまでそれを言いません。アメリカ人のために彼らの違いを理解しにくいと思います。