'Follow me' or 'Come with me'

If I want to tell a person to accompany me, where I will take him to a particular place, which he is looking for, what should I say - ‘Follow me’ or ‘Come with me’?
A friend of mine told me, I can tell ‘follow me’ only when I am speaking to more than one person, which is not the case above. Is that correct?

That is not correct-- your friend’s advice is ridiculous. You can use both phrases synonymously.