Down and up

I often see some phrases that go like this “You’ll have to come down and see for yourself”, I don’t know much about the spacial context but it was said over the phone. I can understand someone saying this in a building where someone needs to go down, but why was it used in that case?

Similar has happened with up and over. “My friend is coming over”, shouldn’t it have more information that that? Over what? Like I said with the phone, over what? Over the phone.

I’ve noticed many different examples on this but that’s the one I remember now. Most of them don’t seem to have a sense of general up and down meaning.

Does anyone get what I’m saying?

Up the road /
down the road
don’t necessarily mean there’s a hill there.

Come over (to my house) /
come down (to my house)
likewise.

That’s the problem. If there’s not a hill there how should I know which one to use when? Can I say “Come down to my house” as much as I could say “come up to my house” without the up/down meaning?

Listen to what the locals use and do the same. There’s no hard and fast rule as to when to use ‘down’, ‘up’ or ‘over’.
It’s not only learners who face that question. Native English speakers will often use them randomly too.