Using 's in the sentence without possession

Hello,

Can you please explain, why we use 's in the sentence like this

" You have to go to the doctor’s immediately " ?

Thank you in advance

doctor’s here means doctor’s place (clinic/hospital).

In my nook it’s common to take that out of the equation:

  • You have to go to the doctor immediately.