I can’t work out which to use. Is it doctor’s appointment because it is an appointment with the doctor and is an appointment belonging to the doctor?
I’ll tell you how I see it.
Your question is not defined enough. Please tell me did you arrange the appointment with the doctor or the doctor arranged the appointment with you or some other one (another doctor, for example).