It won't be a doctor

Talking about his prospective retirement, SAF said, “Nobody knows. Neither do I. It won’t be a doctor that tells me to quit.” which I was reading as: “Among the possible reasons to quit it, my health would be the last thing to consider (=I’m still strong enough)”
–Would you take it as correct?

Yes, I would agree with your interpretation.