Unreal past situations
The perfect infinitive is often used after verbs like mean, be, would like etc., to talk about unreal past situations.
She was to have returned yesterday, but she fell ill.
I meant to have posted the letter, but I forgot. (I did not post the letter.) englishgrammar.org/perfect-infinitives/
Be + to-infinitive can also be used in the past: Mr Jones was to speak at the meeting. (it was arranged and he did)
But was/were + to + perfect infinitive means that the arranged event did not actually happen:
Mr Jones was to have spoken at the meeting, but he had to cancel because of his illness. grammaring.com/be-to-infinitive