Hello.
It’s a poor sort of memory that only works backwards.
If I´m not mistaken it´s from Alice in Wonderland. I haven’t read the book and don’t know what situation the phrase was used in, but what does “to work backwards” mean?
My opinion
backwards ~ towards the past in which the things happened.