Can you say "looking downward" to something?

You can say that if you want. If you used it in an obvious way like “im looking downward to the test coming up”, then sure. go ahead. Ive never heard anyone say it before.

but you couldn’t use it as a replacement word for looking forward to something.

No, you would say “I am not looking forward”…

You can say “looking downward” as the opposite of “looking upward”, as in:

The leader on the Rock-Climb looked downward towards the second to check that he was ready to climb.

You can say “looking backward” as the opposite of “looking forward” as in:

I heard footsteps and looked backward to see who was following me.

When used figuratively, as used in the expression “… I am looking forward to the party this afternoon! “ as you suggest the non-figurative opposite would be “reminisce about” as in:

“ I am reminiscing about the party last summer.

a plausible opposite of “forward” is “back” and the Band “Oasis” use the term “look back” in a some beautiful figurative writing in the song “Dont Look in Anger” ….