Should i back off of him?

Assuming that you are not seriously visually impaired:

….. if you go into a room with (for example) 30 people in it, after a period of only a few seconds, you WILL almost certainly have looked at the faces of all 30 people, and they will almost certainly have looked at your face before you leave the room - if they actually ever became aware of your arrival.

I don’t mean that your eyes will have simply passed over the individual’s location, but that your eyes will methodically scan multiple significant features that allow you to identify those people whom you know (and recognise as friends & relatives), as well as others who you might believe to be hostile towards you.

It makes no difference whether you like, hate, or are indifferent to one another. It is an automatic behaviour that is hard-wired into our brains,

….. and most of the time, we are not even consciously aware of the fact that we are doing it.

Walk into a room in which you had no reason to expect any of your immediate family to be present, and if several of your close relatives happen to be there, I expect that you will have noticed them all within a few seconds (rather than a few minutes). It really does not matter whether or not you “like” them.

“So i was looking at him …” … you looked at everybody else as well …

“… and then he noticed and looked over at me …” … no, he looked at everyone, as well as you, - when he looked at you, he noticed that you were already staring at him …

“… so we locked eyes for like 4 seconds. …” … four seconds? - maybe, maybe not. You are now recalling a specific mundane incident that may have only been a fraction of that time, but you now habitually refer to it as “… like 4 seconds …”.

“… He did not look away until i did …” most guys will not readily allow someone (male or female), who they consider to be physically inferior, to “stare them down”. It is a sort of “macho” thing that they do - they do it to their friends for fun in “staring-out competitions”, they do it to their adversaries in amicable sporting events, and they do it people they dislike or consider to be their enemies.

The incident tells US even less than it tells you: about HIS attitude to YOU.

Your reiterated descriptions tell US a lot more about YOUR attitude to him, than you seem willing to tell HIM directly.