Should you power off a router before unplugging it?

No, you will not damage the hardware by unplugging it. On most of these devices the button is little more than a way to toggle power.

Even if it is, the only thing that powering down really affects is the data not written to disk, and this is a non-issue on routers.

Is it absolutely ideal to power off directly at the mains?
No.

Is it really going to do any harm?
No.

The only difference in reality is if the switch on the router itself switches the low-voltage DC rather than the 110/240v AC, but practically, it's not going to hurt anything 999 out of 1,000 times. These things are designed to survive power cuts & moderate surges.

Wall switch-off is fine.

What can be not fine is switching it on at the wall, then unplugging it before the router finishes booting up.

This can (rarely) result in the router losing saved configuration.

It is not common at all, but with the rolling blackouts ("loadshedding") that we experience here occasionally, I am seeing a fair share of otherwise good home routers losing settings due to the power coming on, stuttering, going off again, then coming on as the power get badly restored. Also seeing tons of dead refrigerator compressors, of course, the same power problem is much more deadly to them. But it is generating maybe 10% of my business as IT support person, this issue of routers losing settings.

Usually seen on the low-mid level home routers. Dlink, Netgear, Tplink, etc. Never on the real routers like Cisco or even the more upmarket home stuff like Ubiquity

I am using Eero router and there is no power on/off on it. I'd think that if powering on/off is required, such expensive and sensitive equipment would have included it in the first place. So, to answer your question, no, you don't need to. Just unplug from the mains. You will be fine. Hope it helps.