Show corresponding icons for each monitor on task bar

In the settings app under the taskbar options, set the option Show Taskbar button under Personalization->Taskbar to "taskbar where window is open"

I have personally tried two options:

DisplayFusion

DisplayFusion is a paid program with lots of features for multiple monitors on Windows. https://www.displayfusion.com/

Dual Monitor Taskbar

Dual Monitor Taskbar is a freeware program that is much simpler and might do the trick - it splits the programs per monitor. https://sourceforge.net/projects/dualmonitortb/

if you change the "Show taskbar Buttons on" dropdown to "Main taskbar and taskbar where window is open" this will do close to what your asking, just that your main monitor will contain all icons. This is found in multiple displays under taskbar settings