How can I "comment out" files in the Startup folder in Windows?

A simple solution: Right-click the shortcut, select Properties, and in the General tab set the shortcut to Hidden, then click OK.

I have just tested it, and it works - a hidden shortcut is not used when booting.

The folder I did it on was
C:\Users\USER-NAME\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup.

To undo, in order to see in Explorer the hidden files, you need in View tab > Options button > View tab to set "Show hidden f‌iles, folders and drives".

Just use the standard startup configuration tools.

This used to be handled by msconfig, which used to have a Startup tab where you could uncheck individual startup items to disable them. This goes way back to Windows XP at least (perhaps even 95 or 98). However, as @Scott pointed out in the comments, running msconfig requires administrator rights.

In Windows 8 and later, this got moved to the Task Manager which got a Startup tab for the same purpose. This move also made it available for non-administrators.

Disabling an item here doesn't alter the link in the Startup folder in any way. It just configures Windows not to run it (via the registry).

I'm adding this answer for historic reasons. This was an "unofficial convention" often followed in the DOS/Windows 3.x days and for some time afterwards (for all sorts of "keeping files" but having them "invisible" - not only the startup menu).

Overall idea: since Windows uses the file extension(1) to determine how a file should be run(2), changing this extension will prevent the file from being "seen" by the application responsible for that file type(3).

(1) = this is the part of the filename after the last . (period). Up until Windows '95 this was limited to 3 characters. The extension is now hidden by default and e.g. File Explorer's settings need to be changed to show it. (Startup files have a .LNK ("link") extension.)

(2) = or more technically: which program/application/executable should take a file of this extension to run with it as input.

(3) = Programs/applications/executables are associated with files of a certain type, as determined from their extension, under Settings: On my Windows 10 system, I open the Start menu (press the Windows key) then type "assoc" and choose "Choose default applications by file type". On the resulting dialog you will see the extension in the left column and the application in the right column. Still works though.

Caveats:

(a) One will need to change settings, or work in the command line window, to see the files' extensions.

(b) This solution involves changing the filenames (and potentially changing them back, once done).

Convention:

The "convention" was to create a backup file, by replacing the file's extension with .BAK (for the historical 3-character limit). Since longer filenames have been introduced on Windows, it is probably safer to preserve the original extension and simply appending .BAK (which makes it easier to remove later on without guesswork).

Sometimes it is helpful to also add a timestamp, especially when creating multiple versions of the same file, e.g. .BAK-20211022.

Longer file name limits obviously allow one to use longer extensions, e.g. .Backup-31-May-2021, but lazy typists try to stay as short as possible :-). It is nice to stay consistent though. Also remember that the ISO date format of YYYYMMDD sorts nicely with an alphabetical sort, as may be done in File Explorer or with DIR.

When I run NirSoft WhatInStartup, right-click a Startup folder item and select "Disable" it will create, inside the Startup folder, a folder named ~Disabled with the hidden attribute set.

I did not actually reboot my machine to verify this but I can't imagine NirSoft would have implemented it this way if Windows then actually opened that folder on startup.