Do I need to have a passphrase for my SSH RSA key?

As you know, the advantage that the passphrase gives you is that if someone is able to read your private key, they are 'unable' to use it.

If someone is able to access that private key, you should take it for granted that they have access(ed)/compromised whatever machines are set up with the public key. Things like .bash_history or .ssh/config only make this easier, even if your .ssh/known_hosts is obfuscated.

Not having a password on your key isn't the end of the world, here are 3 ideas to try and help you secure yourself a little better despite this. (The biggie is the second, read that if nothing else)

1. Don't just use the same key across all machines and users. Generate each user on each machine (that needs to do this kind of thing) its own key pair. This will let you keep fine grained control on what is able to ssh where.

2. When adding the key to your authorized_keys file, you can lock it down to only be able to run a specific command, or use it only from a specific host.

   See man ssh and search for command= and from=

   The syntax is something like:

   from="1.2.3.4",command="/path/to/executable argument" ssh-rsa key name

   i.e. pop 'rsync' in there and only 'rsync' could be called by your key, and only from the IP address 1.2.3.4. Multiple IPs can be separated by ,. Host names are also supported.

3. Another thing that springs to mind is the 'AllowUser' directive in your sshd_config

   AllowUsers

   This keyword can be followed by a list of user name patterns, separated by spaces. If specified, login is allowed only for user names that match one of the patterns. '*' and '?' can be used as wildcards in the patterns. Only user names are valid; a numerical user ID is not recognized. By default, login is allowed for all users. If the pattern takes the form USER@HOST then USER and HOST are separately checked, restricting logins to particular users from particular hosts.

   That basically ensures that the user can only log in from a certain location. (although it accepts wildcards too) Not going to solve all of your problems but it'll at least make it harder for others.

You can use something like keychain to make having a passphrase less painful. This is slightly more secure than using a passwordless login, and can be used in combination with the other answers here. PriceChild's answer was pretty good.