How do I configure custom URL handlers on OS X?

The file you seek is ~/Library/Preferences/com.apple.LaunchServices/com.apple.launchservices.secure.plist.

It holds an array called LSHandlers, and the Dictionary children that define an LSHandlerURLScheme can be modified accordingly with the LSHandlerRole. You need to [convert this binary file][1] to edit it.

Rather than manually editing this file, you can use SwiftDefaultApps. Note that this app is a successor to RCDefaultApp, which has been unmaintained, but it also appears to have some bugs.

Update Jul 2017: RCDefaultApp and lstool no longer work on macOS 10.12 or later. We'll have to find a new solution.

Edit Mar 2018
I found SwiftDefaultApps but can't test as I'm not on High Sierra yet.

Furthering the previous answer, if you would like to do this on the command line you can use the lstool command, which is the core of RCDefaultApp, found under RCDefaultApp.prefPane/Contents/Resources/lstool.

Its usage is straightforward:

[~]$lstool --help
Usage:

lstool read [<lsscheme> [<type>]]
lstool [-n] write <lsscheme> <type> <app>
lstool [-n] setoption <lsscheme> <type> login|ignorecreator YES|NO
lstool [-n] register {<app>}
lstool [-n] unregister {<path> | <app>}
lstool apps

-n means do not make changes
<lsscheme> is one of: internet, media, url, extension, uti, mime, ostype
<app> is the path to an application or a name to be looked up

Open the file with XCode works quite easy.

Using the build in command plutil as described in the answer on https://discussions.apple.com/thread/5815759 to convert between format xml1 or binary1 works similar.

plutil -convert xml1 /file-i-wish

nano /file-i-wish & save file 

plutil -convert binary1 /file-i-wish

No need for 'alien' tools :-)

I actually wrote an application simplifying registration of custom URL protocols, if anyone is interested. It is called LinCastor (https://onflapp.github.io/blog/pages/LinCastor.html). Handlers can be defined as AppleScript or shell script (which can perl, python or what ever).