How do I recover the password of a RAR file on Mac OS X?

You can bypass the hash extraction altogether and use cRARk instead. cRARk is a freeware command-line RAR password cracking utility available for Windows, Mac and Linux. It is also designed to work with CUDA so you may want to take advantage of that if you have a powerful GPU.

One caveat is that it will take very long to crack an archive if you know nothing about the password, and it is > 6 characters in length. If you do know a small detail about the password, such as the approximate number of characters, it allows you to input that as a switch to dramatically shorten cracking time. Even though cRARk is one of the fastest RAR crackers out there using extremely optimized MMX & SSE code, this holds true for any brute force application. When cracking longer passwords, it will take a substantially longer amount of time due to the myriad of possibilities to try.

Here's a sample run of cracking the password 'john':

C:\>crark.exe -c -l4 -g4 Chap7.rar
     cRARk 3.2d (CUDA enabled) Freeware
    Copyright 1995-2001, 2006-09 by P. Semjanov,
         http://www.crark.net
     portions (c) 1993-2005 Eugene Roshal
 (c) PSW-soft Password Cracking Library PCL v. 2.0d by P. Semjanov


Testing archive Chap7.rar : version 2.9
Testing     Chap7.rtf
Choosing best crypto functions.................................................
Chosen: ASM  (Prescott/AMD), SSE2 (P4/Core 2) (-f1114)
Ticks per password expected = 40438280, theoretical = 27000000, CPU rate = 0.67

Processing line 56 of password definition file...
Testing 4-chars passwords ...
ckdk
Passwords tested = 42000 (time = 3:45.00, rate = 186 p/s)
elka
Passwords tested = 78000 (time = 6:58.99, rate = 186 p/s)
john - CRC OK
In hex (PCL style): \6A \6F \68 \6E
Passwords tested = 167844 (time = 15:02.38, rate = 186 p/s)
Total     tested = 167844, slow tests = 20914

Not too shabby ;)