OpenSSL hash function for generating AES key

Fairly sure it's an SHA1 digest algorithm but in all honesty I can't say with 100% certainty.

And who would have thought that something designed to increase obtuseness would have obtuse instructions ;)

EDIT: This may not be helpful in your circumstances but I guess you could always know by doing

openssl enc -d -a -md sha1 -aes-256-cbc -nosalt -p 

It's a concatenation of two MD5 hashes.

It's derived like this:

128bit_Key = MD5(Passphrase + Salt)
256bit_Key = 128bit_Key + MD5(128bit_Key + Passphrase + Salt)

You can check this by doing:

$ echo Testing > file
$ openssl enc -aes-256-cbc -p -in file -out file.aes -salt
: enter aes-256-cbc encryption password: abc
: Verifying - enter aes-256-cbc encryption password: abc
: salt=3025373CA0530C93
: key=E165475C6D8B9DD0B696EE2A37D7176DFDF4D7B510406648E70BAE8E80493E5E
: iv =B030394C16C76C7A94DC22FDDB6B0744
$ perl -e 'print pack "H*", "3025373CA0530C93"' > salt
$ echo -n abc > passphrase
$ cat passphrase > key.128.tmp
$ cat salt >> key.128.tmp
$ md5sum key.128.tmp 
: e165475c6d8b9dd0b696ee2a37d7176d  key.128.tmp
$ perl -e 'print pack "H*", "e165475c6d8b9dd0b696ee2a37d7176d"' > key.128
$ cat key.128 > key.256.tmp
$ cat passphrase >> key.256.tmp
$ cat salt >> key.256.tmp
$ md5sum key.256.tmp 
: fdf4d7b510406648e70bae8e80493e5e  key.256.tmp

Notice how both MD5's of 'key.128.tmp' and 'key.256.tmp' concatenated together form the same key as output at the initial command.

OpenSSL uses AES with SHA1.

If you wish to examine better-written source than OpenSSL, have a look at the article
C++ class that interfaces to OpenSSL ciphers.

The article includes very simple source code that :

allows you to encrypt and decrypt files or strings using the OpenSSL AES-256-CBC cipher and SHA1 digest algorithms. It is interoperable with the openssl command line tool which makes it a good introduction to using OpenSSL for ciphers.