I would have to go with Firefox here, get rid of Outlook (and of course IE) because many of the settings they both use are the same. For example if you change the print margins for somethig in IE then it will change for Outlook as well (don't ask me why... may be the developers just thought it was cool this way?) so you need to keep changing the settings. And not only that since they are some what connected you can get other problems (like William said, it will open anything only in IE).

So if possible why not give Thunderbird a try... it's newer and chances are that you are going to like it...