We struggled with this for days, no exaggeration! And I googled like crazy, but never got an answer, and only accidentally stumbled onto the solution in the end.
The problem was that when I set up my phpmailer (5.1) to send mails using my gmail account, via google smtp (smtp.google.com), the mailer program worked. But if I tried other SMTP servers, I got the following error:
fsockopen() [function.fsockopen]: unable to connect to mail.mydomain.com:25 (Connection refused)