IIRC, it's not just fey creatures. Ghosts, foreign spirits, the Grim Reaper, and the ilk seem to be drawn to him.

If you're playing Pirate England, wouldn't it make sense for him to scamper(?) about with mermaids, have chats with the Flying Dutchman, or even accidentally commandeer a ghost ship and it's crew? Sea myths, anon.