The song is an Irish folk song called Carrickfergus, named after a large town in County Antrim, known locally as Carrick and famed for its Norman Castle, Carrickfergus Castle.
The version playing at the end of the episode, and over the end credits, is performed by Loudon Wainwright III.