away with the fairies

It's an Irish/Scots expression. In Gaelic folkloretradition there are numerous stories of humans who went off with, or visited the homes of, the Sidhe, the "fairies" of Gaelic tradition. These people would stay with the fairies for what seemed like only a few hours, but when they returned to the everyday world they would find that many years had passed.