The problem with book 4 is that it came on the heels of the climactic book 3, yet removed many of the main characters. Right after reading about The Red Wedding, The Purple Wedding, Oberyn's death, Tywin's death, and all that... book 4 doesn't feature Arya, Tyrion, Dany, or Jon. And introduces a bunch of newbies.

After you know where everyone ends up, it's not so bad to read book 4.