uhm... it is sort of hard to find (though you once signed a map to it). Looking at the map now, it is... near Tower Bridge. I remember a very exciting period with Faerlan the day after the first London signing in which we hunted around for it (:
It isn't a free-entry sort of museum though, and the exhibits change fairly regularly (though i went to one exhibit twice because I loved it (: )