I was thinking maybe a big sign that says "Evacuate" at the top of the screen. When trolls attack, a mod could activate it so that it would flash colorfully and everyone would get off the board.
------ Shoo-dono said that posting threads would just make the trolls post more, so making a sign would probably be the same, but at least the other people on the board (save the poor mods...) wouldn't see anything.
Another idea would be for the mods to kick everyone off the website until everything is cleaned up. (Can they do that, mewonders???)