Confused on dashes, colons, semi-colons! HELP?

Both sentences can be correct with the use of a comma. A colon is normally used to precede a list or to explain an independent clause. Therefore, your sentence "The spell was broken, and we both glowered at our friend. Let me rephrase: His friend, my nightmare." does not function this way, let me rephrase is not an independent clause. On a side note, I find the use of "don't" and "I've" to be rather lazy; you are a writer, have some tact.