Wear colours if you want, you'll get called a soft prick/southern wanker/gloryhunting *word censored* but i'm sure you can handle it. That said, they are a scummy bunch with not much between the ears so you can save yourself a lot of hassle if you don't bother.
Saw a group of about 4/5 EDL types giving grief to a Gooner lad of about 12 last season - "Little twat, you don't even live near Arsenal do you? Fucking gloryhunter!" I did point out that he couldn't really be classed as a gloryhunter considering we haven't won anything for 7 years and was told in no uncertain terms to "fuck off"