How bad is it to not wear your retainer at a certain time?

You should follow your orthodontist's instructions. Not everyone will be told the same thing regarding how long to wear the retainer, but the orthodontist is the one who KNOWS how much you should wear it. I'm sure you are excited to have the braces off...don't spoil that perfect smile by 'cutting out' on the retainer. :)