How harry was able to do magic when he feel either scared or angry?

hagrid told him this thing at their first meeting.he did it actually on dudley at zoo and on aunt marge before his 3rd year.even severus did it on petunia as shown in the pensieve(he didn't have wand at that time). but he never did any magic without his wand either at hogwarts or any other place other time no matter how much he was scared or angry.is this sort of magic can be done only on muggles or it can bec done only in muggle world or only by pre-teens wizards?