Bulls do not actually get angry at the color red; in fact, bulls are color blind to the color red. Bulls instead have a habit of charging at things that are moving.
Regardless of the color, bulls will charge at a flag if it is moving. Bullfighters also have a different colored cape from the red one, and bulls still charge at it with the same ferociousness and anger as the red one. One of the main reasons why the last cape of the matador is red is to conceal the blood spilled from the bull as the final kill is made.