Ugh, this is such a hard question, but I will try to give actual reasons while reducing my mental farts:
1.Generally speaking, you should always go for Physics over Math because it is bigger, it requires more intelligence. One can be a mathematician but an awful physicist, if you pick high enough standards, or one can be a physicist who is actually good at math by definition, and by the same standards.
2.You can check IQ differences as well.
3.Math is not a science in the first place, pursuing it pretty much makes you an unscientific philosopher, while pursuing Physics makes you a scientific philosopher. Side note: an exclusive interest in math is unnatural, you will eventually pursue Philosophy if you are not mentally ill, (because math is part of philosophy anyway but whatever)
4.It is cool to learn about advanced math on your own, instead of having to be forcefed with boring bullshit just get a job at a fast food and not starve in the streets like a loser
5. This one requires some thought: All human math is contained in physics. Our brains work inside physics, so just like an hypothetical scenario of a plane falling is inside Physics, all possible brain events are inside Physics. Physics also includes all possible models of reality, even when we rule them out, so in all cases infinite models are being considered in order to advance Physics. Physics corrects our abstraction, you can think of math containing Physics only if you are dumb enough to not realize that you are thinking such thing within Physics in the such place. Not only that, if you think that math contains Physics, you have a very loose definition of Math.
6. Trivia: Physics contains Chemistry and Biology(math doesn't), it contains the reason for our specific languages, and very interestingly, it contains the Turing machines that we popularly call "computers", which are essential to modern math. For fun: en.wikipedia.org/wiki/Curry–Howard_correspondence