Okay, so there are two options:
1. You either recently changed to the math major, after have started out as premed/bio, etc.
2. You're not telling the truth, or have the worst advisor/faculty known to college education.
3. I'm sorely confused, either you aren't in a full Euro/US uni, or something, idk.
How in the world are you a math major but have only taken linear algebra and calculus for scientists, and trig based physics? Generally speaking, no "calculus course" is proof based unless your at a top school, in which case even then it isn't comprehensive. That's not the point of calculus in the current curriculum setup.
Regardless of which one it is, you need an overhaul in perspective on all subject areas related to math/physics.
Some more questions because this is bewildering:
1. Am I correct in assuming you havent gotten past chapter 2 or 3 in spivak?
2. What maths did you have in high school?
3. If you are in a 4-year college, please either tell me which one, or give an outline of what math and science courses you are in, and which semester you took them in.
I dont mean to come across as affrontive or autist investigator, but either your advisor does not care about your education at all, you arent being completely honest, or you just so happen to be extremely misguided, either by unfortunate circumstances or by Veeky Forums.
Either way, you have quite the road ahead of you. I'm a senior math/physics double major at a meh-tier university, but I'm in good shape. I don't pretend to know everything, but something in your story is not adding up. You started this thread looking for a book, but based on your responses, you will more likely need a place to start and direction.
If I am just over analyzing, the short answer to the semester off/lazy semester question is it depends on whether you can afford it scholarship or personal funds wise.