Probably because it's not really a "lifestyle" bluffing game. Most people in the BGG community want a game like BSG or Avalon or Secret Hitler that you can really game the system of hard, build up conventions among you, learn your friends, and struggle your way to a skilled victory.
Secrets is more of a ONUW, Mascarade, Mafia de Cuba kinda bluffing game. It's quirky and bananas and while you can totally game it, it presents itself as far more casual and silly than that - a game where you can't read an opponent because they don't even know who they are, a game where decisions are made purely out of fun-factor sometimes.
That said, I said it presents that way, not that it actually is that way. I think it looks far sillier than it is. If you game this game, I am of the opinion that you will be just as capable at dominating via skill as you would be in any "serious" game.
Also, it's still new. The ratings always start low.
I only played with 6 and 7 players, about equal amounts of each. I think I liked 7 better, but only because all odd-numbered counts get two Hippies instead of 1, and I appreciate what the role of Hippie does for this game tremendously.
(If one ends the game with less points than any other person at the whole table, they instantly win alone. Always puts everyone on their toes.)
I also enjoy that 7+ players start the game by all getting to look at the center chip, because that feels empowering, starting with three data points instead of just two. It also allows the first few turns to start more confidently and give everyone a role to track when the Diplomat, who interacts with the center, is played.
Otherwise, I feel like both operated similarly enough. I don't know if it plays well at 4-5, which would certainly be valuable to know! Nor do I know higher than 7. But I think I'd intentionally go for 7 next time I play. Being able to see the center chip and getting those extra Hippies are both great, great bonuses for me.