The first half, roughly, is history of mathematics, looking at some of the ways mathematicians and physicists have been influenced and inspired by a desire to understand music. The Pythagoreans developed the basic theory of strings, the Enlightenment brought studies of beats and overtones, the mid-18th century saw a "Great String Debate" over exactly what happens when a plucked string vibrates, and the 19th century brought the idea of decomposition into sine waves and an explanation of how our ears can separate a compound tone into components.
Maor suggests that composers and musicians have largely ignored all of this. "In 1877 Lord Raleigh published his The Theory of Sound, a two-volume work of over a thousand pages, the definitive treatise on acoustics up to that date. ... It was never intended, of course, to have a direct influence on music, and indeed it hasn't." But he ends with the introduction of the equal-tempered scale, "perhaps the single greatest gift of mathematics to music". And he touches on two key engineering contributions: the tuning fork and the metronome.
There follow some attempts to explain some basic ideas of music. "Rhythm, Meter, and Metric" is a fairly general introduction to musical time signatures, with the only mathematical connection a tenuous link at the end to Riemann's idea of a local metric: "In my mind, Stravinsky's abrupt meter changes in the Rite bear a striking conceptual similarity to the variable metric of a warped, distorted surface." And "Frames of Reference" explains the basics of keys and the tonal system. This includes some musical notation for illustrative classical passages, but can be read without an understanding of that.
Chapters "Relativistic Music" and "Aftermath" draw a parallel between Einstein and Schoenberg, and between relativity and atonal music — even if, as Maor confesses, "There is no hard evidence to suggest [that] Schoenberg's music was influenced by Einstein's theory of relativity". Relativity has been a huge success; serial music not so much.
There are a lot of digressions in Music by the Numbers. Some are in the main text — a page on Mersenne primes, for example, and lots of one or two paragraph micro-biographies — but there are also some explicitly separated "sidebars" — on the nomenclature used for tones or notes, on the slinky, on the debate over whether key relations have intrinsic emotional meaning or not, and on a novel musical instrument constructed on a logarithmic spiral.
There's nothing too involved in any of this, but even those with a background in both mathematics and music are likely to find something new in it. And it's a lot of fun.