Let me preface it that this is my view of fandom. Your fandom, and view of it, may be different. Also that it includes a broad overview of fannish history, which likely includes both errors and exceptions.
Ur-fandom started around the pulp sf magazines of the 1920s and 1930s: Weird Tales, Astounding, Amazing Stories, et c. There were regional differences, but there were communication between groups by way of the fanzines, the letter columns in the pulps, and the cons started to gather larger amounts of fans together. There were no real differences due to media, as far as I can tell: radio dramas, literature, and comics were all thrown in the melting pot. The early fandom also proved to be a recruiting and growing ground for many influential sf authors and editors: Fred Pohl, Jerry Siegel, Isaac Asimov, Virginia Kidd, and Donald Wollheim are all examples of these early pro-fans.
As science fiction evolved, the pro-fans came to dominate the genre. Sure, there were plenty of other authors who didn't enter via fandom (like, from what I can gather, Robert Heinlein), but the culture they entered was one of symbiosis between science fiction and fandom. There were tensions and differences, but I imagine they all felt part of the one and same culture.
So, when the Hugos started in 1953 it can be viewed as a way of the community of fandom to honour its best and brightest. But due to the symbiosis between science fiction and fandom, and that the Hugo electorate often managed to pick very good winners, the Hugos came to be viewed as the science fiction award too. Or it was the other way around: both viewpoints are quite valid.
But back in the 60s, ur-fandom started to split, or perhaps rather create offshoots. SCA started to develop in the 60s. Media science fiction, like Star Trek and even more Star Wars, turned science fiction from popular culture to mass culture. Media fandom started to develop their own cons and traditions, partly based on their different relations to their pros (that is the actors), partly because they reached a different and larger audience. Special interests within fandom, like filking and costuming, started to gain size and momentum too, gain their own pros, and evolve away from "core" fandom. The growing also changed the landscape of the authors of science fiction: the proportion of pro-fans (as in pros who had been nurtured in fandom) grew smaller relative to authors who came from outside fandom and didn't encounter it until they were more or less established.
In the United States, the ties between these different cultures and groups are still very much present, though they are weakening. The Worldcon has world-class cosplaying, a filk program which can rival just about any filk con in quality, and so on. At the same time we have seen the rise of the mass culture cons, like DragonCon. But a digression to fandom(s) in Sweden.
(Ur-)Fandom came to Sweden in the 1950s. In the early 70s Tolkien societies evolved here from it, in many ways similar to SCA in the United States. The ties between the Tolkien societies and fandom in Sweden are still strong, and we can mingle relatively easily. However, media fandom, cosplay, LARP, and lots of other stuff were direct imports from the United States. Here the cultural differences are much larger and more profound. Partly this is because of the direct import, partly this is because Swedish fandom after the disastrous feuds of the 80s closed in on itself and very much focused on the core of discussing science fiction as books.
Put another way, the splinter lines within all the various off-shoots, special fandoms, and so are much easier to see here in Sweden. But the same tendencies are very much present in the United States, I imagine.
Another thing which has happened, from the 90s forward, is that the Internet has made it much easier to set up special interest groups that can gain critical size and connectivity. Baen's Bar is one early such example, but there are many more nowadays.
So which of these disparate groups do the label "fandom" belong to nowadays? All of them. However, there is a tendency to use the word "fandom" as a shorthand for "the specific fannish group that I happen to be a member of". I believe this is especially true within "core" fandom, the one that evolved around the pulp magazines in the 20s and 30s, with a primary interest in written science fiction. Historically, I think that movement can claim having first dibs on the label, but it helps to remember that fandom nowadays is much bigger and diverse than "core" fandom is.
As for the Hugos, that's much easier. They belong to Worldcon, which still very much is a part of "core" fandom.
The duality of the Hugo award, of being for the best in science fiction (and fantasy), and for "core" fandom to honour its best and brightest, has started to pull in different ways, and I think also helps to explain the way the Hugos nowadays feels less representative of the entire field of science fiction. That tension is also something which I think lies behind a lot of how the Puppies view the award, or rather, I imagine many of them see one part of it ("best in science fiction") but not the other ("best and brightest in core fandom").
Personally, I believe that the status of the Hugos as "the best in science fiction" is a reflection of the success in its mission of being an award for "the best and brightest in core fandom", and that removing the latter would require a permanent split away from the historical Hugo award. But it's a tension to be aware of.