I do accept that we could define all those activities as mathematical, but before we parochially (not in a pejorative sense) do so, we might consider them basic human activities of discrimination and generalization.
We identify things "out there" based on certain characteristics and those items that have identifiable similarities get grouped together, separate from those that we can identify as "different". A lot of this depends on how fine we want to tune our instrument.
Of course, mathematical grouping is different because we are dealing with an abstraction, a conception that seems to be basic and common to human capability.
So in a sense, I can accept all the "what ifs" proposed and have to agree with Ed's final "... it's not that simple and intertwined..." but I'm not sure I can leave it at that. If there is an intertwining that we can identify and build on to capture and incorporate the literate capability into a useful mechanism for improving the teaching and learning of math, then we're well on our way to being significantly better at what we do ... later, mark