The following is an account about what sets are, first I'll write the exposition of this base theory in brief, then I'll discuss some related issues.
Language: FOL + P, Rp
P stands for "is part of"
Rp stands for "represents"
Axioms: Identity theory axioms +
I. Part-hood: P partially orders the universe.
ll. Supplementation: x P y & ~ y P x -> Exist z. z P y & ~ x P z.
Def.) atom(x) <-> for all y. y P x -> x P y
Def.) x atom of y <-> atom(x) & x P y.
Def.) c is a collection of atoms iff for all y. y P c -> Exist z. z atom of y.
Def.) c is atomless <-> ~ Exist x. x atom of c
lll. Representation: x Rp c & y Rp d -> (x=y<->c=d)
lV. Representatives: x Rp c -> atom(x)
V. Null: Exist! x. (Exist c. x Rp c & c is atomless).
A Set is an atom that uniquely represents a collection of atoms or absence of atoms.
Def.) Set(x) <-> Exist c. (c is a collection of atoms or c is atomless) & x Rp c & atom(x)
Here in this theory because of lV there is no need to mention atom(x) in the above definition.
Set membership is being an atom of a collection of atoms that is uniquely represented by an atom.
Def.) x e y iff Exist c. c is a collection of atoms & y Rp c & x atom of c & atom(y)
Here in this theory because of lV there is no need to mention atom(y) in the above definition.
Vl. Composition: if phi is a formula in which y is free but x not, then [Exist y. atom(y) & phi] -> [Exist x. x is a collection of atoms & (for all y. y atom of x <-> atom(y) & phi)] is an axiom.
Vll. Pairing: for all atoms c,d Exist x for all y. y e x <-> y=c or y=d /
This theory can interpret second order arithmetic. And I like to think of it as a base theory on top of which any stronger set theory can have its axioms added to it relativized to sets and with set membership defined as above, so for example one can add all ZFC axioms in this manner, and the result would be a theory that defines a model of ZFC, and thus proves the consistency of ZFC. Anyhow this would only be a representation of those theories in terms of different primitives, and it is justified if one think of those primitives as a more natural than membership, or if one think that it is useful to explicate the later. Moreover this method makes one see the Whole Ontology involved with set\class theories, thus the bigger picture revealed! This is not usually seen with set theories or even class theories as usually presented, here one can see the interplay between sets and classes (collections of atoms), and also one can easily add Ur-elements to this theory and still be able to discriminate it from the empty set at the same time, a simple approach is to stipulate the existence of atoms that do not represent any object. It is also very easy to explicate non well founded scenarios here in almost flawless manner. Even gross violation of Extensionality can be easily contemplated here. So most of different contexts involved with various maneuvering with set\class theories can be easily paralleled here and understood in almost naive manner.
In simple words the above approach speaks about sets as being atomic representatives of collections (or absence) of atoms, the advantage is clearly of obtaining a hierarchy of objects. Of course an atom here refers to indivisible objects with respect to relation P here, and this is just a descriptive atom-hood that depends on discourse of this theory, it doesn't mean true atoms that physically have no parts, it only means that in the discourse of this theory there is no description of proper parts of them, so for example one can add new primitive to this theory like for example the primitive "physical" and stipulate that any physical object is an atom, so a city for example would be an atom, it means it is descriptively an atom as far as the discourse of this theory is concerned, so atom-hood is a descriptive modality here. From this one can understand that a set is a way to look at a collection of atoms from atomic perspective, so the set is the atomic representative of that collection, i.e. it is what one perceives when handling a collection of atoms as one descriptive \discursive whole, this one descriptive\discursive whole is actually the atom that uniquely represents that collection of atoms, and the current methodology is meant to capture this concept.
Now from all of that it is clear that Set and Set membership are not pure mathematical concepts, they are actually reflecting a hierarchical interplay of the singular and the plural, which is at a more basic level than mathematics, it is down at the level of Logic actually, so it can be viewed as a powerful form of logic, even the added axioms to the base theory above like those of ZFC are really more general than being mathematical and even when mathematical concepts are interpreted in it still the interpretation is not completely faithful to those concepts. However this powerful logical background does provide the necessary Ontology required for mathematical objects to be secured and for their rules to be checked for consistency.
But what constitutes mathematics? Which concepts if interpreted in the above powerful kind of logic would be considered as mathematical? This proves to be a very difficult question. I'm tending to think that mathematics is nothing but "Discourse about abstract structure", where abstract structure is a kind of free standing structural universal. Anyhow I'm not sure of the later. I don't think anybody really succeeded with carrying along such concepts.