You are almost correct in speculating that the Romans calculated with an abacus. This was probably true for most people in ordinary life. Keep in mind that the abacus was more of a concept than an artifact -- anyone with something that kept lines fairly straight and a few pebbles ("calcluli") could (and probably did) make an abacus on demand. This is why we don't seem to have many abaci left. Probably there are little pebbles all over the Mediterranean (and maybe elsewhere) which were once used as stones in an abacus, then discarded. There is evidence of certain kinds of tokens with hand-signals for numbers on them; these might have been gaming pieces or might have been specialized for fancy abacus-use, such as might have been the case of a rich merchant.
Probably there were made ruled boards or cloths on which the stones could be moved conveniently and quickly; the little hand-held calculators which are preserved are probably miniature versions. The boards or cloths, of course, were very common (and continued to be during Medieval times), and (just as modern computers are used both for calculation and gaming) became handy places to play games. Several of our most common games, including chess, checkers, and backgammon, are all played on versions of these boards (the common practice of printing modern gaming boards with one side for chess/checkers and the other for backgammon is merely a reflection of the fact that both of these boards were places for calculation).
As to the details of how calculation was performed, the numerals I, X, C, and M were usually represented by pebbles on the lines, while V, L, and D were represented by pebbles in the spaces between the lines (the lines were probably called "digitus" -- modern "digit" -- and the spaces were termed "articulus" -- which originally meant the "joint" between the fingers; the terminology survived in Medieval works on arithmetic). The original versions of Roman numerals exactly reflect the usage of the abacus. Probably even the apparently late and fancy version which uses IV for 4, IX for 9, etc., reflects a specialized placement of stones on or between the lines.
Used in this way, the abacus and Roman numerals were very efficient for ordinary commercial work, and could easily be adapted to handle fractions (the surviving examples show how -- it's really very simple, just adding lines on one side and setting a rule for how many stones could be on each line). In fact, this kind of abacus remains the only calculator ever devised (to my knowledge) which conveniently handles arithmetic in which each symbol in a number can be in a different base.
The classical Roman word for what we call today by the term "abacus" was "tabula", and the counters which moved on it were called "calculi". Obviously the words calculus and table (in modern usage) somewhat reflect the ancient association of these terms with mathematics.
For fancy calculations, such as any kind of multiplication or division which could not be accomplished quickly by repeated addition (or subtraction, as the case may be) ancient people probably went to a specialist who used tables. We still have many of these tables, particularly the ones preserved in Ptolemy's works and in the many Medieval copies of the "Calculus" of Victorius of Aquitaine.
The term "abacus" in ancient times referred to a flat slab (presumably of marble) which was used as a kind of pad for the many columns which were such a characteristic part of Greco-Roman civic architecture. The word is not to be connected with Semitic words for dust, etc., and did not come to be used to mean what we today call an "abacus" until late Medieval times.
The origins of the abacus are lost in antiquity; it could easily predate civilization itself. Denise Schmandt-Bessarat (sp?; I don't have my references at hand) wrote some twenty or thirty years ago about the various specialized counters or tokens used in ancient Near Eastern proto-literate societies. These might well have been used in early versions of the abacus. Particularly interesting about them is that numbers seem to have been associated with what they counted or measured. Thus in the earliest versions of the abacus one could add (using the right tokens) two goats to three goats to get five goats, but not two goats to five bushels of wheat. The pure abstraction of number seems to have been a later development, perhaps around the end of the third millennium BC