Hence my query is why in case 1 x*y is not replaced by b and a whereas in case 2 it got replaced ?
Mathematica understands xy as a single new symbol, and, therefore, it does not interpret it as a product. If you want it to be multiplication you should make a space between x and y. Then it works. Compare: this is your function