Have your fifth grade students used any Investigations units at previous grade levels?
I have used the grade five units with students who have never used Investigations before--and it is very challenging for exactly the reasons you gave. Many of the activities and routines rely on the habits-of-mind developed at the younger grades. In fact, with fifth grade students who are "new" to this program I believe that "Mathematical Thinking at Grade 5" and "Building on Numbers You Know" are two of the most difficult units.
I had more positive student feed back from units in strands other than Number: Picturing Polygons and Containers and Cubes were a big hit!
One suggestion while you are working on the Number units is to make sure to add Ten Minute Math activities to your day, especially the activities involving geometry, data and probability (such as "Quick Images" and "What is Likely?". You'll find them in each unit right before the Student Sheets, or (even better) altogether in the book "Ten Minute Math" (from Dale Seymour).
Another suggestion is to use some of the grade 4 units. You'll notice a grade band on the top of these units indicating that they can be used in grade 4 or 5 (incase you need some justification). You may want to use "Arrays and Shares" to support the grade 5 number units. Again back to my experience, for fraction work I used the grade 4 "Different Shapes, Equal Pieces" before the grade 5 "Name that Portion.