For standard office use, I think you have enough processor. As for RAM, I think you need to consider adding more.
To figure out how much ram you need add up the working set of the apps for each user.
Outlook (32 megs min)
IE ~12 megs a copy.
Word 10 megs min
Think 64 megs per active user.
Plus at least 128 for the sytem.