It seems the most important question is the following: is this off-roading desire a temporary itch that will disappear in a few weeks, or is it something that will become a hobbie for the next several years, and will create fun for both you and the whole family?
If it's the latter, then you have no choice. Go for it. But if you're anything like me, you can't be completely sure it's not the former.