If you go that route, and assuming you’re in the US, I’d recommend looking for a government civilian job rather than a contractor position. The pay will be slightly lower, but you’ll have pretty steady pay increases year-to-year, the benefits will almost certainly be better, and you’ll have better job security.
The major downside will be that you’ll likely wind up working for/with a bunch of people who are just trying to keep their heads down and coast until retirement. A major upside will be that you’ll almost certainly be able to retire comfortably.