Sounds like you may have ripped a tire balancing weight off driving those lumber roads which wouldn't be noticeable a lower speeds.
On solid front axles a worn track bar bushing issue will cause "death wobble". I'm not familiar enough with the Toyota IFS to point you in the right direction there.