Is this a crazy home building idea?

Yeah but modular homes are sometimes cheaper to buy than build and you cand also have them build to suit your needs. If you really wanted to do it you better be a very good architech and engineer.