The builders have got to start building,” Federal Housing Finance Agency director Bill Pulte said at a Friday conference.