fix: robots-txt not loaded error conditions

fix: robots-txt location

fix: robots-txt cache location

cleanup: use local project code as much as possible

restructure: nolonger a firebase application (#1160)