Archive/parquetTrainingValidationSet/original_launch.txt

(socmapping) vfourel@g058:/lustre/home/vfourel/SOCProject/SOCmapping/balancedDataset$ python nonRandomVal_v2.py --use-gpu
Using device: cuda
/lustre/home/vfourel/SOCProject/SOCmapping/balancedDataset/dataframe_loader.py:301: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
  dataframe['survey_date'] = pd.to_datetime(dataframe['survey_date'])
Initial shape: (30451, 7)
Final filtered shape: (16514, 7)
Loaded 16514 rows
/home/vfourel/miniforge3/envs/socmapping/lib/python3.8/site-packages/scipy/stats/_continuous_distns.py:709: RuntimeWarning: invalid value encountered in sqrt
  sk = 2*(b-a)*np.sqrt(a + b + 1) / (a + b + 2) / np.sqrt(a*b)
/home/vfourel/miniforge3/envs/socmapping/lib/python3.8/site-packages/scipy/optimize/_minpack_py.py:178: RuntimeWarning: The iteration is not making good progress, as measured by the 
  improvement from the last ten iterations.
  warnings.warn(msg, RuntimeWarning)
/home/vfourel/miniforge3/envs/socmapping/lib/python3.8/site-packages/scipy/stats/_distn_infrastructure.py:2789: RuntimeWarning: invalid value encountered in scalar multiply
  Lhat = muhat - Shat*mu
Best fitting distribution: Inverse Gamma
Parameters: {'a': 3.5093212085018015, 'loc': -0.2207140712018134, 'scale': 55.73445932737795}
KS statistic: 0.0566
Flipping 1917 points (distance < 1.7 km)
Validation set size 0.39% < 10.0%. Increasing ratio.
Flipping 2253 points (distance < 1.7 km)
Validation set size 0.35% < 10.0%. Increasing ratio.
Flipping 2550 points (distance < 1.7 km)
Validation set size 0.56% < 10.0%. Increasing ratio.
Flipping 2885 points (distance < 1.7 km)
Validation set size 0.53% < 10.0%. Increasing ratio.
Flipping 3191 points (distance < 1.7 km)
Validation set size 0.67% < 10.0%. Increasing ratio.
Flipping 3486 points (distance < 1.7 km)
Validation set size 0.89% < 10.0%. Increasing ratio.
Flipping 3813 points (distance < 1.7 km)
Validation set size 0.91% < 10.0%. Increasing ratio.
Flipping 4131 points (distance < 1.7 km)
Validation set size 0.98% < 10.0%. Increasing ratio.
Flipping 4444 points (distance < 1.7 km)
Validation set size 1.08% < 10.0%. Increasing ratio.
Flipping 4763 points (distance < 1.7 km)
Validation set size 1.16% < 10.0%. Increasing ratio.
Flipping 5058 points (distance < 1.7 km)
Validation set size 1.37% < 10.0%. Increasing ratio.
Flipping 5328 points (distance < 1.7 km)
Validation set size 1.73% < 10.0%. Increasing ratio.
Flipping 5636 points (distance < 1.7 km)
Validation set size 1.87% < 10.0%. Increasing ratio.
Flipping 5922 points (distance < 1.7 km)
Validation set size 2.14% < 10.0%. Increasing ratio.
Flipping 6218 points (distance < 1.7 km)
Validation set size 2.34% < 10.0%. Increasing ratio.
Flipping 6472 points (distance < 1.7 km)
Validation set size 2.80% < 10.0%. Increasing ratio.
Flipping 6703 points (distance < 1.7 km)
Validation set size 3.41% < 10.0%. Increasing ratio.
Flipping 7013 points (distance < 1.7 km)
Validation set size 3.53% < 10.0%. Increasing ratio.
Flipping 7196 points (distance < 1.7 km)
Validation set size 4.42% < 10.0%. Increasing ratio.
Flipping 7486 points (distance < 1.7 km)
Validation set size 4.67% < 10.0%. Increasing ratio.
Flipping 7599 points (distance < 1.7 km)
Validation set size 5.98% < 10.0%. Increasing ratio.
Flipping 7820 points (distance < 1.7 km)
Validation set size 6.64% < 10.0%. Increasing ratio.
Flipping 7937 points (distance < 1.7 km)
Validation set size 7.93% < 10.0%. Increasing ratio.
Flipping 8111 points (distance < 1.7 km)
Validation set size 8.88% < 10.0%. Increasing ratio.
Flipping 8210 points (distance < 1.7 km)
Full dataset size: 16514
Final validation set: 1698 (10.28%)
Final training set: 14816
Minimum distance (km): 1.43
Validation OC distribution matched to Inverse Gamma with parameters: {'a': 3.5093212085018015, 'loc': -0.2207140712018134, 'scale': 55.73445932737795}