v0.74.0 | 21 Apr 2024 Additions Added: new function to remove rows that do not have timestamp info (NaT) (diive.core.times.times.remove_rows_nat and diive.core.times.times.TimestampSanitizer) Added: new settings VARNAMES_ROW and VARUNITS_ROW in filetypes YAML files, allows better and more specific configuration when reading data files (diive/configs/filetypes) Added: many (small) example data files for various filetypes, e.g. ETH-RECORD-TOA5-CSVGZ-20HZ Added: new optional check in TimestampSanitizer that compares the detected time resolution of a time series with the nominal (expected) time resolution. Runs automatically when reading files with ReadFileType, in which case the FREQUENCY from the filetype configs is used as the nominal time resolution. (diive.core.times.times.TimestampSanitizer, diive.core.io.filereader.ReadFileType) Added: application of TimestampSanitizer after inserting a timestamp and setting it as index with function insert_timestamp, this makes sure the freq/freqstr info is available for the new timestamp index (diive.core.times.times.insert_timestamp) Notebooks General: Ran all notebook examples to make sure they work with this version of diive Added: new notebook for reading EddyPro fluxnet output file with DataFileReader parameters (notebooks/ReadFiles/Read_single_EddyPro_fluxnet_output_file_with_DataFileReader.ipynb) Added: new notebook for reading EddyPro fluxnet output file with ReadFileType and pre-defined filetype EDDYPRO-FLUXNET-CSV-30MIN (notebooks/ReadFiles/Read_single_EddyPro_fluxnet_output_file_with_ReadFileType.ipynb) Added: new notebook for reading multiple EddyPro fluxnet output files with MultiDataFileReader and pre-defined filetype EDDYPRO-FLUXNET-CSV-30MIN (notebooks/ReadFiles/Read_multiple_EddyPro_fluxnet_output_files_with_MultiDataFileReader.ipynb) Changes Renamed: function get_len_header to parse_header(diive.core.dfun.frames.parse_header) Renamed: exampledata files (diive/configs/exampledata) Renamed: filetypes YAML files to always include the file extension in the file name (diive/configs/filetypes) Reduced: file size for most example data files Tests Added: various test cases for loading filetypes (tests/test_loaddata.py) Added: test case for loading and merging multiple files (tests.test_loaddata.TestLoadFiletypes.test_load_exampledata_multiple_EDDYPRO_FLUXNET_CSV_30MIN) Added: test case for reading EddyPro fluxnet output file with DataFileReader parameters (tests.test_loaddata.TestLoadFiletypes.test_load_exampledata_EDDYPRO_FLUXNET_CSV_30MIN_datafilereader_parameters) Added: test case for resampling series to 30MIN time resolution (tests.test_time.TestTime.test_resampling_to_30MIN) Added: test case for inserting timestamp with a different convention (middle, start, end) (tests.test_time.TestTime.test_insert_timestamp) Added: test case for inserting timestamp as index (tests.test_time.TestTime.test_insert_timestamp_as_index) Bugfixes Fixed: bug in class DetectFrequency when inferred frequency is None (diive.core.times.times.DetectFrequency) Fixed: bug in class DetectFrequency where pd.Timedelta() would crash if the input frequency does not have a number. Timedelta does not accept e.g. the frequency string min for minutely time resolution, even though e.g. pd.infer_freq() outputs min for data in 1-minute time resolution. TimeDelta requires a number, in this case 1min. Results from infer_freq() are now checked if they contain a number and if not, 1 is added at the beginning of the frequency string. (diive.core.times.times.DetectFrequency) Fixed: bug in notebook WindDirectionOffset, related to frequency detection during heatmap plotting Fixed: bug in TimestampSanitizer where the script would crash if the timestamp contained an element that could not be converted to datetime, e.g., when there is a string mixed in with the regular timestamps. Data rows with invalid timestamps are now parsed as NaT by using errors='coerce' in pd.to_datetime(data.index, errors='coerce'). (diive.core.times.times.convert_timestamp_to_datetime and diive.core.times.times.TimestampSanitizer) Fixed: bug when plotting heatmap (diive.core.plotting.heatmap_datetime.HeatmapDateTime) What's Changed Update read csv and notebooks by @holukas in https://github.com/holukas/diive/pull/93 Added new and updated test cases by @holukas in https://github.com/holukas/diive/pull/94 Full Changelog: https://github.com/holukas/diive/compare/v0.73.0...v0.74.0