queens.data_processors package#
Data Processor.
Modules for extracting and processing data from simulation output files.
Submodules#
queens.data_processors.csv_file module#
Data processor class for csv data extraction.
- class CsvFile(file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None)[source]#
Bases:
DataProcessorClass for extracting data from csv files.
- use_cols_lst#
List with column numbers that should be read-in.
- Type:
lst
- filter_type#
Filter type to use.
- Type:
str
- header_row#
Integer that determines which csv-row contains labels/headers of the columns. Default is ‘None’, meaning no header used.
- Type:
int
- skip_rows#
Number of rows that should be skipped to be read-in in csv file.
- Type:
int
- index_column#
Column to use as the row labels of the DataFrame, either given as string name or column index.
Note: index_column=False can be used to force pandas to not use the first column as the index. index_column is used for filtering the remaining columns.
- Type:
int, str
- use_rows_lst#
In case this options is used, the list contains the indices of rows in the csv file that should be used as data.
- Type:
lst
- filter_range#
After data is selected by use_cols_lst and a filter column is specified by index_column, this option selects which data range shall be filtered by providing a minimum and maximum value pair in list format.
- Type:
lst
- filter_target_values#
Target values to filter.
- Type:
list
- filter_tol#
Tolerance for the filter range.
- Type:
float
- returned_filter_format#
Returned data format after filtering.
- Type:
str
- classmethod check_valid_filter_options(filter_options_dict)[source]#
Check valid filter input options.
- Parameters:
filter_options_dict (dict) – dictionary with filter options
- expected_filter_by_range = {'range': [1.0, 2.0], 'tolerance': 0.0, 'type': 'by_range'}#
- expected_filter_by_row_index = {'rows': [1, 2], 'type': 'by_row_index'}#
- expected_filter_by_target_values = {'target_values': [1.0, 2.0, 3.0], 'tolerance': 0.0, 'type': 'by_target_values'}#
- expected_filter_entire_file = {'type': 'entire_file'}#
queens.data_processors.ensight_file module#
Data processor class for reading vtk-ensight data.
- class EnsightFile(file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None, external_geometry=None, experimental_data_reader=None)[source]#
Bases:
DataProcessorClass for data-processing ensight output.
- experimental_data#
dict with experimental data.
- Type:
dict
- coordinates_label_experimental#
List of (spatial) coordinate labels of the experimental data set.
- Type:
lst
- time_label_experimental#
Time label of the experimental data set.
- Type:
str
- external_geometry#
QUEENS external geometry object.
- Type:
obj
- target_time_lst#
Target time list for the ensight data, meaning time for which the simulation state should be extracted.
- Type:
lst
- time_tol#
Tolerance for the target time extraction.
- Type:
float
- vtk_field_label#
Label defining which physical field should be extraced from the vtk data.
- Type:
str
- vtk_field_components#
List with vector components that should be extracted from the field.
- Type:
lst
- vtk_array_type#
Type of vtk array (e.g. point_array).
- Type:
str
- geometric_target#
List with information about specific geometric target in vtk data (This might be dependent on the simulation software that generated the vtk file).
- Type:
lst
- geometric_set_data#
Dictionary describing the topology of the geometry
- Type:
dict
- filter_and_manipulate_raw_data(raw_data)[source]#
Filter the ensight raw data for desired time steps.
- Parameters:
raw_data (obj) – Raw data from file.
- Returns:
processed_data (np.array) – Cleaned, filtered or manipulated data_processor data.
queens.data_processors.numpy_file module#
Data processor class for numpy data extraction.
- class NumpyFile(file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None)[source]#
Bases:
DataProcessorClass for extracting data from numpy binaries.
- filter_and_manipulate_raw_data(raw_data)[source]#
Filter and manipulate the raw data.
In this case we want the raw data as it is. Of course this method can implement more specific data processing in the future.
- Parameters:
raw_data (np.array) – Raw data from file.
- Returns:
processed_data (np.array) – Cleaned, filtered or manipulated data_processor data.
queens.data_processors.pvd_file module#
Data processor class for pvd data extraction.
- class PvdFile(field_name, file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None, time_steps=None, block=0, point_data=True)[source]#
Bases:
DataProcessorClass for extracting data from pvd.
- field_name#
Name of the field to extract data from
- Type:
str
- time_steps#
Considered time steps (last time step by default)
- Type:
lst
- block#
Considered block of MultiBlock data set (first block by default)
- Type:
int
- data_attribute#
‘point_data’ or ‘cell_data’
- Type:
str
queens.data_processors.txt_file module#
Data processor class for txt data extraction.
- class TxtFile(file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None, remove_logger_prefix_from_raw_data=True, logger_prefix='\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3} - queens\\.drivers\\.driver_\\d* - INFO -', max_file_size_in_mega_byte=200)[source]#
Bases:
DataProcessorClass for extracting data from txt files.
Provides basic functionality for extracting data from txt files, however the final implementation is up to the user.
- The implementation of the filter_and_manipulate_raw_data method is up to the user.
- Throws:
MemoryError: We throw a conservative MemoryError if the txt file is larger than 200 MB. This is due to the current design, which loads the entire content of the .txt file into memory.
- Potential Improvement:
Use a generator for reading the content of the file in chunks. This however requires a more advanced logic with the possibility to nest functions calls in a flexible way.
- filter_and_manipulate_raw_data(raw_data)[source]#
Filter the raw data from the txt file.
The TxtFile class provides some basic filtering functionality, however it is up to the user to define the specifics of how the raw data should be filtered.
- Parameters:
raw_data (lst) – List of strings Raw data from file.
- Returns:
To be implemented by user.