queens.data_processor package#

Data Processor.

Modules for extracting and processing data from simulation output files.

Submodules#

queens.data_processor.data_processor module#

Module for data processing of simulation results.

class DataProcessor(file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None)[source]#

Bases: object

Base class for data processing.

files_to_be_deleted_regex_lst#

List with paths to files that should be deleted. The paths can contain regex expressions. The paths are relative to the particular simulation output folder: experiment_dir/<job_id>/output/<here comes your regex> .

Type:: lst

file_options_dict#

Dictionary with read-in options for the file.

Type:: dict

file_name_identifier#

Identifier for files. The file prefix can contain BASIC regex expression and subdirectories. Examples are wildcards * or expressions like [ab].

Type:: str

abstract filter_and_manipulate_raw_data(raw_data)[source]#

Filter or clean the raw data for given criteria.

Parameters:: raw_data (obj) – Raw data from file.
Returns:: processed_data (np.array) – Cleaned, filtered or manipulated data_processor data.

get_data_from_file(base_dir_file)[source]#

Get data of interest from file.

Parameters:: base_dir_file (Path) – Path of the base directory that contains the file of interest
Returns:: processed_data (np.array) – Final data from data processor module

abstract get_raw_data_from_file(file_path)[source]#

Get the raw data from the files of interest.

Parameters:: file_path (str) – Actual path to the file of interest.
Returns:: raw_data (obj) – Raw data from file.

queens.data_processor.data_processor_csv module#

Data processor class for csv data extraction.

class DataProcessorCsv(file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None)[source]#

Bases: DataProcessor

Class for extracting data from csv files.

use_cols_lst#

List with column numbers that should be read-in.

Type:: lst

filter_type#

Filter type to use.

Type:: str

header_row#

Integer that determines which csv-row contains labels/headers of the columns. Default is ‘None’, meaning no header used.

Type:: int

skip_rows#

Number of rows that should be skipped to be read-in in csv file.

Type:: int

index_column#

Column to use as the row labels of the DataFrame, either given as string name or column index.

Note: index_column=False can be used to force pandas to not use the first column as the index. index_column is used for filtering the remaining columns.

Type:: int, str

use_rows_lst#

In case this options is used, the list contains the indices of rows in the csv file that should be used as data.

Type:: lst

filter_range#

After data is selected by use_cols_lst and a filter column is specified by index_column, this option selects which data range shall be filtered by providing a minimum and maximum value pair in list format.

Type:: lst

filter_target_values#

Target values to filter.

Type:: list

filter_tol#

Tolerance for the filter range.

Type:: float

returned_filter_format#

Returned data format after filtering.

Type:: str

classmethod check_valid_filter_options(filter_options_dict)[source]#

Check valid filter input options.

Parameters:: filter_options_dict (dict) – dictionary with filter options

expected_filter_by_range = {'range': [1.0, 2.0], 'tolerance': 0.0, 'type': 'by_range'}#

expected_filter_by_row_index = {'rows': [1, 2], 'type': 'by_row_index'}#

expected_filter_by_target_values = {'target_values': [1.0, 2.0, 3.0], 'tolerance': 0.0, 'type': 'by_target_values'}#

expected_filter_entire_file = {'type': 'entire_file'}#

filter_and_manipulate_raw_data(raw_data)[source]#

Filter the pandas data-frame based on filter type.

Parameters:: raw_data (DataFrame) – Raw data from file.
Returns:: processed_data (np.array) – Cleaned, filtered or manipulated data_processor data.

get_raw_data_from_file(file_path)[source]#

Get the raw data from the files of interest.

This method loads the desired parts of the csv file as a pandas dataframe.

Parameters:: file_path (str) – Actual path to the file of interest.
Returns:: raw_data (DataFrame) – Raw data from file.

queens.data_processor.data_processor_ensight module#

Data processor class for reading vtk-ensight data.

class DataProcessorEnsight(file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None, external_geometry=None, experimental_data_reader=None)[source]#

Bases: DataProcessor

Class for data-processing ensight output.

experimental_data#

dict with experimental data.

Type:: dict

coordinates_label_experimental#

List of (spatial) coordinate labels of the experimental data set.

Type:: lst

time_label_experimental#

Time label of the experimental data set.

Type:: str

external_geometry#

QUEENS external geometry object.

Type:: obj

target_time_lst#

Target time list for the ensight data, meaning time for which the simulation state should be extracted.

Type:: lst

time_tol#

Tolerance for the target time extraction.

Type:: float

vtk_field_label#

Label defining which physical field should be extraced from the vtk data.

Type:: str

vtk_field_components#

List with vector components that should be extracted from the field.

Type:: lst

vtk_array_type#

Type of vtk array (e.g. point_array).

Type:: str

geometric_target#

List with information about specific geometric target in vtk data (This might be dependent on the simulation software that generated the vtk file).

Type:: lst

geometric_set_data#

Dictionary describing the topology of the geometry

Type:: dict

filter_and_manipulate_raw_data(raw_data)[source]#

Filter the ensight raw data for desired time steps.

Parameters:: raw_data (obj) – Raw data from file.
Returns:: processed_data (np.array) – Cleaned, filtered or manipulated data_processor data.

get_raw_data_from_file(file_path)[source]#

Read-in EnSight files using the vtkEnsightGoldBinaryReader.

Parameters:: file_path (str) – Actual path to the file of interest.
Returns:: raw_data (obj) – Raw data from file.

static read_geometry_coordinates(external_geometry)[source]#

Read geometry of interest.

This method uses the QUEENS external geometry module.

Parameters:: external_geometry (queens.fourc_dat_geometry) – QUEENS external geometry object
Returns:: dict – set with 4C topology

queens.data_processor.data_processor_ensight_interface module#

Data processor module for vtk ensight boundary data.

class DataProcessorEnsightInterfaceDiscrepancy(file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None)[source]#

Bases: DataProcessor

Discrepancy measure for boundaries and shapes.

data_processor class uses full ensight result in vtk to measure distance of surface from simulations to experiment.

time_tol#

Time tolerance for given reference time points.

Type:: float

visualization_bool#

Boolean for vtk visualization control.

Type:: bool

displacement_fields#

String with exact field names for displacement to apply.

Type:: str

problem_dimension#

String to determine problems in spatial dimension.

Type:: string

experimental_ref_data_lst#

Experimental reference data to which the discrepancy measure is computed.

Type:: list

compute_distance(intersection_points, measured_points)[source]#

Find the furthest point for a set of intersection points.

Parameters:

intersection_points (list) – intersection point coordinates
measured_points (list) – pair of points from monitor file

Returns:

distance (float) – signed distance between root point and furthest outward intersection point; positive if in positive direction from root

deformed_grid(raw_data, time)[source]#

Read deformed grid from Ensight file at specified time.

Initially, the undeformed grid is read from the Ensight file Afterward, warpbyvector applies the displacement of structure field at time time such that the final result is the deformed grid at the specified time.

Parameters:

raw_data (obj) – Raw data from file
time (float) – Time value for data processing

Returns:

deformed_grid (vtkUnstructuredGrid) – Deformed grid for given time

filter_and_manipulate_raw_data(raw_data)[source]#

Get deformed boundary from vtk.

Create vtk representation of deformed external_geometry_obj and evaluate surface distance measurement for every given time step from the experiment.

Parameters:: raw_data (obj) – Raw data from file.
Returns:: residual (list) – Full residual from this data_processor class

get_raw_data_from_file(file_path)[source]#

Read-in EnSight file using vtkGenericEnSightReader.

Parameters:: file_path (str) – Actual path to the file of interest.
Returns:: raw_data (obj) – Raw data from file.

static read_monitorfile(path_to_experimental_reference_data)[source]#

Read Monitor file.

The Monitor File contains measurements from the experiments.

Parameters:: path_to_experimental_reference_data (path obj) – Path to experimental reference data
Returns:: monfile_data (list) – Data from monitor file in numbers

stretch_vector(vec1, vec2, scalar)[source]#

Extend a vector by scalar factor on both ends.

Parameters:

vec1 (list) – root point coordinates
vec2 (list) – directional point coordinates
scalar (float) – scalar multiplier

Returns:

vec (list) – vector from modified root to modified direction point

queens.data_processor.data_processor_numpy module#

Data processor class for numpy data extraction.

class DataProcessorNumpy(file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None)[source]#

Bases: DataProcessor

Class for extracting data from numpy binaries.

filter_and_manipulate_raw_data(raw_data)[source]#

Filter and manipulate the raw data.

In this case we want the raw data as it is. Of course this method can implement more specific data processing in the future.

Parameters:: raw_data (np.array) – Raw data from file.
Returns:: processed_data (np.array) – Cleaned, filtered or manipulated data_processor data.

get_raw_data_from_file(file_path)[source]#

Get the raw data from the files of interest.

This method loads the numpy binary data from the file.

Parameters:: file_path (str) – Actual path to the file of interest.
Returns:: raw_data (np.array) – Raw data from file.

queens.data_processor.data_processor_pvd module#

Data processor class for pvd data extraction.

class DataProcessorPvd(field_name, file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None, time_steps=None, block=0, point_data=True)[source]#

Bases: DataProcessor

Class for extracting data from pvd.

field_name#

Name of the field to extract data from

Type:: str

time_steps#

Considered time steps (last time step by default)

Type:: lst

block#

Considered block of MultiBlock data set (first block by default)

Type:: int

data_attribute#

‘point_data’ or ‘cell_data’

Type:: str

filter_and_manipulate_raw_data(raw_data)[source]#

Filter and manipulate the raw data.

Parameters:: raw_data (pv.PVDReader) – PVDReader object.
Returns:: processed_data (np.array) – Cleaned, filtered or manipulated data_processor data.

get_raw_data_from_file(file_path)[source]#

Get the raw data from the files of interest.

Parameters:: file_path (str) – Actual path to the file of interest.
Returns:: raw_data (pv.PVDReader) – PVDReader object.

queens.data_processor.data_processor_txt module#

Data processor class for txt data extraction.

class DataProcessorTxt(file_name_identifier=None, file_options_dict=None, files_to_be_deleted_regex_lst=None, remove_logger_prefix_from_raw_data=True, logger_prefix='\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3} - queens\\.drivers\\.driver_\\d* - INFO -', max_file_size_in_mega_byte=200)[source]#

Bases: DataProcessor

Class for extracting data from txt files.

Provides basic functionality for extracting data from txt files, however the final implementation is up to the user.

The implementation of the filter_and_manipulate_raw_data method is up to the user.

Throws:: MemoryError: We throw a conservative MemoryError if the txt file is larger than 200 MB. This is due to the current design, which loads the entire content of the .txt file into memory.
Potential Improvement:: Use a generator for reading the content of the file in chunks. This however requires a more advanced logic with the possibility to nest functions calls in a flexible way.

filter_and_manipulate_raw_data(raw_data)[source]#

Filter the raw data from the txt file.

The DataProcessorTxt class provides some basic filtering functionality, however it is up to the user to define the specifics of how the raw data should be filtered.

Parameters:: raw_data (lst) – List of strings Raw data from file.
Returns:: To be implemented by user.

get_raw_data_from_file(file_path)[source]#

Load the text file into memory.

Parameters:: file_path (str) – Actual path to the file of interest.
Returns:: raw_data (lst) – A list of strings read in from file_path.