2.1.1.1.1.15. emicroml.modelling.cbed.distortion.estimation.MLModelTester

class MLModelTester(ml_dataset_manager, device_name=None, output_dirname='results', misc_model_testing_metadata={}, skip_validation_and_conversion=False)[source]

Bases: _MLModelTester

A machine learning model tester.

The current class is a subclass of fancytypes.PreSerializableAndUpdatable.

The current class represents a machine learning (ML) model tester that can be used to test ML models represented by the class emicroml.modelling.cbed.distortion.estimation.MLModel.

See the documentation for the method emicroml.modelling.cbed.distortion.estimation.MLModelTester.test_ml_model() for a discussion on how performance metrics are calculated and tracked during ML model testing.

Parameters:

ml_dataset_manageremicroml.modelling.cbed.distortion.estimation.MLDatasetManager

The ML dataset manager to use during ML model testing. The ML dataset manager must specify at least a ML testing dataset. Any ML training and validation datasets specified are ignored.

Note that ml_dataset_manager stores an integer ml_dataset_manager.core_attrs["mini_batch_size"] which specifies the mini-batch size to be used in evaluating ML models. This is different from the mini-batch size used for calculating mini-batch losses during testing, which is always equal to unity. Hence, each mini-batch loss is equivalent to the loss of a single ML data instance. Generally speaking, the higher the value of ml_dataset_manager.core_attrs["mini_batch_size"], the faster the testing of ML models since more parallelization is being used.

device_namestr | None, optional

This parameter specifies the device to be used to perform computationally intensive calls to PyTorch functions and to store intermediate arrays of the type torch.Tensor. If device_name is a string, then it is the name of the device to be used, e.g. ”cuda” or ”cpu”. If device_name is set to None and a GPU device is available, then a GPU device is to be used. Otherwise, the CPU is used.

output_dirnamestr, optional

The relative or absolute path to the directory in which all output files are saved.

misc_model_testing_metadatadict, optional

Miscellaneous ML model testing metadata. Can be any dict object that is serializable, i.e. import json; json.dumps(misc_model_testing_metadata) must not raise an exception. Note that misc_model_testing_metadata is not used to test ML models, but is serialized and saved as output. See the documentation for the method emicroml.modelling.cbed.distortion.estimation.MLModelTester.test_ml_model(), for details on how misc_model_testing_metadata is saved as output.

skip_validation_and_conversionbool, optional

Let validation_and_conversion_funcs and core_attrs denote the attributes validation_and_conversion_funcs and core_attrs respectively, both of which being dict objects.

Let params_to_be_mapped_to_core_attrs denote the dict representation of the constructor parameters excluding the parameter skip_validation_and_conversion, where each dict key key is a different constructor parameter name, excluding the name "skip_validation_and_conversion", and params_to_be_mapped_to_core_attrs[key] would yield the value of the constructor parameter with the name given by key.

If skip_validation_and_conversion is set to False, then for each key key in params_to_be_mapped_to_core_attrs, core_attrs[key] is set to validation_and_conversion_funcs[key] (params_to_be_mapped_to_core_attrs).

Otherwise, if skip_validation_and_conversion is set to True, then core_attrs is set to params_to_be_mapped_to_core_attrs.copy(). This option is desired primarily when the user wants to avoid potentially expensive deep copies and/or conversions of the dict values of params_to_be_mapped_to_core_attrs, as it is guaranteed that no copies or conversions are made in this case.

Attributes:

core_attrs: dict: The “core attributes”.
de_pre_serialization_funcs: dict: The de-pre-serialization functions.
pre_serialization_funcs: dict: The pre-serialization functions.
validation_and_conversion_funcs: dict: The validation and conversion functions.

Methods

`de_pre_serialize`([serializable_rep, ...])	Construct an instance from a serializable representation.
`dump`([filename, overwrite])	Serialize instance and save the result in a JSON file.
`dumps`()	Serialize instance.
`get_core_attrs`([deep_copy])	Return the core attributes.
`get_de_pre_serialization_funcs`()	Return the de-pre-serialization functions.
`get_pre_serialization_funcs`()	Return the pre-serialization functions.
`get_validation_and_conversion_funcs`()	Return the validation and conversion functions.
`load`([filename, skip_validation_and_conversion])	Construct an instance from a serialized representation that is stored in a JSON file.
`loads`([serialized_rep, ...])	Construct an instance from a serialized representation.
`pre_serialize`()	Pre-serialize instance.
`test_ml_model`(ml_model)	Test a machine learning model.
`update`(new_core_attr_subset_candidate[, ...])	Update a subset of the core attributes.

execute_post_core_attrs_update_actions

Methods

`de_pre_serialize`	Construct an instance from a serializable representation.
`dump`	Serialize instance and save the result in a JSON file.
`dumps`	Serialize instance.
`execute_post_core_attrs_update_actions`
`get_core_attrs`	Return the core attributes.
`get_de_pre_serialization_funcs`	Return the de-pre-serialization functions.
`get_pre_serialization_funcs`	Return the pre-serialization functions.
`get_validation_and_conversion_funcs`	Return the validation and conversion functions.
`load`	Construct an instance from a serialized representation that is stored in a JSON file.
`loads`	Construct an instance from a serialized representation.
`pre_serialize`	Pre-serialize instance.
`test_ml_model`	Test a machine learning model.
`update`	Update a subset of the core attributes.

Attributes

`core_attrs`	dict: The "core attributes".
`de_pre_serialization_funcs`	dict: The de-pre-serialization functions.
`pre_serialization_funcs`	dict: The pre-serialization functions.
`validation_and_conversion_funcs`	dict: The validation and conversion functions.

property core_attrs

dict: The “core attributes”.

The keys of core_attrs are the same as the attribute validation_and_conversion_funcs, which is also a dict object.

Note that core_attrs should be considered read-only.

property de_pre_serialization_funcs

dict: The de-pre-serialization functions.

de_pre_serialization_funcs has the same keys as the attribute validation_and_conversion_funcs, which is also a dict object.

Let validation_and_conversion_funcs and pre_serialization_funcs denote the attributes validation_and_conversion_funcs pre_serialization_funcs respectively, the last of which being a dict object as well.

Let core_attrs_candidate_1 be any dict object that has the same keys as validation_and_conversion_funcs, where for each dict key key in core_attrs_candidate_1, validation_and_conversion_funcs[key](core_attrs_candidate_1) does not raise an exception.

Let serializable_rep be a dict object that has the same keys as core_attrs_candidate_1, where for each dict key key in core_attrs_candidate_1, serializable_rep[key] is set to pre_serialization_funcs[key](core_attrs_candidate_1[key]).

The items of de_pre_serialization_funcs are expected to be set to callable objects that would lead to de_pre_serialization_funcs[key](serializable_rep[key]) not raising an exception for each dict key key in serializable_rep.

Let core_attrs_candidate_2 be a dict object that has the same keys as serializable_rep, where for each dict key key in validation_and_conversion_funcs, core_attrs_candidate_2[key] is set to de_pre_serialization_funcs[key](serializable_rep[key]).

The items of de_pre_serialization_funcs are also expected to be set to callable objects that would lead to validation_and_conversion_funcs[key](core_attrs_candidate_2) not raising an exception for each dict key key in core_attrs_candidate_2.

Note that de_pre_serialization_funcs should be considered read-only.

classmethod de_pre_serialize(serializable_rep={}, skip_validation_and_conversion=False)

Construct an instance from a serializable representation.

Parameters:

serializable_repdict, optional

A dict object that has the same keys as the attribute validation_and_conversion_funcs, which is also a dict object.

Let validation_and_conversion_funcs and de_pre_serialization_funcs denote the attributes validation_and_conversion_funcs de_pre_serialization_funcs respectively, the last of which being a dict object as well.

The items of serializable_rep are expected to be objects that would lead to de_pre_serialization_funcs[key](serializable_rep[key]) not raising an exception for each dict key key in serializable_rep.

Let core_attrs_candidate be a dict object that has the same keys as serializable_rep, where for each dict key key in serializable_rep, core_attrs_candidate[key] is set to de_pre_serialization_funcs[key](serializable_rep[key])``.

The items of serializable_rep are also expected to be set to objects that would lead to validation_and_conversion_funcs[key](core_attrs_candidate) not raising an exception for each dict key key in serializable_rep.

skip_validation_and_conversionbool, optional

Let core_attrs denote the attribute core_attrs, which is a dict object.

If skip_validation_and_conversion is set to False, then for each key key in serializable_rep, core_attrs[key] is set to validation_and_conversion_funcs[key] (core_attrs_candidate), with validation_and_conversion_funcs and core_attrs_candidate_1 being introduced in the above description of serializable_rep.

Otherwise, if skip_validation_and_conversion is set to True, then core_attrs is set to core_attrs_candidate.copy(). This option is desired primarily when the user wants to avoid potentially expensive deep copies and/or conversions of the dict values of core_attrs_candidate, as it is guaranteed that no copies or conversions are made in this case.

Returns:

instance_of_current_clsCurrent class: An instance constructed from the serializable representation serializable_rep.

dump(filename='serialized_rep_of_fancytype.json', overwrite=False)

Serialize instance and save the result in a JSON file.

Parameters:

filenamestr, optional: The relative or absolute path to the JSON file in which to store the serialized representation of an instance.
overwritebool, optional: If overwrite is set to False and a file exists at the path filename, then the serialized instance is not written to that file and an exception is raised. Otherwise, the serialized instance will be written to that file barring no other issues occur.

Returns:

dumps()

Serialize instance.

Returns:

serialized_repdict: A serialized representation of an instance.

get_core_attrs(deep_copy=True)

Return the core attributes.

Parameters:

deep_copybool, optional

Let core_attrs denote the attribute core_attrs, which is a dict object.

If deep_copy is set to True, then a deep copy of core_attrs is returned. Otherwise, a shallow copy of core_attrs is returned.

Returns:

core_attrsdict: The attribute core_attrs.

classmethod get_de_pre_serialization_funcs()

Return the de-pre-serialization functions.

Returns:

de_pre_serialization_funcsdict: The attribute de_pre_serialization_funcs.

classmethod get_pre_serialization_funcs()

Return the pre-serialization functions.

Returns:

pre_serialization_funcsdict: The attribute pre_serialization_funcs.

classmethod get_validation_and_conversion_funcs()

Return the validation and conversion functions.

Returns:

validation_and_conversion_funcsdict: The attribute validation_and_conversion_funcs.

classmethod load(filename='serialized_rep_of_fancytype.json', skip_validation_and_conversion=False)

Construct an instance from a serialized representation that is stored in a JSON file.

Users can save serialized representations to JSON files using the method fancytypes.PreSerializable.dump().

Parameters:

filenamestr, optional

The relative or absolute path to the JSON file that is storing the serialized representation of an instance.

filename is expected to be such that json.load(open(filename, "r")) does not raise an exception.

Let serializable_rep=json.load(open(filename, "r")).

Let validation_and_conversion_funcs and de_pre_serialization_funcs denote the attributes validation_and_conversion_funcs de_pre_serialization_funcs respectively, both of which being dict objects as well.

filename is also expected to be such that de_pre_serialization_funcs[key](serializable_rep[key]) does not raise an exception for each dict key key in de_pre_serialization_funcs.

Let core_attrs_candidate be a dict object that has the same keys as de_pre_serialization_funcs, where for each dict key key in serializable_rep, core_attrs_candidate[key] is set to de_pre_serialization_funcs[key](serializable_rep[key])``.

filename is also expected to be such that validation_and_conversion_funcs[key](core_attrs_candidate) does not raise an exception for each dict key key in serializable_rep.

skip_validation_and_conversionbool, optional

Let core_attrs denote the attribute core_attrs, which is a dict object.

Let core_attrs_candidate be as defined in the above description of filename.

If skip_validation_and_conversion is set to False, then for each key key in core_attrs_candidate, core_attrs[key] is set to validation_and_conversion_funcs[key] (core_attrs_candidate), , with validation_and_conversion_funcs and core_attrs_candidate being introduced in the above description of filename.

Otherwise, if skip_validation_and_conversion is set to True, then core_attrs is set to core_attrs_candidate.copy(). This option is desired primarily when the user wants to avoid potentially expensive deep copies and/or conversions of the dict values of core_attrs_candidate, as it is guaranteed that no copies or conversions are made in this case.

Returns:

instance_of_current_clsCurrent class: An instance constructed from the serialized representation stored in the JSON file.

classmethod loads(serialized_rep='{}', skip_validation_and_conversion=False)

Construct an instance from a serialized representation.

Users can generate serialized representations using the method dumps().

Parameters:

serialized_repstr | bytes | bytearray, optional

The serialized representation.

serialized_rep is expected to be such that json.loads(serialized_rep) does not raise an exception.

Let serializable_rep=json.loads(serialized_rep).

Let validation_and_conversion_funcs and de_pre_serialization_funcs denote the attributes validation_and_conversion_funcs de_pre_serialization_funcs respectively, both of which being dict objects as well.

serialized_rep is also expected to be such that de_pre_serialization_funcs[key](serializable_rep[key]) does not raise an exception for each dict key key in de_pre_serialization_funcs.

Let core_attrs_candidate be a dict object that has the same keys as serializable_rep, where for each dict key key in de_pre_serialization_funcs, core_attrs_candidate[key] is set to de_pre_serialization_funcs[key](serializable_rep[key])``.

serialized_rep is also expected to be such that validation_and_conversion_funcs[key](core_attrs_candidate) does not raise an exception for each dict key key in serializable_rep.

skip_validation_and_conversionbool, optional

Let core_attrs denote the attribute core_attrs, which is a dict object.

If skip_validation_and_conversion is set to False, then for each key key in core_attrs_candidate, core_attrs[key] is set to validation_and_conversion_funcs[key] (core_attrs_candidate), with validation_and_conversion_funcs and core_attrs_candidate_1 being introduced in the above description of serialized_rep.

Otherwise, if skip_validation_and_conversion is set to True, then core_attrs is set to core_attrs_candidate.copy(). This option is desired primarily when the user wants to avoid potentially expensive deep copies and/or conversions of the dict values of core_attrs_candidate, as it is guaranteed that no copies or conversions are made in this case.

Returns:

instance_of_current_clsCurrent class: An instance constructed from the serialized representation.

property pre_serialization_funcs

dict: The pre-serialization functions.

pre_serialization_funcs has the same keys as the attribute validation_and_conversion_funcs, which is also a dict object.

Let validation_and_conversion_funcs and core_attrs denote the attributes validation_and_conversion_funcs and core_attrs respectively, the last of which being a dict object as well.

For each dict key key in core_attrs, pre_serialization_funcs[key](core_attrs[key]) is expected to yield a serializable object, i.e. it should yield an object that can be passed into the function json.dumps without raising an exception.

Note that pre_serialization_funcs should be considered read-only.

pre_serialize()

Pre-serialize instance.

Returns:

serializable_repdict: A serializable representation of an instance.

test_ml_model(ml_model)[source]

Test a machine learning model.

See the summary documentation of the class emicroml.modelling.cbed.distortion.estimation.MLModelTester for additional context.

Let core_attrs be the attribute core_attrs, output_dirname be core_attrs["output_dirname"], and misc_model_testing_metadata be core_attrs["misc_model_testing_metadata"].

The only output file that is generated by the end of the ML model testing is the ML model testing summary output data file, which is an HDF5 file generated at the file path output_dirname+"/ml_model_testing_summary_output_data.h5". The HDF5 file is guaranteed to contain the following HDF5 objects:

ml_model_tester_params: <HDF5 1D dataset>
total_num_ml_testing_data_instances: <HDF5 0D dataset>

ml_data_instance_metrics: <HDF5 group>
- testing: <HDF5 group>
  - epes_of_adjusted_distortion_fields <HDF5 1D dataset>
    - dim_0: “ml testing data instance idx”

Note that the sub-bullet points listed immediately below a given HDF5 dataset display the HDF5 attributes associated with said HDF5 dataset. Some HDF5 datasets have attributes with names of the form "dim_{}".format(i) with i being an integer. Attribute "dim_{}".format(i) of a given HDF5 dataset labels the i th dimension of the underlying array of the dataset.

The HDF5 dataset at the HDF5 path "/ml_model_tester_params" stores a serialized version of the attribute core_attrs, which is essentially the construction parameters used to construct an instance of the current class. From the output HDF5 file, users can reconstruct the instance of the current class that generated said output file by:

import h5pywrappers
import emicroml.modelling.cbed.distortion.estimation

filename = (output_dirname 
            +"/ml_model_testing_summary_output_data.h5")

kwargs = {"filename": filename,
         "path_in_file": "ml_model_tester_params"}
json_document_id = h5pywrappers.obj.ID(**kwargs)

serializable_rep = h5pywrappers.json.document.load(json_document_id)

MLModelTester = \
    emicroml.modelling.cbed.distortion.estimation.MLModelTester
ml_model_tester = \
    MLModelTester.de_pre_serialize(serializable_rep)

where ml_model_tester is the reconstructed instance of the current class, and serializable_rep is a “pre-serialized” version of it. See the documentation for the class fancytypes.PreSerializable for a discussion on pre-serialization.

The zero-dimensional HDF5 dataset, i.e. scalar, at the HDF5 path "/total_num_ml_testing_data_instances" stores the total number of ML testing data instances.

The HDF5 group at the HDF5 path "/ml_data_instance_metrics" stores the performance metrics that are tracked during testing.

The HDF5 dataset at the HDF5 path "/ml_data_instance_metrics/testing/epes_of_adjusted_distortion_fields" stores the end-point errors (EPEs) of the “adjusted” standard distortion fields specified by the predicted standard coordinate transformation parameter sets, during testing. For every nonnegative integer m less than the the total number of ML testing data instances, the m th element of the aforementioned HDF5 dataset is the EPE of the adjusted standard distortion field specified by the m th predicted standard standard coordinate transformation set, during testing. See the summary documentation of the class emicroml.modelling.cbed.distortion.estimation.MLModelTrainer for a definition of an adjusted standard distortion field, and how the EPE is calculated exactly.

Parameters:

ml_modelemicroml.modelling.cbed.distortion.estimation.MLModel: The ML model to test.

update(new_core_attr_subset_candidate, skip_validation_and_conversion=False)

Update a subset of the core attributes.

Parameters:

new_core_attr_subset_candidatedict, optional

A dict object.

skip_validation_and_conversionbool, optional

Let validation_and_conversion_funcs and core_attrs denote the attributes validation_and_conversion_funcs and core_attrs respectively, both of which being dict objects.

If skip_validation_and_conversion is set to False, then for each key key in core_attrs that is also in new_core_attr_subset_candidate, core_attrs[key] is set to validation_and_conversion_funcs[key] (new_core_attr_subset_candidate).

Otherwise, if skip_validation_and_conversion is set to True, then for each key key in core_attrs that is also in new_core_attr_subset_candidate, core_attrs[key] is set to new_core_attr_subset_candidate[key]. This option is desired primarily when the user wants to avoid potentially expensive deep copies and/or conversions of the dict values of new_core_attr_subset_candidate, as it is guaranteed that no copies or conversions are made in this case.

property validation_and_conversion_funcs

dict: The validation and conversion functions.

The keys of validation_and_conversion_funcs are the names of the constructor parameters, excluding skip_validation_and_conversion if it exists as a construction parameter.

Let core_attrs denote the attribute core_attrs, which is also a dict object.

For each dict key key in core_attrs, validation_and_conversion_funcs[key](core_attrs) is expected to not raise an exception.

Note that validation_and_conversion_funcs should be considered read-only.