IAGOS.apps.workflow.qaqc

See also

The system uses the QA/QC framework Autom8QC. See also: https://autom8qc.readthedocs.io/en/latest/index.html

Overview

Base Components

The base components are dedicated to performing one specific task and don’t know anything about the context. They save the metadata and implements only one method. This approach ensures that they are flexible and can easily adapt. All components inherit from the abstract base class BaseComponent since they share the same properties (DRY). Moreover, the inherited components are abstract as well (Open-Closed) and can be customized.

QA/QC Components

The QA/QC components combine base components to complex components. They cache the results and have a specific behavior. To ensure that new QA/QC components can be added, they all inherit from the abstract base class autom8qc.qaqc.base.QAQCComponent. The abstract class has the abstract method perform and the property results.

FilterAttributes

class IAGOS.apps.workflow.qaqc.models.FilterAttributes(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the table filter_attrs. The filter are used to filter the series.

Parameters

name (string) – Name of the series (e.g., T_INSTR)
not_contains (string) – Part that shouldn’t be contained (e.g., analog)

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_series(task): Not finished yet

GroupItem

class IAGOS.apps.workflow.qaqc.models.GroupItem(*args, **kwargs)

Bases: django.db.models.base.Model

An instance of the class can be used to add a TestManager, a TestSequence and a TestGroup to a test group. Each item can be weighted. Therefore, use the attribute weight. The weight will be used by measures.

See also

IAGOS.apps.workflow.qaqc.models.QAQCTestGroup

Parameters

name (str) – Name of the item
weight (int) – Weight of the test
manager (QAQCTestManager) – Manager to handle the test
sequence (QAQCSequence) – Test Sequence
group (QAQCTestGroup) – TestGroup
weight – Weight of the test

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_instance(task)

Returns the instance of the model.

Parameters: task (Task) – Task
Returns: Item of a group
Return type: autom8qc.qaqc.base.GroupItem

get_test(task)

Returns the test of the item (manager, sequence or group)

Parameters: task (Task) – Task
Returns: Test
Return type: autom8qc.qaqc.base.QAQCComponent

QAQCContainer

class IAGOS.apps.workflow.qaqc.models.QAQCContainer(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the table qaqc_container. A container is a collection of QA/QC components which provides the method perform. The method performs all components inside the container and is used by the TaskManager.

See also

IAGOS.apps.workflow.qaqc.models.QAQCTestManager
IAGOS.apps.workflow.qaqc.models.QAQCTestGroup
IAGOS.apps.workflow.qaqc.models.QAQCSequence

Parameters

managers (ManyToManyField) – Test Managers
groups (ManyToManyField) – Test Groups
sequences (ManyToManyField) – Test Sequences

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

perform(task)

Performs all items of the container.

Parameters: task (Task) – Task
Returns: None
Return type: None

QAQCFunction

class IAGOS.apps.workflow.qaqc.models.QAQCFunction(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_function. Each function has to refer to a type and individual parameters.

See also

IAGOS.apps.workflow.qaqc.models.QAQCParameter
IAGOS.apps.workflow.qaqc.models.QAQCFunctionType

Parameters

function_type (QAQCFunctionType) – Type of the function (e.g., Linear Interpolation)
parameters (ManyToManyField) – Parameters of the test

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_instance()

Returns the instance of the model.

Returns: QAQCFunction
Return type: autom8qc.qaqc.base.BaseFunction

import_parameters(parameters)

Imports the given parameters.

Parameters: parameters (autom8qc.core.ParameterList) – Parameters of function
Returns: None
Return type: None

QAQCFunctionType

class IAGOS.apps.workflow.qaqc.models.QAQCFunctionType(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_function_type. The type stores the metadata of a function (name, description). With this approach, you can make sure that the functions always share the same metadata.

Parameters

name (string) – Name of the type (e.g. Linear Interpolation)
description (string) – Description of the type (optional)

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

create_function(parameters)

Creates a function with the given parameters.

Parameters: parameters (autom8qc.core.ParameterList) – Parameters of test tests
Returns: Instance of the class QAQCFunction with the given parameters
Return type: QAQCFunction

load_reference()

Loads the class references.

Returns: None
Return type: None

property supported_parameters

Returns the supported parameters of the function.

Returns: Supported parameters
Return type: autom8qc.core.ParameterList

QAQCMapper

class IAGOS.apps.workflow.qaqc.models.QAQCMapper(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_mapper. Each mapper has to refer to a type and individual parameters.

See also

IAGOS.apps.workflow.qaqc.models.QAQCParameter
IAGOS.apps.workflow.qaqc.models.QAQCMapperType

Parameters

mapper_type (QAQCMapperType) – Type of the mapper
parameters (ManyToManyField) – Parameters of the test

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_instance()

Returns the instance of the model.

Returns: QAQCMapper
Return type: autom8qc.qaqc.base.BaseMapper

import_parameters(parameters)

Imports the given parameters.

Parameters: parameters (autom8qc.core.ParameterList) – Parameters of mapper
Returns: None
Return type: None

QAQCMapperType

class IAGOS.apps.workflow.qaqc.models.QAQCMapperType(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_mapper_type. The type stores the metadata of a mapper (name, description). With this approach, you can make sure that the mappers always share the same metadata.

Parameters

name (string) – Name of the type (e.g. Logical-Threshold-Mapper)
description (string) – Description of the type (optional)

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

create_mapper(parameters)

Creates a mapper with the given parameters.

Parameters: parameters (autom8qc.core.ParameterList) – Parameters of test tests
Returns: Instance of the class QAQCTest with the given parameters
Return type: QAQCTest

load_reference()

Loads the class references.

Returns: None
Return type: None

property supported_parameters

Returns the supported parameters of the test.

Returns: Supported parameters
Return type: autom8qc.core.ParameterList

QAQCMeasure

class IAGOS.apps.workflow.qaqc.models.QAQCMeasure(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_measure. Each measure has to refer to a type and individual parameters.

See also

IAGOS.apps.workflow.qaqc.models.QAQCParameter
IAGOS.apps.workflow.qaqc.models.QAQCMeasureType

Parameters

measure_type (QAQCMeasureType) – Type of the measure (e.g. Weighted Mean)
parameters (ManyToManyField) – Parameters of the test

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_instance()

Returns the instance of the model.

Returns: QAQCRule
Return type: autom8qc.qaqc.base.BaseRule

import_parameters(parameters)

Imports the given parameters.

Parameters: parameters (autom8qc.core.ParameterList) – Parameters of measure
Returns: None
Return type: None

QAQCMeasureType

class IAGOS.apps.workflow.qaqc.models.QAQCMeasureType(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_measure_type. The type stores the metadata of a measure (name, description). With this approach, you can make sure that the measures always share the same metadata.

Parameters

name (string) – Name of the type (e.g. Weighted Mean)
description (string) – Description of the type (optional)

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

create_function(parameters)

Creates a measure with the given parameters.

Parameters: parameters (autom8qc.core.ParameterList) – Parameters of test tests
Returns: Instance of the class QAQCMeasure with the given parameters
Return type: QAQCMeasure

load_reference()

Loads the class references.

Returns: None
Return type: None

property supported_parameters

Returns the supported parameters of the measure.

Returns: Supported parameters
Return type: autom8qc.core.ParameterList

QAQCParameter

class IAGOS.apps.workflow.qaqc.models.QAQCParameter(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the table qaqc_parameter. Each parameter has a name and a value. Instances of the class can be used by QAQCTest, QAQCMapper, QAQCFunction, QAQCMeasure and QAQCRule.

See also

IAGOS.apps.workflow.qaqc.models.QAQCFunction
IAGOS.apps.workflow.qaqc.models.QAQCMapper
IAGOS.apps.workflow.qaqc.models.QAQCMeasure
IAGOS.apps.workflow.qaqc.models.QAQCRule
IAGOS.apps.workflow.qaqc.models.QAQCTest

Parameters

name (string) – Name of the parameter (e.g. Number of neighbors)
value (string) – Value of the type

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

QAQCRule

class IAGOS.apps.workflow.qaqc.models.QAQCRule(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_rule. Each rule has to refer to a type and has individual parameters.

See also

IAGOS.apps.workflow.qaqc.models.QAQCParameter
IAGOS.apps.workflow.qaqc.models.QAQCRuleType

Parameters

rule_type (QAQCRuleType) – Type of the rule (e.g. Weighted Mean)
parameters (ManyToManyField) – Parameters of the test

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_instance()

Returns the instance of the model.

Returns: QAQCRule
Return type: autom8qc.qaqc.base.BaseRule

import_parameters(parameters)

Imports the given parameters.

Parameters: parameters (autom8qc.core.ParameterList) – Parameters of rule
Returns: None
Return type: None

QAQCRuleType

class IAGOS.apps.workflow.qaqc.models.QAQCRuleType(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_rule_type. The type stores the metadata of a rule (name, description). With this approach, you can make sure that the rules always share the same metadata.

Parameters

name (string) – Name of the type (e.g. Weighted Mean)
description (string) – Description of the type (optional)

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

create_rule(parameters)

Creates a function with the given parameters.

Parameters: parameters (autom8qc.core.ParameterList) – Parameters of test tests
Returns: Instance of the class QAQCRule with the given parameters
Return type: QAQCRule

load_reference()

Loads the class references.

Returns: None
Return type: None

property supported_parameters

Returns the supported parameters of the rule.

Returns: Supported parameters
Return type: autom8qc.core.ParameterList

QAQCSequence

class IAGOS.apps.workflow.qaqc.models.QAQCSequence(*args, **kwargs)

Bases: django.db.models.base.Model

A TestSequence allows you to create a sequence of several tests that are based on each other. If a data point failed a test, it won’t pass to the next test. A data point failed the test, if the probability is lower than the defined threshold. With this approach, you can ensure that invalid data points don’t affect the next test. Especially, if you use tests that consider the global range of the data (e.g., autom8qc.qaqc.outlier.LOFTest) it’s highly recommended to use this data structure.

Important

If you don’t pass a measure to the constructor, the autom8qc.measures.probabilities.WorstProbabilityMeasure will be used.

Warning

A test group handles the QA/QC tests isolated and combines the results of the tests. If your tests are not depending on each other, you need to use a TestGroup.

See also

IAGOS.apps.workflow.qaqc.models.QAQCTest
IAGOS.apps.workflow.qaqc.models.QAQCMapper
IAGOS.apps.workflow.qaqc.models.QAQCMeasure
IAGOS.apps.workflow.qaqc.models.QAQCFunction
IAGOS.apps.workflow.qaqc.models.QAQCRule

Execution Pipeline:

Preprocessing (optional)
Perform tests (required)
Apply rule on the probabilities (optional)
Map the probabilties to another domain (optional)
Apply rule on the mapped values (optional)
Postprocessing (optional)

Parameters

filter_attr (FilterAttributes) – Filter to access the data
items (ManyToManyField) – Sequence items
mapper (QAQCMapper) – Mapper to map the final results
pre_function (QAQCFunction) – Function that will be applied before the sequence
post_function (QAQCFunction) – Function that will be applied after the sequence
prob_rule (QAQCRule) – Rule that will be applied on the probabilities
mapped_rule (QAQCRule) – Rule that will be applied on the mapped values
measure (QAQCMeasure) – Measure for the total result

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_instance(task)

Returns the instance of the model.

Parameters: task (Task) – Task
Returns: TestManager
Return type: autom8qc.qaqc.base.TestManager

get_mapped_rule()

Returns the rule for mapped values. If no rule is set, None will be returned.

Returns: QAQCRule
Return type: autom8qc.qaqc.base.BaseRule

get_mapper()

Returns the mapper. If no mapper is set, None will be returned.

Returns: QAQCMapper
Return type: autom8qc.qaqc.base.BaseMapper

get_measure()

Returns the measure. If no function is set, None will be returned.

Returns: QAQCFunction
Return type: autom8qc.qaqc.base.BaseFunction

get_post_func()

Returns the postprocessing function. If no function is set, None will be returned.

Returns: QAQCFunction
Return type: autom8qc.qaqc.base.BaseFunction

get_pre_func()

Returns the preprocessing function. If no function is set, None will be returned.

Returns: QAQCFunction
Return type: autom8qc.qaqc.base.BaseFunction

get_prob_rule()

Returns the rule for probabilites. If no rule is set, None will be returned.

Returns: QAQCRule
Return type: autom8qc.qaqc.base.BaseRule

perform(task)

Performs all items of the container.

Parameters: task (Task) – Task
Returns: None
Return type: None

QAQCTest

class IAGOS.apps.workflow.qaqc.models.QAQCTest(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_test. Each test has to refer to a type and has individual parameters.

See also

IAGOS.apps.workflow.qaqc.models.QAQCParameter
IAGOS.apps.workflow.qaqc.models.QAQCTestType

Parameters

test_type (TestType) – Type of the test
parameters (ManyToManyField) – Parameters of the test

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_instance()

Returns the instance of the model.

Returns: QAQCTest
Return type: autom8qc.qaqc.base.QAQCTest

import_parameters(parameters)

Imports the given parameters.

Parameters: parameters (autom8qc.core.ParameterList) – Parameters of the test
Returns: None
Return type: None

QAQCTestCategory

class IAGOS.apps.workflow.qaqc.models.QAQCTestCategory(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_test_category. The class is used by the class QAQCTestType, since every type has to refer to a category.

See also

IAGOS.apps.workflow.qaqc.models.QAQCTestType

Parameters: name (string) – Name of the category (e.g. Outlier detection)

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

QAQCTestGroup

class IAGOS.apps.workflow.qaqc.models.QAQCTestGroup(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table test_group. A test group is a collection of tests. A group has a unique name and a short description.

See also

IAGOS.apps.workflow.qaqc.models.QAQCTest
IAGOS.apps.workflow.qaqc.models.QAQCMapper
IAGOS.apps.workflow.qaqc.models.QAQCMeasure
IAGOS.apps.workflow.qaqc.models.QAQCFunction
IAGOS.apps.workflow.qaqc.models.QAQCRule

Execution Pipeline:

Perform tests (required)
Apply rule on the probabilities (optional)
Map the probabilties to another domain (optional)
Apply rule on the mapped values (optional)
Postprocessing (optional)

Parameters

filter_attr (FilterAttributes) – Filter to access the data
items (ManyToManyField) – Sequence items
mapper (QAQCMapper) – Mapper to map the final results
pre_function (QAQCFunction) – Function that will be applied before the sequence
post_function (QAQCFunction) – Function that will be applied after the sequence
prob_rule (QAQCRule) – Rule that will be applied on the probabilities
mapped_rule (QAQCRule) – Rule that will be applied on the mapped values
measure (QAQCMeasure) – Measure for the total result

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_instance(task)

Returns the instance of the model.

Parameters: task (Task) – Task
Returns: TestManager
Return type: autom8qc.qaqc.base.TestManager

get_mapped_rule()

Returns the rule for mapped values. If no rule is set, None will be returned.

Returns: QAQCRule
Return type: autom8qc.qaqc.base.BaseRule

get_mapper()

Returns the mapper. If no mapper is set, None will be returned.

Returns: QAQCMapper
Return type: autom8qc.qaqc.base.BaseMapper

get_measure()

Returns the measure. If no function is set, None will be returned.

Returns: QAQCFunction
Return type: autom8qc.qaqc.base.BaseFunction

get_post_func()

Returns the postprocessing function. If no function is set, None will be returned.

Returns: QAQCFunction
Return type: autom8qc.qaqc.base.BaseFunction

get_prob_rule()

Returns the rule for probabilites. If no rule is set, None will be returned.

Returns: QAQCRule
Return type: autom8qc.qaqc.base.BaseRule

perform(task)

Performs all items of the container.

Parameters: task (Task) – Task
Returns: None
Return type: None

QAQCTestManager

class IAGOS.apps.workflow.qaqc.models.QAQCTestManager(*args, **kwargs)

Bases: django.db.models.base.Model

A manager simplifies the execution of a test and caches the results. Each manager needs a BaseStructure to handle the data. Moreover, you have to define the test that should execute. Optionally, you can specify a mapper that maps the probabilities to other values (e.g., validities). Moreover, you have the option to define pre-and-post functions to preprocess the data (e.g., linear interpolation) and post-process the results (e.g., filling gaps).

See also

IAGOS.apps.workflow.qaqc.models.QAQCTest
IAGOS.apps.workflow.qaqc.models.QAQCMapper
IAGOS.apps.workflow.qaqc.models.QAQCFunction
IAGOS.apps.workflow.qaqc.models.QAQCRule

Execution Pipeline:

Preprocessing (optional)
Perform test (required)
Apply rule on the probabilities (optional)
Map the probabilties to another domain (optional)
Apply rule on the mapped values (optional)
Postprocessing (optional)

Parameters

filter_attr (FilterAttributes) – Filter to access the data
test (QAQCTest) – Test
mapper (QAQCMapper) – Mapper to map the probabilities
pre_func (QAQCFunction) – Function that will be applied before the test
post_func (QAQCFunction) – Function that will be applied after the test
prob_rule (QAQCRule) – Rule that will be applied on the probabilities
mapped_rule (QAQCRule) – Rule that will be applied on the mapped values

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_instance(task)

Returns the instance of the model.

Parameters: task (Task) – Task
Returns: TestManager
Return type: autom8qc.qaqc.base.TestManager

get_mapped_rule()

Returns the rule for mapped values. If no rule is set, None will be returned.

Returns: QAQCRule
Return type: autom8qc.qaqc.base.BaseRule

get_mapper()

Returns the mapper. If no mapper is set, None will be returned.

Returns: QAQCMapper
Return type: autom8qc.qaqc.base.BaseMapper

get_post_func()

Returns the postprocessing function. If no function is set, None will be returned.

Returns: QAQCFunction
Return type: autom8qc.qaqc.base.BaseFunction

get_pre_func()

Returns the preprocessing function. If no function is set, None will be returned.

Returns: QAQCFunction
Return type: autom8qc.qaqc.base.BaseFunction

get_prob_rule()

Returns the rule for probabilites. If no rule is set, None will be returned.

Returns: QAQCRule
Return type: autom8qc.qaqc.base.BaseRule

perform(task)

Performs all items of the container.

Parameters

task (Task) – Task
save (bool) – Save the validities (default: False)

Returns

None

Return type

None

QAQCTestType

class IAGOS.apps.workflow.qaqc.models.QAQCTestType(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table qaqc_test_type. The type stores the metadata of a test (name, description, category). With this approach, you can make sure that the tests always share the same metadata.

See also

IAGOS.apps.workflow.qaqc.models.QAQCTest

Parameters

name (string) – Name of the type (e.g. Local Outlier Factor algorithm)
description (string) – Description of the type (optional)
test_category (TestCategory) – Category of the test

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

create_test(parameters)

Creates a test with the given parameters.

Parameters: parameters (autom8qc.core.ParameterList) – Parameters of test tests
Returns: Instance of the class QAQCTest with the given parameters
Return type: QAQCTest

load_reference()

Loads the class references.

Returns: None
Return type: None

property supported_parameters

Returns the supported parameters of the test.

Returns: Supported parameters
Return type: autom8qc.core.ParameterList

SequenceItem

class IAGOS.apps.workflow.qaqc.models.SequenceItem(*args, **kwargs)

Bases: django.db.models.base.Model

An instance of the class is used by a TestSequence. Each item has a test and a threshold. If a test will be performed, the results of the tests will be cached.

See also

IAGOS.apps.workflow.qaqc.models.QAQCSequence

Parameters

name (str) – Name of the item
test (QAQCTest) – Test that should be performed
threshold (float) – Threshold to filter the good points
weight (int) – Weight of the item
position (int) – Position of the item

exception DoesNotExist: Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: django.core.exceptions.MultipleObjectsReturned

get_instance()

Returns the instance of the model.

Returns: Item of a sequence
Return type: autom8qc.qaqc.base.SequenceItem