IAGOS.apps.workflow.exports
Overview
The ExportManager is the last instance of the task execution and combines the components Exporter and Writer. The Exporter accesses the data of the database and prepares (e.g., resample the data) it for the export. The Writer writes the prepared data into a specific format (e.g., netCDF).
Interfaces
Interface - Writer
This module defines the interface for the writers. Each writer has to inherit from the abstract base class BaseWriter. This approach makes sure that the writer can be integrated into the workflow without modifying code. Each writer has to provide its metadata. Moreover, each writer has to implement the abstract method perform which writes the given data into a specific format (e.g., netCDF).
See also
- class IAGOS.apps.workflow.exports.writers.base.BaseWriter
Every writer has to inherit from this abstract base class. This approach ensures that new writers can be integrated into the workflow without modifying the existing code. Each writer has to provide its metadata (NAME, DESCRIPTION) and has to implement the abstract method write.
Warning
If you add additional attributes in the inherited classes, make sure that you reset them in the method write. Moreover, ensure that call the super-constructor.
- Parameters
NAME (string) – Name of the reader
DESCRIPTION (string) – Description of the reader
- check_description()
Checks if the DESCRIPTION is set and contains at least 1 character. If the DESCRIPTION is not valid, the method will raise a ValueError or a TypeError.
- Raises
ValueError – If DESCRIPTION is not set or empty
TypeError – If DESCRIPTION is not a string
- Returns
None
- Return type
None
- check_metadata()
Checks if the metadata is correct.
- Raises
ValueError – If NAME is not set or empty
TypeError – If NAME is not a string
ValueError – If DESCRIPTION is not set or empty
TypeError – If DESCRIPTION is not a string
- Returns
None
- Return type
None
- check_name()
Checks if the NAME is set and contains at least 1 character. If the NAME is not valid, the method will raise a ValueError or a TypeError.
- Raises
ValueError – If NAME is not set or empty
TypeError – If NAME is not a string
- Returns
None
- Return type
None
- abstract write(data, data_info, filename, addition)
Writes the data into a specific format.
- Parameters
data (pd.DataFrame) – Data in a pd.DataFrame
data_info (dict) – Data info (e.g. header in Nasa-Ames)
filename (string) – Name of the file
addition (string (JSON)) – Additional information
- Returns
Data
- Return type
object
Interface - Exporter
In this module, the base classes for all exporters are implemented. Every exporter has to be inherited from one of theses classes. With this approach, you can implement a new exporter without changing the existing code. Each exporter has to provide its metadata. Furthermore, every exporter has to implement the method run that executes the export process.
- class IAGOS.apps.workflow.exports.exporter.base.BaseExporter
Every exporter has to inherit from this abstract base class. This approach ensures that new exporters can be integrated into the workflow without modifying the existing code. Each exporter has to provide its metadata (NAME, DESCRIPTION) and has to implement the abstract methods run. The method prepares the data for the export. After it, the writer will be used to write the prepared data into a specific format.
Warning
If you add additional attributes in the inherited classes, make sure that you reset them in the method run. Moreover, ensure that call the super-constructor.
- Parameters
NAME (string) – Name of the exporter
DESCRIPTION (string) – Description of the exporter
error_status (Status) – Error status that describes why the export failed
process_status (List(ProcessStatus)) – All process status of the import
exported_series (List(DataSeries)) – All imported series
exported_metadata (List(Metadata)) – Exported metadata
- check_description()
Checks if the DESCRIPTION is set and contains at least 1 character. If the DESCRIPTION is not valid, the method will raise a ValueError or a TypeError.
- Raises
ValueError – If DESCRIPTION is not set or empty
TypeError – If DESCRIPTION is not a string
- Returns
None
- Return type
None
- check_metadata()
Checks if the metadata is correct.
- Raises
ValueError – If NAME is not set or empty
TypeError – If NAME is not a string
ValueError – If DESCRIPTION is not set or empty
TypeError – If DESCRIPTION is not a string
- Returns
None
- Return type
None
- check_name()
Checks if the NAME is set and contains at least 1 character. If the NAME is not valid, the method will raise a ValueError or a TypeError.
- Raises
ValueError – If NAME is not set or empty
TypeError – If NAME is not a string
- Returns
None
- Return type
None
- get_error_status()
The method returns the error status of the export. If there is no error, None will be returned. This method will be used by the task manager to inform the user that an error had occurred during the export process.
Important
Note that the error status describes that the whole export failed. Don’t use it as a warning.
- Returns
Status that describes why the export failed.
- Return type
- get_exported_metadata()
Returns all exported metadata.
- Returns
Exported metadata
- Return type
List<Metadata>
- get_exported_series()
Returns all exported series.
- Returns
Exported series
- Return type
List<DataSeries>
- get_process_status()
Returns all process-status of the export.
- Returns
All process-status of the export
- Return type
List<ProcessStatus>
- reset_parameters()
Reset the parameters of the exporter.
- Returns
None
- Return type
None
- abstract run(task, writer, destination, addition)
The exporter prepares the data for the export. That could be synchronizing, resampling, or just collect the data from the database. If the data is prepared, the exporter writes the data into the needed specific format. Therefore, the abstract method write of the writer will be used and the written data returned.
- Parameters
task (Task) – Related task
writer (BaseWriter) – Writer that writes the data
destination (string) – Destination of the export (e.g. filename)
addition (dictionary) – Additional parameters for the import.
- Returns
Data for the transfer
- Return type
object
- class IAGOS.apps.workflow.exports.exporter.base.BaseSeriesExporter
This class implements the base for the series exporter and inherits from the base class BaseExporter. The exporter for the series provides more information about the data.
Note
The attributes NAME and DESCRIPTIONS are class attributes. That means that all readers of the same class, refer always the same NAME and the DESCRIPTION.
Warning
If you add additional attributes in the inherited classes, make sure that you reset them in the method read. Otherwise this could create side effects and is very error prone.
- Parameters
NAME (string) – Name of the exporter
DESCRIPTION (string) – Description of the exporter
error_status (Status) – Error status that describes why the export failed
process_status (List<ProcessStatus>) – All process status of the import
exported_series (List<DataSeries>) – All imported series
exported_metadata (List<Metadata>) – Exported metadata
- get_comments()
Returns additional comments.
- Returns
Comments
- Return type
List<string>
- get_data_availability()
Returns the data availabilities. One value (percentage of availability, integer between 0 and 1)
- Returns
Data availability
- Return type
List<float>
- get_data_description()
Returns data descriptions.
- Returns
Data descriptions
- Return type
List<string>
- get_data_info()
Returns all information about the data.
- Returns
Data information
- Return type
dict
- get_data_level()
Returns the data level of the data.
- Returns
Data level
- Return type
integer
- get_deployment()
Returns the deployment.
- Returns
Related deployment
- Return type
- get_imported_series(task, name, exclude=None)
Returns the series with the given name. Optional, you can use the exclude to define a part of the name, that shouldn’t be considered.
Note
If the series doesn’t exist, the previous task will be considered.
- Parameters
task (Task) – Related task
name (string) – Name of the series
exclude (string) – Exclude string (optional)
- Returns
Series
- Return type
- get_instrument()
Returns the string representation of the instrument.
- Returns
Representation of the instrument
- Return type
string
- get_invalid_values()
Returns the values that mark invalid values (e.g. -9999).
- Returns
Invalid values
- Return type
List<float>
- get_parameters()
Returns the parameter descriptions [name, cf_convention, unit, comment] Example:
NO_P2b
mole_fraction_of_nitrogen_monoxide_in_air
ppb
Measured by IAGOS package 2b
- Returns
Parameters description
- Return type
List<string>
- get_pi()
Returns the PI of the job.
- Returns
PI of the job
- Return type
User
- get_resolution()
Returns the resolution in seconds of the data.
- Returns
Resolution in seconds
- Return type
integer
- get_scale_factors()
Returns the scale factors of the parameters.
- Returns
Scale factors
- Return type
List<float>
- get_utc_reference_date()
Returns the UTC reference date (for UTC seconds).
- Returns
UTC reference date
- Return type
datetime
- reset_parameters()
Reset the parameters of the exporter.
- Returns
None
- Return type
None
- abstract run(task, writer, destination, addition)
The duty of the exporter is to prepare the data for the export. That could be synchronizing, resampling, or just collect the data from the database. If the data is prepared, the exporter writes the data into the needed specific format. Therefore, the abstract method write of the writer will be used and the written data returned.
- Parameters
task (Task) – Related task
writer (BaseWriter) – Writer that writes the data
destination (string) – Destination of the export (e.g. filename)
addition (dictionary) – Additional parameters for the import.
- Returns
Data for the transfer
- Return type
object
ExportInfo
- class IAGOS.apps.workflow.exports.models.ExportInfo(*args, **kwargs)
Bases:
django.db.models.base.Model
This class implements the table export_info. An instance of the class stores all information about an export process.
- Parameters
export_manager (ExportManager) – Manager to export the data
destination (string) – Destionation of the export
exported_series (models.ManyToManyField) – Exported series
exported_metadata (models.ManyToManyField) – Exported metadata
min_level (integer) – Minimum level to wait for confirmation of the PI
process_status (models.ManyToManyField) – Several status of the process (e.g. missing data)
status (Status) – Status of the task (e.g. completed)
- exception DoesNotExist
Bases:
django.core.exceptions.ObjectDoesNotExist
- exception MultipleObjectsReturned
Bases:
django.core.exceptions.MultipleObjectsReturned
ExportManager
- class IAGOS.apps.workflow.exports.models.ExportManager(*args, **kwargs)
Bases:
django.db.models.base.Model
This class implements the database table exporter_manager. An instance of class handles the export process. Therefore the exporter and the writer will be used to export the data into a specific format. After that, the manager will transfer the data to the external system or store it in the volume.
See also
IAGOS.apps.workflow.transfer.models.TransferHandler*
- Parameters
destination (string) – Destination of the export
transfer_handler (TransferHandler) – Handles the transfer of the data
exporter (BaseExporter) – Exporter
writer (BaseWriter) – Writer to write the data in a specific format
parameters (models.ManyToManyField) – Parameters that should be exported
- exception DoesNotExist
Bases:
django.core.exceptions.ObjectDoesNotExist
- exception MultipleObjectsReturned
Bases:
django.core.exceptions.MultipleObjectsReturned
- close()
Closes the connection.
- Returns
None
- Return type
None
- connect()
Connects with the host.
- Returns
None
- Return type
None
- run(task, export_info)
Runs the export.
- Parameters
task (Task) – Related task
export_info (ExportInfo) – Exports info
- Returns
None
- Return type
None
Exporter
- class IAGOS.apps.workflow.exports.models.Exporter(*args, **kwargs)
Bases:
django.db.models.base.Model
This class implements the database table exporter. An instance of the model stores the metadata of an exporter. The exporter provides the data which is needed for the export. This data will be written in the required format by the writer.
See also
- Parameters
name (string) – Name of the exporter
description (string) – Description of the exporter (optional)
- exception DoesNotExist
Bases:
django.core.exceptions.ObjectDoesNotExist
- exception MultipleObjectsReturned
Bases:
django.core.exceptions.MultipleObjectsReturned
- get_error_status()
Returns the error status of the export. If there is no error, None will be returned. This method will be used by the task manager to inform the user that an error had occurred.
Important
Note that the error status describes that the whole export failed. Don’t use it for warnings!
- Returns
Status that describes why the import failed.
- Return type
- get_exported_series()
Returns all exported series.
- Returns
Exported series
- Return type
List<DataSeries>
- get_process_status()
Returns all process status of the export.
- Returns
All process status of the export
- Return type
List<ProcessStatus>
- load_instance()
Loads the exporter with the name. If the exporter is already loaded, the method doesn’t load the instance again.
Note
All exporter are defined in the module exporter.all_exporter. Make sure that your writer can be loaded by the method get_exporter_by_name.
- Returns
None
- Return type
None
- run(task, writer, destination, addition)
Runs the export process.
- Parameters
task (Task) – Related task
writer (BaseWriter) – Writer that writes the data
destination (string) – Destination of the export (e.g. filename)
addition (dictionary) – Additional parameters for the import.
- Returns
Data
- Return type
object
Writer
- class IAGOS.apps.workflow.exports.models.Writer(*args, **kwargs)
Bases:
django.db.models.base.Model
This class implements the database table writer. An instance of the model writes the data into a specific format.
See also
IAGOS.apps.workflow.exports.models.ExporterManager
- Parameters
name (string) – Name of the writer
description (string) – Description of the writer (optional)
- exception DoesNotExist
Bases:
django.core.exceptions.ObjectDoesNotExist
- exception MultipleObjectsReturned
Bases:
django.core.exceptions.MultipleObjectsReturned
- load_instance()
Loads the exporter with the name. If the exporter is already loaded, the method doesn’t load the instance again.
Note
All writers are defined in the module writers.all_writer. Make sure that your writer can be loaded by the method get_writer_by_name.
- Returns
None
- Return type
None
- write(data, data_info, filename, addition)
Writes the data into a specific format.
- Parameters
data (pd.DataFrame) – Data in a pd.DataFrame
data_info (dict) – Data info (e.g. header in Nasa-Ames)
filename (string) – Name of the file
addition (string (JSON)) – Additional information
- Returns
Data
- Return type
object