IAGOS.apps.workflow.exports

Overview

../_images/export.svg

The ExportManager is the last instance of the task execution and combines the components Exporter and Writer. The Exporter accesses the data of the database and prepares (e.g., resample the data) it for the export. The Writer writes the prepared data into a specific format (e.g., netCDF).

Interfaces

Interface - Writer

This module defines the interface for the writers. Each writer has to inherit from the abstract base class BaseWriter. This approach makes sure that the writer can be integrated into the workflow without modifying code. Each writer has to provide its metadata. Moreover, each writer has to implement the abstract method perform which writes the given data into a specific format (e.g., netCDF).

class IAGOS.apps.workflow.exports.writers.base.BaseWriter

Every writer has to inherit from this abstract base class. This approach ensures that new writers can be integrated into the workflow without modifying the existing code. Each writer has to provide its metadata (NAME, DESCRIPTION) and has to implement the abstract method write.

Warning

If you add additional attributes in the inherited classes, make sure that you reset them in the method write. Moreover, ensure that call the super-constructor.

Parameters
  • NAME (string) – Name of the reader

  • DESCRIPTION (string) – Description of the reader

check_description()

Checks if the DESCRIPTION is set and contains at least 1 character. If the DESCRIPTION is not valid, the method will raise a ValueError or a TypeError.

Raises
  • ValueError – If DESCRIPTION is not set or empty

  • TypeError – If DESCRIPTION is not a string

Returns

None

Return type

None

check_metadata()

Checks if the metadata is correct.

Raises
  • ValueError – If NAME is not set or empty

  • TypeError – If NAME is not a string

  • ValueError – If DESCRIPTION is not set or empty

  • TypeError – If DESCRIPTION is not a string

Returns

None

Return type

None

check_name()

Checks if the NAME is set and contains at least 1 character. If the NAME is not valid, the method will raise a ValueError or a TypeError.

Raises
  • ValueError – If NAME is not set or empty

  • TypeError – If NAME is not a string

Returns

None

Return type

None

abstract write(data, data_info, filename, addition)

Writes the data into a specific format.

Parameters
  • data (pd.DataFrame) – Data in a pd.DataFrame

  • data_info (dict) – Data info (e.g. header in Nasa-Ames)

  • filename (string) – Name of the file

  • addition (string (JSON)) – Additional information

Returns

Data

Return type

object

Interface - Exporter

In this module, the base classes for all exporters are implemented. Every exporter has to be inherited from one of theses classes. With this approach, you can implement a new exporter without changing the existing code. Each exporter has to provide its metadata. Furthermore, every exporter has to implement the method run that executes the export process.

class IAGOS.apps.workflow.exports.exporter.base.BaseExporter

Every exporter has to inherit from this abstract base class. This approach ensures that new exporters can be integrated into the workflow without modifying the existing code. Each exporter has to provide its metadata (NAME, DESCRIPTION) and has to implement the abstract methods run. The method prepares the data for the export. After it, the writer will be used to write the prepared data into a specific format.

Warning

If you add additional attributes in the inherited classes, make sure that you reset them in the method run. Moreover, ensure that call the super-constructor.

Parameters
  • NAME (string) – Name of the exporter

  • DESCRIPTION (string) – Description of the exporter

  • error_status (Status) – Error status that describes why the export failed

  • process_status (List(ProcessStatus)) – All process status of the import

  • exported_series (List(DataSeries)) – All imported series

  • exported_metadata (List(Metadata)) – Exported metadata

check_description()

Checks if the DESCRIPTION is set and contains at least 1 character. If the DESCRIPTION is not valid, the method will raise a ValueError or a TypeError.

Raises
  • ValueError – If DESCRIPTION is not set or empty

  • TypeError – If DESCRIPTION is not a string

Returns

None

Return type

None

check_metadata()

Checks if the metadata is correct.

Raises
  • ValueError – If NAME is not set or empty

  • TypeError – If NAME is not a string

  • ValueError – If DESCRIPTION is not set or empty

  • TypeError – If DESCRIPTION is not a string

Returns

None

Return type

None

check_name()

Checks if the NAME is set and contains at least 1 character. If the NAME is not valid, the method will raise a ValueError or a TypeError.

Raises
  • ValueError – If NAME is not set or empty

  • TypeError – If NAME is not a string

Returns

None

Return type

None

get_error_status()

The method returns the error status of the export. If there is no error, None will be returned. This method will be used by the task manager to inform the user that an error had occurred during the export process.

Important

Note that the error status describes that the whole export failed. Don’t use it as a warning.

Returns

Status that describes why the export failed.

Return type

Status

get_exported_metadata()

Returns all exported metadata.

Returns

Exported metadata

Return type

List<Metadata>

get_exported_series()

Returns all exported series.

Returns

Exported series

Return type

List<DataSeries>

get_process_status()

Returns all process-status of the export.

Returns

All process-status of the export

Return type

List<ProcessStatus>

reset_parameters()

Reset the parameters of the exporter.

Returns

None

Return type

None

abstract run(task, writer, destination, addition)

The exporter prepares the data for the export. That could be synchronizing, resampling, or just collect the data from the database. If the data is prepared, the exporter writes the data into the needed specific format. Therefore, the abstract method write of the writer will be used and the written data returned.

Parameters
  • task (Task) – Related task

  • writer (BaseWriter) – Writer that writes the data

  • destination (string) – Destination of the export (e.g. filename)

  • addition (dictionary) – Additional parameters for the import.

Returns

Data for the transfer

Return type

object

class IAGOS.apps.workflow.exports.exporter.base.BaseSeriesExporter

This class implements the base for the series exporter and inherits from the base class BaseExporter. The exporter for the series provides more information about the data.

Note

The attributes NAME and DESCRIPTIONS are class attributes. That means that all readers of the same class, refer always the same NAME and the DESCRIPTION.

Warning

If you add additional attributes in the inherited classes, make sure that you reset them in the method read. Otherwise this could create side effects and is very error prone.

Parameters
  • NAME (string) – Name of the exporter

  • DESCRIPTION (string) – Description of the exporter

  • error_status (Status) – Error status that describes why the export failed

  • process_status (List<ProcessStatus>) – All process status of the import

  • exported_series (List<DataSeries>) – All imported series

  • exported_metadata (List<Metadata>) – Exported metadata

get_comments()

Returns additional comments.

Returns

Comments

Return type

List<string>

get_data_availability()

Returns the data availabilities. One value (percentage of availability, integer between 0 and 1)

Returns

Data availability

Return type

List<float>

get_data_description()

Returns data descriptions.

Returns

Data descriptions

Return type

List<string>

get_data_info()

Returns all information about the data.

Returns

Data information

Return type

dict

get_data_level()

Returns the data level of the data.

Returns

Data level

Return type

integer

get_deployment()

Returns the deployment.

Returns

Related deployment

Return type

Deployment

get_flight()

Returns the flight.

Returns

Related flight

Return type

Flight

get_imported_series(task, name, exclude=None)

Returns the series with the given name. Optional, you can use the exclude to define a part of the name, that shouldn’t be considered.

Note

If the series doesn’t exist, the previous task will be considered.

Parameters
  • task (Task) – Related task

  • name (string) – Name of the series

  • exclude (string) – Exclude string (optional)

Returns

Series

Return type

DataSeries

get_instrument()

Returns the string representation of the instrument.

Returns

Representation of the instrument

Return type

string

get_invalid_values()

Returns the values that mark invalid values (e.g. -9999).

Returns

Invalid values

Return type

List<float>

get_parameters()

Returns the parameter descriptions [name, cf_convention, unit, comment] Example:

  • NO_P2b

  • mole_fraction_of_nitrogen_monoxide_in_air

  • ppb

  • Measured by IAGOS package 2b

Returns

Parameters description

Return type

List<string>

get_pi()

Returns the PI of the job.

Returns

PI of the job

Return type

User

get_resolution()

Returns the resolution in seconds of the data.

Returns

Resolution in seconds

Return type

integer

get_scale_factors()

Returns the scale factors of the parameters.

Returns

Scale factors

Return type

List<float>

get_utc_reference_date()

Returns the UTC reference date (for UTC seconds).

Returns

UTC reference date

Return type

datetime

reset_parameters()

Reset the parameters of the exporter.

Returns

None

Return type

None

abstract run(task, writer, destination, addition)

The duty of the exporter is to prepare the data for the export. That could be synchronizing, resampling, or just collect the data from the database. If the data is prepared, the exporter writes the data into the needed specific format. Therefore, the abstract method write of the writer will be used and the written data returned.

Parameters
  • task (Task) – Related task

  • writer (BaseWriter) – Writer that writes the data

  • destination (string) – Destination of the export (e.g. filename)

  • addition (dictionary) – Additional parameters for the import.

Returns

Data for the transfer

Return type

object

ExportInfo

class IAGOS.apps.workflow.exports.models.ExportInfo(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the table export_info. An instance of the class stores all information about an export process.

Parameters
  • export_manager (ExportManager) – Manager to export the data

  • destination (string) – Destionation of the export

  • exported_series (models.ManyToManyField) – Exported series

  • exported_metadata (models.ManyToManyField) – Exported metadata

  • min_level (integer) – Minimum level to wait for confirmation of the PI

  • process_status (models.ManyToManyField) – Several status of the process (e.g. missing data)

  • status (Status) – Status of the task (e.g. completed)

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

ExportManager

class IAGOS.apps.workflow.exports.models.ExportManager(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table exporter_manager. An instance of class handles the export process. Therefore the exporter and the writer will be used to export the data into a specific format. After that, the manager will transfer the data to the external system or store it in the volume.

See also

Parameters
  • destination (string) – Destination of the export

  • transfer_handler (TransferHandler) – Handles the transfer of the data

  • exporter (BaseExporter) – Exporter

  • writer (BaseWriter) – Writer to write the data in a specific format

  • parameters (models.ManyToManyField) – Parameters that should be exported

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

close()

Closes the connection.

Returns

None

Return type

None

connect()

Connects with the host.

Returns

None

Return type

None

run(task, export_info)

Runs the export.

Parameters
  • task (Task) – Related task

  • export_info (ExportInfo) – Exports info

Returns

None

Return type

None

Exporter

class IAGOS.apps.workflow.exports.models.Exporter(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table exporter. An instance of the model stores the metadata of an exporter. The exporter provides the data which is needed for the export. This data will be written in the required format by the writer.

Parameters
  • name (string) – Name of the exporter

  • description (string) – Description of the exporter (optional)

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

get_error_status()

Returns the error status of the export. If there is no error, None will be returned. This method will be used by the task manager to inform the user that an error had occurred.

Important

Note that the error status describes that the whole export failed. Don’t use it for warnings!

Returns

Status that describes why the import failed.

Return type

Status

get_exported_series()

Returns all exported series.

Returns

Exported series

Return type

List<DataSeries>

get_process_status()

Returns all process status of the export.

Returns

All process status of the export

Return type

List<ProcessStatus>

load_instance()

Loads the exporter with the name. If the exporter is already loaded, the method doesn’t load the instance again.

Note

All exporter are defined in the module exporter.all_exporter. Make sure that your writer can be loaded by the method get_exporter_by_name.

Returns

None

Return type

None

run(task, writer, destination, addition)

Runs the export process.

Parameters
  • task (Task) – Related task

  • writer (BaseWriter) – Writer that writes the data

  • destination (string) – Destination of the export (e.g. filename)

  • addition (dictionary) – Additional parameters for the import.

Returns

Data

Return type

object

Writer

class IAGOS.apps.workflow.exports.models.Writer(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table writer. An instance of the model writes the data into a specific format.

See also

Parameters
  • name (string) – Name of the writer

  • description (string) – Description of the writer (optional)

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

load_instance()

Loads the exporter with the name. If the exporter is already loaded, the method doesn’t load the instance again.

Note

All writers are defined in the module writers.all_writer. Make sure that your writer can be loaded by the method get_writer_by_name.

Returns

None

Return type

None

write(data, data_info, filename, addition)

Writes the data into a specific format.

Parameters
  • data (pd.DataFrame) – Data in a pd.DataFrame

  • data_info (dict) – Data info (e.g. header in Nasa-Ames)

  • filename (string) – Name of the file

  • addition (string (JSON)) – Additional information

Returns

Data

Return type

object