Export your Data

The ExportManager is the last instance of the task execution and combines the components Exporter and Writer. The Exporter accesses the data of the database and prepares (e.g., resample the data) it for the export. The Writer writes the prepared data into a specific format (e.g., netCDF).

Exporter

Interface

class IAGOS.apps.workflow.exports.exporter.base.BaseExporter

Bases: abc.ABC

Every exporter has to inherit from this abstract base class. This approach ensures that new exporters can be integrated into the workflow without modifying the existing code. Each exporter has to provide its metadata (NAME, DESCRIPTION) and has to implement the abstract methods run. The method prepares the data for the export. After it, the writer will be used to write the prepared data into a specific format.

Warning

If you add additional attributes in the inherited classes, make sure that you reset them in the method run. Moreover, ensure that call the super-constructor.

Parameters

NAME (string) – Name of the exporter
DESCRIPTION (string) – Description of the exporter
error_status (Status) – Error status that describes why the export failed
process_status (List(ProcessStatus)) – All process status of the import
exported_series (List(DataSeries)) – All imported series
exported_metadata (List(Metadata)) – Exported metadata

IAGOS.apps.workflow.exports.exporter.base.BaseExporter.run(self, task, writer, destination, addition)

The exporter prepares the data for the export. That could be synchronizing, resampling, or just collect the data from the database. If the data is prepared, the exporter writes the data into the needed specific format. Therefore, the abstract method write of the writer will be used and the written data returned.

Parameters

task (Task) – Related task
writer (BaseWriter) – Writer that writes the data
destination (string) – Destination of the export (e.g. filename)
addition (dictionary) – Additional parameters for the import.

Returns

Data for the transfer

Return type

object

Template

from IAGOS.apps.workflow.exports.exporter.base import BaseExporter

class ExampleExporter(BaseExporter):
   """
   Here you can describe the functionality of your exporter!
   """

   NAME = "Example"
   DESCRIPTION = "Example for demonstrations"

   def __init__(self):
      super().__init__()

   def run(self, task, writer, destination, addition):
      pass

Example & Registration

To enable a smooth start, we have prepared a simple example. The Exporter prepares a time series and additional information for the export. After it, the writer will write the data into a CSV file.

Download the Python module demo.py
Move the module to the folder application/IAGOS/apps/workflow/exports/writers
Open the module application.IAGOS.apps.workflow.exports.writers.all_writer.py

Import the writer

from IAGOS.apps.workflow.exports.writers.demo import ICHCSVWriter

Add your writer to the dictionary

WRITERS = {
   ICHCSVWriter.NAME: ICHCSVWriter
}

Register it by executing the following command in the project directory
```
make update-data
```

Writer

Interface

class IAGOS.apps.workflow.exports.writers.base.BaseWriter

Bases: abc.ABC

Every writer has to inherit from this abstract base class. This approach ensures that new writers can be integrated into the workflow without modifying the existing code. Each writer has to provide its metadata (NAME, DESCRIPTION) and has to implement the abstract method write.

Warning

If you add additional attributes in the inherited classes, make sure that you reset them in the method write. Moreover, ensure that call the super-constructor.

Parameters

NAME (string) – Name of the reader
DESCRIPTION (string) – Description of the reader

IAGOS.apps.workflow.exports.writers.base.BaseWriter.write(self, data, data_info, filename, addition)

Writes the data into a specific format.

Parameters

data (pd.DataFrame) – Data in a pd.DataFrame
data_info (dict) – Data info (e.g. header in Nasa-Ames)
filename (string) – Name of the file
addition (string (JSON)) – Additional information

Returns

Data

Return type

object

Template

from IAGOS.apps.workflow.exports.writers.base import BaseWriter

class ExampleWriter(BaseWriter):
   """
   Here you can describe the functionality of your writer!
   """

   NAME = "Example"
   DESCRIPTION = "Example for demonstrations"

   def __init__(self):
      super().__init__()

   def write(self, data, data_info, filename, addition):
      pass

Example & Registration

To enable a smooth start, we have prepared simple examples. We have prepared a writer that writes the data into a csv format with additional information.

Download the Python module demo.py
Move the module to the folder application/IAGOS/apps/workflow/exports/writers
Open the module application.IAGOS.apps.workflow.exports.writers.all_writer.py

Import the writer

from IAGOS.apps.workflow.exports.writers.demo import ICHCSVWriter

Add your writer to the dictionary

WRITERS = {
   ICHCSVWriter.NAME: ICHCSVWriter
}

Register it by executing the following command in the project directory
```
make update-data
```