.. _import_interface:

****************
Import your Data
****************

The **ImportManager** combines the components **Reader** and **Importer**. The
**Reader** reads the data and parses it to a standardized format that the
**Importer** can interpret. The Importer imports and processes the parsed
data. Since the importer expects the standardized format, it doesn't have
to know the format. This allows you to implement different **Readers** for
different formats and to use the same importer. If you import processed data
or final data, you can ignore the data processing part. Within the **IAGOS**
project, the raw data will be read and then processed. The data processing
is a part of the **Importer** because in some use cases the data needs to
be synchronized. It's recommended to synchronize the data first and then to
import only the synchronized data. 

.. figure:: ../../graphics/import_steps.svg
   :width: 100%

Reader
======

Interface
---------

.. autoclass:: IAGOS.apps.workflow.imports.readers.base.BaseReader
   :show-inheritance:
   :noindex:

.. autofunction:: IAGOS.apps.workflow.imports.readers.base.BaseReader.read
    :noindex:

.. autofunction:: IAGOS.apps.workflow.imports.readers.base.BaseReader.read_list
    :noindex:

.. autofunction:: IAGOS.apps.workflow.imports.readers.base.BaseReader.get_source_name
    :noindex:

Template
--------

.. code-block:: python

   from IAGOS.apps.workflow.imports.readers.base import BaseReader

   class ExampleReader(BaseReader):
      """
      Here you can describe the functionality of your reader!
      """

      NAME = "Example"
      DESCRIPTION = "Example for demonstrations"

      def __init__(self):
         super().__init__()

      @staticmethod
      def read_list(source, additional_source, additional_info):
         pass

      def read(self, source):
         pass

      def get_source_name(self):
         pass

.. _reader_example:

Example & Registration
----------------------

To enable a smooth start, we have prepared simple examples. We have prepared a
reader to read the metadata of the ICH unit (instrument, deployment, calibration)
from an Excel file :download:`ich_metadata.xls <../../media/ich_metadata.xls>`
and a reader to read a measured time series from a netCDF file
:download:`H2O2018061616204002.nc <../../media/H2O2018061616204002.nc>`. With the
following steps, you can register them.

1. Download the Python module :download:`demo.py <../../media/demo_reader.py>`

2. Move the module to the folder **application/IAGOS/apps/workflow/imports/readers**

3. Open the module **application.IAGOS.apps.workflow.imports.readers.all_readers.py**

4. Import the readers

   .. code-block:: python

      from IAGOS.apps.workflow.imports.readers.demo import ICHMetaDataReader
      from IAGOS.apps.workflow.imports.readers.demo import ICHSeriesReader

5. Add your reader to the dictionary

   .. code-block:: python

      READERS = {
         ICHMetaDataReader.NAME: ICHMetaDataReader,
         ICHSeriesReader.NAME: ICHSeriesReader,
      }

6. Register them by executing the following command in the project directory

   .. code:: console

      make update-data

Importer
========

Interface
---------

.. autoclass:: IAGOS.apps.workflow.imports.importer.base.BaseImporter
   :show-inheritance:
   :noindex:

.. autofunction:: IAGOS.apps.workflow.imports.importer.base.BaseImporter.run
    :noindex:

.. autofunction:: IAGOS.apps.workflow.imports.importer.base.BaseImporter.rerun
    :noindex:

Template
--------

.. code-block:: python

   from IAGOS.apps.workflow.imports.importer.base import BaseImporter

   class ExampleImporter(BaseImporter):
      """
      Here you can describe the functionality of your importer!
      """

      NAME = "Example"
      DESCRIPTION = "Example for demonstrations"

      def __init__(self):
         super().__init__()

      def run(self, reader, parameters, flags, addition):
         pass

      def rerun(self, task, reader, parameters, flags, addition):
         pass

Example
-------

To enable a smooth start, we have prepared simple examples. We have prepared an
importer to read the parsed data and an importer to read the
time series (see also: :ref:`reader_example`).

1. Download the Python module :download:`demo_importer.py <../../media/demo_importer.py>`

2. Move the module to the folder **application/IAGOS/apps/workflow/imports/importer**

3. Open the module **application.IAGOS.apps.workflow.imports.importer.all_importer.py**

4. Import your importer

   .. code-block:: python

      from IAGOS.apps.workflow.imports.importer.demo import ICHMetadataImporter
      from IAGOS.apps.workflow.imports.importer.demo import ICHSeriesImporter

5. Add them to the dictionary

   .. code-block:: python

      IMPORTERS = {
         ICHMetadataImporter.NAME: ICHMetadataImporter,
         ICHSeriesImporter.NAME: ICHSeriesImporter,
      }

6. Register them by executing the following command in the project directory

   .. code:: console

      make update-data