IAGOS.apps.workflow.transfer

Overview

../_images/transfer.PNG

The TansferHandler handles the communication between the application and external systems. It is used by the ImportManager and the ExportManager. The handler combines the three components TransferType, ConnectionHandler and DataProvider. The component TransferType defines the transfer protocol (e.g., REST). Note that the transfer type is an interface that could be extended with new protocols if needed. The ConnectionHandler handles the authentication and the connection to the external systems. Note that the authentication method often differs between the systems for security purposes. Therefore the component provides an interface that allows adding new authentication methods. In many use cases, only the authentication method differs. Assume you have to request a token for authentication; you can use OAuth2 or connect directly with your credentials. But after the token is requested, the REST requests would be the same. Thus the type and authentication are split into two components which can be combined individually. Finally, there is the component DataProvider. It records only some information about the external data provider.

Interfaces

Interface - TransferType

In this module, all transfer types will be implemented. A transfer type handles communication with other systems. It could be the FTP protocol, a REST-API, and also a simple directory. Every type has to inherit from the base class and implements the interface.

class IAGOS.apps.workflow.transfer.transfer_types.base.BaseTransferType

This class implements the base for every transfer type.

Parameters

  • NAME (string) – Name of the handler

  • DESCRIPTION (string) – Description of the handler

get(source, connection)

Returns the data of the source.

Parameters

  • source (string) – Source of the data

  • connection (object) – Connection (e.g. Token or FTP connection)

Returns

Data

Return type

object

static get_description()

Returns the description of the transer.

Returns

Description of the transfer type

Return type

string

get_list(source, connection, addition, timestamp)

Returns a list of sources. For example, if the source is a directory, the method will return all directory files.

Parameters

  • source (string) – Source (e.g. directory)

  • connection (object) – Connection (e.g. Token or FTP connection)

  • addition (string (JSON)) – Additional parameters (e.g. Filter attributes)

  • timestamp (datetime) – Timestamp to detect new data

Returns

List of the sources

Return type

object

static get_name()

Returns the name of the transfer type.

Returns

Name of the transfer type

Return type

string

post(destination, data, connection)

Send the data to the destination.

Parameters

  • destination (string) – Destination of the data

  • data (object) – Data that should be transfered

  • connection (object) – Connection (e.g. Token or FTP connection)

Returns

Destination

Return type

string

put(destination, data, connection)

Updates the destination’s data.

Parameters

  • destination (string) – Destination of the data

  • data (object) – Data that should be transfered

  • connection (object) – Connection (e.g. Token or FTP connection)

Returns

Destination

Return type

string

Interface - ConnectionHandler

In this module, the interface for the connection methods is implemented. Some systems need a special connection method to increase security. The base class defines the interface to make sure that the workflow could use every connection method. Therefore each connection method has a unique name and a short description. Moreover, each method has to implement the abstract methods connect and close.

class IAGOS.apps.workflow.transfer.connection_handler.base.BaseConnectionHandler

This class implements the interface for the connection methods. Every subclass of the base has to implement the methods get_name, get_description, connect, and close.

Parameters

  • NAME (string) – Name of the handler

  • DESCRIPTION (string) – Description of the handler

close(connection)

Closes the connection with the host.

Parameters

connection (object) – Connection

Returns

None

Return type

None

connect(host=None, user=None, password=None, addition=None)

Connects with the given host.

Parameters

  • host (string) – URL for login

  • user (string) – Username

  • password (string) – Userpassword

  • addition (string (JSON)) – Additional information (e.g. Port)

Returns

Connection (e.g. Token) & True if connected, False otherwise

Return type

(object, bool)

static get_description()

Returns the description of the connection method.

Returns

Name of the connection method

Return type

string

static get_name()

Returns the name of the connection method.

Returns

Name of the connection method

Return type

string

is_valid(connection)

This method returns the status of the connection. If the connection is still valid, the method will return True. Otherwise, the method will return False. Note that some connections are limited in time and have a timeout.

Parameters

connection (object) – Connection

Returns

True if valid, False otherwise.

Return type

bool

ConnectionHandler

class IAGOS.apps.workflow.transfer.models.ConnectionHandler(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table connection_method. Often systems use its login method to increase the security of the system. The class stores the metadata of the connection method.

Parameters

  • name (string) – Name of the provider (e.g. IAGOS Data Portal)

  • description (string) – Description of the provider [optional]

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

close()

Closes the connection.

Returns

None

Return type

None

connect(host=None, user=None, password=None, addition=None)

Connects with the given host.

Parameters

  • host (string) – URL for login

  • user (string) – Username

  • password (string) – Userpassword

  • addition (string (JSON)) – Additional information (e.g. Port)

Returns

Connection (e.g. Token) & True if connected, False otherwise

Return type

(object, bool)

DataProvider

class IAGOS.apps.workflow.transfer.models.DataProvider(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table data_provider. Each provider has a name and a description. The data provider can be used for imports and exports. An example of a provider is the IAGOS Data Portal or Package 2b.

Parameters

  • name (string) – Name of the provider (e.g. IAGOS Data Portal)

  • description (string) – Description of the provider [optional]

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

TransferHandler

class IAGOS.apps.workflow.transfer.models.TransferHandler(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table transfer. An instance of the model stores the transfer information and handle the transfer. The information includes the data provider, the username, the password, and the transfer type. Note that the user and the password are only required when the application needs to authenticate.

Parameters

  • data_provider (DataProvider) – Data provider (e.g. IAGOS PORTAL)

  • transfer_type (TransferType) – Transfer type (e.g. REST-API)

  • connection_method (Special login method [optional]) – ConnectionMethod

  • auth_link (string) – Authentication link [optional]

  • user (string) – Username [optional]

  • password (string) – Password [optional]

  • addition (string (JSON)) – Additional information for the connection (e.g. Port)

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

close()

Closes the connection.

Returns

None

Return type

None

connect()

Connects with the host.

Returns

None

Return type

None

get(source)

This method requires the data from the source. If authentication is needed, the method uses the attribute user and password.

Parameters

source (string) – Source (e.g. REST-API request)

Returns

Data

Return type

object

get_list(source, addition, timestamp)

Returns a list of sources. For example, if the source is a directory, the method will return all directory files.

Parameters

  • source (string) – Source (e.g. REST-API request)

  • addition (string (JSON)) – Additional parameters (e.g. Filter attributes)

  • timestamp (datetime) – Timestamp to detect new data

Returns

Data

Return type

object

post(destination, data)

Send the data to the source. If authentication is needed, the method uses the attribute user and password.

Parameters

  • destination (string) – Destination of the data (e.g. Filename)

  • data (File or similar) – Written data

Returns

Destination

Return type

string

put(destination, data)

Updates the destination’s data.

Parameters

  • destination (string) – Destination of the data

  • data (object) – Data that should be transfered

Returns

Destination

Return type

string

TransferType

class IAGOS.apps.workflow.transfer.models.TransferType(*args, **kwargs)

Bases: django.db.models.base.Model

This class implements the database table transfer_type. An instance of the model is used by the importer to ensure how the data can be accessed. A transfer type has a name and a short description. An example of a transfer is REST-API.

Parameters

  • name (string) – Name of the transfer type (e.g. FTP or REST-API)

  • description (string) – Description of the provider [optional]

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

get(source, connection)

Returns the data of the source.

Parameters

  • source (string) – Source of the data

  • connection (object) – Connection (e.g. Token or FTP connection)

Returns

Data

Return type

object

get_list(source, connection, addition, timestamp)

Returns a list of sources. For example, if the source is a directory, the method will return all directory files.

Parameters

  • source (string) – Source (e.g. REST-API request)

  • connection (object) – Connection (e.g. Token or FTP connection)

  • addition (string (JSON)) – Additional parameters (e.g. Filter attributes)

  • timestamp (datetime) – Timestamp to detect new data

Returns

List of the sources

Return type

object

load_instance()

Loads an instance of the transfer type. Therefore the method get_transfer_type_by_name is used, which returns the related class.

Returns

None

Return type

None

post(destination, data, connection)

Send the data to the destination.

Parameters

  • destination (string) – Destination of the data

  • data (object) – Data that should be transfered

  • connection (object) – Connection (e.g. Token or FTP connection)

Returns

Destination

Return type

string

put(destination, data, connection)

Updates the destination’s data.

Parameters

  • destination (string) – Destination of the data

  • data (object) – Data that should be transfered

  • connection (object) – Connection (e.g. Token or FTP connection)

Returns

Destination

Return type

string