TIXstream MFT – Installation Guide

The Linux install script specifies the service account tixstream as well as the group of the same name, under which the MFT services are started. If the account of such user already exists on the system, it will be used without any changes.

The transfer system services require a Windows user account that has access to the directories of the storage systems (e.g. CIFS) used for transfers. Administrative rights are not required. In the following, it is assumed that this is the user tixstream.

The following is an example of the steps required to create a local user account tixstream with password xsecret. These can be skipped if the account to be used has already been created by Windows administration.

A license is required to operate the MFT system. It is alternatively provided as a USB dongle (license dongle) or as a license file (license file). In both cases, Wibu’s CodeMeter software is used.

Depending on the licensing medium, a parameter in the configuration file tixstream.conf (in the directory config ) has to be adapted or created:

License file (default if not specified):

License dongle:

Explanations on installation and configuration can be found in the sections on the operating systems.

Evaluation licenses do not contain any restrictions, apart from the expiration time. Upon purchasing, they are exchanged for permanent licenses for production. However the basic licensing mechanism remains the same.

Regular USB dongles contain a permanent license (without expiration time), which is not linked to a specific system. If an MFT system fails, you can start up an identical configured cold standby system by (physically or virtually) re-plugging the USB dongle.

There are also dongles with cold standby licenses available. These are valid for a period of 14 days starting at the time of first use, i.e. access by the application. Thus a cold standby system can be put into operation in the event of a failure of the primary system without re-plugging dongles. Within the 14-day period, the dongle with the permanent license can be plugged into the active system or the primary system can be put into operation again.

No dongle is required for installation and configuration. Therefore, a cold standby system can also be set up (usually by cloning the primary system), without the need for a dongle at all. A valid license is only required for actual data transfer.

A cold standby dongle can be provided with a new license again after expiration of the usage period. Please contact TIXEL.

In addition to a standard Windows system, the following software is required:

The installation archive contains the TIXEL software packages of the application stack of an MFT system and some utilities (tools).

For installation, the archive is unpacked into any directory, here, for example, D:\sw\. The created main folder (e.g. tixstream-mft-windows-bundle-v1.2.3-0-gda4787b) contains an installation script install.cmd.

Please execute this script with run as administrator:

This results in the following folder structure, e.g. in`D:\bin\tixel`:

A license file (*.WibuCmRaU) may be provided separately.

The content in logs and tmp can be cleared for housekeeping purposes. The location for directories can be configured in the .properties files. However, if any changes are made, please keep in mind possible dependencies of other services.

The read/write and execution rights should be restricted to the user tixstream so that regular users may not have access to configuration files.

Please adapt the following two parameters to your system:

The following script removes the MFT windows services from the system.

The script stops the services (please ignore error messages if the services have already been inactive) and removes them from the system. The programs files will not be removed.

Under certain circumstances (e.g. active service panel window, "Computer Management") the services can not be immediately removed from the system. In this case they are marked as "for removal" and effectively removed later.

In addition to the Windows services, the system-wide environment variable TIXEL_HOME will also be unset.

TIXstream MFT provides the option to be operated on HA cluster nodes. For details please refer to: TIXstream MFT Cluster Setup for Windows.

In addition to a CentOS7 or an Ubuntu 18.04 base OS installation (package selection "minimal") the following packages will be required.

The installation package is available as a tgz archive and a checksum file ( *.tgz.sha256). After downloading the checksum should be verified:

The following section describes the initial installation of the MFT software. The update of an existing installation is similar, see section Update.

The archive will be unpacked, e.g. in the home directory of user tixel, $HOME (/home/tixel).

The new directory tixel now contains the following data:

This directory also contains the installation script install.sh.

The following command is used to install the services as root in /opt/tixel/.

Call the install script with the --help or -h parameter to see other available options, e.g. to provide user, group and umask settings for the services.

Please follow the instructions for switching between license dongle and license file (see License).

The license will be activated (1 successful update done) and is available for use with TIXstream.

The following command is used to verify license installation:

The output includes e.g. the following information (here: evaluation license):

For permanent licenses no expiration date will be displayed.

Finally it can be verified if a TIXstream license is (still) valid. For this purpose the following command displays TIXstream version information. In case of an invalid license or an inactive Wibu runtime TIXstream refuses to start.

If the current version along with the license information is displayed, the license has been successfully installed.

The expiration date of a Cold Standby License can be check with cmu -l -x. The output shows information about the usage period Usage Period: 14 days (1209600 seconds) as well as the activation status (not yet activated).

You can explicitly delete temporary licenses from the system. However, this is not required, as a current license will automatically replace an expired one.

To check all current installed licenses and the respective serial numbers use the command cmu --list. The following command (replace NNN-NNNNNNN by a valid serial number) removes a license permanently from the system.

In the following section the host name mft01.example.com and system alias (name) mft01 will be used as an example for the MFT system.

The install script already transmits the current host name and the derived system alias in the configuration file. The necessary changes to these parameters are described as follows.

If necessary, adjust the following two entries

in the file /opt/tixel/config/transfer-job-manager.properties , e.g.:

Changes will be effective after restarting of the transfer job manager (systemctl restart transfer-job-manager)

After install.sh has been executed, all services can be started (as root) with /opt/tixel/admin/ctl-services.sh start; alternatively the services will be started automatically by rebooting of the system. Stopping the services can be done with /opt/tixel/admin/ctl-services.sh stop.

The services

can be controlled individually with the commands systemctl start, stop, restart, status. journalctl shows log information, e.g.

Services are deactivated by systemctl disable <servicename>. Thus a service does not start automatically with the system start. To (re-) activate the service, use systemctl enable <servicename>.

To perform actions on all TIXEL services simultaneously, the following script can be used encompassing all of the mentioned systemctl functions (start, stop, restart, status, disable, enable):

The following TCP ports are used for communication among MFT systems:

The ports are configured in the .properties-files of the respective services. Please note that manual changes might require further adjustments to other files or systems. The configuration files of the installation package are pre-configured, so that usually no changes are necessary.

Currently there is only one hostname per machine supported.

If the computer cannot contact itself via the public DNS name, it must be registered in the local hosts file and point to the IP address used for communication with other transfer systems.

Therefore please edit (as root or administrator) the file /etc/hosts (Linux) or C:\Windows\System32\drivers\etc\hosts (Windows) and create an entry with IPv4 address (here: 10.10.10.48) and its hostname (here mft01.sample.com):

TIXEL Control Center enables you to configure TIXEL services via a web interface. The respective access to the Web UI for a transfer system administrator is designed in a way that it is configured and available without external services being operational. Thus it can be used independent from data base servers, directory services (LDAP) as well as Windows user accounts.

For using the Web UI you need a recent web browser (Firefox, Chrome, Safari, Edge, IE) with activated JavaScript (IE: "Active Scripting"). In case of malfunction with older versions of IE, the entry add_header X-UA-Compatible "IE=edge"; in the section server of the configuration file nginx.conf might help.

Open the TIXEL Control Center URL http://localhost:9010/tixel-control-center/ with a web browser. You can login with default credentials (user admin, password verysecret). The password can be changed via the top right User icon.

The user admin has rights for both, configure the transfer system as well as to manage (create, monitor and control) file transfers.

The http access on port 9010 works directly on the MFT system (localhost or 127.0.0.1). If you have configured https, you can also access the web UI via https://<FQDN>/tixel-control-center/ using standard https port 443. Depending on configuration, http access from other systems might be denied, so that only https access will be available in this case.

Local and CIFS shares can be configured here. Click on the + symbol and enter values for Name, Local Path and Public URI, e.g.

Here videoserver is the network hostname of the CIFS share in the LAN, scratch`is the name of the CIFS share (in the notion of CIFS/Samba). The full local path can be accessed via directory browser `…​. On Linux, a CIFS share has to be mounted first, e.g. as /mnt/cifs/videoserver/scratch, also see Local Path.

In Permissions tick checkbox Write. Values for Description and Expiration Date are optional. Click Save to enable the new share.

Creating a share for an FTP server is similar. While CIFS credentials are handled by the operating system, for an FTP share they are handled by MFT.

Use of SFTP is similar to FTP. The default port is usually the SSH port (22).

For SFTP access currently only username/password authentication is supported. The Microsoft PowerShell OpenSSH implementation has some limitations resulting in limited functionality: a root folder must be specified and it is not possible to select subfolders in TCC.

By clicking + , in the field Address you may enter a Service-URI of a Node. Even for local transfers your own system has to be entered here. The address has the format http://<FQDN>:60000/transfer-job-manager/v1(or https and Port 60001), e.g.

If system to be added is online, the stored system name and its associated address appears in the node list. The name of the other system and its configured destination shares will automatically be made available to the local system. If the remote system is not accessible (offline, missing network connection) a corresponding error message will be displayed.

It is recommended to set up the system list by using one of the import functions described below. Already configured systems, which are not included in the import list, are not deleted but remain and can be deleted individually if required. The list contains one line per address in the above described format.

With Import you can select a local file which contains the service URIs of transfer systems line by line. Each line contains a transfer system URI in the above format.

You can also retrieve a transfer system list via network. The format corresponds to the one of the local file. The address used in this case can be configured in the system administrator section (see TIXEL Control Center)

Click on the cloud icon with the downward-pointing arrow. No systems will be deleted during an import, but new systems will be added only.

TIXstream MFT allows the administrator to configure certain environment specific capabilities on per-node basis. After a node has been imported, parameters can be configured which are applied to all transfers with this specific node.

Before each transfer sender and receiver negotiate these parameters, the integer values are determined by using the minimum of sender and receiver parameter. Thereby the sender uses the outgoing parameters defined by the administrator of the sender node for the receiver node. The receiver uses the incoming parameters defined by the administrator of the receiver node for the sender node.

Based on the policy parameters, a boolean value is determined which either enables or disables a feature. The following table shows how the boolean value is determined.

The transfer strategy is determined based on the following table.

In both cases the sender is the dominant part. In case of an error, the transfer is rejected and a policy mismatch is reported.

Deleting systems is done by selecting them in the list and clicking on the trash can icon. Already created transfers using this system are no longer possible.

A deleted system can be re-entered at any time resulting in a procedure exactly like adding a new system.

Deletion is affecting the local system only, i.e it has no impact on the referred (deleted) system other than that no transfers between the two systems are possible anymore.

The general availability of the remote system can be checked with ping.

In many networks as the ICMP messages used by the standard Ping are discarded, the use of a TCP-based Ping tool is recommended. Under Windows, for example, psping from Sysinternals can be used (PsTools from http://www.sysinternals.com).

The test should be carried out by each participating sender and receiver system on ports 60000 and 60002 to ensure the communication capability of the Transfer Job Manager (60000) and the transfer engine (60002).

The ports used for the actual file transfer (60004-60009) can not be verified with psping because they are only active at the beginning of a file transfer.

Permanent Ping, quit with Ctrl-C:

limited to 10 iterations:

Example output:

The following list contains all available configuration parameters and their default values for MFT. All parameters that are unavailable in the GUI can be modified directly in the .properties file.

The password strength can be defined by specifying a regular expression (regex). The expression in the standard configuration provides the following password requirements:

For username the expression ^[A-Za-z0-9_@.-]{4,64}$ is predefined. This applies:

See section Database Setup.

For authentication and authorization with LDAP see section LDAP.

The parameter custom.security.ldap.enabled may contain the following values:

In the Transfer Job Manager the following basic settings are available. Default values are shown in parentheses.

On a system running https only, the Listen Address can be limited to localhost (127.0.0.1). Note that the http access to the TJM just like the TCC is also realized by the nginx proxy port 80. However, if systems (other transfer systems or local third party systems) use http and port 60000 for access, leave the setting, using the default 0.0.0.0.

Please note that the maximum number of simultaneous outgoing transfers (e.g. 5) comprises already the Number of reserved priorization slots (e.g. 1). Using the previous example, without prioritization, 4 outgoing transfers would be processed in parallel.

See section database setup . Database-Setup.

E-mail settings are used for sending transfer system notifications.

Templates for e-mail notifications can be saved in the folder config/templates which is in the TIXEL installation directory.

Example JobStatusNotification.html:

In the template, following variables can be used:

The following is an example notification of a completed and confirmed transfer:

Templates are read at the service start. Thus for changes to take effect, please restart the Transfer Job Manager service.

The following settings are related to the transfer engine (TIXstream).

Details regarding rescheduling interval: if a transfer job can not be started because the sender is busy with other jobs, the receiver will start a new transfer attempt after a random period of time between 1 and 5 minutes. The upper and lower limits of this interval can be set, wherein the minimum value is 45s.

A list of all configuration parameters and their default settings:

The maximum metadata size can be limited (similar to whitelist/blacklist) on the input side. Note that for using of large datasets (greater than 400 KBytes) several conditions must be met:

In general, use of large metadata records can delay transfer job creation and further processing. If any abortions occur due to timeouts, adjust the parameters custom.transfer-job-manager.rest-connect-timeout-ms or, in particular,` custom.transfer-job-manager.rest-read-timeout-ms`. The system was originally designed for metadata sets of up to 400000 characters. Notably when processing jobs with up to 300 files and complex XML schema, data sets with the size of up to 8 megabytes can occur.

Explanation of internal advanced settings, usually not to be changed:

The following X.509 certificates and keys are required for using the system with TLS/HTTPS:

The host certificate contains its hostname (FQDN) as CN (common name) as the certificate subject or, alternatively, in the subject alternative name (SAN).

On the server side, https connections are terminated by an nginx reverse proxy. Moreover, the services transfer engine (tixstream), transfer job manager (TJM) and TIXEL control center (TCC) also require host key and truststore for mutual authentication. For https access to TCC and TJM by third-party systems, certificates are used for regular (i.e. non-mutual) server based authentication. For sending notifications via https, TJM requires truststore access as well.

Certificates are required in PEM (base64 encoded) as well as in PKCS12 (PFX/P12) format. PEM is used for nginx and tixstream while the Java services TJM and TCC require host certificates and keys as a PKCS12 Java keystore. A Java keystore can be created in different formats, such as JKS (Java Keystore) or PKCS12 (".p12").

To summarize, certificates are used in different formats by the following services:

These services require read permissions of the respective files. On Linux you may want to create a group tixcerts and assign users` tixstream` and nginx to this group. The tixcerts group is then allowed read access to these files.

The truststore file trusted.pem contains concatenated trusted certificates. Typically, these are (only) the root CA certificates, analogous e.g. to the cacerts file of curl or a Java truststore. In addition, individual host certificates to be trusted can also be stored here. If the host certificate was not derived from a stored CA, the truststore must also include the own host certificate in order to allow access to the metadata editor which is connected via an internal service (metadata clipboard).

When creating certificates, keep in mind that the certificate is used as a server certificate as well as a client certificate. This is the case for a certificate generated by openssl using standard options. When using the corresponding extensions, it must be taken into account during certificate creation (extendedKeyUsage = serverAuth, clientAuth).

When you need your certificate to be signed by a root CA, create a (secret) key and a Certificate Signing Request (CSR, * .csr file) which you pass to your signing authority. The CA will return the signed certificate. The commonly used file name extension is .cer which corresponds to the PEM format required here.

Note that host-crt.pem must contain the derivation chain up to its root CA. The host’s certificate must be the first certificate inside this file. E.g. creating the required certificate chain in host-crt.pem from host certificate host-only-crt.pem, intermediate CA` inter-crt.pem` and root-CA ca-crt.pem, can be done as follows (Windows Shell):

For Linux:

In order to create a PKCS12 from this X.509 certificate and the associated key (host-crt.pem, host-key.pem), you can use openssl (please replace "myhost" with your actual hostname):

The created file host.p12 can be used as a Java keystore, which can be verified with keytool -v -list -storepass changeit -keystore host.p12. The keytool is included in the JRE or JDK (in Windows, for example,c:\Program Files\Java\jre1.8.0_111\bin\keytool.exe).

Certificates present as .pem files (root CA certificates or host certificates) can be included in a keystore by using the keytool, e.g. (please replace myrootca accordingly):

This allows you to add additional trusted certificates. If the import fails, make sure that the PEM is not UTF-16 encoded (e.g. by editing it with a Windows editor).

Place the certificate files in the config directory of the MFT installation under the following filenames:

The web server nginx is used as a reverse proxy for https. The MFT software package contains a configuration file nginx_https.conf that has all relevant settings prepared and can be activated manually.

To enable TLS for the TIXstream transfer engine, set parameter internal-client-tlsverify to 1 in the configuration file config/tixstream.conf. With value 0 the remaining internal-client-tls*-parameters are ignored. Certificate names and paths are already pre-configured.

Java services require the certificates in a Java keystore. Even a truststore containing certificates only is always protected by a password (default: changeit). Details about keystore and truststore are to be specified as options when starting the service (i.e. java command-line). During installation, the services are created with following configuration:

If the keystore files names and/or passwords are modified, they must be adjusted in the configuration files of the services (Windows: /config/*.service or – prior installation – in install-tix-services.cmd, Linux: systemd modules and install.sh).

The Transfer Job Manager notifies the sender system about the port of the receiver system to be used by tixstream.

For encrypted connections select port 60003 (port for http: 60002). In TCC, this setting can be modified in Transfer Job Manager – Advanced Settings – TIXstream internal port.

Alternatively, the parameter can be changed in config/transfer-job-manager.properties:

Do not change setting custom.tixstream.external.ssl=false.

Transfer system configuration is described in TIXstream MFT Peer Nodes. To use https instead of http, modify the address as follows:

Optionally, you may also activate data channel encryption per transfer system. This is independent of control channel encryption, but makes no sense without encrypted data channel since, in this case, the key for user data would be transmitted via an unsecured channel. Data channel encryption is activated by the sender.

As a summary here is a checklist for migrating a system for use with TLS:

To check whether transfer job manager can be reached using mutual authentication via https port 60001 go to tixel\config directory and issue the following command (replacing FQDN with your MFT hostname):

Windows:

Linux:

This command should display the configured system alias.

If this fails, using curl with an additional -v may provided further details. If you get "SSL certificate verify ok" but 400 ("SSL certificate error"), it is probably due to nginx configuration. Please check the server section for port 60001 and 60003, especially if the certificate file names have been changed:

Use of file-based H2 databases is pre-configured so that an MFT system can be put into operation without external dependencies. The database files are located in the config directory as service name (with underscore) and suffix .mv.db, e.g. access_manager.mv.db..

A sample configuration of a MariaDB or MySQL database server for use with MFT transfer job manager is described below.

A sample configuration of an MSSQL server for MFT Transfer Job Manager is described below.

Based on a standard MSSQL installation the following configurations have to be made:

Usually the following steps are required:

The following description is based on the example below. The LDAP groups (groupOfUniqueNames), used for mapping user roles are particularly relevant for the transfer system; in the example tixel-user and tixel-admin.

In setup mode of the TIXEL Control Center, create the following entries in Access Manager – LDAP Settings. Values have to be adapted to the actual LDAP configuration.

For the LDAP configuration from the example above:

To use LDAPS (LDAP over TLS/SSL) LDAP URL must be adapted accordingly (e.g. ldaps://dstst.example.com:636). If not done already, import the certificate or the root CA chain in the truststore used by the Access Manager (default: ${TIXEL_HOME}/config/trusted.p12), see also TLS/HTTPS. The use of STARTTLS via port 389 is not supported.

If the LDAP server does not allow anonymous access, parameters LDAP Manager DN and LDAP Manager Secret must be specified for LDAP access. In the access-manager.properties file, these are the parameters custom.security.ldap.manager-dn and custom.security.ldap.manager-password.

The value from LDAP root DN is always added to the search path. Thus, do not repeat this value again (in the example, search path for users is ou=Users, dc=tixel, dc=it).

Upon log in, users enter their uid as user name. Thus User Search Filter is (uid={0}). If users are supposed to log on with their e-mail address instead, User Search Filter would be (mail={0}).

In the example, groups were created with objectClass groupOfUniqueNames. These contain members with complete DN (FDN, Fully Distinguished Name) in uniqueMember (e.g. uniqueMember: uid=ernie, ou=Users, dc=tixel, dc=it). Using groupOfNames behaves similar, here group members are stored as member.

If you are using posixGroups (which contain the uid as memberUid, e.g. memberUid: uid=ernie), you must change the group search filter from (uniqueMember={0}) to (memberUid={1}), since {1} contains the login name (uid), while {0} contains the FDN of the user.

Finally, the Group Role Attribute specifies which attribute contains the group identifier (here: cn). These identifiers are used in role assignment described in the next section.

In TIXEL Control Center, User administration – Roles, Directory server group, the appropriate LDAP groups can be assigned to each role (currently 6 different roles, for example, "Access Manager Admin"). Different roles can be assigned to the same LDAP groups. In this example, only two groups tixel-admin and tixel-user are used:

The following roles are assigned to default users:

The following roles from a previous version were discarded:

For each service, a subfolder with the service name is created in directory logs, e.g. tixelcontrolcenter. Each subfolder contains:

The basic status of the service can be obtained from the start log. In the service logs, service specific information can be found.

Settings for logging can to be made separately for each service in TIXEL Control Center setup mode.

The used web proxy NGINX generates various log files that can be rotated or deleted separately. For this purpose, the script nginxlogrotate.cmd (in admscripts) is available. It can be started periodically by using Windows task scheduling.

Individual adjustments can be made inside this script by using a text editor:

The other parameters do not need to be changed in an unmodified installation.

Example configuration (excerpt from nginxlogrotate.cmd):

In general, proxy logging can be minimized or disabled during normal operation, and only be activated temporarily for troubleshooting. With the value off as log file names, access logs can be disabled in nginx.conf. However, error log cannot be disabled.

If you have installed the TIXstream MFT services using specific service user and group settings (instead of the defaults tixstream:tixstream), you are asked if you would like to keep those settings (recommended).

If you use specific umask settings in the corresponding service files, you need to provide umask setting explicitly (again) when invoking the install script, e.g.: