Using Extract Data (FTP) Template

To convert unstructured data into structured data from an FTP Source, you need to create a Transaction using the pre-defined Template.

<ch:anchor>It is recommended that the provided pre-built Templates are not edited. Based on your requirement if you need to make changes to the Template, you may do so by creating a copy of the provided pre-built Templates.  </ch:anchor> While reusing the pre-built Templates, note that these Templates support only File Based Triggers.

It is recommended that all the activities belonging to the AI Project are not edited.

Prerequisites

Before using the Template ensure that:

• You have successfully created a security policy for WS Consumers with OAuth authentication and associated it with the following REST Consumers:

  • Unstructured_Extractfields_API

  • CallUnstructuredAPI
     

    If you are an on-premise customer, please contact your Adeptia application admin or support@adeptia.com for assistance with this prerequisite.

    Refer to this page to learn more about creating a security policy with OAuth authentication.

• You have specified the URL of the Web Service to be invoked in the REST Consumer activity, UnstructuredFieldSelection. To do so, enter the Endpoint URL of the REST Provider activity, Unstructured_FileSlection_WSP, in the URL field of the REST Consumer.

Using Extract Data (FTP Upload) Template

Follow the steps given below to create a Transaction:

1. Click Transactions > T****emplates.

2. On the Templates screen, navigate to the Extract Data (FTP Upload) Template and click .

3. Provide the name and description of the Transaction. The name and description of the Transaction are generated automatically based on the selected Template. You can edit the Transaction name and description that can be up to 48 and 300 characters long.

   

4. Click Continue.

5. In the Select Partner field, select a partner for whom you are creating the transaction.

6. Click Next.

7. On the Parameter Values screen, enter the following details:
   
    

   1. Click  to upload an unstructured file to be processed.
      On uploading the file, the Unstructured Data Fields displays.

   2. If you want to extract selected fields from the unstructured document, do the following:  

      1. In the Unstructured Data Fields, select the fields from the dropdown that need to be extracted into the target file.  
         

   3. In the Structured Data Format Type field, select from the dropdown the format of the structured data. 
      

   4. If you have selected JSON as the Structured Data Format Type, do the following to specify the output format in JSON that you would like to receive.
      
       

      1. In the  Structured JSON Format field, enter the names that you want to provide to the fields in the Target files in JSON structure.
         For example, you have three files containing employee data, namely  File A, File B, and File C. And each file contains distinct field names: Emp ID in File A, employee Id in File B, and EMP ID in File C. If you want the Target files to have a standard name for these fields as "Employee ID". Then you may define "Employee ID" as the field name in a valid JSON format in the Structured JSON Format field. Consequently, the generated Output files will have the field name Employee ID for these fields. 

         The JSON Format for defining this field is illustrated in the example screenshot. 
         

   5. Click Next.

8. On the Transaction Settings screen, do the following:

   1. Enter the details as described in the following table.
      

      Field Name		Description
      Contact User		Email id(s) listed here will receive a notification email whenever the Transaction is executed successfully or if there is an error during execution. Multiple email ids are separated by comma. If Adeptia Connect is running on multiple environments and you want to receive the environment name in the emails, then you need to specify the environment name in the relevant property (Account > Settings > Expand Application Settings > Common > Expand Personalization).
      When to notify the contact user thru email?		Always: Send an email whenever a Transaction executes. On Error: Send an email if an error occurred during the execution of the Transaction. Do Not Notify: Select this option if you do not want to receive an email notification for Transaction execution.
      Email Notification Format		This field is visible only when you have selected the Always or On Error option to send the notification to the user. Select the email notification format that you want to use for notification emails sent to the user for successful and failed Transaction. Default: If you select Default, the system will use the default email notification format. Customized Notification Format: If you select this option, a Custom Error Notification Format field appears. In this field, you can select an existing custom email notification format or create a new one, and use it for email notifications.

   2. Expand Advanced Settings to set the Logging Level, Repository File Retention, and Activity Logging Retention. The available options for the fields are as follows:
       

      • Logging Level- INFO, DEBUG, and ERROR

      • Repository File Retention- DONT DELETE, DELETE, DELETE ON SUCCESS, and DELETE ON SUCCESS EXCEPT SOURCE DATA

      • Activity Logging Retention- DONT DELETE, DELETE ON SUCCESS, and NO LOGGING. To learn more about these options, refer to this page.
        

   3. Click Next to define the Unstructured data file Source.

9. Select the FTP account information for the source file. You can either use an existing account or create a new one. Click to add a new account. For information on how to add a new account, refer to Supported applications. 

   

   Click Test to test the connection to your account.

10. Click Next.

11. On the Choose Trigger screen, select the trigger from the drop-down list based on your requirement. 

    

12.  On the Select File Location screen, do the following:

    1. Provide the configuration details of your account.
       

       Field Name	Description
       Folder Location	Folder location from where to pick files.
       Process Empty File	This option will process empty (0 bytes) files.
       Process all the existing records/files when this transaction is turned on	This option will process all the existing records or files on first activation of this connection.
       Don't process duplicate files	This option will not process any new file that has the same name as the file that has already been processed.
       Send mail notification for no data on trigger	Selecting this option will send the mail notification whenever there is no data found on trigger.
       Send mail notification for error on trigger	Selecting this option will send the mail notification whenever there is an error on trigger.
       Delete file on success	Selecting this option will automatically delete the source file after the transaction runs successfully.
       File type	File format to process the files. Multiple file formats are separated by a comma. Use *.* to process all file formats.
       File exclude	Type the file format to exclude the file type while the transaction is running. Multiple file formats are separated by a comma. For example, if you want to exclude text files from the FTP location, type .txt in the text box.
       Trigger Type	You can set trigger through POLLING and CRON EXPRESSION. POLLING: The time interval during which the Transaction triggers. By default, Transaction triggers after every 10 minutes. Select No polling option to trigger Transaction manually. CRON EXPRESSION: Define the specific time schedule for the Transaction to trigger. For more information on how to define Cron Expression, click here.   It is mandatory to define the first cron expression if you have selected cron as a Trigger type. For example, if you have set an FTP Event to run every 5 minutes from Monday to Sunday, you can set the cron expression as "0 0/5 * ? * MON, TUE, WED, THU, FRI, SAT * ". You may make use of the second cron expression to define another condition or address a condition supplementary to the first one. For example, if you have set an FTP Event to run every 5 minutes from Monday to Sunday, but do not want the same event to run from 12 AM to 12 PM on Sundays, the second cron expression can be defined as "0 0/5 13-23? * SUN * ".  The firing schedule you define in the second cron expression must not overlap the one defined in the first cron expression.
       File Stable Time	You can set file stable time in the text box and select the time unit from the drop-down list. This option is applicable for On File Created and On File Modified Trigger Type. The trigger will wait for the specified time for the file to become stable. For example, if you choose 30 Seconds in the stable time then the system will wait for 30 seconds for the file to go stable before it triggers the transaction.

    2. Expand Other Settings and provide the details.
       

       Select Trigger in Sequence check box to trigger the Transaction in a sequential manner. If you do not select the Trigger in Sequence check box then the associated Transaction will trigger concurrently.

       If you want to enable misfire handling for your file event then, select the Run Once option from the Action on Misfire Event drop-down list. If you do not wish to enable misfire handling for your file event then, select the Ignore Misfire option.  For more details, refer to Handling Misfire Event.

       The Run Once option will run only one collective instance of all the misfires for a particular event. Whereas the Ignore Misfire option will not run a misfired event.  The default value for Action on Misfire Event field is set to Run Once for all of the earlier events.

       The Connector drop-down lists the APIs to connect to the FTP Server. The options change as per your selection of the protocols that you want to use to establish an FTP connection. 

       Protocol	Options
       FTP	Native  Secured Inet Factory
       SFTP	J2SCH (VFS)  Secured Inet Factory  J2SSH
       FTPS	J2SCH (VFS)  Secured Inet Factory

       Select the Verbose check box, if you want to create a log file of your FTP activity. This check box is enabled only if you select Secure Inet Factory from the Connector drop-down list box.

    3. Click Next to define the Structured file Target account. 

13. Select the FTP account information for the Target file. You can either use an existing account or create a new one. Click to add a new account. For information on how to add a new account, refer to Supported applications. 

14. Click Next.

15. On the Select File Location screen, do the following:
    

    1. Provide the folder location from where to pick files. 

    2.  Select Auto Generated if you want the Target file name to be the same as the Source file name or source application entity name and Company name. Else, select User-Defined to provide your own file name. 

    3. Specify the file extension for the Target file in the Target file extension field. 

       This field appears only when the Target file name is Auto Generated.

    4. Select the Overwrite this file checkbox if you want to overwrite a file with the same name. 

16. Click Save & Exit to save the Transaction.
    This creates a Transaction.