User Tools

Site Tools


5. Jobs

5.1 Definition

While Transformations are about moving and transforming rows from source to target, jobs are more about high level flow control: executing transformations, sending mails on failure, etc.

5.2 Documenting

The documentation of jobs and job entries follows the same logic as documenting transformations and transformation steps. Please refer to the Documenting section of the Transformation chapter.

5.3 Job entries

As transformation steps are the building blocks of a transformation, so are job entries for a job. However, there are differences between job entries and transformation steps.

Characteristics of job entries

  • A job entry needs to have a unique name with the exception of shadow copies.
  • You can create shadow copies of a job and use them on multiple locations. Editing one copy of a shadow copy job will edit all copies. All copies of the same entry have the same name;
  • Most of the time, jobs are executed in sequence meaning that each entry of a job waits that the previous entry is finished before starting. Unlike transformations, parallelism is most of the time not a characteristic of jobs;
  • Job entries can include other jobs or transformations.

Job entry descriptions

Here is a list of job entries categories:

Here is a list of job steps available in Spatialytics ETL.

General

Step name Description
StartStart is where the job starts to execute and is required before the job can be executed
DummyUse the Dummy job entry to do nothing in a job. This can be useful to make job drawings clearer or for looping
Abort JobUse this job entry if you want to abort a job
Display MsgBox infoThis job entry allow you to display a message box in job
Dummy plugin
JobYou can use the Job job entry to execute a previously defined job
Ping a hostYou can use the Ping a host job entry to ping a host using the ICMP protocol
Send SNMP TrapSend SNMP trap to a target host
Set variablesSet one or several variables
SuccessThis step clears any error state encountered in a job and forces it to a success state
TransformationYou can use the Transformation job entry to execute a previously defined transformation
Truncate tablesThis job entry allows you to truncate one or several tables at once without writing 0 SQL command
Wait for SQLScan a database and success when a specified condition on returned rows is true
Write to logThis step writes a specific string to the PDI logging system

Mail

Step name Description
Get mails from popThis step allows you to retrieve mails from a POP server and store them into one or more files
MailYou can use the Mail job entry to send a text or HTML e-mail with optional file attachments
Mail validatorThis step checks if an email address is valid or not

File Management

Step name Description
Add filename to resultThis job entry allows you to add a set of files or folders to the result list of the job entry
Compare foldersThis job entry compares 2 folders and sees if the content of the 2 is identical or not
Copy filesYou can copy one of more files or folders with this job entry
Copy or move result filenames
Create a folderThis job entry simply creates a folder at the specified location
Create fileYou can use the Create a file job entry to create an empty file
Delete fileYou can use the Delete a file job entry to delete a file
Delete filenames from resultUse this job entry to delete all the filenames that are in the result files list of the last job entry
Delete filesYou can use the Delete files job entry to delete muultiple files
Delete foldersThis job entry deletes one or more folders and all the content in it
File compareYou can use the File compare job entry to compare the contents of 2 files and control the flow of the job by it
HTTPYou can use the HTTP job entry to get a file from a web server using the HTTP protocol
Move filesThis job entry allows you to move a number of files and/or folders to alternate locations on your file system
Unzip fileYou can unzip one or more files with this job entry
Wait for fileYou can use the Wait for file job entry to wait for a file
Zip fileThis step creates a standard ZIP archive using the options you specify in the dialog

Condition

Step name Description
Check if a folder is emptyThis job entry verifies if a folder is empty
Checks if files exist
Columns exist in tableThis job entry verifies that one or more columns exist in a database table
Evaluate rows number in tableWith this job entry you can define the success condition that can be evaluated and branch the job process by the result
File existsYou can use the File Exists job entry to verify if a certain file exists on the server on which Spatialytics ETL runs
Simple evaluationWith this step, you can evaluate contents of a variable or a field in the result stream
Table existsYou can use the Table Exists job entry to verify if a certain table exists on a database
Wait forYou can use the Wait for to wait a delay before running the next job entry

Scripting

Step name Description
JavascriptEvaluates certain conditions in the “JavaScript” job entry
SQLYou can use the SQL job entry to execute a SQL script
ShellYou can use the Shell job entry to execute a shell script on the host where the job is running

Bulk loading

Step name Description
Bulkload from MYSQL to fileLoad from a mysql table into a file
Bulkload into MSSQLLoad data from a file into a MSSQL table
Bulkload into MYSQLThis step loads data from a text file into a MySQL table
MS Access Bulk loadLoad data into a Microsoft Access table from CSV file format

XML

Step name Description
Check if xml file is well formedThis job entry verifies if one or more files contain well formed (allowed) XML content
DTD validatorShort for Document Type Definition, DTD allows you to verify if an XML file corresponds to a certain structure or format
XSD validatorThis job entry performs an XSD validation against data in a file or in an input field
XSL transformationXSL Transformation job entry is designed to transform (by applying XSL document ) XML documents into other documents (XML or other format, such as HTML or plain text)

File Transfer

Step name Description
FTP deleteDelete files in a remote host
Get a file with FTPYou can use the FTP job entry to get one or more files from an FTP server
Get a file with SFTPYou can use the SFTP job entry to get one or more files from an FTP server using the Secure FTP protocol
Put a file with FTP
Put a file with SFTPYou can use the Put files with SFTP job entry to put one or more files from an FTP server using the Secure FTP protocol
SSH2 GetGet files using SSH2
SSH2 PutPut files in a remote host using SSH2

Repository

Step name Description
Check if connected to repositoryReturn true if we are connected to a repository
Export repository to xml fileThis job entry exports repository objects from a given repository into one or multiple files in the XML format

5.4 Job Hops

Besides the execution order, a hop also specifies the condition on which the next job entry will be executed. You can specify the evaluation mode by right clicking on the job hop:

  • “Unconditional” specifies that the next job entry will be executed regardless of the result of the originating job entry.

  • “Follow when result is true” (green arrow) specifies that the next job entry will only be executed when the result of the originating job entry was true, meaning successful execution, file found, table found, without error, evaluation was false, …
  • “Follow when result is false” (red arrow) specifies that the next job entry will only be executed when the result of the originating job entry was false, meaning unsuccessful execution, file not found, table not found, error(s) occurred, evaluation was false, …

en/spatialytics_etl/005_jobs.txt · Last modified: 2013/02/18 10:14 by olussier

Page Tools