User Tools

Site Tools


CookBook

Introduction

Spatialytics ETL – Cookbook is a powerful tool that automates the generation of ETL chains documentation. The produced documentation is composed of HTML files. So, it is easy to access and share on the network. Spatialytics ETL – Cookbook comprises different transformations and jobs which aims at retrieving the relevant meta-information in jobs and transformations stored in a repository or in a simple directory, at structuring it and at producing an interactive documentation based on it. It uses the meta-information stored in each step (resp. job entry) of each transformation (resp. job). In order for Spatialytics ETL – CookBook to work adequately, the user must then complete the following tasks:

  • describe what each transformation step does by adding description (right click step / Edit step description). Do the same for each job entry (right click on a jobEntry / Edit JobEntry description);
  • add a general description for each transformation or job (right click / transformation settings or job settings);
  • add notes describing what does the transformation or the job do (right click / new note);
  • describe parameters and variables used in the transformations (right click / transformation settings / parameters tab / description column) and the jobs (right click / jobs settings / parameters tab / description column);
  • describe connections to databases (to define a new connection: Click on the View tab / right click on the Database connection node / new connection or new connection wizard. For an existing connection: Click on the View tab of an existing connection / expand the Database connection node / right click on the connection / Edit).

Spatialytics ETL – Cookbook transformations and jobs allowing the generation of the documentation can be found in the pdi subdirectory of the cookbook directory in the Spatialytics ETL installation path.

ETL chains stored in a directory

To generate the documentation for ETL chains stored in a directory (as .ktr or .kjb files), do the followings:

  • launch the document-folder.kjb job. This job takes three (3) parameters that must be provided at runtime. These parameters also have default values that should be modified to meet the requirements of the user (right click / job settings / Parameters tab / default value column):
    • INPUT_DIR: Path to the transformations and jobs directory for which the user wants to generate the documentation;
    • OUTPUT_DIR : Directory in which the generated documentation files will be stored;
    • DOC_LANG : Defines the language in which documentation will be generated. Default language is EN, for english. To generate it in French, set DOC_LANG parameter value to FR.
Note on parameter

The SAXON parameter of this job should not be modified. Therefore, keep the default value.

ETL chains stored in a repository

When jobs and transformations are stored in a repository, launch the generate_Documentation_ from_repository job. This job performs the following tasks:

  1. It extracts transformations and jobs from the repository to a directory specified by the user (INPUT_DIR), that defaults to the cookbook subdirectory of the Spatialytics ETL installation path;
  2. It launches the document-folder.kjb job in order to generate the documentation and store it in the destination directory (OUTPUT_DIR).

To performs theses tasks, the main job uses the following parameters:

  • REP_USERNAME : Name of the user accessing the repository;
  • REP_PASSWORD: Password of the user accessing the repository;
  • INPUT_DIR: Directory in which extracted transformations and jobs from the repository will be stored. Before generating the documentation, Spatialytics ETL - Cookbook extracts all transformations and jobs from the repository and store them locally in the directory specified by the user. This directory is then used to generate the documentation;
  • OUTPUT_DIR : Directory in which the generated documentation files will be stored;
  • DOC_LANG : Defines the language in which the documentation will be generated. Default language is EN, for English. To generate it in French, set DOC_LANG parameter value to FR.

Once the documentation has been generated, the user can delete the folder pointed by the INPUT_DIR parameter, since it contains the same transformations and jobs as the repository.

Access to the documentation

The user can access the generated documentation in the directory defined by the OUTPUT_DIR parameter. This directory contains some HTML files. To access the documentation home page, the user has just to open the index.html file in his browser. This page comprises two parts (see Figure 1):

Figure 1: Home page of a documentation generated by Spatialytics ETL - Cookbook – Categories tab selected.

  1. The left part: Lists all the documented objects (see Figure 1). These objects are grouped by categories (connections, transformations, jobs) when the Categories tab is selected and as a list when the Hierarchy tab is selected. On this page, the user can also find a useful links to navigate in the documentation: Home to go back to the home of the documentation and About that provides a short description of the Spatialytics ETL – Cookbook tool.
  2. The right part: Gives details about elements (connections, transformations, jobs) selected in the left part (see Figure 2). This description is structured acording to the following sections (the user can also retrieve these section titles as clickable links at the top of the page in order to easily navigate in the documentation):
    1. Upper section: Provides the element name and other relevant information such as its creation / modification date, version, status and the username of the user that created the element;
    2. Parameters: Describes the parameters used by the job or the transformation;
    3. High level diagram: Describes briefly the element;
    4. Diagram: Gives an overview of the element (transformation or job):
    5. Database connection: Describes the database connections used by the transformation or the job;
    6. Steps (resp. job entry): Describes in details each step (resp. job entry) of the transformation (resp. Job).

Figure 2: Details of an element selected in the left pane of the documentation generated by Spatialytics ETL - Cookbook.

Additional remarks

Spatialytics ETL – Cookbook is a powerfull tool. It automates the generation of ETL jobs and transformations documentation. It uses meta information stored in each ETL object. In order to produce a better documentation, the user has to provide description and information for each transformation, job, step, job entry and database connection.

en/spatialytics_etl/006_modules/cookbook.txt · Last modified: 2013/02/04 11:40 by sbedard

Page Tools