Table of Contents
List of Tables
List of Examples
Did you ever want to transfer data from one JDBC source to another and do some kind of transformation during copy, like: normalize unnormalized data / create artificial keys / execute SQL statements during / before / after transfer ?
Did you ever want to create a database like the following in one step: create the tables, load initial data, create indexes, create primary keys, create foreign keys ?
If your answer is yes, you need this Octopus !
WebDocWf Octopus loads data from a JDBC source database into JDBC target databases. Octopus could process different types of databases. Possible source/target database types are: MSSQL, MySql, Access, Excel, Csv, PostgreSQL, Qed, InstantDB, BorlandJDataStore, Oracle, HSQL and DB2 database.
There are several loading possibilities: creating databases, creating tables, inserting data into an empty database, inserting data in to a non empty database, updating constant columns, updating relation columns, updating columns with system time, updating columns with a user ID, executing every possible SQL statement, creating artificial keys using the Enhydra DODS objected logic, ...
Rules of transformations are written in a special XML file (input XML loadJob file). The number of transformations in one XML loadJob file is not limited.
Octopus could execute SQL statements to load data (possibility to load data from different source tables. Octopus could create new database (target database), tables, relations and indexes. Octopus also could execute all sql statements in target database.
Restart mode: the possibility to continue the process where it was stopped (if it was stopped in the middle of the process) without losing or multiplying data.
There is possibility to track the user who starts the loading process.
Every time the application is started, a log file is written. Default format of log file name is: LoaderLog-YYYY-MM-DD-HH-mm-SS.txt (Example: loaderLog-2002-04-25-11-05-30.txt).
Octopus creates log files in the current working directory (as default). Also if you set Octopus parameter -l (one of Octopus' starting parameter) , you can choose other directory for log files.
There are three possible log modes:
"none" - writes only that the application is started, the names of the import definitions, time when the loading process was started/ended and time when the application has been finished;
"normal" - writes main events (number of import job, the names of the importDefinitions, time when the application was started/ended and the number of committed rows) and
"full" - writes detailed information about the loading process.
Data columns are written to the target table using several modes:
Key mode - defines key column of the target table, the value is only written if a new row is inserted;
Overwrite mode - overwrite the value of the target column in the appropriate row or insert the value with a new row;
Update mode - update the value in an empty column of the appropriate row (no overwrite of existing value);
SetIfCreated mode - the value is only written if a new row is inserted;
SetNull mode - value is only written if the new value is NULL.
There are two possible ways to start the Octopus: as a standalone application or as a Java API.
Table of Contents
The following sections list the new or updated features associated with Octopus 2.1 beta3
Support for global unique objectID's as used by TOS (Together object server)
New installation for windows based on NSIS 2.0b
New installation for linux based on rpm packets
subCounterColumns are implemented with SetIfCreated mode
Fixed problem with different source types in same load job
Fixed JTurbo bugs with SQL server and selectStatements
Added SQL server 7 supports for JTurbo driver. Changes in OctopusDbVendors.xml
Updated csv jdbc driver ver. 0.9 with Octopus changes (writing option, add 'where' in SQL statements).
Implemented new echo element with message attribute.
Added driverName attribute in OctopusDbVendors and jdbcParameters element because dbVendor is not unique (MSSQL).
Implemented Returncode parameter, and attribute in case of Octopus errors
Produced regression tests with JUnit integrated into Maven
Qed database vers. 2.0 included in Octopus as a test database
Implemented counterColumns for auto increment columns in target tables.
Added time measuring of each importDefinition
Improved csv jdbc driver with basic insert, update and create table sql commands. With Octopus 2.0 you can use csv tables as a target tables using csv jdbc driver.
Separated jar files (Octopus-2.0.jar, OctopusTest.jar and OctopusTask.jar
Added selectStatement attribute in importDefinition element for loading source data using external sql statements (you can use data from more source tables)
Added Driver Class Name as a parameter in OctopusDbVendors configuration file
Support for BorlandJDataStore, HSQL, Oracle and DB2 databases
Added classes which support using Octopus as a JUnit extension.
Added Octopus ant task for starting Octopus from ant
Updated documentation in DocBook format.
New example classes for using Octopus as a test case
Also, Octopus 1.7 works with Microsoft JDBC driver for SQL server 2000
Octopus 1.7 uses xerces 1.4.4.
Octopus 1.7 uses xalan 2.3.1. and fop 0.19 for creating PDF files
Octopus 1.7 is extension of JUnit 3.8
Building Octopus using MAVEN
Table of Contents
Parameter is URL to XML file, example:
java org.webdocwf.util.loader.Loader MyLoadJob.xml
where org.webdocwf.util.loader is a package name where Loader class belongs, and MyLoadJob.xml is input XML file.
Using native Java VM commands you can start the Octopus with the following syntax:
java org.webdocwf.util.loader.Loader [options] loadJob_xml_filename
"loadJob_xml_filename" is the name of the XML file which contains all processing instructions.
Table 3.1. Options.
option | meaning | sample |
---|---|---|
-m | defines the default logmode. Possible values are "none", "normal" (is the default) and "full". | -m none |
-r | starts the Octopus in restart mode after abnormal termination in a prior execution. | -r |
-u | defines the current UserID used in UserID value columns. | -u r004d\meier |
-v | defines variables used in variable columns. | -v varname1=varval1;varname2=varval2 |
-l | defines the logfile directory. The default is the current working directory. | -l c:\my_directory_for_logfiles |
-f | defines the logfile name. The default is "LoaderLog-YYYY-MM-DD-HH-mm-SS.txt". | -f c:\somewhere\down\..\mydir2\mylog.txt |
-d | the filename of the XML DB-vendor configuration file. The default is "OctopusDBVendors.xml". | -d MyVendor.xml |
-e | defines to set the default of "onErrorContinue" to "true". Is false otherwise. | -e |
-p | Extend the classpath with additional paths. | -p c:\webdocwf\webdocwf310\lib\csvjdbc.jar;c:\webdocwf\webdocwf310\lib\JTurbo.jar |
-c | Sets the default commit count. System default is "100". | -c 1000 |
-rc | Sets the default return code to the environment if some element of loadjobs fails | -rc 10 |
The order of the options is not relevant.
A complete start command example:
java -cp c:\webdocwf\webdocwf310\lib\wdw.jar;c:\webdocwf\webdocwf310\lib\webdocwf.jar org.webdocwf.util.loader.Loader -m none -r -u myID -v myvar1=value1;myvar2=value2;myvar3=value3 -l c:\mylogs -f loadlogs\mylog.txt -d MyDB.xml -e -p c:\webdocwf\webdocwf310\lib\csvjdbc.jar -c 1000 MyLoadJob.xml -rc 1
Another one:
load -u myID MyLoadJob.xml
WebDocWf contains startup scripts under the "bin" directory to start the Octopus. Under Windows 2000 simply use the "load" command.
Be sure to include necessary entries in the classpath for extra JDBC drivers which are used in your loader jobs or simply use the -p Parameter !
The Octopus can be used from every Java application by its public constructor.
public Loader(String loadJobFileName, String mode, String userID, String logDirName, String logFileName, boolean restartIndicator, Map variableValues, String vendorFileName, boolean onErrorContinue, String additionalPaths, int commitCount, int returnCode)
To start the load process simply does the following:
Loader myOctopus=new Loader("MyLoadJob.xml", Loader.LOGMODE_NONE, "myID", null, null, false, null, null, true, null, 0, 1); myOctopus.load();
The logmodes are defined as public String constants in the Loader class. Their names are: Loader.LOGMODE_NONE, Loader.LOGMODE_NORMAL, Loader.LOGMODE_FULL.
There is a second public constructor using defaults for all options:
public Loader(String loadJobFileName)
You can also use the "getxxx" and "setxxx" methods for the additional options as defined in the API documentation.
Octopus Task class (LoaderTask) extends jakarta-ant Task class and uses to start Octopus application as a jakarta-ant task in build.xml file.
Attributes and subelements of LoaderTask represent Octopus parameters:
Table 3.2. Table LoaderTask arguments. Arguments are:
argument | meaning | sample |
---|---|---|
loadJob | defines input loaderJob xml file with rules for loading data from source to target database | loadJob="c:\test\loadJobExample.xml" |
mode | defines the default logmode. Possible values are "none", "normal" (is the default) and "full". Is "normal" otherwise | mode=" none" |
restartIndicator | starts the Octopus in restart mode after abnormal termination in a prior execution. | restartIndicator="true" |
userID | defines the current UserID used in UserID value columns. | userID=" r004d\meier" |
logDir | defines the logfile directory. The default is the current working directory. | logDir="c:\my_directory_for_logfiles" |
logFile | defines the logfile name. The default is "LoaderLog-YYYY-MM-DD-HH-mm-SS.txt". | logFile="c:\somewhere\down\..\mydir2\mylog.txt" |
vendorFile | the filename of the XML DB-vendor configuration file. The default is "OctopusDBVendors.xml". | vendorFile="MyVendor.xml" |
onErrorContinue | defines to set the default of "onErrorContinue" to "true". Is false otherwise. | onErrorContinue="true" |
commitCount | Sets the default commit count. System default is "100". | commitCount="1000" |
Table 3.3. Table LoaderTask subelements. Subelements are:
subelement | consists of subelements | argument | meaning |
---|---|---|---|
variables | variable | name, value | "name" is name of variable that is used in Octopus variable columns (loaderJob xml file). "value" is value of variable that is used in Octopus variable columns (loaderJob xml file). |
additionalPaths | additionalPath | path | File to add in classpath |
Example 3.1. Example of xml file starting OctopusTask:
<taskdef name="LoaderTask" classname="org.webdocwf.util.loader.task.LoaderTask"/>
<LoaderTask mode="full" restartIndicator="yes" userID="admin" logDir="c:\my_directory_for_logfiles" logFile="c:\somewhere\down\..\mydir2\mylog.txt" vendorFile="MyVendor.xml" onErrorContinue="yes" commitCount="1000" loadJob="GRECOOMDSBROWSER/ObjectLoader/LoadOMDSProject.xml"> <additionalPaths> <additionalPath path="../lib/xerces.jar"/> <additionalPath path="../lib/loader-1.6.jar"/> <additionalPath path="../lib/freetds_jdbc.jar"/> </additionalPaths> <variables> <variable name="name" value="varval"/> </variables> </LoaderTask>
The following walks you through the core classes that make up Octopus as a JUnit extension.
DatabaseOperation (org.webdocwf.util.loader.test)
It is an abstract class and you will find yourself using implementations or subclasses.
The DatabaseOperation abstract class defines operations performed on a target database before and after each test. Again, the framework provides several implementations.
DO_NOTHING
CreateDatabaseOperation
DropDatabaseOperation
LoaderOperation
The DatabaseOperation class declaration:
public abstract class DatabaseOperation { public static final String NONE = "NONE"; public static final String CREATE = "CREATE"; public static final String DROP = "DROP"; public static final String LOADER = "LOADER"; public static DatabaseOperation DO_NOTHING = new DummyAction(); /** * Executes this operation on the specified database using the specified * connection to the database. * * @param conn the database connection. */ public abstract void execute(Connection conn) throws SQLException; /** * Returns type of database operation * */ public abstract String getDatabaseOperationType(); }
This operation creates new database using SQL commands "CREATE DATABASE <database_name>", where is <database_name> is defined by method getDatabaseName(). Default value is "LoaderTest", and if you want to change that value, you have to override this method in class that extends LoaderTestCase class. Sql commands will be executed using Connection to the target database defined by getConnection() method.
This operation deletes database using SQL commands "DROP DATABASE <database_name>", where is <database_name> is defined by method getDatabaseName(). Default value is "LoaderTest", and if you want to change that value, you have to override this method in class that extends LoaderTestCase class.) . Sql commands will be executed using Connection to the target database defined by getConnection() method.
This operation loads data from source tables into target tables into the database using Octopus.The transformation commands are based on a special XML DTD/schema. The number of transformations in one XML loadJob file is not limited: creating databases, creating tables, inserting data into an empty database, inserting data in to an existing database, updating columns with constant values, updating relations, updating columns with system-date/time, updating columns with a current user ID, executing every possible SQL statement, creating artificial keys using the Enhydra DODS objectid logic, ...
To write a test case, please follow the steps described below.
Your test will obviously need some data to work with. This means you must create database and load data from source tables into target tables. You can create database using CreateDatabaseOperation or executing SQL statement using Octopus and its sql subelement in loaderJob configuration file.
Now you need to create a test class. The easiest way to use Octopus as a test case is to make your test class extend the LoaderTestCase class. LoaderTestCase extends the JUnit TestCase class. Two template methods are required to be implemented: getConnection() to return a connection to your database and getLoader() to return the instance of Loader class which will be used for loading data.
Following is a sample implementation that returns a connection to a MSSQL database and an instance of Loader class which uses LoadOMDSProjectTest.xml file.
public class LoaderTest extends LoaderTestCase { public LoaderTest(String name) { super(name); } public Connection getConnection() throws Exception { Class driverClass = Class.forName("com.internetcds.jdbc.tds.Driver"); Connection jdbcConnection = DriverManager.getConnection( "jdbc:freetds:sqlserver://localhost:1433/master", "sa", ""); return jdbcConnection; } /** * Override method getDatabaseName() to rename test database. */ public String getDatabaseName() throws Exception { return "OMDSTest"; } public Loader getLoader() throws Exception { Loader loadJob= new Loader("LoadOMDSProjectTest.xml"); loadJob.setUserID("admin"); return loadJob; }
Note: Please read Octopus documentation, for using setXXX Octopus' methods.
By default, LoaderTestCase performs a CreateDatabaseOperation and execute Octopus with xml file defined by getLoader() method, before executing each test and performs DropDatabaseOperation afterward. You can modify this behavior by overriding getSetUpOperation() and getTearDownOperation().
The following example demonstrates how you can easily override the operation executed before or after your test.
public class SampleTest extends DatabaseTestCase { public DatabaseOperation[] getSetUpOperation() throws Exception { // Executing loaderJob defined in xml file. DatabaseOperation[] dbOperation = new DatabaseOperation[1]; dbOperation[0]=new LoaderOperation(getLoader()); return dbOperation; } public DatabaseOperation[] getTearDownOperation() throws Exception { // Do nothing... DatabaseOperation[] dbOperation = new DatabaseOperation[1]; dbOperation[0]=DatabaseOperation.DO_NOTHING; return dbOperation; } You can use more DatabaseOperations in startUp or at the end of test. /** * Returns the database operations executed in test setup. */ public DatabaseOperation[] getSetUpOperation() throws Exception { // Creating test database..... DatabaseOperation[] dbOperation = new DatabaseOperation[4]; dbOperation[0]=new CreateDatabaseOperation(getDatabaseName()); // Creating tables..... dbOperation[1]=new LoaderOperation(getLoader()); // Inserting data..... Loader loadJob1= new Loader("LoadOMDSProjectInsertData.xml"); loadJob1.setUserID("admin"); dbOperation[2]=new LoaderOperation(loadJob1); // Creating indexes, foreign keys..... Loader loadJob2= new Loader("LoadOMDSProjectCreateIndex.xml"); dbOperation[3]=new LoaderOperation(loadJob2); return dbOperation; }
In this example, before the start of the test, four database operations would be executed.
Table of Contents
A Octopus job is a XML file correspondig to the Octopus DTD/schema. XML file is the key part for loading criteria. XML tags define how the loading job will be done.
The main tag is <loaderJob>, and it could occur just once in XML input file.
The root tag of the XML file is <loaderJob>. The element <loaderJob> may have up to fifteen attributes. These attributes are default values for the attributes with the same name in the <importDefinition> and <sql> elements. If those attributes are not specified in an <importDefinition> or <sql> element, the Octopus takes the default values from <loaderJob> element.
Example 4.1. Example of a <loaderJob> tag:
<loaderJob logMode="full" objectIDIncrement="5" objectIDTableName="myAdminObjectID" objectIDColumnName="myNext" onErrorContinue="false" commit="true" userID="r004d\meier" logDir="c:\mylogs" logFile="myFile.txt" vendorConfig="MyVendorConfig.xml" objectIDAutoCreate="true" objectIDStartValue="150000" commitCount="1000" oidLogic="false" tableMode="Cache" returnCode="1"> ... </loaderJob>
Table 4.1. <loaderJob> attributes:
attribute | default for | description | param |
---|---|---|---|
logMode | <importDefinition> and <sql> | defines the default logmode. Possible values are: "normal", "none" and "full". The system default is "normal". | -m |
objectIDIncrement | <importDefinition> | Defines the incremental value for global unique objectID's as used by Enhydra DODS. Default is 200. | none |
objectIDTableName | <importDefinition> | Defines the table name for global unique objectID's as used by Enhydra DODS. Default is "objectid". | none |
objectIDColumnName | <importDefinition> | Defines the column name for global unique objectID's as used by Enhydra DODS. Default is "next". | none |
objectIDNameColumnName | <importDefinition> | Defines the column name for OID name defined by TOS. If this value is set, Loader uses OID logic as used by TOS. | none |
objectIDNameColumnValue | <importDefinition> | Defines the column name for OID values defined by TOS. If this value is set, Loader uses OID logic as used by TOS. Type of this column is VARCHAR. | none |
onErrorContinue | <sql> | Defines if the load job should continue if an error in a SQL statement occurs. Default is "false" | -e |
commit | <sql> | Defines if SQL statement blocks should be commited separately. Default is "false" | none |
userID | none | Defines the value for userID columns. | -u |
logDir | none | Defines the target directory for the logfile. Default is the current working directory | -l |
logFile | none | Defines the logfile name. Default is "LoaderLog-YYYY-MM-DD-HH-mm-ss.txt" | -f |
vendorConfig | none | Defines the name of the DB vendor configuration XML file. Default is "OctopusDBVendors.xml" | -d |
returnCode | none | Defines the default java.exe return code to the environment if some element of loadjob fails. | -rc |
objectIDAutoCreate | <importDefinition> | Defines if the objectID table should be created automatically if not present. Default is "false". | none |
objectIDStartValue | <importDefinition> | Defines the start value for objectIDs ONLY if the objectID table is created automatically. Default is "1" | none |
commitCount | <importDefinition> | Defines the default commit count. System default is "100" | -c |
oidLogic | <importDefinition> <table> | Defines the default if tables use the DODS oid logic. The system default is "true". | none |
tableMode | <importDefinition> <table> | Defines the default access method for tables. The system default is "Query". | none |
Attributes of the loaderJob tag are overridden by the corresponding parameters of the application/constructor !
LoaderJob element may have several subelements: <restartCounter>, <variables>, <jdbcDefaultParameters>, <definitionInclude>, <importDefinition>, <sql> and <echo>
Example 4.2. Subelements
<loaderJob ...>ONE TIME SUBELEMENTS:
<restartCounter.../> <variables> ... </variables> <jdbcDefaultParameters> ... </jdbcDefaultParameters>MULTIPLE SUBELEMENTS IN EVERY ORDER/COMBINATION:
<sql> ... <sql> <definitionInclude> ... </definitionInclude> <echo message="..."/> <importDefinition> ... </importDefinition> </loaderJob>
Defines table and column name of the restart counter. This is an optional element/feature of the Octopus. Jobs are only restartable if this element is specified. Additionally importDefinitions must have a defined value sort order to be able to restart. <restartCounter> element has three attributes: tableName, importDefinitionColumnName, restartCounterColumnName and restartAutoCreate. All attributes are optional.
Table 4.2. Attributes:
attribute | description |
---|---|
tableName | Table name in target database where is restart column value. The system default is "LOADERRESTART". |
importDefinitionColumnName | Column name in the tableName where is name of importDefinition. The system default is "IMPORTDEFINITION". |
restartCounterColumnName | Column name in the tableName where is value of restartCounter for determined importDefinition. The system default is "RESTARTCOUNTER". |
restartAutoCreate | The system default is "false". |
If a loaderJob is started without the "-r" parameter all entries in this table are deleted !
Example 4.3. Example of a restartCounter
<loaderJob ...> <restartCounter tableName="MyRestart" importDefinitionColumnName="MyName" restartCounterColumnName="MyCounter" restartAutoCreate="true"/> ... </loaderJob>
This element together with it's subelements defines logical variables to be used in constantColumns, SQL-statements, source-data and JDBC parameters.
Table 4.3. Attributes:
attribute | description |
---|---|
prefix | Default value for attribute prefix in each <variable> tag. |
suffix | Default value for attribute suffix in each <variable> tag. |
override | Default value for attribute override in each <variable> tag. |
replaceInConstants | Default value for attribute replaceInConstants in each <variable> tag. |
replaceInSQL | Default value for attribute replaceInSQL in each <variable> tag. |
replaceInData | Default value for attribute replaceInData in each <variable> tag. |
replaceInJDBC | Default value for attribute replaceInJDBC in each <variable> tag. |
The system defaults are: prefix="%" suffix="%" override="true" replaceInConstants="false" replaceInSQL="false" replaceInData="false" replaceInJDBC="false".
All occurences of prefix+name+suffix will be replaced by value or the override value specified in the application/constructor parameters (if the override="true"). The replace will take place where is replaceIn... attribute is equal "false". The system defaults are: prefix="%" suffix="%" override="true" replaceInConstants="false" replaceInSQL="false" replaceInData="false" replaceInJDBC="false".
Table 4.4. Attributes:
attribute | description | default value |
---|---|---|
name | See Note. | |
value | See Note. | |
prefix | See Note. | % |
suffix | See Note. | % |
override | See Note. | true |
replaceInConstants | The replace take place in Constants. | false |
replaceInSQL | The replaces take place in SQL. | false |
replaceInData | The replaces take place in Data. | false |
replaceInJDBC | The replaces take place in JDBC. | false |
Note: prefix+name+suffix will be replaced with value or the override value specified in the application/constructor parameters (if the override="true"). name is required parameter.
Example 4.4. Example of variables
<loaderJob ...> ... <variables prefix="!" suffix="!" override="false" replaceInConstants="true" replaceInSQL="false" replaceInData="true" replaceInJDBC="true"> <variable name="myvar1" value="myVal1" prefix="$" suffix="$" override="true" replaceInConstants="false" replaceInSQL="true" replaceInData="false" replaceInJDBC="false"/> <variable name="myvar2" prefix="$" suffix="$"/> <variable name="myvar3" value="myVal3"/> </variables> ... </loaderJob>
In this example the following will be done:
All occurences of "$myvar1$" will be replaced by "myVal1" or the override value specified in the application/constructor parameters. The replace will take place in SQL statements only.
All occurences of "$myvar2$" will be replaced by the value specified in the application/constructor parameters. The replace will take place in constant columns, source data and JDBC parameters. If there is no value specified in the application/constructor parameters for this variable name an exception will occur. A NULL value for myvar2 can be specified by "-v myvar2=;myvar3=myval3"
All occurences of "!myvar3!" will be replaced by the value "myVal3" which can NOT be overridden by parameters. Conflicts between variable definitions and parameters lead to a runtime exception ! The replace will take place in constant columns, source data and JDBC parameters.
The system defaults are: prefix="%" suffix="%" override="true" replaceInConstants="false" replaceInSQL="false" replaceInData="false" replaceInJDBC="false"
Defines default JDBC parameters to enable connection on source and target database. It can occur only ones. <jdbcParameters> tag has two child tags: <jdbcSourceParameters> and <jdbcTargetParameters>. This tag doesn't have any attribute.
<jdbcSourceParameters> defines JDBC source parameters. This tag may occur only once. This tag has two attributes - dbVendor and driverName. dbVEndor is name of database or database type, driverName. Depending of those value Octopus reads driver parameters for that database type from Octopus configuration file - OctopusDBVendors.xml. Possible values for attribute dbVendor as well possible source database types are : mssql (MSSQL server), mysql (MySlq database), excel (Source table in Excel format), cvs (separators: ';' or ',')), access (Access database), PostgreSQL, Qed, instantDB, Hsqldb(HSQL database), Oracle, JDataStore (BorlandJDataStore database) and DB2. Default value of dbVendor is csv. Default value of driverName is csv too. Child element of this tag is <jdbcSourceParameter>.
<jdbcSourceParameter> tag defines JDBC parameters: JDBC driver, connection URL, user and password. This tag occurs at least once (for connection URL). JdbcTargetParameter for JDBC driver is optional and its default values Octopus reads from OctopusDbVendors.xml configuration file. User and password are optional.
Table 4.5. Attributes:
attribute | description |
---|---|
name | Name of the JDBC parameters for source database. Possible values are: JdbcDriver, Connection.Url, User and Password. |
value | Value of the appropriate parameter, depends on the value of
the attribute name. name="JdbcDriver" attribute value is JDBC driver; name="Connection.Url" attribute value is connection URL; name="User" attribute value is user name, tag with this attribute name is optional; name="Password" attribute value is password, tag with this attribute name is optional. |
<jdbcTargetParameters> defines JDBC source parameters. This tag may occur only once. This tag has two optional attributes - dbVendor and driverName. dbVEndor is name of database or database type, driverName. Depending of those value Octopus reads driver parameters for that database type from Octopus configuration file - OctopusDBVendors.xml. Possible values for attribute dbVendor as well possible source database types are : mssql (MSSQL server), mysql (MySlq database), excel (Source table in Excel format), cvs (separators: ';' or ',')), access (Access database), PostgreSQL, Qed, instantDB, Hsqldb(HSQL database), Oracle, JDataStore (BorlandJDataStore database) and DB2. Default value of dbVendor is mssql. Default value of driverName is jTurbo. Child element of this tag is <jdbcTargetParameter>.
<jdbcTargetParameter> tag defines JDBC parameters: JDBC driver, connection URL, user and password. This tag occurs at least once (for connection URL). JdbcTargetParameter for JDBC driver is optional and its default values Octopus reads from OctopusDbVendors.xml configuration file. User and password are optional. If there is not values for user and password, application is automatically asking for those values later.
Table 4.6. Attributes:
attribute | description |
---|---|
name | Name of the JDBC parameters for target database. Possible values are: JdbcDriver, Connection.Url, User and Password. |
value | Value of the appropriate parameter, depends on the value of
the attribute name. name="JdbcDriver" attribute value is JDBC driver; name="Connection.Url" attribute value is connection URL; name="User" attribute value is user name, tag with this attribute name is optional; name="Password" attribute value is password, tag with this attribute name is optional. |
Example 4.5. Example of JDBC default parameters
<loaderJob ...> ... <jdbcDefaultParameters> <jdbcSourceParameters> <jdbcSourceParameter name="JdbcDriver" value="org.myDriver.JDBCDriver"/> <jdbcSourceParameter name="Connection.Url" value="jdbc:myDriver:MySourceDatabase"/> <jdbcSourceParameter name="User" value="myDBUserName"/> <jdbcSourceParameter name="Password" value="myDBPassword"/> </jdbcSourceParameters> <jdbcTargetParameters> <jdbcTargetParameter name="JdbcDriver" value="org.myDriver2.JDBCDriver2"/> <jdbcTargetParameter name="Connection.Url" value="jdbc:myDriver2:MyTargetDatabase"/> <jdbcTargetParameter name="User" value="myDBUserName2"/> <jdbcTargetParameter name="Password" value=""/> </jdbcTargetParameters> </jdbcDefaultParameters> ... </loaderJob>
If there is no specified password you will be asked during runtime.
If there is no specified JdbcDriver class Octopus will use class name from OctopusDbVendors.xml configuration file.
All values can use the variable-replace functionality defined in the variables tag.
Defines SQL statements executing in target database. Target database parameters are in the <jdbcTargetParameters> tag. <sql> tag is optional, and there could be more in loaderJob. <sql> tag has four attributes: name, logMode, onErrorContinue and commit; and it also has two child tags: <jdbcTargetParameters> and <sqlStmt>.
Table 4.7. Attributes:
attribute | description |
---|---|
name | Name of the <sql> tag. This attribute is required. |
logMode | This attribute is optional.This mode defines which steps of the loading process will be written in log file. Possible values: none, normal and full. None - start and finish of the application will be notified; Normal - writes start and finish of each importDefinition process, importDefinition name, number of committed rows; Full - writes detailed information about the loading process into the log file.This attribute is required. For each of these modes, time start/end and the name of the job will be written. The system default is Normal. |
onErrorContinue | Value is true or false. When value is true, if an error occurs, error message is shown and application continue. When value is false, if an error occurs, error message is shown and application will be finished. This attribute is optional. The system default is false. |
commit | Value is true or false. If value is true, SQL statement will be committed, else it will be committed at the end of the application.This attribute is optional. The system default is true. |
returnCode | Defines the default java.exe return code to the environment if some element of loadjob fails.This attribute is optional. The system default is 1. |
Example 4.6. Example of the sql tag
<loaderJob ...> ... <sql logMode="none" onErrorContinue="true" commit="true"> <jdbcTargetParameters> <jdbcTargetParameter name="JdbcDriver" value="org.myDriver2.JDBCDriver2"/> <jdbcTargetParameter name="Connection.Url" value="jdbc:myDriver2:MyTargetDatabase"/> <jdbcTargetParameter name="User" value="myDBUserName2"/> <jdbcTargetParameter name="Password" value=""/> </jdbcTargetParameters> <sqlStmt> CREATE DATABASE Together; CREATE TABLE tab1 (col1 varchar(250)); </sqlStmt> </sql> ... <sql logMode="full" onErrorContinue="false" commit="false"> <jdbcTargetParameters> <jdbcTargetParameter name="JdbcDriver" value="org.myDriver2.JDBCDriver2"/> <jdbcTargetParameter name="Connection.Url" value="jdbc:myDriver2:MyTargetDatabase"/> <jdbcTargetParameter name="User" value="myDBUserName2"/> <jdbcTargetParameter name="Password" value=""/> </jdbcTargetParameters> <sqlStmt> <include href="c:\mySQLFile1" parse="text"/> </sqlStmt> </sql> ... </loaderJob>
<jdbcTargetParameters> defines JDBC source parameters. This tag may occur only once. This tag has two optional attributes - dbVendor and driverName. dbVEndor is name of database or database type, driverName. Depending of those value Octopus reads driver parameters for that database type from Octopus configuration file - OctopusDBVendors.xml. Possible values for attribute dbVendor as well possible source database types are : mssql (MSSQL server), mysql (MySlq database), excel (Source table in Excel format), cvs (separators: ';' or ',')), access (Access database), PostgreSQL, Qed, instantDB, Hsqldb(HSQL database), Oracle, JDataStore (BorlandJDataStore database) and DB2. Default value of dbVendor is mssql. Default value of driverName is jTurbo. Child element of this tag is <jdbcTargetParameter>.
<jdbcTargetParameter> tag defines JDBC parameters: JDBC driver, connection URL, user and password. This tag occurs at least once (for connection URL). JdbcTargetParameter for JDBC driver is optional and its default values Octopus reads from OctopusDbVendors.xml configuration file. User and password are optional. If there is not values for user and password, application is automatically asking for those values later.
Table 4.8. Attributes:
attribute | description |
---|---|
name | Name of the JDBC parameters for target database. Possible values are: JdbcDriver, Connection.Url, User and Password. |
value | Value of the appropriate parameter, depends on the value of
the attribute name. name="JdbcDriver" attribute value is JDBC driver; name="Connection.Url" attribute value is connection URL; name="User" attribute value is user name, tag with this attribute name is optional; name="Password" attribute value is password, tag with this attribute name is optional. |
<sqlStmt> defines SQL Statement. This tag is required, and it may occur only once. <sqlStmt> tag doesn't have any attribute. It has text (sql statement). Child of this tag is <include> tag.
<include> tag defines file to be included. This tag is optional, and it may occur only once. <include> tag has two attributes: href and parse. This tag doesn't have any child tag.
Table 4.9. Attributes:
attribute | description |
---|---|
href | file to be included |
parse | file's type (xml or text) |
If parse = "xml", Octopus substitutes include element with another XML file.
If parse = "text", Octopus substitutes include element with another text file with SQL commands.
Example 4.7. Example of the sqlStmt tag:
<sqlStmt> CREATE DATABASE Together; </sqlStmt>
or:
<sqlStmt> <include href="SQLproba.sql" parse="text"/> </sqlStmt>
<definitionInclude> tag defines another xml file with import definitions to be included. This tag is optional, and it may occur more than once. <definitionInclude> tag has one child element (<include>) with two attributes: href and parse. Tag <include> is required and may could occur only once.
<include> tag defines file to be included. This tag is required, and it may occur only once. <include> tag has two attributes: href and parse. This tag doesn't have any child tag.
Table 4.10. Attributes:
attribute | description |
---|---|
href | file to be included |
parse | file's type (xml or text) |
Example 4.8. Example of the definitionInclude tag
<loaderJob ...> ... <definitionInclude> <include href="c:\MyImportDefinitions1" parse="xml"/> </definitionInclude> ... </loaderJob>
The included file must have the following structure/content:
Example 4.9. Example of an included definition file
<definitionInclude> <importDefinition...> ... </importDefinition...> <importDefinition...> ... </importDefinition...> ... </definitionInclude>
Echo element is implemented in parallel to importDefinitions and sql's. This element has only one attribute (required) message. Octopus writes this messages in log output (screen and log file) in logmode normal and logmode full.
Example 4.10. Example of an echo elemnt
<echo message ="Log message..."/>
<importDefinition> defines loading process. It is parent tag to other tags connected with the same loading process. It is optional tag, and there could be more. <importDefinition> tag has seven child tags: <sortColumns>, <jdbcParameters>, <valueColumns>, <variableColumns>, <relationColumns>, <constantColumns>, <counterColumns> and <tables>.
Table 4.11. Attributes:
attribute | description |
---|---|
name | Name of the import definition. The name attribute of importDefinitions must be unique within one loaderJob in order to make the restart functionality work correctly. This attribute is required. |
tableName | Name of the source table in the source database. This attribute is optional.If not exists, you must set selectStatement for loading data using sql statement |
selectStatement | Select statement for loading data from source tables. This attribute is optional. Required if table name doesn't exist. |
commitCount | Number of rows in source table after which the target database will be committed. This attribute is optional. If the attribute doesn't exist, the default (from loaderJob) will be used. |
logMode | This mode defines which steps of the loading process will
be written in log file. Possible values: none (Loader.LOGMODE_NONE),
normal (Loader.LOGMODE_NORMAL) and full
(Loader.LOGMODE_FULL).None - start and finish of the application will be notified; Normal - writes start and finish of each importDefinition process, importDefinition name, number of committed rows; Full - writes detailed information about the loading process into the log file.This attribute is required. For each of these modes, time start/end and the name of the job will be written. This attribute is optional. The system default is Loader.LOGMODE_NORMAL |
objectIDIncrement | This number defines the OID increment value. This attribute is optional and the default is 200. |
objectIDTableName | Defines table name where is oid number. This attribute is optional. Default value is ObjectID. |
objectIDColumnName | Defines column name where is oid number. This attribute is optional. Default value is next. |
objectIDNameColumnName | Defines the column name for OID name defined by TOS. If this value is set, Loader uses OID logic as used by TOS. This attribute is optional. |
objectIDNameColumnValue | Defines the column name for OID values defined by TOS. If this value is set, Loader uses OID logic as used by TOS. Type of this column is VARCHAR. This attribute is optional. |
returnCode | Defines the default java.exe return code to the environment if some element of loadjob fails. |
Example 4.11. Example of the importDefinition tag
<loaderJob ...> ... <importDefinition name="MyDef1" tableName="MySourceTable1" logMode="full" objectIDIncrement="2" objectIDTableName="Myoid" objectIDColumnName="MyNext" objectIDAutoCreate="true" objectIDStartValue="500" commitCount="10000" returnCode="1"> <sortColumns> <sortColumn entry="ColName1"/> <sortColumn entry="ColName2"/> </sortColumns> <jdbcParameters> <jdbcSourceParameters> <jdbcSourceParameter name="JdbcDriver" value="org.myDriver.JDBCDriver"/> <jdbcSourceParameter name="Connection.Url" value="jdbc:myDriver:MySourceDatabase"/> <jdbcSourceParameter name="User" value="myDBUserName"/> <jdbcSourceParameter name="Password" value="myDBPassword"/> </jdbcSourceParameters> <jdbcTargetParameters> <jdbcTargetParameter name="JdbcDriver" value="org.myDriver2.JDBCDriver2"/> <jdbcTargetParameter name="Connection.Url" value="jdbc:myDriver2:MyTargetDatabase"/> <jdbcTargetParameter name="User" value="myDBUserName2"/> <jdbcTargetParameter name="debug" value="on"/> </jdbcTargetParameters> </jdbcParameters> <valueColumns> <valueColumn sourceColumnName="Col19" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col17" valueMode="Key"/> <valueColumn sourceColumnName="Col20" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col18" valueMode="Update"/> <valueColumn sourceColumnName="Col21" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col19" valueMode="Overwrite"/> <valueColumn sourceColumnName="Col22" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col20" valueMode="SetIfCreated"/> <valueColumn sourceColumnName="Col22" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col21" valueMode="SetNull"/> <valueColumn sourceColumnName="Col31" targetTableName="TargetTab2" targetTableID="0" targetColumnName"Col17" valueMode="Key"/> <valueColumn sourceColumnName="Col32" targetTableName="TargetTab3" targetTableID="0" targetColumnName"Col18" valueMode="Key"/> <valueColumn sourceColumnName="Col33" targetTableName="TargetTab4" targetTableID="0" targetColumnName"Col19" valueMode="Key"/> <valueColumn sourceColumnName="Col34" targetTableName="TargetTab5" targetTableID="0" targetColumnName"Col20" valueMode="Key"/> <valueColumn sourceColumnName="Col35" targetTableName="TargetTab5" targetTableID="1" targetColumnName"Col21" valueMode="Key"/> </valueColumns> <variableColumns> <variableColumn name="var1" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col11" valueMode="Key"/> <variableColumn name="var2" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col12" valueMode="Update"/> <variableColumn name="var3" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col13" valueMode="Overwrite"/> <variableColumn name="var4" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col14" valueMode="SetIfCreated"/> <variableColumn name="var5" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col15" valueMode="SetNull"/> <userIDColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col22" valueMode="Key"/> <userIDColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col23" valueMode="Update"/> <userIDColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col24" valueMode="Overwrite"/> <userIDColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col25" valueMode="SetIfCreated"/> <userIDColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col26" valueMode="SetNull"/> <timeStampColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col32" valueMode="Key"/> <timeStampColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col33" valueMode="Update"/> <timeStampColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col34" valueMode="Overwrite"/> <timeStampColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col35" valueMode="SetIfCreated"/> </variableColumns> <constantColumns> <constantColumn constantValue="myVal%var12345%" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col47" valueMode="Key"/> <constantColumn constantValue="myVal2" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col48" valueMode="Update"/> <constantColumn constantValue="myVal3" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col49" valueMode="Overwrite"/> <constantColumn constantValue="myVal4" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col50" valueMode="SetIfCreated"/> <constantColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col51" valueMode="Overwrite"/> <constantColumn targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col52" valueMode="SetIfCreated"/> </constantColumns> <counterColumns counterTableName="counterTable" counterNameColumn=Col01 counterValueColumn="valueCol" > <counterColumn counterName="counter1" counterStartValue="100" counterIncrement="1" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col88" valueMode="SetIfCreated" counterStartValueReset="true"/> <counterColumn counterName="counter2" counterStartValue="1" counterIncrement="10" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col89" valueMode="SetIfCreated" counterStartValueReset="false"/> <subCounterColumn constantValue="myVal3" targetTableName="TargetTab1" targetTableID="0" targetColumnName="Col49" valueMode="Overwrite"/> <subCounterKeyColumn keyColumnName="Col99"/> </subCounterColumn> </counterColumns> <relationColumns> <relationColumn relationSourceTableName="TargetTab2" relationSourceTableID="0" relationSourceColumnName="Key21" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col61" valueMode="Key"/> <relationColumn relationSourceTableName="TargetTab2" relationSourceTableID="0" relationSourceColumnName="Key22" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col62" valueMode="Key"/> <relationColumn relationSourceTableName="TargetTab3" relationSourceTableID="0" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col63" valueMode="Update"/> <relationColumn relationSourceTableName="TargetTab4" relationSourceTableID="0" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col64" valueMode="Overwrite"/> <relationColumn relationSourceTableName="TargetTab5" relationSourceTableID="0" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col65" valueMode="SetIfCreated"/> <relationColumn relationSourceTableName="TargetTab5" relationSourceTableID="1" targetTableName="TargetTab1" targetTableID="0" targetColumnName"Col66" valueMode="SetNull"/> </relationColumns> <tables> <table tableName="TargetTab1" tableID="0" insert="true" tableMode="Query" oidLogic="true"/> <table tableName="TargetTab2" tableID="0" insert="false" tableMode="Cache" oidLogic="false"/> <table tableName="TargetTab3" tableID="0" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="TargetTab4" tableID="0" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="TargetTab5" tableID="0" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="TargetTab5" tableID="1" insert="false" tableMode="Cache" oidLogic="true"/> </tables> </importDefinition> ... </loaderJob>
The "name" attribute of importDefinitions must be unique within one loaderJob in order to make the restart functionality work correctly.
<sortColumns> defines columns of the source table which are used as a sort criteria. This is child tag of the <importDefinition>. The <sortColumn> tag is optional, and it may occur only once. It has one child tag: <sortColumn>.
<sortColumn> tag defines column of source table, which is used for sorting source data. There could be more <sortColumn> tags. It has one attribute: entry, and it hasn't any child tag.
Table 4.12. Attributes:
attribute | description |
---|---|
entry | column name which will be sort column in the loading process |
Example 4.12. Example of the <sortColumn> tag
<sortColumn entry="KeyValue"/>
Also, Octopus checks if sort columns in source table have unique values. If source table has more rows with equal values in sort columns return Exception.
<jdbcParameters> defines JDBC parameters to enable connection on source and target database. It can occur only once. <jdbcParameters> tag has two child tags: <jdbcSourceParameters> and <jdbcTargetParameters>. This tag doesn't have any attribute.
<jdbcSourceParameters> defines JDBC source parameters. This tag may occur only once. This tag has two attributes - dbVendor and driverName. dbVEndor is name of database or database type, driverName. Depending of those value Octopus reads driver parameters for that database type from Octopus configuration file - OctopusDBVendors.xml. Possible values for attribute dbVendor as well possible source database types are : mssql (MSSQL server), mysql (MySlq database), excel (Source table in Excel format), cvs (separators: ';' or ',')), access (Access database), PostgreSQL, Qed, instantDB, Hsqldb(HSQL database), Oracle, JDataStore (BorlandJDataStore database) and DB2. Default value of dbVendor is csv. Default value of driverName is csv too. Child element of this tag is <jdbcSourceParameter>.
Child element of this tag is <jdbcSourceParameter>.
<jdbcSourceParameter> tag defines JDBC parameters: JDBC driver, connection URL, user and password. This tag occurs at least once (for connection URL). JdbcTargetParameter for JDBC driver is optional and its default values Octopus reads from OctopusDbVendors.xml configuration file. User and password are optional.
Table 4.13. Attributes:
attribute | description |
---|---|
name | Name of the JDBC parameters for source database. Possible values are: JdbcDriver, Connection.Url, User and Password. |
value | Value of the appropriate parameter, depends on the value of the attribute name. name="JdbcDriver" attribute value is JDBC driver; name="Connection.Url" attribute value is connection URL; name="User" attribute value is user name, tag with this attribute name is optional; name="Password" attribute value is password, tag with this attribute name is optional. |
Example 4.13. Example of the <jdbcSourceParameter> tag
<jdbcSourceParameter name="JdbcDriver" value="sun.jdbc.odbc.JdbcOdbcDriver"/> <jdbcSourceParameter name="Connection.Url" value="jdbc:odbc:Loader"/> <jdbcSourceParameter name="User" value="sa"/> <jdbcSourceParameter name="Password" value="as"/>
<jdbcTargetParameters> defines JDBC source parameters. This tag may occur only once. This tag has two optional attributes - dbVendor and driverName. dbVEndor is name of database or database type, driverName. Depending of those value Octopus reads driver parameters for that database type from Octopus configuration file - OctopusDBVendors.xml. Possible values for attribute dbVendor as well possible source database types are : mssql (MSSQL server), mysql (MySlq database), excel (Source table in Excel format), cvs (separators: ';' or ',')), access (Access database), PostgreSQL, Qed, instantDB, Hsqldb(HSQL database), Oracle, JDataStore (BorlandJDataStore database) and DB2. Default value of dbVendor is mssql. Default value of driverName is jTurbo. Child element of this tag is <jdbcTargetParameter>.
Child element of this tag is <jdbcTargetParameter>.
<jdbcTargetParameter> tag defines JDBC parameters: JDBC driver, connection URL, user and password. This tag occurs at least once (for connection URL). JdbcTargetParameter for JDBC driver is optional and its default values Octopus reads from OctopusDbVendors.xml configuration file. User and password are optional.
Table 4.14. Attributes:
attribute | description |
---|---|
name | Name of the JDBC parameters for target database. Possible values are: JdbcDriver, Connection.Url, User and Password. |
value | Value of the appropriate parameter, depends on the value of the attribute name. name="JdbcDriver" attribute value is JDBC driver; name="Connection.Url" attribute value is connection URL; name="User" attribute value is user name, tag with this attribute name is optional; name="Password" attribute value is password, tag with this attribute name is optional. |
Example 4.14. Example of the <jdbcTargetParameter> tag
<jdbcTargetParameter name="JdbcDriver" value="sun.jdbc.odbc.JdbcOdbcDriver"/> <jdbcTargetParameter name="Connection.Url" value="jdbc:odbc:Loader"/> <jdbcTargetParameter name="User" value="sa"/> <jdbcTargetParameter name="Password" value="as"/>
<valueColumns> defines connections between source and target columns in source and target databases. This tag has child tag <valueColumn>. This tag is optional.
<valueColumn> tag defines source column and target table and column and value mode between them. This tag must occur one or more times.
Table 4.15. Attributes:
attribute | description |
---|---|
sourceColumnName | Column name in source table in source database. This attribute is required. |
targetTableName | Table name in target database. This attribute is required. |
targetColumnName | Column name in table in target database. This attribute is required. |
targetTableID | Defines logical target table. It is possible to have more logical target tables then target tables. This attribute can take values: 0, 1, 2, ... This attribute is required. This attribute defines more logical tables at the same physical table (use same table several times). |
valueMode | Defines updating mode. Possible modes are: Key,
Overwrite, Update, SetIfCreated and SetNull. This attribute is required.
|
Example 4.15. Example of the <valueColumn> tag
<valueColumn sourceColumnName="LONGNAME" targetTableName="PERSISTENTTREETYPE" targetColumnName="LONGNAME" targetTableID="0" valueMode="Overwrite"/> <valueColumn sourceColumnName="KEYVALUE" targetTableName="PERSISTENTTREETYPE" targetColumnName="KEYVALUE" targetTableID="0" valueMode=" Key"/> <valueColumn sourceColumnName="DATASTRING01DESC" targetTableName="PERSISTENTTREETYPE" targetColumnName="DATASTRING01DESC" targetTableID="0" valueMode="Update"/> <valueColumn sourceColumnName="XMLNAME" targetTableName="PERSISTENTTREETYPE" targetColumnName="XMLNAME" targetTableID="0" valueMode="SetIfCreated"/>
<variableColumns> define target columns with current time value and user ID value. The updating mode is defined too. This tag is optional. Child tags are: <variableColumn>, <userIDColumn> and <timeStampColumn>.
<variableColumn> tag defines value columns with parameter. This tag is optional. The Octopus parameter -v (defines variables) and variable tag is required for using this option. Value of attribute name (in the tag <variableColumn>) will be replaced with value of the attribute value in variable column. nameValue.
Table 4.16. Attributes:
attribute | description |
---|---|
name | Name of variable column. This value (value of the attribute name) will be replaced with value of the Loader class constructor parameter nameValue. |
targetTableName | Defines target table name. This attribute is required. |
targetColumnName | Defines column name in target table. This attribute is required. |
targetTableID | Defines logical target table. It is possible to have more logical target tables then target tables. This attribute can take values: 0, 1, 2, ... This attribute is required. This attribute defines more logical tables at the same physical table (use same table several times). |
valueMode | Defines updating mode. Possible modes are: Overwrite,
Update, SetIfCreated and SetNull. This attribute is required.
|
After replacing values (in attribute name) Octopus processes this tag as a usual value columns.
Example 4.16. Example of the <variableColumn> tag
<variableColumn name="-v_name" targetTableName="Person" targetColumnName="PersHobby" targetTableID="0" valueMode="Overwrite"/>
<userIDColumn> tag defines a table and column in the target database where the value of the Octopus argument "userID". This tag is optional.
Table 4.17. Attributes:
attribute | description |
---|---|
targetTableName | Defines target table name. This attribute is required. |
targetColumnName | Defines column name in target table. This attribute is required. |
targetTableID | Defines logical target table. It is possible to have more logical target tables then target tables. This attribute can take values: 0, 1, 2, ... This attribute is required. This attribute defines more logical tables at the same physical table (use same table several times). |
valueMode | Defines updating mode. Possible modes are: Key,
Overwrite, Update, SetIfCreated and SetNull. This attribute is required.
|
Octopus processes UserID variable column as a relation column (write data into target table using relations criteria).
Example 4.17. Example of the <userIDColumn> tag
<userIDColumn targetTableName="OMDSBENUTZER" targetColumnName="USERID" targetTableID="0" valueMode="Key"/>
<timeStampColumn> tag defines target column in witch to overwrite or update current time value. Overwrite or update depends on value mode. This tag is optional.
Table 4.18. Attributes:
attribute | description |
---|---|
targetTableName | Defines target table name. This attribute is required. |
targetColumnName | Defines column name in target table. This attribute is required. |
targetTableID | Defines logical target table. It is possible to have more logical target tables then target tables. This attribute can take values: 0, 1, 2, ... This attribute is required. This attribute defines more logical tables at the same physical table (use same table several times). |
valueMode | Defines updating mode. Possible modes are: Overwrite,
Update, SetIfCreated and SetNull. This attribute is required.
|
Octopus processes timeStamp column as a relation column (write data into target table using relations criteria).
Example 4.18. Example of the <timeStampColumn> tag
<timeStampColumn targetTableName="Geschaeftsfallmenue" targetColumnName="ERZEUGTAMUM" targetTableID="0" valueMode="SetIfCreated"/>
<relationColumns> defines relations between target tables. This tag is optional. Child tag is <relationColumn>.
<relationColumn> tag defines a relation between columns in target tables. Those relations are between tables in target database. This tag must occur one or more times.
Table 4.19. Attributes:
attribute | description |
---|---|
relationSourceTableName | Table name of the source table in relations. This table is in target database. This attribute is required. |
relationSourceColumnName | Source column name. This attribute is optional. Default value is oid. |
relationSourceTableID | Defines logical source table. It is possible to have more logical source tables then source tables. This attribute can take values: 0, 1, 2, ... This attribute is required. This attribute defines more logical tables at the same physical table (use same table several times). |
targetTableName | Table name of the target table in relations. This table is in target database. This attribute is required. |
targetColumnName | Target column name. This attribute is optional. Default value is oid (when oidLogic = true for that table). |
targetTableID | Defines logical target table. It is possible to have more logical target tables then target tables. This attribute can take values: 0, 1, 2, ... This attribute is required. This attribute defines more logical tables at the same physical table (use same table several times). |
valueMode | Defines updating mode. Possible modes are: Key,
Overwrite, Update and SetNull. This attribute is required.
|
Example 4.19. Example of the <relationColumn> tag
<relationColumn relationSourceTableName="PERSISTENTTREE" SourceColumnName="OID" relationSourceTableID="0" TargetTableName="PERSISTENTTREE" TargetColumnName="PERSISTENTTREETYPEOID" relationTargetTableID="0" Mode="Overwrite"/>
<constantColumns> inserts constant values in the target columns. This tag is optional. Child tag is <constantColumn>.
<constantColumn> inserts constant values in the target columns. This tag must occur one or more times. It has five attributes: targetTableName, targetColumnName, targetTableID, valueMode and constantValue; and it hasn't any child tag.
Table 4.20. Attributes:
attribute | description |
---|---|
targetTableName | Table name of the target table. This attribute is required. |
targetColumnName | Target column name. This attribute is optional. Default value is oid. |
targetTableID | Defines logical target table. It is possible to have more logical target tables then target tables. This attribute can take values: 0, 1, 2, ... This attribute is required. This attribute defines more logical tables at the same physical table (use same table several times). |
valueMode | Defines updating mode. Possible modes are: Key,
Overwrite, Update, SetIfCreated and SetNull. This attribute is required.
|
constantValue | Constant value. This attribute is optional. If a constantColumn has no constantValue attribute the value is NULL. |
Example 4.20. Example of the <constantColumn> tag
<constantColumn targetTableName="OMDSBENUTZER" targetColumnName="GESCHLECHTOID" targetTableID="0" valueMode="Overwrite" constantValue="2"/>
<counterColumns> inserts auto increment values in the target columns. This tag is optional. Subelements of this tag are <counterColumn>(inserting simple auto increment values in counterColumn) and subCounterColumn (inserting auto increment values in counterColumn for each different combination of key values defined by subCounterKeyColumns).
Table 4.21. Attributes:
attribute | description |
---|---|
counterTableName | Table of counters and its current values. This attribute is required. |
counterNameColumn | Counter name column. Must be unique for all counters in one importDefinition. This attribute is required. |
counterValueColumn | Column with current counter values for each counterColumn name. This attribute is required. |
<counterColumn> inserts auto increment values in the target columns. This element is optional. It has eight attributes: targetTableName, targetColumnName, targetTableID, valueMode, counterName, counterStartValue, counterIncrement and counterStartValueReset.
Table 4.22. Attributes:
attribute | description |
---|---|
counterName | Name of counter. It must be unique into one importDefinition. This attribute is required. |
counterStartValue | Start value of counter. This attribute is optional. Default value is 1. |
counterIncrement | Increment of counter. This attribute is optional. Default value is 1. |
counterStartValueReset | Possible values are true and false. If value is false counter starts from value which read from counterTable. If value is true, counter start value is counterStartValue. This attribute is required. |
targetTableName | Table name of the target table. This attribute is required. |
targetColumnName | Target counter column name. This attribute is required. |
targetTableID | Defines logical target table. It is possible to have more logical target tables then target tables. This attribute can take values: 0, 1, 2, ... This attribute is required. This attribute defines more logical tables at the same physical table (use same table several times). |
valueMode | Defines updating mode. Possible mode is SetIfCreated (other modes not implemented yet). This attribute is required. |
<subCounterColumn> inserts auto increment values in counterColumn for each different combination of key values defined by subCounterKeyColumns (subelements of subCounterColumn). This element is optional. It has seven attributes: targetTableName, targetColumnName, targetTableID, valueMode, counterName, counterStartValue and counterIncrement.
Table 4.23. Attributes:
attribute | description |
---|---|
counterName | Name of counter. Must be unique into one importDefinition. This attribute is required. |
counterStartValue | Start value of counter. This attribute is optional. Default value is 1. |
counterIncrement | Increment of counter. This attribute is optional. Default value is 1. |
targetTableName | Table name of the target table. This attribute is required. |
targetColumnName | Target counter column name. This attribute is required. |
targetTableID | Defines logical target table. It is possible to have more logical target tables then target tables. This attribute can take values: 0, 1, 2, ... This attribute is required. This attribute defines more logical tables at the same physical table (use same table several times). |
valueMode | Defines updating mode. Possible mode is SetIfCreated (other modes not implemented yet). This attribute is required. |
Example 4.21. Example of the <counterColumns> tag
<counterColumns counterTableName="counterTable" counterNameColumn=Col01 counterValueColumn="valueCol" > <counterColumn counterName="counter1" counterStartValue="100" counterIncrement="1" targetTableName="TargetTab1" targetTableID="0" targetColumnName"ColCounter" valueMode="SetIfCreated" counterStartValueReset="true"/> <subCounterColumn constantValue="myVal3" targetTableName="TargetTab1" targetTableID="0" targetColumnName="Col49" valueMode="Overwrite"/> <subCounterKeyColumn keyColumnName="Col1"/> <subCounterKeyColumn keyColumnName="ForeignKey1"/> </subCounterColumn> </counterColumns>
<tables> defines target tables. This tag is required.
<table> tag defines target table, insert mode, table mode and oid logic. Target table is table in target database. This tag must occur one or more times.
Table 4.24. Attributes:
attribute | description |
---|---|
tableName | Table name in the target database. This attribute is required. |
tableID | Defines logical target table. It is possible to have more logical target tables then target tables. This attribute can take values: 0, 1, 2, ... This attribute is required. This attribute defines more logical tables at the same physical table (use same table several times). |
insert | Possible values are true or false. If value is true, data will be inserted otherwise data won't be inserted. This attribute is required. |
tableMode | Possible values are Cache or Query. If value is Cache, all data will be read from the source table and put into memory and finally put into target database. If value is Query, data will be read and load raw by raw. This attribute is required. |
oidLogic | Possible values are true or false. If value is true, target table has oid column and supports oid logic. This attribute is required. |
Example 4.22. Example of the <table> tag
<table tableName="PERSISTENTTREETYPE" tableID="0" insert="true" tableMode="Cache" oidLogic="true"/>
The important thing in loading process is to define Key columns in target tables. These columns are defined in valueColumn, variableColumn and relationColumn tags. Where the value of the attribute valueMode, variableMode and relationMode in this tags respectively is Key. All this columns are Key columns.
Octopus class version 2.0 uses configuration file for different types of sorce and target databases or tables types. OctopusDBVendors.xml is configuration file for ditterent types of databases. It is written in XML format. It consists of several elements dbVendor. Each dbVendor element describes some sprecial characteristics for jdbc driver of that type database. All of these characteristics are sub elements of main element dbVendor:
DriverName Name of driver used for connecting to the database
ClassNameName of jdbc driver class. Default value if this data is not set in jdbcParameters tag.
RequiredUser may have value "true" or "false". If database requires user and password for connecting value is "true" else if not value is "false".
FirstColumnResult may have value "0" or "1". This element has value of first column in result set.
EnableJumpInResult may have value "true" or "false". If jdbc driver support absolute(int ) and relative(int ) methods (jumping in result set) value is "true" else if not value is "false".
EnablePreviousRecord may have value "true" or "false". If jdbc driver supports previous() method (return for one row back in result set) value is 'true" and if it doesn't support, value is "false".
BeforeFirstRow may have value "true" or "false". If jdbc driver supports moving pointer before the first row in result set (position 0) value is "true" and if it doesn't support, value is "false".
AfterLastRow may have value "true" or "false". If jdbc driver supports moving pointer after the last row in result set value is "true" and if it doesn't support, value is "false".
EnableOrderBy may have value "true" or "false". If jdbc driver supports "order by" sql statement in sql commands value is "true" and if it doesn't support, value is "false".
Example 5.1.
<dbVendor name="Csv"> <DriverName value="csvjdbc"/> <ClassName value="org.relique.jdbc.csv.CsvDriver"/> <RequiredUser value="false"/> <FirstColumnResult value="0"/> <EnableJumpInResult value="false"/> <EnablePreviousRecord value="false"/> <BeforeFirstRow value="true"/> <AfterLastRow value="true"/> <EnableOrderBy value="false"/> </dbVendor>
This version of Octopus supports loading data from source table of these types:
Csv
Excel
Access
MSSQL (Microsoft, JTurbo, FreeTds JDBC driver)
MySql
InstantDB
PostgreSQL
Qed
HSQL
Oracle
DB2
BorlandJDataStore
If OctopusDBVendors.xml file doesn't exist, the system default is MSSQL database (parameters for this database).
Table of Contents
In this chapter we will help you to create your own input XML file.
Main tag of the XML file is <loaderJob> tag, first we need to define <loaderJob> attributes:
<loaderJob logMode="full" objectIDIncrement="1" commitCount="1">
Child tag is <jdbcDefaultParameters>. It defines JDBC default source and target parameters. If the database is MSSQL there must be four tags because database requires user and password. Jdbc parameters for Access, Csv and Excel may have two tags, because for these databases user and password isn't required. JdbcDriver tag defines database driver. Connection.Url tag defines connection URL. In the case when database is:
Csv, value attribute is folder URL where the files are;
Excel, value attribute is the name of the system ODBC;
Access, value attribute is the name of the system ODBC;
MSSQL, value attribute is the URL of the database.
<jdbcDefaultParameters> <jdbcSourceParameters> <jdbcSourceParameter name="JdbcDriver" value="org.myDriver.JDBCDriver" /> <jdbcSourceParameter name="Connection.Url" value = "jdbc:myDriver:MySourceDatabase" /> </jdbcSourceParameters> <jdbcTargetParameters> <jdbcTargetParameter name="JdbcDriver" value="org.myDriver2.JDBCDriver"/> <jdbcTargetParameter name="Connection.Url" value = "jdbc:myDriver2:MySourceDatabase2" /> <jdbcTargetParameter name="User" value="sa"/> <jdbcTargetParameter name="Password" value=""/> </jdbcTargetParameters> </jdbcDefaultParameters>
Next child tag of the <loaderJob> tag is <variables> tag. It defines logical variables to be used in constantColumns, SQL-statements, source-data and JDBC parameters.
<variables prefix="!" suffix="!" override="false" replaceInConstants="true" replaceInSQL="false" replaceInData="true" replaceInJDBC="true"> <variable name="var1" value="myVal1" prefix="$" suffix="$" replaceInConstants="false" replaceInSQL="true" replaceInData="false" replaceInJDBC="false"/> <variable name="var2" value="myVal2"/> </variables>
Next child tag of the <loaderJob> tag is <sql> tag. We need to define its name, logMode, if continue on error and if it will be commited.
<sql name="DropDatabase" onErrorContinue="true" commit="true" logMode="full">
Child tag is <jdbcTargetParameters>. It defines JDBC target parameters.
<jdbcTargetParameters> <jdbcTargetParameter name="JdbcDriver" value = "org.myDriver2.JDBCDriver" /> <jdbcTargetParameter name="Connection.Url" value = " jdbc:myDriver2:MySourceDatabase3"/> <jdbcTargetParameter name="User" value="sa"/> <jdbcTargetParameter name="Password" value=""/> </jdbcTargetParameters>
Next child tag is <sqlStmt> tag. It defines sql statement.
<sqlStmt> DROP DATABASE MyDatabase </sqlStmt> </sql>
Next child tag of the <loaderJob> tag is <sql> tag too. We need to create MyDatabase.
<sql name="CreateDatabase" onErrorContinue="false" commit="true" logMode="full"> <jdbcTargetParameters> <jdbcTargetParameter name="JdbcDriver" value="com.ashna.jturbo.driver.Driver"/> <jdbcTargetParameter name="Connection.Url" value="jdbc:JTurbo://localhost:1433/master"/> <jdbcTargetParameter name="User" value="sa"/> <jdbcTargetParameter name="Password" value=""/> </jdbcTargetParameters> <sqlStmt> CREATE DATABASE MyDatabase </sqlStmt> </sql>
Next child tag of the <loaderJob> tag is <importDefinition> tag, source table name is FIRSTOMDSBENUTZER,
<importDefinition name="FIRSTOMDSBENUTZER" tableName="FIRSTOMDSBENUTZER">
If the Jdbc source and target parameters are not defined the default will be used.
Next step is valueColumns tag. It defines which source column to load into which target table and column.
<valueColumns> <valueColumn sourceColumnName="VORNAME" targetTableName="OMDSBENUTZER" targetColumnName="VORNAME" targetTableID="0" valueMode="Overwrite"/> <valueColumn sourceColumnName="NACHNAME" targetTableName="OMDSBENUTZER" targetColumnName="NACHNAME" targetTableID="0" valueMode="Overwrite"/> <valueColumn sourceColumnName="TITEL" targetTableName="OMDSBENUTZER" targetColumnName="TITEL" targetTableID="0" valueMode="Overwrite"/> <valueColumn sourceColumnName="EMAILADRESSE" targetTableName="OMDSBENUTZER" targetColumnName="EMAILADRESSE" targetTableID="0" valueMode="Overwrite"/> <valueColumn sourceColumnName="TELEFONNUMMER" targetTableName="OMDSBENUTZER" targetColumnName="TELEFONNUMMER" targetTableID="0" valueMode="Overwrite"/> </valueColumns>
Next step is variableColumns. This tag defines columns which has the time stamp and user ID.
<variableColumns> <timeStampColumn targetTableName="OMDSBENUTZER" targetColumnName="ERZEUGTAMUM" targetTableID="0" valueMode="SetIfCreated"/> <timeStampColumn targetTableName="OMDSBENUTZER" targetColumnName="GEAENDERTAMUM" targetTableID="0" valueMode="Overwrite"/> <userIDColumn targetTableName="OMDSBENUTZER" targetColumnName="USERID" targetTableID="0" valueMode="Key"/> </variableColumns>
Next step is constantColumns. This tag defines constant columns which has the constant values and defines the value.
<constantColumns> <constantColumn targetTableName="OMDSBENUTZER" targetColumnName="GESCHLECHTOID" targetTableID="0" valueMode="Overwrite" constantValue="2"/> <constantColumn targetTableName="OMDSBENUTZER" targetColumnName="ERZEUGTVONOID" targetTableID="0" valueMode="SetIfCreated" constantValue="1"/> <constantColumn targetTableName="OMDSBENUTZER" targetColumnName="GEAENDERTVONOID" targetTableID="0" valueMode="Overwrite" constantValue="1"/> </constantColumns>
You can add counterColumns for auto increment columns. CounterColumn defines simple auto increment columns with different counterName for all columns in one importDefinition. If you wish to continue using the same counter values in next importDefinition, just give the same name to the counterColumn in next importDefinition.
SubCounterColumn inserts auto increment values in counterColumn depending on values in keyColumns. For each different value of key columns (or combination of keyColumns - in case of more key columns in one subCounterColumn element) Octopus inserts auto increment values beginig from counterStartValue.
<counterColumns counterTableName="counterTable" counterNameColumn="counteName" counterValueColumn="counterValue" > <counterColumn counterName="counter1" counterStartValue="100" counterIncrement="1" targetTableName="OMDSBENUTZER" targetTableID="0" targetColumnName"NUMMER" valueMode="SetIfCreated" counterStartValueReset="true"/> <subCounterColumn constantValue="myVal3" targetTableName="OMDSBENUTZER" targetTableID="0" targetColumnName="SUBNUMMER" valueMode="Overwrite"/> <subCounterKeyColumn keyColumnName="GESCHLECHTOID"/> </subCounterColumn> </counterColumns>
Next step is table tag. It defines target tables:
name;
insert logic - if it is true, data will be inserted, otherwise, the table is going to be only a part of a relation;
table mode - defines Query or Cache mode;
oid logic - if it is true, target table has oid column, otherwise doesn't.
<tables> <table tableName="OMDSBENUTZER" tableID="0" insert="true" tableMode="Query" oidLogic="true"/> </tables>
The end of import definition and loaderJob:
</importDefinition> </loaderJob>
This is the end of simple XML without relation columns. The most complicated tag is relation tag, so it will be discussed more detailed in the next chapter.
If we want to create relations we have to notice a few parameters:
how many target tables, which establish relations, are in loading process;
does the values in relation columns changing during the loading process and
are there any Key column, which are not oid columns, in both tables connected with relations.
Generally speaking in relations we make connections between target tables. Relation tag has source table name, source table column name, target table name and target table column name. Only thing to do is to set this attribute values and relation is done.
Problem is how Octopus will know which row from source table to connect with appropriate row from target table?
The simplest problem is: when there is one source table which is spread on two target tables. XML will have one importDefinition. There will be two table tags and valueColumn tags will be as many as target columns in both target tables are. There will be at least two Key valueMode, for each target table at least one. There will be one relation tag where source and target tables will be tables from the target database. Source and target columns will be appropriate columns in these tables.
Next example is complicated one with different kind of relations. This example is continuing from the example in section "Simple XML File without Relation Columns".
This importDefinition has two relation tags. These relations are between two target tables. First one is between source table OMDSBENUTZER and target table OMDSSTAMMGESCHLECH, value from oid column (where USERID is sort column) from source table is set into ERZEUGTVONOID column in target table. Second is between source table OMDSBENUTZER and target table OMDSSTAMMGESCHLECH, value from oid column (where USERID is sort column) from source table is set into GEAENDERTVONOID column in target table.
Table 6.1.
Table: | OMDSBENUTZER (source) | OMDSSTAMMGESCHLECH (target) |
Columns: | OID (relation1-source, relation2-source) | ERZEUGTVONOID (relation1-target) |
GEAENDERTVONOID (relation2-target) |
<importDefinition name="OMDSSTAMMGESCHLECHT" tableName="OMDSSTAMMGESCHLECHT" logMode="normal" objectIDIncrement="1" objectIDTableName="oidAdminData" objectIDColumnName="myNext"> <valueColumns> <valueColumn sourceColumnName="KEYVALUE" targetTableName="OMDSSTAMMGESCHLECHT" targetColumnName="KEYVALUE" targetTableID="0" valueMode="Key"/> <valueColumn sourceColumnName="KURZBEZ" targetTableName="OMDSSTAMMGESCHLECHT" targetColumnName="KURZBEZ" targetTableID="0" valueMode="Overwrite"/> <valueColumn sourceColumnName="LANGBEZ" targetTableName="OMDSSTAMMGESCHLECHT" targetColumnName="LANGBEZ" targetTableID="0" valueMode="Overwrite"/> </valueColumns> <variableColumns> <timeStampColumn targetTableName="OMDSSTAMMGESCHLECHT" targetColumnName="ERZEUGTAMUM" targetTableID="0" valueMode="SetIfCreated"/> <timeStampColumn targetTableName="OMDSSTAMMGESCHLECHT" targetColumnName="GEAENDERTAMUM" targetTableID="0" valueMode="Overwrite"/> <userIDColumn targetTableName="OMDSBENUTZER" targetColumnName="USERID" targetTableID="0" valueMode="Key"/> <userIDColumn targetTableName="OMDSBENUTZER" targetColumnName="USERID" targetTableID="1" valueMode="Key"/> </variableColumns> <relationColumns> <relationColumn relationSourceTableName="OMDSBENUTZER" relationSourceTableID="0" relationTargetTableName="OMDSSTAMMGESCHLECHT" relationTargetColumnName="ERZEUGTVONOID" relationTargetTableID="0" relationMode="SetIfCreated"/> <relationColumn relationSourceTableName="OMDSBENUTZER" relationSourceTableID="1" relationTargetTableName="OMDSSTAMMGESCHLECHT" relationTargetColumnName="GEAENDERTVONOID" relationTargetTableID="0" relationMode="Overwrite"/> </relationColumns> <tables> <table tableName="OMDSBENUTZER" tableID="0" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="OMDSBENUTZER" tableID="1" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="OMDSSTAMMGESCHLECHT" tableID="0" insert="true" tableMode="Query" oidLogic="true"/> </tables> </importDefinition>
Next importDefinition has three relation tags, difference between the one above is that the relationMode is Key. In first relation tag that means the sort columns in target table PERSISTENTTREETYPE are: KEYVALUE and OID. In second relation tag sort columns in target table PERSISTENTTREETYPE are: KEYVALUE and OID. In third relation tag sort column in target table PERSISTENTTREE is KEYVALUE.
Table 6.2.
Table: | PERSISTENTTREETYPE (relation1-source, relation2-source) | PERSISTENTTREE (relation1-target, relation2-target, relation3-source, target) |
Columns: | OID (relation1-source, relation1-sort, relation2-source, relation2-sort) | OID (relation3-source, relation3-sort) |
KEYVALUE (relation1-sort, relation2-sort) | PARENTTREEOID (relation3-target) | |
PERSISTENTTREETYPEOID (relation1-target, relation2-target) | ||
KEYVALUE (relation3-sort) |
<importDefinition name="PARENTTREE" tableName="PERSISTENTTREE" logMode="normal"> <valueColumns> <valueColumn sourceColumnName="KEYVALUE" targetTableName="PERSISTENTTREE" targetColumnName="KEYVALUE" targetTableID="0" valueMode="Key"/> <valueColumn sourceColumnName="TYPEKEY" targetTableName="PERSISTENTTREETYPE" targetColumnName="KEYVALUE" targetTableID="1" valueMode="Key"/> <valueColumn sourceColumnName="PARENTKEY" targetTableName="PERSISTENTTREE" targetColumnName="KEYVALUE" targetTableID="2" valueMode="Key"/> <valueColumn sourceColumnName="PARENTTYPEKEY" targetTableName="PERSISTENTTREETYPE" targetColumnName="KEYVALUE" targetTableID="3" valueMode="Key"/> </valueColumns> <relationColumns> <relationColumn relationSourceTableName="PERSISTENTTREETYPE" relationSourceTableID="3" relationTargetTableName="PERSISTENTTREE" relationTargetColumnName="PERSISTENTTREETYPEOID" relationTargetTableID="2" relationMode="Key"/> <relationColumn relationSourceTableName="PERSISTENTTREETYPE" relationSourceTableID="1" relationTargetTableName="PERSISTENTTREE" relationTargetColumnName="PERSISTENTTREETYPEOID" relationTargetTableID="0" relationMode="Key"/> <relationColumn relationSourceTableName="PERSISTENTTREE" relationSourceTableID="2" relationTargetTableName="PERSISTENTTREE" relationTargetColumnName="PARENTTREEOID" relationTargetTableID="0" relationMode="Overwrite"/> </relationColumns> <tables> <table tableName="PERSISTENTTREETYPE" tableID="1" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="PERSISTENTTREE" tableID="0" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="PERSISTENTTREETYPE" tableID="3" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="PERSISTENTTREE" tableID="2" insert="false" tableMode="Cache" oidLogic="true"/> </tables> </importDefinition>
Next two importDefinitions are example of deep relations. In this cases first must be done first level of relation than the second one. First importDefinition has one relation. Source table is GENERICENUMTYPEDEFINITION source column is OID target table is GENERICENUMVALUEDEFINITION target column name is ENUMTYPEDEFINITIONOID and sort column is KEYVALUE. Second importDefinition has for relations:
Table 6.3.
Table: | GENERICENUMTYPEDEFINITION (relation1-source, relation2-source) | GENERICENUMVALUEDEFINITION (relation1-target, relation2-target, relation3-source, relation4-source) | GENERICSTATETRANSITION (relation3-target, relation4-target) |
Column: | OID (relation1-source, relation1-sort, relation2-source, relation2-sort) | OID (relation3-source, relation3-sort, relation4-source, relation4-sort) | ENUMVALUEDEFINITIONFROMOID (relation3-target, relation4-target) |
KEYVALUE (relation1-sort, relation2-sort) | ENUMTYPEDEFINITIONOID (relation1-target, relation2-target) | ||
KEYVALUE (relation3-sort, relation4-sort) |
<importDefinition name="GENERICENUMVALUEDEFINITION" tableName="GENERICENUMVALUEDEFINITION" logMode="normal" objectIDIncrement="1" objectIDTableName="oidAdminData" objectIDColumnName="myNext"> <valueColumns> <valueColumn sourceColumnName="ENUMTYPEDEFINITIONKEY" targetTableName="GENERICENUMTYPEDEFINITION" targetColumnName="KEYVALUE" targetTableID="0" valueMode="Key"/> <valueColumn sourceColumnName="BEGINSTATE" targetTableName="GENERICENUMVALUEDEFINITION" targetColumnName="BEGINSTATE" targetTableID="0" valueMode="Overwrite"/> <valueColumn sourceColumnName="ENDSTATE" targetTableName="GENERICENUMVALUEDEFINITION" targetColumnName="ENDSTATE" targetTableID="0" valueMode="Overwrite"/> <valueColumn sourceColumnName="LONGNAME" targetTableName="GENERICENUMVALUEDEFINITION" targetColumnName="LONGNAME" targetTableID="0" valueMode="Overwrite"/> <valueColumn sourceColumnName="SHORTNAME" targetTableName="GENERICENUMVALUEDEFINITION" targetColumnName="SHORTNAME" targetTableID="0" valueMode="Overwrite"/> <valueColumn sourceColumnName="KEYVALUE" targetTableName="GENERICENUMVALUEDEFINITION" targetColumnName="KEYVALUE" targetTableID="0" valueMode="Key"/> <valueColumn sourceColumnName="XMLNAME" targetTableName="GENERICENUMVALUEDEFINITION" targetColumnName="XMLNAME" targetTableID="0" valueMode="Overwrite"/> </valueColumns> <relationColumns> <relationColumn relationSourceTableName="GENERICENUMTYPEDEFINITION" relationSourceTableID="0" relationTargetTableName="GENERICENUMVALUEDEFINITION" relationTargetColumnName="ENUMTYPEDEFINITIONOID" relationTargetTableID="0" relationMode="Key"/> </relationColumns> <tables> <table tableName="GENERICENUMTYPEDEFINITION" tableID="0" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="GENERICENUMVALUEDEFINITION" tableID="0" insert="true" tableMode="Cache" oidLogic="true"/> </tables> </importDefinition> <importDefinition name="GENERICSTATETRANSITION" tableName="GENERICSTATETRANSITION" logMode="normal" objectIDIncrement="1" objectIDTableName="oidAdminData" objectIDColumnName="myNext"> <valueColumns> <valueColumn sourceColumnName="ENUMTYPEDEFINITIONKEY" targetTableName="GENERICENUMTYPEDEFINITION" targetColumnName="KEYVALUE" targetTableID="0" valueMode="Key"/> <valueColumn sourceColumnName="ENUMTYPEDEFINITIONKEY" targetTableName="GENERICENUMTYPEDEFINITION" targetColumnName="KEYVALUE" targetTableID="1" valueMode="Key"/> <valueColumn sourceColumnName="ENUMVALUEDEFINITIONFROMKEY" targetTableName="GENERICENUMVALUEDEFINITION" targetColumnName="KEYVALUE" targetTableID="0" valueMode="Key"/> <valueColumn sourceColumnName="ENUMVALUEDEFINITIONTOKEY" targetTableName="GENERICENUMVALUEDEFINITION" targetColumnName="KEYVALUE" targetTableID="1" valueMode="Key"/> </valueColumns> <relationColumns> <relationColumn relationSourceTableName="GENERICENUMTYPEDEFINITION" relationSourceTableID="0" relationTargetTableName="GENERICENUMVALUEDEFINITION" relationTargetColumnName="ENUMTYPEDEFINITIONOID" relationTargetTableID="0" relationMode="Key"/> <relationColumn relationSourceTableName="GENERICENUMTYPEDEFINITION" relationSourceTableID="1" relationTargetTableName="GENERICENUMVALUEDEFINITION" relationTargetColumnName="ENUMTYPEDEFINITIONOID" relationTargetTableID="1" relationMode="Key"/> <relationColumn relationSourceTableName="GENERICENUMVALUEDEFINITION" relationSourceTableID="0" relationTargetTableName="GENERICSTATETRANSITION" relationTargetColumnName="ENUMVALUEDEFINITIONFROMOID" relationTargetTableID="0" relationMode="Key"/> <relationColumn relationSourceTableName="GENERICENUMVALUEDEFINITION" relationSourceTableID="1" relationTargetTableName="GENERICSTATETRANSITION" relationTargetColumnName="ENUMVALUEDEFINITIONTOOID" relationTargetTableID="0" relationMode="Key"/> </relationColumns> <tables> <table tableName="GENERICENUMTYPEDEFINITION" tableID="0" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="GENERICENUMTYPEDEFINITION" tableID="1" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="GENERICENUMVALUEDEFINITION" tableID="0" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="GENERICENUMVALUEDEFINITION" tableID="1" insert="false" tableMode="Cache" oidLogic="true"/> <table tableName="GENERICSTATETRANSITION" tableID="0" insert="true" tableMode="Cache" oidLogic="true"/> </tables> </importDefinition> </loaderJob>
Table of Contents
Tables used by TOS have global unique OID logic as a DODS tables. TOS tables uses OID column (VARCHAR type) as a primary key column. Also TOS creates Updt column (timestamp), LID, MKeys, VKeys, DKeys columns. OID column is unique for each table.
Difference between TOS and DODS oid logic are:
type of OID column (decimal vs varchar)
in TOS table there are no version column
objectID table has two columns: one for name of OID counter, other for its value
In Octopus you can load data into TOS table if objectIDNameColumnName and objectIDNameColumnValue attributes are set in loadJob element (default values). If these attributes are set in loadJob configuration file Octopus automatically uses TOS tables format.
Also attribute objectIDAutoCreate sets Octopus to auto create objectID table or not.
Example 7.1. Example: Setting Octopus attributes to work with TOS tables
<loaderJob logMode="normal" objectIDIncrement="1" objectIDTableName="objectid" objectIDColumnName="next" objectIDNameColumnName="Column1" objectIDNameColumnValue="test"/>
In importDefinition you can create variableColumn to fill Updt column (update time).
<variableColumns> <timeStampColumn targetTableName="TargetTable1" targetColumnName="Updt" targetTableID="0" valueMode="Overwrite"/> </variableColumns>
Table of Contents
To start example Octopus application LoadOMDSProject:
copy your all files in Octopus directory into your hard drive
install MSSQL server (version 7 or 2000)
start MSSQL server
if your MSSQL server is not installd on your local computer you have to change path in freetds drivers (in xml files in GRECOOMDSBROWSER\ObjectLoader directory). (e.g. jdbc:freetds:sqlserver://localhost:1433/Together into jdbc:freetds:sqlserver://MSSQL_host:MSSQL_port/master)
start LoadOMDSProject.bat
LoadOMDSProject loads data from csv files (situated in GRECOOMDSBROWSER/ObjectLoader/LoaderInput/Browser/daten directory) using csv driver (csvjdbc.jar) into MSSQL database (Together) using freetds jdbc driver. Rules for loading data is written in xml files (in ObjectLoader directory). Main xml file is LoadOMDSProject.xml file. Also, test example creates tables in new database using sql files (situated in ObjectLoader/includes directory).
Examples:
LoadExample2: This example shows using variables in configuration xml files to substitute values in source tables.
LoadExample3: This example shows using SQL statements instead source tables.
LoadExample4: This example shows using csv jdbc driver for loading data into csv target tables.
ExampleCounters: This example shows using counterColumns for simple auto increment columns.
To start example Octopus application LoadOMDSProject using ant task please start:
start_using_ant.cmd
Ant uses build.xml file in the same directory to execute Octopus.
There are three classes to demonstrate how to use Octopus as a test case (for manipulating with databases). To start these examples just start LoadTest with one of the following arguments:
LoaderTest LoaderTest2 LoaderTest3
For running these examples, you need to set junit.jar into you classpath. (JUnit version 3.8 is placed in lib directory of Octopus).
Table of Contents
This is a sneak peek of a free type 4 JDBC driver for SQLServer and Sybase ASE. Currently it has only been tested mostly on SQLServer 7.0 and SQLServer 2000.
The name of the driver class for using in the Class.forName() is
com.internetcds.jdbc.tds.Driver
and it accepts any of the following forms of URLs:
jdbc:freetds:sqlserver://servername/database jdbc:freetds:sqlserver://servername:port/database jdbc:freetds:sybase://servername/database jdbc:freetds:sybase://servername:port/database jdbc:freetds://servername/database jdbc:freetds://servername:port/database
(These last two forms are obsolete and should not be used.)
The driver can also recognize several properties in the URL. The properties include:
Table 9.1. Connection URL properties
CHARSET | Character set to use. Defaults to iso_1. |
user | user name that will be used to connect to the database. |
password | password used to connect to the database. |
APPNAME | Application name to tell to the database. Defaults to "jdbclib". |
PROGNAME | Program name to tell to the database. Defaults to "java_app". |
TDS | TDS protocol version to use. Can be "4.2", "5.0", or "7.0". Defaults to "7.0". |
e.g. "jdbc:freetds:sqlserver://servername/database;CHARSET=cp950;TDS=7.0"
The class com.internetcds.jdbc.tds.SybaseDriver is no longer needed and has been retained only for backward compatibility. It should not be used for new code.
Right now unless you specify sybase it defaults to port 1433 for SQLServer and port 7100 for Sybase. Generally you will need to specify the port number for Sybase, but not for SQLServer.
Table of Contents
The complete connection URL format used with the driver manager and driver class are:
com.microsoft.jdbc.sqlserver.SQLServerDriver jdbc:microsoft:sqlserver://hostname:port[;property=value...]
examples:
jdbc:microsoft:sqlserver://servername;DatabaseName="YourDatabase" jdbc:microsoft:sqlserver://servername:port;DatabaseName="YourDatabase"
If you don't specify database name, driver connects to master database.
The driver can also recognize several properties in the URL. The properties include:
hostname | is the TCP/IP address or TCP/IP host name of the server to which you are connecting. NOTE: Untrusted applets cannot open a socket to a machine other than the originating host. |
port | is the number of the TCP/IP port. |
property=value | specifies connection properties. See "Connection String Properties" for a list of connection properties and their values. |
The following example shows a typical connection URL:
jdbc:microsoft:sqlserver://server1:1433;user=test;password=secret
You can use the following connection properties with the JDBC driver manager or SQL Server 2000 Driver for JDBC data sources.
Table 3-1 lists the JDBC connection properties supported by the SQL Server driver, and describes each property. The properties have the form:
property=value
NOTE: All connection string property names are case-insensitive. For example, PortNumber is the same as portnumber.
Table 10.1. SQL Server Connection String Properties
Property | Description |
---|---|
DatabaseName OPTIONAL | The name of the SQL Server database to which you want to connect. |
HostProcess OPTIONAL | The process ID of the application connecting to SQL Server 2000. The supplied value appears in the "hostprocess" column of the sysprocesses table. |
NetAddress OPTIONAL | The MAC address of the network interface card of the application connecting to SQL Server 2000. The supplied value appears in the "net_address" column of the sysprocesses table. |
Password | The case-insensitive password used to connect to your SQL Server database. |
PortNumber OPTIONAL | The TCP port (use for DataSource connections only). The default is 1433. |
ProgramName OPTIONAL | The name of the application connecting to SQL Server 2000. The supplied value appears in the "program_name" column of the sysprocesses table. |
SelectMethod | SelectMethod={cursor | direct}. Determines whether
database cursors are used for Select statements. Performance and behavior of the driver are
affected by the SelectMethod setting. Direct-The direct method sends the complete result set
in one request to the driver. It is useful for queries that only produce a small amount of data
that you fetch completely. You should avoid using direct when executing queries that produce
a large amount of data, as the result set is cached completely on the client and constrains
memory. In this mode, each statement requires its own connection to the database. This is
accomplished by "cloning" connections. Cloned connections use the same connection properties
as the original connection; however, because transactions must occur on a single connection,
auto commit mode is required. Due to this, JTA is not supported in direct mode. In addition,
some operations, such as updating an insensitive result set, are not supported in direct mode
because the driver must create a second statement internally. Exceptions generated due to the
creation of cloned statements usually return an error message similar to "Cannot start a cloned
connection while in manual transaction mode."Cursor-When the SelectMethod is set to cursor, a server-side cursor is generated. The rows are fetched from the server in blocks. The JDBC Statement method setFetchSize can be used to control the number of rows that are fetched per request. The cursor method is useful for queries that produce a large amount of data, data that is too large to cache on the client. Performance tests show that the value of setFetchSize has a serious impact on performance when SelectMethod is set to cursor. There is no simple rule for determining the value that you should use. You should experiment with different setFetchSize values to find out which value gives the best performance for your application. The default is direct. Octopus supports cursor method. If you don't set SelectMethod to cursor Octopus automatically change this property. |
SendStringParameters AsUnicode | SendStringParametersAsUnicode={true | false}. Determines whether string parameters are sent to the SQL Server database in Unicode or in the default character encoding of the database. True means that string parameters are sent to SQL Server in Unicode. False means that they are sent in the default encoding, which can improve performance because the server does not need to convert Unicode characters to the default encoding. You should, however, use default encoding only if the parameter string data that you specify is consistent with the default encoding of the database. The default is true. |
ServerName | The IP address (use for DataSource connections only). |
User | The case-insensitive user name used to connect to your SQL Server database. |
Table of Contents
This is a JDBC driver for SQLServer 7.0 and SQLServer 2000.
The name of the driver class for using in the Class.forName() is
com.ashna.jturbo.driver.Driver
and it accepts any of the following forms of URLs:
jdbc:JTurbo:sqlserver://servername:port/database jdbc:JTurbo:sqlserver://servername:port/database/parameter=value
The driver can also recognize several properties in the URL. The properties include:
Table 11.1. Connection URL properties
user | user name that will be used to connect to the database. |
password | password used to connect to the database. |
sql70 | To support SQL Server 7. Can be "true" or "false". Default is "false". If use sql70=true parameter, you have to uncomment setings for JTurbo driver (sql70=true parameter) in OctopusDBvendors.xml file. |
e.g. "jdbc:JTurbo:sqlserver://servername/database/sql70=true"