Automate Open Query in MS SQL Server

This Table and set of Stored Procedures will enable you to automate the transfer of data from Oracle and Progress databases into SQL Server.

Creating The Get Open Query Modules

We are assuming that you have the knowledge and permission to create tables, stored procedures, Linked Server objects and jobs, without this then there may be issues that we cannot help you with.

When running this, we tend to place it in a Utilities type database where we keep shared functions and maintenance objects.

First task we have is to create a master table to manage all of the data flow, a few of the Columns to keep an eye out for include;

GetOpenQueryStream - This is an integer field that allows you to batch process sets of tables, and will run in the order set in the column GetOpenQueryOrder
GetOpenQueryName - This should be either GetOpenQuery_Progress or GetOpenQuery_Oracle unless you have adapted the code.
GetOpenQueryLinkServ - Is the linked server name, which needs to be set up in Server Objects
GetOpenQuerySourceSchema - Is the source database schema
GetOpenQueryDatabase, GetOpenQuerySchema, GetOpenQueryTable form the three part identifier for where the data is set to be copied to.
You can add in WHERE, INNER and TOP statements
GetOpenQuerySkipCols - this will remove the columns from the table when importing.

Create Table

CREATE TABLE dbo.GetOpenQuery(
GetOpenQueryID int IDENTITY(1,1) NOT NULL,GetOpenQueryStream int NULL,GetOpenQueryName nvarchar(100) NULL,GetOpenQueryLinkServ nvarchar(100) NULL,GetOpenQueryDatabase nvarchar(100) NULL,GetOpenQuerySchema nvarchar(max) NULL,GetOpenQueryTable nvarchar(100) NULL,GetOpenQueryWHERE nvarchar(1000) NULL,GetOpenQueryTOP nvarchar(100) NULL,GetOpenQuerySourceSchema nvarchar(100) NULL,GetOpenQuerySkipCols nvarchar(max) NULL,GetOpenQueryINNER nvarchar(max) NULL,GetOpenQuerySkipTruncate bit NULL DEFAULT ((0)),GetOpenQueryOrder bit NULL DEFAULT ((0)))

A hub

Jobs can then be hooked to pass in a Stream ID and loop through the relevant tables via this stored Procedure. This loops through the table above, and fires off the relevant code for either Oracle or Progress linked server.

This then becomes the hub by which all other jobs can be called.

Create Stored Procedure

CREATE PROC [dbo].[GetOpenQuery_Data](@Stream INT) AS BEGINDECLARE @GetOpenQueryName NVARCHAR(100),@GetOpenQueryLinkServ NVARCHAR(100),@GetOpenQueryDatabase NVARCHAR(100),@GetOpenQuerySchema NVARCHAR(MAX),@GetOpenQueryTable NVARCHAR(100),@GetOpenQueryWHERE NVARCHAR(1000),@GetOpenQueryTOP NVARCHAR(100),@SrcScheme NVARCHAR(100),@SkipCols NVARCHAR(MAX),@GetOpenQueryINNER NVARCHAR(MAX),@GetOpenQuerySkipTruncate BIT DECLARE @SQL NVARCHAR(MAX),@SQLRows BIGINT,@SQLRowCount BIGINT,@SQLOutPut NVARCHAR(100)='@SQLRows BIGINT' DECLARE C CURSOR FAST_FORWARD FORSELECTGetOpenQueryName, GetOpenQueryLinkServ, GetOpenQueryDatabase, GetOpenQuerySchema, GetOpenQueryTable,GetOpenQueryWHERE, GetOpenQueryTOP, GetOpenQuerySourceSchema, GetOpenQuerySkipCols, GetOpenQueryINNER, GetOpenQuerySkipTruncateFROM GetOpenQueryWHERE GetOpenQueryStream=@StreamORDER BY GetOpenQueryOrderOPEN CFETCH NEXT FROM CINTO @GetOpenQueryName,@GetOpenQueryLinkServ,@GetOpenQueryDatabase,@GetOpenQuerySchema,@GetOpenQueryTable,@GetOpenQueryWHERE,@GetOpenQueryTOP,@SrcScheme,@SkipCols,@GetOpenQueryINNER,@GetOpenQuerySkipTruncateWHILE @@FETCH_STATUS = 0 BEGIN       BEGIN TRY       SET @SQL='EXEC '+@GetOpenQueryName+' '''+       @GetOpenQueryLinkServ+''','''+       @GetOpenQueryDatabase+''','''+       @GetOpenQuerySchema+''','''+       @GetOpenQueryTable+''','+       ISNULL(NULLIF(''''+@GetOpenQueryWHERE+'''',''),'NULL')+','+       ISNULL(NULLIF(@GetOpenQueryTOP,''),'NULL')+','+       ISNULL(NULLIF(@SrcScheme,''),'NULL')+','+       ISNULL(NULLIF(''''+@SkipCols+'''',''),'NULL')+','+       ISNULL(NULLIF(''''+@GetOpenQueryINNER+'''',''),'NULL')+','+(CASE WHEN @GetOpenQuerySkipTruncate=1 THEN '1' ELSE '0' END)       EXEC sp_executesql @SQL       SET @SQLRows=@@ROWCOUNT       SELECT @SQLRows       END TRY       BEGIN CATCH        PRINT @GetOpenQueryLinkServ+'; Table '+@GetOpenQueryTable+' Failed'       END CATCH       PRINT @SQL       FETCH NEXT FROM C       INTO @GetOpenQueryName,@GetOpenQueryLinkServ,@GetOpenQueryDatabase,@GetOpenQuerySchema,@GetOpenQueryTable,       @GetOpenQueryWHERE,@GetOpenQueryTOP,@SrcScheme,@SkipCols,@GetOpenQueryINNER,@GetOpenQuerySkipTruncateEND CLOSE C;DEALLOCATE C;ENDGO

How to call the jobs

Now that these steps have been created, we can plumb in the remaining SPROC's to handle both Oracle and Progress.

To fire off Stream 1, create an SQL Agent Job with the following code;

EXEC Utilities.dbo.GetOpenQuery_Data 1