The Setup

• Windows 7 64-bit
• Visual Studio 2008
• SQL Server 2008 64-bit

The Process

1. Create a signing key (See also, How to: Create a Public/Private Key Pair)
   1. Open Visual Studio 2008 command Prompt – the regular command prompt will not work
   2. Change to a friendly directory: cd %userprofile%\Desktop
   3. Create the key file: sn -k key.snk
2. Sign the assembly – There are a few ways to do this, but I found this to be the easiest. If you want to sign it some other way, check out How to: Sign an Assembly with a Strong Name
   1. Right-click the Project
   2. Select “Properties”
   3. Navigate to the “Signing” tab
   4. Browse to strong name key file (which was created in the previous step)
   5. Recompile the project
3. Copy the re-compiled assembly to your GAC
   1. gacutil -i “C:\Path\to\CustomAssemblyName.dll”
4. Copy assembly to “%programfiles(x86)%\Microsoft SQL Server\100\SDK\Assemblies”
5. Add reference in script task. Repeat this for every Script Task you want to access this assembly from
   1. Right-click References
   2. Click “Add Reference…”
      

      Add Reference...

   3. On the .NET tab, scroll to find your assembly
   4. Press “OK”
   5. The Assembly should now appear under the References list
6. Add a reference to the assembly in code, at the top
   1. (C#) Using CustomAssemblyName;
   2. (VB.NET) Imports CustomAssemblyName
7. You should now have full access to the imported DLL

Caveats

This method works pretty well, but deployment isn’t exactly seamless – you’ll have to repeat this for each server and re-register & copy the DLL separately for any updates. Additionally, there is no way to globally add the assembly reference to the entire project or package. Instead, you’ll have to repeat step 6 (adding the reference) for every Script Task.

1. Download and install MySQL Connector/NET 6.0
2. Start a new Integration Services project in BIDS
3. Right-click in Connection Managers and create a new ADO.NET Connection

   

4. In the Provider dropdown, expand .Net Providers and select MySQL Data Provider. Press "OK"

   

5. Fill out the Server name, User name, Password and select the database name for the target MySQL server. Be sure to test the connection and press “OK”

   

6. Rename the connection to “MySQLDB”

   

7. Open up the Toolbox and drag a Data Flow Task from the toolbox onto the Control Flow surface
   

   Add Dataflow Task

8. Double-click the Data Flow Task to switch to the Data Flow view
9. Create a new variable, “MySQLResult” with the Data Type of Object. We will be using this as the final destination for the data, so we don’t need to connect to a file or database to store the data from this test
   

   MySQLResult Variable

10. Drag a new DataReader Source component onto the Data Flow surface
    

    Add DataReader Source

11. Double-click the DataReader Source to open the Advanced Editor. On the Connection Managers tab, select the previously-created MySQLDB connection
    

    DataReader Source Connection Managers

12. Switch to the Component Properties tab and enter the SQL query in the SqlCommand property. Note that the query must be compatible with MySQL syntax, not SQL Server.
    

    DataReader Source Component Properties

13. Switch to the Column Mappings tab to verify that the query is successful and the all the columns were pulled from the database. When done, press “OK”.
    

    DataReader Source Column Mappings

14. Create a new Recordset Destination by dragging it from the toolbox to the Data Flow surface
    

    Add Recordset Destination

15. Drag the green Data Flow Path from DataReader Source to Recordset Destination, so they connect
    

    Connect Source to Destination

16. Double-click the Recordset Destination to open its Advanced Editor
17. Under Custom Properties, select the dropdown for VariableName and select the variable we created before, User::MySQLResult
    

    Recordset Destination Component Properties

18. Switch to the Input Columns tab and select those columns that you want stored in the Recordset Destination. When complete, click “OK”
    

    Recordset Destination Input Columns

19. Right-click the green Data Flow Path and choose “Data Viewers…”
    

    Data Flow Path Data Viewers

20. Select “Data Viewers” from the left pane and click the “Add…” button
    

    Data Flow Path Editor

21. Under the General tab, select Grid and press “OK”
    

    Configure Data Viewer

22. Run the package
23. If you’ve done everything correctly, you should see a Data Reader Output Data Viewer window pop up with the contents of the query we specified earlier.
    

    Data Viewer Output

SQL Server Integration Services makes connecting to other systems very easy. The MySQL ADO.NET provider works well, but requires more configuration than a native Source component.

Ingredients

I’ve created an SSIS package that will import the articles and pages into a SQL Server 2005 database. To do this, you’ll first need to gather a few files:

• Latest Wikipeda pages/articles dump (Download enwiki-latest-pages-articles.xml.bz2, approximately 4.1GB at time of writing)
• MediaWiki XSD (originally located on Manual:XML Import file manipulation in CSharp)
• SSIS Package & Database Scripts
  

You’ll need approximately 75GB of free space – 50GB for the database and 20GB for the XML file. Also, this will likely take several hours, if not longer. If you have separate drive spindles, it would certainly help to separate the XML and database files. My example uses C:\wikipedia\ as the working folder; if you prefer another location, we’ll configure it later. If not, this is what the structure should look like:

Directory Listing

Recipe

Note: If you’re planning on importing the XML file into a remote server, I highly recommend performing all these operations on the server itself through Remote Desktop. Aside from having to re-transfer the gigantic XML dump, debugging SSIS packages is much easier when working locally.

1. Connect to the SQL Server database and run WikipediaImport\WikipediaImport\DatabaseCreate.sql
   1. This creates the database (cleverly named “Wikipedia”) on C:\wikipedia\. If you want the data and log files located elsewhere, find 50GB of free space and update the CREATE DATABASE statement.
   2. Because we know the database is going to grow immediately, I’ve told the script to allocation 40GB for data and 10GB for log, so this step may take a while to run.
2. Open WikipediaImport\WikipediaImport.sln in Visual Studio 2005
3. Enable the Variables window if it is not already visible
   1. Select Data Flow
   2. Select the View menu
   3. Select Other Windows
   4. Select Variables
      

      Enable Variables Menu

4. If the working files were placed somewhere besides C:\wikipedia\, you can configure that in the Variables window. Be sure to update both PageArticlesXML and PageArticlesXSD
   

   Set Variable Values

5. Additionally, if you’re not importing to localhost, configure the database connection variable (named DatabaseConnection)
6. Verify there are no warnings or errors and build the solution (Ctrl + Shift + B or Build?Build WikipediaImport)
7. If the build succeeds, go ahead and run it (F5 or Debug?Start Debugging)
8. If all goes well, the XML file should now be streaming into the database. This will likely take hours, even with a fast RAID. Notice that the file only requires a single pass, rather than scanning it once per table.
   

   Import Progress

9. Switch back to SSMS and run WikipediaImport\WikipediaImport\IndexCreate.sql
   1. This step is technically optional, but is going to help speed up your queries significantly
   2. If we had created the indexes before the import, the import would have been even slower
   3. This will take a while!
10. Run a test query

    SELECT * FROM dbo.vw_Articles WHERE title = 'Green Day'

Behind the Scenes

Getting this to work took a while of tweaking, but there are a few highlights I’d like to point out.

Data Types

XSD: The provided XML Schema Definition file does not contain any information about the intended length of the string data. Fortunately, through testing, I was able to shrink some of those sizes down, although they do not strictly conform to the official database schema. Specifically, I have shrunk page.restrictions and text.space from nvarchar(255) to nvarchar(50). Most other items conform as close as possible. In addition to these, I had to update text.text to a nvarchar(max). SSIS initially suggested an nvarchar(255), but articles are obviously much longer than this. To perform these changes, right-click the XML Source (named PageArticles) in the Import XML Data Flow Task and select “Show Advanced Editor…”

Advanced Editor Menu

Expand out each of the changed columns (both External and Output) and update the DataType. For example, SQL Server-specifc nvarchar(max) is a more general “Unicode text stream [DT_NTEXT]” in SSIS. For the others, just update the length from 255 to 50.

Set Data Types

Extraneous information

There are many more fields in the XML file than I’ve decided to import. Unfortunately, I can’t just turn them off completely, lest SSIS complains. Instead I have chosen to suffer the lesser fate of “Warning”. Additionally, I changed the Error Output to “Ignore failure” on Error. This screen can be accessed by double-clicking the XML Source, PageArticles, then selecting the Error Output page.

Configure Error Output

Wrap-up

SSIS is very picky about metadata, making this a somewhat difficult project to get running, however, it is actually running. This could be further extended with an automated download and increased amount of data imported, but for now it serves its purpose.

I know not everyone will get this on first run, so if you have a problem, please leave a comment below and I’ll do my best to answer them in a timely manner.

1. Ensure a string variable is setup with the password. For this demo, I’m using the name, “FTPPassword”
2. Add a Script Task to your package
3. Edit the task
4. On the Script page, click the elipsis for ReadOnlyVariables and check the box for User::FTPPassword.
5. Click the “Edit Script…” button
6. Change your entry point (Main, by default) to look like the below code. Save, close, and hit OK.

1
2
3
4
5
6
7
8

public void Main()
{
	ConnectionManager FTPConn;
	FTPConn = Dts.Connections["FTPServer"];
	FTPConn.Properties["ServerPassword"].SetValue(FTPConn, Dts.Variables["FTPPassword"].Value);
 
	Dts.TaskResult = (int)ScriptResults.Success;
}

A couple notes:

• Ensure you update lines 4-5 to reflect the actual connection name. My example uses the name FTPServer.
• Just to reiterate, my password is stored in the variable name FTPPassword. If yours is different make this change on line 5.

That’s it. Make sure you’ve got this task being executed before your actual FTP task and everything should work fine. Cheers!

"12036","Company Name, Inc.","555-555-1234","3.14159"

That’s fine, but what happens when the field also contains quotes, such as this:

"52665","Best "Kept" Secret Storage Facility","555-555-9876","2.71828"

Note: This post originally assumed you were working with SQL Server 2008 because the included code was only provided in C#. The post has been updated to include a VB .NET code sample as well.

Now, depending on how the parser interprets this line, it could see the double-quote before  “Kept” and expect a field terminator (a comma in our case). Well, it so happens, that for whatever reason, Integration Services exhibits this type of behavior and will fail on this type of line. It’s a well-documented bug and oft-requested fix, but even as of SQL Server 2008, the issue is still present (Note: This apparently should be a supported configuration according to RFC 4180). Ideally, you can simply import a different format – using a different text qualifier is probably the easiest change (ever tried the thorn?). If this is not feasible, a variety of other solutions have been suggested, the most flexible being, writing your own Script Transformation to custom-parse the rows.

The setup is simple: configure the Flat File Source to import the whole record as a single field. Pipe that output to your script component, and then connect that to the destination database or rest of your process. Here’s what my Data Flow Task looks like:

Data Flow Task Overview

And here are the settings I used for my Flat File Connection:

• General
  • Text qualifier: <none>
  • Header row delimiter: {CR}{LF}
• Columns
  • Row delimiter: {CR}{LF}
  • Column delimiter: [blank]
• Advanced
  • Name your column. My example user the unimaginative name, “line”.
  • DataType: string [DT_STR]
  • OutputColumnWidth: 8000

And the steps to get started:

1. Go ahead and add a Flat File Source to your Data Flow Task and configure it to use this Flat File Connection
2. Now, place a Script Component on your Data Flow and select “Transformation” when prompted
3. Drag the green output arrow from Flat File Source to Script Component
4. Edit the Script Component and switch over to the “Inputs and Outputs” page
5. Rename the output to “Match Rows”, or anything else of your choosing.
6. Set ExclusionGroup = 1
7. Add the correct output columns, with necessary names. If you set the types to string [DT_STR], you can convert them in a data convesion transformation later. Or, if you prefer to set the correct types now, you’ll need to case them specifically in the script component. Add an extra column for the line number with type of “four-byte signed integer [DT_I4]“. I called mine MatchLineNum.
8. Add another output and call it “Error Rows” or anything else of your choosing.
9. Set ExclusionGroup = 1
10. Set “Synchronous InputID” to your input. In my case, it is named “Input 0″
11. Add only two columns, ErrorLine (string [DT_STR] 8000) and ErrorLineNum (four-byte signed integer [DT_I4]).

Now, a quick explanation on what all this was for: Because we are only importing the whole row, we want to configure the Script Component to output the individual columns. We did this part when we completed steps 5-7. Next, we configured an additional output for rows that we are not able to successfully parse. This is why we defined the Error Rows output in steps 8-10. Be aware, that if you’re supremely confident that you can parse every row, you could optionally skip the creation of the Error Rows output, but I would advise against it.

Now, we will create the actual script to parse the rows manually.

1. Switch back to the Script page and click the “Edit Script…” button. This will open up a script editor that looks like a stripped-down Visual Studio.
2. Take note of the code below, and I will explain it below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

/* Microsoft SQL Server Integration Services Script Component
*  Write scripts using Microsoft Visual C# 2008.
*  ScriptMain is the entry point class of the script.*/
 
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
 
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    //Declare LineNum to keep track of the number of rows
    int LineNum;
 
    public override void PreExecute()
    {
        base.PreExecute();
 
        //Initalize LineNum
        LineNum = 0;
    }
 
    public override void PostExecute()
    {
        base.PostExecute();
    }
 
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
        //Increment LineNum  to keep track of the number of rows
        LineNum += 1;
 
        //split the input column. This will be highly dependent on the format and will probably need adjustment
        string[] columns = System.Text.RegularExpressions.Regex.Split(Row.line, "(?<="),(?=")");
        //If the number of elements is not expected, we assume there was a problem
        if (columns.Length != 4)
        {
            Row.ErrorLine = Row.line;
            Row.ErrorLineNum = LineNum;
            Row.DirectRowToErrorRows();
        }
        //Everything looks good, let's move forward
        else
        {
            Row.MatchLineNum = LineNum;
            Row.ID = StripQualifier(columns[0], """);
            Row.Company = StripQualifier(columns[1], """);
            Row.PhoneNumber = StripQualifier(columns[2], """);
            Row.FavoriteIrrationalNumber = StripQualifier(columns[3], """);
            Row.DirectRowToMatchRows();
        }
 
    }
 
    public static string StripQualifier(string InputString, string Qualifier)
    {
        //This is a helper function only to remove surrounding text qualifiers
 
        string OutputString;
 
        if(
            InputString.Substring(0, Qualifier.Length) == Qualifier
            && InputString.Substring(InputString.Length - Qualifier.Length, Qualifier.Length) == Qualifier
          )
            OutputString = InputString.Substring(Qualifier.Length, InputString.Length - (2 * Qualifier.Length));
        else
            OutputString = InputString;
 
        return OutputString;
    }
 
}

• Line 14: Declare LineNum for counting our position in the file.
• Line 21: Initialize LineNum to 0. In the above script, we increment before parsing, so the first record will have a LineNum of 1. If you happen to have a header row and would prefer it to be 2 to coincide with the actual lines in your flat file, feel free to change the initalization to -1.
• Line 31: Increment LineNum for keeping track of our position in the file
• Line 35: This is the real nugget to the whole post. It utilizes regular expressions to correctly parse out the row, even with an embedded text qualifier. This is a  fairly simplistic implementation. If you need to tweak the parsing routine to your situation do it here!
• Line 37: In my example I am expecting 4 fields, so I check that my string array contains four entries. If not, I assume it’s just a bad row and will inspect later.
• Line 39-40: Set the appropriate output columns.
• Line 41: Force the rows to the “Error Rows” output. Remember that extra output we created? It’s specifically for this situation, so we can redirect the “bad” rows to a different location.
• Line 44: If we have 4 elements in our array, assume the best! You probably want to perform additional sanity checks here, including wrapping the whole thing in a try-catch block, but for our example, I’m keeping it simple.
• Line 47-50: Set the appropriate output columns
• Line 51: Redirect output to “Match Rows”
• Line 56-71: This is simply a helper method to strip the text qualifiers off the text. Discussing it is outside the scope of this post, but I’ve tested it and it works fine.

Here is a similar script in Visual Basic .NET that will work for SQL Server 2005:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

' Microsoft SQL Server Integration Services user script component
' This is your new script component in Microsoft Visual Basic .NET
' ScriptMain is the entrypoint class for script components
 
Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper
 
Public Class ScriptMain
    Inherits UserComponent
 
    'Declare and initialize LineNum to keep track of the number of rows
    Dim LineNum As Int32 = 0
 
 
    Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
        'Increment LineNum  to keep track of the number of rows
        LineNum += 1
 
        'split the input column. This will be highly dependent on the format and will probably need adjustment
        Dim columns As String() = System.Text.RegularExpressions.Regex.Split(Row.line, "(?<=""),(?="")")
 
        'If the number of elements is not expected, we assume there was a problem
        If columns.Length <> 4 Then
            Row.ErrorLine = Row.line
            Row.ErrorLineNum = LineNum
            Row.DirectRowToErrorRows()
            'Everything looks good, let's move forward
        Else
            Row.MatchLineNum = LineNum
            Row.ID = StripQualifier(columns(0), """")
            Row.Company = StripQualifier(columns(1), """")
            Row.PhoneNumber = StripQualifier(columns(2), """")
            Row.FavoriteIrrationalNumber = StripQualifier(columns(3), """")
            Row.DirectRowToMatchRows()
        End If
 
    End Sub
 
    Public Function StripQualifier(ByRef InputString As String, ByRef Qualifier As String) As String
        'This is a helper function only to remove surrounding text qualifiers
 
        Dim OutputString As String
 
        If InputString.Substring(0, Qualifier.Length) = Qualifier And InputString.Substring(InputString.Length - Qualifier.Length, Qualifier.Length) = Qualifier Then
            OutputString = InputString.Substring(Qualifier.Length, InputString.Length - (2 * Qualifier.Length))
        Else
            OutputString = InputString
        End If
 
        Return OutputString
    End Function
 
 
End Class

Be sure and build the script (Ctrl+Shift+B or use the Build menu) before closing out. If not, you may receive a validation error. After a successful build, close and click “OK” on the Script Transformation Editor dialog. You can now connect the outputs from your Script Component to any destination, be it an OLE DB Destination or another flat file. The important thing to note is that you connect the Error Rows output to analyze any potential problems. You have access to both the record number and the original line read in. This should provide you with sufficient information to start analyzing the problem, and hopefully fix the parsing routine in the Script Component to deal with it.

Note, that performing this extra Script Component severely affected the processing rate versus SSIS natively supporting the import. Again, the best solution would be to correct the source, if possible.