Given a table with columns:
A: int, primary key, identity
B: int, not null

How can I set a default for column B to be the value in column A?

This is an interesting problem on a couple levels. First, a computed column won’t work, because we need the ability to arbitrarily update column B. We could program a trigger, but I prefer to avoid them when possible. Preferably, we could actualize the purpose of the DEFAULT option. Unfortunately, this isn’t immediately an apparent choice because SQL Server doesn’t allow you to reference a column in a DEFAULT specification. In fact, according to BOL, “Only a constant value, such as a character string; a scalar function (either a system, user-defined, or CLR function); or NULL can be used as a default.”

Fortunately, the column which we’re trying to set a default from had an identity property, and system functions dealing with identity values is the key to using the DEFAULT specification. First, a little code:

CREATE TABLE dbo.IdentDefault
(
	A INT NOT NULL IDENTITY (1, 1) PRIMARY KEY,
	B INT NOT NULL DEFAULT SCOPE_IDENTITY()
)
 
INSERT INTO dbo.IdentDefault DEFAULT VALUES
INSERT INTO dbo.IdentDefault DEFAULT VALUES
INSERT INTO dbo.IdentDefault DEFAULT VALUES
 
UPDATE dbo.IdentDefault SET B = 10 WHERE A = 1
 
SELECT * FROM dbo.IdentDefault

And the result:

A           B
----------- -----------
1           10
2           2
3           3

As you can see from the above result set, column B defaults to whatever column A contains, yet we can still update column B to whatever value necessary.

The Setup

• Windows 7 64-bit
• Visual Studio 2008
• SQL Server 2008 64-bit

The Process

1. Create a signing key (See also, How to: Create a Public/Private Key Pair)
   1. Open Visual Studio 2008 command Prompt – the regular command prompt will not work
   2. Change to a friendly directory: cd %userprofile%\Desktop
   3. Create the key file: sn -k key.snk
2. Sign the assembly – There are a few ways to do this, but I found this to be the easiest. If you want to sign it some other way, check out How to: Sign an Assembly with a Strong Name
   1. Right-click the Project
   2. Select “Properties”
   3. Navigate to the “Signing” tab
   4. Browse to strong name key file (which was created in the previous step)
   5. Recompile the project
3. Copy the re-compiled assembly to your GAC
   1. gacutil -i “C:\Path\to\CustomAssemblyName.dll”
4. Copy assembly to “%programfiles(x86)%\Microsoft SQL Server\100\SDK\Assemblies”
5. Add reference in script task. Repeat this for every Script Task you want to access this assembly from
   1. Right-click References
   2. Click “Add Reference…”
      

      Add Reference...

   3. On the .NET tab, scroll to find your assembly
   4. Press “OK”
   5. The Assembly should now appear under the References list
6. Add a reference to the assembly in code, at the top
   1. (C#) Using CustomAssemblyName;
   2. (VB.NET) Imports CustomAssemblyName
7. You should now have full access to the imported DLL

Caveats

This method works pretty well, but deployment isn’t exactly seamless – you’ll have to repeat this for each server and re-register & copy the DLL separately for any updates. Additionally, there is no way to globally add the assembly reference to the entire project or package. Instead, you’ll have to repeat step 6 (adding the reference) for every Script Task.

Steps

1. First, fire up Visual Studio (2005 or 2008 – it doesn’t matter).
2. Create a new project – name it something clever, like “RegEx”
3. After creating the project, you should be prompted to connect to a database where you’ll eventually want to deploy the project. This is completely optional and can be changed later.
4. Right-click the project name (“RegEx”) and choose Add → User-Defined Function
   

   Add User Defined Function

5. Name the file RegExMatch.
6. Paste the following code into that file
7. When done, simple build (Ctrl+Shift+B) and deploy (Right-click the project name (“RegEx”) → Deploy).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
using System.Text.RegularExpressions;
 
public partial class UserDefinedFunctions
{
    [Microsoft.SqlServer.Server.SqlFunction(IsDeterministic = true, IsPrecise = true)]
    public static SqlString RegExMatch(SqlString expression, SqlString pattern)
    {
        if (expression.IsNull || pattern.IsNull)
            return SqlString.Null;
 
        Match match = new Regex(pattern.ToString()).Match(expression.ToString());
 
        return match.Success ? new SqlString(match.Value) : SqlString.Null;
 
    }
};

Code Explanation

• Line 1-6: Including necessary assemblies. The only item you need to add is line 6 – System.Text.RegularExpressions
• Line 11: We indicate the function requires 2 parameters, the input string and the regular expression to apply.
• Line 13-14: Check if either input string is NULL. If so, return NULL and do nothing else.
• Line 16: There’s a lot packed on this line, but essentially, it creates an object named match based on the results of the regular expression match operation.
• Line 18: Use the ternary operator to check if the match was a success. If so, return the matching string. Otherwise, return a NULL.

Examples

We’re going to use a simple regular expression to check for a valid US postal code (AKA Zipcode + 4):

^\d{5}(-\d{4})?$

This regular expressions checks for exactly 5 digits followed by an option group of hyphen and 4 more digits.

SELECT
	  dbo.RegExMatch('90210','^\d{5}(-\d{4})?$')
	, dbo.RegExMatch('90210-1234','^\d{5}(-\d{4})?$')
	, dbo.RegExMatch('90210-','^\d{5}(-\d{4})?$')
	, dbo.RegExMatch('9021A','^\d{5}(-\d{4})?$')

And results:

90210	90210-1234	NULL	NULL

Other notes

If you need to change the target database, do so in the project’s properties:

1. Right-click the proeject name (“RegEx”) → Properties
2. Select the Database tab (2nd item from the bottom)
3. Click “Browse…” to create and test the connection string.
   

   Database Properties

Thanks to Jeff’s SQL Server Blog for the initial Regular Expression Replace code.

1. Download and install MySQL Connector/NET 6.0
2. Start a new Integration Services project in BIDS
3. Right-click in Connection Managers and create a new ADO.NET Connection

   

4. In the Provider dropdown, expand .Net Providers and select MySQL Data Provider. Press "OK"

   

5. Fill out the Server name, User name, Password and select the database name for the target MySQL server. Be sure to test the connection and press “OK”

   

6. Rename the connection to “MySQLDB”

   

7. Open up the Toolbox and drag a Data Flow Task from the toolbox onto the Control Flow surface
   

   Add Dataflow Task

8. Double-click the Data Flow Task to switch to the Data Flow view
9. Create a new variable, “MySQLResult” with the Data Type of Object. We will be using this as the final destination for the data, so we don’t need to connect to a file or database to store the data from this test
   

   MySQLResult Variable

10. Drag a new DataReader Source component onto the Data Flow surface
    

    Add DataReader Source

11. Double-click the DataReader Source to open the Advanced Editor. On the Connection Managers tab, select the previously-created MySQLDB connection
    

    DataReader Source Connection Managers

12. Switch to the Component Properties tab and enter the SQL query in the SqlCommand property. Note that the query must be compatible with MySQL syntax, not SQL Server.
    

    DataReader Source Component Properties

13. Switch to the Column Mappings tab to verify that the query is successful and the all the columns were pulled from the database. When done, press “OK”.
    

    DataReader Source Column Mappings

14. Create a new Recordset Destination by dragging it from the toolbox to the Data Flow surface
    

    Add Recordset Destination

15. Drag the green Data Flow Path from DataReader Source to Recordset Destination, so they connect
    

    Connect Source to Destination

16. Double-click the Recordset Destination to open its Advanced Editor
17. Under Custom Properties, select the dropdown for VariableName and select the variable we created before, User::MySQLResult
    

    Recordset Destination Component Properties

18. Switch to the Input Columns tab and select those columns that you want stored in the Recordset Destination. When complete, click “OK”
    

    Recordset Destination Input Columns

19. Right-click the green Data Flow Path and choose “Data Viewers…”
    

    Data Flow Path Data Viewers

20. Select “Data Viewers” from the left pane and click the “Add…” button
    

    Data Flow Path Editor

21. Under the General tab, select Grid and press “OK”
    

    Configure Data Viewer

22. Run the package
23. If you’ve done everything correctly, you should see a Data Reader Output Data Viewer window pop up with the contents of the query we specified earlier.
    

    Data Viewer Output

SQL Server Integration Services makes connecting to other systems very easy. The MySQL ADO.NET provider works well, but requires more configuration than a native Source component.

Actual employee structure to be flattened

This structure will act as our example for the Development Department. The Department Head (Employee 0) has asked for a report of all employees within each team of the development group. Notice that different groups have varying levels of depth. The Database team only has a Manager with two direct reports. The UI team is a single person. The Middle Tier group is much larger, with a Manager having two direct reports, and one of those employees having several employees beneath him. The requested report should group all employees, flattening the relationship to a single Manager, but excluding himself (since that would mean every employee would simply roll up to him). This “Desired Relationship” is represented below:

The desired result of the query

To recreate this scenario, I’m providing some code for both the setup and query. If you’re not familiar with Common Table Expressions, I would suggest that you familiarize yourself with the SQL Server BOL entry. The main concept you should understand to be able to adapt this to your own data is that for a CTE to act recursively, you need both an anchor and recursive query. I’ll explain their parts later.

First, let’s instantiate the tables and populate them with data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

USE tempdb
go
 
IF OBJECT_ID('dbo.Employees') IS not null
	DROP TABLE dbo.Employees
go
 
CREATE TABLE dbo.Employees
(
	  EmployeeID INT PRIMARY KEY
	, EmployeeName VARCHAR(50) not null
	, ManagerEmployeeID INT null
)
go
 
INSERT INTO dbo.Employees (EmployeeID, EmployeeName, ManagerEmployeeID)
SELECT 0, 'Employee 0', null
UNION all
SELECT 1, 'Employee 1', 0
UNION all
SELECT 2, 'Employee 2', 0
UNION all
SELECT 3, 'Employee 3', 0
UNION all
SELECT 4, 'Employee 1.1', 1
UNION all
SELECT 5, 'Employee 1.2', 1
UNION all
SELECT 6, 'Employee 3.1', 3
UNION all
SELECT 7, 'Employee 3.2', 3
UNION all
SELECT 8, 'Employee 3.1.1', 6
UNION all
SELECT 9, 'Employee 3.1.1.1', 8
go

This script will simply create the example employees table and populate it with data to relationally represent what is displayed in the first picture. Here are what the various lines accomplish:

• 1-2: Switch the database context to tempdb. Since the contents of tempdb are cleared upon service restart, we’re simply ensuring that this will be cleaned up eventually.
• 4-6: Check if dbo.Employees exists. If so, drop it.
• 8-14: Create a table to hold our example data.
• 16-36: Manually populate the example data. Note that we’re using a UNION ALL between the selects, so only a single INSERT occurs.

As for the query to manipulate this data, here is the actual CTE:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

-- Use a CTE to flatten an organization structure to each department
WITH AllMyChildren (ManagerEmployeeID, EmployeeID, GroupManagerID, EmployeeName, DEPTH)
AS
(
	-- The anchor statement. This selects each department head
	-- The important part is to alias the original EmployeeID as the GroupManagerID
	SELECT ManagerEmployeeID, EmployeeID, EmployeeID [GroupManagerID], EmployeeName, 0 [DEPTH]
	FROM dbo.Employees WHERE EmployeeID in
		-- We want all employees 1 level below below a certain level, so we'll utilize a subquery to find them
		(
		-- Get EmployeeIDs that are children to their common parent
		SELECT EmployeeID FROM dbo.Employees WHERE ManagerEmployeeID = 0
		)
	UNION ALL
	-- The recursive statement which finds all employees under each department
	SELECT pc.ManagerEmployeeID, pc.EmployeeID, amc.GroupManagerID, pc.EmployeeName, DEPTH + 1
	FROM dbo.Employees pc
	INNER join AllMyChildren amc ON pc.ManagerEmployeeID = amc.EmployeeID
)
-- The CTE is primed, but we still need to execute it with the statement below
SELECT EmployeeName, GroupManagerID, DEPTH 
FROM AllMyChildren
ORDER BY GroupManagerID, DEPTH
-- Use the MAXRECURSION query hint to avoid default recursion limit of 100
OPTION (MAXRECURSION 0)

I’ve commented liberally, but I’ll go through this line-by-line to explain it in more detail.

• 2-4: Define the CTE (AllMyChildren) and the columns that it will output (ManagerEmployeeID, EmployeeID, GroupManagerID, EmployeeName, Depth)
• 7-8: This is the anchor statement. SELECT the fields we eventually want outputted. The key to this is that EmployeeID is being included a second time, but aliased as GroupManagerID.
• 10-13: This is a subquery in the WHERE condition of the anchor statement. Because we want to discard some levels (only the topmost level in our example), we need to tell the anchor to fetch all children whose parent EmployeeID is 0. Adjust this sub-query to affect at what level the teams should roll up to.
• 14-18: UNION ALL the anchor to the recursive portion of the CTE. The recursive part of the query is similar to the anchor in that it must SELECT similar columns. Notice that it again selects from dbo.Employees, but it also performs an INNER JOIN against the CTE, linking the EmployeeID and ManagerEmployeeID. This is what causes to recursion to occur. Additionally, we increment Depth by 1. The inclusion of the level is completely optional in the CTE, but may be helpful for reporting.
• 19: Close the CTE block. At this point, it is defined, but we have yet to execute it.
• 21-23: Perform a SELECT against the CTE to start it.
• 25: By default, recursion in Common Table Expressions is limited to 100 iterations. If you anticipate having more than 100 records, you’ll need to specify MAXRECURSION 0. When testing, you can limit it to any number up to 32,767 to prevent wild-running queries. Here – since this is tested and working – we specify “0″ for unlimited iterations.

And this is the result of the above query:

The tabular results of the query

Configure a MySQL DSN

The first step is to configure a MySQL data source by running the ODBC Data Source Administrator. This step is technically entirely optional, but allows a simpler configuration in the SQL Server Linked Server settings. Instead of composing a complicated MySQL connection string, we can use a simple GUI application.

Run odbcad32

If you’re using Windows Server 2003, bring up a Run dialog box with Start→Run or WinKey+R. If you’re using Windows Server 2008, use the Start Menu search box directly. In either OS, type in “odbcad32″ and hit Enter.

System DSN

Select the System DSN tab to configure a data source for the entire system. If you only want to create the DSN for a specific user (such as your service account), use the User DSN tab. In either scenario, select the “Add…” button.

Create New Data Source

Scroll down in the Create New Data Source window and select “MySQL ODBC 3.51 Driver” and click “Finish”.

MySQL Connector Login Settings

Once added, clicking the “Configure…” button will bring up the Connector/ODBC 3.51 Configure Data Source application. This is where you can specify all the connection settings for connecting SQL Server to MySQL. Select a Data Source Name – I typically name it after the application or database I’m connecting to. The Server, User, Password, and Database should all be obvious.

Test ODBC Connection

After you’ve entered all the required parameters, click the “Test” button to ensure a connection can be made to the MySQL server.

These settings are the bare minimum required to connect MySQL and SQL Server via a linked server, but I like to specify additional options to optimize the connection between the servers. Without these, I have run into “Out of Memory” errors that require restarting the service.

MySQL Connector Advanced Flags 1 Settings

Select the Advanced tab and you’ll be placed on the “Flags 1″ sub-tab. Check the boxes labeled “Allow Big Results” and “Use Compressed Protocol”.

MySQL Connector Advanced Flags 2 Settings

Next, switch to the “Flags 2″ tab and select “Don’t Cache Result (forward only cursors)”. This can actually be a performance penalty if you perform the same query multiple times to the same linked server. However, in my experience, the reason to connect SQL Server to MySQL, is to pull data into a single server, in which case, this option is perfectly suited.

MySQL Connector Advanced Flags 3 Settings

On the “Flags 3″ tab, select “Force Use Of Forward Only Cursors”. When you’re done setting all these options, select the “Ok” button.

Configure Linked Server Provider

Adjusting the Linked Server Provider is simple, but it comes with a caveat: When adjusting a provider, you are adjusting it for all connections using that provider. I am not aware of any way to change these settings on a per-connection basis.

Provider Properties

Drill down to Server Object → Linked Servers → Providers, right-click MSDASQL, and select “Properties”.

Set Provider Options

The Provider Options for Microsoft OLE DB Provider for ODBC Drivers dialog box will open allowing you to configure several options. Ensure the following four options are checked:

• Nested queries
• Level zero only
• Allow inprocess
• Supports ‘Like’ Operator

All other options should be unchecked. When done, click “OK”.

Create Linked Server to MySQL

Finally, the last step in our process is to create the actual MySQL Linked Server.

Create a New Linked Server

You should already have Linked Servers expanded in the Object Explorer tree. If not, find it in Server Objects → Linked Server. Once there, right-click Linked Servers and select “New Linked Server…”

New linked Server Settings

The New Linked Server dialog box will open. Because we specified all our connection settings in the ODBC Data Source Administrator, this last step is very simple. Name the linked server. As with the Data Source Name, I like to name it after the product or database I’m connecting to. In my example, I used MYSQLAPP. Ensure that the “Other data source” option is selected and choose “Microsoft OLE DB Provider for ODBC Drivers” from the Provider dropdown. Lastly, specify the Product name and Data source. The Product name doesn’t matter so much as the Data source must match what you provided in the MySQL Connector/ODBC configuration. Press “OK” when complete.

Testing the SQL Server to MySQL connection

If everything has been set correctly, you should be able to execute a query directly again the MySQL database from SQL Server Management Studio. For example:

SELECT TOP 10 TABLE_NAME FROM MYSQLAPP...tables WHERE TABLE_TYPE != 'MEMORY'

If you’ve done everything correctly, you should get back a result set. There are several error message you might receive:

OLE DB provider "MSDASQL" for linked server "MYSQLAPP" returned message "[Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified".
Msg 7303, Level 16, State 1, Line 1
Cannot initialize the data source object of OLE DB provider "MSDASQL" for linked server "MYSQLAPP".

The message indicates that the Data source name you’ve specified for the linked server does not match that of the Data Source Name specified in the MySQL Connector.

Msg 7313, Level 16, State 1, Line 1
An invalid schema or catalog was specified for the provider "MSDASQL" for linked server "MySQLApp".

This uninsightful error is a result of not correctly setting the options for the Linked Server Provider.

Msg 7399, Level 16, State 1, Line 1
The OLE DB provider "MSDASQL" for linked server "MySQLApp" reported an error. The provider did not give any information about the error.
Msg 7312, Level 16, State 1, Line 1
Invalid use of schema or catalog for OLE DB provider "MSDASQL" for linked server "MySQLApp". A four-part name was supplied, but the provider does not expose the necessary interfaces to use a catalog or schema.

This “four-part name” error is due to a limitation in the MySQL ODBC driver. You cannot switch catalogs/schemas using dotted notation. Instead, you will have to register another DSN and Linked Server for the different catalogs you want to access. Be sure and follow the three-dot notation noted in the example query.

If, however, you want to access other schemas, you can do so utilizing OPENQUERY. This is also a great way to test your connection if you’re receiving problems. The syntax looks like this:

SELECT * FROM OPENQUERY(MYSQLAPP, 'SELECT * FROM INFORMATION_SCHEMA.TABLES LIMIT 10')

Notice that the actual query syntax in the string must be in the MySQL format (SQL Server does not support the LIMIT keyword). Additionally, you can specify a different schema using SCHEMA.TABLENAME in the query.

Conclusion

Creating a linked server between SQL Server and MySQL is a simple process. The first time requires you to install the software and configure the Linked Server Provider, but all subsequent connections require only a DSN and Linked Server.

SELECT CONVERT(DATETIME, 0)
--Result: 1900-01-01
 
--Store the date to a DATETIME variable first
DECLARE @DATE DATETIME
SET @DATE = '1900-01-01'
SELECT CAST(@DATE AS INT)
--Result: 0

You can use cast and convert for either direction, there is no restriction. Note in the second example, I stored the date in a variable first, because there is no way to natively pass a DATETIME type into the query window; SQL Server must always convert it from a string. By taking the intermediary step of storing it into a variable, we ensure that the Database Engine truly understands the value as a DATETIME and not a VARCHAR. If you had attempted to convert the VARCHAR value directly, you would receive the following error:

SELECT CAST('1900-01-01' AS INT)
--Msg 245, Level 16, State 1, Line 1
--Conversion failed when converting the varchar value '1900-01-01' to data type int.

The trick works for any value valid in the DATETIME range (January 1, 1753, through December 31, 9999):

SELECT CONVERT(DATETIME, 2958463)
--Result: 9999-12-31

Try to go too far, however, and you’ll get an error:

SELECT CONVERT(DATETIME, 2958464)
--Msg 8115, Level 16, State 2, Line 1
--Arithmetic overflow error converting expression to data type datetime.

Manipulating Dates

Fortunately Microsoft has provided some very useful functions for manipulating the DATETIME data type, negating our need to perform complicated math.

DATEADD

According to MSDN, DATEADD returns a specified date with the specified number interval (signed integer) added to a specified datepart of that date. The function prototype looks something like this:

DATEADD (DATEPART, number, DATE)

Because any date can be referenced as a simple integer, adding dates becomes trivial with the DATEADD function. Let’s compare adding days with both as a DATETIME and as an INT:

SELECT DATEADD(DAY, 1, '1900-1-1')
--Result: 1900-01-02

And compare that with adding 1 date to 0

SELECT DATEADD(DAY, 1, 0)
--Result: 1900-01-02

Notice the result is the same. We can also add any other datepart. Want the start of 2009?

SELECT DATEADD(YEAR, 109, 0)
--Result: 2009-01-01

DATEDIFF

DATEDIFF acts in an opposite capacity of DATEADD: it calculates the date or time difference between DATETIME values. The prototype looks like this:

DATEDIFF (DATEPART, startdate , enddate)

And an actual example:

SELECT DATEDIFF(YEAR, '1900-01-01', '2009-01-01')
--Result: 109

As before, we can certainly specify any of the dates as a INT. Since we know that 1900 has a corresponding integer value of 0, let’s try that:

SELECT DATEDIFF(YEAR, 0, '2009-01-01')
--Result: 109

We can take it a step further and compare two INT values (again, as long as they fall within the valid DATETIME range). Let’s find how many days are between 1900-01-01 and 9999-12-31:

SELECT DATEDIFF(YEAR, 0, 2958463)
--Result: 8099

The important aspect to note is that DATEDIFF “returns the number of date and time boundaries crossed between two specified dates.” (MSDN) In plain language, that means SQL Server never rounds the time period, but rather calculates the time periods completed. For example, let’s check the number of years between Leap Day in 2000 and Christmas 2008:

SELECT DATEDIFF(YEAR, '2000-02-29', '2008-12-25')
--Result: 8

Although the total time is nearly 8.9 years, only 8 actual year boundaries have been crossed. This principle is the basis for calculating beginnings of time periods. When we combine DATEDIFF with DATEADD, the combination will predictably give us the start of any time period. Let’s say we wanted to calculate the number of years since 1900-01-01:

--Assume GETDATE() returns 2009-02-12 22:23:00.000
SELECT DATEDIFF(YEAR, 0, GETDATE())
--Result: 109

We could easily take this result and place it in a DATEADD and see that we get 1900 + 109 = 2009

SELECT DATEADD(YEAR, 109, '1900-01-01')
--Result: 2009-01-01

Or, as before, let’s use the integer value for 1900-01-01:

SELECT DATEADD(YEAR, 109, 0)
--Result: 2009-01-01

Since DATEDIFF returns an INT and DATEADD adds an INT to a DATETIME, we can combine the two and return a natural DATETIME in a single query:

--Assume GETDATE() returns 2009-02-12 22:23:00.000
SELECT DATEADD(YEAR, DATEDIFF(YEAR, 0, GETDATE()), 0)
--Result: 2009-01-01

To analyze what is actually occurring, let’s start with the DATEDIFF in the middle <DATEDIFF(year, 0, GETDATE()>. This calculates the number of year boundaries crossed between 1900 and 2009, which is 109. The outer portion <DATEADD(year, 109, 0)> adds that result of 109 to 1900 and gives us the start of the year 2009. This technique allows us to find the start of any date or time period based on a relative starting point. That starting point is commonly GETDATE(), but can be any valid DATETIME. If you want the beginning of the month, swap “year” for “month”. The same can be used for day, week, or any other datepart. This can help us greatly when querying a DATETIME field to fall in a range of values (i.e. “Select all rows created this year.”) What if we wanted the start of next year (2010)?

--Assume GETDATE() returns 2009-02-12 22:23:00.000
SELECT DATEADD(YEAR, DATEDIFF(YEAR, 0, GETDATE()) + 1, 0)
--Result: 2010-01-01

Notice the only change was a “+ 1″ next to the DATEDIFF calculation. Because we’re calculating the number of years since 1900 (109), we can adjust for years forward by adding, or years back by subtracting. By adding only 1, we jump forward a single year.

Query efficiency

Using DATEADD and DATEDIFF is much faster than string manipulation with CAST and CONVERT. Additionally, it allows us to check the bounds of a DATETIME column without using the YEAR, MONTH, or DAY function. Why is this good? Any modification applied to a column in the WHERE condition, negates the ability for SQL Server to utilize an index on that column. This is because the index contains only the actual values, not some calculation of them. Think of it this way: If you looked in the index of a book and wanted to know the pages of topics whose third letter is an “e” is, you would have to scan the entire article list, because the index is based on the first letter, not third. This is similar to how a YEAR function might affect an index on a DATETIME column. To get around this, rather than altering the column value in the condition, we alter our lookup condition. For example, instead of YEAR(DateColumn) = YEAR(GETDATE()), we can specify DateColumn >= @BeginningOfThisYear AND < @BeginningOfNextYear. Or, more specifically:

SELECT *
FROM TableName
WHERE DateColumn >= DATEADD(YEAR, DATEDIFF(YEAR, 0, GETDATE()), 0)
	AND DateColumn < DATEADD(YEAR, DATEDIFF(YEAR, 0, GETDATE()) + 1, 0)

In English, it reads something like this: “Select everything from TableName where DateColumn is at least the start of the year and also before next year”. Or, even simpler, “Show me where DateColumn is the current year”. This has a similar effect of YEAR(DateColumn), but will allow an index on DateColumn to be used. Obviously, this will not guarantee the index will be used – if one exists at all – but, it at least leaves the option available to the query optimizer.

Conclusion

It is very common to check for records falling in a specified range such as current year or last month, but many people resort to nasty functions forcing a table scan. With the above knowledge and tools, developers can more efficiently leverage the DATETIME data type and speed up slow-running queries.

Ingredients

I’ve created an SSIS package that will import the articles and pages into a SQL Server 2005 database. To do this, you’ll first need to gather a few files:

• Latest Wikipeda pages/articles dump (Download enwiki-latest-pages-articles.xml.bz2, approximately 4.1GB at time of writing)
• MediaWiki XSD (originally located on Manual:XML Import file manipulation in CSharp)
• SSIS Package & Database Scripts
  

You’ll need approximately 75GB of free space – 50GB for the database and 20GB for the XML file. Also, this will likely take several hours, if not longer. If you have separate drive spindles, it would certainly help to separate the XML and database files. My example uses C:\wikipedia\ as the working folder; if you prefer another location, we’ll configure it later. If not, this is what the structure should look like:

Directory Listing

Recipe

Note: If you’re planning on importing the XML file into a remote server, I highly recommend performing all these operations on the server itself through Remote Desktop. Aside from having to re-transfer the gigantic XML dump, debugging SSIS packages is much easier when working locally.

1. Connect to the SQL Server database and run WikipediaImport\WikipediaImport\DatabaseCreate.sql
   1. This creates the database (cleverly named “Wikipedia”) on C:\wikipedia\. If you want the data and log files located elsewhere, find 50GB of free space and update the CREATE DATABASE statement.
   2. Because we know the database is going to grow immediately, I’ve told the script to allocation 40GB for data and 10GB for log, so this step may take a while to run.
2. Open WikipediaImport\WikipediaImport.sln in Visual Studio 2005
3. Enable the Variables window if it is not already visible
   1. Select Data Flow
   2. Select the View menu
   3. Select Other Windows
   4. Select Variables
      

      Enable Variables Menu

4. If the working files were placed somewhere besides C:\wikipedia\, you can configure that in the Variables window. Be sure to update both PageArticlesXML and PageArticlesXSD
   

   Set Variable Values

5. Additionally, if you’re not importing to localhost, configure the database connection variable (named DatabaseConnection)
6. Verify there are no warnings or errors and build the solution (Ctrl + Shift + B or Build?Build WikipediaImport)
7. If the build succeeds, go ahead and run it (F5 or Debug?Start Debugging)
8. If all goes well, the XML file should now be streaming into the database. This will likely take hours, even with a fast RAID. Notice that the file only requires a single pass, rather than scanning it once per table.
   

   Import Progress

9. Switch back to SSMS and run WikipediaImport\WikipediaImport\IndexCreate.sql
   1. This step is technically optional, but is going to help speed up your queries significantly
   2. If we had created the indexes before the import, the import would have been even slower
   3. This will take a while!
10. Run a test query

    SELECT * FROM dbo.vw_Articles WHERE title = 'Green Day'

Behind the Scenes

Getting this to work took a while of tweaking, but there are a few highlights I’d like to point out.

Data Types

XSD: The provided XML Schema Definition file does not contain any information about the intended length of the string data. Fortunately, through testing, I was able to shrink some of those sizes down, although they do not strictly conform to the official database schema. Specifically, I have shrunk page.restrictions and text.space from nvarchar(255) to nvarchar(50). Most other items conform as close as possible. In addition to these, I had to update text.text to a nvarchar(max). SSIS initially suggested an nvarchar(255), but articles are obviously much longer than this. To perform these changes, right-click the XML Source (named PageArticles) in the Import XML Data Flow Task and select “Show Advanced Editor…”

Advanced Editor Menu

Expand out each of the changed columns (both External and Output) and update the DataType. For example, SQL Server-specifc nvarchar(max) is a more general “Unicode text stream [DT_NTEXT]” in SSIS. For the others, just update the length from 255 to 50.

Set Data Types

Extraneous information

There are many more fields in the XML file than I’ve decided to import. Unfortunately, I can’t just turn them off completely, lest SSIS complains. Instead I have chosen to suffer the lesser fate of “Warning”. Additionally, I changed the Error Output to “Ignore failure” on Error. This screen can be accessed by double-clicking the XML Source, PageArticles, then selecting the Error Output page.

Configure Error Output

Wrap-up

SSIS is very picky about metadata, making this a somewhat difficult project to get running, however, it is actually running. This could be further extended with an automated download and increased amount of data imported, but for now it serves its purpose.

I know not everyone will get this on first run, so if you have a problem, please leave a comment below and I’ll do my best to answer them in a timely manner.