I ran into this bug and had trouble explaining it to my colleagues. Part of the trouble comes from having no agreement on what a CSV file actually looks like.

Now the code runs, however all the rows in up going to the error table.

I am assuming that the regex parsing is not correct for my data.

My data is comma delimenated fields.

Not all field values are double quotes around the data. Which I believe you handle.

The fields that are delimenated with double quotes may have embedded commas or double quotes. It appears the embedded quotes are escaped with another double quote.

I assume that I can check for the embedded extra quote after removing the delimenating double quotes.

All that being said it appears that the regex expression is where I need help.

Forrest

The issue is with how picky SSIS is with data types. There’s a discrepancy between what is read from the file (ASCII string) and where it should be stored in the database (Unicode string). If these assumptions are true, you can simply add a Data Conversion component in the stream, converting the offending column from ASCII to Unicode with an appropriate code page (probably 1252). Just make sure to modify the database inset component to use this new, Unicode column.

I am getting a conversion error when I write it to the database table. The data coming out appears to be DT_STR and the database table wants DT_WSTR. How should by database columns be defined. They are current nchar(##), shold they be nvarchar(##).

Forrest

public override void Input0_ProcessInputRow(Input0Buffer Row)
{
        Regex rCSV = new Regex(",(?=(?:[^\"]*\"[^\"]*\")*(?![^\"]*\"))");
        Regex rQout = new Regex("\"\"");
 
        string[] fields = rCSV.Split(Row.line);
 
 
        if (fields.Length == EXPECTED_FIELDS)
        {
                // Case 1: Quoted field with "" embedded text qualifiers
                Row.FIELD1 = Format(field[0], rQout);
 
                // Case 2: Non-quoted field
                Row.FIELD2 = Convert.ToDESIRED_TYPE(field[1]);
        }
}
 
public static string Format(string input, Regex rQout)
{
        return (!String.IsNullOrEmpty(input)) ? rQout.Replace(input.Substring(1, input.Length - 2), "\"") : "";
}

——————————————————-

This has also the case where there are emtpy fields:

For example

“Hi”,123,,”my name is Marco the “”programmer”"”,”other text”

Will produce the output:

Hi
123

my name is Marco the “programmer”
other text

“Clark, Bob”,”123 Main St”,1234.00,45,”Something Else”,….

…so I am having trouble writing the appropriate parsing function because I cannot parse on just comma or on quote-comma-quote!

Thanks for listening to my frustration…{-:

Yes, thanks for pointing out that a line break as data would not work for this solution due to parsing each line as a single record.

And you’re right insofar as using DTS. This is what infuriates many people, is that this very old tool more correctly parses CSV than SSIS, despite the longstanding issue. At this point, it’s clear this is not a priority for Microsoft, so unless someone develops a custom Data Flow Source, all we can do is wait and hope for a patch or fix in the next version of SQL Server.

1,”Hello, this field
is a “”real”" pain!”,”4/27/2009″

Yes, that’s one row where:

Field 1 = 1
Field 2 (represented on one line with line break escaped for readability) = Hello, this field \r\nis a “real” pain!
Field 3 = 4/27/2009

The destination table is:

create table dbo.LogInfo
(
RecordID int,
LogInfo varchar(500),
LogDateTime datetime
)

I have looked into reading the file where each line equates to a single column and parsing from there as you suggested however the embedded line break prevents me from using that method.

Any further pointers on how to import csv files using SSIS would be much appreciated. Against some long-standing personal bias I am actually considering recommending using DTS, a 10+ year old technology, to solve the issue since is does a capable job of parsing and importing csv files and SSIS does not provide an easy path to process what many would consider a most common file format.

Thanks for reading.