Forum Discussion

photon's avatar
photon
Contributor
2 days ago
Solved

importing a file with variable columns

We have been tasked with importing a file that has a large number of variable columns. For the sake of easy explanation, let's say the first five columns are standard (time, entity, ud1, ud2, ud3) bu...
  • RobbSalzmann's avatar
    9 hours ago

    I would use approach 1 or 2.  The deciding factor is the size of the file, number of records.

    Since you know the first n columns are fixed to specific dimensions, and the remaining columns are specific to accounts, assuming column header = account name, this is pretty easy to pivot, something like this:

    Function PivotAccounts(inputTable As DataTable) As DataTable
        ' Prepare output table
        Dim outputTable As New DataTable()
        outputTable.Columns.Add("time", GetType(String))
        outputTable.Columns.Add("entity", GetType(String))
        outputTable.Columns.Add("ud1", GetType(String))
        outputTable.Columns.Add("ud2", GetType(String))
        outputTable.Columns.Add("ud3", GetType(String))
        outputTable.Columns.Add("account", GetType(String))
        outputTable.Columns.Add("amount", GetType(Decimal))
    
        ' Validate column count
        If inputTable.Columns.Count < 7 Then
            Throw New InvalidOperationException("Input table must have at least 7 columns.")
        End If
    
        ' Fixed column indexes
        Dim fixedColCount As Integer = 5
        Dim lastColIndex As Integer = inputTable.Columns.Count - 1
    
        ' Iterate through each row
        For Each row As DataRow In inputTable.Rows
            ' Extract fixed values once per row
            Dim timeVal As String = row(0).ToString()
            Dim entityVal As String = row(1).ToString()
            Dim ud1Val As String = row(2).ToString()
            Dim ud2Val As String = row(3).ToString()
            Dim ud3Val As String = row(4).ToString()
    
            ' Iterate over the account columns in the current row
            For i As Integer = fixedColCount To lastColIndex - 1
                Dim accountName As String = inputTable.Columns(i).ColumnName
                Dim rawValue As Object = row(i)
    
                If Not IsDBNull(rawValue) AndAlso Decimal.TryParse(rawValue.ToString(), Nothing) Then
                    Dim amountVal As Decimal = Convert.ToDecimal(rawValue)
    
                    ' create a record for each account's amount
                    If amountVal <> 0D Then
                        Dim outputRow As DataRow = outputTable.NewRow()
                        outputRow("time") = timeVal
                        outputRow("entity") = entityVal
                        outputRow("ud1") = ud1Val
                        outputRow("ud2") = ud2Val
                        outputRow("ud3") = ud3Val
                        outputRow("account") = accountName
                        outputRow("amount") = amountVal
                        outputTable.Rows.Add(outputRow)
                    End If
                End If
            Next
        Next
    
        Return outputTable
    End Function