views:

1751

answers:

21

At the risk of being downmodded, I want to ask what the best mechanism (best is obviously subjective for the practice violation inherent here) for viewing data from a table, using C#, with a lot of columns. By a lot, I mean something like 1000.

Now before you get all click happy, or throw out responses like "why the hell would you ever have a table with that many columns" let me say that it's actually part of a design requirement. We are collecting data as fast as we can from 1000 data points. We need to store these as fast as possible, hence the flat table. The data needs to be directly accessible from SQL Server, hence the database (we're using SQL Compact with table-direct).

So let's forget, for now, all that we've learned about proper database design, the rules of normalization, etc. and just focus on the fact that I have a table with 1000 columns and I want to be able to display the data on screen to verify that the data is actually going in there.

I've tried a data grid. It pukes because (not surprisingly) it's not designed to handle that many columns.

I've tried using the viewer in Studio. It pukes after 256, plus the end user won't have Studio installed anyway.

For now the result need not be pretty, it need not be updateable, nor does it need to be sensitive to data changes - just a static snapshot of data in the table at a given point in time.

Relevant (or semi-relevant) info:

  • Table has 1000 columns (read above before getting click happy)
  • Using SQL Compact version 3.5
  • Running on the desktop
  • Looking for a managed-code answer
+2  A: 

try a html table with a very tiny font

if you're concerned about formatting the table use CSS:

td { font-size: 0.2em; text-align: right; }

alternately, if all your numbers are the same size, you could also just generate a "wall of numbers" display, e.g. use a fixed-width font and display columns 5 characters wide in a scrolling panel

Steven A. Lowe
Not a bad call, actually, though then I have to write code to format said HTML. I'm not concerned about scrolling left/right, in fact as crazy as that sounds, it's desired.
ctacke
@ctacke: see edits
Steven A. Lowe
A: 

I feel dirty even for suggesting this, but you could do something along the lines of:

SELECT Field1 + ' - ' + Field2 + ... AS EvilMegaColumn FROM Table

but really I think this falls into the category of "if you're running into this limitation, you're doing something wrong". I really can't see any reason, speed or otherwise to need 1000 columns...

Whisk
The SELECT gets the data - that's trivial. It won't *display* it though. And though you don't see the need, trust me, we've been testing this heavily and a single table, with a boatload of columns gives us the best write performance coupled with queryability from standard SQL Server tools.
ctacke
There's also a limitation on the length of your SQL string, so having a single statement retrieve the data like that is asking for trouble.
Whytespot
A: 

How much of the data is critical for the initial view? I can see doing something like a master/detail type grid where you're putting the critical columns (say like 10) onto the datagrid and when the user clicks to view the details, you can take the remaining columns and display them in a "properties area" or something in that regard.

Dillie-O
If it's 20-30 rows that's probably enough to prove it out. There are no "critical" columns, though, and no way to normalize the data. It's simply a giant array of data pulled of a remote PLC.
ctacke
+1  A: 

Who will read a 1000 column table??? Try to think of way to filter or visualize the data.

Manu
I was going to say that too. If it's an integer, how about each point is a bit position left to right, the number is a color, then each time you take a reading, scroll up one line and repeat. If it's strings--maybe sample a few?
Bill K
I will. I (and the customer) need to have confidence in the temporal consistency of the data, and while it sounds ugly, being able to scroll through it in a big grid allows you to visually find inconsistencies quickly.
ctacke
+3  A: 

do you need to view multiple rows on a single table?

my guess is that this data is numerical, is there any way you could display a single rows data as a 20*50 grid or something like that, then just paginate through the rows?

Eg, row 1, column 1 = colum 1 of the database, row 2, column 1 = column 21 of the database, etc

Id = 1
     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
----|--------------------------------------------------------
  0 |  
 20 |  
 40 |
 60 |
 80 |
100 |
120 |
etc |
Matthew Watson
Yes, for now it's purely numerical (some bit fields probably too), as they are largely input from PLC tag reads. It has to be one row per line though. Anything else will be confusing and unreadable.
ctacke
I think the everything-on-one-line approach is impossible unless you have an incredibly wide monitor or are willing to scroll left and right a huge amount. If the scrolling is OK, then use a one-column datagrid with a fixed-width font, and roll your own 1000 columns.
MusiGenesis
Weird. When I first posted my previous comment, it ended up in the comments for the question.
MusiGenesis
A: 

Perhaps you should investigate a different type of database. I've heard column-oriented databases are good for this sort of thing (whereas a typical RDBMS is row-oriented). Also, if you won't be going back to update rows after they're first inserted, maybe a binary flat file would be better than a giant table?

rmeador
We've already considered this. A different type might be faster - there are historians that are fast and much better and keeping actual storage low. However it's tough for them to use sync services or replication to get that into their back-end.
ctacke
A: 

I would make this a drill-down. In the first page (or at the top of the page) you would have controls that select the row. In the next page (or at the bottom of the page) you would display the data from the selected row. Depending on the required cell width, you might do this as 100 rows of 10 columns, or 1000 rows of 1 column, for example.

This would be fairly easy to do as dynamic client-side javascript -- you could even make it editable this way. I'm not sure how this would work in C#.

Ken Paul
The data is not relational in and way (except that each table represents a different PLC in the line), so tehre's no logical "drill down", and it's not stored normalized anyway.
ctacke
+4  A: 

What about storing the data in a csv file, which would give you options for viewing. If your user has excel or Open Office Calc, they could easily import the data (not sure if there is a column limit on Calc, but excel 2007 can hold 16384 columns) and view it through that program?

Jason Down
CSV is an option we already tried. Direct table access to an SQL CE table is actually faster because there's no string manipulation.
ctacke
Ya we've found it to be quite fast using SQL CE... I think we were using direct table access as well, but I can't remember off hand.
Jason Down
A: 

If you are just after a verification could you not check each field programatically and report that entire row is ok!. Then you need a much simple data grid which lists the rows that are not so good.

They can then be examined by whatever technique you can apply to a single row as you will not need to browse the fields in most cases. I am assuming here that you can view the entire row somehow already and are after a way to browse several rows at the same time looking for missing data (automating this will make it much more reliable).

Brody
Sure, I can report that, and the SQL to do so is straightforward. However when the customer pulls data off the factory line, especially when vetting the system, they'd rather trust their eyes than my say-so.
ctacke
A: 

Coming at it from an oblique angle, I'd ask if the user needs to have all the columns "loaded" at one time?

If the users would be happy to have a subset of columns displayed at once (say, 100 at a time, or specfic sets at a time), then I'd use a some kind of data grid (the built in one, or a ListView, or maybe a third party one) to display the subset, with a CheckedListView docked to the side, allowing the subset of interest to be displayed.

Alternatively, could you display some kind of summary data showing the count/average/xxx for groups of 100 columns?

Bevan
An interesting thought, though I was honestly trying to just knock out a really low-code, simple way for data verification. Real reports are going to come from a much more elegant tool.Averages don't help - we're looking for proof of temporal consistency.
ctacke
+1  A: 

It depends a bit on how pretty it needs to be. If this is just a debug/spot check tool, you could put several DataGrids side by side, each one displaying a selection of columns. Would be kind of ugly, but would be workable.

OTOH, if you need a semi-polished tool, you might want to come up with a custom control to handle it. Basically, you would load the section of the database being viewed, with a bit of buffer, and when the user scrolled off the currently loaded data, run a new query.

Jeff Olhoeft
Pretty is not required (and if you'd seen any of my UI's you'd know it's unobtainable for me anyway). It's an interesting thought I might play with tomorrow.
ctacke
A: 

If all you need is to make sure the data is being populated then why not have every column with a default value, say, 'void', 'blank', etc.

Then you can iterate through while counting non-default/total to show a percentage.

Now you can visualize the data completeness with a percentage value, maybe even record which columns had the default values (like to a list/array) for further investigation.

Karl R
Making sure there is data and making sure the data is *right* are two different animals.
ctacke
+1  A: 

A DataGrid (or even a ListView) should be able to handle a table with 32 columns and 32 rows, which would allow you to display an entire DB row's worth of data at once. This would allow you to instantly see whether some cells were missing data or not.

MusiGenesis
So you propose rotaing the table, with rows becoming columns? Now *that* is interesting and would provide the same visual markers we need.
ctacke
Cool idea... and so simple!
Jason Down
What I'm proposing is more like word-wrapping, where you show 32 columns of data on each row (row 1 is columns 1-32, row 2 is columns 33-64 etc.), for 32 rows.
MusiGenesis
My suggestion was the same as Matthew Watson's above, but I didn't read your comment about everything being on one line.
MusiGenesis
A: 

You might consider checking with your user base and seeing what they really need to see, then set up views for each distinct need, in order to get the column count down.

Another option would be to read the data, and create a whopping big static set of html pages from it. Then you could invoke the browser from within your program to view it.

EvilTeach
+12  A: 

You could format all numbers as n-character strings with spaces and then display them in a fixed width font.

1       2       3       4       6      36     436    6346
2       3       4       6      36     436    6346       0
3       4       6      36     436    6346       3       4
4       6      36     436    6346     333     222     334
configurator
+1 for the time-tested "wall of numbers" display
Steven A. Lowe
Don't laugh. If civilization collapses and we're thrown back to 1950's era technology, he'll have a competitive edge.
MusiGenesis
The simplest ideas alwasy seem to elude me. This seems really right. I could even draw a grid between columns and rows with ASCII to give it that 80's feel.
ctacke
As a personal favor to me, would you make it a green Courier New font on a black background? Thank you.
MusiGenesis
A: 

Have a scrollable pane and show 10 columns at a time (these can be actively loaded or cached or whatever you need). When you are scrolled left, show the first ten. As you scroll right, show the latter sequence of columns. So all in all, only 10 columns are active at any given point. Trying to actually display 1000 columns would be nuts any other way in my opinion. PS: This is nothing more than an ideal guess; I'm not really sure if it's remotely possible.

Joe Philllips
+14  A: 
MusiGenesis
This would work quite well, though it's probably too much labor for what we're after right now.
ctacke
It's a fun thing to write. I mostly write software for PDAs, so I run into your problem (too many columns to reasonably display on a screen) all the time.
MusiGenesis
Actually, I think this is a really good idea.
configurator
A: 

I would recommend investigating something other than a flat layout. In my experience, databases have restrictions on column counts and row byte sizes.

  • Your SQL may allow for 1000 columns to be defined.
  • A SQL row cannot exceed the row byte limit.

Each database implementation has a page size (4k / 8k), and a single row must fit within this data size. NULLs are typically freebies. This means that 1000 ints 1000 x 4 bytes will just fit within a 4k page size.

If you are talking data with varchars, then the problem is worse. How many characters are in each column? How many columns can be filled in? If you have 10 characters on average, and your page size is 8k, then you lose the data with a SQL error.

Laugh if you must, but this situation did occur with a particularly long winded typist in a flat datatable that I knew was pushing the limits.

Kieveli
+9  A: 

Ok, what turned out to be the right answer for me was to use the ReportViewer control, but not in any manner documented in MSDN. The problem is that I have dynamic data, so I need a dynamic report, and all of the tutorials, etc. seem to assume you have the luxury of knowing everything at design time so you can point and click your way through a Wizard.

The solution ended up requiring a couple pieces. First, I had to create code to dynamically generate the RDLC that the ReportViewer uses to describe the report layout and what data fields map to what. This is what I came up with:

public static Stream BuildRDLCStream(
    DataSet data, string name, string reportXslPath)
{
  using (MemoryStream schemaStream = new MemoryStream())
  {
    // save the schema to a stream
    data.WriteXmlSchema(schemaStream);
    schemaStream.Seek(0, SeekOrigin.Begin);

    // load it into a Document and set the Name variable
    XmlDocument xmlDomSchema = new XmlDocument();
    xmlDomSchema.Load(schemaStream);        
    xmlDomSchema.DocumentElement.SetAttribute("Name", data.DataSetName);

    // load the report's XSL file (that's the magic)
    XslCompiledTransform xform = new XslCompiledTransform();
    xform.Load(reportXslPath);

    // do the transform
    MemoryStream rdlcStream = new MemoryStream();
    XmlWriter writer = XmlWriter.Create(rdlcStream);
    xform.Transform(xmlDomSchema, writer);
    writer.Close();
    rdlcStream.Seek(0, SeekOrigin.Begin);

    // send back the RDLC
    return rdlcStream;
  }
}

The second piece is an XSL file that I took right off of Dan Shipe's blog. The RDLC code there was pretty worthless as it was all intended for Web use, but the XSL is pure gold. I've put it at the bottom of this post for completeness in case that blog ever goes offline.

Once I has those two pieces, it was simply a matter of creating a Form with a ReportViewer control on it, then using this bit of code to set it up:

ds.DataSetName = name;

Stream rdlc = RdlcEngine.BuildRDLCStream(
    ds, name, "c:\\temp\\rdlc\\report.xsl");

reportView.LocalReport.LoadReportDefinition(rdlc);
reportView.LocalReport.DataSources.Clear();
reportView.LocalReport.DataSources.Add(
    new ReportDataSource(ds.DataSetName, ds.Tables[0]));
reportView.RefreshReport();

The key here is that 'ds' is a DataSet object with a single DataTable in it with the data to be displayed.

Again, for completeness, here's the XSL - sorry about the size:

 <?xml version="1.0"?>
 <!-- Stylesheet for creating ReportViewer RDLC documents -->
 <xsl:stylesheet version="1.0"
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
   xmlns:msxsl="urn:schemas-microsoft-com:xslt"
   xmlns:xs="http://www.w3.org/2001/XMLSchema"
   xmlns:msdata="urn:schemas-microsoft-com:xml-msdata"
   xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner"  xmlns="http://schemas.microsoft.com/sqlserver/reporting/2005/01/reportdefinition"
   >

  <xsl:variable name="mvarName" select="/xs:schema/@Name"/>
  <xsl:variable name="mvarFontSize">8pt</xsl:variable>
  <xsl:variable name="mvarFontWeight">500</xsl:variable>
  <xsl:variable name="mvarFontWeightBold">700</xsl:variable>


  <xsl:template match="/">
   <xsl:apply-templates select="/xs:schema/xs:element/xs:complexType/xs:choice/xs:element/xs:complexType/xs:sequence">
   </xsl:apply-templates>
  </xsl:template>

  <xsl:template match="xs:sequence">
   <Report xmlns:rd="http://schemas.microsoft.com/SQLServer/reporting/reportdesigner" xmlns="http://schemas.microsoft.com/sqlserver/reporting/2005/01/reportdefinition"&gt;
    <BottomMargin>1in</BottomMargin>
    <RightMargin>1in</RightMargin>
    <LeftMargin>1in</LeftMargin>
    <TopMargin>1in</TopMargin>
    <InteractiveHeight>11in</InteractiveHeight>
    <InteractiveWidth>8.5in</InteractiveWidth>
    <Width>6.5in</Width>
    <Language>en-US</Language>
    <rd:DrawGrid>true</rd:DrawGrid>
    <rd:SnapToGrid>true</rd:SnapToGrid>
    <rd:ReportID>7358b654-3ca3-44a0-8677-efe0a55c7c45</rd:ReportID>

    <xsl:call-template name="BuildDataSource">
    </xsl:call-template>

    <xsl:call-template name="BuildDataSet">
    </xsl:call-template>

    <Body>
     <Height>0.50in</Height>
     <ReportItems>
      <Table Name="table1">
       <DataSetName><xsl:value-of select="$mvarName" /></DataSetName>
       <Top>0.5in</Top>
       <Height>0.50in</Height>
       <Header>
        <TableRows>
         <TableRow>
          <Height>0.25in</Height>
          <TableCells>

           <xsl:apply-templates select="xs:element" mode="HeaderTableCell">
           </xsl:apply-templates>

          </TableCells>
         </TableRow>
        </TableRows>
       </Header>
       <Details>
        <TableRows>
         <TableRow>
          <Height>0.25in</Height>
          <TableCells>

           <xsl:apply-templates select="xs:element" mode="DetailTableCell">
           </xsl:apply-templates>

          </TableCells>
         </TableRow>
        </TableRows>
       </Details>
       <TableColumns>

        <xsl:apply-templates select="xs:element" mode="TableColumn">
        </xsl:apply-templates>

       </TableColumns>
      </Table>
     </ReportItems>
    </Body>
   </Report>
  </xsl:template>

  <xsl:template name="BuildDataSource">
   <DataSources>
    <DataSource Name="DummyDataSource">
     <ConnectionProperties>
      <ConnectString/>
      <DataProvider>SQL</DataProvider>
     </ConnectionProperties>
     <rd:DataSourceID>84635ff8-d177-4a25-9aa5-5a921652c79c</rd:DataSourceID>
    </DataSource>
   </DataSources>
  </xsl:template>

  <xsl:template name="BuildDataSet">
   <DataSets>
    <DataSet Name="{$mvarName}">
     <Query>
      <rd:UseGenericDesigner>true</rd:UseGenericDesigner>
      <CommandText/>
      <DataSourceName>DummyDataSource</DataSourceName>
     </Query>
     <Fields>

      <xsl:apply-templates select="xs:element" mode="Field">
      </xsl:apply-templates>

     </Fields>
    </DataSet>
   </DataSets>
  </xsl:template>

  <xsl:template match="xs:element" mode="Field">
   <xsl:variable name="varFieldName"> 
    <xsl:value-of select="@name" />
   </xsl:variable>

   <xsl:variable name="varDataType">
    <xsl:choose>
     <xsl:when test="@type='xs:int'">System.Int32</xsl:when>
     <xsl:when test="@type='xs:string'">System.String</xsl:when>
     <xsl:when test="@type='xs:dateTime'">System.DateTime</xsl:when>
     <xsl:when test="@type='xs:boolean'">System.Boolean</xsl:when>
    </xsl:choose>
   </xsl:variable>

   <Field Name="{$varFieldName}">
    <rd:TypeName><xsl:value-of select="$varDataType"/></rd:TypeName>
    <DataField><xsl:value-of select="$varFieldName"/></DataField>
   </Field>
  </xsl:template>

  <xsl:template match="xs:element" mode="HeaderTableCell">
   <xsl:variable name="varFieldName"> 
    <xsl:value-of select="@name" />
   </xsl:variable>

   <TableCell>
    <ReportItems>
     <Textbox Name="textbox{position()}">
      <rd:DefaultName>textbox<xsl:value-of select="position()"/>
      </rd:DefaultName>
      <Value><xsl:value-of select="$varFieldName"/></Value>
      <CanGrow>true</CanGrow>
      <ZIndex>7</ZIndex>
      <Style>
       <TextAlign>Center</TextAlign>
       <PaddingLeft>2pt</PaddingLeft>
       <PaddingBottom>2pt</PaddingBottom>
       <PaddingRight>2pt</PaddingRight>
       <PaddingTop>2pt</PaddingTop>
       <FontSize><xsl:value-of select="$mvarFontSize"/></FontSize> 
       <FontWeight><xsl:value-of select="$mvarFontWeightBold"/></FontWeight> 
       <BackgroundColor>#000000</BackgroundColor> 
       <Color>#ffffff</Color>
       <BorderColor>
        <Default>#ffffff</Default>
       </BorderColor>
       <BorderStyle>
        <Default>Solid</Default>
       </BorderStyle>
      </Style>
     </Textbox>
    </ReportItems>
   </TableCell>
  </xsl:template>

  <xsl:template match="xs:element" mode="DetailTableCell">
   <xsl:variable name="varFieldName"> 
    <xsl:value-of select="@name" />
   </xsl:variable>

   <TableCell>
    <ReportItems>
     <Textbox Name="{$varFieldName}">
      <rd:DefaultName><xsl:value-of select="$varFieldName"/></rd:DefaultName>
      <Value>=Fields!<xsl:value-of select="$varFieldName"/>.Value</Value>
      <CanGrow>true</CanGrow>
      <ZIndex>7</ZIndex>
      <Style>
       <TextAlign>Left</TextAlign>
       <PaddingLeft>2pt</PaddingLeft>
       <PaddingBottom>2pt</PaddingBottom>
       <PaddingRight>2pt</PaddingRight>
       <PaddingTop>2pt</PaddingTop>
       <FontSize><xsl:value-of select="$mvarFontSize"/></FontSize> 
       <FontWeight><xsl:value-of select="$mvarFontWeight"/></FontWeight> 
       <BackgroundColor>#e0e0e0</BackgroundColor> 
       <Color>#000000</Color> 
       <BorderColor>
        <Default>#ffffff</Default> 
       </BorderColor>
       <BorderStyle>
         <Default>Solid</Default>
       </BorderStyle>
      </Style>
     </Textbox>
    </ReportItems>
   </TableCell>
  </xsl:template>

  <xsl:template match="xs:element" mode="TableColumn">
   <TableColumn>
    <Width>0.75in</Width>
   </TableColumn>
  </xsl:template>

  <xsl:template name="replace-string">
   <xsl:param name="text"/>
   <xsl:param name="from"/>
   <xsl:param name="to"/>
   <xsl:choose>
    <xsl:when test="contains($text, $from)">
     <xsl:variable name="before" select="substring-before($text, $from)"/>
     <xsl:variable name="after" select="substring-after($text, $from)"/>
     <xsl:variable name="prefix" select="concat($before, $to)"/>
     <xsl:value-of select="$before"/>
     <xsl:value-of select="$to"/>
     <xsl:call-template name="replace-string">
      <xsl:with-param name="text" select="$after"/>
      <xsl:with-param name="from" select="$from"/>
      <xsl:with-param name="to" select="$to"/>
     </xsl:call-template>
    </xsl:when>
    <xsl:otherwise>
     <xsl:value-of select="$text"/>
    </xsl:otherwise>
   </xsl:choose>
  </xsl:template>
 </xsl:stylesheet>
ctacke
Is this SO's longest answer? :)
MusiGenesis
A: 

.. to verify that the data is actually going in there.

May be it is outdated, but you could use pixel map where single pixel is representing single cell of table (is screen is more then 1000) or 10 cells for one pixel with zoom region on click.

The color of pixel will be data dependent. It could be black/white for empty/data. Or it could be color to show value grows or decrease with every row. Or red for sudden jumps of data. All anomalies you could catch normally with your eye in data grid.

Then all you need is to catch click coordinates in the area of interest and use small table to show that part of table without any scrolling.

Just click to go back to pixel-map.

Malx
A: 

Given that the user will have to scroll horizontally anyway, you could use a regular data grid showing a reasonable number of columns (say, 50). Then you have a horizontal scrollbar positioned under the grid that selects a subset of columns to show. When the scrollbar is at the left you show columns 1-50, when you click the right arrow you go to 2-51, etc.

This gives you the scrolling capability without ever having to overload a grid control with data. While you would lose the ability to freely cursor around in the table or make large rectangular selections, it doesn't sound like that would be an issue for this application.

Gabe