Unacceptable Performance for Data Export (.ten, Excel, etc.)

Walt · Post by **Walt** » Tue Aug 09, 2005 10:01 pm

TeeChart.NET is extremely slow at exporting moderate amounts of data (40k data points). I am using TeeChart.NET v1, but I have tried this with v2 Evaluation as well and it was still extremely slow.

1. Simply open up the demo app (for v1 or v2) and get to a chart editor.
2. Add a ColorGrid series or Bubble series.
3. Set the data source to "random" and set # of points to 200 for ColorGrid (which for ColorGrid means 200 * 200 = 40,000 points) or 40,000 for Bubble.
4. Notice that ColorGrid renders fast enough, but Bubble renders a little slow compared to the ActiveX control (try zooming with animation or just resize the form).
5. Now, attempt to export the data by saving to a ".ten" file with the data included or by using copy to clipboard in Excel format (and probably any other format).
6. Notice that it takes 3-5 minutes or more, or even crashes (see crash dump below).

Our original application was written using ActiveX v5 using the Bubble series type and has no trouble with large amounts of data. I am experimenting with using the ColorGrid series type because it makes sense for our data (we are currently plotting many square bubbles with 0.5 radius). I find the performance of the .NET control to be disappointing in general, and for data export it is unacceptable.

Are you using a ton of string concatenation to export the data? Your data export code must have some sort of exponential algorithm because it is unbelievably slow. I'm certain that with some effort, you can make a dramatic improvement in this area.

Please look into this issue.

Thanks,

Walt Dell

Here is the crash dump I got trying to export a 40,000 data point ColorGrid series using TeeChart.NET v2:

---------------------------------------------------------------
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.Runtime.InteropServices.ExternalException: The requested clipboard operation failed.
at System.Windows.Forms.Clipboard.SetDataObject(Object data, Boolean copy)
at Steema.TeeChart.Export.DataExportFormat.CopyToClipboard()
at Steema.TeeChart.Editors.Export.ExportEditor.button1_Click(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

************** Loaded Assemblies **************
mscorlib
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.2032
CodeBase: file:///c:/windows/microsoft.net/framework/v1.1.4322/mscorlib.dll
----------------------------------------
TeeChartNetExamples
Assembly Version: 2.0.1992.14012
Win32 Version: 2.0.1992.14012
CodeBase: file:///C:/Program%20Files/Steema%20Software/TeeChart%20for%20.NET%20v2%20Evaluation/Examples/DemoProject/bin/ExecutableDemo/TeeChartNetExamples.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.2032
CodeBase: file:///c:/windows/assembly/gac/system.windows.forms/1.0.5000.0__b77a5c561934e089/system.windows.forms.dll
----------------------------------------
System
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.2032
CodeBase: file:///c:/windows/assembly/gac/system/1.0.5000.0__b77a5c561934e089/system.dll
----------------------------------------
System.Drawing
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.2032
CodeBase: file:///c:/windows/assembly/gac/system.drawing/1.0.5000.0__b03f5f7f11d50a3a/system.drawing.dll
----------------------------------------
AxInterop.SHDocVw
Assembly Version: 1.1.0.0
Win32 Version: 1.1.0.0
CodeBase: file:///C:/Program%20Files/Steema%20Software/TeeChart%20for%20.NET%20v2%20Evaluation/Examples/DemoProject/bin/ExecutableDemo/AxInterop.SHDocVw.DLL
----------------------------------------
Interop.SHDocVw
Assembly Version: 1.1.0.0
Win32 Version: 1.1.0.0
CodeBase: file:///C:/Program%20Files/Steema%20Software/TeeChart%20for%20.NET%20v2%20Evaluation/Examples/DemoProject/bin/ExecutableDemo/Interop.SHDocVw.DLL
----------------------------------------
Accessibility
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.573
CodeBase: file:///c:/windows/assembly/gac/accessibility/1.0.5000.0__b03f5f7f11d50a3a/accessibility.dll
----------------------------------------
TeeChart
Assembly Version: 2.0.1992.14011
Win32 Version: 2.0.1992.14011
CodeBase: file:///C:/Program%20Files/Steema%20Software/TeeChart%20for%20.NET%20v2%20Evaluation/Examples/DemoProject/bin/ExecutableDemo/TeeChart.DLL
----------------------------------------
System.Design
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.2032
CodeBase: file:///c:/windows/assembly/gac/system.design/1.0.5000.0__b03f5f7f11d50a3a/system.design.dll
----------------------------------------
System.Data
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.2032
CodeBase: file:///c:/windows/assembly/gac/system.data/1.0.5000.0__b77a5c561934e089/system.data.dll
----------------------------------------
System.Xml
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.2032
CodeBase: file:///c:/windows/assembly/gac/system.xml/1.0.5000.0__b77a5c561934e089/system.xml.dll
----------------------------------------
System.Web
Assembly Version: 1.0.5000.0
Win32 Version: 1.1.4322.2037
CodeBase: file:///c:/windows/assembly/gac/system.web/1.0.5000.0__b03f5f7f11d50a3a/system.web.dll
----------------------------------------

************** JIT Debugging **************
To enable just in time (JIT) debugging, the config file for this
application or machine (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the machine
rather than being handled by this dialog.

Miguel · Post by **Miguel** » Wed Aug 10, 2005 8:50 am

Hi Walt,

Thanks for your observations. In the current release, 2.0.2040.15119, the results for the same tests (in a P4, 2.5GHz, 512Mb RAM) are:

Color Grid, 200x200, exporting into a .ten file: about 1 second.
Bubble, 40000 points, exporting into a .ten file: about 10 seconds.

Walt · Post by **Walt** » Wed Aug 10, 2005 6:13 pm

Miguel wrote:Thanks for your observations. In the current release, 2.0.2040.15119, the results for the same tests (in a P4, 2.5GHz, 512Mb RAM) are:

Color Grid, 200x200, exporting into a .ten file: about 1 second.
Bubble, 40000 points, exporting into a .ten file: about 10 seconds.

Are you sure that you checked "include series data"? If so, please apply that bug fix to version 1. Did you also test copying the data to the clipboard in Excel format?

I will consider an upgrade to version 2, but it wasn't very long ago that I bought the version 1 license (just before version 2 came out) and I don't like to upgrade just for bug fixes (and I don't think my manager would like to either).

Is it possible for me to download an evaluation copy of version 2.0.2040.15119? My client downloads only lists a "June 2005" edition, and I think that is the version I did my tests with. I will have to review the new features to see if there is something that would justify the upgrade to my management (other than bug fixes).

Thanks.

Miguel · Post by **Miguel** » Thu Aug 11, 2005 3:00 pm

Hi Walt,

Currently we don't have a evaluation version for the latest release. We will probably update it in a few weeks. In any case, until then you can use June version to evaluate the new features of version 2 and see the differences with version 1.

Narcís · Post by **Narcís** » Tue Aug 16, 2005 7:15 am

Hi Walt,

Evaluation version is already up to date to our latest August 2005 release.

Walt · Post by **Walt** » Wed Aug 17, 2005 12:08 am

narcis wrote: Evaluation version is already up to date to our latest August 2005 release.

It was still labeled as "June 2005", but the file size was bigger, so I installed it and the ".ten" export is indeed faster. However, rendering of the bubble series is still slow and export to Excel is still very, very slow.

It would be nice if those issues could also be addressed, however, I have gotten authorization to upgrade to version 2.0, so I will do so. Thanks.

Narcís · Post by **Narcís** » Wed Aug 17, 2005 8:10 am

Hi Walt,

I've been able to reproduce what you report with our latest release and have added those issues to our wish list to be enhanced for future releases.

It was still labeled as "June 2005"

Also could you please be more specific on this? Where can you find this "label"?

Post by **Christopher** » Wed Aug 17, 2005 4:00 pm

Hi Walt,

Yes, you were right, string concatenation was the bottle neck. I've implemented the StringBuilder class in its place meaning that the following code:

Code: Select all

		private void Form1_Load(object sender, System.EventArgs e) 
		{ 
			commander1.Chart = tChart1;
			bubble1.FillSampleValues(40000);
		}

		private System.Timers.Timer timer1 = new System.Timers.Timer(1);
		private int count;

		private void button1_Click(object sender, System.EventArgs e)
		{
			timer1.Elapsed +=new System.Timers.ElapsedEventHandler(timer1_Elapsed);
			count = 0;
			timer1.Start();
			tChart1.Export.Data.Excel.Series = bubble1;
			tChart1.Export.Data.Excel.Save(@"C:\Temp\bubble.xls");
			timer1.Stop();
			label1.Text = count.ToString();
		}

		private void timer1_Elapsed(object sender, System.Timers.ElapsedEventArgs e) 
		{
			++count;
		}

Now runs in 14 milliseconds. I have made similar changes to all Data export formats (text, xml, html and xls) and these changes will be available in the next maintenance release.

Walt · Post by **Walt** » Wed Aug 17, 2005 5:34 pm

Christopher Ireland wrote:Yes, you were right, string concatenation was the bottle neck. I've implemented the StringBuilder class in its place...

Thanks very much. Although I was disappointed with v1 of TeeChart.NET, I am happy to have the responsive technical support on this forum and that makes me (and my management) a loyal customer. Please continue the focus on quality.

Walt · Post by **Walt** » Wed Aug 17, 2005 5:52 pm

narcis wrote:
It was still labeled as "June 2005"
Also could you please be more specific on this? Where can you find this "label"?

Under my client access page for TeeChart.NET v1 (license #8129718). It was an old-style page at this URL:

http://www.steema.com/cgi-bin/teegetR.cgi

(Old-style as in how your site used to look a long time ago.)

Walt · Post by **Walt** » Thu Aug 18, 2005 12:15 am

I have another performance issue I hope you guys can address (it occurs in the latest v2 release). I have been experimenting with using the ColorGrid series to show my data, which works pretty well except for one thing...

I need to display marks when zoomed in on some detailed data (imagine zooming to a 10x10 portion of a 200x200 ColorGrid). I already have code with the bubble series that enables or disables marks depending on the zoom level (and I also scale the font size). This part works pretty well with the bubble series, but enabling marks on the ColorGrid series type significantly slows it down, even when you can only see a 10x10 portion of the grid. I did turn on the "Clipped" setting, but it seems that you are still trying to draw all the marks (and relying on Windows to do the clipping).

Please review your marks clipping algorithm in the ColorGrid series type vs. the Bubble series type.

I think you need an optimization where you don't even try to draw a mark if its data point is totally off the chart (if you wish, you can use this optimization only when "Clipped" is enabled). Only partially shown data points need their marks to be "clipped".

For now, I might be able to work around this issue by dividing my data into many smaller ColorGrid series, then detect when only one of these smaller series is visible and enable marks only on that visible series. This would be a lot of work for me, though, so I would highly appreciate it if you could look at this issue.

Thank you very much for your support.

Post by **Christopher** » Thu Aug 18, 2005 12:17 pm

Hi Walt,

Well, I've managed to get SeriesMark drawing for a 10 x 10 portion of a 200 x 200 ColorGrid from 5690 milliseconds down to 310 milliseconds.

Again, this improvement will be available in the next maintenance release.

Walt · Post by **Walt** » Mon Aug 22, 2005 5:42 pm

Christopher Ireland wrote:Well, I've managed to get SeriesMark drawing for a 10 x 10 portion of a 200 x 200 ColorGrid from 5690 milliseconds down to 310 milliseconds.

Thanks very much!

Walt · Post by **Walt** » Wed Oct 26, 2005 4:59 pm

Christopher Ireland wrote:Again, this improvement will be available in the next maintenance release.

What is the estimated timeframe for the next maintenance release?

Thanks.

Narcís · Post by **Narcís** » Thu Oct 27, 2005 7:39 am

Hi Walt,

Our release policy, approximatelly, consists of 3 maintenance releases and one major release every year. According to this you may expect one release every 3-4 months. Considering that last maintenance release is from 2nd August 2005 a new maintenance release should be out soon. However, I can't still give you a date when will it be out.

To reduce the waiting time for our customers, we started the Debug Build Initiative. So you may want check those versions periodically.