What is the performance difference of processing large csv files vs isf files?

Hi,

I am doing some research and I am looking to find the speed difference of reading and performing calculations on a large file(~5 million data points) saved in an isf format versus the same file saved in a csv format. Both would be generated from a Tektronix oscilloscope. I would like to perform FFT on both formats and I am curious to know how much faster it would be with the isf file.

Thank you!

Answer 1

[dpb]

Give it a try and see...no klew what an 'isf' format is, specifically. If it's an unformatted binary transfer it'll undoubtedly be quite a lot quicker; if not, perhaps "not so much".

If you're looking for speed, find the way to do unformatted ("stream" or "binary") transfers.

[Noah]

Thank you for the information! This was going to be my next step once I get my hands on a scope. I just wanted to see if anyone has had any previous experience with this or any advice I should know about.

[dpb]

From the Tek site...

:WFMPRE:BYT_NR 2;BIT_NR 16;ENCDG BIN;BN_FMT RI;BYT_OR MSB;NR_PT 10000;WFID "Ch1, DC coupling, 2.0E0 V/div, 1.0E-5 s/div, 10000 points, Sample mode";PT_FMT Y;XINCR 1.0E-8;PT_OFF 0;XZERO 3.5E-4;XUNIT "s";YMULT 3.125E-4;YZERO 0.0E0;YOFF 0.0E0;YUNIT "V";:CURVE #520000<binary data>

The values from the “:” at the beginning of the file to the “;” just before the “:curve” are the preamble. The binary block begins with “:CURVE #”. The next value is referred to in the manual as X. This is the ASCII representation of the number of bytes that follow that represent the record length. The next bytes, referred to in the manual as the YYY bytes are the ASCII representation of the record length field. In this example the 5 tells us that the next 5 bytes represent the record length. The next 5 bytes are 20000, which tells us that the number of bytes returned in the curve will be 20,000. Since we can see in the preamble that the number of bytes for each acquired sample point is 2 (BYT_NR 2) we know that the number of sample acquired was 20,000 / 2 or 10,000 samples.

Hence, while you'll have to either find a routine posted on the File Exchange (I didn't look) or search the Tek site to see if they do have a m-file for Matlab (didn't see anything in a quick search so my guess is "not"), if you do take the time to write a function to parse the header and then suck up the binary data undoubtedly that will be quite a lot faster than the "convert to csv then parse" route. Doesn't look too hard...