Can MATLAB be slowed down by recieving a lot of data via Serial Port?

So I'm having some issues with a National Instruments DAQ device. I'll try to keep this question focused though, and I'll add more details if they become necessary or relevant.

I can read 1000 samples of data in 0.22 seconds. It takes 50.32 seconds to read 5000 data points. For 10,000 it's 126.74 seconds. I'm storing all of these data into one matrix.

Is MATLAB to blame for this problem, or is it the code or hardware?

Thanks.