Feeding in large amounts of data from an online database

1 view (last 30 days)
Would appreciate someone pointing me in the right direction:
Currently I am using urlread to feed in data from a password protected database. I read in all the data at once as a block and then start breaking it up with textscan. Currently, I place a limit on the amount of data I'm pulling in. You can see it below (password removed):
url = 'http://assa104/eurofot/csv';
block = urlread(url, 'Authentication', 'Basic', 'Post', {'login', '***', 'passwd', '***', 'trips', '1332244409', 'signals', '1.AVL_BRTORQ_SUM.AVL_BRTORQ_SUM_DVCH', 'limit', '300000','content_type', 'text'});
but when I remove this limit, I get the following errors:
Error using urlreadwrite (line 97)
Error downloading URL. Your network connection may be down
or your proxy settings improperly configured.
Error in urlread (line 36)
[s,status] =
urlreadwrite(mfilename,catchErrors,url,varargin{:});
My question is two-fold: 1. Any idea why I am returning such an error? 2. This method takes a while (almost a minute with my limit at 300k) and I want to read in even more data in the future. Is there an alternative to my current method? Maybe one that doesn't save the whole set in a block at once but one that reads in rows at a time?
I'm not very experienced with urlread, so any assistance would be greatly appreciated.
Thanks in advance.
  1 Comment
Muneer
Muneer on 11 Feb 2014
Here is the message for the exception. Hopefully this sheds some light on the problem.
Java exception occurred:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.grow(Unknown Source)
at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
at java.io.ByteArrayOutputStream.write(Unknown Source)
at com.mathworks.mlwidgets.io.InterruptibleStreamCopier.copyStream(InterruptibleStreamCopier.java:99)
at com.mathworks.mlwidgets.io.InterruptibleStreamCopier.copyStream(InterruptibleStreamCopier.java:76)

Sign in to comment.

Answers (0)

Categories

Find more on Downloads in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!