How do I use webread with HTTPS?

I would like to read JSON data from a URL in MATLAB. Specifically, I am interested in pulling data from the USDA. But I get a "Connection reset" error.

>> webread('https://quickstats.nass.usda.gov/api/api_GET/?key=1C757E50-5169-30CC-BEFD-40A5C3E2A43D&format=JSON&or_desc=CROPS&domain_desc=TOTAL&agg_level_desc=COUNTY&state_name=ALABAMA&county_name=AUTAUGA&year=2012')
Error using webread (line 128)
The server returned the message: "Connection reset" for URL,
'https://quickstats.nass.usda.gov/api/api_GET/?key=1C757E50-5169-30CC-BEFD-40A5C3E2A43D&format=JSON&sector_desc=CROPS&domain_desc=TOTAL&agg_level_desc=COUNTY&state_name=ALABAMA&county_name=AUTAUGA&year=2012'
(with HTTP response code unknown).

Calling webread over the HTTP protocol works just fine:

>> webread('http://maps.googleapis.com/maps/api/geocode/json?address=google')
ans = 
    results: [1x1 struct]
    status: 'OK'

Unfortunately the USDA only offers data via HTTPS. Calling the Google Maps API over HTTPS gives me a different error.

>> webread('https://maps.googleapis.com/maps/api/geocode/json?address=google')
Error using webread (line 128)
The server returned the message: "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid
certification path to requested target" for URL, 'https://maps.googleapis.com/maps/api/geocode/json?address=google' (with HTTP response code unknown).

What must I do to use MATALB's webread with HTTPS URLs?

Answer 1

[Guillaume]

Both of your https webread work for me (well, the first one errors but only because of the typo § instead of & in the url, which is completely irrelevant to your problem).

So, the reason it does not work for you is outside of matlab. Most likely, there is a proxy server between you and the internet and this proxy alters/filters the request incorrectly. Possibly the proxy does not recognise matlab's user agent so you could try overriding that with weboptions. E.g. if the connection works from your browser you could grab its user agent string, and use that:

webread('https://quickstats.nass.usda.gov/api/api_GET/?key=1C757E50-5169-30CC-BEFD-40A5C3E2A43D&format=JSON&or_desc=CROPS&domain_desc=TOTAL&agg_level_desc=COUNTY&state_name=ALABAMA&county_name=AUTAUGA&year=2012', ...
         weboptions('UserAgent', 'http://www.whoishostingthis.com/tools/user-agent/'))
%with typo fixed

But otherwise, you'll have to contact your IT people for help as it's something of theirs interfering with your connection.

[Tony Florida]

Unfortunately, using weboptions as you have described doesn't work for me.

The connection does indeed work from my browser and also from Python using urllib.request.urlopen. Because I am able to pull the data by other means, I don't think the issue cold be related to a local network interference.

[Guillaume]

Well, the googlapi error is telling you that the certificate for the google site could not be validated. Since it can be assumed that google is not sending you an invalid certificate, it's either something between you and google intercepting traffic and replacing the certificate, something done by some proxies, or the Java certification chain on your computer is not right.

You could install whatever certificate you're receiving into the certificate store as explained here but then you may as well be using http.

[Tony Florida]

Prior to posing my question here, I already tried installing the certificate into my certificate store by following your linked instructions and that still didn't work for me. I am going to work with IT to see if we can pinpoint proxies or the Java cert chain as the issue.