HtmlUnit not work for a page in weather.com

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

HtmlUnit not work for a page in weather.com

Xue-Feng Yang
Hi, I used HtmlUnit for getting some other web pages. It works great.

However, when I tried https://weather.com/weather/monthly/l/27560:4:US , I got the page source code without running the javascript code.

Here are the summary of my system:

OS: win 10
Java: jdk1.8.0_131
HtmlUnit: htmlunit-2.27-bin

eclipse-debug gives the result HtmlUnit got. The main code is as follows:

        webClient = new WebClient(BrowserVersion.FIREFOX_45);
        webClient.getOptions().setTimeout(600 * 1000);
        webClient.waitForBackgroundJavaScript(600 * 1000);
        webClient.getOptions().setRedirectEnabled(true);
        webClient.getOptions().setJavaScriptEnabled(true);
        webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setCssEnabled(false);

        htmlPage = webClient.getPage(_url);
        page = htmlPage.asXml();

The return page is a subset of view-source in Firefox. I found the JavaScript inspector in Firefox has the full HTML tree.

My question is how to get the HTML tree of this page by use of HtmlUnit.

Thanks,

Xuefeng

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user