Re: HTMLUnit-2.23: 100% CPU Utilization and memory issues.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: HTMLUnit-2.23: 100% CPU Utilization and memory issues.

asashour
Hi,

- Please sent to user-list, not dev-list.
- You must subscribe to the list before posting 
- Try to use 2.24 
- Try to profile the application. 
- Provide your complete case (with URL).

Ahmed



----- Forwarded Message -----
From: "[hidden email]" <[hidden email]>
To: [hidden email]
Sent: Tuesday, January 10, 2017 4:19 PM
Subject: Auto-discard notification

----- Forwarded Message -----

The attached message has been automatically discarded.Hi,

We have recently updated HTMLUnit from 2.15 to 2.23 after which we are
facing memory and cpu utilization issues. Below is the code snippet for open
and close connections for webclient. Please review and provide some
feedback:

openSession(){
                        WebClient _webClient = new
WebClient(BrowserVersionFactory.getBrowserVersion(browserVersion));
            _webClient.getOptions().setSSLClientCertificate(url.toURL(), new
String(selfSignedCertificate), "JKS");
            _webClient.getOptions().setUseInsecureSSL(true);
            _webClient.getOptions().setTimeout(Config.getTwbTimeout() * 1000);
            _webClient.getOptions().setCssEnabled(false);
            _webClient.getCookieManager().setCookiesEnabled(true);
            _cookieManager = _webClient.getCookieManager();
            _cookieManager.setCookiesEnabled(true);
            _webClient.getOptions().setJavaScriptEnabled(true);
            _webClient.getOptions().setRedirectEnabled(true);
            _webClient.getOptions().setThrowExceptionOnScriptError(false);
            _webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
            _webClient.setAjaxController(new NicelyResynchronizingAjaxController());
}


closeSession(){
              try{
            if(_webClient != null){
                List<WebWindow> windows = _webClient.getWebWindows();
                for (WebWindow wd : windows) {
                    wd.getJobManager().removeAllJobs();
                }
                _webClient.getCookieManager().clearCookies();
                _webClient.close();
            }
        }catch (Exception e) {
            TPLog.writeLogMessage(Log.WARN, ERROR_LOG+command+e.getMessage(), e);
        }
}


Hi,

We have recently updated HTMLUnit from 2.15 to 2.23 after which we are
facing memory and cpu utilization issues. Below is the code snippet for open
and close connections for webclient. Please review and provide some
feedback:

openSession(){
                        WebClient _webClient = new
WebClient(BrowserVersionFactory.getBrowserVersion(browserVersion));
                        _webClient.getOptions().setSSLClientCertificate(url.toURL(), new
String(selfSignedCertificate), "JKS");
                        _webClient.getOptions().setUseInsecureSSL(true);
                        _webClient.getOptions().setTimeout(Config.getTwbTimeout() * 1000);
                        _webClient.getOptions().setCssEnabled(false);
                        _webClient.getCookieManager().setCookiesEnabled(true);
                        _cookieManager = _webClient.getCookieManager();
                        _cookieManager.setCookiesEnabled(true);
                        _webClient.getOptions().setJavaScriptEnabled(true);
                        _webClient.getOptions().setRedirectEnabled(true);
                        _webClient.getOptions().setThrowExceptionOnScriptError(false);
                        _webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
                        _webClient.setAjaxController(new NicelyResynchronizingAjaxController());
}


closeSession(){
               try{
                        if(_webClient != null){
                                List<WebWindow> windows = _webClient.getWebWindows();
                                for (WebWindow wd : windows) {
                                    wd.getJobManager().removeAllJobs();
                                }
                                _webClient.getCookieManager().clearCookies();
                                _webClient.close();
                        }
                }catch (Exception e) {
                        TPLog.writeLogMessage(Log.WARN, ERROR_LOG+command+e.getMessage(), e);
                }
}




--
View this message in context: http://htmlunit.10904.n7.nabble.com/HTMLUnit-2-23-100-CPU-Utilization-and-memory-issues-tp40582.html
Sent from the HtmlUnit - Dev mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: HTMLUnit-2.23: 100% CPU Utilization and memory issues.

manojsakarwal
Hi All,

Yesterday, we upgraded HTML Unit to 2.24 as suggested but we are still seeing the same results. Memory usage is higher, near to 95% and around 98% of CPU usage is observed after we upgraded from HTML Unit 2.15.

We have also tried to solve it from our side by adding    
 wd.getJobManager().shutdown();

Following is the new change we have made it close method to solve it.

public void closeSession()
        {
                String command = "CloseSession";
                try{
                        if(_webClient != null){
                                List<WebWindow> windows = _webClient.getWebWindows();
                                for (WebWindow wd : windows) {
//    wd.getJobManager().removeAllJobs();  //removed this to replace with shutdown.
                                    wd.getJobManager().shutdown();
                                }
                                _webClient.getCookieManager().clearCookies();
                        }
                }catch (Exception e) {
                        Log.writeLogMessage(Log.WARN, ERROR_LOG+command+e.getMessage(), e);
                }finally{
                        if(_webClient != null){
                                _webClient.close();
                        }
                }
        }

but this also did not help.



Please help us in troubleshooting.

Thanks,
Manoj


Reply | Threaded
Open this post in threaded view
|

Re: HTMLUnit-2.23: 100% CPU Utilization and memory issues.

adam afeltowicz-2
Hi,

I've also observed higher CPU usage since mouseover event started to clean computed stylesheet cache (after looking into commits I think it is connected with :hover pseudoclass handling). I tried to modify code to clean only some elements from cache, but this simply doesn't worked when application has complex CSS. Maybe this is same issue (I haven't yet registered a bug)?

Adam

2017-02-02 12:31 GMT+01:00 manojsakarwal <[hidden email]>:
Hi All,

Yesterday, we upgraded HTML Unit to 2.24 as suggested but we are still
seeing the same results. Memory usage is higher, near to 95% and around 98%
of CPU usage is observed after we upgraded from HTML Unit 2.15.

We have also tried to solve it from our side by adding
 wd.getJobManager().shutdown();

Following is the new change we have made it close method to solve it.

public void closeSession()
        {
                String command = "CloseSession";
                try{
                        if(_webClient != null){
                                List<WebWindow> windows = _webClient.getWebWindows();
                                for (WebWindow wd : windows) {
//                                  wd.getJobManager().removeAllJobs();  //removed this to replace
with shutdown.
                                    wd.getJobManager().shutdown();
                                }
                                _webClient.getCookieManager().clearCookies();
                        }
                }catch (Exception e) {
                        Log.writeLogMessage(Log.WARN, ERROR_LOG+command+e.getMessage(), e);
                }finally{
                        if(_webClient != null){
                                _webClient.close();
                        }
                }
        }

but this also did not help.

<http://htmlunit.10904.n7.nabble.com/file/n40902/Pasted_image_at_2017_02_01_10_441AM.png>

Please help us in troubleshooting.

Thanks,
Manoj






--
View this message in context: http://htmlunit.10904.n7.nabble.com/Re-HTMLUnit-2-23-100-CPU-Utilization-and-memory-issues-tp40587p40902.html
Sent from the HtmlUnit - General mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: HTMLUnit-2.23: 100% CPU Utilization and memory issues.

manojsakarwal
Hi,

We did profiling of our systems to investigate the memory leaks with HTML Unit(2.24). We found that one of the object of HTML Page is using 2 GB of memory. Following are the screenshots. Can you help us to find out why it is eating that much of memory.

HTMLPageRetained HeapClass hirerchy

Thanks,
Manoj Sakarwal
Reply | Threaded
Open this post in threaded view
|

Re: HTMLUnit-2.23: 100% CPU Utilization and memory issues.

Ronald Brill
Hi Manjo,

i know that hunting for memory leaks is not that simple. But believe me it is nearly impossible to do based on screen shots.
HtmlUnit is maintained in our spare time - this implies that you have to help us a bit if you like to have any support.

1. please try to use the latest SNAPSHOT build. The quality of the snapshot builds is the same as the release build because they are done only if the whole test suite is green.
There is no time for us to support older versions

2. We need more details. In your case you have to figure out, which collection is your problem. And we have to know, which entries are inside and why this is not correct. And we need the page
source code to be able to understand your problem.

3. In general there is a good chance to get a fix if you are able to nail down the problem to the root cause.

Hope that helps - waiting for your details to make HtmlUnit better


    RBRi
    --------------------------
    WETATOR
    Smart web application testing
    http://www.wetator.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: HTMLUnit-2.23: 100% CPU Utilization and memory issues.

Preston
This post has NOT been accepted by the mailing list yet.
Hi Ronald,

I've been working with Manoj to better understand what's going on with this potential memory leak. I'm fairly new to htmlunit so some of the questions I'm asking are to help orient me.

We an instance of htmlpage consuming ~2.3 GB of memory. The memory is mostly held in the DomListeners and AttributeListeners collections. They have the same number of instances (~33m) so whatever is adding to one collection is also adding to the other.

The interesting part is that it's only one instance of htmlpage that's having this issue. I don't know what's creating it or who owns the thread. Our app creates many instances of webclient but then tears them down. I would think if there was a problem in that we'd see many instances of htmlpage but there's only the one when we inspect the heapdump. Based on the size of the collections, it's as if this instance is hanging out somewhere and continually getting new listeners added to it.

Lastly, I'll work towards implementing your suggestions above.

Questions:
1) Based on what you know about html unit is there any sort of singleton or shared htmlpage?
2) It looks like its related to javascript. What creates this thread?
3) We don't have any code that explicitly adds to DomListners or AttributeListeners, what inside htmlunit might be adding to those collections?