ScriptableObjects piling up on the heap

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

ScriptableObjects piling up on the heap

jason franklin-stokes
Hi

I am looking through about 30 pages (webclient.getPage(someurl))
Over time I seem to have ScriptableObjects (slots) collecting on the heap which are not getting removed - and yep I always use  webclient.closeAllWindows.

The scriptableObjects seem to be holding onto the instances of pages which also pile up on the heap.  

Any ideas what might be causing this - or if there is anything I can do to flush them (i 100% totally do a closeAllWindows after looking at a page)

best and thanks for any help.

Jason.


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: ScriptableObjects piling up on the heap

reinhard schwab
hi,

i'm crawling several thousands of pages in one batch job and
interpreting it with htmlunit, not disabling javascript.
the only memory issue i have encountered so far has been some creative
usage of top level windows, but
this has been disabled now.
so far i have not looked into the deep details about memory usage as my
box is capable of handling it. 2 gb memory.

which version of htmlunit are you using?
can you provide more details about the heap and the heap size?
do you cache anything in memory?

reinhard

jason franklin-stokes schrieb:

> Hi
>
> I am looking through about 30 pages (webclient.getPage(someurl))
> Over time I seem to have ScriptableObjects (slots) collecting on the heap which are not getting removed - and yep I always use  webclient.closeAllWindows.
>
> The scriptableObjects seem to be holding onto the instances of pages which also pile up on the heap.  
>
> Any ideas what might be causing this - or if there is anything I can do to flush them (i 100% totally do a closeAllWindows after looking at a page)
>
> best and thanks for any help.
>
> Jason.
>
>
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Htmlunit-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>  


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: ScriptableObjects piling up on the heap

RuralHunter
In reply to this post by jason franklin-stokes
I am doing this maybe you want to have a try:

protected void closeWebClient(WebClient wc) {
         List<WebWindow> windows = wc.getWebWindows();
         for (WebWindow wd : windows) {
             //wd.getThreadManager().interruptAll();
             wd.getJobManager().removeAllJobs();
         }
         wc.closeAllWindows();
}

2010-3-21 18:47, jason franklin-stokes wrote:

> Hi
>
> I am looking through about 30 pages (webclient.getPage(someurl))
> Over time I seem to have ScriptableObjects (slots) collecting on the heap which are not getting removed - and yep I always use  webclient.closeAllWindows.
>
> The scriptableObjects seem to be holding onto the instances of pages which also pile up on the heap.
>
> Any ideas what might be causing this - or if there is anything I can do to flush them (i 100% totally do a closeAllWindows after looking at a page)
>
> best and thanks for any help.
>
> Jason.
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Htmlunit-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>    


------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: ScriptableObjects piling up on the heap

jason franklin-stokes
Thanks a million - that was the right pointer for me.

I am using htmlunit 2.7 and - this was the code that did it for me.

* * * jruby code * * *

          def destroy
            webclient.getWebWindows.each {  |window|
              window.getJobManager.removeAllJobs
              window.getJobManager.shutdown
             }
            webclient.closeAllWindows
          end

* * * end * * *

I could not find any references to window.getThreadManager do I left it out (seems to work fine without it).

Now I have very healthy looking garbage collection cycles.

The only thing that gets me is that I thought closeAllWindows was supposed to kill all the java threads and get rid of the scriptable objects - why is that not working as expected??

Also

window.getJobManager

is subject to change at anytime, so its not exactly a stable candidate to use for the next releases......

any thoughts?

Jason.




On Mar 22, 2010, at 4:40 AM, Rural Hunter wrote:

> I am doing this maybe you want to have a try:
>
> protected void closeWebClient(WebClient wc) {
>         List<WebWindow> windows = wc.getWebWindows();
>         for (WebWindow wd : windows) {
>             //wd.getThreadManager().interruptAll();
>             wd.getJobManager().removeAllJobs();
>         }
>         wc.closeAllWindows();
> }
>
> 2010-3-21 18:47, jason franklin-stokes wrote:
>> Hi
>>
>> I am looking through about 30 pages (webclient.getPage(someurl))
>> Over time I seem to have ScriptableObjects (slots) collecting on the heap which are not getting removed - and yep I always use  webclient.closeAllWindows.
>>
>> The scriptableObjects seem to be holding onto the instances of pages which also pile up on the heap.
>>
>> Any ideas what might be causing this - or if there is anything I can do to flush them (i 100% totally do a closeAllWindows after looking at a page)
>>
>> best and thanks for any help.
>>
>> Jason.
>>
>>
>> ------------------------------------------------------------------------------
>> Download Intel&#174; Parallel Studio Eval
>> Try the new software tools for yourself. Speed compiling, find bugs
>> proactively, and fine-tune applications for parallel performance.
>> See why Intel Parallel Studio got high marks during beta.
>> http://p.sf.net/sfu/intel-sw-dev
>> _______________________________________________
>> Htmlunit-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>
>>
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Htmlunit-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user


------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: ScriptableObjects piling up on the heap

RuralHunter
I agree with you that this should be done inside of  
Webclient.closeAllWindows(). I didn't check source so I'm not sure
what's done there for these threads. maybe developers wathing this mail
list can give an answer.

On 2010-3-22 17:27, jason franklin-stokes wrote:

> Thanks a million - that was the right pointer for me.
>
> I am using htmlunit 2.7 and - this was the code that did it for me.
>
> * * * jruby code * * *
>
>            def destroy
>              webclient.getWebWindows.each {  |window|
>                window.getJobManager.removeAllJobs
>                window.getJobManager.shutdown
>               }
>              webclient.closeAllWindows
>            end
>
> * * * end * * *
>
> I could not find any references to window.getThreadManager do I left it out (seems to work fine without it).
>
> Now I have very healthy looking garbage collection cycles.
>
> The only thing that gets me is that I thought closeAllWindows was supposed to kill all the java threads and get rid of the scriptable objects - why is that not working as expected??
>
> Also
>
> window.getJobManager
>
> is subject to change at anytime, so its not exactly a stable candidate to use for the next releases......
>
> any thoughts?
>
> Jason.
>
>
>
>
> On Mar 22, 2010, at 4:40 AM, Rural Hunter wrote:
>
>    
>> I am doing this maybe you want to have a try:
>>
>> protected void closeWebClient(WebClient wc) {
>>          List<WebWindow>  windows = wc.getWebWindows();
>>          for (WebWindow wd : windows) {
>>              //wd.getThreadManager().interruptAll();
>>              wd.getJobManager().removeAllJobs();
>>          }
>>          wc.closeAllWindows();
>> }
>>
>> 2010-3-21 18:47, jason franklin-stokes wrote:
>>      
>>> Hi
>>>
>>> I am looking through about 30 pages (webclient.getPage(someurl))
>>> Over time I seem to have ScriptableObjects (slots) collecting on the heap which are not getting removed - and yep I always use  webclient.closeAllWindows.
>>>
>>> The scriptableObjects seem to be holding onto the instances of pages which also pile up on the heap.
>>>
>>> Any ideas what might be causing this - or if there is anything I can do to flush them (i 100% totally do a closeAllWindows after looking at a page)
>>>
>>> best and thanks for any help.
>>>
>>> Jason.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download Intel&#174; Parallel Studio Eval
>>> Try the new software tools for yourself. Speed compiling, find bugs
>>> proactively, and fine-tune applications for parallel performance.
>>> See why Intel Parallel Studio got high marks during beta.
>>> http://p.sf.net/sfu/intel-sw-dev
>>> _______________________________________________
>>> Htmlunit-user mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>
>>>
>>>        
>>
>> ------------------------------------------------------------------------------
>> Download Intel&#174; Parallel Studio Eval
>> Try the new software tools for yourself. Speed compiling, find bugs
>> proactively, and fine-tune applications for parallel performance.
>> See why Intel Parallel Studio got high marks during beta.
>> http://p.sf.net/sfu/intel-sw-dev
>> _______________________________________________
>> Htmlunit-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>      
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Htmlunit-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>    


------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: ScriptableObjects piling up on the heap

jason franklin-stokes
It seems to be that some jobs are not being removed after window.getJobManager.removeAllJobs

I added an exception for such cases and it gets thrown occasionally. (see below)

I am assuming that your line, wd.getThreadManager().interruptAll(); makes sure that they can be removed,
unfortunately I cannot find anything like this is the 2.7 API.

Any Ideas? Thanks a million, Jason.

          def destroy
            @webclient.getWebWindows.each do |window|
              window.getJobManager.removeAllJobs
              raise "JOBS STILL RUNNING!!" if window.getJobManager.jobCount > 0
              window.getJobManager.shutdown
            end
            @webclient.closeAllWindows
          end






On Mar 22, 2010, at 2:50 PM, Rural Hunter wrote:

> I agree with you that this should be done inside of  
> Webclient.closeAllWindows(). I didn't check source so I'm not sure
> what's done there for these threads. maybe developers wathing this mail
> list can give an answer.
>
> On 2010-3-22 17:27, jason franklin-stokes wrote:
>> Thanks a million - that was the right pointer for me.
>>
>> I am using htmlunit 2.7 and - this was the code that did it for me.
>>
>> * * * jruby code * * *
>>
>>           def destroy
>>             webclient.getWebWindows.each {  |window|
>>               window.getJobManager.removeAllJobs
>>               window.getJobManager.shutdown
>>              }
>>             webclient.closeAllWindows
>>           end
>>
>> * * * end * * *
>>
>> I could not find any references to window.getThreadManager do I left it out (seems to work fine without it).
>>
>> Now I have very healthy looking garbage collection cycles.
>>
>> The only thing that gets me is that I thought closeAllWindows was supposed to kill all the java threads and get rid of the scriptable objects - why is that not working as expected??
>>
>> Also
>>
>> window.getJobManager
>>
>> is subject to change at anytime, so its not exactly a stable candidate to use for the next releases......
>>
>> any thoughts?
>>
>> Jason.
>>
>>
>>
>>
>> On Mar 22, 2010, at 4:40 AM, Rural Hunter wrote:
>>
>>
>>> I am doing this maybe you want to have a try:
>>>
>>> protected void closeWebClient(WebClient wc) {
>>>         List<WebWindow>  windows = wc.getWebWindows();
>>>         for (WebWindow wd : windows) {
>>>             //wd.getThreadManager().interruptAll();
>>>             wd.getJobManager().removeAllJobs();
>>>         }
>>>         wc.closeAllWindows();
>>> }
>>>
>>> 2010-3-21 18:47, jason franklin-stokes wrote:
>>>
>>>> Hi
>>>>
>>>> I am looking through about 30 pages (webclient.getPage(someurl))
>>>> Over time I seem to have ScriptableObjects (slots) collecting on the heap which are not getting removed - and yep I always use  webclient.closeAllWindows.
>>>>
>>>> The scriptableObjects seem to be holding onto the instances of pages which also pile up on the heap.
>>>>
>>>> Any ideas what might be causing this - or if there is anything I can do to flush them (i 100% totally do a closeAllWindows after looking at a page)
>>>>
>>>> best and thanks for any help.
>>>>
>>>> Jason.
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Download Intel&#174; Parallel Studio Eval
>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>> proactively, and fine-tune applications for parallel performance.
>>>> See why Intel Parallel Studio got high marks during beta.
>>>> http://p.sf.net/sfu/intel-sw-dev
>>>> _______________________________________________
>>>> Htmlunit-user mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>
>>>>
>>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download Intel&#174; Parallel Studio Eval
>>> Try the new software tools for yourself. Speed compiling, find bugs
>>> proactively, and fine-tune applications for parallel performance.
>>> See why Intel Parallel Studio got high marks during beta.
>>> http://p.sf.net/sfu/intel-sw-dev
>>> _______________________________________________
>>> Htmlunit-user mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>
>>
>> ------------------------------------------------------------------------------
>> Download Intel&#174; Parallel Studio Eval
>> Try the new software tools for yourself. Speed compiling, find bugs
>> proactively, and fine-tune applications for parallel performance.
>> See why Intel Parallel Studio got high marks during beta.
>> http://p.sf.net/sfu/intel-sw-dev
>> _______________________________________________
>> Htmlunit-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>
>>
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Htmlunit-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user


------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: ScriptableObjects piling up on the heap

RuralHunter
No idea....wd.getThreadManager().interruptAll() is from old HtmlUnit
API. I believe it's not in HtmlUnit 2.7 any more.
wd.getJobManager().removeAllJobs() is the one I found that seems to do
the similar job.

On 2010-3-22 23:20, jason franklin-stokes wrote:

> It seems to be that some jobs are not being removed after window.getJobManager.removeAllJobs
>
> I added an exception for such cases and it gets thrown occasionally. (see below)
>
> I am assuming that your line, wd.getThreadManager().interruptAll(); makes sure that they can be removed,
> unfortunately I cannot find anything like this is the 2.7 API.
>
> Any Ideas? Thanks a million, Jason.
>
>            def destroy
>              @webclient.getWebWindows.each do |window|
>                window.getJobManager.removeAllJobs
>                raise "JOBS STILL RUNNING!!" if window.getJobManager.jobCount>  0
>                window.getJobManager.shutdown
>              end
>              @webclient.closeAllWindows
>            end
>
>
>
>
>
>
> On Mar 22, 2010, at 2:50 PM, Rural Hunter wrote:
>
>    
>> I agree with you that this should be done inside of
>> Webclient.closeAllWindows(). I didn't check source so I'm not sure
>> what's done there for these threads. maybe developers wathing this mail
>> list can give an answer.
>>
>> On 2010-3-22 17:27, jason franklin-stokes wrote:
>>      
>>> Thanks a million - that was the right pointer for me.
>>>
>>> I am using htmlunit 2.7 and - this was the code that did it for me.
>>>
>>> * * * jruby code * * *
>>>
>>>            def destroy
>>>              webclient.getWebWindows.each {  |window|
>>>                window.getJobManager.removeAllJobs
>>>                window.getJobManager.shutdown
>>>               }
>>>              webclient.closeAllWindows
>>>            end
>>>
>>> * * * end * * *
>>>
>>> I could not find any references to window.getThreadManager do I left it out (seems to work fine without it).
>>>
>>> Now I have very healthy looking garbage collection cycles.
>>>
>>> The only thing that gets me is that I thought closeAllWindows was supposed to kill all the java threads and get rid of the scriptable objects - why is that not working as expected??
>>>
>>> Also
>>>
>>> window.getJobManager
>>>
>>> is subject to change at anytime, so its not exactly a stable candidate to use for the next releases......
>>>
>>> any thoughts?
>>>
>>> Jason.
>>>
>>>
>>>
>>>
>>> On Mar 22, 2010, at 4:40 AM, Rural Hunter wrote:
>>>
>>>
>>>        
>>>> I am doing this maybe you want to have a try:
>>>>
>>>> protected void closeWebClient(WebClient wc) {
>>>>          List<WebWindow>   windows = wc.getWebWindows();
>>>>          for (WebWindow wd : windows) {
>>>>              //wd.getThreadManager().interruptAll();
>>>>              wd.getJobManager().removeAllJobs();
>>>>          }
>>>>          wc.closeAllWindows();
>>>> }
>>>>
>>>> 2010-3-21 18:47, jason franklin-stokes wrote:
>>>>
>>>>          
>>>>> Hi
>>>>>
>>>>> I am looking through about 30 pages (webclient.getPage(someurl))
>>>>> Over time I seem to have ScriptableObjects (slots) collecting on the heap which are not getting removed - and yep I always use  webclient.closeAllWindows.
>>>>>
>>>>> The scriptableObjects seem to be holding onto the instances of pages which also pile up on the heap.
>>>>>
>>>>> Any ideas what might be causing this - or if there is anything I can do to flush them (i 100% totally do a closeAllWindows after looking at a page)
>>>>>
>>>>> best and thanks for any help.
>>>>>
>>>>> Jason.
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Download Intel&#174; Parallel Studio Eval
>>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>>> proactively, and fine-tune applications for parallel performance.
>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>> _______________________________________________
>>>>> Htmlunit-user mailing list
>>>>> [hidden email]
>>>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> ------------------------------------------------------------------------------
>>>> Download Intel&#174; Parallel Studio Eval
>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>> proactively, and fine-tune applications for parallel performance.
>>>> See why Intel Parallel Studio got high marks during beta.
>>>> http://p.sf.net/sfu/intel-sw-dev
>>>> _______________________________________________
>>>> Htmlunit-user mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>
>>>>          
>>> ------------------------------------------------------------------------------
>>> Download Intel&#174; Parallel Studio Eval
>>> Try the new software tools for yourself. Speed compiling, find bugs
>>> proactively, and fine-tune applications for parallel performance.
>>> See why Intel Parallel Studio got high marks during beta.
>>> http://p.sf.net/sfu/intel-sw-dev
>>> _______________________________________________
>>> Htmlunit-user mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>
>>>
>>>        
>>
>> ------------------------------------------------------------------------------
>> Download Intel&#174; Parallel Studio Eval
>> Try the new software tools for yourself. Speed compiling, find bugs
>> proactively, and fine-tune applications for parallel performance.
>> See why Intel Parallel Studio got high marks during beta.
>> http://p.sf.net/sfu/intel-sw-dev
>> _______________________________________________
>> Htmlunit-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>      
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Htmlunit-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>    


------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: ScriptableObjects piling up on the heap

jason franklin-stokes

ok - it seems to be that it is enough to call :

window.getJobManager.shutdown for each window.

the shutdown method correctly stops the jobs and removes them.




On Mar 22, 2010, at 4:35 PM, Rural Hunter wrote:

> No idea....wd.getThreadManager().interruptAll() is from old HtmlUnit
> API. I believe it's not in HtmlUnit 2.7 any more.
> wd.getJobManager().removeAllJobs() is the one I found that seems to do
> the similar job.
>
> On 2010-3-22 23:20, jason franklin-stokes wrote:
>> It seems to be that some jobs are not being removed after window.getJobManager.removeAllJobs
>>
>> I added an exception for such cases and it gets thrown occasionally. (see below)
>>
>> I am assuming that your line, wd.getThreadManager().interruptAll(); makes sure that they can be removed,
>> unfortunately I cannot find anything like this is the 2.7 API.
>>
>> Any Ideas? Thanks a million, Jason.
>>
>>           def destroy
>>             @webclient.getWebWindows.each do |window|
>>               window.getJobManager.removeAllJobs
>>               raise "JOBS STILL RUNNING!!" if window.getJobManager.jobCount>  0
>>               window.getJobManager.shutdown
>>             end
>>             @webclient.closeAllWindows
>>           end
>>
>>
>>
>>
>>
>>
>> On Mar 22, 2010, at 2:50 PM, Rural Hunter wrote:
>>
>>
>>> I agree with you that this should be done inside of
>>> Webclient.closeAllWindows(). I didn't check source so I'm not sure
>>> what's done there for these threads. maybe developers wathing this mail
>>> list can give an answer.
>>>
>>> On 2010-3-22 17:27, jason franklin-stokes wrote:
>>>
>>>> Thanks a million - that was the right pointer for me.
>>>>
>>>> I am using htmlunit 2.7 and - this was the code that did it for me.
>>>>
>>>> * * * jruby code * * *
>>>>
>>>>           def destroy
>>>>             webclient.getWebWindows.each {  |window|
>>>>               window.getJobManager.removeAllJobs
>>>>               window.getJobManager.shutdown
>>>>              }
>>>>             webclient.closeAllWindows
>>>>           end
>>>>
>>>> * * * end * * *
>>>>
>>>> I could not find any references to window.getThreadManager do I left it out (seems to work fine without it).
>>>>
>>>> Now I have very healthy looking garbage collection cycles.
>>>>
>>>> The only thing that gets me is that I thought closeAllWindows was supposed to kill all the java threads and get rid of the scriptable objects - why is that not working as expected??
>>>>
>>>> Also
>>>>
>>>> window.getJobManager
>>>>
>>>> is subject to change at anytime, so its not exactly a stable candidate to use for the next releases......
>>>>
>>>> any thoughts?
>>>>
>>>> Jason.
>>>>
>>>>
>>>>
>>>>
>>>> On Mar 22, 2010, at 4:40 AM, Rural Hunter wrote:
>>>>
>>>>
>>>>
>>>>> I am doing this maybe you want to have a try:
>>>>>
>>>>> protected void closeWebClient(WebClient wc) {
>>>>>         List<WebWindow>   windows = wc.getWebWindows();
>>>>>         for (WebWindow wd : windows) {
>>>>>             //wd.getThreadManager().interruptAll();
>>>>>             wd.getJobManager().removeAllJobs();
>>>>>         }
>>>>>         wc.closeAllWindows();
>>>>> }
>>>>>
>>>>> 2010-3-21 18:47, jason franklin-stokes wrote:
>>>>>
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I am looking through about 30 pages (webclient.getPage(someurl))
>>>>>> Over time I seem to have ScriptableObjects (slots) collecting on the heap which are not getting removed - and yep I always use  webclient.closeAllWindows.
>>>>>>
>>>>>> The scriptableObjects seem to be holding onto the instances of pages which also pile up on the heap.
>>>>>>
>>>>>> Any ideas what might be causing this - or if there is anything I can do to flush them (i 100% totally do a closeAllWindows after looking at a page)
>>>>>>
>>>>>> best and thanks for any help.
>>>>>>
>>>>>> Jason.
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Download Intel&#174; Parallel Studio Eval
>>>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>>>> proactively, and fine-tune applications for parallel performance.
>>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>>> _______________________________________________
>>>>>> Htmlunit-user mailing list
>>>>>> [hidden email]
>>>>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Download Intel&#174; Parallel Studio Eval
>>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>>> proactively, and fine-tune applications for parallel performance.
>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>> _______________________________________________
>>>>> Htmlunit-user mailing list
>>>>> [hidden email]
>>>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>>
>>>>>
>>>> ------------------------------------------------------------------------------
>>>> Download Intel&#174; Parallel Studio Eval
>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>> proactively, and fine-tune applications for parallel performance.
>>>> See why Intel Parallel Studio got high marks during beta.
>>>> http://p.sf.net/sfu/intel-sw-dev
>>>> _______________________________________________
>>>> Htmlunit-user mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>
>>>>
>>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download Intel&#174; Parallel Studio Eval
>>> Try the new software tools for yourself. Speed compiling, find bugs
>>> proactively, and fine-tune applications for parallel performance.
>>> See why Intel Parallel Studio got high marks during beta.
>>> http://p.sf.net/sfu/intel-sw-dev
>>> _______________________________________________
>>> Htmlunit-user mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>
>>
>> ------------------------------------------------------------------------------
>> Download Intel&#174; Parallel Studio Eval
>> Try the new software tools for yourself. Speed compiling, find bugs
>> proactively, and fine-tune applications for parallel performance.
>> See why Intel Parallel Studio got high marks during beta.
>> http://p.sf.net/sfu/intel-sw-dev
>> _______________________________________________
>> Htmlunit-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>
>>
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Htmlunit-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user


------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: ScriptableObjects piling up on the heap

RuralHunter
Great findings. I didn't try the shutdown method.

On 2010-3-24 5:06, jason franklin-stokes wrote:

> ok - it seems to be that it is enough to call :
>
> window.getJobManager.shutdown for each window.
>
> the shutdown method correctly stops the jobs and removes them.
>
>
>
>
> On Mar 22, 2010, at 4:35 PM, Rural Hunter wrote:
>
>    
>> No idea....wd.getThreadManager().interruptAll() is from old HtmlUnit
>> API. I believe it's not in HtmlUnit 2.7 any more.
>> wd.getJobManager().removeAllJobs() is the one I found that seems to do
>> the similar job.
>>
>> On 2010-3-22 23:20, jason franklin-stokes wrote:
>>      
>>> It seems to be that some jobs are not being removed after window.getJobManager.removeAllJobs
>>>
>>> I added an exception for such cases and it gets thrown occasionally. (see below)
>>>
>>> I am assuming that your line, wd.getThreadManager().interruptAll(); makes sure that they can be removed,
>>> unfortunately I cannot find anything like this is the 2.7 API.
>>>
>>> Any Ideas? Thanks a million, Jason.
>>>
>>>            def destroy
>>>              @webclient.getWebWindows.each do |window|
>>>                window.getJobManager.removeAllJobs
>>>                raise "JOBS STILL RUNNING!!" if window.getJobManager.jobCount>   0
>>>                window.getJobManager.shutdown
>>>              end
>>>              @webclient.closeAllWindows
>>>            end
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mar 22, 2010, at 2:50 PM, Rural Hunter wrote:
>>>
>>>
>>>        
>>>> I agree with you that this should be done inside of
>>>> Webclient.closeAllWindows(). I didn't check source so I'm not sure
>>>> what's done there for these threads. maybe developers wathing this mail
>>>> list can give an answer.
>>>>
>>>> On 2010-3-22 17:27, jason franklin-stokes wrote:
>>>>
>>>>          
>>>>> Thanks a million - that was the right pointer for me.
>>>>>
>>>>> I am using htmlunit 2.7 and - this was the code that did it for me.
>>>>>
>>>>> * * * jruby code * * *
>>>>>
>>>>>            def destroy
>>>>>              webclient.getWebWindows.each {  |window|
>>>>>                window.getJobManager.removeAllJobs
>>>>>                window.getJobManager.shutdown
>>>>>               }
>>>>>              webclient.closeAllWindows
>>>>>            end
>>>>>
>>>>> * * * end * * *
>>>>>
>>>>> I could not find any references to window.getThreadManager do I left it out (seems to work fine without it).
>>>>>
>>>>> Now I have very healthy looking garbage collection cycles.
>>>>>
>>>>> The only thing that gets me is that I thought closeAllWindows was supposed to kill all the java threads and get rid of the scriptable objects - why is that not working as expected??
>>>>>
>>>>> Also
>>>>>
>>>>> window.getJobManager
>>>>>
>>>>> is subject to change at anytime, so its not exactly a stable candidate to use for the next releases......
>>>>>
>>>>> any thoughts?
>>>>>
>>>>> Jason.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mar 22, 2010, at 4:40 AM, Rural Hunter wrote:
>>>>>
>>>>>
>>>>>
>>>>>            
>>>>>> I am doing this maybe you want to have a try:
>>>>>>
>>>>>> protected void closeWebClient(WebClient wc) {
>>>>>>          List<WebWindow>    windows = wc.getWebWindows();
>>>>>>          for (WebWindow wd : windows) {
>>>>>>              //wd.getThreadManager().interruptAll();
>>>>>>              wd.getJobManager().removeAllJobs();
>>>>>>          }
>>>>>>          wc.closeAllWindows();
>>>>>> }
>>>>>>
>>>>>> 2010-3-21 18:47, jason franklin-stokes wrote:
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> Hi
>>>>>>>
>>>>>>> I am looking through about 30 pages (webclient.getPage(someurl))
>>>>>>> Over time I seem to have ScriptableObjects (slots) collecting on the heap which are not getting removed - and yep I always use  webclient.closeAllWindows.
>>>>>>>
>>>>>>> The scriptableObjects seem to be holding onto the instances of pages which also pile up on the heap.
>>>>>>>
>>>>>>> Any ideas what might be causing this - or if there is anything I can do to flush them (i 100% totally do a closeAllWindows after looking at a page)
>>>>>>>
>>>>>>> best and thanks for any help.
>>>>>>>
>>>>>>> Jason.
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------------------
>>>>>>> Download Intel&#174; Parallel Studio Eval
>>>>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>>>>> proactively, and fine-tune applications for parallel performance.
>>>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>>>> _______________________________________________
>>>>>>> Htmlunit-user mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>> ------------------------------------------------------------------------------
>>>>>> Download Intel&#174; Parallel Studio Eval
>>>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>>>> proactively, and fine-tune applications for parallel performance.
>>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>>> _______________________________________________
>>>>>> Htmlunit-user mailing list
>>>>>> [hidden email]
>>>>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>>>
>>>>>>
>>>>>>              
>>>>> ------------------------------------------------------------------------------
>>>>> Download Intel&#174; Parallel Studio Eval
>>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>>> proactively, and fine-tune applications for parallel performance.
>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>> _______________________________________________
>>>>> Htmlunit-user mailing list
>>>>> [hidden email]
>>>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> ------------------------------------------------------------------------------
>>>> Download Intel&#174; Parallel Studio Eval
>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>> proactively, and fine-tune applications for parallel performance.
>>>> See why Intel Parallel Studio got high marks during beta.
>>>> http://p.sf.net/sfu/intel-sw-dev
>>>> _______________________________________________
>>>> Htmlunit-user mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>>
>>>>          
>>> ------------------------------------------------------------------------------
>>> Download Intel&#174; Parallel Studio Eval
>>> Try the new software tools for yourself. Speed compiling, find bugs
>>> proactively, and fine-tune applications for parallel performance.
>>> See why Intel Parallel Studio got high marks during beta.
>>> http://p.sf.net/sfu/intel-sw-dev
>>> _______________________________________________
>>> Htmlunit-user mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>>
>>>
>>>        
>>
>> ------------------------------------------------------------------------------
>> Download Intel&#174; Parallel Studio Eval
>> Try the new software tools for yourself. Speed compiling, find bugs
>> proactively, and fine-tune applications for parallel performance.
>> See why Intel Parallel Studio got high marks during beta.
>> http://p.sf.net/sfu/intel-sw-dev
>> _______________________________________________
>> Htmlunit-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>>      
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Htmlunit-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>
>    


------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user