Navigate website after logging in using WebRequest

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Navigate website after logging in using WebRequest

minersail
I have tried to do the typical log in with HTMLUnit by filling in inputs and clicking the submit button. However, that didn't work, even with giving the webpage 20 seconds to load.

So instead, I've tried to log in using a POST WebRequest, which has worked. The cookies successfully updated and the site itself in chrome told me that I was logged in elsewhere. The page returned by the POST was not a valid html page though, so I could not navigate directly from there.

When I now use the webclient to navigate through the website, it does not register me as logged in. I've tried both navigating to the website regularly with client.getPage() as well as a GET request with the updated cookies attached.

How would I get the logged in state from the POST request to transfer over when navigating through the rest of the website?

I have omitted code for brevity but can post any relevant code necessary.
Reply | Threaded
Open this post in threaded view
|

Re: Navigate website after logging in using WebRequest

Ronald Brill
Please provide a complete sample (code and maybe account data) (private mail). We will try to reproduce.

        RBRi

On Sun, 25 Jun 2017 15:44:06 -0700 (MST) minersail wrote:

>
>I have tried to do the typical log in with HTMLUnit by filling in inputs and
>clicking the submit button. However, that didn't work, even with giving the
>webpage 20 seconds to load.
>
>So instead, I've tried to log in using a POST WebRequest, which has worked.
>The cookies successfully updated and the site itself in chrome told me that
>I was logged in elsewhere. The page returned by the POST was not a valid
>html page though, so I could not navigate directly from there.
>
>When I now use the webclient to navigate through the website, it does not
>register me as logged in. I've tried both navigating to the website
>regularly with client.getPage() as well as a GET request with the updated
>cookies attached.
>
>How would I get the logged in state from the POST request to transfer over
>when navigating through the rest of the website?
>
>I have omitted code for brevity but can post any relevant code necessary.
>
>
>
>--
>View this message in context: http://htmlunit.10904.n7.nabble.com/Navigate-website-after-logging-in-using-WebRequest-tp42263.html
>Sent from the HtmlUnit - General mailing list archive at Nabble.com.
>
>------------------------------------------------------------------------------
>Check out the vibrant tech community on one of the world's most
>engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>_______________________________________________
>Htmlunit-user mailing list
>[hidden email]
>https://lists.sourceforge.net/lists/listinfo/htmlunit-user
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Htmlunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/htmlunit-user
Reply | Threaded
Open this post in threaded view
|

Re: Navigate website after logging in using WebRequest

minersail
I have created a dummy account on MarineTraffic to use.

Method 1: Manipulating form and clicking button:

String username = "htmlunit888@gmail.com";
String password = "password1";
               
WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setCssEnabled(false);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
                       
HtmlPage webPage = (HtmlPage)webClient.getPage("https://www.marinetraffic.com/");
HtmlForm loginForm = (HtmlForm)webPage.getElementById("login_form_REACT");
                       
loginForm.getInputByName("data[email]").setValueAttribute(username);
loginForm.getInputByName("data[password]").setValueAttribute(password);
                       
HtmlPage webPage2 = (HtmlPage)((HtmlButton)loginForm.getFirstByXPath("//button[@type='submit']")).click();
webClient.waitForBackgroundJavaScript(20 * 1000);
                       
System.out.println(webPage2.asXml());

----------------------------------------------------
Method 2: Logging using POST WebRequest:

WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setCssEnabled(false);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
                       
HtmlPage webPage = (HtmlPage)webClient.getPage("https://www.marinetraffic.com/");
URL cookieURL = new URL("https://www.marinetraffic.com/");
String cookies = webClient.getCookies(cookieURL).toString();

URL loginurl = new URL("https://www.marinetraffic.com/en/users/ajax_login");
WebRequest requestSettings = new WebRequest(loginurl, HttpMethod.POST);

requestSettings.setAdditionalHeader(":authority", "www.marinetraffic.com");
requestSettings.setAdditionalHeader(":method", "POST");
requestSettings.setAdditionalHeader(":path", "/en/users/ajax_login");
requestSettings.setAdditionalHeader(":scheme", "https");
requestSettings.setAdditionalHeader("accept", "*/*");
requestSettings.setAdditionalHeader("accept-encoding", "gzip,deflate,sdch");
requestSettings.setAdditionalHeader("accept-language", "en-US,en;q=0.8");
requestSettings.setAdditionalHeader("content-type", "application/x-www-form-urlencoded; charset=UTF-8");
requestSettings.setAdditionalHeader("cookie", cookies);
requestSettings.setAdditionalHeader("origin", "https://www.marinetraffic.com");
requestSettings.setAdditionalHeader("referer", "https://www.marinetraffic.com/en/ais/home/centerx:-33.1/centery:21.4/zoom:4");
requestSettings.setAdditionalHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36");
requestSettings.setAdditionalHeader("x-requested-with", "XMLHttpRequest");
                       
                        requestSettings.setRequestBody("_method=POST&email=htmlunit888%40gmail.com&password=password1&is_ajax=true");

Page redirectPage = webClient.getPage(requestSettings);
webClient.waitForBackgroundJavaScript(10 * 1000);
                       
System.out.println(redirectPage.getWebResponse());

------------------------------------------------------------------------

Method 1 I have not gotten to work past verifying the inputs were filled in. Method 2 I have been able to log in, but not been able to navigate the website thereafter.