Jsup and JavaScript dependent web pages?
So I had trouble formulating the correct syntax for selecting this element from the web page This is the path on the inspect element interface on Firefox
Here is my current code:
Element prices = doc.select("body[class =en page-type-search page-type-group-shelf og ress] " + "div#wrap " + "div#main-wrap " + "div#jalapeno-template " + "div[class=zone zone3 wgrid-10of12 wgrid-6of8 wgrid-4of4] " + "section#shelf-page " + "div#shelf-thumbs " + "div.shelf-thumbs " + "div.price-current " + "span.product-price-analytics").first(); String priceOne = prices.attr("data-analytics-value");
And very clearly, the attribute I want is' data analysis - value 'because it gives the exact price
I think I have all the right grammar, so what did I do wrong? When I run the program, it gives me a NullPointerException Any help is appreciated!
[update] I changed princeone to doc Tostring(), and it says that the web browser does not run JavaScript and needs JavaScript to view the Walmart website. Any solution?
Solution
After trying to use Android WebView without luck, I accidentally found a solution to set up my useragent. All I did was change
Jsoup.connect(url).get();
reach
Jsoup.connect(url).userAgent("YOUR_USER_AGENT_HERE").get();
It's like a charm Thanks for Fred's reply!