There's a website that I want to scrape (FWIW it's svpply) and there's a button that shows up sometimes saying "Show All", it's an HTML
<button> element. Is there any way to use Ruby to emulate a click of this button, and get the content of the full page that results after clicking that button, since the button reveals more content?
What will work are tools that drive an actual browser such as watir and selenium. I installed watir-webdriver and successfully got it to click the button and show additional products.
require 'watir-webdriver' b = Watir::Browser.new b.goto 'svpply.com/editors_pick' #count products puts b.elements(:xpath => '//li[@data-class="Product"]').count #=> 30 #Now click button show_all = b.button(:id => "btn_all") show_all.click sleep 4 #count products again puts b.elements(:xpath => '//li[@data-class="Product"]').count #=>60
Mechanize can do this nicely for you.