I'm working on an automated testing project where I need to capture the current state of the page's HTML to validate some dynamic content. I've successfully navigated to the site using the driver, but I'm unsure which method allows me to grab the entire DOM as a string so I can save it to a file or parse it. Does Selenium provide a built-in function to get the "View Source" equivalent, and will it include changes made to the DOM by JavaScript after the initial load?
3 answers
In Selenium WebDriver, the most efficient way to retrieve the HTML source of the currently loaded page is by using the getPageSource() method. This method is available directly on the WebDriver instance. You can simply call String source = driver.getPageSource(); and then print it or write it to a local file using a FileWriter. It is important to note that unlike the "View Source" option in a browser—which only shows the static HTML sent by the server—getPageSource() returns the current state of the DOM. This means if JavaScript has executed and modified elements, added classes, or injected data, those changes will be reflected in the string you retrieve. This makes it an incredibly powerful tool for debugging single-page applications (SPAs) built with frameworks like React or Angular.
If the page uses infinite scrolling or heavy AJAX loading, does getPageSource() wait for those elements to finish rendering, or do I need to implement an explicit wait before calling the method to ensure I get the complete data? I’ve noticed sometimes the captured source looks "empty" compared to what I see on the screen.
The quickest way is definitely driver.getPageSource(). I usually combine this with a regex or a library like Jsoup to parse specific tags once I have the raw HTML string.
I agree with Brandon. Using Jsoup in tandem with Selenium is a fantastic strategy. While Selenium is great for navigating and interacting, Jsoup is much faster at parsing and extracting data from the HTML string once you've captured it. It’s the standard workflow for most advanced scraping tasks I've handled recently.
Gregory, that is a common hurdle! getPageSource() does not wait for asynchronous calls to complete; it simply takes a snapshot of the DOM at the exact millisecond it is called. To fix this, you should use WebDriverWait along with ExpectedConditions to wait for a specific element that signifies the page has finished loading (like a footer or a data table). Once that element is present, calling the method will provide the fully rendered HTML. Without a wait, you’re often just capturing the initial skeletal HTML or a loading spinner.