I am writing a Selenium script in Python to scrape data from several financial portals, but every time I click a PDF link, the browser opens the file in a built-in viewer instead of saving it to my disk. I need this process to be fully automated. How can I configure the Chrome or Firefox options in my WebDriver setup to bypass the PDF preview and force a download to a specific directory on my computer? Is there a set of experimental preferences I need to pass during the driver initialization?
3 answers
To force a PDF download in Chrome using Selenium, you must modify the ChromeOptions. You need to set a dictionary of experimental preferences. Specifically, you should set download.default_directory to your target path and plugins.always_open_pdf_externally to True. This second setting is the "magic key" that tells Chrome to stop using its internal viewer and just hand the file to the OS download manager. In Python, you would apply this using options.add_experimental_option("prefs", prefs). This ensures that your automation script isn't stalled by a manual "Save As" window or an embedded viewer that Selenium cannot easily interact with.
I've implemented the Chrome options you mentioned, but my script sometimes fails if the download takes too long. Is there a built-in Selenium "Wait" command that detects when a file has finished downloading, or do I need to write a manual loop in Python that checks my local folder for the appearance of the new .pdf file?
For Firefox users, the logic is similar but uses FirefoxOptions. You’ll need to set pdfjs.disabled to true and configure the browser.helperApps.neverAsk.saveToDisk preference with the PDF MIME type.
I agree with Patricia; the MIME type part is vital. If you don't specify application/pdf, Firefox might still prompt you with a dialog box. This is a common hurdle in cross-browser automation testing, so always keep a list of common MIME types handy for your Selenium configurations.
Steven, Selenium doesn't actually track the OS file system, so it won't know when the download is complete. The standard Software Development practice here is to write a "polling" function. Use Python's os.path.exists() inside a while loop with a timeout. You should also check that the file doesn't have a .crdownload extension, which Chrome uses for files still in progress. Once the extension is just .pdf, your script can safely move to the next task in your data science pipeline.