C# .NET Selenium HtmlAgilityPack- scraper fixes

1.

The problem involves only the Chrome browser and affects the browser itself, and appeared long after the development of a fully functional scraper. The web portal uses CloudFlare, which, if the user agent is swapped, returns a 403 error after interacting with the login module.

If the ua is changed only after logging into a regular Chrome instance, you can navigate the entire portal as a logged-in user. When you log out (with the changed ua) it triggers a 403.

The 403 is triggered when the ua is swapped:

- Selenium Chromedriver

- Chrome UA Spoofer plugin

- Modheader plugin

403 is not triggered, and portal login runs smoothly when the ua is swapped within Developer Tools in a regular browser.

The 403 is triggered in Selenium Chromedriver if the scraper is started with the default ua (obsolete for the portal), swapped later by the hitherto successful Developer Tools.