Job description
We need a search engine, collecting data from various websites and partner websites, and presenting offers collected from these websites to users (illustrations, brands, models, parameters and prices), allowing filtering results by categories (up to four levels) and parameters and I appreciate.
The system collecting offers is supposed to be unmanned, it should be universal and collect data from several thousand different websites and partner websites. It can not be adapted to individual pages or websites - its operation is based on searching for keywords referring to categories, sub-categories, brands, models and other product parameters. These words (in nine languages, with the possibility of future expansion) will be delivered to the contractor - there are several hundred thousand of them. The system is to work with partner websites and services added cyclically in the future.
The system is to periodically check the list of partner websites and websites and index their content - finding and saving individual sales offers or updating existing ones. If the offer is not found - does not exist, it has to be deleted.
In addition, the system must be able to search pages whose content is loaded dynamically by JavaScript and match the found offers to an extensive list of categories (with subcategories). It may not require integration from the website (API, special tags or XML files) or configuration for a specific site - only the link to the site will be in the database.
Experience in the construction of similar systems is required.
