Price scraping is, in essence, about keeping track of competitors’ price trends through the use of technology. As a technique, price scraping alludes to that area in which bot can extract information from each competitor’s online catalogue, including prices and all other information revolving around the sale of a product on the web.
The main differentiator of price scraping is the method that is used to extract data from competitors’ web pages. A price scraping software acts in the same way as a real user would do to search for information about the price of a product, naturally browsing the web, while being the result of a mathematical algorithm.
Price scraping and dynamic pricing
Like competitive monitoring in its broadest definition, price scraping maintains a close relationship with dynamic pricing. And what does price scraping bring to a dynamic pricing strategy? With the data that emerges from the use of this price scraping technique, it is possible to monitor when competitors’ pricing goes up or down and to what extent, and to adjust one’s own pricing choices to those of key competitors. It is always essential, however, to investigate the cost structure of competitors, so as not to erode your profit margin by following the pricing choices of other companies.
Typically, price scraping projects use solutions that extract competitor data by inserting code in a programming language, be it Javascript, Python or LXML.
Price scraping, the technologies
Price scraping is therefore the extraction of information about the published online prices of products or services using a bot or web crawler. Bots search, find and copy data from a website to make it available for specific analysis. However, scraping becomes more difficult every day as many websites use tools designed to prevent such activities.
Scientifically conducting pricing analysis is critical to being able to take into account all factors that have a real impact on demand when pricing a product or service. In most markets, looking only at your production costs, the availability of a good or the profit you would like to make on each sale is not enough to maintain a certain market share.
Price scraping, markets
For example, concerning the ticketing and live entertainment market, whether or not to attend an event can be influenced by a series of external variables that cannot be controlled and defined in advance by the organizers, such as the weather or the conjunction with other events aimed at the same target consumers.
Not only that, in most markets (eCommerce, retail, tourism…) it is now essential, given the ease for the consumer to access in a few clicks information about the offer of the competition, to monitor the behaviour and strategies of competitors.
If finding information on the quantities of products sold by competitors is complex, unless the competitor provides this information directly, it is easier to find information on the prices applied. To support companies, technology provides sophisticated scraping techniques.
Price scraping is a technique that allows you to “visit” the websites of competitors and acquire information about the prices of their products to compare them with the price choices made by the company itself. This is an activity that until some time ago would have been counted as industrial espionage and, as such, companies are increasingly trying to find solutions capable of blocking these scraping activities to maintain an advantage, at least informative, over the competition.
Once you have obtained, through scraping, the prices of your competitors, how can they be used to optimize your pricing choices?
Price scraping for competitive positioning
A preliminary activity to the correct definition of the price is to define a ranking of the competitors based on their importance, the average prices they apply, the market shares and other significant indicators to understand which positioning objective to pursue concerning these other players.
Let’s now assume that the scientific pricing analysis mentioned earlier suggests a price that can maximize a certain business objective (revenue maximization, maximization of units sold..) for each of our products. These prices should be compared with the equivalent prices of all major competitors to understand their price positioning in the market.
Price scraping, the role of marketing
At this point, in addition to mathematics, Marketing comes into play. Unless the client company has established a priori that, to achieve the desired positioning, it is necessary to propose a price that is always at least 5€ above competitor X, the prices of competitors can be used as input to the pricing algorithm and not as simple downstream rules to “adjust” the price.
Whatever are the indications conveyed by Marketing, it is necessary, in order not to ruin the balance of the entire market, to avoid putting in place a price war by positioning oneself, for example, always 1€ below a certain competitor, without considering the differences between companies in the cost structure. Not only that, but it will also be necessary to avoid becoming victims of a certain immobilism, waiting for a new move by competitors and losing potential opportunities.
As it turns out, the range of possibilities for action regarding the use of competitors’ pricing information is very wide, but much depends on the market and how free companies are to act and modify their pricing choices independently.
Beyond these aspects, knowing the prices of competitors, even if not constantly updated as happens with some scraping systems, is of crucial importance, at least to understand if the company is wrongly positioning itself in the market also for the values and identity that it promotes.
