Evaluation of Web Crawling and Scraping Technologies
First written in FY2017. 3 of 4 entries say "Advanced", first in FY2018. Latest entry: FY2021.
No other company in this theme has enough history to compare yet.
- FY2021AdvancedAnnual report
「During the fiscal year, we tested the automated collection of data using a web crawler and image editing using a fully automated image-processing tool. Based on the test results, we evaluated the potential for applying these technologies to future service development.」
For web crawling technology, the company progressed from conventional testing to testing its use for automated data collection, and also tested image editing using a fully automated image-processing tool. - FY2019AdvancedAnnual report
「During the current fiscal year, we conducted research on “investigating the store attributes of a specific chain using image recognition based on web scraping technology and a machine-learning API” and “a proof of concept for text mining to use survey response data.”」
The company has advanced its testing of web scraping from conventional automated information gathering to applying image recognition using a machine-learning API to investigate the store attributes of a specific chain. - FY2018AdvancedAnnual report
「We tested the effectiveness and challenges of web crawling and scraping as methods for automatically gathering information from websites over the internet. We also tested whether it would be possible to automatically gather information from individual websites using general-purpose rules, while limiting the information collected and using past research findings and other information as reference data.」
The company has progressed its work on web crawling and scraping to testing automated information gathering using limited data, past research findings and general-purpose rules. - FY2017First statedAnnual report
「We tested the effectiveness and challenges of web crawling and scraping as methods for automatically gathering information from websites over the internet. Having confirmed their effectiveness, we will continue testing and consider applying them to our services.」
The company has described for the first time a research and development theme involving the continued testing of automated information gathering from websites for application to its services, after confirming its effectiveness.