Applying particle swarm optimization-based dynamic adaptive hyperlink evaluation to focused crawler for meteorological disasters

Traditional semantic-based focused crawlers calculate the topical priority of hyperlink by linearly integrating topical similarity evaluation metrics and empirical weights. However, the manually pre-determined weights may introduce bias in evaluating hyperlinks, resulting in topic deviation during crawling. To address this problem, we propose a dynamic adaptive procedure based on particle swarm optimization which dynamically updates weights in every crawling step and put forward a new focused crawler, called FCPSO. In FCPSO, we utilize domain ontology for topic representation and a comprehensive priority evaluation method to evaluate the topical priority of hyperlink. Furthermore, we construct a multi-objective optimization model for hyperlink selection, in which the strategy of the non-dominant sorting with the nearest farthest candidate solution is proposed to select Pareto-optimal hyperlinks and guide the crawling direction. Extensive experiments demonstrate the effectiveness of FCPSO over other strategies that it can obtain more topic-relevant webpages with less time consumption.