StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm. It is provided under Apache License and is written mostly in Java.
storm-crawler alternatives and similar libraries
Based on the "Web Crawling" category.
Alternatively, view storm-crawler alternatives based on common mentions on social networks and blogs.
8.1 7.3 L2 storm-crawler VS Apache NutchApache Nutch is an extensible and scalable web crawler
4.7 7.6 storm-crawler VS SparklerSpark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.
Do you think we are missing an alternative of storm-crawler or a related project?
The version of Storm to use must match the one defined in the pom.xml file of your topology. The major version of StormCrawler mirrors the one from Apache Storm, i.e whereas StormCrawler 1.x used Storm 1.2.3, the current version now requires Storm 2.3.0. Our Ansible-Storm repository contains resources to install Apache Storm using Ansible.
Once Storm is installed, the easiest way to get started is to generate a brand new StormCrawler project using :
mvn archetype:generate -DarchetypeGroupId=com.digitalpebble.stormcrawler -DarchetypeArtifactId=storm-crawler-archetype -DarchetypeVersion=2.1
You'll be asked to enter a groupId (e.g. com.mycompany.crawler), an artefactId (e.g. stormcrawler), a version and package name.
This will not only create a fully formed project containing a POM with the dependency above but also the default resource files, a default CrawlTopology class and a configuration file. Enter the directory you just created (should be the same as the artefactId you specified earlier) and follow the instructions on the README file.
Alternatively if you can't or don't want to use the Maven archetype above, you can simply copy the files from archetype-resources.
Have a look at the code of the CrawlTopology class, the crawler-conf.yaml file as well as the files in src/main/resources/, they are all that is needed to run a crawl topology : all the other components come from the core module.
DigitalPebble Ltd provide commercial support and consulting for StormCrawler.
YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.
We are very grateful to our sponsors for their continued support.
*Note that all licence references and agreements mentioned in the storm-crawler README section above are relevant to that project's source code only.