Skip to main content

​
What is Node-crawler?

Node-crawler is a specialized crawling framework that manages a request queue and concurrency. It includes built-in features like priority queuing and rate limiting. Use Oculus Proxies for bulk crawling, your IP will be flagged almost instantly without a proxy. This allow you to distribute that massive load across thousands of residential IPs, making the crawl appear as many individual users.
If you’re using Oculus to scrape search engines like Google, Bing, or Yandex and facing connection issues, your proxy type may need adjustment. ISP Premium Proxies provide stable and unrestricted access, preventing blocks that standard proxies may encounter. Switching to ISP Premium Proxies can improve performance.

​
How to Use Oculus Proxies With Node-crawler

1

Install package

Install via cmd:
npm install crawler
2

Example code

Use the following code as an example for Nightmare.js. Make sure to replace the correct proxy credentials from your dashboard inside c.
const Crawler = require('crawler');

const c = new Crawler({
proxy: 'http://USERNAME:[email protected]:PORT', 
callback: (error, res, done) => {
    if (!error) console.log('Crawler Data:', res.body);
    done();
}
});
c.queue('http://httpbin.org/ip');