Merge pull request #130 from yujiosaka/browser_cache_option

Browser cache option

CHANGELOG.md

@@ -6,6 +6,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+- Support `browserCache` for [crawler.queue()](https://github.com/yujiosaka/headless-chrome-crawler#crawlerqueueoptions)'s options.
+- Support `depthPriority` option again.
+
 ## [1.3.4] - 2018-02-22
 
 ### changed

README.md

@@ -182,7 +182,7 @@ browserWSEndpoint, ignoreHTTPSErrors
 Also, the following options can be set as default values when [crawler.queue()](#crawlerqueueoptions) are executed.
 
 ```
-url, allowedDomains, deniedDomains, timeout, priority, depthPriority, delay, retryCount, retryDelay, jQuery, device, username, password, evaluatePage
+url, allowedDomains, deniedDomains, timeout, priority, depthPriority, delay, retryCount, retryDelay, jQuery, browserCache, device, username, password, evaluatePage
 ```
 
 > **Note**: In practice, setting the options every time you queue equests is redundant. Therefore, it's recommended to set the default values and override them depending on the necessity.
@@ -222,7 +222,7 @@ ignoreHTTPSErrors, headless, executablePath, slowMo, args, ignoreDefaultArgs, ha
 Also, the following options can be set as default values when [crawler.queue()](#crawlerqueueoptions) are executed.
 
 ```
-url, allowedDomains, deniedDomains, timeout, priority, depthPriority, delay, retryCount, retryDelay, jQuery, device, username, password, evaluatePage
+url, allowedDomains, deniedDomains, timeout, priority, depthPriority, delay, retryCount, retryDelay, jQuery, browserCache, device, username, password, evaluatePage
 ```
 
 > **Note**: In practice, setting the options every time you queue the requests is redundant. Therefore, it's recommended to set the default values and override them depending on the necessity.
@@ -251,6 +251,7 @@ url, allowedDomains, deniedDomains, timeout, priority, depthPriority, delay, ret
  * `retryCount` <[number]> Number of limit when retry fails, defaults to `3`.
  * `retryDelay` <[number]> Number of milliseconds after each retry fails, defaults to `10000`.
  * `jQuery` <[boolean]> Whether to automatically add [jQuery](https://jquery.com) tag to page, defaults to `true`.
+ * `browserCache` <[boolean]> Whether to enable browser cache for each request, defaults to `true`.
  * `device` <[string]> Device to emulate. Available devices are listed [here](https://github.com/GoogleChrome/puppeteer/blob/master/DeviceDescriptors.js).
  * `username` <[string]> Username for basic authentication. pass `null` if it's not necessary.
  * `screenshot` <[Object]> Screenshot option, defaults to `null`. This option is passed to [Puppeteer's page.screenshot()](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagescreenshotoptions). Pass `null` or leave default to disable screenshot.

lib/crawler.js

@@ -80,6 +80,7 @@ class Crawler {
  this._preventNewTabs(),
  this._authenticate(),
  this._emulate(),
+ this._setCacheEnabled(),
  this._setUserAgent(),
  this._setExtraHeaders(),
  this._handlePageEvents(),
@@ -118,6 +119,15 @@ class Crawler {
  return this._page.emulate(devices[this._options.device]);
  }
 
+ /**
+ * @return {!Promise}
+ * @private
+ */
+ _setCacheEnabled() {
+ if (this._options.browserCache) return Promise.resolve();
+ return this._page.setCacheEnabled(false);
+ }
+
  /**
  * @return {!Promise}
  * @private

lib/hccrawler.js

@@ -111,6 +111,7 @@ class HCCrawler extends EventEmitter {
  retryCount: 3,
  retryDelay: 10000,
  jQuery: true,
+ browserCache: true,
  persistCache: false,
  skipDuplicates: true,
  depthPriority: true,