爬虫 知乎用户信息
- 对用url进行缓存，防止重复请求，url来源与关注和被关注列表
- 用户信息存入mongodb
- 知乎进行了限制，所有需要调试最佳的请求间隔，监控日志，如果出现error....则用程序里面的cookie刷新页面（所以浏览器就不要清cookied）,填写验证码就可以继续爬了
- 增加了代理，可查看本人的代理服务dynamic_proxy_pool,用户可选择性的使用
-
Notifications
You must be signed in to change notification settings - Fork 0
MicroCountry/my_zhihu_spider
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
爬虫－知乎用户信息
Topics
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published