Fajri Abdillah

Scrapy Bhinneka Crawler

Just a playing project, to capture all price data on bhinneka.com

Challenge

  • Python

    Write a good python code is fun, because it indented very well

  • Scrapy & XPATH

    Confusing at first, but it goes well, trial and error using xpath code just to capture some in string in HTML element

  • Pagination

    Every product has paginated result, to crawl a thread I have to know the pagination. It's challenging for me

Year

2012

Stack

  • Python 2.6
  • Ubuntu Server 12.04
  • Apache 2
  • Scrapy 0.16
  • Redis

Code

Scrapy Bhinneka Crawler

Lesson learned

Scrapy is absolutely a web crawler framework, and fast.