Usage

Use crawlpropery.py to make manual crawls, just modify the CrawlPropertyReactor parameters to yours needs:

dedomeno/idealista/crawlproperty.py

if __name__ == "__main__":
    spider = CrawlPropertyReactor(property_type='land', transaction='sale', provinces=['salamanca'])
    spider.conf()
    spider.run()

Use django + celery + flower + scrapy to do programmatic crawls and visualize the output:

# 1. start django server, http://127.0.0.1:8000/
python manage.py runserver
# 2. remove all tasks from queue: celery. (Only works when RabbitMQ is up)
celery -A proj purge
# Enter admin http://127.0.0.1:8000/admin and change django-celery-beat to schedule the periodic task in the db
# 3. Run the RabbitMQ message broker
sudo rabbitmq-server
# 4. Celery:
# 4.1 Start the celery worker
celery -A dedomeno worker --loglevel=INFO
# 4.2 Run Flower, a web based tool for monitoring and administrating Celery clusters
celery -A dedomeno flower
# 4.3 Start the celery beat (schedule tasks)
celery -A dedomeno beat -l info -S django