dedomeno

A Spanish real estate (Idealista) python scraper

• Free software: MIT license

• Documentation: https://dedomeno.readthedocs.io.

Features

First untested alpha version.

Use crawlpropery.py to make manual crawls, just modify the CrawlPropertyReactor parameters to your needs:

dedomeno/idealista/crawlproperty.py

if __name__ == "__main__":
    spider = CrawlPropertyReactor(property_type='land', transaction='sale', provinces=['salamanca'])
    spider.conf()
    spider.run()

I used django + celery + flower + scrapy to do programmatic crawls and visualize the output:

# 1. start django server, http://127.0.0.1:8000/
python manage.py runserver
# 2. remove all tasks from queue: celery. (Only works when RabbitMQ is up)
celery -A proj purge
# Enter admin http://127.0.0.1:8000/admin and change django-celery-beat to schedule the periodic task in the db
# 3. Run the RabbitMQ message broker
sudo rabbitmq-server
# 4. Celery:
# 4.1 Start the celery worker
celery -A dedomeno worker --loglevel=INFO
# 4.2 Run Flower, a web based tool for monitoring and administrating Celery clusters
celery -A dedomeno flower
# 4.3 Start the celery beat (schedule tasks)
celery -A dedomeno beat -l info -S django

Built With

• Django - Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.

• Scrapy - An open source and collaborative framework for extracting the data you need from websites.

• Celery - An asynchronous task queue/job queue based on distributed message passing.

Credits

• Gino Palazzo ginopalazzo@gmail.com

• This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.