This article will tell you how to install python scrapy and how to fix the error Running setup.py install for Twisted … error during the installation process.
1. Install Scrapy In Python Steps & Fix Can Not Install Twisted Error.
- Open a terminal and run the command
pip install scrapyto install scrapy. - During the installation, you may encounter below error messages.
Using legacy 'setup.py install' for Twisted, since package 'wheel' is not installed Installing collected packages: Twisted, scrapy Running setup.py install for Twisted ... error
- This means the python scrapy needed Twisted library is not installed successfully.
- Go to the python Twisted GitHub page, we can see that it supports python version 3.5, 3.6 and 3.7. But my python version is 3.9. So I need to install python 3.7 to fix it ( please read article Installed Python 3 On Mac OS X But It Still Use Python 2.7 to learn how to install multiple python version ).
$ python -V Python 3.9.3
- If the error still exists after you install python 3.7, you can run the command
pip install Twistedto install the Twisted library manually.pip install Twisted
- But above command may still encounter the error again.
- Now you can go to https://pypi.org/project/Twisted/#files to download the related Twisted library installation file to your local directory.
- Please choose the download file carefully. The download file name is something like Twisted-20.3.0-cp27-cp27m-macosx_10_6_intel.whl, I use macOS, so I download the macosx version. The cp27 in the download file name means this Twisted packages only support python 2.7.
- Because my python version is 3.7, so when I run the command
$ pip install Twisted-20.3.0-cp27-cp27m-macosx_10_6_intel.whlto install it, I meet below error messages.$ pip install Twisted-20.3.0-cp27-cp27m-macosx_10_6_intel.whl ERROR: Twisted-20.3.0-cp27-cp27m-macosx_10_6_intel.whl is not a supported wheel on this platform.
- So to install Twisted on python version 3.7, I should download the file Twisted-20.3.0-cp37-cp37m-macosx_10_6_intel.whl, and run below command to install it.
$ pip install Twisted-20.3.0-cp37-cp37m-macosx_10_6_intel.whl $ pip show Twisted Name: Twisted Version: 20.3.0 Summary: An asynchronous networking framework written in Python Home-page: https://twistedmatrix.com/ Author: Twisted Matrix Laboratories Author-email: [email protected] License: MIT Location: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages Requires: PyHamcrest, constantly, zope.interface, incremental, attrs, hyperlink, Automat Required-by: Scrapy
- Now run
pip install scrapycommand again, it will install python scrapy successfully.$ pip install scrapy $ pip show scrapy Name: Scrapy Version: 2.4.1 Summary: A high-level Web Crawling and Web Scraping framework Home-page: https://scrapy.org Author: Scrapy developers Author-email: None License: BSD Location: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages Requires: zope.interface, pyOpenSSL, cssselect, protego, lxml, Twisted, itemadapter, w3lib, cryptography, itemloaders, PyDispatcher, parsel, service-identity, queuelib Required-by: