Python requests module provide functions to write web request process code easy and simple. It can send both http get and post request to web server. When you send get request to web server, you can use python request module to pass parameters. When you send post request to web server, you can use python request module to post form data. It can also send or get http headers, cookies and send files. This article will focus on send get or post request, we will write other article to show headers, cookies and session management.
1. How To Install Python Requests Module.
1.1 Install Python Requests Module Use Pip Command.
- Run
$ pip (pip3) show requests
command to check whether python requests module has been installed on your operating system or not. If nothing print out, then it means python requests module has not been installed.$ pip3 show requests
- If python requests module is not installed, then run
$ pip3 install requests
in terminal to install it like below.$ pip3 install requests Collecting requests Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB) |████████████████████████████████| 61kB 124kB/s Collecting idna<2.9,>=2.5 (from requests) Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB) |████████████████████████████████| 61kB 152kB/s Collecting certifi>=2017.4.17 (from requests) Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB) |████████████████████████████████| 163kB 127kB/s Collecting chardet<3.1.0,>=3.0.2 (from requests) Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB) |████████████████████████████████| 143kB 127kB/s Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests) Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB) |████████████████████████████████| 153kB 104kB/s Installing collected packages: idna, certifi, chardet, urllib3, requests Successfully installed certifi-2019.3.9 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
- Now run
$ pip (pip3) show requests
again, you can see python requests module installation information like below, this means it has been installed successfully.$ pip3 show requests Name: requests Version: 2.22.0 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: me@kennethreitz.org License: Apache 2.0 Location: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages Requires: urllib3, certifi, idna, chardet Required-by:
1.2 Install Python Requests Module From Source Code.
Beside use pip command line, you can also install python request module from source code.
- Run command
$ wget https://github.com/requests/requests/tarball/master
in a terminal to get python requests module source code to a local directory, the directory name is master.$ wget https://github.com/requests/requests/tarball/master --2019-06-17 10:11:48-- https://github.com/requests/requests/tarball/master Resolving github.com (github.com)... 52.74.223.119 Connecting to github.com (github.com)|52.74.223.119|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://github.com/kennethreitz/requests/tarball/master [following] --2019-06-17 10:11:49-- https://github.com/kennethreitz/requests/tarball/master Reusing existing connection to github.com:443. HTTP request sent, awaiting response... 302 Found Location: https://codeload.github.com/kennethreitz/requests/legacy.tar.gz/master [following] --2019-06-17 10:11:49-- https://codeload.github.com/kennethreitz/requests/legacy.tar.gz/master Resolving codeload.github.com (codeload.github.com)... 13.229.189.0 Connecting to codeload.github.com (codeload.github.com)|13.229.189.0|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/x-gzip] Saving to: ‘master’ master [ <=> ] 2.92M 760KB/s in 4.0s 2019-06-17 10:11:54 (741 KB/s) - ‘master’ saved [3066122]
- Run
$ ls -l
in command line to verify the master directory has been created.$ ls -l total 6208 ...... -rw-r--r-- 1 songzhao staff 3066122 Jun 17 10:11 master ......
- Run
$ mv master requests.tgz
command to move the master directory to a zip file. Run$ ls -l
command again to verify the zip file has been created.$ mv master requests.tgz $ ls -l ...... -rw-r--r-- 1 songzhao staff 3066122 Jun 17 10:11 requests.tgz ......
- Run
$ tar xzf requests.tgz
command to uncompress the tgz file to a local folder. Runls -l
to check the local folder. From below output, we can see the local folder name is kennethreitz-requests-4983a9b.$ tar xzf requests.tgz $ ls -l total 6208 ...... drwxr-xr-x 26 songzhao staff 832 Jun 3 22:01 kennethreitz-requests-4983a9b -rw-r--r-- 1 songzhao staff 3066122 Jun 17 10:11 requests.tgz ......
- Run
cd kennethreitz-requests-4983a9b
to go into the uncompressed directory, and you can list all the files in the directory.$ cd kennethreitz-requests-4983a9b/ $ ls -l total 280 -rw-r--r-- 1 songzhao staff 7688 Jun 3 22:01 AUTHORS.rst -rw-r--r-- 1 songzhao staff 96 Jun 3 22:01 CODE_OF_CONDUCT.md -rw-r--r-- 1 songzhao staff 3010 Jun 3 22:01 CONTRIBUTING.md -rw-r--r-- 1 songzhao staff 48996 Jun 3 22:01 HISTORY.md -rw-r--r-- 1 songzhao staff 582 Jun 3 22:01 LICENSE -rw-r--r-- 1 songzhao staff 122 Jun 3 22:01 MANIFEST.in -rw-r--r-- 1 songzhao staff 861 Jun 3 22:01 Makefile -rw-r--r-- 1 songzhao staff 442 Jun 3 22:01 Pipfile -rw-r--r-- 1 songzhao staff 36746 Jun 3 22:01 Pipfile.lock -rw-r--r-- 1 songzhao staff 4037 Jun 3 22:01 README.md drwxr-xr-x 3 songzhao staff 96 Jun 3 22:01 _appveyor -rw-r--r-- 1 songzhao staff 1659 Jun 3 22:01 appveyor.yml drwxr-xr-x 13 songzhao staff 416 Jun 3 22:01 docs drwxr-xr-x 4 songzhao staff 128 Jun 3 22:01 ext -rw-r--r-- 1 songzhao staff 33 Jun 3 22:01 pytest.ini drwxr-xr-x 20 songzhao staff 640 Jun 3 22:01 requests -rw-r--r-- 1 songzhao staff 63 Jun 3 22:01 setup.cfg -rwxr-xr-x 1 songzhao staff 3312 Jun 3 22:01 setup.py drwxr-xr-x 15 songzhao staff 480 Jun 3 22:01 tests -rw-r--r-- 1 songzhao staff 84 Jun 3 22:01 tox.ini
- Now run
$ pip install .
to install it.$ pip install . Processing /Users/songzhao/kennethreitz-requests-4983a9b Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from requests==2.22.0) (3.0.4) Requirement already satisfied: idna<2.9,>=2.5 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from requests==2.22.0) (2.8) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from requests==2.22.0) (1.25.3) Requirement already satisfied: certifi>=2017.4.17 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from requests==2.22.0) (2019.3.9) Installing collected packages: requests Found existing installation: requests 2.22.0 Uninstalling requests-2.22.0: Successfully uninstalled requests-2.22.0 Running setup.py install for requests ... done Successfully installed requests-2.22.0
2. How To Send Http Get Request Use Python Requests Module.
You can use python requests module’s get function to send http get request, the get function has a string value parameter which is a web page url. The page url must start with http or https protocol, if not it will throw a requests.exceptions.MissingSchema exception. Below is an example.
- Open a terminal and run command python3 to enter python interactive mode.
$ python3 Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21) [Clang 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>
- First you should import python requests module as below.
>>> import requests
- Then call requests module’s get function with provided url string, the url string should start with http or https protocol.
>>> response = requests.get('http://www.dev2qa.com')
- If the url string do not start with http or https protocol, then it will throw requests.exceptions.MissingSchema exception.
>>> response = requests.get('www.dev2qa.com') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/sessions.py", line 519, in request prep = self.prepare_request(req) File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/sessions.py", line 462, in prepare_request hooks=merge_hooks(request.hooks, self.hooks), File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/models.py", line 313, in prepare self.prepare_url(url, params) File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/models.py", line 387, in prepare_url raise MissingSchema(error) requests.exceptions.MissingSchema: Invalid URL 'www.dev2qa.com': No schema supplied. Perhaps you meant http://www.dev2qa.com? >>>
- When the requests module’s get function execute successfully, you can get a response object, you can call various methods or property of the response object, such as get http response status code with status_code property.
>>> response.status_code 200
- Get http response encoding with encoding property.
>>> response.encoding 'UTF-8'
- Get request url web page content use content or text property. Please note the content property return the binary format of the web page, you can see the content is prefixed with b. The text property return the text format of the web page content.
>>> response.content b'<!DOCTYPE html><html lang="en-US">....</html> >>> response.text '<!DOCTYPE html><html lang="en-US">....</html>'
- Get response raw object by call response.raw property. We can see the raw property return a urllib3.response.HTTPResponse object in below code.
>>> response.raw <urllib3.response.HTTPResponse object at 0x1101b5ef0>
3. How To Pass Parameters In Http Get Request URL.
Create a python dictionary object to save query string key value pair, then pass this dictionary object to requests module’s get method’s params argument.
>>> import requests >>> # Create dictionary object to save query string key value pair. >>> query_string = {'user_name':'jerry', 'password':'jerry'} >>> # Pass above object to get function's params argument. >>> response = requests.get('http://www.dev2qa.com', params=query_string) >>> # Display the request url. You can see the query string in the url. >>> response.url 'https://www.dev2qa.com/?user_name=jerry&password=jerry'
4. How To Send Http Post Request Use Python Requests Module.
Invoke requests module’s post function to send post data to special form url. The post data should be saved in a python dictionary object. Then pass this dictionary object to the post function’s data parameter.
# First import python requests module. >>> import requests >>> # Create a python dictionary object to contain post data. The dictionary object item data format is form_element_name: form_element_value. >>> post_data={'user_name':'jerry', 'password':'password'} >>> # Invoke post function to post above dictionary data to special form url. >>> response = requests.post("http://www.dev2qa.com", data=post_data) >>> >>> response.url 'https://www.dev2qa.com/'