I created an command line to tool to use Hacker news and Google sheets API to store 'Who wants to be hired?' post comment in a google sheet. I am learning to use API's by building these applications and also learning how to do TDD(test driven development)
Here is its repository
here is how
Usually ever month starting a bot automatically generates 3 posts with titles
- Who's Hiring?
- Who Wants to Be Hired?
- Freelancer/Looking for Freelancer?
What i was targeting was only "Who wants to be hired?" in the V1 of this application.
Objective
$ hackerjobs --help
the above command should give me the
Usage: hackerjobs [OPTIONS] COMMAND [ARGS]...
This function is the entry point for 'the hackerjobs' command
Options:
--help Show this message and exit.
Commands:
config Config DocString
who-wants-to-be-hired This function adds data to the Google Sheet of...
and the command should have various commands like
$ hackerjobs who-wants-to-be-hired
$ hackerjobs config --api-name 'sheets'
$ hackerjobs config --secret-file 'home/$USER/tokens/token.json'
$ hackerjobs config --spread-sheet-id <google-sheet-id-of-your-sheet>
$ hackerjobs config --api-version 'v4'
$ hackerjobs config --client-secret-file-name <name-of-the-json-file-you-are-using>
'who-wants-to-be-hired' should get all the values of the comments under recent 'Who wants to be hired?' post and store them in a google sheet which you created
'config' commands should add the arguments to config file in the home folder i.e, /home/$USER/.config/hackerjobs/ (by the way you should create a hackerjobs folder before running this program)
all the config commands are required for the program to run. Below i will explain all the config inputs
API-name and API-version:
Since we are storing this to sheets you should use 'sheets' and 'v4' as arguments. I could have added that in-built I just want to type those every time for fun
secret-file
This will the google sheets json file created from OAuth client id credentials which i will explain how to create.
You have to mention the directory at which this file is located so that it could move to ~/.config/hacker/
SpreadSheet Id:
You have to mention the id of the google spread sheet. following is the example of the id
docs.google.com/spreadsheets/d//edit
you should give the to the --spread-sheet-id argument for it to upload the data to that exact sheet
Secret-file-name:
this is the name of the file of json you stored in ~/.config/hackerjobs/ for it to read from it
Now coming in detailed into project
Requirements
- Google sheets credentials
- creating 'hackerjobs' folder in /home/$USER/.config
- python 3.7 or above
- conda environment or any virtual environments
All the dependencies you can from 'requirements.txt' by going to the repo directory and
$ pip install -e .
- You have to download google oauth client id json file. You can see how from this link
you will find detailed instructions on how to install and run from my readme.md file. I am here to tell you what I learnt during building this project
Things I Learnt
I first learnt how to structure a project first. The most important things I learnt was
Always create README.md file when creating a repo the first thing you should do is to create a README.md and add all titles and fill the readme as you go further building the project
create a setup.py and mention the requirements and entry points in it at first and then you can manage them later
make requirements.txt manually not just by using
$ pip freeze > requirements.txtmanually entering requirements will let to mention if you can use a version greater than or in a range of versions because sometime it may cause problems if you mention partcular version
Always write a function and its test function at the same time this will help you when you come back comeback to previous commit and gives a structure to project
Always use doc strings and write in detail about the function and what it does because if spend most of the time reading the code rather than writing the code so it is important to understand the code better
Importance of Git. learn git in detail not just 'git pull', 'git push','git add','git commit'. you can do so much stuff with git you can experiment heavily on your code and need not to worry about your project if you know better.
What I did in the project
At first I learnt how to use Hackernews api. You can go through the documentation by yourself if you want to learn it. This is a good documentation
At first I created module named Hacker_news since it is good practice to use modular projects(i know the name 'Hacker_news' is not good implementation regarding nomenclature wise i known after completing that i should use capital letters at first now i don't want change because I have to change all the imports also)
Next I created get_news.py which objective was to fetch the data from the Hackernews API
First I bring objects of all the top stories of the month using HackerNews().ask_stories() and then I looped through those objects until I found the title name of the object which has "who wants to be hired" in it and i then returned the id of the post to next function(get_post_comments()) which then fetched all the details about the post in the form of object. Then get_post_comments() will all the comments from the post object
make_list() function will return a dictionary with list of lists and list of dicts of all each details of comment posts.
Google Sheets API
I learnt a lot on how to use API by reading sheets API
First I downloaded sheets_instance.py from internet to create sheets instance someday I will learn how to do that but for my current application the work is done without me learning it.
You can go through Google API documentation to learn about the api.I created sheets instance to 'service' variable and since i am using spreadsheets i use service.spreadsheets()
I used only selected commands from documentation like value().get(), value().update(), value().append()
I get the already stored row values from sheets in order to check whether the current data we are adding isn't the same as the previous. We compare the new_thread_id() and last row id from service.spreadsheets().value().get() and we give arguments to get().
Note: While going through docs it is mentioned that range attribute of get() update() and update() is mentioned in format "!:" example "sheet1!A1:D5" but that wont work if we add sheets name it gives range error instead use only start and end cells like "A1:D5".
Example:
result = service.spreadsheets().values().append(spreadsheetId=spreadsheet_id,range="A2:J2",valueInputOption='USER_ENTERED',insertDataOption="INSERT_ROWS",
body=body).execute()
Using Click:
click makes normal function executable through command line interface. In setup.py we can give starting point as cli script file we can execute file as command name.
First I created the commands group called cli like
@click.group()
def cli:
pass
This will create a group of commands in which we can add up on existing commands. I created the config command and its arguments using click option
@cli.command(<command-name>)
@cli.option(<option-name>)
by default the command name will be name of function but that can hinder our naming since we cant use '-' in function names so we can use .command()
I called the commands under these function so that when we call this CLI commands our functionality will be executed
Using Pytest:
I learnt Mocking using pytest. Even though we have default mocking in pytest I preferred to use unittest.mock and patch with pytest
I used @patch to mock a function
usually mocking is done if there is a API call or another function is used inside testing function so that we can create fake data.
I used .side_effect to mimic a mocking function with another function and return the values I want.
For these mock functions we also need mock response we can get it by getting single value from previous input.
I am an amateur programmer trying to learn new things by building projects please feel free to give feedback