Discover a Houseplant App

Did you know that January 10th was National Houseplant Day? It was a great coincidence that around that time, I was working on an app that is supposed to recommend similar houseplants to people so that they can discover different plants they might like.

Motivation

I got the inspiration to create an app revolving around houseplants because my mom, cousin, and best friend are plant lovers. Their houses are basically mini-forests. I scraped all my data off of houseplant411.com, created a crude recommendation system and dressed it up in a flask app which was lastly deployed on Heroku.

Data

The data I scraped included images of the 136 different popular house plants, their descriptions, and other information related to their care such as light, water, fertilizer, temperature, humidity, flowering, pests, diseases, soil, pot size, pruning, propagation, special occasion, and poisonous plant information. I used beautiful soup to do all of my web-scraping.


The Recommendation System

At the core of my app is a recommendation system made possible by using cosine similarity on vectors created using Tfidfvectorizer. The vectors came from the text descriptions of the plants and care information related to the plants. I compared the vectors that came from just the text descriptions using cosine similarity to create a cosine similarity matrix. I also compared the vectors that came from care information using cosine similarity to create another matrix. I then found the average of the two matrices to create my last matrix which I used to recommend 5 different, yet similar plants to a plant of the user's choice.

The App

The application can be found here - https://discover-a-houseplant.herokuapp.com/

The idea of the app is that three random plants are generated from my list of 136 plants. A user chooses one of the plants he or she likes and then they are brought to a new page which has 5 similar, yet different plants to the plant he or she chose. There is a photo, a name and a description of each plant on each page of the application.

Results

You can decide if you like the recommendations put forth by this app or not! But, I think this is a fun way to discover new plants.

Code

The code for this project can be found on my Github at https://github.com/morningkaren/discover-a-houseplant .

US Traffic Accidents Trends (Part 1)

Happy 2020!

A new year calls for a new project. I decided to look at a Kaggle dataset about US traffic accidents from February 2016 to March 2019. Since the data set is really large, with over 2.5 million rows and 49 columns, I believe there is a lot to analyze. This entry is just part 1 of my analysis, but it already has attempted to answer some interesting questions that I came up with.

I thought this dataset would be interesting mainly because it is so large. But, I am also interested to see if I can find any insights in traffic accident data to understand more about the behavior of drivers. Some preliminary questions I wanted to answer was how does weather affect the severity of traffic accidents, how does the time of day affect the severity of accidents, and where do the most accidents occur?

Those questions require some assumptions and definitions. Severity is ranked between 1 and 4, where 1 indicates the least impact on traffic and 4 indicates a significant impact on traffic. I also assume that the dataset is representative of all the accidents that occurred in the US and that there is little to no bias of the amount of accidents reported in each location. (The dataset was gathered by tracking multiple Traffic APIs.)

Most accidents occur in California

 484,706 of them. I am not surprised by this finding. California is a huge state and has the most people living there in the U.S. Texas is second place, but that makes sense too. It is also a huge state and has a lot of people living there.

Big Picture - A Bimodal Distribution of Accidents Over the Hours

If I analyze the distribution of accidents over 24 hours, I see a bimodal distribution in all Severity levels. Take a look.

It seems as if the number of accidents peak at around 7 am, dips in the afternoon and comes up in the late afternoon around 5-6pm. The very early hours and the very late hours have the least amount of accidents. If you look a the y axis which represents the number of accidents in each graph, we can see that there are relatively few Severity Level 1 accidents and the most Severity Level 2 accidents. (Level 2 > Level 3 > Level 4 > Level 1)

Zooming In: Top 10 Urban Cities vs Top 10 Rural States

Another question that came up was whether the big picture trend, that the number of accidents have a bimodal distribution is consistent if we zoom in on urban ares and rural areas. I found the top 10 urban cities (2014) on this website. (The website takes into account more than 100 variables to determine the urban-ness of a city.) I also found the most rural states from this Quora post. (I based a state's rural-ness by the amount of land that is 'rural'.) Since according to the 2010 Census, some 80% of the U.S. population lives in urban areas, I felt comparing the top 10 urban cities and the top 10 rural states made sense. 

The top urban cities are

1. New York, NY

2. San Francisco, CA

3. Boston, MA

4. Jersey City, NJ

5. Washington, DC

6. Miami, FL

7. Chicago, IL

8. Seattle, WA

9. Philadelphia, PA

10. Minneapolis, MN

The top rural states are

1. Alaska

2. Wyoming

3. Montana

4. North Dakota

5. South Dakota

6. Idaho

7. New Mexico

8. Nebraska

9. Nevada

10. Utah

* Data from Alaska does not exist in the dataset since the dataset only has data from the contiguous U.S.

Analysis of the top urban cities show that the trend in Severity Level 2 and 3 accidents do look like the big picture trend, but take a look at Severity Level 4 accidents - 

The bimodal trend is not seen in Severity Level 4 accidents in the 10 most urban cities. The most number of Severity Level 4 accidents occur in the morning at around 6-7am. Rush hour. Then, it drops throughout the day. 

Analysis of the top rural states show that the trend in Severity Level 2 and 3 accidents also have a bimodal trend, but Severity Level 4 accidents look like this - 

The bimodal trend is not seen here either! In fact, very severe accidents occur in the middle of the day  at around 12-1pm. 

This is quite interesting and I wonder why the curve looks the way it does in urban and rural areas of the U.S.

Weather Condition Might Not Have An Effect on Severity of Accidents

It is quite interesting, but it may be the case that weather conditions, especially bad weather conditions might not have an effect on severity of accidents. For one, the most number of accidents occur on Clear days. So, we can say that if the day is clear, there may be more accidents. But, that is simply because more people will drive on clear days. There are fewer accidents on non-clear days, and the trend of the severity levels is still about the same for clear and non clear days, with the most number of Severity 2 accidents and least number of Severity 1 accidents.

Out of the 2.25 million accidents reported, 808,182 of those accidents occurred on clear days. The next weather condition to have the most number of accidents is overcast, with 382,482 accidents.

Only one traffic accident occurred in weather conditions of heavy smoke, dust whirls, and blowing sand. 

Wrapping up brief EDA

This exploratory data analysis of the U.S. traffic accident dataset was really fun. At first, I didn't know where to begin because I had so much data and so many questions. But, tackling one question at a time was a good start. As a conclusion, I would like to say that sqlite3 saved the day. I remember learning about sqlite3 at Metis very briefly and didn't understand why I would ever use it. 

sqlite3 turns out to be great when I have a csv file and want to run SQL queries on it in a Jupyter notebook. I simply create an empty database, to_sql my csv or a dataframe into the empty database, then run my query on that new table in my database. 

Here is the skeleton of the code I used to run a SQL query using sqlite3 in a jupyter notebook setting: 

Through this EDA, I was able to really understand aggregating over a CASE WHEN statement too. I struggled with that idea and how to implement it so that I get the data in a format that I wanted to. I used SQL and pandas to do groupbys, but since I was practicing a lot of SQL, I found that I gravitated towards using CASE WHEN with SQL first, rather than using pandas groupby function.

Overall, this EDA was a lot of fun and the insights were interesting. I will be continuing this EDA project, as there are so many other questions to ask and so many different variables to look at (there are 49 columns!). 

For now, thanks for coming along with me on my first EDA of 2020.

An Exploration of Classification Models using Mobile Dataset

I wanted to review some classification algorithms, so I found a clean data set on Kaggle about classifying mobile price ranges to practice with logistic regression, decision tree algorithm, and random forest. I will be walking through my thought process when attempting this exploration of the data set from Kaggle.

The Dataset

The data set has its own story about how Bob wants to start his own mobile company, but does not know how to estimate the price of mobiles his company creates. He decides to collect sales data of mobile phones from various companies and decides to find some relation between the features of the phone and its selling price.

The features of the phone include battery power, bluetooth capabilities, clock speed, dual sim option, front camera mega pixels, 4G capabilities, internal memory, mobile depth, mobile weight, number of core processors, primary camera mega pixels, pixel height resolution, pixel width resolution, RAM, screen height, screen width, talk time with a single battery charge, 3G capabilities, touch screen, and wifi capabilities.

The data set has a test and train file, but because there were no price ranges in the test file, I decided to just use my training data and create a separate test file out of some of my training data. This decreased the amount of data I could train, but I felt that it was ok for this exploration.

Exploratory Data Analysis

Some things I did before I ran any models was to look at pairplots using seaborn. Since there were so many features, I separated the dataset into three groups with about 7-8 features in each set of pairplots and compared them with the price range, which is denoted  0, 1, 2, or 3. I looked at pairplots to see if there are any features that are linearly separable. (This would show me that I could use logistic regression and it would work well.)

How can I tell if some features are linearly separable or not? Well, I looked at each pairplot to see if there were features that overlapped the least. For example, if I look at the distribution plot of battery power with the different price ranges as the "hue" attribute, I see that the blue and red hues are clearly separated. There is definitely overlap, but there is a huge separation between the blue and red distributions.

There is some separation in the pixel width feature too.

But the separation with the RAM is most significant.

So you might ask what are some examples of features that do not look separable? Well, I would say the front camera pixels feature is not linearly separable, since there seems to be an equal distribution of all the four colors in the same way. Look at the front camera pixels distribution-- the colors overlap almost perfectly.

So, now I know that certain features like RAM, battery power, and pixel width may be very telling of the price range of mobile phones, but something like the front camera pixels might not be very telling.

Are classes balanced?

Another thing to look at before doing any modeling is to check the class balance of the target. My target has four classes that tell me how expensive a mobile phone is-- class 0, 1, 2, and 3. Looking at a distribution plot of the price range, I see that it is uniform. This rarely happens in real life, but again, this is a clean Kaggle data set.

Balanced classes tell me that I would not need to do any over or undersampling or class weights or change the threshold of my classes after modeling.

Logistic Regression Model

I used sci-kit learn's modules to run my logistic regression model. I did two train, test, splits to divide my data set into a train-validate-test set of 60-20-20.  I also imported StandardScaler to standardize my features. I looked at many metrics, including F1, accuracy, precision, and recall. I also decided to look at a confusion matrix, to practice and make sure that I am getting the metrics right.

For my logistic regression model, I just decided to focus on the three features I found that looked more linearly separable than the rest of the features--battery_power, ram, and px_width. I standard scaled my features, then fit my model to my training set. I predicted using my validation set and used a classification report to see the precision, recall, F1, and accuracy scores. Using just three features, I got an accuracy of 82%, and an F1 score of predicting class 0 as .91 and class 3 as .93, class 1 as .68, and class 2 as .71. That's not bad.

Logistic Regression-Confusion Matrix

This part will cover how to calculate precision, recall, and accuracy using the confusion matrix. The confusion matrix basically tells me the actual vs. predicted values of my target. For example, in this confusion matrix, there are 95 cases of phones predicted as class 0 and were actually class 0.

So, what is accuracy in relation to the confusion matrix? Accuracy is the total number of true positives over the total number of cases. (An example of a true positive is a case that is predicted to be 0 and is actually 0.) In this case, it is the sum of the diagonal that goes from top left to bottom right over the sum of the numbers in the whole matrix- 325/328 which rounds to .82 (same number as reported above).

What is precision in relation to the confusion matrix? Precision is the total number of true positives over the sum of true positives and false positives (an example of a false positive is a case that is predicted to be 0 but is not 0.). We will have to look at precision class to class. In this case, the total number of true positives for class 0 is 95. The total number of false positives is 19. So, the precision is 95/114 which rounds to .83.

What is recall in relation to the confusion matrix? Recall is the total number of true positives over the sum of true positives and false negatives. (An example of a false negative is a case that is predicted to not be 0 but is actually 0.) In this case, the total number of true positives for class 0 is 95 and the number of false negatives is 0, so precision is 1.

What is F1 in relation to precision and recall? F1 is actually the harmonic mean of the precision and recall scores.

Decision Tree

Using sklearn's decision tree algorithm is simple. You instantiate the DecisionTreeClassifier and set a max depth for the tree. Then you fit your training set and then predict using your validation set. The decision tree performed poorly in comparison to the logistic regression algorithm, with an accuracy of .26. We can guess the classes of the mobile phones and do just as well as the decision tree classifier.

I want to make a note that instead of using just three features, I used all the features available to me when training and fitting my decision tree.

The F1 score for class 0 was .28, class 1 was .33, class 2 was .14, and class 3 was .27.

Random Forest

Similarly, random forest did not perform that well compared to logistic regression. I used 1000 estimators (basically individual decision trees) and a max depth of 2 in my random forest model. The random forest basically averages out many decision trees to give a more robust and accurate prediction, but in this case, the accuracy score turned out o be .24. The F1 score for class 0 was .28, class 1 was .31, class 2 was .12, and class 3 was .27.

However, what was interesting to see was the feature importances in my random forest model. The following feature importance graph shows the relative importance of each feature in determining the information gained from splitting at that feature. Here, ram, battery power, px_height, and px_width showed the information gain. This is consistent with the fact that I noticed that ram, battery power and px_width showed much linear separability from the pairplots done in the exploratory data analysis portion above.

Logistic Regression worked best

A reason why I think logistic regression worked best in this scenario is that there were features that were linearly separable. In cases where there was little linearly separability, random forest might have worked better.

A final note

Improvements could be made on my model with more investigation of how to obtain a F1 score of when predicting class 1 and class 2 mobiles. Perhaps running a logistic regression model on all with all of the features and lowering the C parameter (a regularization parameter, which I set to 1 which means no regularization was used) to make sure the model doesn't over-fit. That would be interesting to see.

2019 Year In Reflection

As 2019 ends, I find myself being very excited for all the things I will be doing in 2020. But, first, it is important to reflect on the past year, express my gratitudes, and learn from my mistakes.

I knew the first thing of 2019 I will be doing is going to be the end (perhaps) of my career in teaching. I left the a teaching position in the Bronx and decided to study for an actuarial exam (FM) full time. But, I always had the feeling that an actuarial career path was a wrong puzzle piece in the picture of my life. I still tried hard to see if that puzzle piece would fit though. But, I tried other puzzle pieces too through networking with people from different walks of life and starting to dabble in some data projects. I connected with a data scientist friend and he showed me the work he was doing for his company at the time-- SQL queries, A/B testing, coding with Python, influencing decisions using insights gathered from data. Is this the future...(my future?) I thought.

When I passed FM in April, I made another decision. I will not continue to take actuarial exams. Instead, I will pursue this data science thing full-time. Was I fickle? Do I get swept away with interesting ideas and new things too easily? In retrospect, it seems like I was simply following my instincts and my instincts told me that data science could be right for me. But, instincts doesn't really give you all the technical skills you need to be a leader in data science. Although I had already studied some Python and done some data projects on my own, I was lacking in everything a modern day data scientist needed.

Therefore, I joined Metis, a 12 week data science bootcamp to skill up. Really, it was the start of a career change and a new chapter in my life where I met new friends and learned the basics and the latest of machine learning and data science. Although I say that Metis was the start of a career change, I believe my mindset has already been set in the beginning that I will change my career with success.

9 months into 2019, I have left my teaching job, passed an actuarial exam, and graduated from a data science bootcamp. The next three months was spent in coffee shops, yoga studios, cross-fit boxes, libraries, networking events, at home, but I had one main goal. I was going to find my first full time position as a data scientist...or analyst.

I can say that I have been scouted by a company and although it is for an internship, I am really happy that a door opened up for me in data science.

I know that 2019 has been really focused on my career change. My career change, although scary, has been the one thing that remained constant throughout the year.

As for all the people who made this year possible, I would have to thank my parents, sister, boyfriend, and all my friends. I'm grateful for the life I had lived.

Coming into the new year, I think there is one thing among many things that I would like to work on. It is being aware. I want to be more aware of myself, body and mind, and of the filters I put in the world. I believe being aware has been an unconscious goal of mind and I am really happy that I became aware of this goal of mine this time around and am able to articulate it with words.

Thank you 2019.

Notes on Root Mean Squared Error and R^2

Linear Regression Metrics

Two popular linear regression metrics are the root mean squared error and R^2.

Say you have a data set and have plotted the line of best fit. How well does the line of best fit actually model the data set? Visually, if the data points are very close to the line of best fit, you can say that it is a pretty good model compared to data points that are scattered everywhere and not particularly hugging the line of best fit. But, there is a more rigorous way of telling whether your linear model is good or not.

SSE- Sum of Squared Errors 

The most straightforward way to see whether or not your model is a good fit compared to other models is to find the sum of the squared errors. This means you sum up all the squared values of the difference of the actual values and the predicted values. The lower the sum, the better the model.

MAE- Mean Absolute Error

Another way to compare models is to find the mean absolute error, which is just the average of the absolute values of the difference of the actual values and predicted values. Again, the lower the mean, the better the model. SSE is more sensitive to outliers (since we are squaring the values) than MAE is. But, MAE requires less refinement to be interpretable since the MAE returns the original units of the values. You would need to divide SSE by the total number of values and then take the square root of that for it to be more interpretable. Which leads us to...

Root Mean Squared Error

The root mean squared error is simply the square root of the SSE divided by the total number of values. This creates a more interpretable metric for how good our model is. Again, RSME, like SSE is more sensitive to outliers and would penalize them more than MAE.

R^2

R^2, or the coefficient of determination, is another metric you could use to see how good your model is. R^2 is usually between 0 and 1, but there are cases where R^2 can be negative. (That's really 'bad'.) The idea behind R^2 is to measure the target "variance explained" by the model. (How much variance of the target can be explained by the variance of the inputs?)

SST, or the sum of the squares total) is the target's intrinsic variance from its mean. It is given by the sum of the squares of the difference of the actual values and the mean of the actual values.

SSE/SST gives us the percent of the variance that remains once you've modeled the data. The sum of the squared errors over the sum of the squares total is the portion of the variance that the model cannot explain. However, we can subtract SSE/SST from 1 and get the portion of the variance that the model can explain.

1 - SSE/SST is the coefficient of determination, or R^2. "R^2 measures how good [your] regression fit is relative to a constant model that predicts the mean".

(Usually SSE is smaller than SST, assuming that our model does a little or much better than just predicting the mean all the time, so R^2 is between 0 and 1.)

Forest Fires in Brazil

Context

As a person who cares about environmental issues, coming across a data set about forest fires in Brazil on Kaggle was very exciting. The data set contains the number of forest fires in 23 states in each month over the span of years from 1998 to 2017 reported by the Brazilian government. I decided to analyze the data set to find the trend of the number of forest fires, to find when forest fires occur with the highest frequency, and to create a Tableau dashboard to visualize the change in the number of forest fires over time.

engine = 'python'

When reading the csv file with the data set, I had to set the engine parameter to python because UTF-8 couldn't decode some characters.
I realized that the month names were in Portuguese, so I decided to change all the month names to numeric names. I created a new column in my dataframe that are the numeric equivalents to the month names.

Geocoding

 I wanted to have the latitude and longitude for each state in the data set so that I can graph it in the future. So, I used locationiq's API to find the latitude and longitude for each state.

I then was able to create a dictionary of the states and their respective latitude and longitude values. I was able to use two apply functions to apply to each row the correct latitude and longitude based on the state.

Average Number of Forest Fires Per Year in Each State

The visual I wanted to create is one of a map which shows the average number of forest fires per year in each state in the data set with a circle. The larger and darker the circle, the higher the average number of forest fires per year. I would need to create a dataframe with the average number of forest fires per year in each state first. Here is the code, which required use of a groupby:


Total Number of Forest Fires Per Year

It would be great if I could see the overall trend of forest fires throughout the years. I decided to use another groupby to find the sum of all forest fires over the years. I found that there was an increasing trend.

A dashboard where the user could click on a point on the line graph which corresponds to a specific year and the number of forest fires would filter the map of Brazil to the corresponding year and show the average number of forest fires in each state would be pretty nice. So, I did that.

Monthly Trends in Forest Fires

Before I show the Tableau dashboard, I would like to show a graph which represents the monthly trend of forest fires over the years. As you can see, the graph below shows that the number of forest fires are low in the beginning of the year, increases quickly in June, peaks at July, drops a bit in September, then spikes again in October.

Looking at a bit more granular of a level, you can see the shift in the lines upward since 1998 of the number of forest fires over months. This supports the positive trend seen in the number of forest fires over the years.

An Increasing Trend of Forest Fires in Brazil

Lastly, I would like to present the Tableau dashboard I created with a short video. You can see the change in the sizes and shade of the circles that represents the average number of forest fires in each state over time. The trend line of the total number of forest fires each year is also there. Sao Paulo always has a very large average number of forest fires each year.

Once Upon A Time- NLP on Disney Movie Scripts & Their Original Stories

Introduction

Practically everyone knows some Disney movies, but not everyone knows about their original stories. The movies' original stories come from many places. For example, The Little Mermaid comes from a short story by Christian Andersen and The Hunchback of Notre Dame comes from a gothic novel written by Victor Hugo. It would be interesting to see the parallels between the Disney movie scripts and their original writings, which is exactly what my project seeks to do. In addition to finding interesting relationships between main characters in the movie scripts and original stories, my project also culminates in a rudimentary flask app that is also a book/movie recommendation system.

Methodology

I first gathered 17 Disney movie scripts and their corresponding original stories. Then, I tokenized my data in two different ways-- on the word level and on the sentence level. I tokenized my data on the word level with TF-IDF, removed punctuation and used stop-words, lemmatization, and parts of speech tagging to only keep nouns and adjectives. I did this pre-processing in order to feed my data into an NMF model to do topic modeling and also used Word2Vec and PCA to find relationships between main characters in the original stories and in the movie scripts. I also tokenized my data on the sentence/line level, keeping punctuation in order to do sentiment analysis with Vader and applied dynamic time warping techniques to compare sentiment change over time in the books and movies. With my NMF model and sentiment analysis, I was able to come up with a rudimentary book/movie recommendation system. The user puts in a story/movie from the list of 34 stories/movies they liked, a sentiment similarity weight and a topic similarity weight and the system returns a book or a movie that matches the criteria. 

My methodology is summed up in this picture:

Findings

Word2Vec & PCA

Using Word2Vec and reducing dimensions to 2 with PCA, I was able to visualize the spread of main characters in the stories and in the movies. In the original stories, there is a greater spread of main characters, with some characters closer to the words "love" and "happy" and some characters rather far away from them. However, in the movies, the main characters are bunched together and everyone seems to be relatively close to the words "love" and "happy". Does this mean that the Disney movies are just more similar to each other and sugar-coats the original stories? This idea makes sense since a lot of the original stories are more dark and does not always have happy endings. For example, Victor Hugo's novel of the hunchback of Notre Dame has a lot of the main characters dying, but the Disney version is a lot happier. 

Sentiment Analysis 

Using Vader, I was able to assign a sentiment score to each line or sentence of the movie script or story. This created a time series of sentiment change over the plot of the movie or story. I then used dynamic time warping techniques to compare sentiment change over time to compare how similar the movie and their corresponding story 'feels'. My findings show that the most similar book and movie pair is Winnie the Pooh by A.A. Milne and  The Adventures of Winnie the Pooh. The Winnie the Pooh book/movie pair received a dynamic time warping distance of 7, the lowest of all the other pairs of books and movies.  The most dissimilar book and movie according to sentiment is The Hunchback of Notre Dame by Victor Hugo and The Hunchback of Notre Dame.  This book/movie pair received a dynamic time warping distance of 33. You can see the change in sentiment over time in the following graphs. (I used a rolling mean of 5% of the lines/sentences to smooth out the graphs.)

Recommendation System

Finally, my recommendation system was created by doing topic modeling over the 34 total number of books and movies and dynamic time warping scores. NMF was able to create 34 topics that corresponded to the movie or a book very well. I also wanted to be able to create a recommendation based on how similar the user wanted the recommended book or movie to feel to their chosen book or movie and how similar in topic the user wanted the recommended book or movie to be to their chosen book or movie. To give users those options, I used cosine similarity to compare each book or movie against another book or movie using the topic weights vectors I received from NMF. I also used the dynamic time warping distances to compare how similar each book/movie was to another book/movie. 

Here is a demonstration of the flask app I built:

I will end with a quote: 

"Begin at the beginning," the King said, very gravely, "and go on till you come to the end: then stop."

-Lewis Caroll, Alice in Wonderland