Split available dataset into training and test

 How to  split data set into training and test data set

 
We can train the model using data which we call as training data or training set. The training data is the one which already has the actual value that the model should have predicted and thus the algorithm changes the value of parameters to account for the data in the training set.

But how do we know after training the model is overall good ?
For that, we have test data/test set which is basically a different data for which we know the values but this data was never shown to the model before. Thus if the model after training is performing good on test set as well then we can say that the Machine Learning model is good.

If the model is not tested and is made such that it just perform good on training data then parameters will be such that they are only good enough to predict the value for data which was in training set. That is not general. This is called overfitting.

So we don’t land making a useless model which is only good for the training set and not general enough.

split data set into training and test data set in python :

first we load the data set


now we declare x & y variable which we all know that we have to pass this in fit method during training our model.


Now we use sklearn library's train_test_split module and split the data into training and test data.


Here  test_size = 0.3 means 30% of data is test data set and 70% of data we use for train our ml model.

Lets check how X_train & y_train look . NOTE that it's randomly select data


Hope you understand.

if you want to download data set then click here.








Comments

Post a Comment

Popular posts from this blog

Multivariate logistic regression in Python

Image
 Multivariate logistic regression Difference between single variate and multivariate regression :  Suppose we have data of the customer will buy insurance or not? Here you can see your prediction or target is only yes or no. you can also say that 0 or 1 .  now suppose you have to  build model in which you have to decide which party a person is going to vote for? so there are many target value for this like democratic,republic,independent and many more,Here you have more then two possible outcome so its called  multivariate regression. Multivariate logistic regression in Python: Here problem is  given that we have hand written digit(0 to 9) data set and we have to build machine learning model that predict the which digit is it. So you can clearly notice that our target value is 0 to 9 which means total 10 possible outcome. so it is the example of Multivariate regression. We will use logistic regression for this. First of all import library Now we load t...
Read more

Decision tree for titanic dataset in Python

Image
  Build decision tree model to predict survival based on certain parameters : import library Read data set using panda library's read_csv method.   In this file using following columns build a model to predict if person would survive or not  1.Pclass 2.Sex 3.Age 4.Fare so we will drop unnecessary columns :   Now we separate dependent and independent   data frame for pass into our model Gender column contain categorical data so we have too convert it into numerical for this we will use map method of pandas. and also it contain missing value which we will handle using fillna method.   Now train test split : Building Decision tree model for titanic dataset in Python Confusion matrix : if you want to download data set then click here.
Read more

K Means Cluster Algorithm

Image
 K Means Cluster  Its a distance based algorithm. The k-means algorithm is a  clustering algorithm . That means that you have a bunch of points in some space, and you want to guess what groups they seem to be in.  K-Means cluster in Python : First of all import necessary  library and read the csv file using panda library Lets visualize data    Now we build the KMeans  cluster Now we find the center of group  You can clearly saw that center is not perfectly fit in group because we didn't performing scaling  on over data set. so first we do the scaling on over data set. Now again we build KMeans cluster model Now we again find center of our data groups Now question is How we know the value of K ?  We can find the value of K using elbow method. In this method we plot the data and where you find the major difference in graph that point is your value of K. You can download data set by click here.
Read more