Gender Recognition on Dutch Tweets - CLIN Journal

Computational Linguistics in the Netherlands Journal 4 (2014) 171-190

Submitted 06/2014; Published 12/2014

Gender Recognition on Dutch Tweets Hans van Halteren Nander Speerstra

[email protected] [email protected]

Radboud University Nijmegen, CLS, Linguistics

Abstract In this paper, we investigate gender recognition on Dutch Twitter material, using a corpus consisting of the full Tweet production (as far as present in the TwiNL data set) of 600 users (known to be human individuals) over 2011 and 2012. We experimented with several authorship profiling techniques and various recognition features, using Tweet text only, in order to determine how well they could distinguish between male and female authors of Tweets. We achieved the best results, 95.5% correct assignment in a 5-fold cross-validation on our corpus, with Support Vector Regression on all token unigrams. Two other machine learning systems, Linguistic Profiling and TiMBL, come close to this result, at least when the input is first preprocessed with PCA.

1. Introduction In the Netherlands, we have a rather unique resource in the form of the TwiNL data set: a daily updated collection that probably contains at least 30% of the Dutch public tweet production since 2011 (Tjong Kim Sang and van den Bosch 2013). However, as any collection that is harvested automatically, its usability is reduced by a lack of reliable metadata. In this case, the Twitter profiles of the authors are available, but these consist of freeform text rather than fixed information fields. And, obviously, it is unknown to which degree the information that is present is true. The resource would become even more useful if we could deduce complete and correct metadata from the various available information sources, such as the provided metadata, user relations, profile photos, and the text of the tweets. In this paper, we start modestly, by attempting to derive just the gender of the authors1 automatically, purely on the basis of the content of their tweets, using author profiling techniques. For our experiment, we selected 600 authors for whom we were able to determine with a high degree of certainty a) that they were human individuals and b) what gender they were. We then experimented with several author profiling techniques, namely Support Vector Regression (as provided by LIBSVM; (Chang and Lin 2011)), Linguistic Profiling (LP; (van Halteren 2004)), and TiMBL (Daelemans et al. 2004), with and without preprocessing the input vectors with Principal Component Analysis (PCA; (Pearson 1901); (Hotelling 1933)). We also varied the recognition features provided to the techniques, using both character and token n-grams. For all techniques and features, we ran the same 5-fold cross-validation experiments in order to determine how well they could be used to distinguish between male and female authors of tweets. In the following sections, we first present some previous work on gender recognition (Section 2). Then we describe our experimental data and the evaluation method (Section 3), after which we proceed to describe the various author profiling strategies that we investigated (Section 4). Then follow the results (Section 5), and Section 6 concludes the paper.

1. For whom we already know that they are an individual person rather than, say, a husband and wife couple or a board of editors for an official Twitterfeed.

c

2014 van Halteren and Speerstra.

2. Gender Recognition Gender recognition is a subtask in the general field of authorship recognition and profiling, which has reached maturity in the last decades (for an overview, see e.g. (Juola 2008) and (Koppel et al. 2009)). Currently the field is getting an impulse for further development now that vast data sets of user generated data is becoming available. Narayanan et al. (2012) show that authorship recognition is also possible (to some degree) if the number of candidate authors is as high as 100,000 (as compared to the usually less than ten in traditional studies). Even so, there are circumstances where outright recognition is not an option, but where one must be content with profiling, i.e. the identification of author traits like gender, age and geographical background. In this paper we restrict ourselves to gender recognition, and it is also this aspect we will discuss further in this section. A group which is very active in studying gender recognition (among other traits) on the basis of text is that around Moshe Koppel. In (Koppel et al. 2002) they report gender recognition on formal written texts taken from the British National Corpus (and also give a good overview of previous work), reaching about 80% correct attributions using function words and parts of speech. Later, in 2004, the group collected a Blog Authorship Corpus (BAC; (Schler et al. 2006)), containing about 700,000 posts to blogger.com (in total about 140 million words) by almost 20,000 bloggers. For each blogger, metadata is present, including the blogger’s self-provided gender, age, industry and astrological sign. This corpus has been used extensively since. The creators themselves used it for various classification tasks, including gender recognition (Koppel et al. 2009). They report an overall accuracy of 76.1%. Slightly more information seems to be coming from content (75.1% accuracy) than from style (72.0% accuracy). However, even style appears to mirror content. We see the women focusing on personal matters, leading to important content words like love and boyfriend, and important style words like I and other personal pronouns. The men, on the other hand, seem to be more interested in computers, leading to important content words like software and game, and correspondingly more determiners and prepositions. One gets the impression that gender recognition is more sociological than linguistic, showing what women and men were blogging about back in 2004. A later study (Goswami et al. 2009) managed to increase the gender recognition quality to 89.2%, using sentence length, 35 non-dictionary words, and 52 slang words. The authors do not report the set of slang words, but the non-dictionary words appear to be more related to style than to content, showing that purely linguistic behaviour can contribute information for gender recognition as well. Gender recognition has also already been applied to Tweets. Rao et al. (2010) examined various traits of authors from India tweeting in English, combining character N-grams and sociolinguistic features like manner of laughing, honorifics, and smiley use. With lexical N-grams, they reached an accuracy of 67.7%, which the combination with the sociolinguistic features increased to 72.33%. Burger et al. (2011) attempted to recognize gender in tweets from a whole set of languages, using word and character N-grams as features for machine learning with Support Vector Machines (SVM), Naive Bayes and Balanced Winnow2. Their highest score when using just text features was 75.5%, testing on all the tweets by each author (with a train set of 3.3 million tweets and a test set of about 418,000 tweets).2 Fink et al. (2012) used SVMlight to classify gender on Nigerian twitter accounts, with tweets in English, with a minimum of 50 tweets. Their features were hash tags, token unigrams and psychometric measurements provided by the Linguistic Inquiry of Word Count software (LIWC; (Pennebaker et al. 2007)). Although LIWC appears a very interesting addition, it hardly adds anything to the classification. With only token unigrams, the recognition accuracy was 80.5%, while using all features together increased this only slightly to 80.6%. Bamman et al. (2014) examined about 9 million tweets by 14,000 Twitter users tweeting in American English. They used lexical features, and present a very good breakdown of various word types. When using all user tweets, they reached an accuracy of 88.0%. An interesting observation is that there is a clear class of misclassified users who have a majority of opposite gender users in their social network. 2. When adding more information sources, such as profile fields, they reach an accuracy of 92.0%.

172

For Tweets in Dutch, we first look at the official user interface for the TwiNL data set, http://www.twiqs.nl. Among other things, it shows gender and age statistics for the users producing the tweets found for user specified searches. These statistics are derived from the users’ profile information by way of some heuristics. For gender, the system checks the profile for about 150 common male and 150 common female first names, as well as for gender related words, such as father, mother, wife and husband. If no cue is found in a user’s profile, no gender is assigned. The general quality of the assignment is unknown, but in the (for this purpose) rather unrepresentative sample of users we considered for our own gender assignment corpus (see below), we find that about 44% of the users are assigned a gender, which is correct in about 87% of the cases. Another system that predicts the gender for Dutch Twitter users is TweetGenie (http://www.tweetgenie.nl), that one can provide with a Twitter user name, after which the gender and age are estimated, based on the user’s last 200 tweets. The age component of the system is described in (Nguyen et al. 2013). The authors apply logistic and linear regression on counts of token unigrams occurring at least 10 times in their corpus. The paper does not describe the gender component, but the first author has informed us that the accuracy of the gender recognition on the basis of 200 tweets is about 87% (Nguyen, personal communication).3 In later experiments, Nguyen et al. (2014) did a crowdsourcing experiment, in which they asked human participants to guess the gender and age on the basis of 20 to 40 tweets. When using a majority vote to represent the crowd’s opinion, the crowd’s perception of the gender on the basis of the tweets coincided with the actual gender in about 84% of the cases. The conclusion is not so much, however, that humans are also not perfect at guessing age on the basis of language use, but rather that there is a distinction between the biological and the social identity of authors, and language use is more likely to represent the social one (cf. also (Bamman et al. 2014)). Although we agree with Nguyen et al. on this, we will still take the biological gender as the gold standard in this paper, as our eventual goal is creating metadata for the TwiNL collection.

3. Experimental Data and Evaluation In this section, we first describe the corpus that we used in our experiments (Section 3.1). Then we outline how we evaluated the various strategies (Section 3.2). 3.1 Corpus Used in the Experiments We selected our experimental material from the TwiNL data set (Tjong Kim Sang and van den Bosch 2013), which was collected by searching for tweets with any of a number of probably Dutch words, after which a character n-gram language filter was applied. The collection is estimated to contain 30-40% of all public Dutch tweets. From this material, we considered all tweets with a date stamp in 2011 and 2012. In all, there were about 23 million users present. Of these, we only considered the ones who produced 2 to 10 tweets on average per day over 2011 and 2012. The minimum ensured a sufficient amount of text (1500-7300 tweets) for classification; the maximum served to avoid very high volume users, who might be professional (multi-user/edited) feeds or even twitterbots. This restriction brought the number of users down to about 270,000. We then progressed to the selection of individual users. We aimed for 600 users. We selected 500 of these so that they get a gender assignment in TwiQS, for comparison, but we also wanted to include unmarked users in case these would be different in nature. All users, obviously, should be individuals, and for each the gender should be clear. From the about 120,000 users who are assigned a gender by TwiQS, we took a random selection in such a manner that the volume distribution (i.e. from 2 to 10 tweets per day average) is equally spread throughout the range and approximately equal for men and women. We checked gender manually for all selected users, mostly on the basis 3. As in our own experiment, this measurement is based on Twitter accounts where the user is known to be a human individual.

173

of the profile texts and profile photo’s, and only included those for which we were convinced of the gender.4 Later even more detailed rechecks, after a few extremely unlikely classification results, served to clean up the (hopefully) last gender assignment errors.5 The final corpus is not completely balanced for gender, but consists of the production of 320 women and 280 men. However, as research shows a higher number of female users in all as well (Heil and Piskorski 2009), we do not view this as a problem. From each user’s tweets, we removed all retweets, as these did not contain original text by the author. Then, as several of our features were based on tokens, we tokenized all text samples, using our own specialized tokenizer for tweets. Apart from normal tokens like words, numbers and dates, it is also able to recognize a wide variety of emoticons. The tokenizer is able to identify hashtags and Twitter user names to the extent that these conform to the conventions used in Twitter, i.e. the hash (#) resp. at (@) sign are followed by a series of letters, digits and underscores. URLs and email addresses are not completely covered. The tokenizer counts on clear markers for these, e.g. http, www or one of a number of domain names for URLs. Assuming that any sequence including periods is likely to be a URL proves unwise, given that spacing between normal words is often irregular. And actually checking the existence of a proposed URL was computationally infeasible for the amount of text we intended to process. Finally, as the use of capitalization and diacritics is quite haphazard in the tweets, the tokenizer strips all words of diacritics and transforms them to lower case. 3.2 Evaluation We divided our corpus in five parts, each containing (approximately) the same number of male and female authors.6 We used this division in all experiments, each time using four parts as training material and one as test material. For those techniques where hyperparameters need to be selected, we used a leave-one-out strategy on the test material. For each test author, we determined the optimal hyperparameter settings with regard to the classification of all other authors in the same part of the corpus, in effect using these as development material. In this way, we derived a classification score for each author without the system having any direct or indirect access to the actual gender of the author. We then measured for which percentage of the authors in the corpus this score was in agreement with the actual gender. These percentages are presented below in Section 5.

4. Profiling Strategies In this section, we describe the strategies that we investigated for the gender recognition task. As we approached the task from a machine learning viewpoint, we needed to select text features to be provided as input to the machine learning systems, as well as machine learning systems which are to use this input for classification. We first describe the features we used (Section 4.1). Then we explain how we used the three selected machine learning systems to classify the authors (Section 4.2). 4.1 Machine Learning Features We restricted ourselves to lexical features for our experiments. The use of syntax or even higher level features is (for now) impossible as the language use on Twitter deviates too much from standard Dutch, and we have no tools to provide reliable analyses. However, even with purely lexical features, 4. On the examined users, the gender assignment of TwiQS proved about 87% correct. 5. Several errors could be traced back to the fact that the account had moved on to another user since 2012. 6. We could have used different dividing strategies, but chose balanced folds in order to give a equal chance to all machine learning techniques, also those that have trouble with unbalanced data. If, in any application, unbalanced collections are expected, the effects of biases, and corrections for them, will have to be investigated.

174

there are still various options from which to choose. Most of them rely on the tokenization described above. We will illustrate the options we explored with the tweet @ANONYMISED Hahaha...ik geloof dat meneer Bee heeft ingezet op een plan v´ oo ´r het slapen ;-))ng <@> hahaha ... I believe that mister B has opted for a plan before sleeping ng which after preprocessing becomes <@> hahaha ... ik geloof dat meneer bee heeft ingezet op een plan voor het slapen ng The first option for machine learning features is a traditional one. Top 100 Function Words The most frequent function words (see (Kestemont 2014) for an overview). We used the 100 most frequent, as measured on our tweet collection, of which the example tweet contains the words ik, dat, heeft, op, een, voor, and het. Then, we used a set of feature types based on token n-grams, with which we already had previous experience (Van Bael and van Halteren 2007). For all feature types, we used only those features which were observed with at least 5 authors in our whole collection (for skip bigrams 10 authors). Unigrams Single tokens, similar to the top function words, but then using all tokens instead of a subset. About 47K features. In the example tweet, we find e.g. ik, ingezet, and . Bigrams Two adjacent tokens. About 265K features. In the example tweet, e.g. heeft ingezet, slapen , and <@>, where the double underscore represents the start of the tweet. Trigrams Three adjacent tokens. About 355K features. In the example tweet, e.g. op een plan and <@> hahaha. Skip bigrams Two tokens in the tweet, but not adjacent, without any restrictions on the gap size. About 580K features. In the example tweet, e.g. dat heeft and hahaha . Finally, we included feature types based on character n-grams (following (Kjell et al. 1994)). We used the n-grams with n from 1 to 5, again only when the n-gram was observed with at least 5 authors. However, we used two types of character n-grams. The first set is derived from the tokenizer output, and can be viewed as a kind of normalized character n-grams. Normalized 1-gram About 350 features. In the example tweet, e.g. i and @. Normalized 2-gram About 4K features. In the example tweet, e.g. ik and twice ng. Normalized 3-gram About 36K features. In the example tweet, e.g. gez and n v, where the underscore represents a space. Normalized 4-gram About 160K features. In the example tweet, e.g. slap and op e. Normalized 5-gram About 420K features. In the example tweet, e.g. ingez and <@> , now with a double underscore for the beginning of the tweet and a single underscore for a space. The second set of character n-grams is derived from the original tweets. This type of character n-gram has the clear advantage of not needing any preprocessing in the form of tokenization. Original 1-gram About 420 features. In the example tweet, e.g. e and ;. 175

Original 2-gram About 8K features. In the example tweet, e.g. Be and

@.

Original 3-gram About 77K features. In the example tweet, e.g. ing and ))n. Original 4-gram About 260K features. In the example tweet, e.g. plan and ;-)). Original 5-gram About 580K features. In the example tweet, e.g. r Bee and a...i. 4.2 Machine Learning Techniques Having determined the features we would be working with, we next needed to select a machine learning system. Again, we decided to explore more than one option, but here we preferred more focus and restricted ourselves to three systems. Our primary choice for classification was the use of Support Vector Machines, viz. LIBSVM (Chang and Lin, 2001). We chose Support Vector Regression (ν-SVR to be exact) with an RBF kernel, as it had shown the best results in several research projects (e.g. (van Halteren 2008)). With these main choices, we performed a grid search for well-performing hyperparameters, with the following investigated values: the cost factor C is set to respectively 1/32, 1, 32, 1024, and 32768, γ to 1/4, 1/2, 1, 2 and 4 times LIBSVM’s default of one divided by the number of features, and ν to 0.1, 0.3, 0.5 and 0.7. The second classification system was Linguistic Profiling (LP; (van Halteren 2004)), which was specifically designed for authorship recognition and profiling. Roughly speaking, it classifies on the basis of noticeable over- and underuse of specific features. Before being used in comparisons, all feature counts were normalized to counts per 1000 words, and then transformed to Z-scores with regard to the average and standard deviation within each feature. LP has four hyperparameter settings, three of which weight the relative importance of each feature/dimension in the feature vector when comparing a text’s feature vector to the profile vector (in this case the average of the feature vectors for all the training texts for a given gender), and one determining the threshold for feature Zscores to be taken into account. Here the grid search investigated: the hyperparameter emphasizing the difference between text feature and profile feature to polynomial exponents set to 0.1, 0.4, 0.7, ..., 2.7 and 3; the hyperparameters for emphasizing text feature size to 0 or 1; the hyperparameter for emphasizing profile feature size to -1, 0, 1, and 2; and the threshold hyperparameter also to 0 or 1. Finally, we added TiMBL (Daelemans et al. 2004), a k-nearest neighbour classification system, which is used extensively in-house for various machine learning tasks, but which we had so far not used for authorship tasks. As the input features are numerical, we used IB1 with k equal to 5 so that we can derive a confidence value. The only hyperparameters we varied in the grid search are the metric (Numerical and Cosine distance) and the weighting (no weighting, information gain, gain ratio, chi-square, shared variance, and standard deviation). However, the high dimensionality of our vectors presented us with a problem. For such high numbers of features, it is known that k-NN learning is unlikely to yield useful results (Beyer et al. 1999). This meant that, if we still wanted to use k-NN, we would have to reduce the dimensionality of our feature vectors. We chose to use Principal Component Analysis (PCA; (Pearson 1901), (Hotelling 1933)).7 And, now we had the principal component vectors, we decided also to provide them to SVR and LP. For each system, we provided the first N principal components for various N. In effect, this N is a further hyperparameter, which we varied from 1 to the total number of components (usually 600, as there are 600 authors), using a stepsize of 1 from 1 to 10, and then slowly increasing the stepsize to a maximum of 20 when over 300. Rather than using fixed hyperparameters, we let the control shell choose them automatically in a grid search procedure, based on development data. When running the underlying systems 7. To be exact, we used the function prcomp in R (R Development Core Team 2008), with the instruction scale = T RU E to force normalization of the vectors before the principal components were determined. As scaling is not possible when there are columns with constant values, such columns were removed first.

176

themselves, we used various hyperparameter settings, as listed above. For each setting and author, the systems report both a selected class and a floating point score, which can be used as a confidence score.8 For each individual author, the control shell examined the scores for all other authors in the same fold.9 It then calculated a class separation value, namely the difference between the mean scores for each of the two classes (male and female), divided by the sum of the two standard deviations.10 The optimal hyperparameter settings are assumed to be those where the two classes are separated most, i.e. where the class separation value is highest. In order to improve the robustness of the hyperparameter selection, the best three settings were chosen and used for classifying the current author in question. A final detail that we exploited is that SVR and LP are asymmetric in the modeling of the classes. For LP, this is by design. A model, called “profile”, is constructed for each individual class, and the system determines for each author to which degree they are similar to the class profile. For SVR, one would expect symmetry, as both classes are modeled simultaneously, and differ merely in the sign of the numeric class identifier. However, we do observe different behaviour when reversing the signs. For this reason, we did all classification with SVR and LP twice, once building a male model and once a female model. For both models the control shell calculated a final score, starting with the three outputs for the best hyperparameter settings. It normalized these by expressing them as the number of non-model class standard deviations over the threshold, which was set at the class separation value. The control shell then weighted each score by multiplying it by the class separation value on the development data for the settings in question, and derived the final score by averaging. It then chose the class for which the final score is highest. In this way, we also get two confidence values, viz. the model score for the chosen class (how male/female the author writes) and the difference between the two scores (how much more female/male the author writes than male/female).

5. Results In this section, we will present the overall results of the gender recognition. We start with the accuracy of the various features and systems (Section 5.1). Then we will focus on the effect of preprocessing the input vectors with PCA (Section 5.2). After this, we examine the classification of individual authors (Section 5.3 and the distinguishing power of features (Section 5.4). 5.1 Overall Quality Table 1 shows the accuracy of the recognition, using the desccribed features and systems. For the systems, both SVR and LP are used with the original case vectors as well as with PCA preprocessing, where TiMBL, for reasons mentioned above, is used only with preprocessed vectors. For the measurements with PCA, the number of principal components provided to the classification system is learned from the development data. Below, in Section 5.2, we will examine what the systems are capable of at fixed numbers of principal components. Starting with the systems, we see that SVR (using original vectors) consistently outperforms the other two. For only one feature type, character trigrams, LP with PCA manages to reach a higher accuracy than SVR, but the difference is not statistically significant. LP and TiMBL are closely matched, although LP appears to be slightly better when combined with PCA, but the next section will shed new light on this comparison. From the measurements here, we can conclude that LP profits from PCA preprocessing, but SVR is better off with the original vectors. 8. For SVR and LP, these are rather varied, but TiMBL’s confidence value consists of the proportion of selected class cases among the nearest neighbours, which with k at 5 is practically always 0.6, 0.8, or 1.0. 9. This gives the best chances that the selected optimal hyperparameters generalize to the author in question. 10. The class separation value is a variant of Cohen’s d (Cohen 1988). Where Cohen assumes the two distributions have the same standard deviation, we use the sum of the two, practically always different, standard deviations.

177

Table 1: Accuracy Percentages for various Feature Types and Techniques. For each feature type, the best percentage is bolded, and all percentages are italicized that are not statistically significantly different at the 5% level. Feature type Techniques Support Vector Regression Linguistic Profiling TiMBL original with PCA original with PCA with PCA Top 100 Function Words 84.8 83.7 76.7 84.7 75.8 Token Unigram 95.5 94.3 91.8 93.0 91.7 Token Bigram 94.5 93.7 89.8 91.0 91.8 Token Trigram 92.5 89.7 89.0 92.0 87.2 Token Skip Bigram 93.5 92.7 88.3 92.7 92.3 Char 1-gram-n 81.8 79.5 77.2 79.7 76.2 Char 2-gram-n 88.7 89.7 84.2 86.2 78.8 Char 3-gram-n 94.0 90.5 89.2 89.3 86.5 Char 4-gram-n 94.2 92.2 89.0 91.0 89.2 Char 5-gram-n 94.3 93.0 90.5 93.5 91.2 Char 1-gram-o 81.7 82.2 76.8 77.8 75.2 Char 2-gram-o 86.0 85.0 83.2 85.3 81.2 Char 3-gram-o 86.6 86.0 86.7 88.0 85.5 Char 4-gram-o 92.0 90.5 86.7 89.0 87.3 Char 5-gram-o 92.8 92.3 89.7 92.2 88.7

As for features types, the token unigrams are clearly the best choice. In fact, for all the tokens n-grams, it would seem that the further one goes away from the unigrams, the worse the accuracy gets. An explanation for this might be that recognition is mostly on the basis of the content of the tweet, and unigrams represent the content most clearly. Possibly, the other n-grams are just mirroring this quality of the unigrams, with the effectiveness of the mirror depending on how well unigrams are represented in the n-grams. Below (Section 5.4), we will have a closer look at this hypothesis. For the character n-grams, our first observation is that the normalized versions are always better than the original versions. This means that the content of the n-grams is more important than their form. This is in accordance with the hypothesis just suggested for the token n-grams, as normalization too brings the character n-grams closer to token unigrams. The best performing character n-grams (normalized 5-grams), will be most closely linked to the token unigrams, with some token bigrams thrown in, as well as a smidgen of the use of morphological processes. However, we cannot conclude that what is wiped away by the normalization, use of diacritics, capitals and spacing, holds no information for the gender recognition. To test that, we would have to experiment with a new feature types, modeling exactly the difference between the normalized and the original form. 5.2 Effects of PCA In the measurements above, the number of principal components provided to the classification systems was learned on the basis of the development sets. This number was treated as just another hyperparameter to be selected. As a result, the systems’ accuracy was partly dependent on the quality of the hyperparameter selection mechanism. In this section, we want to investigate how strong this dependency may have been. 178

1.0 0.9 0.8 0.7 0.6 0.5

Attribution Accuracy

SVM LP TiMBL

0

2

4

6

8

Log2(Number of Principal Components)

Figure 1: Recognition accuracy as a function of the number of principal components provided to the systems, using token unigrams. The dotted line represents the accuracy of SVR without PCA preprocessing.

Figures 1, 2, and 3 show accuracy measurements for the token unigrams, token bigrams, and normalized character 5-grams, for all three systems at various numbers of principal components. The dotted line is at the accuracy of SVR without PCA. For the unigrams, SVR reaches its peak (94.7%) around 30-40 principal components, with a second peak around 80-90. TiMBL closely follows SVR, but only reaches its best score (94.2%) at the latter peak (80-90). Interestingly, it is SVR that degrades at higher numbers of principal components, while TiMBL, said to need fewer dimensions, manages to hold on to the recognition quality. LP peaks much earlier (93.8%) at only 8-10 principal components. However, it does not manage to achieve good results with the 80-100 principal components that were best for the other two systems. Furthermore, LP appears to suffer some kind of mathematical breakdown for higher numbers of components. If we look at these measurements, it would seem we should prefer TiMBL over LP, which is in contradiction to what we see in Table 1. Although LP performs worse than it could on fixed numbers of principal components, its more detailed confidence score allows a better hyperparameter selection, on average selecting around 9 principal components, where TiMBL chooses a wide range of numbers, and generally far lower than is optimal. We expect that the performance with TiMBL can be improved greatly with the development of a better hyperparameter selection mechanism. For the bigrams (Figure 2), we see much the same picture, although there are differences in the details. SVR now already reaches its peak (94.3%) at 10 principal components, and stays at almost the same quality until around 200. TiMBL peaks a bit later at 200 with 94.7%, even slightly higher than SVR without PCA. And LP just mirrors its behaviour with unigrams. For the normalized character 5-grams, SVR is clearly better than TiMBL, with peaks (94.2%) from 40 to 100. LP keeps its peak at 10, but now even lower than for the token n-grams (92.8%). All in all, we can conclude that SVR without PCA is still the best choice. However, all systems are in principle able to reach the same quality (i.e. not significantly lower) with the optimal number of principal components. Even with an automatically selected number, LP already profits clearly 179

1.0 0.9 0.8 0.7 0.6 0.5

Attribution Accuracy

SVM LP TiMBL

0

2

4

6

8

Log2(Number of Principal Components)

Figure 2: Recognition accuracy as a function of the number of principal components provided to the systems, using token bigrams. The dotted line represents the accuracy of SVR without PCA preprocessing.

from PCA, but (for this task) lags behind. And TiMBL is currently underperforming, but might be a challenger to SVR when provided with a better hyperparameter selection mechanism. 5.3 Analysis of Author Classifications In this section, we will examine some aspects of author classifications. We will focus on the token n-grams and the normalized character 5-grams. As for systems, we will involve all five systems in the discussion. However, our starting point will always be SVR with token unigrams, this being the best performing combination. We will only look at the final scores for each combination, and forgo the extra detail of any underlying separate male and female model scores (which we have for SVR and LP; see above). As can be seen in Figure 4, the two scores for SVR match almost completely anyway (Pearson Correlation -0.993).11 The major exception to the symmetry is author 543, lying clearly in the male area, but quite a bit above the dotted line (at around -2,4 in Figure 4). When we look at his tweets, we see a kind of financial blog, which is an exception in the population we have in our corpus. The exception also leads to more varied classification by the different systems, yielding a wide range of scores. SVR tends to place him clearly in the male area with all the feature types, with unigrams at the extreme with a score of -3.497.12 SVR with PCA on the other hand, is less convinced, and even classifies him as female for unigrams (1.136) and skipgrams (3.946). LP and TiMBL also show scores all over the range. Figure 4 shows that the male population contains some more extreme exponents than the female population. The most obvious male is author 430, with a resounding -6.050. Looking at his texts, we indeed see a prototypical (young) male Twitter user: the addressed topics mainly consist of soccer, gaming, school, and music (all of which we will see again below, when examining the most gender 11. This is rather different for LP, but the focus is on SVR here. 12. From this point on in the discussion, we will present female confidence as positive numbers and male as negative.

180

1.0 0.9 0.8 0.7 0.6 0.5

Attribution Accuracy

SVM LP TiMBL

0

2

4

6

8

Log2(Number of Principal Components)

Figure 3: Recognition accuracy as a function of the number of principal components provided to the systems, using normalized character 5-grams. The dotted line represents the accuracy of SVR without PCA preprocessing.

specific unigrams). All systems have no trouble recognizing him as a male, with the lowest scores (around 1) for the top 100 function words. If we look at the rest of the top males (Table 2), we may see more varied topics, but the wide recognizability stays. Unigrams are mostly closely mirrored by the character 5-grams, as could already be suspected from the content of these two feature types. For the other feature types, we see some variation, but most scores are found near the top of the lists. Table 2: Top ranking males in SVR on token unigrams, with ranks and feature types. Feature type 430 344 564 Unigram 1: -6.050 2: -5.243 3: -4.886 4: Bigram 1: -5.776 20: -2.875 12: -30.72 8: Trigram 1: -3.916 64: -1.523 49: -1.663 15: Skipgram 1: -3.186 8: -2.808 3: -3.123 21: Char 5-gram 1: -5.228 3: -4.398 4: -4.239 6: Top 100 Function 4: -1.946 144: -0.602 42: -1.114 5:

scores for SVR with various 335 -4.356 -3.225 -2.338 -2.191 -3.754 -1.903

5: 6: 17: 15: 5: 31:

454 -4.218 -3.375 -2.277 -2.385 -4.001 -1.211

On the female side, everything is less extreme. The best recognizable female, author 264, is not as focused as her male counterpart. There is much more variation in the topics, but most of it is clearly girl talk (of the type described in Section 5.4), again putting the best recognition at a prototypical young Twitter user. In scores, too, we see far more variation. Even the character 5-grams have ranks up to 40 for this top-5. Another interesting group of authors is formed by the misclassified ones. Taking again SVR on unigrams as our starting point, this group contains 11 males and 16 females. We show the 5 most 181

6

* * *

2 0

* * ***** *** * * ****** * * **** ****** ** ** ** * * * ***** ************** *** *************** ******************** * * ** * ****************** * * ** ******** * * * ********* *** * ******** * * ***** ** *********** ** * * ** *** ********* * * *********** * *** * * ************ * * ************* * * ******************* *** ************ * * ************** * * * ******* * ************** * * ******* ** *** * * * * ***** * **

*

−4

−2

Score from Male Profile

4

** * * * ** *

−6

−4

−2

0

2

4

Score from Female Profile

Figure 4: Confidence scores for gender assignment with regard to the female and male profiles built by SVR on the basis of token unigrams. The dashed line represents the separation threshold, i.e. a higher score for one gender than for the other. The dotted line represents exactly opposite scores for the two genders.

182

Table 3: Top ranking females in SVR on token unigrams, with ranks and scores for SVR with various feature types. Feature type 264 13 75 43 298 Unigram 1: 3.372 2: 3.196 3: 3.154 4: 3.102 5: 2.964 Bigram 6: 3.011 11: 2.847 78: 2.001 33: 2.472 43: 2.377 Trigram 43: 2.147 49: 2.070 179: 1.136 69: 1.911 157: 1.252 Skipgram 10: 2.546 11: 2.525 13: 2.462 118: 1.405 76: 1.704 Char 5-gram 2: 3.319 9: 2.847 14: 2.687 40: 2.194 29: 2.383 Top 100 Function 9: 1.658 43: 1.215 140: 0.679 88: 0.904 174: 0.553

Table 4: Most strongly misclassified males in SVR on token unigrams, various feature types. Feature type 352 355 386 566 Unigram 0.841 0.826 0.753 0.388 Bigram 0.361 0.047 1.214 0.348 Trigram 0.300 -0.723 1.016 0.339 Skipgram 0.104 1.059 0.655 0.750 Char 5-gram 0.819 0.345 0.870 0.147 Top 100 Function 1.088 0.163 0.477 0.711

with scores for SVR with 389 0.362 1.472 1.226 2.080 1.388 1.508

strongly misclassified ones of each gender in Tables 4 and 5. With one exception (author 355 is recognized as male when using trigrams), all feature types agree on the misclassification. This may support our hypothesis that all feature types are doing more or less the same. But it might also mean that the gender just influences all feature types to a similar degree. In addition, the recognition is of course also influenced by our particular selection of authors, as we will see shortly. Apart from the general agreement on the final decision, the feature types vary widely in the scores assigned, but this also allows for both conclusions. The male which is attributed the most female score is author 352. On (re)examination, we see a clearly male first name and also profile photo. However, his Twitter network contains mostly female friends. This apparently colours not only the discussion topics, which might be expected, but also the general language use.13 Another interesting case is author 389. The unigrams do not judge him to write in an extremely female way, but all other feature types do. When looking at his tweets, we 13. This has also been remarked by Bamman et al. (2014).

Table 5: Most strongly misclassified females in various feature types. Feature type 103 Unigram -3.285 Bigram -1.689 Trigram -0.830 Skipgram -1.243 Char 5-gram -2.914 Top 100 Function -0.667

SVR on token unigrams, with scores for SVR with 261 -0.933 -0.105 -0.033 -0.226 -0.203 -0.170

183

49 -0.700 -1.149 -0.747 -0.508 -0.672 -0.099

226 -0.691 -0.895 -0.494 -1.027 -1.009 -0.683

61 -0.659 -0.547 -0.415 -0.435 -0.814 -0.589

get the impression that Dutch is not his native language, which is supported by his name. There is an extreme number of misspellings (even for Twitter), which may possibly confuse the systems’ models. The most extreme misclassification is reserved for a female, author 103. This turns out to be Judith Sargentini, a member of the European Parliament, who tweets under the name @judithineuropa.14 Although clearly female, she is judged as rather strongly male (-3.285) when using unigrams and character 5-grams, and male by all system-feature combinations15 except three. LP with PCA on skipgrams assigns her a female score of 1.321 and SVR with PCA (just as with author 543) arrives at a clearly female attribution with character 5-grams (4.554) and unigrams (5.149). In this case, it would seem that the systems are thrown off by the political texts. If we search for the word parlement (“parliament”) in our corpus, which is used 40 times by Sargentini, we find two more female authors (each using it once), as compared to 21 male authors (with up to 9 uses). Apparently, in our sample, politics is a male thing.16 It is intriguing that both here and with the male financial blogger, the erroneous misclassification with unigrams is reversed when using PCA on the unigrams. We did a quick spot check with author 113, a girl who plays soccer and is therefore also misclassified often; here, the PCA version agrees with and misclassified even stronger than the original unigrams (-0.707 versus -0.248). In later research, when we will try to identify the various user types on Twitter, we will certainly have another look at this phenomenon. 5.4 Features In the analysis so far, we have wondered several times what kind of features are responsible for the rather accurate classification. Are they mostly targeting the content of the tweets, i.e. related to the activities of the authors in real life, or the style, i.e. the way they use the basic building blocks of the Dutch language? In this section, we will attempt to get closer to the answer to this question. Again, we take the token unigrams as a starting point. However, looking at SVR is not an option here. Because of the way in which SVR does its classification, hyperplane separation in a transformed version of the vector space, it is impossible to determine which features do the most work. Instead, we will just look at the distribution of the various features over the female and male texts. Figure 5 shows all token unigrams. The ones used more by women are plotted in green, those used more by men in red. The position in the plot represents the relative number of men and women who used the token at least once somewhere in their tweets. However, for classification, it is more important how often the token is used by each gender. We represent this quality by the class separation value that we described in Section 4.2, and show it in the form of font size, i.e. the more distinguishing tokens are bigger. As the separation value and the percentages are generally correlated, the bigger tokens are found further away from the diagonal, while the area close to the diagonal contains mostly unimportant (and therefore unreadable) tokens. On the female side, we see a representation of the world of the prototypical young female Twitter user. It is a very emotional place, with omg (“Oh My God!”) in a central position, but also containing giggling (hihi) and lots of emotionally loaded adjectives, such as lief and lieve (“sweet”), schattig (“cute”), leuk and leuke (“nice”). And also some more negative emotions, such as haat (“hate”) and pijn (“pain”). Next we see personal care, with nagels (“nails”), nagellak (“nail polish”), makeup (“makeup”), mascara (“mascara”), and krullen (“curls”). Clearly, shopping is also important, as is watching soaps on television (gtst ). The age is reconfirmed by the endearingly high presence of mama and papa. As for style, the only real factor is echt (“really”). The word haar may be the pronoun “her”, but just as well the noun “hair”, and in both cases it is actually more related to the 14. Identity disclosed with permission. 15. And by TweetGenie as well. 16. An alternative hypothesis was that Sargentini does not write her own tweets, but assigns this task to a male press spokesperson. However, we received confirmation that she writes almost all her tweets herself (Sargentini, personal communication).

184

60

haar

50

omg

eng

lief pijn

leuk

wil

ik heb

haat heel

word niet

leuke

kwijt kan bang snel hele helemaal niemand doet jou blij snap ben lachen ogen vergetennooit gezicht voel zo=n jongens vind kon hoofd lang meer ze samen jaloers dood durf roze kamer voelt want foto=s moest vallen zoveel steeds toen raar liedje vertellen terwijl moet zin bruinlacht lach waarom proberen verpest me gesprekken koud boos zingen erger ging ga enge belt ruikt omdat altijd kast elke iemand zo naast maken val bent lukt liedjes huilengezellig vertel allesveranderen erg dingen klas bellen gekregen gemaakt krijg nemen hoef lieve kind schrok oog meisje typen viel veranderd lezen sliep moe hou misselijk ene met praten film mis verliefd denken zonder gesprek dom buikpijn verder moeder gevallen zoooo kleur mezelf sturen gevoel raam mensen keek vraagt missenbuik ( arm denkt alsof foto bikinishoppen hoofdpijn kindjes = zegt grond soms voeten voelen ruzie vriendin schrik vroeger oeps regent liggen oke wiskunde vriendinnen liefste gebeld rare voelde hahaha verslaafd grappig eraf blond verbrand vrienden tranen vergeet stem liep super zooo tas kent neeeeomazingtvoet knuffel krullen gekocht sommige gemist er stond spreken gtst stomme stom lijk wachten slim msn droom bedje agenda gingen jezelf i fietsen perfect opeensoplader warme tijden even noemt broek opruimen jaaa hart medelijden warm leuks wekker schreeuwen geslapen oh lelijk diegene ijsje spijt lig knap taart mobiel kleren ? liefde reageren gelachen verkouden boeken kindje * heeel neee we gemeen irritant verveel verteld liet papa x gedoe spullen erge tekenen boek stuk honger lag − verwijderd jaaaa mee o zeuren ; bitch douchen leukste denk kussenlelijke heeeel hakken janken wow dokter bios betekend totdat hallo kus chocola mascara opa reageert geleerd verwijderen slaap vrolijk waarheid ookal sloom domme nog smsen armen uur trouwen redden einde hard wiltpraat vriendje pff persoon slaapt een nat staan 8 leren happy _ vraag sexy op jas koekjes dankje zijn nee neeeee vest $ love dm you yes fouten lakken liegen groetjes aangesproken wijf zou haalt tante gevoelens jaaaaa make−up zeg ping opschieten afspreken blijft stuurde chocolade spiegel kleding doei dekennegeer bank beterbinnen sneller zooooo schatjes smsjeneefjelaptop hij bb serieus kwartiertje moeite schat dankjewel hartje wauw rot beneden eind zelfs kom 4 dan bio telefoon gedroomd van zenuwachtig ava huiswerk badkamer trui vanavond week zal schoonmaken frans nagellakjurkje opmakengillenuggs keelpijn plaats lijkt brekenhahahahaha onvoldoende staat pyjama mn krijgt volgende kusje minuutjes rest aankleden aandoen hahahahah koffer in aaah god jammer fijn meeste haal kopen bericht hahahaha hahahahha thee nichtje verven 2 hyves vlieg jaa weetje weer 10 makkelijk hahhaha valt 11 genoeg hetzelfde gevraagd grote past jezuspogingbesef duidelijk nieuwe cadeautje heeeeel cadeautjes & hey wordt vanaf aaaahspin zoiets vriendinnetje schatje 1 tijd last eens gebruik arme eerst vorige schoool kut laatste goed nichtketting tongvertrouw of zondag pfffverpesten veeel omkleden zouden afscheid opgeruimd irriteer oook hoog verwacht maaar zwaar 40 let okeee vanacht mini hondje moee xxx xx hik dansen dingetjes gehaald beetje laatst uiteten zak rond ook gegeven geeen zaterdag smsjes dans hoesje ak seconde vaag gebruiken google woordjes knippen oppassenxxxx loveyou huil zich wie hoer minuut bah dr hoezo snapte zielige lullig gehoord rust 5 daarom harde winkels deel worden penhahahha langs handig erbij onzin mislukt cake douche vertelt huisje trek cola lucht knuffelen 12 erop hoger pingen omgg t zusjebriefje +gtstcute dekens shirtje irriteren hell schaam buurt vriendinnetjes jaar jurk hooor zing vreselijk waar gedoucht jawel lijst 16 100 rekenen ja so lieverd helft ... kaartje aap upblackberry nummer meter amp arrogantverdrietig 1000 werkt volgend noooit chinees tenminste ineens hoelaat allerliefste gebruikt gebeuren paarhoeveel pf mop dun voice bad iegrap keuze ervoor kinderachtig herinneringen zonnen kost pak yess daagjes bday plannen zware vorig wooow downloaden schrikken please halve rij2012 begonnen vriendschap winkelen gaaan waarschijnlijk klopt afkijken one pff. huilt trouwens gesmst 2e nouu verschilrustigschiet erin eerste kletsen zwanger zorg ff eventjes rijd broekje weeer vrouwen % horror yesss meid weekend logeren jouuomg. mening tijdens land extra 26 verdriet aller reis 20 koudd voeg opeten bye geslaagd gedronken berichtje paard draaien make sterk harder links 21 minder bedankt bekijken boom klaarmakennegeertaangekleed stoot flikker eraan verwarming jeej blijkt probleem smst 80 22 avond rechts vooruit vuurwerk zus mooi verliezen makeup zeker afgelopen ipv mam eh 200 thuiss hahahhaha winterjas dronken programma duur xoxo goedd aaan mams ontmoet niett besteld meedoen oorbellen 18 paars omfg tekening kauwgom geniaal stofzuigen opzich koopweinig staart uhh naaar hahaah zonde stap feest mail regelen liefjehihihi spontaan opstaan gestoord vermoord jeetje auw succes 25 cutie awh uh musical bedenk verdienen kaas nogsteeds vrij de 50 gewerkt zover15 lippen just rijk jahoor never 35 band hoesten laatmaar shirt betalen damn mega voor lieff+loveit goodnight lip concentreren vertelde verschillende via lul bord feit jasje nederland meis kaart opgeven spoort lady jaja heletijd positief m=nhumeur 19 doel well ooooh stik enkel verf moeee 17 btgdorpknappe geslagen januari compleet halfuurtje spel msn.fml weeet warmte 13 klaaar kaal beeld k chagerijnig nogal chagrijnig natuurlijk draai tweede geld kutwijf gooi goeie dankjee proefwerk +koud uitlachen saaai zooi stijlen doood betere popcorn lastig magister schijt kusjes mensjes dansje zeer omggg iloveyou xxxxx tent veeeel binnenkort nope wind hoek kapper nouuu gratis ahw lolly aangekomen opmerkingen neeeeee glimlach aardrijkskunde ever 00 kans alcohol radio uitt fles terecht vanuit grrrhelemaaal doenn optreden 28sluit nummers 45boer lekkerder sowieso scherm ach drink with goede ooook +geenzin mannetje+gezellig kregen rekening bijles accepteerahh jeeej jouw bbq wezen 2000 vanwege lachte schattiexo vermoorden netjes win gedrag yeaaah leugens politie simpel woow tassen vent 500 interessant klaarwakker vecht ahahaha okeeee huiss idd ondanks verstand lisa overleef ?? piercing spulletjes verkopen leukkoeeh jaren afgepakt schatjee hyper xxxxxx karma derde gele das benieuwd komende 70 schoolfoto koortsaaaaah fake nagel 60 30 mogelijk euro per paps opgegeten gebelt eitje roos mannenredelijkduitsland kei uhmm her thanks byna nageltjes gosh winnen niveau vergeven ergs vandaaag duizelig uitslag u slapenn +help33 paarden snoepjes vrije optie waaar tussenuur start staren beide vandaar versie nieuws gay gekkebelgie weertje sweet 2011 loser justin nachtmerrie standaard pa bak werkbekend geregeld kluisje slot 300 kleden volle ahhh inderdaad cd project stalker mmm lieffie speel jouuu augustus paul stoere 333 niezen actie ieder iel bh whahah hiero verlies nederlandse actief spelen grr liefswaaarom geval geel droomde wandelen drinkt strak file tje schattige gelakt verloren w=app hahahahahha pot drank draagt gekkie girl klasje poepie schiphol 31drie verplicht allebij too nina stelt irriteert verdiend groeten balen opgemaakt britthahhah bouwen leugen speciale park prik duitse k3 oehh klagen jan parfum +pfff ohnee iedereeen rotterdam etenn checken plus kaarten gewooonmoederdag zorgt pyama tjaa speelde pfoex. vullen check homo talentgewone sport stylen regel opleiding lamp hahhahaha ten mtv binnekort vlak krabbel +fijn dusss tril rokje forever miljoen betaald veld gewonnen goeed genieten hatelijk areshitt adres kerk italie serie liefjes haaat by eigelijk gadver big medicijnen rijdt popje vuur stemmen heeeeeel ned steun tochh iss tekort ruim helaas voorbeeld traan jong hempje wint gezocht+zinin berg −− heeee sixpacknooo knopje 2de kennis bar slag zieke paarse moppfohnen onderweg traag buurmeisje ziekjes plaatsen lijn natte gatver prijs ruimte chillen grappen twilight gekker opdoen tyfus tip juiste ijskoud rijbewijs weeeer woooow auto=s c1000 roddelen zaal informatie info rugpijn drogen allerbeste hihih swag vissen appen poepen pony legging aardige belangrijke snoepje persoonlijk download direction kyken daarvan hy mama=s 90 speelt chill jy storing raaar centrum 32 bomen echtt controle tis huisdier sterker liefie kleertjes jonge korting hoogte ter meezingenn bby jeeeejmaaaar tenzij periode hahahhahaha vakantiee schatt laaatzyn neehoor uitstaan bb. verboden luie +tienerfeiten ofniettrappen baas aantal ballen gelogen goud papa=s huistelefoon goedkoop ervaring honey nest zachte bevalt diehard how boo tosti=s morgenn wangen poppetje melig eeuwig nichtjes +lief verwijdert waauw 75 michelle onzeker conditie dubbele titelafwachten pet pro ek doodmoe pik wedstrijd verkocht trainen held myn gehaalt puur middelvinger nieet oplossing glee inslaapijsjeswakkerr panty tevreden but dance conclusie nationale betaal februarimoney flink oeeeh kosten wenkbrauwen gfrank yessss altydgeloofde luchtje tandjes apple terras utrechttheorie gast schuur aahw +loveher iris brrr verwend voordeel fluiten geshopt dekentje awhh kansen finalevoetbal team bal gezellie chaggie 3333 hello verstandig schooltje 1ste duiken ee aankomende +linkdelieve nikssvlinders och mark combinatie blablabla tim engeland afhalen primark jeugd +wateenmeisjewil verliest gozer gevolgd +bye shirtjes vreselijke wijntje uuur enso tessa studie boete gevaarlijk centraal kroelen hersens hooigesprekje waarin overhalen leeuk paardrijden vorm punten knot rap gitaar opgesloten strakjes patatjes compliment vervoer stikkenjariggvoetjes gedroomt danst mario henkvoorbereiden vestje stijltangsuperleuk man km geirriteerd baby=s behoorlijk byee doeii lovely thankyou uw hockey kapsel dame scherp +irritant toeval ontstoken sirieus melissa trok thank fans fit gescoord wy spangas joggingbroek shampoo paal innerlijk jeee moeeee tom house bakje lars naomi leukst blabla vreetzak amber laarzen ft gemiddelde algemeen fotoshoot maskertje beker direct birthday hongerrmood praatte +dutchteenagers afgeleid cupcakes kijkenn waaauw omgggg tussenuren werkte wauwie okii schooll nutteloze ist shoarma bff outfit riipings vertrek knallen gespeeld real iig prachtige glad beschrijven b−day wimpers daphne xxxxxxx toee tram hierr linker woehoe chocomelk aawh levenloos nederlanders chil hihihihi verveeel aanwezig club gaar galaxy collega=s snelweg langskomen jaaaaaaa oneerlijk smiley cakejes slaaap ugh paracetamol +auwbeyonce starten stand box oorbel weltrusten boeiepagina doden lache rechter resultaat chille +sweetdreams moi +fail nl. reken gooische update training pyn bedjeenieeet qua getypt fantasie tog tydfleur badje uhhh sorryy paranormal hope status merel wiee ehhh zaken media hihii doeen omaa boys minimaal scoren +eng gespot g nogwel geluidje gebeten geverfd heuj sophie afpakken excuses trut bahh bedd knuffelsheaa niels grens oooook vlek keken inde mb album maat pitten basis geeeen onhandig geniale leerling kryg haai zojuist hartjes slaapkop kleuters aahh dijknogmaals eer 250 snelle hoppa interview ham chanel armband inmiddels tumblr koffie spanning verkoop doeeen lieverds swagg terugg belangrijker college jazeker lonely bly bijpraten kadootje ouwe tour smste buurjongen 3000 aienkele blehhondjes awkward kees rickleen boxenvolledig maxime woww sfeer morge seizoen +tienerthings zaterdagavond always zinnetjes jurkjes jordy hek cursus pokemon boodschapjes 140 vriendloos sander hamburgers kouddd sis innemen lies linda +vakantie achterin +dtv fixen nicekampioen morgenvroeg pure porno gedaaan walg lichtje loslaten dankjeee slijmen bijvoorbeeld nadeel samenvattingen aaahw nicole bestwel lauw black testen vakantieee cafe loveyouu prima djzaak partij150 wedstrijden grappigs uitgelachen leerlingen gouden gelyk beauty juul bijkomen wijven kleurtje kapsalon wass zussen orde moois loveu makenn zienn aaahh etui lalala heeeeeeelups beat +chill beelden lerenn fissa kill kale beats toernooi gapen persee topper onderzoek tempo wegleggen kga gedraaid wouw zuur +blond ilse blazen baardmooiboy lll gezelligg genaaid spellen bedrijf kloten garage superr game groningen geschoten bing opgeruimt goeeedangela waarschuwing smaken volop +durftevragen uitspraak superveel xd. treintje weeeeer os brownies bloedheet zwolle oeff fantastisch friesland him fml.brrzijnn schatjess studio uitzending lam waarde chilling slopen rustige gestemd mandy +grrr warmm +nietleuk meiss okieht ruben kammen doooodahwwdaddy vuile snell bahbah hoooor zonnebankhvj vertrekt mischien getraindduitsers fuckyou ronde topp voorschut b=day daaag hooop job dienst jessie creme come positieve http can=t w=appen route shirts ouderwets uhmmm allesdoen levensverhaal serviceamerikaanse bestel heeeft dankjewell zuipen vs kwaliteit el peter hans 2−1 games gevangen opsturen +jeej sportschool eemroyvergelijken manon mnr mr.geslaagde maarten set nasi kranten besta nieuwste mattiebetaalt hardcore ivmgemeente ofnie felicitaties stefan ronaldo nogmeer pauzeemom gruwelijk marathon +gaoverdestreep logolid schade pitte praktijkbetreft lieverdje autoo leuukzoveeel 2010 winnaar huize nederlander bc=s snol niksenalmere level bro zinn smurfen potje kerel 1−0 grotere ebuddy navelpiercing nieuwschierig prettig paardje chantal mooii werke stal nicee klantkoppen chaosvergadering interesse geplaatst jongeren +boysengirls regend niee pittig persoonlijke mjaa tired xoxoxo +xmz borst dure iewl stadd maaike +linkdedownie invloed cup breed volk rob arnhem hih awhhh floor frituren faka koning mopje gasten htc aanvragen zanger griekenland +megazinnen gebruikte ohmygod iedergeval aanbieding +genieten inschrijven awpingg sweater andre biedenkleedkamer downs allessmanege eline ookk schaal edkaulo aii daarbij locatie banden nuljoost bizar effect wilders youubuitenn voetballen ajax groet winne ontslagen tanken degrassi byebye migraine hoesjes 700 uiteraard aldus whaaa +kutzooi bliksem haven +weekend helm cuties steeeds noah mweh besteed waarderen toffe verkoopt bier city klasse barcelona gewn garekapotte melding krygt ing complimenten gebreksocial belgen dinnetje jeah ohyeah kyke rijtje mekaar skip moppieeowhh suus xx. meerr tamara 65 installeren scheren zend meidd tfoee ritideaal flinke +link gok beschikbaar liter 0−0 +regen gevuld geldt bijnaa jah +puberfact cc wast gevolgenairco 538 systeem halenn doeiii denemarken kilometer kantoor zekers 44 benzine danielle salaris +tienersonline hieer jhaa televisie fitness sessie fietsenn keeper stadion joyce kykt omygod dagelijkse jepp https politiek pissen jellerijkeopgepakt xox maten mediamarkt klanten huhh whaa matties gemeld jonguh daarzo aanpakken bouw sweetdreams voetbalveld verzoek tocht solo slachtoffer officiele wk publiek won spelers wissel coach wiel contract 3−1 trainer aanpassen trapt uitdaging stevig koningin zuigt43 plank +mooiezinnetjes +iloveyou joo balletje bedrijven potjes koptelefoon 54 geschikt +sbs6 +bamstukkie erik angry huus ahhw intremix correctie lens duurder inclusiefgoestudenten functie mafkees jaman motor snelheid +nice 1−1 taalgebruik kgmatneeman geheel journaal maartje liefjee onse iritant bond cd=s cultuur macht sociale htt fix aight gel playstation johan plaat appe barca2−0 geintje overleg halverwege waterpijp dope race focus bakkie +causeitstrue zowel hout schatties autist italiaanse aanstaande taakhardstyle sick daaro drerijlesopening flitsschandalig arena carriere nuchter train scoort oppeens temperatuur spot gearriveerd string goedkomen gerookt jongeman suggesties belg kroeg golffitnessen pikken petje score motto 120 aai +voetbal buis omdt klussen verkeer drenthe birds heer lievee zy q.avaaa veronica erheen prijzen nos mede twente xbox fc zegg agent posten comedy ziggo vervolg complete bijv. miljard judith beroep vos aldi interessante inzetmaximaalvoetballer stukk brein khad onderdeel kwestie kick jantje directeur fantastische fransen definitief afgelast azgamen competitie speler illegaal geschorst geinteresseerd trainingspak afsluiting hoofdpyn druppel v. bovenaan store 52 darten lara beiden noorden bike rock ronaldgeweld voetbalschoenen bakt fzo maaltijdkutten ofwel chef goal kabel android 3−0 gevaar +mooiaangehouden zayn scheids proost burger jeeh principe carmen nare marcel lat +teenshit american rabobank verdomd +thatsoundstrue sorrie bruno allerlei hoedafgerond mentaal verdorie stiefvader eigenaar positie101 winst wegens joden manager johnny organisatie jaap pakkie ron tut voort nette vaart w8en 1200 brian bekent naait technisch relaxed komst zweden helden artiesten zombies flex initiatief updaten biefstuk voorbereiding persie feyenoord +held selectie fluit ps3 pers 2−2 chelsea genot aanslag king overigens partijtje prestatie afterparty ewa aangepast scorro teams master 130 evi larissa kzit shine checke +twitter thuus communicatie inkomen donders hitler +rtl7 ad track mix akkoord fok aanbod cod ado a1 1−2 elftal gegevens rappers jagen regering instellingen agh gb sjaak hierbij somaar loesoe mulder herman overval voetballe aanklede eminem donalds komend lidl rusland losseneasy anita levert theo kzie112 makkie frituur grensrechter nummertje machtig gebied aanrijding leiding kay puck mans vs.bitterballen steve overheid rellen brommer aansluiten wederom hd leden ofwa hood spits biertje 4−0 yoo popo omroep pomp bespreking onderwijs passie judo joop assen o.a. heineken psv eredivisie tonkuip entree 2−3 milan traine dollar michelmatch kil20min verlengen c1 pompen scheidsrechter b1 management kzag regio samenwerking belasting briljant join kpn dubstep prof 360 meldt pagga echter leider terwel skool relaxt killer knul skeere +ned3 nadal arbeid aka software tegenstander student b2 doelpunt schot mang mw3 vivitesseheerenveen gta smerige digitale +slechtegrappen keurig jw lelystad pijp rijn politieke sprint penis oa gevoetbalt noob +modieus +ikzougraagwillen kzou ios dab zuiden doner broeder kebab michiel bod friso +3fm partijen geblesseerd 180 onterecht huntelaar 4−1 sp1−4 gebak marketing guido leroy wes vvv manchester osso 1−5dordrecht 1−3c2 bass ekt blockxl leon bazen heftribune ac gevoetbald achterstand scoorde windows popp burgemeester ingezet belang keepen scoor +os2012 +os12 wielrennen keet lan+ah milaan thx rtl7 wkkr sneijder kappertje speakers supporters bayern janssen 3−2 wellicht zwembroek blessure immers madrid pvv messi dalyk a.s. houdoe mixtape nac ant vant kdenk kzal supporter d1 vvd demo handballen siem docu tracks zaalvoetbal joinen toto6−0voorsprong opstelling goals penalty +ajax 5−0 verdediging ops sgoed epic telegraaf sc rutte kalm beleid kanen bestuur +ajatwe matepvda doelpunten 0−2 robben controller voorspelling coldplay oosten nmmr pokoe c3 munchen wasted +lekkah fixe f1 loting sar oefenwedstrijd ploeg overwinning kampioenschap 0−1 a2biertjes balotelli arsenal nec chez bsm barkie xboxen ajaciedkabinetcommentator kratje iscool excelsior bommel +tdf 5−1 knvb clubsleague poule4−2 010 jc cruijff united voorzitter fraai spijkenisse fa2 gaal lyonroda pilsje sparta koploper pelle osm sv kunstgras +houdoe +vvd barneveld +vi rkc +pvv +pvda +pownews etappekoeman +psv server boyz ballie brada pils matchday +feyenoord +out +later cda +ned1 niffo strootman mertens +feyaja vermeer inter cl politici +bier transfer +cda +voetbalpings buitenspel guidetti broeders stream +ps3 divisie +fifa12 +sport1 eriksen+fifa battlefield dortmund m narsingh eagles joure mw2 fifa12 suarez champions wijnaldum liverpool clasie m m gesink bf3 m m headset

schattig

zielig

mama

,

hoe

en

dus

doe

af

om

hihi

iets zitten

je

vakantie

weet elkaar

zeggen

deed

overal

niks

mag

alleen

wou

laat

weten zoeken wist

aan

blijven hand

maakt

hier

klein

het

laten

komt

willen

gister

ligt

al

morgen

zien

gaat

werd

weg

uitleggen waarvan blauwe engels

nagels

vragen doen

normaal trap donker speciaal witte

pas

dicht

moesten

leuker

iedereen liefst loopt nergens

was

dat

over

dit

zei

houdenzettengegaan begin oude hebt

gedaan

hoor langer dezelfde zie

helpt school allemaal

keer

zit

zat

andere

gewoon kijkt zwarte blauw anders enige duurt mocht vinden ooit leest had zichzelf gebroken kwam trekkenvaak ziek beste verkeerde hebben gestuurd lijken maandag stuurt halen half geen gezien nu familie hun ander horen manier fout geven loop geloven wc alvast stuur ofzo leven wacht geeft hem als eigenlijkgezet deurbrengen stil bestaatbegon best ziet waren dacht . ergens gelukkig moeilijk helpen wanneer he kwamen uit behalve handen kunnenhoort hoeft door geluid kop 7 veel liever sla rodemoment kreeg krijgen nou sta eet ochtend kwartierding welke buren heet idee merk verhaalwonen rug open gezeten snapt oor gelopen volgens opnieuw hoop trots vertrouwen nederlands bloed heen druk later zelfde ! vieren snappen help vandaag nodigverkeerd vrijdag wat haha 24 stiekem wereld nek allebei kort die vroeg tegen terug begrijpen achtergrond belangrijk slapen neemt midden ons reden oud geleden eigen vinger dinsdagsinds donderdag kiezen verjaardag geef reactie mist bed vond woorden fiets negeren mogen zorgen geweest leg gelukt mij naargevondenvast neem naam roosterboeit zet meestal vader belmomenten woord red schuld zomervakantie sorry leggen leeg brengt langzaam woensdag neus maak huis zomer precieshoorde =s 3 mond kant hadden ken stoel schoenen belde viezevoordattwitter achtertussen slaat zag gezegt probeer chips degene geweldig gesproken tegelijk geworden enzo noemendag wees filmpje blijfbedoelde woont kondenstopt komen onder echte vandaan inhalen films gaan allangmaakte stukje kijk jij mentor vannacht slecht zodat boven keihard huhgegeten las passen nepkinderen stopgedacht hangt eindelijk bijna gepakt inpakken been weken welk hoeven verhalen alsjeblieft aandacht muziek sokken misschien vaker nieuw typ kapot spierpijnmijne problemen zolang ongeluk wij jongen ver neer zwart alle leer luister tafel ei ophalen tekst zelf vol kerst kijken reageer gebeurt dagen filmpjesplezier < afgesproken vrouw tweets elk straat hahahah verzinnen regenen koude durft lees ermeeouder verdient eerder bedenken stoppen spelletje gaf meenemen mens mevrouwhekel stoerpakken vervelend koken bestaan recht bedoel minuten wit groot gelijk ijs meenwakker zaten sukkel leraren antwoorden brief water bakken vak oudersgrapje onze leeft cijfers pizza zoekt genomen getweet gestopt schrijvensite diep beterschap probeert uren plek jullie draait ruik kennen lenen les zoals hel steek leef regen graag grootste gezeik uurtje kip gebeurd gezegd uitslapen zijverslapen beginnen oven lui leeftijd ziekenhuis it dromen saaie tweetspringen eruitgelezen luisteren rood straks kougebracht vechten leek mooier houd gehad aardig pakt optijd raken tuin winkel antwoord klaar kleine nachthoelang mooiste dik reclame teveel duits eten soep groter gebleven volgt geborenwens hangen oom sneeuw meest meteen 6 deze betekent sinterklaas totaal muur vanzelf h eerlijk lange voorbij gekomen omhoog ) oren licht retweeten tot tanden vriend kwart saai normale lekkere slechte maand internet verleden haten gang zullen snoep boodschappen energie oranjeuurtjes teen schelden haren winterbedoelt 3e duren tl boekje spreekdelen stad schoon tv 9 doos zeiden klok hahah bleef facebook groep weekje opdracht begrijpt onweer zn roepen stress hoi kont bedacht maanden erachter ondertussen alweer thuisbestellen / gene vingers juist gezonddikke maarja toetsen netdaarvoor the relatie hahahahahaha vreemd drama bril raakt lokaal tandarts waardmiddag bedoeld fuck leraar yeah hee poetsen makkelijker eentje opzoeken gefietst tweeten vanochtend auto bui uitval moeders wildeverveling tellen zoek los afmaken gepraat gedachten glas ontbijt tuurlijk geloof plan noem to rt lukken bek inleveren verhuizen buurman beest slaan late dorst pesten gekeken wassen batterij blijkbaar schrijf hopen drinken and zeiken 14 ergste whatsapp opgehaald vliegen ontbijten sms slet okeezonnetje strand anderhalf schoot kies humorkaartjes houdtschrijft korte roept gat klinkt gooit lekker uiteindelijk spannend dubbel hmmm verlaten merken taal groenanderen ongeveer keertje gisteravond logisch geluk koppijn pffff meneer all doorheenkunt datum keuken vooral soort stel veelstehome nadenken berichtengun meegenomen zagen groene zee poep rapport presentatie geweldige mijnyoutube april gooien begrijp schijnt moeilijke broodjes lieg woon waarmee ervan fan namen oeh vervelen sleutel lessen verwachten steken amper verslagdagje vanmiddag enigste erover zoenen mac alsnog gelooft toetsavonds lekkers ruilen uitzoeken drukke link hmm bezig gedachte leert reclames ex zwembad lullen online vis zacht fb hw vliegtuig toevallig tegenwoordig hopelijk appeloortjes zwemmen account contact hoefde voldoende date mug fotos min meiden alone triest lichaam wtf miss computer goedzo schooldag perongeluk vliegt jarigedaarna zoo papier 4e zon camera voortaan toe heee lied kat onthouden toekomst ochtends life kleiner hoofdstuk gelaten flauw joh sws +not letterlijk meinamelijk heerlijk gelegen gisteren meegemaakt know basisschool rtl shit oefenen der toevoegen ring geschiedenis droog geholpen verschrikkelijk opdrachten plekken vans m getrouwd r 1e melden hahahahahah jongetje overleven schoen teken deden b feestje brand mooie pest like party doorgaat aflevering vergaat aangezien broodje perfecte afspraak slaaplekker cijfer zinnen ah vier holland koelkast neefopladen top dragen lekkerding reageerde vriendjes gestolen pech doorgaan gehouden bepaalde zomaar nespelletjes linken hond daardoor rook z=n e schieten allen geknipt sloeg haartjes vet cadeau bam vakken tel telkens luistertvoren trekt december23 punt officieel my respect apartwonder beugelstadje broertje engelse volg oja hield new honden sterkte cent reed cool !! faal offlineherken ingepakt verlang oordopjes koekje achterop dichtbij a sterren havo breektwaarop bereiken klote beurt markt dachten fack iemands ruikengooide opnemen bedoeling gedragen breng pakjebanaan tijdje sjaalplakken ohja kermis zozo verzend koe hema hahahahahahaha duss bruine moooi broer werden xd lust visite economie verneukt bos hahha muggen gaten patat klap hulp plaatje rijden gvd stijl twijfel ma zweer restaurant meent tv. kun aub knie not boeiend seks back twitteren afsluiten overleden pauze gezellige groeien naja moppie +kenjedat nouja twijfelen weekendje rondje ww ren test 6e brood do uitkomen eenmaal melk nl fietst dann goodmorning tegenover = jee geheim automatisch laag kende gedownload veilig baby nam meee sneu oooh steen 5e heten telaat what verander ziekte stellen baan zonnebril okeoke hahahaa verandert duim discussiehotel beloofd face wifi overmorgen boekverslag spring down + gekozen liegtverwijder d wennen hhaha letters beters haast vindt tosti nachtje godver draag nut gegooid onderwerp achjadurven slechterblind seconden aula amerika genoemd bezoek aah uhm nerd verstuurd btw tien andersom zwaaien openen stappen == kampbullshit volgen wss zowat voeren ooh at sneeuwt daarmee waardoorbrug nick naakt scheld rent blik hemel stonden volhoudengwn +fml fietste for geprobeerd droge tering spanje slotje klikgrappige pannekoeken dankzij pfff. DQUO savonds aarde vogel slimme kleuren meerdere zodra vroege na godverdomme hoge uitlegnachts gebakken skypedownie wachtwoord jouwe ontbijtje bepaald no ipod stelletje facking doornemen ohh uitlaten nadat dak 27 bootplanning l opzoek feesten maag accepteren uitprinten why yeahh samenvatting fucking ipad bewegen weekjes on wrmgood ikea stressen raden lader docent chick aankomen krachtdruktegewend vreemde schaatsen toetsenbord spongebob lult letter lol gs schouder gras geil pap ontvangen followers overdagiedere be avondje verbinding gekookt zusjes gefeliciteerd vette +love ahaha weetniet doorlezen concentratie letten smaak zakken la bereik krant ahahah y plekje sparen troep vergeef aangereden single bankje uiterlijk +honger mongoolmeld tja profiel warmer achternaam zeurt commentaar tweettefranse tattoo bieber heus bus kerstboomtandenpoetsen aantrekken drukken zekerheid carnaval wedden damesverdien brak raad werken woonkamer schoolfeest wang your i=m pc heerlijke geluiden biologie muziekje gedraag praatjes voorlopig juni jaartje ophouden zeur eeen wensen haasten alleeen bemoei blonde walibi duurde dochter bereikbaar herinneren gezeur pop apppersonen vervolgens graden ramenspringt yeaah inplaats bekijkgeduldspreekt parijs plat war 7e abbo hah thema ontbeten smaaktcode showgesloopt have halfuur kilo f opzetten post uitgeslapen drugs morgenochtend zovan prikken oo ware spul ov ideeen plassen concert 29 hartstikke avondeten ok buurvrouw hoest station feliciterenzette idioot +wtf sporten geluisterd heledag maybe blaadje konijn toetje iphonemaart september efteling hardlopen piet priveopletten kopje juli men s vwo tss aanzetten mooizo kfc +moe gereageerd zelfmoord reacties voorstellen teminste ruimen bijt stelen vonden onbekend rol favoriete donder wistenkoopt go verveelt type grapjes aangedaan toegevoegd dusja 2013 e. +tvoh stink pijnlijkvanaaf lukte riep hahaakaften onvoldoendes betekenis whahaha ortho vage ontzettend vals irritante kotsen max oktober a=dam straf overleefd massage expres gegooit slippers oogjes jep richting uitgaan goh appie ooo meeer v frankrijk twee 06 miste mentions voorhoofd uitzetten wolken deuren dieren wats bureau kantine rollen muis bruiloft laura uitvalt poes pw typtzegmaardon=t teleurgesteld gemiddeld gadverdamme geniet volgensmij negatief bewijs mailtje boy stilte +heerlijk mochten wijs pfffff mes keus waarvoor zwak random hen vaders werkstuk branden mike steekt dvd wtf. tenen +np baal oorlog woordje moeilijker55 waait day examen belachelijk gezelligheid geraakt kutdagbinne gereden lopend pad id vlees pannenkoeken ohhh bil schouders benauwd techniek nachten vloer sint bakker broeken falen rampgroepjezandvangenbowlenlente opgestaan noemde waarbij gedachtes herhaling o. onee tsss cm zometeen hahahahahahahaha zeikomgaantrue adem fruit laden kanten klootzak leveren min. overdreven ey lege legt hierna hang knopsportdag maan vanmorgen sportief clip heeerlijk hoekje naa omw hij/zij ow +blij ster dude hahahahhaha knieen dora +ziek hehe eierenklikken timeline nietsgestaan vertrekken beesten whahahaha bekende filmen nooooit foute opmerking stemming gevecht schooljaarnah heette morgenavond fijne w enzoo smeren mar drankje sec katten doordat whaha kutt ahahahaha hahahahahahah coole gaaf ikke this lunchen kunnevolgers ahah ehh deal mailen blz followen hi follow saunastukjes opgegeven p wegg perse verlegen ~ overdrijven8e bob jwz tijdstip reageerd verkering jaaaaaa appeltaart gesloten bemoeien barbie hahaaha klassen +bah keuzes +kut time sluiten herfst ofzoo sanne beroerd wallen kutweer behoefte grijze getrokken gala vooor klappen flesje jumbo acc nablijven pepernoten turkije trouwmogool out gejat reet beslotensex vermoeiend idk merkt laad rekenmachine kids acht welkom du trein kim opp spoor negen gauw samsungwijze today onmogelijk tasje nja nieuwsgierig whatsappen geheugenhitte gezwommen camping high t. hap zijn/haar ha energy scooter naartoe tweete hahahaah +haat uitvallen simon d=r turksesnachts handschoenen zolder hhahaha hierheen ikk trillen =t gevraagt beltegoed stage prachtig +goedbezig gewacht hamburger verandering heeeerlijk tijdelijk lesjediploma plaatjes mann eco style ste ahhaha masker modelspinnenkots bezetsleutels daan telt bleek mention kerstvakantie kwaadkutzooi zender verkloot verzin haaha wiet oef ofso verrot weggaan geretweet poster get wild regels okay eindigt ginger +yeah lift kloppen pinkonthoud asafspraken momentje examens mm sloot uitkomt starbucks hea bestond aanmaken slimmer kneus jup blok afstand ho bereikt betekenen vodafone doof hmm. t/m+eindelijk nf zoooooo grrrr geurachteraf ong o_o verstopt inloggen rijst korter prins voorzichtig middagje grijs gezakt hobby gebouwjack muts slagroom rondjes +pff enn zwerver indd ehm invullen boel kikker +dom dm=s 0000 ciao haah +faal londen +twexit bloemen verslavend lek neit jeuk scheef bende november schrift scheelt uhuverzenden lerares scheikunde hmmmm smiddags vogelsaanval kutje jo vijf msnen monster 0 01 gn kanker+trots nask tvoh doorweekt zucht molnutteloosenz. +leuk breek dode zitte den neger hahahhaa beer unfollowen milkshake agressief zul stroom chris bram faalt gas opgenomen vervangen proeven gourmetten spaced10000 opstarten mja mongolen lagenbeland nieuwjaar ingeleverd wijnbijzonder omgekleed au piep waneer kruipen an reizen retweet bart hahhaa hartaanval daten schreeuwt sukkels billengemerkt tieten blote bevroren rookt hm ll off wii stof +sorry schijnen schoentjes uitnodiging supermarkt terugkijken stoned opschool jeroen aankijken gestoken dichterbij afgeslotenaha tips oma=s zoon telefoontje em fietsje sneeuwen openbaar versta 36 varken thuisss whahahah beseffen ofzoiets bieb broers kado wis t−shirt teksten bussen knal fris amsterdam hahahahaa +haha bespreken email vele haters fik adidas zegge gm +omg roep gaaat uitgenodigd instagram groeit player mc neppeaftellen heftiginhoud enorme um beloof dier overschrijven werkelijkangst pasta sms=en probeerde plastic menen vretenadvies aangenomen vaste taxi 08boterham ahahha ontmoeten onderbroek hebbe volwassen verdomme willem leve busje ahahahah dekbed strafwerk trilt fock gymen gewassen doorgeven icarly jippie vrouwtje slappe daarop eindhoven +lekker stofzuiger gezin depressief aa verkrachtthat fun +school wee 600 =m zulk+verveling albertexcuushoeren hoevaak huid nakomen dromenland +goodnight herkent balkon achterkantpan kassahomo=s live sterke hahahh bas +saai wasmachine +hahaha luxe favoriet pp ontvang toppunt isgoed ophangen zogenaamd getekend vertraging +kapot ninja inspiratie owja schaap verzonnen varen wt hetewoudensticker 400 suikerontdektstaken haahah stalken s=avonds chicks ahha stinken duik uitzicht eronder twittert kunst kater le herinner bitches gone gekk vn gewoonlijk typisch gepland +loveyou mondeling kijke +slapen zenuwen focking cammen lampje oww family eva worde polsunfollow lijf ns gerust maandje wrts 39 maccie kansloos babe +herkenbaar woordenboek zocht ghehe inhouden schelen dr. paraplu blad whats smoel natuurkunde zult kraan goeds oeioverheen indruk bom erna motivatie treinen uitverkocht buite huizen kachel touch duizend apen way sigaret geweten aangevallen ojaa bioscoop geredmoete danny belangrijkste hoogste maatje goeiemorgen gedicht xtje +doei wauww marco z us enorm annespuiten et lekkerste omwt kome record overkomen beeldscherm +lol aardbeien brb regende ezel kak debiel + alarm bankhangen minstens verbeteren buitenland roken robin tweetdeck kk pasen gunt schuldig gij frietjes j uitgevallen printer will afleiding ontploft foutje camwoonde s. sprong rozen sam braaf maa [ glazen c mmmm versturen mot storm brievenbus keiharde velhopeloos lenzen zombielevend docenten need ellende +linkdefollower kruidvat douch oooo wachte schreef ht kevin // opgelost dennis tof afrika geest streep portugal +happy eigenwijs maja+retweet berichtjes achterlijk geak harry aangemaakt afleveringen tweeling jb uitpakken kap 05 lager sprak mense printen es holwekkertje noo ] likkengaande ssverhaaltje pedo ookalweer bewaarnuttigs erom demi tl. it=s duwen titanic gaga iikgevangenis slikken deo drm +etenmuseum aids neuken tell sporen zovaak video leiden uitrusten bushalte mi gezongen gunnen wagen spamboeren paragraaf kippenvel bbq=en hinderlijk broertjes ventje hoopt night vloog gezelschap diepestoelen +goodlife gelegd 34 pingthaahaha knoop dinges yeahhh dinge ingevuld juf emo +slajezelf m. etc. contacten rustenkeren owh volume scheetsnapje zuid toetsweek eeh belle liepen twit patatje villa favo fijner gaa pakte koenvergaan vervelende absoluut kn kamertje ge verstoppen stapt verslagen godsnaam overleggen lucky dam plegen gepingt homealone jesse plein haalde jaloezielaaste afstandsbediening blije +rtl4 +tweetbreak enz se operatie repetitie nouen inn x_x johh+tweetoff eenzaam piano joepienaaien strijd toppie watt koel sochtends clown lotteverslaving kleed hale tennis meemaken schamen kweet di dooie uitschelden kleuter verlaat hoofdstukken ff. +raar perry nogg amy +vroeg ng wordfeud p. tegaan +slaaplekker hup duimen grammatica kopieren bezorgd gierig slape drop terrasje powerpoint beginne neejoh pakjesvrolijke proef anna oow snor zweetpatrick 38 d. pinpasdushi lovee rende hhahah dm.kabouter little herkenbaar moed riem hapje ckv steunen zeeland voedsel landen turk hersenen achteruit verveeld bekeken bn daaar dichter romantisch lunch noodles herrie wi friet lijstje tik mogdingetje prate he−le−maal lage bijtenvraagje mongol gevoelig gemaild vergeleken slapennn skypen yolo dodelijk hjb mode maatschappij graf fietse plak snijden toon pillen worst wetgoedkoper hamster bv dagelijks daarin mospannende virus david bultdankbaar ambulance mooooikinder emotioneel geplaktpreek lekkerr dm=en verdwenen bami eiland welterusten snackbar vry opwarmen spaans opgeladen regenboog lasagne lengte wegbrengen volgde sommigen horloge collega heey wapp weghalen yeaaaah kyk smerig slaapkamer oplossen +jammer bewijzen sukkeltje maandjes fatsoenlijkbot nakijken +rtaanmelden verstaan 09 neiging boterpuntjes uitdoen slome rinickelodeon kon. sushi dt oohh wachtend bollevertalen wille pakke boze worldbepalen verzamelen onbekende donkere uitziet duh loos ligge durfde lik stout verschijnen podium schijnheilig activity zere again jeuj verheug yoghurtrt=ensnurken pindakaas verschrikkelijke gebeurde dutje metro gedouched vrijdagavond jonger weggooien vooraan hardop kleintjes populair m=s zooooooo am kleinste so=sademen mruitgebreid stenen vat stront lekkker luisterd +kutschool halloween +wiestaateronder twitternaam who +hgt huisarts +balen beleven situatie malleslechts hebb adam koos vlakbij noh goor +oeps weggaat gepestanouksimpele genoten tegenegatieve uitgestuurd middags geregend opgehangen begreep opblijven kellydierentuintfoe uitzien stapel buikje paniek +jaloers nar retweets vaatwasser avatar apps 48 fam herkende zicht aant truste voorkant zachterolifant aleen hoorr derest klassers vereerd iknow hbgeheimen beet dee herhalen im toegeven beantwoorden alexhiermee kiest ritme stukken schooool heren opweg zong oppas procent spaghetti ernstig programma=s vlag ongesteld gaatjes oe ij bedoelen eige domste neeeeeeeqqgeheimpje now belooft esmee kapott neefjes see schoolplein profielfoto etcstokbrood meot daden matras +slecht kluisafsteken irri koeien scholen studeren klimmen +beter huilend beroemd overnieuw versieren verbranden krabben sarcastisch redenen minute heerst vermaak serieuze marktplaats controleren chatopkomen kader ferb sterf elf schone express romy yo pinnen pwwwaarover pussy verzorging gedraagt print jerry +geweldig disneybuddy begrafenis beweging genegeerd uitgezet haag valentijnsdag verassing sleep +zielig valentijn ikga kerstdag mistig menu mvrmoeilijkste wouter 13e 800 schema hoeveelste zeikt twaalf nieteens blokken dats endanhoere vermaken boverzonden schaduw koek middelbare actiesmartijn geile guy bol zeven hierzo shock luchtbed stilloverleeft mediatheek kelder aanrader aaaaaah q verdwaald +hihi vin song gers schop rep turks terugweg tegemoet antwerpen herinnering trainingsbroek gevoeld gaatje herkansing sjors soo poffertjes onweren spijkerbroek dalijk appeltje nauwelijks klopte voer uberhaupt strijkenfollowt kleintje thijs missie evevaderdag curve pin respectloos zes nrprullenbak next noeteindigen west 37 limburg 99 ubersocial meende niettt action hate echofcourse doorsturen bananen chocomel luisterde kouder dies kook wedde tiefavorieten ka feestjes deuk 5minbesteden halteapparaat vierdegemakbereid boem bold web iceachtertuin nokia ali gedeelte klonk krukken bc ofs wajo lef music forceren friends beseft herfstvakantie wesp gesprongen blikje wegen enig 42 topo +msn afgezet inkopen trampoline geopereerd stoplicht hielp gemiste daag aardappels bbqen gordijnen shop uitgeven bosjesleerde uitgevonden hongerrr voorstelling soepje simsveeeeel uitreiking geitsigaretten ranzig expeditie goedemorgen mars voornemen flipt ooow trauma vreet 07 drukt slechtste dikker brandweer moord achtbaan blaar aanraken look vla spotten s=ochtends modder relaties vielen daarheen wonderen china nie we=ll daarovertsja luck hahahahahahahahaha twitt wisselen usbbloedneus zakt bedanken zoiezo femke frikandel feliciteer meegaan mapafwassen gepoetst ontvangt oprotten opstaat geheime omgeving jet weleens schoolspullen opschrijven weetjewel knapper doorkijken verdwijnen people maatschappijleer lope hotmail puh aapje hoopte phineas onduidelijk opslaan > chillings weerr schoongemaakt can plafond merbevallen vor salade awesome coma zoen wachtkamer mt w8 verkeerds lover depri kutschool verveelde blokuur edwin hintingehaald buienradar achteraan tekens noord bert dank ii hadde uiverrassing linkt hardstgrapjee +feit t−mobile snurkt verbaasd lot afwezig boren ds joey vd zeep u_u dooor arrogante voegt got meeten overstappen poten gelukkige yeaahh gevochten grapjas gezichten opendag werkdag bezorgen tekeningen girls jordans slapmeester mcflurry+mtv aww ava=s smakelijk vrees speeltuin spieren heijnthomas julie koppie maarjaa mouwen schatjeee +hoofdpijn onweert +tv zout homee here macaroni sok disco heks onverwachts itunes kutkind kuttt hahahahhahaha meissiebenn muchvlekken loner sharon hooikoorts batt manieren rommel dis +lui keizer+zucht overkant brabant lichterotonde rooie wachtende honderd dd hot begrip +gn schreeuw afvallen inkt+nietdus ofz schuif buschauffeur fail 3de sterven rappenbosch you=re pauzes peer step aangestoken opgeslagen betrapt nachtjes jawell pfoeepardon vrachtwagen uitproberen plas spreekbeurt gatverdamme gehoort give scheveningen honingwoede +kangewoon heftige gelinkt bijzonders right bergen ingeschreven wijk nuu morning va+seksfeiten proefwerken sss zwaait noooo wifey zen fysio vul danku rose =n utomdraaien stort spuit extreem hijs barst hha think tussendoor hahahahahhaha saaaai maker veder aaawh +wishmeluck tand ahahahahaha elkaarstiener drol robinson lawaai redbull slam eend eikeloliebollen enthousiast bikkelenleger nachtmerries voorkomen lucas macdonalds smit bewaren mainflessen +slim d8 dna ft. bied handje wtff fm genezen crazy eey slagen groente stampen +respect sgatjerihanna gio opgeschreven begrippen handtekening hing gelul ingenomen ongelofelijk +buikpijn tript gerend mailtjes kippen poesje thuisblijven whapp holy tussenuurtje +spannend phone jointgehackt bezoeken zakje aaa voooordelen strax 000opgezocht hahhahahucs junior denise tirol pffff.konijnen kookt uitmaakt +liefde kin talen wokken schrikt viert zinloos lichter tikt droomt hok minstopgezet schudden stekker dar huren crisis vermist schande hilarisch dontshowen festival onderuit ps verseverband klikt beantwoord verbouwen luukrondlopen oudere haahha ke skien uithalen +1 ruziestaartje weggegaan geroepen u__ublondje vliegveld wendy geaccepteerd mjhahaahah gesneden maagd blaadjes logica nuttig jaah duidelijkheid gekste uitgelaten handjes papiertje godsamme gedag tijger hahahahahahahah jatgoedkope begrepen jatten opgestuurd prinses racen pardoel dop mavo optocht wa kramp iq nijmegen vers baggerlekke vnv vmbozowiezo +true 02 +rtl5 5000 esther simkaart hoera uitstellen truien whahahahah kerstman dave welll ehmmtweeet lame steelt +niet nietmeer instellen hiervan haa omlaag nike maastricht losse eb bon kheb chickie levens huisss brownie +leren gescheurd douchenn nix voorraad thnxspaanse verfen riksmoesjes +yes pennen srry recept aparte poot bandje jes dringend +vorigetweet pagina=s pieter klasgenoot aantekeningen prikt vergist bladzijde sjoerd gordon origineel prinsesje eindelyk sytycd lievelings jank juliaafgevallen pil aantrekkelijk pakketje +engels onwijs zachtjes u. huwelijk speelgoed pizza=s heilig peukschapen uitkijken rustiger hoofddorp reuze moer same accounts wete accent 4000 8ste endmeerijden benede feiten +brb herkennen free hater herkansen krokodil creatief benden +tb vertelle ahhahaha charlie dekken chocolademelk zeiknat verkoudheid denkik made undercover handdoek kust 47vasthouden kieken mob dirk+werken verzint kben nachtrust vertrokken sociaal vlucht neuk artikel momenteel helder flatapeldoorn overhoren zalm wakkker don skeer website +besef duivel nickikrassen adhd tablet hahahhah euh egtgezelliger blut k.verneuken mengen tape sgat maargoed joggen geblokt gekneusd gecondoleerd gedrukt geboekt boeg puntje nagedacht geblokkeerd gewaarschuwd wandeling achtig wedstrijdje tweetjail komop zoe bekijkt wajoo ass ly jongetjes +sytycd ohjaa vrede hip tj nadoen thankyouu ennn zelfgemaakte heeeeerlijk zoontje mannn opdr knipahahahha pappie verplaatsen pasje strijden emmer leventje jochieomheen skippen uploaden ej uitgelegd kapoet saus wnr or rome natuur raarste rt=selleboog lichten +tienershit+vt blaas turken cavia kriebels lachwekkend schilderen andermans komkommer adele daaardappel steel skmannetjes 12−12−12 bedenkt winkelcentrum london dickzoveelste condoom lekkah +omdathetkan dilemma damnn dannn meids dramatisch vanalles sufanderhalve kwil yay gen zappen streng guus typte appart wintersport tegelijkertijd +waarginghetmis slapende +gvd vogeltjeslow gesport cking groei toestemming joe oudste+thuis bodyvroegen jr soorten oostenrijk bijwerken roosje geprikt vrijheid pannekoek overnemen allergisch glaasje uitzondering ensbewerken emma pffffffbolletje a.achterlijke ws un oefen walvis smoothie shittt +tvk donuts dn weetik vork schut raakte houde +wiskunde hollands haarlem ban dikzak niceee dwars veels beurs combi netwerk des michael leeuw loon 4de kritiek +druk ohw jupp mislukte meivakantie hahahaaha dees maria boeien wilden follower smurf duif progamma +gm+wakker gavegeopend vrage ong. aangezet tegenaan gangster ontspannen toppers jongee vrydag joggingsbroek geenzin geboorte longen spelling godsdienst apotheek jeffrey verplaatst lijden svengewenst blijve asjeblieft panda kluisjes funny setje york special junk bootje berlijn bijdehand baantje wesley polen sarcasme +moee focktop filmpie wer hamer aaruitspreken ven wanner avaa +jippie oktweeekend rocgenaamd zometeeen rat uutopbouwen zoet potter straten suc6 1500verlopen ikweet whahahahaha risico koffers uuh about ofsoo niterugkomen anti engel bi baksteen hahaahaha lisanne ietbasketbal let=s +party bbmbijnaam kkr that=s alweeer datt ahhhhookwel +hahahaha losers donutbidden euhm realiseer rede dunne 10min eeeh nein hmmm. frisse heur theater effe australie kaak choco france oost getrapt fk sneaky sorrry gwn. overgeven augaankomt voegen xpverspillen nastysterretje poort computers magnetron meegaat paste zuinig langste ginge ap blegh nigvooruitzicht opdagen gedanst eetsmakelijk bladzijdes +wauw maarr cv wrijfsollicitatie afgemaakt bepaal +lachjerot opschiet sletje +warm dphelemaaaal jesusdoodop jennifer ajb +heelherkenbaar ananas whap lomp care+hatelijk stijn workshop strandje twijfelsstrepen eitjes eej eeey kappen bijv marcmobiele wauuw tusse naataanvallen ssstbeloftes 03 besluit trieste bin kamers whahha dombo oehhh melanie ha−ha naakte broekzak take decuitgezocht alkmaar klinken drinke taken uitmaken da=savontuur kuch call beloven beterr merkte bestelt mazzel ted stoken misbruik breda delete 5de brown bloem likt +stress oflinemammie refresht afwas formulier rtl4 gehoopt pc. agmasseren grrrrr negers paas po tonen ongelooflijk whoop gips geloofd figuur gooie only r=dam aangemeld minaj grieppaspoort +fuckingfeiten aardbeving wejo klasgenoten lepel stijf daggie menn flip fanatiek +nee mevr caravan mean tssss meet tm yep kogel sate tevoren 51 klaasvrijwillig 56verdieping 59 loveeeu___u +hahah kanndouchee smile alfabet stefbewezen oooooh baar heulbeslissen chinezen ko onnodig fucked bullschoppen kanaal details behandelen 41 cherso done borsten work fsoallemaaal welcome ahahahahah relaxen nino +widm verdacht kok maal celine hardd +foreveralone heh hiervooropent 9euithangen il +gelukkig verzorgen guys g. bedank ryan poept portemonnee onveilig gevlogen hagel gucci something tegenwind droger gewild banga getal gewicht slaag gesteldeisen presenteren bestelling gemeentehuis +wieuitjeklas papieren kijkje presentaties niey pleur piemel jajaja boodschap vrijjhockeyen laptopje joun songfestival oefeningen mijzelf john fijnn << +truste vouwen eu geert toilet renee adios wnnr neenee ome oeehhahahah meeneemt ete tt dieet 46champagne blikken eff isa smse boerenkool roelvolleybal avonden zilver onderaan dol hoorn okk +ik twitcam folders =ie movie olie lever mel +mls hahahahahahha whahahha vriende texel csitijdmachine wei +boring timo horochtendje wollah flikken zkr kenne heleboel vinde leukere kaolo verbazen circus chauffeur enschede hal space hoeso tmf gangnam spammen posters owjaa +hyves doorgelezen oohja jip dorpje slang jamie log verkleed getwitterd kontje morgenmiddag volger joketelefoons etentje zonnig olympische toekomstige dylan bijzondere pinge +sleepwell kutkop kloptt vera uitstappen pokke reserveren flesjes san planten turnen toeter trusten 6000 joris 900 kroket stoep bezit cito battle dnekcharlotte hai +shit stres +brrr hoebedoel auditie kmoet tennissen waardeloos nood vriezer draak hak blog xdd handvaardigheid wol ikheb krijge zulle idioten denke lamabolletjes ominpartner machine wahaha aff supper +nf heeey draad en/of faalde hw. if bikkel oftewel say gaap geeneens egohagelslag ny rinus dinernamens rapper klus robert vries europa o__o tess gert egypte pret +grapje altijdd minutjes pinda klaag schijn jasper pijpen vrouwelijke hopgewonne gijs veters denkje kempi dadelijkofwat verslaap spek centjes ondertiteling beentjes byeee potlood ripautomaat corvee run wauwww moes swa csdoorzetten +waarom dafuq enne nounou +nw verzet troost 04 49 alleenmaar sarah zaandam tieners ooohh lieke kanniet afzetten ubermobieltje kutleven vandoor flowhandiger amersfoort rotzooimoge tikken rtl5aansteker tunnel mehdumpen when ojaaadenhaag houten zussie downies +nederlands eindstand handel neukt ahahhaha schoonmaakster uiit bestand nikki jammie zegen grenzen +tevroeg regelmatig gezondheid volgends +goodmorning sigaretje jongste geinstalleerd jemig joodopdrinken ownee afdeling gr duitser uufollowback jay maare miljoenen sas hoertje +jammie okey opdonderen tuintje lampen postbode waa essenfcking veldjeuitgerust uitvoeren wr tweetten tomtom wattan sst rten story zadel stansnapback erwin hu jeffdanmaar hbo genakt bossrolstoel bassie rachel werkenn maud +lovee marloes krokobil merci hett kroatie dion leregestuurt vierkant sites harte vermoedenwo rand bianca induiken beau languit und lv neuhdommer mit fv eigenlyk eg skieen ya waarneer scarylekkerst qq. +byee jha mpp klaargemaakt +jijbenthet brenge alsje shore werkelijkheid +geniaal hardgelopen taylor pis sow hoedan noa pim wijfie mwa lappie voornemens spier vip scheenparisvermaakt eeuw cel >> pakket fftjes comp vett art 95 stille ichleestoets werkse trailer +nietnormaal achtste draw valle weeek luilak s=nachts zowieso boksen +amsterdam tickets snowboarden zoeke drake relax beef brussel inbox luc dien bf opvang eef fffvolleyballen liefe till lil scheuren wappen wapen eyy x__x mett organiseren gevaarlijke sjonge uitgegeven dying sgool vndaag eig +duurtlang koninginnedag beslissing opgelet geneukt +lovefeeling blyf smh joa vanvond jersey neejmorgens bezoekje toetsje jaaah leippersoneel verderop matthijs polska paardjes jimmygespeelt joep aangifte curacao +sarcasme vv zweten rens vo break zure fase +auto tentamen tc +hmm toet dese doeg cont 10efotograaf lb daarnet nfbghehehe kauwen btw. drukker middel haan mien stok marieke hilversum bean sloop arie chickies abonnement doeei vnd roti +hoedan +xfactor lekka loesboerderij lesuren turbo bla ab bezem zielonbeperkt ontdekken hollandse borrel debat geve ys luistere trug levensbeschouwing slagharen gekneust gezonde hangover gevonde gefixtontslag helikopter hiphop poppie sjans zwemme vijfde baasjetilburg +zon vriendelijk ju vijver 85 1800deventer 1−8 zuster cover ventilator bewust ruk kalender airtankstation lau enalles heei nual thankss amstelveen skat sie lobi ellen heid deenwakkeer simone nehmarokko abo crack ticket verwerken mwah richard rene gerard bron vincent its +weltrusten aiiiknakworst klere ballon intro flikkers flodder +ihvh llll v/d nogmaar smorgens gegunt ptawhappen okeej stijve beverwijk heerlyk spulle +jwz grapig poeh ji goes remco kope schijten cody jamaar spele dage danique ashley ennu njaa gehehe scene fy ovr vam xoxoxmeisjee eah raare wejoo menno oss proefwerkweek tentamens 53 chinese inval mie laatse dead steden onderhand loll wiskut omklede lekkr dinokijkende claudia habbo +drama leeuwarden dolle ictproducten ruudschuren +bzv sit geslape naah maf loss 7de e−mail b. daarnaast sbs bocht +werk ffies +ff brieven +grrrr+twexitt cap casper groote keelontsteking vamiddag +goedemorgen sjongejonge terschelling bestemming belgische +herkenjedatt daantje helpechatten knuppel leipe johannes aanrijden alphenslager dominique jim accuoefening non schip +pffff st+geenzinin weert constant buss hahaaa ia sensatie muil +toetsweek zinin lois denkk kvind fuckinchristus mbo +feest louis +loesoetweets liam oowh snik wessel dnk bove +welterusten weltruste wies tedoen uuhm +sinterklaas intake ieks twexit daniel vuistswagger viezerik wok +stageverloot lkkr macbook banken zi hoewelsteven og caps +top nicky+vreemd heavy trending zonnige isgoedd julian jasmijn nii +moeeehahax frituurpan heej +linkjefollowers peukie mietje pleas } aanfietsen zuidplein ec thom =ns ooknog neven vrijwel ritje kiekupdates sture teun pascal street 1−6 administratie duo lopende xdxdmonique chilll lord tellie lulle 1−7 +ciao +sr12 mounir verloten blue 6de moven eehmgheghe biken buitenlandse btr tank +dwdd nit mooiso uf faking nk emmen +gone appartement nigger berrygezelig somige wnt luna kerstgala youri jooo sandra bp portfolio +geschiedenis getraint voetbalwedstrijd betoog pool peuken taaldorp bietje slapie twasbacardi bjorn jewel plzmentoruur rea hannah suzanne move c. anus james wijffie babey vr exact scoot steets 1900vergete +smh homie dich kee jessica nienke mick com hebbiemegool lo twittere invaller dronk hehehe almelo elo yup moetje italiaans kooi bnt ondernemen uhhm conducteur seg yeey wap sofie +pauze +topmoppen pwswerkweek gdzandvoort +hongerr +verjaardag meldingen herkansingen stevige okayy janneke floortje krantjes ahoy twintig joel rijlessen enjoy sean zow irritand +truetweets leo hugokortom watdan kaka +hoi gwoonnouw togg wahahaha eric duiven +foodfromheaven whahahahahaha hoar mier delft advocaat lkker gezopen pleintje vandag gerrit instelling neeej joaa carlomamma hengelo stephanie blijfen probere pleuris ramonsporthal slaapie hp spotify schijfobama zoetermeer mart sterre miami doelen chrome mainstreet ede meeting oowk niekgekrege bewolkt wodka rdam thanx drummen dronke lool +yeahhmarijn joy uitbed tgn s/o schoene kiki fietsie friestrainings ofseu sara vespa rutger ges spar goan qqqq wayne qqqhoogvliet pappa+pww n_n faya jordi jmmr cynthia pb jj kimberly jill whatsap paula chagarijnig diego jazz tnx handbal grieks jos latijn melvin saar pfft marit +ns ppff tweetoff lola deka hahahaaa niiet inge ofzow leonie balle basketballen nathalie chiller mainnslaapen viese kword olc wal klappe para zeilen hartelijk coco +mlw poosje okej ofja marleen hiep hooft hebjemoskee loempia cas bijeenkomst maas kampen nw gistere rr owkee aan=t sabine martin nien tostie kb w8ten xddd insta ind douwe neeh danisuriname bk valk florisjochem kwas lex linde haas ciaoo dammmyrthe literatuur +slamfm +lml maik fred kwt sem ofsow megan spang praag segt gegetewand jelmer nigel gino jens brokko joeri vomar dagt pien +gym hare ophale corramadan mrja +retweetje jackie pepijn purmerend +yolo zalig vervalt +sr11gouda 4s aaai fking renne hoowrdingensiedee din det isabel plzz mitch waker swek kud damian elise venlo wim roermond victor frissschiedam freek +gnx yara +doeii shag eingelijk jari volges rico +xtypischtieners sijn zecht kanjer cheyenne +expeditierobinson benje ekte jonko schietpartij afspreke mss glenn netzo someteen xlove skatebaan oohw exit jose bussie roffa +duits slachtoffers ku maahn informatica column heracles db rijde kar ehv grunn gr. parra geselligeiggmuwhaha guusje dig neet adje niggers anw +koffie boog mun yaay wiepennigga waz mies fotograferen aaron +jeuj wor koulo +trusten niall eik werkk kats vnaaf goeiinpakke shisha roosendaal drachten +trainen zekker waha ef stiff +wordfeud volendam maffen vh zwager veul neen soraya +top2000 hoogeveen gieten meppel aju giel mitchell mannekarten ea jow ta ortothailand rokeschagen sluis +itz fve claire fck ivo minister ho=s +adios sin alls hein lorenzo dancetour waal cker vk hbt yannick +penw eeyyhellemaal dagga boeng waggie vndg proficiat bakkerij beetj fakking hej rox hilde coen zoowfanclub su gehe goeiemorge mert luun jeez yael mila x=d kuil ffkes libie +lekkerweer rein ominglisettevaleriewrom robbin gsm coop ingrid syl juaa +shitdamnsjeez gina +twetixmahn z. srs slapee nt zwz dur xpp kerol jouwnwakka bes goon hidde oi mathijs renske boh jorn ruby mhh +in+nuiphone europese roxy jahh togh jarno briesje eyh sjat nynke verkiezingen strx iedreen 020 ckr eigk sneek loten fox dtc alsk andrea lauren sebas hallentakkie vodka europees trg opt mijnheer niej graafschap premier lijp alltijd sophia senna zown zoom pinkpop mos ekkes ofss richelle tele uitgeest amai lokale whoo fe dronten aad drek doetinchem +fb ike lowlands mrgn mirthe ig milo nag e_e gerald aisha dian +baas wallah iran mbt ard dgn jona stein w88 emiel px +groningen groenlinks mogguh fokking esme alissadaarlijkdaniek zina marlou zomenteen hzo +tweetbreakk urk dordt buit tori +off heerlen +huiswoerden +nuandroid adressen +meidendingen whisky weing kristel krantenlopen roxanne okj kaddafi btje zw zijt bermuda valkenswaard ffe pepe arjen weej hgv kgaa sana +sstnl dah libische dalik fain boodemorro +nieuwsuur helle borgerkifesh moss diem cairomorguh weekeinde ep +limburg nej voc+shazam +roermond d2 windstil achilles bee +f1 wagten anass +wateenjongewil +jinek zeau +pils +barneveld +mooileven gek

lopen zulke vies :

rennen meisjes jarig

te moeten

10

20

30

40

benen

stinkt

keel gym

raak

toch

heeft

geschreven buiten

maar

is

begint

<@>

wel bij

daar

fifa

0

Percentage of females using token

185

F gure 5 Percentages of use of tokens by fema e and ma e authors The font s ze of the words nd cates to wh ch degree they d fferent ate between the gender when a so tak ng nto account the re at ve frequenc es of occurrence

echt

0

10

20

30

40

Percentage of males using token

50

60

already observed activities. We do also see more expressions of self, with ik (“I”) and first person verbs such as wil (“want”) and heb (“have”), but these are much less distinguishing. On the male side, we see a rather different world. Apart from bier (“beer”), we see an enormous number of soccer-related words, with fifa at the most extreme, and then club names, scores, competitions, playing, winning, losing, etc. On the right edge of the plot, though, we do also observe some function words. The location adverbs daar (“there”) and waar (“where”) appear to be a more male thing, as well as some prepositions like per (“per”), bij (“at”, “near”) and voor (“for”, “before”). Finally, mentioning other users (<@>) is apparently more often done by men. For both genders, the tokens are dominated by the young person’s world. It is no wonder that classifying different types of authors, such as politicians and financial bloggers, is more problematic.

60

echt wil

ik heb word

50

allesme ’

(

even

; x

40

zo

moet ga bentwaaromiets met dit nu uit !

30

en

mag

aan

het

al dat

over

als he .

door

nou

mij

naar

deze

tot

)

t

...

tegen

jij

te geen we heeft die wat maarmee

nog hij

vanin

jullie

beetje

wie

mijn

ff

!!

k

, om

had hebben :

−

mn

&

hoe

ze

dus je

de

DQUO

istoch

<@> of ook

wel bij voor



10

20

??

er

?

opzou zijn een dan

0

Percentage of females using token

niet kan

jou ben

0

10

20

30

40

50

60

Percentage of males using token

Figure 6: Percentages of use of the 100 most frequent function words and punctuation by female and male authors. The font size of the words indicates to which degree they differentiate between the gender when also taking into account the relative frequencies of occurrence. Although most distinguishing tokens appear to be related to content, we do observe some stylerelated tokens. In Figure 6, we show a plot for the top 100 function words (or rather tokens), which was the only feature type focusing on style in our experiments. We can now observe various distinguishing tokens which were so far lost in the dense cloud of words. They correspond to what earlier research (see Section 2) has observed. Females show more personal pronouns, such as the already mentioned ik (“I”), but also me (“me”) and jou (“you”), as well as the reduced possessive pronoun mn (mijn; “my”). Males write more objective structures, with the mentioned prepositions, and also articles like de (“the”) and een (“a”). Here we find also find more third person constructions, with is (“is”), hij (“he”) and zijn (“his”, or plural “are”). Looking at the bigrams, which we will not plot here, we see a few more style-related constructions appearing. On the female side, we see niet meer (“not anymore”), ik mis (“I miss”) and the 186

intensifying adverb zo combined with various adjectives: zo moe (“so tired”), zo blij (“so happy”), zo zielig (“so pathetic”), and zo leuk (“so nice”). On the male side, there are also mostly combinations of already observed unigrams, but also the more pragmatic ending of tweets with the word man, in man ! (“,man!”), nee man (“no, man”), niet man (“not, man”), and goed man (“good, man”). All in all, there appear to be quite a few features related to style after all. Furthermore, the top 100 function words are doing quite well, with 84.8%, seeing how few features there are compared to the full set of unigrams. On the other hand, we cannot escape the impression that even these style features are more often related to what is being tweeted about, than to personal writing style.

6. Conclusion and Future Work We have investigated how well the gender of authors on Twitter can be determined on the basis of token or character n-grams. We find that recognition is possible with a high accuracy, up to 95.5% on our data set (but see dicussion below). Furthermore, some of the errors are probably related to the fact that the authors in question are different from the typical Twitter users dominating our data set. The best feature type for recognition appears to be the token unigrams, with the most distinguishing tokens linked to the typical activities of the dominant Twitter users. As for classification systems, Support Vector Regression clearly performs best with all feature types. During our investigation into gender recognition, we have also experimented with the use of Principal Component Analysis as a preprocessing step to classification. It was already known that this step was necessary for k-NN learning. We found that SVR is actually hampered rather than helped by the preprocessing. Its accuracy degrades when using PCA, although often not significantly. For Linguistic Profiling, PCA increases accuracy, in some cases enabling it to reach a score which is no longer significantly worse than that of SVR. TiMBL, even with PCA, does not reach the same accuracy level, and only accomplishes scores similar to SVR’s scores for token skip bigrams and unnormalized character trigrams. However, TiMBL’s lower quality is mostly a matter of hyperparameter selection. The number of principal components provided to the learners was determined automatically on the basis of development data. When we examined the systems’ accuracy for fixed numbers of principal components, TiMBL was often at the same accuracy level as SVR, and it was LP that was falling behind. It has remained unclear to which degree gender can be recognized on the basis of style features. Although the use of all unigrams for classification yields far better results than the use of the 100 most frequent function words, the latter are certainly not doing badly. Furthermore, our closer examination in Section 5.4 may imply that it is not the quality, but the number of features that is the reason for the difference in accuracy. We will revisit this question when we have larger n-gram sets available which can be assumed to be largely domain-independent.17 Finally, if we look back at our original goal, the automatic estimation of metadata for the TwiNL data set, we must conclude that we have made a significant step forward, but still only a modest one. Not only did we predict just one user trait, but we also considered just a very select class of users, namely individual users with a significant tweet volume. We will still need to test the minimum number of words on which the classifier can maintain its current high quality. Furthermore, we will need to build classifiers to distinguish between individual user accounts, shared user accounts, accounts controlled by boards of editors, and tweetbots. It may also be useful to distinguish between different uses of Twitter, such as professional communication and social chitchat, and build separate metadata estimators for these different uses. Even more importantly, we will need to look beyond very specific lexical features. If we base metadata on a limited number of such features, we will never be able to use the resulting data for studying language use or social behaviour. If we would try, we would fall victim to circular reasoning, such as observing that only men ever play soccer, 17. We are currently laying the basis for the construction of such sets in other work (van Halteren and Oostdijk Submitted).

187

since this is the information we put in with our metadata determination. Therefore, if we ever want to automatically add metadata, it will have to be with as many information sources as possible, preferably only using that metadata on which various sources agree.

References Bamman, David, Jacob Eisenstein, and Tyler Schnoebelen (2014), Gender identity and lexical variation in social media, Journal of Sociolinguistics. Beyer, Kevin, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft (1999), When is “Nearest Neighbor” meaningful?, In Int. Conf. on Database Theory, pp. 217–235. Burger, John D., John Henderson, George Kim, and Guido Zarrella (2011), Discriminating gender on Twitter, Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK, July 27–31, 2011, pp. 1301–1309. Chang, Chih-Chung and Chih-Jen Lin (2011), LIBSVM: A library for support vector machines, ACM Transactions on Intelligent Systems and Technology 2 (27), pp. 1–27. Cohen, Jacob (1988), Statistical Power Analysis for the Behavioral Sciences (second ed.), Delft: Now Publishers. Daelemans, Walter, Jakub Zavrel, Ko van der Sloot, and Antal Van den Bosch (2004), Timbl: Tilburg memory-based learner, Technical Report ILK-0209, Tilburg University. Fink, C., J. Kopecky, and M. Morawski (2012), Inferring gender from the content of tweets: A region specific example, Proceedings of the International AAAI Conference on Weblogs and Social Media, North America, May 2012. Goswami, Sumi, Sudeshna Sarkar, and Mayur Rustagi (2009), Stylometric analysis of bloggers’ age and gender, Proceedings ICWSM 2009. Heil, B. and M. J. Piskorski (2009), New Twitter research: Men follow men and nobody tweets, Harvard Business Review. Hotelling, H. (1933), Analysis of a complex of statistical variables into principal components, Journal of Educational Psychology 24, pp. 417–441 and 498–520. Juola, Patrick (2008), Authorship Attribution, Lawrence Erlbaum Associates. Kestemont, M. (2014), Function words in authorship attribution. from black magic to theory?, Proceedings of the Third Computational Linguistics for Literature Workshop, co-located with EACL 2014 – the 14th Conference of the European Chapter of the Association for Computational Linguistics (27 April 2014, Gothenburg, Sweden), pp. 59–66. Kjell, Bradley, W. Addison Woods, and Ophir Frieder (1994), Discrimination of authorship using visualization, Inf. Process. Manage. 30 (1), pp. 141–150. Koppel, Moshe, Jonathan Schler, and Shlomo Argamon (2009), Computational methods in authorship attribution, J. Am. Soc. Inf. Sci. Technol. 60 (1), pp. 9–26. Koppel, Moshe, Schlomo Argamon, and Anata Rachel Shimony (2002), Automatically categorizing written texts by author gender, Literary and Linguistic Computing 17 (4), pp. 401–412. Narayanan, Arvind, Hristo Paskov, Neil Zhenqiang Gong, John Bethencourt, Eui Chul, Richard Shin, and Dawn Song (2012), On the feasibility of internet-scale author identification, Proceedings of the 33rd conference on IEEE Symposium on Security and Privacy. IEEE. 188

Nguyen, D., D. Trieschnigg, A. S. Do˘ gru¨ oz, R. Gravel, M. Theune, T. Meder, and F.M.G. de Jong (2014), Why gender and age prediction from tweets is hard: Lessons from a crowdsourcing experiment, Proceedings of COLING 2014. Nguyen, Dong, Rilana Gravel, Dolf Trieschnigg, and Theo Meder (2013), “How old do you think I am?”: A study of language and age in twitter, Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media (ICWSM 2013). Pearson, K. (1901), On lines and planes of closest fit to systems of points in space, Philosophical Magazine 2 (11), pp. 559–572. Pennebaker, J.W., C.K. Chunk, M. Ireland, A. Gonzales, and R.J. Bootk (2007), The development and psychometric properties of LIWC2007, Software Manual. R Development Core Team (2008), R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing. http://www.R-project.org. Rao, D., D. Yarowsky, A. Shreevats, and M. Gupta (2010), Classifying latent user attributes in Twitter, Proceedings of the 2nd international workshop on Search and mining user-generated contents, pp. 37–44. Schler, J., M. Koppel, S. Argamon, and J. Pennebaker (2006), Effects of age and gender on blogging, Proceedings of 2006 AAAI Spring Symposium on Computational Approaches for Analyzing Weblogs. Tjong Kim Sang, Erik and Antal van den Bosch (2013), Dealing with big data: the case of Twitter, Computational Linguistics in the Netherlands Journal 3, pp. 121–134. Van Bael, Christophe and Hans van Halteren (2007), Speaker classification by means of orthographic and broad phonetic transcriptions of speech, Speaker Classification (2), pp. 293–307. van Halteren, Hans (2004), Linguistic Profiling for authorship recognition and verification, Proceedings ACL 2004, pp. 199–206. van Halteren, Hans (2008), Source language markers in Europarl translations, Proceedings of COLING2008, 22nd International Conference on Computational Linguistics, pp. 937–944. van Halteren, Hans and Nelleke Oostdijk (Submitted), Word distribution in Dutch Tweets, Nederlandse Taal en Letterkunde.

189