The table below allows looking at word-use ranking via three different criteria. The first is simple word-count, how many times each word occurred in the total tweet-set, including multiple occurrences of a word within a single tweet. The remaining columns: \(tf\), \(df\), \(p\), \(idf\), \(tf\_idf\), and \( -p \ln (p) \) require more explanation.

\(tf\), for ‘term frequency’ starts out as simply the count of each word, but is then divided by by the square-root of the sum of the squares of the counts of all of the words. This effectively makes \(tf\) into a direction cosine in the N-dimensional vector-space of the conference vocabulary, giving the extent to which the vector of the entire vocabulary points in the direction of each word.

\(df\), for ‘document frequency’ is simply the number of tweets that each word occurred in. Divided by the number of tweets, it becomes \(p\), the sample probability that a word would occur in a tweet at a similar conference or the same conference in parallel universes.

The last needed factor is: \(- ln (p) = ln(1/p) = ln(N/df)\), one of several variants of the inverse document frequency (idf) found in the literature. The purpose of the idf is to reduce the relative weighting of words that are ubiquitous to a set of documents (tweets). Being predictable lowers their information content.

Finally, the last two ranking criteria, \(tf\_idf\), and \( -p \ln (p) \) are obtained by multiplying \(tf\) and \(p\) by idf, respectively, the latter being proportional to Shannon's information content.

You can reorder the ranking of the table below by clicking on the column heads for count, \(tf\_idf\), or \( -p \ln (p) \).

---