Identifying Nonprofits by Scaling Mission and Activity with Word Embedding

Abstract

This study develops a new text-as-data method for organization identification, based on word embedding. We introduce and apply the method to identify identity-based nonprofit organizations, using the U.S. nonprofits’ mission and activity information reported in the IRS Form 990s in 2010–2016. Our results show that such method is simple but versatile. It complements the existing dictionary-based approaches and supervised machine learning methods for classification purposes and generates a reliable continuous measure of document-to-keyword relevance. Our approach provides a nonbinary alternative for nonprofit big data analyses. Using word embedding, researchers are able to identify organizations of interest, track possible changes over time and capture nonprofits’ multi-dimensionality.

Appendices

Appendix A: Learning Distributed Semantics with word2vec

Word2vec has two variants: the Skip-gram model and the continuous bag-of-words (CBOW) model. The difference between the two lies in the specific task of prediction performed. The Skip-gram model predicts context words with the target word; the CBOW model predicts the target word with context words. Thus, the definition of likelihood functions slightly differs, leading to different optimization tasks.

To formalize, we start with the following setup: Consider a sequence of text of T words in total. Let the size of context windows be c (i.e., c’s immediate neighbors before and after a word are considered its context). Let ${\mathbf {v}}(w)$ be the vector of distributed semantics of word w. Let the size of the vocabulary be V, (i.e., there are V unique words in the text). Let $p(w_i | w_j)$ be the probability of word $w_i$ appearing, given word $w_j$. Let $\mathcal {L}$ be the likelihood.

The word2vec uses the a softmax function to link distributed representation of words (word vectors) with their predicted probabilities. Specifically, the probability of word $w_i$ given word $w_j$ in its context window is the exponential of the dot products of the word vectors ${\mathbf {v}}_{w_i}, \tilde{{\mathbf {v}}}_{w_j}$ over the sum of the exponentials of the dot products of $\tilde{{\mathbf {v}}}_{w_j}$ with word vectors of all words in the vocabulary:

$$\begin{aligned} \log p(w_i | w_j)&= \log \frac{\exp ({\mathbf {v}}_{w_i}^T \tilde{{\mathbf {v}}}_{w_j})}{\sum ^{V}_{k=1} \exp ({\mathbf {v}}_{w_k}^T \tilde{{\mathbf {v}}}_{w_j})} \end{aligned}$$

(5)

$$\begin{aligned}&= {\mathbf {v}}_{w_i}^T \tilde{{\mathbf {v}}}_{w_j} - \log \sum ^{V}_{k=1} \exp ({\mathbf {v}}_{w_k}^T \tilde{{\mathbf {v}}}_{w_j}) \end{aligned}$$

(6)

Thus, the log-likelihood of the Skip-gram model is computed as follows. At location t of the text sequence, the joint conditional probability of words in the context window (conditional on the target word at t) is calculated. The conditional probabilities are obtained by applications of softmax on the target word vector against each context word vector. Then the algorithm moves to location $t+1$ and repeat the process until the end of the sequence. The log-likelihood is the sum of all log probabilities. Formally:

$$\begin{aligned} \log \mathcal {L}&= \sum ^{T}_{t=1} \sum _{-c \le j \le c, j \ne 0} \log p(w_{t+j} | w_t) \end{aligned}$$

(7)

$$\begin{aligned}&= \sum ^{T}_{t=1} \sum _{-c \le j \le c, j \ne 0} \log \frac{\exp (\mathbf{v }_{w_{t+j}}^T \tilde{\mathbf{v }}_{w_{t}})}{\sum ^{V}_{k=1} \exp (\mathbf{v }_{w_k}^T \tilde{\mathbf{v }}_{w_{t}})} \end{aligned}$$

(8)

$$\begin{aligned}&= \sum ^{T}_{t=1} \left[ \sum _{-c \le j \le c, j \ne 0} \mathbf{v }_{w_{t+j}}^T \tilde{\mathbf{v }}_{w_t} - \log \sum _{k=1}^{V} \exp ({\mathbf {v}}_{w_k}^T \tilde{{\mathbf {v}}}_{w_t}) \right] \end{aligned}$$

(9)

Similarly, the log-likelihood of the CBOW model is computed as follows: at location t of the text sequence, the probability of target word given context words is calculated. The conditional probability is obtained by a softmax of the target word vector and the average of context word vectors. Then the algorithm moves to location $t+1$ and repeats the process until the end of the sequence. The log-likelihood is the sum of all log probabilities. Formally:

$$\begin{aligned} \log \mathcal {L}&= \sum ^{T}_{t=1} \log p(w_t | w_{t-c}, w_{t-c+1}, ... w_{t+c-1}, w_{t+c}) \end{aligned}$$

(10)

$$\begin{aligned}&= \sum ^{T}_{t=1} \log \frac{\exp ({\mathbf {v}}_{w_t}^T \bar{{\mathbf {v}}}_t)}{\sum ^{V}_{k=1} \exp ({\mathbf {v}}_{w_k}^T \bar{{\mathbf {v}}}_t)} \end{aligned}$$

(11)

$$\begin{aligned}&= \sum ^{T}_{t=1} \left[ {\mathbf {v}}_{w_t}^T \bar{{\mathbf {v}}}_t - \log \sum ^{V}_{k=1} \exp ({\mathbf {v}}_{w_k}^T \bar{{\mathbf {v}}}_t) \right] \end{aligned}$$

(12)

$$\begin{aligned}&\text {where } {\bar{v}}_t = \frac{1}{2c} \sum _{-c \le j \le c, j \ne 0} \tilde{{\mathbf {v}}}_{w_{t+j}} \end{aligned}$$

(13)

Both Skip-gram and CBOW models train vector representations of words to maximize the above defined likelihood. The processing is operationalized as neural networks trained by stochastic gradient descent. In general, they are both neural networks with one hidden layer and two weight matrices. The first weight matrices ${\mathbf {W}}_{V \times N}$ contain vector representations of all V words as targets in the vocabulary: ${\mathbf {W}}_{V \times N} = [{\mathbf {v}}_{w_1}, {\mathbf {v}}_{w_2}, ..., {\mathbf {v}}_{w_N}]^T$. The second weight matrices $\tilde{{\mathbf {W}}}_{N \times V}$ contain vectors of words as context: $\tilde{{\mathbf {W}}}_{N \times V} = [\tilde{{\mathbf {v}}}_{w_1}, \tilde{{\mathbf {v}}}_{w_2}, ..., \tilde{{\mathbf {v}}}_{w_N}]$. Input and output layers are one-hot-encoded words. The differences between Skip-gram and CBOW are evident in the model architectures. Skip-gram (Panel a) uses target words to predict context words, while CBOW (Panel b) uses context words to predict target words. Word vectors are updated with stochastic gradient descent. For the final output, researchers can use either of the two weight matrices ${\mathbf {W}}_{V \times N}, \tilde{{\mathbf {W}}}_{N \times V}^T$ or the two matrices’ average as the representation of distributed semantics.

Training word2vec models can be computationally taxing. Two methods are used to reduce the computational demands of the model: hierarchical softmax and negative sampling. The algorithm in its naïve version described above can be computationally taxing primarily because the complexity of the softmax step (Eq. 5) grows linearly with the vocabulary size (i.e., O(V) complexity): in the forward pass, it takes summations over the whole vocabulary of size V for the denominator; in the backpropagation, it updates all V word vectors in the vocabulary. Two methods have been developed to boost efficiency. First, hierarchical softmax uses a binary tree where words are represented by their leaf units. The probability of a word being the output is estimated by the probability of the path from root to leaf of the word. The method reduces computational complexity from O(V) to $O(\log _2 V)$ given its tree structure. A second and more intuitive method, negative sampling, takes a random sample of words from the vocabulary to approximate the denominator in the forward pass and to update only the sample in the backpropagation. Thus, the computational complexity depends on the size of the negative sample and does not grow with the vocabulary size. The two methods have both demonstrated good performance in existing applications.

Appendix B: Word Cloud of Other Query Terms

Appendix C: Additional Histograms of Cosine Similarity