tethne-services

Tools to enhance metadata-based analysis in Tethne.

Clone this Repo

Clone this repository into whatever directory you'd like to work on it from:

git clone https://github.com/diging/tethne-services.git

Install the following

Python v2.7.12
Tethne
- pip install tethne
pandas v0.19.0
- pip install pandas
scikit-learn v0.18.1
- pip install -U scikit-learn
numpy v1.11.3
- pip install numpy
fuzzywuzzy v0.14.0
- pip install fuzzywuzzy

APIs

In addition to the below examples, an example showing basic classification of 2 author-paper instances can be found here

This package will expose the following APIs:

CorpusParser.py: This module is responsible for parsing a Tethne corpus object and returning a pandas DataFrame with 14 columns. Each row in the DataFrame is an Author-Paper instance. Also, please note that each row in the DataFrame is assigned an index. The index is generated using (concatenation of)the following: 1. Author Last Name
2. Author First Name
3. WOS ID

An example usage of this API is shown below.

from authors.paperinstances import CorpusParser
from tethne.readers import wos
datapath = "/Users/aosingh/tethne-services/tests/data/Boyer_Barbara.txt"
corpus = wos.read(datapath)
corpus_parser = CorpusParser(tethne_corpus=corpus)
df = corpus_parser.parse() # final pandas DataFrame of Author-Paper instances.

#This is how the indices look like for each row
u'BOYERBCWOS:000077556600009',
u'BOYERBCWOS:000086633600013',
u'BOYERBCWOS:000171953200027',
u'BOYERBCWOS:000186338800013',
u'BOYERBCWOS:A1971I198200009',
u'BOYERBCWOS:A1972L780300002',
u'BOYERBCWOS:A1981MP70900043',
u'BOYERBCWOS:A1982QN98300013',
u'BOYERBCWOS:A1983RR86600053',
u'BOYERBCWOS:A1984TR20700060',

#The DataFrame returned has the following colmns
["WOSID", "DATE", "TITLE", "LASTNAME", "FIRSTNAME", "JOURNAL", "EMAILADDRESS",
"PUBLISHER", "SUBJECT", "WC", "AUTHOR_KEYWORDS", "INSTITUTE", "AUTH_LITERAL", "CO-AUTHORS"]

InitialCluster.py: InitialCluster, as the name suggests, groups Author-Paper instances by similar author names. In this process, we do not use any classification or machine learning approach. Initial Clustering is done to limit the size of comparisons when we perform the actual classification. For example: It is not efficient to compare papers by the authors 'BRUCE WAYNE' and 'CLARK KENT' using the classification model. We know they are 2 different people. While we build this initial cluster, we group together author_literals which are similar and have higher probability of actually belonging to the same cluster.

An example usage of this API is shown below

from authors.Cluster import InitialCluster
from tethne.readers import wos
datapath = './data/Albertini_David.txt'
corpus = wos.read(datapath)
initial_cluster_instance = InitialCluster(corpus=corpus)
clusters = initial_cluster_instance.build()

#This is how the dictionary of initial clusters will look like:

 {u'AALBERGJ': {u'AALBERGJ', u'VALBERGPA'},
  u'ABOULGHARM': {u'ABOULGHARM'},
  u'ABSEDANNIE': {u'ABSEDANNIE'},
  u'AHUJAKAMAL': {u'AHUJAKAMAL'},
  u'AINSLIEA': {u'AINSLIEA']},
  u'AKKOYUNLUGOKHAN': {u'AKKOYUNLUGOKHAN'},
  u'ALBERTINDF': {u'ALBERTINDF',
             u'ALBERTINID',
             u'ALBERTINID F',
             u'ALBERTINIDAVID',
             u'ALBERTINIDAVID F',
             u'ALBERTINIDF'},
   u'ALECCIC':{u'ALECCIC'},
   u'ALEXANDREHENRI': {u'ALEXANDREHENRI'},
   u'ALIKANIMINA': {u'ALIKANIMINA', u'GALIANIDALIA'},
   u'ALLWORTHAE': {u'ALLWORTHAE'},
   u'ANDERSENCY': {u'ANDERSENCY', u'ANDERSONE', u'ANDERSONR'}}

IdentityCluster.py: IdentityCluster class uses Machine Learning(RandomForestClassifier) to cluster paper instances belonging to the same author. We first build an initial cluster using the class InitialCluster. Please read the example below to understand it's usage

An example usage of this API is shown below

"""
Algorithm: 

STEP 1 : Parse the TETHNE corpus and return a pandas DataFrame of Author-paper instances. 
         Please note that, an index is assigned to each Author-Paper instance.
         The index is generated using (concatenation of)the following:
            1. WOS ID
            2. Author Last Name
            3. Author First Name

STEP 2 : Use the `IdentityCluster` class to group instances belonging to the same class(Basically, 
         build a dictionary).   
      
STEP 3 : Return the dictionary with LABEL as keys and a set of pandas DataFrame indexes as values. 
These indices are the same which are created in the STEP 1 of the algorithm
"""

 from authors.cluster import IdentityCluster
 from tethne.readers import wos
 from authors.paperinstances import CorpusParser
 datapath = "/Users/aosingh/tethne-services/tests/data/Boyer_Barbara.txt"
 corpus = wos.read(datapath)

 corpus_parser = CorpusParser(tethne_corpus=corpus)
 df = corpus_parser.parse() # STEP 1 in the algorithm

 identity_cluster_instance = IdentityCluster(corpus=corpus)
 identity_clusters = identity_cluster_instance.build() # STEPS 2 and 3 in the algorithm

#This is how the dictionary of final identity clusters will look like:
{u'ARNOLDJM': set([u'ARNOLDJMWOS:A1986E918400022',
                       u'ARNOLDJMWOS:A1988N184500004']),
     u'BOYERB': set([u'BOYERBCWOS:000076265300004',
                     u'BOYERBCWOS:000077556600009',
                     u'BOYERBCWOS:000086633600013',
                     u'BOYERBCWOS:000171953200027',
                     u'BOYERBCWOS:000186338800013',
                     u'BOYERBCWOS:A1971I198200009',
                     u'BOYERBCWOS:A1972L780300002',
                     u'BOYERBCWOS:A1981MP70900043',
                     u'BOYERBCWOS:A1982QN98300013',
                     u'BOYERBCWOS:A1983RR86600053',
                     u'BOYERBCWOS:A1984TR20700060',
                     u'BOYERBCWOS:A1984TR20700065',
                     u'BOYERBCWOS:A1985AUC5500038',
                     u'BOYERBCWOS:A1985AUC5500046',
                     u'BOYERBCWOS:A1986A349100017',
                     u'BOYERBCWOS:A1986C019700010',
                     u'BOYERBCWOS:A1986E918400022',
                     u'BOYERBCWOS:A1986E918400023',
                     u'BOYERBCWOS:A1987G340700002',
                     u'BOYERBCWOS:A1988N184500004',
                     u'BOYERBCWOS:A1988R225500053',
                     u'BOYERBCWOS:A1989CH57500002',
                     u'BOYERBCWOS:A1991GV28500052',
                     u'BOYERBCWOS:A1992KC97700038',
                     u'BOYERBCWOS:A1992KC97700042',
                     u'BOYERBCWOS:A1995RP17800035',
                     u'BOYERBCWOS:A1995TA77100017',
                     u'BOYERBCWOS:A1996VQ71700035',
                     u'BOYERBCWOS:A1996VT14600003',
                     u'BOYERBWOS:A1996UQ10700011']),
     u'HENRYJJ': set([u'HENRYJJWOS:000077556600009',
                      u'HENRYJQWOS:000076265300004',
                      u'HENRYJQWOS:000086633600013',
                      u'HENRYJQWOS:A1995TA77100017',
                      u'HENRYJQWOS:A1996VQ71700035',
                      u'HENRYJQWOS:A1996VT14600003']),
     u'HILLSD': set([u'HILLSDWOS:000171953200027', u'HILLSDWOS:000186338800013']),
     u'KAPLANIM': set([u'KAPLANIMWOS:A1988R225500053',
                       u'KAPLANIMWOS:A1992KC97700042']),
     u'LADURNERP': set([u'LADURNERPWOS:A1996UQ10700011']),
     u'LANDOLFAM': set([u'LANDOLFAMAWOS:A1985AUC5500046',
                        u'LANDOLFAMAWOS:A1986E918400022',
                        u'LANDOLFAMWOS:A1988N184500004']),
     u'MAIRG': set([u'MAIRGWOS:A1996UQ10700011']),
     u'MARTINDALEMQ': set([u'MARTINDALEMQWOS:000077556600009',
                           u'MARTINDALEMQWOS:000086633600013',
                           u'MARTINDALEMQWOS:A1995TA77100017',
                           u'MARTINDALEMQWOS:A1996VQ71700035',
                           u'MARTINDALEMQWOS:A1996VT14600003']),
     u'PALASZEWSKIPP': set([u'PALASZEWSKIPPWOS:A1983RR86600053']),
     u'REITERD': set([u'REITERDWOS:A1996UQ10700011']),
     u'RIEGERR': set([u'RIEGERRWOS:A1996UQ10700011']),
     u'ROONEYLM': set([u'ROONEYLMWOS:A1984TR20700065']),
     u'SALVENMOSERW': set([u'SALVENMOSERWWOS:A1996UQ10700011']),
     u'SANTOSKA': set([u'SANTOSKAWOS:A1988R225500053']),
     u'SMITHGW': set([u'SMITHGWWOS:A1982QN98300013'])}

diging/tethne-services

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Uh oh!

Languages

Packages