Indexing OAI-PMH non-compliant repositories

[wetneb]

So, BASE (http://www.base-search.net) indexes many repositories that have an OAI-PMH interface, which is already very useful and covers most of the respectable repositories.

But unfortunately, many repositories do not provide such an interface. And even when they provide one, the protocol is far from perfect: for instance, it is hard to detect accurately whether a full text is freely available behind a metadata record. Reaching out to the repository admins to encourage them to expose their metadata correctly is, from my experience, not effective at all.

If we want to go beyond this, I think we need to crawl!

• One (conceptually) simple option would be to crawl for PDF files, extract metadata from them and dump this in a proper database with indexes. I think this is what CiteSeerX does, but only for papers within a particular field. In my experience the metadata that comes out of this is quite noisy.
• The other option we have discussed would be to leverage existing scrapers (Zotero) to extract cleaner metadata from HTML pages. Zotero does the scraping pretty well, but I have no clue what crawling software we should use. Any idea? The scrapy framework looks nice but I'm a complete newcommer in this field so I have probably missed better options.

How much resources (servers) do we need for this? Where could we get them?

[mekarpeles]

@wetneb thanks for this excellent summary. We can apply some pressure and communicate the importance of this initiative to the Internet Archive, they may be able to provide us with resources (disk/storage space). I can talk to Brewster this Friday about that possibility (a researcher VM with ~5tb of space, for starters). At very least, hopefully it will help us develop crawlers, write tutorials for the community to follow, and store documents temporarily (we can also use the openjournal user account to upload files directly to the Internet Archive, and they have an S3 style API for bulk upload. Internet Archive also does OCR on papers/pdfs that are uploaded, which can be a big mutual win. Re: noisy data, I am not sure how accurate their OCR is for academic (especially math-heavy) works. Perhaps worth exploring as an experiment.

Regarding crawlers, perhaps we can create a document an awesome-list of all crawling initiatives for academic papers, journals, databases, and metadata. I have ideas on how we can leverage libraries like scrapy -- as you elude to, I'm sure there existing tools (and I imagine creating n source-specific crawlers and reverse-engineering their indexing scheme, and having/maintaining a repository of these crawlers of each source the public to use, will be more success / more comprehensive than a deep crawl)

@pietsch, this is a fairly comprehensive list of publicly accessible sources that use OAI-PMH, right?

[cleegiles]

Please see comments below.

It is important not to reinvent wheels. Some of what we discuss
falls into that domain.

On 11/30/15 5:45 PM, Antonin wrote:

So, BASE (http://www.base-search.net) indexes many repositories that have an OAI-PMH interface, which is already very useful and covers most of the respectable repositories.

But unfortunately, many repositories do not provide such an interface. And even when they provide one, the protocol is far from perfect: for instance, it is hard to detect accurately whether a full text is freely available behind a metadata record. Reaching out to the repository admins to encourage them to expose their metadata correctly is, from my experience, not effective at all.

If we want to go beyond this, I think we need to crawl!

• One (conceptually) simple option would be to crawl for PDF files, extract metadata from them and dump this in a proper database with indexes. I think this is what CiteSeerX does, but only for papers within a particular field. In my experience the metadata that comes out of this is quite noisy.
  No longer, CiteSeerX now crawls for all scholarly documents.
  Please note that some repositories prevent crawling with their
  robots.txt, except for Googlebot.

Yes, it is noisy but not that bad and is always getting better. Others
are doing this and
contributing to the extraction algorithms. Please see our previous list
of extraction methods.
We may have a tutorial at major conference soon.

• The other option we have discussed would be to leverage existing scrapers (Zotero) to extract cleaner metadata from HTML pages. Zotero does the scraping pretty well, but I have no clue what crawling software we should use. Any idea? The scrapy framework looks nice but I'm a complete newcommer in this field so I have probably missed better options.
  Our crawling code is available on the CiteSeerX GitHub. We are always
  crawling and recently with
  Semantic Scholar at AI2. We use Heritrix, an excellent tool.
  How much resources (servers) do we need for this? Where could we get them?
  This is storage and bandwidth intensive. The problems are:

• crawling is very time consuming and needs a great deal of coordinated
  parallel threads.
• crawling brings back unwanted PDFs which must be filtered or
  classified. We have
  a few papers on this if anyone is interested.
• a scholarly paper PDF is about one Meg - a million a Tera. However,
  the others have
  to are usually stored.

We've found a factor of 3 for all documents we expect to crawl. That is
we crawl 3 times
as many PDFs for the same number of PDF files.

We share our crawl seeds if anyone wants to use them.

Who here crawls?

---

Reply to this email directly or view it on GitHub:
#8

[cleegiles]

These papers may be of help. One can publish improvements. We would very
much like to improve ours. Suggestions most welcomed.

Best

Lee

On 11/30/15 7:04 PM, Michael E. Karpeles wrote:

@wetneb thanks for this excellent summary. We can apply some pressure and communicate the importance of this initiative to the Internet Archive, they may be able to provide us with resources (disk/storage space). I can talk to Brewster this Friday about that possibility (a researcher VM with ~5tb of space, for starters). At very least, hopefully it will help us develop crawlers, write tutorials for the community to follow, and store documents temporarily (we can also use the openjournal user account to upload files directly to the Internet Archive, and they have an S3 style API for bulk upload. Internet Archive also does OCR on papers/pdfs that are uploaded, which can be a big mutual win. Re: noisy data, I am not sure how accurate their OCR is for academic (especially math-heavy) works. Perhaps worth exploring as an experiment.

Regarding crawlers, perhaps we can create a document an awesome-list of all crawling initiatives for academic papers, journals, databases, and metadata. I have ideas on how we can leverage libraries like scrapy -- as you elude to, I'm sure there existing tools (and I imagine creating n source-specific crawlers and reverse-engineering their indexing scheme, and having/maintaining a repository of these crawlers of each source the public to use, will be more success / more comprehensive than a deep crawl)

@pietsch, this is a fairly comprehensive list of publicly accessible sources that use OAI-PMH, right?

---

Reply to this email directly or view it on GitHub:
#8 (comment)

[mekarpeles]

@cleegiles Thanks for the excellent contribution. Also, for the record, I view OpenJournal as a working group for reducing (exactly as you say) re-implemented wheels. So I'm on board! I don't anticipate building anything directly through OpenJournal -- if I contribute to something, it will be an existing project (unless there's a very compelling reason why something new needs to be built, and if that's the case, I'd still prefer someone else leads it, i.e. limited bandwidth).

There are several folks affiliated w/ the Internet Archive I know who are working on crawling efforts. Some aren't comfortable announcing. @nthmost can likely share her ideas for pubmed crawling. I have experience crawling, am happy to participate, but do not currently work on a crawler.

@cleegiles I'll try to generate some traffic to this thread by pinging other institutions and seeing if they can weigh in on the status of their crawlers -- thanks for nudging us in that direction.

[wetneb]

@cleegiles Thanks for the clarification, it's great that CiteSeerX is not limited to comp. sci. anymore!
Do not get me wrong, what CiteSeerX does is massively useful. But at http://dissem.in we need to match publications with preprints, which is quite hard as soon as the title or the authors differ by a few words. So, metadata quality is critical for us.

I can imagine that CiteSeerX requires a lot of resources indeed. We cannot afford this for dissemin, and this is the reason why I thought using scrapers would be cheaper (both in terms of bandwidth, storage, and computing) and could potentially yield cleaner metadata. But this option only works for repositories, not home pages.

I have been playing around on indexing researchgate.net recently (with crawling and scraping), and I have been in touch with Mike Taylor who has started something along these lines for SSRN.

@mekarpeles Having a researcher VM would be amazing! And I'm really looking forward to hearing more crawling stories, especially from the Internet Archive!

[davidar]

And even when they provide one, the protocol is far from perfect: for instance, it is hard to detect accurately whether a full text is freely available behind a metadata record.

Yes, this is an issue we're trying to deal with in ipfs-inactive/archives#3 (enriching OAI-PMH metadata with fulltext links).

@cleegiles Is this something that CiteSeerX could help with?

[wetneb]

@davidar that's awesome! I hope you will succeed.

[davidar]

@wetneb well, my plan was basically the same as what you outlined in the OP, so I'm afraid I might be reinventing the wheel? Perhaps this is something we could collaborate on?

[wetneb]

@davidar I would love to! Joining the discussion there then.

[pietsch]

Hi @mekarpeles,

@pietsch, this is a fairly comprehensive list of publicly accessible sources that use OAI-PMH, right?

Yes, these are the 3881 OAI-PMH sources BASE is currently harvesting (in intervals). The other lists I am aware of are the Directory of Open Access Journals (DOAJ) and the official (if incomplete and out-of-date) list of OAI-PMH repositories.

[cleegiles]

We do this with our crawler. If the link goes directly to a non-open
source publisher, there is no reason to
crawl. We have a blacklist and whitelist of where we go now which we can
share. It's fairly complete.

Another way is to parse the link to the document. Links that are not
directly to pdfs are usually not
downloadable. It would be useful to do a sample to see how often this is
true, but we've found it
to nearly always be the case. A counter example is where one has to sign
in but anyone can have
an account.

On 12/1/15 3:03 AM, David A Roberts wrote:

And even when they provide one, the protocol is far from perfect: for instance, it is hard to detect accurately whether a full text is freely available behind a metadata record.
Yes, this is an issue we're trying to deal with in ipfs-inactive/archives#3 (enriching OAI-PMH metadata with fulltext links).

@cleegiles Is this something that CiteSeerX could help with?

---

Reply to this email directly or view it on GitHub:
#8 (comment)

[cleegiles]

The best metadata seems to be the Web of Science (WoS) but it has to be
purchased and is not cheap.

This seems to be a good source of metadata, but we have not compared it
to ours

http://research.microsoft.com/en-us/projects/mag/
We currently have a project to clean up our and other metadata.

On 12/1/15 2:27 AM, Antonin wrote:

@cleegiles Thanks for the clarification, it's great that CiteSeerX is not limited to comp. sci. anymore!
Do not get me wrong, the metadata CiteSeerX extracts is very useful. But at http://dissem.in we need to match publications with preprints, which is quite hard as soon as the title or the authors differ by a few words. So, metadata quality is critical for us.

I can imagine that CiteSeerX requires a lot of resources indeed. We cannot afford this for dissemin, and this is the reason why I thought using scrapers would be cheaper (both in terms of bandwidth, storage, and computing) and could potentially yield cleaner metadata. But this option only works for repositories, not home pages.

I have been playing around on indexing researchgate.net recently (with crawling and scraping), and I have been in touch with Mike Taylor who has started something along these lines for SSRN.

---

Reply to this email directly or view it on GitHub:
#8 (comment)