Lisan Al-Arab: Most sources provide no meaningful structure, except the original book itself

[antaz]

While there are many digitized sources for the book itself, most of them provide no way to distinguish words from definitions which makes parsing extremely challenging

https://shamela.ws/book/1687/23

The original book makes the structure very clear however:

[larrasket]

I don't understand, I can see that both of them are pretty much the same. Let's take أَبَأَ for example, what is the difference between what is in shamela.ws and the screenshot you provided?

[antaz]

@salehmu The content itself is not different, the structure however of the document, the document in shamela colors the words differently from the definitions, but it also colors other things too?

[antaz]

nevermind! I think i found a better way to do this

[larrasket]

@BLOCR have you reached to anything?

[antaz]

@salehmu not much progress to be honest, the sources are either incomplete or hard to scrape

[larrasket]

@BLOCR can you explain how the wiki source is incomplete? It seemed complete to me

[antaz]

@salehmu I just assumed it is since it mentions here that it's not
https://ar.wikisource.org/wiki/%D9%84%D8%B3%D8%A7%D9%86_%D8%A7%D9%84%D8%B9%D8%B1%D8%A8

and in the discussion page too
https://ar.wikisource.org/wiki/%D9%86%D9%82%D8%A7%D8%B4:%D9%84%D8%B3%D8%A7%D9%86_%D8%A7%D9%84%D8%B9%D8%B1%D8%A8

otherwise i'm not sure.

[larrasket]

I see, what about IslamPort? @BLOCR

[antaz]

@salehmu let me check that and report back

[larrasket]

@BLOCR Okay, I started any way writing a utility for scrapping from it, 90% of islamport copys are just the same as shamelas', so it's pretty solid OCR and since the website is so small scrapping won't be a big issue. Also I'm going to consider deserializing the bok extension, I've looked many projects that did that. https://github.com/ojuba-org/thawab-lite

Have a look that thawab-lite, since it's written in python maybe you would have more understanding than me of the way it works, if you are able to reuse it to generate RTF it would be great.

[antaz]

@salehmu alright, but i'm not sure why we should go for such a proprietary format like RTF.
What i'm doing now is collecting all the pages from http://islamport.com/w/lqh/Web/1156/2.htm into a json object, assuming this book is complete and we can separate the words from definitions, we should have a good framework to work with

[larrasket]

@BLOCR i was looking at islamport now http://islamport.com/w/lqh/Web/1156/8.htm the format is pretty bad, I think I will consider using bok or scrapping from shamela instead

[antaz]

@salehmu The problem i found with shamela's version though is that it doesn't have any structure, like how can you seperate the word from its definition?

[larrasket]

@BLOCR Well, I think we are able to at least serialize JSON objects of chapters, i.e. scrapping the data first, after that we will see how we can make use of it (separate the word from its definition). We should consider preserving the text highlighting from shamela and replacing it with more intelligent one to be used as marks.

[antaz]

@salehmu that's a good idea, but i found the highlighting more or less semantically meaningless, they look like some weird OCR artifacts but i don't know