Some thoughts and suggestions

[talolard]

Hey!
Thanks for putting these together. I'm the founder of LightTag. I have some suggestions which I'll try to keep as unbiased as possible.

I think that the repo content currently focuses on DIY annotations, e.g. how to put together your own ad-hoc tool. There are many good tools out there, including a few that are open source. I imagine that the majority of readers don't want to roll there own rather are looking for a way to quickly solve there problem.

The division into binary, multi class and span annotations is very good. These are a progression in terms of complexity and are to a degree distinct use cases. Perhaps consider adding an introductory section that describes what each of these are and how they relate, possibly with pictures gifs (we're happy to provide some)

Aside from actually annotation, the typical user has non functional requirements. Notably, how do I bring a team, can I and how to use active learning, how do I evaluate the quality of annotations and what is a good output format for my use case. Another common consideration is the tokenization scheme used by the annotation interface, e.g. can I annotate subwords and phrases or am I pre-commited to tokens as defined by some external system ? All of these are worth discussing.

Some users will be interested in outsourcing their annotations to mechanical turks or similar services. It's worth discussing the tradeoffs on this (quality/price tradeoff, expertise and language limitations). MTurks has external questions which let the user provide their own annotation interface which is often very handy.

It's worth talking about why you annotate data. People do this for different reasons. The obvious one is to collect labeled data to train machine learning models on. Often, companies will realize in hindsight that they also need to collect a test set with different procedures (test for inter annotator agreement, no active learning to test for bias).

Sometimes, often in fact, people annotate for non ML reasons. Linguists do this all the time, annotate treebanks and such to study linguistic phenomena. This is also common in "computational social sciences", where the point of annotation is to learn something about society, not train a model. It would be useful for the repo to talk about these distinctions, if only to help potential readers articulate what they are trying to do.

Hope this helps, I'm happy to submit some PRs here, but am hesitant because it might bias the repo towards LightTag. We'd love that, but it would undermine what you are trying to do here.

[sudodoki]

I thought I could do everything in one go, but would probably to take few takes at this. Thank you for such an extensive feedback, this is partially overlapping with what I have in mind as well as brings up some items I never thought of.

I think that the repo content currently focuses on DIY annotations, e.g. how to put together your own ad-hoc tool

True that. I have some of the 3rd party tools listed in tools list and at least one in the section for ner/spans. My general vision, at the moment slightly blurry, is to have generic comparison table for tools in terms of pricing / technology used / task solved / formats supported / tokenization / user management / models used to predict labels etc. Ideally I would love for each section to have several most common options and pros/cons for each one. Using jupyter is easy for quickly going through smaller datasets on your own, google spreadsheets is useful as it has user management / history tracking and export to csv out of the box, yet it's somewhat cumbersome to split data into batches / work with bigger dataset

Perhaps consider adding an introductory section that describes what each of these are and how they relate, possibly with pictures gifs (we're happy to provide some)

Good call, maybe some sort of Glossary.md with original text linking to it

how do I evaluate the quality of annotations

Really good points, would love to hear what possible references you might know for this. I know some of the tools have built-in diff viewers for same documents annotated by multiple people, people could try measuring inter annotator agreement / Cohen's kappa, but I don't think I saw lots of information on how to interpret that data or the good ways to converge multiple conflicting annotations (or have parallel version used as ground truth). Probably, hints on writing annotation guidelines would be useful as well

good output format for my use case

Yes, this is important, both for annotators / users as well as future annotation tools creators :)

MTurks has external questions which let the user provide their own annotation interface which is often very handy.

Would love to have each section a info on how to setup labelling on crowdsource platform

It's worth talking about why you annotate data

This and next paragraph brings some points I would love to be further extended, really interesting topics I haven't heard about. Do you have some references on this? Especially - testing sets and active learning and non ML annotaitons?

Somewhat unrelated note, but a possible UX simplification if this repo grows too big after covering everything you mentioned :) (yet it would require extensive argumentation and objectivity) - some sort of decision tree 'how to choose tool to annotate your NLP data' with questionnaire and yielding best match for selected criteria

I couldn't do everything in one sitting, so I'll have to come back to this and review once again/create issues. If you come up with extra thoughts - send 'em this way! :)

[sudodoki]

Items for which I didn't create issues discussed in here, I guess

• why you annotate data
• assessing annotations

Needs additional references/contributors for this. 🤔