Single-word lines do not form events

[kuchenrolle]

What you were trying to do

Turn a one-sentence-per-line corpus into events, where some lines are only one word (headings).

What actually happened

The single-word lines do not form events.

How to reproduce

The problem is in the generation of occurrences. This is what happens there:

occurrences = list()
words = ["test"]
before = 2
after = 1
for ii, word in enumerate(words):
    # words before the word to a maximum of before
    cues = words[max(0, ii - before):ii]
    # words after the word to a maximum of before
    cues.extend(words[(ii + 1):min(len(words), ii + 1 + after)])
    # append (cues, outcomes)
    occurrences.append(("_".join(cues), word))

The loop has only one iteration, in which the word is the outcome without any cues, so it is dropped.

This is not necessarily a bug, but it was at least surprising to me. I would expect (want) that the word is treated as a cue that predicts no outcome. I only noticed this randomly, when a cue was dropped, because it occurred exclusively in those circumstances. I haven't checked, but I'm assuming this holds for context structures other than "line" as well.

[derNarr]

Thank your for the bug report.

To me it seems to only make sense to generate events with the word as cues and no outcomes, if we consistently add an "begining" / "ending" / "empty" marker to the start and the end of the words in the line, i. e.:

In [42]: occurrences = list() 
    ...: words = ["test"] 
    ...: before = 2 
    ...: after = 1 
    ...: words = ['',] + words + ['',] 
    ...: for ii, word in enumerate(words): 
    ...:     # words before the word to a maximum of before 
    ...:     cues = words[max(0, ii - before):ii] 
    ...:     # words after the word to a maximum of before 
    ...:     cues.extend(words[(ii + 1):min(len(words), ii + 1 + after)]) 
    ...:     # append (cues, outcomes) 
    ...:     occurrences.append(("_".join(cue for cue in cues if cue), word)) 
    ...:                                                                        

In [43]: occurrences                                                            
Out[43]: [('test', ''), ('', 'test'), ('test', '')]

The occurrence where 'test' has no cues, gets removed later on.

This would lead to the following behaviour in longer contexts:

In [46]: occurrences = list() 
    ...: words = ["hello", "world", "we", "are", "small"] 
    ...: before = 2 
    ...: after = 1 
    ...: words = ['',] + words + ['',] 
    ...: for ii, word in enumerate(words): 
    ...:     # words before the word to a maximum of before 
    ...:     cues = words[max(0, ii - before):ii] 
    ...:     # words after the word to a maximum of before 
    ...:     cues.extend(words[(ii + 1):min(len(words), ii + 1 + after)]) 
    ...:     # append (cues, outcomes) 
    ...:     occurrences.append(("_".join(cue for cue in cues if cue), word)) 
    ...:                                                                        

In [47]: occurrences                                                            
Out[47]: 
[('hello', ''),
 ('world', 'hello'),
 ('hello_we', 'world'),
 ('hello_world_are', 'we'),
 ('world_we_small', 'are'),
 ('we_are', 'small'),
 ('are_small', '')]

Is this the behaviour you are looking for?

If not should we spit out a warning if words has length one?

[kuchenrolle]

Yeah, I'm not sure. Maybe a warning is best, although it will presumably always be issued. I can see that what you're suggesting is more consistent, but with empty markers added like that, the first and last word(s) are always learned to predict the absence of all non-empty outcomes, which is similarly awkward or probably even worse, as single-word sentences are rather infrequent.

To be honest, it probably doesn't matter that much. But it's good to be aware of.

[Trybnetic]

I agree with adding a warning and reference #107 here to keep that in mind while implementing multilevel logging.