Bugzilla adapter

[miketaylr]

A bugzilla adapter will take as an input a json bug from the HTTP api of bugzilla and convert it, prepare it for sending it to github. Example

This might depend on other issues: Let's put issue numbers beside

• documenting how to write a new adapter Documenting how to write a new adapter #11
• a test case for Mozilla json bugs without comment Test case for Mozilla json bugs with and without comment #12
• a test case for Mozilla json for comments Test case for Mozilla json bugs with and without comment #12
• a command line label for specifying the type Command line label for specifying the adapter input #13
• a function for importing the json markup

The current documentation of the API is documented in two places:

• Bugzilla Server API
• The fields for each bug
• Bugzilla API

Some examples:

• To include all data for a bug:
  https://bugzilla.mozilla.org/rest/bug/35
• To select only some data: https://bugzilla.mozilla.org/rest/bug/35?include_fields=summary,status,resolution
• To get all comments: https://bugzilla.mozilla.org/rest/bug/35/comment

[miketaylr]

https://api-dev.bugzilla.mozilla.org/latest/bug/1013525?include_fields=_default,comments

Attachments with image content type could be embedded in description at the bottom.

comment 0 would be body.

Will need an optional link back to original report (probably just in the body somewhere)

version, op_sys, summary.

So then build up an object from those parts and we'll need a new method to post comments from the comments object.

component: Desktop or Mobile. It might be tricky to build a nice "Browser" and "Version" from op_sys, component and version.

[karlcow]

The new shiny API is at
https://bugzilla.mozilla.org/rest/bug/1013525

[miketaylr]

Thanks @karlcow.

[karlcow]

@miketaylr

I took the freedom to edit the first comment and add a bit of structure and documentation.
I think we need to think a bit of what we want to do. Currently your code is working with

    python issue-importer.py bug1234.json
or cat bug1234.json | python issue-importer.py`

I wonder if we need a --from parameter, so issues importer could have a directory with specific modules for the different adapters that people could develop.

python issue-importer.py --from mozilla bug1234.json

Or do we need to format in advance into the appropriate format for issue-importer.py. Aka if not in the right format, issue-importer is bailing out.

python extract-issue.py --from mozilla mozuri -o mozblah.json
python issue-importer.py mozblah.json

Or maybe you had a 3rd option in mind. :)

[karlcow]

Some progress going on.

[karlcow]

This is not yet operational but in good progress. I would say that there are a couple of details to fix.

• which labels do we import? Do we create a vocabulary
• body needs an additional requests for the comment.
• only dry-run mode for now, needs to plug the real commit.
• test cases for weirdness in data formats

→ python import.py --dry-run -o moz bugzilla_issue.json 
Test Mode Run. Nothing is sent to github
Converting Mozilla bugs

**URL**: http://www.reuters.com/finance/stocks/OPERA.OL/key-developments/article/2833426
**Browser / Version**: Firefox
**Operating System**: android

body is missing. Need to code it

The real data being returned by the mozilla_adapter are:

{'body': 'body is missing. Need to code it', 
'title': u'Reuters.com enters in an infinite loop for *any* mobile browsers', 
'url': u'http://www.reuters.com/finance/stocks/OPERA.OL/key-developments/article/2833426', 
'labels': [u'serversniff', u'firefox', u'android', u'imported'], 
'source_human': 'https://bugzilla.mozilla.org/show_bug.cgi?id=911601', 
'os': u'android', 
'browser': u'Firefox'}```

[miketaylr]

which labels do we import? Do we create a vocabulary

I think if we have some confidence about the UA string, we can add a browser label. We could map [sitewait], [contactready], to "sitewait" and "contactready" labels as well. We don't currently have a notion of "serversniff" and "clientsniff" on webcompat.com but we can discuss adding that, especially when we get to webcompat/webcompat.com#172.

And +1 for "imported". We should make this a non-optional label (or add it ourselves if it's missing).

[miketaylr]

Whoops, forgot to respond to #7 (comment). 😢

Or maybe you had a 3rd option in mind. :)

Hmm, it's hard to say which approach is better. I like the idea of having a standalone program to do the conversion, so you could just pipe or xargs in the transformed data, or use it as part of a workflow in other tools for exporting from bug sources/trackers. But I don't think it really matters--the other option is also good, we just have to pick one and document it. :)

Aka if not in the right format, issue-importer is bailing out.

I think this would have to be a requirement for either approach (the JSON schema validation should catch this).

[karlcow]

@miketaylr About labels.

I wonder if a JSON schema can have predefined list of values. I need to study this :)
Currently in the adapter.
https://github.com/karlcow/issue-importer/blob/dd72146a3f3509034a8413d4bff33af7240cc261/importer/adapters/mozilla_adapter.py#L24-L29

I will get away from the hardcoded to have something more generic for the complete code.

[karlcow]

@miketaylr about import approach.

I'm sticking for now with your original choice:

→ python import.py -o moz bugzilla_issue.json 
→ cat bugzilla_issue.json  | python import.py -o moz 

I decided to add a parameter -o $ORIGIN where $ORIGIN could be moz (mozilla), ms (microsoft), op (opera), etc. (todo: Values to determine). It means basically that the JSON file being treated as the format of one of the chosen repository. So when the origin is given it will call a specific adapter that each company/organization can code. They don't need to export their issue in a specific format, just write the right importer that will the prepare the JSON/message for webcompat.com.

The other strategy would have been to have a fixed format for the issue JSON and the responsibility of other companies to export in a specific format.

[miketaylr]

@karlcow

I wonder if a JSON schema can have predefined list of values. I need to study this :)

It can, see https://github.com/webcompat/issue-importer/blob/master/importer/schema.py#L30 and https://github.com/webcompat/issue-importer/blob/master/importer/__init__.py#L110.

Essentially if you have an "enum" key in the schema you can restrict the set of valid values (which is what I think you're asking). http://json-schema.org/example2.html has a few examples of this.

[karlcow]

OK. No matter I turn it. It's currently impossible to receive the bug and its comments in 1 HTTP request. On the good side of the news is that I can minimize the JSON I want from bugzilla to just what I need. For example, if we request:

→ http -b https://bugzilla.mozilla.org/rest/bug/911601?include_fields=url,summary,creation_time,op_sys,id,whiteboard,status/comment

Then the JSON we receive in return is:

{
    "bugs": [
        {
            "creation_time": "2013-09-01T21:23:19Z", 
            "id": 911601, 
            "op_sys": "Android", 
            "summary": "Reuters.com enters in an infinite loop for *any* mobile browsers", 
            "url": "http://www.reuters.com/finance/stocks/OPERA.OL/key-developments/article/2833426", 
            "whiteboard": "[mobile-compat-form] [serversniff] [country-all] [sitewait]"
        }
    ], 
    "faults": []
}

Which is kind of cool. For getting the comment I need an additional request. All of this seem unrelated to the Adapter. So When writing the adapter I was wondering if I should take care of getting the description (aka getting the comments) of the bug in the adapter. After thinking about it for a little while I would prefer to have a more or less complete JSON file for a bug.

Then the secondary question:

Is the body mandatory aka description of the bug. If we consider yes, then we have an additional issue. Sometimes the description is not useful. It's why we created the tags "description" and "suggested fix" for simplebug.

It means we will need to work on a bug exporter too. :)

Opinions?

There is another question about architecture. I will open a separate issue.

[karlcow]

Created the new issue about the architecture
Architecture of Issue Importer 2.0 #16

[karlcow]

The comments have this format.

{
    "bugs": {
        "911601": {
            "comments": [
                {
                    "attachment_id": null, 
                    "author": "hsteen@mozilla.com", 
                    "bug_id": 911601, 
                    "creation_time": "2013-09-01T21:23:19Z", 
                    "creator": "hsteen@mozilla.com", 
                    "id": 7813000, 
                    "is_private": false, 
                    "raw_text": "Site: http://mobile.reuters.com/do/urlRedirect?URL=http%3A%2F%2Fwww.reuters.com/finance/stocks/OPERA.OL/key-developments/article/2388763?\nredirect loop on Reuters page\n\n:: Steps To Reproduce\n\n1. go to  \n2. error about looping redirect appears\n\n:: Expected Result\n\npage\n\n:: Actual Result\n\nerror message\n\n:: Additional Information\n\nSoftware Version: 24\nReporter's User Agent: Mozilla/5.0 (Android; Mobile; rv:24.0) Gecko/24.0 Firefox/24.0", 
                    "tags": [], 
                    "text": "Site: http://mobile.reuters.com/do/urlRedirect?URL=http%3A%2F%2Fwww.reuters.com/finance/stocks/OPERA.OL/key-developments/article/2388763?\nredirect loop on Reuters page\n\n:: Steps To Reproduce\n\n1. go to  \n2. error about looping redirect appears\n\n:: Expected Result\n\npage\n\n:: Actual Result\n\nerror message\n\n:: Additional Information\n\nSoftware Version: 24\nReporter's User Agent: Mozilla/5.0 (Android; Mobile; rv:24.0) Gecko/24.0 Firefox/24.0", 
                    "time": "2013-09-01T21:23:19Z"
                },