Add a means to compare two results within a single test

[joanmarie]

One example use case is described in the ARIA 1.1 testable statements wiki, namely:

The spec states "User agents SHOULD NOT expose descendants of this element through the platform accessibility API." Because it's a "SHOULD NOT" rather than a "MUST NOT," one might make the case that no testing is needed. However, in order to verify implementation (implementability), we can test that the user agent treats elements with the specified roles in the same way they expose roles which had children-presentational of true in 1.0, such as button, separator, and progressbar.

In other words, the first test step would get -- and save for later comparison -- whether or not a child of a button was exposed. The second test step would get whether or not a child of a tab was exposed. The expected result of the second test step is that the value is the same as what was obtained for the result of the first test step.

[halindrome]

umm... We would need some sort of wiki syntax for this. Otherwise I don't see why not.

[joanmarie]

Do you have any ideas/proposals? It would be nice to run the tests in an automated fashion, and in the near future, if at all possible.

[halindrome]

Not yet. Question: in this example, if the result of the first step is fail... doesn't the whole test fail? I mean - there are no examples in WPT of a fail result meaning that the overall test case passes as far as I know.

[joanmarie]

I'm afraid I don't follow.

Hypothetical example of what I'd like to address:

Given two elements, test1 and test2,
Step 1: Get the accessible child count of the element with id test1.
Step 2: Get the accessible child count of the element with id test2.
Expectation: The value of the child count obtained in step 2 will equal the value of the child count obtained in step 1.

In other words, step 1 isn't a pass/fail test; it's like an event (sorta). But instead of, say, setting focus, we're obtaining a value for later comparison.

[halindrome]

Oh!

Given our syntactic limitations, it would need to be a new "verb" like "is" or "contains".

property|role|rememberIn|{variableName}

then in some other step

property|role|whatever verb|{variableName}

I know role is a horrible example, but you get the idea. The braces are something we are not currently using so it would be a reasonable delimiter I think. Thoughts?

[joanmarie]

I like the idea and think it would work for getting a simple value; however I don't think it will work for everything we might need it to. One of the things we test is for states in a state set, e.g.:

property|states|contains|STATE_FOCUSABLE

In that case, we'd want to store the boolean result for the element with id test1, run the same test for the element with id test2, and compare the two boolean results.

Brainstorming: Could the rememberIn be the test step type and then add an additional comparison test step type?

element test1
property|states|contains|STATE_FOCUSABLE
remeberIn {variableName1}
element test2
property|states|contains|STATE_FOCUSABLE
remeberIn {variableName2}
assert|{variableName1}|is|{variableName2}

[halindrome]

Hmm...

"contains" is an assertion. So it would pass or fail. There's no way around that if we want to retain the current syntax. So no, I don't think that would achieve what you are looking for.

[joanmarie]

Fair enough. But what I write above, while broken for the reason you state, is the sort of thing we need to be able to do. Any brilliant (or otherwise) solutions? :)

[halindrome]

Do you anticipate the evaluation being done in the ATTA or in the client (ATTAcomm.js) ?

[joanmarie]

For the most part, I would think the client could handle it. But if we ever had a case where the two results being compared or examined required platform-specific knowledge, then the ATTA would need to do it. So having the ATTAs do the evalution work seems safer to me.

What do you think?

[halindrome]

I actually don't think the client can handle it.... the client doesn't have any of the information that the ATTAs have. The ATTAs do everything. All we could have the client do is retain information on the reply from the ATTA to a query. Which might be fine...

Extend the syntax of one of the columns to mean "this statement is NOT a test; remember its result in a variable named foo". Then the ATTA could do the initial processing and the client could do any test based upon the result.

element test1
property|states|contains|{STATE_FOCUSABLE:varName1}
element test2
property|states|contains|{STATE_FOCUSABLE:varName2}
evaluate varName1 == varName 2

or something like that.... could do it on the verb as well:

element test1
property|states|{contains:varName1}|STATE_FOCUSABLE
element test2
property|states|{contains:varName2}|STATE_FOCUSABLE
evaluate varName1 == varName 2

Do either of those resonate?