Motivation

• WCAG 1.0 conformance levels (0, A, AA, AAA) are the most broadly accepted qualitative metrics
• They are not accurate enough for several purposes
• A web page fulfilling only priority 1 guidelines would obtain the same value than a web page fulfilling priority 1 and almost all priority 2 guidelines
• It is assumed that if a web page fails to accomplish one guideline in a level it is so unaccessible as it fails to fulfil all of them
• We need more than accept/reject validation

Application Scenarios for Quantitative Metrics I

Quality Assurance within Web Engineering

• Web engineering aims at creating high quality web applications
• Quality Models are defined
  • List of attributes in order to measure the quality of a product
  • Web accessibility is included in Quality Models. WebQEM [Olsina and Rossi, 2002] and 2QCV3Q [Mich et al., 2003]
  • Web accessibility has to be accurately measured
• Ranking different prototypes during the lifecycle according to accessibility
  • Assess the impact of new functionalities
  • Measure the impact of updates

Application Scenarios for Quantitative Metrics II

Information Retrieval Systems

• Ivory et al. state that sorting results according visually impaired users abilities would improve search experience
• Results yielded by search engines could be sorted according accessibility criteria
• Google launched Google Accessible Search

Web Accessibility Monitoring

• Institutional websites are framed by legal restrictions
  • Updates can decrease accessibility level
• An accurate metric is useful to keep track of e-government accessibility levels

Obtaining Quantitative Metrics Automatically

Modular architecture

• Guidelines repository
• Evaluation Engine
• Report Manager works jointly with the Metrics Manager
• External evaluation reports are also included

Metric for Measuring Web Accessibility

Overview

• Metric is based on automatic evaluations
• Accessibility problems are classified in three main grups
  • Automatic tests (errors)
  • Semi-automatic tests (warnings)
  • Generic problems
• Results give an upper bound on the accessibility level
• Strong points of the state-of-art have been considered [Sullivan and Matson, 2000 and Hackett et al., 2004]

Metric for Measuring Web Accessibility

Requirements

• The result of the metric should be normalized
  • A ratio scale from 0 to 100 can express the accessibility in percentage terms
• The metric should also give one value for each accessibility attribute in WCAG 2.0
  • The system is capable to produce a value for each WCAG 2.0 guideline/attribute
  • Guidelines in WCAG 1.0 are mapped onto P,O,U,R guidelines

Metric for Measuring Web Accessibility II

Assumptions

• The number of times each checkpoint has been tested has to be taken into account
  • The metric should be based on the relative numbers of found errors in relation to number tested cases
  • E.g.: 2 accessible images out of 3 in a web page is more accessible than 1 accessible image out of 3
• The priority of an unfulfilled checkpoint should be reflected in the final result
  • Priorities in WCAG are based on the impact on the accessibility level
  • Their weight in the accessibility score obtained by the metric should be different
  • Weights are empirically tunned. Different values were assigned to weights. Priority 1 > Priority 2 > Priority 3
  • Manually applied our metric to some test files with a determined failure rate
  • Similarity of the accessibility score to the failure rate in the test files was the criteria to choose the weights. P1=0.8, P2=0.16, P3=0.04

Metric for Measuring Web Accessibility III

Assumptions

• Generic problems should not have influence on the metric

Facts

• When checkpoints are grouped by priority (1,2,3) and guideline (P,O,U,R)...
• There is a tendency for low failure ratios
  • It is difficult to discriminate among differen web pages
  • Low failure rates interval has to be spread
• Failure rate in semi-automatic tests is highly correlated with automatic test
  • We assume that semi-automatic test failure rate is equivalent

Computing the metric

Algorithm

for i in each checkpoint in a guideline {P,O,U,R} loop
   for j in each type of checkpoint {auto, semi} loop 
      for k in each priority{1,2,3} loop
	     Ai,j=calculate_failure_rate()*priority_weight(k); 
      end loop      
   end loop
   Ai=(Ni,auto*Ai,auto+Ni,semi*Ai,semi)/Ni    
end loop
A=(NP*AP+NO*AO+NU*AU+NR*AR)/N

• WCAG 1.0 checkpoints are grouped by
  • guideline membership in WCAG 2.0
  • guideline type
  • WCAG 1.0 priorities
• Based on evaluation reports, the failure rate is computed for each subgroup
• All subgroups are by merged pondering by the number of guidelines in each subgroup
• Mean accessibility value, A is obtained as a result

Experimental Analysis of the Proposed Metric

Overview

• The metric proved to correlate positively with a research carried out by experts [Arrue et al., 2004]
• A study was carried out to verify its reliability
  • The degree to which it yields comparable results when applied to the same websites but using different tools
• Rationale:
  • If results are similar, the particular tool is not an important decission
  • If results are not similar but do correlate, different tools can be used for scenarios that require ranking websites
  • If results do not correlate, no comparisons are justified

Experimental Analysis of the Proposed Metric

Experimental Design

• We used two accessibility evaluation tools, EvalAccess and LIFT
• We selected 15 websites belonging to universities and online newspapers from all over the world
  • About 100 pages were downloaded for each of them
  • From homepage following a breadth-first strategy
  • This resulted in 1363 pages, 918 belonging to universities and 445 belonging to newspapers
  • These pages were uploaded in a web server and EvalAccess and LIFT were independently launched

Experimental Analysis of the Proposed Metric

Experimental Results

• We obtained two variables A_EVAL and A_LIFT
• Assuming all pages in a website have the same weight we obtained A^*_EVAL and A^*_LIFT
• There is a marked difference between A_EVAL and A_LIFT
  • A_EVAL is higher than A_LIFT.
  • Comparing the means, A_EVAL is larger than A_LIFT by at least 36 percentual points
  • Ranks of A_EVAL and A_LIFT Spearman's correlation is r(1363)=0.719
  • 

Experimental Analysis of the Proposed Metric

Experimental Results II

• By splitting the data site by site, A^*_EVAL is larger than A^*_LIFT
• 
• Spearman's correlation index on A^*_EVAL and A^*_LIFT is r(15)=0.735

Experimental Analysis of the Proposed Metric

Interpretation of results

• A_EVAL and A_LIFT are completely different. A^*_EVAL and A^*_LIFT are also different
• A_EVAL and A_LIFT do correlate and A^*_EVAL and A^*_LIFT do strongly correlate
• The choice of the tool constraints the evaluator when quantitative values are needed
  • QA in Web Engineering scenario
• The metric can be used when ordinal scale of accessibility is needed
  • Information Retrieval
  • Accessibility Monitoring scenarios

References

• Olsina and Rossi. 2002. Measuring Web Application Quality with WebQEM. IEEE Multimedia 9(4), 20-29
• Mich, Franch and Gaio. 2003. Evaluating and Designing the Quality of Web Sites. IEEE Multimedia 10(1), 34-43
• Ivory, Yu, Gronemyer. 2004. Search result exploration: a preliminary study of blind and sighted users' decidion making and performance. CHI 2004, 1453-1456
• Arrue, Vigo, Abascal. 2005. Quantitative Metrics for Web Accessibility Evaluation. ICWE 2005 Workshop on Web Metrics and Measurement