Introduction

• The WWW has a great potential to make life easier for people with disabilities
  • e-learning
  • e-government procedures
  • do the shopping, communicate with peers, etc.
• The WWW still remains non-accessible
  • Bad design practices
  • Lack of awareness
  • Lack of methodologies and supporting tools
• W3C-WAI proposed WCAG 1.0
  • Useful list of good design practices
  • Supported by enacted laws in different countries
  • There exist more guidelines Section 508, IBM Accessibility guidelines...

Information Search in the WWW

• Some authors (Kobayashi & Takeda, 2000) state that 85% of users use search engines when seeking for information on the Web
• Accessibility of search engines interfaces:
  In a study carried out by Andronico et al., 2006 only 38% of visual impaired found search engines results useful
• Accessibility of the results:
  Some authors (Ivory et al., 2004) suggest that,
  • Additional features in results such as cues or hints
  • Re-ranking according to user's accessibility profile
  would be helpful for visually impaired users

Objective of our Work

• Provide a list of results ranked according their accessibility
• Criteria for result sorting: Web Accessibility Quantitative Metric
• A component to calculate accessibility scores automatically
• Exploration of architectural issues
  • Proposal of a framework to include web accessibility issues in traditional Information Retrieval
  • Implementation of a prototype
  • Results when compared to other search engines
• Find weak points to be improved

Motivation for a Web Accessibility Quantitative Metric

• WCAG 1.0 conformance are the most broadly accepted qualitative levels
  • All checkpoints in WCAG 1.0 have a priority assigned (priority 1, priority 2, priority 3)
  • Based on the priorities there are four conformance levels (AAA, AA, A, 0)
    • Web page fulfilling all priority 1 checkpoints obtains A
    • Web page fulfilling all priority 1 and 2 checkpoints obtains AA
    • Web page fulfilling all priority 1,2 and 3 checkpoints obtains AAA
• A web page fulfilling only priority 1 guidelines would obtain the same value than a web page fulfilling priority 1 and almost all priority 2 guidelines
• They are not accurate enough for accurate measurement


• New drafts such as WCAG 2.0 are being developed
• New guideline concept: Perceivable, Operable, Understandable, Robust
• Goal: A metric that give a score for each POUR attribute in addition to a mean value

Limitations of the Web Accessibility Quantitative Metric

• Metric is based on automatic evaluations and scores are automatically calculated
• An evaluation report contains:
  • number of errors
  • number of potential errors
  • priority
  • typology
• Accessibility problems are classified in three main grups
  • Automatic tests (errors)
  • Semi-automatic tests (warnings)
  • Generic problems
• Results give an upper bound on the accessibility level

Web Accessibility Quantitative Metric

Requirements

• The result of the metric should be normalized
  • A ratio scale from 0 to 100 can express the accessibility in percentage terms


• The metric should also give one value for each accessibility attribute in WCAG 2.0
  • The system is capable to produce a value for each WCAG 2.0 guideline/attribute
  • Guidelines in WCAG 1.0 are mapped onto P,O,U,R guidelines

Web Accessibility Quantitative Metric II

Assumptions

• The number of times each checkpoint has been tested has to be taken into account
  • The metric should be based on the relative numbers of found errors in relation to number tested cases
  • E.g.: 2 accessible images out of 3 in a web page is more accessible than 1 accessible image out of 3


• The priority of an unfulfilled checkpoint should be reflected in the final result
  • Priorities in WCAG are based on the impact on the accessibility level
  • Their weight in the accessibility score obtained by the metric should be different
  • Weights are empirically tunned. Different values were assigned to weights. Priority 1 > Priority 2 > Priority 3
  • Manually applied our metric to some test files with a determined failure rate
  • Similarity of the accessibility score to the failure rate in the test files was the criteria to choose the weights. P1=0.8, P2=0.16, P3=0.04

Metric for Measuring Web Accessibility III

Assumptions

• Generic problems should not have influence on the metric

Facts

• When checkpoints are grouped by priority (1,2,3) and guideline membership (P,O,U,R)...
• There is a tendency for low failure ratios
  • It is difficult to discriminate among different web pages
  • Low failure rates interval has to be spread
• Failure rate in semi-automatic tests is highly correlated with automatic test
  • We assume that semi-automatic test failure rate is equivalent

Computing the metric

Algorithm

for i in each checkpoint in a guideline {P,O,U,R} loop
   for j in each type of checkpoint {auto, semi} loop 
      for k in each priority{1,2,3} loop
	     Ai,j=calculate_failure_rate()*priority_weight(k); 
      end loop      
   end loop
   Ai=(Ni,auto*Ai,auto+Ni,semi*Ai,semi)/Ni    
end loop
A=(NP*AP+NO*AO+NU*AU+NR*AR)/N

• WCAG 1.0 checkpoints are grouped by
  • guideline membership in WCAG 2.0 (4)
  • guideline type (2)
  • WCAG 1.0 priorities (3)
• The failure rate is computed for each subgroup (24)
• All subgroups are by merged pondering by the number of guidelines in each subgroup
• In addition to POUR scores a mean value, A is also obtained

Validation of the Metric

• Validity of the metric with respect to evaluation performed by experts
• Conclusion: the metric does highly correlate with automatic+manual evaluations by experts


• Validity of the metric with respect to the evaluation tool
• Conclusion: the metric can be used when ordinal scale of accessibility is needed

Framework Proposal

• Content Analysis Module (CAM): it produces a list of URLs rated according to their suitability for a specific query


• Accessibility Analysis Module (AAM): performs web accessibility evaluation


• Results Collector Module (RCM):
  • Ensures that the information provided by the other two modules is adequately combined
  • Calculates accessibility scores
  • Shows results

Instantiation of the Framework in a Prototype

• Content Analysis Module (CAM):
  • APIs offered by Search Engines: Google, MSN or Yahoo!
  • They provide abstraction layer for accessing a Web Service


• Accessibility Analysis Module (AAM)
  • Tool for automatic web accessibility evaluation, EvalAccess
  • Implemented as a Web Service
  • Evaluation reports are coded in XML

Instantiation of the Framework in a Prototype

• Results Collector Module (RCM)
  • User interface, looks like traditional search engine
  • Handles the query and delegates to the CAM
  • Query results are sent to the AAM
  • Evaluation reports are exploited and metric is calculated for each URL
  • The user obtains a list of results sorted by their accessibility score
  
  

Some screenshots of the UI

• Search interface
• 
• Search result
• 

Results

• Comparing the results of our Customized Search Engine and a Search Engine. E.g.:"cheap flights"

URL	Google	CSE	P1	P2	P3
www.cheapflights.co.uk	1	5	0	15	265
www.easyjet.com	2	2	0	29	91
www.ryanair.com/site/EN	3	6	45	230	995
www.skyscanner.net	4	8	0	9	69
www.cheapflights.com	5	3	0	7	186
www.flightline.co.uk	6	4	0	6	128
www.flybmi.com/bmi	7	1	0	2	60
www.bmibaby.com	8	10	2	75	169
www.aerlingus.com	9	-	0	5	100
www.openjet.com	10	-	0	1	3
www.bargainholidays.com	-	7	29	67	262
www.cheapestflights.co.uk	-	9	8	143	198

• In general, results having less accessibility problems are higher in the rank
• Since the metric relies on failure-rates having the less accessibility errors doesn't guarantee a higher rank

Conclusions

• Some considerations on the prototype
  • The overall latency is higher
  • Trade-off of content ranking vs. accessibility ranking. Re-ranking is not necessary, results can be marked with their accessibility score


• The proposed metric aims at being a general accessibility score


• WCAG guidelines are a general approach


• We are working on user-tailored evaluations and measurement


• Metric and search engine sorting would be tailored to particular user requirements

References

• Kobayashi and Takeda. 2000. Information Retrieval on the Web. ACM Computing Surveys 32(2), 144-173.


• Andronico, Buzzi, Castillo and Leporini. 2006. Improving search engine interfaces for blind users: a case study. Universal Access in the Information Society 5(1), 23-40.


• Ivory, Yu and Gronemyer. 2004. Search result exploration: a preliminary study of blind and sighted users' decidion making and performance. CHI 2004, 1453-1456