Efficiency and effectiveness in case of disaster : a visual damage assessment test

A user study is conducted to evaluate the efficiency and effectiveness of two types of visualizations to identify damages sites in case of disaster. The test consists out 36 trials (18 for each visualisation) and in each trial an area of 1x1km, located in Ghent, is displayed on a screen. This image shows the combined height information from before and after the disaster. The first visualisation, page flipping, is based on greyscale images with height information from the preand post-disaster situation between which users can switch manually. The second visualisation, difference image, is a result of subtracting the heights (before versus after) and assigning a blue-white-red colour ramp. In order to simulate the urgency with which the data is captured, systematic and random imperfections are introduced in the post-disaster data. All participants’ mouse and key interactions are logged, which is further complemented by the registration of their eye movements. This give insights the visualizations’ efficiency, effectiveness and the overall search strategies of the participants.


Introduction
Disasters can struck anywhere and anytime, leaving a trace of destruction.Emergency response teams need to know the location of damaged sites as fast and accurately as possible in order to save lives.A promising option is to use the crowd, which enormous power has been demonstrated in response to the Haiti earthquake in 2010 (Zook et al., 2010).The fact that the volunteers not necessarily need to be present at the disaster site is a huge advantage of crowdsourcing: contributions can come from anywhere, anytime.Nevertheless, most of these volunteers are not necessarily experts in the matter at hand.Therefore, adapted tools and visualizations -easy to handle and interpret -are crucial.This research focusses on the evaluation of the efficiency and effectiveness of two different visualization in a situation where the crowd is asked to identify damaged sites in a crisis situation.

Participants
Students, alumni, and staff from Gävle University College are invited to take part in this test.Their personal characteristics (age, gender, education, etc.) are obtained through a questionnaire, which also probes for familiarity with the displayed area.All participants participated voluntarily and received a lunch coupon.

Stimuli
The user study consists out of a training test followed by 36 trials (18 for each visualisation).In each trial an area of 1x1km, located in Ghent, is displayed on a screen.This image shows the combined height information from be-fore and after the disaster.It can be assumed that height information is available for most regions on the planet.After a disaster, height information can nowadays be gathered quickly using remote sensing techniques (LiDAR, photogrammetry, e.g. by the use of UAVs), as has been applied in response to the 9/11-attack in New York (Huyck et al., 2003) and the Haiti earthquake (Kent, 2010).In order to detect the damaged sites, selecting the correct visualisation is crucial.In this study, two alternatives are evaluated.The first visualisation, page flipping, is based on grey-scale images with height information from the pre-and post-disaster situation between which users can switch manually (see Fig. 1, left and middle image).The second visualisation, difference image, is a result of subtracting the heights (before versus after) and assigning a blue-white-red colour ramp (see Fig. 1, right image).Of particular interest is the effect of change blindness in these visualisations (Simons & Ambinder, 2005).In each of the images, damaged sites are simulated: buildings that have collapsed in different manners.The number of damage sites varies between 6 and 8, but is balanced across all images.In order to simulate the urgency with which the data is captured, systematic and random imperfections are introduced in the post-disaster data.The stimuli are thus subdivided into two groups dependent on the visualization (page flipping versus difference image).Within each group, the stimuli are displayed in a random order to the participants.In order to avoid a learning effect or influence due to fatigue, two measure were taken.First, each participant saw only half of the 72 stimuli (36 in Set 1 or 36 in Set 2).This way the participant never saw the same 'area & damage sites'combination twice.Second, half of the participants started the experiment with the page flipping visualization whereas the other half started with the difference images.Taking into account these measures, a rotation of the experiment conditions across four participants was implemented (see

Tasks and Recordings
Participants have to indicate (by mouse click) the damage sites (6-8) as quickly as possible.When the page flip-ping visualisation is displayed, the participants can manually 'flip' between the pre-and post-disaster image using the left and right arrow keys.The participant can proceed to the next image by pressing the space bar.Nevertheless, a time limit of 75 s is set on each trial to simulate urgency and keep the study manageable.The system logs the time of each mouse click and keyboard interaction and the position of the mouse clicks.Furthermore, the participants' eye movements are recorded to obtain insights in their visual behaviour (search strategies, interpretation difficulties, etc.).The experiment is implemented in a web site that is linked to the eye tracking software.For this experiment, the SMI RED250 was used to record the participant's eye movements (see Fig. 2).
Fig. 2. SMI RED250, with difference image displayed as visualization

Procedure
The experiment follows a structured stepwise approach.First, the participant is given a general introduction on the context and goal of the experiment, the different types of visualizations and the tasks that should be complet-ed.Second, (s)he is asked to fill out a questionnaire which focusses on personal characteristics (age, gender, level of education, color blindness, familiarity with the region, etc.).Third, the participant could complete a short training task related to the first visualization.Fourth, the eye tracking equipment is calibrated before starting with the actual main test (which consists out of 18 trials).After completing this test, the participant could enter feedback on the test or visualization.Next, the procedure is repeated from step three and onwards for the second visualization.

Data Analysis
Two key components complement each other in the analysis: the data gathered through the web site (mouse and key interactions) and the eye movements.The data from the web site is stored in a database from which it is ex-ported.The available data sets are illustrated in Fig. 3. Using these data sets, comparisons between the visualizations can be made based on: trial completion time, number of correct demolitions indications, number of wrong demolition indications, time to first demolition indication, time between last demolition indication and trial end, etc.Furthermore, part of the analysis focusses on the influence of the systematic imperfections on these measurements, complemented by, among others, the number of arrow key presses.

Conclusion and Future Work
This study is a first step in order to evaluate which visualization is most efficient and effective to visually detect damage sites by the crowd in case of disasters.Furthermore, more insights in the participants' search behavior is obtained -where do they start looking, is there a trend in how the image is scanned, which regions are neglected, etc.
-is provides crucial information to propose improvements for these visualizations.Therefore, future work is foreseen to implement other visualizations (e.g.variations in the size of the area that is displayed, additional processing of the data, etc.) and characteristics of the participants (e.g.familiarity with the region).

Fig. 1 .
Fig. 1.Visualizations of pre-versus post disaster height information using (left & middle) page flipping or (right) a difference image

Fig. 3 .
Fig. 3. Data obtained from the web component An overview of the analysis scheme for the eye tracking data is presented in Fig. 4. The data is gathered by software provided by the eye tracking vendor SMI.However, this is a continuous recording of raw data over all trials, which has to be split up to have the data for each trial separately (by synchronising it with the res_trials data

Fig. 4 .
Fig. 4. Data and analysis scheme for the eye tracking component

Table 1 .
Rotation of parameters across participants to avoid learning effect and influences of fatigue