A set of criteria for evaluating map application design in a mobile environment

With the appearance of smartphones and affordable mobile phone plans including (internet) data within the last decade, mobile cartography in the sense of map applications became available to a wide user group including professionals and non-professionals. This paper defines the field of mobile cartography, deals with new research areas and prerequisites concerning mobile cartography and a set of criteria for evaluating map applications has been derived. For this work, the criteria have been applied to several map applications in order to evaluate them and present recent features and possible actions. In particular, the focus has been set on high-quality map applications for Austria and on novel navigation and routing capabilities. Results showed that the guideline presented in this paper can already be well applied but can still be improved upon.


Introduction
In 2021 we are used to handling mobile map applications (apps) in our everyday life. Navigation systems, applications like Google Maps or location-based services (LBS) in general are omnipresent. This has of course not always been the case. Looking back not too far in time, concerning the history of cartography, the digital revolution of the 1980s and 1990s brought a serious shift in map design and distribution by introducing more digital media as well as the internet to the field. Twenty years later the mobile revolution had similar consequences especially for map design (Reichenbacher 2004 andRicker andRoth 2018). Looking at one's favourite web mapping service for example, we can type in a search term, maybe our favourite restaurant, and its location will be immediately shown to us on screen. Only a few words of user input trigger a chain of geospatial functions in the background. Such convenient usage (which is basically a full-on geospatial analysis) makes it obvious that the audience for GIS products has expanded. While this offers a lot of possibilities, it also rises new challenges. Since the audience for GIS based applications has expanded to nonexperts as well, apps must be designed in certain conventions which accommodate the user's level of knowhow. Cartographers and GIS-experts must adapt to the distinct capabilities and constraints of mobile devices. The claim made above led to the following research questions that are dealt with in this paper: What are new design conventions for mobile map applications and how do they contribute to a good application design? Furthermore, how can existing mobile map applications be evaluated through a set of criteria based on broad consent in recent literature? These questions are going to be answered within this paper and subsequently the elaborated criteria are being applied to a quick evaluation of selected applications. Those applications will be chosen from the field of mobile map services with focus on Austrian maps. After evaluating the apps, the initially defined criteria of evaluation are discussed based on the experiences made during their application. The aim of this paper is to evaluate existing cartographic applications using a descriptive approach and therefore making use of a predefined set of criteria. This paper is mainly focusing on the prerequisites a (good) mobile map application must fulfil but its aim is not to provide guidance on how to conceive, design or develop an application from the bottom up. It also does not provide any guidance on programming techniques or software preferences. Before heading into the details of what is distinct about mobile cartography, to further deduct some criteria for our guideline, it is necessary to first define what the term mobile cartography describes. Breaking down the term we are left with the words cartography and mobile. A simplified definition found in the Cambridge Dictionary (which will not be any further commented in this paper) describes cartography as "the science or art of making or drawing maps" (Cambridge University Press 2014). In a more elaborate approach, Spektrum (2021) includes all contents and activities from the conceptualization of maps up to their use. The International Cartographic Association defined cartography in 2019 as follows: "Cartography is the science, art, and technology of making and using maps." (Kraak, 2019). A mobile device can simply be defined as a computing system small enough to be handheld such as a smartphone or tablet" (IGI Global 2021). While few sources give a direct and concise definition of mobile cartography, some describe the concept in a detailed manner. According to Reichenbacher (2001) "Mobile cartography deals with theories and technologies of dynamic cartographic visualisation of spatial data and its interactive use on portable devices anywhere and anytime under special consideration of the actual context and user characteristics". Since we aim to define what mobile cartography is rather than what it refers to, we decided to formulate our own definition of the term based on the knowledge gained above. Mobile Cartography is the technology, visualisation, transmission and usage of spatial data fitted for the special circumstances and capabilities of mobile devices as well as a mobile usage-environment. With this definition now set, the following parts of this paper will proceed to clarify the details of what those aforementioned special circumstances of mobile devices are as well as why those circumstances are of utmost importance when it comes to the design of mobile mapping applications. Finding a suitable set of criteria for evaluating mobile maps includes profound literature research. Firstly, mobile cartography is "located" within the broad field of cartography. This includes also a brief look on important new research areas or neighbouring disciplines of mobile cartography. To derive a set of criteria for answering the research questions, the conceptual framework of mobile cartography proposed by Reichenbacher (2001) should be born in mind as well as some prerequisites for mobile cartography and the two main strategies for digital mapping applications.

Important New Research Areas
At the beginning of the 20th century, cartography could be divided in theoretical and practical cartography. Today, however, the view on cartography has changed into a general cartography and an applied cartography (Hake et al. 2002). Mobile cartography can be regarded a field within applied cartography. By having access to the web and determining the location of the device, users are enabled to obtain plenty of information to the local environment. This is what we know as "location-based services", one of the new research areas. Using LBS (location-based services), the maps within the applications adapt themselves (both zoom-level and extent) depending on the user's position on earth. Within the field of mobile mapping for example, a map view with the user's own location as an arrow pointing to the user's orientation can be created -as we know it from navigation systems -so that we have a forward view, not a north-orientated view. This is called "egocentric map view" (Meng 2005, quoted from Ricker and Roth 2018). An everyday example for location-based services is sharing your position with others as possible in Google Maps or WhatsApp. LBS facilitate wayfinding as well by supporting turn-by-turn navigation (Klippel et al. 2010, quoted from Ricker and Roth 2018, Gartner et al. 2007). The term "TeleCartography" refers to the distribution of cartographic presentation forms via wireless data transfer interfaces and mobile devices. Another term that can be associated with mobile cartography is "ubiquitous cartography": Cartography, which is available anywhere and at any time and permits users to create and use maps at any place and at any time to resolve geospatial problems. However, how "ubiquitous" a map app can be might as well depend on its requirements concerning sensor activity and established connections (e.g. internet access, GPS). Augmented reality (AR) can be also considered to be a part of mobile cartography (Slocum et al. 2010). For this reason, we decided to include Google Maps in this paper since it offers a quite new AR-technique for visualizing route directions. Another new research area is adaptive cartography, which will be presented in the following paragraph. Map designs can change according to the kind-of-use and the context. Other related research areas that deal with mobile map design are volunteered geographic information (crowdsourced information) and locational privacy, which means the protection of personal spatial information from public communication without permission (Ricker and Roth 2018, Cartwright and Ruas, 2021, Kubíček et al. 2019, Elwood, 2012, Konečný and Staněk. 2010). Reichenbacher (2001) introduced an extended framework for mobile cartography which the following paragraphs are referring to. Figure 1 shows an adapted illustration of this framework. The user can be described by parameters such as knowledge, preferences, abilities and personality. These parameters influence a personalised extraction of information and visualisation. The user demands spatial information from the internet as a huge information pool (containing heterogeneous data differing in scale, quality, price etc. like maps or geographic features that can be geocoded), which is then filtered depending on the context in which the information needs to be gathered and visualised. The most important context element is the user's location or the desired input location, followed by weather or time for example. The visualisation is personalised by the user and controlled by the type of information, which can be in turn retrieved by different analysis methods like spatial analyses (e.g. proximity analysis).

Methods
The methods used in this paper can be divided into two sections: • Literature research to derive criteria for evaluating mobile cartography apps; • Evaluating selected mobile cartography apps based on these criteria and revealing positive features as well as usability issues. To find suitable criteria, it is first necessary to find out what distinguishes mobile maps from common maps and to have a look at the capabilities and the constraints of mobile mapping.

Distinctiveness
Mobile cartography is multimedia, which is what distinguishes mobile cartography from static digital maps in raster or vector format. For mobile cartography, it is crucial to select different modalities (perception channels) depending on context and user, since not all the user tasks necessarily afford a visual information representation.

Capabilities and Constraints
Mobile devices can be taken literally to any place on the earth's surface or beneath. Nowadays, they are so handy that they can be taken along easily even for a walk. Equipped with a battery, they do not need permanent external power supply. Another capability is the usage of many connection types and sensors: Built-in internet connection together with a SIM enables access to the huge information pool of the web also out in the field while bringing along the capability to use wireless LAN. Image sensors make taking pictures and videos possible and microphones enable audio recording. Most modern smartphones are also equipped with further sensors like Bluetooth, GNSS sensors (e.g. NAVSTAR GPS, Galileo, GLONASS, Beidou), accelerometers, gyroscopes, barometric pressure sensors etc. Besides the capabilities, there are constraints as well: Although mobile device screens have become larger and larger during the last years, the screen size is still reduced, and a restricting factor compared to print products or desktop applications. Moreover, the resolution varies a lot depending on the model. Compared to desktop computer systems, we might consider the reduced processing power and disk and memory capacities. Furthermore, the data connection is less reliable and the bandwidth is reduced. The coverage gradient between urban and rural or sparsely populated areas should be considered. External factors such as sunlight, extreme temperatures, precipitation, wind, noise and congestion must be taken account of because they can produce variable viewing and interaction conditions and make designs less reliable across conditions. Reichenbacher (2001) defined several prerequisites in order to make visualisation in a mobile context possible: Mobile, handheld consumer devices; High resolution displays; Positioning and orienteering techniques; Broadband wireless internet connection; Interoperability and several others.

New Design Conventions
At the beginning, digital maps were intended for being viewed on "big" screens (connected to desktop systems). Nowadays, more and more maps are used on "small" screens. Due to the smaller screen size and the special user requirements, the user experience (UX) is limited. At first, the same maps had been used on both smartphones and normal screens. This led to problems with readability and user interaction and therefore the adaption of maps for proper viewing and interaction on smartphone screens became necessary: "Arguably, the mobile revolution in the late 2000s and early 2010s was as transformative for map design as the digital revolution was in the late 1980s and early 1990s" (Ricker and Roth 2018). Two main strategies exist in order to develop digital mapping applications: Web maps and Mobile apps. One way is to create responsive web maps: Responsive web design is a set of strategies that changes the content dynamically and adapts the layout and the style of a website depending on the device and the user context (Ricker and Roth 2018). Responsive websites have potential for a better accessibility and dissemination because they use the Open Web Platform. This approach makes sense when resources are limited and it is usually cheaper in development, but slower in processing and interaction and a continuous established network connection is needed (Ricker and Roth 2018). Conventions for web maps include information panels on one side of the screen, a larger default cartographic scale (including greater map detail and higher label density) and more prominent menus, windows or icons and are optimized for use with a mouse pointerin short WIMP. As far as user experience / user interface (UX/UI) is concerned, there are several things to consider for mobile maps (Trajkovski et al. 2015): • Icons must be big enough to be touched with the thumb and have enough separating space between them; • Mobile maps are mostly used for location-based services in the proximity of the user and while the user is in motion; • Since entering text on small screens can be time consuming and is prone to errors, users might prefer existing lists, which regularly need to be updated; • Interactivity is an important feature: Interactive icons or map symbols with hyperlinks prevent overfilling of the map and they can add multimedia content to the map. With the above given definitions and approaches to mobile cartography and the selected applications mentioned, we can now proceed to the derived set of criteria for evaluating these applications afterwards, which is part of the research results.

Evaluation criteria for mobile cartography apps
Considering the outlined theoretical input including Reichenbacher's conceptual framework, we derived the following set of criteria for analysing and evaluating mobile cartography applications. This set of criteria involves the name of the respective criterion, a yes-no question (in order to make the results comparable) and a tighter definition.
(1) Basemap Quality: Is it appropriate to the app's intended purpose, source, data actuality and validity? (2) Ubiquitous cartography: Can the app be used in a "ubiquitous" manner? (3) UX/UI: Is the design of the user interface appropriate to the app's purpose and the mobile environment? (4) Mobile Design Conventions: Does the app design adhere appropriately to mobile design conventions? (5) Perception Channels: Does the app make use of all perception channels appropriate for its purpose? (6) Use of capabilities: Does the app make appropriate use of the special capabilities of the mobile device and environment? (7) Adapting to constraints: Does the app design consider the special constraints of the mobile device and environment? (8) Location Based Services (LBS): Does the app make use of LBS appropriate to its purpose?
(9) User tasks: Can the app perform all tasks as appropriate for its purpose? (10) Functional adaption to target audience: Is the functionality of the app designed with regard to its audience?

Austrian Map Mobile (Android)
-General Description "Amap mobile" is the official mobile application for the authoritative topographic maps for the Republic of Austria. It includes all available scales of the Austrian Map: 1:50.000, 1:250.000, 1:500.000, 1:1.000.000. The app offers a tile-based offline service (tiles need to be downloaded for use) with GPS-tracking capabilities and a search function by official geographical names or administrative units.
-Basemap Quality -Source, Validity and Recency The aim of this app is to supply a highly detailed topographic map for all kind of user groups. All data can be regarded as high quality and valid official data.
-Ubiquitous Cartography Initially, the user must purchase the desired area of Austria and a download of the map tiles via mobile data or wireless LAN is required. However, once successfully installed, this application can be used with and without internet connection. Active GPS connection is not a must-have to operate and view map content or names but is required for tracking.
-UX/UI As far as usability and user interface is concerned, the application's performance is lacking: The buttons and icons in the default map view are quite small and in order to change the displayed information, tapping in the middle of the screen's bottom is necessary. All submenus can be reached via a small button down on the right-hand side and are shown as a list of buttons with even more lists. There is (compared to other apps) room for improvement.
-Mobile Design Conventions An advantage of the app is the wide map window, which is essential within an application that has been designed for viewing topographic maps. Furthermore, standard gestures for zooming and panning are included. However, the map is a static raster map and therefore it is not possible to change the orientation manually. What is indeed possible, is to use the map's compass function so that the map is always north-oriented, no matter in which direction the device's top is showing. Tilting the map is not included in the app's functions (no 3D views).
-Perception Channels There are two perception channels: visual and auditory.
-Use of Capabilities The app definitely meets the capabilities of the mobile environment: It makes use of the web (installation, updates of map material), of the GPS sensor, and the gyroscope (compass). In addition, the map material is of high resolution and the app is not dependent on a stable data connection as it is an offline map.
-Adapting to Constraints The performance of the app is good and it deals well with the reduced screen size, since the map viewer is not cluttered. Moreover, the app does not require ongoing internet connection which is beneficial if the user does not have a mobile plan with unlimited data and is dependent on wireless LAN networks. One disadvantage of the app is the disk space that is being used for storing the map data offline on the phone.
-Location Based Services Austrian Map mobile uses location-based services in a limited capacity: Based on the device's GPS position, the map pans automatically and the position is centred.
-User Tasks Locator tasks are performed through determining the user's position and looking up place names. Way descriptions are not directly possible, but a rudimentary compass navigation is included. Distances or areas can be measured (exploratory user task), also a continuous information display of altitude or XY-coordinates is possible. Data collection is another user task included in this app: Tracks can be recorded and locally stored.
-Functional Adaption The included functions are displaying basemap material, a compass function, searching for geographical names and creating GPS tracks. However, since this is the only mobile application of BEV, more functions could be included: Purchasing and displaying separate geodata (which is available in the BEV Shop), a simple data viewer to get attributive information or the integration of vector basemaps.

Bergfex Touren & GPS
-General Description This outdoor application is an extended map viewer where users can create, edit and manage tour data (GPX tracks) and display different kinds of basemaps. The app can be used both online and offline and has also bi-modal navigational functionality (walking and mountain bike) based on OSM data. It offers high quality basemaps from renowned sources in Austria. It is also possible to display a slope overlay.
-Basemap Quality -Source, Validity and Recency The aim of this app is to supply highly detailed outdoor maps from official sources but also from crowdsourced volunteered geographic information (OSM) and for all kind of user groups. The quality of the map material can be described as very good, since the app includes various sources.
-Ubiquitous Cartography Generally, a working internet connection is needed for displaying map material. However, the user can save map tiles of an area of interest and use the app also offline. Active GPS connection is not a must-have to operate the app but is required for tracking and navigation. This criterion can be evaluated as appropriate, but the user must not forget to store data offline before going into the field without any access to the web.
-UX/UI The interface of the app is tidy but at the same time offers most of the necessary functions within several clicks: There are buttons for location, for the map layers, for navigation and a bar on top (search and menu bar).
-Mobile Design Conventions Compared to BEV's Austrian Map, this application offers extended gestures: zooming, panning, tilting and rotating the map. The map window is large enough and the interactions are comprehensible.
-Perception Channels There are multiple perception channels: The visual (for viewing tracks, map sections or other information) and the auditory channel that can be used for navigation and warnings when leaving a track. It is also possible to set push-notifications (sound and vibration).
-Use of Capabilities The app makes use of the web (installation, display and download of map material, synchronisation of track data with online account etc.), of the GPS sensor, the gyroscope (compass) and Bluetooth (heart rate monitoring via sports watches and fitness trackers). In addition, the map material is of high resolution and the app is not dependent on a stable data connection, it can also be used offline.
-Adapting to Constraints The performance of the app is goodalthough sometimes a bit demanding -and the app deals well with the reduced screen size, since the map viewer is not cluttered. Moreover, continuous internet connection is not prerequisite which is good if the user does not have a mobile plan with unlimited data and is dependent on wireless LAN networks.
-Location Based Services Location-based services are being provided: Based on the device's GPS position, the map pans automatically and the position is centred. The user gets suggestions for tours in his proximity that can be further filtered.
-User Tasks Locator tasks are performed through determining the own position and looking up place names. Getting the nearest tours as a result is a proximity task. Bi-modal navigation is also included. However, events are not directly displayed in this offline map, but using it online enables the display of live-traffic blockades and prohibitions via the Google Maps basemap. Crowdsourced-GI tasks and data collection tasks are implemented as well: Tracks can be recorded, stored (including descriptions and attribute information) and shared via the Bergfex platform with other users.
-Functional Adaption This app can be described as an explorative app for tracking tours and following existing tracks. furthermore, it is also a successful navigation system for pedestrians (including warnings when leaving a route) and cyclists and it offers a high-class selection of basemaps.

Google Maps
-General Description Google Maps is "the paragon" of implementing LBS in a map application: It is a map viewer and mobile multimodal navigation system (walking, cycling, motorised private transport and public transport), users can search facilities in their proximity or at any place in the world, display live traffic data and see the world through a photographic view using "Street View". Maps has got novel implementations like listening to music while navigating or following routes using the device's camera in AR. Furthermore, it is also possible to store and label places linked to the user's Google account.
-Basemap Quality -Source, Validity and Recency Google Maps offers a wide palette of functions, but regarding its basemap only the data Google acquired or bought by itself can be called up and used by now. While the basemap quality might be very good in cities and densely populated areas (apart from countries or regions that reject the Western world and especially US-American companies), it might lack of detail in sparsely inhabited regions. The data is also not compulsory verified by reliable sources and includes also volunteered geographic information.
-Ubiquitous Cartography Generally, a working internet connection is needed for displaying map material and using most of Google Maps' functions. However, the user can save map areas for offline use. Active GPS connection is not a must-have to operate the app but is required for tracking and navigation. This criterion cannot be evaluated as appropriate, because the abundance of functions needs a lot of internet data and the app was not designed for use without any connection in the field.
-UX/UI The interface of the app unveils that its purpose is to perform proximity tasks: The search bar and the filter categories are very prominent on top of the map viewer. The icon language is meaningful, and the user detects that the icons are from the same set. Important functions are hidden behind buttons with enough separating space between them and large enough for the thumb. They are all positioned on the right, so right-hand use is preferred.
-Mobile Design Conventions This application offers the same gestures as the bergfex.at app: zooming, panning, tilting and rotating the map. The whole design is based on post-WIMP interactions and speech input is accepted.
-Perception Channels Google Maps addresses the visual and the auditory channel: Visual representations are used in all map-related tasks, while audio guidance is implemented in the navigating software. A special feature is the AR implementation "Live View" in the navigation system: The user can hold the mobile device as if capturing a video (pointing the camera in the streets so that street signs are recognised) and gets to see the route directions and street names directly on the live image. This of course requires an active route and GPS signal, as well as internet connection.
-Use of Capabilities The app meets the capabilities of the mobile environment even more than others: It makes use of the web, the GPS sensor, the camera, microphone and the gyroscope (compass). In addition, the map material is of medium to high resolution and the app is not dependent on a stable data connection, it can also be used offline if the user stores map areas in advance.
-Adapting to Constraints The performance of the app is good and the app deals well with the reduced screen size, since the map viewer is not cluttered. During use, it became clear that some features, in particular the navigation features (normal mode and AR mode), drain the battery and use a lot of mobile data, especially the satellite imagery. To overcome environmental conditionsespecially the daytime -Google implemented a dark mode for safer navigation at night.
-Location Based Services The app incorporates plenty of location-based services: Based on the device's GPS position, the map pans automatically and the position is centred. The user gets suggestions for all sort of POIs (points of interest) in his proximity, which then can further be filtered. Google Maps shows "interesting" places in the local environment, it is possible to share the user's own location with other users (so they can see each other's location on the map), there is a highly-sophisticated navigation system implemented and Google stores a location history for every user.
-User Tasks Locator tasks are performed through determining the own position and looking up place names, but also through sharing the location with other users or querying attributes connected to different objects. Proximity tasks can be done for persons and objects (e.g. Who of the people I shared my location with is around here? Where is the next petrol station?). Navigation tasks are supported by the multimodal routing capabilities and events are displayed as well: traffic jams, barriers or other special occurrences. When using the navigation view, the current speed is continuously displayed (secondary positioning task). It is also possible to contribute to Google Maps: locations and images can be added, the map can be edited (which is afterwards verified) and own places can be stored for further use. Google Maps therefore also covers geocoding, crowdsourced-GI and data collection tasks.
-Functional Adaption This app is designed for providing location-based services to users that are mostly connected to the internet and for being used as a pedestrian's or cyclist's navigation system or instead of a built-in or external navigation system in

Comparing Cartography App Results
The comparison of the evaluation results is shown in Table  1 where Yes-No questions are completed.  Fig. 2 illustrates what the three tested applications represent in their core functionality: a topographic basemap, pedestrian navigation and high-quality basemaps and innovative routing. On the left, the digital ÖK 50 map is showing a section of Vienna. The middle part of Fig. 2 shows a pedestrian route in Graz, supported by another Austrian authoritative map product of high quality (basemap.at). The app's routing network is based on OSM data. On the right-hand side, Google Maps' beta AR routing functionality is shown: Using the device's camera and GPS sensor, the navigation symbols are directly drawn on the camera image.

Discussion
As demonstrated above, we were successful in creating a guideline which compiles the broad consent in literature of what good mobile map application design needs to fulfil into a set of criteria that can be used to evaluate mobile mapping apps in a descriptive and qualitative manner. It was also possible to make the guideline somewhat comparable. However, during application of the criteria, we determined room for improvement in our guideline which shall now be discussed. One issue that stood out were overlaps between some of the criteria. While it is natural for these overlaps to exist since most of the criteria are substantially connected (i.e. being able to fulfil user tasks and audience adaption), it also caused redundancies writing down the results of the evaluation. Further issues concern the comparison table. The inherent simplification of the results for a table creates the impression that the results of the evaluation might be quantifiable. While this might be true to a certain extent (one could for example replace the letters by numbers ranking -1 to +1 and sum them up), quantifying the results is inadvisable for multiple reasons. One reason is that not all criteria are of the same importance. While the questions were phrased in a way to always reflect the app's purpose, mobile design conventions are still not as important as the user being able to fulfil the task (they are, however, connected). Relatedly the importance of certain criteria might differ between apps. Another important point to note is that a full row of "Yes" in the table might convey the impression of a perfect app. While such an app would certainly fulfil all criteria there is always room for improvement. This issue is strengthened by the fact that the table also does not reflect reasons for any respective assessment and should therefore not be taken out of context with the textual evaluation. To further improve upon the guideline, the aforementioned flaws should be eliminated. The most obvious solution for the redundancy issue for example would be to try and phrase the criteria and their descriptions more sharply to reduce overlaps in content. While slightly flawed, the guideline was of much help getting insight into the design of the applications and provides a basis for recommendations of improvement. It turned out that Austrian Map mobile and Bergfex are two high-quality outdoor offline applications regarding data quality, but with different target audiences and app functionality. The second app might seem more convincing to users, since most of the Austrian Map's functions are also included in the cheaper Bergfex app. Google Maps however has its intentions mainly in online LBS and tries to push mobile cartography in a less "ubiquitous", but more innovative direction.

Conclusion
In this paper an attempt was made to create a guideline for the (self-)reflective evaluation of mobile mapping applications. To do so, subject literature of mobile cartography was studied to find a consent on characteristics a mobile mapping application should possess. From these characteristics, a set of criteria was derived and arranged to a guideline with which chosen applications from the field of mobile map services were subsequently evaluated. In order to find criteria for the guideline, firstly the field of mobile cartography has been defined as being "the technology, visualisation, transmission and usage of spatial data fitted for the special circumstances and capabilities of mobile devices as well as a mobile usageenvironment". Subsequently the field and its research areas "location-based services", "TeleCartography" and "adaptive cartography" were further elaborated. Reichenbacher's Conceptual Framework of Mobile Cartography was found to be an important indicator for evaluation criteria. The distinctiveness of mobile cartography (in comparison to traditional cartography), as well as the special capabilities and constraints of mobile devices and mobile design conventions have also been considered. As the first result of this paper a guideline has been introduced, consisting of 10 criteria: Basemap Quality, Ubiquitous Cartography, UX/UI, Mobile Design Conventions, Perception Channels, Use of Capabilities, Adapting to Constraints, Location Based Services, User Tasks, Functional Audience Adaption. Subsequently, the results of the evaluations have been presented. It was shown that, while most apps already had satisfying design, there was still room for improvement. Discussing the results, not only recommendations for improvements of the applications were made but also some flaws in the guideline presented by this paper have been pointed out for future correction.