Smart Screenshots – using image and text recognition to make screenshots more awesome.

The screenshots feature on iPhone, which allows people to take a snapshot of the contents of their screen, has gone unchanged since Apple introduced it seven years ago. The feature was never meant to be used by the average smartphone user, which is why Apple has never advertised it in their user manuals. But as people become more proficient in smartphone usage, the knowledge of its existence has spread largely by word of mouth. Now more and more people are relying on the feature for all kinds of tasks.


An opportunity to look into how people actually use screenshots.

Here's an opportunity to look into how people actually use screenshots and suggest better ways to support their behaviour. I asked 30 people either in person or through a survey to state the circumstances and contents of when they took their last screenshot.

    Examples of people's iPhone screenshots and the various intentional and unintentional purposes.

    Examples of people's iPhone screenshots and the various intentional and unintentional purposes.


    People take screenshots as part of a larger goal – not to fill their photostreams.

    It is likely that the reason why taking screenshots is used for the tasks below is that holding down two buttons is an interaction that is relatively easy to remember. Also the outcome of the action – all screenshots are stored in your photo album – is consistent. The problem is that taking a screenshot is seldom the main goal to be achieved, much rather it is a means to achieve a larger task. 


    The challenge is to better support people's goals without making screenshotting cumbersome.

    I looked at the way people hold their phones to press the home and lock buttons at the same time. From this it becomes obvious that ideally the suggested action ought to be placed towards the center top edge of the screen.


    The Solution: Context aware screenshots can suggest actions and shortcuts.

    The proposed solution is to suggest an action based on the context and content of the screenshot. The content of screenshots could be analysed using image and text recognition technologies. Additionally, the iOS knows which app is currently running on the users phone and would be able to infer from common usage patterns what actions to suggest. Because people use screenshots for very different purposes, these suggestions do not have to be acted upon – taking screenshots ought not get more complicated. Given these two findings, and the new interactive notifications in iOS 8, using a similar pattern makes sense. After taking a screenshot, the suggested action stays visible for a couple of seconds before disappearing if not acted upon.

    Necessary interactions to send a screenshot to a friend using the conventional methods in iOS7 (left) and using context-aware screenshots (right).

    Necessary interactions to send a screenshot to a friend using the conventional methods in iOS7 (left) and using context-aware screenshots (right).


    Sharing Conversations

    Giving people access to content  or snippets of conversations that they don't have access to.

    Messaging is what Lin does most on her phone. There's hardly an hour in the day where no message arrives that needs to be answered. Right now, Neil just texted her where and when his farewell party will take place. Lin is super excited to go, but wants her friend Vivian to tag along. To find out if Vivian will be able to join she takes a screenshot of the conversation and forwards it to her friend Vivian, hoping that they would still see it in time.

    People tend to take screenshots of conversations in order to forward them to one of their close friends. Here's an opportunity to eliminate a lot of interactions involved with messaging a screenshot.

    Ensuring Access

    Making sure that information can be accessed regardless of internet availability.

    Malin is excited about taking her friend out for dinner for a special occasion. They are staying in Switzerland for the first time and because she is unsure whether her phone will be able to look up the location once they are on the go, she takes a screenshot of the map showing the restaurant's location in the city. Just to be on the safe side.

    People tend to take screenshots of maps because they are unsure whether they will be able to access the internet on the go. This is a great opportunity to make that part available offline and have Maps suggest directions on the lock screen as soon as people get close.

    Quicker Access

    Making important information instantly available without needing to search for it.

    Lars always checks in online the day before his departure. Usually, he gets a confirmation email with his booking reference, seat number and QR code. Because he is not sure if he can access it at the airport and because he hates weeding through his inbox in the security queue he takes a screenshot of the confirmation email. He really likes that he can access the information a lot quicker when he needs it.

    People often take screenshots in order to access information quicker. In case of boarding passes, they could be converted immediately to Passbook* passes. Also optical text recognition could help identify other types of passes and store them in the Reminders app.

    Passbook keeps things like airline boarding passes all in one place (Source).


    Saving Inspiration

    Keeping track of interesting finds as a source of inspiration later on.

    The internet is such an amazing source of inspiration for Malin. She is always on the look out for things that will spark that next great idea. Often that comes from photographs or illustrations that have little to do with her work. But when it happens, she needs a quick and easy way to document that image. The easiest way to do so is to pinch, zoom and snap (take a screenshot) and later sort out the images on her computer - but honestly - that never really happens anyways.

    People take screenshots of illustrations and other content that cannot be saved another way. There is an opportunity to at least make sure they do not need to crop out the interface elements they don't want to have on their pictures afterwards and automatically add them to a 'inspiration' photo album.

    Accidental Usage

    Pictures of lock screens filling up people's photo streams.

    Looking through the photos on his phone, Lars points out that he seems to have a special talent to take screenshots of his wallpaper. They just seem to happen somehow when his phone is in his pocket. Or when he tries to unlock the phone and is not paying attention to what he is doing. He usually deletes them when he notices it happening - but that doesn't happen very often.

    One of the major annoyances of screenshots are that they are often triggered accidentally. In this case a smart screenshot feature could make sure to prevent duplicates from happening and require an additional confirmation gesture.

    Client Personal Project
    Year Summer 2014
    Skills Interaction Design

    Kevin Gaunt is an Interaction Design student at Umeå Institute of Design Masters Programme – around 200 kilometres from the arctic circle. Kevin is currently on a study break to intern at design companies across the globe. He'd be happy to hear from you.