Smart Screenshots – using image and text recognition to make screenshots more awesome.

The screenshots feature on iPhone, which allows people to take a snapshot of the contents of their screen, has gone unchanged since Apple introduced it seven years ago. The feature was never meant to be used by the average smartphone user, which is why Apple has never advertised it in their user manuals. But as people become more proficient in smartphone usage, the knowledge of its existence has spread largely by word of mouth. Now more and more people are relying on the feature for all kinds of tasks.

 
 
 
hero_2x_iphone.png
 
 
 

An opportunity to look into how people actually use screenshots.

Here's an opportunity to look into how people actually use screenshots and suggest better ways to support their behaviour. I asked 30 people either in person or through a survey to state the circumstances and contents of when they took their last screenshot.

     
     
     
    Examples of people's iPhone screenshots and the various intentional and unintentional purposes.

    Examples of people's iPhone screenshots and the various intentional and unintentional purposes.

     
     
     

    People take screenshots as part of a larger goal – not to fill their photostreams.

    It is likely that the reason why taking screenshots is used for the tasks below is that holding down two buttons is an interaction that is relatively easy to remember. Also the outcome of the action – all screenshots are stored in your photo album – is consistent. The problem is that taking a screenshot is seldom the main goal to be achieved, much rather it is a means to achieve a larger task. 

     
     
     
    3.jpg

    The challenge is to better support people's goals without making screenshotting cumbersome.

    I looked at the way people hold their phones to press the home and lock buttons at the same time. From this it becomes obvious that ideally the suggested action ought to be placed towards the center top edge of the screen.

     
     
     

    The Solution: Context aware screenshots can suggest actions and shortcuts.

    The proposed solution is to suggest an action based on the context and content of the screenshot. The content of screenshots could be analysed using image and text recognition technologies. Additionally, the iOS knows which app is currently running on the users phone and would be able to infer from common usage patterns what actions to suggest. Because people use screenshots for very different purposes, these suggestions do not have to be acted upon – taking screenshots ought not get more complicated. Given these two findings, and the new interactive notifications in iOS 8, using a similar pattern makes sense. After taking a screenshot, the suggested action stays visible for a couple of seconds before disappearing if not acted upon.

     
     
     
    Necessary interactions to send a screenshot to a friend using the conventional methods in iOS7 (left) and using context-aware screenshots (right).

    Necessary interactions to send a screenshot to a friend using the conventional methods in iOS7 (left) and using context-aware screenshots (right).

     
     
    conversation-nugget-animated.gif
     

    Sharing Conversations

    Giving people access to content  or snippets of conversations that they don't have access to.

    Messaging is what Lin does most on her phone. There's hardly an hour in the day where no message arrives that needs to be answered. Right now, Neil just texted her where and when his farewell party will take place. Lin is super excited to go, but wants her friend Vivian to tag along. To find out if Vivian will be able to join she takes a screenshot of the conversation and forwards it to her friend Vivian, hoping that they would still see it in time.

    People tend to take screenshots of conversations in order to forward them to one of their close friends. Here's an opportunity to eliminate a lot of interactions involved with messaging a screenshot.
     
     
     
     

    Ensuring Access

    Making sure that information can be accessed regardless of internet availability.

    Malin is excited about taking her friend out for dinner for a special occasion. They are staying in Switzerland for the first time and because she is unsure whether her phone will be able to look up the location once they are on the go, she takes a screenshot of the map showing the restaurant's location in the city. Just to be on the safe side.

    People tend to take screenshots of maps because they are unsure whether they will be able to access the internet on the go. This is a great opportunity to make that part available offline and have Maps suggest directions on the lock screen as soon as people get close.
    screenshot-map-animated.gif
     
    convert-passcode-animated.gif
     

    Quicker Access

    Making important information instantly available without needing to search for it.

    Lars always checks in online the day before his departure. Usually, he gets a confirmation email with his booking reference, seat number and QR code. Because he is not sure if he can access it at the airport and because he hates weeding through his inbox in the security queue he takes a screenshot of the confirmation email. He really likes that he can access the information a lot quicker when he needs it.

    People often take screenshots in order to access information quicker. In case of boarding passes, they could be converted immediately to Passbook* passes. Also optical text recognition could help identify other types of passes and store them in the Reminders app.

    Passbook keeps things like airline boarding passes all in one place (Source).

     
     
     
     

    Saving Inspiration

    Keeping track of interesting finds as a source of inspiration later on.

    The internet is such an amazing source of inspiration for Malin. She is always on the look out for things that will spark that next great idea. Often that comes from photographs or illustrations that have little to do with her work. But when it happens, she needs a quick and easy way to document that image. The easiest way to do so is to pinch, zoom and snap (take a screenshot) and later sort out the images on her computer - but honestly - that never really happens anyways.

    People take screenshots of illustrations and other content that cannot be saved another way. There is an opportunity to at least make sure they do not need to crop out the interface elements they don't want to have on their pictures afterwards and automatically add them to a 'inspiration' photo album.
    hero-image-saving-inspiration.gif
     
    hero-image-avoid-accidental-usage.gif
     

    Accidental Usage

    Pictures of lock screens filling up people's photo streams.

    Looking through the photos on his phone, Lars points out that he seems to have a special talent to take screenshots of his wallpaper. They just seem to happen somehow when his phone is in his pocket. Or when he tries to unlock the phone and is not paying attention to what he is doing. He usually deletes them when he notices it happening - but that doesn't happen very often.

    One of the major annoyances of screenshots are that they are often triggered accidentally. In this case a smart screenshot feature could make sure to prevent duplicates from happening and require an additional confirmation gesture.
     
     
     

    Client Personal Project
    Year Summer 2014
    Skills Interaction Design

    Kevin Gaunt is an Interaction Design student at Umeå Institute of Design Masters Programme – around 200 kilometres from the arctic circle. Kevin is currently on a study break to intern at design companies across the globe. He'd be happy to hear from you.


     

     

     

    portfolio-thumnail-image-chi.jpg

    CHI'14 Student Design Competition: A Critical View on Quantified Self

    With the goal of learning more about ourselves we are starting to build sensors into all of our possessions, but what happens when we start quantifying those that cannot decide for themselves?

    Update: Our paper was accepted (acceptance rate 12.5%) and we are looking forward to presenting our concept in Toronto this April.

    Competition CHI'14 Student Design Competition
    Project Type Extracurricular
    Year Fall 2013
    Skills Interviewing, Research, Critical Thinking, Brainstorming, Academic Writing
    Team Members Marcel Penz, Júlia Nacsa

    The call for participation of the CHI 2014 Student Design Competition asked students to concern themselves with "BodyData: Designing for Qualities of the Quantified Self". In an extra-curricular collaboration with Julia Nacsa and Marcel Penz we explored both our own and our peers' attitudes towards the Quantified Self. To spur the discussion we designed realistic a future vision of a quantified baby product that we named Baby Lucent. Our future vision was based on existing or in-development products and plays on parental worries identified in interviews with parents in Umeå. Baby Lucent is our critical interpretation of how a Quantified Baby product might look in the near future and is aimed at first-time parents and allows them to reflect upon and improve their parenting behavior based on real data. In the following discussion we then showed our designs to 14 young designers. Their initial enthusiasm slowly gave way to more critical thought regarding the purpose and consequences of such products.  Finally, we concerned ourselves with how Quantified Baby ought to be designed to avoid the three biggest pitfalls: Increasing the distance between parent and child, increasing parental anxiety and inhibiting parental intuition.

    pacifier.103-only.jpg

    The Smart Pacifier contains sensors arranged as pores on the mouthpiece to analyze the infant’s saliva for bacteria levels and body temperatures. The pacifier periodically takes new readings and sends it to the Baby Lucent Internet service.

    pacifier.103-feedingbottle.jpg

    The Smart Feeding Bottle measures both the infant’s food intake for its nutritional values. The idea being that parents not only can see the optimum feeding quantities and time, but also can even reflect on the mother’s behavior and diet if the feeding bottle is filled with breast milk.

    app.jpg

    The Smartphone Application acts as the primary interface for the parent. They can identify their child’s likelihood of picking up infection in areas close by (e.g. the playground around the corner), the optimal time to feed the baby, a comparison of actual and recommended nutrient levels of the baby’s food.

    DSC00610.jpeg

    Smile Bar – Inviting People to Smile More

    Umeå's winters are cold and dark. The Smile Bar is an interactive installation for the Umeå Institute of Design that highlights the benefits of smiling and encourages students to smile more. Research shows that the act of smiling is enough to make us feel happy, a particular reason to do so is not that important.  The Smile Bar is a different take on a photo booth – reducing it to just taking a picture of their mouth. After a smile was capture the Smile Bar prints a fun receipt containing funny statistics for the student to keep.

    School Umeå Institute of Design
    Class Experience Prototyping –  Rapid Prototyping for Data Driven Experiences
    Tutors Kaveh Shirdel & Friedrich Foerstner (DesignIt Munich)
    Year Fall 2013 (4 days)
    Skills Low-Fi Prototyping, Physical Prototyping (Arduino), User Testing, Processing, Lean Business Canvas, Elevator Pitch
    Team Members Doris Feurstein, Shelagh McLellan, Linus Persson

    The Challenge
    The challenge was to come up with a fun and inviting product in four days that fits into the category 'Health'. We were encouraged to iterate quickly and to build multiple prototypes.

    The Result
    Motivated to fail early and often we came up with multiple low-fi prototypes and tested them with people around school until we arrived at the Smile Bar concept in our third iteration. Smile Bar works similarly to a interactive Photo Booth but exclusively focusses on the subjects' smiles. All smile photos are projected onto the wall behind the installation, inviting people to try smile bar for themselves. It consists of a podium, a webcam, a huge button, a mac mini, an Arduino board, a thermal printer and a projector.

    My Contribution
    Most of the project was done in close collaboration with all team members. We all collectively brainstormed, ideated, built simple prototypes out of paper and tested them with people in school. On the last day before the presentation amongst other things I wrote the processing code that displayed all the smile photographs and tutored Shelagh on how to get the thermal printer to print using Arduino.


    DSC01934.JPG
     

    The challenge was to design a multimodal interface that supports drivers of highly automated truck convoys in 2030.

    One of the main challenges was how to combine both the truckers' needs and values of 2013 with the future scenarios of 2030. We were trying to come up with a highly usable design concept and simultaneously faced the issue of grounding the scenarios in reality. 

    Given these challenges we wanted to humanize the automation technology within the truck, making it a trustworthy companion on the truck operators journey. We also wanted to give technology a clear point of reference within the truck and finally aimed to make sure truck drivers remain aware of the automation's actions and decisions at all time.

     
     

    The solution is the heart of automation: a skin-conductive steering wheel, unobtrusive voice control and a context-aware heads-up display.

    The steering wheel has been used to interact with other drivers with a honk of the horn. The horn's original purpose was to be used in cases of impending danger, however it's function has inevitably broadened beyond a single message. How might the truck operator communicate with others on the highway in 2030? Introducing the heart of Scania.

    The primary output will happen on the trucks windshield though an advanced heads up display. These types of displays have significant safety benefits associated with heads up displays. They make sure that the relevant information is directly in the drivers view and minimize eye movement. Given that when trucks are platooning, most of their field of vision is severely limited by the back of the truck driving in front, we can make use of this dead space for our heads-up display information (see screens in storyboard). Additionally, we will use both ambient displays and sound feedback. We envision the usage of different elements in the drivers' cabin that can subtly change colour to indicate the difference in the automation status. This has the benefit that we can reduce the clutter displayed on the heads up display. Audio feedback can be utilized to help operators understand the necessity to address their attention to the system interface. 

     
    Heads–up display interface when the truck driver is actually in full manual control of his truck. The heads–up control only shows the most essential elements.

    Heads–up display interface when the truck driver is actually in full manual control of his truck. The heads–up control only shows the most essential elements.

    The heads–up display interface when the driver has joined a platoon. Now the interface elements take up more of the windshield as not his full attention is required.

    The heads–up display interface when the driver has joined a platoon. Now the interface elements take up more of the windshield as not his full attention is required.

    In full-automation mode (while being part of a truck convoy) the drivers interface strongly resembles a entertainment system. This is possible, because the lead platooner is responsible for the safety of all drivers.

    In full-automation mode (while being part of a truck convoy) the drivers interface strongly resembles a entertainment system. This is possible, because the lead platooner is responsible for the safety of all drivers.

     
    Press Down – confirm actions or suggestions made by automation system. Possibly problematic as action associated with the car horn.

    Press Down – confirm actions or suggestions made by automation system. Possibly problematic as action associated with the car horn.

    Hold – hold steering wheel centre to start voice command inputs. Skin conducting material could make this possible.

    Hold – hold steering wheel centre to start voice command inputs. Skin conducting material could make this possible.

    Retract Steering Wheel – unlock and retract steering wheel to give up manual control of the truck. Pull back to regain control.

    Retract Steering Wheel – unlock and retract steering wheel to give up manual control of the truck. Pull back to regain control.

    Point to Select – a 3D trackpad enables you select items from the choices displayed on the heads up display.

    Point to Select – a 3D trackpad enables you select items from the choices displayed on the heads up display.

     

    sfasf

     

    The process involved finding a way to design for truck drivers operating with technology that does not exist yet.

    It became clear to us that for automated platooning to be adopted by the truck driver it needs to respect the driver’s fundamental values. To determine these values and to determine what characteristics truck drivers share, we created a set of behavioural variables based on the findings of our contextual inquiry. The other teams then added their input to allow us to compare the characteristics of 6 different truck drivers. This contextual inquiry made us realise that there is no archetypical trucker. We also found that most of our interviewees described the concept of “Freedom” in their job that they liked. The data that we managed to collect let us then build a realistic truck driver persona. To understand what situation our persona faces throughout a workday in 2030 we created a user journey. With our user journey we visualised the context, touch-points and both the users and systems actions at during each important event. This gave us seven relevant scenarios to work with.

    To be able to decide on interactions that match truck operators expectations we decided to put ourselves in the situation and act out each interaction (bodystorming). We used simple props made out of elements we could find around school. This gave us the following two main insights: Firstly, it became obvious how important the human communication between truck drivers is. This finding is backed up by our user research. Secondly, there was an explicit need to see each other's actions. We therefore decided on defining the lead platooner’s role aspirational and full of responsibility. 

    We unconsciously went into the user research with a trucker stereotype, which we quickly found not to be true at all.

    We then moved forward to storyboarding our design solutions through storytelling. We picked the two most important scenarios to start with: joining a platoon and exiting a platoon. We separated and came up with four different storyboards that each conveyed different design interventions. Using these results we discussed the advantages and shortcomings of each intervention and summarized what worked for each. Collectively, we then scripted our story and produced a low-fi version of the storyboards for both main scenarios. This forced us to think about all interfaces involved and forced us come up with preliminary sketches for the interfaces. We then focussed on detailing the storyboards and thereby the in-truck interfaces. 

    Final Keynote Presentation.033.jpg

    The main lesson learned was to find a way where augmented reality feels true to me, in that it is used to visualise things that are actually invisible to our eyes.

    Things I have learned

    • a way of designing for future concepts
    • importance of a positive team spirit throughout the project
    • visualising is often better than talking
    • user interviews can be very exhausting and need a clear goal
    • working with people with different strengths is a lot of fun
    • patent considerations can take a long time

    Things I would do differently

    • engage in more co-design activities rather than talking for extended periods of time
    • talk about each others expectations of the projects and agreeing on a group contract from the very beginning 
    • keep the production values simpler, After Effects can be a huge sinkhole
    • find better ways of recapping user research