T 2630/17 (Feedback on gesture input/APPLE) 28-05-2019

The invention

1. The application relates to a gesture-based procedure for unlocking the touch screen of a portable computing device such as a mobile phone - or, more generally for "transitioning [...] between" first and second "user interface states" (see paragraphs 2-6) - and to providing visual feedback (referred to as "sensory" in paragraph 6) for supporting the user in the process.

1.1 A typical interface according to the invention is depicted in figures 4A-B and 5A-C. To unlock the device, the user "drags" an "unlock image" (402) along a predefined path (404). This mechanism has become known as "slide-to-unlock".

1.2 The application discusses only one specific input gesture explicitly, namely a linear swipe. Paragraph 55 hints at other gestures also being possible by stating that the "horizontal movement" is merely an "example".

1.3 It is proposed to give "sensory feedback of the progress" of the required gesture by "transitio­ning an optical intensity of one or more user interface objects associated with the second user interface" (see paragraph 82). The "optical intensity of a user-inter­face object" is said to be "the object's degree of visu­al materialization" in a broad sense, subsuming transparency or brightness effects or other means of "visual diffe­ren­tiation" (see paragraphs 85-87 and figures 8A-8C).

1.4 The following typical situation is described (see paragraph 91 and figures 7A-7D). A mobile phone is initially locked, displaying the unlock image (702, 704), when a prompt on the screen (706) indicates an incoming call. The user then starts dragging the unlock image in order to unlock the device so as to be able to accept or decline the call. Once the unlock gesture has started, the interface displays "a set of virtual buttons" (see paragraph 92 and figure 7C; 708), the optical intensity of which increases as the user drags the unlock image. The user cannot activate these buttons - in particular to decline or accept the incoming call - until the device has been unlocked (see paragraphs 92 and 93).

1.5 The application also discloses that several unlock images may be displayed, each of which being associated with an individual application such as a messaging or a music application (see paragraphs 105-107 and figures 11A-11F).

The prior art

2. D8 is a user's guide to the "Neonode" mobile phone. In order to unlock the device, the user must press the power button once and is then instructed to "Right sweep to unlock" on the touch screen (see page 8, last paragraph, page 9, "Keylock - Unlocking the unit", and page 11).

3. D6 discusses alternative designs for toggle switches on touch screen interfaces. Inter alia it discloses a slider toggle (see figure 2, bottom left) which the user can "grab" and "slide" to the other side (see page 668, left column, paragraph 5), and that the slider "lever or pointer should highlight when touched to signify that the user now has control over it" (page 668, right column, lines 8-10).

Inventive step, Article 56 EPC

4. The examining division started its inventive step assessment from D8, and both the board and the appellant agree with this choice.

5. D8 discloses the subject-matter of the preamble and the last two lines of claim 1.

5.1 D8 does not disclose anything being displayed on the screen in response to the "right sweep".

5.2 More specifically, D8 does not disclose that progress of the gesture input is indicated by "transitioning an optical intensity" of "appearing" user interface objects "associated with" the second user-interface state.

5.3 These differences are essentially the same for all requests, although the auxiliary requests qualify some of them. In particular:

Auxiliary request 1 specifies that the optical intensity changes "continuously with progress of the gesture input", auxiliary request 3 that the increase is "smooth[] at a predefined rate in accordance with completion of the gesture" and auxiliary request 4 that the "transitioning" takes place "in accordance with progress towards completion of the gesture input".

Auxiliary request 2 specifies that the user can interact with the claimed user-interface objects in the second user-interface state and that they are not displayed prior to the gesture input, auxiliary request 6 states that the user-interface objects "are not included in the first user-interface state" and auxiliary request 5 specifies that the user-interface states differ in how the electronic device responds to user input.

5.4 In the following analysis, these qualifications are all considered in combination. That is, it is assumed that the optical intensity is transitioned smoothly accor­ding to auxiliary request 3 (which is more specific than the corresponding feature in auxiliary requests 1 and 4) and that the user-interface objects pertain to the second user-interface state, appear on gesture input and have no function in the first user-interface state (as claimed in auxiliary requests 2, 5 and 6).

5.5 In view of the board's conclusion that this combination of features lacks inventive step, it can be concluded that claim 1 of each of the pending requests lacks inventive step, as they specify the invention in more general terms.

6. The examining division found that the differences over the then pending request addressed the problem of "how to provide visual feedback to a user about the [ongoing] transition" from one to another interface state (see reasons 16.5 and 18.5). The appellant proposed to consider a similar, if slightly more specific, formulation, mentioning that the feedback not only related to an "ongoing transition" but also indicated the "progress of [the] transition" (see grounds of appeal, page 3, paragraph 4 from the bottom).

6.1 The examining division further took the view that the skilled person, in order to solve the problem posed, would have considered combining the teachings of D8 and D6. The board agrees that the skilled person, seeking to provide feedback on the unlocking gesture of D8, would have considered the recommendations of D6 on how to display toggle buttons or switches.

6.2 The appellant stated that "In seeking to address this problem the skilled person would not (indeed could not) combine the teaching of D8 with that of D6 to arrive at the claimed subject matter" (see grounds of appeal, page 3, paragraph 3 from the bottom). However, as the board understands the appellant's subsequent discussion (see in particular the grounds of appeal, page 5, first sentence and paragraph 4), it does not dispute that the skilled person would consider a combination of D8 and D6, but merely that, when doing so, the skilled person would have arrived at the claimed invention.

6.3 The examining division interpreted the teaching of D6 to highlight the slider lever or pointer when touched as disclosing an element appearing on the screen and, ipso facto, increasing in optical intensity (see reasons 16.7, first paragraph). The independent claims of the main request thus lacked inventive step over D8 in view of D6 (see the decision, reasons 16.7, paragraphs 2 and 3).

6.4 The board does not understand the "highlighting" according to D6 in the sense of a newly appearing user interface object. Page 668, left column, lines 20 to 22, mentions the slider having a three-step animation of a "yellow pointer", which the board understands to mean that the pointer can only be in the left-, middle- or right-position and is always yellow. The board understands page 668, right column, lines 5 to 10, in the sense that the (yellow) slider is always visible but becomes "more visible", i.e. is "highlighted", when touched by the user. The board understands "highlighting" in this context as "making more visible", which covers a change in colour as well as a change in intensity.

6.5 Nonetheless, highlighting (in the sense of making more visible) an object on a user interface is an obvious option. For example, a text object "ON" might be made to appear on the slider during the input gesture.

6.6 The independent claims require the newly appearing user interface object to "appear and increas[e] in optical intensity". In the board's judgment, these are two different require­ments, whereas the examining division argued that the optical intensity of the user interface object was zero before it appeared and increased when it appeared.

6.7 The board disagrees with this interpretation, in particular because claim 1 also requires that the "transitioning of the optical intensity indicat[e] [...] progress of the gesture input". Any highlighted "element" according to D6, however, would be present at the start of the input gesture and not change during its progress and thus could not serve as an indicator of progress.

6.8 The board therefore concludes that neither D8 nor D6 discloses or suggests that a user interface element appears at the start of the gesture input and increases in optical intensity to provide "feedback" about its progress.

7. According to established jurisprudence of the boards of appeal, it remains to be assessed whether the difference features (see point 5.4 above) solve a technical problem or, instead, merely "refer[] to an aim to be achieved in a non-technical field" (see, in particular, T 641/00, headnote 2). In the latter case, the difference features would not support an inventive step (see T 641/00, headnote 1), even if they happened to be non-obvious over the prior art.

8. The appellant suggested that the optical intensity display by the graphical user interface (GUI) provided information about the internal state of the gesture recognition engine.

8.1 In T 528/07, several board of appeal decisions relating to this argument are discussed and it is concluded that "giving visual indications [...] about conditions prevailing in an apparatus or system" can only be considered a technical problem if these are technical conditions (see, in particular, reasons 3.3 and 3.5). That decision found, for example, that "data relating to a business undertaking" was not a technical condition of a computer program running business software (see reasons 3.6).

8.2 In agreement with T 528/07, the board considers that not every internal state variable in a running computer can be considered a "technical condition" prevailing in the computer and that, therefore, the display of an internal state variable of a running computer program cannot, by itself, establish that a technical problem is solved.

9. The examining division, applying the principles set out in T 336/14, found that indicating the progress of gesture input "assist[ed] the user in performing the technical task of switching user interface states by means of a continued and guided human-machine interaction process" (see the decision, reasons 18.4.1, 2 and 4).

9.1 The board agrees that decision T 336/14 provides useful hints for assessing whether or not features of a GUI go beyond presenting infor­ma­tion to the human user or providing aesthetic effects by having a technical effect (and therefore may contribute to inventive step).

9.2 T 336/14 suggests in its headnote that it has to be analysed "whether the GUI together with the content presented credibly assists the user in performing a technical task [...] by means of a continued and/or guided human-machine interaction process". The board notes in this regard that the second clause of this sentence (beginning with "by means") cannot be ignored. A (graphical) user interface may well assist the user in solving a technical task with non-technical means, such as the means of advising the user to take a break (see also T 1741/08, reasons 2.1.12).

10. In the present case, the board considers gesture input on a touch screen to be a technical process, in that it implies the tracking of the user's finger position to identify a gesture and the comparison of that gesture with a required, reference gesture.

10.1 The board also accepts that the claimed graphical user interface provides a form of "feedback" about a "continued [...] human-machine interaction process" but considers that this feedback does not guide the user or, more importantly, assist the user in inputting the gesture correctly. The feedback does not, for instance, indicate an error between the gesture being input and the reference gesture.

10.2 According to the claimed invention, users are merely continuously informed as to whether they have input the required gesture correctly so far. That is, as long as they observe a change in optical intensity while gesturing they will understand that the partial gesture input so far has been correct.

10.3 This may reassure (inexperienced) users about what they are doing but does not help them decide what to do next. The board considers that providing reassuring feedback - i.e. a confirmation that the user has been doing the right thing so far - is not in itself a technical effect. For illustration, the board notes that a user who knows the required gesture, in particular if it is a short, simple swipe as required by D8, may decide to ignore the graphical feedback altogether without any loss in input precision or speed. In other words, the user need not heed the feedback in order to successfully enter the required gesture.

11. The appellant argued that the user would notice when the optical intensity stopped changing and understand this to mean that the gesture input had failed at that point. The user might then decide to stop inputting the gesture and to start again. Saving the user time in this manner was a technical effect.

11.1 The board does not disagree in principle that this might be a possible technical effect. However, the board is not persuaded by this argument for three reasons.

11.2 Firstly, the description does not discuss or allude to this effect.

11.3 Secondly, the claimed "transitioning an optical intensity" need not, in general, be a monotonic increase. For instance, the optical intensity might increase in a few discrete steps between which no change occurs. Hence, the lack of such a change could, in such a situation, not be interpreted as an indication that the gesture input had failed. Only claim 1 of auxiliary request 2 might allow the user to draw this conclusion, as it requires a smooth increase at a predefined rate in accordance with completion of the gesture input, which, in the board's judgment, the skilled person would take to imply a monotonically increasing intensity in view of figures 8A to 8C.

11.4 Thirdly, the claims do not indicate how the user could recognise that the gesture input was complete. The transition itself between the user-interface states may not be visible to the user, and the level of optical intensity obtained after complete and successful gesture input (such as, maybe, zero transparency; cf. paragraph 85 of the description) is not defined in the claim, nor disclosed in the application. Hence, as the claims stand, the user might have to interpret the optical intensity ceasing to change as an indication that the gesture was complete and correct. The same GUI response would then no longer be available to uniquely indicate an erroneous input, as suggested by the appellant.

11.5 The board thus concludes that this effect, of saving the user time by indicating that a gesture input error has occurred, cannot be ascribed to the claimed invention in its full breadth.

12. The appellant also argued that the optical intensity display by the GUI allowed the user to see whether the gesture recognition was working properly.

12.1 A priori, the board accepts that the optical intensity display may serve two purposes. As is discussed in the application, the user might receive - and appreciate - continuous, reassuring feedback on whether the gesture has been input correctly so far. The scenario implied by the appellant is that the user might receive feedback from the gesture recognition engine as to whether it is working properly and has recognized a gesture correctly. In the board's understanding, this is a testing scenario in which the user is a developer who knows without doubt that the input gesture is correct.

12.2 The board has discussed the first scenario above and found that it did not help the user solve a technical problem. The second option, however, is entirely without basis in the application as filed. The application only refers to the situation in which a user would assume the gesture recognition engine to be working properly, rather than wanting to establish whether it does. Also, the claims are not limited to the testing or debugging scenario, and cannot be (within the limits of Article 123(2) EPC) because the application does not disclose this limitation.

12.3 Therefore, the board concludes that the claimed visual feedback cannot be accepted as solving a problem in a testing or debugging situation. It may be left open whether, if it did, some such improvements would be technical.

13. Finally, the appellant argued that displaying user-interface elements pertaining to the second user-interface state simplified the GUI because it informed the user about what the gesture being input was about to achieve. If users were to inadvertently input a gesture leading to a "second" user-interface state, although a "third" such state was actually desired, they would notice and immediately stop inputting the wrong gesture.

13.1 As before, the board points out that this scenario is entirely without basis in the application. Although the application discloses the existence of alternative unlock images depending on the action the user might want to take after unlocking the device (see e.g. figures 7D and 11D), the application consistently discusses a single linear unlock action, rather than one of several gestures leading to one of several user-interface states, respectively.

13.2 Also, the claims do not set out a "third" user-inter­face state or a choice between two target states that the user might want to make. The claimed invention and the Neonode device known from D8 both have in common the fact that the swiping gesture is required before the user can use the device in any way. The situation disclosed in D8 is inconsistent with the idea that the user might inadvertently input a swiping gesture and might not, actually, want to unlock the mobile phone but rather want to do something else with it.

13.3 As a consequence, the board also does not accept that the claimed invention has this effect.

14. Summarizing the above, the appellant could not persuade the board that the characterising GUI features help users to solve a technical problem.

14.1 The board concludes that, starting from D8, the characterizing features merely animate the gesture input of D8 in an aesthetically or otherwise pleasing manner and do not have a technical effect. Following established jurisprudence (see, again, T 641/00, headnote 2), such features may legitimately appear in the problem to be solved vis-à-vis D8, in particular as a "constraint that ha[s] to be met".

14.2 More specifically, the problem solved by the claimed invention is to provide an implementation of the features listed under point 5.4.

14.3 The implementation being straightforward, the board concludes that the claimed invention of all requests lacks inventive step, Article 56 EPC.