T 1403/20 (Ink to text conversion/MICROSOFT) 17-11-2022
Download and more information:
Ink to text representation conversion
I. The appellant (applicant) appealed against the decision of the examining division refusing European patent application No. 14730385.3, which was published as international application WO 2014/189812.
II. The contested decision cited, inter alia, the following documents:
D1:|WO 2005/029391 A1, 31 March 2005;|
D5:|G. Hardock et al.: "A marking based interface for collaborative writing", Proceedings of the 6th Annual ACM Symposium on User Interface Software and Technology (UIST '93), December 1993, pp. 259-266;|
D9:|US 2008/0178126 A1, 24 July 2008.|
The examining division decided that the subject-matter of independent claims 1 and 7 of both the main request and the auxiliary request lacked inventive step over document D1. The reasoning for the decision also makes reference to documents D5 and D9.
III. In its statement of grounds of appeal, the appellant maintained the requests considered in the contested decision. The appellant requested that the decision under appeal be set aside and that a patent be granted on the basis of the claims of the main request or, in the alternative, of the auxiliary request.
IV. In the communication accompanying the summons to oral proceedings, the board expressed the preliminary view that claim 1 of the main request was not clear and that the subject-matter of claim 1 of both requests lacked inventive step both over document D1 and over document D5.
V. With a letter filed in preparation for the oral proceedings, the appellant maintained its pending requests as main request and auxiliary request 1 and filed auxiliary requests 2 to 5.
VI. During the oral proceedings, which were held on 17 November 2022, the appellant made auxiliary request 3 its new main request, maintained auxiliary request 1 and withdrew its other requests. At the end of the oral proceedings, the Chair announced the board's decision.
VII. The appellant's final requests were that the decision under appeal be set aside and that a patent be granted on the basis of the claims of the new main request or, in the alternative, of auxiliary request 1.
VIII. Claim 1 of the main request reads as follows:
"A method executed on a computing device for converting handwritten ink input to text representations, the method comprising:
detecting an action to provide ink input on content (106) viewed on a computing device;
analyzing the ink input;
processing the ink input to differentiate a comment (214), content (204, 208, 210) to be inserted into the document and an instruction (217) for an action intended to be performed on the content;
when the ink input is detected as a comment (214) or as content (204, 208, 210) to be inserted into the document,
converting the ink input to a text annotation, and
displaying (750) the text annotation in an annotation view; and
when the ink input is detected as an instruction (217) to make a change to the content,
matching the instruction to an action by searching a data store storing instructions and associated actions,
converting the instruction to an associated action, and
applying the associated action to a portion of the content."
IX. Claim 1 of auxiliary request 1 reads as follows:
"A method executed on a computing device for converting handwritten ink input to text representations, the method comprising:
detecting an action to provide ink input on content (106) viewed on a computing device;
analyzing the ink input;
converting the ink input to a suggested text annotation based on a context of the ink input on the content and a grammar and proofreading analysis;
enabling the user to accept or reject the suggested text annotation; and
displaying (750) the suggested text annotation in an annotation view."
X. The appellant's arguments, where relevant to this decision, are discussed in detail below.
1. The application relates to converting handwritten text input to text annotations and is illustrated by Figure 2:
FORMULA/TABLE/GRAPHIC
The left-hand side of Figure 2 shows a tablet computing device displaying text ("content") to which the user adds handwritten annotations, referred to in the application as "ink input" (see paragraphs [0023] and [0024] of the international publication).
The handwritten annotations include annotations:
- to add text (204, 210, 216);
- to delete text (208, 218);
- to underline text (206, 213);
- to add a comment (214);
- to give an editing instruction (217).
The handwritten annotations are converted into text annotations and integrated into the "annotation view" 220 on the right-hand side of Figure 2, for example when the user selects the "Convert Ink" conversion button 240 (paragraph [0032]). The two views can also be shown side-by-side (see Figure 4 of the application).
Main request
2. Admission into the appeal proceedings
2.1 The main request was filed, as a third auxiliary request, after the notification of the board's summons to oral proceedings and represents a reasonable reaction to clarity and added-matter objections which were raised for the first time in the board's communication accompanying the summons to oral proceedings.
2.2 The admission of this request into the appeal proceedings is therefore justified by an exceptional circumstance, as required by Article 13(2) RPBA 2020.
3. Inventive step
3.1 Document D5 discloses a collaborative stylus-based annotation tool referred to as "MATE" (see page 259, left-hand column, lines 2 to 7 and 19 to 25; page 260, left-hand column, lines 23 to 30).
3.2 The MATE tool functions in three modes: edit mode, annotation mode and incorporation mode (page 260, left-hand column, lines 24 to 26).
In the edit mode, ink input is processed immediately upon the user lifting the stylus (Figure 2 and its caption). As a prototype, the tool supports delete, move and insert commands (page 260, left-hand column, lines 28 to 32).
The annotation mode is intended to allow collaborators of the primary author of the document to review the document. In this mode, ink input entered by a collaborator is treated as handwritten annotations (page 260, right-hand column, lines 4 to 14, and Figure 3).
In the incorporation mode, the primary author performs the actual revision. In this mode, the MATE tool displays two views of the document, as shown in Figure 4:
FORMULA/TABLE/GRAPHIC
The left-hand ("annotation") view displays the markup document received from the collaborators with the annotations made and allows additional annotations to be added (page 260, right-hand column, lines 18 to 22). The right-hand ("edit") view shows the current version of the document. Handwritten annotations added to this view are immediately processed as in the edit mode (page 260, right-hand column, lines 22 to 26).
The incorporation mode allows the user to point to an annotation in the left-hand view and ask the system to perform it, resulting in a change to the document shown in the right-hand view (page 260, right-hand column, lines 27 to 29). As explained in the caption of Figure 4, the annotations in the annotation view that have not yet been executed appear as thick lines (the "Reword" comment annotation and the annotation indicating that "that" is to be moved to the position after "some"), whereas those which have been executed appear as thin lines (the insertion of "ed" and the deletion of "specific").
3.3 The edit view, not the annotation view, of document D5 corresponds to the annotation view of the application, as it displays the converted text annotations integrated into the displayed content in the same manner as annotation view 220 of Figure 2 of the application (see point 1. above).
3.4 In document D5, the ink input can represent an "insert" command specifying content to be inserted into the document (page 260, left-hand column, lines 28 to 32; Figure 4, the annotation of the word "mark"). However, as the typed letters "ed" in the left-hand view of Figure 4 suggest, text to be inserted has to be entered by keyboard (page 260, left-hand column, line 32).
3.5 The ink input can also represent a "delete" or "move" command (page 260, left-hand column, lines 28-32; Figure 4, the annotations to move "that" and to delete "specific"). As discussed during the oral proceedings before the board, these commands are instructions "for an action intended to be performed on the content".
3.6 The ink input does not need to be an editing command but can also represent a handwritten comment such as "Reword" (page 260, right-hand column, lines 10 to 13; Figure 4).
3.7 Document D5 does not disclose that the system implements recognition of handwritten text, and the fact that text to be inserted has to be entered by keyboard suggests that it does not.
Moreover, document D5 does not unambiguously disclose that comment annotations in the left-hand view of Figure 4, when selected for processing by the user, are added to the right-hand view, and the lack of handwriting recognition suggests that they are not.
3.8 In its statement of grounds of appeal, the appellant argued that document D5 disclosed neither differentiating between annotations nor converting an instruction to an associated action, because it was the user who executed a command by selecting the annotation with the stylus. At the oral proceedings, the appellant referred to the passage on page 260, right-hand column, lines 5 to 10, which discloses that, in the annotation mode, markings are not interpreted as commands but treated strictly as annotations.
As explained in point 3.2 above, in document D5 ink input is entered either in an annotation view (Figure 3 and left-hand view of Figure 4), in which case it will be processed further when the user selects it in the incorporation mode (Figure 4), or in an edit view (Figure 2 and right-hand view of Figure 4), in which case it will be processed immediately. The board notes that claim 1 neither requires nor rules out that processing ink input is triggered by a further user action.
Moreover, when the system of document D5 processes ink input, resulting in text being inserted, deleted or moved or a comment being added, it necessarily must analyse the ink input to differentiate between the various kinds of annotation and associate the input with the required action to be taken.
3.9 In sum, document D5 discloses:
- detecting an action to provide ink input on content viewed on a computing device (Figures 2, 3 and 4);
- analysing and processing the ink input to differentiate between a comment (handwritten "Reword" comment in Figure 4), content to be inserted into the document (insertion of "ed" in Figure 4) and an instruction for an action intended to be performed on the content (instructions to move "that" and to delete "specific" in Figure 4);
- in both the edit mode and the incorporation mode of D5, converting ink input into text annotations and displaying the text annotations in an "annotation view" (referred to as "edit view" in D5), including applying instructions "for an action intended to be performed on the content".
However, no handwriting recognition takes place: text to be inserted has to be entered by keyboard, and comments are not converted into text and displayed in the annotation view (i.e. the "edit view" of D5).
3.10 The subject-matter of claim 1 therefore differs from the disclosure of document D5 in that:
(a) content to be inserted is allowed to be handwritten and is converted into text;
(b) handwritten comments are converted into text and displayed in the annotation view;
(c) instructions are matched to an action "by searching a data store storing instructions and associated actions".
3.11 At the priority date of the present application, i.e. in May 2013, it would have been obvious to add handwriting recognition to the annotation tool described in document D5, which was published in 1993, and thus to allow content to be inserted to be fully handwritten and to convert handwritten comments into text for display in the annotation view in accordance with the distinguishing features (a) and (b).
3.12 As for the distinguishing feature (c), it is a standard programming technique to provide a data store, for example in the form of a list data structure, containing a mapping of instructions to actions.
3.13 Hence, the subject-matter of claim 1 of the main request lacks an inventive step over document D5 (Article 56 EPC).
Auxiliary request 1
4. Inventive step
4.1 Document D1 relates to techniques for processing handwritten ink input in the context of providing ink annotations in electronic documents (see paragraphs [01], [02] and [81]).
An application program detects an action to provide ink input 306 on content viewed on a computing device (paragraphs [62], [63], [81] and [82]; Figures 3 and 7A).
The application program calls a parser 310, which analyses the electronic ink input and classifies it as one of various ink types (Figure 3 and paragraphs [63] and [65]), such as a container annotation, a highlight/underline annotation (paragraphs [65] and [82]; Figure 7A) or a "margin comment" annotation (paragraphs [65] and [88] to [90]; Figure 10A).
As part of the analysis process, the parser may call a handwriting recogniser. It returns the results of the analysis to the application program (paragraph [66]).
4.2 The appellant did not contest the examining division's finding that paragraphs [14], [90] and [96] of document D1, which disclose that a document data structure is updated using the results of the analysis process, implicitly discloses the step of displaying the text annotation in an annotation view. The board sees no reason to deviate from the examining division's analysis.
4.3 Hence, the subject-matter of claim 1 differs from the disclosure of document D1 in that:
(a) the ink input is converted to a text annotation on the basis of "a context of the ink input on the content and a grammar and proofreading analysis";
(b) the user is allowed to accept or reject the text annotation.
At the oral proceedings, the appellant confirmed that feature (a) encompassed taking into account "a context of the ink input on the content and a grammar and proofreading analysis" when performing handwriting recognition.
4.4 In the board's view, feature (a) expresses how a human would attempt to give meaning to a handwritten annotation. If some of the individual letters of handwritten text are ambiguous, a human will perform a "grammar and proofreading analysis" and also look at the text in which the annotation is to be incorporated to determine which possible interpretation of the handwriting makes the most sense.
Since the claim specifies no implementation details, the board concludes that feature (a) merely expresses an obvious desire.
4.5 As for feature (b), the board notes that it is well known in the field of computers to give the user the possibility to undo/reject a change and to otherwise accept it.
4.6 In its statement of grounds of appeal, the appellant argued that the use of grammar and proofreading analysis improved processing latency.
However, processing latency is neither mentioned nor alluded to in the application as filed. Moreover, adding more processing typically increases processing latency and certainly does not decrease it over the whole scope of the claim.
4.7 The appellant further argued that the distinguishing features did not merely implement a concept that a human would apply. If input was "not understood" due to incorrect grammar and proofreading, the obvious approach would be to display a user prompt for "asking the user" to enter the correct text. This corresponded to what a human would do in such a situation. The distinguishing features (a) and (b) avoided these additional actions and steps by allowing the user to simply accept the suggested text annotation without further interactions.
However, this argument does not contradict that a human would first attempt to understand the annotation, taking into account "a context of the ink input on the content and a grammar and proofreading analysis". Only when this attempt has failed would the human ask for help. Feature (a) does not rule out that the user is prompted for the correct text if the ink input cannot be converted to a text annotation even when "a context of the ink input on the content and a grammar and proofreading analysis" are taken into account.
4.8 Hence, the subject-matter of claim 1 of the auxiliary request lacks an inventive step over document D1 (Article 56 EPC).
5. Since neither of the appellant's requests is allowable, the appeal is to be dismissed.
For these reasons it is decided that:
The appeal is dismissed.