T 0646/23 (Story album/HUAWEI) 09-12-2024

1. The application relates to displaying "story albums".

2. The appellant's requests

2.1 In its letter of 30 September 2024, the appellant requested consideration of the main and first to fourth auxiliary requests as filed on 17 August 2022 if the board "should object to any of the amendments of the Main and First Auxiliary to Fourth Auxiliary Request as filed herewith". The requests filed on 17 August 2022 are the requests considered in the decision under appeal. Given that the letter of 30 September 2024 was accompanied only by a set of claims according to a sixth auxiliary request, the board understands "the Main and First Auxiliary to Fourth Auxiliary Request as filed herewith" to be the main request and the first to fourth auxiliary requests filed with the statement of grounds of appeal.

2.2 The statement of grounds of appeal included similarly worded statements. In its communication, the board indicated that it understood those statements to mean that the appellant withdrew the main request and first to fourth auxiliary requests considered in the decision under appeal only in so far as the corresponding amended requests filed with the statement of grounds of appeal were admitted into the appeal proceedings. The appellant has not disputed this interpretation.

2.3 Hence, the appellant requested that the decision under appeal be set aside and that a patent be granted on the basis of the claims of the main request or, in the alternative, of one of the first to sixth auxiliary requests, where the main request and the first to fourth auxiliary requests are understood to be the amended requests filed with the statement of grounds of appeal if admitted into the appeal proceedings and to be the requests filed with the letter of 17 August 2022 otherwise.

Main request and first to third auxiliary requests

3. Admission into the appeal proceedings

3.1 Since the main request and the first, second and third auxiliary requests as filed with the statement of grounds of appeal differ from the corresponding requests considered in the decision under appeal only in a minor correction, the board admits them into the appeal proceedings (Article 12(4) RPBA).

3.2 As a consequence, the main request and the first, second and third auxiliary requests as filed with the letter of 17 August 2022 need not be considered in this decision (see point 2. above).

Main request

4. Clarity

4.1 Claim 1 of the main request includes a step of "performing, by the terminal, screening on the N images in the gallery, wherein the performing screening comprises identifying content of each image in the gallery". The claim does not specify in what sense the content of each image is identified, i.e. what is the concrete result of the content identification. Moreover, none of the other features of the claim refers to the result of the content identification.

4.2 In its letter of 30 September 2024, the appellant submitted that identifying content could readily be understood as "generally obtaining an information of said content of the image" and could relate to a direct identification of the objects shown in the image or to an intermediate identification, for example based on metadata of the image. If an identification of displayed objects was used, the concrete result would be information concerning the identified objects. If an intermediate identification was used, the concrete result would be information relating to the metadata. The skilled person understood that the content identification related to the claimed step of removing an invalid image based on an image similarity:

"removing, by the terminal, an invalid image from the N images, to obtain the story album comprising M images, wherein the invalid image comprises an image with an invalid label, or one or more images with an image similarity greater than a threshold".

The "performing" step of claim 1 was broad but not unclear.

4.3 The board notes, however, that the claimed "removing" step does not in any way link the step of "identifying content" or its result to the "invalid label" or to the determination of images "with an image similarity greater than a threshold". Claim 1 does not express that "screening" an image including "identifying content" of the image means, in one alternative, retrieving a metadata label of the image. Nor does claim 1 express that "image similarity" between one or more images is determined on the basis of the identification of the content of the images.

Although the skilled reader of claim 1 may understand that some kind of link between "identifying content" and the "removing" step is intended, on the basis of the wording of the claim they can only speculate what that link might be.

The board therefore concludes that claim 1 of the main request is not clear (Article 84 EPC).

4.4 For the sake of completeness, the board notes that the explanations given by the appellant are not supported by the description. Although paragraph [0123] of the translation of the description as filed starts by stating that "the terminal may identify content of each image in the initial story album", this paragraph then explains that, for example, image similarity between two images can be determined as a hamming distance by using the "pHash" perceptual hash algorithm. The application of a perceptual hash algorithm on image data does not, however, result in an identification of objects shown in the image.

First, second and third auxiliary requests

5. Clarity

5.1 Claim 1 of each of the first and second auxiliary requests include the same "performing" and "removing" steps as claim 1 of the main request. These claims are therefore not clear for the same reason as given in point 4. above (Article 84 EPC).

5.2 Claim 1 of the third auxiliary request adds features specifying that the "removing" step further comprises:

- displaying, by the terminal, X images with a similarity greater than the threshold that are identified from the N images, wherein X >= 2, and

- in response to a target image selected by a user from the X images,

- retaining, by the terminal, the target image in the story album, and

- removing a remaining image from the X images other than the target image.

Since these further features do not relate to the determination of the X images which have image similarity greater than a threshold, the clarity objection raised in point 4. above also applies to claim 1 of the third auxiliary request (Article 84 EPC).

Fourth auxiliary request

6. Admission into the appeal proceedings

6.1 The fourth auxiliary request as filed with the statement of grounds of appeal differs from the corresponding request considered in the decision under appeal only in a minor correction and the deletion of the feature which, in point 4. above, was found to create a lack of clarity. Therefore, the board admits it into the appeal proceedings (Article 12(4) RPBA).

6.2 As a consequence, the fourth auxiliary request as filed with the letter of 17 August 2022 need not be considered (see point 2. above).

7. Inventive step

7.1 Claim 1 of the fourth auxiliary request is directed to a "story album display method". According to claim 1:

(a) A terminal first selects ("aggregates") N images from an image collection on the basis of photographing time or photographing location.

(b) The terminal then removes a number of images that are determined to be "invalid" to obtain M remaining images, where M < N. An image may be determined to be invalid if it has an "invalid label" or if its "image similarity" to another image is greater than a threshold ("wherein the invalid image comprises an image with an invalid label, or one or more images with an image similarity greater than a threshold"). The remaining M images form a "story album".

(c) On the basis of "an image feature of at least one of the M images", the terminal determines a "theme associated with the M images". The terminal modifies this theme, referred to as "the theme of the story album", on the basis of "association information that is in a target application and that is associated with the theme of the story album".

(d) The terminal then displays an "album cover" of the story album.

7.2 As to the term "invalid label" in item (b), in its letter of 30 September 2024, the appellant explained that "any suitable definition of what labels are invalid and what labels are valid may be applied". According to paragraph [0127] of the translated description, whether a label is invalid may be determined by checking whether the label is in an "invalid label list", and examples of invalid labels are "scissors", "keyboard" and "screenshot".

As to item (c), in the example given in paragraph [0152] of the description, the initially determined theme is "party", which is modified to "mother's birthday party" on the basis of calendar information indicating "mother's birthday".

As to item (d), the appellant explained that "album cover" referred to the first page of the "story album" and that "any suitable album cover" could be chosen. According to paragraph [0160] of the description, the "album cover" may be one of the M images that form the story album.

7.3 Terminals, such as smartphones, on which image collections are stored and which display images selected from the image collection were well known in the art at the filing date of the application (5 December 2017). The board considers such a prior-art terminal to be a suitable starting point for assessing inventive step.

7.4 The subject-matter of claim 1 differs from this prior-art terminal in that the terminal includes software functionality to select a subset of images from the image collection on the basis of criteria involving image similarity and image metadata, where the image metadata includes photographing time, photographing location and image labels or tags (items (a) and (b)). An image of this subset is then selected for display as "album cover" (item (d)).

This software functionality, however, does not make a technical contribution. Indeed, it cannot be seen what specific technical effect is achieved by the selection of a subset of images from a larger set of images on the basis of these criteria or by the selection of an "album cover" image from the selected subset of images.

Although a specific implementation in software or hardware of such non-technical functionality may in certain cases have technical aspects, for example relating to determining image similarity, technical details of such an implementation are not claimed.

7.5 The subject-matter of claim 1 further differs from the prior-art terminal in that a "theme" is determined for the subset of M images on the basis of an "image feature" of one of the M images and "association information" such as calendar information, in accordance with item (c).

A method of classifying (or recognising) images on the basis of features extracted from the visual content of the images may in certain cases be technical, in particular if it is based on technical considerations relating to image processing. However, the claim leaves the kind of "image feature" undefined, and paragraph [0137] of the description explains that the theme may be determined on the basis of "image features" reflected by the textual labels associated with the M images. Hence, the determination of a "theme" in the present case encompasses a mere processing of non-technical textual information, which does not make a technical contribution.

7.6 The appellant argued that the invention achieved "improved accuracy for a story album and for the displaying of its cover", which allowed a user to search and retrieve images more efficiently and faster.

The board does not consider a more efficient or faster search and retrieval to be credible for the reason alone that the method of claim 1 merely displays an "album cover", which can be a single image.

Moreover, even if the claimed image selection somehow reliably returns what the user is looking for (which is unlikely given that the method does not require the user to give an indication of what they are looking for), improving the accuracy of search results to correspond more closely to the user's subjective wishes is in principle not a technical effect.

7.7 The appellant considered the present case to be directly analogous to the case underlying decision T 643/00.

The board does not see any similarity with the facts underlying decision T 643/00, which was concerned with a tool which allowed a user to search a collection of images in a new (and efficient) way by, inter alia, displaying thumbnail images side by side. The user still had to look at and evaluate each image, but the invention provided a user-interface mechanism that allowed them to browse through the images more quickly.

In the present case, claim 1 does not define any user-interface mechanism for allowing a user to search an image, let alone a new mechanism.

7.8 The appellant also submitted that removing images on the basis of similarity, even if the claim did not specify that the removed images were physically removed from storage, at least reduced working memory usage. In this respect, the appellant argued that the claim required the actual images to be processed, which had to be done in working memory. Fewer images in the story album therefore resulted in lower resource consumption.

7.8.1 The board notes that claim 1 requires all the N images in the initial image selection to be processed for the purpose of determining image similarity, including those that are removed. If anything, the "removing" step of claim 1 therefore does not decrease but increase the use of working memory for processing images.

Claim 1 does not specify in any detail how the "story album" image selection of first N images and later M < N images is represented in memory. Such an image selection can be represented, for example, as a bitmap indicating which images of the complete collection of images stored on the phone are part of the selection and which are not. In this case, the "removal" of an image from the selection amounts to flipping the bit corresponding to the image from 1 to 0, and no reduction of memory usage is achieved by reducing the number of images in the selection from N to M.

It is therefore not credible that a reduction in working memory usage is achieved over essentially the whole scope of the claim.

7.8.2 Moreover, even if the removal of an image from the "story album" selection did reduce memory usage, this would merely be the physical consequence of the non-technical decision to remove certain images. Memory usage could be even further reduced by removing all but one image, or by not creating the selection at all. Such normal physical consequences of non-technical decisions cannot support an inventive step (see decision T 2230/10, Reasons 3.7 and 3.8).

7.9 Hence, the subject-matter of claim 1 lacks an inventive step over a well-known prior-art smartphone.

Fifth auxiliary request

8. Admission into the appeal proceedings

8.1 The fifth auxiliary request is based on the fourth auxiliary request and was filed for the first time with the statement of grounds of appeal.

8.2 The appellant argued that the fifth auxiliary request should be admitted because the amendments served to overcome the inventive-step objection raised in the decision against the fourth auxiliary request, which had not been discussed in detail before, and because it "merely" introduced features taken from a dependent claim and from the description.

The board notes that the fourth auxiliary request was filed in an attempt to overcome the objection of lack of inventive step over document D1, which had already been raised in the European search opinion. The fifth auxiliary request is likewise an attempt to overcome that same objection, and the board is not aware of a reason why it could not have been filed in the first-instance proceedings.

8.3 In its letter of 30 September 2024, the appellant further argued that the fifth auxiliary request should be admitted because it addressed the clarity objection to the feature "the M images are associated with a theme of the story album" raised in the board's communication.

However, the board has been able to deal with the fourth auxiliary request under Article 56 EPC (see point 7. above). The clarity objection raised in the board's communication therefore does not represent a compelling reason to admit an (allegedly) clarifying amendment.

8.4 The board therefore does not admit the fifth auxiliary request into the appeal proceedings (Article 12(4) RPBA).

Sixth auxiliary request

9. Admission into the appeal proceedings

9.1 The sixth auxiliary request is based on the fourth auxiliary request and was filed after notification of the board's communication under Article 15(1) RPBA.

9.2 Claim 1 of the sixth auxiliary request adds to claim 1 of the fourth auxiliary request features specifying that the "removing" step further comprises:

- displaying, by the terminal, Y labels identified from the N images, wherein a label of each image is used to reflect an image feature of the image, and Y >= 2, and

- in response to a target label selected by the user from the Y labels,

- retaining, by the terminal, an image with the target label in the story album, and

- removing an image without the target label.

In addition, it adds:

- wherein the album cover comprises a title of the story album, and the title is associated with at least one of the theme of the story album, a photographing time of the N images, and a photographing location of the N images.

9.3 The appellant argued that the sixth auxiliary request should be admitted into the appeal proceedings because it addressed the clarity objections to "invalid label" and "album cover" raised for the first time in the board's communication.

The board notes that, at least prima facie, the features added to claim 1 of the sixth auxiliary request do not provide a clarification of the expression "invalid label", as they do not even refer to this expression.

In any event, since the board has dealt with the fourth auxiliary request not under Article 84 EPC but under Article 56 EPC (see point 7. above), in the present case the board's clarity objections to "invalid label" and "album cover" do not represent exceptional circumstances which justify the admittance of amendments filed after notification of the board's communication under Article 15(1) RPBA.

9.4 Hence, the board does not admit the sixth auxiliary request into the appeal proceedings (Article 13(2) RPBA)

10. Since none of the requests admitted into the appeal proceedings is allowable, the appeal is to be dismissed.