T 0076/22 08-09-2023

1. The appeal is admissible.

2. Introduction

The patent relates to the automated detection of the location of animal teats for controlling an automated milking apparatus (see published patent specification, paragraph [0001]). It is known to use optical sensors to determine the attitude and location in three dimensions of the teats in order to guide the milking apparatus (see paragraph [0002]). The invention uses a light source and a camera having a two-dimensional array of pixels that are capable of individually providing time of flight data in an automated milking system, so that three-dimensional (3D) data can be acquired from the imaged teats (see paragraph [0014] and all versions of the independent claims).

3. Main request, claim 1, inventive step starting from D1 with D18

3.1 The Board agrees with the opposition division (see impugned decision, reasons 6.1) that D1 is a good starting point from which to assess inventive step, since it discloses an automated milking system having a milking apparatus with milking cups for extracting milk from the teats of an animal (see D1, page 21, line 31 to page 23, line 6 with figures 13 to 15). The system includes a movable apparatus, namely a robot arm 13 - best seen in figure 13 - which is adapted to receive instructions to move to the location of a teat, as explained on page 22, lines 1 to 4 and 31 to 36. D1's system also has light sources 101 and 102 for illuminating the area where the teats are (see page 22, lines 9 to 20 and 31 to 36) and a CCD camera 17, i.e. with a two-dimensional sensor array, which images the milking cups and teats in one field of view (see figure 14). It also discloses electronic means for capturing the camera output to deliver data characterising the location of the teats, including depth data (see for example page 20, line 23 to page 23 line 6 with figures 10 to 13). In D1 the light sources may, for example, project scanning light beams that move in a zigzag or within a plane (see page 20, lines 23 to 32, page 22, lines 26 to 29 with figures 10, 11, 14 and 15) in order to locate the teats in three-dimensional space.

3.2 Therefore, the subject-matter of claim 1 differs from D1 in that the pixels of the camera's two-dimensional array of pixels are capable of individually providing time of flight data from which the [3D] location characterisation is made. The difference boils down to a time of flight (ToF) sensor for locating the teats.

3.3 The patent (see paragraph [0012]) sees the object of the invention as providing a teat sensing system that is compact, easy to manipulate and effective. The Board also agrees with the appellant-proprietor that D1's camera already solves the aspect of the problem of compactness (see page 15, lines 4 to 6 and page 22, lines 1 to 7 with figure 13). Therefore, a different problem can be formulated. Turning to the aspects of simpler and more effective sensing, the Board also agrees with the appellant-proprietor that this can be understood in terms of improved accuracy or reliability in locating the teats for subsequent attachment mentioned in the patent at paragraph [0028]. Compared with D1's scanning of projected light, the invention's ToF sensor, directly providing depth information for every pixel without scanning, appears to be a simpler and faster way of characterising the location of the teats in 3D space (see published patent specification, paragraphs [0006] and [0028]). As animals may move, a faster characterisation of teat positions will lead to more accurate characterisation of their actual positions. Moreover, teats do not always hang vertically but may also be inclined (see D1, page 23, lines 7 to 17 with figure 16). In this case, D1's system may not accurately recognise a teat and the robot may guide teat cups away from the teat. For these reasons, the Board finds it plausible that the ToF sensor system of the invention, with its accurate and real time 3D characterisation of what the camera views, will lead to a more accurate or reliable teat location and subsequent attachment of the teat cups, as the appellant-proprietor has argued.

3.4 Therefore, the objective technical problem can be formulated as: how to modify the teat sensing system of D1's automated milking system to make it simpler and more accurate and thus more reliable in terms of attaching teat cups to teats.

3.5 The question of inventive step must be considered from the perspective of the relevant skilled person. In accordance with established jurisprudence, (see Case Law of the Boards of Appeal, 10th edition, 2022 (CLBA) I.D.8.1.1 and the cited prior art, in particular T0026/98, reasons 6.1 to 6.3), amongst other things, the skilled person can be expected to look for suggestions in a general technical field if they are aware of such fields. Moreover, in advanced technical fields the competent "skilled person" could be taken to be a team of experts from the relevant technical branches. The Board holds that the relevant skilled person would, according to both these criteria, be a person skilled in machine vision systems.

3.5.1 Firstly this is because automated milking systems typically use machine vision to guide a robot to attach teat cups (cf. published patent specification, paragraph [0002]), as indeed is the case in the closest prior art D1, so the relevant skilled person would be very much aware of this field. In this regard, the present case is different from T 0422/93, cited by the proprietor, in which the appropriate skilled person's basic knowledge was found to not include that of a specialist in the different technical field to which the proposed solution belonged because the closest prior art gave no indication that the solution was to be sought in this other field.

3.5.2 Secondly, such milking systems are complex systems, involving machine vision amongst other things. Therefore, the person skilled in the art of automated milking systems can be seen as a team having expertise in machine vision systems amongst other fields.

3.5.3 From the above, the arguments of the appellant-proprietor that the skilled person must be considered to have only partial or limited knowledge of the field of machine vision, knowing of such systems but not their particular advantages and that they would know only of off-the-shelf systems, but not research disclosures such as D18, are moot.

3.6 Turning now to the combination of D1 and D18 and the question of obviousness of the solution, this is to be considered starting from D1 with the eye of the skilled person looking for a solution to the objective technical problem, rather than seeking to overcoming any drawback explained in D1. Therefore, the appellant-proprietor's argument that D1 itself does not motivate the skilled person to replace its machine vision system is moot.

3.7 Because the skilled person is skilled in the field of machine vision, they would be aware of D18 which discloses a ToF camera. Indeed, the patent itself (see published specification, paragraph [0023]) confirms that the person skilled in the art of machine vision knows about such cameras.

3.8 Faced with the objective technical problem (making the D1 sensing arrangement simpler and more accurate), the Board holds that the skilled person would immediately see that D18 (see abstract and the conclusion) offers a simpler way of 3D mapping in that it is able to entirely capture its environment in all three spatial dimensions, generating a distance map for each single pixel without the need for scanning (see for example last sentence of section 2.2 and the paragraph above figure 8: A simple graphic user interface enables the read out of the 3D image data [...]).

3.9 The skilled person would also immediately realise that D18 offers a sufficiently accurate way of detecting objects in a field of view because it offers sub-centimeter depth resolution (see title). Moreover, in the context of imaging a moving animal, the skilled person would realise that D18's real time 3D imaging can but offer a more accurate way of detecting actual position compared to D1's scanning system. In solving the objective technical problem, the skilled person would therefore replace D1's machine vision system with D18's time of flight camera system comprising a light source, control electronics and camera sensor, packaged in a housing (see for example figure 8 and the paragraph above it). In so doing, the skilled person would arrive at the subject-matter of claim 1, as a matter of obviousness.

3.10 In this regard, contrary to how the appellant-proprietor has argued, the Board considers that the fact that D18 does not disclose to apply its ToF camera system to robot milking systems would be no barrier to the skilled person replacing D1's vision system with that of D18. This is because the skilled person would be very well aware that the underlying task performed by D1's optical system is to characterise 3D space. Put differently, they would know that the accuracy and complexity shortcomings of D1's machine vision system are related to how its sensor system renders its field of view in 3D, with its multiple scanning steps and inherently inaccurate perception of moving or inclined objects. This is all the more true because D18 discloses (see section 7. Outlook) that its imaging system offers the possibility of entering completely new application fields and that the added value of depth information would allow the replacement of traditional sensors and cameras [with D18's ToF system].

3.11 From the above, the Board concludes that claim 1 lacks inventive step, Article 56 EPC, so that the main request fails.

4. Auxiliary request A, claim 1, inventive step

4.1 In its preliminary opinion (see section 6.2), the Board considered that claim 1 lacked inventive step for the same reasons as the main request because, compared to the main request, it only adds features that are known from D1. The Board wrote the following:

"In the Board's view, claim 1 of this request adds features which are all known from D1. In particular, as seen in D1, figure 13 and explained on page 21, line 31 to page 22, line 8, the moving apparatus is a robot arm and the milking cups 14 are mounted on the robot arm 13. Moreover, a light source and camera unit 10 is mounted on the robot arm, which would likewise be the case if the unit 10 was replaced with D18's time of flight unit. It goes without saying that such a unit contains electronics for providing the relevant output. Therefore, the subject matter of auxiliary request A appears to lack inventive step for the same reasons as apply to the main request."

At the oral proceedings before the Board, the appellant made no further comment with regard to inventive step starting from D1, nor had it done so in writing after the communication. In the light of this, the Board confirms its provisional opinion that claim 1 of this request lacks inventive step for the same reasons as apply to the main request (D1 with D18).

5. Auxiliary request B, claim 1, inventive step

Claim 1 of auxiliary request B reads as for auxiliary request A. Therefore, this request fails for the same reason as apply to auxiliary request A (lack of inventive step starting from D1 with D18).

6. Auxiliary request C (auxiliary request 2 in opposition proceedings), claim 4, added subject-matter

6.1 According to established jurisprudence (see CLBA, II.E.1.9 and the decisions cited therein) it is normally not allowable to base an amended claim on the extraction of isolated features from a set of features originally disclosed only in combination. Such an amendment results in an "intermediate generalisation" and is justified only in the absence of any clearly recognisable functional or structural relationship among the features of the specific combination.

6.2 In the present case, claim 1 is based on original method claim 3. This claim defined (in summary) using a light source to illuminate a plurality of teats and using a two-dimensional array camera to capture an image of a teat; and to process the image to provide three-dimensional location information. The present claim adds the steps of providing a robot arm and using a sensor housing mounted on the robot arm that includes the light source and the two-dimensional array camera, amongst other features. The opposition division found (see impugned decision, reasons 13 to 15 with 11.1.1 and 11.2.1) that claiming these features without electronic means, constituted an unallowable intermediate generalisation. The Board agrees.

6.3 It is common ground that the features relating to the robot arm and sensor housing are taken from the detailed description of the preferred embodiment (see published patent application, page 5 of the, lines 8 to 22 with figures 1 and 2).

6.3.1 The first sentence of page 5 (see also, figure 1) introduces the idea of a sensor housing mounted on the robot arm for sensing the location and attitude of teats. The next sentence explains that the output of the sensor is used to control the application [of milking cups]. Thus the term "sensor" here seems to designate a complete apparatus for sensing teat location in 3D space.

6.3.2 The paragraph that follows explains what the sensor housing includes: A modulated light source, a camera, readout electronics and suitable optics (see figure 2). Here, the readout electronics are disclosed in a structural relationship with the camera and light source because they are all contained in the sensor housing.

6.3.3 The next paragraph (see page 5, lines 24 to 28 with figure 3) explains that the camera 26 consists of a two-dimensional array of pixels, each of which being able to report time of flight data synchronized to the light source 22, as well as intensity. In the Board's view, the synchronism between camera data and the [modulated] light source can but mean that both the light source and the camera are part of a ToF sensor, rather than the camera 26 being a ToF sensor in its own right, with inbuilt readout electronics, as the appellant-proprietor would have it. The Board's interpretation is confirmed by the next paragraph (page 5, last three lines) which explains that the image captured on the two-dimensional array must be rendered as 3D data using the time of flight information to determine range. Thus the raw data from the camera array does not contain depth data, the latter must be obtained by processing. It can only be the readout electronics 28 which does this. To do so, they must be connected to the sensor's light source and the camera. Thus the readout electronics are originally disclosed in a close structural relationship with the light source, camera and sensor housing.

6.4 Because the readout electronics 28 enable the sensor in the housing 10 to render a view in 3D, rather than being there to merely pass on pre-rendered 3D data, they are also originally disclosed in a close functional relationship with the light source and camera. Nor, given that they are central to generating depth data, can they be said to be trivial and thus not necessary to claim as the appellant has argued.

Moreover, in a system that measures the flight time of light travelling short distances, the Board holds that the propagation time of the electrical signals between sensor components would be significant. Consequently, there is a functional relationship between locating the camera, light source and readout electronics in the housing and thus proximate to each other (cf. figure 2).

6.5 For these reasons, the Board considers that it is not justified to extract from the detailed description of the embodiment the features of the light source and camera being in the sensor housing on the robot arm and incorporate these into the claim, without also including the readout electronics. Therefore, the claim represents an unallowable intermediate generalisation.

6.6 The appellant-proprietor has argued that it is implicit that a time of flight camera has readout electronics and so the claim need not explicitly mention them. Whatever the term time of flight camera might imply, the claim does not use it, so the argument is moot. At most the claim only defines components, including a camera, that make up a ToF sensor.

6.7 Moreover, the Board is not convinced by the appellant-proprietor's argument that the claim need not define the readout electronics because it is a method claim. The claim defines the step of providing a robot arm and using a sensor housing that is mounted to this arm. Therefore, far from the claim merely defining process steps devoid of implications for any physical entities, just as with the robot arm, in order to be used, the sensor housing and its contents must be provided.

6.8 Lastly, although the claim step of processing the image from the camera to provide three-dimensional information might imply there to be readout electronics somewhere, since the feature does not define where the processing occurs, it is not equivalent to a feature defining readout electronics being included in the sensor housing.

6.9 For all these reasons, the Board holds that claim 4 adds subject-matter extending beyond the application as filed.

7. Auxiliary request D, admittance

7.1 In its communication (see section 5), the Board commented on the admittance of auxiliary request D (and B) as follows:

"5. Admissibility of certain auxiliary requests

5.1 According to Article 12(2) RPBA 2020, [i]n view of the primary object of the appeal proceedings to review the decision under appeal in a judicial manner, a party's appeal case shall be directed to the requests, facts, objections, arguments and evidence on which the decision under appeal was based.

5.2 Article 12(4) RPBA 2020 states that: Any part of a party's appeal case which does not meet the requirements in paragraph 2 is to be regarded as an amendment, unless the party demonstrates that this part was admissibly raised and maintained in the proceedings leading to the decision under appeal. Any such amendment may be admitted only at the discretion of the Board. The party shall clearly identify each amendment and provide reasons for submitting it in the appeal proceedings. In the case of an amendment to a patent application or patent, the party shall also indicate the basis for the amendment in the application as filed and provide reasons why the amendment overcomes the objections raised.

In exercising its discretion, according to Article 12(6) RPBA 2020 second sentence: The Board shall not admit requests, facts, objections or evidence which should have been submitted, or which were no longer maintained, in the proceedings leading to the decision under appeal, unless the circumstances of the appeal case justify their admittance.

5.3 In the present case, auxiliary requests B and D were not on file in the opposition proceedings but were first filed in appeal and thus represent amendments to the appeal case of the proprietor in the sense of Article 12(4) RPBA. In its appeal grounds (see sections E and G), the appellant-proprietor has submitted that it was unable to present them before because the reasons for the added subject-matter objections to auxiliary requests 1 and 2 (now auxiliary requests A and C respectively) became apparent for the first time in the reasoned decision. The Board does not find this convincing.

At the oral proceedings before the opposition division (see minutes page 6), there was a discussion of added subject-matter of auxiliary request 1 (now auxiliary request A). Amongst other things, the opponent 1 argued that, electronic means mounted on the robot arm were missing from the independent method claim, claim 5. The proprietor defended the claim with counter arguments. The division then deliberated on the matter and subsequently announced the conclusion that claim 5 contained added subject matter and that this would likewise apply to the independent method claim of auxiliary request 2 (now auxiliary request C). The appellant-proprietor was therefore aware of the problem (absence of electronic means in the method claim) at the oral proceedings, even though it had not seen the detailed written reasons as to why the absence of this feature added subject-matter. Moreover, the appellant-proprietor was given the opportunity to file a new request to respond to this finding (see minutes page 7, lines 1 to 7). It chose to do so by filing a new auxiliary request (2A) in which the method claims were deleted, rather than amending the method claims by adding the missing feature. In the Board's view, since the appellant-proprietor knew at the opposition division oral proceedings which feature (electronic means) was considered by its absence to add subject-matter, it had all the information required to make a suitable amendment to the method claim and indeed should have done so, had it wished the patent to be maintained with such a method claim.

Therefore, the Board intends not to admit auxiliary requests B and C [sic] into the appeal proceedings in accordance with Article 12(6) RPBA 2020 with Article 114(2) EPC.

7.2 As is clear from the context of the last paragraph, the Board intended not to admit auxiliary request D. In the absence of any comments from the parties on these issues, either in writing or at the oral proceedings, the Board sees no reason to depart from its preliminary opinion. Therefore, the Board decides not to admit auxiliary request D into the proceedings.

8. Since the appellant's main request and auxiliary requests A to C do not meet the requirements of the EPC and auxiliary request D has not been admitted into the proceedings, the Board must dismiss the appeal.