T 1252/22 12-12-2024
Download and more information:
A SYSTEM FOR VISUALISING AN ANATOMICAL TARGET
Novelty - (yes)
Inventive step - (yes)
Request to allow an accompanying person to make oral submissions - not granted
I. The opponent filed an appeal against the interlocutory decision of the opposition division to maintain the contested patent in amended form on the basis of the auxiliary request 1 then on file.
II. In its decision, the opposition division found that the subject-matter of claim 1 of that request was novel in view of D6 and involved an inventive step starting from any of D2 to D5 in combination with D9.
D2 to D5, D6 and D9 are the following documents:
D2|S. Zenbutsu et al., "3D ultrasound assisted laparoscopic liver surgery by visualization of blood vessels", 2013 Joint UFFC, EFTF and PFM Symposium (2013), pages 840-843|
D3|S. Zenbutsu et al., "Development of Blood Vessel Depth Displaying Method for Laparoscopic Surgery Guidance", Journal of Medical Imaging and Health Informatics 3 (2013), pages 101-106|
D4|C. Hansen et al., "Illustrative visualization of 3D planning models for augmented reality in liver surgery", International Journal of Computer Assisted Radiology and Surgery (2010), pages 1-10|
D5|C. Hansen et al., "Illustration of Vascular Structures for Augmented Reality in Liver Surgery", IFMBE Proceedings 25/IV (2009), pages 2113-2116|
D6|"Clinical User Guide", Revision 1.2, of the software "VectorVision cranial/ENT", Version 7.8, Copyright 2008, BrainLAB AG Germany|
D9|N. Hirai et al., "Image-guided neurosurgery system integrating AR-based navigation and open-MRI monitoring", Computer Aided Surgery 10(2) (2005), pages 59-71|
III. The patent proprietor, who had also filed an appeal against that decision, withdrew it during the oral proceedings before the Board on 12 December 2024.
IV. The parties' final requests at the oral proceedings before the Board were as follows:
(a) The appellant (opponent) requested that the decision under appeal be set aside and that the patent be revoked.
(b) The respondent (patent proprietor) requested that the patent be maintained on the basis of one of auxiliary requests 5, 6, 8, 9, 12, 13, 13a, 14, 14a and 15 to 22 filed with its statement of grounds of appeal, or one of auxiliary requests 5a to 5c, 6a, 8a, 9a, 12a to 12c, 13b to 13d, 14b to 14d, 15a, 16a to 16c, 17a, 21a to 21c, and 22a filed with its letter of 18 November 2024. Auxiliary request 5 is identical to the auxiliary request 1 found allowable by the opposition division.
V. During the oral proceedings before the Board, the appellant's representative requested that Mr Robert Niemann, Director Intellectual Property of the appellant, who accompanied the representative at the hearing, be allowed to speak as a technical expert on the issue of the basic principles of CT modalities, with which the respondent disagreed.
VI. Claim 1 of auxiliary request 5 (hereinafter "claim 1") reads as follows (with the feature numbering used in the decision under appeal):
1 |"A system for visualizing an anatomical target, comprising: |
1.1 | an imaging device (105) configured to collect real-time images of an anatomical target; |
1.2 | a three-dimensional model (136) generated from pre- or intra-operative images and including images of structures below a surface of the anatomical target not visible in the images from the imaging device;|
1.3.1| an image processing module (148) configured to generate an overlay (107) registered to the real-time images |
1.3.2|which displays the structures below the surface of the anatomical target not visible in the images from the imaging device and |
1.3.3|indicates a depth of the structures below the surface; and |
1.4 | a display device (118) configured to concurrently display the real-time images and the overlay;wherein the system further comprises |
1.5 | a visualization feature (125) configured to display an internal view of at least a portion of the structures below the surface; and |
1.6 |wherein the display device (118) is configured to concurrently display the real-time images, the overlay and the internal view, |
1.7a |wherein the internal view (506) includes a three-dimensional fly-through image or |
1.7b |a virtual cross-section of the structures below the surface." |
VII. The appellant's arguments relevant to the present decision can be summarised as follows.
Hearing of Mr Niemann as a technical expert
Mr Niemann should be allowed to make oral contributions during the oral proceedings as a technical expert on the issue of the basic principles of CT modalities. Mr Niemann was very knowledgeable in this technical field and his oral contributions could shed light on why the person skilled in the art would consider a reconstructed CT image to be a "virtual cross-section of the structures below the surface". This question was relevant for the discussion of inventive step.
Inventive step starting from any of D2 to D5 in combination with D9
The subject-matter of claim 1, which differed from the systems disclosed in any of D2 to D5 by features 1.5 to 1.7b, did not involve an inventive step over the combination of any of these documents with D9. The line of argument was the same when starting from any of D2 to D5.
When starting from any of D2 to D5 and faced with the objective technical problem of providing additional information to the medical personnel, the person skilled in the art, being a software engineer experienced in developing medical software for computer assisted surgery, would have considered D9. It was irrelevant that D2 to D5 primarily related to liver surgery.
D9 disclosed an augmented reality-based system in which wireframe models of a tumour to be resected and various anatomical landmarks of interest, such as surrounding nerves or blood vessels, extracted from volumetric CT or MRI data, were overlaid on real-time endoscope video images (Figure 2).
D9 suggested that the surgeon's understanding of the anatomical structures around a region of interest could be enhanced by additionally displaying sectional images, or "slices", reconstructed from CT or MRI data (right column on page 60 and left column on page 61 in combination with Figure 2). These sectional images were derived from the very same dataset from which the three-dimensional frame model had been calculated to serve as an overlay registered with the endoscope video images.
Thus, the person skilled in the art starting from any of D2 to D5 and seeking a solution to the above technical problem would have inferred from D9 that the same volumetric dataset from which the overlay complementing the video images was retrieved provided all the necessary data for sectional images that could be additionally displayed to a surgeon so as to improve the understanding of the patient's anatomy during a medical intervention. Applying this teaching, the person skilled in the art would have arrived at the subject-matter of claim 1 without an inventive step.
Novelty in view of D6
D6, a manual for the "VectorVision" software sold by the appellant, had been made available to the public by being delivered to two customers in 2008, i.e. before the priority date of the contested patent, as substantiated in the appellant's reply to the respondent's statement of grounds of appeal. It was therefore part of the prior art. The subject-matter of claim 1 was not novel in view of this document.
D6 disclosed (page 291) that a system equipped with that software could display multiple windows concurrently, the contents of which could be selected at will by the user. For example, the system could concurrently display a microscope video view along with various additional two- or three-dimensional views.
In a view mode called "Image Injection" (page 366), the "contours of outlined objects" were injected into, i.e. registered and overlaid on, the microscope video view, with "the intersection of [the] objects with the focal plane of the microscope" being "displayed as a solid contour line", and "the extension of objects behind the focal plane" being "indicated by a dotted line". Since, in use, the focal plane of the microscope had to coincide with the outer surface of the anatomical target in order to obtain sharp images thereof, this view mode anticipated the display of structures below the surface of the anatomical target which were not visible in the images of the imaging device, as defined by feature 1.3.2.
Furthermore, the additional two- or three-dimensional views which could be displayed concurrently with this injected microscope video view were reconstructed from three-dimensional volumetric CT or MRI data and were thus virtual images. Because a reconstructed two-dimensional CT/MRI image of an anatomical target structure represented a planar "slice" through the target, as discussed for D9, such an image also showed an internal view including a cross-section of any internal anatomical structures that crossed the image plane. In particular, D6 disclosed a so-called "Inline" view mode (page 310), in which a three-dimensional volume cut along the plane parallel to the microscope axis was reconstructed. The CT/MRI image displayed in this mode thus showed the cross-section of any structure below the surface of the anatomical target that crossed the image plane of the image. Therefore, D6 disclosed the combination of features 1.5, 1.6 and 1.7b.
In addition, in the absence of a specific definition in the contested patent, any image providing a three-dimensional representation of an anatomical structure was to be interpreted as a "three-dimensional fly-through image" according to feature 1.7a. This applied not only to the three-dimensional "Inline" view, which was dynamically updated as the microscope was moved relative to the patient, but also to the "Cubic Cut" view further disclosed on page 308, in which internal structures were dynamically displayed along a three-dimensional cut surface. Moreover, the "VectorVision" software was also capable of displaying a three-dimensional representation of internal structures such as bones under the skin of a patient (page 305). As a result, D6 also disclosed the combination of features 1.5, 1.6 and 1.7a.
VIII. The respondent's arguments relevant to the present decision can be summarised as follows.
Hearing of Mr Niemann as a technical expert
The request to allow Mr Niemann to make oral submissions on the question of the basic principles of CT modalities should have been made in advance of the oral proceedings and not at the oral proceedings themselves. Moreover, Mr Niemann's qualifications in this area were unclear. Therefore, this request should not be granted.
Inventive step starting from any of D2 to D5 in combination with D9
The subject-matter of claim 1 differed from the systems disclosed in D2 to D5 at least by features 1.5 to 1.7b and involved an inventive step when starting from any of these documents, even taking into account D9.
Firstly, the person skilled in the art starting from any of D2 to D5, being experienced in ultrasound techniques, liver laparoscopy and liver surgery, would not have combined it with D9 because of the completely different technical fields to which these documents belonged.
Secondly, the appellant's formulation of the objective technical problem was incorrect, as it contained a pointer to the solution. Rather, the correct technical problem was to provide a system for visualising an anatomical target which allowed for an improved assistance of a physician during liver surgery.
Thirdly, the combination of any of D2 to D5 with D9 would in any event not have led to the subject-matter of claim 1. Indeed, the additional views displayed concurrently with the real-time endoscope images and the overlaid wireframe models as disclosed in D9 were merely sectional CT or MR images, i.e. real images, and not virtual cross-sections, let alone virtual cross-sections of those structures modelled by the wireframe models overlaid on the endoscope images. Rather, these CT or MR images did not show said structures and were merely displayed to allow the surgeon to confirm the position and orientation of the endoscope tip (see, for example, the sentence bridging the two columns on page 67). Therefore, D9 would not have motivated the person skilled in the art to (i) include in the systems disclosed in any of D2 to D5 a visualisation feature configured for displaying an internal view including a virtual cross-section specifically of those structures below the surface which were displayed by the overlay and whose depth was indicated by the latter (features 1.5 and 1.7b), and (ii) configure the display device to display this internal view concurrently with the real-time endoscope images and the overlay (feature 1.6).
Novelty in view of D6
It had not been established that D6 had become publicly available before the priority date of the contested patent.
In any event, the subject-matter of claim 1 was novel in view of D6. At least features 1.5 to 1.7b were not disclosed in D6. For the same reasons as discussed in relation to D9, the CT/MRI images displayed by the system of D6 were real images and not virtual cross-sections as required by feature 1.7b. Furthermore, these images would in any case intersect the objects being injected in the microscope live view - and thus display an internal view such as a cross-section of these objects - only by coincidence or if the user specifically selected the position and the orientation of these images on an ad hoc manner. This did not anticipate features 1.5 and 1.6, which required the visualisation feature and the display device to be configured and not merely suitable for displaying an internal view of at least a portion of those structures below the surface displayed by the overlay and whose depth was indicated by the overlay, concurrently with the real-time images and the overlay.
1. Subject-matter of the contested patent
The contested patent relates to a system as defined in claim 1 which provides improved visualisation of an anatomical target and improved guidance of a surgeon during a medical intervention (see paragraphs [0011] to [0013] of the patent specification).
For this purpose, the claimed system comprises (see paragraph [0006]) an imaging device, such as an endoscope camera (paragraph [0018]), configured to collect real-time images of an anatomical target. A three-dimensional model is generated from pre- or intra-operative images, such as CT or MRI images (paragraph [0021]), and includes images of structures below a surface of the anatomical target which are not visible in the images from the imaging device. Such structures may be, for example, blood vessels hidden below the surface of a heart.
The system further comprises an image processing module configured to generate an overlay, registered with the real-time images, which displays the structures below the surface of the anatomical target which are not visible in the images of the imaging device, and which indicates a depth of the structures below the surface, for example via a colour gradient. A display device of the system is configured to concurrently display the real-time images and the overlay (see for example Figure 5).
In addition, the system also comprises a visualisation feature configured to display an internal view of at least a portion of the structures below the surface, including a three-dimensional fly-through image or a virtual cross-section of the structures below the surface (see paragraphs [0013] and [0041]). The display device is also configured to display this internal view concurrently with the real-time images and the overlay.
Figure 9, reproduced below, shows an example of the resulting display showing (see paragraph [0043]) a real-time endoscope image (134) with an overlay (107) registered thereon, itself showing an underlying coronary artery (504), together with a virtual cross-section (506) of the artery possibly showing the presence of calcifications or atherosclerotic narrowings.
FORMULA/TABLE/GRAPHIC
2. Hearing of Mr Niemann as a technical expert
During the oral proceedings before the Board, the appellant's professional representative requested that Mr Robert Niemann, Director Intellectual Property of the appellant, who accompanied the representative at the hearing, be allowed to make oral submissions as a technical expert on the issue of the basic principles of CT modalities. The respondent disagreed.
It is common ground that Mr Niemann is to be regarded as an accompanying person within the meaning of the decision G 4/95 of the Enlarged Board of Appeal.
As stated in G 4/95 (see Headnote), oral submissions by an accompanying person in opposition appeal proceedings cannot be made as a matter of right, but only with the permission of and under the discretion of the EPO. The criteria elaborated by the Enlarged Board of Appeal for the exercise of this discretion include that the request should have been made sufficiently in advance, indicating the name and qualifications of the accompanying person and the subject-matter of the proposed oral submissions. Moreover, a request made shortly before or at the oral proceedings should, in the absence of exceptional circumstances, be refused unless the opposing party consents.
In the present case, the Board considers that the appellant's request concerning Mr Niemann does not meet the above criteria.
Mr Niemann was offered as a witness in the appellant's reply to the respondent's statement of grounds of appeal (see last paragraph on page 6), but not in connection with the issue of the basic principles of CT modalities, but to confirm the alleged public availability of D6 before the priority date of the contested patent. A request to allow Mr Niemann to "comment on technical issues", namely to "explain[] the underlying concepts of the 'VectorVision'-Software and how it is used in practice", was then made in the appellant's submission of 15 November 2024 (see second paragraph on page 1 and second paragraph on page 2). Again, however, the subject-matter of the proposed submissions was different from the issue of the basic principles of CT modalities. Rather, the request that Mr Niemann be allowed to speak as a technical expert on this issue was made only at the oral proceedings before the Board.
The appellant did not put forward, and the Board does not see, any exceptional circumstances that would justify granting this late request despite the respondent's objection.
Consequently, the Board decided not to grant the appellant's above request in respect of Mr Niemann.
3. Inventive step starting from any of D2 to D5 in combination with D9
3.1 As stated by the Board in its communication under Article 15(1) RPBA (see point 4.2), each of D2 to D5 discloses a system comprising all features 1 to 1.4. At the oral proceedings before the Board, the respondent did not present any further arguments, but only referred to its written submissions. The Board therefore sees no reason to depart from its preliminary view as expressed in its communication.
D2 discloses a system for visualising an anatomical target, such as a liver during laparoscopic liver surgery. This system is configured to display concurrently real-time laparoscopic images of the surface of the liver and a colour-coded overlay, registered on the real-time images, showing underlying structures hidden below that surface, such as a blood vessel (see Figure 2, right image and Figure 5). This overlay is generated from a three-dimensional model resulting from the segmentation of volumetric ultrasound data acquired on the liver (section II.D), and the colour coding of the overlay indicates the depth of the vessel below the liver surface (section II.E). This understanding was actually not contested by the respondent (see point 1.1 of its letter of 18 November 2024).
D3, D4 and D5 disclose substantially similar systems (see point 4.2 of the Board's communication).
3.2 It is common ground that the subject-matter of claim 1 differs from the system disclosed in each of D2 to D5 by features 1.5 to 1.7b. The subject-matter of claim 1 is therefore novel in view of each of these documents.
3.3 The Board agrees with the respondent that the subject-matter of claim 1 involves an inventive step starting from each of D2 to D5, even considering D9.
3.3.1 Indeed, even if one accepts the objective technical problem formulated by the appellant in respect of features 1.5 to 1.7b, to provide additional image information to the medical personnel, and assumes that the person skilled in the art, starting from any of D2 to D5 and faced with this technical problem, would consider D9 - both of which the respondent contested - the combination of D9 with that starting point would not, contrary to the appellant's argument, have led in an obvious manner and without the benefit of hindsight to the subject-matter of claim 1.
3.3.2 As argued by the appellant, D9 discloses an augmented reality-based navigation system in which three-dimensional wireframe models of various anatomical structures of interest to the surgeon, such as a tumour to be resected and surrounding landmarks such as nerves or arteries (see Figure 2), generated from three-dimensional volumetric operative CT or MRI data, are overlaid - in a manner somewhat similar to the system of claim 1 - on real-time endoscope video images.
It is true that D9 suggests to enhance the surgeon's understanding of the anatomical structures around the region of interest by additionally and concurrently displaying sectional images reconstructed from CT or MRI data (see paragraph bridging the right column of page 60 and the left column of page 61 together with Figure 2).
3.3.3 However, these sectional images are only disclosed in D9 as being generally "three orthogonal views" including "the sectional image perpendicular to the view direction of the endoscope" (page 61, left column, first four lines), in particular "three volume sections (axial, coronal and sagittal sections)" (page 62, right column, last three lines). These images should help the surgeon to "carefully observe the vicinity of the tumor regions to be removed" (abstract), to "comprehend the anatomical structures around the tumor easily and immediately" (page 61, left column, first paragraph), to "easily understand the surgical actions" (page 63, left column, first paragraph), and to "confirm the position and orientation of the endoscope tip"
(sentence bridging the two columns on page 67).
As argued by the respondent, D9 neither directly and unambiguously discloses nor suggests that the system is configured to display, in addition to the real-time endoscope images and the overlaid wireframe models, sectional images which intersect the structures represented by the wireframe models, thereby providing an internal view (and a cross-section) of these structures. In fact, a user of the system of D9 would have no incentive to be provided with such an internal view of the landmarks surrounding a tumour, since the primary objective of the system of D9 is to avoid damage to these structures during surgery, for example during resection of the tumour, and their internal substructure is irrelevant in this regard.
Therefore, even if, as further argued by the appellant, such a sectional image reconstructed from CT or MRI data were considered to be an "internal view including a virtual cross-section" - which the respondent denied - this sectional image would not necessarily be an "internal view including a virtual cross-section" of the landmarks which are overlaid as wireframe models on the real-time endoscope images.
3.3.4 It follows that, at most, D9 might motivate the person skilled in the art to modify the system of any of D2 to D5 so that, in addition to displaying the colour-coded overlay, it also concurrently displays sectional images reconstructed from the pre- or intra-operative images (such as three axial, coronal and sagittal sectional images) which might, but would not necessarily, intersect the structures below the surface which are displayed by the overlay (such as a blood vessel in D2) so as to display an internal view including a virtual cross-section of at least a portion of these structures.
As argued by the respondent, the resulting system would not anticipate features 1.5, 1.6 and 1.7b, which require the visualisation feature and the display to be configured for displaying an internal view (including a virtual cross-section) of at least a portion of those structures below the surface which are displayed via the overlay, concurrently with the real-time images and said overlay. The respondent's argument is consistent with the usual interpretation of the term "configured" in computer-implemented inventions - as is the case for the claimed visualisation feature and display - according to which a means configured to perform a function is specifically adapted, such as programmed, and not merely suitable, to perform that function.
It follows that the combination of D9 would not have led the person skilled in the art to implement features 1.5, 1.6 and 1.7b, without hindsight and in an obvious manner, in any of the systems disclosed in D2 to D5.
3.3.5 The appellant did not contest that feature 1.7a was neither disclosed nor suggested in any of D2 to D5 and in D9. It follows that the combination of D9 with any of these other documents would not have led the person skilled in the art to implement this feature either.
3.3.6 The subject-matter of claim 1 is therefore inventive starting from any of D2 to D5, even considering D9.
4. Novelty in view of D6
4.1 Contrary to the appellant's view, the subject-matter of claim 1 is novel in view of D6. It is therefore not needed for the Board to address the question, on which the parties also disagree, whether D6 belongs to the state of the art pursuant to Article 54 EPC.
4.2 As argued by the appellant, D6 discloses on page 366 that, in the "Image Injection" view mode of a system equipped with the "VectorVision" software, the contours of an "object" can be "injected" into, i.e. registered and overlaid on, the real-time images provided by the microscope. Such "objects" correspond to anatomical structures such as eyes or blood vessels, which have been outlined using other software and which can be arbitrarily made visible or not on the display, as explained on page 246 (see section "Managing Objects" and the mention "Vessel CT" in Figure 177). The system is configured to display injected contours even for anatomical structures, or portions thereof, which may extend behind the focal plane of the microscope, and thus which may not be visible in the microscope real-time images (in this case, the contours would then be displayed using a dotted line). Therefore, D6 discloses features 1.3.1 and 1.3.2.
4.3 It is true that the system of D6 is capable of displaying additional two- or three-dimensional views reconstructed from CT or MRI volumetric data, such as the CT/MRI images displayed in the "Inline" or "Cubic Cut" view modes referred to by the appellant.
However, as argued by the respondent, D6 does not disclose that the system is configured - and not merely suitable - for generating and displaying, concurrently with the above-mentioned injected microscope view, additional views which would intersect the objects injected into the real-time images and thereby display an internal view of these objects, or a part thereof.
It is irrelevant in this respect that such an internal view may be displayed by chance or as a result of the user's ad hoc selection of an appropriate position and orientation of the CT/MRI images, as this does not make the system "configured" to do so.
It follows that, even if such reconstructed CT/MRI images were considered to be "virtual cross-sections" or "three-dimensional fly-through images", as further argued by the appellant - which the respondent denied - D6 does not directly and unambiguously disclose at least features 1.5 and 1.6. For this reason alone, the subject-matter of claim 1 is novel in view of D6.
5. Conclusion
It follows from the foregoing that none of the appellant's objections prejudice the maintenance of the contested patent on the basis of auxiliary request 5, which is identical to the claim request found allowable by the opposition division in the decision under appeal. The appeal is therefore to be dismissed.
For these reasons it is decided that:
The appeal is dismissed.