Get 20M+ Full-Text Papers For Less Than $1.50/day. Subscribe now for You or Your Team.

Learn More →

Synthetic Elastography using B-mode Ultrasound through a Deep Fully-Convolutional Neural Network

Synthetic Elastography using B-mode Ultrasound through a Deep Fully-Convolutional Neural Network Synthetic Elastography using B-mode Ultrasound through a Deep Fully-Convolutional Neural Network R. R. Wildeboer, R. J. G. van Sloun, C. K. Mannaerts, P. H. Moraes, G. Salomon, M.C. Chammas, H. Wijkstra, M. Mischi Abstract—Shear-wave elastography (SWE) permits local es- Ultrasound-based elasticity imaging, that is, ultrasound elas- timation of tissue elasticity, an important imaging marker in tography, has played a major role in these developments biomedicine. This recently-developed, advanced technique as- [6]. So-called quasi-static elastographic (QSE) strain imag- sesses the speed of a laterally-travelling shear wave after an ing allows for the relative assessment of tissue deformation acoustic radiation force “push” to estimate local Young’s moduli due to externally applied stress, but as this stress is often in an operator-independent fashion. In this work, we show how synthetic SWE (sSWE) images can be generated based on manually delivered, the technique remains operator dependent conventional B-mode imaging through deep learning. Using side- and limited to superficial organs. Therefore, more recently, by-side-view B-mode/SWE images collected in 50 patients with dynamic elastography techniques were developed where tissue prostate cancer, we show that sSWE images with a pixel-wise deformation induced by an acoustic radiation force “push” mean absolute error of 4.50.96 kPa with regard to the original pulse is quantified to obtain more objective and reproducible SWE can be generated. Visualization of high-level feature levels through t-Distributed Stochastic Neighbor Embedding reveals measures of elasticity [7]. At this moment, we distinguish substantial overlap between data from two different scanners. especially acoustic radiation force imaging (ARFI) and shear- Qualitatively, we examined the use of the sSWE methodology wave elastography [5], [8]. The first method analyses tissue for B-mode images obtained with a scanner without SWE displacement resulting from a push pulse along the beam path, functionality. We also examined the use of this type of network whereas the latter relies on the speed of transversally-travelling in elasticity imaging in the thyroid. Limitations of the technique reside in the fact that networks have to be retrained for different shear waves to estimate tissue elasticity. The tissue elasticity is organs, and that the method requires standardization of the quantified by the Young’s modulus, that is, the ratio between imaging settings and procedure. Future research will be aimed stress and strain. at development of sSWE as an elasticity-related tissue typing strategy that is solely based on B-mode ultrasound acquisition, and the examination of its clinical utility. SWE requires advanced ultrafast acquisition schemes with Index Terms—Shear-Wave Elastography, Deep Learning, Con- frame rates of 1000 Hz to accurately assess tissue deforma- volutional Neural Networks, B-mode Ultrasound tion and shear-wave dynamics [7], [9]. Moreover, ultrasound transducers have to be sufficiently equipped to allow for the I. INTRODUCTION generation of acoustic radiation force pulses as well as ultrafast imaging of the shear wave displacements [10]. Although sev- Tissue elasticity is an important biomarker of cancer. Prostate cancer, for example, is characterized by increased eral techniques and sequences have been developed to enable stiffness [1], thyroid and liver nodules can be discriminated SWE on commercial scanners, the frame rate of conventional based on their elasticity [2], [3], and also breast lesions are B-mode ultrasound cannot be reached as it requires long typically diagnosed based on their elastic properties [4]. It is settling times and multiple “push” pulses to reliably generate also increasingly used to image musculoskeletal pathologies an elastogram. in e.g. muscles, tendons, and ligaments [5]. Over the last few decades, this has spurred considerable advances in the Realizing that conventional B-mode ultrasound assesses development of elasticity imaging. tissue echogenicity rather than tissue elasticity, we here R.R. Wildeboer, R.J.G van Sloun, H. Wijkstra and M. Mischi are propose that both properties can be expected to be linked with the Lab of Biomedical Diagnostics, Department of Electrical Engi- through their dependence on the underlying tissue structure. neering, Eindhoven University of Technology, The Netherlands. (e-mail: In this work, we exploit this fact by designing a deep r.r.wildeboer@tue.nl; r.j.g.v.sloun@tue.nl) C.K. Mannaerts and H. Wijkstra are with the Department of Urology, fully-convolutional neural network (DCNN) that is able to Academic University Medical Centres, University of Amsterdam, The Nether- assess echogenic patterns in B-mode ultrasound that are lands. useful for elasticity-related tissue typing (see Figure 1). G. Salomon is with the Department of Urology, University Hospital Hamburg-Eppendorf, Germany. Whereas deep-learning strategies were already proposed for P. H. Moraes and M.C. Chammas are with the Department of Radiology, estimation of speed of sound [11], extraction of strain images Universidade de Sao ˜ Paulo Faculdade de Medicina Hospital das Cl´ ınicas. from radio frequency data [12], [13], and for processing of 2020 IEEE. Personal use of this material is permitted. Permission from conventional SWE sequences [14], we train our network to IEEE must be obtained for all other uses, in any current or future media, directly map B-mode ultrasound towards the corresponding including reprinting/republishing this material for advertising or promotional elasticity images obtained through SWE. purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. arXiv:1908.03573v2 [eess.IV] 4 Apr 2020 2 quality. Pre-processing involved alignment of the side-by-side B-mode and SWE data, followed by physical regridding. This entailed downsampling onto a conveniently-scaled 9664 grid from an original extracted size of 600400 pixels by means of bilinear interpolation. The B-mode images were subsequently scaled from 0-255 to 0-1. Likewise, the elastography data were scaled by 100 kPa so that clinically-relevant Young’s moduli also scale from 0 to 1. In addition, full-screen B-mode images (sized 600900 with a pixel spacing of 0.095 mm, focus in the centre of the gland, and a zoom of 120%) were obtained in roughly the same imaging planes using the B-mode-only acquisition protocol. As these B-mode images are formed outside of the (side- by-side) SWE sequence, they exhibit a different resolution before regridding, slightly different image features, and any information leak from the SWE formation into the B-mode is prevented. These images were not used for training. In order to establish the use of sSWE in a device that does not feature SWE itself, B-mode and QSE recordings were performed in 10 patients at the Academic Medical Cen- tre (University Hospital, Amsterdam) using an iU22 scanner (Philips Healthcare, Bothell, WA) equipped with a C10-3v probe. The extracted images were 450300 pixels in size (i.e., downsample factor of 4.7), had a focus positioned in the middle of the prostate, and a pixel size of 0.12 mm. QSE allows for the extraction of relative stiffness by assessment of tissue compression and decompression upon cyclic manual pressure asserted by the ultrasound operator [7]. These quasi- static elastograms allow a qualitative evaluation of sSWE, as these images reflect elasticity on a larger scale than SWE’s pixelwise quantification due to the operator-dependency of Fig. 1: Schematic implementation of conventional SWE and QSE and its relative rather than absolute nature. synthetic SWE. In addition, we studied the applicability of sSWE in other organs by using SWE recordings of thyroid nodules. For this, we used a different dataset collected in 215 patients at the II. MATERIALS AND M ETHODS Hospital das Cl´ ınicas da Faculdade de Medicina da Universi- A. Data Acquisition dade in Sao ˜ Paulo, Brazil. The recordings were obtained using a LOGIQ E9 ultrasound device (GE HealthCare, Wauwasota, At the Martini Clinic in Hamburg, supersonic shear imaging WI, USA) equipped with a 9L probe. Approximately three was performed in 50 patients that were diagnosed with prostate images per patient were collected, with an extracted size of cancer using the Aixplorer ultrasound scanner (SuperSonic 480320 pixels (i.e., downsample factor of 5), pixel size of Imagine, Aix-en-Provence, France). At least 3 image planes 0.12 mm, and focus at approximately 2 cm. Here, the shear- (basal, mid-gland and apical orientation) were recorded per wave speed was extracted from the images as measure for patient, defining regions of interest (ROIs) that covered the elasticity. Subsequently, the elastograms were subjected to entire prostate and smaller ROIs that only covered one side or normalisation (i.e., by 10 m/s, also to scale the clinically- a suspicious area. The acquired SWE images were obtained relevant speeds between 0 and 1) and the same pre-processing with minimal preload and such that a steady position was procedure as the prostate recordings. The confidence of the maintained for 5 s [15]; the resulting images had a pixel shear-wave speed estimation (comparable to the confidence spacing of 0.16 mm, a zoom of 120%, a focus positioned above measure available in the prostate recordings) could not be the anterior edge of the gland and a maximum elasticity scale extracted. of 70 kPa. At least 9 and at most 15 images were obtained per patient. B. Neural Network Architecture In some cases, the operator chose to make additional SWE images of suspicious structures or repeat an acquisition to We designed a DCNN that serves as an end-to-end nonlinear obtain a higher-quality image; to avoid introducing any bias mapping function transforming 2D B-mode ultrasound images by image selection, all images were used for training. We to 2D synthetic SWE images. To this end, we employ a U- extracted the Young’s modulus data from the SWE acquisi- net-like architecture [16], featuring a general encoder-decoder tions, as well as the estimation confidence (ranging from 0 to shape in which a hierarchy of features is consecutively ex- 1) calculated by the machine to reflect the local elastogram tracted from the B-mode data to yield a latent feature space. 3 Fig. 2: Schematic representation of the proposed DCNN architecture for the synthesis of shear-wave elastography from conventional B-mode ultrasound. These features are subsequently used to construct an SWE network comprises a total of 158,177 trainable parameters. image by a decoding network that approximately mirrors the Max-pooling layers in the decoder are replaced by upsample encoding part. This type of network has been used frequently layers that restore the original image dimensions through for image segmentation and reconstruction tasks [17]–[19]. nearest-neighbour interpolation. The final output layer consists The network was contained direct skip connections from the of a sigmoid activation function that maps the network outputs encoder filter layer to its equally-sized decoder counterpart, as to the normalized Young’s modulus. The use of a sigmoid introduced by [16]. By transferring the encoder layer output activation function forces the network to be the most sensitive across the latent space and concatenating it to the larger-scale to values around 0.5 [22]. Due to the normalization by 100 model features during decoding, we enable our network to kPa, the network therefore focuses the most clinically relevant combine fine and course level information and generate higher- Young’s moduli which are in the range between 25 kPa and resolution SWE estimations. See Figure 2 for an overview of 75 kPa [23]. The same holds for the shear-wave speed range the DCNN architecture. of 2.5 to 7.5 m/s within the thyroid [24]. The convolutional layers of the proposed network comprised a bank of 2D 33-pixel convolutional filters (described by C. Training Strategy the filter weights) and biases of which the results were Optimization of the trainable DCNN parameters  was subsequently passed through a non-linear activation function. achieved through minimization of the root-mean-square pre- Every convolution layer maps its input to 32 feature maps. diction error (RMSE). Given a set of SWE images Y and Leaky Rectified Linear Units (Leaky ReLUs; i.e., f(x) = corresponding B-mode images X, we iteratively update the max(  x; x)) with an -value of 0.1 were adopted as non- parameters  in our network such that the loss of the estimated linear activation functions to minimize the risk of vanishing sSWE images F(X ; ) with regard to Y is minimized: gradients [20]. Every two convolutional layers were followed by a 22 u X spatial max-pooling operation with stride of 2, reducing the L () = jY F(X ; )j : (1) RMSE i i image dimensions with a factor 2 and forcing the network to i=1 subsequently learn larger-scale features that are less sensitive In this formulation, N is the number of training images. to local variations. The max pooling operation reduces a kernel of four pixels into one by projecting only the highest value Network parameters were learned by employment of the onto the smaller grid [21]. In total, the encoder consists of stochastic optimization method Adam [25] in 2,100 epochs, 4 convolutional and 2 max-pooling layers mapping the input using a mini-batch size of 64 training samples for each itera- images into the latent space, which consists of 2 convolutional tion. These values were chosen after performing a preliminary layers as well. With the decoder being a mirrored version grid-search-based optimization procedure including the batch of the encoder layer, appended with a final output layer, the size (i.e., 16, 32, and 64), the number of layers (i.e., 2, 3, 4 Fig. 3: Examples from the ten test patients, with (a) B-mode ultrasound imaging, (b) shear-wave elastographic acquisition, (c) corresponding synthetic SWE (sSWE) image by deep learning, and (d) difference image between sSWE and SWE showing the error as a percentage of the original sSWE value. 5 and 4 sets of 2-layer blocks in the encoder), and the number allows us to put more weight on the accurate estimation of of epochs (i.e., 2,100; 2,450; and 2,800), using 30 patients occasionally-occurring lesions in otherwise low-to-medium- from the training set for training and 10 for validation. The elasticity images. For validation we also considered the mean relatively small batch size is favourable for its looser memory error (ME), requirements, while preserving an appropriate convergence rate [26]. All filter weights were initialized by a random L () = (Y F(X ; )); (3) ME i i uniform kernel initializer over the range [-0.05, 0.05] and i=1 all biases were initialized to zero. An adaptive learning rate a measure that reflects a potential bias towards higher or reduction strategy was used to reduce the learning rate once lower Young’s moduli. the optimization reached a plateau for 10 epochs. In order to study to what extent higher-level features Whereas B-mode data were available for the full image are independent from the machine used from the B-mode space, SWE values are only estimated in a certain region acquisition, we encoded both the B-mode images recorded of interest. Moreover, SWE analysis allows for a measure of with the Philips iU22 scanner and the B-mode images from estimation confidence and, usually, low-confidence values are the test set obtained with the original SuperSonic Aixplorer displayed more transparently or not at all. We exploited this device. Subsequently, we examined the latent feature space information by only propagating loss gradients for those pixels through t-Distributed Stochastic Neighbor Embedding (t-SNE), presenting an SWE label of sufficient quality, using a >0.75- a probabilistic approach to dimensionality reduction [29]. confidence threshold as determined by qualitative assessment As tissue structures differ from organ to organ, the thyroid of the confidence maps. DCNN was separately trained with imaging data of 165 Generalizability was promoted through data augmentation, patients and subsequently validated against the test set of altering a heuristically chosen 90% of the mini-batch data the remaining 50 patients. No alterations with regard to the before being fed into the network [27]. Data augmentation network architecture, processing steps, and training procedure entailed mirroring and cropping of the image by maximum used for the prostate dataset were made, other than the use 5% on all sides, contrast reduction or amplification with a of the normalised SWE speed as a measure of elasticity. The maximum of 50%, random rotation with a maximum of 10 same performance measures were used. degrees, and full image translation with a maximum of 50% laterally and 10% axially.. All coordinate transformations were also applied to the SWE labels. Furthermore, we applied drop- III. RESULTS out after each max-pooling step to avoid overfitting [28]. This regularization method involves the removal of (in our case In Figure 3, sSWE examples from five test patients are 50% of the) nodes in a random fashion at each training epoch, depicted alongside the B-mode and corresponding SWE im- while switching on all units during testing. As a consequence, ages. Over the test set, we were able to reach a per-patient inference is based on an approximate average of all these RMSE of 8.81.7 kPa, an ME of -1.61.6 kPa and an MAE trained dropout networks [28], acting as an ensemble. of 4.50.96 kPa. The negative ME reveals that the model is The model was implemented using Keras with the Ten- slightly biased towards higher SWE estimates. Qualitatively, sorFlow (Google, Mountain View, CA) back-end. Both for tumour locations recognizable on SWE seem to be well training and inference, we employed a Titan XP (NVIDIA, estimated also by the sSWE. Outside of the prostate, the Santa Clara, CA). SWE as well as the sSWE are generally of lower quality. The importance of data augmentation is demonstrated by the fact that the performance of the network drops when augmenta- D. Validation methodology tion is omitted from the procedure, exhibiting an RMSE of Prior to training, our prostate dataset was divided in a 9.92.3 kPa (p=0.0035), an ME of -1.71.6 kPa (p=0.59) training set of 40 patients (consisting of 375 transrectal side- and an MAE of 4.81.2 kPa (p=0.026). The reported p-values by-side B-mode-SWE images with a varying region-of-interest reflect the significance of the improvement as evaluated by a size, as the elastogram size was adjusted to fit (half of) the paired t-test. prostate cross-section during the acquisition) and a test set Using full-screen B-mode acquisition of the same imaging of 10 patients (30 images). All images from the training-set planes, we demonstrate the ability of sSWE to generalise to B- patients were used to maximize the training input and reduce mode images outside the SWE module. These B-mode images the impact of artefacts, whereas only the three full-prostate exhibit a different resolution and contrast compared to the images of each test patient were used during testing to ensure side-by-side B-mode images, and since they were obtained that all prostate regions equally contributed to the validation. separately, the use of shear waves in the acquisition sequence To evaluate the performance of the DCNN, both the RMSE of side-by-side imaging cannot have played a confounding and mean absolute error (MAE) were monitored: role. Nevertheless, even though we allowed the probe to put X more pressure on the prostate, generally bringing the prostate L () = jY F(X ; )j: (2) MAE i i closer into view, Figure 4 shows how the results of these i=1 images as input for the trained sSWE model compare well The RMSE was chosen as loss function because it more qualitatively to the corresponding SWE images. This suggests heavily penalizes large errors than the similar MAE, and thus that the DCNN extracts higher-level features that are shared 6 Fig. 4: Examples of sSWE generalisation to full-screen B-mode acquisitions in the test patients depicted in the upper part of Figure 3, with (a) B-mode ultrasound imaging, (b) corresponding shear-wave elastographic acquisition, and (c) corresponding synthetic SWE (sSWE) image by deep learning. false reading. These are indicated by white arrows. Although the prostate-based network could not be applied to generate accurate sSWE images of the thyroid, training of the exact same network architecture with a set of thyroid SWE images resulted in sSWE with a per-patient RMSE of 0.730.24 m/s, an ME of 0.0430.21 m/s and an MAE of 0.340.14 m/s. Without retraining the network on thyroid data, we obtained a per-patient RMSE of 1.010.33 m/s, an ME of 0.260.29 m/s and an MAE of 0.460.20 m/s. Fig. 5: Network output for the full B-mode image of the Typical examples of thyroid sSWE images alongside the last prostate in Figure 4. Extra-prostatic features such as the actual SWE recordings are depicted in Figure 8; sSWE shows bladder (indicated by the white arrow) and a vessel structure general agreement in the differentiation between stiff and soft (indicated by the blue arrow) are mapped as high-elastic areas. regions, but the networks capability to show details is limited. As we did not have access to the SWE confidence, artefacts could not be excluded from training. among transrectal B-mode images in general. As the network is only trained on prostate tissue, the output outside of the prostate boundaries is highly variable and remains unvalidated. IV. D ISCUSSION As highlighted in Figure 5, it is unlikely that even tissues close to the prostate are characterized with the same accuracy. In this work, we describe and validate a DCNN archi- As can be appreciated in Figure 7, depicting the results of t- tecture that provides synthetic SWE images based on B- SNE of the latent feature space, there is only a slight difference mode ultrasound. This approach is in line with other recently- in how data from the iU22 and Aixplorer US scanners is proposed inter-modality image synthesis techniques, such as computed tomography from magnetic resonance images [19], mapped into the resulting two-dimensional subspace. This [30], [31] or vice versa [32]. Validation in 30 full-prostate suggests that the information encoded in the features that SWE images from 10 patients demonstrated a pixel-wise MAE are not specific to the machine with which the data were of 4.50.96 kPa, less than 10% deviation in the clinically- obtained. Figure 6 depicts B-mode images obtained using a relevant elasticity range of 0-70 kPa. Similar results were Philips scanner without an SWE option. It demonstrates that achieved in the thyroid. Accordingly, it seems that B-mode some stiff regions as revealed by sSWE correspond to those ultrasound (patterns) harbours information that can be linked found by QSE, which was available on the device. The fourth example also shows an example of a false-negative, high- tissue elasticity. stiffness region that in healthy individuals marks the transition Although the results in this article show the technical zone of the prostate [23]; the second example shows a similar feasibility of such an approach, the current study is limited 7 Fig. 6: Examples of sSWE results in a non-SWE ultrasound device, with (a) B-mode ultrasound imaging, (b) quasi-static elastographic acquisition, and (c) corresponding synthetic SWE (sSWE) image by deep learning. A second limitation is in that sSWE does not use mechanical stimulation and can therefore only be considered as a surrogate for elasticity imaging. In this respect, we see sSWE as an elasticity-guided method of tissue typing rather than an alterna- tive to US elasticity imaging techniques. As a consequence of relying solely on B-mode acquisitions, however, sSWE would be less sensitive to e.g. probe pressure, motion artefacts, and region of interest [9]. The clinical interpretability of sSWE, also in relation to SWE, should in the future be examined in a blind fashion. A third limitation resides in the fact that networks have to be retrained when imaging another organ. This emerged by applying sSWE in both the prostate and the thyroid, which are positioned differently and composed of different tissues. On the one hand, this shows that standardization of the imaging procedure is essential for sSWE. Training an sSWE network for organ-specific imaging would therefore be an important part of the standardization procedure. On the other hand, it could indicate that the deep network used in sSWE is to some extent tuned to the tissues that are imaged. In terms Fig. 7: Visualization of B-mode images from both the original of performance, sSWE of the thyroid only seems to capture SuperSonic Aixplorer ultrasound scanner and the Philips iU22 stiff and elastic regions on a higher scale. This could be the scanner encoded into high-level features by the DCNN. Reduc- result of a more diverse range of tissues, a lower degree of tion of the dimensionality was carried out through t-Distributed standardization in the acquisition procedure and settings, or the Stochastic Neighbor Embedding into two dimensions. absence of means to exclude low-confidence SWE estimation from the analysis. It should furthermore be noted that these results are prelim- in a few aspects. First of all, the clinical utility of sSWE inary in the sense that only a small dataset of a specific organ remains to be investigated. In fact, the use of SWE itself is and a limited number of machines has been taken into account. still being studied in the clinic [33], [34]. The prostatic SWE To provide more robust evidence for the proof-of-principle images used as input in this study were previously clinically work presented in this paper, a larger and more variant SWE examined for their use in prostate cancer detection, revealing dataset containing different organs and acquisitions should be diagnostic potential, especially when used concurrently with examined. The availability of a higher variety of data might other US-based prostate imaging modalities [35]. The gained also allow the training of a deeper network, which may result experience formed the basis for the qualitative comparison of in more robust and potentially more accurate sSWE estimation. lesion persistence from SWE to sSWE in this work. An in-depth study of SWE images that were incorrectly 8 Fig. 8: Examples of sSWE of the thyroid, with (a) B-mode ultrasound imaging, (b) corresponding shear-wave elastographic acquisition, (c) corresponding synthetic SWE (sSWE) image by deep learning, and (d) difference image between sSWE and SWE showing the error as a percentage of the original sSWE value. estimated might guide towards more effective augmentation identification of anatomical zones [40] and a possible role of techniques or highlight the type of acquisitions that should be sSWE features in computer-aided detection approaches could more abundant in the training set for future data collection. be taken into account [41]. If proven useful, sSWE would be a fast addition to the clinical workflow in situations where In the future, as we already found indications that sSWE conventional SWE is not available or not possible. might be generalisable to other ultrasound machines, the use of domain adaptation techniques to ensure high-quality, machine- independent sSWE should be investigated [36]. As shown in Figure 7, the high-level feature values generally differ little and V. CONCLUSION minimal domain adaptation strategies could already enforce full overlap. For this, for example, shift techniques could In conclusion, we have proposed a DCNN architecture that be utilized to adjust the mean and variance of the latent generates synthetic SWE images based on B-mode ultrasound throughput. Moreover, the proposed network could possibly acquisitions. Although further validation of the method is still be extended with a concurrent estimation of SWE confidence required, development of this technique paves the way towards to identify low-confidence regions due to shear-wave artefacts elasticity-guided tissue characterisation without the need for such as signal voids in (pseudo)liquid lesions or B-mode complex SWE imaging schemes, using B-mode characteristics artefacts such as shadowing or reverberation. Alternatively, to infer mechanical properties. This would eventually enable an sSWE implementation could be extended to predict other SWE-like analysis by basic US scanners, which could even be elasticity-related parameters than the Young’s modulus or low-end systems. shear-wave speed, such as viscosity [37], which is considered an additional biomarker for cancer in e.g. the prostate [38]. At the present moment, however, there is still a lack of accurate VI. ACKNOWLEDGEM ENTS techniques that can assess tissue viscoelastic properties at high spatial resolution allowing the development of such networks. This study has received funding from the Dutch Cancer However, before using sSWE in the clinic, the clinical Society (#UVA2013-5941) and a European Research Council potential of the technique for the diagnosis of e.g. prostate Starting Grant (#280209), and was performed within the cancer should first be investigated. Also its use in registration framework of the IMPULS2-program within the Eindhoven technology using mechanical properties [39], the (automatic) University of Technology in collaboration with Philips. 9 REFERENCES [21] F.-J. H. Marc’Aurelio Ranzato, Y.-L. Boureau, and Y. LeCun, “Unsuper- vised learning of invariant feature hierarchies with applications to object recognition.” [1] J.-M. Correas, A.-M. Tissier, A. Khairoune, G. Khoury, D. Eiss, and [22] J. Han and C. Moraga, “The influence of the sigmoid function parameters O. Hel ´ enon, ´ “Ultrasound elastography of the prostate: State of the art,” on the speed of backpropagation learning,” in International Workshop Diagnostic and Interventional Imaging, vol. 94, no. 5, pp. 551–560, may on Artificial Neural Networks. Springer, 1995, pp. 195–201. [23] O. Rouviere, ` C. Melodelima, A. Hoang Dinh, F. Bratan, G. Pagnoux, [2] F. Sebag, J. Vaillant-Lombard, J. Berbis, V. Griset, J. F. Henry, P. Petit, T. Sanzalone, S. Crouzet, M. Colombel, F. Mege-Leche ` vallier, and and C. Oliver, “Shear Wave Elastography: A New Ultrasound Imaging R. Souchon, “Stiffness of benign and malignant prostate tissue measured Mode for the Differential Diagnosis of Benign and Malignant Thyroid by shear-wave elastography: a preliminary study,” European Radiology, Nodules,” The Journal of Clinical Endocrinology & Metabolism, vol. 95, vol. 27, no. 5, pp. 1858–1866, 2017. no. 12, pp. 5281–5288, dec 2010. [24] C.-K. Zhao and H.-X. Xu, “Ultrasound elastography of the thyroid: [3] R. G. Barr, “Shear wave liver elastography,” Abdominal Radiology, principles and current status,” Ultrasonography, vol. 38, no. 2, p. 106, vol. 43, no. 4, pp. 800–807, 2018. [4] J. M. Chang, W. K. Moon, N. Cho, A. Yi, H. R. Koo, W. Han, D.-Y. [25] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” Noh, H.-G. Moon, and S. J. Kim, “Clinical application of shear wave arXiv preprint arXiv:1412.6980, 2014. elastography (SWE) in the diagnosis of benign and malignant breast [26] D. Csiba and P. Richtarik, ´ “Importance sampling for minibatches,” The diseases,” Breast Cancer Research and Treatment, vol. 129, no. 1, pp. Journal of Machine Learning Research, vol. 19, no. 1, pp. 962–982, 89–97, 2011. [5] M. S. Taljanovic, L. H. Gimber, G. W. Becker, L. D. Latt, A. S. Klauser, [27] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification D. M. Melville, L. Gao, and R. S. Witte, “Shear-Wave Elastography: Ba- with deep convolutional neural networks,” in Advances in neural infor- sic Physics and Musculoskeletal Applications,” RadioGraphics, vol. 37, mation processing systems, 2012, pp. 1097–1105. no. 3, pp. 855–870, may 2017. [28] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut- [6] R. M. S. Sigrist, J. Liau, A. El Kaffas, M. C. Chammas, and J. K. dinov, “Dropout: a simple way to prevent neural networks from over- Willmann, “Ultrasound elastography: review of techniques and clinical fitting,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. applications,” Theranostics, vol. 7, no. 5, p. 1303, 2017. 1929–1958, 2014. [7] J.-L. Gennisson, T. Deffieux, M. Fink, and M. Tanter, “Ultrasound [29] L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,” elastography: Principles and techniques,” Diagnostic and Interventional Journal of machine learning research, vol. 9, no. Nov, pp. 2579–2605, Imaging, vol. 94, no. 5, pp. 487–495, 2013. [8] K. Nightingale, “Acoustic Radiation Force Impulse (ARFI) Imaging: a [30] T. Huynh, Y. Gao, J. Kang, L. Wang, P. Zhang, J. Lian, and D. Shen, Review,” Current medical imaging reviews, vol. 7, no. 4, pp. 328–339, “Estimating CT image from MRI data using structured random forest nov 2011. and auto-context model,” IEEE transactions on medical imaging, vol. 35, [9] P. Bouchet, J.-L. Gennisson, A. Podda, M. Alilet, M. Carrie, ´ and no. 1, pp. 174–183, 2016. S. Aubry, “Artifacts and Technical Restrictions in 2D Shear Wave [31] J. M. Wolterink, A. M. Dinkla, M. H. F. Savenije, P. R. Seevinck, C. A. T. Elastography TT - Artefakte und technische Einschrankungen ¨ bei der van den Berg, and I. Isgum, ˇ “Deep MR to CT synthesis using unpaired 2D-Scherwellen-Elastografie,” Ultraschall in Med, no. EFirst, 2018. data,” in International Workshop on Simulation and Synthesis in Medical [10] A. P. Sarvazyan, O. V. Rudenko, S. D. Swanson, J. Fowlkes, and S. Y. Imaging. Springer, 2017, pp. 14–23. Emelianov, “Shear wave elasticity imaging: a new ultrasonic technology [32] C.-B. Jin, W. Jung, S. Joo, E. Park, A. Y. Saem, I. H. Han, J. I. Lee, of medical diagnostics,” Ultrasound in Medicine & Biology, vol. 24, and X. Cui, “Deep ct to mr synthesis using paired and unpaired data,” no. 9, pp. 1419–1435, 1998. arXiv preprint arXiv:1805.10790, 2018. [11] M. Feigin, D. Freedman, and B. W. Anthony, “A deep learning frame- [33] L. Sang, X.-M. Wang, D.-Y. Xu, and Y.-F. Cai, “Accuracy of shear work for single sided sound speed inversion in medical ultrasound,” wave elastography for the diagnosis of prostate cancer: A meta-analysis,” arXiv preprint arXiv:1810.00322, 2018. Scientific reports, vol. 7, no. 1, p. 1949, may 2017. [12] S. Wu, Z. Gao, Z. Liu, J. Luo, H. Zhang, and S. Li, “Direct reconstruction [34] A. van Hove, P.-H. Savoie, C. Maurin, S. Brunelle, G. Gravis, N. Salem, of ultrasound elastography using an end-to-end deep neural network,” in and J. Walz, “Comparison of image-guided targeted biopsies versus International Conference on Medical Image Computing and Computer- systematic randomized biopsies in the detection of prostate cancer: a Assisted Intervention. Springer, 2018, pp. 374–382. systematic literature review of well-designed studies,” World journal of [13] M. G. Kibria and H. Rivaz, “GLUENet: Ultrasound Elastography Using urology, vol. 32, no. 4, pp. 847–858, 2014. Convolutional Neural Network BT - Simulation, Image Processing, and [35] C. K. Mannaerts, R. R. Wildeboer, S. Remmers, R. A. van Kollenburg, Ultrasound Systems for Assisted Diagnosis and Navigation.” Cham: A. Kajtazovic, J. Hagemann, A. W. Postema, R. J. van Sloun, M. Roobol, Springer International Publishing, 2018, pp. 21–28. D. Tilki, M. Mischi, H. Wijkstra, and G. Salomon, “Multiparametric [14] T. Ahmed and M. Hasan, “SHEAR-net: An End-to-End Deep Learning ultrasound for prostate cancer detection and localization: Correlation of Approach for Single Push Ultrasound Shear Wave Elasticity Imaging,” B-mode, shearwave elastography and contrast-enhanced ultrasound with arXiv preprint arXiv:1902.04845, 2019. radical prostatectomy specimens.” Journal of Urology, vol. 202, no. 6, [15] C. K. Mannaerts, R. R. Wildeboer, A. W. Postema, J. Hagemann, pp. 1166–1173, jun 2019. L. Budaus, ¨ D. Tilki, M. Mischi, H. Wijkstra, and G. Salomon, “Multi- [36] M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” parametric ultrasound: evaluation of greyscale, shear wave elastography Neurocomputing, vol. 312, pp. 135–153, 2018. and contrast-enhanced ultrasound for prostate cancer detection and [37] R. van Sloun, R. Wildeboer, H. Wijkstra, and M. Mischi, “Viscoelasticity localization in correlation to radical prostatectomy specimens,” BMC Mapping by Identification of Local Shear Wave Dynamics,” IEEE urology, vol. 18, no. 1, p. 98, nov 2018. Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, [16] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks vol. 64, no. 11, pp. 1666–1673, 2017. for biomedical image segmentation,” International Conference on Med- [38] M. Zhang, P. Nigwekar, B. Castaneda, K. Hoyt, J. V. Joseph, ical image computing and computer-assisted intervention, vol. 18, pp. A. di Sant’Agnese, E. M. Messing, J. G. Strang, D. J. Rubens, and 234–241, 2015. K. J. Parker, “Quantitative Characterization of Viscoelastic Properties of [17] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep con- Human Prostate Correlated with Histology,” Ultrasound in Medicine & volutional encoder-decoder architecture for image segmentation,” IEEE Biology, vol. 34, no. 7, pp. 1033–1042, jul 2008. transactions on pattern analysis and machine intelligence, vol. 39, [39] R. R. Wildeboer, R. J. G. van Sloun, A. W. Postema, C. K. Mannaerts, no. 12, pp. 2481–2495, 2017. M. Gayet, H. P. Beerlage, H. Wijkstra, and M. Mischi, “Accurate vali- [18] H. Noh, S. Hong, and B. Han, “Learning deconvolution network for dation of ultrasound imaging of prostate cancer: a review of challenges semantic segmentation,” in Proceedings of the IEEE international con- in registration of imaging and histopathology,” Journal of Ultrasound, ference on computer vision, 2015, pp. 1520–1528. vol. 21, no. 3, pp. 197–207, 2018. [19] X. Han, “MRbased synthetic CT generation using a deep convolutional [40] R. J. van Sloun, R. R. Wildeboer, C. K. Mannaerts, A. W. Postema, neural network method,” Medical physics, vol. 44, no. 4, pp. 1408–1419, M. Gayet, H. P. Beerlage, G. Salomon, H. Wijkstra, and M. Mischi, 2017. “Deep Learning for Real-time, Automatic, and Scanner-adapted Prostate [20] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities im- (Zone) Segmentation of Transrectal Ultrasound, for Example, Magnetic prove neural network acoustic models,” in 30 th International Conference Resonance Imagingtransrectal Ultrasound Fusion Prostate Biopsy,” Eu- on Machine Learning, Atlanta, Georgia, 2013. ropean Urology Focus, vol. in press, 2019. 10 [41] G. Lemaˆ ıtre, R. Mart´ ı, J. Freixenet, J. C. Vilanova, P. M. Walker, and F. Meriaudeau, “Computer-Aided Detection and diagnosis for prostate cancer based on mono and multi-parametric MRI: A review,” Computers in Biology and Medicine, vol. 60, pp. 8–31, may 2015. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Electrical Engineering and Systems Science arXiv (Cornell University)

Synthetic Elastography using B-mode Ultrasound through a Deep Fully-Convolutional Neural Network

Loading next page...
 
/lp/arxiv-cornell-university/synthetic-elastography-using-b-mode-ultrasound-through-a-deep-fully-0l0VOWUNrC

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

ISSN
0885-3010
eISSN
ARCH-3348
DOI
10.1109/TUFFC.2020.2983099
Publisher site
See Article on Publisher Site

Abstract

Synthetic Elastography using B-mode Ultrasound through a Deep Fully-Convolutional Neural Network R. R. Wildeboer, R. J. G. van Sloun, C. K. Mannaerts, P. H. Moraes, G. Salomon, M.C. Chammas, H. Wijkstra, M. Mischi Abstract—Shear-wave elastography (SWE) permits local es- Ultrasound-based elasticity imaging, that is, ultrasound elas- timation of tissue elasticity, an important imaging marker in tography, has played a major role in these developments biomedicine. This recently-developed, advanced technique as- [6]. So-called quasi-static elastographic (QSE) strain imag- sesses the speed of a laterally-travelling shear wave after an ing allows for the relative assessment of tissue deformation acoustic radiation force “push” to estimate local Young’s moduli due to externally applied stress, but as this stress is often in an operator-independent fashion. In this work, we show how synthetic SWE (sSWE) images can be generated based on manually delivered, the technique remains operator dependent conventional B-mode imaging through deep learning. Using side- and limited to superficial organs. Therefore, more recently, by-side-view B-mode/SWE images collected in 50 patients with dynamic elastography techniques were developed where tissue prostate cancer, we show that sSWE images with a pixel-wise deformation induced by an acoustic radiation force “push” mean absolute error of 4.50.96 kPa with regard to the original pulse is quantified to obtain more objective and reproducible SWE can be generated. Visualization of high-level feature levels through t-Distributed Stochastic Neighbor Embedding reveals measures of elasticity [7]. At this moment, we distinguish substantial overlap between data from two different scanners. especially acoustic radiation force imaging (ARFI) and shear- Qualitatively, we examined the use of the sSWE methodology wave elastography [5], [8]. The first method analyses tissue for B-mode images obtained with a scanner without SWE displacement resulting from a push pulse along the beam path, functionality. We also examined the use of this type of network whereas the latter relies on the speed of transversally-travelling in elasticity imaging in the thyroid. Limitations of the technique reside in the fact that networks have to be retrained for different shear waves to estimate tissue elasticity. The tissue elasticity is organs, and that the method requires standardization of the quantified by the Young’s modulus, that is, the ratio between imaging settings and procedure. Future research will be aimed stress and strain. at development of sSWE as an elasticity-related tissue typing strategy that is solely based on B-mode ultrasound acquisition, and the examination of its clinical utility. SWE requires advanced ultrafast acquisition schemes with Index Terms—Shear-Wave Elastography, Deep Learning, Con- frame rates of 1000 Hz to accurately assess tissue deforma- volutional Neural Networks, B-mode Ultrasound tion and shear-wave dynamics [7], [9]. Moreover, ultrasound transducers have to be sufficiently equipped to allow for the I. INTRODUCTION generation of acoustic radiation force pulses as well as ultrafast imaging of the shear wave displacements [10]. Although sev- Tissue elasticity is an important biomarker of cancer. Prostate cancer, for example, is characterized by increased eral techniques and sequences have been developed to enable stiffness [1], thyroid and liver nodules can be discriminated SWE on commercial scanners, the frame rate of conventional based on their elasticity [2], [3], and also breast lesions are B-mode ultrasound cannot be reached as it requires long typically diagnosed based on their elastic properties [4]. It is settling times and multiple “push” pulses to reliably generate also increasingly used to image musculoskeletal pathologies an elastogram. in e.g. muscles, tendons, and ligaments [5]. Over the last few decades, this has spurred considerable advances in the Realizing that conventional B-mode ultrasound assesses development of elasticity imaging. tissue echogenicity rather than tissue elasticity, we here R.R. Wildeboer, R.J.G van Sloun, H. Wijkstra and M. Mischi are propose that both properties can be expected to be linked with the Lab of Biomedical Diagnostics, Department of Electrical Engi- through their dependence on the underlying tissue structure. neering, Eindhoven University of Technology, The Netherlands. (e-mail: In this work, we exploit this fact by designing a deep r.r.wildeboer@tue.nl; r.j.g.v.sloun@tue.nl) C.K. Mannaerts and H. Wijkstra are with the Department of Urology, fully-convolutional neural network (DCNN) that is able to Academic University Medical Centres, University of Amsterdam, The Nether- assess echogenic patterns in B-mode ultrasound that are lands. useful for elasticity-related tissue typing (see Figure 1). G. Salomon is with the Department of Urology, University Hospital Hamburg-Eppendorf, Germany. Whereas deep-learning strategies were already proposed for P. H. Moraes and M.C. Chammas are with the Department of Radiology, estimation of speed of sound [11], extraction of strain images Universidade de Sao ˜ Paulo Faculdade de Medicina Hospital das Cl´ ınicas. from radio frequency data [12], [13], and for processing of 2020 IEEE. Personal use of this material is permitted. Permission from conventional SWE sequences [14], we train our network to IEEE must be obtained for all other uses, in any current or future media, directly map B-mode ultrasound towards the corresponding including reprinting/republishing this material for advertising or promotional elasticity images obtained through SWE. purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. arXiv:1908.03573v2 [eess.IV] 4 Apr 2020 2 quality. Pre-processing involved alignment of the side-by-side B-mode and SWE data, followed by physical regridding. This entailed downsampling onto a conveniently-scaled 9664 grid from an original extracted size of 600400 pixels by means of bilinear interpolation. The B-mode images were subsequently scaled from 0-255 to 0-1. Likewise, the elastography data were scaled by 100 kPa so that clinically-relevant Young’s moduli also scale from 0 to 1. In addition, full-screen B-mode images (sized 600900 with a pixel spacing of 0.095 mm, focus in the centre of the gland, and a zoom of 120%) were obtained in roughly the same imaging planes using the B-mode-only acquisition protocol. As these B-mode images are formed outside of the (side- by-side) SWE sequence, they exhibit a different resolution before regridding, slightly different image features, and any information leak from the SWE formation into the B-mode is prevented. These images were not used for training. In order to establish the use of sSWE in a device that does not feature SWE itself, B-mode and QSE recordings were performed in 10 patients at the Academic Medical Cen- tre (University Hospital, Amsterdam) using an iU22 scanner (Philips Healthcare, Bothell, WA) equipped with a C10-3v probe. The extracted images were 450300 pixels in size (i.e., downsample factor of 4.7), had a focus positioned in the middle of the prostate, and a pixel size of 0.12 mm. QSE allows for the extraction of relative stiffness by assessment of tissue compression and decompression upon cyclic manual pressure asserted by the ultrasound operator [7]. These quasi- static elastograms allow a qualitative evaluation of sSWE, as these images reflect elasticity on a larger scale than SWE’s pixelwise quantification due to the operator-dependency of Fig. 1: Schematic implementation of conventional SWE and QSE and its relative rather than absolute nature. synthetic SWE. In addition, we studied the applicability of sSWE in other organs by using SWE recordings of thyroid nodules. For this, we used a different dataset collected in 215 patients at the II. MATERIALS AND M ETHODS Hospital das Cl´ ınicas da Faculdade de Medicina da Universi- A. Data Acquisition dade in Sao ˜ Paulo, Brazil. The recordings were obtained using a LOGIQ E9 ultrasound device (GE HealthCare, Wauwasota, At the Martini Clinic in Hamburg, supersonic shear imaging WI, USA) equipped with a 9L probe. Approximately three was performed in 50 patients that were diagnosed with prostate images per patient were collected, with an extracted size of cancer using the Aixplorer ultrasound scanner (SuperSonic 480320 pixels (i.e., downsample factor of 5), pixel size of Imagine, Aix-en-Provence, France). At least 3 image planes 0.12 mm, and focus at approximately 2 cm. Here, the shear- (basal, mid-gland and apical orientation) were recorded per wave speed was extracted from the images as measure for patient, defining regions of interest (ROIs) that covered the elasticity. Subsequently, the elastograms were subjected to entire prostate and smaller ROIs that only covered one side or normalisation (i.e., by 10 m/s, also to scale the clinically- a suspicious area. The acquired SWE images were obtained relevant speeds between 0 and 1) and the same pre-processing with minimal preload and such that a steady position was procedure as the prostate recordings. The confidence of the maintained for 5 s [15]; the resulting images had a pixel shear-wave speed estimation (comparable to the confidence spacing of 0.16 mm, a zoom of 120%, a focus positioned above measure available in the prostate recordings) could not be the anterior edge of the gland and a maximum elasticity scale extracted. of 70 kPa. At least 9 and at most 15 images were obtained per patient. B. Neural Network Architecture In some cases, the operator chose to make additional SWE images of suspicious structures or repeat an acquisition to We designed a DCNN that serves as an end-to-end nonlinear obtain a higher-quality image; to avoid introducing any bias mapping function transforming 2D B-mode ultrasound images by image selection, all images were used for training. We to 2D synthetic SWE images. To this end, we employ a U- extracted the Young’s modulus data from the SWE acquisi- net-like architecture [16], featuring a general encoder-decoder tions, as well as the estimation confidence (ranging from 0 to shape in which a hierarchy of features is consecutively ex- 1) calculated by the machine to reflect the local elastogram tracted from the B-mode data to yield a latent feature space. 3 Fig. 2: Schematic representation of the proposed DCNN architecture for the synthesis of shear-wave elastography from conventional B-mode ultrasound. These features are subsequently used to construct an SWE network comprises a total of 158,177 trainable parameters. image by a decoding network that approximately mirrors the Max-pooling layers in the decoder are replaced by upsample encoding part. This type of network has been used frequently layers that restore the original image dimensions through for image segmentation and reconstruction tasks [17]–[19]. nearest-neighbour interpolation. The final output layer consists The network was contained direct skip connections from the of a sigmoid activation function that maps the network outputs encoder filter layer to its equally-sized decoder counterpart, as to the normalized Young’s modulus. The use of a sigmoid introduced by [16]. By transferring the encoder layer output activation function forces the network to be the most sensitive across the latent space and concatenating it to the larger-scale to values around 0.5 [22]. Due to the normalization by 100 model features during decoding, we enable our network to kPa, the network therefore focuses the most clinically relevant combine fine and course level information and generate higher- Young’s moduli which are in the range between 25 kPa and resolution SWE estimations. See Figure 2 for an overview of 75 kPa [23]. The same holds for the shear-wave speed range the DCNN architecture. of 2.5 to 7.5 m/s within the thyroid [24]. The convolutional layers of the proposed network comprised a bank of 2D 33-pixel convolutional filters (described by C. Training Strategy the filter weights) and biases of which the results were Optimization of the trainable DCNN parameters  was subsequently passed through a non-linear activation function. achieved through minimization of the root-mean-square pre- Every convolution layer maps its input to 32 feature maps. diction error (RMSE). Given a set of SWE images Y and Leaky Rectified Linear Units (Leaky ReLUs; i.e., f(x) = corresponding B-mode images X, we iteratively update the max(  x; x)) with an -value of 0.1 were adopted as non- parameters  in our network such that the loss of the estimated linear activation functions to minimize the risk of vanishing sSWE images F(X ; ) with regard to Y is minimized: gradients [20]. Every two convolutional layers were followed by a 22 u X spatial max-pooling operation with stride of 2, reducing the L () = jY F(X ; )j : (1) RMSE i i image dimensions with a factor 2 and forcing the network to i=1 subsequently learn larger-scale features that are less sensitive In this formulation, N is the number of training images. to local variations. The max pooling operation reduces a kernel of four pixels into one by projecting only the highest value Network parameters were learned by employment of the onto the smaller grid [21]. In total, the encoder consists of stochastic optimization method Adam [25] in 2,100 epochs, 4 convolutional and 2 max-pooling layers mapping the input using a mini-batch size of 64 training samples for each itera- images into the latent space, which consists of 2 convolutional tion. These values were chosen after performing a preliminary layers as well. With the decoder being a mirrored version grid-search-based optimization procedure including the batch of the encoder layer, appended with a final output layer, the size (i.e., 16, 32, and 64), the number of layers (i.e., 2, 3, 4 Fig. 3: Examples from the ten test patients, with (a) B-mode ultrasound imaging, (b) shear-wave elastographic acquisition, (c) corresponding synthetic SWE (sSWE) image by deep learning, and (d) difference image between sSWE and SWE showing the error as a percentage of the original sSWE value. 5 and 4 sets of 2-layer blocks in the encoder), and the number allows us to put more weight on the accurate estimation of of epochs (i.e., 2,100; 2,450; and 2,800), using 30 patients occasionally-occurring lesions in otherwise low-to-medium- from the training set for training and 10 for validation. The elasticity images. For validation we also considered the mean relatively small batch size is favourable for its looser memory error (ME), requirements, while preserving an appropriate convergence rate [26]. All filter weights were initialized by a random L () = (Y F(X ; )); (3) ME i i uniform kernel initializer over the range [-0.05, 0.05] and i=1 all biases were initialized to zero. An adaptive learning rate a measure that reflects a potential bias towards higher or reduction strategy was used to reduce the learning rate once lower Young’s moduli. the optimization reached a plateau for 10 epochs. In order to study to what extent higher-level features Whereas B-mode data were available for the full image are independent from the machine used from the B-mode space, SWE values are only estimated in a certain region acquisition, we encoded both the B-mode images recorded of interest. Moreover, SWE analysis allows for a measure of with the Philips iU22 scanner and the B-mode images from estimation confidence and, usually, low-confidence values are the test set obtained with the original SuperSonic Aixplorer displayed more transparently or not at all. We exploited this device. Subsequently, we examined the latent feature space information by only propagating loss gradients for those pixels through t-Distributed Stochastic Neighbor Embedding (t-SNE), presenting an SWE label of sufficient quality, using a >0.75- a probabilistic approach to dimensionality reduction [29]. confidence threshold as determined by qualitative assessment As tissue structures differ from organ to organ, the thyroid of the confidence maps. DCNN was separately trained with imaging data of 165 Generalizability was promoted through data augmentation, patients and subsequently validated against the test set of altering a heuristically chosen 90% of the mini-batch data the remaining 50 patients. No alterations with regard to the before being fed into the network [27]. Data augmentation network architecture, processing steps, and training procedure entailed mirroring and cropping of the image by maximum used for the prostate dataset were made, other than the use 5% on all sides, contrast reduction or amplification with a of the normalised SWE speed as a measure of elasticity. The maximum of 50%, random rotation with a maximum of 10 same performance measures were used. degrees, and full image translation with a maximum of 50% laterally and 10% axially.. All coordinate transformations were also applied to the SWE labels. Furthermore, we applied drop- III. RESULTS out after each max-pooling step to avoid overfitting [28]. This regularization method involves the removal of (in our case In Figure 3, sSWE examples from five test patients are 50% of the) nodes in a random fashion at each training epoch, depicted alongside the B-mode and corresponding SWE im- while switching on all units during testing. As a consequence, ages. Over the test set, we were able to reach a per-patient inference is based on an approximate average of all these RMSE of 8.81.7 kPa, an ME of -1.61.6 kPa and an MAE trained dropout networks [28], acting as an ensemble. of 4.50.96 kPa. The negative ME reveals that the model is The model was implemented using Keras with the Ten- slightly biased towards higher SWE estimates. Qualitatively, sorFlow (Google, Mountain View, CA) back-end. Both for tumour locations recognizable on SWE seem to be well training and inference, we employed a Titan XP (NVIDIA, estimated also by the sSWE. Outside of the prostate, the Santa Clara, CA). SWE as well as the sSWE are generally of lower quality. The importance of data augmentation is demonstrated by the fact that the performance of the network drops when augmenta- D. Validation methodology tion is omitted from the procedure, exhibiting an RMSE of Prior to training, our prostate dataset was divided in a 9.92.3 kPa (p=0.0035), an ME of -1.71.6 kPa (p=0.59) training set of 40 patients (consisting of 375 transrectal side- and an MAE of 4.81.2 kPa (p=0.026). The reported p-values by-side B-mode-SWE images with a varying region-of-interest reflect the significance of the improvement as evaluated by a size, as the elastogram size was adjusted to fit (half of) the paired t-test. prostate cross-section during the acquisition) and a test set Using full-screen B-mode acquisition of the same imaging of 10 patients (30 images). All images from the training-set planes, we demonstrate the ability of sSWE to generalise to B- patients were used to maximize the training input and reduce mode images outside the SWE module. These B-mode images the impact of artefacts, whereas only the three full-prostate exhibit a different resolution and contrast compared to the images of each test patient were used during testing to ensure side-by-side B-mode images, and since they were obtained that all prostate regions equally contributed to the validation. separately, the use of shear waves in the acquisition sequence To evaluate the performance of the DCNN, both the RMSE of side-by-side imaging cannot have played a confounding and mean absolute error (MAE) were monitored: role. Nevertheless, even though we allowed the probe to put X more pressure on the prostate, generally bringing the prostate L () = jY F(X ; )j: (2) MAE i i closer into view, Figure 4 shows how the results of these i=1 images as input for the trained sSWE model compare well The RMSE was chosen as loss function because it more qualitatively to the corresponding SWE images. This suggests heavily penalizes large errors than the similar MAE, and thus that the DCNN extracts higher-level features that are shared 6 Fig. 4: Examples of sSWE generalisation to full-screen B-mode acquisitions in the test patients depicted in the upper part of Figure 3, with (a) B-mode ultrasound imaging, (b) corresponding shear-wave elastographic acquisition, and (c) corresponding synthetic SWE (sSWE) image by deep learning. false reading. These are indicated by white arrows. Although the prostate-based network could not be applied to generate accurate sSWE images of the thyroid, training of the exact same network architecture with a set of thyroid SWE images resulted in sSWE with a per-patient RMSE of 0.730.24 m/s, an ME of 0.0430.21 m/s and an MAE of 0.340.14 m/s. Without retraining the network on thyroid data, we obtained a per-patient RMSE of 1.010.33 m/s, an ME of 0.260.29 m/s and an MAE of 0.460.20 m/s. Fig. 5: Network output for the full B-mode image of the Typical examples of thyroid sSWE images alongside the last prostate in Figure 4. Extra-prostatic features such as the actual SWE recordings are depicted in Figure 8; sSWE shows bladder (indicated by the white arrow) and a vessel structure general agreement in the differentiation between stiff and soft (indicated by the blue arrow) are mapped as high-elastic areas. regions, but the networks capability to show details is limited. As we did not have access to the SWE confidence, artefacts could not be excluded from training. among transrectal B-mode images in general. As the network is only trained on prostate tissue, the output outside of the prostate boundaries is highly variable and remains unvalidated. IV. D ISCUSSION As highlighted in Figure 5, it is unlikely that even tissues close to the prostate are characterized with the same accuracy. In this work, we describe and validate a DCNN archi- As can be appreciated in Figure 7, depicting the results of t- tecture that provides synthetic SWE images based on B- SNE of the latent feature space, there is only a slight difference mode ultrasound. This approach is in line with other recently- in how data from the iU22 and Aixplorer US scanners is proposed inter-modality image synthesis techniques, such as computed tomography from magnetic resonance images [19], mapped into the resulting two-dimensional subspace. This [30], [31] or vice versa [32]. Validation in 30 full-prostate suggests that the information encoded in the features that SWE images from 10 patients demonstrated a pixel-wise MAE are not specific to the machine with which the data were of 4.50.96 kPa, less than 10% deviation in the clinically- obtained. Figure 6 depicts B-mode images obtained using a relevant elasticity range of 0-70 kPa. Similar results were Philips scanner without an SWE option. It demonstrates that achieved in the thyroid. Accordingly, it seems that B-mode some stiff regions as revealed by sSWE correspond to those ultrasound (patterns) harbours information that can be linked found by QSE, which was available on the device. The fourth example also shows an example of a false-negative, high- tissue elasticity. stiffness region that in healthy individuals marks the transition Although the results in this article show the technical zone of the prostate [23]; the second example shows a similar feasibility of such an approach, the current study is limited 7 Fig. 6: Examples of sSWE results in a non-SWE ultrasound device, with (a) B-mode ultrasound imaging, (b) quasi-static elastographic acquisition, and (c) corresponding synthetic SWE (sSWE) image by deep learning. A second limitation is in that sSWE does not use mechanical stimulation and can therefore only be considered as a surrogate for elasticity imaging. In this respect, we see sSWE as an elasticity-guided method of tissue typing rather than an alterna- tive to US elasticity imaging techniques. As a consequence of relying solely on B-mode acquisitions, however, sSWE would be less sensitive to e.g. probe pressure, motion artefacts, and region of interest [9]. The clinical interpretability of sSWE, also in relation to SWE, should in the future be examined in a blind fashion. A third limitation resides in the fact that networks have to be retrained when imaging another organ. This emerged by applying sSWE in both the prostate and the thyroid, which are positioned differently and composed of different tissues. On the one hand, this shows that standardization of the imaging procedure is essential for sSWE. Training an sSWE network for organ-specific imaging would therefore be an important part of the standardization procedure. On the other hand, it could indicate that the deep network used in sSWE is to some extent tuned to the tissues that are imaged. In terms Fig. 7: Visualization of B-mode images from both the original of performance, sSWE of the thyroid only seems to capture SuperSonic Aixplorer ultrasound scanner and the Philips iU22 stiff and elastic regions on a higher scale. This could be the scanner encoded into high-level features by the DCNN. Reduc- result of a more diverse range of tissues, a lower degree of tion of the dimensionality was carried out through t-Distributed standardization in the acquisition procedure and settings, or the Stochastic Neighbor Embedding into two dimensions. absence of means to exclude low-confidence SWE estimation from the analysis. It should furthermore be noted that these results are prelim- in a few aspects. First of all, the clinical utility of sSWE inary in the sense that only a small dataset of a specific organ remains to be investigated. In fact, the use of SWE itself is and a limited number of machines has been taken into account. still being studied in the clinic [33], [34]. The prostatic SWE To provide more robust evidence for the proof-of-principle images used as input in this study were previously clinically work presented in this paper, a larger and more variant SWE examined for their use in prostate cancer detection, revealing dataset containing different organs and acquisitions should be diagnostic potential, especially when used concurrently with examined. The availability of a higher variety of data might other US-based prostate imaging modalities [35]. The gained also allow the training of a deeper network, which may result experience formed the basis for the qualitative comparison of in more robust and potentially more accurate sSWE estimation. lesion persistence from SWE to sSWE in this work. An in-depth study of SWE images that were incorrectly 8 Fig. 8: Examples of sSWE of the thyroid, with (a) B-mode ultrasound imaging, (b) corresponding shear-wave elastographic acquisition, (c) corresponding synthetic SWE (sSWE) image by deep learning, and (d) difference image between sSWE and SWE showing the error as a percentage of the original sSWE value. estimated might guide towards more effective augmentation identification of anatomical zones [40] and a possible role of techniques or highlight the type of acquisitions that should be sSWE features in computer-aided detection approaches could more abundant in the training set for future data collection. be taken into account [41]. If proven useful, sSWE would be a fast addition to the clinical workflow in situations where In the future, as we already found indications that sSWE conventional SWE is not available or not possible. might be generalisable to other ultrasound machines, the use of domain adaptation techniques to ensure high-quality, machine- independent sSWE should be investigated [36]. As shown in Figure 7, the high-level feature values generally differ little and V. CONCLUSION minimal domain adaptation strategies could already enforce full overlap. For this, for example, shift techniques could In conclusion, we have proposed a DCNN architecture that be utilized to adjust the mean and variance of the latent generates synthetic SWE images based on B-mode ultrasound throughput. Moreover, the proposed network could possibly acquisitions. Although further validation of the method is still be extended with a concurrent estimation of SWE confidence required, development of this technique paves the way towards to identify low-confidence regions due to shear-wave artefacts elasticity-guided tissue characterisation without the need for such as signal voids in (pseudo)liquid lesions or B-mode complex SWE imaging schemes, using B-mode characteristics artefacts such as shadowing or reverberation. Alternatively, to infer mechanical properties. This would eventually enable an sSWE implementation could be extended to predict other SWE-like analysis by basic US scanners, which could even be elasticity-related parameters than the Young’s modulus or low-end systems. shear-wave speed, such as viscosity [37], which is considered an additional biomarker for cancer in e.g. the prostate [38]. At the present moment, however, there is still a lack of accurate VI. ACKNOWLEDGEM ENTS techniques that can assess tissue viscoelastic properties at high spatial resolution allowing the development of such networks. This study has received funding from the Dutch Cancer However, before using sSWE in the clinic, the clinical Society (#UVA2013-5941) and a European Research Council potential of the technique for the diagnosis of e.g. prostate Starting Grant (#280209), and was performed within the cancer should first be investigated. Also its use in registration framework of the IMPULS2-program within the Eindhoven technology using mechanical properties [39], the (automatic) University of Technology in collaboration with Philips. 9 REFERENCES [21] F.-J. H. Marc’Aurelio Ranzato, Y.-L. Boureau, and Y. LeCun, “Unsuper- vised learning of invariant feature hierarchies with applications to object recognition.” [1] J.-M. Correas, A.-M. Tissier, A. Khairoune, G. Khoury, D. Eiss, and [22] J. Han and C. Moraga, “The influence of the sigmoid function parameters O. Hel ´ enon, ´ “Ultrasound elastography of the prostate: State of the art,” on the speed of backpropagation learning,” in International Workshop Diagnostic and Interventional Imaging, vol. 94, no. 5, pp. 551–560, may on Artificial Neural Networks. Springer, 1995, pp. 195–201. [23] O. Rouviere, ` C. Melodelima, A. Hoang Dinh, F. Bratan, G. Pagnoux, [2] F. Sebag, J. Vaillant-Lombard, J. Berbis, V. Griset, J. F. Henry, P. Petit, T. Sanzalone, S. Crouzet, M. Colombel, F. Mege-Leche ` vallier, and and C. Oliver, “Shear Wave Elastography: A New Ultrasound Imaging R. Souchon, “Stiffness of benign and malignant prostate tissue measured Mode for the Differential Diagnosis of Benign and Malignant Thyroid by shear-wave elastography: a preliminary study,” European Radiology, Nodules,” The Journal of Clinical Endocrinology & Metabolism, vol. 95, vol. 27, no. 5, pp. 1858–1866, 2017. no. 12, pp. 5281–5288, dec 2010. [24] C.-K. Zhao and H.-X. Xu, “Ultrasound elastography of the thyroid: [3] R. G. Barr, “Shear wave liver elastography,” Abdominal Radiology, principles and current status,” Ultrasonography, vol. 38, no. 2, p. 106, vol. 43, no. 4, pp. 800–807, 2018. [4] J. M. Chang, W. K. Moon, N. Cho, A. Yi, H. R. Koo, W. Han, D.-Y. [25] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” Noh, H.-G. Moon, and S. J. Kim, “Clinical application of shear wave arXiv preprint arXiv:1412.6980, 2014. elastography (SWE) in the diagnosis of benign and malignant breast [26] D. Csiba and P. Richtarik, ´ “Importance sampling for minibatches,” The diseases,” Breast Cancer Research and Treatment, vol. 129, no. 1, pp. Journal of Machine Learning Research, vol. 19, no. 1, pp. 962–982, 89–97, 2011. [5] M. S. Taljanovic, L. H. Gimber, G. W. Becker, L. D. Latt, A. S. Klauser, [27] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification D. M. Melville, L. Gao, and R. S. Witte, “Shear-Wave Elastography: Ba- with deep convolutional neural networks,” in Advances in neural infor- sic Physics and Musculoskeletal Applications,” RadioGraphics, vol. 37, mation processing systems, 2012, pp. 1097–1105. no. 3, pp. 855–870, may 2017. [28] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut- [6] R. M. S. Sigrist, J. Liau, A. El Kaffas, M. C. Chammas, and J. K. dinov, “Dropout: a simple way to prevent neural networks from over- Willmann, “Ultrasound elastography: review of techniques and clinical fitting,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. applications,” Theranostics, vol. 7, no. 5, p. 1303, 2017. 1929–1958, 2014. [7] J.-L. Gennisson, T. Deffieux, M. Fink, and M. Tanter, “Ultrasound [29] L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,” elastography: Principles and techniques,” Diagnostic and Interventional Journal of machine learning research, vol. 9, no. Nov, pp. 2579–2605, Imaging, vol. 94, no. 5, pp. 487–495, 2013. [8] K. Nightingale, “Acoustic Radiation Force Impulse (ARFI) Imaging: a [30] T. Huynh, Y. Gao, J. Kang, L. Wang, P. Zhang, J. Lian, and D. Shen, Review,” Current medical imaging reviews, vol. 7, no. 4, pp. 328–339, “Estimating CT image from MRI data using structured random forest nov 2011. and auto-context model,” IEEE transactions on medical imaging, vol. 35, [9] P. Bouchet, J.-L. Gennisson, A. Podda, M. Alilet, M. Carrie, ´ and no. 1, pp. 174–183, 2016. S. Aubry, “Artifacts and Technical Restrictions in 2D Shear Wave [31] J. M. Wolterink, A. M. Dinkla, M. H. F. Savenije, P. R. Seevinck, C. A. T. Elastography TT - Artefakte und technische Einschrankungen ¨ bei der van den Berg, and I. Isgum, ˇ “Deep MR to CT synthesis using unpaired 2D-Scherwellen-Elastografie,” Ultraschall in Med, no. EFirst, 2018. data,” in International Workshop on Simulation and Synthesis in Medical [10] A. P. Sarvazyan, O. V. Rudenko, S. D. Swanson, J. Fowlkes, and S. Y. Imaging. Springer, 2017, pp. 14–23. Emelianov, “Shear wave elasticity imaging: a new ultrasonic technology [32] C.-B. Jin, W. Jung, S. Joo, E. Park, A. Y. Saem, I. H. Han, J. I. Lee, of medical diagnostics,” Ultrasound in Medicine & Biology, vol. 24, and X. Cui, “Deep ct to mr synthesis using paired and unpaired data,” no. 9, pp. 1419–1435, 1998. arXiv preprint arXiv:1805.10790, 2018. [11] M. Feigin, D. Freedman, and B. W. Anthony, “A deep learning frame- [33] L. Sang, X.-M. Wang, D.-Y. Xu, and Y.-F. Cai, “Accuracy of shear work for single sided sound speed inversion in medical ultrasound,” wave elastography for the diagnosis of prostate cancer: A meta-analysis,” arXiv preprint arXiv:1810.00322, 2018. Scientific reports, vol. 7, no. 1, p. 1949, may 2017. [12] S. Wu, Z. Gao, Z. Liu, J. Luo, H. Zhang, and S. Li, “Direct reconstruction [34] A. van Hove, P.-H. Savoie, C. Maurin, S. Brunelle, G. Gravis, N. Salem, of ultrasound elastography using an end-to-end deep neural network,” in and J. Walz, “Comparison of image-guided targeted biopsies versus International Conference on Medical Image Computing and Computer- systematic randomized biopsies in the detection of prostate cancer: a Assisted Intervention. Springer, 2018, pp. 374–382. systematic literature review of well-designed studies,” World journal of [13] M. G. Kibria and H. Rivaz, “GLUENet: Ultrasound Elastography Using urology, vol. 32, no. 4, pp. 847–858, 2014. Convolutional Neural Network BT - Simulation, Image Processing, and [35] C. K. Mannaerts, R. R. Wildeboer, S. Remmers, R. A. van Kollenburg, Ultrasound Systems for Assisted Diagnosis and Navigation.” Cham: A. Kajtazovic, J. Hagemann, A. W. Postema, R. J. van Sloun, M. Roobol, Springer International Publishing, 2018, pp. 21–28. D. Tilki, M. Mischi, H. Wijkstra, and G. Salomon, “Multiparametric [14] T. Ahmed and M. Hasan, “SHEAR-net: An End-to-End Deep Learning ultrasound for prostate cancer detection and localization: Correlation of Approach for Single Push Ultrasound Shear Wave Elasticity Imaging,” B-mode, shearwave elastography and contrast-enhanced ultrasound with arXiv preprint arXiv:1902.04845, 2019. radical prostatectomy specimens.” Journal of Urology, vol. 202, no. 6, [15] C. K. Mannaerts, R. R. Wildeboer, A. W. Postema, J. Hagemann, pp. 1166–1173, jun 2019. L. Budaus, ¨ D. Tilki, M. Mischi, H. Wijkstra, and G. Salomon, “Multi- [36] M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” parametric ultrasound: evaluation of greyscale, shear wave elastography Neurocomputing, vol. 312, pp. 135–153, 2018. and contrast-enhanced ultrasound for prostate cancer detection and [37] R. van Sloun, R. Wildeboer, H. Wijkstra, and M. Mischi, “Viscoelasticity localization in correlation to radical prostatectomy specimens,” BMC Mapping by Identification of Local Shear Wave Dynamics,” IEEE urology, vol. 18, no. 1, p. 98, nov 2018. Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, [16] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks vol. 64, no. 11, pp. 1666–1673, 2017. for biomedical image segmentation,” International Conference on Med- [38] M. Zhang, P. Nigwekar, B. Castaneda, K. Hoyt, J. V. Joseph, ical image computing and computer-assisted intervention, vol. 18, pp. A. di Sant’Agnese, E. M. Messing, J. G. Strang, D. J. Rubens, and 234–241, 2015. K. J. Parker, “Quantitative Characterization of Viscoelastic Properties of [17] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep con- Human Prostate Correlated with Histology,” Ultrasound in Medicine & volutional encoder-decoder architecture for image segmentation,” IEEE Biology, vol. 34, no. 7, pp. 1033–1042, jul 2008. transactions on pattern analysis and machine intelligence, vol. 39, [39] R. R. Wildeboer, R. J. G. van Sloun, A. W. Postema, C. K. Mannaerts, no. 12, pp. 2481–2495, 2017. M. Gayet, H. P. Beerlage, H. Wijkstra, and M. Mischi, “Accurate vali- [18] H. Noh, S. Hong, and B. Han, “Learning deconvolution network for dation of ultrasound imaging of prostate cancer: a review of challenges semantic segmentation,” in Proceedings of the IEEE international con- in registration of imaging and histopathology,” Journal of Ultrasound, ference on computer vision, 2015, pp. 1520–1528. vol. 21, no. 3, pp. 197–207, 2018. [19] X. Han, “MRbased synthetic CT generation using a deep convolutional [40] R. J. van Sloun, R. R. Wildeboer, C. K. Mannaerts, A. W. Postema, neural network method,” Medical physics, vol. 44, no. 4, pp. 1408–1419, M. Gayet, H. P. Beerlage, G. Salomon, H. Wijkstra, and M. Mischi, 2017. “Deep Learning for Real-time, Automatic, and Scanner-adapted Prostate [20] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities im- (Zone) Segmentation of Transrectal Ultrasound, for Example, Magnetic prove neural network acoustic models,” in 30 th International Conference Resonance Imagingtransrectal Ultrasound Fusion Prostate Biopsy,” Eu- on Machine Learning, Atlanta, Georgia, 2013. ropean Urology Focus, vol. in press, 2019. 10 [41] G. Lemaˆ ıtre, R. Mart´ ı, J. Freixenet, J. C. Vilanova, P. M. Walker, and F. Meriaudeau, “Computer-Aided Detection and diagnosis for prostate cancer based on mono and multi-parametric MRI: A review,” Computers in Biology and Medicine, vol. 60, pp. 8–31, may 2015.

Journal

Electrical Engineering and Systems SciencearXiv (Cornell University)

Published: Aug 9, 2019

There are no references for this article.