Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Fully Automated Left Atrium Segmentation from Anatomical Cine Long-axis MRI Sequences using Deep Convolutional Neural Network with Unscented Kalman Filter

Fully Automated Left Atrium Segmentation from Anatomical Cine Long-axis MRI Sequences using Deep... This study proposes a fully automated approach for the left atrial segmentation from routine cine long- axis cardiac magnetic resonance image sequences using deep convolutional neural networks and Bayesian ltering. The proposed approach consists of a classi cation network that automatically detects the type of long-axis sequence and three di erent convolutional neural network models followed by unscented Kalman ltering (UKF) that delineates the left atrium. Instead of training and predicting all long-axis sequence types together, the proposed approach rst identi es the image sequence type as to 2, 3 and 4 chamber views, and then performs prediction based on neural nets trained for that particular sequence type. The datasets were acquired retrospectively and ground truth manual segmentation was provided by an expert radiologist. In addition to neural net based classi cation and segmentation, another neural net is trained and utilized to select image sequences for further processing using UKF to impose temporal consistency over cardiac cycle. A cyclic dynamic model with time-varying angular frequency is introduced in UKF to characterize the variations in cardiac motion during image scanning. The proposed approach was trained and evaluated separately with varying amount of training data with images acquired from 20, 40, 60 and 80 patients. Evaluations over 1515 images with equal number of images from each chamber group acquired from an additional 20 patients demonstrated that the proposed model outperformed state-of-the-art and yielded a mean Dice coecient value of 94.1%, 93.7% and 90.1% for 2, 3 and 4-chamber sequences, respectively, when trained with datasets from 80 patients. Keywords: Left Atrial Segmentation, Magnetic Resonance Imaging, Long-Axis Sequences, Deep Convolutional Neural Network, Unscented Kalman Filter 1. Introduction the anatomical assessment, the functional assess- ment of the left atrium is often performed using cine MRI or echocardiography. The advances in cardiac The assessment of left atrial function is becoming MRI technology have led to the generation of a large increasingly important due to its role in prognosis amount of imaging data with an increasingly high and risk strati cation of several cardiovascular dis- level of quality. However, segmenting a large num- eases including cardiomyopathy, ischemic heart dis- ber of left atrial images in cardiac MRI over the ease and valvular heart disease (Hoit, 2014; Kowal- entire cardiac cycle is a tedious and complex task lick et al., 2014; Tobon-Gomez et al., 2015). While for clinicians, who have to manually extract im- both the Computed Tomography (CT) and Mag- portant information (Despotovi c et al., 2015). The netic Resonance Imaging (MRI) could be used for manual analysis is often time-consuming and sub- ject to inter- and intra-operator variability. Due to these diculties, there is an increasing demand Corresponding author: Kumaradevan Punithakumar (Email: punithak@ualberta.ca) Preprint accepted by Medical Image Analysis arXiv:2009.13627v2 [cs.CV] 22 Nov 2020 for computerized methods to shorten segmentation quences. First, a classi cation network is intro- time and improve the accuracy of cardiac diagnosis duced to automatically detect the input sequence and treatment. into three groups according to the number of car- A number of cardiac image segmentation meth- diac chambers visible in the images, i.e., the net- ods using CT or MRI have been proposed in the work will classify the sequence as 2-chamber, 3- literature to improve the accuracy of the cardiac chamber or 4-chamber long-axis cine MRI. Then, chamber delineations (Peng et al., 2016; Zheng a U-net (Ronneberger et al., 2015) based model et al., 2008; Budge et al., 2008; Kaus et al., 2004; which was trained with 5-fold cross-validation for Radau et al., 2009). However, most of these meth- the detected sequence is applied. In order to im- ods focus on automated delineation of the left ven- prove the temporal consistency of the delineations, tricle only (Peng et al., 2016; Petitjean and Dacher, a second classi cation network is trained to detect 2011; Zreik et al., 2016). The left atrium is more image sequences which tend to yield low Dice co- dicult to segment (Tobon-Gomez et al., 2015). ecient in comparison to manual contours and the The diculties arise from following reasons (Zhu unscented Kalman lter (UKF) with a periodic dy- et al., 2012): 1) The wall of the left atrium is namic model is utilized to improve the accuracy of relatively thin compared to the left ventricle; 2) the selected sequences. Evaluations over 1515 im- Boundaries are not clearly de ned when the blood ages acquired from 20 patients in comparison to pool of the left atrium goes into pulmonary veins; expert manual delineation for each chamber group and 3) The shape variability of the left atrium is demonstrated that the proposed method yielded large between di erent patients. There are a few signi cantly better results than U-net alone. More- methods proposed in the literature to achieve au- over, this study empirically assesses the impact of tomated left atrial segmentation using model based the number f training samples on the network per- methods (Ecabert et al., 2008, 2011; Ordas et al., formance by comparing the results of four di erent 2007), which rely on static images obtained using scenarios where the size of training sets are 20, 40, CT imaging (Hunold et al., 2003). In comparison 60 and 80 patients. to CT imaging, MRI produces higher temporal res- olution, without the use of ionizing radiation or in- 2. Method travenous contrast media, and cine MRI allows for assessment of motion over the entire cardiac cycle The objective of the proposed method is to auto- (Parry et al., 2005). matically delineate the left atrium from cine long- A few notable exceptions to studies proposed for axis 2D MRI sequences correspond to 2-chamber, 3- the segmentation of the left atrium from MRI im- chamber and 4-chamber views that are acquired in ages include a method based on traditional image routine clinical scans. The overall system diagram analysis techniques such as salient feature and con- in Fig. 1 depicts the components of the proposed tour evolution by Zhu et al. (2012) and deep learn- method which integrates a classi cation network to ing based method by Mortazi et al. (2017). In ad- detect chamber type, three U-net based frameworks dition, a public segmentation challenge to delineate for semantic segmentation followed by another clas- the chamber from gadolinium-enhanced 3D MRI se- si cation network to detect results with low accu- quences was hosted at the Statistical Atlases and racy and unscented Kalman lter to improve tem- Computational Models of the Heart (STACOM) poral consistency. Both proposed classifying net- workshop in the Medical Image Computing and works are based on convolutional neural networks Computer Assisted Intervention (MICCAI) confer- and shortened as Classnet-A and Classnet-B in ence in 2018 (Pop et al., 2019). More than 15 re- rest of the paper. Each input image is resized to search teams participated in the challenge and a V- 256 256 using zero-padding scheme along the im- net (Milletari et al., 2016) based convolutional neu- age borders. Instead of training blindly with all ral network approach by Xia et al. (2019) yielded image data, three separate U-net based neural nets the best performance. However, these methods rely corresponding to 2, 3 and 4-chamber sequences were on 3D MRI scans of a single point in the cardiac constructed and trained after classifying the se- phase which may not be suited for the functional quence using the Classnet-A. The network architec- assessment of the left atrium (Zhang et al., 2018). ture of the proposed Classnet-A is given in Fig. 2. This study proposes a fully automated left atrial The standard U-net model is limited when process- segmentation approach for long-axis cine MRI se- ing image sequences as the predictions are made 2 independently where the temporal consistency is Dice score less than the threshold 0.9 are designated not maintained. In order to improve the accu- as Class 1 and the rest of the images in the set that racy, the unscented Kalman lter to exploit time- yield a Dice score of 0.9 and above are designated serial aspect of cardiac motion in the MRI sequence as Class 2. The rationale for the selection is that data. The second classi er neural net, Classnet-B, images with certain degraded quality lead to poor is trained using to the results from the U-net in prediction performance using U-net. The Classnet- comparison to expert manual ground truth to de- B was trained using the comparative results by U- tect image sequences that require further process- net predictions and expert manual ground truth for ing. The unscented Kalman lter is then applied as the training data sets. Upon training, the Classnet- a post-processing step for image sequences for the B is applied to the testing set to identify image selected sequences. frames that will likely yield diminished accuracy (Dice score < 0:9) with U-net based predictions, without relying on ground truth labels. If any of 2.1. Classnet Architecture the image frames in a sequence is classi ed as Class As shown in Fig. 2, the image is processed by 1 by the Classnet-B, the entire sequence is selected a 3 3 32 convolutional layer with ReLU activa- for post-processing with the UKF. tion function followed by a 2 2 max pooling layer. Another 3  3  64 convolutional and 2  2 max 2.4. Unscented Kalman Filter pooling layers are added afterwards. The output of the max pooling layer is connected to a fully con- The UKF is utilized for sequences which were nected block with drop out (0.5). The nal fully detected by the Classnet-B that are likely to connected layer outputs the classi cation result of yield diminished U-net prediction accuracy in each the chamber group after a softmax activation. chamber group to further improve the predic- tion accuracy. In contrast to the earlier state- 2.2. U-net Architecture space model proposed for cardiac motion estimation Three U-net based models are applied exclusively (Punithakumar et al., 2010), the proposed method after classifying the input images using Classnet-A. utilizes an angular time-varying frequency compo- An example U-net model used in this study is de- nent. The input to the lter is set as contour points picted in Fig. 3. Similar to the original U-net ar- in Cartesian coordinates obtained from predicted chitecture (Ronneberger et al., 2015), each U-net image masks after boundary detection and uniform based model contains convolutional with ReLU ac- sampling. The UKF rst iterates over the uniformly tivation function, max pooling, skip connection and sampled points and then iterates over entire frames upsampling operations. It then connects to a fully in the cardiac cycle. The overall iterative process convolutional layer with sigmoid activation function detailing the UKF steps for each contour point is to obtain output masks. The labelled masks for shown in Fig. 5. The UKF relies on a state tran- training and testing are both scaled to [0; 1]. The sition model to enforce temporal smoothing. The loss function is chosen as the negative of Dice met- UKF utilizes the Kalman gain, which is computed ric given in (27). The input images are normalized at each iteration in the process to deal with mea- st th with respect to the 1 and 99 percentile of pixel surement uncertainty resulting from the U-net pre- intensity values of each image. dictions. 2.3. Image Selection for Post-processing 2.4.1. Uniform sampling of contour points After obtaining semantic segmentation results The semantic segmentation results for the 2, 3 from U-net based models, the Classnet-B is applied and 4-chamber sequences by U-net models were to select the images for post-processing. The train- converted into contour points. Each contour cor- ing and testing process is shown in Fig. 4. The layer responding to di erent cardiac frames is sampled details follow the same notation de ned in Fig. 2. to 60 points, and the sampling is performed by The Classnet-B neural net is trained using input transforming the contours to the polar coordinate MRI images from the training set with binary an- system. The origin of the polar coordinate system notations consisting of Class 1 and Class 2 where is selected as the centroid of the segmented region the classes are de ned based on the U-net predic- and the sampling points are selected at 6 inter- tion accuracy. The training set images which yield a vals along the theta-axis using linear interpolation. 3 2ch U-net U-net+UKF MRI Training Unscented Classnet-A Classnet-B Training Dice Output Sequences Scenario Kalman Filter Threshold Train 20 Train 20 3ch U-net Train 40 Train 40 Train 60 Train 60 Train 80 Train 80 4ch U-net Figure 1: Overall system diagram showing the components of the proposed method as well as di erent test scenarios of this study to empirically assess the impact of the size of the training set on the performance of the U-net and the proposed U- net+UKF algorithms. Four di erent test scenarios were created. The accuracy of the algorithms were evaluated using overlap and distance metrics in comparison to expert manual delineation of the left atrium from 2, 3 and 4-chamber long-axis MRI sequences. Channel size 2 chamber Image size ratio 3x3 Convolution layer Flatten layer 3 6 with ReLU activation 5 3 2 4 1/4 Input images 1/2 3 chamber Node size Classnet-A Dropout layer Fully connected layer with 0.5 4 chamber Layer details Training and testing Figure 2: Classnet-A architecture proposed in this paper to automatically detect the input image sequence and classify the sequence type. The input images are resized to 256  256. The image size is changed by applying a 2  2 maxpooling layer. A fully connected block with softmax activation is applied in the nal layer to obtain output class. 1 1 2 2 2 2 1 1 3 3 6 6 6 6 3 3 2 2 5 5 5 5 2 2 2 2 4 4 4 4 2 2 8 8 6 6 6 6 8 8 1/16 1/8 Input images U-net predictions 1/4 1/2 1x1 fully convolutional 1 layer with sigmoid Training and testing Skip connection activation Figure 3: U-net architecture for each chamber group. The convolution layer follows the same layer details in Fig. 2. Final output is a 256  256 image mask corresponding to the left atrial semantic segmentation. 4 Training labels Extract entire sequence 1 1 1 1 Training Dice 6 6 6 6 Threshold 1/8 1/4 Entire sequence for Testing inputs 1/2 selected frames 1 Classnet-B U-net predictions UKF Class 1 Class 1 1 1 1 1 4 2 6 6 6 6 1/8 1/4 U-net predictions Training inputs 1/2 Classnet-B Output segmentation Testing Class 2 Training Class 2 Figure 4: Visualization of the training and application of the Classnet-B neural net of the proposed method. The algorithm was trained using the input MRI images along with the classi cation labels obtained based on the Dice score values of the U-net predictions where a threshold of 0.9 was used in deciding the class labels. During application or testing stage, the Classnet-B predictions for the input MRI images were used in deciding the application of the UKF for the U-net predictions. Unscented Kalman Filter Prediction Update (based on the Measurement Update (calculate physical model and previous state) residual 𝑦 from measurement 𝑧 𝑘 𝑘 𝑆 𝑆 0 𝑘 with uncertainty) χ = SigmaFunc(𝑆 ,𝑃 ) 𝑧 𝑘 𝑘 𝑘 𝑘 𝑚 𝑐 𝑊 ,𝑊 = WeightFunc(𝑑 ,α,β,κ) 𝑍 = 𝐻 У +𝜂 𝑃 𝑃 𝑘 𝑘 𝑘 0 𝑘 𝑚 𝑐 У = 𝑓 χ +𝑣 𝑘 𝑘 𝑘 𝑘 𝜇 ,𝑃 = 𝑈𝑇 𝑍 ,𝑊 ,𝑊 ,cov(𝜂 ) z 𝑧 𝑘 𝑚 𝑐 𝑆 ′,𝑃 ′= 𝑈𝑇 У ,𝑊 ,𝑊 ,cov(𝑣 ) Filter Previous state 𝑘 𝑘 𝑘 𝑘 𝑦 = 𝑧 −𝜇 𝑘 𝑘 z initialization Contour measurement Calculate Kalman Gain State and Covariance Update (≈ (belief in measurement)/(belief in state)) 𝑘 +1 𝑆 = 𝑆 ′+𝐾 𝑦 𝑘 +1 𝑘 𝑘 𝑘 𝑐 ′ 𝑇 −1 ′ 𝑇 𝐾 = ෍𝑤 (У −𝑆 ) 𝑍 −𝜇 𝑃 𝑘 +1 𝑃 = 𝑃 −𝐾 𝑃 𝐾 𝑘 𝑘 𝑘 z 𝑧 𝑘 +1 𝑘 𝑘 𝑧 𝑘 Current state Figure 5: Visualization of the UKF post-processing process for each sample contour point over the cardiac cycle. The objective of the UKF post-processing is to enforce temporally consistent segmentation results by considering the image frames in the entire cardiac sequence in contrast to the processing by U-net which segments each frame independently. UT denotes unscented transform. The proposed method uses a state variable with dimension d = 7 and unscented transformation parameters = 0:1; = 2;  = 1. 5 Algorithm 1: Unscented Kalman lter for enforcing temporal consistency 1 Input: U-net prediction after uniform sampling with shape N  K  2. K is the total number of frames in each cardiac sequence and N is the number of samples in contour. 2 For contour point i = 1; 2; 3; :::; N do i T i i i ^ ^ ^ ^ 3 Initialize: S = [x_ x ^ x  y_ y^ y ! ] in (21)-(24), P in (25) and (26), Q in (17), R in (20). 1 1 1 1 1 1 1 1 1 1 1 4 For time frame k = 1; 2; 3; :::; K do: 5 Prediction update: i i i 6  = SigmaFunction(S ; P ) k k k m c 7 W ; W = WeightFunction(d; ; ; ) i i 8 Y = f ( ) + v k k k k 0 0 i i i m c i 9 S ; P = UnscentedTransform(Y ; W ; W ; Q ) k k k 10 Measurement update: 11 Z = H Y + k k i i m c i 12  ; P = UnscentedTransform(Z; W ; W ; R ) z z 1 i i i 13 y = z k k 14 Calculate Kalman gain: i c i i i T i 1 15 K = [ w (Y S )(Z  ) ](P ) z z k k k 16 State and covariance update: i i i i 17 S = S + K y k+1 k k k i i i i i T 18 P = P K P (K ) k+1 k k k 19 Output: Filtered contour with shape N  K  2 and improved temporal consistency. Centroid (x ; y ) of the segmented region is com- the points on contour obtained after resizing the c c puted using (1), where n is the number of raw con- prediction masks of U-net to original image size th tour points and (x ; y ) is the p point on raw con- and consider the corresponding state vector  = p p tour set [x  x x _ ] which describes the dynamics of point in x-coordinate direction. The terms x  and x _ denote n n X X mean position and velocity over the entire cardiac x = x ; y = y : (1) c p c p cycle, respectively. We exploit the periodicity fea- p=1 p=1 ture in each cardiac cycle and propose a state-space We convert each point (x ; y ) on the automated p p model (3) corresponding to the oscillation: left atrial contour from the Cartesian coordinate 2 3 2 3 system to the polar coordinate system using the 0 0 0 1 0 6 7 6 7 following equations: 6 7 6 7 (t) = 6 7  (t) + 6 7 w(t) ; (3) 0 0 1 0 0 4 5 4 5 2 2 ! ! 0 0 1 2 2 r = x + y ;  = tan : (2) p p p p where ! is the angular frequency and w(t) is the vector-valued white noise. The model describes A linear interpolation is then applied to nd uni- the cardiac motion as a simple harmonic motion. formly sampled points along the angular coordi- Nonetheless, cardiac movement is much more com- nates in the polar coordinate system. These points plex than a simple harmonic motion and a time- in the polar coordinate system are converted back varying angular velocity is added to account for the to the Cartesian coordinate system and used in the rate of oscillation variation and the discrete equiv- subsequent processing using unscented Kalman l- alence of the model is described in ter. = F  + w ; (4) k+1 k k k 2.5. Dynamic model for temporal periodicity A dynamic model is designed to characterize where k is the frame number between each heart cardiac motion. Assume that (x; y) be one of beat and the maximum value K denotes the num- 6 ber of frames in a cardiac cycle which is typically Small q and q will enforce the conformity of car- 1 2 between 25 to 30. F is given in the following: diac motion to the model, thereby a ect the accu- racy of the prediction results when motion deviates 2 3 1 0 0 from the model. Large values of q and q allow 1 2 6 7 6 7 for high uncertainties within the model, which may F = 6 7 ; k 1 cos(! T ) cos(! T ) sin(! T ) k k k 4 5 lead to poor temporal consistency. In our case, the ! sin(! T ) ! sin(! T ) cos(! T ) k k k k k noise parameters q = q = 1  10 are chosen 1 2 (5) empirically. where w is a discrete-time white noise, T is the k Let S = [x  x x _ y  y y _ ! ] be the state k k k k k k k k sampling interval and ! is the discrete angular fre- k vector that describes the corresponding dynamics quency. The initial estimated value of the angular at time frame k. Elements x , x _ , y , y _ denote, re- frequency ! is computed as 1 spectively, the mean position and velocity in x- coordinate and mean position and velocity in y- 2  Heart Rate coordinate. The discrete-time model that describes ! = ; (6) the cyclic motion of the point is given in and the time interval between two consecutive car- S = f (S ) + v ; (15) k+1 k k k diac frames T is calculated as where f (S ) is de ned by 60 k k T = : (7) 2 3 Heart Rate K F 0 0 k 33 31 6 7 The covariance of process noise Q is given by 6 7 f (S ) = 6 7 S ; (16) k k 0 F 0 k 33 k 31 4 5 Q = [q ] ; (8) k ij 33 0 0 1 33 33 where q is de ned in (9){(14): ij and v denotes a Gaussian process noise sequence with zero-mean and covariance that accommodates q = q T ; (9) unpredictable error due to model uncertainties q (! T sin(! T )) k k given in q = q = ; (10) 12 21 k 2 3 q = q = q (1 cos(! T )) ; (11) Q 0 0 13 31 k k 33 31 6 7 6 7 2 2 cov(v ) = ; (17) 6 7 k 0 Q 0 33 k 31 q ! (3! T 4 sin(! T ) 1 k k k 4 5 0 0 1 33 33 + cos(! T ) sin(! T )) k k + q (! T cos(! T ) sin(! T )) k k k where Q is de ned in (8). The measurement equa- q = ; tion is given in 2! (12) Z = H S +  ; (18) k k k 2 2 2 q ! (1 2 cos(! T ) + cos (! T )) k k 1 k where H is de ned in + q sin (! T ) " # q = q = ; 23 32 2! 0 1 0 0 0 0 0 H = ; (19) (13) 0 0 0 0 1 0 0 2 2 q ! (cos(! T ) sin(! T ) ! T ) 1 k k k and  is a zero-mean Gaussian noise sequence to model the measurement uncertainties such as po- q (cos(! T ) sin(! T ) + ! T ) k k k q = ; 33 tential false positives resulting from inaccurate U- 2! net predictions, with covariance demonstrated in (14) " # r 0 where q and q describe uncertainties associated 1 2 cov( ) = : (20) with the mean position and velocity respectively. 0 r 7 The parameter r characterizes the uncertainties as- where  is given in sociated with observations. Smaller values of r en- 2 3 r r force the conformity of the estimation to the mea- k kT 6 7 surements while larger values allow for associating 6 7 r r = 6 7 : (26) 1 r k T more uncertainty with the measurements. In the 4 5 r r 2r proposed study, U-net based model showed ade- kT T T quate raw accuracy of the prediction masks and r was set to 1 10 . Apart from the measurement 2.7. Iterative prediction and update uncertainties modeled using  , the UKF updates After generating sigma points using the method its state and covariance based on the relative ra- proposed in (Julier and Uhlmann, 2004), prediction tio of the belief in measurement over the belief in and update steps were performed iteratively based state by calculating Kalman gain shown in Fig. 5 on (15) and (18). The entire process of the UKF is to improve prediction performance. If the Kalman summarized in Algorithm 1. gain is high, the lter will rely more on the current measurement. Otherwise, it will rely more on the predicted value by the lter. 3. Experimental Setup and Results 2.6. Filter initialization 3.1. Dataset, evaluation criteria and implementa- tion details It is not possible to have prior information of The U-net and the proposed algorithms were the exact initial value of S in the proposed study. trained and tested over 100 MRI patient datasets Therefore, a two-point di erencing method pro- acquired retrospectively at the University Alberta posed by Bar-Shalom et al. (2004) is applied to Hospital. The study was approved by the Health initialize the position and velocity components of Research Ethics Board of the University of Al- the state. For instance, the initial position x ^ and berta. The details of the datasets used in the pro- velocity x_ in x-coordinate of the ith sample point: posed model are given in Table 1. The evaluations ix x ^ = z ; (21) were performed over 1515 images acquired from 20 patients in comparison to the expert manual seg- ix ix mentation of the left atrium from the 2, 3 and z z 2 1 x_ = ; (22) 4-chamber cine steady state processions long-axis MR sequences where each chamber group consists ix where z ; k = 1; 2; :::; K is the observation in x- of 505 images. The 2, 3 and 4-chamber sequences coordinate obtained from kth frame. The mean are shortened as 2ch, 3ch and 4ch in the rest of position over cardiac cycle illustrated in (23) was the paper. The ground truth manual segmentation initialized by taking the average of observation in was initially performed by a medical student using x-coordinate a semi-automated software and the contours were corrected by an experienced radiologist. ix x  = z : (23) K In order to empirically evaluate the impact of k=1 training sample size on the algorithm performance, Similarly, the initial state elements in y-coordinate four di erent scenarios were created by increasing can be computed using the same strategy and the the sample size from 20 to 80 patients with an incre- nal initial state vector input S is given in (24) ment of 20 patients per scenario (Refer to Fig. 1). These scenarios are shortened as Train 20-80 in the ^ ^ ^ ^ S = [x_ x ^ x  y_ y^ y ! ] : (24) 1 1 1 1 1 1 1 1 rest of the paper. For each case, the U-net based neural nets were trained and tested using 5-fold of The corresponding initial covariance is given in cross validation to prevent over- tting. The train- 2 3 ing loss function was chosen as the negative of the 0 0 Dice coecient. To demonstrate the e ectiveness of 1 33 31 6 7 6 7 the proposed approach, the results were compared P = 6 7 ; (25) 1 0  0 33 1 31 4 5 with the algorithm that was trained using a merged 0 0 1 set of all cardiac long-axis chambers. 33 33 8 Table 1: Details of data set used in the proposed study. Description Training Set Test Set Number of subjects 20 80 20 Patient's sex 40 Males / 40 Females 10 Males / 10 Females Scanner protocol Siemens AERA Siemens AERA Patient's age 4 85 years 29 73 years Magnetic strength 1.5 T 1.5 T Number of frames (K) 25 or 30 25 or 30 Image size (176, 184) (256, 208) pixels (192, 208) (256, 256) pixels Pixel spacing 1:445 1:797 mm 1:445 1:660 mm Repetition time (TR) 21:12 89:10 ms 27:10 67:25 ms Echo time (TE) 1:09 1:43 ms 1:12 1:18 ms 3.2. Implementation 3.3.1. Dice coecient Dice coecient illustrated in (27) measures the overlap between two delineated regions, where set The U-net based models, Classnet-A and A is the prediction image set and M is the manually Classnet-B were implemented in Python program- labelled image set ming language using Keras machine learning mod- ule with Tensor ow backend and cudnn. The l- 2jA\ Mj terpy Python module Labbe (2014) was used for DC = : (27) jAj+jMj the implementation of Bayesian based unscented Kalman lter. Three U-net based architectures were utilized with linear variable learning rate with 3.3.2. Hausdor distance 4 5 Adam optimizer from 1 10 in initial to 1 10 The Hausdor distance measures the maximum 5 6 for 2ch U-net and 1  10 in initial to 1  10 deviation between two contours in terms of the Eu- for 3ch and 4ch. The decay was uniform per each clidean distance. The Hausdor distance between epoch. Batch size was set as 32 and epoch num- manual and automatic contours was computed as ber is set to 500 in a 5-fold cross validation scheme. follows: The proposed model was tested on a desktop with Intel core i7-7700 HQ CPU with 16 GB RAM. The i j d (a; m) = max max min d(p ; p ) ; neural net models were trained and tested with a m i j an NVIDIA GTX 1080 Ti graphics processing unit i j with 12 GB memory. The approximate training max min d(p ; p ) ; (28) a m j i time for each chamber group is 7 hours and the prediction time is 4 ms per image for the proposed where d() is the Euclidean distance, fp g denotes method. the set of points on automated contour and fp g denotes the set of points on manual contour. 3.3. Quantitative Evaluation Metrics 3.3.3. Reliability metric The proposed method and the U-net based ap- The reliability metric is evaluated using the func- proach were evaluated quantitatively using Dice co- tion given in (29). The complementary cumulative ecient (DC), Hausdor distance (HD) and relia- distribution function is de ned for each d 2 [0; 1] as bility metric. the probability of obtaining Dice coecient higher 9 than d over all images 8 out of 12 scenarios tested with di erent chamber groups and training sets. Images with Dice The best performances for each chamber group coecient higher than d in terms of Dice coecient and Hausdor distance R(d) = P (DC > d) = : Total images were obtained using the proposed U-net+UKF us- (29) ing Classnet-A approach with Train 80 except for the 4ch segmentation measured in terms of Dice score where a small decrease of 0.1% in performance 3.4. Experimental results was observed with the application of Classnet-A. 3.4.1. Quantitative assessment The approach yielded average Dice coecients of The Classnet-A and Classnet-B 's performance 94.1%, 93.7% and 90.1% for 2ch, 3ch, and 4ch, re- are reported in Table 2 and Table 3, respectively. spectively. The corresponding average Hausdor Both classi cation networks achieved 100% accu- distance values were 6.0 mm, 5.5 mm and 8.4 mm, racy in the Train 80 scenario where the approaches respectively. have correctly classi ed all 1515 images to the re- The accuracy of the proposed U-net+UKF with spective chamber group or image classi cation into Classnet-A method steadily increased with the size Class 1 and Class 2 based on U-net segmentation of the training data for all chamber groups as re- accuracy. The results from both tables show that ported in Tables 4{6. The increase in accuracy with the accuracy of the classi cation algorithms im- the size of training data set is also observed in the proves with the size of training set. The number reliability assessment as depicted in Fig. 10. The of classes on ground truth remained the same for improvement was signi cant from Train 20 to Train Train 20, 40, 60, and 80 scenarios for Classnet-A, as 40 in 2-chamber and marginal from Train 40 to indicated in Table 2, since it is based on the cham- Train 80. In 3-chamber group, the improvement ber views on the test set. However, the number was critical from Train 40 to Train 60 while in 4- of classes on ground truth varied for di erent sce- chamber group the overall improvement is marginal narios for Classnet-B, as indicated in Table 3, due with the training size. to varying U-net prediction accuracy on the test- 3.4.2. Visual assessment ing set. For instance, the U-net trained with Train 80 scenario resulted in more images (n = 1177) in Fig. 6 shows example automatic contours by the Class 2 (Dice score  0:9) than the one trained with proposed U-net+UKF approach (blue curve) as Train 20 (n = 345). well as U-net (red curve) with the application of The quantitative assessment of the automated Classnet-A. The manual ground truth delineation is segmentation results for 2ch, 3ch and 4ch are given depicted by the yellow area. Fig. 7 shows compara- in Tables 4, 5, and 6, respectively. The tables re- tive examples of U-net and U-net+UKF prediction port the mean and standard deviation of Dice coe- results on selected frames over a complete cardiac cient and Hausdor distance values for U-net+UKF cycle for 2, 3, and 4 chamber views where the U-net and U-net alone approaches with and without the yielded temporally inconsistent results. The exam- application of Classnet-A neural net for di erent ples show that the proposed application of temporal sizes of training data sets. As shown in Tables 4{6, smoothing for image sequences consisting of image the proposed U-net+UKF method with Classnet-A frames with inaccurate U-net predictions led to im- outperformed U-net for all three chamber groups in proved segmentation accuracy. Fig. 8 shows the terms of Hausdor distance in Train 80 scenario. trajectories of left atrial boundary points over the The method also outperformed U-net in 2ch and complete cardiac cycle, which demonstrates the ad- 3ch sequences in Train 80 scenario. The reported vantage of applying UKF for better temporal con- values in Tables 4{6 demonstrate that the proposed sistency compared to U-net alone. As shown in U-net+UKF approach outperformed U-net alone Fig. 8(a), the U-net yielded unrealistic trajectories method in terms of Dice coecient in 23 out of 24 that do not match with the left atrial dynamics, and di erent tested scenarios that consist of with and the trajectories are corrected with the application without Classnet-A, Train 20 { Train 80 scenarios, of the UKF, as shown in Fig. 8(b). The segmenta- and 2ch { 4ch chamber groups. The reported values tion results by the proposed approach over di erent also demonstrate that the Classnet-A approach im- temporal frames for 2ch, 3ch and 4ch is given in proved performance in terms of Dice coecient in Fig. 9. 10 2ch Train 80 3ch Train 80 4ch Train 80 DC: U-net+UKF = 93:3% DC: U-net+UKF = 94:1% DC: U-net+UKF = 95:2% DC: U-net = 91:7% DC: U-net = 89:6% DC: U-net = 91:5% Figure 6: Representative example results by the proposed method (blue curve) and U-net (red curve) approaches in comparison to the expert manual ground truth (yellow area). The proposed method yielded more conformity with manual ground truth than U-net approach. Table 2: Classi cation result for Classnet-A proposed in this paper. The test images consisted of 505 images for each chamber group. Overall accuracy is reported for each training scenario in the last row. Prediction Result Ground Truth Train 20 Train 40 Train 60 Train 80 2ch 3ch 4ch 2ch 3ch 4ch 2ch 3ch 4ch 2ch 3ch 4ch 2ch 505 0 0 505 0 0 505 0 0 505 0 0 3ch 20 485 0 0 480 25 0 505 0 0 505 0 4ch 0 24 481 0 17 488 0 30 475 0 0 505 Accuracy(%) 97.1 97.2 98.0 100.0 Table 3: Classi cation result for Classnet-B proposed in this paper. The test images consisted of 1515 images from all chamber groups (each consists of 505 images). Class 1 indicates the images that need to be post-processed using the UKF and Class 2 indicates the images that do not need to be post-processed due to their higher accuracy. Overall accuracy is reported for each training scenario in the last row. Prediction Result Ground Truth Train 20 Train 40 Train 60 Train 80 Class 1 Class 2 Class 1 Class 2 Class 1 Class 2 Class 1 Class 2 Class 1 1167 3 634 0 542 0 338 0 Class 2 52 293 0 881 0 973 0 1177 Accuracy(%) 96.4 100.0 100.0 100.0 11 3.4.3. Ablation study 2. The impact of Classnet-A on the overall perfor- mance becomes unpredictable with smaller training Table 7 reports the results for U-net+UKF ap- sets such as Train 20 scenario where Classnet-A led proach without Classnet-A and Classnet-B evalu- to inaccurate classi cations in addition to the vary- ated in terms of Dice score and Hausdor distance. ing performances of the corresponding U-Net algo- The reported values in the table demonstrate the rithms. Although the U-Net with Classnet-A led impact of Classnet-B in improving the segmenta- to improved performance over the U-Net without tion accuracy by automatically selecting the images Classnet-A for Train 20 scenario for 4-chamber se- that require post-processing with UKF instead of quences as reported in Table 6, no similar improve- applying it to the entire image set. The improve- ments were observed for 2 and 3 chamber sequences ment was observed in all chamber views in terms of as reported in tables 4 and 5. Future studies will Dice coecient as reported in Table 7 as well as in explore the performance of neural networks with column 3 of Tables 4, 5, and 6. smaller training sets. The intent of Classnet-B is to detect images 4. Discussion that will lead to less accurate left atrial segmenta- tion by U-net, and therefore, the training phase of In this study, we proposed and validated a fully Classnet-B utilized the ground truth data from the automated segmentation approach to delineate left training set. Upon trained, the Classnet-B neural atrium from MRI sequences using deep convolu- net parameters were no longer updated and none of tional neural networks and Bayesian ltering. Pre- the ground truth delineations from the test set was vious automated left atrial segmentation method utilized in the process. attempted to delineate the chamber from a single Among the three long-axis sequences, the 3- time point in the cardiac phase using static images chamber view requires more experience to segment (Hunold et al., 2003; Ordas et al., 2007; Ecabert as the anterior margin of the mitral valve, being ad- et al., 2008, 2011; Zhu et al., 2012; Mortazi et al., jacent to the aortic valve, is harder to delineate than 2017; Pop et al., 2019). The proposed study relied in other views. The results of this study show that on 2-chamber, 3-chamber and 4-chamber 2D MRI the performance is lower for the 3-chamber view in sequences, which may or may not be orthogonal to comparison to other views when the proposed ap- each other. These long-axis sequences do not form proach was trained with images from 20 patients. a 3D MRI dataset, as in the case of STACOM MIC- However, as indicated in Table 5, a signi cant per- CAI 2018 challenge Pop et al. (2019), however, o er formance improvement was achieved in 3-chamber a series of images to assess the temporal characteris- view results when the training set was increased to tics of the left atrium over the cardiac cycle. To the 80 patients. best of authors' knowledge, this is the rst study to In this study, the test set MRI sequences were propose a fully automated approach to delineate selected from 20 patients, and none of the scans left atrium over the entire cardiac cycle that could were excluded in the analysis regardless of the im- serve as the basis for the functional assessment of age quality, which led to high inhomogeneity among the chamber. the test images and a high standard deviation was The intent of Classnet-A is to automatically iden- observed for the evaluation metrics reported in Ta- tify the 2-chamber, 3-chamber and 4-chamber views bles 4, 5, and 6. Table 8 reports the numbers using image pixel information only. The experimen- of samples used for training the U-Net approach tal analysis showed that the Classnet-A achieved with and without Classnet-A. Due to the di er- perfect classi cation performance without a single ent number of samples used for training the U-Net misclassi cation when trained with images from 80 approach, the performance of the algorithm varies patients. The results presented in Tables 4, 5, and even when the Classnet-A o ers perfect classi ca- 6 show that the application of Classnet-A led to an tions. For instance, the results for the Train 80 improvement on segmentation accuracy measured scenario reported in columns 3 and 5 for the U-Net in terms of Dice score in seven cases, slight reduc- approach with and without Classnet-A in Tables 4, tion (< 1%) or similar performance in four cases, 5, 6 are di erent where Classnet-A yielded perfect and a more considerable reduction in one case for classi cation. Train 60 and Train 80 scenarios where the algorithm Clinical importance of the left atrial functional yielded a perfect classi cation as indicated in Table analysis has been emphasized in many echocardio- 12 2ch Frame 1 2ch Frame 6 2ch Frame 11 2ch Frame 16 2ch Frame 21 Proposed= 96:0% Proposed = 96:2% Proposed = 96:0% Proposed = 95:1% Proposed = 94:1% U-net = 94:2% U-net = 83:7% U-net = 82:4% U-net = 91:7% U-net = 91:6% 3ch Frame 1 3ch Frame 6 3ch Frame 11 3ch Frame 16 3ch Frame 21 Proposed= 93:2% Proposed = 93:5% Proposed = 93:0% Proposed = 91:3% Proposed = 92:5% U-net = 95:7% U-net = 92:7% U-net = 68:8% U-net = 71:4% U-net = 88:9% 4ch Frame 1 4ch Frame 6 4ch Frame 11 4ch Frame 16 4ch Frame 21 Proposed= 86:8% Proposed = 90:7% Proposed = 90:5% Proposed = 91:2% Proposed = 91:3% U-net = 76:9% U-net = 71:9% U-net = 73:5% U-net = 66:0% U-net = 72:4% Figure 7: An example showing the prediction results on selected frames over a complete cardiac cycle (K=25) for 2, 3 and 4-chamber views (from top to bottom) where the U-net yielded temporally inconsistent segmentation results. The yellow area is the manually labelled left atrium. The blue and red curves are the prediction results by the proposed U-net+UKF and U-net alone, respectively. A higher conformance was observed between expert manual segmentation and the results by the proposed fully automated approach which demonstrates the bene ts of applying temporal smoothing with the UKF. 13 Table 4: Evaluation of the automated segmentation results over 20 patients data sets using Dice coecient and Hausdor distance metrics. The results of the U-net and the proposed U-net+UKF methods were compared against expert manual ground truth produced by clinicians with and without the application of Classnet-A neural net. The evaluations were performed over 1515 images obtained from the long-axis cine 2-chamber view. The chamber group consisted of 505 images. 2ch Dice Coecient (%) 2ch Hausdor Distance (mm) Method without Classnet-A with Classnet-A without Classnet-A with Classnet-A U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF Train 20 84:3 6:7 85:4 6:0 82:5 13:5 86:8 9:7 9:4 5:4 9:1 5:0 12:2 8:5 10:2 7:7 Train 40 90:8 4:0 91:6 3:2 91:8 7:6 92:9 4:2 9:5 6:9 8:5 5:0 8:6 6:8 7:7 6:2 Train 60 91:1 3:8 91:8 3:0 93:7 2:6 93:7 2:7 8:8 6:9 7:6 4:6 6:7 3:5 6:0 2:9 Train 80 93:4 3:0 93:5 3:0 93:5 5:8 94:1 3:7 7:7 6:4 6:6 4:3 7:0 4:3 6:0 3:3 Table 5: Evaluation of the automated segmentation results over 20 patients data sets using Dice coecient and Hausdor distance metrics from the long-axis cine 3-chamber view. The chamber group consisted of 505 images. 3ch Dice Coecient (%) 3ch Hausdor Distance (mm) Method without Classnet-A with Classnet-A without Classnet-A with Classnet-A U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF Train 20 83:0 6:6 84:8 8:0 77:0 9:5 80:4 8:7 19:7 20:9 17:1 16:2 11:2 9:3 10:5 9:3 Train 40 89:3 5:0 90:1 10:7 79:1 16:2 83:4 14:3 17:0 22:8 13:9 16:5 16:6 15:6 9:4 8:8 Train 60 90:9 5:3 91:3 10:5 92:5 8:4 92:6 8:4 13:9 18:3 12:4 15:1 10:4 15:0 7:4 5:4 Train 80 92:4 4:0 92:5 9:1 90:3 15:2 93:7 8:2 16:3 22:9 12:3 14:8 5:7 3:2 5:5 3:0 Table 6: Evaluation of the automated segmentation results over 20 patients data sets using Dice coecient and Hausdor Distance metrics from the long-axis cine 4-chamber view. The chamber group consisted of 505 images. 4ch Dice Coecient (%) 4ch Hausdor Distance (mm) Method without Classnet-A with Classnet-A without Classnet-A with Classnet-A U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF Train 20 82:0 10:1 84:4 5:1 85:6 5:6 86:4 5:6 17:3 12:8 14:9 10:4 11:7 5:7 9:3 5:1 Train 40 87:2 12:6 88:0 8:1 88:9 5:9 89:6 5:1 12:9 11:8 10:4 7:8 9:9 11:7 8:5 8:1 Train 60 87:9 11:2 88:5 10:1 87:1 16:5 88:4 13:4 12:7 11:8 10:4 7:9 11:6 10:8 10:1 10:1 Train 80 88:9 11:3 90:2 4:2 88:9 10:7 90:1 8:1 11:8 10:9 10:0 7:6 12:2 5:6 8:4 4:2 Table 7: Evaluation of automated segmentation results over 20 patient data sets for the U-Net+UKF approach without Classnet-A and Classnet-B in terms of Dice coecient and Hausdor distance metrics for the long-axis cine 2, 3, 4-chamber views. The test set for each chamber group consisted of 505 images. Dice Coecient (%) Hausdor Distance (mm) Method 2ch 3ch 4ch 2ch 3ch 4ch Train 20 84:6 6:5 83:3 6:8 83:5 8:4 9:5 3:3 9:6 2:8 9:7 3:8 Train 40 90:9 4:0 89:4 4:9 87:7 10:9 7:5 2:4 10:0 7:3 9:7 7:5 Train 60 91:2 3:8 91:0 5:2 88:2 10:6 7:0 2:5 8:6 3:5 10:0 7:8 Train 80 93:5 2:9 92:5 4:0 89:9 9:9 7:0 2:6 9:1 3:2 8:5 6:1 14 Table 8: The number of images utilized in training the U-Net with and without Classnet-A for 2, 3, 4-chamber sequences. The image sequences have either 25 or 30 images leading to slight variations in the number of images for 2, 3, and 4 chamber training sets. With Classnet-A Without Classnet-A Method 2ch 3ch 4ch Combined 2,3,4ch Train 20 500 505 505 1510 Train 40 1020 1025 1025 3070 Train 60 1535 1540 1540 4615 Train 80 2115 2015 2090 6220 though such delineations could be used for comput- ing functional parameters such as volume or volume rate over time, some additional parameters such as strain or strain-rate require open contours repre- senting the boundary of the chamber walls that ex- cludes anatomical structures corresponds to the mi- tral valve. Future studies will attempt to address this limitation. (a) U-net (b) U-net+UKF 5. Conclusion Figure 8: An example showing the trajectories of the left This paper proposes a fully automated approach atrial boundary points over the entire cardiac cycle (K=25) to segment the left atrium from routine long-axis estimated using U-net alone, and U-net+UKF approaches. The example demonstrates the bene ts of applying the UKF cardiac MR image sequences acquired over the en- to enforce the temporal consistency in estimating the con- tire cardiac cycle. The proposed methods intro- tour points as the method allows for correcting inaccurate duces a neural net approach to classify the input im- segmentation results by the U-net in a few arbitrary image ages and loads the corresponding U-net based con- frames. volutional neural network architecture for semantic segmentation. A second classi er network is devel- oped to select sequences for further processing with graphy studies particularly the prognostic value of left atrial measurements for patients with heart fail- the Unscented Kalman lter to impose temporal pe- ure with preserved ejection fraction (Morris et al., riodicity over the cardiac cycle. The proposed algo- 2011; Santos et al., 2014, 2016; Cameli et al., 2017; rithm was trained separately with di erent scenar- Liu et al., 2018). Clinical studies that used MRI ios with varying amount of training data with im- ages acquired from 20, 40, 60, 80 patients. The pre- sequences for cardiac functional measurements also diction results were compared against expert man- led to similar ndings and demonstrated the prog- ual delineations over an additional 20 patients in nostic values of the left atrial functional parameters terms of the Dice coecient and Hausdor distance. in heart failure patients (Von Roeder et al., 2017; The quantitative assessment showed that the pro- Chirinos et al., 2018). One of the signi cant ad- posed method performed better than recent U-net vantages of the proposed method is that it is fully algorithm. The proposed approach yielded 100% automated and, therefore, will allow for analyzing accuracy for chamber classi cation and a mean Dice a large number of MRI images which otherwise te- coecient of 94:1%, 93:7% and 90:1% for 2, 3 and 4- dious and time-consuming. Further clinical studies chamber sequences after training using 80 patients are warranted to objectively measure the left atrial data. function and assess its predictive ability for heart failures with preserved and reduced ejection frac- tion. Acknowledgments The proposed approach performs semantic seg- mentation to delineate left atrium where pixels are X. Zhang was supported by the Mitacs Globalink classi ed either foreground or background. Al- Research Internship program. D. Glynn Martin was 15 2ch Frame 1 2ch Frame 6 2ch Frame 11 2ch Frame 16 2ch Frame 21 3ch Frame 1 3ch Frame 6 3ch Frame 11 3ch Frame 16 3ch Frame 21 4ch Frame 1 4ch Frame 6 4ch Frame 11 4ch Frame 16 4ch Frame 21 Figure 9: An example showing the prediction results over a complete cardiac cycle (25 frames in total) for 2, 3 and 4-chamber groups (from top to bottom). The yellow area is the manually labelled region of interest corresponds to left atrium and the blue curve is the prediction results by the proposed U-net+UKF. A higher conformance was observed between expert manual segmentation and the results by the proposed fully automated approach 2-chamber reliability 3-chamber reliability 4-chamber reliability Figure 10: Reliability versus Dice metric for the proposed U-net+UKF with Classnet-A approach with varying training set data. The training were performed using data sets acquired from 20, 40, 60 and 80 patients for each chamber group. The evaluations were performed over 505 images for each chamber group acquired from 20 patients. 16 supported by the University of Alberta Radiology atrial systolic function and inter-atrial dyssynchrony may contribute to symptoms of heart failure with preserved left Endowed Fund. The work in-part was supported ventricular ejection fraction: A comprehensive assessment by a grant from the Servier Canada Inc. by echocardiography. International Journal of Cardiology 257, 177{181. Milletari, F., Navab, N., Ahmadi, S., 2016. V-Net: Fully con- References volutional neural networks for volumetric medical image segmentation, in: 2016 Fourth International Conference Bar-Shalom, Y., Li, X.R., Kirubarajan, T., 2004. Estima- on 3D Vision (3DV), pp. 565{571. tion with applications to tracking and navigation: theory Morris, D.A., Gailani, M., Vaz P erez, A., Blaschke, F., Dietz, algorithms and software. John Wiley & Sons.  R., Haverkamp, W., Ozcelik, C., 2011. Right ventricular Budge, L.P., Sha er, K.M., Moorman, J.R., Lake, D.E., myocardial systolic and diastolic dysfunction in heart fail- Ferguson, J.D., Mangrum, J.M., 2008. Analysis of in ure with normal left ventricular ejection fraction. Journal vivo left atrial appendage morphology in patients with of the American Society of Echocardiography 24, 886{897. atrial brillation: a direct comparison of transesophageal Mortazi, A., Karim, R., Rhode, K., Burt, J., Bagci, U., 2017. echocardiography, planar cardiac ct, and segmented three- Cardiacnet: Segmentation of left atrium and proximal dimensional cardiac ct. Journal of interventional cardiac pulmonary veins from MRI using multi-view cnn, in: In- electrophysiology 23, 87{93. ternational Conference on Medical Image Computing and Cameli, M., Mandoli, G.E., Mondillo, S., 2017. Left atrium: Computer-Assisted Intervention, Springer. pp. 377{385. the last bulwark before overt heart failure. Heart Failure Ordas, S., Oubel, E., Leta, R., Carreras, F., Frangi, A.F., Reviews 22, 123{131. 2007. A statistical shape model of the heart and its appli- Chirinos, J.A., Sardana, M., Ansari, B., Satija, V., Kuri- cation to model-based segmentation, in: Medical Imaging akose, D., Edelstein, I., Oldland, G., Miller, R., Gad- 2007: Physiology, Function, and Structure from Medical dam, S., Lee, J., Suri, A., Akers, S.R., 2018. Left Atrial Images, International Society for Optics and Photonics. Phasic Function by Cardiac Magnetic Resonance Feature p. 65111K. Tracking Is a Strong Predictor of Incident Cardiovascular Parry, D.A., Booth, T., Roland, P.S., 2005. Advantages of Events. Circulation. Cardiovascular imaging 11, e007512. magnetic resonance imaging over computed tomography Despotovi c, I., Goossens, B., Philips, W., 2015. MRI seg- in preoperative evaluation of pediatric cochlear implant mentation of the human brain: challenges, methods, and candidates. Otology & Neurotology 26, 976{982. applications. Computational and mathematical methods Peng, P., Lekadir, K., Gooya, A., Shao, L., Petersen, S.E., in medicine 2015. Frangi, A.F., 2016. A review of heart chamber segmen- Ecabert, O., Peters, J., Schramm, H., Lorenz, C., von Berg, tation for structural and functional analysis using cardiac J., Walker, M.J., Vembar, M., Olszewski, M.E., Subra- magnetic resonance imaging. Magnetic Resonance Mate- manyan, K., Lavi, G., et al., 2008. Automatic model- rials in Physics, Biology and Medicine 29, 155{195. based segmentation of the heart in CT images. IEEE Petitjean, C., Dacher, J.N., 2011. A review of segmentation transactions on medical imaging 27, 1189. methods in short axis cardiac MR images. Medical image Ecabert, O., Peters, J., Walker, M.J., Ivanc, T., Lorenz, C., analysis 15, 169{184. von Berg, J., Lessick, J., Vembar, M., Weese, J., 2011. Pop, M., Sermesant, M., Zhao, J., Li, S., McLeod, K., Young, Segmentation of the heart and great vessels in CT im- A.A., Rhode, K., Mansi, T., 2019. Statistical Atlases ages using a model-based adaptation framework. Medical and Computational Models of the Heart: Atrial Segmen- image analysis 15, 863{876. tation and LV Quanti cation Challenges: 9th Interna- Hoit, B.D., 2014. Left atrial size and function: Role in prog- tional Workshop, STACOM 2018, Held in Conjunction nosis. Journal of the American College of Cardiology 63, with MICCAI 2018, Granada, Spain, September 16, 2018, 493{505. doi:10.1016/j.jacc.2013.10.055. Revised Selected Papers. volume 11395. Springer. Hunold, P., Vogt, F.M., Schmermund, A., Debatin, Punithakumar, K., Ayed, I.B., Islam, A., Ross, I.G., Li, S., J.F., Kerkho , G., Budde, T., Erbel, R., Ewen, K., 2010. Regional heart motion abnormality detection via Barkhausen, J., 2003. Radiation exposure during car- information measures and unscented Kalman ltering, in: diac CT: e ective doses at multi{detector row CT and International Conference on Medical Image Computing electron-beam CT. Radiology 226, 145{152. and Computer-Assisted Intervention, Springer. pp. 409{ Julier, S., Uhlmann, J., 2004. Unscented ltering and non- linear estimation 92, 401{422. Radau, P., Lu, Y., Connelly, K., Paul, G., Dick, A., Wright, Kaus, M.R., Von Berg, J., Weese, J., Niessen, W., Pekar, G., 2009. Evaluation framework for algorithms segment- V., 2004. Automated segmentation of the left ventricle in ing short axis cardiac mri. The MIDAS Journal-Cardiac cardiac mri. Medical image analysis 8, 245{254. MR Left Ventricle Segmentation Challenge 49. Kowallick, J.T., Kutty, S., Edelmann, F., Chiribiri, A., Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: Con- Villa, A., Steinmetz, M., Sohns, J.M., Staab, W., Bet- volutional networks for biomedical image segmentation, tencourt, N., Unterberg-Buchwald, C., Hasenfu, G., in: International Conference on Medical image comput- Lotz, J., Schuster, A., 2014. Quanti cation of left atrial ing and computer-assisted intervention, Springer. pp. 234{ strain and strain rate using Cardiovascular Magnetic Res- onance myocardial feature tracking: A feasibility study. Santos, A.B.S., Kraigher-Krainer, E., Gupta, D.K., Zile, Journal of Cardiovascular Magnetic Resonance 16, 60. M.R., Pieske, B., Voors, A.A., Lefkowitz, M.P., Packer, doi:10.1186/s12968-014-0060-6. M., Mcmurray, J.J., Shah, A.M., Solomon, S.D., 2014. Im- Labbe, R., 2014. Kalman and bayesian lters in python. paired left atrial function in heart failure with preserved Liu, S., Guan, Z., Zheng, X., Meng, P., Wang, Y., Li, Y., ejection fraction: A speckle tracking echocardiography Zhang, Y., Yang, J., Jia, D., Ma, C., 2018. Impaired left study. European Journal of Heart Failure 16, 1096{1103. 17 Santos, A.B.S., Roca, G.Q., Claggett, B., Sweitzer, N.K., Zhang, X., Martin, D.G., Noga, M., Punithakumar, K., Shah, S.J., Anand, I.S., Fang, J.C., Zile, M.R., Pitt, B., 2018. Fully automated left atrial segmentation from MR Solomon, S.D., Shah, A.M., 2016. Prognostic relevance image sequences using deep convolutional neural network of left atrial dysfunction in heart failure with preserved and unscented Kalman lter, in: IEEE International Con- ejection fraction. Circulation: Heart Failure 9, 1{11. ference on Bioinformatics and Biomedicine, pp. 2316{ Tobon-Gomez, C., Geers, A.J., Peters, J., Weese, J., Pinto, 2323. K., Karim, R., Ammar, M., Daoudi, A., Margeta, J., San- Zheng, Y., Barbu, A., Georgescu, B., Scheuering, M., Co- doval, Z., et al., 2015. Benchmark for algorithms segment- maniciu, D., 2008. Four-chamber heart modeling and ing the left atrium from 3D CT and MRI datasets. IEEE automatic segmentation for 3-d cardiac ct volumes us- transactions on medical imaging 34, 1460{1473. ing marginal space learning and steerable features. IEEE Von Roeder, M., Rommel, K.P., Kowallick, J.T., Blazek, S., transactions on medical imaging 27, 1668{1681. Besler, C., Fengler, K., Lotz, J., Hasenfu, G., Luc  ke, C., Zhu, L., Gao, Y., Yezzi, A., MacLeod, R., Cates, J., Tan- Gutberlet, M., Schuler, G., Schuster, A., Lurz, P., 2017. nenbaum, A., 2012. Automatic segmentation of the left In uence of Left Atrial Function on Exercise Capacity and atrium from MRI images using salient feature and con- Left Ventricular Function in Patients with Heart Failure tour evolution, in: Engineering in Medicine and Biology and Preserved Ejection Fraction. Circulation: Cardiovas- Society (EMBC), 2012 Annual International Conference cular Imaging 10, 1{9. of the IEEE, IEEE. pp. 3211{3214. Xia, Q., Yao, Y., Hu, Z., Hao, A., 2019. Automatic Zreik, M., Leiner, T., de Vos, B.D., van Hamersvelt, R.W., 3D Atrial Segmentation from GE-MRIs Using Volumet- Viergever, M.A., I sgum, I., 2016. Automatic segmenta- ric Fully Convolutional Networks, in: Lecture Notes in tion of the left ventricle in cardiac CT angiography us- Computer Science, Statistical Atlases and Computational ing convolutional neural networks, in: Biomedical Imag- Models of the Heart Workshop on Atrial Segmentation ing (ISBI), 2016 IEEE 13th International Symposium on, and LV Quanti cation Challenges. Springer International IEEE. pp. 40{43. Publishing. volume 11395, pp. 211{220. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Electrical Engineering and Systems Science arXiv (Cornell University)

Fully Automated Left Atrium Segmentation from Anatomical Cine Long-axis MRI Sequences using Deep Convolutional Neural Network with Unscented Kalman Filter

Loading next page...
 
/lp/arxiv-cornell-university/fully-automated-left-atrium-segmentation-from-anatomical-cine-long-Ar7nTnQ6F8

References (34)

ISSN
1361-8415
eISSN
ARCH-3348
DOI
10.1016/j.media.2020.101916
Publisher site
See Article on Publisher Site

Abstract

This study proposes a fully automated approach for the left atrial segmentation from routine cine long- axis cardiac magnetic resonance image sequences using deep convolutional neural networks and Bayesian ltering. The proposed approach consists of a classi cation network that automatically detects the type of long-axis sequence and three di erent convolutional neural network models followed by unscented Kalman ltering (UKF) that delineates the left atrium. Instead of training and predicting all long-axis sequence types together, the proposed approach rst identi es the image sequence type as to 2, 3 and 4 chamber views, and then performs prediction based on neural nets trained for that particular sequence type. The datasets were acquired retrospectively and ground truth manual segmentation was provided by an expert radiologist. In addition to neural net based classi cation and segmentation, another neural net is trained and utilized to select image sequences for further processing using UKF to impose temporal consistency over cardiac cycle. A cyclic dynamic model with time-varying angular frequency is introduced in UKF to characterize the variations in cardiac motion during image scanning. The proposed approach was trained and evaluated separately with varying amount of training data with images acquired from 20, 40, 60 and 80 patients. Evaluations over 1515 images with equal number of images from each chamber group acquired from an additional 20 patients demonstrated that the proposed model outperformed state-of-the-art and yielded a mean Dice coecient value of 94.1%, 93.7% and 90.1% for 2, 3 and 4-chamber sequences, respectively, when trained with datasets from 80 patients. Keywords: Left Atrial Segmentation, Magnetic Resonance Imaging, Long-Axis Sequences, Deep Convolutional Neural Network, Unscented Kalman Filter 1. Introduction the anatomical assessment, the functional assess- ment of the left atrium is often performed using cine MRI or echocardiography. The advances in cardiac The assessment of left atrial function is becoming MRI technology have led to the generation of a large increasingly important due to its role in prognosis amount of imaging data with an increasingly high and risk strati cation of several cardiovascular dis- level of quality. However, segmenting a large num- eases including cardiomyopathy, ischemic heart dis- ber of left atrial images in cardiac MRI over the ease and valvular heart disease (Hoit, 2014; Kowal- entire cardiac cycle is a tedious and complex task lick et al., 2014; Tobon-Gomez et al., 2015). While for clinicians, who have to manually extract im- both the Computed Tomography (CT) and Mag- portant information (Despotovi c et al., 2015). The netic Resonance Imaging (MRI) could be used for manual analysis is often time-consuming and sub- ject to inter- and intra-operator variability. Due to these diculties, there is an increasing demand Corresponding author: Kumaradevan Punithakumar (Email: punithak@ualberta.ca) Preprint accepted by Medical Image Analysis arXiv:2009.13627v2 [cs.CV] 22 Nov 2020 for computerized methods to shorten segmentation quences. First, a classi cation network is intro- time and improve the accuracy of cardiac diagnosis duced to automatically detect the input sequence and treatment. into three groups according to the number of car- A number of cardiac image segmentation meth- diac chambers visible in the images, i.e., the net- ods using CT or MRI have been proposed in the work will classify the sequence as 2-chamber, 3- literature to improve the accuracy of the cardiac chamber or 4-chamber long-axis cine MRI. Then, chamber delineations (Peng et al., 2016; Zheng a U-net (Ronneberger et al., 2015) based model et al., 2008; Budge et al., 2008; Kaus et al., 2004; which was trained with 5-fold cross-validation for Radau et al., 2009). However, most of these meth- the detected sequence is applied. In order to im- ods focus on automated delineation of the left ven- prove the temporal consistency of the delineations, tricle only (Peng et al., 2016; Petitjean and Dacher, a second classi cation network is trained to detect 2011; Zreik et al., 2016). The left atrium is more image sequences which tend to yield low Dice co- dicult to segment (Tobon-Gomez et al., 2015). ecient in comparison to manual contours and the The diculties arise from following reasons (Zhu unscented Kalman lter (UKF) with a periodic dy- et al., 2012): 1) The wall of the left atrium is namic model is utilized to improve the accuracy of relatively thin compared to the left ventricle; 2) the selected sequences. Evaluations over 1515 im- Boundaries are not clearly de ned when the blood ages acquired from 20 patients in comparison to pool of the left atrium goes into pulmonary veins; expert manual delineation for each chamber group and 3) The shape variability of the left atrium is demonstrated that the proposed method yielded large between di erent patients. There are a few signi cantly better results than U-net alone. More- methods proposed in the literature to achieve au- over, this study empirically assesses the impact of tomated left atrial segmentation using model based the number f training samples on the network per- methods (Ecabert et al., 2008, 2011; Ordas et al., formance by comparing the results of four di erent 2007), which rely on static images obtained using scenarios where the size of training sets are 20, 40, CT imaging (Hunold et al., 2003). In comparison 60 and 80 patients. to CT imaging, MRI produces higher temporal res- olution, without the use of ionizing radiation or in- 2. Method travenous contrast media, and cine MRI allows for assessment of motion over the entire cardiac cycle The objective of the proposed method is to auto- (Parry et al., 2005). matically delineate the left atrium from cine long- A few notable exceptions to studies proposed for axis 2D MRI sequences correspond to 2-chamber, 3- the segmentation of the left atrium from MRI im- chamber and 4-chamber views that are acquired in ages include a method based on traditional image routine clinical scans. The overall system diagram analysis techniques such as salient feature and con- in Fig. 1 depicts the components of the proposed tour evolution by Zhu et al. (2012) and deep learn- method which integrates a classi cation network to ing based method by Mortazi et al. (2017). In ad- detect chamber type, three U-net based frameworks dition, a public segmentation challenge to delineate for semantic segmentation followed by another clas- the chamber from gadolinium-enhanced 3D MRI se- si cation network to detect results with low accu- quences was hosted at the Statistical Atlases and racy and unscented Kalman lter to improve tem- Computational Models of the Heart (STACOM) poral consistency. Both proposed classifying net- workshop in the Medical Image Computing and works are based on convolutional neural networks Computer Assisted Intervention (MICCAI) confer- and shortened as Classnet-A and Classnet-B in ence in 2018 (Pop et al., 2019). More than 15 re- rest of the paper. Each input image is resized to search teams participated in the challenge and a V- 256 256 using zero-padding scheme along the im- net (Milletari et al., 2016) based convolutional neu- age borders. Instead of training blindly with all ral network approach by Xia et al. (2019) yielded image data, three separate U-net based neural nets the best performance. However, these methods rely corresponding to 2, 3 and 4-chamber sequences were on 3D MRI scans of a single point in the cardiac constructed and trained after classifying the se- phase which may not be suited for the functional quence using the Classnet-A. The network architec- assessment of the left atrium (Zhang et al., 2018). ture of the proposed Classnet-A is given in Fig. 2. This study proposes a fully automated left atrial The standard U-net model is limited when process- segmentation approach for long-axis cine MRI se- ing image sequences as the predictions are made 2 independently where the temporal consistency is Dice score less than the threshold 0.9 are designated not maintained. In order to improve the accu- as Class 1 and the rest of the images in the set that racy, the unscented Kalman lter to exploit time- yield a Dice score of 0.9 and above are designated serial aspect of cardiac motion in the MRI sequence as Class 2. The rationale for the selection is that data. The second classi er neural net, Classnet-B, images with certain degraded quality lead to poor is trained using to the results from the U-net in prediction performance using U-net. The Classnet- comparison to expert manual ground truth to de- B was trained using the comparative results by U- tect image sequences that require further process- net predictions and expert manual ground truth for ing. The unscented Kalman lter is then applied as the training data sets. Upon training, the Classnet- a post-processing step for image sequences for the B is applied to the testing set to identify image selected sequences. frames that will likely yield diminished accuracy (Dice score < 0:9) with U-net based predictions, without relying on ground truth labels. If any of 2.1. Classnet Architecture the image frames in a sequence is classi ed as Class As shown in Fig. 2, the image is processed by 1 by the Classnet-B, the entire sequence is selected a 3 3 32 convolutional layer with ReLU activa- for post-processing with the UKF. tion function followed by a 2 2 max pooling layer. Another 3  3  64 convolutional and 2  2 max 2.4. Unscented Kalman Filter pooling layers are added afterwards. The output of the max pooling layer is connected to a fully con- The UKF is utilized for sequences which were nected block with drop out (0.5). The nal fully detected by the Classnet-B that are likely to connected layer outputs the classi cation result of yield diminished U-net prediction accuracy in each the chamber group after a softmax activation. chamber group to further improve the predic- tion accuracy. In contrast to the earlier state- 2.2. U-net Architecture space model proposed for cardiac motion estimation Three U-net based models are applied exclusively (Punithakumar et al., 2010), the proposed method after classifying the input images using Classnet-A. utilizes an angular time-varying frequency compo- An example U-net model used in this study is de- nent. The input to the lter is set as contour points picted in Fig. 3. Similar to the original U-net ar- in Cartesian coordinates obtained from predicted chitecture (Ronneberger et al., 2015), each U-net image masks after boundary detection and uniform based model contains convolutional with ReLU ac- sampling. The UKF rst iterates over the uniformly tivation function, max pooling, skip connection and sampled points and then iterates over entire frames upsampling operations. It then connects to a fully in the cardiac cycle. The overall iterative process convolutional layer with sigmoid activation function detailing the UKF steps for each contour point is to obtain output masks. The labelled masks for shown in Fig. 5. The UKF relies on a state tran- training and testing are both scaled to [0; 1]. The sition model to enforce temporal smoothing. The loss function is chosen as the negative of Dice met- UKF utilizes the Kalman gain, which is computed ric given in (27). The input images are normalized at each iteration in the process to deal with mea- st th with respect to the 1 and 99 percentile of pixel surement uncertainty resulting from the U-net pre- intensity values of each image. dictions. 2.3. Image Selection for Post-processing 2.4.1. Uniform sampling of contour points After obtaining semantic segmentation results The semantic segmentation results for the 2, 3 from U-net based models, the Classnet-B is applied and 4-chamber sequences by U-net models were to select the images for post-processing. The train- converted into contour points. Each contour cor- ing and testing process is shown in Fig. 4. The layer responding to di erent cardiac frames is sampled details follow the same notation de ned in Fig. 2. to 60 points, and the sampling is performed by The Classnet-B neural net is trained using input transforming the contours to the polar coordinate MRI images from the training set with binary an- system. The origin of the polar coordinate system notations consisting of Class 1 and Class 2 where is selected as the centroid of the segmented region the classes are de ned based on the U-net predic- and the sampling points are selected at 6 inter- tion accuracy. The training set images which yield a vals along the theta-axis using linear interpolation. 3 2ch U-net U-net+UKF MRI Training Unscented Classnet-A Classnet-B Training Dice Output Sequences Scenario Kalman Filter Threshold Train 20 Train 20 3ch U-net Train 40 Train 40 Train 60 Train 60 Train 80 Train 80 4ch U-net Figure 1: Overall system diagram showing the components of the proposed method as well as di erent test scenarios of this study to empirically assess the impact of the size of the training set on the performance of the U-net and the proposed U- net+UKF algorithms. Four di erent test scenarios were created. The accuracy of the algorithms were evaluated using overlap and distance metrics in comparison to expert manual delineation of the left atrium from 2, 3 and 4-chamber long-axis MRI sequences. Channel size 2 chamber Image size ratio 3x3 Convolution layer Flatten layer 3 6 with ReLU activation 5 3 2 4 1/4 Input images 1/2 3 chamber Node size Classnet-A Dropout layer Fully connected layer with 0.5 4 chamber Layer details Training and testing Figure 2: Classnet-A architecture proposed in this paper to automatically detect the input image sequence and classify the sequence type. The input images are resized to 256  256. The image size is changed by applying a 2  2 maxpooling layer. A fully connected block with softmax activation is applied in the nal layer to obtain output class. 1 1 2 2 2 2 1 1 3 3 6 6 6 6 3 3 2 2 5 5 5 5 2 2 2 2 4 4 4 4 2 2 8 8 6 6 6 6 8 8 1/16 1/8 Input images U-net predictions 1/4 1/2 1x1 fully convolutional 1 layer with sigmoid Training and testing Skip connection activation Figure 3: U-net architecture for each chamber group. The convolution layer follows the same layer details in Fig. 2. Final output is a 256  256 image mask corresponding to the left atrial semantic segmentation. 4 Training labels Extract entire sequence 1 1 1 1 Training Dice 6 6 6 6 Threshold 1/8 1/4 Entire sequence for Testing inputs 1/2 selected frames 1 Classnet-B U-net predictions UKF Class 1 Class 1 1 1 1 1 4 2 6 6 6 6 1/8 1/4 U-net predictions Training inputs 1/2 Classnet-B Output segmentation Testing Class 2 Training Class 2 Figure 4: Visualization of the training and application of the Classnet-B neural net of the proposed method. The algorithm was trained using the input MRI images along with the classi cation labels obtained based on the Dice score values of the U-net predictions where a threshold of 0.9 was used in deciding the class labels. During application or testing stage, the Classnet-B predictions for the input MRI images were used in deciding the application of the UKF for the U-net predictions. Unscented Kalman Filter Prediction Update (based on the Measurement Update (calculate physical model and previous state) residual 𝑦 from measurement 𝑧 𝑘 𝑘 𝑆 𝑆 0 𝑘 with uncertainty) χ = SigmaFunc(𝑆 ,𝑃 ) 𝑧 𝑘 𝑘 𝑘 𝑘 𝑚 𝑐 𝑊 ,𝑊 = WeightFunc(𝑑 ,α,β,κ) 𝑍 = 𝐻 У +𝜂 𝑃 𝑃 𝑘 𝑘 𝑘 0 𝑘 𝑚 𝑐 У = 𝑓 χ +𝑣 𝑘 𝑘 𝑘 𝑘 𝜇 ,𝑃 = 𝑈𝑇 𝑍 ,𝑊 ,𝑊 ,cov(𝜂 ) z 𝑧 𝑘 𝑚 𝑐 𝑆 ′,𝑃 ′= 𝑈𝑇 У ,𝑊 ,𝑊 ,cov(𝑣 ) Filter Previous state 𝑘 𝑘 𝑘 𝑘 𝑦 = 𝑧 −𝜇 𝑘 𝑘 z initialization Contour measurement Calculate Kalman Gain State and Covariance Update (≈ (belief in measurement)/(belief in state)) 𝑘 +1 𝑆 = 𝑆 ′+𝐾 𝑦 𝑘 +1 𝑘 𝑘 𝑘 𝑐 ′ 𝑇 −1 ′ 𝑇 𝐾 = ෍𝑤 (У −𝑆 ) 𝑍 −𝜇 𝑃 𝑘 +1 𝑃 = 𝑃 −𝐾 𝑃 𝐾 𝑘 𝑘 𝑘 z 𝑧 𝑘 +1 𝑘 𝑘 𝑧 𝑘 Current state Figure 5: Visualization of the UKF post-processing process for each sample contour point over the cardiac cycle. The objective of the UKF post-processing is to enforce temporally consistent segmentation results by considering the image frames in the entire cardiac sequence in contrast to the processing by U-net which segments each frame independently. UT denotes unscented transform. The proposed method uses a state variable with dimension d = 7 and unscented transformation parameters = 0:1; = 2;  = 1. 5 Algorithm 1: Unscented Kalman lter for enforcing temporal consistency 1 Input: U-net prediction after uniform sampling with shape N  K  2. K is the total number of frames in each cardiac sequence and N is the number of samples in contour. 2 For contour point i = 1; 2; 3; :::; N do i T i i i ^ ^ ^ ^ 3 Initialize: S = [x_ x ^ x  y_ y^ y ! ] in (21)-(24), P in (25) and (26), Q in (17), R in (20). 1 1 1 1 1 1 1 1 1 1 1 4 For time frame k = 1; 2; 3; :::; K do: 5 Prediction update: i i i 6  = SigmaFunction(S ; P ) k k k m c 7 W ; W = WeightFunction(d; ; ; ) i i 8 Y = f ( ) + v k k k k 0 0 i i i m c i 9 S ; P = UnscentedTransform(Y ; W ; W ; Q ) k k k 10 Measurement update: 11 Z = H Y + k k i i m c i 12  ; P = UnscentedTransform(Z; W ; W ; R ) z z 1 i i i 13 y = z k k 14 Calculate Kalman gain: i c i i i T i 1 15 K = [ w (Y S )(Z  ) ](P ) z z k k k 16 State and covariance update: i i i i 17 S = S + K y k+1 k k k i i i i i T 18 P = P K P (K ) k+1 k k k 19 Output: Filtered contour with shape N  K  2 and improved temporal consistency. Centroid (x ; y ) of the segmented region is com- the points on contour obtained after resizing the c c puted using (1), where n is the number of raw con- prediction masks of U-net to original image size th tour points and (x ; y ) is the p point on raw con- and consider the corresponding state vector  = p p tour set [x  x x _ ] which describes the dynamics of point in x-coordinate direction. The terms x  and x _ denote n n X X mean position and velocity over the entire cardiac x = x ; y = y : (1) c p c p cycle, respectively. We exploit the periodicity fea- p=1 p=1 ture in each cardiac cycle and propose a state-space We convert each point (x ; y ) on the automated p p model (3) corresponding to the oscillation: left atrial contour from the Cartesian coordinate 2 3 2 3 system to the polar coordinate system using the 0 0 0 1 0 6 7 6 7 following equations: 6 7 6 7 (t) = 6 7  (t) + 6 7 w(t) ; (3) 0 0 1 0 0 4 5 4 5 2 2 ! ! 0 0 1 2 2 r = x + y ;  = tan : (2) p p p p where ! is the angular frequency and w(t) is the vector-valued white noise. The model describes A linear interpolation is then applied to nd uni- the cardiac motion as a simple harmonic motion. formly sampled points along the angular coordi- Nonetheless, cardiac movement is much more com- nates in the polar coordinate system. These points plex than a simple harmonic motion and a time- in the polar coordinate system are converted back varying angular velocity is added to account for the to the Cartesian coordinate system and used in the rate of oscillation variation and the discrete equiv- subsequent processing using unscented Kalman l- alence of the model is described in ter. = F  + w ; (4) k+1 k k k 2.5. Dynamic model for temporal periodicity A dynamic model is designed to characterize where k is the frame number between each heart cardiac motion. Assume that (x; y) be one of beat and the maximum value K denotes the num- 6 ber of frames in a cardiac cycle which is typically Small q and q will enforce the conformity of car- 1 2 between 25 to 30. F is given in the following: diac motion to the model, thereby a ect the accu- racy of the prediction results when motion deviates 2 3 1 0 0 from the model. Large values of q and q allow 1 2 6 7 6 7 for high uncertainties within the model, which may F = 6 7 ; k 1 cos(! T ) cos(! T ) sin(! T ) k k k 4 5 lead to poor temporal consistency. In our case, the ! sin(! T ) ! sin(! T ) cos(! T ) k k k k k noise parameters q = q = 1  10 are chosen 1 2 (5) empirically. where w is a discrete-time white noise, T is the k Let S = [x  x x _ y  y y _ ! ] be the state k k k k k k k k sampling interval and ! is the discrete angular fre- k vector that describes the corresponding dynamics quency. The initial estimated value of the angular at time frame k. Elements x , x _ , y , y _ denote, re- frequency ! is computed as 1 spectively, the mean position and velocity in x- coordinate and mean position and velocity in y- 2  Heart Rate coordinate. The discrete-time model that describes ! = ; (6) the cyclic motion of the point is given in and the time interval between two consecutive car- S = f (S ) + v ; (15) k+1 k k k diac frames T is calculated as where f (S ) is de ned by 60 k k T = : (7) 2 3 Heart Rate K F 0 0 k 33 31 6 7 The covariance of process noise Q is given by 6 7 f (S ) = 6 7 S ; (16) k k 0 F 0 k 33 k 31 4 5 Q = [q ] ; (8) k ij 33 0 0 1 33 33 where q is de ned in (9){(14): ij and v denotes a Gaussian process noise sequence with zero-mean and covariance that accommodates q = q T ; (9) unpredictable error due to model uncertainties q (! T sin(! T )) k k given in q = q = ; (10) 12 21 k 2 3 q = q = q (1 cos(! T )) ; (11) Q 0 0 13 31 k k 33 31 6 7 6 7 2 2 cov(v ) = ; (17) 6 7 k 0 Q 0 33 k 31 q ! (3! T 4 sin(! T ) 1 k k k 4 5 0 0 1 33 33 + cos(! T ) sin(! T )) k k + q (! T cos(! T ) sin(! T )) k k k where Q is de ned in (8). The measurement equa- q = ; tion is given in 2! (12) Z = H S +  ; (18) k k k 2 2 2 q ! (1 2 cos(! T ) + cos (! T )) k k 1 k where H is de ned in + q sin (! T ) " # q = q = ; 23 32 2! 0 1 0 0 0 0 0 H = ; (19) (13) 0 0 0 0 1 0 0 2 2 q ! (cos(! T ) sin(! T ) ! T ) 1 k k k and  is a zero-mean Gaussian noise sequence to model the measurement uncertainties such as po- q (cos(! T ) sin(! T ) + ! T ) k k k q = ; 33 tential false positives resulting from inaccurate U- 2! net predictions, with covariance demonstrated in (14) " # r 0 where q and q describe uncertainties associated 1 2 cov( ) = : (20) with the mean position and velocity respectively. 0 r 7 The parameter r characterizes the uncertainties as- where  is given in sociated with observations. Smaller values of r en- 2 3 r r force the conformity of the estimation to the mea- k kT 6 7 surements while larger values allow for associating 6 7 r r = 6 7 : (26) 1 r k T more uncertainty with the measurements. In the 4 5 r r 2r proposed study, U-net based model showed ade- kT T T quate raw accuracy of the prediction masks and r was set to 1 10 . Apart from the measurement 2.7. Iterative prediction and update uncertainties modeled using  , the UKF updates After generating sigma points using the method its state and covariance based on the relative ra- proposed in (Julier and Uhlmann, 2004), prediction tio of the belief in measurement over the belief in and update steps were performed iteratively based state by calculating Kalman gain shown in Fig. 5 on (15) and (18). The entire process of the UKF is to improve prediction performance. If the Kalman summarized in Algorithm 1. gain is high, the lter will rely more on the current measurement. Otherwise, it will rely more on the predicted value by the lter. 3. Experimental Setup and Results 2.6. Filter initialization 3.1. Dataset, evaluation criteria and implementa- tion details It is not possible to have prior information of The U-net and the proposed algorithms were the exact initial value of S in the proposed study. trained and tested over 100 MRI patient datasets Therefore, a two-point di erencing method pro- acquired retrospectively at the University Alberta posed by Bar-Shalom et al. (2004) is applied to Hospital. The study was approved by the Health initialize the position and velocity components of Research Ethics Board of the University of Al- the state. For instance, the initial position x ^ and berta. The details of the datasets used in the pro- velocity x_ in x-coordinate of the ith sample point: posed model are given in Table 1. The evaluations ix x ^ = z ; (21) were performed over 1515 images acquired from 20 patients in comparison to the expert manual seg- ix ix mentation of the left atrium from the 2, 3 and z z 2 1 x_ = ; (22) 4-chamber cine steady state processions long-axis MR sequences where each chamber group consists ix where z ; k = 1; 2; :::; K is the observation in x- of 505 images. The 2, 3 and 4-chamber sequences coordinate obtained from kth frame. The mean are shortened as 2ch, 3ch and 4ch in the rest of position over cardiac cycle illustrated in (23) was the paper. The ground truth manual segmentation initialized by taking the average of observation in was initially performed by a medical student using x-coordinate a semi-automated software and the contours were corrected by an experienced radiologist. ix x  = z : (23) K In order to empirically evaluate the impact of k=1 training sample size on the algorithm performance, Similarly, the initial state elements in y-coordinate four di erent scenarios were created by increasing can be computed using the same strategy and the the sample size from 20 to 80 patients with an incre- nal initial state vector input S is given in (24) ment of 20 patients per scenario (Refer to Fig. 1). These scenarios are shortened as Train 20-80 in the ^ ^ ^ ^ S = [x_ x ^ x  y_ y^ y ! ] : (24) 1 1 1 1 1 1 1 1 rest of the paper. For each case, the U-net based neural nets were trained and tested using 5-fold of The corresponding initial covariance is given in cross validation to prevent over- tting. The train- 2 3 ing loss function was chosen as the negative of the 0 0 Dice coecient. To demonstrate the e ectiveness of 1 33 31 6 7 6 7 the proposed approach, the results were compared P = 6 7 ; (25) 1 0  0 33 1 31 4 5 with the algorithm that was trained using a merged 0 0 1 set of all cardiac long-axis chambers. 33 33 8 Table 1: Details of data set used in the proposed study. Description Training Set Test Set Number of subjects 20 80 20 Patient's sex 40 Males / 40 Females 10 Males / 10 Females Scanner protocol Siemens AERA Siemens AERA Patient's age 4 85 years 29 73 years Magnetic strength 1.5 T 1.5 T Number of frames (K) 25 or 30 25 or 30 Image size (176, 184) (256, 208) pixels (192, 208) (256, 256) pixels Pixel spacing 1:445 1:797 mm 1:445 1:660 mm Repetition time (TR) 21:12 89:10 ms 27:10 67:25 ms Echo time (TE) 1:09 1:43 ms 1:12 1:18 ms 3.2. Implementation 3.3.1. Dice coecient Dice coecient illustrated in (27) measures the overlap between two delineated regions, where set The U-net based models, Classnet-A and A is the prediction image set and M is the manually Classnet-B were implemented in Python program- labelled image set ming language using Keras machine learning mod- ule with Tensor ow backend and cudnn. The l- 2jA\ Mj terpy Python module Labbe (2014) was used for DC = : (27) jAj+jMj the implementation of Bayesian based unscented Kalman lter. Three U-net based architectures were utilized with linear variable learning rate with 3.3.2. Hausdor distance 4 5 Adam optimizer from 1 10 in initial to 1 10 The Hausdor distance measures the maximum 5 6 for 2ch U-net and 1  10 in initial to 1  10 deviation between two contours in terms of the Eu- for 3ch and 4ch. The decay was uniform per each clidean distance. The Hausdor distance between epoch. Batch size was set as 32 and epoch num- manual and automatic contours was computed as ber is set to 500 in a 5-fold cross validation scheme. follows: The proposed model was tested on a desktop with Intel core i7-7700 HQ CPU with 16 GB RAM. The i j d (a; m) = max max min d(p ; p ) ; neural net models were trained and tested with a m i j an NVIDIA GTX 1080 Ti graphics processing unit i j with 12 GB memory. The approximate training max min d(p ; p ) ; (28) a m j i time for each chamber group is 7 hours and the prediction time is 4 ms per image for the proposed where d() is the Euclidean distance, fp g denotes method. the set of points on automated contour and fp g denotes the set of points on manual contour. 3.3. Quantitative Evaluation Metrics 3.3.3. Reliability metric The proposed method and the U-net based ap- The reliability metric is evaluated using the func- proach were evaluated quantitatively using Dice co- tion given in (29). The complementary cumulative ecient (DC), Hausdor distance (HD) and relia- distribution function is de ned for each d 2 [0; 1] as bility metric. the probability of obtaining Dice coecient higher 9 than d over all images 8 out of 12 scenarios tested with di erent chamber groups and training sets. Images with Dice The best performances for each chamber group coecient higher than d in terms of Dice coecient and Hausdor distance R(d) = P (DC > d) = : Total images were obtained using the proposed U-net+UKF us- (29) ing Classnet-A approach with Train 80 except for the 4ch segmentation measured in terms of Dice score where a small decrease of 0.1% in performance 3.4. Experimental results was observed with the application of Classnet-A. 3.4.1. Quantitative assessment The approach yielded average Dice coecients of The Classnet-A and Classnet-B 's performance 94.1%, 93.7% and 90.1% for 2ch, 3ch, and 4ch, re- are reported in Table 2 and Table 3, respectively. spectively. The corresponding average Hausdor Both classi cation networks achieved 100% accu- distance values were 6.0 mm, 5.5 mm and 8.4 mm, racy in the Train 80 scenario where the approaches respectively. have correctly classi ed all 1515 images to the re- The accuracy of the proposed U-net+UKF with spective chamber group or image classi cation into Classnet-A method steadily increased with the size Class 1 and Class 2 based on U-net segmentation of the training data for all chamber groups as re- accuracy. The results from both tables show that ported in Tables 4{6. The increase in accuracy with the accuracy of the classi cation algorithms im- the size of training data set is also observed in the proves with the size of training set. The number reliability assessment as depicted in Fig. 10. The of classes on ground truth remained the same for improvement was signi cant from Train 20 to Train Train 20, 40, 60, and 80 scenarios for Classnet-A, as 40 in 2-chamber and marginal from Train 40 to indicated in Table 2, since it is based on the cham- Train 80. In 3-chamber group, the improvement ber views on the test set. However, the number was critical from Train 40 to Train 60 while in 4- of classes on ground truth varied for di erent sce- chamber group the overall improvement is marginal narios for Classnet-B, as indicated in Table 3, due with the training size. to varying U-net prediction accuracy on the test- 3.4.2. Visual assessment ing set. For instance, the U-net trained with Train 80 scenario resulted in more images (n = 1177) in Fig. 6 shows example automatic contours by the Class 2 (Dice score  0:9) than the one trained with proposed U-net+UKF approach (blue curve) as Train 20 (n = 345). well as U-net (red curve) with the application of The quantitative assessment of the automated Classnet-A. The manual ground truth delineation is segmentation results for 2ch, 3ch and 4ch are given depicted by the yellow area. Fig. 7 shows compara- in Tables 4, 5, and 6, respectively. The tables re- tive examples of U-net and U-net+UKF prediction port the mean and standard deviation of Dice coe- results on selected frames over a complete cardiac cient and Hausdor distance values for U-net+UKF cycle for 2, 3, and 4 chamber views where the U-net and U-net alone approaches with and without the yielded temporally inconsistent results. The exam- application of Classnet-A neural net for di erent ples show that the proposed application of temporal sizes of training data sets. As shown in Tables 4{6, smoothing for image sequences consisting of image the proposed U-net+UKF method with Classnet-A frames with inaccurate U-net predictions led to im- outperformed U-net for all three chamber groups in proved segmentation accuracy. Fig. 8 shows the terms of Hausdor distance in Train 80 scenario. trajectories of left atrial boundary points over the The method also outperformed U-net in 2ch and complete cardiac cycle, which demonstrates the ad- 3ch sequences in Train 80 scenario. The reported vantage of applying UKF for better temporal con- values in Tables 4{6 demonstrate that the proposed sistency compared to U-net alone. As shown in U-net+UKF approach outperformed U-net alone Fig. 8(a), the U-net yielded unrealistic trajectories method in terms of Dice coecient in 23 out of 24 that do not match with the left atrial dynamics, and di erent tested scenarios that consist of with and the trajectories are corrected with the application without Classnet-A, Train 20 { Train 80 scenarios, of the UKF, as shown in Fig. 8(b). The segmenta- and 2ch { 4ch chamber groups. The reported values tion results by the proposed approach over di erent also demonstrate that the Classnet-A approach im- temporal frames for 2ch, 3ch and 4ch is given in proved performance in terms of Dice coecient in Fig. 9. 10 2ch Train 80 3ch Train 80 4ch Train 80 DC: U-net+UKF = 93:3% DC: U-net+UKF = 94:1% DC: U-net+UKF = 95:2% DC: U-net = 91:7% DC: U-net = 89:6% DC: U-net = 91:5% Figure 6: Representative example results by the proposed method (blue curve) and U-net (red curve) approaches in comparison to the expert manual ground truth (yellow area). The proposed method yielded more conformity with manual ground truth than U-net approach. Table 2: Classi cation result for Classnet-A proposed in this paper. The test images consisted of 505 images for each chamber group. Overall accuracy is reported for each training scenario in the last row. Prediction Result Ground Truth Train 20 Train 40 Train 60 Train 80 2ch 3ch 4ch 2ch 3ch 4ch 2ch 3ch 4ch 2ch 3ch 4ch 2ch 505 0 0 505 0 0 505 0 0 505 0 0 3ch 20 485 0 0 480 25 0 505 0 0 505 0 4ch 0 24 481 0 17 488 0 30 475 0 0 505 Accuracy(%) 97.1 97.2 98.0 100.0 Table 3: Classi cation result for Classnet-B proposed in this paper. The test images consisted of 1515 images from all chamber groups (each consists of 505 images). Class 1 indicates the images that need to be post-processed using the UKF and Class 2 indicates the images that do not need to be post-processed due to their higher accuracy. Overall accuracy is reported for each training scenario in the last row. Prediction Result Ground Truth Train 20 Train 40 Train 60 Train 80 Class 1 Class 2 Class 1 Class 2 Class 1 Class 2 Class 1 Class 2 Class 1 1167 3 634 0 542 0 338 0 Class 2 52 293 0 881 0 973 0 1177 Accuracy(%) 96.4 100.0 100.0 100.0 11 3.4.3. Ablation study 2. The impact of Classnet-A on the overall perfor- mance becomes unpredictable with smaller training Table 7 reports the results for U-net+UKF ap- sets such as Train 20 scenario where Classnet-A led proach without Classnet-A and Classnet-B evalu- to inaccurate classi cations in addition to the vary- ated in terms of Dice score and Hausdor distance. ing performances of the corresponding U-Net algo- The reported values in the table demonstrate the rithms. Although the U-Net with Classnet-A led impact of Classnet-B in improving the segmenta- to improved performance over the U-Net without tion accuracy by automatically selecting the images Classnet-A for Train 20 scenario for 4-chamber se- that require post-processing with UKF instead of quences as reported in Table 6, no similar improve- applying it to the entire image set. The improve- ments were observed for 2 and 3 chamber sequences ment was observed in all chamber views in terms of as reported in tables 4 and 5. Future studies will Dice coecient as reported in Table 7 as well as in explore the performance of neural networks with column 3 of Tables 4, 5, and 6. smaller training sets. The intent of Classnet-B is to detect images 4. Discussion that will lead to less accurate left atrial segmenta- tion by U-net, and therefore, the training phase of In this study, we proposed and validated a fully Classnet-B utilized the ground truth data from the automated segmentation approach to delineate left training set. Upon trained, the Classnet-B neural atrium from MRI sequences using deep convolu- net parameters were no longer updated and none of tional neural networks and Bayesian ltering. Pre- the ground truth delineations from the test set was vious automated left atrial segmentation method utilized in the process. attempted to delineate the chamber from a single Among the three long-axis sequences, the 3- time point in the cardiac phase using static images chamber view requires more experience to segment (Hunold et al., 2003; Ordas et al., 2007; Ecabert as the anterior margin of the mitral valve, being ad- et al., 2008, 2011; Zhu et al., 2012; Mortazi et al., jacent to the aortic valve, is harder to delineate than 2017; Pop et al., 2019). The proposed study relied in other views. The results of this study show that on 2-chamber, 3-chamber and 4-chamber 2D MRI the performance is lower for the 3-chamber view in sequences, which may or may not be orthogonal to comparison to other views when the proposed ap- each other. These long-axis sequences do not form proach was trained with images from 20 patients. a 3D MRI dataset, as in the case of STACOM MIC- However, as indicated in Table 5, a signi cant per- CAI 2018 challenge Pop et al. (2019), however, o er formance improvement was achieved in 3-chamber a series of images to assess the temporal characteris- view results when the training set was increased to tics of the left atrium over the cardiac cycle. To the 80 patients. best of authors' knowledge, this is the rst study to In this study, the test set MRI sequences were propose a fully automated approach to delineate selected from 20 patients, and none of the scans left atrium over the entire cardiac cycle that could were excluded in the analysis regardless of the im- serve as the basis for the functional assessment of age quality, which led to high inhomogeneity among the chamber. the test images and a high standard deviation was The intent of Classnet-A is to automatically iden- observed for the evaluation metrics reported in Ta- tify the 2-chamber, 3-chamber and 4-chamber views bles 4, 5, and 6. Table 8 reports the numbers using image pixel information only. The experimen- of samples used for training the U-Net approach tal analysis showed that the Classnet-A achieved with and without Classnet-A. Due to the di er- perfect classi cation performance without a single ent number of samples used for training the U-Net misclassi cation when trained with images from 80 approach, the performance of the algorithm varies patients. The results presented in Tables 4, 5, and even when the Classnet-A o ers perfect classi ca- 6 show that the application of Classnet-A led to an tions. For instance, the results for the Train 80 improvement on segmentation accuracy measured scenario reported in columns 3 and 5 for the U-Net in terms of Dice score in seven cases, slight reduc- approach with and without Classnet-A in Tables 4, tion (< 1%) or similar performance in four cases, 5, 6 are di erent where Classnet-A yielded perfect and a more considerable reduction in one case for classi cation. Train 60 and Train 80 scenarios where the algorithm Clinical importance of the left atrial functional yielded a perfect classi cation as indicated in Table analysis has been emphasized in many echocardio- 12 2ch Frame 1 2ch Frame 6 2ch Frame 11 2ch Frame 16 2ch Frame 21 Proposed= 96:0% Proposed = 96:2% Proposed = 96:0% Proposed = 95:1% Proposed = 94:1% U-net = 94:2% U-net = 83:7% U-net = 82:4% U-net = 91:7% U-net = 91:6% 3ch Frame 1 3ch Frame 6 3ch Frame 11 3ch Frame 16 3ch Frame 21 Proposed= 93:2% Proposed = 93:5% Proposed = 93:0% Proposed = 91:3% Proposed = 92:5% U-net = 95:7% U-net = 92:7% U-net = 68:8% U-net = 71:4% U-net = 88:9% 4ch Frame 1 4ch Frame 6 4ch Frame 11 4ch Frame 16 4ch Frame 21 Proposed= 86:8% Proposed = 90:7% Proposed = 90:5% Proposed = 91:2% Proposed = 91:3% U-net = 76:9% U-net = 71:9% U-net = 73:5% U-net = 66:0% U-net = 72:4% Figure 7: An example showing the prediction results on selected frames over a complete cardiac cycle (K=25) for 2, 3 and 4-chamber views (from top to bottom) where the U-net yielded temporally inconsistent segmentation results. The yellow area is the manually labelled left atrium. The blue and red curves are the prediction results by the proposed U-net+UKF and U-net alone, respectively. A higher conformance was observed between expert manual segmentation and the results by the proposed fully automated approach which demonstrates the bene ts of applying temporal smoothing with the UKF. 13 Table 4: Evaluation of the automated segmentation results over 20 patients data sets using Dice coecient and Hausdor distance metrics. The results of the U-net and the proposed U-net+UKF methods were compared against expert manual ground truth produced by clinicians with and without the application of Classnet-A neural net. The evaluations were performed over 1515 images obtained from the long-axis cine 2-chamber view. The chamber group consisted of 505 images. 2ch Dice Coecient (%) 2ch Hausdor Distance (mm) Method without Classnet-A with Classnet-A without Classnet-A with Classnet-A U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF Train 20 84:3 6:7 85:4 6:0 82:5 13:5 86:8 9:7 9:4 5:4 9:1 5:0 12:2 8:5 10:2 7:7 Train 40 90:8 4:0 91:6 3:2 91:8 7:6 92:9 4:2 9:5 6:9 8:5 5:0 8:6 6:8 7:7 6:2 Train 60 91:1 3:8 91:8 3:0 93:7 2:6 93:7 2:7 8:8 6:9 7:6 4:6 6:7 3:5 6:0 2:9 Train 80 93:4 3:0 93:5 3:0 93:5 5:8 94:1 3:7 7:7 6:4 6:6 4:3 7:0 4:3 6:0 3:3 Table 5: Evaluation of the automated segmentation results over 20 patients data sets using Dice coecient and Hausdor distance metrics from the long-axis cine 3-chamber view. The chamber group consisted of 505 images. 3ch Dice Coecient (%) 3ch Hausdor Distance (mm) Method without Classnet-A with Classnet-A without Classnet-A with Classnet-A U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF Train 20 83:0 6:6 84:8 8:0 77:0 9:5 80:4 8:7 19:7 20:9 17:1 16:2 11:2 9:3 10:5 9:3 Train 40 89:3 5:0 90:1 10:7 79:1 16:2 83:4 14:3 17:0 22:8 13:9 16:5 16:6 15:6 9:4 8:8 Train 60 90:9 5:3 91:3 10:5 92:5 8:4 92:6 8:4 13:9 18:3 12:4 15:1 10:4 15:0 7:4 5:4 Train 80 92:4 4:0 92:5 9:1 90:3 15:2 93:7 8:2 16:3 22:9 12:3 14:8 5:7 3:2 5:5 3:0 Table 6: Evaluation of the automated segmentation results over 20 patients data sets using Dice coecient and Hausdor Distance metrics from the long-axis cine 4-chamber view. The chamber group consisted of 505 images. 4ch Dice Coecient (%) 4ch Hausdor Distance (mm) Method without Classnet-A with Classnet-A without Classnet-A with Classnet-A U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF U-net U-net+UKF Train 20 82:0 10:1 84:4 5:1 85:6 5:6 86:4 5:6 17:3 12:8 14:9 10:4 11:7 5:7 9:3 5:1 Train 40 87:2 12:6 88:0 8:1 88:9 5:9 89:6 5:1 12:9 11:8 10:4 7:8 9:9 11:7 8:5 8:1 Train 60 87:9 11:2 88:5 10:1 87:1 16:5 88:4 13:4 12:7 11:8 10:4 7:9 11:6 10:8 10:1 10:1 Train 80 88:9 11:3 90:2 4:2 88:9 10:7 90:1 8:1 11:8 10:9 10:0 7:6 12:2 5:6 8:4 4:2 Table 7: Evaluation of automated segmentation results over 20 patient data sets for the U-Net+UKF approach without Classnet-A and Classnet-B in terms of Dice coecient and Hausdor distance metrics for the long-axis cine 2, 3, 4-chamber views. The test set for each chamber group consisted of 505 images. Dice Coecient (%) Hausdor Distance (mm) Method 2ch 3ch 4ch 2ch 3ch 4ch Train 20 84:6 6:5 83:3 6:8 83:5 8:4 9:5 3:3 9:6 2:8 9:7 3:8 Train 40 90:9 4:0 89:4 4:9 87:7 10:9 7:5 2:4 10:0 7:3 9:7 7:5 Train 60 91:2 3:8 91:0 5:2 88:2 10:6 7:0 2:5 8:6 3:5 10:0 7:8 Train 80 93:5 2:9 92:5 4:0 89:9 9:9 7:0 2:6 9:1 3:2 8:5 6:1 14 Table 8: The number of images utilized in training the U-Net with and without Classnet-A for 2, 3, 4-chamber sequences. The image sequences have either 25 or 30 images leading to slight variations in the number of images for 2, 3, and 4 chamber training sets. With Classnet-A Without Classnet-A Method 2ch 3ch 4ch Combined 2,3,4ch Train 20 500 505 505 1510 Train 40 1020 1025 1025 3070 Train 60 1535 1540 1540 4615 Train 80 2115 2015 2090 6220 though such delineations could be used for comput- ing functional parameters such as volume or volume rate over time, some additional parameters such as strain or strain-rate require open contours repre- senting the boundary of the chamber walls that ex- cludes anatomical structures corresponds to the mi- tral valve. Future studies will attempt to address this limitation. (a) U-net (b) U-net+UKF 5. Conclusion Figure 8: An example showing the trajectories of the left This paper proposes a fully automated approach atrial boundary points over the entire cardiac cycle (K=25) to segment the left atrium from routine long-axis estimated using U-net alone, and U-net+UKF approaches. The example demonstrates the bene ts of applying the UKF cardiac MR image sequences acquired over the en- to enforce the temporal consistency in estimating the con- tire cardiac cycle. The proposed methods intro- tour points as the method allows for correcting inaccurate duces a neural net approach to classify the input im- segmentation results by the U-net in a few arbitrary image ages and loads the corresponding U-net based con- frames. volutional neural network architecture for semantic segmentation. A second classi er network is devel- oped to select sequences for further processing with graphy studies particularly the prognostic value of left atrial measurements for patients with heart fail- the Unscented Kalman lter to impose temporal pe- ure with preserved ejection fraction (Morris et al., riodicity over the cardiac cycle. The proposed algo- 2011; Santos et al., 2014, 2016; Cameli et al., 2017; rithm was trained separately with di erent scenar- Liu et al., 2018). Clinical studies that used MRI ios with varying amount of training data with im- ages acquired from 20, 40, 60, 80 patients. The pre- sequences for cardiac functional measurements also diction results were compared against expert man- led to similar ndings and demonstrated the prog- ual delineations over an additional 20 patients in nostic values of the left atrial functional parameters terms of the Dice coecient and Hausdor distance. in heart failure patients (Von Roeder et al., 2017; The quantitative assessment showed that the pro- Chirinos et al., 2018). One of the signi cant ad- posed method performed better than recent U-net vantages of the proposed method is that it is fully algorithm. The proposed approach yielded 100% automated and, therefore, will allow for analyzing accuracy for chamber classi cation and a mean Dice a large number of MRI images which otherwise te- coecient of 94:1%, 93:7% and 90:1% for 2, 3 and 4- dious and time-consuming. Further clinical studies chamber sequences after training using 80 patients are warranted to objectively measure the left atrial data. function and assess its predictive ability for heart failures with preserved and reduced ejection frac- tion. Acknowledgments The proposed approach performs semantic seg- mentation to delineate left atrium where pixels are X. Zhang was supported by the Mitacs Globalink classi ed either foreground or background. Al- Research Internship program. D. Glynn Martin was 15 2ch Frame 1 2ch Frame 6 2ch Frame 11 2ch Frame 16 2ch Frame 21 3ch Frame 1 3ch Frame 6 3ch Frame 11 3ch Frame 16 3ch Frame 21 4ch Frame 1 4ch Frame 6 4ch Frame 11 4ch Frame 16 4ch Frame 21 Figure 9: An example showing the prediction results over a complete cardiac cycle (25 frames in total) for 2, 3 and 4-chamber groups (from top to bottom). The yellow area is the manually labelled region of interest corresponds to left atrium and the blue curve is the prediction results by the proposed U-net+UKF. A higher conformance was observed between expert manual segmentation and the results by the proposed fully automated approach 2-chamber reliability 3-chamber reliability 4-chamber reliability Figure 10: Reliability versus Dice metric for the proposed U-net+UKF with Classnet-A approach with varying training set data. The training were performed using data sets acquired from 20, 40, 60 and 80 patients for each chamber group. The evaluations were performed over 505 images for each chamber group acquired from 20 patients. 16 supported by the University of Alberta Radiology atrial systolic function and inter-atrial dyssynchrony may contribute to symptoms of heart failure with preserved left Endowed Fund. The work in-part was supported ventricular ejection fraction: A comprehensive assessment by a grant from the Servier Canada Inc. by echocardiography. International Journal of Cardiology 257, 177{181. Milletari, F., Navab, N., Ahmadi, S., 2016. V-Net: Fully con- References volutional neural networks for volumetric medical image segmentation, in: 2016 Fourth International Conference Bar-Shalom, Y., Li, X.R., Kirubarajan, T., 2004. Estima- on 3D Vision (3DV), pp. 565{571. tion with applications to tracking and navigation: theory Morris, D.A., Gailani, M., Vaz P erez, A., Blaschke, F., Dietz, algorithms and software. John Wiley & Sons.  R., Haverkamp, W., Ozcelik, C., 2011. Right ventricular Budge, L.P., Sha er, K.M., Moorman, J.R., Lake, D.E., myocardial systolic and diastolic dysfunction in heart fail- Ferguson, J.D., Mangrum, J.M., 2008. Analysis of in ure with normal left ventricular ejection fraction. Journal vivo left atrial appendage morphology in patients with of the American Society of Echocardiography 24, 886{897. atrial brillation: a direct comparison of transesophageal Mortazi, A., Karim, R., Rhode, K., Burt, J., Bagci, U., 2017. echocardiography, planar cardiac ct, and segmented three- Cardiacnet: Segmentation of left atrium and proximal dimensional cardiac ct. Journal of interventional cardiac pulmonary veins from MRI using multi-view cnn, in: In- electrophysiology 23, 87{93. ternational Conference on Medical Image Computing and Cameli, M., Mandoli, G.E., Mondillo, S., 2017. Left atrium: Computer-Assisted Intervention, Springer. pp. 377{385. the last bulwark before overt heart failure. Heart Failure Ordas, S., Oubel, E., Leta, R., Carreras, F., Frangi, A.F., Reviews 22, 123{131. 2007. A statistical shape model of the heart and its appli- Chirinos, J.A., Sardana, M., Ansari, B., Satija, V., Kuri- cation to model-based segmentation, in: Medical Imaging akose, D., Edelstein, I., Oldland, G., Miller, R., Gad- 2007: Physiology, Function, and Structure from Medical dam, S., Lee, J., Suri, A., Akers, S.R., 2018. Left Atrial Images, International Society for Optics and Photonics. Phasic Function by Cardiac Magnetic Resonance Feature p. 65111K. Tracking Is a Strong Predictor of Incident Cardiovascular Parry, D.A., Booth, T., Roland, P.S., 2005. Advantages of Events. Circulation. Cardiovascular imaging 11, e007512. magnetic resonance imaging over computed tomography Despotovi c, I., Goossens, B., Philips, W., 2015. MRI seg- in preoperative evaluation of pediatric cochlear implant mentation of the human brain: challenges, methods, and candidates. Otology & Neurotology 26, 976{982. applications. Computational and mathematical methods Peng, P., Lekadir, K., Gooya, A., Shao, L., Petersen, S.E., in medicine 2015. Frangi, A.F., 2016. A review of heart chamber segmen- Ecabert, O., Peters, J., Schramm, H., Lorenz, C., von Berg, tation for structural and functional analysis using cardiac J., Walker, M.J., Vembar, M., Olszewski, M.E., Subra- magnetic resonance imaging. Magnetic Resonance Mate- manyan, K., Lavi, G., et al., 2008. Automatic model- rials in Physics, Biology and Medicine 29, 155{195. based segmentation of the heart in CT images. IEEE Petitjean, C., Dacher, J.N., 2011. A review of segmentation transactions on medical imaging 27, 1189. methods in short axis cardiac MR images. Medical image Ecabert, O., Peters, J., Walker, M.J., Ivanc, T., Lorenz, C., analysis 15, 169{184. von Berg, J., Lessick, J., Vembar, M., Weese, J., 2011. Pop, M., Sermesant, M., Zhao, J., Li, S., McLeod, K., Young, Segmentation of the heart and great vessels in CT im- A.A., Rhode, K., Mansi, T., 2019. Statistical Atlases ages using a model-based adaptation framework. Medical and Computational Models of the Heart: Atrial Segmen- image analysis 15, 863{876. tation and LV Quanti cation Challenges: 9th Interna- Hoit, B.D., 2014. Left atrial size and function: Role in prog- tional Workshop, STACOM 2018, Held in Conjunction nosis. Journal of the American College of Cardiology 63, with MICCAI 2018, Granada, Spain, September 16, 2018, 493{505. doi:10.1016/j.jacc.2013.10.055. Revised Selected Papers. volume 11395. Springer. Hunold, P., Vogt, F.M., Schmermund, A., Debatin, Punithakumar, K., Ayed, I.B., Islam, A., Ross, I.G., Li, S., J.F., Kerkho , G., Budde, T., Erbel, R., Ewen, K., 2010. Regional heart motion abnormality detection via Barkhausen, J., 2003. Radiation exposure during car- information measures and unscented Kalman ltering, in: diac CT: e ective doses at multi{detector row CT and International Conference on Medical Image Computing electron-beam CT. Radiology 226, 145{152. and Computer-Assisted Intervention, Springer. pp. 409{ Julier, S., Uhlmann, J., 2004. Unscented ltering and non- linear estimation 92, 401{422. Radau, P., Lu, Y., Connelly, K., Paul, G., Dick, A., Wright, Kaus, M.R., Von Berg, J., Weese, J., Niessen, W., Pekar, G., 2009. Evaluation framework for algorithms segment- V., 2004. Automated segmentation of the left ventricle in ing short axis cardiac mri. The MIDAS Journal-Cardiac cardiac mri. Medical image analysis 8, 245{254. MR Left Ventricle Segmentation Challenge 49. Kowallick, J.T., Kutty, S., Edelmann, F., Chiribiri, A., Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: Con- Villa, A., Steinmetz, M., Sohns, J.M., Staab, W., Bet- volutional networks for biomedical image segmentation, tencourt, N., Unterberg-Buchwald, C., Hasenfu, G., in: International Conference on Medical image comput- Lotz, J., Schuster, A., 2014. Quanti cation of left atrial ing and computer-assisted intervention, Springer. pp. 234{ strain and strain rate using Cardiovascular Magnetic Res- onance myocardial feature tracking: A feasibility study. Santos, A.B.S., Kraigher-Krainer, E., Gupta, D.K., Zile, Journal of Cardiovascular Magnetic Resonance 16, 60. M.R., Pieske, B., Voors, A.A., Lefkowitz, M.P., Packer, doi:10.1186/s12968-014-0060-6. M., Mcmurray, J.J., Shah, A.M., Solomon, S.D., 2014. Im- Labbe, R., 2014. Kalman and bayesian lters in python. paired left atrial function in heart failure with preserved Liu, S., Guan, Z., Zheng, X., Meng, P., Wang, Y., Li, Y., ejection fraction: A speckle tracking echocardiography Zhang, Y., Yang, J., Jia, D., Ma, C., 2018. Impaired left study. European Journal of Heart Failure 16, 1096{1103. 17 Santos, A.B.S., Roca, G.Q., Claggett, B., Sweitzer, N.K., Zhang, X., Martin, D.G., Noga, M., Punithakumar, K., Shah, S.J., Anand, I.S., Fang, J.C., Zile, M.R., Pitt, B., 2018. Fully automated left atrial segmentation from MR Solomon, S.D., Shah, A.M., 2016. Prognostic relevance image sequences using deep convolutional neural network of left atrial dysfunction in heart failure with preserved and unscented Kalman lter, in: IEEE International Con- ejection fraction. Circulation: Heart Failure 9, 1{11. ference on Bioinformatics and Biomedicine, pp. 2316{ Tobon-Gomez, C., Geers, A.J., Peters, J., Weese, J., Pinto, 2323. K., Karim, R., Ammar, M., Daoudi, A., Margeta, J., San- Zheng, Y., Barbu, A., Georgescu, B., Scheuering, M., Co- doval, Z., et al., 2015. Benchmark for algorithms segment- maniciu, D., 2008. Four-chamber heart modeling and ing the left atrium from 3D CT and MRI datasets. IEEE automatic segmentation for 3-d cardiac ct volumes us- transactions on medical imaging 34, 1460{1473. ing marginal space learning and steerable features. IEEE Von Roeder, M., Rommel, K.P., Kowallick, J.T., Blazek, S., transactions on medical imaging 27, 1668{1681. Besler, C., Fengler, K., Lotz, J., Hasenfu, G., Luc  ke, C., Zhu, L., Gao, Y., Yezzi, A., MacLeod, R., Cates, J., Tan- Gutberlet, M., Schuler, G., Schuster, A., Lurz, P., 2017. nenbaum, A., 2012. Automatic segmentation of the left In uence of Left Atrial Function on Exercise Capacity and atrium from MRI images using salient feature and con- Left Ventricular Function in Patients with Heart Failure tour evolution, in: Engineering in Medicine and Biology and Preserved Ejection Fraction. Circulation: Cardiovas- Society (EMBC), 2012 Annual International Conference cular Imaging 10, 1{9. of the IEEE, IEEE. pp. 3211{3214. Xia, Q., Yao, Y., Hu, Z., Hao, A., 2019. Automatic Zreik, M., Leiner, T., de Vos, B.D., van Hamersvelt, R.W., 3D Atrial Segmentation from GE-MRIs Using Volumet- Viergever, M.A., I sgum, I., 2016. Automatic segmenta- ric Fully Convolutional Networks, in: Lecture Notes in tion of the left ventricle in cardiac CT angiography us- Computer Science, Statistical Atlases and Computational ing convolutional neural networks, in: Biomedical Imag- Models of the Heart Workshop on Atrial Segmentation ing (ISBI), 2016 IEEE 13th International Symposium on, and LV Quanti cation Challenges. Springer International IEEE. pp. 40{43. Publishing. volume 11395, pp. 211{220.

Journal

Electrical Engineering and Systems SciencearXiv (Cornell University)

Published: Sep 28, 2020

There are no references for this article.