uRP: An integrated research platform for one-stop analysis of medical images

1Medical imaging is widely employed in clinical research to investigate effects on diagnosis, staging, treatment planning, and follow-up evaluations. Medical imaging contains multiple imaging sequences or modalities, such as magnetic resonance imaging (MRI), computed tomography (CT), and positron emission tomography (PET), providing complementary information. The processing and quantitative analysis of medical images ensure their clinical utility in a variety of medical applications, from general research to clinical workflows.

2Most recently, machine learning- and deep learning-based intelligent imaging analyses have shown enormous advantages in providing consistent and accurate image quantifications in multiple applications, including image segmentation, registration, classification, etc.. A series of algorithm architectures and strategies have been developed to meet different requirements. For example, U-Net, V-Net, and nnU-Net exhibit accurate segmentation performance; affine models (i.e., FLIRT, A-SIFT) and deformable models [i.e., FNIRT, ANTS, VoxelMorph, Dual-PRNet, LDDMM] assist to image registration; ResNet, DenseNet and their variants have attracted much attention in classification tasks. Also, varied attention mechanisms and loss functions have been utilized to optimize the deep learning network and improve its robustness. The accurate analysis of medical images accelerates the development and upgrading of intelligent algorithms that can be integrated into the software to enable easy-to-use clinical research.

3Numerous choices of medical image analysis tools integrating advanced algorithms are available. For example, MATLAB, Python (https://www.python.org/), 3D Slicer (https://www.slicer.org/), and Mimics (Materialize, Leuven, Belgium) allow general image processing, while FreeSurfer (https://surfer.nmr.mgh.harvard.edu/), chest imaging platform (CIP, https://chestimagingplatform.org/), and OpenSim are proprietarily applied to the brain, lung, knee joint analyses, respectively. Meanwhile, a number of software is dedicated to a specific modality, such as resting-state fMRI data analysis toolkit (REST) and statistical parametric mapping (SPM, UCL Queen Square Institute of Neurology, London, UK) designed for functional MR images; DtiStudio and medical imaging interaction toolkit (MITK) applied to diffusion images; SenseCare provides a range of artificial intelligence (AI) toolkits for specific clinical scenarios such as lung cancer diagnosis and radiotherapy planning. Overall, the software greatly simplifies image processing and makes it easy for clinicians to understand and use.

4However, users still face a series of challenges in using the software to achieve one-stop image analysis. First, complex image analysis requires introduction of multiple software to adapt to the respective modalities and organs, which makes it difficult to integrate information from different modalities organically. Second, different software relies on specific environments (e.g., Windows and Linux) and programming languages (e.g., Python, C++, and R), requiring extensive computer knowledge to be used in practice. Third, the feasibility of integrating the latest AI models into the software to iteratively optimize performance is yet to be assessed. Finally, manual contouring regions of interest (ROIs) is always required in scientific research to serve as the gold standard or to extract quantitative metrics, which is time-consuming and may suffer from low reproducibility and consistency due to intra- and inter-observer variability. Therefore, it is desired to design an integrated platform for one-stop analysis of medical images, which needs to be compatible, advanced, easy to use, extensible, and reproducible. In addition, the ideal platform should offer cloud-based services (public or private) to reduce configuration requirements on the user end and allow multiple clients to work simultaneously. A dedicated data management module is also essential for organizing multiple clinical projects and massive medical data, which allows the integration of large-scale data from multiple centers to develop robust algorithms and facilitate collaborative research.

5In line with the trend, we propose a multifunctional platform, called uRP (uAI research portal, https://www.uii-ai.com/en/uai/scientific-research), to perform accurate image processing and analysis on demand. The uRP can satisfy the following requirements: (1) Integrating a variety of algorithms with respect to the image's modality (i.e., MRI, CT, PET), body part (i.e., head, chest, abdomen, pelvis), and processing task (i.e., segmentation, registration, classification), to be suitable for diverse applications; (2) Offering friendly interactive user interface (UI) to make clinicians easy to understand and independently implement complete AI-related research; (3) Possessing the extendable capability to enrich existing modules and ensure reusable and reproducible analysis across clinicians, even hospitals; (4) Achieving automatic or semi-automatic image processing (e.g., delineation) to ensure efficient and accurate analysis; (5) Supporting cloud-based computing services with high concurrency and owing a dedicated data management module.

6In the below sections, we present an overview of uRP's architecture and major functional modules, including semi-automatic delineation, deep learning-based image segmentation, registration, and classification, as well as radiomics and statistics. The clinical utility of the uRP is demonstrated by three domain analyses of the brain, pneumonia, and knee joint. Representative use cases are exampled to illustrate some of the outcomes that clinicians have achieved by using the uRP. In addition, the modules of the uRP will continue to be developed and extended, and we prospect future design concepts and directions to achieve a more intelligent platform for scientific research and even clinical uses.

7In this article, we propose the uRP to realize the one-stop medical image analysis. The architecture and main modules are shown in the following parts.

8From 2018, we began to build the uRP to promote one-stop advanced medical image analysis in the context of integrating AI modules. It is intended to facilitate scientific research for clinicians and is therefore designed as flexible modules that can be used directly, combined, or customized for specific application domains. Here, we start by describing its architecture and key components (Figure 1).

9The design of uRP architecture takes a modular and layered approach. Internally, the software consists of three layers: (1) The lower level is composed of hardware drivers, such as graphics processing unit (GPU) accelerated using NVIDIA CUDA, and cloud servers, such as Amazon web services (AWS), that efficiently use graphics resources of the host system; (2) At the middle level, there is application programming interface (API), primarily Python and C++, contributing a range of algorithms (e.g., segmentation, registration, classification) and providing higher-level functionality and abstractions. Additionally, a variety of open-source libraries are embedded in the uRP, where the DICOM toolkit (DCMTK) is used to support DICOM format data, and Qt to provide a cross-platform graphical user interface (GUI) framework; (3) The higher level presents UIs of multiple algorithms and builds blocks to the end users for domain-specific analysis (Figure 1). All computational demanding modules can run in parallel on many processors at once, using message passing between processes and/or shared-memory threads.

10Since its inception, uRP has been evolving with major architecture, UIs, and functional redesigns. The uRP's version is updated every 2 months, and each release is formally tested on a variety of platform configurations to ensure its stability.

11uRP is a web-based platform supporting GPU cloud computing. It supports plug-ins to deliver task-specific functionality to the user. uRP mainly consists of three functional modules, (1) data management, supporting the upload of imaging data and non-imaging features, extraction of subject information from DICOM tags, and image search based on specific criteria; (2) image processing, including semi-automatic delineation, deep learning-based segmentation, registration, and classification; (3) radiomics and statistics, for classification and regression tasks. Images in DICOM or NIFIT formats from clinical picture archiving and communication system (PACS) or external drives are supported with various imaging modalities, including MR, CT, PET, x-ray, and digital radiography (DR) images. These technical modules can support a variety of applications on demand (Figure 1).

12Delineation of ROIs is essential in clinical research as the primary step for quantification and feature extraction. Manual delineation is generally considered as the gold standard, but it is limited by its tedious, time-consuming, and error-prone characteristics, and thus difficult to achieve high-quality and efficient annotation especially with the massive amount of medical images. In view of this, various automatic delineation algorithms have been developed, including fully automatic and semi-automatic methods. It is worth noting that fully-automatic delineation is convenient but sometimes hard to reach the desired accuracy, while semi-automatic delineation allows human-computer interaction and thus can optimize the results, which is a time-saving alternative to manual delineation. Our proposed uRP platform contains a collection of tools for fully-automatic, semi-automatic, and manual delineation. As shown in Figure 2, a smart annotation tool and a ROI modification tool are integrated into the uRP to assist the delineation process.

13The uRP offers several smart annotation tools (SATs) for medical images, including (1) intelligent interactive segmentation and (2) annotation propagation, which enables fast extraction of the target from the complex background and 3D propagation of the annotation.

14Intelligent interactive segmentation is a technique that allows the user to adjust the region of interest (ROI) by manipulating seeds. In use, the user firstly draws a rectangle to cover the ROI, where the Canny edge detection algorithm is then performed within the rectangle to generate the target boundary. After that a positive seed is generated at the centroid, and 4 negative seeds are placed at the vertices of the user-defined rectangle, serving as the control points for the boundary (Figure 2A, Supplementary Video 1). We can then update the shape and size of the generated boundary by adjusting the positive / negative seeds. Also, more positive seeds (green) can be added to enlarge the ROI by clicking on the area outside the edge, and negative seeds (red) can be added in the ROI to remove specific regions. Canny's approach is a widely used edge detection method with tweakable parameters and thus suitable to be integrated with a GUI. The algorithm can be divided into the following 5 steps: (1) Use a Gaussian filter to smooth the image and reduce the noise; (2) Calculate the gradient intensity and direction for each pixel point in the image; (3) Edge candidates are identified by applying the non-maximal or critical suppression to the gradient magnitude; (4) Apply double-threshold detection to determine real and potential edges; and (5) Finalize the edge detection by suppressing isolated weak edges. The implementation for the Canny edge detection algorithm could be found in the OpenCV library.

15Another tool is annotation propagation, which applies the current annotation in one slice to its adjacent slices. Specifically, the current annotation serves as the initial mask, and the tool can automatically propagate annotations across entire image frames (Supplementary Video 2). Especially, the user can optionally keep the correct region when the ROI is propagated to other slices and divided into multiple regions. Users can save annotations for following radiomics and deep learning analyses on the uRP, or download them for future review.

16After labeling the ROI, a variety of specific preprocessing needs to be performed on the mask to meet diverse image analyses. For example, studying tumor microenvironment requires obtaining the peritumor region, whereas studying liver fat requires extracting the ROI by intensity. The uRP offers a morphological modification tool to dilate or erode the selected ROI according to user-defined distances in the x, y, and z directions. For existing ROIs, separation can be performed by user-defined split intervals, connectivity, or minimum size (voxels). It can also merge, intersect, and complement multiple ROIs by performing linear operations on the pixels at each position of the ROI (Figure 2B). When conducting multimodal research, ROI can be duplicated across modalities. The above ROI modification tools can be applied to 2D and 3D ROI, and users can expand research directions with a variety of ROI preprocessing tools available on the platform.

17With the advancement of deep learning and its wide application in medical image processing, this technology has shown great potential in organ segmentation, disease diagnosis, etc. Some powerful online trainable deep learning modules are available on the uRP, such as the segmentation module, registration module, and classification module. The network architecture of each module is shown in Figure 3. The segmentation module can segment various organs of the whole body using a cascade coarse-to-fine framework (Figure 3A); the registration module performs unsupervised registration from a moving image to a reference image (Figure 3B), and the classification module can focus on ROI to classify the input image (Figure 3C). These deep learning modules are directly invoked through a simple parameter configuration, which can be flexibly applied to various research scenarios.

18The input of the segmentation module can be single- or multimodal 2D or 3D data, where models can be trained to automatically delineate ROIs, such as organs or tumors. The uRP integrates a segmentation toolkit named VB-Net. Briefly, we use a V-Net as the backbone for the segmentation task, which consists of a compression path, an expansion path, and skip connections. The compression path extracts high-level context information and the expansion path upsamples the signal to recover its original size, where skip connections allow the extracted high-level context information to be fused with fine-grained local information. To reduce model parameters and GPU memory cost, bottleneck layers are added in the down block and up block of the network (Figure 3A). Moreover, a variety of optimization strategies are embedded in the network to improve segmentation performance and extend application scenarios, as described below:

19(1) Adaptive input module, which adds convolutional layers to large-size images to ensure the network adapts to various input images; (2) A cascade coarse-to-fine strategy, in which the coarse-resolution model aims to localize ROIs in the original image by leveraging the global 3D context, and the fine-resolution model focuses on refining detailed boundaries of ROIs (Figure 3A); (3) Self-attention mechanism, which accelerates network convergence and improves segmentation accuracy; (4) Various loss functions, such as Dice loss, focal loss, and boundary loss, which can effectively constrain the segmented targets in different tasks.

20Scientific research users can invoke the deep learning segmentation module directly through the configuration file, requiring no coding skills in the process (Supplementary Video 3). The configuration file provides the following functions: (1) setting paths of training data, model storage, and output results; (2) hyperparameters during the training process (e.g., GPU selection, batch size, training epochs); (3) data augmentation (e.g., flipping, rotating, scaling), (4) data preprocessing (e.g., sampling method, cropped size, padding type, normalization); (5) configurations of the loss function, optimizer, and learning rate scheduler; (6) segmentation evaluation metrics (i.e., Dice, average symmetrical surface distance, Hausdorff distance); (7) configurations of networks and strategies, e.g., cascading of different networks and selection of attention mechanisms. The configuration file should be configured in a standard format (a template is provided on the uRP) and called in the training process.

21Moreover, the segmentation module integrates more than 100 high-precision organ segmentation models throughout the body. Segmentation of organs at risk using the uRP's segmentation module has shown great advantages in terms of speed (0.7 s vs. 20 s per organ), accuracy (average Dice score 96.6% vs. 84.3%), and robustness (successful rate 98.6% vs. 83.3%) compared to conventional methods.

22Image registration is to align a moving image to a reference image, which is a critical procedure in the analyses of multimodal images and longitudinal data. The uRP provides both traditional image registration (i.e., rigid, affine transformation) and deep learning-based nonlinear registration. Briefly, the nonlinear registration model consists of a registration network, spatial transform block, and hybrid loss calculation module. A hybrid loss is calculated to strengthen the alignment constraints of different structures, which combines image dissimilarity, deformation regularization, and segmentation dissimilarity with different weights (Figure 3B). The available image dissimilarity metrics include mean square difference (MSD), normalized correlation (NC), mutual information (MI), etc. Image registration can be performed between the images of the same modality or of different modalities; for example, MI is generally selected as the image dissimilarity for cross-modal registration.

23Notably, the segmentation results obtained automatically from the segmentation module could also be used in the registration process. In previous studies, organ segmentations served as soft constraint in the loss function to provide auxiliary information in the training of the registration model. Compared to using only the intensity image for registration, these studies found that the large deformation can be more readily estimated with the help of the segmentation result.

24The uRP integrates a classification module and can be used for two-class and multi-class tasks. ResNet is used as the classification backbone and optimized by several strategies: (1) an online attention module as CAM and Grad-CAM, that ensures the network to focus on ROIs and increases the model interpretability (Figure 3C); (2) a balanced sampling mechanism, which can alleviate the imbalanced distribution of the input data; (3) different sampling methods for inputs; (4) various loss functions, e.g., focal loss, Ap loss, and CAM loss. A lot of classic and mature classification networks such as DenseNet and EfficientNet have been embedded in the uRP. Moreover, the classification module can be flexibly invoked via a configuration file, similar to the segmentation module.

25Radiomics is a quantitative image analysis technique through extracting quantitative features from medical images, that aims to link large-scale data mining of images with clinical and biological endpoints. The uRP implements a one-stop analytic pipeline of radiomics, providing clinical researchers with a simple UI for image visualization, image processing, feature analysis, model construction and evaluation, and statistical analysis (Figure 4).

26Previous studies have shown that radiomics features are sensitive to variations in gray level, pixel size, and slice thickness of images. However, it is difficult to standardize parameters during image acquisition for all patients in a clinical setting. The platform supports a variety of normalization algorithms to normalize image signal intensity, such as mean_std, max_min, and center_width. And it resamples the image and ROI mask to specified pixel spacing to achieve a standardized variable pixel size and slice thickness by resampling algorithms, such as the nearest neighbor, linear interpolation, and B-spline interpolation. A pre-defined bin number of 64 is used for all analyses. In image processing and feature calculation, we follow the guidelines of the imaging biomarker standardization initiative (IBSI).

27The uRP supports three data grouping methods, namely customized grouping, random grouping according to proportion, and cross-validation grouping. The training set is used for feature selection and model construction, and the model performance is evaluated on the testing set.

28After preprocessing the images, a total of 2,264 radiomics features can be automatically extracted from each ROI (Supplementary Table S1). The first-order statistics include 18 features that reflect the quantitative depiction of the distribution of voxel intensity in medical images. The shape-based features include 14 features that reflect the shape and size of a region. The textural features include 21 gray level co-occurrence matrix (GLCM) features, 16 gray level run length matrix (GLRLM) features, 16 gray level size zone matrix (GLSZM) features, 5 neighboring gray-tone difference matrix (NGTDM) features, and 14 gray levels dependent matrix (GLDM) features, which quantify regional heterogeneity differences. Additionally, the derived images are obtained by applying 24 filters (box mean, additive Gaussian noise, binomial blur, curvature flow, box-sigma, normalize, Laplacian sharpening, discrete Gaussian, mean, speckle noise, recursive Gaussian, shot noise, LoG (sigma: 0.5, 1, 1.5, 2), and wavelets (LLL, LLH, LHL, LHH, HLL, HLH, HHL, HHH)), and are used to extract first-order statistics and textural features (2,160 derived features). Most features defined in the uRP conform to feature definitions described in the IBSI.

29To ensure the clinical utility of the model, features beyond radiomics should also be considered to improve the model's generalizability, such as demographic information and biological data. Considering that radiomics features and clinical features have different ranges, feature standardization algorithms are also provided in the uRP, such as z-score_scaler, min_max_scaler, quantitle_transformer, yeojohnson_transformer, boxcox_transformer, L1_normalization, L2_normalization, and max_abs_scaler.

30The feature selection module is used for feature selection or dimension reduction to improve model performance. It includes variance thresholding for removing low variance features, SelectKBest for removing high p-value features, as well as least absolute shrinkage and selection operator (LASSO) for sparse feature selection. It also includes factor analysis, independent component analysis (ICA), linear discriminant analysis (LDA), principal component analysis (PCA), and more than 10 algorithms for dimension reduction to get fewer new features formed by original features.

31Based on the selected features, various machine learning-based models can be constructed for classification or regression tasks. Our proposed uRP integrates 13 machine learning algorithms, including adaptive boosting (AdaBoost), bagging decision tree, decision tree, Gaussian process, gradient boosting decision tree (GBDT), K-nearest neighbors (KNN), random forest (RF), logistic regression (LR), extreme gradient boosting (XGBoost), stochastic gradient descent (SGD), support vector machine (SVM), quadratic discriminant analysis (QDA), partial least squares-discriminant analysis (PLS-DA), and allows for hyperparameters adjustment. The nomogram model is also included in this module, which combines radiomics and clinical factors to facilitate the clinical utility of predictive models. It is necessary to first calculate the score of each predictive variable, obtain the total point, and then find the probability of the disease outcome corresponding to the total score.

32To evaluate model performance, the uRP provides two sets of quantitative metrics for classification and regression models.

33(1) Classification model: The platform can automatically generate receiver operating characteristic (ROC) curves and calibration curves in the training and validation cohorts, and calculate multiple metrics, such as the area under the ROC curve (AUC, with 95% confidence interval), F1 score, precision, sensitivity, specificity, and accuracy. It also provides a visual representation of the confusion matrix and supports the comparison of multiple models. In addition, the histogram, box chart, violin chart, correlation analysis heatmap, and clustering analysis heatmap are optionally plotted to establish the relationship between features. For clinical applications, the decision curve and clinical impact curve can be plotted to assess the clinical usefulness of models by quantifying net benefits at different risk thresholds.

34(2) Regression model: The platform automatically generates prediction curves and scatter plots to visualize the results, and calculates mean absolute error (MAE), mean absolute error (MSE), R-squared, and Pearson correlation coefficient for regression model evaluation.

35To illustrate the clinical utility of the uRP, we list three domain analyses (i.e., brain, pneumonia, and knee joint) and representative use cases.

36It is worth noting that uRP can be extended and applied to a variety of designated applications (Figure 1), e.g., brain structural analysis, pneumonia analysis, and knee joint analysis. In this section, we focus on these three specific scenarios to describe the versatility and scalability of uRP, covering segmentation, quantitative analysis of ROI, classification for disease prediction, and prognosis, where image analysis modules are integrated into sequential to form an automatic analysis pipeline.

37Neuroimaging shows tremendous potential in the early diagnosis of neurodegenerative diseases, in which structural imaging (MRI) serves as a foundation to provide brain tissue and parcellation information (Figure 5).

38First, uRP can handle high-resolution MR images and hierarchically segment brain structures (Figure 5A). The workflow mainly involves: (1) the bias field correction, (2) removal of the skull, (3) tissue segmentation of white matter, gray matter, and cerebrospinal fluid (CSF), (4) bilateral segmentation, and (5) parcellation of 109 sub-regions. A total of 109 sub-regions includes 22 temporal lobe structures, 20 frontal lobe structures, 12 parietal lobe structures, 8 occipital lobe structures, 8 cingulate gyrus structures, 2 insular structures, 12 subcortical gray matter structures, cerebral white matter structures, ventricles, cerebellum, and other structures. To emphasize, this segmentation process depends on VB-Net, achieving efficient, precise, and end-to-end segmentation of multiple sub-regions. The model was trained on T1 images of 1,800 subjects and tested on 295 subjects with an average Dice of 0.92, where the images were acquired from the Consortium for Reliability and Reproducibility (CoRR) dataset and Chinese brain molecular and functional mapping (CBMFM) project. Based on the segmentation results, the volume, volume ratio of each sub-region, and the asymmetry index of paired sub-region are calculated quantitatively and compared to the relevant parameters from the gender- and age-matched normal dataset (Figure 5B). Abnormal brain sub-regions are identified, where those below the 5th percentile of the normal range are considered likely to be abnormally atrophic, and those above the 95th percentile are considered abnormally enlarged (Figure 5C). On the other hand, the medial temporal lobe atrophy (MTA) score, also known as Schelten's scale, is developed with an AI model for automated assessment of the hippocampus atrophy status. The score ranges from 0 to 4, in which the higher the score, the more severe the hippocampal atrophy (Figure 5B). It should be noted that the segmentation and computation of the 109 sub-regions take less than 1 min.

39Based on this, uRP holds enormous ability in the follow-up data analysis, i.e., (1) comparing volume changes of ROIs over time to explore pathological progression of neurodegenerative diseases; (2) constructing AI models to predict the odds of other diseases such as Parkinson's disease (PD) or mental diseases for early diagnosis and early intervention; (3) promoting the brain functional analysis with established structural ROIs, as well as fiber connectivity analysis in diffusion tensor imaging (DTI) (Figure 5C).

40With the worldwide spread of coronavirus disease (COVID-19), the early diagnosis and prognostic analysis of pneumonia have become an urgent need, which inspires a large number of related researches to serve the clinic. CT has been popularly used to monitor pneumonia's progression and measure the disease severity. Based on chest CT scans, three important issues need to be explored: (1) the location of the pneumonia infection, (2) the severity of the infection, and (3) the etiology of the disease. uRP meets these requirements by integrating segmentation, classification, and registration algorithms (Figure 6A).

41The first step is to locate the infected lesions. First, the whole lung is obtained by embedded VB-Net, followed by bilateral segmentation. Then, the left lung is segmented into 2 lung lobes (superior and inferior lobes), while the right lung is segmented into 3 lung lobes (superior, middle, and inferior lobes). Afterward, 5 lung lobes are then finely segmented into 18 bronchopulmonary segments (Figure 6B). Infected lesions are also auto-contoured in this process, and can be visualized from the UI. Noted that the human-in-the-loop strategy is designed to iteratively update VB-Net to address the problem of limited annotated data. To be specific, an initial segmentation model based on a small amount of delineated data is applied to the new data, and segmentation results are manually corrected and then fed into the model, so that a more robust model will be trained through 3∼4 iterations, greatly improving the efficiency of delineation.

42Following the segmentation, a diverse set of handcrafted features are calculated to quantitatively assess the severity of the pneumonia infection, including (1) 26 volumetric features—the volume and percentage of infections in each lobe and pulmonary segment, (2) 31 numeric features—the number of infected lobes and pulmonary segments, (3) 32 histogram features—the histogram distribution of CT intensity, (4) 7 surface features—the surface area of infections and lung boundary. A total of 96 location-specific features are displayed in the UI to reflect the severity of pneumonia infection. In addition, follow-up data can be registered with previously acquired images to extract changes in infection-specific features to monitor the progression of pneumonia and to accurately determine the severity (Figure 6C).

43Importantly, uRP can also distinguish different types of pneumonia and predict possible pneumonia causes via uRP's built-in classification algorithms. Based on the segmented masks and extracted features, dual-sampling attention 3D ResNet is used to diagnose COVID-19 from community acquired pneumonia (CAP). Moreover, masks, handcrafted features, as well as radiomics features can be used to classify the cause of pneumonia (e.g., viruses, fungi, and bacteria), and to report the corresponding probabilities.

44Therefore, uRP-based pneumonia analyses involve automatic segmentation of infected lesions, extraction and visualization of quantitative metrics, and classification of different types of pneumonia, which largely accelerates scientific research on pneumonia.

45Knee osteoarthritis (OA), known as a degenerative joint disease, results from the wear, tear, and progressive loss of articular cartilage, which may eventually lead to disability. The severity staging of knee OA should be carefully taken into consideration for the treatment, which relies on a range of morphological parameters, including the volume and thickness of articular cartilage, and minimal joint space width (mJSW). To benefit clinical practice, uRP implements a complete analysis pipeline that automatically segments knee tissues and calculates morphological metrics.

46At first, a cascade coarse-to-fine strategy is applied to obtain fast and accurate segmentation results (Figure 3A). Through two 3D-VNet segmentation models, multiple knee joint tissues are accurately segmented, including bones (i.e., femoral bone, tibial bone, patella bone), cartilages (i.e., femoral cartilage, tibial cartilage, patella cartilage), as well as menisci, and visualized in the UI of uRP (Figure 7A).

47At the same time, a series of morphological parameters are automatically calculated from corresponding segmented masks, including (1) volumes of cartilages, menisci, and cartilage damage, (2) the mean thicknesses of cartilages, (3) the medial and lateral mJSWs, calculated with the minimal Euclidean distance between the femoral and tibial surfaces, (4) the severity assessment—the ratio of the tibial cartilage covered by menisci, the grade of cartilage damage and corresponding probability (Figure 7B). Severity grading is also performed by a classification algorithm (Figure 7C). All the above results can be viewed from the uRP, which is helpful for clinicians to make a quick diagnosis of knee OA.

48Overall, the knee joint analysis module on uRP can handle MRI images in a fully automatic manner, in which knee joint tissues are segmented and key features (i.e., volume, thickness, mJSW) are calculated to identify the severity of knee OA, thus guiding the optimal treatment.

49To illustrate the utility of the uRP, we example several use cases and organize them into the following four parts:

50The automatic segmentation module on the uRP has been used in many medical scenarios, for example, to diagnose COVID-19 infections. Chest CT scans of 549 patients were collected from Shanghai Public Health Clinical Center and several other hospitals, and were automatically segmented and quantified the infected regions throughout the lungs. Researchers used the pneumonia analysis module of the uRP, yielding a Dice similarity coefficient of 91.6% ± 10.0% for the segmentation of infected areas, and a mean estimation error of the infected percentage of 0.3% for the whole lung on the validation dataset (300 patients). Besides, to predict the severity of COVID-19 patients, quantitative features of 5 lung lobes and 18 bronchopulmonary segments were calculated and used to construct a classification model based on the SVM algorithm. The best accuracy of severity prediction was 73.4% ± 1.3%, which demonstrated that uRP's pneumonia analysis module owned good performance on patient severity prediction.

51The image classification module built into the uRP has been experimentally explored in various applications such as disease diagnosis, risk classification, and treatment selection. Gastrointestinal stromal tumors (GISTs) are mesenchymal neoplasms with variable malignant potentials. In clinics, accurate preoperative risk classification is important for surgical resection and adjuvant treatment. Researchers from Shandong Provincial Hospital and the Affiliated Hospital of Qingdao University collected contrast-enhanced CT images and clinicopathological characteristics from 733 patients and the goal was to develop a model for predicting the GISTs risk stratification. A deep learning model with an attention mechanism was constructed on the uRP's classification module to divide patients into three categories (i.e., low-malignant, intermediate-malignant, and high-malignant). The obtained AUCs were 0.90, 0.80, and 0.89 on the testing sets for low-malignant, intermediate-malignant, and high-malignant GISTs, respectively. Therefore, this multi-center study demonstrated that the quantitative CT and deep learning-based approach can be an objective means of predicting the risk stratification of GISTs.

52Image registration is widely needed for the analyses of multimodal images and longitudinal data. For example, precise registration of dynamic contrast enhanced MR images (DCE-MEIs) with pre-contrast images can be used to obtain accurate subtraction images, which helps to better differentiate the true enhancement of residual viable tumors from coagulative necrosis. In our recent study, Qian ­et al. collected 3D liver DCE-MRI series from 97 patients, with each series including pre-contrast T1-weighted, post-contrast T1-weighted scan at the arterial phase, and post-contrast T1-weighted scan at the portal venous phase. To overcome the intensity enhancement (due to the contrast agent) and spatial distortions of the liver, the cascade registration framework integrated into uRP was used to register the post-contrast images to the pre-contrast images. The registration performance of the proposed method was compared with the traditional registration method SyN in the ANTs toolkit, and the results demonstrated that the proposed framework embedded in the registration module of uRP owned a comparable performance and significantly improved efficiency.

53The radiomics module is particularly suitable for image-based classification and regression tasks, deployed and used well in multiple centers. Radiomics analysis was conducted in the Fourth Affiliated Hospital of Harbin Medical University to discriminate acute myocardial infarction from unstable angina. A total of 210 patients with coronary computed tomography angiography (CCTA) images were retrospectively collected and randomly divided into the training and validation cohorts. Following the workflow of radiomics analysis in uRP, three vessel-based pericoronary adipose tissue (PCAT) radiomics features and fat attenuation index (FAI) were extracted from CCTA images, and then selected features were used to construct the classification model. Results demonstrated that the combined model achieved superior performance with AUC values of 0.97 and 0.95 for training and validation cohorts, respectively. The Affiliated Hospital of Southwest Medical University applied a radiomics model to predict the T stage, perineural invasion, and microvascular invasion of extrahepatic cholangiocarcinoma (CCA). This retrospective trial included 101 CCA patients scanned with four MR images, including T1-weighted imaging (T1WI), T2-weighted imaging (T2WI), diffusion-weighted imaging (DWI), and apparent diffusion coefficient (ADC) map. Radiomics features were extracted from four MR images, followed by dimension reduction, and selected features were used to construct three classification models corresponding to the three tasks. The AUC values of models in the testing cohort for predicting T stage, perineural invasion, and microvascular invasion were 0.962, 1.000, and 1.000, respectively. Similarly, researchers from Zhongshan Hospital constructed a multi-parametric radiomics nomogram for predicting the microvascular invasion (MVI) based on multiple MR sequences from 130 patients pathologically confirmed with intrahepatic CCA. The nomogram incorporating tumor size, intrahepatic duct dilatation, and the radiomics model, achieved good prediction performance with AUC values of 0.953, 0.861, and 0.819 in the training, validation (n = 33), and time-independent testing cohorts (n = 24), respectively.

54Over the years, uRP has gained broad acceptance within the medical image analysis community, which can be attributed to its breadth of functionality, extensibility, and cross-platform portability.

55One key strength of uRP is the modularization for customized extensibility, where plug-ins can be designed for specific purposes, and can be freely combined to accomplish complex analyses, suitable for a variety of scenarios. uRP's extensible structure means that new functionality can be integrated on top of the existing platform, rather than being created from scratch, showing significant advantages over monolithic software. The modular design benefits users in several ways: Reusability: uRP has integrated multiple algorithms that are embedded into axiomatic building blocks and are invoked for specific analysis workflows. An algorithm may participate in many tasks and perform a similar function, meaning that advanced techniques can be reused in new research areas. In addition to existing algorithms, new algorithms can also be developed and integrated into the uRP, for example, transfer learning, transformer networks, etc., to meet clinical needs. Besides, uRP can be extended with third-party software (i.e., 3D Slicer) to import historical annotation data or clinical information.Reproducibility: The design of plug-ins of uRP follows the criteria of standardization and interoperability, which can be easily shared among research groups, thus minimizing the need for duplication and facilitating reproducibility and consensus building. uRP collects detailed parameters set by the user and generates a report summarizing quantitative metrics and results so that a study can be replicated by different researchers or even different institutions. Equally important, uRP can be used for data management and fair data repositories are essential for reproducible research. Meanwhile, uRP owns reproducibility at scale, e.g., producing high-dimensional radiomics features for ROI in each image or applying the same quantitative analysis in high-throughput images.Community: uRP has an active community of more than 50 hospitals in China. The feedbacks provided by users fuel improvements of uRP, especially the development of innovative algorithms for clinical needs.

56Reusability: uRP has integrated multiple algorithms that are embedded into axiomatic building blocks and are invoked for specific analysis workflows. An algorithm may participate in many tasks and perform a similar function, meaning that advanced techniques can be reused in new research areas. In addition to existing algorithms, new algorithms can also be developed and integrated into the uRP, for example, transfer learning, transformer networks, etc., to meet clinical needs. Besides, uRP can be extended with third-party software (i.e., 3D Slicer) to import historical annotation data or clinical information.

57Reproducibility: The design of plug-ins of uRP follows the criteria of standardization and interoperability, which can be easily shared among research groups, thus minimizing the need for duplication and facilitating reproducibility and consensus building. uRP collects detailed parameters set by the user and generates a report summarizing quantitative metrics and results so that a study can be replicated by different researchers or even different institutions. Equally important, uRP can be used for data management and fair data repositories are essential for reproducible research. Meanwhile, uRP owns reproducibility at scale, e.g., producing high-dimensional radiomics features for ROI in each image or applying the same quantitative analysis in high-throughput images.

58Community: uRP has an active community of more than 50 hospitals in China. The feedbacks provided by users fuel improvements of uRP, especially the development of innovative algorithms for clinical needs.

59The uRP platform has shown great potential for one-stop image analyses in multiple scientific researches. There are several other platforms for image analysis, such as 3D Slicer, SenseCare, MITK. The proposed uRP has several advantages. (1) uRP can provide cloud-based services allowing for high user concurrency, a data management module to facilitate collaborations, and batch processing capability for efficient analysis. (2) Some of these platforms are for clinical diagnostic and treatment planning usages, with modules such as lung cancer diagnosis, radiotherapy planning for head and neck cancers. uRP is designed for clinical scientific research, supporting image analysis and research idea validation. (3) The uRP integrates machine learning algorithms and statistical analysis methods to provide a more powerful analytical tool for clinical research. (4) The uRP also offers smart annotation tools for medical images, to remedy the time-consuming manual annotation process that serves as a prerequisite for ROI based analysis.

60Although uRP already has many applications as a one-stop medical image analysis platform, some issues are still to be addressed. Firstly, uRP currently focuses on the analysis of radiological images, while other types of images could be future supported, such as those from pathology. Secondly, most existing applications are oriented towards the adult population, while applications specific for the fetal, infant, and children are needed. We have only two applications for now, i.e., infant brain segmentation, skeletal age prediction, and would be further increased in the future. While uRP cannot meet all the scientific needs, it is fortunately a dynamic software that evolves together with the new scientific research derived from clinical problems. In the future, we will continue to develop more tools and domain-specific methods, including algorithms, statistics, as well as radiomics, to improve the efficiency, accuracy, robustness, and generalization of one-stop analysis. Represented by ChatGPT, generative AI is a hot research topic for now. It would bring many improvements to the current scientific research, such as better integrating multi-omics data to assist clinical diagnosis and prognosis assessment. We are also exploring the possibility of integrating generative AI in the platform at suitable scenarios. We anticipate that the uRP can be applied to diverse domains covering an increasing number of analytic pipelines for diverse pathological diseases.

61In summary, uRP is a one-stop medical image analysis software for scientific research, and it not only supports versatile visualizations, but also provides advanced functionality such as automatic segmentation, registration, and classification for a variety of application domains. More specifically, it has three major merits, (1) advanced built-in algorithms (>100) applicable to multiple imaging modalities (i.e., CT, MR, PET, DR), diseases (i.e., tumor, neurodegenerative disease, pneumonia), and applications (i.e., diagnosis, treatment planning, follow-up); (2) an iterative deep learning-based training strategy for fast delineation of ROIs of large-scale datasets, thereby greatly saving clinicians' time and obtaining novel and more robust models; (3) a modular architecture with customization and extensibility, where plugins can be designed for specific purposes. As a result, it will be necessary to investigate and develop new algorithms and strategies to expand application domains and really solve clinical problems.