Serverless Architecture for Data Processing and Detecting Anomalies with the Mars Express MARSIS Instrument

Serverless computing is a cloud computing model that abstracts the underlying infrastructure, operating system, and server management from developers. In a serverless environment, the application logic is commonly implemented as a set of stateless functions that are triggered by events (e.g., API calls, message queues or scheduled tasks), and are executed by containerized or microVM based run time environments.

The main benefit of the serverless model (Baldini et al. 2017; Eivy & Weinman 2017) is that developers can focus solely on writing code and deploying their applications, without worrying about the infrastructure or scaling requirements. The cloud provider takes care of the heavy lifting of server management, capacity planning, and scaling, allowing developers to build and deploy applications more quickly and efficiently. In addition, serverless computing reduces the cost of computing by charging only for the resources consumed during the execution of the function.

Platforms implementing this serverless model are categorized as Function as a Service (FaaS). FaaS platforms enables developers to upload their code to the cloud provider, define the trigger that will activate it, and specify the resources required to run the function. When a trigger occurs, the cloud provider automatically spins up the required resources, runs the code, and then releases the resources when the function has completed execution.

Some examples of commercial FaaS platforms are Amazon AWS Lambda, ⁶ Google Cloud Functions, ⁷ and Microsoft Azure Functions. ⁸ There are also some open source serverless initiatives, such as Apache OpenWhisk ⁹ (Quevedo et al. 2019), OpenLambda ¹⁰ (Hendrickson et al. 2016), and Knative. ¹¹

Serverless computing or FaaS platforms are suitable for the deployment and execution of scientific workloads (John et al. 2019; Burkat et al. 2021; Eismann et al. 2020; Malawski et al. 2020) for two main reasons. On the first hand, serverless computing has revealed itself as a valuable paradigm for complex high-throughput applications environments that demand optimal resource provision (Raman et al. 1998). This is because dynamic load peaks can be effectively attended via automatic resource management. With serverless, the cloud provider automatically handles the scaling of resources based on the needs of the application. This can lead to improved performance and availability for the end user. On the second hand, serverless platforms offer the possibility of function chaining (Daw et al. 2020; Balla et al. 2021; Sabbioni et al. 2021), where the completion of a function can trigger the invocation of another function. This allows for the creation of complex workflows by linking together multiple simple functions. For example, each task in the workflow can be implemented by a different function, and when a task is completed, its output is placed in an intermediate storage that triggers the execution of the next function. In this work, we propose the use of this serverless function chaining scenario to deploy our scientific application in an efficient way.

In particular, we have focused on Amazon Web Services (AWS), one of the leading public cloud providers, which pioneered the provision of serverless computing services. Its FaaS Lambda ¹² has been used in our paper because it allows an easy code execution without managing infrastructure elements while defining events that trigger the functions (Villamizar et al. 2016).

AWS Lambda operates by executing each of its functions within a dedicated container, created when the function is first created. These containers are then executed on a multitenant cluster of machines managed by AWS. To ensure that each function is allocated the necessary resources, each container is allocated the required RAM and CPU capacity before execution. When a function completes, the RAM allocation is multiplied by the function's run time, and customers are charged accordingly based on the allocated memory and run time. AWS Lambda allows many instances of the same or different functions to run concurrently, making it possible to deploy highly scalable applications. To implement function chaining in AWS Lambda, developers can use a variety of methods, including AWS Step Functions, AWS EventBridge, or the Simple Storage System (S3), among others. The scientific workflow implemented in this work is based on S3 buckets that are configured to trigger a lambda function when new objects are added or updated, so that a sequence of lambda functions are executed, with each function processing the data in a specific way before passing it on to the next function in the workflow.

The total function running time in AWS Lambda comprises the execution of the code itself and the initialization performed by Lambda (Fouladi et al. 2017). The code is executed inside a container that is deployed to this end. This container will be reused if the function is triggered again in the next 15 minutes, reducing the initialization time (Vazquez-Poletti & Llorente2018).

In order to understand the proposed architecture, its building blocks are explained in depth:

• 1.  
  S3 buckets are a type of cloud storage system with some special characteristics. In particular, we take advantage of the triggering produced by new files appearing in the bucket. This way, each new MARSIS ionogram uploaded to S3 will force the execution of an operation we have previously defined on it. Multiple triggerings will produce multiple executions that will run in parallel.
• 2.  
  A lambda function is a cloud computing component that allows to execute a certain code (different languages are supported ¹³ ) in a set of given conditions (execution deadline and maximum memory). As it can be deduced from the previous S3 explanation, lambda functions are invoked through defined triggers. The main advantage is its high level of parallelism, allowing up to 1000 instances of the same function to be executed at the same time. In our study, we conducted tests on the architecture with a limit of 1000 parallelization functions; however, it is worth noting that a significantly higher number of simultaneous executions can be requested on-demand through Amazon Web Services (AWS).

There are different programming languages for developing lambda functions. In the present work, Python was chosen for its versatility when performing mathematical calculations and working with advanced color patterns. We have been using both 3.8 and 3.9 run times because not all lambda layers work with the same version.

Additionally, lambda developers do not interact with any operating system. However, the function execution takes place on a container running Amazon Linux.

The proposed architecture is decomposed in several chained lambda functions that will use S3 as swap memory, as the output files of a function will trigger one or more other functions. These functions and buckets are organized in modules, as it can be seen in Figure 2.

Zoom In Zoom Out Reset image size

The first module (COMMON PRE-PROCESSING) is common to all MARSIS input files, and it prepares the data to be processed by the following modules.

The next classification module (CUSTOM MEASURE MODULE) launches simultaneously the initial functions of different detection modules and on the same input data. In this work we have focused on the GREEN DETECTION function, which moves the process to the module for anomaly in the green spectrum detection, and finally to the module for the oblique echoes detection. Returning to the CUSTOM MEASURE MODULE, the first function of additional detection algorithms besides the oblique echoes one would be placed here.

The final result of the proposed architecture are the ionograms classified in different folders on an S3 bucket, according to what has been detected on them.

3.2.1. Common Preprocessing Module

MARSIS ionograms are uploaded to the S3_INIT input bucket. Each file upload triggers the INIT_MODULE_MARS function, which classifies the ionogram according to the latitude, longitude, and solar zenith angle. This process is aided by a custom layer named Tesseract that parses the previous mentioned parameters directly from the image (as they are written on it).

Once the images are classified, their white borders are removed. This way each image is properly centered and deposited in the S3_CROP bucket (see the example shown in Figure 3). This action triggers the next modules.

Zoom In Zoom Out Reset image size

As explained before, in this contribution we have focused on the measured module (CUSTOM MEASURE MODULE).

3.2.2. Custom Measure Module

Upon detecting a new file in the S3_CROP bucket, different functions are simultaneously triggered (GREEN DETECTION, BLUE DETECTION, HARMONICS DETECTION). The present work covers the GREEN DETECTION function, which removes the blue spectrum in the ionogram. This procedure isolates its green zones using color masks converts the color space of the image from the RGB color space to the HSV color space using the cv2.inRange function; this range falls between 60° and 120° on the HSV color space. This function returns a binary mask where pixels that fall within the specified range of colors are set to white, while the rest are set to black.

Then the cv2.bitwise_and function applies the mask to the original image, resulting in a new one where only the pixels that fall within the specified color range are visible, resulting in a cleaner version that is stored in the S3_NO_BLUES bucket (see the example shown in Figure 4). This triggers the GREEN PATH MODULE.

Zoom In Zoom Out Reset image size

3.2.3. Green Path Module

This module is the most complex in the architecture, because it involves several functions to prepare the ionogram for oblique echoe detection.

The first function of this module is MARSIS_AREAS, which works with a dictionary that provides possible detection locations within the image. This way the ionogram is cropped depending on the frequency or area where potential anomalies should arise. For the specific case of detecting oblique echoes, we perform an image cropping in the frequency range of 2.5–5.5 MHz to isolate the areas where these phenomena may occur. This technique is commonly used in ultrasound imaging to enhance the visibility of oblique structures and improve diagnostic accuracy. By selectively removing unwanted noise and artifacts from the image, the cropped frequency range can provide a clearer and more detailed representation of the targeted area of interest. The result is stored in the S3_AREAS bucket (see the example shown in Figure 5), which triggers the BINARY_AND_NOT function.

Zoom In Zoom Out Reset image size

Finally, the resulting image is written to a file at a given path using OpenCV's imwrite function. The resulting image is then saved to an S3 container.

In this particular case, the purpose of this image processing is to isolate the green areas of the image and filter out the blue color values. This is intended to identify the anomalies, which may appear in green.

The process then focuses in identifying the relative position of the objects in the figure. This is accomplished by the BINARY_AND_NOT function, which produces a binary image with black background and white objects, followed by a color invertion (white background and black objects) and file upload to S3_BINARY.

The next function triggered is MARSIS_AIS_RESIZED, which isolates the image according to the 800 × 600 px standard adopted by ESA in this public database. The function rescales the image, leaving the detection area in the middle. This prevents each loose pixel from being treated as a separate object from the whole image. The result is loaded to S3_RESIZED (see the example shown in Figure 6), triggering the MARSIS_NO_NOISE function.

Zoom In Zoom Out Reset image size

This function removes the abovementioned loose pixels by detecting their proximity with lines. It uses OpenCV's imread function at the beginning and then converts the color space of the image from RGB to grayscale using the cvtColor function.

Then it applies a binary threshold to the grayscale image using OpenCV's threshold function. This creates a binary image where pixels with intensity values greater than 127 are set to 255 (white), and pixels with intensity values less than or equal to 127 are set to 0 (black).

Next, the function finds contours in the binary image using OpenCV's findContours function. The contours are retrieved using the RETR_TREE mode, which retrieves all the hierarchical contours, and the CHAIN_APPROX_SIMPLE method, which compresses horizontal, vertical, and diagonal segments and leaves only their end points.

Then, a mask with the same shape as the original image is created using NumPy's zeros function. The code then loops through the contours and checks if they have no child contours and if their area is less than 70. If these conditions are met, the contour is drawn on the mask with a white color.

Finally, OpenCV's bitwise_not function is used to invert the colors of the original image using the mask. The resulting image is then written to a file at a given path using OpenCV's imwrite function. The resulting image is then saved to an S3 container.

In this particular case, the purpose of this image processing is to detect and isolate small objects in the image that have an area less than 70 pixels. The detected objects are then converted to a mask and used to remove those objects from the original image. The result of this last function is stored in the S3_NOT_NOISE bucket, triggering the ECHO DETECTION MODULE.

3.2.4. Echo Detection Module

The primary objective of this module is to identify the number of objects and their relative position. At least two of these objects arranged in parallel are needed for detecting oblique echoes, as it will be explained later.

The process starts with the NUMER_OBJECT function, which counts the objects in the image. It first reads the ionogram and converts it to grayscale using the cv2.cvtColor() function. Then, it applies a threshold to the grayscale image using the cv2.threshold() function, with a minimum threshold value of 225, a maximum threshold value of 255, and an inverted binary threshold type (cv2.THRESH_BINARY_INV).

Next, it creates a 2 × 2 kernel using np.ones() function and performs dilation on the thresholded image using the cv2.dilate() function with the created kernel and 10 iterations. The resulting image is used to find contours using cv2.findContours() function with an external retrieval mode (cv2.RETR_EXTERNAL) and simple approximation method (cv2.CHAIN_APPROX_SIMPLE).

As said before, oblique echo detection needs at least two objects: an original and a replica underneath. When these (or more) are detected by the architecture, the image is loaded into the TMP bucket, which triggers the ECHO_DETECTION function. This function takes the relative positions of the objects into account and, if the detection is positive, it stores the original image along with the detection proof (see the example shown in Figure 7) in the final bucket (TO_EDGE).

Zoom In Zoom Out Reset image size

We have considered three different computing platforms to run our experiments:

• 1.  
  Computer_1: an Intel Core i9 10980XE (18 cores, 36 threads) computer with 64 GB RAM and SSD (M2) hard disks. Its estimated price is €3500.
• 2.  
  Computer_2: an Intel Core i7 vPro i7-10875H (8 cores, 16 threads) computer with 64 GB RAM and SSD (M2) hard disks. Its estimated price is €3250.
• 3.  
  Serverless: the AWS Lambda system described in this contribution with a 128 MB memory limit.

In order to compare these three platforms, the code of the serverless architecture shown in Figure 2 has been adapted to run locally in Computer_1 and Computer_2. During the initial experiments we considered the following:

• 1.  
  An execution deadline of 1 hr was established for all experiments, so local resources were not busy for too long. Due to the low time variations in the same execution, this deadline was considered enough for result extrapolation.
• 2.  
  Due to the restrictions imposed by the Tesseract custom layer each execution will have to take place in a separate environment and assigned to an available processor core.

The reason for the last consideration is that text recognition may not be accurate, especially when the image quality is poor. Therefore, it is crucial to optimize the image before using this layer to enhance the recognition accuracy.

Using the Tesseract in a Python function requires an additional compiled layer. Unfortunately, the size limit of the lambda function layers is 250 MB, which can be a problem when more layers are added. This limitation prevents the inclusion of other required dependencies and resources in the same function, which can impact the overall performance (Section 3.2.1).

The serverless solution processed the entire data set in 20 minutes, Computer_1 managed to process 4320 (0.009%) and Computer_2 1440 ionograms (0.003%). With this in mind, Table 1 shows the execution time for the data set on each considered platform.

At this point it is important to indicate that using the Tesseract in Python has been deemed as a complex undertaking, particularly when attempting local execution. Despite the creation of scripts aiming to automate the process, manual execution remains a necessity. Throughout the testing phase, maintaining consistent code has been a priority in order to ensure accurate comparisons.

Parallelization on a desktop environment presents its own set of challenges, as it would need a distinct code base from that utilized in AWS Lambda. Moreover, threading in Python is a notoriously intricate and error-prone endeavor. While alternative solutions may be achievable through the employment of various tools in the C programming language, the advantages of cloud-based parallelization would be undeniable.

By leveraging the power of cloud computing, the achieved parallelization far surpasses what would be possible using all available threads on a local machine. As a result, the decision has been made to compare the same unmodified code in both environments, avoiding the falsification of outcomes and ensuring a fair comparison between the two distinct implementations.

The main advantage of the AWS Lambda approach is its extreme parallelization capability, which clearly exceeds those of the local resources that were used during the experiments. This added to its autoscaling mechanisms result in the performance and cost numbers shown below.

Data for each function's performance have been gathered through the Amazon CloudWatch tool. ¹⁴ These data include execution times and costs. Table 2 shows the average execution times for each AWS Lambda function depicted in Figure 2.

Considering that AWS limits the simultaneous execution of lambda functions to 1000, we have estimated 3.9 s as the average execution time for each batch of 1000 ionograms.

The AWS Lambda cost is determined by the memory used by functions (128 MB in the proposed architecture), the maximum execution time per function (5 s in the proposed architecture), and the number of function calls. This results in an average of $0.000000084 per processed GB.

On the other hand, the S3 cost is subject to the stored data size. However, the proposed solution does not store intermediate data (just input and final output files). We would like to highlight that we are generating data in each intermediate S3 bucket. However, we have implemented a mechanism in each lambda function to delete all intermediate files once they have been processed. Therefore, no intermediate data are stored in the system as they are deleted automatically. This limits the S3 cost to $0.00000023. This way, the total cost of processing the 441,919 ionograms is $0.00001.

In order to provide a fair and accurate comparison, tests were conducted using exactly the same Python code in both local and lambda environments. The code was set up locally without a lambda handler, meaning that it does not have a trigger like in AWS, but the internal code is the same. While it is possible to use threads and different cores within Python, the code was not modified for parallelization in lambda in order to avoid falsifying results. Even with this limitation, the number of threads and cores available on a local computer still cannot match the parallelization resources available in lambda.

The results obtained were based solely on the times required to execute exactly the same code, with no modifications for parallelization using threads or cores. This approach allowed for a fair comparison of the performance of local computing resources versus the parallelization capabilities of lambda.

Overall, the results clearly demonstrated the benefits of lambda's extreme parallelization capabilities. Despite the fact that the same code was used in both environments, the lambda environment consistently outperformed local resources. This underscores the power and scalability of lambda for organizations that need to process large amounts of data quickly and efficiently.

The final cost for the anomaly-detection classifier in the Mars atmosphere includes hosting in S3, but without cold storage, as intermediate images are deleted in each step of the process because they are not longer needed. As a result, the overall cost for processing 120 GB of data is reduced while the architecture's efficiency is maximized.

To perform statistical analysis on a large data set of approximately 400,000 images of the Mars atmosphere, a random sampling of 120 images (0.03%) was taken. The sampling process was designed to be representative and unbiased, with random selection of different ranges of zenith angle, latitude, longitude, and dates. This ensured that the sample was diverse and representative of the overall data set.

After this random selection process, each of the 120 sampled images was manually verified to ensure accuracy and validity. This verification process involved analyzing each image for any anomalies or irregularities, and cross-referencing the results with the statistical analysis of the larger data set.

These are the results:

• 1.  
  The number of oblique echoes was 90, with the rest being false positives.
• 2.  
  These 30 false positives correspond mainly (20 ionograms) to cyclotron echoes, which have a similar pattern (see Figures 8 and 9).

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

While a false-detection rate of 25% may seem high, the technique used to generate an initial data set is still extremely valuable because no oblique echoes were undetected. By using automated processes to detect anomalies in the Martian atmosphere, we are able to generate an initial data set quickly and efficiently without the need for manual inspection of every image.

This initial data set can then be used to train the first machine-learning model for detecting this type of anomalies, with the understanding that the initial model will have a higher false-detection rate due to the limitations of the initial data set. However, this can still be improved over time by incorporating feedback from manual inspections and incorporating additional data to the refining process.

Eventually, with the help of the distributed architecture and trained machine-learning models, we can expect to achieve an extremely efficient classification model with a much lower false-positive detection rate.