Ecological Application of UAVs for Monitoring and Eliminating Oil Product Spills on the Sea Surface

The relevance of developing new technologies for environmental monitoring of marine waters and for eliminating local oil pollution using UAVs has increased significantly at present. The integration of Artificial Intelligence (AI) frameworks significantly enhances the operational capabilities of UAV technologies in ensuring environmental safety. The emergence of advanced deep learning techniques and specialized neural architectures significantly augments the capabilities of UAV-based methods for oil spill detection and mitigation. This is especially true for a no single drone, but multi-agent drone system for marine environmental safety, where each agent performs its functionality in the mission.

The requirements for such a system are formulated as follows. The ecological UAV system should provide a comprehensive solution to the problem: detection of oil spills, approximate assessment of oil product volumes; remediation of the oil spill by spraying dispersants; collection of data for assessing the damage caused to the marine ecosystem both by the oil spill itself and by dispersants used for spraying with UAVs; detection of oil product grades and provision of data for identifying the source of pollution.

All tasks included above in the mission of ensuring the environmental safety of the sea during an oil spill can be divided into two functionalities: monitoring and eliminating the oil slick. Accordingly, the ecological multi-agent system should include two types of agents: UAV monitoring and UAV spill elimination. In turn, the application of monitoring and elimination tasks includes many functions that can be performed only using modern environmental monitoring methods integrated with artificial intelligence elements that implement these methods and ensure autonomous mission execution. This paper describes the results of the sequential development of the hardware and software framework for the multi-agent system and the testing of each of its functionalities, using the example of an experimental multi-agent system consisting of only two agents: UAV monitoring and UAV elimination of oil product slicks on the sea surface. All experiments were conducted on oil product spills in situ in the waters of Peter the Great Bay during the summer from 2019 to 2024. Slick thicknesses varied within 0.05–500 μm, and spill areas reached a square kilometer.

The article presents the results of the validation of the main provisions of the new concept and methodology of an integrated mission of environmental safety of marine waters in case of pollution by oil products using a multi-agent UAV system. It includes both detecting spills and eliminating them using the dispersant-spraying method, and assessing the impact of these processes on the state of phytoplankton communities. The adaptation of AI tools for the detection and assessment of oil spill volumes has been performed: procedures for primary and secondary segmentation of an oil slick on the sea surface have been developed. This approach enables the detection of oil spills, the measurement of their total area, and the segmentation of the slick according to varying thickness levels. This is necessary to correctly determine the amount of dispersant that should be sprayed over each segment to eliminate it. The prospects of the LIF spectroscopy technique for the rapid determination of the effects of marine pollution processes with petroleum products and their elimination processes using dispersants on the state of phytoplankton communities have been experimentally demonstrated. The most effective deep learning methods for solving the problems of determining the grades of petroleum products polluting the water area using the LIF methodology have been identified. This advances the capabilities of multi-agent environmental systems to identify possible sources of pollution. The development of hard and soft frameworks for the multi-agent UAV systems made it possible to practically implement the concept of ensuring comprehensive safety of marine waters during oil spills by the UAV systems for the first time.

Nowadays, agriculture is the most advanced sector in the civil application of various multi-agent systems [1]. There are good examples of multi-agent UAV systems for monitoring natural disasters [2], transporting payloads to their intended destinations [3], ensuring environmental safety and fire monitoring [4,5], and searching for sources of anthropogenic air pollution [6]. In marine applications, single drones are mainly used for island mapping [7], monitoring and preventing biological hazards [8], preventing and eliminating the consequences of natural and man-made marine disasters [9,10], analyzing the physical parameters of the sea surface [11], detecting and assessing the spatiotemporal distribution of anthropogenic debris [12], monitoring aquaculture development zones and the fishing process in situ [13], analyzing various marine ecosystems and marine fauna (for example, coral reefs, glaciers, wetlands and mangroves, assessing the health and biological characteristics of the animals studied) [14,15,16].

An analysis of the applied aspects of UAV use in the oil and gas marine sector for environmental monitoring and detecting oil spills and leaks of petroleum products is provided in review papers [17,18]. Research results are analyzed to develop UAVs for each application considered. To date, only theoretical works on the development of concepts for the use of a multi-agent UAV systems for monitoring and eliminating pollution of marine surfaces with oil products are known [19]. Thus, in works [20,21], the theoretical aspects of dynamic cleaning of oil spills are investigated, and a concept for a system of autonomous hardware and software UAV complexes designed to clean the sea surface by spraying dispersants is proposed.

Using a multi-agent UAV system has high potential in the Arctic and freezing waters. In some cases, this may be the only possible method for eliminating oil product slicks on the sea surface of freezing waters.

Local pollution of the sea with oil products is becoming one of the main sources of marine pollution. The number of large oil spills has been decreasing annually since 1970 [22]. Estimates of the volume of oil pollution show that more than 40% of the total water pollution is not due to global spills but to unspecified (local) sources [23]. Contaminated wastewater is considered the greatest threat to coastal waters, so special programs are being developed to combat such local pollution [24].

Local pollution is dynamic—it appears quickly and exists for a limited time. As a rule, it emerges where ships are bunkered, when ships discharge bilge water, when oil products are washed off coastal areas during rains, etc. For this reason, it is difficult to detect them and hard to remediation them using standard methods. In many cases, standard methods for eliminating small-scale pollution are ineffective and economically impractical.

Small-scale spills usually cause “hidden” damage, which does not manifest in the death of marine fauna and flora on a significant scale. However, as a rule, they occur in the coastal zone, on the shelf, near recreation areas, within the city limits, or near aquaculture facilities and require prompt elimination. Such frequent pollution of the sea with oil products causes damage, which is manifested mainly in the form of the impact of dissolved oil products in seawater on the state of phytoplankton communities. The impact results are the subject of the research at present. The literature contains the most contradictory results of studies on the impact of oil products dissolved in seawater on phytoplankton communities, from cell death to increased activity and increased concentrations of certain phyto and nanoplankton cells. Each marine area where spills occur can be characterized by the species composition of phytoplankton or the most likely types of oil products spilled on the sea surface. Therefore, the results of studies can vary significantly [25,26,27,28]. To study the impact of small-scale oil spills on the state of phytoplankton communities, it is necessary to conduct numerous in situ studies. These studies can be carried out using specialized research methods adapted to UAVs.

Spraying dispersants is one of the efficient ways to eliminate oil product slicks on the sea surface. By now, exact rules for monitoring and assessing the volume of oil spills and measures for remediation using aviation equipment have been developed and implemented [29,30]. However, in many situations, an autonomous ecological multi-agent UAV system will be more effective and economically justified.

Numerous types of dispersants based on solvents and surfactants are recommended for use and have varying degrees of impact on marine ecosystems. The development of new dispersants continues to create the “greenest” dispersants, so there would be the least impact on the state of phytoplankton cells [31,32,33,34,35,36,37]. However, the issue of the impact of dispersants on marine ecosystems, especially on phytoplankton communities, is also being actively studied at present [38,39,40,41]. The results of studies on the interaction between dispersants and phytoplankton communities obtained to date are not unambiguous. Therefore, it is crucial to develop an operational method for in situ studying of the impact of dispersants on the photosynthetic apparatus of phytoplankton cells [42,43]. It is very important to develop sensors specifically designed for UAVs to measure chlorophyll A concentration and dissolved organic matter in seawater. So systematic studies of the impact of these processes on phytoplankton communities, as well as the development of new methods for assessing the damage caused to phytoplankton communities by oil spills and the destruction of oil slicks by dispersants in a specific marine area, may be realized.

The monitoring functionality includes three functions: primary segmentation and oil slick detection function, secondary segmentation or color code segmentation function, and LIF spectrometry.

The first function of the multi-agent system is to detect oil spills on the sea surface and determine the outer boundaries of the slick, i.e., outline the slick. An artificial intelligence element is integrated into the computer vision of UAV monitoring, the element is trained to detect a slick on the water surface and determine the contours of the spot, i.e., to segment the entire slick, highlighting its boundaries (primary segmentation). Two stages at once—spot detection and highlighting its contour on the surface are combined in this function. The computer vision of the UAV monitoring is provided by the IMX219 video camera (Raspberry Pi Camera Module V2, Sinoseen, Shenzhen, China), which works together with the NVIDIA JETSON on-board computer with ROS software (ROS 2). When an oil slick is detected, the area on the image is outlined along the contour of the detected object. The resulting image is analyzed by an algorithm that is trained to recognize an oil slick on the sea surface and perform slick segmentation. The analyzed slick is highlighted as a selected area on the image. To solve the recognition problem (semantic segmentation) [44], an AI element is needed in the form of a software component, using the corresponding neural network architecture as an algorithm, which allows for the analysis of images and videos. The Mask R-CNN neural network [45] and the Python programming language were used to create this software component. The OpenCV library of open-source computer vision, image processing, and general-purpose numerical algorithms was used to work with slick images. Mask R-CNN, implemented using the Keras library with the Tensorflow framework, was chosen as the implementation system for the Recognition Algorithm element component for object segmentation. The image dataset was used to train an AI element to analyze images and detect oil spills on the sea surface. Convolutional neural networks and supervised learning were used [46]. The dataset required to train the algorithm included more than 3000 images of real oil spills obtained from photos and videos. The data set also included images of the same spill taken from different angles in order to increase the number of images and expand the learning environment. During the process of forming the dataset, areas with visible oil contamination were identified on all images. The “Uplearning” procedure of the segmentation algorithm was applied to images of slicks not included in the dataset, thereby increasing the algorithm’s accuracy (about 150 such slicks were used). Figure 1 shows an example of primary segmentation (detection) of an oil slick using the created AI detection element. Thus, the primary segmentation consisted of determining the outer and inner contours of the oil slick. The slicks shift and change shape and size over time. The example of the evolution entire slick, which takes place over two days, is presented in Figure 2. The slick was shifted and divided into three parts by surface currents and winds.

The accuracy of segmentation was assessed as the relative difference in area between the true value—S_t and the area of the segment that the AI element detected—S_s:

If S_s ≥ 2S_t, then $$\Delta$$ is equal to 0. The range of $$\Delta$$ values was 60–80% for the analyzed slicks of real spills that were recorded during the monitoring period. The difference between S_t and S_s was calculated for several selected spill images to evaluate the primary segmentation error under various sea states and monitoring conditions. The corresponding program counted the number of pixels in the original image (S_t) and then in the segmented (contoured) image (S_s). The difference was multiplied by the film area of one pixel, thus achieving the value S_t − S_s. The size of one pixel was previously calculated based on the known dimensions of objects within the frame.

The lowest values of $\Delta$ were obtained for images of light marine fuel spills in wind conditions with a wind speed of more than 5 m/s. Small-scale ripples were observed on the water surface (example Figure 1a), thin and long structures, such as filaments, were observed at the boundaries of the slick, and surfaces of clear water were observed inside the spots in this case. The maximum values of $$\Delta$$ corresponded to calm or wind with a speed of <5 m/s.

Given the known grades of oil products and sea surface temperatures, the results of the primary segmentation already allow for estimating the volume of oil spills. Further improvement of the accuracy of primary segmentation is associated with an increase in the number of images in the dataset and fine-tuning of the neural network.

Oil slick elimination by spraying dispersants is the next step of the environmental safety mission multi-agent UAV system. To proceed to this step, it is necessary to perform secondary segmentation of the slick image. It consists of dividing the slick into segments that have the same thickness. The volume of dispersant that must be sprayed to eliminate a given slick segment depends on the thickness and area of the segment. We used the method adopted in the Bonn Agreement [29] for secondary segmentation of the oil slick image. As in the case of primary segmentation, the secondary segmentation of the slick is achieved by deep learning of the neural network and the creation of a corresponding dataset of oil slick images for training. Secondary segmentation of the slick is provided by the following. The area already allocated by the primary segmentation must be divided into several other related codes (sheen, rainbow, metallic). We used these three color codes only at this stage of training. The thickness range of each color is given in Table 1. The color of each pixel in the selected area is analyzed. By selecting the pixels related to the rainbow and counting their number, we can say what part of the total area is occupied by the rainbow pixels.

The remaining part can be divided by shade into lighter and darker areas, and their ratio can be estimated; these will be sheen and metallic. An example of such segmentation is shown in Figure 3 (red color corresponds to the sheen color code, green—rainbow, and blue—metallic). Thus, the slick outlined during primary segmentation is divided into segments of equal thickness (to which specific colors are assigned) during secondary segmentation. Secondary segmentation allows for the calculation of the volume of dispersant required to eliminate the slick in each segment.

It should be noted that the selected areas of equal thickness represent connected areas of the surface only in the absence of small-scale ripples on it (example in Figure 3b), when the wind force is less than 5 m/s or in calm weather. In the case of stronger winds and the presence of “ripples” on the surface (as is the case in Figure 3a), the task of determining the volumes and trajectory of spraying becomes more complicated. And it is necessary to include additional software blocks in the program for controlling the spray volumes for UAV slick elimination, which should take into account the presence of such a complex structure in the distribution of areas of the same thickness over the surface of the slick. The procedure of secondary segmentation by color codes was carried out for image areas that were at angles close to the nadir during shooting. Areas of the slick that are at angles greater than 40° to the nadir direction are identified by the sheen color code in Figure 3b. However, when examining images of these areas obtained during the UAV flight over them (in the nadir direction), all three color codes are present.

The diagram of the functionality of the primary and secondary segmentation of the oil product image is presented in Figure 4.

The next function of UAV monitoring functionality is the laser fluorimetry of oil products directly in the spills and dissolved in seawater. This function is used to implement in the monitoring practice a method for identifying types of marine fuel and to develop new methods for assessing the damage caused to phytoplankton communities by pollution from oil products and dispersants used to eliminate them.

The UAV monitoring can be equipped with a laser fluorimeter to measure LIF spectra. Seawater fluorescence spectra are measured directly under the slick or in the immediate vicinity of the spill when the UAV lands on the water. The procedure for measuring spectra, the methodology, and the laser fluorimeter itself were described in [42,43]. Further development of this method was carried out in the direction of expanding the archive of LIF spectra from petroleum products and using the most effective deep learning methods for these tasks. Laboratory studies of LIF spectra were conducted at various stages of degradation and weathering of the petroleum product slick. The obtained data were used to create a dataset, which was used to conduct machine learning to recognize types of petroleum products found in a spill or solution of seawater. An analysis of existing machine learning algorithms for the classification of induced fluorescence spectra of seawater with dissolved oil products was conducted. The analysis was carried out using the k-fold cross-validation method [47]. It showed that the most optimal algorithms for the classification of induced fluorescence spectra are the Linear Discriminant Analysis (LDA) [48] and Linear Support Vector Machines (Linear SVC) [48] algorithms. An important aspect is the possibility of using these algorithms on relatively low-performance processor boards. This allows them to be used to implement compact hardware and software systems for use on UAVs.

Analysis of experiments in a laboratory pool showed that the recognition algorithm is highly likely to identify types of dissolved petroleum products under a slick that are located on a calm surface. This is true if the LIF was measured within no more than a few hours from the moment the slick appeared for light grades of marine fuel and no more than 2 days for heavy grades, and in conditions of no direct sunlight. During the degradation of petroleum products, the characteristic features of the LIF spectra for different grades change differently, and recognition algorithms reduce the accuracy of determining individual grades of oil. The LIF spectra of degraded petroleum products differ from the reference spectra obtained in the laboratory. The creation of a dataset of LIF spectra obtained under natural conditions, with varying degrees of degradation, remains relevant for implementing this function. UAV monitoring with a laser spectrometer is the optimal tool for this task. This will improve the accuracy of the identification process by accounting for the degree of oil spill degradation.

Arrays of spectra of different types of petroleum products were used to compile the dataset to extract features from them. The data matrix formed from feature descriptions was used to train the LDA and Linear SVC methods. An example of the operation of the created model in the case of using the Linear SVC algorithm is shown in Figure 5.

Thirty spectral characteristics of solutions of heavy marine fuel type RMB 30 with different concentrations were taken as test spectra. The spectra were obtained in laboratory conditions and were not used in the training process. The horizontal axis (Figure 5a) shows the grades of marine fuel, and the vertical axis shows the probabilities that the given spectra correspond to the specified type of marine fuel.

As a first approximation, an assessment of the impact of seawater pollution processes with oil products and the influence of dispersants during oil spill elimination processes can be carried out based on changes in the concentration of chlorophyll A in seawater. Currently, this issue is the subject of many studies. Both remote methods and laboratory methods of sample analysis are used [21,39,40,49]. Laboratory experiments and results of spill monitoring by a LIF spectrometer have demonstrated the possibility of in situ research of the impact of dissolved petroleum products and dispersants on the state of the photosynthetic apparatus of phytoplankton cells.

Figure 6 shows the LIF spectra, which demonstrate the possibilities of monitoring changes in the concentration of chlorophyll A under the influence of both processes. The response of phytoplankton cells to dispersants was studied by changing the chlorophyll A fluorescence line (680 nm) upon excitation of LIF with a laser wavelength of 532 nm. Figure 6 shows the fluorescence line of a clean seawater sample collected in the Amur Bay on 30 April 2022 (violet line). The chlorophyll A concentration in the sample was 19 μg/L. When only a DMA solution at a concentration of about 90 ppm was added to the clean seawater sample, the chlorophyll A concentration decreased by about 2 times (green line). When 0.1 mL of UNITOR SEACARE.

OSD2 dispersant was added to the same sample with a DMA solution; an additional decrease in the chlorophyll A fluorescence line intensity was observed (yellow graph). The measurements were carried out 24 h after the addition of marine fuel and dispersant solutions to the sample. We interpreted this decrease in intensity as resulting from phytoplankton cell death. This result was recorded for seawater samples collected during the spring algae bloom period in Amur Bay. The numerically dominant groups of phytoplankton were diatoms (44–70%), small flagellates (10–29%), and dinoflagellates (7–18%) [50].

Our laboratory experiments demonstrated that dissolved hydrocarbons and dispersants have a comparable impact on phytoplankton cells, with both cases resulting in a mortality rate of approximately 50%. However, the effects of oil pollution and the dispersants used to mitigate it on phytoplankton are highly complex. The outcome depends on the specific types of pollutants and dispersants applied to the sea surface, as well as the particular algae species involved.

Field studies are necessary to accurately determine the impact in each specific instance. Our findings highlight the potential of using UAV-based Laser-Induced Fluorescence (LIF) spectroscopy for the rapid monitoring of aquatic environments. The example provided here demonstrates the feasibility of using UAVs equipped with LIF spectrometers to monitor the state of phytoplankton communities. A more comprehensive answer to this question would require dedicated scientific research. This paper presents results showing that such studies are now possible using the multi-agent UAV system described above. This approach is significantly faster, more accessible, and more cost-effective than the traditional methods discussed in the literature review.

The main mission of the UAV in this case is to spray dispersant to eliminate the oil slick. The flight route and spray volume are pre-calculated based on the results of the oil slick monitoring mission. A specialized UAV was developed to test the method to eliminate oil product slicks. The UAV was equipped with a dispersant spray system and a 20-L dispersant tank and was capable of landing and taking off from the sea surface. Test missions to eliminate oil product slicks were conducted on small-area marine fuel spills in the waters of Amur Bay, which occurred during ship bunkering. Figure 7 shows an example of such a spill. The image of the marine fuel slick on the sea surface obtained by UAV monitoring is shown in Figure 7a.

The results of secondary segmentation of oil spill images using the color code method are shown in Figure 7b. Here, as before, the red color corresponds to the sheen color code, green to rainbow, and blue to metallic. The volume of dispersant and the optimal flight route for eliminating the slicks were calculated based on the results of secondary segmentation of the image of oil spills. The volume of dispersant sprayed over each segment is varied by changing the pressure in the dispersant spray system on the UAV. The areas of the unsegmented spots in Figure 7a were calculated on a laboratory computer, and only for slicks that were detected and segmented as shown in Figure 7b. The segmentation accuracy $$\Delta$$, calculated by Formula (1), was approximately 70%.

The areas of each of the three color codes were calculated, and the slick thickness was normalized to the average value of each code (see Table 1). The volume of oil product estimated in this way was 42 L in total for all the slicks in Figure 7b. The route points at which it was necessary to change the intensity of dispersant spraying in accordance with the slick thickness were pre-calculated. Spraying was carried out from a height of 7 m, so the diameter of the spot with sprayed dispersants on the surface of the oil product slick was approximately 6 m. The total mission to eliminate the oil spill took about 20 min, with the UAV moving at a speed of no more than 20 km/h (the calculated UAV route for eliminating the oil product slick is shown in Figure 7a). The dispersant was sprayed in calm conditions; there were no surface currents, and the spots remained motionless. The water surface was completely cleared of oil product slicks after the elimination.

The practical use of the essential two-agent system for ensuring the environmental safety of the sea allows us to obtain the right solutions for adapting all the functionalities to a multi-agent system with a large number of agents. The experience of a multi-agent system for monitoring and eliminating oil spills will show how to improve the system in the future.

The most controversial point in the elimination is the spraying of dispersants onto areas of the slick surface, which was segmented by color codes. Now the secondary segmentation of the slick can take a long time. And, in this case, such a function can be beneficial for a stationary slick. In the presence of strong winds or surface currents, the autonomy of the mission requires the expansion of the functionality of the software elements of the system due to the element of flight route correction and the UAV spraying process for elimination. In this case, the UAV monitoring should trace the movement and change in shape of the spot and make adjustments to navigation and the process of dispersant distribution. To ensure the autonomy of the entire mission, it is necessary to develop a software framework for agent interaction in a multi-agent system that implements this function.

It is required to significantly increase the datasets to improve the reliability of primary and secondary segmentation, and the identification of grades of petroleum products. Adapt more productive neural networks and deep learning methods to these tasks. Expand the functionality of the soft-hard framework to use a large number of agents.

LIF functionality is of great importance, especially for assessing the damage to phytoplankton communities. Small oil spills (example in Figure 7) do not cause significant visible damage that results in the death of waterfowl. And the only damage that can be recorded in this case is the death of phytoplankton cells, which is caused by a solution of certain concentrations of petroleum products in the seawater. The LIF effectively monitors this process by analyzing the intensity of chlorophyll A fluorescence. Similar changes can occur when exposed to a dispersant solution, which is used to eliminate oil slicks in the water. This exposure can occur due to excessive dispersant being sprayed or if the spraying extends beyond the slick-covered area. But it is extremely insufficient to develop a methodology based only on laboratory experiments. Research is needed in natural conditions, so the dynamics of LIF spectra in the presence of natural factors influencing the intensity of chlorophyll A fluorescence may be studied. This also applies to the accuracy of determining the grades of petroleum products using LIF spectra. Type identification by AI tools will be reliable if the database is filled with LIF spectra from slicks of various petroleum products in various stages of degradation and weathering. In this version of the LIF sensor, a huge limitation is the solar background, which limits it during sunny times. It is crucial to develop a closed flow system for pumping seawater, which will eliminate the influence of background radiation and conduct measurements of LIF spectra on sunny days. The development of a multi-agent UAV system and its practical use for both monitoring and eliminating oil product slicks significantly advances the solution of these problems.

Conceptualization, O.A.B. and V.Y.; methodology, O.A.B. and D.P.; software, A.C. and D.P.; validation, D.K., I.B., D.B.; formal analysis, S.G. All authors have read and agreed to the published version of the manuscript.

Not applicable.

Not applicable.

Not applicable.

This research was funded by the Russian Ministry of Transport (State Task No.110-00022-21-00).

The authors declare no conflict of interest.