Image processing: Image processing applications for baking process monitoring

Image processing applications for baking process monitoring

Computer vision and image processing offer the development and employment of non-invasive, contactless, non-destructive, and fast measurement methods with a wide range of applications within the baking industry. These include both the visualization (qualitative measurement) and the exploration of shape and surface properties (quantitative measurement).

+As a subcategory of signal processing, image processing is used in satellite imaging, medical imaging, and industrial imaging for process and even product assessment. Image processing uses camera technologies to capture two- or three-dimensional data, followed by data transfer to a processing unit to extract product or process information.

The usual captured light range is the visible (VIS) light range (400–800nm) because it is captured by both CMOS (complementary metal-oxidesemiconductor) and CCD (charge-coupled device) cameras, which can also include the initial nearinfrared (NIR) range (800–900nm). Silicon sensors can also detect the extended VIS/NIR range. The NIR range up to 1700nm is captured using InGaAs sensors, e.g., with shortwave infrared (SWIR) cameras. For the mid-infrared range (MIR, 2.5–25.0μm), used to capture thermograms, microbolometers are utilized. Microbolometers can also be used to capture ultraviolet and Xradiation. Other camera types for specific parts of the electromagnetic spectrum are available, and these are constantly being developed and adapted to match changing requirements.

Image data capture occurs mainly in line scan mode, in which a dynamic scene is captured line by line, or in area scan mode, where the whole scene is captured in one shot. For image processing, these single recorded lines are sequentially stitched together for image reconstruction. For example, with conveyer belts, typically line scan cameras are used to capture continuous webs of raw material or products.

The obtained image data are available as twodimensional matrices with intensity values. For monochrome cameras, the form is n×m×1, and for RGB cameras, the form is n×m×3. The light or color intensity forms the third dimension for each pixel inside the layers in the n×m matrix. InGaAs-cameras and microbolometers capture n×m×1 monochrome data. By using filters for the separation into individual wavelengths, so-called spectral observations can be carried out. In the resulting spectral cube, the captured data is in the form n×m× ‘number of observed wavelengths/ regions’. The number of captured image data (sampling frequency) is specified by sampling criteria and indicated by frames per second (fps).

Image data processing can be applied at different computational levels. First is the description of the image using statistical tools, e.g., the mean, minimum and maximum value of the color, contrast, or histogram analysis. The image data

Figure 1: Overview of typical image processing applications

Figure 2: Camera system for proofing monitoring and captured images of a dough piece from 0° - 90° can be processed point-wise or in groups, and operators such as thresholding for object isolation can be applied. More sophisticated operations are template matching methods, e.g., the ViolaJones algorithm or neural networks, which can be trained to detect particular objects in scenes. The advantage of these higher-level methods is the robust and real-time object detection. However, the training of the methods in all variations of the object, especially in food manufacturing, where not all objects have the same shape, could be extensive. After the object is detected or recognized, feature extraction can be performed.

Figure 1 gives an overview of the mentioned image processing methods and typical applications. The VIS image processing can measure properties such as object size, color or simply be used as a presence sensor. When applied over time, the evolution of size or discoloration can be measured and kinetics can be established. A spectral imaging system captures three-dimensional spectral data cubes, where the first two dimensions are the spatial information and the third dimension is the wavelength-discretized signal intensity. Chemical information from the region of interest or segmented objects can be measured this way. Dynamic laser speckle imaging can be used for laser speckle contrast or rheology analysis.

Current developments in image processing Digital image processing in the VIS range is applied in the baking industry for raw material analysis by scanning single kernels, analyzing the color, and evaluating, e.g., in fungus contaminations, glossy or non-glossy, whole and broken kernels. With high-speed camera technology,

© TUM

Figure 3: Segmented dough pieces inside the proofing chamber

the raw materials can be analyzed inline and in real-time. Digital image processing is also applied for product or process assessment. In the project AiF 18123 N (German Federation of Industrial Research Associations) [1], a camera system was applied to capture dough pieces during the proofing process. Figure 2 shows the set-up used to measure the dough pieces over time during the proofing. The camera rotated from 0° to 90° around the dough pieces and captured the front and side view to calculate the three dimensions of the pieces: width, height, and depth.

By using segmentation algorithms, single dough pieces were detected automatically. Figure 3 shows the 0° view of single-segmented dough pieces with the measured width and depth (yellow boxes) during proofing. As data was captured over time, the growth kinetics were calculated and are available for process control and automation.

Furthermore, object capturing was applied for a consecutive proofing and baking process. Two cameras, one with top view and one with side view, captured the dough pieces. After applied segmentation, the width and depth, browning index (BI) [2] (see Eq. 1), and texture homogeneity [3] (see Eq. 2) of single pieces were analyzed. L, a, and b are the mean color values of the Lab color space of an object and t is the process time.

Where i indicates the row and j the column of the gray-level co-occurrence matrix (GLCM), a homogeneity of 1 specifies homogenous, a value of 0 an inhomogeneous texture.

These object features were captured over time throughout the proofing and baking process. Figure 4 shows the results of one batch where the browning index, texture homogeneity, and dimensions (height and width) are calculated over the proofing and baking process.

In Figure 4, the texture homogeneity and color changes during proofing and baking, especially the crust browning during the baking process, are shown. Furthermore, the growth due to the proofing and the oven rise during baking is apparent.

Image processing in the mid-infrared spectral range was also applied to observe the baking process. One such implementation was in the project AiF 17735 N [4]. By using MIR imaging, the process and products were evaluated during

Figure 4: Mean browning index (BI), homogeneity, and dimensions (height and width) of the captured dough pieces during proofing and baking

© TUM

INTERVIEW IMAGE PROCESSING APPLICATIONS FOR BAKING PROCESS MONITORING

Figure 5: MIR range image capture of bread, in a) pseudocolor representation of the bread inside the oven deck and in b) the segmented bread with indicated height and width

Figure 6: a) Schematic representation of a laser-optical system. The laser beam is expanded and directed towards the sample. A camera records the laser speckle images and transfers them to the computer. Depending on the application, the laser speckle images can be used to perform b) Laser speckle contrast analysis or c) Laser speckle rheology.

© TUM the baking process [5]. Figure 5 a) shows a captured thermogram of bread in a pseudocolor representation during the baking process. Figure 5 b) shows the segmented object and object borders used to determine the object’s width and height. As a reference, the samples were measured with a volumeter laser scanning device. The deviation between the volumeter laser scanning device and MIR-based measurements was 1.01 mm in width and 1.49 mm in height. Furthermore, the growth kinetics, especially the oven rise, can also be calculated based on MIR image capture over the baking process.

The development over recent years of thermal camera techniques and thermal housings has enabled more reliable measurements and more applications to the baking process; also, it has enabled measurement of other parameters such as crust cracks, heat losses, or even thermal distributions within baking plants. Another method for visualizing the surface properties of dough and baked goods is laser speckle imaging – in which coherent laser light illuminates the optically rough surface of a sample and the light is scattered from the surface and within the layers beneath the surface. Interference effects of randomly phased partial waves originating from different scattering sites create a pattern of brighter and darker spots. These randomly distributed grains of light forming high contrast patterns are called laser speckles (see Figure 6) [6].

The fluctuation of the laser speckle intensity is time-dependent and called dynamic speckle. The movement of the speckle pattern provides information about the movement of the scattering centers. A technique called laser speckle contrast

analysis (LASCA) utilizes this relationship to map flows and regions with different motion patterns and visualizes them directly on the depicted object [7].

LASCA is used to monitor the surface properties of baked goods in the context of AiF project 20495 N [8]. Two expanded laser beams, which are at an angle of 90° to each other, are directed onto a dough piece within an oven. Two cameras acquire the speckle pattern, as well as the shape and size of the dough piece. The technology can be used to visualize surface structures, which are invisible to the human eye. It allows the visualization of the gas distribution beneath the surface during proofing and the gas permeability through the crust as well as the crust formation during baking (see Figure 6). These spatial and timeresolved insights contribute to the optimization and intelligent inline control of the proofing and baking process.

Alongside mapping of the flow, the thermal motion of endogenous scattering particles can be observed. The fluctuations of the temporal speckle intensity are correlated to the mean square displacement (MSD) of light scattering particles undergoing Brownian motion and thus also to the viscoelastic properties of the surrounding medium. Based on this approach, laser speckle rheology (LSR) was developed to measure the mechanical properties of biological tissue [9, 10]. Compared to LASCA, where contrast reduction is used to distinguish areas within a region of interest in terms of motion, samples in LSR must be homogeneous and uniform. Through careful selection of light source, camera, and image acquisition specifications, an optimum speckle pattern as needed to accomplish LSR measurements can be achieved.

Furthermore, LSR was used to develop an innovative, non-invasive and comparatively inexpensive inline method to detect surface and rheological properties of dough during the kneading process [11]. The state of the gluten network shows a characteristic correlation with the rheological key figures from which the kneading optimum can be derived by analyzing the light scattering at the dough surface. This measurement method reduces both the time of analysis and the sample preparation requirements compared to the reference method.

Outlook: automatization based on image processing

All the approaches or methods described above are characterized by their process capability. Measuring the product or process parameters online in a non-destructive, non-invasive manner and in real-time enables a process control ability and opens up further application areas. Image processing has been applied in various projects, e.g., one of the last projects concerned the imagebased measurement of product width and depth in a proofing chamber and corrected possible process faults. In detail, a control strategy was developed for the fermentation of dough pieces, which could counteract a lower yeast quantity. A fuzzy strategy was conceived to overcome this challenge based on control of temperature, humidity, and growth rate determined by timedependent measurements of dough depth and width, and in the first step, the process corridor

Figure 7: Fuzzy-control approach from input, fuzzification by fuzzy-sets, fuzzy rule base, defuzzification, and output of the control variables

1. If (BI is normal) and (Temp is low) then (Temp is veryHigh) (1) 2. If (BI is normal) and (Temp is normal) then (Temp is normal) (1) 3. If (BI is high) and (Temp is hight) then (Temp is lowl) (1) 4. If (BI is high) and (Temp is normal) then (Temp is lowl) (1) 5. If (Temp is low) and (Growth is low) then (Temp is veryHigh) (1) 6. If (Temp is normal) and (Growth is low) then (Temp is veryHigh) (1) 7. If (Temp is high) and (Growth is low) then (Temp is high) (1) 8. If (Temp is low) and (Growth is normal) then (Temp is normal) (1) 9. If (Temp is normal) and (Growth is normal) then (Temp is normal) (1) 10. If (Temp is normal) and (Growth is high) then (Temp is normal) (1) 11. If (Temp is hight) and (Growth is high) then (Temp is normal) (1) 12. If (Temp is hight) and (Growth is high) then (Temp is low) (1)

© TUM

INTERVIEW IMAGE PROCESSING APPLICATIONS FOR BAKING PROCESS MONITORING

of the ‘normal’ process was captured and designed. Subsequently, based on this data and expert knowledge, a fuzzy rule base was designed to reflect the expert knowledge. Figure 7 shows a more detailed pathway of the fuzzy control system. First, the temperature, humidity, and object growth rate are fuzzified and taken into the fuzzy control. In this step, the input parameters are transformed to fuzzy values by membership functions. Subsequently, the fuzzy values are processed and, if possible, combined by the fuzzy rule base to determine the fuzzy-ciphered values. In the final step, the concrete output parameters are defuzzified output rules.

The investigations were performed with dough pieces for bread rolls with half of the ‘normal’ yeast content. Two scenarios were examined, the first without and the second with the fuzzy system to control the proofing chamber climate. The recorded relative size of the dough pieces shows the fuzzy control was able to move the process slightly towards the optimal growth of the dough pieces.

Authors

Ronny Takacs, Stefan Steinhauser, Dominik Geier, Prof. Dr. Thomas Becker: Chair of Brewing and Beverage Technology, Technical University of Munich, Freising

Ronny Takacs has studied brewing and trained as a brewer. At the Technical University of Munich, he graduated with a MSc in Brewing and Beverage Technology in 2014. He has worked as a member of the scientific staff in the BioPAT and Digitization workgroup of the Chair of Brewing and Beverage Technology of the Technical University of Munich since 2014. His research focuses on process monitoring, automatization, and sensor development, in particular image processing and spectral image processing for the brewing, baking, and biotechnological industries. E-mail: ronny. takacs@tum.de

Stefan Steinhauser studied Food Technology and Biotechnology at the Technical University of Munich, where he graduated with a MSc in 2016. He has worked as a member of the scientific staff in the BioPAT and Digitalization workgroup of the Chair of Brewing and Beverage Technology since 2016. His research focuses on the development and application of laser-optical measurement systems for the food industry.

Dominik Geier studied Brewing and Beverage Technology at the Technical University of Munich, where he graduated in engineering (Dipl.-Ing.) in 2011. He has worked as a member of the scientific staff of the Chair of Brewing and Beverage Technology of the Technical University of Munich since 2011. His research centers on process monitoring and sensor development, especially ultrasonic measuring systems and imaging methods. He has also headed the BioPAT and Digitization workgroup since 2016. E-mail: dominik.geier@tum.de

Thomas Becker studied at the Technical University of Munich, where he obtained his doctorate in 1995 with his doctoral thesis entitled “Development of a computer-assisted enzyme-integrated flow injection system and its use in biotechnology process control engineering and quality monitoring”. In 2002, he concluded his habilitation on the topic “Management of bioprocesses through modeling and cognitive tools” at the Chair of Fluid Mechanics and Process Automation of the Technical University of Munich. From 2004 to 2009, he was a professor at the Chair of Process Analysis and Cereal Technology at the University of Hohenheim. He became professor of the Chair of Brewing and Beverage Technology of the Technical University of Munich in 2009. He was also Dean of the TUM School of Life Sciences of the Technical University of Munich, from 2016 to 2021.

Further developments will address the baking process to monitor more properties such as crust formation, gas permeability, or possible product failures such as crust cracks that are to be detected during the process. In this way, further process control strategies will be developed to continually increase product quality and prevent faulty production. +++

Acknowledgments The IGF Projects AiF 18123 N, AiF 17735 N, AiF 18906 N, and AiF 20495 N of the FEI were supported via AiF within the program for promoting the Industrial Collective Research (IGF) of the German Ministry of Economic Affairs and Energy (BMWi), based on a resolution of the German Parliament.

References

[1] AiF 18123 N, Intelligente Gärsteuerung: Entwicklung einer intelligenten Gärsteuerung zur optimierten Herstellung von Teiglingen mittels digitaler Bildauswertung und erfahrungsbasierter Fuzzyregelung. Bonn: Forschungskreis der Ernährungsindustrie e.V. (FEI), 2014-2016. [2] G. Dadalı, D. Kılıç Apar, and B. Özbek, “Color Change

Kinetics of Okra Undergoing Microwave Drying,” Drying Technology, vol. 25, no. 5, pp. 925–936, 2007, doi: 10.1080/07373930701372296. [3] R. M. Haralick, K. Shanmugam, and I.'H. Dinstein, “Textural

Features for Image Classification,” IEEE Trans. Syst., Man,

Cybern., SMC-3, no. 6, pp. 610–621, 1973, doi: 10.1109/

TSMC.1973.4309314. [4] AiF 17735 N, Volumetrischer keramischer Brenner: Systematische Untersuchungen zur Einsatzqualifizierung einer innovativen Backofentechnik mit volumetrischem keramischem Brenner (VKB) einstellbaren Wellenlängenspektrums sowie hoher Regeldynamik und Energieeffizienz.

Bonn: Forschungskreis der Ernährungsindustrie e.V. (FEI), 2014-2017. [5] R. Takacs, V. Jovicic, A. Zbogar-Rasic, D. Geier, A. Delgado, and T. Becker, “Evaluation of baking performance by

means of mid-infrared imaging,” Innovative Food Science & Emerging Technologies, vol. 61, p. 102327, 2020, doi: 10.1016/j.ifset.2020.102327. [6] J. W. Goodman, Speckle phenomena in optics: Theory and applications. Englewood, Colo.: Roberts, 2007. [7] D. Briers et al., “Laser speckle contrast imaging: theoretical and practical limitations,” Journal of biomedical optics, vol. 18, no. 6, p. 66018, 2013, doi: 10.1117/1.

JBO.18.6.066018. [8] AiF 20495 N, Analyse des Gär- und Backprozesses durch

Dynamic Laser Speckle Imaging. Bonn: Forschungskreis der Ernährungsindustrie e.V. (FEI), 2019-2021. [9] T. G. Mason, “Estimating the viscoelastic moduli of complex fluids using the generalized Stokes-Einstein equation,” Rheologica Acta, vol. 39, no. 4, pp. 371–378, 2000, doi: 10.1007/s003970000094. [10] Z. Hajjarian and S. K. Nadkarni, “Evaluation and correction for optical scattering variations in laser speckle rheology of biological fluids,” PloS one, vol. 8, no. 5, e65014, 2013, doi: 10.1371/journal.pone.0065014. [11] AiF 18906 N, Zusammenhang zwischen Teigrheologie und

Oberflächeneigenschaften. Bonn: Forschungskreis der

Ernährungsindustrie e.V. (FEI), 2016-2018.

INTERVIEW IMAGE PROCESSING APPLICATIONS FOR BAKING PROCESS MONITORING