Kng JHJ and Fadeyi MO (2022). Development of a mobile phone-based application for detecting and communicating building defects Built Environment Applied Research Sharing #10 (Technical Note). ISSUU Digital Publishing.
© BEARS reserves the right to this applied research article
Sustainable Infrastructure Engineering (Building Services) Programme, Singapore Institute of Technology, 10, Dover Drive, Singapore 138683, Singapore
*Corresponding author’s email: fadeyi.moshood@singaporetech.edu.sg
Accurate detection of building defects and communication to stakeholders with prudent use of invested resources, e.g., time, manpower, and money, increases value delivery to all stakeholders involved in the building delivery. This paper presents a prototype developed to facilitate enhanced value delivery for building defects inspection and communication during building construction and management. With image classification and cloud database technologies, this study proposes a mobile (phone based) machine learning application to raise awareness of building defects located in buildings and the implications to motivate building stakeholders to promptly reduce the risk level. The benefits of the developed prototype can further be enhanced when the developed prototype is integrated with drones. The adoption of the developed prototype will vary depending on the cause and context of the defined building defects detection and communication problems to be addressed.
Keywords: Building defects, Defects detection, Defects communication, Mobile cloud computing, Machine learning 1.0 INTRODUCTION
In Singapore, under Section 28 of the Singapore Building Control Act 1989 (2020 Revised Edition),allbuildingownersarelegallyrequiredtoengageaProfessionalEngineertoconduct PeriodicStructuralInspectionstoensurethatstructuraldefectsduetothelackofmaintenance can be detected and rectified early to keep buildings structurally sound. This legislature has ensured that buildings, after construction, are maintained to a certain standard that does not poseasasafetyhazardtooccupantsandmembersofthepublic.
BasedontheguidelineslaidoutintheBuildingControlAct,thefollowingactionsarerequired
for the successful conduct of a Periodic Structural Inspection. The Building and Construction Authoritywill informbuilding ownersbyserving aNotice for Periodic Structural Inspections of buildings. Secondly, building owners will need to appoint a Professional Engineer in the CivilandStructuralengineeringdisciplinetoconductastructuralinspectionoftheirbuildings.
Lastly,theProfessionalEngineermustsubmitareporttoBuildingandConstructionAuthority, andthebuildingownermaychoosetoactontherecommendationsoftheProfessionalEngineer ifnecessary.
To improve the inspection process, government agencies and research institutions have been exploring ways to develop and deploy innovative robotic solutions to (a) standardise how building inspections are being performed; (b) improve the quality of inspections down to the microscopic level; (c) minimise safety hazards by ensure that no area is unchecked; and (d) improving the productivity of building inspections (Shariq and Hughes, 2020). Ultimately, using robotics aims to reduce human intervention in the inspection process and increase the valueofallstakeholders involved.Digitalsolutions,suchastheQuicaBot, whichexistsinthe market,performbuildinginspectionstodetectdefectssignificantlyfasterthanbuildinghuman inspectors (Yan et al., 2018). However, robotic solutions are not often available on many construction sites or during building management due to their high costs. Building configurationalsoposesalimitationtotheusageofmanyroboticsolutions.
On the other hand, mobile phones usage is very common and can be carried to any part of buildings or construction sites. Thus, the use of a mobile phone application for accurate building defects detection is financially friendly and easily deployable with little or no constraints.Thisstudyaimstodevelop amobilephone-based applicationforcheap,easy,and accurate building defects detection and communication to provide timely information and knowledgetoallbuildingstakeholderstomotivatethemtotakethenecessaryactionstoprotect themselvesoreliminatehazardsintheirbuildingsorbuildingconstructionsites.
This section provides information on the premise of the prototype's development. It also providesinformationonthedemonstrationofthedevelopedprototype.Avideodemonstration ofthedevelopedprototypecanbefoundinthesupplementaryinformationsectionofthispaper.
Theproposedmobilemachinelearningapplicationaimstoraiseawarenessofbuildingdefects that may lead to catastrophes in any built environment to motivate decision makers to act promptly to assess, manage, and avoid risks. The system consists of the TensorFlow Lite Androidimageclassificationlibrary,androidmobiledevices,andacloudstoragesystem.The TensorFlow library contains logic and knowledge of machine learning networks trained to solve problems in the field of image classification, object detection, smart reply, pose estimation or segmentation. The versatility of the library allows programmers to import the TensorFlow models onto mobile or Internet of Things (IoT) devices. For the android library, TensorFlow uses the device's rear-facing camera to classify images, and this system will be slightlymoreconsistentthanvisualinspection.
The proposed mobile machine learning application is aimed at minimising human effort in building inspections, streamline information and knowledge dissemination to building stakeholders,andmotivatebuildingownerstoacttoassess,manage,andavoidrisks.Userscan use the android device's rear-facing camera to record images of building defects, and the applicationwillautomaticallyidentifythedefectsandpredictdifferentmodesoffailures.These potential failures will be stored in a cloud database, and the information will be disseminated toallbuildingoccupantsusingtheapplication,allowingthemtounderstandthepotentialrisks, however small, of being in the building. Each component of this proposedsolution, including the Android mobile device, TensorFlow Lite Android image classification library, and Cloud Firestore,arebrieflydescribedbelow.
Themobile device used to simulate the application is the Samsung Galaxy S9, whichruns on the Android operating system. The mobile application was developed as the front-end user interface, and primarily allows users to (a) identify building defects using the rear-facing camera;(b) sendinformationtothe database;and (c) receive information on building hazards in real time. The programme was developed using Java in Android Studio, an Integrated DevelopmentEnvironmentforandroidapplicationdevelopment.Forthisapplication,twomain libraries arerequired, includingthe TensorFlow Lite Androidimage classification libraryand Cloud Firestore library, both of which are well documented and readily available for
implementation.
TensorFlowLiteAndroidimageclassificationlibrary isamachinelearningnetworktrainedto solve image classification and object detection problems. The network has pre-trained TensorFlow models that can solve various machine learning problems, with a built-in transfer learningfunctionthatallowsdeveloperstotakeatrainedmodelandre-trainittoperformother tasks.Re-trainingisaconvenientmethodofcreatingnewcategoriesofimagesasitrequiresless time and data than training a model from scratch. The image classification model can be retrained throughout the development and deployment of the application to recognise new categories of the product, and for this application, this includes structural, mechanical, and electricalcomponents.
CloudFirestoreisaflexible,scalable,non-sequential(NoSQL)databaseusedtostorebuilding defect information and contact details of all decision makers and building occupants for informationandknowledgedissemination.Non-sequentialdatabaseshavelittletonostructure, and information can be added and removed freely from all data fields without causing errors. This non-sequential datastoragecapabilityisessentialfor the development and growth of the application, as more information gathered from the machine learning network means more fieldsarerequiredtostoretheinformation.CloudFirestorereadsandwritesdataasJavaScript Object Notation (JSON) in the database, but this can easily be translated to human reading languagewhenwrittenontheuserinterface.
Cloud Firestore instantly disseminatesany new informationstoredin the database to building stakeholders as a real-time database.This real time data transfer capability is essential for the operation of the application, as one of the success factors for this application is the ability to disseminate quality information and knowledge to all building stakeholders promptly. The valueofthisapplicationliesinitsabilitytodisseminateinformationandknowledgetobuilding occupants in the right place at the right time, with little to no input required by humans, but thiscanonlybeachievedifthefunctionsofCloudFirestoreissuccessfullyintegratedwiththe TensorFlowLiteAndroidimageclassificationlibrary.
Figure 2 shows the mobileapplication control logic with the TensorFlowand Cloud Firestore implementation. For the mobile application to disseminate information to building stakeholders, there is a need for all stakeholders to register themselves as a form of identification. After the registration is complete, stakeholders will be categorised into two groups, namely decision makers and building occupants. An admin must further authorise stakeholders to register as decision makers. Upon successful login, both categorises of users willhaveaccesstothebuildingdefectidentificationfunction,allowingstakeholderstoidentify and report building defects toCloudFirestore. Additionally, the dissemination of information andknowledgetobuildingstakeholdersisafunctionaccessiblebyallbuildingstakeholders.
Decision makers have access to a wider array of functions to simplify decision making processes, making it more convenientto effect changes. These functions include (a) viewinga list of building defects; (b) comprehensive breakdown of all building defects; (c) list of contractorsforeachbuildingdefects;and(d)defectresolutiontoinformbuildingoccupantsthat the problem has been rectified. Thepurpose for building occupants inusing this application is to be informed on the condition and safety of the building. Thus, the functions that they can access are geared towards information and knowledge dissemination. These functions include
(a)viewingalistofallbuildingdefectsand(b)basicinformationregardingeachbuildingdefect.
Using this information, building occupants can decide on their course of action for selfpreservation.
All functions provided by this mobile application can be classified into one of three different phases,includingtheinput,communication,andoutputphase.Functionscategorisedunderthe inputphaserequireloggingoroverwritinginformationonCloudFirestore,includingidentifying building defects using TensorFlow and resolving issues. Functions categorised under the communicationphaseautomaticallydisseminateinformationwithouthumanintervention,such asbasicinformationonbuildingdefectsandinformationdisseminationtobuildingstakeholders.
Functions categorized under the output phase allow users to take necessary actions, including thefullbreakdownofbuildingdefects,viewingbuildingdefects,andcontactingthecontractor.
The development of the mobile machine learning application can be divided into three subcategories,including(a)frontendservicesfortheuserinterface;(b)backendservicesusing Cloud Firestore; and (c) Google Cloud Platform for the image classification training and verification. The frontend and backend services will be explained as a single entity, and a separatesectionwillbededicatedfortheimageclassificationlibrarydoneusingGoogleCloud Platform.Forclarity,alluserinterfacesforthemobileapplicationwillbereferredtoasactivity inthissection.
Before users can use the application, there is authorisation and authentication process, which are handled using the register and login pages. Asseen in Figure 3, users willfirst land on the login page to log in to the application and access the functions. Users without an account will be required to register for a new account under the register button. New users can register for an account in the register activity using an email and a password. Email authentication is preferredcomparedtoanonymousauthenticationbecausethereisaneedtostoretheemailused to register the building occupants so that emails can be sent to them whenever a new building defectisreported.Thereisnoneedtostorebuildingoccupant’sinformationastheverification ofthedefectreportisautomaticallydonebytheapplication.
Figures 4 to 6 show the interaction between the register activity, the authentication database, and the list of users on Cloud Firestore. Once a user account is successfully created, basic information such as the email, date of account creation, and last login date are recorded in the authentication database. User passwords are encrypted and inaccessible to developers for security reasons. Upon verification by the authentication database, a separate function is used tocreateaninstanceoftheuserintheFirestoreDatabase,asseeninFigure6.Auserdocument contains the email used to register the account and the user's default role. A user of the application can be classified as either a “user” or a “admin”, but another admin must do the upgradefromusertoadminthroughthedatabase.
Figure5.AuthenticationDatabase
Figure6.UsercollectiononCloudFirestore
Upon successful authentication, users will be redirected to the home activity, containing a list ofdefects.
Asthehomeactivityisrendering,afunctioniscalledtoretrieveandstoretheuser'srolelocally. This function will run in the background whenever the user logs into the application to constantlyupdatetheuserpermissioninthedevice.Thiscallmustbemadebeforealltheactivity startsbecausethebehaviourandlayoutoftheapplicationchangesdependingontheroleofthe users,whichwillbefurtherelaboratedbelow.
AsseeninFigure7,eachbuildingdefectisstoredinacardlayout,andallthebasicinformation, includingthedefect,size,length,date,andpriority,areshowndisplayedonthecard.Themost importantaspectofthisfunctionistoarrangethedefectbasedontheirseverity.Thepriorityof eachdefectisdeterminedbythesizeandlengthofthedefect,andthedefectswithhigherrisks willbeprioritised.Ashighlightedintheredbox,thecardsareorderedbasedontheirpriorities, withthehigherprioritydefectbeingplacedatthetopofthelist.Theprioritytextcolourcoded foreasierreferenceandidentification.
Therecyclerview library is used to populate thelist of buildingdefects on the home activity. Theflexibilityoftherecyclerviewmakesiteasytodisplaylargesetsofdataonascrollinglist efficiently.Thelibraryisgreatforhandlinglargedatasetsasitcanlisten,inrealtime,forstate changesandperformthenecessarystepstomodifythestateofthepage.Statechanges,inthis case, refer to the addition, modification, or removal of any document in Cloud Firestore. For
example, when a building occupant reports a new building defect through Device A, all building occupants can see the new defect on their device without using the pull to refresh function or restarting the application, thus preventing users from using the pull to refresh functionmissinganyupdates.
The real time capabilities of Cloud Firestore is crucial for timely information transfer, which is an important factor for providing quality information. The home activity is programmed to read all the information from the “BuildingDefects” collection in Cloud Firestore, as seen in Figure8.Thedatabasehierarchy forthisapplicationcontainsthe collection,whichisthebase directory,followedbythedocument,whichisthesubdirectory.ThefilepathofFigure8shows “/BuildingDefects/1623229573278”, which means users are currently accessing document 1623229573278inthe“BuildingDefects”collection.
Figure8.BuildingdefectcollectiononCloudFirestore
InformationinthedatabaseisstoredasJavaScriptObjectNotations,whicharekey-valuepairs used to extract certain information from each document. Figure 9 shows the key-value pair of theabovedocument,andeverythingbeforethecolonisthekeys,andeverythingafterthecolon is the respective values. For this example, the “component” is the key, and “cement cracking” is the value. Essentially, there is no limitation to the value, except that it must conform to the primitive type, such asstring orinteger. However, the key mustbe uniqueas the behaviour of thedatabasewillcauseittooverwritetheinitialvalueofanykeytoanewvalue.
Anotherfunctionalitythatprovidesmoreinformationandknowledgeonthebuildingdefectsis thehistoryactivity,asseeninFigure10.Userscanaccessthispagebyclickingonthebuilding defectlistitemonthehomeactivity.Withinthehistoryactivity,userswillbeshowninformation onthecatastrophesthathaveoccurredduetothedefects,includingtheimages,thedescription, and the date of occurrence. This function aims to motivate decision makers to take action to preventacrisisandbuildingoccupantstotakeactiontoprotectthemselves.
TheFirestorecollectionassociatedwiththehistoryactivityusesthekey-valueofthe“defect”, asshowninFigure9.Figure11showsasampleofthe“Large”defectcollectioninthedatabase, with information such as the building that failed, the date of occurrence, the description, and animageoftheaftermathofthecatastrophe.Similartothecardlayoutwithdefectinformation, the card layout in Figure 10 uses the recycler view adapter to display the value of each key
from the document. Each key in Figure 11 has been displayed at a specified location on the card,asshowninFigure10.
The breakdown activity is used to contact the supplier and remove the building defect data if the issue has been resolved. These are top-level functions that should only be accessible by admins. For non-administrative users, the application will not render the default page except foratoastinformingthemthattheyarenotanadmin,asseeninFigure12.
The application renders a dropdown menu for administrative users and calls for all building defects currently stored in Cloud Firestore, as seen in Figure 13. For identification purposes, each dropdown item includes the defective component and the document ID of the building defect.Initially,theactivityisemptyexceptforthedropdownmenu,butmoreinformationwill bepopulatedonthescreenwhenanitemisselectedfromthedropdownmenu.
Afterabuildingdefectisselectedfromthedropdownmenu,thepagerendersalltheinformation ofthatbuildingdefect,asseeninFigure14.Whenfullyrendered,thepagecontainsallthebasic informationofthebuilding defectasstoredinCloudFirestore,with theaddition ofthecontact supplieranddefectresolutionbuttonstosimplifytheworkflowfordecisionmakers.
Thecontactsupplierbuttonislinkedtothe Usercollection onCloudFirestore, containingthe supplier information. Upon clicking the contact supplier button, an automatic email is sent to the maintenance or defect rectification team to inform them of the defect and prompt them to opentheapplicationtoviewmoreinformationonthedefect,asseeninFigure15.Thepurpose of this function is to reduce the amount of time decision makers spend looking for and
contacting the appropriate vendors to solve the issue, thus increasing the value of the application.
Afterthedefecthasbeenresolved,decisionmakerscanclickonthedefectresolutionbuttonto removethebuildingdefectfromthe“BuildingDefects”collection.Figure16showstheupdated homeactivityafteradefecthasbeenremovedfromthecollection.Figure17showstheupdated “BuildingDefects”collection,whichshowsonlyoneitemascomparedtotheoriginalcollection fromFigure8.Theredboxdemarcatingthetimeshowsthattheinformationisupdatedforevery devicealmostinstantaneously.
Figure16.Updatedhomeactivityupondefectrectification
Figure17.Updatedbuildingdefectscollection
Thepurposeof the report activityis to allow all building occupants to report building defects wherever they are. The report activity simplifies the building inspection process by using machinelearningtoclassifyandlabeldefectivecomponentsaccurately.Beforedevelopingthe function on the mobile application, AutoML Vision by Google Cloud Platform was used to trainthemachinelearningmodel,whichproducesseveraloutputfilesthatneedtobeintegrated into the application's assets folder. 100 images of each unique label were designed to be uploaded onto the Cloud Platform for training and evaluation to train the machine learning modelandgenerateaTensorFlowLitefilefortheapplication.Theimagesused forthemodel havealengthandwidthof244pixelseachtooptimizememoryandnetworkparameters.Figure 18showsasampleimageofboththeimagesthatwereuploaded.
Thetypeofclassificationusedforthismachinelearningmodelisthemulti-labelclassification, where each label represents a different classification task, but the tasks are somehow related. Before uploading, all 200 images are grouped based on the defect's size, and each file had a specific naming convention, as shown in Figure 19. The machine learning model uses this namingconventiontotrainandtesttheaccuracyofthemodel.Intermsoftraining,validation, andtesting,AutoMLVisionuses80%oftheimagesfortraining,10%forvalidation,and10% fortesting.OntopoftheTensorFlowLitefile,adictionarytextfilecontainingtheuniquelabels is also generated after the model is trained. This text file, which must also be integrated into
the assets folder, will be used to classify all images taken by the building users and display themontheapplication.
After the training was completed, the remaining images that were not used for training or validation were used to evaluate the trained model. Table 2 shows the evaluation results, as presented on the Cloud Platform, after training the model. Out of the 20 images used for evaluation,themodelaccuracycanbesaidtobe100%basedontheoutputsofboththeprecision and recall. More testing was done on the mobile device after the files were successfully integratedtoensurethemodel'saccuracy.
Table2:Evaluationresults(ExtractedfromGoogleCloudPlatform)
Actiontaken Value
TotalImages 180
TestImages 20
Precision(Ahighprecisionmodelproduces fewerfalsepositives) 100%
Recall(Ahighrecallmodelproducesfewer falsenegatives) 100%
Netron,aviewerforneuralnetworksandmachinelearningmodels,was usedtounderstandthe machine learning model better. Created by Lutz Roeder, this versatile, cross-platform tool was used to visualize, inspect, and communicate the architecture of the model. Figure 20 shows a snippet of the machine learning model during training. Based on the architecture presented in Netron, the Convolutional Neural Network (Convolution 2D and Depthwise Convolution 2D) wasusedtoassignimportancetovariousaspectsofanimage,suchasedgesorcolors,tobeable to differentiate one object from another. The rectified linear unit (Relu) was the activation
functionusedforthismodel.Therectifiedlinearactivatefunctionisapiecewiselinearfunction thatreturnszeroforanynegativeinputandreturnstheinputforanypositiveinput.Thisactivation function is commonly used for many types of neural networks because it is easier to train and oftenachievesbetterperformance.
Figure 21 shows the architecture of the machine learning model after all the layers are trained and connected. Machine learning bias is the difference between the average prediction of the model and thecorrect value that the application is trying to predict. For this model, the bias is low,suggestingthatthereisalowerrorontrainingandtestingthedatasets,whichverifiesthe results shown in Table 2. After which, the memory requirements and computational cost are dramatically reduced through quantization. The scoring, or prediction process, is when the machinelearningmodel generates a value based on thetrainedmachinelearningmodel,using thetestimagesreserved.
AfterinspectingandverifyingthemachinelearningmodelthroughNetron,theTensorFlowlite fileanddictionarytextfilewereintegratedintotheassetsfolderofthemachinelearningmobile application. Figure 22 shows the layout of the report activity, containing an image button, a classifybutton,anda reportbutton.To report abuildingdefect,userswill first clicktheimage button and snap a picture ofthe building defect. Next, they will clickon the classify buttonto call the predictive function. Finally, they will click the report button to upload the predicted
building defect into the “BuildingDefects” collection in Cloud Firestore. The text above the imagebuttonchangesaccordingtotheworkflowtoprovideinstructionsforusers.
Figure23showsthereportactivityaftertheuserhastakenavalidpictureandtheclassifybutton has been clicked. The text field above each image changes based on the unique labels in the dictionarytextfile.Classificationofthebuildingdefectsvalidatesthattheimagetakenisindeed abuildingdefectandshouldberectified.Thisremovesthehumanaspectofabuildinginspection astheapplicationcanquicklyassessanddeterminethedefectsandinformtheuserinrealtime.
When the report button is clicked, the application parses the unique label and sends the information as key-value pair to Cloud Firestore. Additionally, an email is sent to inform all
registered building occupants of the existence of a new building defect to update them on the riskof beingin the existing building. This emailfunction reusesthe code fromthe breakdown activitytoreducecodemaintainability.
This technical note only reports the development of the initial stage of the prototype. Further development and testing of the prototype in real-life settings will be a subject of future endeavour.
Thepurposeofthisstudywastodevelopamobile(phonebased)machinelearningapplication that can (a) perform timely classification of building defects; (b) disseminate information and knowledgetoallbuildingoccupantsforthepurposeofriskavoidance;and(c)motivatedecision makerstoconductproperriskassessmentandmanagementtoprotectthevulnerable.
Intermsofthedevelopmentoftheapplication,twolimitations,includingthesizeofthedatabase and the time taken to train the machine learning model, might hinder the improvement of the application. Basic information dissemination processes, such as sending email, are still relatively quick because of the small number of users. However, as the application grows, the application will take more time to send out emails, which might defeat the purpose of the application. Additionally, the time taken to train machine learning models will increase as the number of defects in the database grows. The scalability of the mobile machine learning applicationisessentialforfuturedevelopment.
The benefits of the developed prototype for accurate detection of building defects and communication to stakeholders with limited invested resources, e.g., time, manpower, and money, to increase value delivery to all stakeholders involved can further be enhanced when integratedwithdrones.Itisimportanttonotethattheadaptionofthedevelopedprototypewill vary depending on the cause and context of building defect detection and communication problemstobeaddressed.
ThesupportoftheSingaporeInstituteofTechnologyincarryingoutthisappliedresearchstudy
isgratefullyacknowledged.Informationreportedinthistechnicalnoteispartoftheworkdone by Mr. Jason Kng as part of his MEngTech capstone project in the Sustainable Infrastructure Engineering (Building Services) programme. Dr. Moshood Olawale Fadeyi supervised the project.Dr.Fadeyialsocontributedtothedevelopmentofthisarticle.
BuildingControlAct1989 (2020RevisedEdition).TheStatuteofthe RepublicofSingapore. Prepared and published by the law revision commission under the authority of the revised editionofthelawsact1983.
Shariq, M. H., and Hughes, B. R. (2020). Revolutionising building inspection techniques to meetlarge-scaleenergydemands:Areviewofthestate-of-the-art. RenewableandSustainable EnergyReviews, 130,109979.
Yan, R. J., Kayacan, E., Chen, I. M., Tiong, L. K., and Wu, J. (2018). QuicaBot: Quality inspection and assessment robot. IEEE Transactions on Automation Science and Engineering, 16(2),506-517.
Clickthelinkbelowtoviewthevideodemonstrationofthedevelopedprototype. https://www.dropbox.com/s/b9db90ng834yotm/Video%20Demonstration%20of%20the%20P rototype.mp4?dl=0