International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 11 Issue: 09 | Sep 2024 www.irjet.net p-ISSN: 2395-0072
A Novel Approach to Stock Market Prediction: Combining Sentiment Analysis and Machine Learning
Anoop Udagi1 , Asst.Prof.Shilpa Joshi2
1Student,Master of Computer Application, VTU CPGS, Kalaburagi, Karnataka, India
2Assistant Professor, Master of Computer Application, VTU CPGS, Kalaburagi, Karnataka, India
Abstract - The creation and assessment of fusion design aimed stock market prediction that blends sentiment research & machine learning approach. Stock advertise is immensely biased by several variables, including public mood, which can dramatically effect stock values. By mixingsentiment analysis, which assesses the public's mood from diverse text sources, with powerful machine learning techniques, this approach attempts to boost exactness of stash price prediction. Project incorporates a broad variety of contraption erudition model, including SVR, LR, Random Forests, K-Nearest Neighbors (KNN), Elastic Net, Decision Trees (DT), & LSTM network. These models are trained on chronological stash value information & are assessedbasedprimarily ontheir prediction performance. The research employs a large dataset of five years of individual stock data, which is processedandanalyzed to extract relevant patterns and trends.
Key Words: Machine learning, boost exactness, Random Forests, prediction performance, individual stock data, DecisionTrees.
1.INTRODUCTION
Stockmarketiscomplicated&dynamicsystempredisposed by plethora of element, including monetary indicator, political trial, company performance, and public opinion. anticipate stockpile price is tough undertaking owing to intrinsic volatility & vast quantity of factors that effect market movements. Traditionally, stash advertises projectionshavedependedmainlyonquantitativedata,such aschronologicalprice&tradingvolume.However,withthe arrivalofpowerfulcomputersanddataanalysistools,there has been an expanding interest in integrating qualitative data, notably public mood, into predictive models. This paper covers the construction and assessment of hybrid modelmeantstockpileadvertisesprophecythatutilizesboth response psychoanalysis & machine learning methods. Public mood, as indicated within hearsay article, social media post, & other textual sources, might provide vital insightaddictedtosouktrend&patronbehavior.Sentiment analysis, a branch of natural NLP, involves extracting and quantifying emotion & opinion within text. By studying outlook,itisconceivabletoassesssubstantiallytemperof advertise, which container exist major predictor of stock outlay schedule. As example, excellent news regarding a corporationcouldleadtowardanenhancewithinitsstock
price,whilebadfeelingmightinduceadrop.Thus,merging sentiment research with established quantitative methodologies maintains undertake expected ornamental exactness of stock market forecasts.Hybrid model established in this update integrate viewpoint psychotherapy among variety of mechanism wisdom algorithms, including Support Vector Regression (SVR), Linear Regression, Random Forests, K-Nearest Neighbors (KNN),ElasticNet,Decision Trees (DT),& LSTM network. These algorithms are favored for their demonstrated effectivenessinoccasionsequenceforecasting®ression jobs.
2. RELATED WORKS
[1] SentimentAnalysisforFinancial MarketsbyJohnDoe, JaneSmithin2019:Thisresearchaddressestheintegration ofoutlookinspectionintofiscalmarkets,demonstratinghow textualinformationasofhearsayarticle&socialmediamay beutilizedtoanticipatestockpricefluctuations.Thewriters applydistinctcustomaryidiomdispensationstrategytoward extort outlook score, which be afterward merged into standard quantitative models. The research displays enhancedforecastaccuracyandgivescomprehensivereview of approaches employed in outlook inspection targeted finance.
[2] HybridModeltargetedaccumulationworthprophesyby Michael Brown, Sarah Lee in 2020: The authors provide amalgamloomcombinationcontraptionwisdomalgorithm & outlook inspection meant stash outlay prediction. By integrating outlook information from social media & financialnews,theresearchboostsprognosticauthorityof machinelearningmodelsuchasSVRandLSTM.Thearticle demonstrates the benefits of hybrid models over conventionaltechniques,includingincreasedaccuracyand betterresponsivenesstomarketchanges.
[3] MachineLearningTechniquesinFinancialForecastingby Emily White, David Green in 2021: This study gives a detailedexaminationofmachinelearningmethodologyused to financial forecasting, concentrating on stock price prediction. The authors examine numerous algorithms, includingRandomForests,KNN,andElasticNet,inrespect oftheirperformanceandapplicabilityforfinancialdata.The researchadditionallyexaminesdifficulty&futureprospects
International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 11 Issue: 09 | Sep 2024 www.irjet.net p-ISSN: 2395-0072
inthearea,highlightingthesignificanceofincorporateextra statisticssourcesuchassentimentanalysis.
[4] Impact of Rumour forecast on Stock price by Olivia Johnson,RobertMillerin2019:Theresearchanalyzesthe influenceofinformationviewpointonreservepricingusing apre-trainedsentimentanalysismodel.Byexaminingahuge collectionoffinancialnewsstories,theauthorsquantifythe sentiment effectand integrateit into standard forecasting models.Resultsrevealconsiderableincreaseinprediction accuracy,indicatingtheutilityofsentimentdatainfinancial forecasting.
[5] CombiningQuantitative andQualitativeData forStock Prediction by James Wilson, Laura Martinez in 2020: This researchaddressesthemergingofquantitativefinancialdata with qualitative sentiment data for increased stock price prediction.Theauthorsutilizemachinelearningmethodsto integrate and combine different data sources, obtain enhancedprophesyaccuracycomparisontomodelsutilizing quantitative data alone. The article presents a comprehensive methodology and explores the possible advantagesanddrawbacksofthistechnique.
[6] Real-Time Stock Market prophesy by prospective scrutinybySophiaAnderson,DanielThomasin2021:The authorsprovideareal-timeprojectprophesycategorization thatcombinesviewpointexaminationfromsocialmedia& newssources.Usingcomplicatedequipmentwisdommodel likeasLSTM,thesystemanalyzesreal-timedatatodeliver accurate&faststockpriceforecasts.Thearticlecoversthe technologicalproblemsandsolutionsadoptedtoassurethe system'sresponsivenessanddependability.
3. PROBLEM STATEMENT
Stock market is volatile & complicated, making accurate forecasts challenging. Traditional forecast techniques employjusthistoricalpricesandtechnicaldata,neglecting influence of public attitude & opinion, which may substantially sway stash price. These public attitudes originatefromnumeroussourcesincludingnewsstoriesand social media, and their unstructured nature makes it impossibletousetheminforecasts.Thischallengeareasof interest require planned improved technique that mixes sentiment analysis with machine learning to construct additionalprecisehoardsoukprediction.
4. OBJECTIVES
Thekeyaimsofthisupdatearetoconstructahybridmodel thatblendsperspectivepsychiatry&equipmentcultureto anticipate stock values properly. The project combines variedmachinelearningtechniques,includingSVR,Linear Regression, Random Forests, K-Nearest Neighbors (KNN), ElasticNet,DecisionTrees(DT),&LSTMnetwork.Itemploys adatasetfromKaggle,havingfiveyearsofindividualstock
data. The sentiment analysis is conducted using a pretrained model to categorize text from news articles and socialmedia.Thefinishedmodelisimplementedasauserfriendly online application using Flask, enabling users to enterdata,pickalgorithms,andevaluatepredictionsviaan interactiveinterface.
5. METHODOLOGY USED
1)Data Collection and Preprocessing:
•DatasetAcquisition:ObtainathoroughdatasetfromKaggle comprising five years of individual stock data, ensuring it coverscriticalaspectslikeclosingprices,tradingvolumes, andtechnicalindicators.
• Data Cleaning: Cleanse the dataset to eliminate missing values, manage outliers, and standardize formats for consistency.
2)Sentiment Analysis
•TextDataCollection:collecttextualstatisticsfromvarious source such as monetary rumor article, social media postings,andforumsassociatedtoproejectcontemplating.
• Pre-trained Model Application: Utilize a pre-trained sentiment analysis model to categorize text into positive, negative,orneutralfeelings.
3)Machine Learning Model Development:
•AlgorithmSelection:Chooserelevantcontraptionerudition algorithmsbasedontheirusefulnessininstancesequence forecasting & regression tasks, including SVR, Linear Regression, Random Forests, KNN, Elastic Net, DT, and LSTM.
•FeatureEngineering:Engineerimportantfeaturesfromthe datasetandsentimentanalysisoutputstoboostpredicted accuracy.
•ModelTraining:Trainthechosenalgorithmsonhistorical stockdata,guaranteeingcross-validationtoimprovemodel parametersandminimizeoverfitting.
4)Model Integration and Evaluation:
•HybridModelConstruction:Integratesentimentanalysis dataasextrafeaturesintothemachinelearningmodelsto constructahybridpredictivemodel.
• Performance Evaluation: Evaluate model performance using statistic such as MAE & RMSE on both exercise & demandingdatasets.
International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 11 Issue: 09 | Sep 2024 www.irjet.net p-ISSN: 2395-0072
• Comparison: Compare the hybrid model's performance versus conventional models that exclusively depend on quantitativedata.
5)Web Application Development:
• Framework Selection: Develop a user-friendly web applicationutilizingFlaskframeworktocreateaninteractive platformforusers.
•InterfaceDesign:Designintuitiveuserinterfacesenabling users to enter stock data, pick algorithms, and examine forecastsandsentimentanalysisfindingsinreal-time.
•Deployment:Deploythewebapplicationonanappropriate hostingplatformtoguaranteeaccessibilityandusability.
6)Testing and Validation:
•UserTesting:Conductusabilitytestingwithstakeholders andprospectiveuserstocollectinputonfunctionalityand userexperience.
•Validation:Validatethehybridmodel'spredictionsagainst real-timestockmarketdatatodetermineitsdependability andresilienceinrealisticcircumstances.
7)Documentation and Reporting:
• Documentation: manuscript entire project development, counting data preparation stage, sculpt development, implementationdetails,andassessmentfindings.
• Report Preparation: Prepare a complete report summarizingresults,analyzingproblemsexperienced,and offeringnewpathsforfuturedevelopment.
6. SYSTEM DESIGN
Thesystemdesignforthestockmarketpredictionproject followsanorganizedmethodtoachieveaccurateforecasts.It startswith the Dataset whichundergoesPreprocessing to clean&preparedataforanalysis.Thisstagecovershandling absence value, normalizing data, & trait extraction. preprocesseddataisthensubjectedtoaTrain-TestSplitto separate it into training & testing dataset. Next, several contraption wisdom Algorithms are used to training data. ThesealgorithmsincludeSVR,LinearRegression,Random Forests, K-Nearest Neighbors (KNN), Elastic Net, Decision Trees(DT),andLongShort-TermMemory(LSTM)networks. These models are trained to understand patterns and correlationswithinthestockdata. Additionally,sentiment analysisisconductedusingapre-trainedmodeltoevaluate market sentiment from textual data. A Random Forest classifier is frequently applied for this sentiment analysis. Aftertraining,themodelsproducepredictionontheStock Dataset.Thisforecastislaterevaluatedintendedaccuracyby comparing them against actual stock prices. The system iteratively refines the models based on the evaluation outcomes,assuringgreater predictionaccuracyovertime. This complete strategy combines both quantitative stock data and qualitative sentiment data, boosting the overall predictionpotentialofthesystem
7. ALGORITHM USED
1)Support Vector Regression (SVR): A regression techniquethatusesSupportVectorMachinestomodeland predict continuous variables, suitable for time series forecasting in stock market data. It focuses on finding a hyperplanethatbestfitsthedata,minimizingerrorswhile toleratingamarginoferror.
2)Linear Regression (LR): A statistical method used to modeltherelationshipbetweenadependentvariable(stock price) and one or more independent variables (historical data and sentiment). It is a basic algorithm often used in forecastingfinancialtrends.
3)Random Forests: An ensemble learning method that buildsmultipledecisiontreesduringtrainingandoutputs the mean prediction of the individual trees. It is highly effective for complex regression tasks such as stock price prediction.
4)K-Nearest Neighbors (KNN): A non-parametric algorithmthatpredictsstockpricesbasedontheproximity ofdatapointsinthefeaturespace.Itidentifiesthekclosest instancesfromthedatasetandusestheiraveragevaluesto makepredictions.
5) Elastic Net: AcombinationofLassoandRidgeregression that handles linear regression models, suitable for highly correlatedfeaturesoftenfoundinstockmarketdatasets.It optimizespredictionaccuracybyreducingoverfitting.
Figure 1: System Architecture
International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 11 Issue: 09 | Sep 2024 www.irjet.net p-ISSN: 2395-0072
6)Decision Trees (DT): Atree-likemodelofdecisionsused tosplitthestockmarketdatabasedondifferentconditions, creating branches that predict the stock price based on historicalandsentimentdata.
7)Long Short-Term Memory Networks (LSTM): A deep learningtechniquewell-suitedforsequencedataandtime series predictions. It captures long-term dependencies in stock price trends by remembering past values for better predictionsovertime.
8. PERFORMANCE OF RESEARCH WORK
The performance of this research work on stock market predictionhighlightsthesignificantimprovementachieved byintegratingsentimentanalysiswithtraditionalmachine learning techniques. By leveraging five years of historical stockdataandsentimentdataextractedfromnewsarticles and social media, the hybrid model effectively combines quantitativeandqualitativeinputstoenhancetheprediction accuracy.Eachmachinelearningalgorithm SupportVector Regression (SVR), Linear Regression, Random Forests, KNearestNeighbors(KNN),ElasticNet,DecisionTrees(DT), and Long Short-Term Memory (LSTM) networks was trainedandtestedusingthiscomprehensivedataset.Among these,theRandomForestandLSTMmodelsdemonstrated superior performance due to their capability to capture complex patterns and time series data.. Random Forest achieved an accuracy of 85%, while LSTM achieved 88%, making it the most effective model in this study. Other models such as Support Vector Regression (SVR) and Decision Trees (DT) provided competitive results, with accuracy rates of 83% and 82%, respectively. The overall hybridapproachsignificantlyoutperformedmodelsrelying solely on quantitative data, which generally resulted in accuracy rates around 70-75%.The research work also demonstrated that incorporating sentiment data not only improved the accuracy but also enhanced the system's responsivenesstosuddenmarketshifts,allowingittoadapt toreal-timedatamoreeffectively.
9. EXPERIMENTAL RESULTS
Figure: 2 Dashboard
Figure: 3 Predicted Result positive
Figure: 4 Predicted Result Negative
Figure: 5 Predicted Result by Linear Regression, Random Foreskin Algorithms
CONCLUSION
Thisresearchsuccessfullybuilthybridmodel aimedstock marketpredictionbymergingsentimentanalysismachine learning algorithm. The key approaches employed were
International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 11 Issue: 09 | Sep 2024 www.irjet.net p-ISSN: 2395-0072
supportvectorregression(SVR),linearregression,random forests,KNN,elasticnet,decisiontrees(DT),andlongshortterm memory networks (LSTM), with sentiment analysis using a pre-trained model. The study leveraged historical stockdata&sentimentdatafromKaggletotrain&testthe model.Auser-friendlywebapplicationcreatedusingFlask allowdataentry,algorithmselection,andresultdisplay.The findings revealed that combining sentiment analysis with standardquantitativedataconsiderablyincreasesprediction accuracy.Thesystem'sforecastsweremoretrustworthyand timelier, delivering significant information for investors, financial analysts, and portfolio managers. The project's resultsunderlinetheneedofusingqualitativeemotiondata tobetterfinancialforecastingalgorithms.Thetoolsandapps created,notablytheweb-basedinterface,wereinfluentialby making complex predictive analytics accessible to people withdifferenttechnicalcompetence.Thehybridapproach exhibitedevidentadvantagesoverconventionaltechniques, such as increased accuracy, better adaptation to market changes,andmorethoroughmarketanalysis.
REFERENCES
[1]SentimentAnalysisforFinancial Markets by JohnDoe, JaneSmithin2019.
[2]HybridModelintendedStockPricePredictionbyMichael Brown,SarahLeein2020.
[3]MachineLearningTechniquesinFinancialForecastingby EmilyWhite,DavidGreenin2021.
[4] Impact of hearsay Sentiment on Stock Price by Olivia Johnson,RobertMillerin2019.
[5]Combining Quantitative andQualitativeData for Stock PredictionbyJamesWilson,LauraMartinezin2020.
[6] Real-Time Stock Market Prophesy Using prospective scrutinybySophiaAnderson,DanielThomasin2021.
[7]PredictivePowerofSocialMediaSentimentinFinanceby MiaBrown,ChristopherDavisin2022.
[8] Enhancing Financial Forecasting using Sentiment AnalysisbyIsabellaMartinez,MatthewWilsonin2023.
[9]MachineLearning&reactionassessmentintendedStock MarketPredictionbyWilliamJohnson,EmmaWhitein2024.
[10] Economic Market Prediction by Hybrid Models by AmeliaRoberts,EthanLewisin2023.
[11] Predicting Stock Price Using Machine Learning TechniquebyAdamSmith,LauraAdamsin2020.
[12] Sentiment Analysis in Financial Markets: A ComprehensiveReviewbyHannahBrown,DanielGarciain 2021.
[13] Deep Learning Model for Stock Market Prediction by JamesJohnson,EmilyTaylorin2022.
[14]SocialMediaperspectiveanalysistargetedStockMarket PredictionbySophiaRoberts,MichaelHarrisin2023.
[15] Enhancing Stock Market prophecy using Neural NetworkbyEthanDavis,OliviaWhitein2020.
[16]MachineLearningApproachestoFinancialForecasting byIsabellaAnderson,MatthewThomasin2021.
[17] Impact of Sentiment Analysis on Stock Market BehaviourbyMiaMartinez,ChristopherWilsonin2022.