Sentiment Analysis of Oil and Gas Sustainability Reports using Azure Cognitive Services via Cognitive Search

In my last blog post ‘A Data Scientist’s introduction to Sustainable Finance’ I covered what sustainable finance is, what are the current challenges and current data sources and models being used. With attention on sustainability continuing to grow, alongside the many challenges, many have started to leverage different data sources such as: satellite data, supply chain data from input-output tables, IoT sensors, company databases, contracts, Microsoft Office API’s, and more to understand business processes and the impact of the environment to initiate improvements or compare investment opportunities. In this post I will take the example of sustainability reports to try to understand how sustainability sentiment has changed over time within the oil and gas industry, whilst also exploring how some of Microsoft’s out of the box AI models perform.

Sustainability reports can give insights into how organisations are intending to change their processes and strategies toward greener methods of operation, reducing emissions and contribute to communities and different social groups. When changes in sentiment from sustainability publications are compared over time, it can inform us of which organisations are better to invest in when considered alongside other data, and thus contribute to the long term goal of sustainable finance to invest 600 billion in green investments each year for the next 15 years, to global warming below 2.0 degrees by 2100. However, sustainability reports are self-reported so inevitably biased, despite GRI standards requiring a balance of positive and negative aspect’s of a company’s performance be reported, companies use language which makes it challenging to distinguish between positive, neutral and negative sentiment.

Labelling different sentiments are challenging to establish from a human perspective, let alone training a machine learning model to identify the differences, as demonstrated in a case study by KPMG in which they were asked to build a tool to measure the balance of sentiment in sustainability reports. The KPMG team took 900 sustainability reports across different sectors and identified 15 key topic areas which covered environmental statements (emissions, waste, and water) and social statements (HR, customer service, employee training). Training data (3000 sentences) was manually labelled, which turned out to be particularly challenging as even negative statements are worded positively. For example, statements such as ‘ These incidents are of concern to us and led us to strengthen our efforts to prevent such tragedies in the future’ are ambiguous as whether to classify as negative, neutral or positive. Additionally, the sentiment classes were highly imbalanced with 3% of sentences classified as negative, 21% classified as positive and 76% classified as neutral. In this instance, many typical NLP models such as: Rossetta, Stanford Sentiment Treebank and self-trained neural networks failed to show adequate performance. BERT turned out to be the most performant due its ability to understand subtle context and accurately classified 71% of negative sentences, 80% of positive statements and 94% of neutral sentences. When dissecting where the BERT model did not classify correctly it was illustrated that confusion was always between neutral and negative or neutral and positive sentences, also inconsistencies in human labelling were also considered. To address this concern, 3 other people were asked to label sentences from sustainability reports. It was shown that there were differences between how individuals classified each of the sentences but also there were a small amount of inconsistencies in their own labelling.

The oil and gas industry is often depicted in a negative frame of light, and quite rightly so with industrial disasters such as the Deepwater Horizon oil spill which discharged 4.9million barrels of oil discharged into the Gulf of Mexico which catastrophically effected thousands of marine species and habitats (Beyer et al, 2016). However, many oil and gas companies have decided to ride the waves of the sustainability disruption and give themselves a green makeover. BP, for example, is actively shrinking its oil and gas business and increasing offshore wind power and developing solar and battery storage. BP intend to invest and build renewable energy capacity of 20GW by 2025 and 50GW by 2030. These renewable investments are expected to yield a return in range of 8-10%.

So how much insight into the oil and gas sustainable trends increasing over time do sustainability reports give us? To explore this, I have taken sustainability reports from BP and Royal Dutch Shell from 2013 to 2019 and uploaded to an Azure Data Lake and connected to Azure Cognitive Search. Through Cognitive Search, Azure Cognitive Services AI enrichments were leveraged to carry out key phrase extraction, named-entity-recognition (NER- names, locations, organisations), extract PII data, and classify sentiment of sentences and saved as a knowledge store output and wrangled with python and surfaced with Power BI.

Reference: Knowledge Stores in Azure Cognitive Search- Microsoft Developer

Before exploring the process and outputs, I will briefly cover the different Azure services and tools used in this use case. For a great overview checkout this episode of the AI show.

What is cognitive search?

These days people are very used to the search capabilities found on Google, Bing, Travel websites, job websites, etc. Search boxes are found in most applications today. Rather than having to build complex search functions across data developers can now use Azure Cognitive Search to upload all types of data, structured and unstructured (powerpoint, pdf’s, word documents, scanned documents, etc). These documents are then cracked open to extract text, images, tables, metadata and indexed. Once indexed, the API can be leveraged to integrate search capabilities (keyword search, faceting, geospatial support, auto-complete, proximity search, synonyms, cognitive skills, etc) in various services. Cognitive Search is a ‘search-as-a-service’ offering and an AI first approach to content understanding, allowing billions of documents to be searched for specific phrases.

In addition Cognitive Search integrates really well with Cognitive services allowing you to further enrich your data. Additional enrichments include the ability to: identify names entities, key phrases, PII data, extract different languages, and classify sentiment.

If organisations or use cases require custom searches, you can also create your own custom skills and integrate with the cognitive search pipeline. For example a construction company may want to be able to identify specific materials (timbre, steel, etc), or a legal team may want to quickly identify documents where certain clauses are mentioned. These types of custom skills can easily be developed with tools such as Azure Machine Learning or Azure Functions and the endpoints integrated with the Cognitive Search pipeline.

What is cognitive services?

Azure Cognitive services are a set of API’s that allow developers to harness the power of pre-built Microsoft AI models to solve a range of use cases in the areas of vision (computer vision, face recognition, form recognizer, video indexer), speech, language (language understanding, QnA maker, text analytics, translator, immersive reader) and decision making (anomaly detector, content moderator, metrics advisor, personalizer).

What are Knowledge Stores?

Knowledge stores are a relatively new addition to Cognitive Search which allow you to project all additional enrichments and surface projections in other consumable environments like Power BI. Knowledge stores can make three types of projections: tables, files, and objects and can be easily previewed with storage explorer. When knowledge stores are projected to an Azure Table, enriched outputs can easily be consumed by Power BI, and Microsoft have even provided a useful Power BI template which breakdown insights into a few reports: general analytics (key-phrase word cloud, geographical regions of interest, proportion of different entity types mentioned, the actual content, etc), graphical representation of key phrases, graphical representation of different entities, and representation of images within documents. To build upon this report the knowledge store Azure Tables can be wrangled further in python using the Azure SDK.

Exploring BP and Shell Sustainability Reports from 2013 to 2019

The following experimentation has been carried out using the cognitive search wizard via the Azure Portal and then wrangled further with python

but it is possible to initiate cognitive search programmatically too. Once the BP and Shell sustainability reports were uploaded to the Azure Data Lake, Cognitive Search and Cognitive Services resources were created in the sustainable finance resource group. Open the Cognitive Search resource and select import data, which will take you to the wizard.

Next, connect Cognitive Search to the datastore, by providing relevant information such as the connection string.

Once the datastore has been connected, the next step allows for a Cognitive Services resource to be linked and additional enrichments to be specified based on how the content is to be considered. In this instance, we are particularly interested in sentiment analysis, so content is considered at the sentence level.

At this stage, the enriched output can also be specified to be projected to a knowledge store.

Finally, the option to customise the indexer and specify scheduling options if more data is to be added periodically can be defined. Once the progress of documents being progressed can be seen on the main Cognitive Search page, and will be able to be viewed and queried once complete.

Following completion the Power BI template was loaded and connected to the knowledge store. The first page of the report highlights on word cloud that common references are BP and Shell, but also terms such as operation, people, society, etc which is in line with the nature of sustainability reports. In addition, the majority of the entities recognised relate to actual quantities or values, and as expected the geographical locations are diversely spread.

The graphical representation of key phrases also provides some interesting insight. In particular, it is surprising renewables is somewhat isolated given the press reports around oil and gas companies heavily investing in this area.

To gain further understanding around the classification of sentiment in oil and gas sustainability reports, the knowledge store Azure Tables related to sentiment and some metadata tables were read into python using the Azure SDK. Alternatively, you can export the Azure table directly from storage explorer.

The metadata was joined to the sentiment table and the organisation and year were extracted out as separate columns. There were 7 sustainability reports for BP and 7 for Shell which were tokenised into 20,644 sentences. The sentiment score returned ranges from 0 to 1, which can be further categorised to reflect positive, negative and neutral sentiment. The average sentiment score for sentences from sustainability reports for both BP and Shell remained around 0.5 across time.

number of sentences per org and year.png

When the sentiment score was categorised (negative (0 - 0.3), neutral (0.31 – 0.7), positive (0.71 – 1)), it was evident that there was little variation in distribution of different sentiments across time for both BP and Shell. The majority of sentences were classed as neutral, with a small proportion of sentences classed as negative or positive. This distribution matches the distribution patterns of the KPMG manually labelled data, which provides some confidence. There is a slight trend showing the proportion of Shell positive sentiment sentences is slightly increasing through time.

distribution of sentiment across time.png

For a more interactive experience the wrangles data can be read into Power BI. Similar visualisations, which demonstrate the sentiment class distribution over time for BP and Shell. Likewise the average sentiment score across time per organisation can be seen to be consistently around 0.5 across time. The donut chart shows there were slightly more BP tokenised sentences than Shell, and the table shows the raw data which the sentences could be added.

How well is sentiment captured? Whilst the distribution of sentiment captured through Cognitive Services bears some resemblance to similar use cases, the accuracy remains questionable. Through some very basic eye-balling of sentences with high sentiment score in storage explorer, it appears numeric values are given a higher sentiment score, as well as other sentences that would otherwise appear neutral.

In conclusion, Azure Cognitive Search and Cognitive Services provide some powerful tools to get quick insights which would normally require a significant data science effort to process, clean and model data. Whilst some of the performance is questionable, this initial insight provides a good basis to iterate over and compare results of new models. Sustainability reports remain a questionable data source for evaluating the ESG performance given the lack of standardisation and audit of self-reported facts.

References

PyData: KMPG Sentiment Analysis on Sustainability Reporting

Environmental effects of the Deepwater Horizon oil spill: A review (Beyer et al, 2016)

BP Gas and low carbon energy

Big Oil’s Green Makeover

BP Sustainability Reports

Royal Dutch Shell Sustainability Reports

Azure Cognitive Search:

What is Azure Cognitive Search?

Cognitive Search – Azure Search with AI | Azure Friday

Search Results cognitive search (jj09.net)

(266) Knowledge mining with Azure Cognitive Search - YouTube