Insightful consulting is built on good data; good data drives decisions. But like bonanza grades, good data can be hard to find. A strategic investment to cultivate data on the front end of a major study can unlock the full value of dollars spent on consulting by utilizing budget thinking about the data: thinking about the relationships between different datasets, exploration of insight, and the pursuit of serendipity rather than spending hours on the agony of number manipulation and quality checking.
On a recent mine closure project, the need to cultivate data arose from a multi-decade legacy of hydrology, hydrogeology, meteorological, and water quality data. The dataset was unruly in size and surly in disposition, in other words, a drain on resources to manage. To maximize client value the data were “cultivated” by a process of data collection, post-processing, quality checks, and visualization using a web-based data analytics platform to allow synthesis of multiple data streams.
Constant Change
Historically the mining industry has been slow to adapt to technology. To cultivate data, you must enter a nebulous modern world and leverage the tools of the data age: cloud computing, cloud storage, and data analytics. We have all most certainly been exposed to buzzwords like “big data”, but what is data cultivation? The words “data cultivation” invoke a notion of taking a collection of singular data and developing them into a collective commodity. A commodity that can be used to support actions and inform decisions.
The era of automated data collection and data storage has led to an exponential increase in the volume of data to manage. So much so we now have “data gathering engines”, for example. As the sophistication of data gathering increases and the volume of data swells, our familiar methods for dealing with datasets have become cumbersome (Microsoft Excel), and outdated (Microsoft Access?).
A variety of third-party data management platforms have risen in recent years to tackle the challenges posed by big datasets and provide robust solutions for complex data management problems. Most often we see this solution adopted by capital generating operations, such as mines with complex processing and water treatment monitoring needs. Third-party platforms can be a good option, but for some clients, this option is not preferred due to cost or hesitance to commit to a proprietary platform, among a variety of reasons.
In capital recruiting projects, such as mine closure, we have seen a need for off-the-shelf, cost-effective, and nimble data management solutions. An upfront investment by a small team of technical experts and data scientists can cultivate a dataset and democratize the data archive. This is quickly becoming the standard of practice for complex projects.
Navigating the Maze
Cultivating data is the process of converting raw data into data that drives decisions through a data management process: data collection and ingestion, transformation, and synthesis. Synthesis comes from investing brainpower into understanding the data and that’s where consultants provide value. But what about the tedious collection/ingestion/transformation part? Can we skip that part and just stick with excel sheets? In short, No. SRK relies on a mix of data scientists and consultants who leverage technical expertise and cloud computing to collate large datasets into practical databases in a shareable format.
SRK has developed a flexible methodology to cultivate data for unique mine sites using automated data collection and off-the-shelf cloud services. Although there are many available tools, an example of one workflow we use is:
- Automated data collection using cell networks, satellite, radio, WiFi, etc. for large arrays of sensors (see an example at Long Canyon)
- Automated upload of data files to cloud storage
- Data transformation/post-processing (Azure Data Factory is one example)
- Load data into cloud database
- Visualize data using web-based data analytics platform (i.e. PowerBI, but there are others)
- Data validation by a technical expert
- Publish final data to a web-based browser with a shareable link
Storing data on a cloud-based server and accessing the data via BI tools enables the complete database to be shared with a link, viewed on a web browser, and easily downloaded. I can share 40 years of water levels, water quality, and weather data with a web link. BI tools give us the power to quickly graph, filter, and perform functions on a dataset with a few clicks. To be clear, the cloud cannot do everything for us; data automation and cloud services are not a substitute for a knowledgeable technical expert. It is essential that data validation is done by a technical expert near the data.
Harnessing available technology to modernize and automate data management reduces the consultant hours required to convert raw data into usable data. Data cultivation is more than cloud computing, it is a hybrid of technology, data science, and technical expertise that has emerged as a new discipline. Please join us in navigating the digital maze together, we know the way.
View the original article on LinkedIn
