Intelligent Intelligence Gathering by Mining Data on the Internet

Recorded Future can predict at least some aspects of the future by monitoring the Internet. Lots of the Internet.

The two-year old Massachusetts-based firm which is partly funded by Google  Ventures and the CIA’s VC arm (In-Q-Tel, which makes investments to benefit the United States intelligence community)  thinks this aggregated and analyzed Internet information will be especially useful in three areas — government, finance and competitive intelligence.

But will it tell a trader which stocks to buy? Actually, yes. By collecting information and sentiment, it can pick the 50 most popular and the 50 least popular, said  Christopher Ahlberg, the company’s founder and CEO. By going long on the former and short on the latter, traders can make money.

“Organizing the news and companies by decile, you can prove the top decile significantly outperforms the bottom. This can turn out to be a very nice trading strategy, either as a standalone factor or incorporating the data in an overlay to an existing model.” From May to August the Recorded Future strategy had a 10 percent gain against a 10 percent loss for the S&P 500.

Ahlberg founded Spotfire, a company which turned complex data into analytic dashboards for fast action and collaboration. He sold it to Tibco. Recorded Future is a step further into the depths of information. It takes 300,000 Web pages per hour from 40,000 to 50,000 Internet sources and digests the information to create a database that can answer questions like “Which heads of state visited Libya in 2010?” or “What pharma companies are releasing new products in the first quarter of 2012?”

The company repeatedly says it is not a search engine, although I think it may supplement or replace search engines for time-pressed users who want specific information without spending an hour or two wading through the results from Google or Bing. At $149 a month for individual users, it could pay for itself pretty quickly in consulting and research.

As Steffan Truve, co-founder and chief scientist at the company said in avirtual users group conference recently, the company faces questions about what it offers in social media, media and government reports that a user can’t get from Google.

Try asking Google what is happening in Stockholm next week, he suggested. Google does not have a semantic understanding of information. Recorded Future works with structure such as people, places, products and companies; events such as meetings, travels, acquisitions, earning calls and natural disasters; and ontologies or hierarchies that explain groupings such as world leaders, corporations or technology areas. And the company’s analysis treats time in several ways — when an event was reported and when something occurred or is expected to occur. It also measures momentum or media buzz.

An example for competitive analysis: “What companies are working on fuel cell products expected between 2012 and 2015?

The query brings up a moving chart of companies, places and products over a timeline extending to the end of December 2014. Sliders for time and size of the graph give you control over the view, and the user can explore Toyota, see mentions of its fuel cell plans out to 2015, go back and see mentions of the Prius, each with the source indicated. A network view shows which companies are connected, so it displays the links between Ballard Power Systems andHonda Motor, Shanghai Automotive and Roewe 550 and information on the Chosun lblo fuel cell technology.   A source view produces blocks showing major sources of the news — Fast Company, the New York Times, Engadget, Techcrunch, auto blogs and PR Newswire, and where the sources are located — mostly in the U.S. but also the UK, Spain, Romania, Canada and China. A third view shows momentum by recording which firms are getting the most mentions going out to midway through 2015.

“A new report finds that Daimler and Honda are the automakers best positioned for the commercialization of fuel cell vehicles (FCVs) beginning in the 2014/2015 timeframe.” That report is from Pike Research.

It’s easy to see how useful this would be to government analysts. In a recent New York Times story, Quentin Hardy sat with Ahlberg for a demo. Among the topics they looked at — relationships among individuals in the Chinese Communist Party who seem to have unusually strong ties and an Iranian barter company that developed relations in Belarus as sanctions increased

Hardy wrote that Recorded Future  plans to move further into Chinese and Arabic news sources.

And this is just using public information. If the company is backed by the intelligence community, one would assume — or from a taxpayer’s point of view, hope — that the NSA is using this on the tons of material it gathers from private email, land lines and mobile phones. Expansion of Recorded Future into Chinese and Arabic would suggest an intelligence interest. Ahlberg told the New York Times that one client wants to see Chinese coverage of events that is longer in the Chinese version than in the English publication. That reminds me of Thomas Friedman who has often said that the West should pay attention to what Arab leader say in Arabic, which is often quite different from what they say in English. (One can only speculate about what Chinese and Arab intelligence agencies are reporting to their leaders about the contenders for the Republican presidential nomination. Just because it is public doesn’t make it, or them, intelligible.) 

To think how an intelligence analyst would approach this sort of work without a broad quantitative gathering of information suggests what a big step this is. Remember Kremlinologists who tried to divine the hierarchy by who was standing next to whom in reviewing parades? Or think of a China watcher trying to peer into the opaque organization of the Communist Party from a perch in the American embassy in Beijing or CIA offices in Langley. What snippets of information would she have and how would she hope to stitch them together, barring an inside information source who could be trusted, and the record indicates the U.S. has developed very few of those in the history of the USSR and China.

Longer term, Mr. Ahlberg says, “Why not process the world’s 100 million blogs as well?” These could be graded and ranked for reliability, so interested people would know which experts to read.

The software can find  signs of Nokia relationships with Microsoft in the months before it entered a legal agreement to use the Microsoft Windows mobile operating system. (Although, to be fair, some people predicted this when Nokia hired CEO Stephen Elop, who had been a Microsoft executive.) Ahlberg said that the company can look at business relationships like Microsoft and Nokia and surround the key players with products in its reporting.

“The primary applications we are in are intelligence, government, defense and trading but we have early users in competitive intelligence.”

Truve said Recorded Future relies on large numbers which helps achieve a good prediction even from quite noisy data, and it also assigns a confidence level, which can change over time, to the sources it uses.

Steve Holden, Recorded Future’s community manager, said the in an areas like competitive analysis a user could learn about Zynga’s business relationships over the last two years in a few minutes. Or Recorded Future can map the evolving relationships of Chinese oil companies to see when investments have taken place.

Steve Schohn, federal project engineer at Recorded Future, said it can provide government users with some situational awareness, such as what is going to happen in Baghdad next week. Many of the analysts use the company’s API to plug the data into their own applications — Google Earth maps are one favorite — to get a view of what is happening or likely to happen. That way reports of protests or riots can be turned into visualization over time. (See Occupy Wall Street timeline for an example on the company web site.)

The Recorded Future has notes on sources – an interesting one was TV of Iran which apparently found the protests in the U.S. of great interest for its Arab audience.

Ahlberg said about 10 companies are doing interesting work in this area. He keeps track of them with Recorded Future.

Talent Deficit to Handle and Generate Value from Big Data

“Now you see Big Data stories almost everyday. The pace of the development is amazing.”

Education hasn’t kept up with development, said Deng.

“We are facing a huge deficit in people to not only handle big data, but more importantly to have the knowledge and skills to generate value from data — dealing with the non-stop tsunami. 

How do you aggregate and filter data, how do you present the data, how do you analyze them to gain insights, how do you use the insights to aid decision-making, and then how do you integrate this from an industry point of view into your business process? The whole thing is hugely important for the future.”

Deng’s College of Computing and Informatics is already offering Master’s and Ph.D. programs in Bioinformatics. Partnering with the College of Health and Human Services and the University Graduate School, they have developed the first Health Informatics Professional Science Master’s program in North Carolina. “This degree is to train people with solid skills of technology and analytics while being expert in the business of health and healthcare, and making the two working together.”

Now, working with the Belk College of Business at UNCC, the College of Computing is developing a new Professional Science Master’s degree in Business Analytics and Informatics, integrating big data and analytics with business process and management concepts. Their industry advisory group has attracted experts from Bank of America, IBM, SAS, Cisco, McKinsey, Lowes and Ernst & Young.

Because the university is still growing – the enrollment at Deng’s college has increased by 60% in four years – it is offered an opportunity to introduce new programs without ithreatening existing faculty.

“Our faculty are quite receptive to this idea and help to drive these developments,” said Deng.

They realize the need for an interdisciplinary approach to the integration of technology with vertical education in areas such as finance, banking, healthcare, medicine, energy or defense.

“I expect that in two or three years we will have a set of collaborative and interdisciplinary degree programs focused on data and how it integrates with business processes and key industry sectors.”

The university is also working on executive education programs to improve the skills of employees who have to keep up with fast moving changes in their business. The McKinsey study estimated that the US needs to retrain 1.5 million managers who can understand the value of data and know what questions to ask.

Big Data -- Big Money Says It Is A Paradigm Buster

Big Data is attracting big money — $100 million at Accel Partners, the VC firm.

“Big data is one of the biggest transformational changes in the data center and IT landscape,” said Ping Li, a partner at the VC firm Accel Partners, which is running a $100 million Big Data fund. “It happens once in a generation,” he told the audience at a Churchill Club panel in Silicon Valley.

Gartner predicts that data will grow 800 percent over the next five years and 80 percent of the data will be unstructured.

And just what constitutes Big Data? After SC2011, the US supercomputing conference in Seattle in November, Addison Snell, an industry analyst, blurred the lines a bit in a podcast with HPCwire.

“There is small Big Data just as there is entry level high performance computing,” he said. Someone who has worked in gigabytes and now has to work in terabytes is dealing with Big Data, added the CEO of Intersect360. “It’s relative to the infrastructure you had before.” It may incorporate complex event processing, data mining and complex real-time analytics. Big Data can have many elements — large files, large volumes and real-time I/O within a short data life span. Every vendor at SC2011 was talking about big data, agreed Nicole Hemsoth, editor of Datanami.

Or to put Snell’s observation another way, Big Data breaks existing systems and ways of working.

“A lot of people know how to work with data,” observed Anand Rajaraman, “but now there is a lot more data so the kinds of things you can do with it and the way you work with it can are very different. The founder of companies which have been acquired by Amazon and Walmart, Rajaraman is now senior vice president at Walmart Global e-commerce and co-founder @WalmartLabs, and a professor at Stanford.

“The tools [for Big Data] are very different. Many of the fundamental algorithms for predictive analytics depend crucially on keeping the data in main memory with a single CPU to access it. Big Data breaks that condition. The data can’t all be in memory at the same time, so it needs to be processed in a distributed fashion. That requires a new programming model.”

This can be hard for traditional data users to understand, He watches students attack Big Data problems by creating a sample, but that defeats the value of Big Data with all its potentially informative outliers.

Businesses are catching on to the promise of Big Data said Luke Lonergan, chief technology officer at Greenplum, an analytics company that was acquired by EMC.

“Every business is looking for ways to get tighter connection with its  customers, to improve prediction and and move them along a trajectory. We see a certain urgency around Big Data.”

Traditional users of Big Data — retail, telecom and intelligence — are already comfortable with it, said Lonergan. The next big set of users are in mobile-social, especially incorporating geo-location. Some areas have been underserved, such as health care, which he described as the third rail because it has been too hard and too slow. But now health care is experiencing fundamental change similar to what retail felt when customers came in armed with smartphones and had more information than sales people. Patients are starting to acquire more information and health care providers are developing more analytics.

Big Data is also going to change science, said Rajaraman.

“The way science is done has evolved over the centuries — observation, notebooks and then you come up with a theory. That was followed by algorithmic science. Now data analysis is the fourth paradigm for the new way science is being done ranging from earth science, to chemistry to biology and psychology.   Science is about more and more data.”

(For one example, see my story on D.E. Shaw’s biological research.)

Big Data is not an American fad — it is a global phenomenon, said Accel’s Li.

“It’s amazing how global this is. Adoption of Big Data is happening faster in some emerging markets where they don’t have to content with legacy databases.” Instead they are using open source tools with commodity servers and storage.

“I think we are in the very early days of transformation,” Li added. Accel is looking for companies working in data management and platforms.

“A lot of the applications that ride on top of these new data platforms have yet to be invented. Right now users are importing legacy business intelligence and ERP, putting  it on top and calling it Big Data. I think we will see new applications that are Big Data. We are just starting to see the seeds as people recognize the potential of the new platform. We will see startups that will build new applications which will break the existing platforms and start again.”

Rajaraman agreed.

“Big Data is at a stage where early innovators are out there.” This is like the early days of  e-commerce when pioneers had to figure out components like fraud and payments, he added.

“Each of those is now a big company. In Big Data, each of the new use cases will get commercialized and become a big company on its own. Data growth today is faster than Moore’s law and faster than bandwidth growth, so we need new ways of organizing data, new computing and algorithms because computational power is not keeping up.”

People underestimate the degree to which this can change their business, said Greenplum’s Lonergan.

“Those with data science teams begin to understand; others don’t see how much it can do. Rather than any single stovepipe, we are about predicting the future. Some expect they can buy an application to make Big Data happen, but it doesn’t work that way. Businesses should see an explosion of new activity around the explosion of Big Data.”

Tom GroenfeldtContributor

+ Follow on Forbes

show less

I write about finance and technology, which can range from supercomputers to iPhones and iPads, which have made inroads into enterprise computing in the last year. It's a fast-moving world with just a bit of hype, so figuring out what's real is a challenge, and spotting emerging trends is a feat that often comes from listening and deciphering themes amidst the chatter. For a change of pace I turn to photography -- I like shooting on the streets of New York, and when I am home in Wisconsin, I turn to our dogs for inspiration, as you can see on Facebook.

Big Data Provides a 360-Degree View of Banking Customers

Banks can use big data to gain a 360-degree view of customers, even as they lose the personal interaction of branch banking.

As banks close branches and lose some of their personal interaction with customers they will use big data to monitor customers from different angles such as social media, said Karthik Krishnamurthy, AVP in Cognizant’s data warehousing group. Big data lets users process large amounts of data very fast. He cited smart meters in utilities as an example of big data — the meters will generate large volumes of data that the utility has to process. Consumer finance will have something similar.

“Banks spend millions trying to understand how customers interact across multiple products and touchpoints. Now we are looking at world where you move from transaction to interaction — which pages do they spend time on, what accounts do they have with other banks? Big data can help with customer sentiments and give access to the customer across all the touchpoints the bank has with them.”

Financial firms are sometimes reluctant to look at big data because they associate it with the open source Hadoop, he added. One bank CIO told Krishnamurthy bringing Hadoop into the organization seemed at odds with his role of maintaining structure in IT.

Krishnamurthy thinks big data is in many ways a reversion to the old IT approach of building systems and then letting users figure out what to do with them, instead of the standard approach of the last 15 to 20 years of gathering user requirements first.

“”We are now going back to data sources, putting the solutions in place and then taking it to the users and giving them the opportunity to explore and come up with newer solutions.”

Krishnamurthy said this is still in an experimental mode at the enterprise although some banks have deployed big data at a department or line of business level.

“The real value will come when big data goes mainstream. Look at the tech industry — Yahoo, E*Trade, eBay and Intuit have brought big data into the mainstream. Now it is focused on analytical value, but when big data gets adopted within a bank, that’s when you really start seeing business value.

Big Data Drives ‘Loyalty Trifecta’ for Banks

With their rewards programs no longer fueled by interchange income, banks are looking to ‘big data’ and mobile capabilities to build customer loyalty.BYJAMES MAROUS
Feb 28, 2012  |  0 Comments

In a post-Durbin Amendment environment, financial institutions (FIs) are faced with the challenge of changing the foundation of rewards programs that were previously funded by interchange income from both credit and debit cards. With debit interchange funding gone, banks still need to continue to find ways to improve bank loyalty and drive the desired debit and credit transactional and balance behavior. In addition, financial institutions need to better leverage “big data” and the mobile shopping phenomenon in the hopes that they can replace some of the revenue lost as a result of Reg E and the Durbin Amendment.

Optimally, the future of rewards and loyalty will allow banks and credit unions to take advantage of the “Loyalty Trifecta,” my term for bringing together the benefits of 1) transactional insight, 2) targeted offers/communication as well as 3) mobile marketing and payments.

These topics will be discussed at the upcoming BAI Payments Connect 2012 Conference & Expo in a panel discussion that I will moderate entitled “Rewards in a Mobile Banking Environment” with participation from Tom Beecher, CEO, Cartera Commerce Inc.; Rob Heiser, President and CEO, Segmint; Schwark Satyavolu, CEO, Truaxis; and Rod Witmond, senior vice president, Product Management & Marketing, Cardlytics Inc.

Prior to the conference, I posed the following questions to my co-panelists to assess their preliminary stance on these issues:

Q: What’s the current status of the banking rewards environment today and how can it be improved upon?

Beecher: The scope and strategies for banking rewards have changed dramatically in the past two years. Durbin has forced banks to re-imagine how loyalty programs are designed and funded. Also, the development of card-linked offers – where consumers earn cashback or points when using their bank’s payment card at participating merchants – has opened up new incremental revenue opportunities for banks. Finally, the growth of Groupon and deals in general has made consumers (and banks) much more aware of the power and importance of local merchants and online offers.

Witmond: Previously, U.S. banks brought offers to customers in a separate section of the bank website – often referred to as an “online mall.” Only a small percentage of their customers went there. It was not a loyalty solution. Various bank rewards solutions required the customer to enroll their card at a separate site and then hope they remembered to shop at a group of merchants providing lackluster discounts. Low engagement or difficult-to-use approaches won’t strengthen a retailer’s relationship with customers or move the needle on sales – for the merchant or the bank.

The banks’ business cases for the early generation, merchant-funded rewards programs promised significant earnings to the banks driven by large revenue shares. For the reasons stated above, retailers did not see these solutions as adding value to their current marketing mix and budgets did not shift. U.S. banks ended up with a big piece of a very small pie. New enhancements from loyalty vendors have refined the early approaches on several fronts.

Satyavolu: Most banking rewards in the past had four defining aspects: 1) they were mostly available on credit cards and less frequently on debit cards (due to being funded by interchange from merchants); 2) they were mostly one-size-fits-all (everybody gets the same extra points/cash-back on certain categories whether or not you shop there); 3) they were typically limited to cash-back or points back benefits; and 4) merchants were not involved in the creation of these benefits.

Heiser: The way FIs interact, engage and communicate is driven more and more by their customers’ technological lifestyles. While merchant-funded reward programs were one of the first to react to this shift, success today involves the application and technology adoption that is driven by transaction intellect − knowing and understanding the needs of customers.

Q: How can your own solution be leveraged in a mobile environment as opposed to an online banking or bricks and mortar environment?

Witmond: The Cardlytics solution is already leveraged in a mobile environment. We have bank solutions for SMS, mobile, and email in the marketplace. Additionally, we have ATM and social media solutions close to deployment. Most banks start with online banking because it provides the greatest exposure to the rewards platform. However, they quickly recognize the value of extending into mobile applications where they have complete control over the data and data fields. As such, they can drive mobile solutions at their own speed. Where a bank cannot deploy a mobile solution quickly, we offer a white-label mobile solution that can be deployed alongside or within an existing FI application.

Beecher: Mobile is an increasingly important channel for communicating with consumers -- particularly with the growth of in-store (national and local) offers. Cartera powers mobile apps that show consumers where they can use their payment card to redeem card-linked offers from nearby merchants. As Cartera partners roll out support for mobile wallets, this capability will become even more powerful by enabling consumers to find and redeem offers entirely via their smartphone.

Satyavolu: Truaxis’s StatementRewards product easily integrates with a FI’s existing mobile banking app to provide additional benefits to banking customers. Through the existing mobile app, bank customers will be able to view all of their rewards, both purchased and available, via the user dashboard. From this user dashboard, customers can instantly view, purchase and redeem rewards directly while they’re on the go.

Heiser: Segmint is not a merchant-funded rewards provider and, as such, our philosophy is grounded on generating loyalty through digital engagement with customers. Segmint is device-agnostic and can deliver across virtually any electronic medium. There is no doubt that opportunities exist within the mobile environment, but as with all mediums/channels, success revolves around the actual content delivery.

Q: What innovation do you see on the horizon around loyalty and reward platforms, both in banking and non-banking industries, in terms of leveraging social media?

Witmond: We have banks that have already designed how our solution can extend into social media and are deploying the same. The challenge with social media is that it is a “social experience” all about engaging on a person-to-person basis. That being the case, the extension of the core platform into social is only the first stage and the true challenge is in making the rewards solution one that engages on a person-to-person basis.

Beecher: Innovations in payments, big-data-driven marketing, and loyalty are all merging together to form what will ultimately be a new playbook for companies in these spaces and a new set of winners, including the new card-linked offers space. Mobile payments are seeing new non-banking entrants, all realizing that the incorporation of offers into the wallet is central to consumer adoption.

One of the new frontiers of leveraging big data with marketing is anonymous payment data, where new technologies and entrants are helping banks use transaction data that preserves privacy and provides real benefits to consumers. An example would be my purchase at McDonald’s alerting Burger King to make an offer to me. The entire funding model for bank loyalty programs is being turned on its head with merchants paying consumers through banks to shop with them rather than banks focused on taking money from merchants (through interchange) and then funding rewards themselves.

Satyavolu: The biggest innovation for these platforms will be the continued use of data to drive personalization and cut-costs. Both banking and non-banking industries are sitting on piles of data that they both don’t have the resources to utilize and if they did, they wouldn’t know where to begin. By working with third-party vendors like Truaxis, these companies will finally be able to utilize this data through innovative new techniques.

Analyzing transaction data from FIs is only the tip of the iceberg. As these platforms become more integrated across multiple channels and industries, companies will be able to understand and connect with their customers to provide them with the most value and ensure that each customer has a completely personalized experience that provides them with exactly what they need and want.

The data buried in social networks adds an interesting new twist to the personalization capabilities that are made possible, when you add them to the transaction data streams that FIs already have today. The concept of loyalty marketing will undergo a quantum shift in how it operates and who is in the key enabler seat for merchants, where FIs have a huge opportunity and upside to facilitate these interactions.

Heiser: Social media is a huge game changer for FIs and will become the “biggest bank branch” they operate. With nearly a billion active monthly users on Facebook, FIs must become socially actionable and interact with customers in their channel of choice. Last year Segmint introduced SegmintSocial, our social media technology solution that gives FIs the power to precisely identify their customers on the bank’s Facebook page, customize their experience and engage them in real-time, personalized dialogue.

Mr. Marous is senior director, marketing services, at San Antonio, Tex.-basedHarland Clarke Corp., and authors the Bank Marketing Strategy blog atjimmarous.blogspot.com. He can be reached at james.marous@harlandclarke.com.

RELATED ARTICLES

1. Three Keys to Success in Mobile Banking

2. Mobile Primer: What Does it All Mean?

3. Going Slow in Mobile is Going Prudently

Stay connected to Expert Perspectives, Research and Intelligence — subscribe to BAI Banking Strategies now!